Compare commits

...
Author SHA1 Message Date
Garry TanandMastra Code c07c8e8930 fix(test): repair typecheck failures in admin-sse and lifecycle tests (#3598, #3599)
Export AdminSseResponse, HttpServerLifecycle, and SignalSource from
serve-http.ts so test fakes can reference them. Cast structural fakes
through `as unknown as T` where the fake return types (EventEmitter,
plain object) cannot structurally match the full Node/Express originals.

Co-Authored-By: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
2026-07-29 16:39:13 -07:00
daragao3andClaude 945fed6105 fix(engine): enforce static engine-live import boundaries (#3596)
* docs: design engine dynamic-import reconciliation

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): reconcile dynamic import hardening

Co-Authored-By: Claude <noreply@anthropic.com>

* test(engine): guard dynamic import policy

* docs: plan engine dynamic-import reconciliation

Record the approved TDD sequence for selective engine-path hardening,
repository guard wiring, documentation, and local verification. Preserve
the no-version-bump and no-publication boundaries for the remaining work.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(engine): record static import invariant

* fix(engine): parse block comments in import guard

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): parse dynamic imports with TypeScript

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close import guard bypasses

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close parser guard edge cases

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): aggregate parser diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): bound dynamic import marker directive

Require the line-level opt-out marker to be standalone inside real comment trivia so negated or incidental longer tokens cannot authorize an import. Preserve the existing general marked-line contract and pin it with focused regression coverage.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close Unicode marker boundary bypasses

Treat Unicode identifier continuations as marker-token characters and inspect adjacent text by code point so supplementary-plane characters cannot turn longer comment tokens into approvals.\n\nCo-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-29 16:08:08 -07:00
Mikhail Merkulov a175dd0047 fix admin SSE handshake through reverse proxies (#3598) 2026-07-29 16:07:59 -07:00
Mikhail Merkulov 2118f02fc7 fix HTTP server lifecycle retention (#3599) 2026-07-29 16:07:50 -07:00
Sean Gearin a12ab5eabc fix(import): quote frontmatter values and omit absent conversation id (#3600)
Review follow-up to #3549.

An envelope is a third-party file, so interpolating source_provider raw let a
provider string carrying a newline close the scalar and inject arbitrary
frontmatter keys. title: on the line above was already quoted; source: now
matches it.

memvelope_conversation_id emitted the literal string undefined when a
conversation carried no id, which asserts a value rather than reporting
absence: every id-less conversation claims the same id, so anything grouping
or deduping on that key merges unrelated pages. The key is now omitted.

Filename and frontmatter read one hasId predicate so they cannot disagree
about whether an id exists. Tests add both cases; the injection case parses
emitted frontmatter with js-yaml rather than substring-matching it.
2026-07-29 16:07:42 -07:00
Tony Guan e98249a624 fix(files): display zero-byte file sizes (#3608) 2026-07-29 16:07:32 -07:00
Sean Gearin 1057bf4368 feat(import): standalone importer seeding a brain from envelope-v0 chat-history files (#3549)
One Markdown page per conversation from an envelope-v0 file (format spec:
github.com/memvelope/memvelope), written into a directory gbrain sync
ingests. Zero dependencies, deterministic, no network; does not call gbrain.

Filenames are date + conversation id (collision-proof natural key; duplicate
ids overwrite their own file and warn on stderr). Frontmatter carries
type: conversation, source provider, conversation id, and origin. Bodies keep
message-id citations per speaker turn.

Ships as script + test + fixture only; usage and verification steps live in
the script header.
2026-07-29 11:56:20 -07:00
913d2d7f79 fix(test): give slow setup hooks a real timeout budget (#3566)
bun ignores bunfig.toml's timeout key, and beforeAll/beforeEach hooks do
NOT inherit a test's third-arg timeout — a bare `bun test` gives every
hook the 5000ms default even when all tests in the file declare 30s+.
Measured on bun 1.3.14: a 6s hook dies at ~5001ms with the signature
`(unnamed) [5001ms] ... hook timed out` (the #3545 jsonb-parity CI
failure); both `beforeAll(fn, ms)` and the CLI `--timeout` flag are
enforced hook budgets (kills observed at exactly the configured ms).

Fixes:
- e2e.yml (jsonb-parity, tier1, tier2) and release.yml ran bare
  `bun test`; they now pass --timeout=60000 like every scripts/ runner.
- test/e2e/jsonb-roundtrip.test.ts (the #2339 double-encode guard, which
  only real Postgres can surface) additionally carries per-hook 60s
  budgets so a bare local run can't flake either — same pattern as its
  sibling op-checkpoint-jsonb-parity.test.ts.
- scripts/check-bun-test-timeout.sh: CI guard (run from test.yml's
  verify job) failing any future bare `bun test` in workflows/scripts.
- scripts/run-e2e.sh: correct the comment claiming --timeout is
  per-test-only (it covers hooks; the outer gtimeout exists for
  sync-blocking WASM hangs where no timer can fire).

Proof: with Postgres paused for 6s during setupDB's connect, the
unfixed file fails at 5001.81ms with the exact CI signature; the fixed
file passes the identical condition (5 pass, 6.57s). 396 slow
before-hooks across 362 test files lack per-hook budgets; all of them
run through --timeout-passing invocations after this change, enforced
by the new guard.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 20:08:36 -07:00
f9349ba07f fix(doctor,cycle): stop permanent cycle_freshness FAILs on multi-source installs (#2540) (#3562)
Closes the paths #3382 left open (its author said it narrowed the issue
rather than closing it):

1. checkCycleFreshness iterates EVERY local_path source, so an install
   that nightly-dreams one vault via --dir showed a permanent FAIL for
   every other federated source — and for any source added minutes ago.
   'Never completed a full cycle' is now a WARN with the dream/autopilot
   hint; a source that HAS cycled and then went stale still escalates
   through the 6h warn / 24h fail thresholds (the regression signal the
   check exists for). This is the reporter's actual case: the permanent
   red eroded doctor's signal until real staleness hid inside it.

2. resolveSourceForDir's exact-match lookup had no archived filter and
   no ORDER BY, so an archived (or duplicate) alias of the same path
   could shadow the active source; dream's archived guard then refused
   the stamp and the ACTIVE source stayed unstamped forever. The lookup
   now excludes archived rows and orders deterministically, matching
   the canonical-path fallback's posture. The fallback's fail-closed
   ambiguity handling is deliberately unchanged.

3. #3382's own regression test (ii) was environment-sensitive: it
   assumed unsetting OPENAI_API_KEY/ANTHROPIC_API_KEY makes the embed
   phase fail, which is false wherever another embedding provider
   resolves (the cycle then reports 'clean' and the test flips). It now
   fails the sync phase against a vanished checkout — deterministic on
   every machine, same property pinned (a genuinely failing enabled
   phase must prevent the stamp).

New pins fail on unmodified master and pass here: never-cycled→warn
(x2, doctor) and the archived-alias shadow (dream --dir stamp).

Fixes #2540

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:50:44 -07:00
MasaandTime Attakc e72d93fdb5 fix(sync): reconcile the stale old row when a rename falls back to add (#3056) (#3479)
master's rename loop swallows updateSlug failures with an empty catch
("treat as add"), and updateSlug returns void — so a zero-row UPDATE
(old slug absent) and a thrown collision are both invisible. Either way
the run falls through to importFile at the new path while the old row
stays behind live: slug occupied, 0 chunks after the next embed pass,
page count unchanged. A rename that didn't rename, with no trace.

The fix reconciles the duplicate:

- updateSlug returns the number of rows moved in both engines (a
  zero-row UPDATE does not throw; the count is the only way to see it).
- When the cheap rename didn't move a row AND the destination
  demonstrably materialized — imported, or an errorless skip AT the new
  slug (NOT an identity-dedup skip against the old row, which would mean
  nothing landed and deleting the old row would destroy the only copy) —
  the stale row is located positively by source_path = from and deleted.
  No source_path match → nothing is deleted (code-strategy imports don't
  populate source_path and fall back safely to leaving the row).
- A failed reconcile delete records a <rename:…> sentinel: the failure
  gate hard-blocks the bookmark, the auto-skip valve can never
  chronic-skip it (which would bank the duplicate permanently after a
  multi-run outage), and the rename is not checkpointed — the next run
  retries the same diff and clears the sentinel on convergence.

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 19:32:55 -07:00
Time AttakcandGarry Tan 85286a556c fix(search): stop boosting compiled_truth at default detail (#3430) (#3514)
* fix(search): stop boosting compiled_truth at default detail (#3430)

COMPILED_TRUTH_BOOST = 2.0 is applied AFTER RRF normalization, and RRF's whole
dynamic range over a 100-deep pool is 1/60 -> 1/160 (a factor of 2.67). So a
2.0x multiplier consumes roughly three quarters of the range: break-even is
`2/(60+r) >= 1/60`, i.e. r <= 60, which means ANY boosted chunk inside the
first 60 ranks outranks an unboosted rank-1 chunk. That is a categorical
filter, not a tilt.

Measured against master's own rrfFusion, with the correct answer in a
fenced_code chunk at vector rank 0:

  compiled_truth chunks in pool | final rank | in top-20
  10                            | 10         | yes
  20                            | 20         | NO
  40                            | 40         | NO
  80                            | 59         | NO

With the boost off the answer stays at rank 0 in every case.

The gate was spelled `detail !== 'high'` -- written as though `high` were the
special case. The documented contract in src/core/operations.ts is
"low (compiled truth only), medium (default, all with dedup), high (all
chunks)", which makes LOW the special one: `low` already restricts to
compiled_truth, so a boost there is a no-op among equals, while `medium` and
`high` are both meant to see everything. So the default detail was silently
compiled-truth-only, contradicting the op's own description.

Three changes:

1. The three fusion call sites now route through a named predicate,
   `shouldBoostCompiledTruth(detail)`, returning true only for 'low'.
   Extracted rather than left inline precisely because an inline expression is
   only reachable through a full hybridSearch round trip -- which is why the
   inversion went unnoticed. The predicate is directly unit-testable.

2. KNOBS_HASH_VERSION 13 -> 14. Results are cached AFTER fusion, so rows
   ranked under the old semantics would otherwise be served under the new ones
   for the whole TTL (3600s default). One-time miss spike on upgrade.

3. test/search-compiled-truth-boost-scope.test.ts pins both the mapping and
   the arithmetic, and documents the displacement it prevents.

Verified the tests discriminate: stubbing the OLD predicate body into master
(so the failure is behavioral rather than a missing export) gives 4 fail /
3 pass; with the fix, 7 pass. typecheck clean, verify 32/32, and 144 pass /
0 fail across the search + fusion + cache suites.

* fix(test): update the three remaining KNOBS_HASH_VERSION pins to 14 (#3430)

Missed in the first pass because I ran a targeted set of test files instead of
the full suite. CI shards 3, 8 and 10 caught them:

  test/search/knobs-hash-reranker.test.ts:67
  test/cross-modal-phase1.test.ts:139,149
  test/search-alias-resolved-boost.test.ts:93

Each carries the running history of why the version moved, so each gets the
13→14 rationale appended rather than just the number swapped. No pins at 13
remain anywhere in test/.

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-28 19:27:46 -07:00
a8a3b6df9f fix(engine): exclude soft-deleted pages from getHealth counts (#1305) (#3556)
getStats() has excluded soft-deleted pages since v0.26.5, but getHealth()
kept counting raw pages rows: page_count, the islanded/orphan scan, the
entity_pages CTE (link/timeline coverage denominators), and most_connected
all included deleted pages, so brain_score never moved when a user
soft-deleted pages. Repro: 50 pages, soft-delete 40 -> getStats 10 vs
getHealth 50, orphan_pages 50, brain_score byte-identical.

Fix: every page-scoped count in getHealth now filters deleted_at IS NULL,
identically in both engines (engine-parity SQL shapes match).

Deliberate boundary: chunk/link storage counts (embed_coverage,
missing_embeddings, link_count, dead_links) stay raw until the purge phase
runs — matching getStats' documented posture — and destructive-removal
counts (#2235) deliberately keep counting all rows. stale_pages already
filtered via buildStalePagesWhere.

Test: test/health-soft-delete.test.ts — 3 of 4 tests fail behaviorally on
unmodified master (page_count 10 vs 4, orphan_pages 8 vs 0, link_coverage
0.5 vs 1), all pass with the fix.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:14:39 -07:00
3aa064bcc6 fix(test): make resetGateway restore the test baseline instead of unconfiguring (#3554) (#3557)
* fix(test): make resetGateway restore the test baseline instead of unconfiguring (#3554)

bunfig.toml's legacy-embedding-preload pins the gateway once at process
start to openai:text-embedding-3-large @ 1536, but resetGateway() wiped
that pin to _config = null. The next test file's beforeAll engine-connect
then reconfigured from the SHIPPED default (zembed-1 @ 1280) before the
preload's per-test beforeEach could restore anything, and every 1536-d
fixture in that file failed with `expected 1280 dimensions, not 1536`.
Which file pairs collided depended on shard bin-packing, so adding ANY
test file reshuffled the mines (this is what blocks #3545).

Fix: the preload registers its config as a reset baseline via a new
test-only seam (__setGatewayResetBaselineForTests); resetGateway() clears
all module state as before, then re-applies the baseline. All 93 existing
resetGateway() call sites get the correct behavior with zero edits.
Production is untouched: nothing in src/ calls resetGateway() or the
setter, so the baseline is never registered outside tests and
resetGateway() still fully unconfigures there.

Five tests genuinely need an unconfigured gateway (no_gateway_config
diagnosis, isAvailable=false, the #2590 cold-gateway path, the registry
builtin-default tier); they switch to the new __unconfigureGatewayForTests.
Two files' hand-rolled "restore the legacy pin in afterAll/finally"
workarounds for this exact bug are now redundant and simplified away.

Guard test (test/ai/gateway-reset-baseline.test.ts) pins the contract:
1536/openai immediately after resetGateway(), transports still cleared,
hard-unconfigure still available.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(test): restore preload's GBRAIN_AUDIT_DIR instead of deleting it (#3554 sibling)

Same bug class as the gateway fix in this PR: state set once by a bunfig
preload (audit-dir-preload's scratch GBRAIN_AUDIT_DIR), wiped by one
file's cleanup, blast radius decided by shard bin-packing. In shard 6,
test/minions-shell.test.ts (position 12) unconditionally deleted the var
in afterAll; test/audit/audit-dir-preload.test.ts (position 93) then
found it undefined and failed 3 tests — and every file in between wrote
audit fixtures toward the operator's real ~/.gbrain/audit/.

Fix: capture the prior value at file load and conditionally restore it,
the same inline save/restore pattern 11 sibling files already use.
test/e2e/skill-brain-first.test.ts had the identical unconditional
delete in afterEach; fixed the same way. Sweep of every
`delete process.env.GBRAIN_AUDIT_DIR` in test/ confirms all remaining
sites are conditional restores.

Ordered-pair proof (minions-shell.test.ts then
audit/audit-dir-preload.test.ts, one process): 3 fail on master,
43/43 pass with this fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 17:08:27 -07:00
6136e13997 fix(cli): stop parseOpArgs hanging on a non-TTY stdin with no input (#3513) (#3546)
* fix(cli): stop parseOpArgs hanging on a non-TTY stdin with no input (#3513)

parseOpArgs read stdin for stdin-capable ops via readFileSync(0),
assuming non-TTY implies piped content. In a non-TTY with no piped
input — a CI step, a cron job, an agent harness that inherits a non-TTY
stdin without ever writing to it — that call never returns.

The stdin fill moves out of parseOpArgs into an async applyStdinParam
with a bounded read (readStdinBounded), called by the op dispatch right
after arg parsing:

- TTY: skipped, as before.
- Regular file / /dev/null (fstat says not a pipe/socket): readFileSync
  returns without blocking — `gbrain put x < file` and `< /dev/null`
  behave exactly as before (empty-but-readable still yields '').
- FIFO/socket: stream-read with a deadline on the FIRST byte only
  (default 5000ms, GBRAIN_STDIN_TIMEOUT_MS overrides). Real pipes
  (`echo foo | gbrain put x`, heredocs) deliver their first byte in
  milliseconds; once any data arrives the deadline lifts and the read
  drains to EOF, so slow producers keep working. An empty pipe that
  closes yields ''. A pipe that never delivers a byte times out, the
  param stays unset, and the existing required-param usage error fails
  fast with exit 1.

Regression tests spawn the real CLI with a held-open, never-written
pipe (hangs 20s+ on pre-fix code; exits ~1s fixed) plus parity cases
for data pipes, /dev/null, and empty closed pipes, and a subprocess
driver pinning content preservation through applyStdinParam.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(build): restore the executable bit on src/cli.ts

check:cli-exec requires mode 100755; the edit in this branch landed it as
100644, failing `bun run verify` (1/32) on an otherwise-green PR. Mode only,
no content change.

* fix(cli): keep the R4-pinned stdin branch shape in applyStdinParam (#3513)

Shard 10's R4 regression pin (test/cycle/regression-pr-wave-r1-r2-r4.test.ts,
protecting PR #1325's Windows /dev/stdin → fd 0 fix) asserts three source
literals in src/cli.ts: `readFileSync(0, ...)`, the `op.cliHints?.stdin` +
`MAX_STDIN = 5_000_000` branch, and the `!process.stdin.isTTY` gate. The
bounded-read refactor kept the first two but inverted the TTY gate into a
positive early-return, dropping the pinned `!process.stdin.isTTY` spelling.

Restore the original branch shape inside applyStdinParam (guard + read +
cap + assign), unchanged semantics. The #1325 protection itself was never
at risk: no '/dev/stdin' anywhere, readFileSync(0) remains the read for
non-pipe stdin, and pipes drain through process.stdin (fd 0, cross-platform).
The pin now passes unmodified. A comment marks the shape as R4-pinned so
the next refactor doesn't trip it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 15:56:50 -07:00
b3b43d0f91 fix(sources): make the __all__ sentinel work in every resolution tier (#1712) (#3524)
The `__all__` sentinel had two spellings and only one worked: as a
per-call source_id param, resolveRequestedScope understood it; as
--source __all__ or GBRAIN_SOURCE=__all__, SOURCE_ID_RE (which forbids
underscores) made all three resolver entry points throw. makeContext's
blanket catch then silently fell back to sourceId 'default' — making
the documented span-everything flag STRICTLY NARROWER than passing no
flag at all, because the catch also discarded the #2561/#3242 federated
widening:

  unqualified read (federated brain)  -> {"sourceIds":["default","src-a","src-b"]}
  --source __all__ (pre-fix)          -> {"sourceId":"default"}
  --source __all__ (post-fix)         -> {}   (spans the brain)

Fix:
- src/core/source-id.ts: export ALL_SOURCES = '__all__'. SOURCE_ID_RE
  itself is NOT loosened — it still guards source creation, lock ids,
  and path joins, and its underscore rejection is what makes the
  sentinel collision-free.
- src/core/source-resolver.ts: the explicit and env tiers of
  resolveSourceId / resolveSourceIdEngineFree / resolveSourceWithTier
  pass the sentinel through verbatim (skipping the regex and
  assertSourceExists). Covers --source (#1712/#2289) and
  GBRAIN_SOURCE (#2140), local and thin-client alike.
- src/core/operations.ts: sourceScopeOpts — the single choke point every
  read-side scope helper delegates to — translates ctx.sourceId ===
  ALL_SOURCES into {} for trusted local callers (strictly remote ===
  false) and keeps the unsatisfiable literal for remote/untrusted
  callers, so the sentinel can never widen past a caller's grant
  (fail-closed). A federated grant still wins over the sentinel.
- src/cli.ts: makeContext's catch now rethrows when an explicit
  --source was passed — a source that genuinely fails to resolve errors
  loudly instead of silently becoming 'default' (the silent fallback is
  what turned three bug reports into debugging sessions).

Tests (test/all-sources-sentinel.test.ts) fail on unmodified master
(9/13, behaviorally) and pass with the fix; the 4 that pass on both
sides pin invariants that must hold on both (remote fail-closed
literal, grant precedence, invalid-id rejection).

Closes #1712. #2289 and #2140 were closed as duplicates of it.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:55:38 -07:00
5b9a87f1a3 fix(cli): make backfill command reachable — add to CLI_ONLY (#3224) (#3529)
'backfill' had a fully implemented handler (case 'backfill' dispatching
to commands/backfill.ts) but was missing from the CLI_ONLY set, so
dispatch rejected every invocation with 'Unknown command: backfill'.
Same drift class as #2900 (reconcile-links) and #2035 (calibration).

Also lists backfill in the main --help TOOLS section.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:34:28 -07:00
661f1f05cc fix(extract): make receipt shortRunId canonical under slugifySegment (#3443) (#3542)
shortRunId() truncated run ids to their first 8 chars, so propose_takes
run ids ('propose-<timestamp>-<uuid>') shortened to 'propose-' with a
trailing hyphen. slugifySegment() strips boundary hyphens during repo
sync, so the DB receipt slug and its Git-backed markdown slug disagreed
— writing the receipt through to the system-of-record repo created a
normalized sibling/collision instead of materializing the existing page.

shortRunId now trims boundary hyphens after truncation (invariant:
slugifySegment(shortRunId(x)) === shortRunId(x)), with a non-empty
fallback for pathological all-separator prefixes. All other run-id
families (atoms-, efacts-, concepts-, ecf-) are unchanged.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:12:22 -07:00
3fec2123d2 fix(takes): add 'list' subcommand — 'list' was parsed as a page slug (#2079) (#3540)
'gbrain takes list' printed 'No takes on list.' even when the brain held
many takes: the CLI had no list subcommand, so cmdList treated the word
'list' as a page slug and looked up a page named 'list'. The failure
read exactly like an empty takes table, so agents concluded there were
no takes and moved on.

'takes list' now lists all active takes (CLI parity with the takes_list
op), prefixing each row with its page slug. 'takes <slug>' unchanged.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:12:15 -07:00
176836f84d fix(embed): stamp the real model on chunk provenance and keep contextual prefixes on --stale (#3461, #3507) (#3538)
#3461 — insertChunks stamped DEFAULT_EMBEDDING_MODEL on chunk provenance
whenever the AI gateway was unconfigured: getEmbeddingModel() throws
rather than returning falsy, so the reland's '|| resolvedModel' guard
(e1919fab) was dead code and the catch path kept the compile-time
constant. Both engines now fall back to the brain's own
config.embedding_model row on the throw path, with the compiled default
as last resort. Same-line sibling: the ON CONFLICT clause overwrote
'model' via COALESCE even when the existing vector was preserved —
'model' now mirrors the 'embedding' CASE branch-for-branch so the label
always describes whichever vector wins the upsert. The initSchema
sizing sites drop their dead '||' terms too.

#3507 — every plain re-embed path (embed <slug>, --all, --stale, and
the embed-backfill Minion loop) embedded raw chunk_text, silently
stripping the contextual-retrieval prefixes the sync path applied —
and embed --stale is the NORMAL post-model-migration path. All four
sites now wrap through wrapChunkTextsForStoredMode(), reproducing the
page's STORED convention (pages.contextual_retrieval_mode):
title/per_chunk_synopsis pages get the title-tier prefix, fenced_code
chunks stay unwrapped (D20-T4), unstamped pages stay raw. A fully
re-embedded per_chunk_synopsis page is restamped to 'title' so the
mode column keeps describing the vectors actually in the DB.

Deliberately NOT folding the contextual mode into
currentEmbeddingSignature(): the convention is already recorded
per-page (mode + corpus_generation), pages legitimately differ
per-page so a global signature cannot represent it, and bumping
signature semantics would force a full-corpus re-embed on upgrade.
No KNOBS_HASH_VERSION change (no collision with #3514).

Tests fail on unmodified master (6) and pass with the fix; postgres.js
path verified against a real pgvector instance in addition to the
PGLite suite.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:12:08 -07:00
539d015cc5 fix(minions): stamp 10-min default timeout on facts-absorb jobs (#3207) (#3535)
facts-absorb performs one LLM extraction call per page — the same shape
as chronicle_extract — but was missing from HANDLER_DEFAULT_TIMEOUT_MS,
so it inherited the tight null-default wall-clock budget and was
dead-lettered mid-generation on slow chat providers. Nothing was
inserted; the page's facts were silently lost.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:12:01 -07:00
91464564cd fix(cycle): scope the extract_facts guard to its source and fix the drain advice (#2646, #3526) (#3528)
Defect A (#3526): the empty-fence guard COUNT had no source_id predicate,
so one pending legacy row in any mounted source jammed extract_facts for
every source in the brain. The count now binds f.source_id = $1 to the
run's sourceId (source-isolation invariant).

Defect B: the guard advised `gbrain apply-migrations --yes`, which is a
proven no-op once the v0.32.2 ledger entry is complete (the runner
classifies it as already-applied and Phase B never re-runs). The warning
(and cycle.ts's phase hint) now gives the drain path verified to work
end-to-end on a real brain: `apply-migrations --force-retry 0.32.2`
then `apply-migrations --yes` (Phase B is idempotent), or forget_fact
per row.

New test proves a pending legacy row in source A does not jam extraction
for source B (fails on master, passes with the fix).

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:11:54 -07:00
bd049d2969 fix(jobs): honor --dry-run on jobs prune instead of silently deleting (#2712) (#3525)
gbrain jobs prune --dry-run used to silently drop the flag and run the
destructive default — rows were really deleted while the operator
believed they were previewing.

MinionQueue.prune now takes dryRun: count the would-be-pruned rows
without deleting; the CLI parses --dry-run and labels the output.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:11:47 -07:00
784358f5fd fix(sync): preserve non-Latin scripts in slugs (#3417) (#3522)
slugifySegment stripped every character outside [a-z0-9._-] + four CJK
ranges, so filenames in Hebrew, Arabic, Cyrillic, Greek, Thai (and every
other script) collapsed to empty segments. Distinct files then mapped to
the SAME slug (their shared directory prefix) and silently overwrote each
other, last-writer-wins, with import reporting 0 errors.

All three slug grammars move together so sync never emits a slug that
put_page rejects:

- sync.ts SLUGIFY_KEEP_RE + SLUG_SEGMENT_PATTERN — now keep
  \p{Ll}\p{Lm}\p{Lo}\p{M}\p{N} (new single-source SLUG_WORD_CHARS in
  cjk.ts), with the u flag
- cjk.ts PAGE_SLUG_SEG — rebuilt on SLUG_WORD_CHARS; consumers
  (validatePageSlug, SlugRegistry SLUG_RE, dream-cycle SUMMARY_SLUG_RE,
  takes-fence HOLDER_REGEX) all gain the required u flag

CJK_SLUG_CHARS is untouched — it also drives the countCJKAwareWords
chunking density heuristic and must not move with slug grammar.

Kebab-casing, lowercasing, Latin accent-strip (cafe), dots/underscores,
path handling, and the NFC re-normalize (NFD macOS filenames converge
with NFC git filenames) are all unchanged and regression-pinned in
test/slug-unicode-scripts.test.ts. No data migration: the collapse was
N-to-1, so only one row per group ever persisted; a normal gbrain sync
recreates the lost pages under their real slugs.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:11:41 -07:00
d58bb2b0bb fix(check-update): resolve the latest version from VERSION, not the empty releases API (#486) (#3520)
The repo publishes zero GitHub releases, so releases/latest is a permanent
404 and fetchLatestRelease() could never succeed — the entire upgrade
notification subsystem was a silent no-op, and refreshUpdateCache() cached
a fabricated up_to_date marker on every failure.

- Resolve the latest version from raw.githubusercontent.com/.../master/VERSION
  (same trusted host fetchChangelog already uses). Bounded, shape-gated parse;
  handles legacy 3-segment and -suffix channel forms.
- Discriminate network_error from no_releases in the --json error field and
  human output.
- Never write up_to_date on a failed check: preserve the last-known-good
  marker (mtime bump keeps the TTL throttle) or write nothing.
- Rejected the issue's proposed npm fallback: the gbrain npm package is an
  unrelated GPU library (#505) and would produce false upgrade prompts.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 14:11:34 -07:00
b252acfce3 fix(test): correct the deadline inversion in the durability-hook serial tests (#2943) (#3537)
Three layers, verified by 10-run tallies (master: 7 pass/3 fail; branch: 10/10):

1. Deadline inversion: the hook tests' internal 8s poll deadlines sat behind
   bun's 5000ms default because no third-arg timeout was passed — and bun
   1.3.14 ignores bunfig.toml's `timeout` key, so bare `bun test` runs died
   at 5s with the 8s budget unreachable. All three tests now pass 60_000
   explicitly; deadlines raised to 30s for loaded-shard headroom. Same class
   fixed in test/e2e/jsonb-roundtrip.test.ts (four tests had no explicit
   timeout).

2. Root cause of the CI assertion failures: the test's git() helper spawned
   git WITHOUT `env: process.env`. Bun snapshots process.env at startup (the
   #2747 quirk), so the post-commit hook under test resolved GBRAIN_HOME to
   the operator's real ~/.gbrain — writing its log there (polluting the real
   brain-push.log every run) while the test polled the temp log. The
   LOCAL-ONLY assertion only passed when beforeEach's scaffolding hook push
   happened to still be in flight, lose the ref race, and retry after origin
   pointed at the dead path — a load-dependent accident. env is now passed
   everywhere, making the intended signal deterministic (~1s).

3. index.lock race (the third observed CI form): beforeEach now waits for the
   scaffolding commit's detached brain_push to write its terminal log line
   before handing the repo to the test, so its pull-rebase fallback can't
   take .git/index.lock under the test body's own git calls.

Poll loops untouched — they were already correct; the 150 is the poll interval.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 13:58:09 -07:00
cybernaut6404 6920744dd8 fix(ai): enable OpenRouter query expansion (#3499) 2026-07-28 11:56:06 -07:00
daragao3andClaude Opus 5 3df20f9f18 v0.42.67.0 fix(build): force LF for shell scripts and route package.json checks through bash (#3506)
* v0.42.67.0 fix(build): force LF for shell scripts and route package.json checks through bash

Two independent defects left `bun run test`, `verify`, `ci:local` and
`test:e2e` dead on Windows. All four dispatch through bash.

First, every tracked *.sh is checked out with CRLF. The committed blobs are
clean LF; system-level core.autocrlf=true rewrites them on checkout, and a
strict bash then dies at run-unit-parallel.sh line 23 with
"$'\r': command not found". A root .gitattributes pinning `*.sh text eol=lf`
overrides autocrlf regardless of the contributor's git config.

Second, 33 package.json scripts invoked `scripts/foo.sh` directly, which bun
cannot exec via shebang on Windows. They now go through `bash`, matching the
11 that already did; all 59 tracked *.sh files are bash-shebanged (52
`#!/usr/bin/env bash`, 7 `#!/bin/bash`), so the change is uniform. The five
`scripts/*.ts` entries still run under bun.

Measured on this base, `bun run verify` goes from pass=1 fail=31 to pass=25
fail=7. Every one of the baseline's 29 `command not found: scripts/...`
errors is gone; those were the shebang defect, and they account for the
measured delta.

The line-ending defect is verified structurally rather than by that number,
because the bash on PATH for this measurement tolerates CR and so cannot
exhibit it: under the new attribute all 59 tracked *.sh check out LF-only
(0/59 carry a CR byte, against 59/59 before), and `git add --renormalize .`
is a no-op, confirming the index was always correct and only the working
tree was wrong. Zero content churn.

All 7 residual failures also fail on the pristine baseline: four exceed the
harness's 120s cap (standalone `bun run typecheck` exits 0), and check:wasm,
check:skill-brain-first and check:resolver are pre-existing content or
environment issues. check:resolver is not even a shell script.

No behavior change on Linux or macOS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: sync docs to v0.42.67.0

CONTRIBUTING.md gains a Windows section: the `.gitattributes` LF pin makes a
fresh clone correct with no extra steps, working copies cloned earlier need a
one-time `git rm --cached -r . -q && git reset --hard`, and new shell-script
checks must be registered as `bash scripts/<name>.sh`.

docs/TESTING.md records the shell-dispatch convention alongside the command-tier
table, and notes that the table's wallclock figures are Mac numbers: on Windows
`check:privacy`, `check:test-names`, `check:test-isolation` and `typecheck` can
exceed run-verify-parallel.sh's 120s per-check cap while passing on Linux and
macOS. It also flags that the Cygwin bash shipped with Git for Windows tolerates
CRLF where a strict bash does not, so a green local run is not evidence that a
script is CRLF-clean.

CHANGELOG.md's itemized list covers both doc updates.

`bun run build:llms` regenerates byte-identical bundles: docs/TESTING.md is
linked rather than inlined, so llms.txt / llms-full.txt do not move.
`bun test test/build-llms.test.ts` passes 12/12.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 11:45:09 -07:00
cybernaut6404 e58abd652c fix(extract): preserve facts on parse warnings (#3494) 2026-07-28 11:44:26 -07:00
MasaandTime Attakc a104f98dca fix(cycle): extract_facts guard counts only active legacy rows; reconcile preserves forget records (#2646) (#3474)
master's empty-fence guard counts soft-expired legacy rows
(row_num IS NULL, expired_at set), so forget_fact — the sanctioned
removal path, which soft-expires rather than deletes — can never drain
the backlog: apply-migrations no-ops (already marked applied) and the
guard stays triggered forever, jamming extract_facts.

Narrowed re-send of #3252-sibling #3234, scoped to exactly what the
maintainer named reviewable: "one predicate plus the
reconcile-preservation guard."

- Guard predicate: the legacy COUNT adds `AND f.expired_at IS NULL`,
  so each forget_fact visibly drains the pending counter.
- Reconcile preservation: listExistingFactsForPage excludes soft-expired
  legacy rows (they are never fence-owned, so they must neither read as
  perpetually stale nor mask a fence row), and the two wipe call sites
  pass `preserveExpiredLegacy: true` so deleteFactsForPage keeps the
  forget record. The option is the minimal seam for that guard —
  implemented identically in both engines (~6 lines each).

Explicitly NOT included from #3234 (per the close review): the
drift-repair lane, the re-runnable migration orchestrator path, and the
race-accounting layer.

Fence-is-canonical semantics are preserved and pinned by test: if the
fence still carries an expired legacy row's claim, the reconcile
reinserts it as a fresh active fence-owned row (legacy DB-only forgets
are documented non-durable; the expired row survives as the audit
record of the forget).

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 11:44:17 -07:00
cybernaut6404 2ac6959b46 test(schema): exercise v121 bootstrap coverage (#3489) 2026-07-28 11:44:12 -07:00
paul-0320andTime Attakc 18ec732e1b fix(chunker): CJK-aware oversize measurement + close capOversizedChunks' fallback-path gaps (#3477)
Shape requested in #3475's closing review: make capOversizedChunks use a
CJK-aware estimate instead of adding a parallel opt-in cap.

Measurement (vs the Qwen3-Embedding tokenizer, the strict-backend class
from #2826): cl100k matches embedding-family tokenizers on pure-ASCII
source (identical counts on English prose and JSON) but undercounts
MIXED CJK+ASCII chunks — −31% on URL-dense Korean text. The heuristic
fallback (~3.5 chars/token) undercounts CJK ~2.5×.

estimateEmbedTokens(): for chunks containing CJK, max(cl100k, per-char-
class overestimate — CJK 1.0 / other non-ws 0.75 / ws 0.1). ASCII-only
chunks short-circuit to estimateTokens verbatim (bit-identical, pinned);
CJK-DOMINANT text is unchanged too (cl100k already exceeds the weighted
form, so max() returns today's value — pinned). Only mixed-script
chunks, the measured divergence class, estimate higher. Reuses cjk.ts's
existing exports — no new module, no config.

Also: only the empty-AST branch routed its fallback through
capOversizedChunks. The no-language, parse-timeout, no-semantic-nodes
(every JSON/YAML fence — their node types aren't in TOP_LEVEL_TYPES) and
parse-throw branches shipped word-counted chunks unchecked, letting a
14K-char JSON fence emit ~2,700-token chunks past the 2,000 default cap.
Hoist the cap into fallbackChunks so all five emission paths share the
net. Hard-split slice budget becomes 1 char/token for CJK-bearing pieces
(the weighted estimate can reach 1 token/char).

Refs #2826

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 11:41:08 -07:00
Sailesh SivakumarandClaude Fable 5 fd8be831c5 fix(jobs): rehydrate wire-format timestamps in thin-client list/get (#3026) (#3027)
The thin-client branches receive MinionJob rows as parsed JSON off the
MCP wire — every timestamp an ISO string — while formatJob /
formatJobDetail and the stalled-detection comparison hold a Date
contract (locally hydrated by MinionQueue.rowToJob). `jobs get <id>` on
a thin client crashed with "job.started_at.toISOString is not a
function" the moment the remote routing actually worked (unmasked by
the #2951 scratch-engine fix).

Rehydrate once at the unpack boundary via an exported helper that
coerces valid ISO strings to Dates, leaves Dates/nulls/malformed
strings untouched, and preserves the input type. Unit tests +
source-audit pins for both unpack sites.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 23:47:32 -07:00
Eungoo Jung 9664cad329 fix(doctor): sync_freshness falls back to content-lag when the clone is unavailable — stop false stale/FAIL after stateless-container restarts (#2908)
On stateless deploys (Docker on EB/K8s/Fly — what the cloud recipes
produce), a container restart wipes federated clones; each is only
re-materialized when that source's next sync job runs. Until then the
v0.41.27.0 git short-circuit cannot probe HEAD at all, and the check
fell through to raw wall-clock age — which no-op syncs never advance —
so every QUIET source read as stale/FAIL right after a restart.
Observed live: 16-source brain, 12 clones gone after a config-update
restart, doctor 70 -> 25-35, monitor alert storm (score < threshold)
while every clone that DID exist was byte-identical to origin HEAD.

Fix: classify the probe three ways (probeSourceGitState: unchanged /
changed / unavailable). 'unavailable' + chunker match borrows the
REMOTE path's newest_content_at lag (v0.41.32.0) — DB-only, no
subprocess — so a quiet source reads healthy while real missed work
(content newer than last sync) still reports stale. 'changed'
(readable clone, HEAD moved / dirty) keeps wall-clock exactly as
before, and a chunker mismatch disables the fallback (D7: a pending
re-chunk is never masked). isSourceUnchangedSinceSync stays as a
boolean facade so source-health.ts is untouched.

Tests: 6 new doctor cases (F1-F6, incl. three-bucket invariant) +
7 probeSourceGitState unit cases; existing suites green
(doctor 90, git-head 21, source-health 28), tsc --noEmit clean.
2026-07-27 23:46:17 -07:00
fcc6e670f2 fix(search): make no-embedding early-return multimodal-aware (#2319)
The no-embedding-provider short-circuit in hybridSearch probed only the
text column's provider. On a multimodal-only install (text embedding
provider absent, a multimodal provider such as Voyage multimodal-3
present), the function returned to the keyword-only path before the
image/unified vector routing ever ran -- so image and unified queries
silently degraded to keyword search (vector_enabled:false) even though a
usable multimodal vector path existed.

Add a willTryMultimodal guard that probes the multimodal embedding
provider (embedding_multimodal_model) so the early-return does not fire
when multimodal vectoring is still possible, and tighten the unified and
image branches' bare aiIsAvailable('embedding') (global-default) checks
to probe the multimodal provider too.

Adds a focused regression test (search-multimodal-no-embed.serial) that
configures a text-provider-absent / multimodal-present install and
asserts image + unified queries reach the multimodal vector path.

Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 23:44:52 -07:00
Eoin O'BrienandClaude Opus 5 2a17a4dab5 fix(repo): untrack node_modules symlink, guard against tracked symlinks (#3463)
Commit faf5cdba tracked `node_modules -> /tmp/fleet/repo/node_modules`.
That path exists only on the sandbox that produced it, so every other
clone materialized a dangling symlink and `bun install` aborted with
`ENOENT: could not open the "node_modules" directory`. That also broke
`gbrain upgrade` on bun-link installs, which shells out to bun install
and then prints a manual fallback that fails identically.

Three changes:

- Untrack the symlink (`git rm --cached node_modules`).
- Drop the trailing slash from the .gitignore node_modules patterns. A
  `node_modules/` pattern matches directories only, which is why a
  symlink of the same name was never ignored in the first place.
- Add scripts/check-no-tracked-symlinks.sh, wired into `bun run verify`
  and `check:all`. The .gitignore fix alone is not sufficient, since
  `git add -f` bypasses it; the guard fails on any mode-120000 entry.
  The repo has no legitimate tracked symlinks, so it starts with an
  empty allowlist.

Covered by test/no-tracked-symlinks-guard.test.ts, which builds a
throwaway repo containing the exact symlink shape and asserts the guard
exits 1 and names the offender.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:37:09 -07:00
ddd66e1d25 fix(ai): declare DashScope's documented 10-item embedding batch cap (#2643 concept, refs #2103 #2405) (#3451)
DashScope's OpenAI-compatible /embeddings endpoint rejects requests with
more than 10 input items (documented Model Studio cap). The generic
per-recipe max_batch_items field + gateway capBatchItems pre-split
already exist (#1281); the in-tree dashscope recipe just never declared
the cap, so large embed backfills would send oversized batches and get
rejected server-side. Declare max_batch_items: 10 on the dashscope
embedding touchpoint; max_batch_tokens stays as the aggregate
token-size guard.

Test: pins dashscope's max_batch_items === 10 (+ max_batch_tokens
unchanged) and that 25 items pre-split into groups of at most 10 via
capBatchItems, alongside the existing llama-server cap pin.

No new models or recipes; no error-sniffing/halving recovery — the
pre-split makes the failure unreachable. Item-cap concept credited to
declined community PRs #2643 and #2405.

Also verified (no code change needed): #2103's litellm three-way dead
end is already fixed on master by a25209bb (#2271) via trust_custom_dims.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Yicong <charlieyiconghuang@gmail.com>
Co-authored-by: Cheng Zijun <robotics.chengzijun@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 18:55:23 -07:00
faf5cdba54 feat(conversation-facts): parse Slack block format + route granular collector page-types (#2357)
extract-conversation-facts extracted nothing on brains that store chat in the
collector's native page types. Two stacked gaps:

- Type routing: the allowlist exact-matched {conversation,meeting,slack,email}
  against pages.type and passed each straight to listPages({type}), so
  --types slack matched zero rows on a brain carrying slack-dm-day /
  slack-thread / email-digest. Add ALLOWED_TYPE_ALIASES + pageTypesForAllowed()
  to expand logical -> concrete (canonical name first so consolidated brains are
  unaffected), wired into both the single-slug filter and the listPages loop.

- Block-format parsing: the 14 built-in patterns are single-line; the Slack
  collector emits a header + indented-body block (`- **Name** (Mon 11:18)` then
  body on following lines) that none match -> phase:'no_match', 0 messages, and
  the LLM fallback is not wired. Add normalize-block.ts, a strict-no-op pre-pass
  in parseConversation that collapses the block into the canonical
  `**Name** (HH:MM): body` line the bold-paren-time pattern handles; the
  per-message date fills in downstream via fallbackDate. 12h am/pm normalized to
  24h; day-of-week dropped.

Verified on a 13.7K-page comms brain whose facts table was empty: a 12-page
Slack sample went 0/12 parsed (no_match) -> 12/12 (regex_match), 103 messages,
13 segments; extraction wrote 58 facts across 16 entities (~$0.09) and
find_trajectory returns a populated points list for a local/owner caller where
it previously returned empty.

Tests: +13 normalize-block (detection, multi-paragraph collapse, 12h->24h,
no-op on canonical, parseConversation integration) + 7 pageTypesForAllowed.
typecheck clean; verify 30/30.


Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-27 18:06:01 -07:00
e9fa962929 feat(extract): --infer-dates anchors timeline from a page's content date when its body has none (#2341)
* feat(extract): --infer-dates anchors timeline from a page's content date when its body has none

parseTimelineEntries only reads in-body date lines (`- **YYYY-MM-DD** | ...`).
Comms- and calendar-dominated brains keep the date in frontmatter or the
filename (slug `2026-04-24-...`), so those pages yield zero timeline entries
and find_trajectory stays blind even though the page is firmly dated.

`--infer-dates` (opt-in, DB-source) anchors ONE timeline entry at the page's
already-computed `effective_date` for pages whose body parse returns nothing.
Trustworthy sources only (frontmatter event_date/date/published or the filename
date) — never the `updated_at` fallback. Applied solely on the zero-entry path
so it can never shadow a real in-body timeline.

- new pure helper `deriveTimelineAnchor()` in link-extraction.ts (+6 unit tests)
- `getPage()` now projects effective_date/effective_date_source in BOTH engines
  (engine parity)
- on a comms-heavy ~13.7K-page brain this lifts a dry-run timeline yield from 1
  to 11,006 entries (timeline coverage 0% -> ~80%)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy

* docs(extract): correct deriveTimelineAnchor comment — feeds page timeline, not find_trajectory

find_trajectory reads the facts table by entity_slug; the page-level `timeline`
table this helper populates feeds get_timeline + the brain-score timeline_coverage
component instead. Comment-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E5wtDU4ZLKewXUYkPLQHSy

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-27 18:04:46 -07:00
0413c93e72 feat: CJK entity extraction for Chinese/Japanese/Korean names (#1637)
* feat: CJK entity extraction for Chinese/Japanese/Korean names

- Add hasCJK() / cjkCharCount() detection helpers
- Lower min name length for CJK entities from 4 to 2 chars
- Fix tokenizeTitle() to handle pure CJK titles as single tokens
  (was returning [] for CJK-only titles, excluding them from gazetteer)
- Add CJK substring matching pass in findMentionedEntities
- NER extraction works without schema pack (plain mentions fallback)

Verified: gbrain extract links --by-mention creates 27 links from
456 pages with 3 CJK entity pages in gazetteer.

* feat: Chinese link type inference + timeline date formats

Link types:
- CN_FOUNDED_RE: 创立/创办/成立/创建 → founded
- CN_INVESTED_RE: 投资/入股/融资 → invested_in
- CN_ADVISES_RE: 顾问/咨询/指导 → advises
- CN_WORKS_AT_RE: 任职/就职/担任 → works_at
- CN_CITED_RE: 引用/提到/提及 → cited

Timeline:
- TIMELINE_LINE_RE_CN: YYYY年M月D日 | event
- Auto-normalizes to YYYY-MM-DD format
- Falls through to English format if CN doesn't match

* fix: CJK tokenizer uses char-level tokens (reviewer feedback)

Addresses all 4 concerns from review of PR #1637:

1. tokenizeForScan now emits CJK characters as individual tokens
   — normal scan path reaches CJK gazetteer entries naturally,
     eliminating the separate O(P×C×N) substring fallback pass.

2. tokenizeTitle splits pure CJK titles into individual chars
   — e.g. '纳瓦尔' → ['纳','瓦','尔'], matching body-level CJK tokens.

3. Removed O(P×C×N) CJK substring pass — no longer needed.
   Performance now O(P × N_tokens) for both ASCII and CJK.

4. Renamed CN_*_RE → ZH_*_RE in link-extraction.ts with a comment
   clarifying these are Chinese-only (entity NAME extraction in
   by-mention.ts covers CJK scripts, link TYPE extraction is zh only).

Added 12 CJK-specific tests (10 pure + 2 engine integration).
All 51 existing + new tests pass.

* review-repair(#1637): scope CN timeline regex to 年月日, revert off-scope extract-ner no-pack change, cosmetics

- TIMELINE_LINE_RE_CN required only [年-] separators, so non-bold ASCII
  dates (- 2020-01-02 - text) started parsing as timeline entries — an
  English-default regression. Now requires the 年/月 markers.
- Dropped the dead 'm = cm as any' assignment.
- src/core/extract-ner.ts reverted to origin/master: the no-pack →
  plain-mentions walk was off-scope for a CJK PR, duplicated the
  existing --by-mention pass, and hardcoded pack_unavailable:false
  (breaking the CLI hint).
- by-mention.ts: fixed stray indentation + restored EOF newline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-27 18:03:31 -07:00
56aac51a08 feat(extract): quarantine lane for auto-extracted entities from untrusted input (#160) (#3458)
* feat(extract): quarantine lane for auto-extracted entities from untrusted input (#160)

extractAndEnrich regex-extracts entity names from arbitrary ingested text
and creates people/ + companies/ stub pages. Those writes are now trust-
gated end to end:

- src/core/extraction-review.ts: new marker module (sibling of
  quarantine.ts / embed-skip.ts, frontmatter-key pattern, no migration).
  Untrusted-input stubs carry `provenance: auto-extracted` +
  `status: unverified`; the shared unverifiedExtractionFragment() is the
  single SQL source of truth for every consumer.
- enrichment-service: enrichEntity/enrichEntities/extractAndEnrich take
  EnrichmentTrustOptions; only an explicit trusted:true writes
  authoritative pages (fail-closed, mirrors the OperationContext.remote
  invariant). Also threads sourceId through the write path.
- retrieval: unverified stubs rank as ordinary content — skipped by the
  compiled-truth fusion boost (stampUnverifiedExtractions pre-fusion on
  all three hybrid paths + keyword-only opt-out) and by the people//
  companies/ namespace source-boost (guard inside buildSourceFactorCase,
  shared by both engines' search SQL). Results carry `unverified: true`.
  New engine method getUnverifiedExtractionPageIds in BOTH engines.
- ops (contract-first): extract_entities (direct write only for
  ctx.remote === false + --trusted-extraction; everything else
  quarantines), extraction_pending (read, source-scoped list),
  extraction_review (owner-only batch promote/reject; promote flips
  status to verified keeping provenance for audit, reject soft-deletes).
- doctor: unverified_extractions check warns on stubs older than N days
  (default 7) with the exact review commands.

Tests: test/extraction-review.test.ts (PGLite: fail-closed matrix incl.
remote-unset, fusion boost skip, review queue, doctor, hostile-transcript
e2e proving fake entities land quarantined and rank below a verified page
of equal lexical relevance) + test/e2e/extraction-review-postgres.test.ts
(live Postgres parity, verified against pgvector:pg16). sql-ranking
expectations updated to current state. Docs: KEY_FILES + RETRIEVAL +
llms rebuild.

Closes #160

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(extract): close vector-arm source-boost gap + harden extract_entities (#160 review round)

Adversarial review of the quarantine lane found the people//companies/
1.2x source factor still applied to unverified stubs inside searchVector's
pre-LIMIT re-rank (a different multiplier from the fusion-level 2.0x the
lane already cancels — and applied early enough to evict legitimate pages
from the candidate pool, which nothing downstream can restore).

- buildSourceFactorCase gains an optional unverifiedGuardColumn for the
  bare-slug re-rank form; both engines' hnsw_candidates CTEs now project
  the guard predicate as `unverified_stub` and the factor CASE checks it
  first. Wrong "fusion covers the vector arm" comment corrected.
- extract_entities resource guards: 200k-char input cap (loud reject),
  200-entity cap surfaced as `truncated` + `entities_found`; the library
  extractAndEnrich gets the same default cap. (OperationContext has no
  abort signal field — caps are the bound.)
- extraction_review promote is now a targeted JSONB-merge UPDATE instead
  of putPage, so non-carried columns (page_kind, content_hash) can't be
  reset by the upsert.
- extraction_pending applies buildVisibilityClause (archived-source stubs
  no longer list).
- Wording: op description + module header now state the marker-strip
  assumption plainly (markers are ordinary frontmatter; the boundary
  against wholesale rewrite is put_page write authz) and document the
  CREATE-only scope of the lane.

Tests: vector-arm factor-1.0 pinned on BOTH engines (PGLite unit + live
Postgres e2e, identical basis embeddings → score ratio is the factor);
resource-guard test (oversize reject + 300-entity flood capped at 200);
guard-column form pinned in the buildSourceFactorCase unit test.
search/ suite (340), sql-ranking, searchvector-maxpool, title-retrieval-
arm, rrf-source-key, doctor, ops, cli suites all green; JSONB guards clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 18:01:35 -07:00
0bbaed2e48 v0.42.67.0 feat(migrate): provider-agnostic embedding migration — the path off ZeroEntropy (#3390, fixes #3391) (#3459)
* feat(migrate): provider-agnostic embedding migration service — the path off ZeroEntropy (#3390)

- gbrain migrate embeddings --to <provider:model> (alias: retrieval-upgrade):
  plan + cost preflight, consent gate (--yes / TTY confirm / non-TTY exit 2),
  live probe against the target provider before any mutation, env-override
  gate, schema dimension transition via the shared runSchemaTransition,
  dual-plane config write, NULL-signature-inclusive invalidation, query-cache
  purge, resumable re-embed through the standard embed pipeline (single-flight
  locks, backoff, pacing, stderr progress). Killed runs resume by re-running
  the same command; the NULL-embedding column is the checkpoint.
- #3391 root-cause fix (both engines): countStaleChunks / sumStaleChunkChars /
  invalidateStaleSignatureEmbeddings accept includeNullSignature to lift the
  v108 grandfather clause; embed --stale warns loudly when a model swap
  leaves NULL-signature pages in the old embedding space, and
  --include-null-signature re-embeds them. Default sweep behavior unchanged.
- knobs_hash v=12 → v=13 (prov=default legacy callers must not be served
  pre-migration cache rows).
- migrate_embeddings op: scope admin, localOnly, hidden cliHints, hard
  remote refusal, needs_confirmation without yes=true.
- One-shot post-upgrade ZE-sunset banner (ze_sunset_notice_shown) for brains
  resolving to a zeroentropyai:* embedding model or reranker.
- doctor's dimension-mismatch repair hint now names the real command.
- Docs: docs/guides/embedding-migration.md, KEY_FILES entries, spend-controls
  gate row. Tests: PGLite unit + full-lifecycle flow (interrupted-run resume),
  real-Postgres e2e (pgvector DDL path + #3391 predicate parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): satisfy check:test-isolation + bump the remaining knobs_hash pins

- test/migrate-embeddings-flow.test.ts → .serial.test.ts: the file holds a
  temp GBRAIN_HOME + an installed fake embed transport for its whole
  lifecycle (beforeAll → afterAll), which withEnv() can't wrap. This also
  fixes the CI shard-pollution failure in
  test/ai/recipes-existing-regression.test.ts (that file passes solo on both
  master and this branch; the flow test's configureGateway + provider-key
  deletion was leaking into it inside the same shard process).
- test/embedding-migration.test.ts: env-override case now uses withEnv().
- Bump the three remaining KNOBS_HASH_VERSION pins to 13
  (cross-modal-phase1, search-alias-resolved-boost, search/knobs-hash-reranker).
- Docs + llms bundles follow the test rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(test): wire the new Postgres e2e into the smart e2e selector map

Changes to embed.ts / embedding-migration.ts / retrieval-upgrade-planner.ts /
postgres-engine.ts now trigger test/e2e/migrate-embeddings-postgres.test.ts —
the #3391 stale predicates and runSchemaTransition's DDL path behave
differently on real pgvector than on PGLite, so the smart selector has to know.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(migrate): consult spend.posture in the embedding-migration consent gate

The brief asked the gate to honor spend.posture; it previously didn't read it
at all. Now it does — but deliberately does NOT bypass on tokenmax: posture
waives the spend CEILING, and this gate also guards a destructive schema
rebuild (existing vectors dropped, retrieval degraded until the re-embed
finishes). Under tokenmax the dollar figure is marked informational on stderr
and the confirmation is still asked; --yes stays the single scripted bypass.

Pinned by a new case in the flow test so a later refactor can't quietly turn
posture into a bypass. Guide + spend-controls table updated to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* wip: blocker fixes

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 17:48:37 -07:00
148 changed files with 10343 additions and 468 deletions
+16
View File
@@ -0,0 +1,16 @@
# Line-ending policy.
#
# Shell scripts MUST be checked out with LF endings on every platform.
# Git for Windows installs with `core.autocrlf=true` by default, which
# rewrites LF -> CRLF on checkout. A strict bash (WSL, Linux CI, macOS)
# then chokes on the trailing CR:
#
# scripts/run-unit-parallel.sh: line 23: $'\r': command not found
# scripts/run-unit-parallel.sh: line 24: set: pipefail : invalid option name
# scripts/run-unit-parallel.sh: line 32: syntax error near unexpected token `$'{\r''
#
# That silently disabled `bun run test`, `bun run verify`, `bun run ci:local`
# and `bun run test:e2e` for Windows contributors, since all four dispatch
# through bash. `eol=lf` pins the checkout regardless of the user's
# core.autocrlf setting.
*.sh text eol=lf
+6 -3
View File
@@ -61,7 +61,10 @@ jobs:
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
# --timeout also raises bun's 5s default hook budget (beforeAll/afterAll
# do NOT inherit a test's third-arg timeout; verified on bun 1.3.x).
# Every runner script in scripts/ passes it; bare invocations must too.
run: bun test --timeout=60000 test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
@@ -88,7 +91,7 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -155,7 +158,7 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test --timeout=60000 test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+3 -1
View File
@@ -29,7 +29,9 @@ jobs:
with:
bun-version: 1.3.13
- run: bun install
- run: bun test
# --timeout matches every scripts/ runner and covers hook budgets too
# (bunfig.toml's timeout key is ignored by bun; hooks default to 5s).
- run: bun test --timeout=60000
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Attest build provenance
+5
View File
@@ -113,6 +113,11 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
# default regardless of per-test third-arg timeouts. Runs directly
# (not via verify's CHECKS array) to avoid a package.json edit.
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
+5 -2
View File
@@ -1,4 +1,7 @@
node_modules/
# No trailing slash: a bare `node_modules/` pattern matches directories only,
# so a *symlink* named node_modules slips past it and can be committed
# (that's how the /tmp-pointing symlink in faf5cdba got in). Match any type.
node_modules
bin/
.DS_Store
*.log
@@ -15,7 +18,7 @@ supabase/.temp/
# self-contained binaries (the bun --compile path embeds it via
# `import path from 'admin/dist/index.html' with { type: 'file' }`).
# Build via: cd admin && bun install && bun run build.
admin/node_modules/
admin/node_modules
.idea
eval/reports/
eval/data/world-v1/world.html
+39
View File
@@ -2,6 +2,45 @@
All notable changes to GBrain will be documented in this file.
## [0.42.67.0] - 2026-07-28
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
`bun run test`, `bun run verify`, `bun run ci:local` and `bun run test:e2e` all hand off to shell scripts, and on Windows that hand-off was broken in two separate places. The commands did not stop with an obvious error. They reported a result, so a run could look finished when barely any of the checks had actually inspected anything. On a clean Windows clone, `bun run verify` got 1 check to pass and 31 to fail. It now gets 25 to pass and 7 to fail, and none of the 7 are caused by this change.
The first problem was line endings. Git for Windows installs with `core.autocrlf=true`, which rewrites shell scripts to Windows line endings when you clone or check out. Bash refuses to run those, so a script died on its second line before doing any work. The scripts stored in the repository were always correct; only the copy on your disk was wrong. A new `.gitattributes` pins every `.sh` file to Unix line endings at checkout, no matter how your Git is configured.
The second problem was how the checks were started. Thirty three of them pointed straight at a `.sh` file. On macOS and Linux the shell reads the `#!/usr/bin/env bash` line at the top of the script and runs it correctly. Bun on Windows does not do that, so those commands failed the moment they were called. They now go through `bash` explicitly, the same way the other eleven were already written.
Nothing changes for macOS and Linux. No stored file content moves, and no check behaves differently on those platforms.
## To take advantage of v0.42.67.0
Only Windows contributors need to do anything, and only once. `.gitattributes` applies at checkout time, so shell scripts already sitting on your disk keep their old line endings until you refresh them.
1. **Refresh the working copy** from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
```
2. **Confirm bash can read the scripts:**
```bash
bash -n scripts/run-unit-parallel.sh
```
Silence means it worked. `$'\r': command not found` means step 1 did not take effect.
3. **Run the gate:**
```bash
bun run verify
```
### Itemized changes
- New root `.gitattributes` pins `*.sh text eol=lf`, so shell scripts check out with Unix line endings regardless of the contributor's `core.autocrlf` setting. All 59 tracked `.sh` files were already stored with Unix endings, so `git add --renormalize .` reports nothing to do and no stored content changes.
- `package.json` now routes the remaining 33 `.sh` check commands through `bash`, matching the 11 that already did. Every tracked `.sh` file carries a bash shebang (52 `#!/usr/bin/env bash` and 7 `#!/bin/bash`), so the treatment is uniform across all of them.
- The five `scripts/*.ts` entries still run under bun and are untouched.
- `CONTRIBUTING.md` gains a Windows section covering the one-time working-copy refresh and the `bash scripts/<name>.sh` convention for new checks.
- `docs/TESTING.md` records how the test commands dispatch through bash, and notes that three tree-walking checks plus `typecheck` can exceed the 120s per-check cap on Windows while passing on Linux and macOS.
## [0.42.66.1] - 2026-07-27
### Fixed
+13
View File
@@ -67,6 +67,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
+22
View File
@@ -11,6 +11,28 @@ bun test
Requires Bun 1.0+.
### Windows
`bun run test`, `verify`, `ci:local` and `test:e2e` all dispatch through bash, so
the shell scripts under `scripts/` must be checked out with Unix line endings.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
correct with no extra steps.
If you cloned before that pin existed, your working copy still has the old
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
it once, from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
```
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
rather than relying on the shebang, because bun on Windows cannot exec a `.sh`
directly. Keep that prefix when you add a new shell-script check.
## Project structure
```
+27 -4
View File
@@ -1,5 +1,25 @@
# TODOS
## v0.42.67.0 follow-ups (Windows build tooling)
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
`bash` prefix on the 33 `package.json` check commands). Both items are newly
observable: before that release these checks never executed on Windows at all,
so nothing about their runtime was measurable.
- [ ] **P2 — three guard scripts exceed the 120s `run-verify-parallel.sh` cap on Windows.**
With the dispatch fixed, `bun run verify` on Windows gets 25 passes and 7 failures, and
`check:privacy`, `check:test-names` and `check:test-isolation` are timeouts rather than
real failures (they pass on Linux and macOS well inside the cap). They walk the tree with
per-file shell loops, which is far slower under Windows process creation. Either raise the
cap for these three, or replace the per-file loop with a single `grep -r` pass. Same cap
swallows `typecheck`, though standalone `bun run typecheck` exits 0.
- [ ] **P3 — `check:wasm` cannot create its `node_modules` symlink on Windows.**
`scripts/check-wasm-embedded.sh` fails with `ln: failed to create symbolic link
'/tmp/gbrain-wasm-check.XXXX/node_modules': No such file or directory`. Unprivileged
Windows accounts cannot create symlinks without developer mode. Consider a junction, a
copy, or skipping the check with a clear message when symlink creation is unavailable.
## community fix-wave follow-ups (filed v0.42.60.0)
- [x] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
@@ -62,17 +82,20 @@ Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
- [x] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
native providers (anthropic/openai/google) do. To make expansion work on
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
chat-capable recipes AND add a `generateObject``generateText` capability fallback for
backends without strict structured outputs. Feature-shaped; overlaps the general
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
starting point. Implemented by #2373 plus the DeepSeek/Groq/Together recipe wave,
LiteLLM chat/expansion support, and the OpenRouter expansion touchpoint. Where:
`src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [x] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
LiteLLM proxy is a full LLM backend, not embedding-only. Implemented by #2208.
The general OpenAI-compat proxy story.
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
+1 -1
View File
@@ -1 +1 @@
0.42.66.1
0.42.67.0
+23
View File
@@ -19,6 +19,29 @@ Seven test command tiers, each with a clear scope:
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### Shell dispatch and Windows
All four of `test`, `verify`, `ci:local` and `test:e2e` hand off to shell scripts
under `scripts/`, so every `check:*` entry in `package.json` invokes its script as
`bash scripts/<name>.sh` instead of relying on the shebang — bun on Windows cannot
exec a `.sh` directly. Add a new shell-script check with that same prefix. The
`scripts/*.ts` entries run under bun and take no prefix.
The scripts must also be on disk with Unix line endings. A strict bash (WSL, Linux
CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygwin
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
pick it up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
tree-walking checks (`check:privacy`, `check:test-names`, `check:test-isolation`)
plus `typecheck` can exceed the 120s per-check cap in `run-verify-parallel.sh`
there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
File diff suppressed because one or more lines are too long
+9
View File
@@ -87,6 +87,15 @@ embedding proximity. Four layers, added after the incident in
deciding "is this page already here, safe to NOT write a duplicate?" keys off
`create_safety`, not a raw blended score.
**Extraction quarantine lane (issue #160):** pages carrying the unverified
auto-extracted markers (frontmatter `provenance: auto-extracted` +
`status: unverified`, see `src/core/extraction-review.ts`) rank as ordinary
content — they are skipped by the compiled-truth fusion boost and by the
`people/`/`companies/` namespace source-boost, and every search result from
such a page carries `unverified: true` so agents can label the provenance.
Promote or reject them via `gbrain extraction-pending` / `gbrain
extraction-review`.
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
+138
View File
@@ -0,0 +1,138 @@
# Embedding migration — moving a brain to another embedding provider
`gbrain migrate embeddings` re-embeds an entire brain onto a different
embedding provider/model, safely and resumably. It is the forward path off a
sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
2026-09-04 and is the shipped default for brains that never picked a model) —
but it is provider-agnostic: any configured `provider:model` works as a
target.
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
README reference).
## Quick start
```bash
# Preview the work + cost. Changes nothing.
gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run
# Run it (interactive confirm shows chunk count + $ estimate first).
gbrain migrate embeddings --to openai:text-embedding-3-small
# Non-interactive (cron / scripts): --yes is required, else exit 2.
gbrain migrate embeddings --to voyage:voyage-3-large --yes
```
`--dim <N>` overrides the target width; it defaults to the provider recipe's
declared width and is required for recipes that don't declare one (litellm,
llama-server, and other bring-your-own-model providers).
## What it does, in order
1. **Plan.** Counts every chunk not already in the target embedding space —
including chunks on pages with **no recorded embedding signature**
(pages embedded before the v108 provenance stamp). Prices the re-embed
from the pricing table; unknown providers print "estimate unavailable"
instead of a fabricated number.
2. **Consent gate.** Prints the plan; requires an interactive `y` or `--yes`.
Non-TTY without `--yes` refuses with exit 2 (mirrors the `reindex-code`
gate in [spend-controls](../operations/spend-controls.md)). Unlike the pure
cost gates there, `spend.posture=tokenmax` does **not** bypass this one:
posture waives the spend *ceiling*, and this gate also guards a
destructive schema rebuild. Under `tokenmax` the dollar figure is marked
informational and the confirmation is still asked. `--yes` is the single
scripted bypass.
3. **Live probe.** One tiny embed against the TARGET provider before any
mutation — validates the API key, model id, and dimension support in a
single call. A bad key fails here, with nothing changed.
4. **Env-override gate.** Refuses when `GBRAIN_EMBEDDING_MODEL` /
`GBRAIN_EMBEDDING_DIMENSIONS` would silently defeat the switch at
runtime (the same guard `ze-switch` uses). `--ignore-env-override` for
people running deliberate experiments.
5. **Apply.** When the target width differs from the actual column width,
runs the same atomic schema transition `ze-switch` uses, in one
transaction. It rebuilds **all three dim-pinned text-embedding-space
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
`facts.embedding` — at the new width, preserving each column's type
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
three leaves it silently broken: a narrow `query_cache.embedding` makes
every cache write and read fail *by design* (the cache swallows errors so
it can never break search) for a permanent 0% hit rate, and a narrow
`facts.embedding` fails every per-fact embed write. The image/multimodal
columns ARE deliberately untouched — they use separate models whose
dimensions are independent of the text embedding model.
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
(file plane for the runtime gateway, DB plane for doctor), invalidates
every chunk still in the old space — **including NULL-signature pages**
and purges the semantic query cache so stale cached results can't be
served across the swap.
6. **Re-embed.** The standard embed pipeline (`embed --stale --catch-up`)
with per-source single-flight locks, rate-limit backoff, stderr progress,
and optional DB-contention pacing (`--pace[=mode]`).
## What the rebuild deletes
The dimension change **deletes every stored embedding vector** in the brain —
they are in the old model's space and unusable. They are not recoverable:
going back to the previous provider means paying for a second full re-embed.
`content_chunks` vectors are rebuilt by the re-embed pass, the query cache
refills on the next query, and fact embeddings are rewritten on their next
write (or a `gbrain extract` pass).
## Resume after a kill
The NULL-embedding column is the checkpoint. If the run is killed (or some
pages fail to embed), re-run the **same command**: chunks already embedded on
the target are never re-embedded, the schema/config steps no-op, and the run
continues where it stopped. An in-flight marker (`embedding_migration.state`
in DB config) records the target; it is cleared only when the backlog drains
to zero.
A page whose chunks straddle two stale batches is embedded correctly but not
stamped by the embed loop (which only stamps all-or-nothing per batch), so the
migration runs one reconcile pass after the drain that stamps every
fully-embedded page. Without it a large brain would report "incomplete" and the
re-run would pay again for those pages. `--batch-size N` tunes the batch
(default 2000).
`--no-embed` applies schema + config + invalidation and stops, so you can run
the (potentially long) re-embed later or in the background:
```bash
gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed
gbrain embed --stale --catch-up --include-null-signature --background
```
## During the migration
While the re-embed runs, semantic search returns degraded (lexical-arm-only)
results for not-yet-re-embedded content. Pick a quiet window for large
brains, or use `--pace` to keep the DB responsive.
## Pages without an embedding signature (#3391)
Pages embedded before provenance stamping have `embedding_signature IS NULL`
and are grandfathered by the routine stale sweep (so an upgrade never
surprise-re-embeds a whole corpus). After a provider swap that grandfather
clause would silently leave those pages in the OLD embedding space — mixed
vector spaces in one index, degrading retrieval with nothing in the logs.
- `gbrain migrate embeddings` always includes them.
- Plain `gbrain embed --stale` warns when a model swap leaves NULL-signature
pages behind, and `gbrain embed --stale --include-null-signature` re-embeds
them.
## Reranker
Migrating embeddings does not touch the reranker. If
`search.reranker.model` points at the outgoing provider, the plan prints a
warning; disable it (`gbrain config set search.reranker.enabled false`) or
point it at another provider.
## Self-hosting instead of migrating
If the outgoing model's weights are available (zembed-1's are Apache-2.0),
serving them locally via `llama-server` / `ollama` / a LiteLLM proxy
preserves your existing vectors — no re-embed at all. Point
`embedding_model` at the local recipe and keep the same dimensions. The
migration command is for when you'd rather move to a hosted provider.
+1
View File
@@ -49,6 +49,7 @@ The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to m
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
| `migrate embeddings` consent gate | — (plan + estimate before provider migration) | — | TTY y/N prompt / non-TTY refuse + exit 2 | `--yes` | estimate marked informational, but **still prompts** (guards a destructive schema rebuild, not just spend) |
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
### Sync inline-embed cost gate
+3
View File
@@ -140,6 +140,9 @@ Stable phase names shipped in v0.15.2:
- `import.files`
- `sync.deletes`, `sync.renames`, `sync.imports`
- `migrate.copy_pages`, `migrate.copy_links`
- `migrate.reembed` (the re-embed pass of `gbrain migrate embeddings`; total is the
stale-chunk backlog at the start of the pass, so it can grow slightly if a
writer adds chunks mid-run)
- `repair_jsonb.run`, `repair_jsonb.<table>.<column>`
- `backlinks.scan`
- `lint.pages`
@@ -0,0 +1,690 @@
# Engine Dynamic-Import Reconciliation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reconstruct the missing engine-path static-import hardening, preserve the four load-bearing lazy gateway fallbacks, and prevent unreviewed dynamic imports from returning.
**Architecture:** Make the 13 safe engine/migration import statements static and leave only four line-marked `ai/gateway.ts` imports inside their existing soft-failure `try/catch` boundaries. Enforce that current state with a repository-anchored Bash wrapper delegating to a fail-closed TypeScript AST scanner, a hermetic Bun regression test, package/verify wiring, and current-state architecture documentation.
**Tech Stack:** TypeScript compiler API, Bun test runner, Bash, Git, generated llms documentation bundles.
## Global Constraints
- Reconstruct directly on branch `claude/kind-meitner-330c90`, based on investigated `origin/master` commit `6136e139972a5449630b4f47f5ed7b4cbe5b811b` plus design commit `d7f52d8c`.
- Do not merge or cherry-pick `48ada48f`, `248bfe55`, `ef4cf7a8`, or either historical branch wholesale.
- Do not modify `VERSION`, `CHANGELOG.md`, `TODOS.md`, or release metadata; this is a no-version-bump reconciliation.
- Keep all four `await import('./ai/gateway.ts')` calls lazy: PGLite and Postgres `initSchema`, plus both `_upsertChunksOnce` methods.
- Every allowed lazy gateway line must carry `engine-dynamic-import-ok`; there is no file-level exemption.
- Preserve the stronger gateway rationale: the static closure is large, and eager module evaluation would occur outside the local `try/catch`, potentially converting a recoverable configuration/import failure into a module-load-time hard failure.
- Describe the hoists as engine-path hardening. Do not claim every dynamic import deterministically causes a Windows crash; system-wide commit exhaustion confounded prior measurements.
- Keep shared PGLite/Postgres behavior in parity.
- Invoke repository shell scripts through `bash` in `package.json`.
- Capture complete test/check output to workspace-local `.context/*.txt` files before inspecting it; never pipe a test command directly through `head` or `tail`.
- Use `git log -G`, not `git log -S`, for any additional dynamic-to-static import history work.
- Keep every implementation and verification commit local. Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after local completion.
- Before editing any affected function, run GBrain `code_blast` and `code_callers` for that symbol and inspect any disambiguation candidates.
---
## File Map
- Create `scripts/check-engine-dynamic-import.sh` — repository-anchored Bash wrapper for default and explicit input routing.
- Create `scripts/check-engine-dynamic-import.ts` — TypeScript AST policy scanner for runtime `import()` expressions, parse/read failures, and exact-line comment-trivia opt-outs.
- Create `test/scripts/check-engine-dynamic-import.test.ts` — 22 hermetic adversarial, CRLF, fail-closed, real-tree, and wiring tests.
- Modify `src/core/pglite-engine.ts` — hoist three safe import statements and mark two deliberate gateway imports.
- Modify `src/core/postgres-engine.ts` — hoist eight safe import statements and mark two deliberate gateway imports.
- Modify `src/core/migrate.ts` — hoist two safe migration helper import statements.
- Modify `package.json` — expose `check:engine-dynamic-import` and append it to `check:all` through `bash`.
- Modify `scripts/run-verify-parallel.sh` — add the package check to the authoritative verify dispatcher.
- Modify `CLAUDE.md` — add the cross-cutting current-state invariant.
- Modify `docs/architecture/KEY_FILES.md` — update current-state entries for the three engine-path files.
- Regenerate `llms.txt` and `llms-full.txt` — required derived bundles after CLAUDE/reference documentation changes.
---
### Task 1: Establish and enforce the source invariant
**Files:**
- Create: `scripts/check-engine-dynamic-import.sh`
- Create: `scripts/check-engine-dynamic-import.ts`
- Create: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `src/core/pglite-engine.ts`
- Modify: `src/core/postgres-engine.ts`
- Modify: `src/core/migrate.ts`
**Interfaces:**
- Consumes: shell positional arguments `FILE...`; without arguments, the guard scans the three repository files.
- Produces: `scripts/check-engine-dynamic-import.sh [FILE...]`, exit `0` when every runtime dynamic import is allowed and exit `1` after reporting every `file:line:text` violation plus every read/parse error on stderr.
- Produces: one line-level opt-out token, `engine-dynamic-import-ok`, accepted only in real comment trivia on the same physical line as the deliberately lazy import.
- Fails closed on missing/unreadable inputs, TypeScript parse diagnostics, and scanner/process failures; comments, strings, templates, regex literals, and type-position `import(...)` syntax are not runtime imports.
- [ ] **Step 1: Record call-graph blast radius before touching functions**
First call `sources_list` and select the source whose registered path is this gbrain checkout. Then run `code_blast` and `code_callers` for these qualified symbols with that exact `source_id`, following `did_you_mean`/`candidates` when a method name is ambiguous:
```text
src/core/pglite-engine.ts::PGLiteEngine.initSchema
src/core/pglite-engine.ts::PGLiteEngine.batchRetry
src/core/pglite-engine.ts::PGLiteEngine._upsertChunksOnce
src/core/pglite-engine.ts::PGLiteEngine.mergeOntologyFact
src/core/pglite-engine.ts::PGLiteEngine.getRecentSalience
src/core/postgres-engine.ts::PostgresEngine.disconnect
src/core/postgres-engine.ts::PostgresEngine.initSchema
src/core/postgres-engine.ts::PostgresEngine.batchRetry
src/core/postgres-engine.ts::PostgresEngine._upsertChunksOnce
src/core/postgres-engine.ts::PostgresEngine.mergeOntologyFact
src/core/postgres-engine.ts::PostgresEngine.reconnect
src/core/postgres-engine.ts::PostgresEngine.getRecentSalience
src/core/migrate.ts::runMigrationSQLWithRetry
src/core/migrate.ts::runMigrations
```
Use `depth: 5`, `max_nodes: 200`, and `limit: 100`. Expected: no caller requires a signature or behavior change; the patch only changes module binding time and retains all local fallback/error handling.
- [ ] **Step 2: Write the failing guard regression test**
Create `test/scripts/check-engine-dynamic-import.test.ts` as a hermetic subprocess suite. The completed 22-test surface covers:
- unmarked runtime `import()` rejection, including bare and trivia-separated forms;
- same-line markers in real line or multiline block-comment trivia;
- rejection of markers on prior lines or inside strings, templates, and module paths;
- comments and comment-like delimiters inside strings, templates, and regex literals;
- live code after same-line or multiline block comments close;
- CRLF input and complete multi-file violation aggregation;
- missing/readable mixed inputs and TypeScript parse diagnostics;
- default repository anchoring when invoked from a foreign Git repository;
- the reconciled three-file source scan plus package/parallel-verifier wiring.
Use the TypeScript parser rather than a partial lexical reimplementation. On Windows, set the test default to 30 seconds because each case launches Git Bash and Bun, whose startup can exceed Bun's 5-second per-test default.
- [ ] **Step 3: Run the test to prove the pre-implementation red state**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result captured inside the log. At minimum, the `exists` assertion fails because `scripts/check-engine-dynamic-import.sh` does not exist. Read `.context/engine-dynamic-import-red.txt`; do not infer the result from a truncated pipeline.
- [ ] **Step 4: Add the CRLF-safe, fail-closed guard**
Create `scripts/check-engine-dynamic-import.sh` as a thin LF-terminated wrapper. Resolve its own directory first; when no explicit files are passed, anchor the repository with `git -C "$SCRIPT_DIR/.."` and scan the two engines plus `migrate.ts`. Delegate with `exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"` so scanner failures propagate.
Create `scripts/check-engine-dynamic-import.ts` using the TypeScript compiler API:
- read every requested file and aggregate read failures;
- parse as TypeScript and aggregate parse diagnostics;
- walk the AST for `CallExpression`s whose expression is `ImportKeyword`;
- locate all marker occurrences in the full source and use `ts.getTokenAtPosition` to admit only occurrences outside AST tokens (real comment trivia), recording their physical source lines;
- require each runtime import's line to have an admitted marker or report its original `file:line:text`;
- print every read/parse error and every violation before exiting nonzero.
This preserves CRLF line accounting, ignores comment/literal/type-only false positives, catches every legal runtime `import()` shape the TypeScript parser recognizes, rejects marker spoofing, and fails closed.
- [ ] **Step 5: Run the guard test to prove the source-tree midpoint is still red**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-midpoint.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: the synthetic violation, marker, comments, and CRLF cases pass. The default repository scan fails and reports all 17 current imports: 13 unmarked safe candidates plus the four not-yet-marked gateway calls.
- [ ] **Step 6: Hoist the three safe PGLite import statements**
Replace the existing `retry.ts` import and add the ontology/recency imports near the top of `src/core/pglite-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
```
Delete only these three in-method destructuring imports, leaving their uses unchanged:
```ts
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
- [ ] **Step 7: Mark both PGLite gateway soft-failure boundaries**
In `PGLiteEngine.initSchema`, preserve the `try/catch` and accessors, changing only the rationale and import line:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not configured — use defaults */ }
```
In `PGLiteEngine._upsertChunksOnce`, preserve the config-row and compile-time fallback chain:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 8: Hoist the eight safe Postgres import statements**
Replace the existing `retry.ts` import and add these imports near the top of `src/core/postgres-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logDbDisconnect } from './audit/db-disconnect-audit.ts';
import { logPoolRecovery } from './audit/pool-recovery-audit.ts';
```
Delete the eight safe dynamic-import statements while keeping their surrounding `try/catch` blocks and calls unchanged:
```ts
const { logDbDisconnect } = await import('./audit/db-disconnect-audit.ts');
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { isConnectionEndedError } = await import('./retry-matcher.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
Update the stale `batchRetry` comment from “Lazy-import to avoid a circular dep concern” to current truth:
```ts
// retry.ts is already in this module's static graph through withRetry, so
// classifying the exhausted error does not need a second runtime import.
```
- [ ] **Step 9: Mark both Postgres gateway soft-failure boundaries**
In `PostgresEngine.initSchema`, mirror the PGLite rationale and preserve behavior:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not yet configured — use defaults */ }
```
In `PostgresEngine._upsertChunksOnce`, preserve the DB-config fallback:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 10: Hoist the two migration helper import statements**
Add these static imports at the top of `src/core/migrate.ts`:
```ts
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
isStatementTimeoutError,
isRetryableConnError,
} from './retry-matcher.ts';
import { repairTimelineDedupIndex } from './timeline-dedup-repair.ts';
```
Delete only these two local destructuring imports:
```ts
const { isStatementTimeoutError, isRetryableConnError } = await import('./retry-matcher.ts');
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
```
- [ ] **Step 11: Run the complete guard test and direct guard**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; output contains `check-engine-dynamic-import: ok (3 file(s) scanned)`.
- [ ] **Step 12: Prove the guard leaves exactly four marked dynamic imports**
```bash
git grep -n -F "import('./ai/gateway.ts'); // engine-dynamic-import-ok" -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts > .context/engine-dynamic-import-sites.txt; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exactly four lines, all importing `./ai/gateway.ts` and all carrying `engine-dynamic-import-ok`; no match in `src/core/migrate.ts`.
- [ ] **Step 13: Run focused behavior tests**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. If Windows resource pressure aborts the process, record the exact exit code and rerun the failing file alone; do not relabel an infrastructure abort as a source pass.
- [ ] **Step 14: Commit the source invariant locally**
```bash
git add scripts/check-engine-dynamic-import.sh scripts/check-engine-dynamic-import.ts test/scripts/check-engine-dynamic-import.test.ts src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts
```
```bash
git commit -m "fix(engine): reconcile dynamic import hardening"
```
Expected: one local commit; no version or release files staged.
---
### Task 2: Wire the guard into repository checks
**Files:**
- Modify: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `package.json`
- Modify: `scripts/run-verify-parallel.sh`
**Interfaces:**
- Consumes: `scripts/check-engine-dynamic-import.sh` from Task 1.
- Produces: package script `check:engine-dynamic-import` and verify dry-list entry of the same name.
- [ ] **Step 1: Add failing wiring assertions**
Add these imports/constants to `test/scripts/check-engine-dynamic-import.test.ts`:
```ts
const PACKAGE_JSON = resolve(REPO_ROOT, 'package.json');
```
Append this test block:
```ts
describe('engine dynamic-import guard wiring', () => {
it('is invoked through bash by check:all', () => {
const pkg = JSON.parse(readFileSync(PACKAGE_JSON, 'utf8')) as {
scripts: Record<string, string>;
};
expect(pkg.scripts['check:engine-dynamic-import']).toBe(
'bash scripts/check-engine-dynamic-import.sh',
);
expect(pkg.scripts['check:all']).toContain(
'bash scripts/check-engine-dynamic-import.sh',
);
});
it('is listed by the authoritative verify dispatcher', () => {
const result = spawnSync(BASH, [VERIFY_DISPATCHER, '--dry-list'], {
cwd: REPO_ROOT,
encoding: 'utf8',
timeout: 30_000,
});
expect(result.status).toBe(0);
expect(new Set((result.stdout ?? '').trim().split('\n'))).toContain(
'check:engine-dynamic-import',
);
});
});
```
- [ ] **Step 2: Run the test and verify both wiring assertions fail**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result. The source guard tests remain green; package-script and verify-list assertions fail because the wiring is absent.
- [ ] **Step 3: Add the package scripts**
In `package.json`, add this script alongside the other `check:*` entries:
```json
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh"
```
Append the guard to the existing `check:all` chain, preserving every existing check:
```text
&& bash scripts/check-engine-dynamic-import.sh
```
Do not rewrite any existing shell entry without its `bash` prefix.
- [ ] **Step 4: Add the authoritative verify entry**
In `scripts/run-verify-parallel.sh`, add this stable `CHECKS` entry near the other source-shape guards:
```bash
"check:engine-dynamic-import"
```
- [ ] **Step 5: Run the regression test and package check**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bun run check:engine-dynamic-import > .context/engine-dynamic-import-package-check.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0` and three files scanned.
- [ ] **Step 6: Commit the wiring locally**
```bash
git add package.json scripts/run-verify-parallel.sh test/scripts/check-engine-dynamic-import.test.ts
```
```bash
git commit -m "test(engine): guard dynamic import policy"
```
Expected: one local commit with the guard wiring and its regression assertions.
---
### Task 3: Document the current-state invariant
**Files:**
- Modify: `CLAUDE.md`
- Modify: `docs/architecture/KEY_FILES.md`
- Regenerate: `llms.txt`
- Regenerate: `llms-full.txt`
**Interfaces:**
- Consumes: the four-marked-import source state and the `check:engine-dynamic-import` package surface.
- Produces: current-state contributor guidance and fresh generated documentation bundles.
- [ ] **Step 1: Add the cross-cutting invariant to `CLAUDE.md`**
Add this bullet under “Cross-cutting invariants” near the other language/filesystem guards:
```md
- **Engine-live paths use static imports by default.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, helper modules are top-level imports. The only current
exceptions are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
```
Do not add release tags, Windows-crash certainty, or historical branch names.
- [ ] **Step 2: Update the PGLite current-state entry in `KEY_FILES.md`**
Append this current-state sentence to the existing `src/core/pglite-engine.ts` entry, preserving the entry as one bullet:
```md
Engine-path helper dependencies (`retry`, ontology, recency decay) bind statically; the only lazy imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass.
```
- [ ] **Step 3: Update the Postgres current-state entry in `KEY_FILES.md`**
Append this sentence to the existing `src/core/postgres-engine.ts` entry:
```md
Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite.
```
- [ ] **Step 4: Update the migration current-state entry in `KEY_FILES.md`**
Append this sentence to the canonical `src/core/migrate.ts` entry (the broad runner entry, not the older v95-specific index note):
```md
`retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations.
```
Keep all three entries current-state only: no `v0.42.x`, branch, commit, “previously,” or “was/now” narration.
- [ ] **Step 5: Regenerate the llms bundles**
```bash
bun run build:llms > .context/engine-dynamic-import-build-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; `llms.txt` and/or `llms-full.txt` update according to their configured linked/inlined status. Byte-identical output for a linked source is acceptable; the freshness test is authoritative.
- [ ] **Step 6: Run documentation freshness checks**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-llms-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
```bash
bun run check:doc-history > .context/engine-dynamic-import-doc-history.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; no release-history marker is introduced into current-state reference docs.
- [ ] **Step 7: Confirm prohibited release files remain untouched**
```bash
git diff --name-only d7f52d8c..HEAD -- VERSION CHANGELOG.md TODOS.md
```
Expected: no output.
- [ ] **Step 8: Commit documentation and generated bundles locally**
```bash
git add CLAUDE.md docs/architecture/KEY_FILES.md llms.txt llms-full.txt
```
```bash
git commit -m "docs(engine): record static import invariant"
```
Expected: one local documentation commit. If one generated bundle is byte-identical, Git simply omits it.
---
### Task 4: Verify and review the complete local reconciliation
**Files:**
- Verify all files changed since `d7f52d8c`.
- Do not create or modify release/publication metadata.
**Interfaces:**
- Consumes: Tasks 13.
- Produces: full local verification evidence and an implementation diff ready for user review, not publication.
- [ ] **Step 1: Run the regression test and direct guard again**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-final-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-final-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; three files scanned.
- [ ] **Step 2: Run TypeScript checking**
```bash
bun run typecheck > .context/engine-dynamic-import-typecheck.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. Report exact diagnostics if the branch or current Windows environment has a pre-existing failure.
- [ ] **Step 3: Run the authoritative verify dispatcher**
```bash
bun run verify > .context/engine-dynamic-import-verify.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`, including `check:engine-dynamic-import`. On Windows, classify any per-check timeout from the complete log instead of treating the aggregate result as a source regression without evidence.
- [ ] **Step 4: Re-run focused tests as an ownership check**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-final-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; record any infrastructure abort separately and rerun only the named file before classifying it.
- [ ] **Step 5: Run the llms freshness test after all documentation settles**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-final-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
- [ ] **Step 6: Run whitespace and scope checks**
```bash
git diff --check d7f52d8c..HEAD
```
Expected: exit `0`, no output.
```bash
git diff --name-only d7f52d8c..HEAD
```
Expected files only:
```text
CLAUDE.md
docs/architecture/KEY_FILES.md
docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
llms-full.txt
llms.txt
package.json
scripts/check-engine-dynamic-import.sh
scripts/check-engine-dynamic-import.ts
scripts/run-verify-parallel.sh
src/core/migrate.ts
src/core/pglite-engine.ts
src/core/postgres-engine.ts
test/scripts/check-engine-dynamic-import.test.ts
```
Either generated llms file may be absent if regeneration proves it byte-identical. `VERSION`, `CHANGELOG.md`, and `TODOS.md` must be absent.
- [ ] **Step 7: Review the exact implementation diff**
```bash
git diff --stat d7f52d8c..HEAD && git diff d7f52d8c..HEAD -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts scripts/check-engine-dynamic-import.sh test/scripts/check-engine-dynamic-import.test.ts package.json scripts/run-verify-parallel.sh CLAUDE.md docs/architecture/KEY_FILES.md
```
Expected review findings:
- Exactly 13 safe `await import(...)` statements are removed.
- Exactly four `ai/gateway.ts` imports remain, all marked on the same line.
- All four gateway imports remain inside their original local `try/catch` fallback boundaries.
- No accessor logic, fallback ordering, SQL, public signature, or engine parity behavior changes.
- The parser-backed guard reports all violations plus read/parse failures, preserves CRLF line accounting, ignores comments/literals/type-only syntax, detects every runtime `import()` call expression, and accepts opt-outs only from real comment trivia on the same physical line.
- The package script invokes the shell guard through Bash; `check:all` invokes that shell guard directly, and the parallel verify dispatcher invokes the package check.
- Documentation is current-state and makes no deterministic Windows-crash claim.
**Observed Windows verification classification:** The authoritative aggregate completed with 25 of 33 checks passing. Individual reruns showed `check:test-names` and `typecheck` green; privacy/isolation exceeded Windows timing budgets; WASM failed in unrelated temporary-symlink setup; eval-glossary was CRLF/LF drift; resolver/brain-first findings predated and did not intersect this branch. The focused aggregate produced 103 pass / 5 fail: three setup-hook timeouts reproduced at the untouched base, and the known `migrate-retry` polling failure reproduced there. Its additional race-status assertion did not reproduce at base, so it remains an unresolved timing-sensitive limitation in untouched code—not evidence of an in-scope defect and not claimed as conclusively pre-existing.
- [ ] **Step 8: Commit the approved plan document locally**
The plan is an approved, tracked execution artifact and must not be left as an uncommitted file after implementation:
```bash
git add docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
```
```bash
git commit -m "docs: plan engine dynamic-import reconciliation"
```
Expected: one local plan commit; no release metadata staged.
- [ ] **Step 9: Inspect final status without publishing**
```bash
git status --short --branch
```
Expected: branch `claude/kind-meitner-330c90` with a clean working tree. No push, PR, upstream comment, or other external side effect.
- [ ] **Step 10: Capture the completed milestone to memory**
Before writing, search MemPalace wing `gbrain` for this exact reconciliation to avoid duplication. Add a verbatim drawer recording exact base/head commits, the 13 hoists, four gateway opt-outs and rationale, guard/test/docs files, every verification command with exit code, and any environment-owned failures. Add a GBrain project timeline entry only if there is an existing relevant gbrain project page; do not create duplicate release metadata.
- [ ] **Step 11: Report the local result and ask separately before publication**
Report:
- exact local commits;
- changed files;
- test/check exit codes;
- any blocked or pre-existing failures;
- confirmation that release files were untouched;
- confirmation that nothing was pushed or published.
Do not run any publication command. Wait for explicit user approval before any push, PR, or upstream interaction.
@@ -0,0 +1,142 @@
# Engine dynamic-import reconciliation design
**Date:** 2026-07-28
## Goal
Reconcile the overlapping engine dynamic-import changes from:
- `claude/hungry-edison-8bb1cd` at release commits `48ada48f` and `248bfe55`
- `claude/elegant-gates-e5275e` at `ef4cf7a8`
onto a fresh branch from current `origin/master`, without merging or cherry-picking either lineage wholesale and without adding a release/version bump.
## Established state
At investigation time:
- `origin/master` was `6136e139972a5449630b4f47f5ed7b4cbe5b811b`, version `0.42.67.0`.
- Upstream PR #3511 was still open, so trunk did not contain its two `chronicle/ontology.ts` hoists.
- Neither source branch was an ancestor of trunk.
- Trunk contained 17 dynamic imports in the three engine-path files:
- 13 safe-hoist candidates: two ontology imports, nine engine helper/audit imports, and two migration imports.
- Four `ai/gateway.ts` imports, all inside `try/catch` fallback paths.
- `git log -G` showed the separate ontology, helper, migration, and gateway histories. `git log -S` is not suitable for this dynamic-to-static replacement because the relevant token can remain present while its context changes.
- The guard from `ef4cf7a8` passed against that commit but failed against trunk. It also knew about only two gateway opt-outs because two `_upsertChunksOnce` gateway lookups landed later in trunk.
## Selected approach
Reconstruct the intended current state directly on fresh `origin/master`.
Do not merge or cherry-pick either old lineage. Selectively reproduce the desired source changes, adapt the guard to the current four gateway call sites, and write current-state documentation. This avoids importing stale release metadata, stale TODO claims, and unrelated lineage changes.
## Source changes
### Safe static imports
Hoist all 13 safe candidates:
- `src/core/pglite-engine.ts`
- `valueHash`, `normalizeDimension`, `isNovelDimension` from `chronicle/ontology.ts`
- `isRetryableConnError` through the existing `retry.ts` import
- `resolveRecencyDecayMap`, `DEFAULT_FALLBACK` from `search/recency-decay.ts`
- `src/core/postgres-engine.ts`
- the same ontology, retry, and recency helpers
- `isConnectionEndedError` from `retry-matcher.ts`
- `logDbDisconnect` from `audit/db-disconnect-audit.ts`
- `logPoolRecovery` from `audit/pool-recovery-audit.ts`
- `src/core/migrate.ts`
- `isStatementTimeoutError`, `isRetryableConnError` from `retry-matcher.ts`
- `repairTimelineDedupIndex` from `timeline-dedup-repair.ts`
The implementation must keep the two engines in parity where the behavior is shared. Comments should describe current invariants, not repeat an unproven causal claim that these hoists fix the Windows test-runner crash.
### Deliberately lazy gateway imports
Keep all four `await import('./ai/gateway.ts')` call sites lazy:
- PGLite `initSchema`
- PGLite `_upsertChunksOnce`
- Postgres `initSchema`
- Postgres `_upsertChunksOnce`
Each line receives the explicit `engine-dynamic-import-ok` marker and a concise nearby rationale.
The rationale has two parts:
1. The gateway's static closure includes the AI SDK, provider packages, and validation/config machinery, so eager loading would tax engine startup paths that do not otherwise need it.
2. More importantly, each lookup is inside a `try/catch` that preserves a soft fallback (compiled defaults or the brain's stored embedding-model config). Hoisting the module would evaluate it before that catch can run and could convert a recoverable configuration/import failure into a module-load-time hard failure.
The guard must not allow unmarked gateway imports or a broad file-level exemption.
## Guard and wiring
Add `scripts/check-engine-dynamic-import.sh`, adapted from `ef4cf7a8`, with these properties:
- Default scan set:
- `src/core/pglite-engine.ts`
- `src/core/postgres-engine.ts`
- `src/core/migrate.ts`
- Normalize trailing CR before matching so CRLF checkouts cannot bypass the check.
- Ignore comment-only lines.
- Ignore only lines carrying `engine-dynamic-import-ok`.
- Report every unmarked `await import(` with file and line.
- Explain that contributors should prefer a static import and must justify a real opt-out.
- Avoid asserting that every dynamic import deterministically crashes Windows; the measured evidence supports treating the pattern as an engine-path hardening invariant, while box-level commit exhaustion remained a confound in prior runs.
Wire it into:
- `package.json` as `check:engine-dynamic-import`
- `package.json` `check:all`
- `scripts/run-verify-parallel.sh`
Follow trunk's current rule that package scripts invoke repository shell scripts through `bash`.
## Regression coverage
Add an automated test for the guard. It must cover:
- A real dynamic import produces exit 1 and is reported.
- A line carrying `engine-dynamic-import-ok` is allowed.
- Line comments and block-comment lines do not produce findings.
- The same violation is caught with CRLF input.
- The default repository scan passes after the source reconciliation.
Use a temporary fixture rather than mutating tracked source files. Keep assertions path-portable.
The pre-fix red demonstration is the exact guard from `ef4cf7a8` run against current trunk: it exits 1 and reports the existing unmarked imports. The post-fix guard and test must pass.
## Documentation policy
Preserve current behavior, not either old release narrative:
- Do not modify `VERSION` or add a release `CHANGELOG.md` entry.
- Do not copy old version headings or completed release TODO blocks.
- Do not retain the old TODO claiming that extracting gateway accessors is necessarily the fix; the lazy imports are deliberately protected by their local soft-failure boundaries.
- Add the cross-cutting no-unmarked-dynamic-import invariant to `CLAUDE.md`.
- Update the current-state entries for `src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and `src/core/migrate.ts` in `docs/architecture/KEY_FILES.md` where needed.
- Regenerate `llms.txt` and `llms-full.txt` after the documentation edits.
- Add a TODO only if implementation uncovers a real unresolved action.
Public documentation must use generic language and must not overstate the historical Windows crash causality.
## Verification
Capture full output to files before inspecting summaries. Run, at minimum:
1. The guard regression test.
2. `bash scripts/check-engine-dynamic-import.sh`.
3. Focused tests that exercise the touched engine, migration, retry, audit, and recency modules.
4. `bun run typecheck`.
5. `bun run verify`.
6. `bun run build:llms` followed by `bun test test/build-llms.test.ts`.
7. `git diff --check` and a final clean-status/diff review.
If platform contention or existing Windows suite defects block a broad test, report the exact command, exit code, and ownership classification rather than declaring success from a partial run.
## Git and publication boundary
- Work on `claude/kind-meitner-330c90`, reset locally to the exact investigated `origin/master` base.
- Preserve the previous worktree tip under `claude/kind-meitner-330c90-pre-reconcile`.
- Keep implementation and verification commits local.
- Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after the local result is complete.
+13
View File
@@ -216,6 +216,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
+35 -33
View File
@@ -42,20 +42,21 @@
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bash scripts/run-verify-parallel.sh",
"check:source-config-leak": "scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-key-files-current-state.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "scripts/check-key-files-current-state.sh",
"check:source-config-leak": "bash scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "bash scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "bash scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "bash scripts/check-system-of-record.sh",
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
"check:cli-exec": "bash scripts/check-cli-executable.sh",
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"check:skill-brain-first": "bash scripts/check-skill-brain-first.sh",
"check:wasm": "bash scripts/check-wasm-embedded.sh",
"check:newlines": "bash scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:heavy": "bash scripts/run-heavy.sh",
@@ -65,26 +66,27 @@
"ci:local:diff": "bash scripts/ci-local.sh --diff",
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:search-path": "scripts/check-search-path.sh",
"check:no-double-retry": "scripts/check-no-double-retry.sh",
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:proposal-pii": "scripts/check-proposal-pii.sh",
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
"check:exports-count": "scripts/check-exports-count.sh",
"check:admin-build": "scripts/check-admin-build.sh",
"check:admin-embedded": "scripts/check-admin-embedded.sh",
"check:test-isolation": "scripts/check-test-isolation.sh",
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:jsonb": "bash scripts/check-jsonb-pattern.sh",
"check:search-path": "bash scripts/check-search-path.sh",
"check:no-double-retry": "bash scripts/check-no-double-retry.sh",
"check:batch-audit-site": "bash scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "bash scripts/check-worker-lock-renewal-shape.sh",
"check:source-id-projection": "bash scripts/check-source-id-projection.sh",
"check:privacy": "bash scripts/check-privacy.sh",
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
"check:test-names": "bash scripts/check-test-real-names.sh",
"check:progress": "bash scripts/check-progress-to-stdout.sh",
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
"check:exports-count": "bash scripts/check-exports-count.sh",
"check:admin-build": "bash scripts/check-admin-build.sh",
"check:admin-embedded": "bash scripts/check-admin-embedded.sh",
"check:test-isolation": "bash scripts/check-test-isolation.sh",
"check:fuzz-purity": "bash scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
"postinstall": "bun run scripts/postinstall.ts",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
@@ -145,7 +147,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.66.1",
"version": "0.42.67.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard: every `bun test` invocation in workflows and runner scripts must
# pass an explicit --timeout.
#
# Why: bun ignores bunfig.toml's `timeout` key (verified on 1.3.14), so a bare
# `bun test` gets the 5000ms default for BOTH tests and beforeAll/beforeEach/
# afterAll/afterEach hooks. Hooks do NOT inherit a test's third-arg timeout —
# a file whose tests all declare `}, 30_000)` still has a 5s hook budget, and
# slow setup (Postgres connect + migrations, PGLite cold start) flakes on
# loaded CI runners with the signature `(unnamed) [5001ms] ... hook timed out`
# (the #3545 jsonb-parity failure). The CLI --timeout flag is the one measured
# mechanism that raises the hook budget uniformly; per-hook second-arg
# timeouts work too but don't scale to ~400 slow hooks.
#
# Usage: scripts/check-bun-test-timeout.sh
# Exit: 0 when clean, 1 when a bare `bun test` invocation is found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Match executable `bun test` invocations. Exclude comment lines (#, //, *)
# and lines that already carry --timeout anywhere.
# Scope: workflows + runner scripts (the surfaces CI executes). package.json
# script bodies route through scripts/ already; editing it is out of scope here.
violations="$(grep -rnE '\bbun test\b' .github/workflows scripts 2>/dev/null \
| grep -v -- '--timeout' \
| grep -vE ':[[:space:]]*(#|//|\*)' \
| grep -v 'check-bun-test-timeout' \
|| true)"
if [ -n "$violations" ]; then
echo "FAIL: bare 'bun test' without --timeout (5s default kills slow setup hooks):" >&2
echo "$violations" >&2
echo "" >&2
echo "Add --timeout=60000 (see scripts/run-unit-shard.sh for the convention)." >&2
exit 1
fi
echo "OK: every bun test invocation passes an explicit --timeout."
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Engine-live paths use static imports by default. A line-level
# `engine-dynamic-import-ok` marker is required for a justified lazy import.
#
# Historical Windows runs associated imports on these paths with abrupt Bun
# test-process exits, but system-wide commit exhaustion remained a confound.
# This guard therefore enforces a reviewed engine-path hardening invariant; it
# does not claim every dynamic import deterministically crashes Windows.
#
# Usage:
# bash scripts/check-engine-dynamic-import.sh
# bash scripts/check-engine-dynamic-import.sh FILE [FILE...]
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" || exit 1
if [ "$#" -gt 0 ]; then
FILES=("$@")
else
ROOT="$(git -C "$SCRIPT_DIR/.." rev-parse --show-toplevel 2>/dev/null || true)"
[ -n "$ROOT" ] || ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
cd "$ROOT" || exit 1
FILES=(
src/core/pglite-engine.ts
src/core/postgres-engine.ts
src/core/migrate.ts
)
fi
exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bun
import { readFile } from 'node:fs/promises';
import ts from 'typescript';
const MARKER = 'engine-dynamic-import-ok';
const MARKER_TOKEN_CHAR = /[\p{ID_Continue}$-]/u;
const files = process.argv.slice(2);
const violations: string[] = [];
const readErrors: string[] = [];
for (const file of files) {
let sourceText: string;
try {
sourceText = await readFile(file, 'utf8');
} catch (error) {
const detail = error instanceof Error ? error.message : String(error);
readErrors.push(`ERROR: cannot read input file ${file}: ${detail}`);
continue;
}
const sourceFile = ts.createSourceFile(
file,
sourceText,
ts.ScriptTarget.Latest,
true,
ts.ScriptKind.TS,
);
const lines = sourceText.split(/\r?\n/);
const markerLines = new Set<number>();
if (sourceFile.parseDiagnostics.length > 0) {
const diagnostics = sourceFile.parseDiagnostics
.map((diagnostic) => ts.flattenDiagnosticMessageText(diagnostic.messageText, ' '))
.join('; ');
readErrors.push(`ERROR: cannot parse input file ${file}: ${diagnostics}`);
}
for (let markerPos = sourceText.indexOf(MARKER); markerPos >= 0; markerPos = sourceText.indexOf(MARKER, markerPos + MARKER.length)) {
const before = Array.from(sourceText.slice(0, markerPos)).at(-1);
const after = Array.from(sourceText.slice(markerPos + MARKER.length))[0];
const standaloneMarker = (!before || !MARKER_TOKEN_CHAR.test(before))
&& (!after || !MARKER_TOKEN_CHAR.test(after));
const token = ts.getTokenAtPosition(sourceFile, markerPos);
const insideToken = token.getStart(sourceFile) <= markerPos && markerPos < token.end;
if (standaloneMarker && !insideToken) {
markerLines.add(sourceFile.getLineAndCharacterOfPosition(markerPos).line);
}
}
function visit(node: ts.Node): void {
if (ts.isCallExpression(node) && node.expression.kind === ts.SyntaxKind.ImportKeyword) {
const { line } = sourceFile.getLineAndCharacterOfPosition(node.expression.getStart(sourceFile));
const sourceLine = lines[line] ?? '';
if (!markerLines.has(line)) {
violations.push(` ${file}:${line + 1}:${sourceLine}`);
}
}
ts.forEachChild(node, visit);
}
visit(sourceFile);
}
for (const error of readErrors) console.error(error);
if (violations.length > 0) {
console.error('ERROR: unreviewed dynamic import on an engine-live path:');
console.error();
console.error(violations.join('\n'));
console.error();
console.error('Prefer a static top-level import. If lazy loading is load-bearing,');
console.error("append 'engine-dynamic-import-ok' to that exact line and document");
console.error('the startup or soft-failure boundary that requires it.');
process.exit(1);
}
if (readErrors.length > 0) process.exit(1);
console.log(`check-engine-dynamic-import: ok (${files.length} file(s) scanned)`);
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# CI guard: fail if any symlink is tracked in git.
#
# A symlink committed from a build sandbox points at a path that exists on
# exactly one machine. Everywhere else the checkout produces a dangling
# link, and anything that opens it fails. That is not hypothetical: commit
# faf5cdba landed `node_modules -> /tmp/fleet/repo/node_modules`, which made
# `bun install` abort with `ENOENT: could not open the "node_modules"
# directory` on every fresh clone, and took `gbrain upgrade`'s bun-link path
# down with it (the auto-upgrade runs `bun install`, so the printed manual
# fallback failed the same way).
#
# .gitignore alone does not prevent this. A `node_modules/` pattern with a
# trailing slash matches directories ONLY, so a symlink of the same name is
# never ignored. Dropping the slash closes that hole, but `git add -f` still
# walks straight past it. This guard is the backstop.
#
# The repo has no legitimate tracked symlinks, so the allowlist starts
# empty. If you ever need one, add its exact repo-relative path to ALLOWLIST
# below and explain why — a relative link that resolves inside the repo is
# defensible; an absolute one almost never is.
#
# Usage: scripts/check-no-tracked-symlinks.sh
# Exit: 0 when clean, 1 when a tracked symlink is found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Paths permitted to be tracked symlinks. Empty by design.
ALLOWLIST=()
# Git records symlinks with mode 120000. Field 4 of `ls-files -s` is the path
# (tab-separated from the stage number), so cut on the tab to keep paths with
# spaces intact.
found="$(git ls-files -s | awk '$1 == "120000"' | cut -f2- || true)"
if [ -n "$found" ]; then
filtered="$found"
for f in "${ALLOWLIST[@]:-}"; do
[ -z "$f" ] && continue
filtered="$(echo "$filtered" | grep -vxF "$f" || true)"
done
if [ -n "$filtered" ]; then
echo "ERROR: symlink(s) tracked in git:"
echo
while IFS= read -r path; do
[ -z "$path" ] && continue
target="$(git cat-file blob ":$path" 2>/dev/null || echo '<unreadable>')"
echo " $path -> $target"
done <<< "$filtered"
echo
echo "A committed symlink resolves on the machine that created it and"
echo "nowhere else. Untrack it:"
echo
echo " git rm --cached <path>"
echo
echo "If the path is build output (node_modules, dist, bin), also confirm"
echo "it is covered by .gitignore WITHOUT a trailing slash — a trailing"
echo "slash matches directories only and lets the symlink through."
exit 1
fi
fi
echo "check-no-tracked-symlinks: OK (no tracked symlinks)"
+11 -1
View File
@@ -46,7 +46,15 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
"test/e2e/multi-source-bug-class.test.ts",
"test/e2e/synthesize-bigint-job-id-postgres.test.ts",
],
"src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/commands/embed.ts": [
"test/e2e/multi-source-bug-class.test.ts",
// #3391: the NULL-signature stale predicates differ per engine.
"test/e2e/migrate-embeddings-postgres.test.ts",
],
// #3390: runSchemaTransition's DDL path + the stale predicates behave
// differently on real pgvector than on PGLite.
"src/core/embedding-migration.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
"src/core/retrieval-upgrade-planner.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
"src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"],
// Any minions queue/worker/handler change exercises all minion E2E.
@@ -64,6 +72,8 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
"test/e2e/jsonb-roundtrip.test.ts",
"test/e2e/engine-parity.test.ts",
"test/e2e/schema-drift.test.ts",
// #3391: includeNullSignature stale predicates (engine parity).
"test/e2e/migrate-embeddings-postgres.test.ts",
],
// PGLite bootstrap path + parity guard.
"src/core/pglite-engine.ts": [
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env node
/**
* Import an envelope-v0 file (a JSON serialization of AI chat history; format
* spec: github.com/memvelope/memvelope) into a brain repo as one Markdown page
* per conversation, which `gbrain sync` ingests.
*
* Usage:
* node scripts/envelope-to-gbrain.mjs <envelope.mve.json> [outDir]
*
* Zero dependencies. Deterministic. No network. It does NOT call gbrain — it
* only writes Markdown files.
*
* Output layout:
* - One page per conversation, filename = date + conversation id (shared
* titles cannot collide; the id is the natural key). A duplicate id
* overwrites its own filename and warns on stderr; stdout reports DISTINCT
* files written, not write calls.
* - Frontmatter: `type: conversation` (keeps pages eligible for
* conversation-facts extraction and chronicle behavior after sync), the
* source provider, the conversation id, and `origin: memvelope/envelope-v0`.
* - Page `date` is the first 10 chars of the conversation's ISO-8601
* `created_at`. Body keeps message-id citations beside each speaker turn.
*
* Memory: the whole envelope is held in memory (no streaming); envelopes are
* far smaller than the vendor exports they serialize.
*
* Verify:
* node scripts/envelope-to-gbrain.mjs test/fixtures/memvelope/sample.mve.json /tmp/out
* -> expect "wrote 1 markdown page(s)"
* bun test test/envelope-to-gbrain.test.ts
*
* STATUS: live-verified against gbrain v0.42.56.0 on 2026-07-03: the sample
* fixture -> 1 page; a real 662MB Claude export -> 353 conversations = 353
* distinct pages (no collisions), searchable after sync with provenance and
* message-id citations intact.
*/
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
const [, , envelopePath, outDir = './brain/conversations'] = process.argv;
if (!envelopePath) {
console.error('usage: node envelope-to-gbrain.mjs <envelope.mve.json> [outDir]');
process.exit(1);
}
const env = JSON.parse(readFileSync(envelopePath, 'utf8'));
if (env.memvelope !== 'envelope-v0') {
console.error(`not an envelope-v0 file (memvelope field = ${JSON.stringify(env.memvelope)})`);
process.exit(1);
}
const slug = (s, fallback) =>
(String(s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '') || fallback).slice(0, 60);
mkdirSync(outDir, { recursive: true });
const filesWritten = new Set();
let collisions = 0;
const conversations = env.conversations || [];
for (const [i, c] of conversations.entries()) {
const date = (c.created_at || '').slice(0, 10);
// Name the file by the conversation's own id — the natural unique key — so two
// conversations that share a date and title can never silently overwrite each
// other. The date only leads as a human/chronological sort prefix; the id
// carries uniqueness. Positional fallback keeps names unique and deterministic
// when an envelope omits an id.
// One predicate for "this conversation carries its own id", shared by the
// filename and the frontmatter below. Keeping it in a single place is what
// stops the two from disagreeing about whether an id exists.
const hasId = typeof c.id === 'string' && c.id.trim() !== '';
const convId = hasId ? c.id.trim() : `conv-${i + 1}`;
const name = `${date || '0000-00-00'}-${slug(convId, `conv-${i + 1}`)}.md`;
// gbrain reads YAML frontmatter + markdown body; keep provenance in frontmatter.
// Emit `type: conversation` so gbrain stores these as conversation pages rather
// than defaulting to the generic `concept`. gbrain is open-typed — it takes an
// explicit frontmatter `type` verbatim — and its conversation-aware features
// (conversation-facts extraction, the conversation_format_coverage check,
// chronicle eligibility) key off `type == 'conversation'`.
const front = [
'---',
'type: conversation',
`title: ${JSON.stringify(c.title || 'Untitled conversation')}`,
`date: ${date || 'null'}`,
// Every interpolated value is quoted. An envelope is a third-party file, so
// a provider string carrying a newline would otherwise close this scalar and
// inject arbitrary frontmatter keys into the page gbrain ingests.
`source: ${JSON.stringify(env.meta?.source_provider || 'unknown')}`,
// Omit the key entirely when the envelope carries no id, rather than
// emitting the literal `undefined` or a synthesized `conv-N` — the positional
// fallback names the file, but it is not a memvelope conversation id and
// must not be recorded as one.
...(hasId ? [`memvelope_conversation_id: ${JSON.stringify(convId)}`] : []),
'origin: memvelope/envelope-v0',
'---',
'',
].join('\n');
const body = (c.messages || [])
.map((m) => `**${m.role === 'user' ? 'Me' : 'Assistant'}** (${m.ts || 'no timestamp'} · ${m.id}):\n\n${m.text}`)
.join('\n\n---\n\n');
// Never lose a page silently: if two conversations still map to the same
// filename (e.g. an envelope carrying duplicate ids), warn loudly instead of
// overwriting in silence, and report the count of DISTINCT files written — not
// the number of write calls, which is what hid the old title-collision bug.
if (filesWritten.has(name)) {
collisions += 1;
console.warn(`warning: filename collision on "${name}" — conversation id ${JSON.stringify(c.id)} is not unique; overwriting the earlier page.`);
}
writeFileSync(join(outDir, name), front + `# ${c.title || 'Conversation'}\n\n` + body + '\n');
filesWritten.add(name);
}
console.log(`wrote ${filesWritten.size} markdown page(s) to ${outDir} — point gbrain's sync at this directory.`);
if (collisions) {
console.warn(`warning: ${collisions} filename collision(s) — ${collisions} page(s) overwritten. Deduplicate conversation ids in the envelope to avoid data loss.`);
}
+3 -2
View File
@@ -162,8 +162,9 @@ for f in "${files[@]}"; do
if [ -n "${DATABASE_URL:-}" ]; then
psql "$DATABASE_URL" -At -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND datname = current_database()" >/dev/null 2>&1 || true
fi
# Hard outer timeout (180s per file). bun's --timeout is per-test; if a
# PGLite WASM call hangs in beforeAll/afterAll, --timeout never fires and
# Hard outer timeout (180s per file). bun's --timeout covers tests AND
# hooks (measured on 1.3.14), but it's timer-based: a PGLite WASM call
# that blocks the event loop synchronously never lets the timer fire and
# the file wedges indefinitely. gtimeout/timeout SIGKILLs the file so the
# suite advances. gtimeout (macOS via coreutils) preferred; timeout (Linux)
# fallback; bare bun (no outer cap) if neither is installed.
+2
View File
@@ -42,6 +42,7 @@ CHECKS=(
"check:source-id-projection"
"check:source-config-leak"
"check:progress"
"check:no-tracked-symlinks"
"check:test-isolation"
"check:wasm"
"check:admin-build"
@@ -63,6 +64,7 @@ CHECKS=(
"check:source-scope-onboard"
"check:no-double-retry"
"check:batch-audit-site"
"check:engine-dynamic-import"
"check:worker-lock-renewal-shape"
"typecheck"
)
+136 -14
View File
@@ -9,7 +9,7 @@ installSigchldHandler();
import { installSignalHandlers as installCleanupSignalHandlers } from './core/process-cleanup.ts';
installCleanupSignalHandlers();
import { readFileSync, existsSync, unlinkSync } from 'fs';
import { readFileSync, existsSync, unlinkSync, fstatSync } from 'fs';
import { spawn } from 'child_process';
import {
readUpdateCache,
@@ -55,7 +55,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'backfill']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -107,6 +107,10 @@ const CLI_ONLY_SELF_HELP = new Set([
// `gbrain connect --help` prints its own usage (flags + examples) from
// runConnect; route around the generic one-line short-circuit.
'connect',
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
// (engine transfer) keeps its own dispatch too.
'migrate', 'retrieval-upgrade',
]);
// v114 (#1941): alias -> operation lookup, kept separate from `cliOps` so
@@ -340,6 +344,11 @@ async function main() {
// them out of the engine try/catch is safe and unlocks routing.
const params = parseOpArgs(op, subArgs);
// #3513: stdin fill moved out of parseOpArgs so a non-TTY stdin with no
// piped input can't block the parse forever — the bounded read leaves the
// param unset on timeout and the required-param check below fails fast.
await applyStdinParam(op, params);
// v0.27.1 (`gbrain query --image <path>`): swap the `image` param from
// a filesystem path into base64 bytes + mime. The op accepts base64; the
// CLI accepts a path. Helper is exported so tests can exercise the
@@ -800,18 +809,99 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
}
}
// Read stdin for content params
return params;
}
/**
* #3513: read stdin into an op's stdin-capable param without ever blocking
* forever. The old inline `readFileSync(0)` in parseOpArgs assumed non-TTY
* implies piped content; a non-TTY stdin with NO input (CI step, cron job,
* agent harness holding an unwritten pipe open) blocked the read until kill.
*
* Strategy by fd kind (fstat):
* - TTY: skip, as before (interactive input is not an op-param source).
* - regular file / /dev/null / anything not a pipe or socket: readFileSync
* returns without blocking (`gbrain put x < file`, `< /dev/null` → '').
* - FIFO/socket: stream-read with a deadline on the FIRST byte only. A real
* pipe (`echo foo | gbrain put x`, heredocs) delivers its first byte
* within milliseconds; once any data arrives the deadline is lifted and
* we read to EOF like readFileSync did (slow producers stay supported).
* An empty-but-closed pipe (`: | gbrain put x`) EOFs immediately → ''.
* A pipe that never delivers a byte times out → param stays unset, so
* the existing required-param usage error fires (fail fast, exit 1).
*
* GBRAIN_STDIN_TIMEOUT_MS overrides the first-byte deadline (default 5000).
* Exported for tests; called by the op dispatch right after parseOpArgs.
*/
export async function applyStdinParam(
op: Operation,
params: Record<string, unknown>,
): Promise<void> {
// Branch shape (stdin hint + missing param + `!process.stdin.isTTY` gate +
// 5MB cap) is pinned by the R4 regression test for PR #1325's Windows fix
// (test/cycle/regression-pr-wave-r1-r2-r4.test.ts) — keep the spelling.
if (op.cliHints?.stdin && !params[op.cliHints.stdin] && !process.stdin.isTTY) {
const stdinContent = readFileSync(0, 'utf-8');
const content = await readStdinBounded();
if (content === null) return; // no input arrived — let the required-param check fail fast
const MAX_STDIN = 5_000_000; // 5MB
if (Buffer.byteLength(stdinContent, 'utf-8') > MAX_STDIN) {
if (Buffer.byteLength(content, 'utf-8') > MAX_STDIN) {
console.error(`Error: stdin content exceeds ${MAX_STDIN} bytes. Split into smaller inputs.`);
process.exit(1);
}
params[op.cliHints.stdin] = stdinContent;
params[op.cliHints.stdin] = content;
}
}
return params;
/** First-byte deadline for pipe/socket stdin (#3513). Env-overridable escape hatch. */
function stdinFirstByteTimeoutMs(): number {
const n = Number(process.env.GBRAIN_STDIN_TIMEOUT_MS);
return Number.isFinite(n) && n > 0 ? n : 5000;
}
/**
* Returns the full stdin content, '' for a readable-but-empty stdin, or
* null when stdin is a pipe/socket that never delivered a byte within the
* first-byte deadline (or the fd is closed/unreadable).
*/
export async function readStdinBounded(): Promise<string | null> {
let isPipeOrSocket: boolean;
try {
const st = fstatSync(0);
isPipeOrSocket = st.isFIFO() || st.isSocket();
} catch {
return null; // closed/invalid fd — treat as no input
}
if (!isPipeOrSocket) {
// Regular file redirect, /dev/null, etc. — read returns without blocking.
try {
return readFileSync(0, 'utf-8');
} catch {
return null;
}
}
return await new Promise<string | null>((resolve) => {
const chunks: Buffer[] = [];
let gotData = false;
const timer = setTimeout(() => {
if (!gotData) {
process.stdin.destroy();
resolve(null);
}
}, stdinFirstByteTimeoutMs());
const finish = () => {
clearTimeout(timer);
resolve(Buffer.concat(chunks).toString('utf-8'));
};
process.stdin.on('data', (c: Buffer) => {
if (!gotData) {
gotData = true;
clearTimeout(timer); // deadline applies to the FIRST byte only
}
chunks.push(c);
});
process.stdin.once('end', finish);
process.stdin.once('error', finish);
});
}
/**
@@ -868,7 +958,8 @@ export function applyThinClientSourceScope(
params.source_id = resolved;
}
async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
// Exported for tests (same import-safety contract as applyThinClientSourceScope).
export async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
// 'default'. Wrapped in try/catch so a doctor / single-source brain that
@@ -880,16 +971,21 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
// trusted local boundary) and consumed by federatedSearchScope in
// operations.ts, which additionally gates on ctx.remote === false.
let localFederated: string[] | undefined;
// params.source is set when a CLI flag was parsed for the op (rare; most
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
const explicit = (params.source as string | undefined) ?? null;
try {
const { resolveSourceWithTier, localFederatedSourceIds } = await import('./core/source-resolver.ts');
// params.source is set when a CLI flag was parsed for the op (rare; most
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
const explicit = (params.source as string | undefined) ?? null;
const resolved = await resolveSourceWithTier(engine, explicit);
sourceId = resolved.source_id;
localFederated = await localFederatedSourceIds(engine, resolved.source_id, resolved.tier);
} catch {
// Source resolution failed (e.g. sources table doesn't exist on a fresh
} catch (err) {
// #1712: an EXPLICIT --source that fails to resolve (invalid id, or a
// source that doesn't exist) must error loudly — the blanket swallow
// turned `--source __all__` and typos into a silent `default` scope,
// which is how three bug reports became debugging sessions.
if (explicit) throw err;
// Ambient resolution failed (e.g. sources table doesn't exist on a fresh
// pre-init brain). Leave sourceId unset; engine read methods fall through
// to the cross-source view (D16 back-compat path).
sourceId = undefined;
@@ -1055,7 +1151,7 @@ export function formatResult(opName: string, result: unknown): string {
* `runRemoteDoctor` for thin-client installs.
*/
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'retrieval-upgrade', 'apply-migrations',
'repair-jsonb', 'orphans', 'integrity', 'serve',
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
// the volunteer_context MCP op instead.
@@ -1102,6 +1198,7 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
'extract-conversation-facts': 'extract-conversation-facts runs on the host (requires local engine + chat gateway). Run on the host machine.',
enrich: 'enrich runs on the host (requires local engine + chat gateway for grounded synthesis). Run on the host machine.',
migrate: "migrate runs on the host's local engine. Run on the host machine.",
'retrieval-upgrade': "retrieval-upgrade (embedding migration) rebuilds the host brain's schema + re-embeds. Run on the host machine.",
'apply-migrations': 'schema migrations run on the host. SSH and run there.',
'repair-jsonb': 'repair-jsonb operates on the local DB only.',
integrity: 'integrity scans local files. Run on the host machine.',
@@ -1752,10 +1849,33 @@ async function handleCliOnly(command: string, args: string[]) {
}
// doctor is handled before connectEngine() above
case 'migrate': {
// #3390: `gbrain migrate embeddings --to <provider:model>` — the
// provider-agnostic embedding migration. Everything else stays the
// engine-transfer path (`migrate --to <supabase|pglite>`).
if (args[0] === 'embeddings') {
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
await runMigrateEmbeddings(engine, args.slice(1));
break;
}
if (args.includes('--help') || args.includes('-h')) {
console.log('Usage: gbrain migrate --to <supabase|pglite> [--url <url>] [--path <path>] [--force]');
console.log(' gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes]');
console.log('');
console.log('The first form transfers the brain between engines; the second re-embeds');
console.log('onto a different embedding provider (run `gbrain migrate embeddings --help`).');
break;
}
const { runMigrateEngine } = await import('./commands/migrate-engine.ts');
await runMigrateEngine(engine, args);
break;
}
case 'retrieval-upgrade': {
// The command README.md + doctor.ts promised since v0.36 but never
// dispatched. Alias for `migrate embeddings` (#3390).
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
await runMigrateEmbeddings(engine, args);
break;
}
case 'eval': {
// v0.32 EXP-5: `eval takes-quality {run,trend,regress}` requires a
// brain (samples takes from DB / reads runs table). `replay` was
@@ -2352,6 +2472,7 @@ USAGE
SETUP
init [--pglite|--supabase|--url] Create brain (PGLite default, no server)
migrate --to <supabase|pglite> Transfer brain between engines
migrate embeddings --to <p:model> Re-embed onto another embedding provider
upgrade Self-update
check-update [--json] Check for new versions
doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings)
@@ -2419,6 +2540,7 @@ TOOLS
publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256)
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
backfill <kind|list> v0.30.1: run a registered backfill (effective-date, ...)
orphans [--json] [--count] Find pages with no inbound wikilinks
salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience
anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type)
+71 -27
View File
@@ -8,7 +8,7 @@ import {
semverGt,
semverLte,
} from '../core/semver.ts';
import { writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
import { readUpdateCache, writeUpdateCache, type UpdateMarker } from '../core/self-upgrade.ts';
/** Best-effort cache write — a read-only ~/.gbrain must never make the check throw. */
function safeWriteCache(marker: UpdateMarker): void {
@@ -45,26 +45,53 @@ function upgradeCommandForMethod(method: string): string {
}
}
/** Where the latest version is resolved from. gbrain publishes NO GitHub
* releases (the `releases/latest` API is a permanent 404), so the release
* train's source of truth is the `VERSION` file on master — same trusted host
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
* package on npm is an unrelated GPU library (#505), so it would produce false
* upgrade prompts pointing at a stranger's package. */
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
/** Extract a version from the raw VERSION file body: first line, optional `v`
* prefix, optional `-suffix` channel tag (`0.31.1.1-fixwave` compares as its
* numeric base — fail-safe: a suffix-only bump never prompts). Body is bounded
* before parsing so a malformed/huge response can't blow up the check. */
export function parseVersionFileBody(body: string): string | null {
const firstLine = body.slice(0, 256).trim().split('\n')[0].trim();
const m = firstLine.match(/^v?(\d+\.\d+\.\d+(?:\.\d+)?)(?:[-+][0-9A-Za-z.-]+)?$/);
return m && isValidVersionString(m[1]) ? m[1] : null;
}
export type LatestReleaseResult =
| { ok: true; tag: string; published_at: string; url: string }
| { ok: false; reason: 'network_error' | 'no_releases' };
/**
* Fetch the latest GitHub release. Exported (v0.42) so the self-upgrade refresh
* path and tests can reuse it. 5s timeout (was 10s) — this runs on the detached
* refresh, never the hot path, but a tight bound keeps the refresh cheap.
* Resolve the latest published gbrain version (from VERSION on master — see
* VERSION_SOURCE_URL). Exported (v0.42) so the self-upgrade refresh path and
* tests can reuse it. 5s timeout this runs on the detached refresh, never the
* hot path. Failures are discriminated: `network_error` (offline/timeout) vs
* `no_releases` (endpoint answered but no usable version).
*/
export async function fetchLatestRelease(): Promise<{ tag: string; published_at: string; url: string } | null> {
export async function fetchLatestRelease(): Promise<LatestReleaseResult> {
let res: Response;
try {
const res = await fetch('https://api.github.com/repos/garrytan/gbrain/releases/latest', {
res = await fetch(VERSION_SOURCE_URL, {
headers: { 'User-Agent': `gbrain/${VERSION}` },
signal: AbortSignal.timeout(5_000),
});
if (!res.ok) return null;
const data = await res.json() as any;
return {
tag: data.tag_name || '',
published_at: data.published_at || '',
url: data.html_url || '',
};
} catch {
return null;
return { ok: false, reason: 'network_error' };
}
try {
if (!res.ok) return { ok: false, reason: 'no_releases' };
const tag = parseVersionFileBody(await res.text());
if (!tag) return { ok: false, reason: 'no_releases' };
return { ok: true, tag, published_at: '', url: RELEASE_NOTES_URL };
} catch {
return { ok: false, reason: 'network_error' };
}
}
@@ -118,17 +145,33 @@ export function extractChangelogBetween(changelog: string, from: string, to: str
}
/**
* Fetch the latest release and write the self-upgrade cache (the marker line
* read by the CLI startup hook). Fail-open: on any network failure we cache
* `UP_TO_DATE <current>` so the TTL prevents hammering GitHub on every
* invocation. Returns the resolved marker for callers that want it. This is the
* function the detached single-flight refresh (`gbrain check-update
* --refresh-cache`) invokes.
* A failed check must NEVER write `up_to_date` — that was #486: the fetch
* failed permanently (dead releases API) and every user was told "you're
* current" forever. Instead, re-write the last-known-good marker (bumping its
* mtime so the cache TTL still throttles retries and a network blip can't
* erase a pending upgrade_available notice). No prior marker → write nothing;
* the next invocation retries.
*/
function preserveCacheOnFailedCheck(): void {
try {
const prior = readUpdateCache();
if (prior) safeWriteCache(prior.marker);
} catch {
/* best-effort */
}
}
/**
* Fetch the latest version and write the self-upgrade cache (the marker line
* read by the CLI startup hook). On fetch failure the last-known-good marker is
* preserved (see preserveCacheOnFailedCheck) — never a fabricated `up_to_date`.
* This is the function the detached single-flight refresh (`gbrain
* check-update --refresh-cache`) invokes.
*/
export async function refreshUpdateCache(): Promise<void> {
const release = await fetchLatestRelease();
if (!release) {
safeWriteCache({ kind: 'up_to_date', current: VERSION });
if (!release.ok) {
preserveCacheOnFailedCheck();
return;
}
const latestVersion = release.tag.replace(/^v/, '');
@@ -166,9 +209,8 @@ export async function runCheckUpdate(args: string[]) {
const release = await fetchLatestRelease();
if (!release) {
// Warm the cache fail-open so the startup hook doesn't re-fetch every call.
safeWriteCache({ kind: 'up_to_date', current: VERSION });
if (!release.ok) {
preserveCacheOnFailedCheck();
if (json) {
console.log(JSON.stringify({
current_version: VERSION,
@@ -179,10 +221,12 @@ export async function runCheckUpdate(args: string[]) {
release_url: '',
changelog_diff: '',
published_at: '',
error: 'no_releases',
error: release.reason,
}, null, 2));
} else if (release.reason === 'network_error') {
console.log(`GBrain ${VERSION} — could not check for updates (network unavailable).`);
} else {
console.log(`GBrain ${VERSION} — could not check for updates (no releases found or network unavailable).`);
console.log(`GBrain ${VERSION} — could not determine the latest published version.`);
}
return;
}
+95 -12
View File
@@ -40,7 +40,7 @@ import {
buildBasenameIndex,
queryBasenameIndex,
} from '../core/link-extraction.ts';
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
import { probeSourceGitState } from '../core/git-head.ts';
// v0.41.32.0: remote staleness reads the stored newest_content_at column via
// this pure comparator (no git subprocess on the HTTP MCP doctor path).
import { lagFromContentMs } from '../core/source-health.ts';
@@ -53,6 +53,7 @@ import { isUndefinedColumnError } from '../core/utils.ts';
// drift from what search actually filters.
import { resolveHardExcludes, DEFAULT_HARD_EXCLUDES } from '../core/search/source-boost.ts';
import { escapeLikePattern, buildVisibilityClause } from '../core/search/sql-ranking.ts';
import { unverifiedExtractionFragment } from '../core/extraction-review.ts';
import { hnswIndexExpected, hnswMaxDimsForType } from '../core/vector-index.ts';
export interface Check {
@@ -3620,6 +3621,52 @@ export async function checkLinksExtractionLag(
}
}
/**
* issue #160 unverified_extractions doctor check.
*
* The extraction quarantine lane parks auto-extracted entity stubs
* (frontmatter `provenance: 'auto-extracted'` + `status: 'unverified'`)
* until the owner promotes or rejects them. A queue nobody reviews decays
* into invisible clutter, so this check counts stubs older than N days
* (default 7) and nudges toward the review surface. Exported for direct
* testing (mirrors checkLinksExtractionLag).
*/
export async function checkUnverifiedExtractions(
engine: BrainEngine,
opts?: { sourceId?: string; days?: number },
): Promise<Check> {
const name = 'unverified_extractions';
const days = opts?.days ?? 7;
const sourceId = opts?.sourceId;
try {
const params: unknown[] = [String(days)];
let srcClause = '';
if (sourceId) {
params.push(sourceId);
srcClause = 'AND p.source_id = $2';
}
const rows = await engine.executeRaw<{ n: string | number }>(
`SELECT COUNT(*)::int AS n FROM pages p
WHERE p.deleted_at IS NULL
AND ${unverifiedExtractionFragment('p')}
AND p.created_at < now() - ($1 || ' days')::interval
${srcClause}`,
params,
);
const n = Number(rows[0]?.n ?? 0);
return {
name,
status: n > 0 ? 'warn' : 'ok',
message: n > 0
? `${n} unverified auto-extracted entity stub(s) older than ${days} days awaiting review. List with 'gbrain extraction-pending'; promote/reject with 'gbrain extraction-review <promote|reject> --slugs <slug,...>'.`
: 'No stale unverified extraction stubs',
details: { count: n, days, source_id: sourceId ?? null },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check unverified_extractions: ${(e as Error).message}` };
}
}
/**
* issue #1678 extract_atoms_backlog doctor check.
*
@@ -4015,29 +4062,51 @@ export async function checkSyncFreshness(
// All four must hold; otherwise fall through to the time-based check.
// The chunker version match is computed here (not in the helper)
// because it depends on engine state, not git state.
//
// Clone-unavailable fallback: on stateless deploys (Docker on EB /
// K8s / Fly — the platforms the cloud recipes produce), a container
// restart wipes `local_path` and each clone is only re-materialized
// when that source's next sync job runs. Until then the HEAD probe
// cannot run at all ('unavailable'), which previously fell through to
// raw wall-clock age — and since a no-op sync doesn't advance
// `last_sync_at`, every QUIET source read as stale/FAIL after a
// restart (score-sinking alert storm; observed live: 16-source brain,
// 12 clones gone after a config-update restart, doctor 70→30).
// 'unavailable' + chunker match now reuses the v0.41.32.0 REMOTE lag
// signal (newest_content_at) below — DB-only, no subprocess, and it
// still reports staleness whenever content really is newer than the
// last sync. 'changed' (readable clone with real work) keeps
// wall-clock exactly as before, and a chunker mismatch is never
// masked (D7): it disables the fallback too.
let cloneUnavailable = false;
if (localOnly) {
const gitUnchanged = isSourceUnchangedSinceSync(
const gitState = probeSourceGitState(
source.local_path,
source.last_commit,
{ requireCleanWorkingTree: 'ignore-untracked' },
);
const chunkerMatch = source.chunker_version === currentChunkerVersion;
if (gitUnchanged && chunkerMatch) {
if (gitState === 'unchanged' && chunkerMatch) {
unchanged_count++;
continue;
}
cloneUnavailable = gitState === 'unavailable' && chunkerMatch;
}
// v0.41.32.0: REMOTE path (doctorReportRemote, !localOnly) computes lag
// from the stored newest_content_at column — NO git subprocess on a
// DB-supplied local_path (preserves the v0.41.27.0 trust boundary). A
// quiet repo whose newest commit predates its last sync reports 0; NULL
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock: the
// short-circuit already failed, so the source genuinely has work and
// "hours since last sync" is the right staleness measure. The `ageMs < 0`
// skew check above still runs on raw wall-clock for both paths (A1).
// column → wall-clock fallback. LOCAL fall-through keeps wall-clock when
// the clone is READABLE: the short-circuit failed on real evidence
// (HEAD moved / dirty tree), so the source genuinely has work and
// "hours since last sync" is the right staleness measure. A local clone
// that is UNAVAILABLE (not yet re-materialized, see above) carries no
// evidence either way, so it borrows this same DB-only lag. The
// `ageMs < 0` skew check above still runs on raw wall-clock for both
// paths (A1).
let thresholdAgeMs = ageMs;
if (!localOnly) {
if (!localOnly || cloneUnavailable) {
const contentMs = source.newest_content_at
? new Date(source.newest_content_at).getTime()
: null;
@@ -4280,8 +4349,18 @@ export async function checkCycleFreshness(
: `'${source.id}'`;
const raw = source.config?.last_full_cycle_at;
if (typeof raw !== 'string') {
// #2540: WARN, not FAIL. This check iterates EVERY local_path source,
// so on a multi-source install where only some vaults are cycled
// (e.g. one nightly `gbrain dream --dir <vault>`), a never-cycled
// sibling source turned doctor permanently red — which erodes the
// check's signal until real staleness hides inside the noise (the
// reporter's install masked genuinely stale sources for weeks this
// way). "Never cycled" also fires on a source added minutes ago.
// A source that HAS cycled and then went stale still escalates
// through the warn/fail age thresholds below — that is the
// regression signal this check exists for.
issues.push(`Source ${display} has never completed a full cycle`);
hasFailures = true;
hasWarnings = true;
continue;
}
const last = new Date(raw).getTime();
@@ -4317,7 +4396,7 @@ export async function checkCycleFreshness(
return {
name: 'cycle_freshness',
status: 'warn',
message: `${issues.join('; ')}.`,
message: `${issues.join('; ')}. Run \`gbrain dream --source <id>\` to cycle a source, or start \`gbrain autopilot\`.`,
};
}
return {
@@ -5923,7 +6002,7 @@ export async function buildChecks(
// that doesn't match the gateway's resolved default. Empty-brain vs
// non-empty-brain branching determines the repair hint:
// - empty brain (no embedded chunks) → `gbrain init --force --embedding-model …`
// - non-empty brain → `gbrain retrieval-upgrade --to … --reindex`
// - non-empty brain → `gbrain migrate embeddings --to … --dim …` (#3390)
// The bug-reporter's `rm -rf ~/.gbrain` recovery is never the right answer.
let surfacedUnconfiguredDrift = false;
try {
@@ -5954,7 +6033,7 @@ export async function buildChecks(
if (totalChunks > 0) {
const fix = embeddedCount === 0
? `No embeddings yet — drop the empty schema and re-init at the right dim:\n gbrain init --force --pglite --embedding-model ${configuredModel} --embedding-dimensions ${configuredDims}`
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain retrieval-upgrade --to ${configuredModel} --reindex`;
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain migrate embeddings --to ${configuredModel} --dim ${configuredDims}`;
checks.push({
name: 'embedding_provider',
@@ -6891,6 +6970,10 @@ export async function buildChecks(
checks.push({ name: 'flagged_pages', status: 'ok', message: `Skipped (${msg})` });
}
// issue #160: extraction quarantine lane review nudge.
progress.heartbeat('unverified_extractions');
checks.push(await checkUnverifiedExtractions(engine, { sourceId: orphanRatioSourceId }));
// 11a. Frontmatter integrity (v0.22.4, hardened in v0.38.2.0).
// scanBrainSources walks every registered source's local_path on disk
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
+104 -7
View File
@@ -19,6 +19,26 @@ import {
} from '../core/pace-mode.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
import type { Page } from '../core/types.ts';
/**
* #3507 after a plain re-embed fully re-embedded a `per_chunk_synopsis`
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
* page's CR state to 'title' so `contextual_retrieval_mode` keeps describing
* the vectors actually in the column. The reindex sweep restores the synopsis
* tier later. No-op for every other mode.
*/
export async function restampIfDemotedToTitleTier(
engine: BrainEngine,
page: Pick<Page, 'contextual_retrieval_mode'> | null | undefined,
slug: string,
sourceId: string,
): Promise<void> {
if (page?.contextual_retrieval_mode !== 'per_chunk_synopsis') return;
await engine.updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration());
}
export interface EmbedOpts {
/** Embed ALL pages (every chunk). */
@@ -115,6 +135,16 @@ export interface EmbedOpts {
* Errors/warnings still go to stderr regardless.
*/
quiet?: boolean;
/**
* #3391: widen signature-drift invalidation to pages with NO recorded
* embedding_signature (pre-v108). By default those are grandfathered
* (never invalidated) so a routine upgrade doesn't surprise-re-embed a
* whole corpus but after a provider/model swap the grandfather clause
* silently leaves them in the OLD embedding space, mixing two vector
* spaces in one index. `gbrain migrate embeddings` and
* `gbrain embed --stale --include-null-signature` set this.
*/
includeNullSignature?: boolean;
}
/**
@@ -356,6 +386,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
pacer,
paceMaxConcurrency,
quiet: opts.quiet,
includeNullSignature: opts.includeNullSignature,
}, opts.signal);
} finally {
// E1: surface pacing telemetry (human + structured) when pacing was on.
@@ -469,6 +500,8 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
const catchUp = args.includes('--catch-up');
// #3391: re-embed pages that predate the embedding_signature stamp too.
const includeNullSignature = args.includes('--include-null-signature');
const pace = parsePaceArgs(args);
let opts: EmbedOpts;
@@ -476,11 +509,11 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
} else if (all || stale) {
// E-2: CLI-only single-flight for stale runs (the minion path locks itself).
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) };
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }), ...(includeNullSignature && { includeNullSignature: true }) };
} else {
const slug = args.find(a => !a.startsWith('--'));
if (!slug) {
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]');
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up] [--include-null-signature]');
process.exit(1);
}
opts = { slug, dryRun, sourceId, batchSize, priority, catchUp };
@@ -586,7 +619,11 @@ async function embedPage(
return;
}
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text), { abortSignal: signal });
// #3507: embed with the page's STORED wrapping convention (title-tier
// contextual prefix when the page was embedded wrapped), not raw
// chunk_text — otherwise a re-embed silently strips the contextual
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
@@ -609,6 +646,9 @@ async function embedPage(
// such a page and then stamps it.
if (toEmbed.length === chunks.length) {
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest.
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
}
result.embedded += toEmbed.length;
result.pages_processed++;
@@ -657,6 +697,8 @@ async function embedAll(
paceMaxConcurrency?: number;
/** #394: suppress human stdout summaries (structured-output callers). */
quiet?: boolean;
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
includeNullSignature?: boolean;
},
signal?: AbortSignal,
) {
@@ -748,7 +790,8 @@ async function embedAll(
}
try {
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text));
// #3507: reproduce the page's stored wrapping convention (see embedPage).
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
// Build a map of new embeddings by chunk_index
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
@@ -770,6 +813,11 @@ async function embedAll(
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest.
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
result.embedded += toEmbed.length;
} catch (e: unknown) {
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
@@ -845,6 +893,8 @@ async function embedAllStale(
paceMaxConcurrency?: number;
/** #394: suppress human stdout summaries (structured-output callers). */
quiet?: boolean;
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
includeNullSignature?: boolean;
},
signature?: string,
externalSignal?: AbortSignal,
@@ -852,6 +902,7 @@ async function embedAllStale(
// D7: thread sourceId so source-scoped runs only count + visit
// that source's NULL embeddings.
const sourceOpt = sourceId ? { sourceId } : undefined;
const includeNullSig = !!staleOpts?.includeNullSignature;
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
// swap). dry-run must NOT mutate, so it counts signature-stale via the
@@ -861,16 +912,46 @@ async function embedAllStale(
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
signature,
...(sourceId && { sourceId }),
...(includeNullSig && { includeNullSignature: true }),
});
if (invalidated > 0 && !staleOpts?.quiet) {
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
}
// #3391: the grandfather clause keeps NULL-signature pages on their OLD
// vectors — two embedding spaces mixed in one index. Loud stderr warning
// with the fix, instead of silent retrieval degradation.
//
// Deliberately NOT gated on `invalidated > 0`: the original bug report's
// shape is a brain where EVERY embedded page predates the signature stamp,
// so nothing drifts, nothing is invalidated — and pre-fix that brain got
// no warning AND no work, the exact silent case #3391 is about. The probe
// below computes the left-behind count directly, which is 0 on a healthy
// brain, so an unaffected run stays quiet.
if (!includeNullSig) {
try {
const wide = await engine.countStaleChunks({ ...sourceOpt, signature, includeNullSignature: true });
const narrow = await engine.countStaleChunks({ ...sourceOpt, signature });
const leftBehind = wide - narrow;
if (leftBehind > 0) {
serr(
` [embed] WARNING: ${leftBehind} embedded chunk(s) sit on pages with no recorded ` +
`embedding signature and were NOT invalidated — they remain in the previous model's ` +
`embedding space. Re-run with --include-null-signature (or use ` +
`\`gbrain migrate embeddings\`) to re-embed them.`,
);
}
} catch {
// The warning probe is best-effort; never break the embed run.
}
}
}
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
// dry-run includes signature-drift in the count without mutating.
const staleCount = await engine.countStaleChunks(
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
dryRun && signature
? { ...sourceOpt, signature, ...(includeNullSig && { includeNullSignature: true }) }
: sourceOpt,
);
if (staleCount === 0) {
if (!staleOpts?.quiet) {
@@ -1050,7 +1131,13 @@ async function embedAllStale(
const keySourceId = stale[0]?.source_id ?? 'default';
const slug = stale[0].slug;
try {
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal });
// #3507: fetch the page row for its title + stored CR mode so the
// re-embed reproduces the page's wrapping convention instead of
// silently stripping contextual prefixes — `embed --stale` is the
// NORMAL post-model-migration path, so raw-text embedding here
// quietly converted whole corpora to the unwrapped convention.
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
const staleIdxToEmbedding = new Map<number, Float32Array>();
@@ -1078,6 +1165,14 @@ async function embedAllStale(
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
}
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest. Partially-stale pages
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
if (stale.length === existing.length) {
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
);
}
result.embedded += stale.length;
} catch (e: unknown) {
// Budget/abort-fired cancellations are expected on the way out; don't
@@ -1138,7 +1233,9 @@ async function embedAllStale(
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
const remaining = await engine.countStaleChunks(
signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined),
signature
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
: (sourceId ? { sourceId } : undefined),
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
+41 -2
View File
@@ -149,6 +149,40 @@ export const ALLOWED_TYPES = [
] as const;
export type AllowedType = (typeof ALLOWED_TYPES)[number];
/**
* Granular collector page-types that alias into each canonical conversation
* bucket. The v2 type-consolidation pack retypes these to the canonical names
* (`slack-dm-day`/`slack-thread` `slack`, `email-digest` `email`), but a
* brain that hasn't run that pack still carries the collector's granular types
* in `pages.type`. Without this expansion, `listPages({ type: 'slack' })`
* matches zero rows on such brains and the whole comms corpus is silently
* skipped (facts stay empty `find_trajectory` returns nothing). The canonical
* name is always included first so consolidated brains keep working unchanged.
*/
export const ALLOWED_TYPE_ALIASES: Record<AllowedType, readonly string[]> = {
conversation: ['conversation'],
meeting: ['meeting'],
slack: ['slack', 'slack-dm-day', 'slack-thread'],
email: ['email', 'email-digest'],
imessage: ['imessage'],
'imessage-daily': ['imessage-daily'],
};
/**
* Expand the requested logical types to the concrete `pages.type` values to
* enumerate, canonical-first and de-duplicated. Unknown types pass through
* unchanged so an explicit override is never dropped.
*/
export function pageTypesForAllowed(types: readonly AllowedType[]): string[] {
const out: string[] = [];
for (const t of types) {
for (const concrete of ALLOWED_TYPE_ALIASES[t] ?? [t]) {
if (!out.includes(concrete)) out.push(concrete);
}
}
return out;
}
/**
* Pagination batch size for listPages enumeration. Per-batch memory
* worst case = BATCH × MAX_PAGE_BODY_BYTES = 250MB at default 10
@@ -1264,13 +1298,18 @@ export async function runExtractConversationFactsCore(
}
};
// Expand logical types (conversation/meeting/slack/email) to the concrete
// `pages.type` values to enumerate, so brains on the granular collector
// types are not silently skipped (see ALLOWED_TYPE_ALIASES).
const concreteTypes = pageTypesForAllowed(types);
if (opts.slug) {
const page = await engine.getPage(opts.slug, { sourceId });
if (!page) {
result.pages_skipped_disappeared++;
return;
}
if (!types.includes(page.type as AllowedType)) {
if (!concreteTypes.includes(page.type)) {
result.pages_skipped++;
return;
}
@@ -1284,7 +1323,7 @@ export async function runExtractConversationFactsCore(
// honors AbortSignal at each claim boundary and threads
// BudgetExhausted abort (D13) automatically.
let processedPagesCount = 0;
pageLoop: for (const type of types) {
pageLoop: for (const type of concreteTypes) {
let offset = 0;
// eslint-disable-next-line no-constant-condition
while (true) {
+23 -4
View File
@@ -35,7 +35,7 @@ import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/en
import type { PageType } from '../core/types.ts';
import { parseMarkdown } from '../core/markdown.ts';
import {
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
extractPageLinks, parseTimelineEntries, deriveTimelineAnchor, inferLinkType, makeResolver,
extractFrontmatterLinks, isGlobalBasenameEnabled, LINK_EXTRACTOR_VERSION_TS,
WIKILINK_BASENAME_LINK_TYPE,
buildBasenameIndex, queryBasenameIndex, stripCodeBlocks,
@@ -749,6 +749,12 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
// v0.41.18.0 (A11, T8): --from-meetings extracts timeline entries from
// meeting pages onto each discussed entity. Timeline subcommand only.
const fromMeetings = args.includes('--from-meetings');
// --infer-dates: for pages whose body has NO parseable timeline line, anchor
// one entry at the page's computed effective_date (frontmatter / filename date,
// never the updated_at fallback). Default OFF for back-compat — comms/calendar
// brains opt in to populate timeline from slug/frontmatter dates. DB-source only
// (needs the full Page.effective_date, which getPage projects).
const inferDates = args.includes('--infer-dates');
// v0.41.17.0 (T7, D9): --workers N parsed via the shared validator.
// Honored on the fs-walk inner loops only; DB-source paths stay
// serial in v0.41.17.0 (see ExtractOpts.workers doc).
@@ -963,7 +969,7 @@ Status (v0.42):
result.pages_processed = r.pages;
}
if (subcommand === 'timeline' || subcommand === 'all') {
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter });
const r = await extractTimelineFromDB(engine, dryRun, jsonMode, typeFilter, since, { sourceIdFilter, inferDates });
result.timeline_entries_created = r.created;
result.pages_processed = Math.max(result.pages_processed, r.pages);
}
@@ -1583,7 +1589,7 @@ async function extractTimelineFromDB(
jsonMode: boolean,
typeFilter: PageType | undefined,
since: string | undefined,
opts?: { sourceIdFilter?: string },
opts?: { sourceIdFilter?: string; inferDates?: boolean },
): Promise<{ created: number; pages: number }> {
// v0.32.8: listAllPageRefs enumerates (slug, source_id) pairs so we can
// thread sourceId to getPage and addTimelineEntriesBatch. Pre-fix used
@@ -1592,6 +1598,7 @@ async function extractTimelineFromDB(
// v0.37.7.0 #1204: when sourceIdFilter is set, scope the walk to one
// source so federated brain users can extract per-source.
const sourceIdFilter = opts?.sourceIdFilter;
const inferDates = opts?.inferDates ?? false;
const allRefs = sourceIdFilter
? (await engine.listAllPageRefs()).filter(r => r.source_id === sourceIdFilter)
: await engine.listAllPageRefs();
@@ -1631,7 +1638,19 @@ async function extractTimelineFromDB(
}
const fullContent = page.compiled_truth + '\n' + page.timeline;
const entries = parseTimelineEntries(fullContent);
let entries = parseTimelineEntries(fullContent);
// --infer-dates: pages with no in-body timeline line but a trustworthy
// content date (frontmatter / filename) get one anchor entry at that date.
// Applied ONLY on the zero-entry path so it never shadows a real timeline.
if (entries.length === 0 && inferDates) {
const anchor = deriveTimelineAnchor({
slug,
title: page.title,
effectiveDate: page.effective_date,
effectiveDateSource: page.effective_date_source,
});
if (anchor) entries = [anchor];
}
for (const entry of entries) {
if (dryRunSeen) {
+10 -2
View File
@@ -16,7 +16,7 @@ interface FileRecord {
filename: string;
storage_path: string;
mime_type: string | null;
size_bytes: number;
size_bytes: number | bigint | string | null;
content_hash: string;
metadata: Record<string, unknown>;
created_at: string;
@@ -42,6 +42,14 @@ function fileHash(filePath: string): string {
return createHash('sha256').update(content).digest('hex');
}
export function formatFileSizeKb(rawSizeBytes: number | bigint | string | null): string {
if (rawSizeBytes == null) return '?';
const sizeBytes = Number(rawSizeBytes);
return Number.isFinite(sizeBytes) && sizeBytes >= 0
? `${Math.round(sizeBytes / 1024)}KB`
: '?';
}
export async function runFiles(engine: BrainEngine, args: string[]) {
const subcommand = args[0];
@@ -116,7 +124,7 @@ async function listFiles(engine: BrainEngine, slug?: string) {
console.log(`${rows.length} file(s):`);
for (const row of rows) {
const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?';
const size = formatFileSizeKb(row.size_bytes as FileRecord['size_bytes']);
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
}
}
+37 -5
View File
@@ -143,6 +143,31 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
return parsed;
}
/**
* #3026: the thin-client `list`/`get` branches receive jobs as parsed JSON
* off the MCP wire, where every timestamp is an ISO string but formatJob /
* formatJobDetail (and the stalled-detection comparison) hold a Date
* contract, hydrated locally by MinionQueue.rowToJob. Rehydrate once at the
* unpack boundary so both paths hand the formatters real Dates. Exported for
* unit tests.
*/
const JOB_DATE_FIELDS = [
'created_at', 'updated_at', 'started_at', 'finished_at', 'lock_until', 'delay_until',
] as const;
export function rehydrateJobDates<T>(job: T): T {
if (!job || typeof job !== 'object') return job;
const rec = job as { [k: string]: unknown };
for (const field of JOB_DATE_FIELDS) {
const v = rec[field];
if (typeof v === 'string') {
const d = new Date(v);
if (!Number.isNaN(d.getTime())) rec[field] = d;
}
}
return job;
}
function formatJob(job: MinionJob): string {
const dur = job.finished_at && job.started_at
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
@@ -208,7 +233,7 @@ USAGE
gbrain jobs get <id>
gbrain jobs cancel <id>
gbrain jobs retry <id>
gbrain jobs prune [--older-than 30d]
gbrain jobs prune [--older-than 30d] [--dry-run]
gbrain jobs delete <id>
gbrain jobs stats
gbrain jobs smoke
@@ -496,7 +521,7 @@ HANDLER TYPES (built in)
const raw = await callRemoteTool(cfg!, 'list_jobs', {
status, queue: queueName, limit,
}, { timeoutMs: 30_000 });
jobs = unpackToolResult<MinionJob[]>(raw);
jobs = unpackToolResult<MinionJob[]>(raw).map((j) => rehydrateJobDates(j));
} else {
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
@@ -525,7 +550,7 @@ HANDLER TYPES (built in)
if (isThinClient(cfg)) {
try {
const raw = await callRemoteTool(cfg!, 'get_job', { id }, { timeoutMs: 30_000 });
job = unpackToolResult<MinionJob | null>(raw);
job = rehydrateJobDates(unpackToolResult<MinionJob | null>(raw));
} catch (e) {
// The remote op throws `invalid_params` on not-found; surface as
// the same "Job not found" exit-1 the local path produces.
@@ -608,8 +633,15 @@ HANDLER TYPES (built in)
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000) });
console.log(`Pruned ${count} jobs older than ${days} days.`);
// #2712: --dry-run previews the count without deleting. It used to be
// silently ignored (the destructive default ran anyway).
const dryRun = hasFlag(args, '--dry-run');
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000), dryRun });
if (dryRun) {
console.log(`[dry-run] Would prune ${count} jobs older than ${days} days. Nothing deleted.`);
} else {
console.log(`Pruned ${count} jobs older than ${days} days.`);
}
break;
}
+402
View File
@@ -0,0 +1,402 @@
/**
* `gbrain migrate embeddings --to <provider:model>` (#3390) the
* provider-agnostic forward migration off any embedding provider, built for
* the ZeroEntropy 2026-09-04 sunset but not keyed to it.
*
* Also reachable as `gbrain retrieval-upgrade` the command README.md and
* doctor.ts have promised since v0.36 but which never had a dispatch branch.
*
* Flow (everything heavy is reused, see src/core/embedding-migration.ts):
* 1. plan chunk/char counts via the widened stale predicates,
* cost estimate from embedding-pricing.ts
* 2. preflight print estimate; require --yes or interactive confirm
* (non-TTY without --yes refuses with exit 2, mirroring the
* reindex-code cost gate in docs/operations/spend-controls.md)
* 3. probe one live embed against the TARGET provider BEFORE any
* mutation (validates key + model + dims in one shot)
* 4. apply schema transition (dim change), config (DB + file plane),
* #3391 NULL-signature-inclusive invalidation, cache purge
* 5. re-embed runEmbedCore --stale --catch-up with single-flight locks,
* pacing (--pace), progress reporting. Resumable: a killed
* run re-runs the SAME command; the NULL-embedding cursor is
* the checkpoint and steps 3-4 no-op on the second pass.
*/
import type { BrainEngine } from '../core/engine.ts';
import { serr, slog } from '../core/console-prefix.ts';
import {
planEmbeddingMigration,
applyEmbeddingMigration,
completeEmbeddingMigration,
reconcilePageSignatures,
MIGRATION_STATE_KEY,
type EmbeddingMigrationPlan,
} from '../core/embedding-migration.ts';
import { formatEnvOverrideWarning } from '../core/retrieval-upgrade-planner.ts';
import { parsePaceArgs, runEmbedCore } from './embed.ts';
export interface MigrateEmbeddingsFlags {
to?: string;
dim?: number;
yes: boolean;
dryRun: boolean;
json: boolean;
noEmbed: boolean;
ignoreEnvOverride: boolean;
batchSize?: number;
pace?: ReturnType<typeof parsePaceArgs>;
}
export function parseMigrateEmbeddingsFlags(args: string[]): MigrateEmbeddingsFlags {
const toIdx = args.indexOf('--to');
const dimIdx = args.indexOf('--dim');
const dimRaw = dimIdx >= 0 ? parseInt(args[dimIdx + 1] ?? '', 10) : NaN;
const bsIdx = args.indexOf('--batch-size');
const bsRaw = bsIdx >= 0 ? parseInt(args[bsIdx + 1] ?? '', 10) : NaN;
const batchSize = Number.isFinite(bsRaw) && bsRaw > 0 ? Math.min(10_000, bsRaw) : undefined;
return {
to: toIdx >= 0 ? args[toIdx + 1] : undefined,
dim: Number.isFinite(dimRaw) && dimRaw > 0 ? dimRaw : undefined,
yes: args.includes('--yes') || args.includes('--non-interactive'),
dryRun: args.includes('--dry-run'),
json: args.includes('--json'),
noEmbed: args.includes('--no-embed'),
ignoreEnvOverride: args.includes('--ignore-env-override'),
...(batchSize !== undefined && { batchSize }),
pace: parsePaceArgs(args),
};
}
function printHelp(): void {
process.stdout.write(`Usage: gbrain migrate embeddings --to <provider:model> [flags]
Re-embed the whole brain onto a different embedding provider/model. Handles
dimension changes (schema transition), pages without a recorded embedding
signature (#3391), the query cache, and resume-after-kill. The forward path
off a sunsetting provider.
Flags:
--to <provider:model> Target embedding model (e.g. openai:text-embedding-3-small).
--dim <N> Target dimensions. Defaults to the provider recipe's
declared width; required when the recipe declares none.
--dry-run Plan + cost estimate only; change nothing.
--yes Skip the confirm prompt (required non-interactively).
--json Machine-readable envelope on stdout.
--no-embed Apply schema + config + invalidation, but skip the
re-embed pass (run \`gbrain embed --stale --include-null-signature\`
or \`... --background\` yourself).
--batch-size <N> Stale-chunk batch size for the re-embed (default 2000).
--pace[=mode] DB-contention pacing for the re-embed (off|gentle|balanced|aggressive).
--ignore-env-override Proceed even when GBRAIN_EMBEDDING_* env vars would
override the target at runtime (you know why).
--help Show this help.
A killed run is resumable: re-run the same command. Already-migrated chunks
are never re-embedded twice.
`);
}
function renderPlan(plan: EmbeddingMigrationPlan): string {
const lines: string[] = [];
lines.push('Embedding migration plan');
lines.push(` From: ${plan.from_model} (${plan.from_dims}d${plan.column_dims !== null && plan.column_dims !== plan.from_dims ? `; column is actually ${plan.column_dims}d` : ''})`);
lines.push(` To: ${plan.to_model} (${plan.to_dims}d)`);
if (plan.dim_change) {
lines.push(` DESTRUCTIVE: the embedding column is rebuilt at ${plan.to_dims}d, which DELETES`);
lines.push(' every stored embedding vector in this brain. They are not recoverable —');
lines.push(' going back to the old provider means paying for a second full re-embed.');
lines.push(' Until the re-embed finishes, semantic search is degraded to lexical-only.');
lines.push(` The query cache and fact embeddings are rebuilt at ${plan.to_dims}d too`);
lines.push(' (cache refills on next query; facts re-embed on their next write).');
}
lines.push(` Chunks to re-embed: ${plan.chunks_to_embed}${plan.null_signature_chunks > 0 ? ` (includes ${plan.null_signature_chunks} on pages with no recorded embedding signature)` : ''}`);
lines.push(
plan.price_known
? ` Estimated cost: $${plan.est_cost_usd.toFixed(2)} (${plan.total_chars} chars at the ${plan.to_model} rate)`
: ` Estimated cost: unknown — no pricing entry for ${plan.to_model}. Check the provider's pricing before proceeding.`,
);
if (plan.resuming) {
lines.push(' Resuming: a prior migration to this target was interrupted; continuing it.');
}
if (plan.reranker_warning) {
lines.push(` WARNING: ${plan.reranker_warning}`);
}
return lines.join('\n');
}
/** Single-keypress y/N confirm on stdin. Injectable for tests. */
async function defaultConfirm(question: string): Promise<boolean> {
process.stderr.write(`${question} [y/N] `);
const stdin = process.stdin;
stdin.setRawMode?.(true);
stdin.resume();
const key: string = await new Promise((resolve) => {
stdin.once('data', (d) => resolve(d.toString()));
});
stdin.setRawMode?.(false);
stdin.pause();
process.stderr.write('\n');
return key.trim().toLowerCase().startsWith('y');
}
/**
* One tiny embed against the TARGET provider, BEFORE any mutation: validates
* the API key, the model id, and dimension support in a single call, so a bad
* target fails with the brain untouched instead of after the column is
* dropped. Shared by the CLI and the `migrate_embeddings` op (the op used to
* skip it, which let `yes:true` drop the column against a bad key).
*/
export async function probeTargetProvider(
toModel: string,
toDims: number,
): Promise<{ ok: true } | { ok: false; message: string }> {
try {
const { embed } = await import('../core/ai/gateway.ts');
const vecs = await embed(['gbrain embedding migration probe'], {
embeddingModel: toModel,
dimensions: toDims,
});
const got = vecs[0]?.length ?? 0;
if (got !== toDims) {
return {
ok: false,
message: `Target provider returned ${got}-dim vectors, expected ${toDims}. Pass a valid --dim for ${toModel}.`,
};
}
return { ok: true };
} catch (e) {
return {
ok: false,
message: `Preflight embed against ${toModel} failed — nothing was changed:\n ${e instanceof Error ? e.message : String(e)}`,
};
}
}
/**
* Persist the target model+dims to the FILE plane and reconfigure the
* in-process gateway. The gateway reads file/env config, not the DB plane
* without this the re-embed would silently run against the OLD provider.
* Shared by the CLI command and the `migrate_embeddings` op handler.
*/
export async function persistEmbeddingFileConfig(
toModel: string,
toDims: number,
): Promise<void> {
const { loadConfig, saveConfig } = await import('../core/config.ts');
const { configureGateway } = await import('../core/ai/gateway.ts');
const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts');
const cfg = loadConfig();
if (!cfg) {
// REFUSE rather than warn-and-proceed. Without a file plane to write, the
// switch would not survive this process: the next `gbrain` invocation
// reads file/env config, sees the OLD provider, and re-embeds the brain
// back into the old space (paying twice) — or fails outright against a
// column that is now the new width. Thrown from inside
// applyEmbeddingMigration's try, so it surfaces as status: 'failed'
// BEFORE the config/cache steps and the caller exits non-zero.
throw new Error(
'No ~/.gbrain/config.json found — refusing to migrate.\n' +
' The embed pipeline reads file/env config, so without a file plane this switch\n' +
' would not survive the process and the next run would re-embed into the old space.\n' +
' Fix: run `gbrain init` (or set GBRAIN_EMBEDDING_MODEL + GBRAIN_EMBEDDING_DIMENSIONS\n' +
' in the environment of every gbrain process) and re-run.',
);
}
cfg.embedding_model = toModel;
cfg.embedding_dimensions = toDims;
saveConfig(cfg);
configureGateway(buildGatewayConfig(cfg));
}
export interface RunMigrateEmbeddingsOpts {
/** Test seams. */
confirm?: (question: string) => Promise<boolean>;
isTTY?: boolean;
exit?: (code: number) => never;
}
export async function runMigrateEmbeddings(
engine: BrainEngine,
args: string[],
opts: RunMigrateEmbeddingsOpts = {},
): Promise<void> {
// Explicit `never` annotation so TS control-flow analysis treats every
// exit() call as terminal (required for narrowing after the guard blocks).
const exit: (code: number) => never = opts.exit ?? ((code: number) => process.exit(code));
if (args.includes('--help') || args.includes('-h')) {
printHelp();
exit(0);
}
const flags = parseMigrateEmbeddingsFlags(args);
if (!flags.to) {
serr('Missing --to <provider:model>. Example: gbrain migrate embeddings --to openai:text-embedding-3-small');
serr('Run with --help for all flags.');
exit(1);
}
// From-state as the gateway resolved it (file/env config + defaults) —
// the truth for what embeds run under TODAY.
let fromModel: string | undefined;
let fromDims: number | undefined;
try {
const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts');
fromModel = getEmbeddingModel();
fromDims = getEmbeddingDimensions();
} catch {
// Gateway unconfigured — plan falls back to shipped defaults.
}
let plan: EmbeddingMigrationPlan;
try {
plan = await planEmbeddingMigration(engine, {
to: flags.to!,
...(flags.dim !== undefined && { dim: flags.dim }),
...(fromModel !== undefined && { fromModel }),
...(fromDims !== undefined && { fromDims }),
});
} catch (e) {
serr(e instanceof Error ? e.message : String(e));
exit(1);
return; // unreachable; keeps TS happy for injected exit seams
}
if (flags.json) {
// Human plan goes to stderr so stdout stays JSON-clean.
serr(renderPlan(plan));
} else {
console.log(renderPlan(plan));
}
if (plan.chunks_to_embed === 0 && !plan.dim_change && plan.from_model === plan.to_model) {
if (flags.json) console.log(JSON.stringify({ status: 'skipped_no_work', plan }, null, 2));
else console.log('Nothing to migrate — brain is already on the target model.');
exit(0);
}
if (flags.dryRun) {
if (flags.json) console.log(JSON.stringify({ status: 'planned', plan }, null, 2));
exit(0);
}
// ── Consent gate. Unlike the pure cost gates in
// docs/operations/spend-controls.md, `spend.posture=tokenmax` does NOT
// bypass this one: posture waives the SPEND ceiling, and this gate also
// guards a destructive schema rebuild (existing vectors are dropped, and
// retrieval is degraded until the re-embed finishes). We honor the posture
// by marking the dollar figure informational, and still ask.
if (!flags.yes) {
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const posture = await resolveSpendPosture(engine);
if (posture === 'tokenmax') {
serr(' [migrate] spend.posture=tokenmax: the cost estimate above is informational.');
serr(' [migrate] Confirmation is still required — this rebuilds the embedding column (destructive, not just costly).');
}
const isTTY = opts.isTTY ?? Boolean(process.stdin.isTTY);
if (!isTTY) {
serr('Refusing to migrate without confirmation in a non-TTY environment. Re-run with --yes.');
exit(2);
}
const confirm = opts.confirm ?? defaultConfirm;
const priceNote = plan.price_known ? `~$${plan.est_cost_usd.toFixed(2)}` : 'an UNKNOWN amount';
const ok = await confirm(`Re-embed ${plan.chunks_to_embed} chunks (${priceNote})?`);
if (!ok) {
serr('Aborted. Nothing was changed.');
exit(1);
}
}
// ── Live probe BEFORE any mutation: one tiny embed against the TARGET
// provider validates API key, model id, and dimension support in one call.
const probe = await probeTargetProvider(plan.to_model, plan.to_dims);
if (!probe.ok) {
serr(probe.message);
exit(1);
}
// ── Apply: schema + config + invalidation + cache purge.
const applied = await applyEmbeddingMigration(engine, plan, {
ignoreEnvOverride: flags.ignoreEnvOverride,
persistConfig: (toModel, toDims) => persistEmbeddingFileConfig(toModel, toDims),
});
if (applied.status === 'refused') {
if (flags.json) console.log(JSON.stringify(applied, null, 2));
else serr(formatEnvOverrideWarning(applied.warning));
exit(1);
}
if (applied.status === 'failed') {
if (flags.json) console.log(JSON.stringify(applied, null, 2));
else serr(`Migration apply failed: ${applied.reason}`);
exit(1);
}
serr(` [migrate] schema ${applied.schema_transitioned ? `rebuilt at ${plan.to_dims}d` : 'unchanged'}; ` +
`${applied.invalidated} chunk(s) invalidated; query cache purged (${applied.cache_cleared} row(s)).`);
if (flags.noEmbed) {
const msg = 'Config + schema migrated. Re-embed deferred — run: gbrain embed --stale --catch-up --include-null-signature';
if (flags.json) console.log(JSON.stringify({ ...applied, status: 'applied_no_embed', plan }, null, 2));
else console.log(msg);
exit(0);
}
// ── Re-embed. All the machinery (locks, pacing, backoff, progress,
// signature stamping) is the standard embed pipeline.
const { createProgress } = await import('../core/progress.ts');
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
let progressStarted = false;
const embedResult = await runEmbedCore(engine, {
stale: true,
catchUp: true,
singleFlight: true,
includeNullSignature: true,
quiet: flags.json,
...(flags.batchSize !== undefined && { batchSize: flags.batchSize }),
...(flags.pace && { pace: flags.pace }),
onProgress: (done, total) => {
if (!progressStarted) {
progress.start('migrate.reembed', total);
progressStarted = true;
}
progress.tick(1);
},
});
if (progressStarted) progress.finish();
// Reconcile signatures BEFORE the completion probe: pages straddling a
// stale-batch boundary are embedded correctly but left unstamped by the
// embed loop's all-or-nothing stamp rule. Without this the probe would call
// a fully-migrated brain "incomplete" and the re-run would pay again.
const reconciled = await reconcilePageSignatures(engine, plan);
if (reconciled > 0) {
serr(` [migrate] reconciled the embedding signature on ${reconciled} fully-embedded page(s) (batch-boundary pages).`);
}
const remaining = await engine.countStaleChunks({
signature: `${plan.to_model}:${plan.to_dims}`,
includeNullSignature: true,
});
if (remaining === 0) {
await completeEmbeddingMigration(engine, plan);
if (flags.json) {
console.log(JSON.stringify({ status: 'completed', plan, embedded: embedResult.embedded, remaining: 0 }, null, 2));
} else {
slog(`Migration complete: ${embedResult.embedded} chunk(s) embedded on ${plan.to_model} (${plan.to_dims}d).`);
if (plan.reranker_warning) serr(` [migrate] reminder: ${plan.reranker_warning}`);
}
exit(0);
} else {
if (flags.json) {
console.log(JSON.stringify({ status: 'incomplete', plan, embedded: embedResult.embedded, remaining }, null, 2));
} else {
serr(`Migration incomplete: ${remaining} chunk(s) still stale (embed failures or an interrupted run).`);
serr('Re-run the same command to resume — completed chunks are never re-embedded.');
}
exit(1);
}
}
/** Re-export for the op handler + tests. */
export { MIGRATION_STATE_KEY };
+2 -1
View File
@@ -35,7 +35,8 @@ export async function runSelfUpgrade(args: string[]): Promise<void> {
const force = args.includes('--force');
const json = args.includes('--json');
const release = await fetchLatestRelease();
const result = await fetchLatestRelease();
const release = result.ok ? result : null;
const latest = release ? release.tag.replace(/^v/, '') : null;
const behind = !!latest && isValidVersionString(latest) && isNewerVersion(VERSION, latest);
+90 -5
View File
@@ -12,6 +12,7 @@
import express from 'express';
import type { Request, Response, NextFunction } from 'express';
import type { Server as HttpServer } from 'http';
import cookieParser from 'cookie-parser';
import cors from 'cors';
import rateLimit from 'express-rate-limit';
@@ -46,6 +47,7 @@ import {
type IngestionEvent,
} from '../core/ingestion/types.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
import { registerCleanup } from '../core/process-cleanup.ts';
/**
* /health endpoint timeout. 3s rather than 5s: Fly.io's default
@@ -55,6 +57,71 @@ import { resolveOwnerHolder } from '../core/owner-holder.ts';
*/
export const HEALTH_TIMEOUT_MS = 3000;
/** Exported so tests can type their structural fakes exactly (#3599). */
export type HttpServerLifecycle = Pick<HttpServer, 'listening' | 'once' | 'off' | 'close'>;
/** Exported so tests can type their structural fakes exactly (#3599). */
export type SignalSource = Pick<NodeJS.Process, 'once' | 'off'>;
type CleanupRegistrar = typeof registerCleanup;
/**
* Keep the HTTP server strongly referenced and make the daemon lifetime
* explicit instead of relying on runtime-specific event-loop behavior for an
* unobserved `app.listen()` return value. The shared abnormal-termination
* cleanup pass closes it before process exit.
*/
export function waitForHttpServerLifecycle(
server: HttpServerLifecycle,
options: {
signals?: SignalSource;
register?: CleanupRegistrar;
} = {},
): Promise<void> {
const signals = options.signals ?? process;
const register = options.register ?? registerCleanup;
return new Promise<void>((resolve, reject) => {
let settled = false;
let closePromise: Promise<void> | null = null;
const closeServer = (): Promise<void> => {
if (closePromise) return closePromise;
closePromise = new Promise<void>((closeResolve, closeReject) => {
if (!server.listening) {
closeResolve();
return;
}
server.close((error?: Error) => {
if (error) closeReject(error);
else closeResolve();
});
});
return closePromise;
};
const deregister = register('http-server', closeServer);
const finish = (error?: Error) => {
if (settled) return;
settled = true;
server.off('close', onClose);
server.off('error', onError);
signals.off('SIGINT', onSigint);
deregister();
if (error) reject(error);
else resolve();
};
const onClose = () => finish();
const onError = (error: Error) => finish(error);
const onSigint = () => {
void closeServer().catch(onError);
};
server.once('close', onClose);
server.once('error', onError);
signals.once('SIGINT', onSigint);
});
}
/**
* v0.36.1.x #1024: bootstrap token resolution.
*
@@ -135,6 +202,25 @@ export type ProbeHealthResult =
| { ok: true; status: 200; body: { status: 'ok'; version: string; engine: string; [k: string]: unknown } }
| { ok: false; status: 503; body: { error: 'service_unavailable'; error_description: string } };
/** Exported so tests can type their structural fakes exactly (#3598). */
export type AdminSseResponse = Pick<Response, 'setHeader' | 'flushHeaders' | 'write'>;
/**
* Complete the admin EventSource handshake immediately.
*
* `flushHeaders()` alone can leave reverse proxies and browsers waiting for
* the first response body bytes. An SSE comment is protocol-valid, ignored by
* EventSource consumers, and makes the stream observable end-to-end without
* fabricating an application event.
*/
export function openAdminSseStream(res: AdminSseResponse): void {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
res.write(': connected\n\n');
}
/**
* Pure async health probe. Races `engine.getStats()` against a timeout,
* returns a tagged result. No Express coupling easy to unit-test with a
@@ -1632,10 +1718,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// SSE live activity feed
// ---------------------------------------------------------------------------
app.get('/admin/events', requireAdmin, (req: Request, res: Response) => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
openAdminSseStream(res);
sseClients.add(res);
req.on('close', () => sseClients.delete(res));
@@ -2410,7 +2493,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// ---------------------------------------------------------------------------
const clientCount = await sql`SELECT count(*)::int as count FROM oauth_clients`;
app.listen(port, bind, () => {
const httpServer = app.listen(port, bind, () => {
console.error(`
GBrain MCP Server v${VERSION.padEnd(37)}
@@ -2435,4 +2518,6 @@ ${bootstrapFromEnv
: `║ Admin Token (paste into /admin login): ║\n║ ${bootstrapToken.substring(0, 50)}\n║ ${bootstrapToken.substring(50).padEnd(50)}\n╚══════════════════════════════════════════════════════╝`}
`);
});
await waitForHttpServerLifecycle(httpServer);
}
+81 -4
View File
@@ -2874,10 +2874,17 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
: await resolveSlugByPathOrSourcePath(engine, from, undefined);
// The new path doesn't yet have a row, so resolve from path only.
const newSlug = resolveSlugForPath(to);
// #3056: the cheap rename is OBSERVED, not assumed. A zero-row UPDATE
// doesn't throw, and a thrown collision used to be swallowed by an
// empty catch — both fell through to importFile, which created/updated
// the row at the new path while the old row stayed behind live. Both
// shapes now fall through to the reconcile below.
let renameApplied = false;
try {
await engine.updateSlug(oldSlug, newSlug, renameOpts);
renameApplied = (await engine.updateSlug(oldSlug, newSlug, renameOpts)) > 0;
} catch {
// Slug doesn't exist or collision, treat as add
// Destination slug occupied or invalid — treat as add; the reconcile
// below removes the stale old row once the destination materialized.
}
// Reimport at new path (picks up content changes). Wrapped to match the
// deletes/adds loops: a malformed renamed file is recorded to failedFiles
@@ -2890,9 +2897,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
// repo (committed symlink pointing out).
const filePath = join(gitContextRoot, to);
let importResult: Awaited<ReturnType<typeof importFile>> | undefined;
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
try {
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
importResult = result;
if (result.status === 'imported') chunksCreated += result.chunks;
else if (result.status === 'skipped' && (result as { error?: string }).error) {
failedFiles.push({ path: to, error: String((result as { error?: string }).error) });
@@ -2901,9 +2910,68 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
failedFiles.push({ path: to, error: e instanceof Error ? e.message : String(e) });
}
}
// #3056 reconcile: the rename fell back to add semantics, so the row
// that still represents the OLD path is the stale half of the rename
// (git reported the old path gone; a plain delete of that path would
// remove this row). Two safety rails, both from the #3252 review:
//
// 1. Delete only after the destination demonstrably materialized —
// `imported`, or an errorless `skipped` AT the new slug. Identity
// dedup can skip against the OLD row (result.slug === oldSlug),
// in which case nothing landed at newSlug and deleting the old
// row would destroy the only copy.
// 2. Locate the stale row POSITIVELY by `source_path = from`, never
// by the oldSlug guess — after a collision, a path-derived
// fallback slug could name an unrelated (e.g. manually curated)
// row. No source_path match → nothing is deleted (this also means
// code-strategy imports, which don't populate source_path, fall
// back safely to leaving the old row rather than guessing).
//
// A failed delete records a `<rename:…>` SENTINEL (not an ordinary
// path failure): the gate hard-blocks the bookmark, and — unlike a
// plain path row — the auto-skip valve can never chronic-skip it after
// N attempts, which would advance the bookmark and make a transient
// delete outage a permanent duplicate. The sentinel clears through the
// ordinary success path once the rename converges on a later run.
let reconcileFailed = false;
if (!renameApplied && importResult !== undefined) {
const destMaterialized = importResult.status === 'imported' ||
(importResult.status === 'skipped' && !importResult.error && importResult.slug === newSlug);
if (destMaterialized) {
try {
const staleMap = await engine.resolveSlugsByPaths([from], { sourceId: opts.sourceId ?? DEFAULT_SOURCE_ID });
const staleSlug = staleMap.get(from);
if (staleSlug !== undefined && staleSlug !== newSlug) {
await engine.deletePage(staleSlug, renameOpts);
deletedSlugs.add(staleSlug); // never hand a deleted slug to auto-embed
serr(` [sync] rename reconciled: removed stale row ${staleSlug} (${from} -> ${to} fell back to add).`);
} else if (staleSlug === undefined) {
serr(` [sync] rename fallback: no row has source_path ${from}; stale row (if any) left in place.`);
}
} catch (e: unknown) {
reconcileFailed = true;
failedFiles.push({
path: `<rename:${to}>`,
error: `rename reconcile failed (stale row for ${from} not removed): ` +
`${e instanceof Error ? e.message : String(e)}`,
});
}
} else {
serr(
` [sync] rename fallback: ${from} -> ${to} did not materialize at ${newSlug} ` +
`(import ${importResult.status}); old row left in place.`,
);
}
}
// Converged (cheap rename, clean reconcile, or nothing to reconcile):
// clear any `<rename:…>` sentinel a previous failing run recorded.
if (!reconcileFailed) succeededPaths.push(`<rename:${to}>`);
pagesAffected.push(newSlug);
deletedSlugs.delete(newSlug); // #1284: rename landed on a previously-deleted slug → embeddable again
await markCompleted(to);
// A failed reconcile must NOT checkpoint: banking `to` would make the
// resume filter skip this rename on the retry run, turning a transient
// delete failure into a permanent duplicate — the exact bug being fixed.
if (!reconcileFailed) await markCompleted(to);
progress.tick(1, newSlug);
}
progress.finish();
@@ -3362,7 +3430,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
if (!gate.advanced) {
const codeBreakdown = formatCodeBreakdown(failedFiles);
if (gate.sentinelBlocked) {
// Two sentinel classes block here: `<head>` (pin ancestry broken) and
// `<rename:…>` (#3056 — a rename-reconcile delete failed and advancing
// would permanently bank the duplicate). Pick the message by which fired.
if (gate.sentinelBlocked && failedFiles.some(f => f.path === '<head>')) {
serr(
`\nSync blocked: repository history changed during sync (force-push / reset).\n` +
`${codeBreakdown}\n\n` +
@@ -3370,6 +3441,12 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
`a commit that doesn't match the indexed tree. Re-run sync to re-pin against ` +
`current HEAD.`,
);
} else if (gate.sentinelBlocked) {
serr(
`\nSync blocked: a rename left a stale duplicate that could not be removed:\n` +
`${codeBreakdown}\n\n` +
`The next 'gbrain sync' retries the reconcile from the same diff.`,
);
} else {
const fileFailCount = failedFiles.filter(f => isSkippablePath(f.path)).length;
serr(
+15 -8
View File
@@ -3,6 +3,7 @@
*
* Subcommands:
* takes <slug> list takes for a page
* takes list list all active takes (#2079)
* takes search "<query>" [--who h] keyword search across all takes
* takes add <slug> ...flags append a take (markdown + DB)
* takes update <slug> --row N ...flags update mutable fields
@@ -129,11 +130,10 @@ function writeBody(path: string, body: string): void {
// --- Subcommands ---
async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
const slug = args[0];
if (!slug) {
console.error('Usage: gbrain takes <slug> [--json]');
process.exit(1);
}
// #2079: slug is optional. `gbrain takes list` (no slug) lists ALL active
// takes — CLI parity with the takes_list operation. A leading flag is not
// a slug.
const slug = args[0] && !args[0].startsWith('-') ? args[0] : undefined;
const json = flagPresent(args, '--json');
const holder = flagValue(args, '--who');
const kind = flagValue(args, '--kind') as string | undefined;
@@ -153,17 +153,19 @@ async function cmdList(engine: BrainEngine, args: string[]): Promise<void> {
return;
}
const scope = slug ?? 'this brain';
if (takes.length === 0) {
console.log(`No takes on ${slug}.`);
console.log(`No takes on ${scope}.`);
return;
}
console.log(`# Takes on ${slug}\n`);
console.log(`# Takes on ${scope}\n`);
for (const t of takes) {
const tag = t.active ? '' : ' [superseded]';
const w = Number(t.weight).toFixed(2);
const since = t.since_date ?? '';
const src = t.source ? `${t.source}` : '';
console.log(`#${t.row_num} [${t.kind}${t.holder} • w=${w}${since ? `${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
const where = slug ? '' : `${t.page_slug} `;
console.log(`${where}#${t.row_num} [${t.kind}${t.holder} • w=${w}${since ? `${since}` : ''}]${tag}\n ${t.claim}${src}\n`);
}
}
@@ -555,6 +557,8 @@ export async function runTakes(engine: BrainEngine, args: string[]): Promise<voi
Subcommands:
takes <slug> [--json] [--who h] [--kind k] [--sort weight|since_date|created_at] [--expired]
List takes for a page
takes list [--json] [--who h] [--kind k] [--sort ...] [--expired]
List all active takes across the brain (#2079)
takes search "<query>" [--limit N] [--json]
Keyword search across all takes
takes add <slug> --claim "..." --kind <fact|take|bet|hunch> --who <holder>
@@ -584,6 +588,9 @@ Common flags:
const rest = args.slice(1);
switch (sub) {
// #2079: `takes list` used to be parsed as page slug "list" and printed
// "No takes on list." — reading exactly like an empty takes table.
case 'list': return cmdList(engine, rest);
case 'search': return cmdSearch(engine, rest);
case 'add': return cmdAdd(engine, rest, await resolveTakesSourceId(engine));
case 'update': return cmdUpdate(engine, rest, await resolveTakesSourceId(engine));
+47
View File
@@ -462,6 +462,53 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
// Banner is cosmetic; never block the upgrade.
}
// #3390: ZeroEntropy sunset notice. ZE announced (2026-07-24) that
// its hosted endpoints — including /models/embed and /models/rerank —
// shut down on 2026-09-04. Any brain resolving to a zeroentropyai:*
// embedding model (including default-config brains that never set
// one) loses SEMANTIC RETRIEVAL ENTIRELY on that date: the query
// embedding uses the same endpoint, so existing vectors become
// unqueryable. One-shot per install, gated by
// `ze_sunset_notice_shown` (same pattern as the search-mode banner).
try {
const shown = await engine.getConfig('ze_sunset_notice_shown');
const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL;
const rerankerModel = await engine.getConfig('search.reranker.model');
const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:');
const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:');
if (shown !== 'true' && (onZeEmbedding || onZeReranker)) {
console.log('');
console.log('═══════════════════════════════════════════════════════════════');
console.log('[gbrain] ACTION REQUIRED: ZeroEntropy hosted API sunsets 2026-09-04.');
if (onZeEmbedding) {
console.log(`[gbrain] This brain embeds with ${effectiveModel}. After the sunset,`);
console.log('[gbrain] semantic retrieval STOPS WORKING (queries can no longer be');
console.log('[gbrain] embedded against your existing vectors).');
}
if (onZeReranker) {
console.log(`[gbrain] The reranker (${rerankerModel}) also sunsets; search falls`);
console.log('[gbrain] back to unreranked ordering.');
}
console.log('═══════════════════════════════════════════════════════════════');
console.log('');
console.log('Migrate before the sunset (resumable; preview cost first):');
console.log(' gbrain migrate embeddings --to <provider:model> --dry-run');
console.log(' gbrain migrate embeddings --to <provider:model>');
console.log('');
console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /');
console.log('ollama also works and preserves your existing vectors — point');
console.log('embedding at the local endpoint instead of migrating.');
if (onZeReranker) {
console.log('Reranker: gbrain config set search.reranker.enabled false (or pick another).');
}
console.log('');
await engine.setConfig('ze_sunset_notice_shown', 'true');
}
} catch {
// Banner is cosmetic; never block the upgrade.
}
// PR1: skill-catalog publish consent. New installs default ON at
// `gbrain init`; EXISTING installs stay OFF (default-OFF runtime = no
// silent capability grant on upgrade) until the owner opts in HERE.
+63 -2
View File
@@ -642,8 +642,42 @@ function warnRecipesMissingBatchTokens(): void {
}
}
/** Reset (for tests). */
export function resetGateway(): void {
/**
* Test-only reset baseline (#3554). The bunfig preload
* (`test/helpers/legacy-embedding-preload.ts`) pins the gateway to the legacy
* OpenAI/1536 config at process start, but `resetGateway()` used to wipe that
* pin to `_config = null`. The next test file's engine connect then
* reconfigured from the SHIPPED default (zembed-1 @ 1280) and every 1536-d
* fixture in that file exploded with `expected 1280 dimensions, not 1536`
* a cross-file mine whose placement depended on shard bin-packing.
*
* When a baseline factory is registered, `resetGateway()` means "back to the
* test baseline" instead of "unconfigured": it clears everything as before,
* then re-applies the factory's config via `configureGateway()`. A factory
* (not a frozen config) so each re-application captures fresh
* `process.env`, matching the preload's original `applyLegacy()` semantics.
*
* Production is untouched: nothing in `src/` calls `resetGateway()` or this
* setter, so in production the baseline is never registered and
* `resetGateway()` still fully unconfigures. Same `__*ForTests` seam
* convention as `__setEmbedTransportForTests` above.
*/
let _resetBaseline: (() => AIGatewayConfig) | null = null;
/**
* Register (or clear, with `null`) the config factory that `resetGateway()`
* re-applies. Called once by the bunfig test preload.
*
* @internal exported for tests; not part of the public gateway API.
*/
export function __setGatewayResetBaselineForTests(
factory: (() => AIGatewayConfig) | null,
): void {
_resetBaseline = factory;
}
/** Clear every piece of module state. Shared by both reset flavors. */
function clearGatewayState(): void {
_config = null;
_modelCache.clear();
_shrinkState.clear();
@@ -655,6 +689,33 @@ export function resetGateway(): void {
_extendedModels.clear();
}
/**
* Reset (for tests). Clears all module state (config, model cache, shrink
* state, transports, warned recipes, extended models), then if a test
* baseline is registered re-applies it so the gateway returns to the
* process-wide test default instead of an unconfigured limbo (#3554).
*/
export function resetGateway(): void {
clearGatewayState();
// configureGateway re-clears _modelCache/_shrinkState/_extendedModels and
// registers the baseline's models; transports are NOT touched by it, so a
// stale test transport can never leak back in through this path.
if (_resetBaseline) configureGateway(_resetBaseline());
}
/**
* Reset AND stay unconfigured, ignoring any registered baseline. For the
* handful of tests that assert genuine no-gateway behavior
* (`no_gateway_config` diagnosis, `isAvailable() === false`, graceful
* degradation paths). The preload's per-test beforeEach restores the
* baseline before the next test, so this cannot leak across tests.
*
* @internal exported for tests; not part of the public gateway API.
*/
export function __unconfigureGatewayForTests(): void {
clearGatewayState();
}
/**
* Test-only seam. Replaces the function the gateway calls to embed a
* sub-batch. Pass `null` to restore the real `embedMany` from the AI SDK.
+5
View File
@@ -31,6 +31,11 @@ export const dashscope: Recipe = {
// path. Conservative declaration so the gateway pre-splits before
// hitting whatever undocumented server-side limit exists.
max_batch_tokens: 8192,
// DashScope's OpenAI-compat /embeddings endpoint rejects requests with
// more than 10 input items (documented Model Studio cap). The gateway's
// capBatchItems pre-split enforces this; max_batch_tokens above keeps
// guarding aggregate token size. Concept from community PRs #2643/#2405.
max_batch_items: 10,
// text-embedding-v3 mixes English + CJK heavily; the tokenizer is
// closer to Voyage density than OpenAI tiktoken for CJK-dominant
// content. Conservative chars_per_token=2 leaves headroom.
+11
View File
@@ -180,6 +180,17 @@ export const openrouter: Recipe = {
// to pre-split batches, NOT per-input. Per-input is enforced upstream.
max_batch_tokens: 300_000,
},
// Expansion uses the same routed OpenAI-compatible language-model endpoint
// as chat. Keep a small cheap/fast advisory set; the openai-compat tier
// still accepts any user-configured OpenRouter provider/model ID.
expansion: {
models: [
'anthropic/claude-haiku-4.5',
'google/gemini-3-flash-preview',
'deepseek/deepseek-chat',
],
price_last_verified: '2026-05-20',
},
chat: {
// Curated entry points (verified against OR's catalog 2026-05-20). The
// openai-compat tier does NOT enforce this list at runtime — users can
+125 -11
View File
@@ -40,6 +40,7 @@ export const LINKABLE_ENTITY_TYPES = ['person', 'company', 'organization', 'enti
* types in.
*/
const MIN_NAME_LENGTH = 4;
const MIN_CJK_NAME_LENGTH = 2;
/**
* Built-in ignore list common ambiguous tokens whose body-text mentions
@@ -104,12 +105,12 @@ export interface FindMentionsOpts {
// ============================================================
/**
* Token-only tokenizer. Returns `[token, offset]` pairs for every
* `[a-zA-Z0-9]+` run, lowercased. Non-ASCII (CJK, accented) is
* deliberately not tokenized in v1 entity gazetteer is English-dominant
* in production today. Widening to `\p{L}+` is a future option once a
* real CJK entity catalog appears (filed under TODO-1 + a TODO for
* Unicode-aware tokenization).
* Token-only tokenizer. Returns `[token, offset]` pairs.
*
* ASCII: each `[a-zA-Z0-9]+` run is a single token, lowercased.
* CJK: each CJK character (Chinese/Japanese/Korean) is an individual
* token, lowercased. This allows the normal maximal-munch scan path
* to reach CJK gazetteer entries without a separate substring pass.
*
* Possessive "Acme's" tokenizes as ['acme', 's'] (single-quote breaks the
* run) single-word "Acme" lookup succeeds at offset 0; the trailing 's'
@@ -127,18 +128,129 @@ function tokenizeForScan(text: string): ScannedToken[] {
const out: ScannedToken[] = [];
TOKEN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
// Collect ASCII token spans first.
const asciiSpans: Array<{ start: number; end: number }> = [];
while ((m = TOKEN_RE.exec(text)) !== null) {
out.push({ text: m[0].toLowerCase(), offset: m.index, length: m[0].length });
asciiSpans.push({ start: m.index, end: m.index + m[0].length });
}
// Walk character-by-character: emit ASCII tokens at their start positions,
// then emit individual CJK characters for non-ASCII positions that fall
// outside ASCII token spans.
let asciiIdx = 0;
for (let i = 0; i < text.length;) {
const cp = text.codePointAt(i) ?? 0;
const isCJK = (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af);
// Advance asciiIdx past any spans that end before or at i.
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) {
asciiIdx++;
}
// If position i is inside an ASCII token span, emit the full ASCII token
// and jump past it.
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
const span = asciiSpans[asciiIdx]!;
const token = text.slice(span.start, span.end);
out.push({ text: token.toLowerCase(), offset: span.start, length: token.length });
i = span.end;
asciiIdx++;
continue;
}
// CJK: emit as individual character token.
if (isCJK) {
const charLen = cp > 0xffff ? 2 : 1; // surrogate pair
const charStr = text.slice(i, i + charLen);
out.push({ text: charStr.toLowerCase(), offset: i, length: charLen });
i += charLen;
} else {
i++;
}
}
return out;
}
function hasCJK(s: string): boolean {
for (const ch of s) {
const cp = ch.codePointAt(0) ?? 0;
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af)) return true;
}
return false;
}
function cjkCharCount(s: string): number {
let count = 0;
for (const ch of s) {
const cp = ch.codePointAt(0) ?? 0;
if ((cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af)) count++;
}
return count;
}
/**
* Tokenize a page title for gazetteer insertion.
*
* ASCII titles: standard `[a-zA-Z0-9]+` tokenization, lowercased.
* CJK titles (no ASCII content): split into individual characters
* e.g. "纳瓦尔" ["纳","瓦","尔"]. This allows normal multi-token
* maximal-munch matching to work with character-level CJK tokens
* produced by `tokenizeForScan`.
* Mixed CJK+ASCII titles: ASCII parts tokenized normally, CJK parts
* split into individual characters.
*/
function tokenizeTitle(title: string): string[] {
const tokens: string[] = [];
TOKEN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
while ((m = TOKEN_RE.exec(title)) !== null) tokens.push(m[0].toLowerCase());
return tokens;
const hasAscii = TOKEN_RE.test(title);
if (hasAscii) {
// Mixed ASCII+CJK or pure ASCII: tokenize ASCII normally, then
// append individual CJK characters in order.
TOKEN_RE.lastIndex = 0;
let m: RegExpExecArray | null;
const asciiSpans: Array<{ start: number; end: number; text: string }> = [];
while ((m = TOKEN_RE.exec(title)) !== null) {
asciiSpans.push({ start: m.index, end: m.index + m[0].length, text: m[0].toLowerCase() });
}
let asciiIdx = 0;
for (let i = 0; i < title.length;) {
while (asciiIdx < asciiSpans.length && asciiSpans[asciiIdx]!.end <= i) asciiIdx++;
if (asciiIdx < asciiSpans.length && i >= asciiSpans[asciiIdx]!.start && i < asciiSpans[asciiIdx]!.end) {
tokens.push(asciiSpans[asciiIdx]!.text);
i = asciiSpans[asciiIdx]!.end;
asciiIdx++;
continue;
}
const cp = title.codePointAt(i) ?? 0;
if (hasCJK(title[i]!)) {
const charLen = cp > 0xffff ? 2 : 1;
tokens.push(title.slice(i, i + charLen).toLowerCase());
i += charLen;
} else {
i++;
}
}
return tokens;
}
// Pure CJK (no ASCII content): split into individual characters.
if (hasCJK(title)) {
for (let i = 0; i < title.length;) {
const cp = title.codePointAt(i) ?? 0;
const charLen = cp > 0xffff ? 2 : 1;
tokens.push(title.slice(i, i + charLen).toLowerCase());
i += charLen;
}
return tokens;
}
// Non-ASCII, non-CJK title (emoji, symbols, etc.) — empty set.
return [];
}
/**
@@ -175,7 +287,9 @@ export async function buildGazetteer(
const gazetteer: Gazetteer = new Map();
for (const row of rows) {
if (!row.title || row.title.length < MIN_NAME_LENGTH) continue;
if (!row.title) continue;
if (!hasCJK(row.title) && row.title.length < MIN_NAME_LENGTH) continue;
if (hasCJK(row.title) && cjkCharCount(row.title) < MIN_CJK_NAME_LENGTH) continue;
if (ignoreSet.has(row.title) && !existingTitles.has(row.title)) continue;
const tokens = tokenizeTitle(row.title);
+42 -7
View File
@@ -20,6 +20,7 @@
import { chunkText as recursiveChunk } from './recursive.ts';
import { buildQualifiedName } from './qualified-names.ts';
import { CJK_SLUG_CHARS, CJK_RANGES_REGEX } from '../cjk.ts';
// Embed the tree-sitter runtime + per-language grammars as files.
// `with { type: 'file' }` returns a path (string) at runtime. Bun bundles
@@ -716,7 +717,7 @@ export async function chunkCodeTextFull(
}
if (chunks.length === 0) {
return { chunks: capOversizedChunks(fallbackChunks(source, filePath, language, opts), filePath, language, opts), edges: rawEdges };
return { chunks: fallbackChunks(source, filePath, language, opts), edges: rawEdges };
}
return { chunks: capOversizedChunks(mergeSmallSiblings(chunks, chunkTarget), filePath, language, opts), edges: rawEdges };
} catch {
@@ -842,10 +843,10 @@ function capOversizedChunks(
opts: CodeChunkOptions,
): CodeChunk[] {
const cap = opts.maxChunkTokens ?? DEFAULT_MAX_CHUNK_TOKENS;
if (!chunks.some((c) => estimateTokens(c.text) > cap)) return chunks;
if (!chunks.some((c) => estimateEmbedTokens(c.text) > cap)) return chunks;
const out: CodeChunk[] = [];
for (const c of chunks) {
if (estimateTokens(c.text) <= cap) {
if (estimateEmbedTokens(c.text) <= cap) {
out.push({ ...c, index: out.length });
continue;
}
@@ -880,17 +881,43 @@ function splitToTokenBudget(text: string, cap: number, opts: CodeChunkOptions):
chunkOverlap: opts.fallbackOverlapWords ?? 50,
}).map((p) => p.text);
for (const piece of pieces) {
if (estimateTokens(piece) <= cap) {
if (estimateEmbedTokens(piece) <= cap) {
out.push(piece);
continue;
}
// ~3.5 chars/token is a conservative cl100k estimate for source text.
const charBudget = Math.max(1, Math.floor(cap * 3.5));
// Hard-split slice size. Pure-ASCII pieces: ~3.5 chars/token is a
// conservative cl100k estimate for source text. CJK-containing pieces:
// the weighted estimate can reach 1 token/char, so budget 1 char/token
// to keep every slice under cap by construction.
const charBudget = Math.max(1, Math.floor(cap * (CJK_RANGES_REGEX.test(piece) ? 1 : 3.5)));
for (let i = 0; i < piece.length; i += charBudget) out.push(piece.slice(i, i + charBudget));
}
return out;
}
const CJK_CHARS_G = new RegExp(`[${CJK_SLUG_CHARS}]`, 'g');
/**
* Embedding-safe token estimate for the oversize cap. estimateTokens
* (cl100k) matches embedding-family tokenizers closely on pure-ASCII source
* (measured identical on English prose and JSON vs Qwen3-Embedding), but
* UNDERCOUNTS mixed CJK+ASCII chunks measured 31% on URL-dense Korean
* text vs the Qwen3 embedding tokenizer, which is exactly the shape that
* overflows strict embedding backends (#2826). For chunks containing CJK,
* take the max of cl100k and a per-char-class overestimate (CJK 1.0/char,
* other non-whitespace 0.75/char, whitespace 0.1/char). CJK-DOMINANT text
* is unaffected too: cl100k already counts it above the weighted form, so
* max() returns the same value as today. Only mixed-script chunks the
* measured divergence class estimate higher.
*/
export function estimateEmbedTokens(text: string): number {
const cjk = (text.match(CJK_CHARS_G) || []).length;
if (cjk === 0) return estimateTokens(text);
const ws = (text.match(/\s/g) || []).length;
const weighted = Math.ceil(cjk + (text.length - cjk - ws) * 0.75 + ws * 0.1);
return Math.max(estimateTokens(text), weighted);
}
// ---------- Internals ----------
function fallbackChunks(
@@ -901,7 +928,7 @@ function fallbackChunks(
): CodeChunk[] {
const size = opts.fallbackChunkSizeWords ?? 300;
const overlap = opts.fallbackOverlapWords ?? 50;
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
const chunks = recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
buildChunk({
body: chunk.text, filePath, language,
symbolName: null, symbolType: 'module',
@@ -909,6 +936,14 @@ function fallbackChunks(
index,
}),
);
// Route every fallback emission through the oversize net. Previously only
// the empty-AST branch wrapped its fallback in capOversizedChunks — the
// no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence:
// their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches
// shipped word-counted chunks unchecked, and the word pipeline undercounts
// exactly the dense content (JSON, minified, CJK-mixed) that overflows
// embedders. Hoisting the cap here covers all five paths at once.
return capOversizedChunks(chunks, filePath, language, opts);
}
function buildChunk(input: {
+28 -5
View File
@@ -21,12 +21,35 @@ export const CJK_SLUG_CHARS = '一-鿿぀-ゟ゠-ヿ가-힯';
export const CJK_RANGES_REGEX = new RegExp(`[${CJK_SLUG_CHARS}]`);
/**
* Page-slug segment grammar (no anchors): alnum-or-CJK lead char, then
* alnum/CJK/hyphen continuation. Single source for validatePageSlug
* (operations.ts), SlugRegistry's SLUG_RE, and the dream-cycle
* SUMMARY_SLUG_RE so every slug validator shares one grammar (#738).
* Slug "word" character class (#3417): every script's letters, not just
* Latin + CJK. Unicode property escapes REQUIRES the `u` flag on any
* regex composed from this string (without `u`, `\p{Ll}` silently matches
* the literal chars `p`, `L`, `l`, `{`, `}`).
*
* \p{Ll} lowercase letters (a-z, Cyrillic/Greek lowercase, đ, )
* \p{Lm} modifier letters
* \p{Lo} caseless-script letters (Hebrew, Arabic, Thai, CJK, Devanagari, )
* \p{M} combining marks that survive the Latin accent-strip pass
* (Hebrew niqqud, Arabic harakat, Thai/Devanagari vowel signs)
* \p{N} numbers (0-9, Arabic-Indic digits, )
*
* Uppercase (\p{Lu}/\p{Lt}) is deliberately excluded: slugifySegment()
* lowercases before filtering, so validators stay lowercase-canonical.
*
* Distinct from CJK_SLUG_CHARS above, which also drives the
* countCJKAwareWords density heuristic do NOT merge the two, or slug
* grammar changes silently change chunking behavior.
*/
export const PAGE_SLUG_SEG = `[a-z0-9${CJK_SLUG_CHARS}][a-z0-9${CJK_SLUG_CHARS}\\-]*`;
export const SLUG_WORD_CHARS = '\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}\\p{N}';
/**
* Page-slug segment grammar (no anchors): word-char lead, then word-char or
* hyphen continuation. Single source for validatePageSlug (operations.ts),
* SlugRegistry's SLUG_RE, and the dream-cycle SUMMARY_SLUG_RE so every slug
* validator shares one grammar (#738). Compose with the `u` flag see
* SLUG_WORD_CHARS.
*/
export const PAGE_SLUG_SEG = `[${SLUG_WORD_CHARS}][${SLUG_WORD_CHARS}\\-]*`;
export const CJK_SENTENCE_DELIMITERS = ['。', '', '']; // 。!?
export const CJK_CLAUSE_DELIMITERS = ['', '', '', '、']; // ;:,、
+13
View File
@@ -145,6 +145,19 @@ export function computeCorpusGeneration(args: {
return h.digest('hex').slice(0, 16);
}
/**
* #3507 the corpus_generation a page lands on when a plain re-embed path
* (`embed --stale` and friends) re-embeds a `per_chunk_synopsis` page at the
* title-only tier (the D14 fallback tier; synopsis re-generation is a paid
* backfill concern). Callers restamp
* `updatePageContextualRetrievalState(slug, sourceId, 'title', titleTierCorpusGeneration())`
* so the stamped mode keeps describing the vectors actually in the column.
* Matches what the inline import path writes for its title-tier pages.
*/
export function titleTierCorpusGeneration(): string {
return computeCorpusGeneration({ crMode: 'title', haikuModel: DEFAULT_HAIKU_MODEL });
}
/**
* Compute source_text_hash for D27 P1-4 cache key composition. The
* synopsis cache invalidates correctly when adjacent text changes (page
@@ -0,0 +1,94 @@
/**
* Block-format conversation normalizer.
*
* Some chat exports notably the Slack collector gbrain's own ingestion
* uses emit a HEADER + indented-body BLOCK per message instead of the
* single-line `**Name** (time): body` shape the built-in patterns
* (`builtins.ts`) recognize:
*
* - **Theo** (Mon 11:18)
* Hey everyone quick update on the renewal.
*
* Second paragraph of the same message.
* - **Juan** (Mon 11:20)
* Reply body...
*
* None of the 14 line-oriented built-ins match this: a leading `- ` list
* marker, a day-of-week + time with no trailing colon, and the message body on
* the following indented lines. Result: `phase: 'no_match'`, zero messages,
* and the whole comms corpus is silently un-extractable (facts stay empty
* `find_trajectory` returns nothing).
*
* This collapses each block into the canonical `**Name** (HH:MM): <body joined
* to one line>` shape so the existing `bold-paren-time` pattern matches; the
* per-message date fills in downstream via `fallbackDate` (the page date).
*
* STRICT no-op unless the block signature is present: the header regex requires
* the paren-group to END the line (no inline `: body`), which is exactly what
* the single-line patterns always produce so feeding already-canonical
* content through this function returns it unchanged.
*/
// `- **Name** (Mon 11:18)` / `- **Name** (11:18 AM)` / `- **Name** (16:36)`.
// Day-of-week optional; 12h/24h time; optional am/pm; the line ENDS at the
// close paren (no inline `: body` — that is what distinguishes a block header
// from the single-line `**Name** (time): body` patterns).
const BLOCK_HEADER =
/^\s*-\s+\*\*(.+?)\*\*\s+\((?:[A-Za-z]{2,9}\.?\s+)?(\d{1,2}):(\d{2})(?::\d{2})?\s*([AaPp][Mm])?\)\s*$/;
/** True when at least one line is a block-format message header. */
export function looksLikeBlockConversation(body: string): boolean {
for (const line of body.split('\n')) {
if (BLOCK_HEADER.test(line)) return true;
}
return false;
}
function to24h(hour: number, ampm?: string): number {
if (!ampm) return hour;
const lower = ampm.toLowerCase();
if (lower === 'pm' && hour < 12) return hour + 12;
if (lower === 'am' && hour === 12) return 0;
return hour;
}
/**
* Collapse block-format messages into canonical single-line `**Name** (HH:MM):
* body` lines. Returns `body` unchanged when no block header is present.
*/
export function normalizeBlockConversation(body: string): string {
if (!looksLikeBlockConversation(body)) return body;
const lines = body.split('\n');
const out: string[] = [];
let current: { name: string; time: string } | null = null;
let bodyParts: string[] = [];
const flush = () => {
if (current) {
const text = bodyParts.join(' ').replace(/\s+/g, ' ').trim();
out.push(`**${current.name}** (${current.time}): ${text}`);
}
current = null;
bodyParts = [];
};
for (const line of lines) {
const m = BLOCK_HEADER.exec(line);
if (m) {
flush();
const hour = to24h(parseInt(m[2], 10), m[4]);
const time = `${String(hour).padStart(2, '0')}:${m[3]}`;
current = { name: m[1].trim(), time };
} else if (current) {
// Body line of the current message. Drop blank lines; keep the rest.
const trimmed = line.trim();
if (trimmed) bodyParts.push(trimmed);
}
// Lines before the first header (page title, leading blanks) are dropped —
// they never matched a pattern anyway.
}
flush();
return out.length > 0 ? out.join('\n') : body;
}
+7
View File
@@ -29,6 +29,7 @@ import {
BUILTIN_PATTERNS,
cleanSpeaker,
} from './builtins.ts';
import { normalizeBlockConversation } from './normalize-block.ts';
import type {
DateContext,
MatchedMessage,
@@ -473,6 +474,12 @@ export function parseConversation(
return { messages: [], phase: 'no_match' };
}
// Pre-pass: collapse block-format chat exports (header + indented body, e.g.
// the Slack collector's `- **Name** (Mon 11:18)\n body…`) into the canonical
// single-line shape the built-in patterns recognize. Strict no-op when no
// block header is present, so already-canonical content is untouched.
body = normalizeBlockConversation(body);
const dateCtx = deriveDateContext(opts);
// Assemble candidate pool: built-ins (minus disabled) + user patterns.
+13 -2
View File
@@ -895,8 +895,17 @@ export async function resolveSourceForDir(
// (the cycleSourceId precedence) or 'default'.
if (brainDir === null) return undefined;
try {
// #2540: exclude archived rows (dream's --source guard refuses to stamp
// them, so an archived alias winning here means the stamp silently never
// lands and doctor's cycle_freshness stays red on a healthy install) and
// order deterministically so a duplicate registration of the same path
// can't shadow the active source on whichever row the engine scans first.
// Ordering matches listAllSources/sources-ops for operator-output parity.
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
`SELECT id FROM sources
WHERE local_path = $1 AND archived = false
ORDER BY (id = 'default') DESC, id
LIMIT 1`,
[brainDir],
);
if (rows[0]) return rows[0].id;
@@ -1179,7 +1188,9 @@ async function runPhaseExtractFacts(
summary: `extract_facts skipped: ${result.legacyRowsPending} legacy v0.31 facts pending fence backfill`,
details: {
legacyRowsPending: result.legacyRowsPending,
hint: 'gbrain apply-migrations --yes',
// A bare `apply-migrations --yes` no-ops once the v0.32.2 ledger
// entry is complete; the retry marker is what re-runs Phase B.
hint: 'gbrain apply-migrations --force-retry 0.32.2 && gbrain apply-migrations --yes',
warnings: result.warnings,
},
};
+89 -28
View File
@@ -17,20 +17,26 @@
* DB rows need cleanup (#1781 the unconditional wipe-and-reinsert
* made every cycle non-idempotent, re-appending duplicate rows).
*
* After the phase, the DB index for every affected page matches the
* fence's canonical (claim, source) row set (modulo embeddings +
* runtime-derived fields). Pages with no fence wipe DB rows for that
* page coordinate only; legacy NULL-source_markdown_slug rows survive
* because deleteFactsForPage targets source_markdown_slug = slug only.
* After the phase, the DB index for every cleanly parsed affected page
* matches the fence's canonical (claim, source) row set (modulo embeddings
* + runtime-derived fields). Warning-bearing parses are non-authoritative
* and preserve that page's existing index. Pages with no fence wipe DB rows
* for that page coordinate only; legacy NULL-source_markdown_slug rows
* survive because deleteFactsForPage targets source_markdown_slug = slug only.
*
* Empty-fence guard (Codex R2-#7; #2484): the phase refuses to do its
* destructive reconciliation pass when genuinely-backfillable legacy
* rows still exist `row_num IS NULL` (never fenced) AND `entity_slug`
* resolves to a live page in this source (so the v0_32_2 migration's
* Phase B could fence them). Status returns `warn` with a hint to run
* `gbrain apply-migrations --yes`. Without the guard, an interrupted
* upgrade where v0_32_2 hasn't run could leave the cycle silently
* misreporting "0 facts on people/alice" while legacy rows linger.
* Empty-fence guard (Codex R2-#7; #2484; #2646): the phase refuses to do
* its destructive reconciliation pass when genuinely-backfillable legacy
* rows still exist in THIS run's source only (`source_id = sourceId`;
* a pending row in source A must not jam extraction for source B the
* source-isolation invariant) `row_num IS NULL` (never fenced) AND
* `entity_slug` resolves to a live page in this source (so the v0_32_2
* migration's Phase B could fence them) AND the row is not soft-expired
* (`expired_at IS NULL`). Status returns `warn` with a hint to re-run
* the v0.32.2 fence backfill (`apply-migrations --force-retry 0.32.2`
* then `--yes` a bare `--yes` is a no-op once the ledger says
* complete). Without the guard, an interrupted upgrade where v0_32_2
* hasn't run could leave the cycle silently misreporting "0 facts on
* people/alice" while legacy rows linger.
*
* The live-page requirement (#2484) is load-bearing: the inline facts
* writer keeps producing `row_num IS NULL, entity_slug IS NOT NULL`
@@ -41,6 +47,10 @@
* the phase jams forever (~16/day observed). Requiring a backing page
* keeps genuine pre-v0.32.2 rows (whose entity page exists) gating
* while excluding the inline-writer's permanent-unfenceable rows.
*
* Soft-expired rows don't count either (#2646): they're what
* `forget_fact` produces, so excluding them lets operators drain the
* backlog through the sanctioned removal path instead of raw SQL.
*/
import type { BrainEngine } from '../engine.ts';
@@ -88,6 +98,24 @@ function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFac
* neither count as "stale" (which would force a wipe every cycle) nor
* be compared against the fence's row set. Mirrors the
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
*
* Also excludes soft-expired legacy rows (#2646: `row_num IS NULL AND
* expired_at IS NOT NULL`) — rows that `forget_fact` expired via its
* legacy DB-only path. They are not fence-owned (fence rows always
* carry a row_num), so they must neither count as "stale" (forcing a
* wipe every cycle) nor mask a fence row from insertion. Mirrors the
* preserveExpiredLegacy filter deleteFactsForPage applies on the wipe.
*
* Deliberate consequence: if the fence still carries the same
* (claim, source) as an expired legacy row, the reconcile inserts it
* as a fresh ACTIVE fence-owned row. That is the fence-is-canonical
* contract working as documented legacy DB-only forgets "DO NOT
* survive rebuild" (see forget.ts header); suppressing the insert
* would instead create silent fenceDB divergence, the exact failure
* mode the empty-fence guard exists to prevent. To durably forget
* such a claim, forget the fence-owned row (forget_fact now takes the
* fence path, which strikes the row through in markdown). The expired
* legacy row survives alongside as the record of the earlier forget.
*/
async function listExistingFactsForPage(
engine: BrainEngine,
@@ -100,6 +128,7 @@ async function listExistingFactsForPage(
WHERE source_id = $1
AND source_markdown_slug = $2
AND COALESCE(source, '') NOT LIKE 'cli:%'
AND NOT (row_num IS NULL AND expired_at IS NOT NULL)
ORDER BY row_num ASC, id ASC`,
[sourceId, slug],
);
@@ -173,7 +202,7 @@ export async function runExtractFacts(
phantomsMorePending: false,
};
// ── Empty-fence guard (Codex R2-#7; #2484) ─────────────────────
// ── Empty-fence guard (Codex R2-#7; #2484; #2646) ──────────────
// Pre-check: if any genuinely-backfillable legacy fact rows exist,
// refuse to run the destructive reconciliation pass — the v0_32_2
// orchestrator must fence them first.
@@ -181,12 +210,13 @@ export async function runExtractFacts(
// A row is a real backfill candidate only when `row_num IS NULL`
// (never fenced) AND its `entity_slug` resolves to a LIVE page in
// this source (the migration's Phase B only fences rows whose
// entity_slug maps to a writable page). #2484: the original
// predicate was just `row_num IS NULL AND entity_slug IS NOT NULL`,
// which ALSO matched structurally-unfenceable hot-memory rows the
// inline writer keeps producing post-migration: the legacy DB-only
// fallback (backstop.ts) writes `entity_slug` (a resolved slug, e.g.
// a slugify-floor or stub-guard-blocked unprefixed slug like
// entity_slug maps to a writable page) AND it is not soft-expired.
// #2484: the original predicate was just `row_num IS NULL AND
// entity_slug IS NOT NULL`, which ALSO matched
// structurally-unfenceable hot-memory rows the inline writer keeps
// producing post-migration: the legacy DB-only fallback
// (backstop.ts) writes `entity_slug` (a resolved slug, e.g. a
// slugify-floor or stub-guard-blocked unprefixed slug like
// `people-jane-doe`) with `row_num` NULL whenever the slug has no
// fenceable page. Those rows can never satisfy the migration's exit
// condition (no page to fence onto, and `apply-migrations` is a
@@ -194,27 +224,49 @@ export async function runExtractFacts(
// — ~16/day, mislabeled "v0.31 pending backfill." We now require a
// live backing page, which both genuine pre-v0.32.2 rows (their
// entity page exists) satisfy and inline-writer unfenceable rows do
// not.
// not. #2646: soft-expired rows (`expired_at IS NOT NULL`) are also
// excluded — `forget_fact`, the officially sanctioned removal path,
// soft-expires legacy rows rather than deleting them, so counting
// expired rows would leave the guard permanently stuck with no
// supported way to drain the backlog.
//
// Source isolation (#3526): the count is scoped to THIS run's
// sourceId. The pre-fix query counted brain-wide, so a single pending
// legacy row in any mounted source jammed extract_facts for every
// source — a cross-source leak of one source's migration state into
// another's cycle (CLAUDE.md source-isolation invariant).
const legacy = await engine.executeRaw<{ n: string }>(
`SELECT COUNT(*) AS n
FROM facts f
WHERE f.row_num IS NULL
WHERE f.source_id = $1
AND f.row_num IS NULL
AND f.entity_slug IS NOT NULL
AND f.expired_at IS NULL
AND EXISTS (
SELECT 1 FROM pages p
WHERE p.source_id = f.source_id
AND p.slug = f.entity_slug
AND p.deleted_at IS NULL
)`,
[sourceId],
);
const legacyCount = parseInt(legacy[0]?.n ?? '0', 10);
result.legacyRowsPending = legacyCount;
if (legacyCount > 0) {
result.guardTriggered = true;
// Drain advice must actually work: a bare `apply-migrations --yes`
// is a no-op once the v0.32.2 ledger entry says complete (the
// runner classifies it as already-applied), so the sanctioned
// re-run path is the explicit retry marker first. Phase B is
// idempotent — it only touches `row_num IS NULL` rows and de-dupes
// against the existing fence — so the re-run is safe. Individual
// rows can instead be drained through `forget_fact` (soft-expired
// rows stop counting).
result.warnings.push(
`extract_facts: ${legacyCount} legacy v0.31 fact rows (entity page present, not yet ` +
`fenced) pending fence backfill. Run \`gbrain apply-migrations --yes\` to complete ` +
`v0_32_2 before this phase can safely reconcile fence → DB.`,
`extract_facts: ${legacyCount} legacy v0.31 fact rows in source "${sourceId}" ` +
`(entity page present, not yet fenced) pending fence backfill. Re-run the v0.32.2 ` +
`fence backfill: \`gbrain apply-migrations --force-retry 0.32.2\` then ` +
`\`gbrain apply-migrations --yes\`. Or drain individual rows via \`forget_fact\`.`,
);
return result;
}
@@ -303,6 +355,11 @@ export async function runExtractFacts(
result.warnings.push(
...parsed.warnings.map(w => `${slug}: ${w}`),
);
// The parser deliberately skips malformed rows and returns any rows it
// could still recover. That partial result is not authoritative: using
// it for reconciliation would interpret skipped rows as deletions.
// Preserve this page's existing index and continue with other pages.
continue;
}
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
@@ -334,9 +391,12 @@ export async function runExtractFacts(
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
// (conversation facts from extract-conversation-facts) are NOT
// fence-owned — the page carries no `## Facts` fence to recreate
// them — so they MUST survive this reconcile.
// them — so they MUST survive this reconcile. #2646: soft-expired
// legacy rows (forget_fact's record of the forget) likewise
// survive via preserveExpiredLegacy.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
preserveExpiredLegacy: true,
});
result.factsDeleted += deleted.deleted;
}
@@ -363,10 +423,11 @@ export async function runExtractFacts(
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
// Fall back to the legacy page-level reconcile when old DB rows must
// be removed. Same delete scoping as above: legacy
// NULL-source_markdown_slug rows and `cli:`-origin conversation
// facts (#1928) survive.
// NULL-source_markdown_slug rows, `cli:`-origin conversation
// facts (#1928), and soft-expired legacy rows (#2646) survive.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
preserveExpiredLegacy: true,
});
result.factsDeleted += deleted.deleted;
toInsert = extracted;
+3 -2
View File
@@ -48,8 +48,9 @@ import { safeSplitIndex } from '../text-safe.ts';
import { PAGE_SLUG_SEG } from '../cjk.ts';
// Slug grammar from validatePageSlug — shared via PAGE_SLUG_SEG (#738).
// Used for the orchestrator-written summary index slug.
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`);
// Used for the orchestrator-written summary index slug. `u` flag required
// by PAGE_SLUG_SEG's \p{...} classes (#3417).
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'u');
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
+1
View File
@@ -112,6 +112,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'takes_weight_grid',
'timeline_coverage',
'unified_multimodal_coverage',
'unverified_extractions',
'voice_gate_health',
]);
+17 -2
View File
@@ -19,7 +19,8 @@
import type { BrainEngine } from './engine.ts';
import type { ChunkInput } from './types.ts';
import { embedBatchWithBackoff } from '../commands/embed.ts';
import { embedBatchWithBackoff, restampIfDemotedToTitleTier } from '../commands/embed.ts';
import { wrapChunkTextsForStoredMode } from './embedding-context.ts';
import { type DbPacer, createNoopPacer, observed } from './db-pacer.ts';
import { AbortError } from './abort-check.ts';
@@ -189,8 +190,15 @@ export async function embedStaleForSource(
const keySourceId = stale[0]?.source_id ?? sourceId;
const slug = stale[0].slug;
try {
// #3507: fetch the page row for its title + stored CR mode so the
// re-embed reproduces the page's wrapping convention instead of
// silently stripping contextual prefixes (mirrors
// src/commands/embed.ts:embedAllStale).
const pageRow = await observed(pacer, () =>
engine.getPage(slug, { sourceId: keySourceId }),
);
const embeddings = await embedFn(
stale.map((c) => c.chunk_text),
wrapChunkTextsForStoredMode(pageRow, stale),
{ abortSignal: signal },
);
const existing = await observed(pacer, () =>
@@ -233,6 +241,13 @@ export async function embedStaleForSource(
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
}
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest (mixed pages stay as-is).
if (stale.length === existing.length) {
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
);
}
result.embedded += stale.length;
result.pagesProcessed += 1;
} catch (e: unknown) {
+38
View File
@@ -186,3 +186,41 @@ export function modeRequiresHaiku(mode: CRMode): boolean {
export function modeRequiresWrapper(mode: CRMode): boolean {
return mode !== 'none';
}
/**
* #3507 build the embedding inputs for a re-embed of EXISTING chunk rows,
* reproducing the wrapping convention the page's vectors were originally
* built under (recorded in `pages.contextual_retrieval_mode`).
*
* Used by every plain re-embed path (`embed <slug>`, `embed --all`,
* `embed --stale`, the embed-backfill Minion loop). Before this helper those
* paths embedded raw `chunk_text`, so any re-embed including the NORMAL
* post-model-migration `embed --stale` silently replaced context-wrapped
* vectors with unwrapped ones, degrading retrieval with no signature change
* to show for it.
*
* Convention rules (embed PRESERVES conventions; changing them is
* sync/reindex's job):
* - mode NULL/undefined/'none' raw chunk_text (status quo).
* - mode 'title' title-only prefix (pure string concat).
* - mode 'per_chunk_synopsis' title-only prefix. Re-generating Haiku
* synopses is a paid backfill concern; title-only is the service's own
* documented fallback tier (D14). Callers that fully re-embed a page
* this way should restamp the page to 'title' so the column stays
* honest (see contextual-retrieval-service.ts:titleTierCorpusGeneration).
* - `fenced_code` chunks are NEVER wrapped (D20-T4), same as sync.
*/
export function wrapChunkTextsForStoredMode(
page:
| { title?: string | null; contextual_retrieval_mode?: CRMode | null }
| null
| undefined,
chunks: ReadonlyArray<{ chunk_text: string; chunk_source?: string | null }>,
): string[] {
const mode = page?.contextual_retrieval_mode;
if (mode == null || !modeRequiresWrapper(mode)) {
return chunks.map((c) => c.chunk_text);
}
const prefix = buildContextualPrefix(page?.title ?? '', null);
return chunks.map((c) => wrapChunkForEmbedding(c.chunk_text, prefix, c.chunk_source));
}
+340
View File
@@ -0,0 +1,340 @@
/**
* Provider-agnostic embedding migration (#3390).
*
* `gbrain migrate embeddings --to <provider:model>` re-embeds a brain onto
* any configured provider the forward path off a sunsetting provider that
* `ze-switch` (ZE-only target) and `ze-switch --undo` (needs a snapshot fresh
* installs don't have) cannot cover.
*
* Deliberately thin: everything heavy is reused
* - runSchemaTransition (retrieval-upgrade-planner.ts) for dimension changes
* - invalidateStaleSignatureEmbeddings + the NULL-embedding cursor for
* staleness + resume (the NULL column IS the checkpoint: a killed run
* re-runs the same command and continues where it stopped)
* - the embed pipeline (src/commands/embed.ts) for the actual re-embed,
* with pacing, backfill locks, rate-limit backoff, and progress
* - lookupEmbeddingPrice / estimateCostFromChars for the preflight estimate
* - detectEnvOverride (the #1421 damage-class gate) before any mutation
*
* #3391 companion fix: the migration widens staleness with
* `includeNullSignature: true` so pages that predate the v108 signature stamp
* are re-embedded too, instead of silently staying in the old embedding space.
*
* The command layer (src/commands/migrate-embeddings.ts) owns everything
* process-shaped: confirm prompts, file-plane config persistence (the gateway
* reads file/env, not the DB plane), gateway reconfiguration, and the embed
* catch-up run. This module is engine-pure so both engines and the op handler
* share one implementation.
*/
import type { BrainEngine } from './engine.ts';
import { resolveRecipe, embeddingDimsForModel } from './ai/model-resolver.ts';
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
import { detectEnvOverride, type EnvOverrideWarning } from './retrieval-upgrade-planner.ts';
import { runSchemaTransition } from './retrieval-upgrade-planner.ts';
import { readContentChunksEmbeddingDim } from './embedding-dim-check.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
/**
* Resume/state marker (DB plane). Present while a migration is in flight so
* a re-run can detect + resume; cleared when the re-embed drains to zero.
*/
export const MIGRATION_STATE_KEY = 'embedding_migration.state';
/** ISO timestamp + summary of the last completed migration (DB plane). */
export const MIGRATION_COMPLETED_KEY = 'embedding_migration.completed';
export interface MigrationState {
to_model: string;
to_dims: number;
from_model: string;
from_dims: number;
started_at: string;
}
export interface EmbeddingMigrationPlan {
from_model: string;
from_dims: number;
/** Actual `content_chunks.embedding` vector(N) width (null = column absent). */
column_dims: number | null;
to_model: string;
to_dims: number;
/** True when the schema column must be rebuilt at a new width. */
dim_change: boolean;
/** Chunks not yet in the target embedding space (the migration workload). */
chunks_to_embed: number;
/** Characters across those chunks (feeds the cost estimate). */
total_chars: number;
/**
* #3391 visibility: embedded chunks on pages with NO recorded signature
* (pre-v108). Included in chunks_to_embed via includeNullSignature.
*/
null_signature_chunks: number;
est_cost_usd: number;
/** False when the target model has no entry in EMBEDDING_PRICING. */
price_known: boolean;
/** True when a prior in-flight migration state matches this target. */
resuming: boolean;
/** Set when the brain's reranker is also on the outgoing provider. */
reranker_warning: string | null;
}
export type MigrationApplyResult =
| { status: 'applied'; invalidated: number; cache_cleared: number; schema_transitioned: boolean }
| { status: 'refused'; reason: 'env_override'; warning: EnvOverrideWarning }
| { status: 'failed'; reason: string };
/** `<provider:model>:<dims>` — must match currentEmbeddingSignature()'s shape. */
export function migrationSignature(toModel: string, toDims: number): string {
return `${toModel}:${toDims}`;
}
/**
* Resolve + validate the target `provider:model` and dimensions.
* Throws with a paste-ready message on an unknown provider or when the
* recipe declares no default dims and the caller passed none.
*/
export function resolveMigrationTarget(to: string, dimFlag?: number): { toModel: string; toDims: number } {
if (!to.includes(':')) {
throw new Error(
`--to must be provider:model (e.g. openai:text-embedding-3-small). Got: ${to}`,
);
}
// Throws AIConfigError with provider list on an unknown provider.
const { recipe } = resolveRecipe(to);
if (!recipe.touchpoints.embedding) {
throw new Error(`Provider ${recipe.id} has no embedding support. Pick an embedding-capable provider:model.`);
}
const toDims = dimFlag ?? embeddingDimsForModel(recipe, to);
if (!toDims || toDims <= 0) {
throw new Error(
`No default dimension known for ${to}. Pass --dim <N> explicitly (see the provider's docs for valid values).`,
);
}
return { toModel: to, toDims };
}
/**
* Pure read: compute the migration workload. Uses the stale-chunk predicates
* with the TARGET signature + includeNullSignature so the count is
* resume-aware a re-plan mid-migration counts only what remains.
*/
export async function planEmbeddingMigration(
engine: BrainEngine,
opts: { to: string; dim?: number; fromModel?: string; fromDims?: number },
): Promise<EmbeddingMigrationPlan> {
const { toModel, toDims } = resolveMigrationTarget(opts.to, opts.dim);
// From-state: caller (CLI) passes the gateway-resolved values; fall back
// to the shipped defaults for gateway-less contexts (unit tests, op probe).
const fromModel = opts.fromModel ?? DEFAULT_EMBEDDING_MODEL;
const fromDims = opts.fromDims ?? DEFAULT_EMBEDDING_DIMENSIONS;
const col = await readContentChunksEmbeddingDim(engine);
const sig = migrationSignature(toModel, toDims);
const wide = await engine.countStaleChunks({ signature: sig, includeNullSignature: true });
const narrow = await engine.countStaleChunks({ signature: sig });
const totalChars = await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true });
const price = lookupEmbeddingPrice(toModel);
const estCostUsd = price.kind === 'known'
? estimateCostFromChars(totalChars, price.pricePerMTok)
: 0;
let resuming = false;
try {
const stateStr = await engine.getConfig(MIGRATION_STATE_KEY);
if (stateStr) {
const state = JSON.parse(stateStr) as MigrationState;
resuming = state.to_model === toModel && state.to_dims === toDims;
}
} catch {
// Corrupt state marker — treat as fresh.
}
// Sunset companion warning: migrating embeddings off a provider whose
// reranker is still configured leaves rerank on the outgoing provider.
let rerankerWarning: string | null = null;
try {
const rr = await engine.getConfig('search.reranker.model');
const outgoingProvider = fromModel.split(':')[0];
const targetProvider = toModel.split(':')[0];
if (rr && outgoingProvider !== targetProvider && rr.startsWith(`${outgoingProvider}:`)) {
rerankerWarning =
`search.reranker.model is still ${rr} (the outgoing provider). ` +
`If that provider is sunsetting, also update or disable the reranker: ` +
`gbrain config set search.reranker.enabled false`;
}
} catch {
// Reranker warning is cosmetic.
}
return {
from_model: fromModel,
from_dims: fromDims,
column_dims: col.dims,
to_model: toModel,
to_dims: toDims,
dim_change: col.dims !== null && col.dims !== toDims,
chunks_to_embed: wide,
total_chars: totalChars,
null_signature_chunks: wide - narrow,
est_cost_usd: estCostUsd,
price_known: price.kind === 'known',
resuming,
reranker_warning: rerankerWarning,
};
}
/**
* Apply the non-embed half of the migration: env gate, state marker, schema
* transition (dim changes only), DB-plane config, file-plane persistence
* (via callback the core module never touches ~/.gbrain), stale-signature
* invalidation (#3391: includeNullSignature), and query-cache purge.
*
* Ordering makes every step idempotent under a crash + re-run:
* state marker schema config invalidate cache purge.
* A crash anywhere leaves the state marker set; the re-run re-executes the
* remaining steps (schema transition no-ops when the column is already at
* the target width via the actual-width probe; invalidation matches nothing
* the second time).
*/
export async function applyEmbeddingMigration(
engine: BrainEngine,
plan: EmbeddingMigrationPlan,
opts: {
ignoreEnvOverride?: boolean;
/** Persist target model+dims to the file plane + reconfigure the gateway. */
persistConfig?: (toModel: string, toDims: number) => void | Promise<void>;
} = {},
): Promise<MigrationApplyResult> {
const envWarning = detectEnvOverride(plan.to_model, plan.to_dims);
if (envWarning.triggered && !opts.ignoreEnvOverride) {
return { status: 'refused', reason: 'env_override', warning: envWarning };
}
try {
// 1. State marker FIRST — a crash after any later step is resumable.
const state: MigrationState = {
to_model: plan.to_model,
to_dims: plan.to_dims,
from_model: plan.from_model,
from_dims: plan.from_dims,
started_at: new Date().toISOString(),
};
await engine.setConfig(MIGRATION_STATE_KEY, JSON.stringify(state));
// 2. Schema transition when the ACTUAL column width differs from the
// target (probe again — the plan may be stale after a resume).
let schemaTransitioned = false;
const col = await readContentChunksEmbeddingDim(engine);
if (col.dims !== plan.to_dims) {
await runSchemaTransition(engine, plan.to_dims);
schemaTransitioned = true;
}
// 3. #3391: mark EVERYTHING not in the target space as stale, including
// NULL-signature (pre-v108) pages. After a schema transition this is
// a cheap no-op (the column rebuild already nulled every embedding).
//
// ORDERING (adversarial review): invalidation MUST precede the config
// writes below. On a SAME-dim provider swap there is no schema
// transition to null the vectors, so a crash between "config says new
// provider" and "old vectors invalidated" would leave NEW-space query
// embeddings scored against OLD-space document vectors — silently
// WRONG results. Invalidating first makes the crash window safe:
// config still says the old provider, and the rows are merely stale
// (empty/degraded results, never wrong ones).
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
signature: migrationSignature(plan.to_model, plan.to_dims),
includeNullSignature: true,
});
// 4. DB-plane config (doctor's embedding_width_consistency reads these).
await engine.setConfig('embedding_model', plan.to_model);
await engine.setConfig('embedding_dimensions', String(plan.to_dims));
// 5. File plane + gateway (the embed pipeline reads file/env, not DB).
await opts.persistConfig?.(plan.to_model, plan.to_dims);
// 6. Purge the semantic query cache. The knobs hash folds provider:model
// for callers that thread KnobsHashContext, but legacy callers fall
// back to 'default' — a row they wrote pre-migration must not be
// served post-migration. Best-effort (cache must never block).
let cacheCleared = 0;
try {
const { SemanticQueryCache } = await import('./search/query-cache.ts');
cacheCleared = await new SemanticQueryCache(engine).clear({});
} catch {
// Table may not exist on old brains; a miss here is harmless.
}
return { status: 'applied', invalidated, cache_cleared: cacheCleared, schema_transitioned: schemaTransitioned };
} catch (err) {
return { status: 'failed', reason: err instanceof Error ? err.message : String(err) };
}
}
/**
* Stamp the target signature on every page that is fully embedded but not yet
* stamped. Call after the re-embed drain, BEFORE the completion probe.
*
* Why this exists (adversarial review): the embed loop only stamps a page when
* `stale.length === existing.length` i.e. when every one of the page's
* chunks was in the SAME batch. `listStaleChunks` is a plain keyset LIMIT with
* no page alignment, so on any corpus larger than one batch (default 2000
* chunks) the page straddling each boundary is embedded correctly but never
* stamped. Without this reconcile the command reports "incomplete" + exit 1 on
* a perfectly-migrated brain, and the re-run re-invalidates and PAYS AGAIN for
* those pages breaking the "already-migrated chunks are never re-embedded"
* contract.
*
* Safety: this is only sound because `applyEmbeddingMigration` invalidated
* (NULLed) every chunk that was NOT already in the target space. So "page has
* zero NULL-embedding chunks" ⇒ "every chunk on this page was embedded in the
* target space during this run". Pages with any remaining NULL chunk (a real
* embed failure) are deliberately left unstamped so the completion probe still
* reports them.
*
* Returns the number of pages stamped.
*/
export async function reconcilePageSignatures(
engine: BrainEngine,
plan: EmbeddingMigrationPlan,
): Promise<number> {
const sig = migrationSignature(plan.to_model, plan.to_dims);
const rows = await engine.executeRaw<{ slug: string }>(
`UPDATE pages p
SET embedding_signature = $1
WHERE p.deleted_at IS NULL
AND (p.embedding_signature IS DISTINCT FROM $1)
AND EXISTS (SELECT 1 FROM content_chunks c WHERE c.page_id = p.id)
AND NOT EXISTS (
SELECT 1 FROM content_chunks c
WHERE c.page_id = p.id AND c.embedding IS NULL
)
RETURNING p.slug`,
[sig],
);
return (rows as unknown[]).length;
}
/**
* Finish bookkeeping after the re-embed drains: clear the in-flight marker,
* stamp the completion record. Call ONLY when countStaleChunks() === 0.
*/
export async function completeEmbeddingMigration(
engine: BrainEngine,
plan: EmbeddingMigrationPlan,
): Promise<void> {
await engine.unsetConfig(MIGRATION_STATE_KEY);
await engine.setConfig(
MIGRATION_COMPLETED_KEY,
JSON.stringify({
to_model: plan.to_model,
to_dims: plan.to_dims,
from_model: plan.from_model,
completed_at: new Date().toISOString(),
}),
);
}
+45 -5
View File
@@ -1005,8 +1005,13 @@ export interface BrainEngine {
* counts across every source in the brain. Operators running
* `gbrain embed --stale --source media-corpus` expect only that
* source's NULLs touched; the caller threads `sourceId` here.
*
* `includeNullSignature` (only meaningful with `signature`, #3391): also
* count embedded chunks whose page has NO recorded signature (v108
* grandfathered). Provider-migration paths set this so pre-stamp pages
* aren't silently left in the old embedding space.
*/
countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number>;
countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number>;
/**
* Sum of LENGTH(chunk_text) over stale chunks the character-count
* backlog the embed phase / embed-backfill will process. Sibling of
@@ -1020,8 +1025,10 @@ export interface BrainEngine {
* model signature (a model/dims swap). NULL signature is GRANDFATHERED
* (never counted) so the post-migration corpus isn't flagged en masse.
* Omit `signature` for the legacy `embedding IS NULL`-only count.
* `includeNullSignature` lifts the grandfather clause (#3391) see
* countStaleChunks.
*/
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number>;
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number>;
/**
* Stamp `pages.embedding_signature = signature` for one page. Called after
* a page's chunks are (re)embedded so a later model swap can detect it as
@@ -1036,8 +1043,15 @@ export interface BrainEngine {
* drift pages flow through the existing NULL-embedding cursor (keeps
* listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL
* signature is never invalidated. `sourceId` scopes the sweep.
*
* `includeNullSignature` (#3391): ALSO invalidate embedded chunks whose
* page signature is NULL (pre-v108 pages that predate the stamp). After a
* provider/model swap those vectors are in the old embedding space; the
* default grandfather clause would silently keep them mixed into the new
* index. `gbrain migrate embeddings` and `embed --stale
* --include-null-signature` set this.
*/
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number>;
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number>;
/**
* Return every chunk where embedding IS NULL, with the metadata needed
* to call embedBatch + upsertChunks. The `embedding` column is omitted
@@ -1327,6 +1341,16 @@ export interface BrainEngine {
getContentFlagsByPageIds(
pageIds: number[],
): Promise<Map<number, { reason: string; detail: string }>>;
/**
* Extraction quarantine lane (issue #160): for a list of page_ids, return
* the subset that are unverified auto-extracted entity stubs (frontmatter
* `provenance: 'auto-extracted'` + `status: 'unverified'`). Used by hybrid
* search to stamp `SearchResult.unverified` pre-fusion so the fusion-level
* compiled-truth boost skips them. Single SQL query, not N+1. Empty input
* empty set (no query). SQL predicate is the shared
* `unverifiedExtractionFragment` (src/core/extraction-review.ts).
*/
getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>>;
/**
* v0.27.0: for a list of slugs, return their updated_at timestamps (or created_at fallback).
* Used by hybrid search recency boost. Single SQL query, not N+1.
@@ -1791,11 +1815,22 @@ export interface BrainEngine {
* never recreate them (the page has no `## Facts` fence). Omitted legacy
* behavior (delete every fact on the page coordinate). NULL/empty `source`
* rows are always deletable (fence default).
*
* #2646: `preserveExpiredLegacy` protects soft-expired legacy rows
* (`row_num IS NULL AND expired_at IS NOT NULL`) the record left by
* `forget_fact`'s legacy DB-only path. Fence rows always carry a
* `row_num`, so these rows are never fence-owned and a wipe would
* destroy the forget record (the audit trail of the forget). Note what
* this does NOT promise: it protects the record, not the forget itself
* if the fence still carries the same claim, fence canonicality
* independently reinserts it as a fresh active row (legacy DB-only
* forgets are documented as non-durable; see extract-facts.ts). Omitted
* legacy behavior.
*/
deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }>;
/**
@@ -1916,8 +1951,13 @@ export interface BrainEngine {
* preserved via stable page_id). `opts.sourceId` scopes the UPDATE without
* it, the bare `WHERE slug = old` matches every row across every source and
* would either rename them all OR violate the (source_id, slug) UNIQUE.
*
* Returns the number of rows moved. 0 means the old slug had no row in the
* scoped source an UPDATE that matches nothing does NOT throw, so callers
* that need to know whether the rename actually happened (the sync rename
* path, #3056) must check the return value rather than rely on the catch.
*/
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void>;
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number>;
rewriteLinks(oldSlug: string, newSlug: string): Promise<void>;
/**
+46 -10
View File
@@ -15,6 +15,7 @@
import type { BrainEngine } from './engine.ts';
import { waitForCapacity } from './backoff.ts';
import { quarantineMarkers } from './extraction-review.ts';
// ---------------------------------------------------------------------------
// Types
@@ -28,9 +29,32 @@ export interface EnrichmentRequest {
tier?: 1 | 2 | 3;
}
/**
* Trust options for the enrichment write path (issue #160).
*
* `trusted: true` the input text comes from the machine owner via the
* trusted local CLI (ctx.remote === false) AND the caller passed an explicit
* opt-in flag. Stubs write direct as authoritative entity pages.
*
* Anything else (undefined, false, absent) is UNTRUSTED fail-closed,
* mirroring the OperationContext.remote invariant ("anything not strictly
* false is remote"). Created stubs land in the quarantine lane: frontmatter
* `provenance: 'auto-extracted'` + `status: 'unverified'`. They are excluded
* from authoritative retrieval boosts and wait in the review queue
* (`extraction_pending` / `extraction_review` ops) until the owner promotes
* or rejects them.
*/
export interface EnrichmentTrustOptions {
trusted?: boolean;
/** Source to read/write in (multi-source brains). Omitted → engine default. */
sourceId?: string;
}
export interface EnrichmentResult {
slug: string;
action: 'created' | 'updated' | 'skipped';
/** True when the created stub landed in the quarantine lane (issue #160). */
quarantined?: boolean;
tier: 1 | 2 | 3;
backlinkCreated: boolean;
timelineAdded: boolean;
@@ -72,11 +96,15 @@ export function entityPagePath(name: string, type: 'person' | 'company'): string
export async function enrichEntity(
engine: BrainEngine,
request: EnrichmentRequest,
opts?: EnrichmentTrustOptions,
): Promise<EnrichmentResult> {
const slug = slugifyEntity(request.entityName, request.entityType);
// Fail-closed: only an explicit `trusted: true` writes authoritative pages.
const trusted = opts?.trusted === true;
const scope = opts?.sourceId ? { sourceId: opts.sourceId } : undefined;
// 1. Count existing mentions for tier auto-escalation
const { mentionCount, mentionSources } = await countMentions(engine, request.entityName);
const { mentionCount, mentionSources } = await countMentions(engine, request.entityName, opts?.sourceId);
// 2. Determine tier (auto-escalate based on mentions)
const suggestedTier = suggestTier(mentionCount, mentionSources, request.context);
@@ -84,7 +112,7 @@ export async function enrichEntity(
const tierEscalated = suggestedTier < (request.tier || 3); // lower tier number = higher importance
// 3. Check if entity page exists
const existingPage = await engine.getPage(slug);
const existingPage = await engine.getPage(slug, scope);
let action: 'created' | 'updated' | 'skipped';
if (existingPage) {
@@ -104,8 +132,11 @@ export async function enrichEntity(
created: new Date().toISOString().split('T')[0],
source: request.sourceSlug,
tier,
// issue #160 quarantine lane: stubs extracted from untrusted input
// carry provenance + unverified markers until the owner reviews them.
...(trusted ? {} : quarantineMarkers()),
},
});
}, scope);
action = 'created';
}
@@ -116,7 +147,7 @@ export async function enrichEntity(
date: new Date().toISOString().split('T')[0] ?? '',
summary: `Referenced in [${request.sourceSlug}](${request.sourceSlug}) — ${request.context}`,
source: request.sourceSlug,
});
}, scope);
timelineAdded = true;
} catch {
// Timeline add failed (page might not support it)
@@ -125,7 +156,7 @@ export async function enrichEntity(
// 5. Add backlink from entity to source
let backlinkCreated = false;
try {
await engine.addLink(slug, request.sourceSlug, `Entity mention from ${request.sourceSlug}`); // gbrain-allow-direct-insert: auto-link reconciliation triggered by entity reference in source markdown
await engine.addLink(slug, request.sourceSlug, `Entity mention from ${request.sourceSlug}`, undefined, undefined, undefined, undefined, opts?.sourceId ? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId } : undefined); // gbrain-allow-direct-insert: auto-link reconciliation triggered by entity reference in source markdown
backlinkCreated = true;
} catch {
// Link might already exist
@@ -134,6 +165,7 @@ export async function enrichEntity(
return {
slug,
action,
...(action === 'created' && !trusted ? { quarantined: true } : {}),
tier,
backlinkCreated,
timelineAdded,
@@ -152,14 +184,14 @@ export async function enrichEntity(
export async function enrichEntities(
engine: BrainEngine,
requests: EnrichmentRequest[],
config?: { throttle?: boolean; onProgress?: (done: number, total: number, name: string) => void },
config?: { throttle?: boolean; onProgress?: (done: number, total: number, name: string) => void } & EnrichmentTrustOptions,
): Promise<EnrichmentResult[]> {
const results: EnrichmentResult[] = [];
for (const req of requests) {
if (config?.throttle !== false) {
await waitForCapacity({ maxAttempts: 5 }); // shorter timeout for batch items
}
const result = await enrichEntity(engine, req);
const result = await enrichEntity(engine, req, { trusted: config?.trusted, sourceId: config?.sourceId });
results.push(result);
config?.onProgress?.(results.length, requests.length, req.entityName);
}
@@ -175,8 +207,11 @@ export async function extractAndEnrich(
engine: BrainEngine,
text: string,
sourceSlug: string,
opts?: EnrichmentTrustOptions & { throttle?: boolean; maxEntities?: number },
): Promise<EnrichmentResult[]> {
const entities = extractEntities(text);
// Bounded by default (#160 hardening): the greedy regex on a large paste
// can produce thousands of hits; each enrichment is several DB round-trips.
const entities = extractEntities(text).slice(0, opts?.maxEntities ?? 200);
if (entities.length === 0) return [];
const requests: EnrichmentRequest[] = entities.map(e => ({
@@ -186,7 +221,7 @@ export async function extractAndEnrich(
sourceSlug,
}));
return enrichEntities(engine, requests);
return enrichEntities(engine, requests, { trusted: opts?.trusted, sourceId: opts?.sourceId, throttle: opts?.throttle });
}
// ---------------------------------------------------------------------------
@@ -197,9 +232,10 @@ export async function extractAndEnrich(
async function countMentions(
engine: BrainEngine,
entityName: string,
sourceId?: string,
): Promise<{ mentionCount: number; mentionSources: string[] }> {
try {
const results = await engine.searchKeyword(entityName, { limit: 100 });
const results = await engine.searchKeyword(entityName, { limit: 100, ...(sourceId ? { sourceId } : {}) });
// Derive sources from slug prefixes since SearchResult has no metadata.skill
const sources = new Set<string>();
for (const r of results) {
+13 -3
View File
@@ -85,11 +85,21 @@ const RUN_ID_SHORT_LEN = 8;
/**
* Truncate a run id to the standard 8-char short form used in slug
* paths. Idempotent passing an already-short id returns it unchanged.
* Non-hex / non-alphanumeric chars survive (op-checkpoint ids may
* include dashes or other separators).
* Non-hex / non-alphanumeric chars survive INSIDE the short form
* (op-checkpoint ids may include dashes or other separators), but
* boundary hyphens are trimmed (#3443): `slugifySegment()` strips
* leading/trailing hyphens during repo sync, so a short form like
* 'propose-' (from propose-<timestamp> run ids) made the DB receipt
* slug and its Git-backed slug disagree writing the receipt through
* to the repo created a normalized sibling instead of materializing
* the existing page. Invariant: slugifySegment(shortRunId(x)) ===
* shortRunId(x) for slug-safe run ids.
*/
export function shortRunId(runId: string): string {
return runId.slice(0, RUN_ID_SHORT_LEN);
// ponytail: truncation-based discrimination is only as good as the run id's
// first 8 chars; families that need per-run uniqueness must front-load it.
const short = runId.slice(0, RUN_ID_SHORT_LEN).replace(/^-+|-+$/g, '');
return short || (runId ? 'run' : '');
}
/**
+88
View File
@@ -0,0 +1,88 @@
/**
* Extraction quarantine lane (issue #160).
*
* `extractAndEnrich` regex-extracts entity names from arbitrary ingested text
* and creates `people/{slug}` / `companies/{slug}` stub pages. When the input
* text comes from an untrusted channel (anything that is not the trusted local
* CLI with an explicit opt-in), those stubs must NOT enter the brain as
* authoritative entity pages. Instead they land in the quarantine lane:
* ordinary pages carrying two frontmatter markers
*
* provenance: 'auto-extracted' HOW the page came to exist
* status: 'unverified' the owner has not reviewed it yet
*
* Consequences of the markers (each enforced at its own site):
* - Search: unverified stubs are excluded from the compiled-truth authority
* boost (they rank as ordinary content) and results carry
* `unverified: true` so agents can label the provenance.
* - Review: `extraction_pending` lists them; `extraction_review` promotes
* (status 'verified', provenance kept for audit) or rejects
* (soft-delete) in batch. Promotion is local-owner-only.
* - Doctor: counts unverified stubs older than N days as a review nudge.
*
* Fail-closed trust rule (mirrors OperationContext.remote): only an explicit
* `trusted: true` writes direct; undefined/false/anything-else quarantines.
*
* Known scope (deliberate, documented not gaps discovered later):
* - CREATE-path only. The enrichment UPDATE path (timeline append + edge
* onto an EXISTING page when a slug collides) is the separately-tracked
* slug-collision finding referenced in issue #160; this lane does not
* gate it.
* - The markers are ordinary frontmatter keys, not put_page-strip-listed
* (#1699). A caller holding generic remote put_page write scope can
* rewrite a stub without them but that caller can author an unmarked
* people/ page directly anyway, so stripping here adds no privilege.
* The promotion OP surface (extraction_review) is what stays owner-only.
*
* Sibling of `src/core/quarantine.ts` / `src/core/embed-skip.ts` same
* marker-as-frontmatter-JSONB pattern, same "SQL fragment lives next to the
* marker key so they can never drift" rule. No schema migration needed.
*/
// ---------------------------------------------------------------------------
// Marker keys + values (stable contract)
// ---------------------------------------------------------------------------
export const EXTRACTION_PROVENANCE_KEY = 'provenance';
export const EXTRACTION_STATUS_KEY = 'status';
export const PROVENANCE_AUTO_EXTRACTED = 'auto-extracted';
export const STATUS_UNVERIFIED = 'unverified';
export const STATUS_VERIFIED = 'verified';
/** Frontmatter markers to spread onto a quarantined stub at create time. */
export function quarantineMarkers(): Record<string, string> {
return {
[EXTRACTION_PROVENANCE_KEY]: PROVENANCE_AUTO_EXTRACTED,
[EXTRACTION_STATUS_KEY]: STATUS_UNVERIFIED,
};
}
/**
* JS-side predicate: true only when BOTH markers match. Requiring the pair
* means user pages that happen to carry their own `status` or `provenance`
* frontmatter are never captured by the review lane.
*/
export function isUnverifiedExtraction(
frontmatter: Record<string, unknown> | null | undefined,
): boolean {
if (!frontmatter) return false;
return (
frontmatter[EXTRACTION_PROVENANCE_KEY] === PROVENANCE_AUTO_EXTRACTED &&
frontmatter[EXTRACTION_STATUS_KEY] === STATUS_UNVERIFIED
);
}
/**
* SQL fragment matching unverified auto-extracted stubs, parameterized on the
* page-table alias. Single source of truth for every SQL-side consumer
* (extraction_pending list, doctor count) so the filter and the marker keys
* can never drift. `pageAlias` is engine-supplied (never user input).
* JSONB `->>` works identically on Postgres and PGLite (PostgreSQL-in-WASM).
*/
export function unverifiedExtractionFragment(pageAlias: string): string {
return (
`(COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ->> '${EXTRACTION_PROVENANCE_KEY}') = '${PROVENANCE_AUTO_EXTRACTED}'` +
` AND (COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ->> '${EXTRACTION_STATUS_KEY}') = '${STATUS_UNVERIFIED}'`
);
}
+55 -13
View File
@@ -96,29 +96,71 @@ export interface GitFreshnessOpts {
}
/**
* Returns true iff `localPath` is a git repo whose current HEAD matches
* `lastCommit`, AND (when `requireCleanWorkingTree`) the working tree
* is clean.
* Three-state git probe verdict for a federated source clone.
*
* - `'unchanged'`: HEAD matches `last_commit` (and, when requested, the
* working tree is clean). Sync has nothing to do.
* - `'changed'`: the clone is readable but HEAD moved, the tree is
* dirty, or the DB never recorded a `last_commit`
* sync genuinely has (or may have) work.
* - `'unavailable'`: the HEAD probe itself could not run the clone
* directory is missing, not a git repo, or git errored.
* On stateless deploys (containers on EB / K8s / Fly,
* where `local_path` dies with the filesystem and is
* lazily re-materialized by the next per-source sync)
* this is a NORMAL steady state for quiet sources, not
* evidence of pending work. Callers can fall back to a
* DB-only freshness signal instead of wall-clock age.
*/
export type SourceGitState = 'unchanged' | 'changed' | 'unavailable';
/**
* Probe a source clone and classify it (see `SourceGitState`).
*
* This is NOT a full mirror of `gbrain sync`'s "do work?" predicate.
* Chunker-version match is computed by the caller because it depends on
* engine state (`sources.chunker_version` vs `CURRENT_CHUNKER_VERSION`).
* See `src/commands/doctor.ts:checkSyncFreshness` for the AND
* combination at the call site.
*
* NULL-input guard stays first: a NULL `last_commit` (legacy row) returns
* `'changed'` WITHOUT running the head probe same short-circuit contract
* `isSourceUnchangedSinceSync` always had (pinned by doctor.test.ts case 4).
*/
export function probeSourceGitState(
localPath: string | null | undefined,
lastCommit: string | null | undefined,
opts?: GitFreshnessOpts,
): SourceGitState {
if (!localPath || !lastCommit) return 'changed';
const head = _headProbe(localPath);
if (head === null) return 'unavailable';
if (head !== lastCommit) return 'changed';
if (opts?.requireCleanWorkingTree) {
const ignoreUntracked = opts.requireCleanWorkingTree === 'ignore-untracked';
const isClean = _cleanProbe(localPath, ignoreUntracked);
// null (probe error) AND false (known dirty) both fail the gate. A clean
// probe error with a READABLE head is not classified 'unavailable' —
// fail toward "may have work" so the gate can only relax, never mask.
if (isClean !== true) return 'changed';
}
return 'unchanged';
}
/**
* Returns true iff `localPath` is a git repo whose current HEAD matches
* `lastCommit`, AND (when `requireCleanWorkingTree`) the working tree
* is clean.
*
* Boolean façade over `probeSourceGitState` `'unavailable'` and
* `'changed'` both collapse to `false`, preserving the v0.41.27.0
* fail-open contract for callers that only care about the short-circuit
* (`src/core/source-health.ts`).
*/
export function isSourceUnchangedSinceSync(
localPath: string | null | undefined,
lastCommit: string | null | undefined,
opts?: GitFreshnessOpts,
): boolean {
if (!localPath || !lastCommit) return false;
const head = _headProbe(localPath);
if (head === null || head !== lastCommit) return false;
if (opts?.requireCleanWorkingTree) {
const ignoreUntracked = opts.requireCleanWorkingTree === 'ignore-untracked';
const isClean = _cleanProbe(localPath, ignoreUntracked);
// null (probe error) AND false (known dirty) both fail the gate.
if (isClean !== true) return false;
}
return true;
return probeSourceGitState(localPath, lastCommit, opts) === 'unchanged';
}
+2 -2
View File
@@ -1092,8 +1092,8 @@ export async function importFromFile(
chunks: 0,
error:
`Filename "${relativePath}" produces no usable slug. ` +
`Add a "slug:" to the frontmatter, or rename the file to use ` +
`ASCII / Chinese / Japanese / Korean characters.`,
`Add a "slug:" to the frontmatter, or rename the file to include ` +
`at least one letter or number (any script).`,
};
}
} else if (parsed.slug !== expectedSlug) {
+75 -11
View File
@@ -12,7 +12,7 @@
*/
import type { BrainEngine } from './engine.ts';
import type { PageType } from './types.ts';
import type { PageType, EffectiveDateSource } from './types.ts';
import { ensureWellFormed } from './text-safe.ts';
/**
@@ -671,6 +671,17 @@ const FOUNDED_RE = /\b(?:founded|co-?founded|started the company|incorporated|fo
// "security advisor to|at", "product advisor to|at", "industry advisor".
const ADVISES_RE = /\b(?:advises|advised|advisor (?:to|at|for|of)|advisory (?:board|role|position|capacity|engagement|partnership|contract|relationship|work)|board advisor|on .{0,20} advisory board|joined .{0,20} advisory board|in an? advisory (?:capacity|role|position)|as an? (?:advisor|security advisor|technical advisor|strategic advisor|industry advisor|product advisor|board advisor|senior advisor)|(?:strategic|technical|security|product|industry|senior|board) advisor (?:to|at|for|of)|consults for|consulting role (?:at|with))\b/i;
// Chinese link type patterns for CJK entity mentions.
// NOTE: These patterns are Chinese-only (zh). Japanese and Korean link
// type extraction is not yet implemented. Entity NAME extraction in
// by-mention.ts covers all three scripts (CJK = Chinese/Japanese/Korean)
// via Unicode-aware tokenization.
const ZH_FOUNDED_RE = /(?:创立|创办|成立|创建|建立|开创|发起)(?:了|的)/;
const ZH_INVESTED_RE = /(?:投资|入股|融资|注资|参股)(?:了|的|了?于)/;
const ZH_ADVISES_RE = /(?:顾问|咨询|指导)(?:了|的)?/;
const ZH_WORKS_AT_RE = /(?:任职|就职|担任|供职|在.{0,10}(?:工作|上班|负责))(?:于|在|的)?/;
const ZH_CITED_RE = /(?:引用|援引|提到|提及|转述|摘录)(?:了|的|自)?/;
// Page-role detection: if the source page describes a partner/investor at
// page level, that's a strong prior for outbound company refs being
// invested_in even when per-edge context lacks explicit investment verbs.
@@ -724,6 +735,12 @@ export function inferLinkType(pageType: PageType, context: string, globalContext
if (INVESTED_RE.test(context)) return 'invested_in';
if (ADVISES_RE.test(context)) return 'advises';
if (WORKS_AT_RE.test(context)) return 'works_at';
// Chinese link type patterns
if (ZH_FOUNDED_RE.test(context)) return 'founded';
if (ZH_INVESTED_RE.test(context)) return 'invested_in';
if (ZH_ADVISES_RE.test(context)) return 'advises';
if (ZH_WORKS_AT_RE.test(context)) return 'works_at';
if (ZH_CITED_RE.test(context)) return 'cited';
// Page-role prior: only fires for person -> company links. Concept pages
// about VC topics naturally contain "venture capital" in their text, but
// their company refs are mentions, not investments. Partner pages mentioning
@@ -1174,6 +1191,10 @@ export interface TimelineCandidate {
// Match: `- **YYYY-MM-DD** | summary` or `- **YYYY-MM-DD** -- summary`
// or `- **YYYY-MM-DD** - summary` or just `**YYYY-MM-DD** | summary`.
const TIMELINE_LINE_RE = /^\s*-?\s*\*\*(\d{4}-\d{2}-\d{2})\*\*\s*[|\-–—]+\s*(.+?)\s*$/;
// Chinese date lines: `- 2020年1月2日 | summary` (bold optional). Requires the
// 年/月 markers so plain ASCII `- 2020-01-02 - text` does NOT match — non-bold
// ASCII dates were never timeline entries and must stay that way.
const TIMELINE_LINE_RE_CN = /^\s*-?\s*(?:\*\*)?(\d{4})年(\d{1,2})月(\d{1,2})日?(?:\*\*)?\s*[|\-–—]+\s*(.+?)\s*$/;
/**
* Parse timeline entries from content. Looks at:
@@ -1190,18 +1211,21 @@ export function parseTimelineEntries(content: string): TimelineCandidate[] {
let i = 0;
while (i < lines.length) {
// Try English format first, then Chinese
const m = TIMELINE_LINE_RE.exec(lines[i]);
if (!m) {
i++;
continue;
let date: string;
let summary: string;
if (m) {
date = m[1];
summary = m[2].trim();
} else {
const cm = TIMELINE_LINE_RE_CN.exec(lines[i]);
if (!cm) { i++; continue; }
// Normalize Chinese date to YYYY-MM-DD
date = `${cm[1]}-${cm[2].padStart(2, '0')}-${cm[3].padStart(2, '0')}`;
summary = cm[4].trim();
}
const date = m[1];
const summary = m[2].trim();
if (!isValidDate(date) || summary.length === 0) {
i++;
continue;
}
if (!isValidDate(date) || summary.length === 0) { i++; continue; }
// Collect optional detail lines (indented, until next date or heading).
const detailLines: string[] = [];
let j = i + 1;
@@ -1266,6 +1290,46 @@ function isValidDate(s: string): boolean {
return dt.getUTCFullYear() === y && dt.getUTCMonth() === mo - 1 && dt.getUTCDate() === d;
}
/** Input for {@link deriveTimelineAnchor}: a page's identity + its computed content date. */
export interface TimelineAnchorInput {
slug: string;
title?: string | null;
effectiveDate?: Date | string | null;
effectiveDateSource?: EffectiveDateSource | null;
}
/**
* Anchor a single timeline entry from a page's computed content date, for pages
* whose body carries no parseable timeline line.
*
* Comms- and calendar-dominated brains keep the date in frontmatter or the
* filename (slug `2026-04-24-...`), not in the prose, so `parseTimelineEntries`
* returns nothing and the page-level `timeline` table stays empty even though
* the page is firmly dated leaving `get_timeline` and the brain-score
* `timeline_coverage` component blind to it. This recovers that signal from the
* already-computed `effective_date` (no re-parsing). (It does NOT feed the
* facts-based `find_trajectory`, which reads the `facts` table by entity_slug.)
*
* Fires ONLY for a trustworthy content date frontmatter (`event_date` / `date`
* / `published`) or the `filename` date never the `fallback` source, which is
* `updated_at` (link-churn noise, not when the thing happened). Returns null
* when no trustworthy date is available. Callers MUST apply this only when body
* parsing yields zero entries, so it can never shadow a real in-body timeline.
*/
export function deriveTimelineAnchor(input: TimelineAnchorInput): TimelineCandidate | null {
const { slug, title, effectiveDate, effectiveDateSource } = input;
if (!effectiveDate) return null;
// 'fallback' === updated_at; the rest ('event_date'|'date'|'published'|'filename')
// are real content dates. null/undefined source is not trustworthy either.
if (effectiveDateSource == null || effectiveDateSource === 'fallback') return null;
const dt = typeof effectiveDate === 'string' ? new Date(effectiveDate) : effectiveDate;
if (!(dt instanceof Date) || Number.isNaN(dt.getTime())) return null;
const iso = dt.toISOString().slice(0, 10);
if (!isValidDate(iso)) return null;
const summary = (title ?? '').trim() || slug.split('/').pop() || slug;
return { date: iso, summary, detail: '' };
}
// ─── Auto-link config ───────────────────────────────────────────
/**
+7 -2
View File
@@ -2,6 +2,13 @@ import type { BrainEngine } from './engine.ts';
import { slugifyPath } from './sync.ts';
import { getFtsLanguage } from './fts-language.ts';
import { hnswMaxDimsForType } from './vector-index.ts';
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
isStatementTimeoutError,
isRetryableConnError,
} from './retry-matcher.ts';
import { repairTimelineDedupIndex } from './timeline-dedup-repair.ts';
/**
* Schema migrations run automatically on initSchema().
@@ -5801,7 +5808,6 @@ async function runMigrationSQLWithRetry(
m: Migration,
sql: string,
): Promise<void> {
const { isStatementTimeoutError, isRetryableConnError } = await import('./retry-matcher.ts');
// GBRAIN_MIGRATE_BACKOFF_MS lets tests skip the 5s/15s/45s backoff. In
// production the env var is unset and the default cadence applies.
const fastBackoff = process.env.GBRAIN_MIGRATE_BACKOFF_MS;
@@ -6071,7 +6077,6 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num
// reach the loop below). Best-effort + idempotent: a no-op on a healthy
// index; `doctor` surfaces it independently if this ever fails.
try {
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
const r = await repairTimelineDedupIndex(engine);
if (r.repaired) {
console.error(
+5
View File
@@ -43,6 +43,11 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
// few writes. Generous 10-min budget (vs the tight null-default) covers a
// slow gateway without the 30-min loop budget.
chronicle_extract: TEN_MIN_MS,
// #3207 — same shape as chronicle_extract: one page = one LLM extraction
// call + a few writes. Was missing from this map, so it inherited the tight
// null-default and got dead-lettered mid-generation on slow chat providers
// (facts silently lost) — exactly the failure this file exists to prevent.
'facts-absorb': TEN_MIN_MS,
// Per-page contextual reindex jobs process chunks sequentially with one
// rate-leased LLM synopsis call per chunk; large transcript pages need more
// than the standard 30-min long-job budget.
+12 -1
View File
@@ -534,10 +534,21 @@ export class MinionQueue {
}
/** Prune old jobs in terminal statuses. Returns count of deleted rows. */
async prune(opts?: { olderThan?: Date; status?: MinionJobStatus[] }): Promise<number> {
async prune(opts?: { olderThan?: Date; status?: MinionJobStatus[]; dryRun?: boolean }): Promise<number> {
const statuses = opts?.status ?? ['completed', 'dead', 'cancelled'];
const olderThan = opts?.olderThan ?? new Date(Date.now() - 30 * 86400000);
// #2712: dryRun counts the would-be-pruned rows without deleting.
// Silent-ignoring a safety flag on a delete path is data loss.
if (opts?.dryRun) {
const rows = await this.engine.executeRaw<{ count: string }>(
`SELECT count(*)::text as count FROM minion_jobs
WHERE status = ANY($1) AND updated_at < $2`,
[statuses, olderThan.toISOString()]
);
return parseInt(rows[0]?.count ?? '0', 10);
}
const rows = await this.engine.executeRaw<{ count: string }>(
`WITH pruned AS (
DELETE FROM minion_jobs
+312 -6
View File
@@ -11,7 +11,7 @@ import type { GBrainConfig } from './config.ts';
import type { PageType } from './types.ts';
import { importFromContent } from './import-file.ts';
import { writePageThrough } from './write-through.ts';
import { hybridSearch, hybridSearchCached, stampContentFlags } from './search/hybrid.ts';
import { hybridSearch, hybridSearchCached, stampContentFlags, stampUnverifiedExtractions } from './search/hybrid.ts';
import { expandQuery } from './search/expansion.ts';
import { dedupResults } from './search/dedup.ts';
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
@@ -21,11 +21,14 @@ import { isFactsBackstopEligible } from './facts/eligibility.ts';
import { stripTakesFence } from './takes-fence.ts';
import { stripFactsFence } from './facts-fence.ts';
import { getContentFlag } from './quarantine.ts';
import { unverifiedExtractionFragment, isUnverifiedExtraction, EXTRACTION_STATUS_KEY, STATUS_VERIFIED } from './extraction-review.ts';
import { buildVisibilityClause } from './search/sql-ranking.ts';
import { bumpLastRetrievedAt } from './last-retrieved.ts';
import { isSearchMode } from './search/mode.ts';
import { stampEvidence } from './search/evidence.ts';
import type { SearchResult } from './types.ts';
import { CJK_SLUG_CHARS, PAGE_SLUG_SEG } from './cjk.ts';
import { ALL_SOURCES } from './source-id.ts';
import * as db from './db.ts';
import { VERSION } from '../version.ts';
import {
@@ -160,10 +163,11 @@ export function validatePageSlug(slug: string): void {
if (slug.length > 255) {
throw new OperationError('invalid_params', 'page_slug exceeds 255 characters');
}
// v0.32.7: CJK ranges (Han / Hiragana / Katakana / Hangul Syllables) allowed
// in segments. ASCII shape rules (lead char, hyphen continuation) preserved.
if (!new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'i').test(slug)) {
throw new OperationError('invalid_params', `Invalid page_slug: ${slug} (allowed: alphanumeric, CJK, hyphens, forward-slash separated segments)`);
// #3417: letters/numbers from any script allowed in segments (u flag required
// for the \p{...} classes in PAGE_SLUG_SEG). Shape rules (lead char, hyphen
// continuation) preserved.
if (!new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`, 'iu').test(slug)) {
throw new OperationError('invalid_params', `Invalid page_slug: ${slug} (allowed: letters/numbers in any script, hyphens, forward-slash separated segments)`);
}
}
@@ -484,6 +488,14 @@ export function sourceScopeOpts(ctx: OperationContext): { sourceId?: string; sou
// value of `[]` MUST NOT widen scope to "all sources" by being interpreted
// as "no filter."
if (allowed && allowed.length > 0) return { sourceIds: allowed };
// #1712: the __all__ sentinel spans the brain — but ONLY for trusted local
// callers (strictly `remote === false`). For remote/untrusted callers the
// literal stays as-is: it can never match a real source id (underscores are
// rejected at creation), so the read fail-closes to empty rather than
// widening past the caller's grant. Do NOT "simplify" this to `{}`.
if (ctx.sourceId === ALL_SOURCES) {
return ctx.remote === false ? {} : { sourceId: ctx.sourceId };
}
if (ctx.sourceId) return { sourceId: ctx.sourceId };
return {};
}
@@ -551,7 +563,7 @@ export function resolveRequestedScope(
sourceIdParam: string | undefined,
allSourcesParam = false,
): { sourceId?: string; sourceIds?: string[] } {
const wantsAll = allSourcesParam || sourceIdParam === '__all__';
const wantsAll = allSourcesParam || sourceIdParam === ALL_SOURCES;
if (wantsAll) {
return ctx.remote === false ? {} : sourceScopeOpts(ctx);
}
@@ -1625,6 +1637,10 @@ const search: Operation = {
// agent-warning channel (hybridSearch stamps it; this branch bypasses
// hybridSearch, so stamp explicitly). Fail-open inside the helper.
await stampContentFlags(ctx.engine, results);
// #160: same for the unverified auto-extracted stub marker (no boost
// to cancel on this path — keyword-only never applies the compiled-
// truth boost — but the provenance marker must still surface).
await stampUnverifiedExtractions(ctx.engine, results);
bumpLastRetrievedAt(ctx.engine, results.map((r) => r.page_id));
maybeCaptureSearch(ctx, queryText, results, Date.now() - startedAt, false);
return results;
@@ -4555,6 +4571,90 @@ const code_traversal_cache_clear: Operation = {
cliHints: { name: 'code_traversal_cache_clear', hidden: true },
};
// --- #3390: provider-agnostic embedding migration ---
const migrate_embeddings: Operation = {
name: 'migrate_embeddings',
description: 'Re-embed the brain onto a different embedding provider/model (#3390): schema dimension transition, NULL-signature (#3391) invalidation, query-cache purge, resumable re-embed. Without yes=true returns the plan + cost estimate only. Local-only admin op; the primary surface is `gbrain migrate embeddings`.',
params: {
to: { type: 'string', required: true, description: 'Target provider:model (e.g. openai:text-embedding-3-small).' },
dim: { type: 'number', description: "Target dimensions. Defaults to the provider recipe's declared width; required when the recipe declares none." },
dry_run: { type: 'boolean', description: 'Plan + cost estimate only; change nothing.' },
yes: { type: 'boolean', description: 'Confirm the re-embed spend + destructive schema change. Required for a live run.' },
},
mutating: true,
scope: 'admin',
localOnly: true,
handler: async (ctx, p) => {
// Belt-and-braces on top of localOnly (the get_recent_transcripts
// pattern): a schema-rebuilding, money-spending op must never be
// reachable from a remote transport even if a future dispatch path
// forgets the localOnly filter.
if (ctx.remote !== false) {
throw new Error('migrate_embeddings is local-only. Run `gbrain migrate embeddings` on the host.');
}
const {
planEmbeddingMigration, applyEmbeddingMigration, completeEmbeddingMigration,
reconcilePageSignatures, migrationSignature,
} = await import('./embedding-migration.ts');
const to = p.to as string;
const dim = p.dim as number | undefined;
let fromModel: string | undefined;
let fromDims: number | undefined;
try {
const { getEmbeddingModel, getEmbeddingDimensions } = await import('./ai/gateway.ts');
fromModel = getEmbeddingModel();
fromDims = getEmbeddingDimensions();
} catch { /* gateway unconfigured — plan falls back to defaults */ }
const plan = await planEmbeddingMigration(ctx.engine, {
to,
...(dim !== undefined && { dim }),
...(fromModel !== undefined && { fromModel }),
...(fromDims !== undefined && { fromDims }),
});
if (ctx.dryRun || p.dry_run === true || p.yes !== true) {
return { status: p.yes === true || p.dry_run === true ? 'planned' : 'needs_confirmation', plan };
}
const { persistEmbeddingFileConfig, probeTargetProvider } = await import('../commands/migrate-embeddings.ts');
// Safety parity with the CLI path: probe the target provider BEFORE any
// mutation. Without this, `yes:true` would drop the embedding column and
// only then discover the key/model/dim is wrong.
const probe = await probeTargetProvider(plan.to_model, plan.to_dims);
if (!probe.ok) return { status: 'failed', reason: probe.message, plan };
const applied = await applyEmbeddingMigration(ctx.engine, plan, {
persistConfig: (m, d) => persistEmbeddingFileConfig(m, d),
});
if (applied.status !== 'applied') return { ...applied, plan };
const { runEmbedCore } = await import('../commands/embed.ts');
// singleFlight parity with the CLI path: takes the same per-source
// embed-backfill lock so this can't race a queued embed-backfill job on
// the NULL→non-NULL upsert (the TODOS:2299 class).
const embedResult = await runEmbedCore(ctx.engine, {
stale: true, catchUp: true, singleFlight: true, includeNullSignature: true, quiet: true,
});
// Stamp batch-boundary pages before probing for completion (see
// reconcilePageSignatures — the embed loop's all-or-nothing stamp rule
// skips any page split across two stale batches).
const reconciled = await reconcilePageSignatures(ctx.engine, plan);
const remaining = await ctx.engine.countStaleChunks({
signature: migrationSignature(plan.to_model, plan.to_dims),
includeNullSignature: true,
});
if (remaining === 0) await completeEmbeddingMigration(ctx.engine, plan);
return {
status: remaining === 0 ? 'completed' : 'incomplete',
plan,
embedded: embedResult.embedded,
remaining,
signatures_reconciled: reconciled,
invalidated: applied.invalidated,
schema_transitioned: applied.schema_transitioned,
cache_cleared: applied.cache_cleared,
};
},
cliHints: { name: 'migrate-embeddings', hidden: true },
};
// --- v0.36 Phase 2: search_by_image (image-as-query) ---
const search_by_image: Operation = {
@@ -5587,6 +5687,208 @@ const chronicle_backfill: Operation = {
cliHints: { name: 'chronicle-backfill' },
};
// ---------------------------------------------------------------------------
// Extraction quarantine lane (issue #160)
//
// `extractAndEnrich` regex-extracts entity names from arbitrary text and
// creates people/ + companies/ stub pages. These three ops are its ONLY
// sanctioned surface:
// - extract_entities — run extraction. Direct authoritative writes need
// BOTH the trusted local CLI (ctx.remote === false)
// AND the explicit --trusted-extraction flag;
// everything else lands in the quarantine lane
// (frontmatter provenance/status markers).
// - extraction_pending — list unverified stubs awaiting review.
// - extraction_review — promote (status → verified) or reject
// (soft-delete) in batch. Owner-only (fail-closed
// on ctx.remote): THIS surface never lets a remote
// caller flip the status markers. Scope note: the
// markers are ordinary frontmatter, so a caller who
// already holds generic remote put_page write scope
// can rewrite the page (markers included) — that
// caller could equally author an unmarked people/
// page directly, so the lane adds no privilege
// there; put_page authz is its own boundary.
// ---------------------------------------------------------------------------
// Resource guards for extract_entities (#160 hardening): bound the work a
// single remote write-scope call can trigger. ponytail: flat caps; make them
// config knobs only if a real workload hits them.
const MAX_EXTRACT_TEXT_CHARS = 200_000;
const MAX_EXTRACT_ENTITIES = 200;
const extract_entities: Operation = {
name: 'extract_entities',
description: 'Extract entity names (people, companies) from text and create/update their brain stub pages. Stubs from untrusted input land in the quarantine lane (frontmatter `provenance: auto-extracted` + `status: unverified`) — excluded from authoritative retrieval boosts until reviewed. Direct authoritative writes require the trusted local CLI AND --trusted-extraction.',
params: {
text: { type: 'string', required: true, description: 'The text to extract entities from (email, transcript, pasted content, …). Max 200k characters — split larger inputs.' },
source_slug: { type: 'string', required: true, description: 'Slug of the source page the text came from (used for backlinks + timeline attribution).' },
trusted_extraction: { type: 'boolean', required: false, description: 'Local CLI only: write stubs directly as authoritative pages, skipping the quarantine lane. Ignored (always quarantined) for remote callers.' },
},
mutating: true,
scope: 'write',
handler: async (ctx, p) => {
// Trust rule (#160, fail-closed like the CV6 provenance gate above):
// `ctx.remote === false` is the ONLY truthy condition that can admit a
// direct authoritative write, and even then the caller must opt in
// explicitly. Remote/unset trust → quarantine lane, flag ignored.
const trusted = ctx.remote === false && p.trusted_extraction === true;
const text = p.text as string;
// Resource guards: the greedy name regex on a huge paste can yield tens
// of thousands of "entities", each costing several DB round-trips. Cap
// input size loudly and entity count softly (surfaced as `truncated`).
if (text.length > MAX_EXTRACT_TEXT_CHARS) {
throw new OperationError(
'invalid_params',
`extract_entities: text is ${text.length} chars (max ${MAX_EXTRACT_TEXT_CHARS}).`,
'Split the input and call extract_entities per section.',
);
}
if (ctx.dryRun) return { dry_run: true, action: 'extract_entities', trusted };
const { extractEntities, enrichEntities } = await import('./enrichment-service.ts');
const found = extractEntities(text);
const capped = found.slice(0, MAX_EXTRACT_ENTITIES);
const results = await enrichEntities(
ctx.engine,
capped.map((e) => ({ entityName: e.name, entityType: e.type, context: e.context, sourceSlug: p.source_slug as string })),
{
trusted,
...(ctx.sourceId ? { sourceId: ctx.sourceId } : {}),
// Pure local DB writes — no external API call to pace, so the
// system-load capacity gate would only stall the caller.
throttle: false,
},
);
return {
status: 'ok',
trusted,
quarantined: results.filter((r) => r.quarantined === true).length,
count: results.length,
entities_found: found.length,
truncated: found.length > capped.length,
entities: results,
};
},
cliHints: { name: 'extract-entities' },
};
const extraction_pending: Operation = {
name: 'extraction_pending',
description: 'List unverified auto-extracted entity stubs awaiting owner review (the quarantine lane from extract_entities). Promote or reject them with extraction_review.',
params: {
limit: { type: 'number', required: false, description: 'Max rows (default 100, cap 500).' },
offset: { type: 'number', required: false, description: 'Pagination offset.' },
},
scope: 'read',
handler: async (ctx, p) => {
const limit = Math.min(Math.max(Number(p.limit ?? 100) || 100, 1), 500);
const offset = Math.max(Number(p.offset ?? 0) || 0, 0);
// Read-side source isolation: route through sourceScopeOpts (federated
// array > scalar > nothing), applied in SQL below.
const scope = sourceScopeOpts(ctx);
const params: unknown[] = [];
let srcClause = '';
if (scope.sourceIds && scope.sourceIds.length > 0) {
params.push(scope.sourceIds);
srcClause = `AND p.source_id = ANY($${params.length}::text[])`;
} else if (scope.sourceId) {
params.push(scope.sourceId);
srcClause = `AND p.source_id = $${params.length}`;
}
params.push(limit, offset);
const rows = await ctx.engine.executeRaw<{
slug: string; title: string; type: string; source_id: string;
extracted_from: string | null; created_at: string;
}>(
`SELECT p.slug, p.title, p.type, p.source_id,
p.frontmatter ->> 'source' AS extracted_from,
p.created_at::text AS created_at
FROM pages p
JOIN sources s ON s.id = p.source_id
WHERE ${unverifiedExtractionFragment('p')}
${buildVisibilityClause('p', 's')}
${srcClause}
ORDER BY p.created_at DESC
LIMIT $${params.length - 1} OFFSET $${params.length}`,
params,
);
return { count: rows.length, pending: rows };
},
cliHints: { name: 'extraction-pending' },
};
const extraction_review: Operation = {
name: 'extraction_review',
description: 'Promote or reject unverified auto-extracted entity stubs (batch). Promote flips `status` to verified (provenance kept for audit); reject soft-deletes the stub. Owner-only: this op is refused for any non-local caller. (The markers are ordinary frontmatter — the boundary against rewriting them wholesale is put_page write authz, same as for any page.)',
params: {
action: { type: 'string', required: true, description: "'promote' or 'reject'." },
slugs: { type: 'array', required: true, items: { type: 'string' }, description: 'Stub slugs to act on (batch).' },
},
mutating: true,
scope: 'write',
localOnly: true,
handler: async (ctx, p) => {
// The review decision IS the trust gate — if a remote caller could
// promote, injected content could self-promote and the quarantine lane
// would be decorative. Fail-closed: only strictly-local callers pass.
if (ctx.remote !== false) {
throw new OperationError(
'permission_denied',
'extraction_review is owner-only: promote/reject decisions must come from the trusted local CLI.',
'Run `gbrain extraction-review <promote|reject> --slugs ...` on the host machine.',
);
}
const action = p.action as string;
if (action !== 'promote' && action !== 'reject') {
throw new OperationError('invalid_params', `extraction_review: action must be 'promote' or 'reject'; got '${action}'.`);
}
// CLI passes `--slugs a,b,c` as one string; MCP passes a real array.
const slugs = Array.isArray(p.slugs)
? (p.slugs as string[])
: typeof p.slugs === 'string'
? p.slugs.split(',').map((s) => s.trim()).filter(Boolean)
: [];
if (slugs.length === 0) {
throw new OperationError('invalid_params', 'extraction_review: slugs must be a non-empty array (CLI: --slugs slug1,slug2).');
}
if (ctx.dryRun) return { dry_run: true, action: `extraction_review:${action}`, slugs };
const results: Array<{ slug: string; status: string }> = [];
for (const slug of slugs) {
const page = await ctx.engine.getPage(slug, ctx.sourceId ? { sourceId: ctx.sourceId } : undefined);
if (!page) {
results.push({ slug, status: 'not_found' });
continue;
}
if (!isUnverifiedExtraction(page.frontmatter)) {
results.push({ slug, status: 'not_unverified' });
continue;
}
if (action === 'promote') {
// Frontmatter-only flip via a targeted JSONB merge — NOT putPage,
// whose upsert would reset non-carried columns (page_kind →
// 'markdown', content_hash, …) for a change that only touches one
// frontmatter key. provenance stays 'auto-extracted' as the audit
// trail of HOW the page came to exist; status → 'verified' records
// the owner's call. jsonb_build_object binds as text (no
// JSON.stringify-into-::jsonb hazard); identical on both engines.
await ctx.engine.executeRaw(
`UPDATE pages
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || jsonb_build_object($1::text, $2::text),
updated_at = now()
WHERE slug = $3 AND source_id = $4`,
[EXTRACTION_STATUS_KEY, STATUS_VERIFIED, slug, page.source_id],
);
results.push({ slug, status: 'promoted' });
} else {
await ctx.engine.softDeletePage(slug, { sourceId: page.source_id });
results.push({ slug, status: 'rejected' });
}
}
return { status: 'ok', action, results };
},
cliHints: { name: 'extraction-review', positional: ['action'] },
};
export const operations: Operation[] = [
// Page CRUD
get_page, put_page, delete_page, list_pages,
@@ -5643,6 +5945,8 @@ export const operations: Operation[] = [
volunteer_chronicle, chronicle_backfill,
// v0.43 (#2095): push-based context
volunteer_context,
// Extraction quarantine lane (#160): gated entity extraction + review queue
extract_entities, extraction_pending, extraction_review,
// v0.31: hot memory (facts table)
extract_facts, recall, forget_fact,
// v0.32.6: contradiction probe MCP surface (M3)
@@ -5657,6 +5961,8 @@ export const operations: Operation[] = [
code_blast, code_flow,
// v0.34 W3b: code_traversal_cache admin clear op
code_traversal_cache_clear,
// #3390: provider-agnostic embedding migration (local-only admin)
migrate_embeddings,
// v0.40.6.0 Schema Cathedral v3: 9 new ops — 7 read + 2 admin (NOT
// localOnly per D2 so remote agents (your OpenClaw, etc.) can author packs).
// schema_apply_mutations is batched per D10 — one MCP tool, N
+3 -2
View File
@@ -72,9 +72,10 @@ export class SlugRegistryError extends Error {
// SlugRegistry
// ---------------------------------------------------------------------------
// Shares the page-slug segment grammar (incl. CJK ranges, #738) with
// Shares the page-slug segment grammar (all scripts, #738/#3417) with
// validatePageSlug; keeps this site's dir/name shape (>= 2 segments).
const SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})+$`);
// `u` flag required by PAGE_SLUG_SEG's \p{...} classes.
const SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})+$`, 'u');
export class SlugRegistry {
constructor(private engine: BrainEngine) {}
+121 -33
View File
@@ -17,7 +17,26 @@ import type {
SourceRow,
} from './engine.ts';
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay, type BatchAuditSite } from './retry.ts';
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatchExhausted } from './audit/batch-retry-audit.ts';
import { runMigrations } from './migrate.ts';
import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
@@ -58,6 +77,7 @@ import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { unverifiedExtractionFragment } from './extraction-review.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
import {
@@ -418,9 +438,13 @@ export class PGLiteEngine implements BrainEngine {
let dims: number = DEFAULT_EMBEDDING_DIMENSIONS;
let model: string = DEFAULT_EMBEDDING_MODEL;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel() || model;
model = gw.getEmbeddingModel();
} catch { /* gateway not configured — use defaults */ }
await this.db.exec(getPGLiteSchema(dims, model));
@@ -978,7 +1002,8 @@ export class PGLiteEngine implements BrainEngine {
const { rows } = await this.db.query(
`SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
effective_date, effective_date_source,
source_kind, source_uri, ingested_via, ingested_at
source_kind, source_uri, ingested_via, ingested_at,
contextual_retrieval_mode
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
params
);
@@ -2072,7 +2097,10 @@ export class PGLiteEngine implements BrainEngine {
// Built on the bare `slug` output column: applied inside the `scored` CTE
// whose FROM is the single relation `hnsw_candidates`, so unqualified
// `slug` resolves cleanly (T1 per-page pool restructure).
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
// issue #160: guard predicate projected as `unverified_stub` in
// hnsw_candidates (parity with postgres-engine) so unverified stubs get
// factor 1.0, not the people/ 1.2x, inside the pre-LIMIT re-rank.
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail, 'unverified_stub');
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const innerLimit = offset + Math.max(limit * 5, 100);
@@ -2148,6 +2176,7 @@ export class PGLiteEngine implements BrainEngine {
CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
(${unverifiedExtractionFragment('p')}) AS unverified_stub,
1 - (cc.${col} <=> ${castSql}) AS raw_score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
@@ -2256,7 +2285,6 @@ export class PGLiteEngine implements BrainEngine {
});
} catch (err) {
if (err instanceof Error && err.name === 'RetryAbortError') throw err;
const { isRetryableConnError } = await import('./retry.ts');
if (isRetryableConnError(err)) {
auditLogBatchExhausted(auditSite, batchSize, opts.maxRetries + 1, err);
}
@@ -2315,15 +2343,28 @@ export class PGLiteEngine implements BrainEngine {
// Provenance fallback for chunks without an explicit `model`: resolve the
// gateway's runtime model, not the compile-time DEFAULT_EMBEDDING_MODEL.
// See postgres-engine.ts _upsertChunksOnce for the full rationale — pglite
// mirrors it for parity.
let resolvedModel: string = DEFAULT_EMBEDDING_MODEL;
// #3461: getEmbeddingModel() THROWS when unconfigured (never returns
// falsy) — on the throw path fall back to the brain's own
// `config.embedding_model` row, then the compile-time default as the
// last resort. See postgres-engine.ts _upsertChunksOnce for the full
// rationale — pglite mirrors it for parity.
let resolvedModel: string | null = null;
try {
const gw = await import('./ai/gateway.ts');
resolvedModel = gw.getEmbeddingModel() || resolvedModel;
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
// Gateway unconfigured (unit tests / pre-connect): keep the default.
try {
const cfg = await this.db.query(
`SELECT value FROM config WHERE key = 'embedding_model'`,
);
resolvedModel = ((cfg.rows[0] as { value?: string } | undefined)?.value) ?? null;
} catch {
// config table unreadable — fall through to the compile-time default.
}
}
if (!resolvedModel) resolvedModel = DEFAULT_EMBEDDING_MODEL;
for (const chunk of chunks) {
const embeddingStr = chunk.embedding
@@ -2376,6 +2417,9 @@ export class PGLiteEngine implements BrainEngine {
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
// only embedding-shaped fields doesn't clobber metadata to NULL.
//
// #3461: `model` mirrors the `embedding` CASE branch-for-branch so the label always
// describes whichever vector wins the upsert. See postgres-engine.ts for rationale.
await this.db.query(
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2389,7 +2433,14 @@ export class PGLiteEngine implements BrainEngine {
THEN EXCLUDED.embedding
ELSE content_chunks.embedding
END,
model = COALESCE(EXCLUDED.model, content_chunks.model),
model = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.model
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.model
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.model
ELSE content_chunks.model
END,
token_count = EXCLUDED.token_count,
embedded_at = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
@@ -2428,15 +2479,21 @@ export class PGLiteEngine implements BrainEngine {
/**
* Build the stale-chunk WHERE clause + positional params. embed_skip is
* always excluded. `signature` widens "stale" to include embedding_signature
* drift (NULL grandfathered never stale). Shared by countStaleChunks +
* drift (NULL grandfathered never stale). `includeNullSignature` (#3391)
* lifts the grandfather clause so pre-stamp pages count as stale too
* (provider-migration paths). Shared by countStaleChunks +
* sumStaleChunkChars so they can't drift.
*/
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } {
const params: unknown[] = [];
const conds: string[] = [];
if (opts?.signature !== undefined) {
params.push(opts.signature);
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
conds.push(
opts.includeNullSignature
? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})`
: `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`,
);
} else {
conds.push(`cc.embedding IS NULL`);
}
@@ -2448,7 +2505,7 @@ export class PGLiteEngine implements BrainEngine {
return { where: conds.join(' AND '), params };
}
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
// D7: source-scoped count for `gbrain embed --stale --source X`. Always
// JOIN pages so embed-skip + signature predicates apply. PGLite is
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
@@ -2464,7 +2521,7 @@ export class PGLiteEngine implements BrainEngine {
return Number(count);
}
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
// length for the sync cost preview. ::bigint guards int4 overflow.
const { where, params } = this.buildStaleChunkWhere(opts);
@@ -2486,24 +2543,29 @@ export class PGLiteEngine implements BrainEngine {
);
}
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number> {
// NULL out embeddings whose page signature is set AND differs from the
// current model signature. GRANDFATHER: NULL signature untouched. Feeds
// the existing NULL-embedding cursor so listStaleChunks stays unchanged.
// current model signature. GRANDFATHER: NULL signature untouched
// UNLESS includeNullSignature (#3391): provider migrations must not
// leave pre-stamp pages in the old embedding space. Feeds the existing
// NULL-embedding cursor so listStaleChunks stays unchanged.
const params: unknown[] = [opts.signature];
let srcClause = '';
if (opts.sourceId !== undefined) {
params.push(opts.sourceId);
srcClause = ` AND p.source_id = $${params.length}`;
}
const sigClause = opts.includeNullSignature
? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)`
: `p.embedding_signature IS NOT NULL
AND p.embedding_signature <> $1`;
const { rows } = await this.db.query(
`UPDATE content_chunks cc
SET embedding = NULL, embedded_at = NULL
FROM pages p
WHERE cc.page_id = p.id
AND cc.embedding IS NOT NULL
AND p.embedding_signature IS NOT NULL
AND p.embedding_signature <> $1${srcClause}
AND ${sigClause}${srcClause}
RETURNING cc.page_id`,
params,
);
@@ -3407,6 +3469,20 @@ export class PGLiteEngine implements BrainEngine {
return result;
}
async getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>> {
if (pageIds.length === 0) return new Set();
// Parity with PostgresEngine.getUnverifiedExtractionPageIds (issue #160).
// Predicate is the shared unverifiedExtractionFragment so this query and
// the SQL-side source-boost guard can never drift.
const { rows } = await this.db.query(
`SELECT id FROM pages
WHERE id = ANY($1::int[])
AND ${unverifiedExtractionFragment('pages')}`,
[pageIds]
);
return new Set((rows as { id: number }[]).map((r) => Number(r.id)));
}
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
if (slugs.length === 0) return new Map();
const { rows } = await this.db.query(
@@ -3788,7 +3864,6 @@ export class PGLiteEngine implements BrainEngine {
async mergeOntologyFact(obs: OntologyObservationInput): Promise<OntologyMergeResult> {
const sourceId = obs.sourceId ?? 'default';
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const dimension = normalizeDimension(obs.dimension);
const vh = valueHash(obs.value);
const conf = obs.confidence ?? 0.7;
@@ -4251,9 +4326,15 @@ export class PGLiteEngine implements BrainEngine {
async deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }> {
const prefixes = opts?.excludeSourcePrefixes;
// #2646: keep soft-expired legacy rows (row_num NULL — never
// fence-owned) so a fence reconcile can't destroy forget_fact's
// legacy DB-only forget record.
const expiredLegacyFilter = opts?.preserveExpiredLegacy
? ` AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
: '';
if (prefixes && prefixes.length > 0) {
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
@@ -4262,13 +4343,13 @@ export class PGLiteEngine implements BrainEngine {
const result = await this.db.query(
`DELETE FROM facts
WHERE source_id = $1 AND source_markdown_slug = $2
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`,
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))${expiredLegacyFilter}`,
[source_id, slug, patterns],
);
return { deleted: result.affectedRows ?? 0 };
}
const result = await this.db.query(
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`,
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2${expiredLegacyFilter}`,
[source_id, slug],
);
return { deleted: result.affectedRows ?? 0 };
@@ -5272,12 +5353,16 @@ export class PGLiteEngine implements BrainEngine {
// pages_with_timeline) and v0.10.3 graph layer (link_coverage, timeline_coverage,
// most_connected). Both coexist: master's brain_score is the composite
// dashboard, v0.10.3 metrics give entity-page-level granularity.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const { rows: [h] } = await this.db.query(`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5302,7 +5387,7 @@ export class PGLiteEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company')
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
ORDER BY link_count DESC
LIMIT 5
`);
@@ -5321,6 +5406,7 @@ export class PGLiteEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`);
const r = h as Record<string, unknown>;
@@ -5415,15 +5501,18 @@ export class PGLiteEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
newSlug = validateSlug(newSlug);
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (mirrors postgres-engine.ts).
await this.db.query(
const result = await this.db.query(
`UPDATE pages SET slug = $1, updated_at = now() WHERE slug = $2 AND source_id = $3`,
[newSlug, oldSlug, sourceId]
);
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.affectedRows ?? 0;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
@@ -5937,7 +6026,6 @@ export class PGLiteEngine implements BrainEngine {
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
+133 -39
View File
@@ -13,7 +13,29 @@ import type {
NewFact, FactListOpts, FactsHealth,
SourceRow,
} from './engine.ts';
import { withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay, type BatchAuditSite } from './retry.ts';
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logDbDisconnect } from './audit/db-disconnect-audit.ts';
import { logPoolRecovery } from './audit/pool-recovery-audit.ts';
import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatchExhausted } from './audit/batch-retry-audit.ts';
import type {
DomainBankSampleOpts, CorpusSampleOpts, DomainBankRow,
@@ -65,6 +87,7 @@ import { logConnectionEvent } from './connection-audit.ts';
import { validateSlug, contentHash, rowToPage, rowToStalePage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, takeHitRowToHit, isUndefinedTableError, warnOncePerProcess } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { unverifiedExtractionFragment } from './extraction-review.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { SOURCE_CONFIG_OBJECT_SQL } from './source-config-sql.ts';
@@ -330,7 +353,6 @@ export class PostgresEngine implements BrainEngine {
// even a no-op disconnect (engine that was never connected) is
// recorded — that case may itself be a caller-side bug worth seeing.
try {
const { logDbDisconnect } = await import('./audit/db-disconnect-audit.ts');
logDbDisconnect('postgres', this._connectionStyle ?? 'unknown');
} catch { /* best-effort; never block disconnect on audit failure */ }
// v0.30.1: tear down the direct pool first if the manager owns one.
@@ -380,9 +402,13 @@ export class PostgresEngine implements BrainEngine {
let dims: number = DEFAULT_EMBEDDING_DIMENSIONS;
let model: string = DEFAULT_EMBEDDING_MODEL;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel() || model;
model = gw.getEmbeddingModel();
} catch { /* gateway not yet configured — use defaults */ }
const sqlText = getPostgresSchema(dims, model);
@@ -1030,7 +1056,8 @@ export class PostgresEngine implements BrainEngine {
const rows = await tx`
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
effective_date, effective_date_source,
source_kind, source_uri, ingested_via, ingested_at
source_kind, source_uri, ingested_via, ingested_at,
contextual_retrieval_mode
FROM pages
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
LIMIT 1
@@ -2120,7 +2147,10 @@ export class PostgresEngine implements BrainEngine {
// innerLimit scales with offset to preserve the pagination contract:
// a fixed cap of 100 would silently empty offset > 100.
const boostMap = resolveBoostMap();
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
// issue #160: the guard predicate is projected as `unverified_stub` in
// hnsw_candidates (frontmatter isn't otherwise available at re-rank), so
// unverified auto-extracted stubs get factor 1.0, not the people/ 1.2x.
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail, 'unverified_stub');
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const innerLimit = offset + Math.max(limit * 5, 100);
@@ -2220,6 +2250,7 @@ export class PostgresEngine implements BrainEngine {
CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
(${unverifiedExtractionFragment('p')}) AS unverified_stub,
1 - (cc.${col} <=> ${castSql}) AS raw_score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
@@ -2373,8 +2404,8 @@ export class PostgresEngine implements BrainEngine {
if (err instanceof Error && err.name === 'RetryAbortError') throw err;
// Best-effort exhausted-retry log. If the error wasn't retryable in
// the first place, isRetryableConnError(err) is false and we skip.
// Lazy-import to avoid a circular dep concern.
const { isRetryableConnError } = await import('./retry.ts');
// retry.ts is already in this module's static graph through withRetry, so
// classifying the exhausted error does not need a second runtime import.
if (isRetryableConnError(err)) {
auditLogBatchExhausted(auditSite, batchSize, opts.maxRetries + 1, err);
}
@@ -2432,14 +2463,30 @@ export class PostgresEngine implements BrainEngine {
// hardcoded default (e.g. zeroentropyai:zembed-1) onto rows whose vectors
// were produced by a different, config-resolved model — corrupting the
// provenance that signature-drift staleness + dim-migration logic trust.
// Mirrors the resolve-then-fallback pattern used for schema sizing above.
let resolvedModel: string = DEFAULT_EMBEDDING_MODEL;
//
// #3461: getEmbeddingModel() THROWS when the gateway is unconfigured —
// it never returns falsy — so an `||` guard here is dead code and the
// catch path used to stamp the compile-time default onto rows whose
// vectors came from the config-resolved provider. On the throw path we
// now fall back to the brain's own `config.embedding_model` row (kept
// current by init / migrate / retrieval-upgrade), which names the model
// that actually produced this brain's vectors. The compile-time default
// is the LAST resort (fresh brain whose config row doesn't exist yet).
let resolvedModel: string | null = null;
try {
const gw = await import('./ai/gateway.ts');
resolvedModel = gw.getEmbeddingModel() || resolvedModel;
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
// Gateway unconfigured (unit tests / pre-connect): keep the default.
try {
const cfg = await sql`SELECT value FROM config WHERE key = 'embedding_model'`;
resolvedModel = (cfg[0]?.value as string | undefined) ?? null;
} catch {
// config table unreadable — fall through to the compile-time default.
}
}
if (!resolvedModel) resolvedModel = DEFAULT_EMBEDDING_MODEL;
for (const chunk of chunks) {
const embeddingStr = chunk.embedding
@@ -2503,6 +2550,11 @@ export class PostgresEngine implements BrainEngine {
// pure re-embed (chunk_text unchanged) COALESCEs so a caller that only carries embedding
// doesn't clobber metadata to NULL. Without this, every embed --stale pass nuked code-def's
// primary index for thousands of chunks at once.
//
// #3461: `model` mirrors the `embedding` CASE branch-for-branch — the label must
// describe whichever vector WINS the upsert. The old COALESCE(EXCLUDED.model, …)
// relabeled preserved (older-model) vectors with the current gateway model on every
// partial re-embed, corrupting provenance without changing the vector.
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2516,7 +2568,14 @@ export class PostgresEngine implements BrainEngine {
THEN EXCLUDED.embedding
ELSE content_chunks.embedding
END,
model = COALESCE(EXCLUDED.model, content_chunks.model),
model = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.model
WHEN content_chunks.embedding IS NULL THEN EXCLUDED.model
WHEN EXCLUDED.embedded_at IS NOT NULL
AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at)
THEN EXCLUDED.model
ELSE content_chunks.model
END,
token_count = EXCLUDED.token_count,
embedded_at = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
@@ -2557,15 +2616,21 @@ export class PostgresEngine implements BrainEngine {
/**
* Build the stale-chunk WHERE clause + positional params for sql.unsafe.
* embed_skip always excluded. `signature` widens "stale" to include
* embedding_signature drift (NULL grandfathered). Shared by
* countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling).
* embedding_signature drift (NULL grandfathered). `includeNullSignature`
* (#3391) lifts the grandfather clause so pre-stamp pages count as stale
* too (provider-migration paths). Shared by countStaleChunks +
* sumStaleChunkChars (parity with the PGLite sibling).
*/
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } {
const params: unknown[] = [];
const conds: string[] = [];
if (opts?.signature !== undefined) {
params.push(opts.signature);
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
conds.push(
opts.includeNullSignature
? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})`
: `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`,
);
} else {
conds.push(`cc.embedding IS NULL`);
}
@@ -2577,10 +2642,11 @@ export class PostgresEngine implements BrainEngine {
return { where: conds.join(' AND '), params };
}
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
// Always JOIN pages so the embed_skip + signature predicates apply.
// D7: source_id scoping. v0.41.31: optional signature widens staleness
// to embedding_signature drift (NULL grandfathered).
// to embedding_signature drift (NULL grandfathered unless
// includeNullSignature, #3391).
const { where, params } = this.buildStaleChunkWhere(opts);
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
@@ -2595,7 +2661,7 @@ export class PostgresEngine implements BrainEngine {
});
}
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
// length for the sync cost preview. ::bigint guards int4 overflow.
const { where, params } = this.buildStaleChunkWhere(opts);
@@ -2617,24 +2683,29 @@ export class PostgresEngine implements BrainEngine {
`;
}
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number> {
// NULL embeddings whose page signature is set AND differs from current.
// GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor
// so listStaleChunks stays unchanged. RETURNING → row count.
// GRANDFATHER: NULL signature untouched — UNLESS includeNullSignature
// (#3391): provider migrations must not leave pre-stamp pages in the old
// embedding space. Feeds the NULL-embedding cursor so listStaleChunks
// stays unchanged. RETURNING → row count.
const params: unknown[] = [opts.signature];
let srcClause = '';
if (opts.sourceId !== undefined) {
params.push(opts.sourceId);
srcClause = ` AND p.source_id = $${params.length}`;
}
const sigClause = opts.includeNullSignature
? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)`
: `p.embedding_signature IS NOT NULL
AND p.embedding_signature <> $1`;
const rows = await this.sql.unsafe(
`UPDATE content_chunks cc
SET embedding = NULL, embedded_at = NULL
FROM pages p
WHERE cc.page_id = p.id
AND cc.embedding IS NOT NULL
AND p.embedding_signature IS NOT NULL
AND p.embedding_signature <> $1${srcClause}
AND ${sigClause}${srcClause}
RETURNING cc.page_id`,
params as Parameters<typeof this.sql.unsafe>[1],
);
@@ -3559,6 +3630,20 @@ export class PostgresEngine implements BrainEngine {
return result;
}
async getUnverifiedExtractionPageIds(pageIds: number[]): Promise<Set<number>> {
if (pageIds.length === 0) return new Set();
const sql = this.sql;
// Predicate is the shared unverifiedExtractionFragment (issue #160) so
// this query and the SQL-side source-boost guard can never drift.
const rows = await sql.unsafe(
`SELECT id FROM pages
WHERE id = ANY($1::int[])
AND ${unverifiedExtractionFragment('pages')}`,
[pageIds] as never,
);
return new Set((rows as unknown as { id: number }[]).map((r) => Number(r.id)));
}
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
if (slugs.length === 0) return new Map();
const sql = this.sql;
@@ -3923,7 +4008,6 @@ export class PostgresEngine implements BrainEngine {
async mergeOntologyFact(obs: OntologyObservationInput): Promise<OntologyMergeResult> {
const sql = this.sql;
const sourceId = obs.sourceId ?? 'default';
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const dimension = normalizeDimension(obs.dimension);
const vh = valueHash(obs.value);
const conf = obs.confidence ?? 0.7;
@@ -4407,10 +4491,16 @@ export class PostgresEngine implements BrainEngine {
async deleteFactsForPage(
slug: string,
source_id: string,
opts?: { excludeSourcePrefixes?: string[] },
opts?: { excludeSourcePrefixes?: string[]; preserveExpiredLegacy?: boolean },
): Promise<{ deleted: number }> {
const sql = this.sql;
const prefixes = opts?.excludeSourcePrefixes;
// #2646: keep soft-expired legacy rows (row_num NULL — never
// fence-owned) so a fence reconcile can't destroy forget_fact's
// legacy DB-only forget record.
const expiredLegacyFilter = opts?.preserveExpiredLegacy
? sql`AND NOT (row_num IS NULL AND expired_at IS NOT NULL)`
: sql``;
if (prefixes && prefixes.length > 0) {
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
@@ -4421,11 +4511,12 @@ export class PostgresEngine implements BrainEngine {
WHERE source_id = ${source_id}
AND source_markdown_slug = ${slug}
AND NOT (COALESCE(source, '') LIKE ANY(${patterns}))
${expiredLegacyFilter}
`;
return { deleted: result.count ?? 0 };
}
const result = await sql`
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug}
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} ${expiredLegacyFilter}
`;
return { deleted: result.count ?? 0 };
}
@@ -5365,12 +5456,16 @@ export class PostgresEngine implements BrainEngine {
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5392,7 +5487,7 @@ export class PostgresEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company')
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
ORDER BY link_count DESC
LIMIT 5
`;
@@ -5411,6 +5506,7 @@ export class PostgresEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`;
const pageCount = Number(h.page_count);
@@ -5502,14 +5598,17 @@ export class PostgresEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
newSlug = validateSlug(newSlug);
const sql = this.sql;
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (which would either rename them all OR fail the
// (source_id, slug) UNIQUE if the new slug already exists in another source).
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
const result = await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.count ?? 0;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
@@ -5748,12 +5847,10 @@ export class PostgresEngine implements BrainEngine {
let isReap = false;
if (ctx?.error !== undefined) {
try {
const { isConnectionEndedError } = await import('./retry-matcher.ts');
isReap = isConnectionEndedError(ctx.error);
} catch { /* classification is best-effort */ }
}
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery(isReap ? 'reap_detected' : 'reconnect_other', ctx?.error);
} catch { /* audit is best-effort */ }
@@ -5777,7 +5874,6 @@ export class PostgresEngine implements BrainEngine {
// New pool is live — discard the old one best-effort.
if (oldSql) { try { await oldSql.end({ timeout: 5 }); } catch { /* swallow */ } }
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery('reconnect_succeeded');
} catch { /* best-effort */ }
} catch (err) {
@@ -5789,7 +5885,6 @@ export class PostgresEngine implements BrainEngine {
this._sql = oldSql;
this.connectionManager = oldManager;
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery('reconnect_failed', err);
} catch { /* best-effort */ }
throw err; // let batchRetry's backoff handle the retry
@@ -6226,7 +6321,6 @@ export class PostgresEngine implements BrainEngine {
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
+92 -1
View File
@@ -63,6 +63,7 @@ import type { BrainEngine } from './engine.ts';
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
import { computeReembedEstimate } from './post-upgrade-reembed.ts';
import { hnswIndexExpected } from './vector-index.ts';
// ============================================================================
// Constants
@@ -551,8 +552,12 @@ export async function undoRetrievalUpgrade(engine: BrainEngine): Promise<
*
* IF NOT EXISTS on CREATE INDEX makes the operation safe to re-run during
* `--resume`.
*
* Exported (#3390) so the provider-agnostic embedding migration
* (src/core/embedding-migration.ts) reuses the SAME dimension-transition
* path instead of duplicating the DDL sequence.
*/
async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise<void> {
export async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise<void> {
// v0.41 fix: only transition the primary text embedding column.
// The embedding_image (v0.27.1) and embedding_multimodal (v0.36 / migration
// v78) columns use SEPARATE multimodal models (e.g. voyage-multimodal-3 at
@@ -595,9 +600,95 @@ async function runSchemaTransition(engine: BrainEngine, targetDim: number): Prom
WHERE embedding_image IS NOT NULL`,
);
}
// #3390: the OTHER two dim-pinned columns that carry TEXT-embedding-space
// vectors. Both are created at brain-birth width (migrate.ts v55 for
// query_cache, v42 for facts) and NO migration ever ALTERs them, so before
// this fix a dimension change left them at the old width:
// - query_cache.embedding stayed narrow → every store() AND lookup()
// silently swallowed the width error (by design, so the cache can
// never break search), i.e. a PERMANENT 0% hit rate.
// - facts.embedding stayed narrow → every per-fact embed write failed
// ($N::vector into the old width), and the doctor check that would
// warn is skipped on PGLite (the DEFAULT engine).
// Both are text-embedding-space columns, so they MUST move with
// content_chunks.embedding. The image/multimodal columns above are the
// deliberate exception (separate models, independent dims).
for (const t of TEXT_EMBEDDING_DIM_PINNED_TABLES) {
await transitionDimPinnedColumn(tx, t.table, t.index, t.indexSql, targetDim);
}
});
}
/**
* The dim-pinned TEXT-embedding-space columns outside content_chunks.
* `indexSql` is a factory because each table's index carries its own partial
* WHERE clause + opclass, and the opclass must match the column TYPE
* (vector_cosine_ops vs halfvec_cosine_ops).
*/
const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
table: string;
index: string;
indexSql: (opclass: string) => string;
}> = [
{
table: 'query_cache',
index: 'idx_query_cache_embedding_hnsw',
indexSql: (opclass) =>
`CREATE INDEX IF NOT EXISTS idx_query_cache_embedding_hnsw
ON query_cache USING hnsw (embedding ${opclass})
WHERE embedding IS NOT NULL`,
},
{
table: 'facts',
index: 'idx_facts_embedding_hnsw',
indexSql: (opclass) =>
`CREATE INDEX IF NOT EXISTS idx_facts_embedding_hnsw
ON facts USING hnsw (embedding ${opclass})
WHERE embedding IS NOT NULL AND expired_at IS NULL`,
},
];
/**
* Rebuild one dim-pinned embedding column at `targetDim`, PRESERVING its
* existing column type (`vector` vs `halfvec` migrate.ts picks halfvec when
* the server supports it, and the HNSW opclass must match). No-op when the
* table or column doesn't exist (fresh/older brains).
*
* Dropping the column discards the stored vectors, which is correct: they are
* in the OLD embedding space and unusable after the swap. query_cache is a
* cache (refills on the next query); facts re-embed on their next write /
* `gbrain extract` pass.
*/
async function transitionDimPinnedColumn(
tx: { executeRaw: <T = unknown>(sql: string, params?: unknown[]) => Promise<T[]> },
table: string,
indexName: string,
indexSql: (opclass: string) => string,
targetDim: number,
): Promise<void> {
const probe = await tx.executeRaw<{ udt_name: string | null }>(
`SELECT udt_name FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = $1 AND column_name = 'embedding'`,
[table],
);
const udt = probe[0]?.udt_name;
if (!udt) return; // table or column absent — nothing to transition
// Preserve the column type; anything unexpected falls back to `vector`.
const columnType: 'vector' | 'halfvec' = udt.toLowerCase() === 'halfvec' ? 'halfvec' : 'vector';
const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
await tx.executeRaw(`DROP INDEX IF EXISTS ${indexName}`);
await tx.executeRaw(`ALTER TABLE ${table} DROP COLUMN IF EXISTS embedding`);
await tx.executeRaw(`ALTER TABLE ${table} ADD COLUMN embedding ${columnType}(${targetDim})`);
// HNSW has a per-type dimension ceiling; above it pgvector refuses the
// index and exact scans remain the (correct, slower) path. Mirrors the
// same guard in migrate.ts's original DDL.
if (hnswIndexExpected(columnType, targetDim)) {
await tx.executeRaw(indexSql(opclass));
}
}
// ============================================================================
// Helpers
// ============================================================================
+107 -9
View File
@@ -48,6 +48,32 @@ import {
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
/**
* Which detail levels get the compiled_truth boost (#3430).
*
* ONLY `low`. The documented contract (`src/core/operations.ts`) is
* "low (compiled truth only), medium (default, all with dedup), high (all
* chunks)" so `low` is the level that privileges compiled truth, and both
* `medium` and `high` are supposed to see everything on equal footing.
*
* This was previously spelled `detail !== 'high'`, i.e. written as though
* `high` were the special case. Because COMPILED_TRUTH_BOOST is applied AFTER
* RRF normalization, and RRF's whole range over a 100-deep pool is 1/60 1/160,
* a 2.0x multiplier is not a tilt break-even is `2/(60+r) >= 1/60`, so any
* boosted chunk inside the first 60 ranks outranks an unboosted rank-1 chunk.
* At the default detail that made search categorically compiled-truth-only:
* a page whose answer lived in a `fenced_code` chunk returned the prose chunk,
* and the code chunk fell out of the window entirely.
*
* Extracted as a named predicate rather than left inline at three call sites so
* the detailboost mapping is directly testable. An inline expression can only
* be covered through a full `hybridSearch` round trip, which is why the
* original inversion went unnoticed.
*/
export function shouldBoostCompiledTruth(detail: string | null | undefined): boolean {
return detail === 'low';
}
const pendingCacheWrites = new Set<Promise<unknown>>();
/**
@@ -76,6 +102,37 @@ export async function stampContentFlags(engine: BrainEngine, results: SearchResu
}
}
/**
* Extraction quarantine lane (issue #160). Stamps `SearchResult.unverified`
* for any result whose page is an unverified auto-extracted entity stub
* (frontmatter `provenance: 'auto-extracted'` + `status: 'unverified'`).
* MUST run PRE-fusion: rrfFusion/rrfFusionWeighted read the flag to skip the
* COMPILED_TRUTH_BOOST for these pages, so a stub fabricated by hostile
* ingested text ranks as ordinary content, never with entity authority.
* One batched query over the candidate arms' page_ids. Fail-open on the
* fetch (a marker-fetch failure must not break retrieval) the boost then
* applies, but the SQL-side source-boost guard still holds.
*/
export async function stampUnverifiedExtractions(
engine: BrainEngine,
results: SearchResult[],
): Promise<void> {
if (results.length === 0) return;
try {
const ids = [...new Set(
results.map((r) => r.page_id).filter((n): n is number => typeof n === 'number' && Number.isFinite(n)),
)];
if (ids.length === 0) return;
const unverified = await engine.getUnverifiedExtractionPageIds(ids);
if (unverified.size === 0) return;
for (const r of results) {
if (unverified.has(r.page_id)) r.unverified = true;
}
} catch {
// best-effort: never break retrieval.
}
}
/**
* v0.42.20.0 bounded drain (was an unbounded `Promise.allSettled`, codex
* confirmed; TODOS retrofit). Mirrors `awaitPendingLastRetrievedWrites`: races
@@ -1101,19 +1158,44 @@ export async function hybridSearch(
// provider (Voyage, ZE) works fine.
const { isAvailable } = await import('../ai/gateway.ts');
const providerProbe = resolvedCol.embeddingModel || undefined;
if (!isAvailable('embedding', providerProbe)) {
// Image/both/unified routing embeds via the MULTIMODAL provider, not the
// text provider — so a multimodal-only install (text provider absent) must
// still reach the multimodal branch below. Probe the multimodal provider
// explicitly and only short-circuit when neither the text provider nor (for
// multimodal-routed queries) the multimodal provider is reachable. Without
// this guard a multimodal-only install would fall to keyword-only here and
// never run the image/unified vector path.
const multimodalProviderProbe =
cfgForColumn?.embedding_multimodal_model ?? 'voyage:voyage-multimodal-3';
// The LLM intent tie-break (below) can escalate a regex-'text' query to
// 'image'/'both'; account for that possibility so an ambiguous query on a
// multimodal-only install still reaches the multimodal branch.
const mayEscalateToMultimodal =
earlyModality === 'text' &&
resolvedMode.cross_modal_llm_intent &&
isAmbiguousModalityQuery(query);
const willTryMultimodal =
(resolvedMode.unified_multimodal === true ||
earlyModality === 'image' ||
earlyModality === 'both' ||
mayEscalateToMultimodal) &&
isAvailable('embedding', multimodalProviderProbe);
if (!isAvailable('embedding', providerProbe) && !willTryMultimodal) {
// v0.43 — fuse the relational arm with keyword so typed-edge answers
// survive on the no-embedding-provider path (the relational win is most
// valuable exactly when vector is unavailable). The title arm fuses here
// too — an exact-title lookup on a keyless install is precisely where
// chunk-grain keyword FTS alone fails (D1).
// issue #160: stamp unverified stubs BEFORE fusion so the compiled-truth
// boost skips them (flag survives fusion's result spread).
await stampUnverifiedExtractions(engine, [...keywordResults, ...titleResults, ...relationalList]);
let noEmbedResults = keywordResults;
if (relationalList.length > 0 || titleResults.length > 0) {
const fk = opts?.rrfK ?? RRF_K;
const noEmbedLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
noEmbedResults = rrfFusionWeighted(noEmbedLists, shouldBoostCompiledTruth(detailResolved));
}
if (noEmbedResults.length > 0) {
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
@@ -1233,7 +1315,10 @@ export async function hybridSearch(
if (unifiedRouting) {
try {
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
if (!aiIsAvailable('embedding')) {
// Probe the MULTIMODAL provider, not the global default — on a
// multimodal-only install the global default (text) is absent but the
// multimodal provider is configured, and unified routing embeds via it.
if (!aiIsAvailable('embedding', multimodalProviderProbe)) {
throw new Error('gateway not configured for embedding — unified multimodal would also fail');
}
const unifiedEmbedding = await embedQueryMultimodal(query);
@@ -1268,7 +1353,10 @@ export async function hybridSearch(
// OR the embed throws, log a structured warning and fall through to text.
try {
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
if (!aiIsAvailable('embedding')) {
// Probe the MULTIMODAL provider, not the global default — the image side
// embeds via the multimodal model, which may be configured even when the
// text/global-default embedding provider is absent (multimodal-only).
if (!aiIsAvailable('embedding', multimodalProviderProbe)) {
throw new Error('gateway not configured for embedding — multimodal would also fail');
}
const imageEmbedding = await embedQueryMultimodal(query);
@@ -1342,13 +1430,16 @@ export async function hybridSearch(
// v0.43: fuse the relational arm with keyword via RRF so typed-edge
// answers survive even when vector is unavailable. The title arm fuses
// here too (same rationale as the no-embedding-provider path — D1).
// issue #160: stamp unverified stubs BEFORE fusion (see the
// no-embedding-provider path for rationale).
await stampUnverifiedExtractions(engine, [...keywordResults, ...titleResults, ...relationalList]);
let fallbackResults = keywordResults;
if (relationalList.length > 0 || titleResults.length > 0) {
const fk = opts?.rrfK ?? RRF_K;
const fallbackLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
fallbackResults = rrfFusionWeighted(fallbackLists, shouldBoostCompiledTruth(detail));
}
if (fallbackResults.length > 0) {
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
@@ -1431,7 +1522,11 @@ export async function hybridSearch(
allLists.push({ list: relationalList, k: baseRrfK });
}
let fused = rrfFusionWeighted(allLists, detail !== 'high');
// issue #160: stamp unverified auto-extracted stubs across ALL candidate
// arms BEFORE fusion so the compiled-truth authority boost skips them.
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
let fused = rrfFusionWeighted(allLists, shouldBoostCompiledTruth(detail));
// Cosine re-scoring before dedup so semantically better chunks survive.
// v0.36 (D9): hydrate from the active embedding column so rescore happens
@@ -1997,7 +2092,9 @@ export function rrfFusionWeighted(
if (maxScore > 0) {
for (const e of entries) {
e.score = e.score / maxScore;
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
// issue #160: unverified auto-extracted stubs (stamped pre-fusion by
// stampUnverifiedExtractions) never get the compiled-truth authority boost.
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' && e.result.unverified !== true ? COMPILED_TRUTH_BOOST : 1.0;
e.score *= boost;
}
}
@@ -2040,8 +2137,9 @@ export function rrfFusion(lists: SearchResult[][], k: number, applyBoost = true)
const rawScore = e.score;
e.score = e.score / maxScore;
// Apply compiled truth boost after normalization (skip for detail=high)
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
// Apply compiled truth boost after normalization (skip for detail=high;
// skip for unverified auto-extracted stubs — issue #160)
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' && e.result.unverified !== true ? COMPILED_TRUTH_BOOST : 1.0;
e.score *= boost;
if (DEBUG) {
+11 -1
View File
@@ -756,7 +756,17 @@ export function attributeKnob<K extends keyof ModeBundle>(
// slugs written by a process without it, and vice versa. Same one-time
// global cold-miss pattern as the bumps above; refills within
// cache.ttl_seconds (3600s default).
export const KNOBS_HASH_VERSION = 12;
//
// bump 12→13 (#3390/#3391): embedding-provider migration wave. The `prov=`
// component only isolates callers that thread KnobsHashContext.embeddingModel;
// legacy callers hash `prov=default` before AND after a provider swap, so a
// cache row computed against the pre-migration embedding space could be
// served post-migration. `gbrain migrate embeddings` purges query_cache
// directly at swap time; this version bump is the belt-and-braces for rows
// written between the #3391 stale-fix (which changes which chunks count as
// current) and the operator's migration run. Same one-time global cold-miss
// pattern as the bumps above.
export const KNOBS_HASH_VERSION = 14;
/**
* v0.36 (D8 / CDX-2) second-arg context for the cache key. The
+22 -1
View File
@@ -18,6 +18,7 @@
*/
import { quarantineFilterFragment } from '../quarantine.ts';
import { unverifiedExtractionFragment } from '../extraction-review.ts';
/**
* Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal.
@@ -63,6 +64,7 @@ export function buildSourceFactorCase(
slugColumn: string,
boostMap: Record<string, number>,
detail: 'low' | 'medium' | 'high' | undefined,
unverifiedGuardColumn?: string,
): string {
// Loose-string guard: agents passing `"HIGH"` or `"high "` over MCP/JSON
// should still hit the temporal-bypass path. TypeScript narrows `detail`
@@ -80,7 +82,26 @@ export function buildSourceFactorCase(
`WHEN ${slugColumn} LIKE ${buildLikePrefixLiteral(prefix)} THEN ${factor}`
).join(' ');
return `(CASE ${whens} ELSE 1.0 END)`;
// Extraction quarantine lane (issue #160): unverified auto-extracted stubs
// never receive the namespace-authority factor (people/ / companies/ 1.2x)
// — they rank as ordinary content until promoted. Two forms:
// - table-qualified slug column ('p.slug'): reference the sibling
// `frontmatter` column inline via unverifiedExtractionFragment.
// - bare column + `unverifiedGuardColumn`: the vector arm's re-rank CTE
// has no frontmatter column, so its inner hnsw_candidates CTE projects
// the predicate as a boolean (`... AS unverified_stub`) and passes the
// column name here. Without this the 1.2x would apply INSIDE the
// scored/best_per_page pipeline pre-LIMIT — an unverified stub could
// outrank AND evict a legitimate page from the candidate pool, which
// nothing downstream can restore.
const alias = slugColumn.includes('.') ? slugColumn.split('.')[0] : null;
const unverifiedGuard = unverifiedGuardColumn
? `WHEN ${unverifiedGuardColumn} THEN 1.0 `
: alias
? `WHEN ${unverifiedExtractionFragment(alias)} THEN 1.0 `
: '';
return `(CASE ${unverifiedGuard}${whens} ELSE 1.0 END)`;
}
/**
+11
View File
@@ -33,6 +33,17 @@
export const SOURCE_ID_RE = /^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$/;
/**
* Sentinel meaning "span every source" (#1712). Deliberately NOT a valid
* source id (underscores are rejected by SOURCE_ID_RE), so it can never
* collide with a real source, be created via `sources add`, or leak into
* lock ids / path joins. The resolver's explicit/env tiers pass it through
* verbatim; `sourceScopeOpts` translates it to an unscoped read for trusted
* local callers and keeps it as an unsatisfiable literal for remote callers
* (fail-closed).
*/
export const ALL_SOURCES = '__all__';
/** Returns true if the string matches the canonical source_id regex. */
export function isValidSourceId(s: unknown): s is string {
return typeof s === 'string' && SOURCE_ID_RE.test(s);
+21 -5
View File
@@ -17,9 +17,13 @@ import { readFileSync, lstatSync, type Stats } from 'fs';
import { join, dirname, resolve } from 'path';
import type { BrainEngine } from './engine.ts';
import { isSourceFederated } from './sources-load.ts';
import { SOURCE_ID_RE, isValidSourceId } from './source-id.ts';
import { SOURCE_ID_RE, isValidSourceId, ALL_SOURCES } from './source-id.ts';
import { isTrustedDotfile, realpathOrResolve } from './path-confine.ts';
// Re-export so scope-resolution call sites can import the sentinel from
// either module (#1712).
export { ALL_SOURCES };
const DOTFILE = '.gbrain-source';
// Canonical SOURCE_ID_RE imported from `source-id.ts` (single source of truth).
// Re-exported below as `__testing.SOURCE_ID_RE` for legacy test imports.
@@ -83,8 +87,11 @@ export async function resolveSourceId(
explicit: string | null | undefined,
cwd: string = process.cwd(),
): Promise<string> {
// 1. Explicit flag wins.
// 1. Explicit flag wins. The __all__ sentinel passes through verbatim
// (#1712) — it is not a source id, so it skips both the regex and
// assertSourceExists; sourceScopeOpts gives it span-everything semantics.
if (explicit) {
if (explicit === ALL_SOURCES) return ALL_SOURCES;
if (!SOURCE_ID_RE.test(explicit)) {
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
}
@@ -92,9 +99,10 @@ export async function resolveSourceId(
return explicit;
}
// 2. Env var.
// 2. Env var. Same __all__ pass-through (#2140).
const env = process.env.GBRAIN_SOURCE;
if (env && env.length > 0) {
if (env === ALL_SOURCES) return ALL_SOURCES;
if (!SOURCE_ID_RE.test(env)) {
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
}
@@ -173,6 +181,7 @@ export function resolveSourceIdEngineFree(
cwd: string = process.cwd(),
): string | null {
if (explicit) {
if (explicit === ALL_SOURCES) return ALL_SOURCES; // #1712 sentinel pass-through
if (!SOURCE_ID_RE.test(explicit)) {
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
}
@@ -180,6 +189,7 @@ export function resolveSourceIdEngineFree(
}
const env = process.env.GBRAIN_SOURCE;
if (env && env.length > 0) {
if (env === ALL_SOURCES) return ALL_SOURCES; // #2140 sentinel pass-through
if (!SOURCE_ID_RE.test(env)) {
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
}
@@ -315,8 +325,11 @@ export async function resolveSourceWithTier(
explicit: string | null | undefined,
cwd: string = process.cwd(),
): Promise<{ source_id: string; tier: SourceTier; detail?: string }> {
// 1. Explicit flag wins.
// 1. Explicit flag wins. __all__ sentinel passes through verbatim (#1712).
if (explicit) {
if (explicit === ALL_SOURCES) {
return { source_id: ALL_SOURCES, tier: 'flag', detail: `--source ${ALL_SOURCES} (spans all sources)` };
}
if (!SOURCE_ID_RE.test(explicit)) {
throw new Error(`Invalid --source value "${explicit}". Must match [a-z0-9-]{1,32}.`);
}
@@ -324,9 +337,12 @@ export async function resolveSourceWithTier(
return { source_id: explicit, tier: 'flag', detail: `--source ${explicit}` };
}
// 2. Env var.
// 2. Env var. Same __all__ pass-through (#2140).
const env = process.env.GBRAIN_SOURCE;
if (env && env.length > 0) {
if (env === ALL_SOURCES) {
return { source_id: ALL_SOURCES, tier: 'env', detail: `GBRAIN_SOURCE=${ALL_SOURCES} (spans all sources)` };
}
if (!SOURCE_ID_RE.test(env)) {
throw new Error(`Invalid GBRAIN_SOURCE value "${env}". Must match [a-z0-9-]{1,32}.`);
}
+13 -8
View File
@@ -11,7 +11,7 @@
* pathToSlug() convert file paths to page slugs
*/
import { CJK_SLUG_CHARS } from './cjk.ts';
import { SLUG_WORD_CHARS } from './cjk.ts';
// v0.37.7.0 #1169 submodule-detection helpers. Bottom-of-file already
// aliases existsSync as `_existsSync` for other purposes; the top-of-file
// import keeps the pruneDir helper's deps near its callsite.
@@ -396,8 +396,10 @@ export function unsyncableReason(path: string, opts: SyncableOptions = {}): Sync
/**
* Character class for the lowercase-canonical form of a slug segment after
* slugifySegment() has run. Lowercase letters, digits, dots, underscores,
* hyphens. Exposed so adjacent code (e.g. takes-fence holder validation,
* slugifySegment() has run. Letters/numbers in any script (lowercase where
* the script has case #3417), dots, underscores, hyphens. Uses \p{...}
* classes, so composed regexes need the `u` flag (this one carries it).
* Exposed so adjacent code (e.g. takes-fence holder validation,
* v0.32 EXP-4) can reuse the actual repo slug grammar instead of inventing
* a stricter parallel one and emitting false-positive warnings on legitimate
* `companies/acme.io` / `people/foo_bar` slugs (codex review #3).
@@ -405,15 +407,18 @@ export function unsyncableReason(path: string, opts: SyncableOptions = {}): Sync
* Pattern is the inner character class only (no anchors); callers wrap it
* in `^...$` or compose it with prefixes like `(?:people|companies)/...`.
*/
export const SLUG_SEGMENT_PATTERN = new RegExp(`[a-z0-9._\\-${CJK_SLUG_CHARS}]+`);
export const SLUG_SEGMENT_PATTERN = new RegExp(`[${SLUG_WORD_CHARS}._\\-]+`, 'u');
/**
* Slugify a single path segment: lowercase, strip special chars, spaces hyphens.
* CJK ranges (Han / Hiragana / Katakana / Hangul Syllables) are preserved (v0.32.7).
* NFC re-normalize after the NFD-strip-accents pass so Hangul Jamo recomposes back
* into precomposed syllables that fall inside the whitelist.
* Letters and numbers from EVERY script are preserved (#3417): previously only
* Latin + CJK survived, so Hebrew/Arabic/Cyrillic/Greek/Thai/... filenames
* collapsed to empty segments and distinct files silently merged onto one slug.
* NFC re-normalize after the NFD-strip-accents pass so Hangul Jamo recomposes
* back into precomposed syllables, and so NFD filenames (macOS) and NFC
* filenames (Linux/git) of the same name produce the SAME slug.
*/
const SLUGIFY_KEEP_RE = new RegExp(`[^a-z0-9.\\s_\\-${CJK_SLUG_CHARS}]`, 'g');
const SLUGIFY_KEEP_RE = new RegExp(`[^${SLUG_WORD_CHARS}.\\s_\\-]`, 'gu');
export function slugifySegment(segment: string): string {
return segment
+1
View File
@@ -134,6 +134,7 @@ export const TAKES_FENCE_END = '<!--- gbrain:takes:end -->';
import { SLUG_SEGMENT_PATTERN } from './sync.ts';
export const HOLDER_REGEX = new RegExp(
`^(?:world|brain|(?:people|companies)/${SLUG_SEGMENT_PATTERN.source}|${SLUG_SEGMENT_PATTERN.source})$`,
'u', // required by SLUG_SEGMENT_PATTERN's \p{...} classes (#3417)
);
/**
+11
View File
@@ -699,6 +699,17 @@ export interface SearchResult {
* Absent when the page is clean.
*/
content_flag?: { reason: string; detail: string };
/**
* Extraction quarantine lane (issue #160): true when the result's page is
* an unverified auto-extracted entity stub (frontmatter
* `provenance: 'auto-extracted'` + `status: 'unverified'`). Such pages are
* excluded from the compiled-truth authority boost and the namespace
* source-boost they rank as ordinary content and this marker tells the
* agent the page has NOT been reviewed by the owner. Stamped pre-fusion by
* `stampUnverifiedExtractions` (hybrid.ts). Absent for reviewed/ordinary
* pages.
*/
unverified?: boolean;
/**
* v0.36 (cross-modal wave): the chunk's modality discriminator from
* content_chunks.modality. 'text' for the existing text-embedding rows,
+7
View File
@@ -110,6 +110,12 @@ export function rowToPage(row: Record<string, unknown>): Page {
const sourceUri = row.source_uri === undefined ? undefined : (row.source_uri as string | null);
const ingestedVia = row.ingested_via === undefined ? undefined : (row.ingested_via as string | null);
const ingestedAt = readOptionalDate(row.ingested_at);
// #3507: the CR tier the page was last embedded under (three-state, same
// pattern as the provenance columns above). Re-embed paths (`embed --stale`
// and friends) read this to reproduce the page's stored wrapping convention.
const contextualRetrievalMode = row.contextual_retrieval_mode === undefined
? undefined
: (row.contextual_retrieval_mode as Page['contextual_retrieval_mode']);
return {
id: row.id as number,
slug: row.slug as string,
@@ -135,6 +141,7 @@ export function rowToPage(row: Record<string, unknown>): Page {
...(sourceUri !== undefined && { source_uri: sourceUri }),
...(ingestedVia !== undefined && { ingested_via: ingestedVia }),
...(ingestedAt !== undefined && { ingested_at: ingestedAt }),
...(contextualRetrievalMode !== undefined && { contextual_retrieval_mode: contextualRetrievalMode }),
// v0.31.12: propagate source_id so downstream callers (embed, reconcile-links)
// can thread it through getChunks / upsertChunks without defaulting to 'default'.
// v0.32.8: Page.source_id is required. Every SELECT feeding rowToPage now
+37
View File
@@ -0,0 +1,37 @@
import { describe, expect, test } from 'bun:test';
import { openAdminSseStream, type AdminSseResponse } from '../src/commands/serve-http.ts';
describe('admin SSE handshake', () => {
test('flushes a protocol-valid comment immediately after the headers', () => {
const calls: string[] = [];
const headers = new Map<string, string>();
openAdminSseStream({
setHeader(name: string, value: string | number | readonly string[]) {
headers.set(name, String(value));
calls.push(`header:${name}`);
return this;
},
flushHeaders() {
calls.push('flush');
},
write(chunk: unknown) {
calls.push(`write:${String(chunk)}`);
return true;
},
} as unknown as AdminSseResponse);
expect(headers).toEqual(new Map([
['Content-Type', 'text/event-stream'],
['Cache-Control', 'no-cache'],
['Connection', 'keep-alive'],
]));
expect(calls).toEqual([
'header:Content-Type',
'header:Cache-Control',
'header:Connection',
'flush',
'write:: connected\n\n',
]);
});
});
+68
View File
@@ -0,0 +1,68 @@
/**
* #3554 resetGateway() must restore the test baseline, not unconfigure.
*
* The bunfig preload (test/helpers/legacy-embedding-preload.ts) pins the
* gateway to openai:text-embedding-3-large @ 1536 at process start and
* registers that config as the reset baseline. Before the fix,
* resetGateway() wiped the pin to _config = null; the next file's beforeAll
* engine-connect then reconfigured from the SHIPPED default (zembed-1 @
* 1280) and every 1536-d fixture in that file failed with
* `expected 1280 dimensions, not 1536`. Which file pairs collided depended
* on shard bin-packing, so adding ANY test file reshuffled the mines.
*
* These assertions pin the contract so it cannot silently rot again.
*/
import { describe, test, expect, afterEach } from 'bun:test';
import {
configureGateway,
resetGateway,
__unconfigureGatewayForTests,
__setChatTransportForTests,
getEmbeddingModel,
getEmbeddingDimensions,
isAvailable,
} from '../../src/core/ai/gateway.ts';
afterEach(() => resetGateway());
describe('resetGateway baseline restore (#3554)', () => {
test('immediately after resetGateway(), the preload baseline is live', () => {
resetGateway();
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getEmbeddingDimensions()).toBe(1536);
});
test('resetGateway() overwrites a file-local config back to the baseline', () => {
configureGateway({
embedding_model: 'zeroentropyai:zembed-1',
embedding_dimensions: 1280,
env: {},
});
expect(getEmbeddingDimensions()).toBe(1280);
resetGateway();
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getEmbeddingDimensions()).toBe(1536);
});
test('resetGateway() still clears test transports (no stale transport leaks back)', () => {
__setChatTransportForTests(async () => {
throw new Error('should have been cleared');
});
resetGateway();
// Baseline config sets no chat key in a keyless env, but the transport
// seam itself must be gone: isAvailable('chat') short-circuits to true
// whenever a chat transport is installed, so with a hard-unconfigured
// gateway it can only be true if the transport survived the reset.
__unconfigureGatewayForTests();
expect(isAvailable('chat')).toBe(false);
});
test('__unconfigureGatewayForTests() gives a genuinely unconfigured gateway', () => {
__unconfigureGatewayForTests();
expect(() => getEmbeddingDimensions()).toThrow(/not configured/);
expect(isAvailable('embedding')).toBe(false);
// And a plain reset brings the baseline back.
resetGateway();
expect(getEmbeddingDimensions()).toBe(1536);
});
});
+6 -1
View File
@@ -2,6 +2,7 @@ import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
import {
configureGateway,
resetGateway,
__unconfigureGatewayForTests,
isAvailable,
embed,
getEmbeddingModel,
@@ -55,6 +56,9 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
beforeEach(() => resetGateway());
test('returns false when gateway not configured', () => {
// resetGateway() restores the preload's test baseline (#3554); go
// genuinely unconfigured for this one assertion.
__unconfigureGatewayForTests();
expect(isAvailable('embedding')).toBe(false);
});
@@ -114,11 +118,12 @@ describe('gateway.isAvailable (silent-drop regression surface)', () => {
// #1135 — an explicit expansion_model pointed at a chat-capable
// OpenAI-compatible provider used to silently yield no expansion because
// the recipe declared no expansion touchpoint.
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together)', () => {
test('expansion available for chat-capable openai-compat providers (deepseek/groq/together/openrouter)', () => {
const cases: Array<[string, Record<string, string>]> = [
['deepseek:deepseek-chat', { DEEPSEEK_API_KEY: 'fake' }],
['groq:llama-3.1-8b-instant', { GROQ_API_KEY: 'fake' }],
['together:meta-llama/Llama-3.3-70B-Instruct-Turbo', { TOGETHER_API_KEY: 'fake' }],
['openrouter:google/gemini-3-flash-preview', { OPENROUTER_API_KEY: 'fake' }],
];
for (const [model, env] of cases) {
resetGateway();
@@ -10,7 +10,7 @@
*/
import { afterAll, beforeAll, describe, expect, mock, test } from 'bun:test';
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
import { capBatchItems, configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
import { listRecipes, getRecipe } from '../../src/core/ai/recipes/index.ts';
describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warning', () => {
@@ -49,6 +49,19 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
expect(r!.touchpoints.embedding?.no_batch_cap).toBeUndefined();
});
test('dashscope declares the documented 10-item embedding cap (max_batch_items: 10)', () => {
// DashScope's OpenAI-compat /embeddings endpoint rejects >10-item batches
// (documented Model Studio cap; concept from community PRs #2643/#2405).
// max_batch_tokens stays as the aggregate token-size guard.
const r = getRecipe('dashscope');
expect(r, 'dashscope not registered').toBeDefined();
expect(r!.touchpoints.embedding?.max_batch_items).toBe(10);
expect(r!.touchpoints.embedding?.max_batch_tokens).toBe(8192);
// 25 items pre-split into DashScope-sized groups of at most 10.
const texts = Array.from({ length: 25 }, (_, i) => `t${i}`);
expect(capBatchItems(texts, 10).map(b => b.length)).toEqual([10, 10, 5]);
});
test('configureGateway does NOT warn for ollama/litellm/llama-server', () => {
warnSpy.mockClear();
resetGateway();
+12
View File
@@ -65,6 +65,18 @@ describe('recipe: openrouter', () => {
).not.toThrow();
});
test('3b. expansion reuses routed chat models and accepts arbitrary provider/model IDs', () => {
const r = getRecipe('openrouter')!;
expect(r.touchpoints.expansion).toBeDefined();
expect(r.touchpoints.expansion!.models.length).toBeGreaterThanOrEqual(3);
expect(() =>
assertTouchpoint(r, 'expansion', 'some/provider-model'),
).not.toThrow();
expect(() =>
assertTouchpoint(r, 'expansion', 'meta-llama/llama-future-2030'),
).not.toThrow();
});
test('4. chat models list — every entry matches provider/model shape (D5 regression)', () => {
// Codex correction: pinning specific slugs creates false confidence (the
// list is advisory; OR's catalog churns). The shape test catches the
+176
View File
@@ -0,0 +1,176 @@
/**
* #1712 (dupes #2289, #2140) the `__all__` sentinel must work in EVERY
* resolution tier, not just as a per-call `source_id` param.
*
* The bug: SOURCE_ID_RE forbids underscores, so `--source __all__` and
* `GBRAIN_SOURCE=__all__` threw in the resolver; the CLI's makeContext
* blanket-caught that and silently fell back to `sourceId: 'default'`
* making the documented span-everything flag STRICTLY NARROWER than passing
* no flag at all (the catch also discarded the #2561/#3242 federated
* widening). Meanwhile sourceScopeOpts treated a ctx.sourceId of '__all__'
* as an unsatisfiable literal.
*
* Uses the literal '__all__' (not the ALL_SOURCES constant) so these tests
* load and run behaviorally against pre-fix trees.
*/
import { describe, test, expect } from 'bun:test';
import { withEnv } from './helpers/with-env.ts';
import {
resolveSourceId,
resolveSourceIdEngineFree,
resolveSourceWithTier,
} from '../src/core/source-resolver.ts';
import {
sourceScopeOpts,
federatedSearchScope,
type OperationContext,
} from '../src/core/operations.ts';
import type { BrainEngine } from '../src/core/engine.ts';
// Stub engine: registered sources + no local_path rows + no default config.
function makeStub(registeredSources: string[]): BrainEngine {
return {
kind: 'pglite',
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
const target = params?.[0];
return registeredSources.includes(target as string)
? [{ id: target } as unknown as T]
: [];
}
return [];
},
getConfig: async () => null,
} as unknown as BrainEngine;
}
function ctxOf(overrides: Partial<OperationContext> = {}): OperationContext {
return {
engine: {} as any,
config: {} as any,
logger: console as any,
dryRun: false,
remote: true,
sourceId: 'default',
...overrides,
};
}
// ── Resolver tiers pass the sentinel through verbatim ──────────────────
describe('source-resolver — __all__ sentinel pass-through', () => {
test('resolveSourceId: explicit --source __all__ resolves (no regex throw, no existence check)', async () => {
// '__all__' is deliberately NOT in the registered set — the sentinel
// must skip assertSourceExists (it is not a source id).
const id = await resolveSourceId(makeStub(['default']), '__all__', '/nonexistent');
expect(id).toBe('__all__');
});
test('resolveSourceId: GBRAIN_SOURCE=__all__ resolves (#2140)', async () => {
await withEnv({ GBRAIN_SOURCE: '__all__' }, async () => {
const id = await resolveSourceId(makeStub(['default']), null, '/nonexistent');
expect(id).toBe('__all__');
});
});
test('resolveSourceIdEngineFree: explicit + env __all__ (thin-client path)', async () => {
expect(resolveSourceIdEngineFree('__all__', '/nonexistent')).toBe('__all__');
await withEnv({ GBRAIN_SOURCE: '__all__' }, () => {
expect(resolveSourceIdEngineFree(null, '/nonexistent')).toBe('__all__');
});
});
test('resolveSourceWithTier: flag and env tiers carry the sentinel', async () => {
const flag = await resolveSourceWithTier(makeStub(['default']), '__all__', '/nonexistent');
expect(flag).toMatchObject({ source_id: '__all__', tier: 'flag' });
await withEnv({ GBRAIN_SOURCE: '__all__' }, async () => {
const env = await resolveSourceWithTier(makeStub(['default']), null, '/nonexistent');
expect(env).toMatchObject({ source_id: '__all__', tier: 'env' });
});
});
test('a genuinely invalid --source still throws (SOURCE_ID_RE not loosened)', async () => {
await expect(resolveSourceId(makeStub(['default']), 'my_source', '/nonexistent'))
.rejects.toThrow(/Invalid --source/);
expect(() => resolveSourceIdEngineFree('my_source', '/nonexistent'))
.toThrow(/Invalid --source/);
});
});
// ── sourceScopeOpts — the single read-scope choke point ─────────────────
describe('sourceScopeOpts — __all__ sentinel', () => {
test('trusted local (remote === false): spans the whole brain (empty scope)', () => {
expect(sourceScopeOpts(ctxOf({ remote: false, sourceId: '__all__' }))).toEqual({});
});
test('remote: keeps the unsatisfiable literal — fail-closed, never widens', () => {
expect(sourceScopeOpts(ctxOf({ remote: true, sourceId: '__all__' })))
.toEqual({ sourceId: '__all__' });
});
test('anything not strictly remote === false is untrusted (fail-closed)', () => {
// undefined / missing remote must behave like remote, per the trust rule.
const ctx = ctxOf({ sourceId: '__all__' });
(ctx as any).remote = undefined;
expect(sourceScopeOpts(ctx)).toEqual({ sourceId: '__all__' });
});
test('a federated grant always wins over the sentinel', () => {
const ctx = ctxOf({
remote: true,
sourceId: '__all__',
auth: { token: 't', clientId: 'c', scopes: [], allowedSources: ['a', 'b'] } as any,
});
expect(sourceScopeOpts(ctx)).toEqual({ sourceIds: ['a', 'b'] });
});
});
// ── Never narrower than passing no flag (#2561 regression shape) ────────
describe('__all__ is never narrower than an unqualified read', () => {
test('local __all__ spans the brain even when federated widening exists', () => {
// Unqualified read on a federated brain widens to the federated array…
const unqualified = ctxOf({
remote: false,
sourceId: 'default',
localFederatedSourceIds: ['default', 'src-a', 'src-b'],
});
expect(federatedSearchScope(unqualified)).toEqual({
sourceIds: ['default', 'src-a', 'src-b'],
});
// …and __all__ must be a superset of that: the whole brain ({}).
const all = ctxOf({ remote: false, sourceId: '__all__' });
expect(federatedSearchScope(all)).toEqual({});
});
});
// ── makeContext — explicit --source failures error loudly ───────────────
describe('cli makeContext — no silent default fallback for explicit --source', () => {
test('--source __all__ produces ctx.sourceId __all__ (was: silent default)', async () => {
const { makeContext } = await import('../src/cli.ts');
const ctx = await makeContext(makeStub(['default']), { source: '__all__' });
expect(ctx.sourceId).toBe('__all__');
expect(ctx.remote).toBe(false);
});
test('an explicit --source that fails to resolve throws instead of becoming default', async () => {
const { makeContext } = await import('../src/cli.ts');
await expect(makeContext(makeStub(['default']), { source: 'ghost' }))
.rejects.toThrow(/not found/);
await expect(makeContext(makeStub(['default']), { source: 'my_source' }))
.rejects.toThrow(/Invalid --source/);
});
test('ambient resolution failure still falls back silently (pre-init brains)', async () => {
const { makeContext } = await import('../src/cli.ts');
const broken = {
kind: 'pglite',
executeRaw: async () => { throw new Error('relation "sources" does not exist'); },
getConfig: async () => { throw new Error('relation "config" does not exist'); },
} as unknown as BrainEngine;
const ctx = await makeContext(broken, {});
expect(ctx.sourceId).toBe('default');
});
});
+48 -10
View File
@@ -11,15 +11,34 @@ import { tmpdir } from 'os';
import { execFileSync } from 'child_process';
import { hardenBrainRepo } from '../src/core/brain-repo-durability.ts';
// #2943 root cause: `env: process.env` is REQUIRED here. Bun snapshots
// process.env at startup, so without it the spawned git — and any post-commit
// hook it fires — is blind to beforeEach's HOME/GBRAIN_HOME mutations (the
// same Bun quirk as #2747, see resolveGbrainCliPath in brain-repo-durability).
// Pre-fix, the hook under test resolved ${GBRAIN_HOME:-$HOME/.gbrain} to the
// OPERATOR'S REAL ~/.gbrain: it wrote its log lines there (polluting the real
// brain-push.log on every run), the LOCAL-ONLY test never saw them in the
// temp log it polls, and the assertion only passed when the scaffolding push
// from beforeEach (spawned by hardenBrainRepo WITH explicit env) happened to
// still be in flight, lose the ref race, and retry AFTER the test had pointed
// origin at the dead path — an accidental, load-dependent signal. That race
// is the CI flake.
function git(cwd: string, ...args: string[]): string {
return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], {
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8',
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', env: process.env,
}).trim();
}
function originHead(bare: string): string {
return git(bare, 'rev-parse', 'refs/heads/main');
}
async function waitForOrigin(bare: string, expectSha: string, ms = 8000): Promise<boolean> {
// #2943: 30s poll deadlines (was 8s) for headroom under loaded CI shards —
// the unreachable-origin path runs ~6 sequential process spawns after the
// hook detaches. Every hook test also passes an explicit 60_000 third-arg
// timeout: bun 1.3.14 IGNORES bunfig.toml's `timeout` key, so a bare
// `bun test` enforces its 5000ms default and killed these tests before the
// internal deadline could even elapse (the runner scripts pass --timeout
// explicitly, which is why the inversion only bit direct local runs).
async function waitForOrigin(bare: string, expectSha: string, ms = 30_000): Promise<boolean> {
const deadline = Date.now() + ms;
while (Date.now() < deadline) {
try { if (originHead(bare) === expectSha) return true; } catch { /* */ }
@@ -28,6 +47,24 @@ async function waitForOrigin(bare: string, expectSha: string, ms = 8000): Promis
return false;
}
/** #2943 (index.lock form): hardenBrainRepo installs the post-commit hook
* BEFORE committing the scaffolding, so that commit fires the hook and
* detaches a background brain_push. If that push loses the ref race against
* hardenBrainRepo's own synchronous push, it falls back to `git pull
* --rebase`, which takes .git/index.lock — racing the test body's first git
* calls ("Unable to create '.../.git/index.lock': File exists"). Wait for the
* detached push's terminal log line before handing the repo to the test. */
async function waitForHookPushSettled(ms = 30_000): Promise<void> {
const log = join(process.env.GBRAIN_HOME!, 'brain-push.log');
const terminal = /\[push\] (ok|lock-timeout|LOCAL-ONLY)/;
const deadline = Date.now() + ms;
while (Date.now() < deadline) {
if (existsSync(log) && terminal.test(readFileSync(log, 'utf-8'))) return;
await new Promise(r => setTimeout(r, 150));
}
throw new Error(`detached hook push did not settle within ${ms}ms (${log})`);
}
let root: string, work: string, bare: string;
let oldHome: string | undefined, oldGbrainHome: string | undefined;
@@ -38,14 +75,15 @@ beforeEach(async () => {
process.env.GBRAIN_HOME = join(process.env.HOME, '.gbrain');
process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1';
bare = mkdtempSync(join(root, 'origin-')) + '.git';
execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore' });
execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore', env: process.env });
work = mkdtempSync(join(root, 'work-'));
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore' });
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore', env: process.env });
git(work, 'config', 'user.email', 't@t.t'); git(work, 'config', 'user.name', 'tester');
writeFileSync(join(work, 'README.md'), 'init\n');
git(work, 'add', 'README.md'); git(work, 'commit', '-qm', 'init'); git(work, 'push', '-q', 'origin', 'main');
git(work, 'remote', 'set-head', 'origin', 'main');
await hardenBrainRepo({ repoPath: work, sourceId: 'wiki', pat: 'ghp_x', installCron: false });
await waitForHookPushSettled();
});
afterEach(() => {
if (oldHome === undefined) delete process.env.HOME; else process.env.HOME = oldHome;
@@ -65,7 +103,7 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
expect(originHead(bare)).toBe(git(work, 'rev-parse', 'HEAD'));
// origin actually has the file
const verify = mkdtempSync(join(root, 'verify-'));
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, verify], { stdio: 'ignore' });
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, verify], { stdio: 'ignore', env: process.env });
expect(existsSync(join(verify, 'people', 'alice.md'))).toBe(true);
});
@@ -102,7 +140,7 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
rmSync(join(work, '.git', 'hooks', 'post-commit'));
// Advance the remote from a second clone so a pull is genuinely needed.
const other = mkdtempSync(join(root, 'other-'));
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' });
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore', env: process.env });
git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other');
writeFileSync(join(other, 'remote.md'), 'from other\n');
git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main');
@@ -128,26 +166,26 @@ describe('post-commit hook (D9 local, D7 self-contained)', () => {
git(work, 'add', 'note.md'); git(work, 'commit', '-qm', 'note'); // fires .git/hooks/post-commit
const head = git(work, 'rev-parse', 'HEAD');
expect(await waitForOrigin(bare, head)).toBe(true);
});
}, 60_000);
test('the hook works even with the committed helper deleted (self-contained)', async () => {
rmSync(join(work, 'scripts', 'brain-commit-push.sh'));
git(work, 'add', '-A'); git(work, 'commit', '-qm', 'remove helper');
const head = git(work, 'rev-parse', 'HEAD');
expect(await waitForOrigin(bare, head)).toBe(true);
});
}, 60_000);
test('logs a clear LOCAL-ONLY line when origin is unreachable', async () => {
git(work, 'remote', 'set-url', 'origin', join(root, 'gone2.git'));
writeFileSync(join(work, 'orphan.md'), 'o\n');
git(work, 'add', 'orphan.md'); git(work, 'commit', '-qm', 'orphan');
const log = join(process.env.GBRAIN_HOME!, 'brain-push.log');
const deadline = Date.now() + 8000;
const deadline = Date.now() + 30_000;
let found = false;
while (Date.now() < deadline) {
if (existsSync(log) && readFileSync(log, 'utf-8').includes('NEEDS ATTENTION')) { found = true; break; }
await new Promise(r => setTimeout(r, 150));
}
expect(found).toBe(true);
});
}, 60_000);
});
+159 -4
View File
@@ -58,12 +58,23 @@ beforeEach(async () => {
// Tiny gazetteer builder for pure-fn cases that don't need engine.
function gazetteerFromEntries(entries: Omit<GazetteerEntry, 'tokens'>[]): Gazetteer {
const TOKEN_RE = /[a-zA-Z0-9]+/g;
const isCJK = (s: string): boolean => {
const cp = s.codePointAt(0) ?? 0;
return (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf) ||
(cp >= 0x3040 && cp <= 0x309f) || (cp >= 0x30a0 && cp <= 0x30ff) ||
(cp >= 0xac00 && cp <= 0xd7af);
};
const hasCJKTitle = (s: string): boolean => [...s].some(isCJK);
const tokenize = (s: string): string[] => {
TOKEN_RE.lastIndex = 0;
const out: string[] = [];
let m: RegExpExecArray | null;
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
return out;
if (!hasCJKTitle(s)) {
const out: string[] = [];
let m: RegExpExecArray | null;
while ((m = TOKEN_RE.exec(s)) !== null) out.push(m[0].toLowerCase());
return out;
}
// CJK: split into individual characters, lowercased.
return [...s].map(c => isCJK(c) ? c.toLowerCase() : '').filter(Boolean);
};
const g: Gazetteer = new Map();
for (const raw of entries) {
@@ -259,6 +270,128 @@ describe('findMentionedEntities — pure cases', () => {
});
});
// ============================================================
// CJK — entity extraction tests
// ============================================================
describe('findMentionedEntities — CJK cases', () => {
test('CJK single-name match — "纳瓦尔" in body → matched', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
]);
const mentions = findMentionedEntities('我最近读了纳瓦尔的书。', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
expect(mentions[0]!.slug).toBe('people/naval');
expect(mentions[0]!.name).toBe('纳瓦尔');
});
test('CJK multi-name — two different CJK entities in one body', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
{ slug: 'people/shuang-xuetao', source_id: 'default', title: '双雪涛' },
]);
const mentions = findMentionedEntities('纳瓦尔和双雪涛都是作家。', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(2);
const slugs = mentions.map(m => m.slug);
expect(slugs).toContain('people/naval');
expect(slugs).toContain('people/shuang-xuetao');
});
test('CJK first-mention-only — repeated name → single link', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
]);
const mentions = findMentionedEntities('纳瓦尔说过。然后纳瓦尔又说过。', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(1);
});
test('CJK self-link guard — entity page mentioning itself is skipped', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
]);
const mentions = findMentionedEntities('纳瓦尔是一位投资人。', g, {
fromSlug: 'people/naval', fromSourceId: 'default',
});
expect(mentions).toEqual([]);
});
test('CJK cross-source guard — entity in different source skipped', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'team-b', title: '纳瓦尔' },
]);
const mentions = findMentionedEntities('纳瓦尔写了这本书。', g, {
fromSlug: 'writing/post-1', fromSourceId: 'team-a',
});
expect(mentions).toEqual([]);
});
test('CJK code-block stripping — CJK name inside ``` is skipped, outside matched', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
]);
// "纳瓦尔" only appears inside code block → should be skipped.
const body = '```\n纳瓦尔\n```\n只有代码块里面有。';
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(0);
});
test('CJK determinism — same output across 10 calls', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
{ slug: 'people/shuang-xuetao', source_id: 'default', title: '双雪涛' },
]);
const body = '纳瓦尔和双雪涛。纳瓦尔再说一次。';
const refs = new Set<string>();
for (let i = 0; i < 10; i++) {
const mentions = findMentionedEntities(body, g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
refs.add(JSON.stringify(mentions));
}
expect(refs.size).toBe(1);
});
test('CJK mixed body — CJK entity matched in body with ASCII around it', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
{ slug: 'companies/acme', source_id: 'default', title: 'Acme' },
]);
const mentions = findMentionedEntities('Acme was founded by 纳瓦尔 in 2020.', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toHaveLength(2);
const slugs = mentions.map(m => m.slug);
expect(slugs).toContain('people/naval');
expect(slugs).toContain('companies/acme');
});
test('CJK empty gazetteer — no false positives', () => {
const g: Gazetteer = new Map();
const mentions = findMentionedEntities('纳瓦尔和双雪涛。', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toEqual([]);
});
test('CJK empty text → empty result', () => {
const g = gazetteerFromEntries([
{ slug: 'people/naval', source_id: 'default', title: '纳瓦尔' },
]);
const mentions = findMentionedEntities('', g, {
fromSlug: 'writing/post-1', fromSourceId: 'default',
});
expect(mentions).toEqual([]);
});
});
// ============================================================
// buildGazetteer — engine-backed tests
// ============================================================
@@ -366,4 +499,26 @@ describe('buildGazetteer — engine integration', () => {
// forces a deliberate change (and a corresponding test update).
expect(LINKABLE_ENTITY_TYPES).toEqual(['person', 'company', 'organization', 'entity']);
});
// CJK — engine-backed tests
test('CJK entity with 2-char title enters gazetteer with char-level tokens', async () => {
await engine.putPage('people/naval', {
type: 'person', title: '纳瓦尔', compiled_truth: 'b', timeline: '', frontmatter: {},
});
const g = await buildGazetteer(engine);
// "纳瓦尔" tokenized as ["纳","瓦","尔"] → key is "纳"
expect(g.has('纳')).toBe(true);
const bucket = g.get('纳')!;
expect(bucket.length).toBe(1);
expect(bucket[0]!.tokens).toEqual(['纳', '瓦', '尔']);
expect(bucket[0]!.slug).toBe('people/naval');
});
test('CJK single-char title (cjkCharCount < 2) excluded from gazetteer', async () => {
await engine.putPage('people/x', {
type: 'person', title: '谢', compiled_truth: 'b', timeline: '', frontmatter: {},
});
const g = await buildGazetteer(engine);
expect(g.size).toBe(0);
});
});
+133 -25
View File
@@ -1,8 +1,10 @@
/**
* Serial (stubs globalThis.fetch): exercises the self-upgrade cache REFRESH
* orchestration end-to-end `refreshUpdateCache()` fetches the latest release
* and writes the correct marker to the shared cache file that the CLI startup
* hook reads. Network is stubbed; the cache write + marker logic are real.
* orchestration end-to-end `refreshUpdateCache()` resolves the latest version
* (from the VERSION file on master, #486 the repo has zero GitHub releases,
* so the old `releases/latest` API path could never succeed) and writes the
* correct marker to the shared cache file that the CLI startup hook reads.
* Network is stubbed; the cache write + marker logic are real.
*
* Quarantined as *.serial.test.ts because it reassigns the process-global
* `fetch` (cross-file-unsafe under the parallel runner).
@@ -13,10 +15,11 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { VERSION } from '../src/version.ts';
import { parseSemver } from '../src/core/semver.ts';
import { readUpdateCache } from '../src/core/self-upgrade.ts';
import { refreshUpdateCache } from '../src/commands/check-update.ts';
import { readUpdateCache, writeUpdateCache } from '../src/core/self-upgrade.ts';
import { fetchLatestRelease, parseVersionFileBody, refreshUpdateCache, runCheckUpdate } from '../src/commands/check-update.ts';
const realFetch = globalThis.fetch;
const realLog = console.log;
let homeDir: string;
let priorHome: string | undefined;
@@ -27,14 +30,13 @@ function bump(kind: 'minor' | 'patch' | 'micro'): string {
return `${v[0]}.${v[1]}.${v[2]}.${v[3] + 1}`;
}
function stubReleaseFetch(tag: string | null, ok = true): void {
/** Stub the VERSION-file fetch. body === null → network throw. */
function stubVersionFetch(body: string | null, status = 200): void {
globalThis.fetch = (async (url: any) => {
const u = String(url);
if (u.includes('/releases/latest')) {
if (tag === null) throw new Error('network down');
return new Response(JSON.stringify({ tag_name: tag, published_at: '2026-01-01T00:00:00Z', html_url: 'https://x' }), {
status: ok ? 200 : 500,
});
if (u.includes('/gbrain/master/VERSION')) {
if (body === null) throw new Error('network down');
return new Response(body, { status });
}
// Changelog fetch (only happens when update available) — return empty.
return new Response('', { status: 200 });
@@ -49,49 +51,155 @@ beforeEach(() => {
afterEach(() => {
globalThis.fetch = realFetch;
console.log = realLog;
if (priorHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = priorHome;
rmSync(homeDir, { recursive: true, force: true });
});
describe('fetchLatestRelease — resolves from the VERSION file, discriminates failures', () => {
test('bare version body → ok with that tag', async () => {
stubVersionFetch('0.99.1.0\n');
expect(await fetchLatestRelease()).toMatchObject({ ok: true, tag: '0.99.1.0' });
});
test('network throw → network_error (NOT no_releases — offline users are not told "no releases exist")', async () => {
stubVersionFetch(null);
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'network_error' });
});
test('HTTP 404 → no_releases', async () => {
stubVersionFetch('Not Found', 404);
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
});
test('garbage body → no_releases', async () => {
stubVersionFetch('<html>rate limited</html>');
expect(await fetchLatestRelease()).toEqual({ ok: false, reason: 'no_releases' });
});
});
describe('parseVersionFileBody — shape gate over the raw fetch body', () => {
test('trailing newline, v prefix, 3-segment legacy, suffix channel', () => {
expect(parseVersionFileBody('0.42.67.0\n')).toBe('0.42.67.0');
expect(parseVersionFileBody('v0.42.67.0')).toBe('0.42.67.0');
expect(parseVersionFileBody('0.31.3\n')).toBe('0.31.3'); // legacy 3-segment
expect(parseVersionFileBody('0.31.1.1-fixwave\n')).toBe('0.31.1.1'); // suffix compares as base
});
test('malformed / huge / injected bodies → null', () => {
expect(parseVersionFileBody('')).toBeNull();
expect(parseVersionFileBody('not a version')).toBeNull();
expect(parseVersionFileBody('$(rm -rf /)')).toBeNull();
expect(parseVersionFileBody('1.2')).toBeNull(); // 2-segment: not a gbrain version
expect(parseVersionFileBody('9'.repeat(10_000_000))).toBeNull(); // bounded, no blowup
});
});
describe('refreshUpdateCache — full refresh orchestration (network stubbed)', () => {
test('minor-bump release → writes upgrade_available marker', async () => {
test('minor-bump VERSION on master → writes upgrade_available marker', async () => {
const latest = bump('minor');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
const entry = readUpdateCache();
expect(entry?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('patch release → writes upgrade_available marker', async () => {
test('patch bump → writes upgrade_available marker', async () => {
const latest = bump('patch');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('micro release → writes upgrade_available marker', async () => {
test('micro bump → writes upgrade_available marker', async () => {
const latest = bump('micro');
stubReleaseFetch(`v${latest}`);
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('network failure → writes up_to_date marker (fail-open, TTL prevents hammering)', async () => {
stubReleaseFetch(null);
test('minor bump published as legacy 3-segment → still detected', async () => {
const v = parseSemver(VERSION)!;
const latest = `${v[0]}.${v[1] + 1}.0`; // 3-segment, no micro
stubVersionFetch(`${latest}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('suffix channel release (X.Y.Z.W-fixwave) → compares as numeric base', async () => {
const latest = bump('micro');
stubVersionFetch(`${latest}-fixwave\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('same version on master → up_to_date marker', async () => {
stubVersionFetch(`${VERSION}\n`);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
});
test('non-OK HTTP → fail-open up_to_date', async () => {
stubReleaseFetch(`v${bump('minor')}`, false);
// The #486 bug class: a failed check must never fabricate "you're current".
test('network failure with NO prior cache → writes NOTHING (never a fabricated up_to_date)', async () => {
stubVersionFetch(null);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
expect(readUpdateCache()).toBeNull();
});
test('garbage tag → fail-open up_to_date (forged/invalid version never cached as upgrade)', async () => {
stubReleaseFetch('v$(rm -rf /)');
test('network failure with prior upgrade_available → pending notice PRESERVED, not erased', async () => {
const latest = bump('minor');
writeUpdateCache({ kind: 'upgrade_available', current: VERSION, latest });
stubVersionFetch(null);
await refreshUpdateCache();
expect(readUpdateCache()?.marker).toEqual({ kind: 'up_to_date', current: VERSION });
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
test('non-OK HTTP → no fabricated up_to_date', async () => {
stubVersionFetch('nope', 500);
await refreshUpdateCache();
expect(readUpdateCache()).toBeNull();
});
test('garbage body → no fabricated marker (forged/invalid version never cached as upgrade)', async () => {
stubVersionFetch('$(rm -rf /)');
await refreshUpdateCache();
expect(readUpdateCache()).toBeNull();
});
});
describe('runCheckUpdate --json — failure discrimination (#486)', () => {
function capture(): string[] {
const lines: string[] = [];
console.log = (...a: unknown[]) => { lines.push(a.join(' ')); };
return lines;
}
test('offline → error: network_error (not "no releases exist")', async () => {
stubVersionFetch(null);
const lines = capture();
await runCheckUpdate(['--json']);
const out = JSON.parse(lines.join('\n'));
expect(out.error).toBe('network_error');
expect(out.update_available).toBe(false);
expect(readUpdateCache()).toBeNull(); // and no fabricated up_to_date cache
});
test('endpoint answers but no usable version → error: no_releases', async () => {
stubVersionFetch('garbage');
const lines = capture();
await runCheckUpdate(['--json']);
expect(JSON.parse(lines.join('\n')).error).toBe('no_releases');
});
test('newer VERSION on master → update_available true with latest_version set', async () => {
const latest = bump('minor');
stubVersionFetch(`${latest}\n`);
const lines = capture();
await runCheckUpdate(['--json']);
const out = JSON.parse(lines.join('\n'));
expect(out.update_available).toBe(true);
expect(out.latest_version).toBe(latest);
expect(readUpdateCache()?.marker).toEqual({ kind: 'upgrade_available', current: VERSION, latest });
});
});
+7
View File
@@ -35,6 +35,13 @@ describe('isNewerVersion', () => {
expect(isNewerVersion('0.42.66.0', '0.42.66.1')).toBe(true);
});
test('orders legacy 3-segment against 4-segment: 0.42.67.0 > 0.42.66.1 > 0.42.66', () => {
expect(isNewerVersion('0.42.66.1', '0.42.67.0')).toBe(true);
expect(isNewerVersion('0.42.66', '0.42.66.1')).toBe(true);
expect(isNewerVersion('0.42.66', '0.42.66.0')).toBe(false); // 3-segment == its .0 micro
expect(isNewerVersion('0.42.67.0', '0.42.66.1')).toBe(false);
});
test('rejects equal, older, and malformed versions', () => {
expect(isNewerVersion('0.42.66.0', '0.42.66.0')).toBe(false);
expect(isNewerVersion('0.42.66.1', '0.42.66.0')).toBe(false);
+97
View File
@@ -0,0 +1,97 @@
/**
* capOversizedChunks CJK-aware oversize measurement (follow-up to #1675,
* shape requested in #3475's closing review).
*
* cl100k (estimateTokens) matches embedding-family tokenizers on pure-ASCII
* source (measured identical on English prose and JSON vs Qwen3-Embedding),
* but undercounts MIXED CJK+ASCII chunks measured 31% on URL-dense Korean
* text (#2826's failure shape). estimateEmbedTokens lifts only that class:
* ASCII-only input short-circuits to estimateTokens verbatim, and max()
* keeps CJK-dominant text at the cl100k count it gets today.
*/
import { describe, test, expect } from 'bun:test';
import { chunkCodeText, estimateTokens, estimateEmbedTokens } from '../../src/core/chunkers/code.ts';
/** URL-dense Korean rollup lines — the measured 31% divergence shape. */
function urlDenseKoreanMix(lines: number): string {
return Array.from({ length: lines }, (_, i) =>
`- 항목 ${i}: 검증용 한국어 문장 · 링크: https://docs.example.com/pages/${String(i).padStart(32, '0')}?v=abcdef0123456789&ref=sample`,
).join('\n');
}
function bigJsonWithKoreanValues(targetChars: number): string {
const entries: string[] = [];
let i = 0;
let len = 0;
while (len < targetChars) {
const row =
` "item_${i}": { "name": "예시-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100}, "memo": "한국어 값이 섞인 예시 데이터" }`;
entries.push(row);
len += row.length;
i++;
}
return `{\n${entries.join(',\n')}\n}`;
}
describe('estimateEmbedTokens — measurement gate', () => {
test('ASCII-only input is bit-identical to estimateTokens (no CJK → short-circuit)', () => {
const en = 'function ordinary() { return compute(42) + helper(); } '.repeat(80);
const json = '{"item": {"name": "sample", "url": "https://example.com/a?b=c", "qty": 42}}, '.repeat(60);
expect(estimateEmbedTokens(en)).toBe(estimateTokens(en));
expect(estimateEmbedTokens(json)).toBe(estimateTokens(json));
});
test('never estimates below estimateTokens (max composition)', () => {
for (const s of [urlDenseKoreanMix(20), '이 문장은 순수 한국어 산문 예시입니다. '.repeat(40), 'plain ascii ', '']) {
expect(estimateEmbedTokens(s)).toBeGreaterThanOrEqual(estimateTokens(s));
}
});
test('mixed CJK+ASCII (the measured divergence class) estimates strictly higher', () => {
const mix = urlDenseKoreanMix(20);
// Real Qwen3-Embedding count for this shape measures ~45% ABOVE cl100k;
// the weighted form stays above the real count (+15% measured margin).
expect(estimateEmbedTokens(mix)).toBeGreaterThan(estimateTokens(mix));
});
});
describe('capOversizedChunks with the CJK-aware estimate', () => {
test('oversized json fence with Korean values re-splits under the default cap', async () => {
const src = bigJsonWithKoreanValues(14_000);
const chunks = await chunkCodeText(src, 'fence.json');
expect(chunks.length).toBeGreaterThan(1);
for (const c of chunks) {
// Small slack for the "[JSON] fence.json:…" header buildChunk re-adds
// after the body-level split.
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
}
// Content preserved — spot-check first / last entries survive.
const joined = chunks.map((c) => c.text).join('\n');
expect(joined).toContain('"item_0"');
expect(joined).toContain('한국어 값이 섞인 예시 데이터');
});
test('hard-split fallback makes progress on whitespace-less CJK-mixed input and stays under cap', async () => {
const blob = '한a민b국c'.repeat(3_000); // 18K chars, no whitespace
const chunks = await chunkCodeText(`{"blob": "${blob}"}`, 'fence.json');
expect(chunks.length).toBeGreaterThan(1);
for (const c of chunks) {
expect(estimateEmbedTokens(c.text)).toBeLessThanOrEqual(2000 + 60);
}
});
test('pure-ASCII chunks are measured by the identical estimator (cap decisions unchanged)', async () => {
const entries = Array.from({ length: 120 }, (_, i) =>
` "item_${i}": { "name": "sample-${i}", "url": "https://example.com/api/v2/items/${i}?token=abc${i}def", "qty": ${i % 100} }`,
);
const src = `{\n${entries.join(',\n')}\n}`;
const chunks = await chunkCodeText(src, 'fence.json');
expect(chunks.length).toBeGreaterThan(0);
for (const c of chunks) {
// For ASCII-only chunks the two estimators are identical (pinned
// above), so cap decisions — and therefore boundaries — are unchanged.
expect(estimateEmbedTokens(c.text)).toBe(estimateTokens(c.text));
}
});
});
+173
View File
@@ -0,0 +1,173 @@
/**
* #3513: parseOpArgs' stdin read must never block forever.
*
* In a non-TTY with no piped input a CI step, a cron job, an agent
* harness that inherits a non-TTY stdin without writing to it the old
* inline `readFileSync(0)` never returned. The fix bounds the read with a
* first-byte deadline (pipes/sockets only) and falls through to the
* existing required-param usage error on timeout.
*
* The load-bearing regression test spawns the REAL CLI with a held-open,
* never-written pipe: on pre-fix code it hangs until our observation window
* kills it; on fixed code it exits 1 with the usage error well inside the
* window. The stdin read + required-param check both run BEFORE engine
* connect, so no brain/DB is touched.
*/
import { describe, expect, test } from 'bun:test';
import { dirname, join } from 'path';
const REPO = dirname(import.meta.dir);
const CLI = join(REPO, 'src', 'cli.ts');
interface CliRun {
exited: boolean;
exitCode: number | null;
stderr: string;
}
/** Narrow Bun's `number | FileSink` stdin union to the pipe sink. */
function pipeSink(proc: { stdin: unknown }): { write(d: string): unknown; end(): unknown } {
const s = proc.stdin;
if (!s || typeof s === 'number') throw new Error('expected a piped stdin sink');
return s as { write(d: string): unknown; end(): unknown };
}
/**
* Spawn the CLI with the given stdin wiring. `holdPipeOpen` keeps the write
* end of the stdin pipe alive without ever writing the #3513 repro. The
* observation window kills the child if it hasn't exited (pre-fix hang).
*/
async function runCliWithStdin(
args: string[],
stdin: 'hold-open' | 'closed-empty' | { data: string } | { file: string },
windowMs: number,
): Promise<CliRun> {
const proc = Bun.spawn(['bun', 'run', CLI, ...args], {
cwd: REPO,
env: { ...process.env, GBRAIN_STDIN_TIMEOUT_MS: '500' },
stdin: typeof stdin === 'object' && 'file' in stdin ? Bun.file(stdin.file) : 'pipe',
stdout: 'pipe',
stderr: 'pipe',
});
if (typeof stdin === 'object' && 'data' in stdin) {
pipeSink(proc).write(stdin.data);
await pipeSink(proc).end();
} else if (stdin === 'closed-empty') {
await pipeSink(proc).end();
}
// 'hold-open': never write, never close — the CI/cron/agent-harness shape.
let exited = true;
const killer = setTimeout(() => {
exited = false;
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
}, windowMs);
const [exitCode, stderr] = await Promise.all([
proc.exited,
new Response(proc.stderr).text(),
]);
clearTimeout(killer);
try { pipeSink(proc).end(); } catch { /* hold-open cleanup */ }
return { exited, exitCode: exited ? exitCode : null, stderr };
}
describe('#3513 — stdin-capable op with a non-TTY, never-written stdin', () => {
test('exits fast with the usage error instead of blocking forever', async () => {
// `put` declares stdin:'content' (required). No inline content, no piped
// input → the bounded read times out at 500ms, content stays unset, and
// the required-param check prints usage and exits 1. Pre-fix: readFileSync(0)
// blocks until the 20s window kills the child.
const run = await runCliWithStdin(['put', 'stdin-hang-test-slug'], 'hold-open', 20_000);
expect(run.exited).toBe(true); // pre-#3513 this is false: the read never returns
expect(run.exitCode).toBe(1);
expect(run.stderr).toContain('Usage: gbrain put');
}, 30_000);
test('a genuine pipe with data is still consumed (no hang, no crash)', async () => {
// Piped content fills `content`; the missing positional slug then fails
// the required check — proving the stream path read stdin and moved on.
const run = await runCliWithStdin(['put'], { data: '# hello\n' }, 20_000);
expect(run.exited).toBe(true);
expect(run.exitCode).toBe(1);
expect(run.stderr).toContain('Usage: gbrain put');
}, 30_000);
test('empty-but-real input (`< /dev/null`) does not hang', async () => {
const run = await runCliWithStdin(['put'], { file: '/dev/null' }, 20_000);
expect(run.exited).toBe(true);
expect(run.exitCode).toBe(1);
}, 30_000);
test('an empty pipe that closes immediately does not hang', async () => {
const run = await runCliWithStdin(['put'], 'closed-empty', 20_000);
expect(run.exited).toBe(true);
expect(run.exitCode).toBe(1);
}, 30_000);
});
describe('#3513 — applyStdinParam content preservation (subprocess driver)', () => {
// Drive the exported helper in a child process so we control the child's
// real fd 0 — bun test's own stdin is not a reliable fixture.
const DRIVER = `
const { applyStdinParam } = await import(${JSON.stringify(CLI)});
const op = { name: 'put', params: { content: { type: 'string', required: true } }, cliHints: { stdin: 'content' } };
const params = {};
await applyStdinParam(op, params);
console.log(JSON.stringify(params));
process.exit(0);
`;
async function runDriver(
stdin: 'hold-open' | 'closed-empty' | { data: string } | { file: string },
): Promise<{ exited: boolean; params: Record<string, unknown> | null }> {
const proc = Bun.spawn(['bun', '-e', DRIVER], {
cwd: REPO,
env: { ...process.env, GBRAIN_STDIN_TIMEOUT_MS: '500' },
stdin: typeof stdin === 'object' && 'file' in stdin ? Bun.file(stdin.file) : 'pipe',
stdout: 'pipe',
stderr: 'pipe',
});
if (typeof stdin === 'object' && 'data' in stdin) {
pipeSink(proc).write(stdin.data);
await pipeSink(proc).end();
} else if (stdin === 'closed-empty') {
await pipeSink(proc).end();
}
let exited = true;
const killer = setTimeout(() => {
exited = false;
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
}, 20_000);
const [stdout] = await Promise.all([new Response(proc.stdout).text(), proc.exited]);
clearTimeout(killer);
try { pipeSink(proc).end(); } catch { /* hold-open cleanup */ }
const line = stdout.trim().split('\n').pop() ?? '';
let params: Record<string, unknown> | null = null;
try { params = JSON.parse(line); } catch { /* child killed before printing */ }
return { exited, params };
}
test('piped data lands in the stdin param verbatim', async () => {
const { exited, params } = await runDriver({ data: '---\ntitle: x\n---\nbody' });
expect(exited).toBe(true);
expect(params?.content).toBe('---\ntitle: x\n---\nbody');
}, 30_000);
test('/dev/null yields empty-string content (readable, empty — pre-fix parity)', async () => {
const { exited, params } = await runDriver({ file: '/dev/null' });
expect(exited).toBe(true);
expect(params?.content).toBe('');
}, 30_000);
test('empty closed pipe yields empty-string content', async () => {
const { exited, params } = await runDriver('closed-empty');
expect(exited).toBe(true);
expect(params?.content).toBe('');
}, 30_000);
test('held-open pipe times out and leaves the param unset', async () => {
const { exited, params } = await runDriver('hold-open');
expect(exited).toBe(true); // completes inside the window instead of hanging
expect(params).toEqual({});
}, 30_000);
});
@@ -0,0 +1,81 @@
import { describe, test, expect } from 'bun:test';
import {
normalizeBlockConversation,
looksLikeBlockConversation,
} from '../src/core/conversation-parser/normalize-block.ts';
import { parseConversation } from '../src/core/conversation-parser/parse.ts';
// A realistic Slack-collector page body (the format gbrain's own collector emits).
const SLACK_DM = `# DM (group) with Hugh, Karyshma, Theo — 2026-06-15
- **Theo** (Mon 11:18)
Hey everyone quick note on the *real fiscal value* we surface after accounting.
It's a huge win at renewal and dents churn.
- **Juan** (Mon 11:20)
Agreed. Let's make sure we capture it for all ongoing customers.`;
describe('looksLikeBlockConversation', () => {
test('detects the block header signature', () => {
expect(looksLikeBlockConversation(SLACK_DM)).toBe(true);
expect(looksLikeBlockConversation('- **Theo** (16:36)\n body')).toBe(true);
expect(looksLikeBlockConversation('- **Theo** (11:18 AM)\n body')).toBe(true);
});
test('is false for canonical single-line content (no false trigger)', () => {
expect(looksLikeBlockConversation('**Theo** (11:18): hi there')).toBe(false);
expect(looksLikeBlockConversation('**Theo** (2026-06-15 11:18): hi')).toBe(false);
expect(looksLikeBlockConversation('just some prose with no chat at all')).toBe(false);
});
});
describe('normalizeBlockConversation', () => {
test('collapses header + indented multi-paragraph body to one canonical line', () => {
const out = normalizeBlockConversation(SLACK_DM).split('\n');
expect(out).toEqual([
"**Theo** (11:18): Hey everyone — quick note on the *real fiscal value* we surface after accounting. It's a huge win at renewal and dents churn.",
"**Juan** (11:20): Agreed. Let's make sure we capture it for all ongoing customers.",
]);
});
test('drops the page-title line and leading blanks', () => {
const out = normalizeBlockConversation(SLACK_DM);
expect(out.startsWith('# DM')).toBe(false);
expect(out.startsWith('**Theo**')).toBe(true);
});
test('converts 12h am/pm to 24h', () => {
expect(normalizeBlockConversation('- **A** (1:05 PM)\n hi')).toBe('**A** (13:05): hi');
expect(normalizeBlockConversation('- **A** (12:00 AM)\n midnight')).toBe('**A** (00:00): midnight');
expect(normalizeBlockConversation('- **A** (12:30 PM)\n noon-ish')).toBe('**A** (12:30): noon-ish');
});
test('keeps day-of-week out of the emitted time', () => {
expect(normalizeBlockConversation('- **A** (Tue 09:07)\n morning')).toBe('**A** (09:07): morning');
});
test('is a strict no-op on canonical single-line content', () => {
const canonical = '**Theo** (11:18): hi\n**Juan** (11:20): yo';
expect(normalizeBlockConversation(canonical)).toBe(canonical);
});
test('a message with no body emits an empty-body line', () => {
expect(normalizeBlockConversation('- **A** (10:00)')).toBe('**A** (10:00): ');
});
});
describe('parseConversation integration — block format now yields messages', () => {
test('Slack-collector body parses to 2 messages via the normalize pre-pass', () => {
const res = parseConversation(SLACK_DM, { fallbackDate: '2026-06-15' });
expect(res.messages.length).toBe(2);
expect(res.messages[0].speaker).toBe('Theo');
expect(res.messages[1].speaker).toBe('Juan');
expect(res.phase).not.toBe('no_match');
});
test('canonical content still parses unchanged (no regression)', () => {
const res = parseConversation('**Theo** (11:18): hi there', { fallbackDate: '2026-06-15' });
expect(res.messages.length).toBe(1);
expect(res.messages[0].speaker).toBe('Theo');
});
});
+58
View File
@@ -12,6 +12,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
isSourceUnchangedSinceSync,
probeSourceGitState,
_setGitHeadProbeForTests,
_setGitCleanProbeForTests,
type GitHeadProbe,
@@ -176,3 +177,60 @@ describe('isSourceUnchangedSinceSync — requireCleanWorkingTree (D7)', () => {
expect(cleanCalls).toBe(0);
});
});
describe('probeSourceGitState — three-state verdict', () => {
test('state 1: HEAD matches + clean → unchanged', () => {
_setGitHeadProbeForTests(() => 'abc123');
_setGitCleanProbeForTests(() => true);
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: 'ignore-untracked' }))
.toBe('unchanged');
});
test('state 2: HEAD probe null (clone missing / not a repo / git error) → unavailable', () => {
_setGitHeadProbeForTests(() => null);
expect(probeSourceGitState('/tmp/gone', 'abc123')).toBe('unavailable');
});
test('state 3: HEAD mismatch → changed', () => {
_setGitHeadProbeForTests(() => 'def456');
expect(probeSourceGitState('/tmp/repo', 'abc123')).toBe('changed');
});
test('state 4: dirty tree with readable HEAD → changed (NOT unavailable)', () => {
_setGitHeadProbeForTests(() => 'abc123');
_setGitCleanProbeForTests(() => false);
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: true }))
.toBe('changed');
});
test('state 5: clean-probe ERROR with readable HEAD → changed (fail toward work)', () => {
_setGitHeadProbeForTests(() => 'abc123');
_setGitCleanProbeForTests(() => null);
expect(probeSourceGitState('/tmp/repo', 'abc123', { requireCleanWorkingTree: true }))
.toBe('changed');
});
test('state 6: NULL inputs → changed, head probe never called (case-4 contract)', () => {
let probeCalls = 0;
_setGitHeadProbeForTests(() => { probeCalls++; return 'abc'; });
expect(probeSourceGitState(null, 'abc')).toBe('changed');
expect(probeSourceGitState('/tmp/repo', null)).toBe('changed');
expect(probeSourceGitState('', '')).toBe('changed');
expect(probeCalls).toBe(0);
});
test('state 7: boolean façade parity — isSourceUnchangedSinceSync === (state is unchanged)', () => {
_setGitHeadProbeForTests(() => 'abc123');
_setGitCleanProbeForTests(() => true);
for (const [path, commit] of [
['/tmp/repo', 'abc123'], // unchanged → true
['/tmp/repo', 'other'], // changed → false
] as const) {
expect(isSourceUnchangedSinceSync(path, commit))
.toBe(probeSourceGitState(path, commit) === 'unchanged');
}
_setGitHeadProbeForTests(() => null); // unavailable → false
expect(isSourceUnchangedSinceSync('/tmp/gone', 'abc123'))
.toBe(probeSourceGitState('/tmp/gone', 'abc123') === 'unchanged');
});
});
+3 -2
View File
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
return resolveSearchMode({ mode: 'balanced' });
}
test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => {
test('KNOBS_HASH_VERSION is 14 (cross-modal still appended; 13→14 compiled_truth boost scope #3430)', () => {
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
@@ -146,7 +146,8 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
// finally reaches asymmetric providers — pre-fix rows were keyed on
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
expect(KNOBS_HASH_VERSION).toBe(12);
// #3430: 13→14 compiled_truth boost no longer applies at detail=medium.
expect(KNOBS_HASH_VERSION).toBe(14);
});
test('flipping unified_multimodal changes the hash', () => {
+14 -8
View File
@@ -38,7 +38,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv, emptyHome } from './helpers/with-env.ts';
import { runCycle, ALL_PHASES } from '../src/core/cycle.ts';
import { mkdtempSync, writeFileSync } from 'fs';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
@@ -139,19 +139,25 @@ describe('#2540 (i) — pack omitting optional phases, all enabled phases comple
describe('#2540 (ii) — an enabled phase that never completes still prevents the stamp', () => {
test('every selected phase failing reports status=failed and does NOT stamp last_full_cycle_at', async () => {
await withEnv({ GBRAIN_HOME: gbrainHome, OPENAI_API_KEY: undefined, ANTHROPIC_API_KEY: undefined }, async () => {
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
await seedSource('always-fails');
expect(await readLastFullCycleAt('always-fails')).toBeNull();
// embed is a real, always-enabled phase (no pack gate, no config
// .enabled toggle). With no embedding provider key configured it
// deterministically fails — this is NOT the fix under test, it's
// the pre-existing "an enabled phase genuinely never completes"
// case the issue says must keep failing doctor's check.
// Deterministic, environment-independent failure: run the sync phase
// against a brain directory that no longer exists. The previous shape
// ('embed' with OPENAI_API_KEY/ANTHROPIC_API_KEY unset) was
// environment-sensitive — on a machine where any OTHER embedding
// provider resolves (Voyage, ZeroEntropy, a local endpoint, …), embed
// with zero stale chunks succeeds and the cycle reports 'clean',
// flipping this test's expectation. A vanished checkout fails the
// sync phase on every machine. This is NOT the fix under test; it's
// the pre-existing "an enabled phase genuinely never completes" case
// the issue says must keep failing doctor's check.
rmSync(brainDir, { recursive: true, force: true });
const report = await runCycle(engine, {
brainDir,
sourceId: 'always-fails',
phases: ['embed'],
phases: ['sync'],
});
expect(report.status).toBe('failed');
+28 -2
View File
@@ -79,12 +79,38 @@ describe('doctor checkCycleFreshness', () => {
expect(result.message).toMatch(/gbrain dream --source/);
});
test('source with NO last_full_cycle_at (never cycled) returns fail', async () => {
test('source with NO last_full_cycle_at (never cycled) returns warn, not fail (#2540)', async () => {
// #2540: never-cycled used to FAIL, which turned doctor permanently red
// on any install that doesn't cycle every local_path source (e.g. one
// nightly `dream --dir <vault>` plus other federated sources) — and on
// any source added minutes ago. It surfaces as a warning; only a source
// that HAS cycled and then went stale escalates to fail.
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('virgin');
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('fail');
expect(result.status).toBe('warn');
expect(result.message).toMatch(/never completed a full cycle/);
expect(result.message).toMatch(/gbrain dream --source/);
});
test('reporter case (#2540): one cycled vault + never-cycled siblings is warn, not permanent fail', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('nightly-vault', agoH(2)); // the one vault dreamt via --dir
await seed('federated-a'); // never cycled
await seed('federated-b'); // never cycled
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('warn');
expect(result.message).toMatch(/federated-a/);
expect(result.message).toMatch(/federated-b/);
expect(result.message).not.toMatch(/nightly-vault/);
});
test('a previously-cycled source gone stale still fails even next to never-cycled sources', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('stale', agoH(72)); // real regression signal
await seed('virgin'); // never cycled — warn-only
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('fail');
});
test('mixed sources: highest severity wins (fail > warn > ok)', async () => {
+4 -3
View File
@@ -143,9 +143,10 @@ describe('checkEmbeddingWidthConsistency', () => {
});
test('gateway unconfigured: skips with ok', async () => {
// Reset gateway so requireConfig() throws.
const { resetGateway } = await import('../src/core/ai/gateway.ts');
resetGateway();
// Hard-unconfigure so requireConfig() throws — resetGateway() would
// restore the preload's test baseline (#3554).
const { __unconfigureGatewayForTests } = await import('../src/core/ai/gateway.ts');
__unconfigureGatewayForTests();
const check = await checkEmbeddingWidthConsistency(engine);
expect(check.status).toBe('ok');
expect(check.message).toContain('gateway not configured');

Some files were not shown because too many files have changed in this diff Show More