Compare commits

...
Author SHA1 Message Date
Time Attakc 8475e34ac8 Merge branch 'master' into fix/doctor-wsl-drive-paths-1835 2026-08-01 05:15:51 +08:00
f84bfb57f2 fix(config): render object-valued fields as JSON in config show (#575) (#3575)
Non-string values interpolated into the template literal printed
'[object Object]' (e.g. provider_base_urls). Objects now render via
JSON.stringify; objects under a sensitive key redact to '***' like
their string counterparts.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 05:03:23 +08:00
ad7114f0ad fix(test): make typecheck hermetic to ambient parent openclaw SDK types (#2729) (#3641)
The real plugin-load e2e test dynamically imports 'openclaw/plugin-sdk'.
TypeScript still resolves and type-checks that bare specifier via upward
node_modules resolution, so 'bunx tsc --noEmit' on a clean checkout could
fail (TS2339 on sdk.registerContextEngine) or pass depending on whichever
undeclared openclaw package happened to exist in an ancestor directory.
The existing @ts-ignore only covered the import line, not the property
access on the following line.

Cast the awaited import to a local structural interface declaring the
one member the test uses (registerContextEngine, optional). TypeScript
never consults the ambient module's types for the property access, so
typecheck output is identical regardless of ancestor node_modules state.
The @ts-ignore stays on the import statement itself and stays @ts-ignore
(not @ts-expect-error) because whether TS2307 fires there is itself
ambient-dependent. Runtime behavior is unchanged: the cast erases at
compile time and the export's presence is still verified at runtime.

Fixes #2729

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:03:12 +08:00
23003a2163 fix(facts): preserve remote fence writes (#3659)
Co-authored-by: gbrain contributor <contributor@example.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:03:00 +08:00
4c0ec60275 fix(cycle): thread cycleSourceId into the schema-suggest phase (#3701)
The phase was calling runSchemaSuggestPhase(engine, { dryRun }) with no
sourceId, so it silently fell back to 'default' on every source's dream
cycle -- same bug class as upstream #1586 (synthesize) and #2666
(patterns/synthesize), just an undiscovered instance for this phase.
Confirmed live: schema-events audit log shows only source=default across
41 entries this week despite calendar/mail/mem/social cycles all running
the phase.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 05:02:48 +08:00
Masa 13d95ba0ab fix(schema): apply mutation batches atomically so a mid-batch failure leaves the pack untouched (#2581) (#3446)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Applying a schema mutation batch was not atomic: a failure partway through left earlier mutations permanently written. Reproduced on disk — a failure at index 2 left mutation 0 applied with no way to tell from the pack's state that it was half-done. The fix validates the whole batch first and writes once, which makes partial application impossible by construction rather than by careful ordering.

Verified before merge: the failure was reproduced by injecting one rather than reasoning about it; the PR's own tests fail when the fix is reverted; typecheck clean; MERGEABLE/CLEAN at 22/22 on the current base after batches 1-4 landed.

Sequenced last deliberately — it collides with #3531 on docs/architecture/KEY_FILES.md and with #3667 on src/core/operations.ts, both of which landed earlier today.

Known gap, recorded rather than hidden: lock contention under concurrent writers was reasoned about, not stress-tested.
2026-08-01 04:09:12 +08:00
Time Attakc 022a443e9b fix(pricing): correct voyage-4-large rate and add the missing voyage-4 family entries (#3480)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

voyage-4-large was billed at the voyage-3-large rate — $0.18 against a published $0.12 — so every cost estimate using it was wrong by 50%. Corrected in the canonical table only, per CLAUDE.md's rule that every other pricing table is a derived view, and the drift guard passes. Rate checked against the live vendor page.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: this PR previously failed the JSONB parity guard on a 32-commit-stale base. I rebased it onto current master and re-ran rather than accepting 'flaky' — the guard passes on the real base, 22/22 green.
2026-08-01 03:53:41 +08:00
Masa e7439828f1 fix(migrate): apply the #1178 invalid-index-guard fix to 10 more historical sites (#3192)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Mechanically applies merged #3191's `dropInvalidConcurrentIndex` to the 10 remaining historical migrations that still had the broken DO-block form. Migration ordering and numbering are untouched — this only changes how each guards its own index creation.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: verified by sequence inspection and the migration suite rather than by replaying all 120 migrations against every engine.
2026-08-01 03:53:37 +08:00
Time Attakc 5773736c63 fix(doctor,docs): warn that the npm name 'gbrain' is unrelated + detect a shadowing npm install (#505) (#3454)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The npm package named `gbrain` is an unrelated squatted package, so `npm install gbrain` gives users something that is not this project. Adds doctor detection that classifies real checkouts correctly, fails open, and is try/catch'd throughout. Classification rests on the bin-shape marker since this repo has no `repository` field — verified e2e.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: the remediation commands were not executed against a real global install, and the Windows `which -a` path is unexercised.
2026-08-01 03:53:33 +08:00
cybernaut6404 56454c6ba8 fix(index): preserve code files containing NUL bytes (#3483)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Code files containing raw NUL bytes hard-failed UTF-8 encoding on import, so they silently never indexed. Sanitized at the single choke point both callers route through, with offsets kept in one coordinate space, and exercised end-to-end on a real engine.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: follow-up to file: reindex-code hash ping-pongs on NUL-containing files — reproduced, bounded, and causes no data loss.
2026-08-01 03:53:29 +08:00
alexey-metaengage 63e79838b9 feat(recipes): declare Gemini embedding batch-token budget (#3651)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The google embedding recipe declared no batch caps, so it rode the no-cap fast path with error backstops shaped for Voyage and OpenAI. Caps verified by behavioral probe — 40 texts split into 3 sub-batches matching the declared math. Sequenced after #3531, which touched the same recipe file.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN with 22/22 checks on the current base after batches 1-3 landed.

Known gap, recorded rather than hidden: Gemini's actual 20k limit was taken from vendor docs rather than a live call; being wrong in either direction is bounded by the cap itself.
2026-08-01 03:53:25 +08:00
Masa 3062859420 fix(autopilot): derive the full-cycle timeout floor from the handler anchors (#2781) (#3656)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Full-cycle maintenance jobs were stamped with the outer 600s timeout instead of the 30-minute handler anchor — a regression from #3338 that killed long cycles mid-run. Fixed with a named `fullCycleTimeoutMs` derived from the handler anchors, which now fail loudly rather than silently defaulting; reverting fails 3 of 8 tests.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: the '38 dead cycles in 24h' figure from the description was not reproduced; the stamp arithmetic was verified by code inspection.
2026-08-01 03:39:20 +08:00
Masa addf03119d fix(recipes): align the X secret name with the resolver — X_API_BEARER_TOKEN (#2789 defect 2) (#3649)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The docs and recipe pinned `X_BEARER_TOKEN` while the resolver only ever read `X_API_BEARER_TOKEN` — so no single name worked and the integration could not be configured by following its own documentation. Renamed the dead documented side; reverting fails exactly 2 of the 3 new tests.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: no live X API call was made.
2026-08-01 03:39:16 +08:00
daragao3 dba0ae7b1e fix(validation): qualify backlink endpoint identity (#3667)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The back-link validator compared bare slugs, so a same-slug page in another source masked a genuinely missing reverse edge — silent under-reporting in exactly the multi-source setup where it matters. Now keyed on the full 4-tuple, per the `(source_id, slug)` uniqueness invariant. Verified on real Docker Postgres with 28/28 parity, which the PR itself had skipped.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: remote MCP serialization of the additive Link fields is untested; the fields are additive JSON.
2026-08-01 03:39:12 +08:00
Masa 7376c0266e fix(integrations): resolve secrets through buildGatewayConfig's config→env folding (#3648)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

`integrations show` printed `[missing]` for config-plane keys that the runtime gateway resolves perfectly well — so the status display disagreed with reality and sent people hunting for a problem that did not exist. Fixed with a single `secretEnv()` helper at all four read sites, preserving precedence. The spawn environment is deliberately left unchanged, which is the correct posture. Sequenced after #3531, which refactored the `buildGatewayConfig` internals this consumes.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batches 1 and 2 landed, not a stale one.

Known gap, recorded rather than hidden: the full-suite env-mutation interaction was not run locally; CI shards are green.
2026-08-01 03:39:07 +08:00
Sean Gearin 002ac8050f fix(search): classify first-person "what do I know about X" as an entity query (#3615) (#3616)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

One-character regex fix with a real user-visible effect: "what do **I** know about X" was classified as a general query while the you/we phrasings were correctly classified as entity queries. The new alternation is a strict superset, so no previously-matching phrasing changes, and stubbing the old regex back fails at the exact assertion. Closes verified issue #3615.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.
2026-08-01 03:26:22 +08:00
Javier Aldape f75dbb4ed6 fix(search): give query embeds a fresh floored deadline (#3690)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

The shared AbortSignal arrived already aborted, which made the 2s embed floor dead code and silently degraded hybrid search to keyword-only — users got results that looked complete and were not. Fixed with a fresh AbortSignal.timeout(remaining) at the single shared seam; stubbing the old behavior back fails exactly the new test. This also closes verified issue #2028.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: the DATABASE_URL e2e claims in the description were not re-run, though no SQL is touched.
2026-08-01 03:26:18 +08:00
Masa 437889c0bd fix(cycle): interleave transcript/page work items so a budget cap can't starve the doctor-visible page backlog (#3384)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

Transcripts-first work ordering permanently starved the doctor-visible page backlog whenever the budget cap bit — the pages never got reached. Page-first interleave at the single merge point, with spend proven order-independent, and stubbing the old ordering back fails 4 of 5 tests. Landing first among the extract-atoms.ts PRs, so #3691 and #3654 rebase onto it.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: no real-LLM budget run; the identical error path was driven synthetically.
2026-08-01 03:26:12 +08:00
Time Attakc 335e470394 fix(ai): fold dashscope + google keys into gateway env, drop the retired Gemini default (#3500, #3510) (#3531)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain. 19 of 62 PRs cleared that bar.

dashscope and google keys now fold into the gateway env — verified live end-to-end — and the retired gemini-1.5-pro default is swept from 9 files. Reverting the change fails 11 of 98 tests at fixed seams, and the budget-cap claim in the description reproduced. Landing first in the gateway/config-key seam, so #3648 rebases onto it.

Verified before merge: the PR's own tests fail when the production change is reverted; typecheck clean; MERGEABLE/CLEAN on the current base after batch 1 landed, not a stale one.

Known gap, recorded rather than hidden: no live provider call was made; the gemini retirement was taken from issue history rather than a vendor check. Minor follow-up to file: deriveEnvKey('google_api_key') yields a dead GOOGLE_API_KEY in the minion shell-inherit path.
2026-08-01 03:26:09 +08:00
Masa 7e21e47c15 fix(extract): stop asserting works_at from bare people/->companies/ adjacency (#3466) (#3642)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

19,497 false `works_at` edges were being asserted from bare people/→companies/ directory adjacency. This is the exact fix prescribed when #3495 was closed — bare `mentions` plus an extractor version bump — and stubbing the old behavior back fails the new test. Retroactive cleanup of already-written rows is explicitly out of scope; filing that follow-up.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:55 +08:00
Paolo Belcastro bf7d706bb4 fix(doctor): probe-health 'Latest' reports the true newest run — sort audit events chronologically (#3366)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

doctor's probe-health "Latest" tail-picked the oldest cross-week event instead of the newest. Chronological sort at the reader seam, matching the writer's own documented contract, with all 14 consumers audited. Follow-up to file: `doctor.ts:1017` `self_upgrade_health` has the identical bug class.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:51 +08:00
Masa bb69aa8b65 fix(cli): stop rerouting sync --timeout into dispatchReadOnlyCommand (#3013) (#3650)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

Both #3013 defects reproduced live: `sync --dry-run --timeout` reported "unsupported command", and a bare `--timeout 60` was mis-scaled to 60ms. Fixed with a per-command dispatch gate plus timeout handback to its two owners, with every `cliOpts.timeoutMs` consumer audited. Landing first in the cli.ts and sync.ts clusters.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:47 +08:00
paul-0320 b4a9c7683d fix(search): fold the FTS configuration name into knobs_hash — stop stale rows surviving a reindex-search-vector language switch (#3677)
Adversarial review: survived a hostile reviewer plus two independent refuters, each told to assume the PR was broken and to default to refuting when uncertain.

`GBRAIN_FTS_LANGUAGE` was absent from the query-cache key, so a language switch served stale pre-switch rows. The hash now folds it (v14→15) with all five pin sites updated; reverting the fix fails 4 of 15 tests at the exact claimed step. Landing first in the knobs_hash cluster — the constant is single-writer, so #3617 rebases onto this and takes 16.

Verified before merge: the PR's own tests fail when the production change is reverted (11 of the previous 32 PRs failed exactly there — one had 7 of 8 new tests passing on master); typecheck clean; MERGEABLE/CLEAN with 22/22 checks green on the current base, not a stale one.
2026-08-01 03:11:42 +08:00
Time Attakc 3c61e25503 fix(reindex-frontmatter): reuse the connected engine instead of self-deadlocking on the PGLite lock (#1963) (#3558)
Adversarial review: survived two independent refuters — the only PR of 32 reviewed this way to do so.

The bug: `gbrain reindex-frontmatter` and `gbrain backfill <kind>` were 100% dead on PGLite. cli.ts takes the data-dir lock, the command modules built a second engine on the same dir, and acquireLock never reaps a live PID — 30s timeout, exit 1, with the error naming the waiting process itself as the holder. Reproduced on the parent commit at 33.2s; passes in 5.0s with the fix. Root-cause fix at the dispatch layer, not a softening of the lock, and the sibling census confirmed these were the only two affected callers.

Postgres path verified before merge (it was the review's one open gap, since the bug is PGLite-only and all verification had gone there while the change itself is connection-teardown ownership). Against real Postgres 16 + pgvector: reindex-frontmatter and all three registered backfills exit 0 with zero residual connections, zero advisory locks, and zero cycle-lock rows — byte-identical output and identical teardown to master on the same database, confirming the change is behavior-neutral there.

Merged tree re-verified after rebase: typecheck clean, pglite-lock + reindex-frontmatter 16 pass, llms bundle fresh, 23/23 CI green.
2026-07-31 10:10:57 -07:00
c6dc0adf26 fix(test): repair typecheck failures in admin-sse and lifecycle tests (#3598, #3599) (#3610)
Export AdminSseResponse, HttpServerLifecycle, and SignalSource from
serve-http.ts so test fakes can reference them. Cast structural fakes
through `as unknown as T` where the fake return types (EventEmitter,
plain object) cannot structurally match the full Node/Express originals.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
2026-07-29 17:12:08 -07:00
daragao3andClaude 945fed6105 fix(engine): enforce static engine-live import boundaries (#3596)
* docs: design engine dynamic-import reconciliation

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): reconcile dynamic import hardening

Co-Authored-By: Claude <noreply@anthropic.com>

* test(engine): guard dynamic import policy

* docs: plan engine dynamic-import reconciliation

Record the approved TDD sequence for selective engine-path hardening,
repository guard wiring, documentation, and local verification. Preserve
the no-version-bump and no-publication boundaries for the remaining work.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(engine): record static import invariant

* fix(engine): parse block comments in import guard

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): parse dynamic imports with TypeScript

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close import guard bypasses

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close parser guard edge cases

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): aggregate parser diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): bound dynamic import marker directive

Require the line-level opt-out marker to be standalone inside real comment trivia so negated or incidental longer tokens cannot authorize an import. Preserve the existing general marked-line contract and pin it with focused regression coverage.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(engine): close Unicode marker boundary bypasses

Treat Unicode identifier continuations as marker-token characters and inspect adjacent text by code point so supplementary-plane characters cannot turn longer comment tokens into approvals.\n\nCo-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-29 16:08:08 -07:00
Mikhail Merkulov a175dd0047 fix admin SSE handshake through reverse proxies (#3598) 2026-07-29 16:07:59 -07:00
Mikhail Merkulov 2118f02fc7 fix HTTP server lifecycle retention (#3599) 2026-07-29 16:07:50 -07:00
Sean Gearin a12ab5eabc fix(import): quote frontmatter values and omit absent conversation id (#3600)
Review follow-up to #3549.

An envelope is a third-party file, so interpolating source_provider raw let a
provider string carrying a newline close the scalar and inject arbitrary
frontmatter keys. title: on the line above was already quoted; source: now
matches it.

memvelope_conversation_id emitted the literal string undefined when a
conversation carried no id, which asserts a value rather than reporting
absence: every id-less conversation claims the same id, so anything grouping
or deduping on that key merges unrelated pages. The key is now omitted.

Filename and frontmatter read one hasId predicate so they cannot disagree
about whether an id exists. Tests add both cases; the injection case parses
emitted frontmatter with js-yaml rather than substring-matching it.
2026-07-29 16:07:42 -07:00
Tony Guan e98249a624 fix(files): display zero-byte file sizes (#3608) 2026-07-29 16:07:32 -07:00
Sean Gearin 1057bf4368 feat(import): standalone importer seeding a brain from envelope-v0 chat-history files (#3549)
One Markdown page per conversation from an envelope-v0 file (format spec:
github.com/memvelope/memvelope), written into a directory gbrain sync
ingests. Zero dependencies, deterministic, no network; does not call gbrain.

Filenames are date + conversation id (collision-proof natural key; duplicate
ids overwrite their own file and warn on stderr). Frontmatter carries
type: conversation, source provider, conversation id, and origin. Bodies keep
message-id citations per speaker turn.

Ships as script + test + fixture only; usage and verification steps live in
the script header.
2026-07-29 11:56:20 -07:00
913d2d7f79 fix(test): give slow setup hooks a real timeout budget (#3566)
bun ignores bunfig.toml's timeout key, and beforeAll/beforeEach hooks do
NOT inherit a test's third-arg timeout — a bare `bun test` gives every
hook the 5000ms default even when all tests in the file declare 30s+.
Measured on bun 1.3.14: a 6s hook dies at ~5001ms with the signature
`(unnamed) [5001ms] ... hook timed out` (the #3545 jsonb-parity CI
failure); both `beforeAll(fn, ms)` and the CLI `--timeout` flag are
enforced hook budgets (kills observed at exactly the configured ms).

Fixes:
- e2e.yml (jsonb-parity, tier1, tier2) and release.yml ran bare
  `bun test`; they now pass --timeout=60000 like every scripts/ runner.
- test/e2e/jsonb-roundtrip.test.ts (the #2339 double-encode guard, which
  only real Postgres can surface) additionally carries per-hook 60s
  budgets so a bare local run can't flake either — same pattern as its
  sibling op-checkpoint-jsonb-parity.test.ts.
- scripts/check-bun-test-timeout.sh: CI guard (run from test.yml's
  verify job) failing any future bare `bun test` in workflows/scripts.
- scripts/run-e2e.sh: correct the comment claiming --timeout is
  per-test-only (it covers hooks; the outer gtimeout exists for
  sync-blocking WASM hangs where no timer can fire).

Proof: with Postgres paused for 6s during setupDB's connect, the
unfixed file fails at 5001.81ms with the exact CI signature; the fixed
file passes the identical condition (5 pass, 6.57s). 396 slow
before-hooks across 362 test files lack per-hook budgets; all of them
run through --timeout-passing invocations after this change, enforced
by the new guard.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 20:08:36 -07:00
f9349ba07f fix(doctor,cycle): stop permanent cycle_freshness FAILs on multi-source installs (#2540) (#3562)
Closes the paths #3382 left open (its author said it narrowed the issue
rather than closing it):

1. checkCycleFreshness iterates EVERY local_path source, so an install
   that nightly-dreams one vault via --dir showed a permanent FAIL for
   every other federated source — and for any source added minutes ago.
   'Never completed a full cycle' is now a WARN with the dream/autopilot
   hint; a source that HAS cycled and then went stale still escalates
   through the 6h warn / 24h fail thresholds (the regression signal the
   check exists for). This is the reporter's actual case: the permanent
   red eroded doctor's signal until real staleness hid inside it.

2. resolveSourceForDir's exact-match lookup had no archived filter and
   no ORDER BY, so an archived (or duplicate) alias of the same path
   could shadow the active source; dream's archived guard then refused
   the stamp and the ACTIVE source stayed unstamped forever. The lookup
   now excludes archived rows and orders deterministically, matching
   the canonical-path fallback's posture. The fallback's fail-closed
   ambiguity handling is deliberately unchanged.

3. #3382's own regression test (ii) was environment-sensitive: it
   assumed unsetting OPENAI_API_KEY/ANTHROPIC_API_KEY makes the embed
   phase fail, which is false wherever another embedding provider
   resolves (the cycle then reports 'clean' and the test flips). It now
   fails the sync phase against a vanished checkout — deterministic on
   every machine, same property pinned (a genuinely failing enabled
   phase must prevent the stamp).

New pins fail on unmodified master and pass here: never-cycled→warn
(x2, doctor) and the archived-alias shadow (dream --dir stamp).

Fixes #2540

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:50:44 -07:00
MasaandTime Attakc e72d93fdb5 fix(sync): reconcile the stale old row when a rename falls back to add (#3056) (#3479)
master's rename loop swallows updateSlug failures with an empty catch
("treat as add"), and updateSlug returns void — so a zero-row UPDATE
(old slug absent) and a thrown collision are both invisible. Either way
the run falls through to importFile at the new path while the old row
stays behind live: slug occupied, 0 chunks after the next embed pass,
page count unchanged. A rename that didn't rename, with no trace.

The fix reconciles the duplicate:

- updateSlug returns the number of rows moved in both engines (a
  zero-row UPDATE does not throw; the count is the only way to see it).
- When the cheap rename didn't move a row AND the destination
  demonstrably materialized — imported, or an errorless skip AT the new
  slug (NOT an identity-dedup skip against the old row, which would mean
  nothing landed and deleting the old row would destroy the only copy) —
  the stale row is located positively by source_path = from and deleted.
  No source_path match → nothing is deleted (code-strategy imports don't
  populate source_path and fall back safely to leaving the row).
- A failed reconcile delete records a <rename:…> sentinel: the failure
  gate hard-blocks the bookmark, the auto-skip valve can never
  chronic-skip it (which would bank the duplicate permanently after a
  multi-run outage), and the rename is not checkpointed — the next run
  retries the same diff and clears the sentinel on convergence.

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-28 19:32:55 -07:00
Time AttakcandGarry Tan 85286a556c fix(search): stop boosting compiled_truth at default detail (#3430) (#3514)
* fix(search): stop boosting compiled_truth at default detail (#3430)

COMPILED_TRUTH_BOOST = 2.0 is applied AFTER RRF normalization, and RRF's whole
dynamic range over a 100-deep pool is 1/60 -> 1/160 (a factor of 2.67). So a
2.0x multiplier consumes roughly three quarters of the range: break-even is
`2/(60+r) >= 1/60`, i.e. r <= 60, which means ANY boosted chunk inside the
first 60 ranks outranks an unboosted rank-1 chunk. That is a categorical
filter, not a tilt.

Measured against master's own rrfFusion, with the correct answer in a
fenced_code chunk at vector rank 0:

  compiled_truth chunks in pool | final rank | in top-20
  10                            | 10         | yes
  20                            | 20         | NO
  40                            | 40         | NO
  80                            | 59         | NO

With the boost off the answer stays at rank 0 in every case.

The gate was spelled `detail !== 'high'` -- written as though `high` were the
special case. The documented contract in src/core/operations.ts is
"low (compiled truth only), medium (default, all with dedup), high (all
chunks)", which makes LOW the special one: `low` already restricts to
compiled_truth, so a boost there is a no-op among equals, while `medium` and
`high` are both meant to see everything. So the default detail was silently
compiled-truth-only, contradicting the op's own description.

Three changes:

1. The three fusion call sites now route through a named predicate,
   `shouldBoostCompiledTruth(detail)`, returning true only for 'low'.
   Extracted rather than left inline precisely because an inline expression is
   only reachable through a full hybridSearch round trip -- which is why the
   inversion went unnoticed. The predicate is directly unit-testable.

2. KNOBS_HASH_VERSION 13 -> 14. Results are cached AFTER fusion, so rows
   ranked under the old semantics would otherwise be served under the new ones
   for the whole TTL (3600s default). One-time miss spike on upgrade.

3. test/search-compiled-truth-boost-scope.test.ts pins both the mapping and
   the arithmetic, and documents the displacement it prevents.

Verified the tests discriminate: stubbing the OLD predicate body into master
(so the failure is behavioral rather than a missing export) gives 4 fail /
3 pass; with the fix, 7 pass. typecheck clean, verify 32/32, and 144 pass /
0 fail across the search + fusion + cache suites.

* fix(test): update the three remaining KNOBS_HASH_VERSION pins to 14 (#3430)

Missed in the first pass because I ran a targeted set of test files instead of
the full suite. CI shards 3, 8 and 10 caught them:

  test/search/knobs-hash-reranker.test.ts:67
  test/cross-modal-phase1.test.ts:139,149
  test/search-alias-resolved-boost.test.ts:93

Each carries the running history of why the version moved, so each gets the
13→14 rationale appended rather than just the number swapped. No pins at 13
remain anywhere in test/.

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-28 19:27:46 -07:00
a8a3b6df9f fix(engine): exclude soft-deleted pages from getHealth counts (#1305) (#3556)
getStats() has excluded soft-deleted pages since v0.26.5, but getHealth()
kept counting raw pages rows: page_count, the islanded/orphan scan, the
entity_pages CTE (link/timeline coverage denominators), and most_connected
all included deleted pages, so brain_score never moved when a user
soft-deleted pages. Repro: 50 pages, soft-delete 40 -> getStats 10 vs
getHealth 50, orphan_pages 50, brain_score byte-identical.

Fix: every page-scoped count in getHealth now filters deleted_at IS NULL,
identically in both engines (engine-parity SQL shapes match).

Deliberate boundary: chunk/link storage counts (embed_coverage,
missing_embeddings, link_count, dead_links) stay raw until the purge phase
runs — matching getStats' documented posture — and destructive-removal
counts (#2235) deliberately keep counting all rows. stale_pages already
filtered via buildStalePagesWhere.

Test: test/health-soft-delete.test.ts — 3 of 4 tests fail behaviorally on
unmodified master (page_count 10 vs 4, orphan_pages 8 vs 0, link_coverage
0.5 vs 1), all pass with the fix.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 19:14:39 -07:00
Garry TanandClaude Opus 5 c0dc8b94a5 fix(doctor): stop reporting Windows-drive image paths as missing under WSL (#1835)
files.storage_path rows written by a Windows install (D:/foo/img.jpg,
D:\foo\img.jpg) are not path.isAbsolute() on POSIX, so the image_assets
check joined them onto the repo root and statted a path that can never
exist — a false-positive 'missing from disk, restore from git' WARN.

New src/commands/doctor-asset-paths.ts (kept out of doctor.ts to avoid
colliding with open PRs rewriting that block):
- Under WSL (linux + 'microsoft' in /proc/version), translate D:/x to
  <automount root>/d/x; automount root read from /etc/wsl.conf
  [automount] root, defaulting to /mnt.
- On macOS / plain Linux the path is unresolvable — skip the stat and
  report '(N Windows-drive path(s) skipped — not resolvable on this
  platform)' instead of inventing a path.
- win32 and POSIX-absolute/relative paths keep their existing behavior.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 13:06:38 -07:00
131 changed files with 6822 additions and 649 deletions
+6 -3
View File
@@ -61,7 +61,10 @@ jobs:
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
# --timeout also raises bun's 5s default hook budget (beforeAll/afterAll
# do NOT inherit a test's third-arg timeout; verified on bun 1.3.x).
# Every runner script in scripts/ passes it; bare invocations must too.
run: bun test --timeout=60000 test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
@@ -88,7 +91,7 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -155,7 +158,7 @@ jobs:
}
EOF
- name: Run Tier 2 skill tests
run: bun test test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
run: bun test --timeout=60000 test/e2e/skills.test.ts test/e2e/zeroentropy-live.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+3 -1
View File
@@ -29,7 +29,9 @@ jobs:
with:
bun-version: 1.3.13
- run: bun install
- run: bun test
# --timeout matches every scripts/ runner and covers hook budgets too
# (bunfig.toml's timeout key is ignored by bun; hooks default to 5s).
- run: bun test --timeout=60000
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Attest build provenance
+5
View File
@@ -113,6 +113,11 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
# default regardless of per-test third-arg timeouts. Runs directly
# (not via verify's CHECKS array) to avoid a package.json edit.
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
+15
View File
@@ -2,6 +2,21 @@
All notable changes to GBrain will be documented in this file.
## [0.42.68.1] - 2026-07-30
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
The embedded database allows one process at a time, and holds a lock to enforce it. These two commands opened a second connection to the same database from inside the process that already held that lock, then waited for a lock that could never be released — because the thing holding it was the waiting process itself. The wait ran its full 30 seconds and the command exited with an error naming a blocking process that was, in fact, itself. Both commands now reuse the connection that is already open.
Nothing changes for brains on Postgres, where a second connection was always allowed.
## To take advantage of v0.42.68.1
Nothing to undo — the commands failed without writing anything. Just run whichever you needed:
```bash
gbrain reindex-frontmatter
```
## [0.42.67.0] - 2026-07-28
**If you develop GBrain on Windows, the test and check commands now actually run. Until this release they were quietly doing almost nothing.**
+13
View File
@@ -67,6 +67,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
+7
View File
@@ -16,6 +16,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
+10
View File
@@ -65,6 +65,16 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
+1 -1
View File
@@ -1 +1 @@
0.42.67.0
0.42.68.1
File diff suppressed because one or more lines are too long
+2 -2
View File
@@ -31,7 +31,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
| `--budget-usd N` | unset | Abort before next call's projected cost would exceed cap. Models without a `pricing.ts` entry fail loud (codex #4). |
| `--source db|fs` | `db` | `fs` is reserved for v0.33+. |
| `--slug-prefix P` | unset | Filter takes to pages whose slug starts with P. |
| `--models a,b,c` | `openai:gpt-4o,anthropic:claude-opus-4-7,google:gemini-1.5-pro` | Comma-separated panel. |
| `--models a,b,c` | `openai:gpt-5.2,anthropic:claude-opus-4-7,google:gemini-2.0-flash` | Comma-separated panel. |
| `--json` | off | Emit the full receipt to stdout. |
## Receipt JSON shape (`schema_version: 1`)
@@ -50,7 +50,7 @@ receipt file from disk and re-renders it. The other modes need the brain.
},
"prompt_sha8": "abcd1234",
"models_sha8": "abcd1234",
"models": ["openai:gpt-4o", "anthropic:claude-opus-4-7", "google:gemini-1.5-pro"],
"models": ["openai:gpt-5.2", "anthropic:claude-opus-4-7", "google:gemini-2.0-flash"],
"cycles_run": 3,
"successes_per_cycle": [3, 3, 2],
"verdict": "pass",
+6
View File
@@ -59,6 +59,12 @@ streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
No cache purge is needed. The resolved language is part of the query-cache
key, so rows written under the previous language are unreachable after the
switch — searches read the retokenized index immediately instead of being
served pre-switch results for up to `search.cache.ttl_seconds`. Switching
back reaches the original rows rather than rebuilding them.
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
@@ -0,0 +1,690 @@
# Engine Dynamic-Import Reconciliation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reconstruct the missing engine-path static-import hardening, preserve the four load-bearing lazy gateway fallbacks, and prevent unreviewed dynamic imports from returning.
**Architecture:** Make the 13 safe engine/migration import statements static and leave only four line-marked `ai/gateway.ts` imports inside their existing soft-failure `try/catch` boundaries. Enforce that current state with a repository-anchored Bash wrapper delegating to a fail-closed TypeScript AST scanner, a hermetic Bun regression test, package/verify wiring, and current-state architecture documentation.
**Tech Stack:** TypeScript compiler API, Bun test runner, Bash, Git, generated llms documentation bundles.
## Global Constraints
- Reconstruct directly on branch `claude/kind-meitner-330c90`, based on investigated `origin/master` commit `6136e139972a5449630b4f47f5ed7b4cbe5b811b` plus design commit `d7f52d8c`.
- Do not merge or cherry-pick `48ada48f`, `248bfe55`, `ef4cf7a8`, or either historical branch wholesale.
- Do not modify `VERSION`, `CHANGELOG.md`, `TODOS.md`, or release metadata; this is a no-version-bump reconciliation.
- Keep all four `await import('./ai/gateway.ts')` calls lazy: PGLite and Postgres `initSchema`, plus both `_upsertChunksOnce` methods.
- Every allowed lazy gateway line must carry `engine-dynamic-import-ok`; there is no file-level exemption.
- Preserve the stronger gateway rationale: the static closure is large, and eager module evaluation would occur outside the local `try/catch`, potentially converting a recoverable configuration/import failure into a module-load-time hard failure.
- Describe the hoists as engine-path hardening. Do not claim every dynamic import deterministically causes a Windows crash; system-wide commit exhaustion confounded prior measurements.
- Keep shared PGLite/Postgres behavior in parity.
- Invoke repository shell scripts through `bash` in `package.json`.
- Capture complete test/check output to workspace-local `.context/*.txt` files before inspecting it; never pipe a test command directly through `head` or `tail`.
- Use `git log -G`, not `git log -S`, for any additional dynamic-to-static import history work.
- Keep every implementation and verification commit local. Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after local completion.
- Before editing any affected function, run GBrain `code_blast` and `code_callers` for that symbol and inspect any disambiguation candidates.
---
## File Map
- Create `scripts/check-engine-dynamic-import.sh` — repository-anchored Bash wrapper for default and explicit input routing.
- Create `scripts/check-engine-dynamic-import.ts` — TypeScript AST policy scanner for runtime `import()` expressions, parse/read failures, and exact-line comment-trivia opt-outs.
- Create `test/scripts/check-engine-dynamic-import.test.ts` — 22 hermetic adversarial, CRLF, fail-closed, real-tree, and wiring tests.
- Modify `src/core/pglite-engine.ts` — hoist three safe import statements and mark two deliberate gateway imports.
- Modify `src/core/postgres-engine.ts` — hoist eight safe import statements and mark two deliberate gateway imports.
- Modify `src/core/migrate.ts` — hoist two safe migration helper import statements.
- Modify `package.json` — expose `check:engine-dynamic-import` and append it to `check:all` through `bash`.
- Modify `scripts/run-verify-parallel.sh` — add the package check to the authoritative verify dispatcher.
- Modify `CLAUDE.md` — add the cross-cutting current-state invariant.
- Modify `docs/architecture/KEY_FILES.md` — update current-state entries for the three engine-path files.
- Regenerate `llms.txt` and `llms-full.txt` — required derived bundles after CLAUDE/reference documentation changes.
---
### Task 1: Establish and enforce the source invariant
**Files:**
- Create: `scripts/check-engine-dynamic-import.sh`
- Create: `scripts/check-engine-dynamic-import.ts`
- Create: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `src/core/pglite-engine.ts`
- Modify: `src/core/postgres-engine.ts`
- Modify: `src/core/migrate.ts`
**Interfaces:**
- Consumes: shell positional arguments `FILE...`; without arguments, the guard scans the three repository files.
- Produces: `scripts/check-engine-dynamic-import.sh [FILE...]`, exit `0` when every runtime dynamic import is allowed and exit `1` after reporting every `file:line:text` violation plus every read/parse error on stderr.
- Produces: one line-level opt-out token, `engine-dynamic-import-ok`, accepted only in real comment trivia on the same physical line as the deliberately lazy import.
- Fails closed on missing/unreadable inputs, TypeScript parse diagnostics, and scanner/process failures; comments, strings, templates, regex literals, and type-position `import(...)` syntax are not runtime imports.
- [ ] **Step 1: Record call-graph blast radius before touching functions**
First call `sources_list` and select the source whose registered path is this gbrain checkout. Then run `code_blast` and `code_callers` for these qualified symbols with that exact `source_id`, following `did_you_mean`/`candidates` when a method name is ambiguous:
```text
src/core/pglite-engine.ts::PGLiteEngine.initSchema
src/core/pglite-engine.ts::PGLiteEngine.batchRetry
src/core/pglite-engine.ts::PGLiteEngine._upsertChunksOnce
src/core/pglite-engine.ts::PGLiteEngine.mergeOntologyFact
src/core/pglite-engine.ts::PGLiteEngine.getRecentSalience
src/core/postgres-engine.ts::PostgresEngine.disconnect
src/core/postgres-engine.ts::PostgresEngine.initSchema
src/core/postgres-engine.ts::PostgresEngine.batchRetry
src/core/postgres-engine.ts::PostgresEngine._upsertChunksOnce
src/core/postgres-engine.ts::PostgresEngine.mergeOntologyFact
src/core/postgres-engine.ts::PostgresEngine.reconnect
src/core/postgres-engine.ts::PostgresEngine.getRecentSalience
src/core/migrate.ts::runMigrationSQLWithRetry
src/core/migrate.ts::runMigrations
```
Use `depth: 5`, `max_nodes: 200`, and `limit: 100`. Expected: no caller requires a signature or behavior change; the patch only changes module binding time and retains all local fallback/error handling.
- [ ] **Step 2: Write the failing guard regression test**
Create `test/scripts/check-engine-dynamic-import.test.ts` as a hermetic subprocess suite. The completed 22-test surface covers:
- unmarked runtime `import()` rejection, including bare and trivia-separated forms;
- same-line markers in real line or multiline block-comment trivia;
- rejection of markers on prior lines or inside strings, templates, and module paths;
- comments and comment-like delimiters inside strings, templates, and regex literals;
- live code after same-line or multiline block comments close;
- CRLF input and complete multi-file violation aggregation;
- missing/readable mixed inputs and TypeScript parse diagnostics;
- default repository anchoring when invoked from a foreign Git repository;
- the reconciled three-file source scan plus package/parallel-verifier wiring.
Use the TypeScript parser rather than a partial lexical reimplementation. On Windows, set the test default to 30 seconds because each case launches Git Bash and Bun, whose startup can exceed Bun's 5-second per-test default.
- [ ] **Step 3: Run the test to prove the pre-implementation red state**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result captured inside the log. At minimum, the `exists` assertion fails because `scripts/check-engine-dynamic-import.sh` does not exist. Read `.context/engine-dynamic-import-red.txt`; do not infer the result from a truncated pipeline.
- [ ] **Step 4: Add the CRLF-safe, fail-closed guard**
Create `scripts/check-engine-dynamic-import.sh` as a thin LF-terminated wrapper. Resolve its own directory first; when no explicit files are passed, anchor the repository with `git -C "$SCRIPT_DIR/.."` and scan the two engines plus `migrate.ts`. Delegate with `exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"` so scanner failures propagate.
Create `scripts/check-engine-dynamic-import.ts` using the TypeScript compiler API:
- read every requested file and aggregate read failures;
- parse as TypeScript and aggregate parse diagnostics;
- walk the AST for `CallExpression`s whose expression is `ImportKeyword`;
- locate all marker occurrences in the full source and use `ts.getTokenAtPosition` to admit only occurrences outside AST tokens (real comment trivia), recording their physical source lines;
- require each runtime import's line to have an admitted marker or report its original `file:line:text`;
- print every read/parse error and every violation before exiting nonzero.
This preserves CRLF line accounting, ignores comment/literal/type-only false positives, catches every legal runtime `import()` shape the TypeScript parser recognizes, rejects marker spoofing, and fails closed.
- [ ] **Step 5: Run the guard test to prove the source-tree midpoint is still red**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-midpoint.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: the synthetic violation, marker, comments, and CRLF cases pass. The default repository scan fails and reports all 17 current imports: 13 unmarked safe candidates plus the four not-yet-marked gateway calls.
- [ ] **Step 6: Hoist the three safe PGLite import statements**
Replace the existing `retry.ts` import and add the ontology/recency imports near the top of `src/core/pglite-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
```
Delete only these three in-method destructuring imports, leaving their uses unchanged:
```ts
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
- [ ] **Step 7: Mark both PGLite gateway soft-failure boundaries**
In `PGLiteEngine.initSchema`, preserve the `try/catch` and accessors, changing only the rationale and import line:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not configured — use defaults */ }
```
In `PGLiteEngine._upsertChunksOnce`, preserve the config-row and compile-time fallback chain:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 8: Hoist the eight safe Postgres import statements**
Replace the existing `retry.ts` import and add these imports near the top of `src/core/postgres-engine.ts`:
```ts
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logDbDisconnect } from './audit/db-disconnect-audit.ts';
import { logPoolRecovery } from './audit/pool-recovery-audit.ts';
```
Delete the eight safe dynamic-import statements while keeping their surrounding `try/catch` blocks and calls unchanged:
```ts
const { logDbDisconnect } = await import('./audit/db-disconnect-audit.ts');
const { isRetryableConnError } = await import('./retry.ts');
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const { isConnectionEndedError } = await import('./retry-matcher.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
```
Update the stale `batchRetry` comment from “Lazy-import to avoid a circular dep concern” to current truth:
```ts
// retry.ts is already in this module's static graph through withRetry, so
// classifying the exhausted error does not need a second runtime import.
```
- [ ] **Step 9: Mark both Postgres gateway soft-failure boundaries**
In `PostgresEngine.initSchema`, mirror the PGLite rationale and preserve behavior:
```ts
try {
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
model = gw.getEmbeddingModel();
} catch { /* gateway not yet configured — use defaults */ }
```
In `PostgresEngine._upsertChunksOnce`, preserve the DB-config fallback:
```ts
try {
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
```
- [ ] **Step 10: Hoist the two migration helper import statements**
Add these static imports at the top of `src/core/migrate.ts`:
```ts
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
isStatementTimeoutError,
isRetryableConnError,
} from './retry-matcher.ts';
import { repairTimelineDedupIndex } from './timeline-dedup-repair.ts';
```
Delete only these two local destructuring imports:
```ts
const { isStatementTimeoutError, isRetryableConnError } = await import('./retry-matcher.ts');
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
```
- [ ] **Step 11: Run the complete guard test and direct guard**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; output contains `check-engine-dynamic-import: ok (3 file(s) scanned)`.
- [ ] **Step 12: Prove the guard leaves exactly four marked dynamic imports**
```bash
git grep -n -F "import('./ai/gateway.ts'); // engine-dynamic-import-ok" -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts > .context/engine-dynamic-import-sites.txt; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exactly four lines, all importing `./ai/gateway.ts` and all carrying `engine-dynamic-import-ok`; no match in `src/core/migrate.ts`.
- [ ] **Step 13: Run focused behavior tests**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. If Windows resource pressure aborts the process, record the exact exit code and rerun the failing file alone; do not relabel an infrastructure abort as a source pass.
- [ ] **Step 14: Commit the source invariant locally**
```bash
git add scripts/check-engine-dynamic-import.sh scripts/check-engine-dynamic-import.ts test/scripts/check-engine-dynamic-import.test.ts src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts
```
```bash
git commit -m "fix(engine): reconcile dynamic import hardening"
```
Expected: one local commit; no version or release files staged.
---
### Task 2: Wire the guard into repository checks
**Files:**
- Modify: `test/scripts/check-engine-dynamic-import.test.ts`
- Modify: `package.json`
- Modify: `scripts/run-verify-parallel.sh`
**Interfaces:**
- Consumes: `scripts/check-engine-dynamic-import.sh` from Task 1.
- Produces: package script `check:engine-dynamic-import` and verify dry-list entry of the same name.
- [ ] **Step 1: Add failing wiring assertions**
Add these imports/constants to `test/scripts/check-engine-dynamic-import.test.ts`:
```ts
const PACKAGE_JSON = resolve(REPO_ROOT, 'package.json');
```
Append this test block:
```ts
describe('engine dynamic-import guard wiring', () => {
it('is invoked through bash by check:all', () => {
const pkg = JSON.parse(readFileSync(PACKAGE_JSON, 'utf8')) as {
scripts: Record<string, string>;
};
expect(pkg.scripts['check:engine-dynamic-import']).toBe(
'bash scripts/check-engine-dynamic-import.sh',
);
expect(pkg.scripts['check:all']).toContain(
'bash scripts/check-engine-dynamic-import.sh',
);
});
it('is listed by the authoritative verify dispatcher', () => {
const result = spawnSync(BASH, [VERIFY_DISPATCHER, '--dry-list'], {
cwd: REPO_ROOT,
encoding: 'utf8',
timeout: 30_000,
});
expect(result.status).toBe(0);
expect(new Set((result.stdout ?? '').trim().split('\n'))).toContain(
'check:engine-dynamic-import',
);
});
});
```
- [ ] **Step 2: Run the test and verify both wiring assertions fail**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-red.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit 0
```
Expected: non-zero Bun result. The source guard tests remain green; package-script and verify-list assertions fail because the wiring is absent.
- [ ] **Step 3: Add the package scripts**
In `package.json`, add this script alongside the other `check:*` entries:
```json
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh"
```
Append the guard to the existing `check:all` chain, preserving every existing check:
```text
&& bash scripts/check-engine-dynamic-import.sh
```
Do not rewrite any existing shell entry without its `bash` prefix.
- [ ] **Step 4: Add the authoritative verify entry**
In `scripts/run-verify-parallel.sh`, add this stable `CHECKS` entry near the other source-shape guards:
```bash
"check:engine-dynamic-import"
```
- [ ] **Step 5: Run the regression test and package check**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-wiring-green.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bun run check:engine-dynamic-import > .context/engine-dynamic-import-package-check.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0` and three files scanned.
- [ ] **Step 6: Commit the wiring locally**
```bash
git add package.json scripts/run-verify-parallel.sh test/scripts/check-engine-dynamic-import.test.ts
```
```bash
git commit -m "test(engine): guard dynamic import policy"
```
Expected: one local commit with the guard wiring and its regression assertions.
---
### Task 3: Document the current-state invariant
**Files:**
- Modify: `CLAUDE.md`
- Modify: `docs/architecture/KEY_FILES.md`
- Regenerate: `llms.txt`
- Regenerate: `llms-full.txt`
**Interfaces:**
- Consumes: the four-marked-import source state and the `check:engine-dynamic-import` package surface.
- Produces: current-state contributor guidance and fresh generated documentation bundles.
- [ ] **Step 1: Add the cross-cutting invariant to `CLAUDE.md`**
Add this bullet under “Cross-cutting invariants” near the other language/filesystem guards:
```md
- **Engine-live paths use static imports by default.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, helper modules are top-level imports. The only current
exceptions are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
```
Do not add release tags, Windows-crash certainty, or historical branch names.
- [ ] **Step 2: Update the PGLite current-state entry in `KEY_FILES.md`**
Append this current-state sentence to the existing `src/core/pglite-engine.ts` entry, preserving the entry as one bullet:
```md
Engine-path helper dependencies (`retry`, ontology, recency decay) bind statically; the only lazy imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass.
```
- [ ] **Step 3: Update the Postgres current-state entry in `KEY_FILES.md`**
Append this sentence to the existing `src/core/postgres-engine.ts` entry:
```md
Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite.
```
- [ ] **Step 4: Update the migration current-state entry in `KEY_FILES.md`**
Append this sentence to the canonical `src/core/migrate.ts` entry (the broad runner entry, not the older v95-specific index note):
```md
`retry-matcher.ts` and `timeline-dedup-repair.ts` are static dependencies because `runMigrations()` executes from live engine initialization; the engine dynamic-import guard scans this file with both engine implementations.
```
Keep all three entries current-state only: no `v0.42.x`, branch, commit, “previously,” or “was/now” narration.
- [ ] **Step 5: Regenerate the llms bundles**
```bash
bun run build:llms > .context/engine-dynamic-import-build-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; `llms.txt` and/or `llms-full.txt` update according to their configured linked/inlined status. Byte-identical output for a linked source is acceptable; the freshness test is authoritative.
- [ ] **Step 6: Run documentation freshness checks**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-llms-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
```bash
bun run check:doc-history > .context/engine-dynamic-import-doc-history.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; no release-history marker is introduced into current-state reference docs.
- [ ] **Step 7: Confirm prohibited release files remain untouched**
```bash
git diff --name-only d7f52d8c..HEAD -- VERSION CHANGELOG.md TODOS.md
```
Expected: no output.
- [ ] **Step 8: Commit documentation and generated bundles locally**
```bash
git add CLAUDE.md docs/architecture/KEY_FILES.md llms.txt llms-full.txt
```
```bash
git commit -m "docs(engine): record static import invariant"
```
Expected: one local documentation commit. If one generated bundle is byte-identical, Git simply omits it.
---
### Task 4: Verify and review the complete local reconciliation
**Files:**
- Verify all files changed since `d7f52d8c`.
- Do not create or modify release/publication metadata.
**Interfaces:**
- Consumes: Tasks 13.
- Produces: full local verification evidence and an implementation diff ready for user review, not publication.
- [ ] **Step 1: Run the regression test and direct guard again**
```bash
bun test test/scripts/check-engine-dynamic-import.test.ts > .context/engine-dynamic-import-final-test.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; the full guard regression suite passes.
```bash
bash scripts/check-engine-dynamic-import.sh > .context/engine-dynamic-import-final-guard.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; three files scanned.
- [ ] **Step 2: Run TypeScript checking**
```bash
bun run typecheck > .context/engine-dynamic-import-typecheck.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`. Report exact diagnostics if the branch or current Windows environment has a pre-existing failure.
- [ ] **Step 3: Run the authoritative verify dispatcher**
```bash
bun run verify > .context/engine-dynamic-import-verify.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`, including `check:engine-dynamic-import`. On Windows, classify any per-check timeout from the complete log instead of treating the aggregate result as a source regression without evidence.
- [ ] **Step 4: Re-run focused tests as an ownership check**
```bash
bun test test/chronicle-ontology.test.ts test/chronicle-ontology-ops.test.ts test/recency-decay.test.ts test/core/retry.test.ts test/retry-matcher.test.ts test/audit/pool-recovery-audit.test.ts test/migrate-retry.test.ts test/timeline-dedup-repair.test.ts > .context/engine-dynamic-import-final-focused.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`; record any infrastructure abort separately and rerun only the named file before classifying it.
- [ ] **Step 5: Run the llms freshness test after all documentation settles**
```bash
bun test test/build-llms.test.ts > .context/engine-dynamic-import-final-llms.txt 2>&1; rc=$?; printf 'EXIT=%s\n' "$rc"; exit "$rc"
```
Expected: exit `0`.
- [ ] **Step 6: Run whitespace and scope checks**
```bash
git diff --check d7f52d8c..HEAD
```
Expected: exit `0`, no output.
```bash
git diff --name-only d7f52d8c..HEAD
```
Expected files only:
```text
CLAUDE.md
docs/architecture/KEY_FILES.md
docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
llms-full.txt
llms.txt
package.json
scripts/check-engine-dynamic-import.sh
scripts/check-engine-dynamic-import.ts
scripts/run-verify-parallel.sh
src/core/migrate.ts
src/core/pglite-engine.ts
src/core/postgres-engine.ts
test/scripts/check-engine-dynamic-import.test.ts
```
Either generated llms file may be absent if regeneration proves it byte-identical. `VERSION`, `CHANGELOG.md`, and `TODOS.md` must be absent.
- [ ] **Step 7: Review the exact implementation diff**
```bash
git diff --stat d7f52d8c..HEAD && git diff d7f52d8c..HEAD -- src/core/pglite-engine.ts src/core/postgres-engine.ts src/core/migrate.ts scripts/check-engine-dynamic-import.sh test/scripts/check-engine-dynamic-import.test.ts package.json scripts/run-verify-parallel.sh CLAUDE.md docs/architecture/KEY_FILES.md
```
Expected review findings:
- Exactly 13 safe `await import(...)` statements are removed.
- Exactly four `ai/gateway.ts` imports remain, all marked on the same line.
- All four gateway imports remain inside their original local `try/catch` fallback boundaries.
- No accessor logic, fallback ordering, SQL, public signature, or engine parity behavior changes.
- The parser-backed guard reports all violations plus read/parse failures, preserves CRLF line accounting, ignores comments/literals/type-only syntax, detects every runtime `import()` call expression, and accepts opt-outs only from real comment trivia on the same physical line.
- The package script invokes the shell guard through Bash; `check:all` invokes that shell guard directly, and the parallel verify dispatcher invokes the package check.
- Documentation is current-state and makes no deterministic Windows-crash claim.
**Observed Windows verification classification:** The authoritative aggregate completed with 25 of 33 checks passing. Individual reruns showed `check:test-names` and `typecheck` green; privacy/isolation exceeded Windows timing budgets; WASM failed in unrelated temporary-symlink setup; eval-glossary was CRLF/LF drift; resolver/brain-first findings predated and did not intersect this branch. The focused aggregate produced 103 pass / 5 fail: three setup-hook timeouts reproduced at the untouched base, and the known `migrate-retry` polling failure reproduced there. Its additional race-status assertion did not reproduce at base, so it remains an unresolved timing-sensitive limitation in untouched code—not evidence of an in-scope defect and not claimed as conclusively pre-existing.
- [ ] **Step 8: Commit the approved plan document locally**
The plan is an approved, tracked execution artifact and must not be left as an uncommitted file after implementation:
```bash
git add docs/superpowers/plans/2026-07-28-engine-dynamic-import-reconciliation.md
```
```bash
git commit -m "docs: plan engine dynamic-import reconciliation"
```
Expected: one local plan commit; no release metadata staged.
- [ ] **Step 9: Inspect final status without publishing**
```bash
git status --short --branch
```
Expected: branch `claude/kind-meitner-330c90` with a clean working tree. No push, PR, upstream comment, or other external side effect.
- [ ] **Step 10: Capture the completed milestone to memory**
Before writing, search MemPalace wing `gbrain` for this exact reconciliation to avoid duplication. Add a verbatim drawer recording exact base/head commits, the 13 hoists, four gateway opt-outs and rationale, guard/test/docs files, every verification command with exit code, and any environment-owned failures. Add a GBrain project timeline entry only if there is an existing relevant gbrain project page; do not create duplicate release metadata.
- [ ] **Step 11: Report the local result and ask separately before publication**
Report:
- exact local commits;
- changed files;
- test/check exit codes;
- any blocked or pre-existing failures;
- confirmation that release files were untouched;
- confirmation that nothing was pushed or published.
Do not run any publication command. Wait for explicit user approval before any push, PR, or upstream interaction.
@@ -0,0 +1,175 @@
# Scalar-source Backlink Validation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make backlink validation compare exact `(source_id, slug)` endpoint identities while preserving existing scalar, unscoped, and federated link-read semantics.
**Architecture:** Enrich every engine link-read row with the source identity of its joined from, to, and visible origin pages. Pass the validated page's scalar or federated scope into validator context; the backlink validator scopes its initial read consistently, groups targets by exact identity, and accepts only an exact reverse endpoint pair. SQL predicates remain unchanged, so trusted scalar cross-source visibility and federated all-endpoint containment remain intact.
**Tech Stack:** TypeScript, Bun test, PGLite, PostgreSQL/postgres.js.
## Global Constraints
- Use strict red-before-green TDD with duplicate slugs across sources.
- Preserve unscoped historical reads, scalar near-endpoint scoping, scalar explicit cross-source visibility, federated all-endpoint containment, and `sourceIds` precedence.
- Keep PostgreSQL and PGLite projections in parity.
- Do not change schema or conditional-write conflict semantics.
- Keep deployment, restart, migration, and push actions outside the implementation tasks; a separately authorized release workflow may perform them after verification.
- Capture full test output to files before inspecting it.
---
### Task 1: Pin the backlink false-negative in PGLite
**Files:**
- Modify: `test/writer.test.ts`
**Interfaces:**
- Consumes: `backLinkValidator.validate(PageValidationContext)` and source-qualified `putPage`/`addLink`.
- Produces: regressions for wrong-source reverse rejection, exact reverse acceptance, cross-source pair acceptance, and exact target deduplication.
- [ ] **Step 1: Add the minimal failing duplicate-slug regression**
Create `default` and `team-x` copies of the origin and target, add `(team-x, origin) -> (team-x, target)` plus the wrong reverse `(team-x, target) -> (default, origin)`, validate with `sourceId: 'team-x'`, and require one warning.
- [ ] **Step 2: Run the focused test and verify RED**
```bash
bun test test/writer.test.ts -t "wrong-source reverse" > "$TEMP/backlink-red.txt" 2>&1
```
Expected: assertion failure because current slug-only validation returns zero findings.
- [ ] **Step 3: Add the remaining behavioral regressions after the first red is recorded**
Add tests proving that the exact reverse clears the warning, a legitimate cross-source forward/reverse pair passes, and two destinations sharing one slug but differing by source are validated independently.
### Task 2: Expose exact endpoint identity from both engines
**Files:**
- Modify: `src/core/types.ts:1204-1229`
- Modify: `src/core/postgres-engine.ts:3021-3124`
- Modify: `src/core/pglite-engine.ts:2941-3037`
- Modify: `test/get-page-federated-scope.test.ts:187-246,289-306`
- Modify: `test/e2e/multi-source-bug-class.test.ts:184-205`
- Modify: `test/e2e/engine-parity.test.ts:813-875`
**Interfaces:**
- Produces: `Link.from_source_id: string`, `Link.to_source_id: string`, and `Link.origin_source_id?: string | null`.
- Preserves: `getLinks(slug, { sourceId?, sourceIds? })` and `getBacklinks(...)` filtering semantics.
- [ ] **Step 1: Add engine-contract assertions before implementation**
Assert scalar cross-source rows expose `beta -> default`, federated rows expose only in-grant endpoint IDs, `sourceIds` still beats scalar `sourceId`, and an out-of-grant origin has both `origin_slug` and `origin_source_id` null.
- [ ] **Step 2: Run the focused contract tests and verify RED**
```bash
bun test test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/link-identity-red.txt" 2>&1
```
Expected: source-ID assertions fail because fields are absent.
- [ ] **Step 3: Extend `Link` and project IDs without changing predicates**
Use this additive contract:
```ts
export interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
link_type: string;
context: string;
link_source?: string | null;
origin_slug?: string | null;
origin_source_id?: string | null;
origin_field?: string | null;
}
```
In all six branches per engine, project:
```sql
f.source_id AS from_source_id,
t.source_id AS to_source_id,
o.source_id AS origin_source_id
```
Keep every `WHERE` and grant-aware origin `LEFT JOIN` unchanged.
- [ ] **Step 4: Re-run contract tests and verify GREEN**
Use the same command and require all focused tests to pass.
### Task 3: Validate exact reverse identities and propagate scope
**Files:**
- Modify: `src/core/output/writer.ts:89-96,240-318`
- Modify: `src/core/output/post-write.ts:36-41,73-118`
- Modify: `src/core/output/validators/back-link.ts:24-47`
- Modify: `src/core/operations.ts:1227-1246`
- Modify: `test/post-write-lint.test.ts:67-130`
**Interfaces:**
- Produces: optional `PageValidationContext.sourceId` and `sourceIds`, with `sourceIds` taking precedence.
- `runPostWriteLint(..., opts)` accepts the same optional scope and loads the validated page through it.
- [ ] **Step 1: Add a post-write nested-read regression and verify RED**
Validate a non-default page with a wrong-source reverse via `runPostWriteLint(..., { force: true, noLog: true, sourceId: 'team-x' })`; require a backlink warning.
- [ ] **Step 2: Implement minimal scope propagation**
Add `sourceId?`/`sourceIds?` to validation context and lint options. Load pages using `sourceIds` when non-empty, otherwise scalar `sourceId`. Pass the same scope into nested validators. In the put-page success hook, call lint with the already-resolved write source ID.
- [ ] **Step 3: Implement exact backlink matching**
Initial outbound reads use the validation scope. Deduplicate rows by all four endpoint identity fields so every distinct expected origin remains represented even when targets share a source-qualified identity. Read each target using the federated grant when present, otherwise the target's exact scalar source. Accept only a row matching all four endpoint fields of the expected reverse.
- [ ] **Step 4: Run writer and post-write tests and verify GREEN**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts > "$TEMP/backlink-green.txt" 2>&1
```
Expected: all tests pass, including the recorded false-negative.
### Task 4: Verify PostgreSQL/PGLite parity and final scope
**Files:**
- Modify: `test/e2e/engine-parity.test.ts:813-875`
- Verify: all files above
**Interfaces:**
- Consumes: exact endpoint fields and unchanged filtering semantics.
- Produces: parity evidence for scalar cross-source and federated reads.
- [ ] **Step 1: Compare complete endpoint tuples across engines**
Compare sorted tuples containing `from_source_id`, `from_slug`, `to_source_id`, `to_slug`, `origin_source_id`, and `origin_slug` for scalar and federated fixtures.
- [ ] **Step 2: Run focused PGLite/source-isolation tests**
```bash
bun test test/writer.test.ts test/post-write-lint.test.ts test/get-page-federated-scope.test.ts test/e2e/multi-source-bug-class.test.ts > "$TEMP/backlink-focused.txt" 2>&1
```
Expected: exit 0.
- [ ] **Step 3: Run PostgreSQL parity when the test database is available**
```bash
bun test test/e2e/engine-parity.test.ts -t "federated sourceIds" --timeout=300000 > "$TEMP/backlink-parity.txt" 2>&1
```
Expected: exit 0; if the configured test database is unavailable, report the exact environmental blocker rather than claiming parity execution.
- [ ] **Step 4: Typecheck and inspect the final diff**
```bash
bun run typecheck > "$TEMP/backlink-typecheck.txt" 2>&1
```
Expected: exit 0. Then run `git diff --check` and confirm no version, schema, migration, deployment, or conditional-write files changed.
@@ -0,0 +1,142 @@
# Engine dynamic-import reconciliation design
**Date:** 2026-07-28
## Goal
Reconcile the overlapping engine dynamic-import changes from:
- `claude/hungry-edison-8bb1cd` at release commits `48ada48f` and `248bfe55`
- `claude/elegant-gates-e5275e` at `ef4cf7a8`
onto a fresh branch from current `origin/master`, without merging or cherry-picking either lineage wholesale and without adding a release/version bump.
## Established state
At investigation time:
- `origin/master` was `6136e139972a5449630b4f47f5ed7b4cbe5b811b`, version `0.42.67.0`.
- Upstream PR #3511 was still open, so trunk did not contain its two `chronicle/ontology.ts` hoists.
- Neither source branch was an ancestor of trunk.
- Trunk contained 17 dynamic imports in the three engine-path files:
- 13 safe-hoist candidates: two ontology imports, nine engine helper/audit imports, and two migration imports.
- Four `ai/gateway.ts` imports, all inside `try/catch` fallback paths.
- `git log -G` showed the separate ontology, helper, migration, and gateway histories. `git log -S` is not suitable for this dynamic-to-static replacement because the relevant token can remain present while its context changes.
- The guard from `ef4cf7a8` passed against that commit but failed against trunk. It also knew about only two gateway opt-outs because two `_upsertChunksOnce` gateway lookups landed later in trunk.
## Selected approach
Reconstruct the intended current state directly on fresh `origin/master`.
Do not merge or cherry-pick either old lineage. Selectively reproduce the desired source changes, adapt the guard to the current four gateway call sites, and write current-state documentation. This avoids importing stale release metadata, stale TODO claims, and unrelated lineage changes.
## Source changes
### Safe static imports
Hoist all 13 safe candidates:
- `src/core/pglite-engine.ts`
- `valueHash`, `normalizeDimension`, `isNovelDimension` from `chronicle/ontology.ts`
- `isRetryableConnError` through the existing `retry.ts` import
- `resolveRecencyDecayMap`, `DEFAULT_FALLBACK` from `search/recency-decay.ts`
- `src/core/postgres-engine.ts`
- the same ontology, retry, and recency helpers
- `isConnectionEndedError` from `retry-matcher.ts`
- `logDbDisconnect` from `audit/db-disconnect-audit.ts`
- `logPoolRecovery` from `audit/pool-recovery-audit.ts`
- `src/core/migrate.ts`
- `isStatementTimeoutError`, `isRetryableConnError` from `retry-matcher.ts`
- `repairTimelineDedupIndex` from `timeline-dedup-repair.ts`
The implementation must keep the two engines in parity where the behavior is shared. Comments should describe current invariants, not repeat an unproven causal claim that these hoists fix the Windows test-runner crash.
### Deliberately lazy gateway imports
Keep all four `await import('./ai/gateway.ts')` call sites lazy:
- PGLite `initSchema`
- PGLite `_upsertChunksOnce`
- Postgres `initSchema`
- Postgres `_upsertChunksOnce`
Each line receives the explicit `engine-dynamic-import-ok` marker and a concise nearby rationale.
The rationale has two parts:
1. The gateway's static closure includes the AI SDK, provider packages, and validation/config machinery, so eager loading would tax engine startup paths that do not otherwise need it.
2. More importantly, each lookup is inside a `try/catch` that preserves a soft fallback (compiled defaults or the brain's stored embedding-model config). Hoisting the module would evaluate it before that catch can run and could convert a recoverable configuration/import failure into a module-load-time hard failure.
The guard must not allow unmarked gateway imports or a broad file-level exemption.
## Guard and wiring
Add `scripts/check-engine-dynamic-import.sh`, adapted from `ef4cf7a8`, with these properties:
- Default scan set:
- `src/core/pglite-engine.ts`
- `src/core/postgres-engine.ts`
- `src/core/migrate.ts`
- Normalize trailing CR before matching so CRLF checkouts cannot bypass the check.
- Ignore comment-only lines.
- Ignore only lines carrying `engine-dynamic-import-ok`.
- Report every unmarked `await import(` with file and line.
- Explain that contributors should prefer a static import and must justify a real opt-out.
- Avoid asserting that every dynamic import deterministically crashes Windows; the measured evidence supports treating the pattern as an engine-path hardening invariant, while box-level commit exhaustion remained a confound in prior runs.
Wire it into:
- `package.json` as `check:engine-dynamic-import`
- `package.json` `check:all`
- `scripts/run-verify-parallel.sh`
Follow trunk's current rule that package scripts invoke repository shell scripts through `bash`.
## Regression coverage
Add an automated test for the guard. It must cover:
- A real dynamic import produces exit 1 and is reported.
- A line carrying `engine-dynamic-import-ok` is allowed.
- Line comments and block-comment lines do not produce findings.
- The same violation is caught with CRLF input.
- The default repository scan passes after the source reconciliation.
Use a temporary fixture rather than mutating tracked source files. Keep assertions path-portable.
The pre-fix red demonstration is the exact guard from `ef4cf7a8` run against current trunk: it exits 1 and reports the existing unmarked imports. The post-fix guard and test must pass.
## Documentation policy
Preserve current behavior, not either old release narrative:
- Do not modify `VERSION` or add a release `CHANGELOG.md` entry.
- Do not copy old version headings or completed release TODO blocks.
- Do not retain the old TODO claiming that extracting gateway accessors is necessarily the fix; the lazy imports are deliberately protected by their local soft-failure boundaries.
- Add the cross-cutting no-unmarked-dynamic-import invariant to `CLAUDE.md`.
- Update the current-state entries for `src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and `src/core/migrate.ts` in `docs/architecture/KEY_FILES.md` where needed.
- Regenerate `llms.txt` and `llms-full.txt` after the documentation edits.
- Add a TODO only if implementation uncovers a real unresolved action.
Public documentation must use generic language and must not overstate the historical Windows crash causality.
## Verification
Capture full output to files before inspecting summaries. Run, at minimum:
1. The guard regression test.
2. `bash scripts/check-engine-dynamic-import.sh`.
3. Focused tests that exercise the touched engine, migration, retry, audit, and recency modules.
4. `bun run typecheck`.
5. `bun run verify`.
6. `bun run build:llms` followed by `bun test test/build-llms.test.ts`.
7. `git diff --check` and a final clean-status/diff review.
If platform contention or existing Windows suite defects block a broad test, report the exact command, exit code, and ownership classification rather than declaring success from a partial run.
## Git and publication boundary
- Work on `claude/kind-meitner-330c90`, reset locally to the exact investigated `origin/master` base.
- Preserve the previous worktree tip under `claude/kind-meitner-330c90-pre-reconcile`.
- Keep implementation and verification commits local.
- Do not push, create a PR, comment upstream, or otherwise publish without explicit user approval after the local result is complete.
@@ -0,0 +1,184 @@
# Scalar-source backlink validation design
## Problem
A page identity in a multi-source brain is `(source_id, slug)`, but the back-link validator currently reasons only about `slug`.
For an outbound edge:
```text
(source-a, concepts/origin) -> (source-a, people/target)
```
the validator accepts any reverse row whose bare slugs are:
```text
people/target -> concepts/origin
```
That can incorrectly accept a row ending at `(default, concepts/origin)` instead of `(source-a, concepts/origin)`.
The bug is not that scalar `getLinks(slug, { sourceId })` permits cross-source destinations. That behavior is intentional: scalar scope qualifies the near/from endpoint while trusted local callers retain visibility into explicit cross-source edges. The gap is that a returned `Link` does not carry the source identity of either endpoint, so callers cannot distinguish same-slug pages.
## Reproduction and evidence
A deterministic PGLite reproduction creates duplicate `concepts/a` and `people/b` pages in `default` and `team-x`, then adds:
```text
(team-x, concepts/a) -> (team-x, people/b)
(team-x, people/b) -> (default, concepts/a)
```
The second edge is not a valid reverse of the first. Nevertheless:
```ts
await engine.getLinks('people/b', { sourceId: 'team-x' })
```
returns the second row, and the current validator accepts it because `to_slug === 'concepts/a'`.
Both engines implement the same scalar rule: filter `f.slug` and `f.source_id`, join the actual destination by `to_page_id`, and do not filter `t.source_id`. Federated `sourceIds` is a separate branch that constrains all visible endpoints and takes precedence over scalar scope.
## Goals
1. Validate back-links by exact source-qualified endpoint identity.
2. Preserve explicit cross-source links for trusted scalar reads.
3. Preserve federated all-endpoint containment and `sourceIds` precedence.
4. Keep PostgreSQL and PGLite behavior identical.
5. Add strict red-before-green regressions using duplicate slugs across sources.
6. Avoid schema migrations and production operational changes.
## Non-goals
- Changing scalar link reads to same-source-only reads.
- Weakening or widening federated reads.
- Changing link write identity or database schema.
- Refactoring the atomic conditional-write branch.
- Coupling deployment, restart, or migration mechanics to the backlink code change. Release operations are handled separately after verification.
## Chosen approach
Extend the engine `Link` result with endpoint source identities and use those fields in the validator.
```ts
interface Link {
from_slug: string;
from_source_id: string;
to_slug: string;
to_source_id: string;
// existing fields
origin_slug?: string | null;
origin_source_id?: string | null;
}
```
All `getLinks` and `getBacklinks` query branches in PostgreSQL and PGLite will project the source IDs from the pages already joined as `f`, `t`, and `o`. No filtering behavior changes.
This approach is preferred over a dedicated `hasExactLink` method because it keeps source identity attached to the link data everywhere, avoids duplicate engine SQL and per-edge existence queries, and matches existing source-qualified link-write and batch-row contracts.
Validator-only raw SQL is rejected because validators should consume the `BrainEngine` contract rather than bypass it with engine-specific schema knowledge.
## Engine semantics
The existing three read modes remain unchanged.
### Unscoped
`getLinks(slug)` returns rows from all same-slug from-pages across sources. Each row identifies the actual source of both endpoints.
### Scalar source
`getLinks(slug, { sourceId })` matches exactly `(sourceId, slug)` on the from side. A destination may belong to another source, and `to_source_id` reveals that exact identity.
The corresponding scalar `getBacklinks` rule continues to match the exact destination/to-page identity while allowing a cross-source referrer.
### Federated sources
`getLinks(slug, { sourceIds })` continues to constrain from and to endpoints to the grant. The origin join continues to redact an out-of-grant origin. `sourceIds` continues to take precedence over scalar `sourceId`.
Adding source IDs to returned in-grant endpoints does not disclose anything new: the existing result already discloses those pages' slugs and edges. An out-of-grant endpoint remains absent.
## Validator algorithm
The validator receives the source scope associated with the page being validated.
For every outbound edge:
```text
(from_source_id, from_slug) -> (to_source_id, to_slug)
```
it requires a reverse row:
```text
(to_source_id, to_slug) -> (from_source_id, from_slug)
```
Duplicate edge rows are deduplicated by the full endpoint pair `(from_source_id, from_slug, to_source_id, to_slug)`, not by bare target slug. This preserves separate reverse requirements when multiple same-slug origin pages point to one exact target.
For each target:
1. Read target outbound links using the target's exact scalar source when validation is scalar-scoped.
2. Under federated validation, retain the caller's `sourceIds` grant rather than converting it to scalar scope.
3. Accept only a returned row whose `from_source_id`, `from_slug`, `to_source_id`, and `to_slug` exactly match the expected reverse identity.
4. Emit the existing warning when no exact reverse exists.
This preserves legitimate cross-source pairs. For example:
```text
(source-a, concepts/origin) -> (source-b, people/target)
(source-b, people/target) -> (source-a, concepts/origin)
```
is valid.
## Validation context propagation
`PageValidationContext` must carry the relevant scalar or federated source scope. The writer and post-write lint paths must load the page with that scope and pass the same scope to nested validator reads.
This change is scoped to source routing needed by validation. It does not modify conditional-write revision or conflict semantics and must not be applied to the atomic conditional-write branch.
## Testing strategy
### PGLite strict-TDD regression
Add duplicate pages across `default` and a second source, then prove before the production fix that:
1. A forward edge in the second source plus a wrong-source reverse produces a warning.
2. Adding the exact reverse removes the warning.
3. A legitimate cross-source forward/reverse pair passes.
4. Two same-slug destination pages are not collapsed into one target identity.
The first assertion must fail against the pre-fix implementation.
### Engine contract tests
For PGLite and PostgreSQL:
1. Assert link rows expose exact from/to source IDs.
2. Assert scalar reads still return explicit cross-source destinations.
3. Assert federated reads still exclude out-of-grant endpoints.
4. Assert `sourceIds` still takes precedence over scalar `sourceId`.
5. Assert origin source identity is null when the origin is redacted by the federated branch.
### Parity and focused verification
Run:
- the focused backlink validator test;
- source-isolation and federated link tests;
- the Postgres/PGLite parity fixture with a test database;
- related writer/post-write tests;
- `bun run typecheck`.
Capture complete command output to files before inspecting summaries. Do not use production databases or restart the live service.
## Compatibility
The `Link` change is additive at runtime. Existing consumers that read only slug or provenance fields continue to work. TypeScript object literals typed as complete `Link` values may need source fields; if compatibility pressure is high, the source fields can initially be optional in the public type while engine implementations and validator tests require their presence. The preferred contract is required endpoint source IDs because every persisted link always has both pages and therefore both source IDs.
No schema migration is required because source IDs already live on the joined `pages` rows.
## Operational constraints
The implementation phase does not deploy, restart GBrain, run production migrations, or alter the atomic conditional-write branch. Release, migration, and restart operations are a separate verified workflow and do not change this design's engine or validator semantics.
+30
View File
@@ -216,6 +216,19 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
imports use static top-level imports. The only current dynamic-`import()` exceptions
are the four `ai/gateway.ts` lookups in both engines'
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
local `try/catch` because the gateway has a large provider/config closure and,
more importantly, eager evaluation would occur before the catch and could
turn a recoverable default/config-row fallback into a module-load failure.
Every exception carries `engine-dynamic-import-ok` on the import line.
`scripts/check-engine-dynamic-import.sh` enforces the rule. For history, use
`git log -G'await[[:space:]]+import\\('`, not `git log -S`: a dynamic-to-static
rewrite can preserve the searched token while changing its context.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -1006,6 +1019,13 @@ If you fetched this file by URL without cloning yet, the companion files live at
## Step 1: Install GBrain
> **NEVER install from the npm registry.** GBrain is not distributed on npm; the npm
> package named `gbrain` is an unrelated package. Do NOT run `npm install -g gbrain` or
> `bun add -g gbrain` (note the missing `github:` prefix — that's the trap). The only
> supported sources are `github:garrytan/gbrain` and a git clone, exactly as shown below.
> If an unrelated npm install is already present, remove it first
> (`npm uninstall -g gbrain` / `bun remove -g gbrain`); `gbrain doctor` also detects this.
Default path (Bun is required — gbrain is a Bun + TypeScript runtime):
```bash
@@ -1559,6 +1579,16 @@ This is the difference between a search engine and a brain. Search finds the pag
## Install
> [!WARNING]
> **GBrain is NOT distributed on npm.** The npm package named `gbrain` is an unrelated
> package with no connection to this project. Do not run `npm install -g gbrain` or
> `bun add -g gbrain` — you'll get something else, and it can shadow the real binary on
> your PATH. Install and upgrade ONLY via the documented paths below
> (`bun install -g github:garrytan/gbrain`, or `git clone` + `bun install && bun link`).
> If you already ran the npm install by mistake: `npm uninstall -g gbrain` /
> `bun remove -g gbrain`, then reinstall from GitHub. `gbrain doctor` detects a
> shadowing npm install and prints the fix.
GBrain is designed to be installed and operated by an AI agent. The fastest path is to have your agent do it for you. The CLI and MCP paths below are for people who want to wire it up themselves.
### Have your agent install it (recommended)
+3 -2
View File
@@ -48,7 +48,8 @@
"check:system-of-record": "bash scripts/check-system-of-record.sh",
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
"check:cli-exec": "bash scripts/check-cli-executable.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh",
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh",
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
@@ -146,7 +147,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.67.0",
"version": "0.42.68.1",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
+15 -7
View File
@@ -1,12 +1,12 @@
---
id: x-to-brain
name: X-to-Brain
version: 0.8.2
version: 0.8.3
description: Twitter timeline, mentions, and keyword monitoring flow into brain pages. Tracks deletions, engagement velocity, OCR on images, and real-time alerts.
category: sense
requires: []
secrets:
- name: X_BEARER_TOKEN
- name: X_API_BEARER_TOKEN
description: X API v2 Bearer token (Basic tier minimum, $200/mo for full archive search)
where: https://developer.x.com/en/portal/dashboard — create a project + app, copy the Bearer Token from "Keys and tokens"
- name: X_HANDLE
@@ -16,7 +16,7 @@ health_checks:
- type: http
url: "https://api.x.com/2/users/by/username/$X_HANDLE"
auth: bearer
auth_token: "$X_BEARER_TOKEN"
auth_token: "$X_API_BEARER_TOKEN"
label: "X API"
setup_time: 15 min
cost_estimate: "$0-200/mo (Free tier: 1 app, read-only. Basic: $200/mo for search + higher limits)"
@@ -118,11 +118,11 @@ Tell the user:
Note: Free tier gives read-only access with low limits. Basic tier ($200/mo)
gives search/recent endpoint and higher limits. Pro tier gets full archive search."
Set both `X_BEARER_TOKEN` and `X_HANDLE` in the environment. Validate immediately
Set both `X_API_BEARER_TOKEN` and `X_HANDLE` in the environment. Validate immediately
(app-only bearer tokens cannot call `/users/me` — that endpoint requires
user-context OAuth — so validation uses the by-username lookup):
```bash
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
curl -sf -H "Authorization: Bearer $X_API_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/$X_HANDLE" \
&& echo "PASS: X API connected" \
|| echo "FAIL: X API token invalid"
@@ -138,7 +138,7 @@ starting with 'AAA...', (3) if you just created the app, the token is valid imme
```bash
# Look up the user's X user ID from their handle
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
curl -sf -H "Authorization: Bearer $X_API_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/$X_HANDLE" | grep -o '"id":"[^"]*"'
```
@@ -210,7 +210,7 @@ The agent should review collected data 2-3x daily and run enrichment.
```bash
mkdir -p ~/.gbrain/integrations/x-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.2","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.3","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
```
## Production Patterns (v0.8.1)
@@ -438,6 +438,14 @@ Free tier works for personal monitoring. Basic tier needed for keyword search.
## Troubleshooting
**Upgrading from recipe v0.8.2 or earlier (token shows [missing] after upgrade):**
- Older versions of this recipe named the token `X_BEARER_TOKEN`. The canonical
name is `X_API_BEARER_TOKEN` — the name the built-in `x_handle_to_tweet`
resolver reads. Rename the variable wherever you set it (shell profile, cron
environment, `.env`) — same value, new name. A collector installed under the
old name keeps running either way; the rename is what makes the integrations
dashboard and the resolver see the token.
**API returns 403:**
- Check your app has the right access level (Read or Read+Write)
- Free tier apps can only use basic endpoints
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard: every `bun test` invocation in workflows and runner scripts must
# pass an explicit --timeout.
#
# Why: bun ignores bunfig.toml's `timeout` key (verified on 1.3.14), so a bare
# `bun test` gets the 5000ms default for BOTH tests and beforeAll/beforeEach/
# afterAll/afterEach hooks. Hooks do NOT inherit a test's third-arg timeout —
# a file whose tests all declare `}, 30_000)` still has a 5s hook budget, and
# slow setup (Postgres connect + migrations, PGLite cold start) flakes on
# loaded CI runners with the signature `(unnamed) [5001ms] ... hook timed out`
# (the #3545 jsonb-parity failure). The CLI --timeout flag is the one measured
# mechanism that raises the hook budget uniformly; per-hook second-arg
# timeouts work too but don't scale to ~400 slow hooks.
#
# Usage: scripts/check-bun-test-timeout.sh
# Exit: 0 when clean, 1 when a bare `bun test` invocation is found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Match executable `bun test` invocations. Exclude comment lines (#, //, *)
# and lines that already carry --timeout anywhere.
# Scope: workflows + runner scripts (the surfaces CI executes). package.json
# script bodies route through scripts/ already; editing it is out of scope here.
violations="$(grep -rnE '\bbun test\b' .github/workflows scripts 2>/dev/null \
| grep -v -- '--timeout' \
| grep -vE ':[[:space:]]*(#|//|\*)' \
| grep -v 'check-bun-test-timeout' \
|| true)"
if [ -n "$violations" ]; then
echo "FAIL: bare 'bun test' without --timeout (5s default kills slow setup hooks):" >&2
echo "$violations" >&2
echo "" >&2
echo "Add --timeout=60000 (see scripts/run-unit-shard.sh for the convention)." >&2
exit 1
fi
echo "OK: every bun test invocation passes an explicit --timeout."
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Engine-live paths use static imports by default. A line-level
# `engine-dynamic-import-ok` marker is required for a justified lazy import.
#
# Historical Windows runs associated imports on these paths with abrupt Bun
# test-process exits, but system-wide commit exhaustion remained a confound.
# This guard therefore enforces a reviewed engine-path hardening invariant; it
# does not claim every dynamic import deterministically crashes Windows.
#
# Usage:
# bash scripts/check-engine-dynamic-import.sh
# bash scripts/check-engine-dynamic-import.sh FILE [FILE...]
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" || exit 1
if [ "$#" -gt 0 ]; then
FILES=("$@")
else
ROOT="$(git -C "$SCRIPT_DIR/.." rev-parse --show-toplevel 2>/dev/null || true)"
[ -n "$ROOT" ] || ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
cd "$ROOT" || exit 1
FILES=(
src/core/pglite-engine.ts
src/core/postgres-engine.ts
src/core/migrate.ts
)
fi
exec bun "$SCRIPT_DIR/check-engine-dynamic-import.ts" "${FILES[@]}"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bun
import { readFile } from 'node:fs/promises';
import ts from 'typescript';
const MARKER = 'engine-dynamic-import-ok';
const MARKER_TOKEN_CHAR = /[\p{ID_Continue}$-]/u;
const files = process.argv.slice(2);
const violations: string[] = [];
const readErrors: string[] = [];
for (const file of files) {
let sourceText: string;
try {
sourceText = await readFile(file, 'utf8');
} catch (error) {
const detail = error instanceof Error ? error.message : String(error);
readErrors.push(`ERROR: cannot read input file ${file}: ${detail}`);
continue;
}
const sourceFile = ts.createSourceFile(
file,
sourceText,
ts.ScriptTarget.Latest,
true,
ts.ScriptKind.TS,
);
const lines = sourceText.split(/\r?\n/);
const markerLines = new Set<number>();
if (sourceFile.parseDiagnostics.length > 0) {
const diagnostics = sourceFile.parseDiagnostics
.map((diagnostic) => ts.flattenDiagnosticMessageText(diagnostic.messageText, ' '))
.join('; ');
readErrors.push(`ERROR: cannot parse input file ${file}: ${diagnostics}`);
}
for (let markerPos = sourceText.indexOf(MARKER); markerPos >= 0; markerPos = sourceText.indexOf(MARKER, markerPos + MARKER.length)) {
const before = Array.from(sourceText.slice(0, markerPos)).at(-1);
const after = Array.from(sourceText.slice(markerPos + MARKER.length))[0];
const standaloneMarker = (!before || !MARKER_TOKEN_CHAR.test(before))
&& (!after || !MARKER_TOKEN_CHAR.test(after));
const token = ts.getTokenAtPosition(sourceFile, markerPos);
const insideToken = token.getStart(sourceFile) <= markerPos && markerPos < token.end;
if (standaloneMarker && !insideToken) {
markerLines.add(sourceFile.getLineAndCharacterOfPosition(markerPos).line);
}
}
function visit(node: ts.Node): void {
if (ts.isCallExpression(node) && node.expression.kind === ts.SyntaxKind.ImportKeyword) {
const { line } = sourceFile.getLineAndCharacterOfPosition(node.expression.getStart(sourceFile));
const sourceLine = lines[line] ?? '';
if (!markerLines.has(line)) {
violations.push(` ${file}:${line + 1}:${sourceLine}`);
}
}
ts.forEachChild(node, visit);
}
visit(sourceFile);
}
for (const error of readErrors) console.error(error);
if (violations.length > 0) {
console.error('ERROR: unreviewed dynamic import on an engine-live path:');
console.error();
console.error(violations.join('\n'));
console.error();
console.error('Prefer a static top-level import. If lazy loading is load-bearing,');
console.error("append 'engine-dynamic-import-ok' to that exact line and document");
console.error('the startup or soft-failure boundary that requires it.');
process.exit(1);
}
if (readErrors.length > 0) process.exit(1);
console.log(`check-engine-dynamic-import: ok (${files.length} file(s) scanned)`);
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env node
/**
* Import an envelope-v0 file (a JSON serialization of AI chat history; format
* spec: github.com/memvelope/memvelope) into a brain repo as one Markdown page
* per conversation, which `gbrain sync` ingests.
*
* Usage:
* node scripts/envelope-to-gbrain.mjs <envelope.mve.json> [outDir]
*
* Zero dependencies. Deterministic. No network. It does NOT call gbrain it
* only writes Markdown files.
*
* Output layout:
* - One page per conversation, filename = date + conversation id (shared
* titles cannot collide; the id is the natural key). A duplicate id
* overwrites its own filename and warns on stderr; stdout reports DISTINCT
* files written, not write calls.
* - Frontmatter: `type: conversation` (keeps pages eligible for
* conversation-facts extraction and chronicle behavior after sync), the
* source provider, the conversation id, and `origin: memvelope/envelope-v0`.
* - Page `date` is the first 10 chars of the conversation's ISO-8601
* `created_at`. Body keeps message-id citations beside each speaker turn.
*
* Memory: the whole envelope is held in memory (no streaming); envelopes are
* far smaller than the vendor exports they serialize.
*
* Verify:
* node scripts/envelope-to-gbrain.mjs test/fixtures/memvelope/sample.mve.json /tmp/out
* -> expect "wrote 1 markdown page(s)"
* bun test test/envelope-to-gbrain.test.ts
*
* STATUS: live-verified against gbrain v0.42.56.0 on 2026-07-03: the sample
* fixture -> 1 page; a real 662MB Claude export -> 353 conversations = 353
* distinct pages (no collisions), searchable after sync with provenance and
* message-id citations intact.
*/
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
const [, , envelopePath, outDir = './brain/conversations'] = process.argv;
if (!envelopePath) {
console.error('usage: node envelope-to-gbrain.mjs <envelope.mve.json> [outDir]');
process.exit(1);
}
const env = JSON.parse(readFileSync(envelopePath, 'utf8'));
if (env.memvelope !== 'envelope-v0') {
console.error(`not an envelope-v0 file (memvelope field = ${JSON.stringify(env.memvelope)})`);
process.exit(1);
}
const slug = (s, fallback) =>
(String(s || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '') || fallback).slice(0, 60);
mkdirSync(outDir, { recursive: true });
const filesWritten = new Set();
let collisions = 0;
const conversations = env.conversations || [];
for (const [i, c] of conversations.entries()) {
const date = (c.created_at || '').slice(0, 10);
// Name the file by the conversation's own id — the natural unique key — so two
// conversations that share a date and title can never silently overwrite each
// other. The date only leads as a human/chronological sort prefix; the id
// carries uniqueness. Positional fallback keeps names unique and deterministic
// when an envelope omits an id.
// One predicate for "this conversation carries its own id", shared by the
// filename and the frontmatter below. Keeping it in a single place is what
// stops the two from disagreeing about whether an id exists.
const hasId = typeof c.id === 'string' && c.id.trim() !== '';
const convId = hasId ? c.id.trim() : `conv-${i + 1}`;
const name = `${date || '0000-00-00'}-${slug(convId, `conv-${i + 1}`)}.md`;
// gbrain reads YAML frontmatter + markdown body; keep provenance in frontmatter.
// Emit `type: conversation` so gbrain stores these as conversation pages rather
// than defaulting to the generic `concept`. gbrain is open-typed — it takes an
// explicit frontmatter `type` verbatim — and its conversation-aware features
// (conversation-facts extraction, the conversation_format_coverage check,
// chronicle eligibility) key off `type == 'conversation'`.
const front = [
'---',
'type: conversation',
`title: ${JSON.stringify(c.title || 'Untitled conversation')}`,
`date: ${date || 'null'}`,
// Every interpolated value is quoted. An envelope is a third-party file, so
// a provider string carrying a newline would otherwise close this scalar and
// inject arbitrary frontmatter keys into the page gbrain ingests.
`source: ${JSON.stringify(env.meta?.source_provider || 'unknown')}`,
// Omit the key entirely when the envelope carries no id, rather than
// emitting the literal `undefined` or a synthesized `conv-N` — the positional
// fallback names the file, but it is not a memvelope conversation id and
// must not be recorded as one.
...(hasId ? [`memvelope_conversation_id: ${JSON.stringify(convId)}`] : []),
'origin: memvelope/envelope-v0',
'---',
'',
].join('\n');
const body = (c.messages || [])
.map((m) => `**${m.role === 'user' ? 'Me' : 'Assistant'}** (${m.ts || 'no timestamp'} · ${m.id}):\n\n${m.text}`)
.join('\n\n---\n\n');
// Never lose a page silently: if two conversations still map to the same
// filename (e.g. an envelope carrying duplicate ids), warn loudly instead of
// overwriting in silence, and report the count of DISTINCT files written — not
// the number of write calls, which is what hid the old title-collision bug.
if (filesWritten.has(name)) {
collisions += 1;
console.warn(`warning: filename collision on "${name}" — conversation id ${JSON.stringify(c.id)} is not unique; overwriting the earlier page.`);
}
writeFileSync(join(outDir, name), front + `# ${c.title || 'Conversation'}\n\n` + body + '\n');
filesWritten.add(name);
}
console.log(`wrote ${filesWritten.size} markdown page(s) to ${outDir} — point gbrain's sync at this directory.`);
if (collisions) {
console.warn(`warning: ${collisions} filename collision(s) — ${collisions} page(s) overwritten. Deduplicate conversation ids in the envelope to avoid data loss.`);
}
+3 -2
View File
@@ -162,8 +162,9 @@ for f in "${files[@]}"; do
if [ -n "${DATABASE_URL:-}" ]; then
psql "$DATABASE_URL" -At -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND datname = current_database()" >/dev/null 2>&1 || true
fi
# Hard outer timeout (180s per file). bun's --timeout is per-test; if a
# PGLite WASM call hangs in beforeAll/afterAll, --timeout never fires and
# Hard outer timeout (180s per file). bun's --timeout covers tests AND
# hooks (measured on 1.3.14), but it's timer-based: a PGLite WASM call
# that blocks the event loop synchronously never lets the timer fire and
# the file wedges indefinitely. gtimeout/timeout SIGKILLs the file so the
# suite advances. gtimeout (macOS via coreutils) preferred; timeout (Linux)
# fallback; bare bun (no outer cap) if neither is installed.
+1
View File
@@ -64,6 +64,7 @@ CHECKS=(
"check:source-scope-onboard"
"check:no-double-retry"
"check:batch-audit-site"
"check:engine-dynamic-import"
"check:worker-lock-renewal-shape"
"typecheck"
)
+2 -2
View File
@@ -139,9 +139,9 @@ edits writes a new receipt).
| Slot | Default | Provider |
|------|---------|----------|
| A | `openai:gpt-4o` | OpenAI |
| A | `openai:gpt-5.2` | OpenAI |
| B | `anthropic:claude-opus-4-7` | Anthropic |
| C | `google:gemini-1.5-pro` | Google |
| C | `deepseek:deepseek-v4-pro` | DeepSeek |
**These MUST be frontier models from DIFFERENT providers.** Using a single
provider's family or budget models defeats the purpose — different families
+38 -10
View File
@@ -1707,14 +1707,12 @@ async function handleCliOnly(command: string, args: string[]) {
// Per-command default: search 30s, sources list 10s. User --timeout=Ns wins.
// Other commands (import, embed, doctor, etc.) keep their existing
// unbounded connect — destructive / long-running commands shouldn't get
// a default kill switch.
const readOnlyDefaultTimeoutMs =
command === 'search' ? 30_000 :
command === 'sources' && (args[0] === 'list' || args[0] === undefined) ? 10_000 :
null;
// a default kill switch. The gate below is per-command (#3013): only the
// commands dispatchReadOnlyCommand handles may enter this path — a
// user-supplied --timeout on a write command must never reroute it here.
const cliOptsResolved = getCliOptions();
const userTimeoutMs = cliOptsResolved.timeoutMs;
const readOnlyTimeoutMs = userTimeoutMs ?? readOnlyDefaultTimeoutMs;
const readOnlyTimeoutMs = resolveReadOnlyDispatchTimeoutMs(command, args, userTimeoutMs);
if (readOnlyTimeoutMs !== null) {
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
@@ -2253,16 +2251,24 @@ async function handleCliOnly(command: string, args: string[]) {
//
// v0.30.1: still works; canonical entrypoint is now `gbrain backfill
// effective_date`. This command stays as a thin alias for back-compat.
//
// #1963: pass the already-connected engine. The command used to build
// + connect its OWN engine here, which self-deadlocked on the PGLite
// data-dir lock (this process already holds it via connectEngine
// above) — 30s spin, then exit 1, on every PGLite invocation.
const { reindexFrontmatterCli } = await import('./commands/reindex-frontmatter.ts');
await reindexFrontmatterCli(args);
return; // reindexFrontmatterCli handles its own engine lifecycle
await reindexFrontmatterCli(engine, args);
break;
}
case 'backfill': {
// v0.30.1: first-class generic backfill command. Subcommand dispatch
// is inside runBackfillCommand (kind | list | --help).
// #1963: same double-connect class as reindex-frontmatter — reuse the
// connected engine instead of building a second one on the same
// PGLite data dir.
const { runBackfillCommand } = await import('./commands/backfill.ts');
await runBackfillCommand(args);
return;
await runBackfillCommand(engine, args);
break;
}
case 'code-callers': {
// v0.20.0 Cathedral II Layer 10 (C4): "who calls <symbol>?"
@@ -2305,6 +2311,28 @@ async function handleCliOnly(command: string, args: string[]) {
}
}
/**
* #3013: decide whether an invocation enters the read-only connect+dispatch
* timeout path, and with what wallclock. Returns null for every command
* dispatchReadOnlyCommand can't handle. The gate used to be "a timeout is
* present" so a user-supplied --timeout on a write command (`sync`,
* `embed`, `import`, ...) hijacked dispatch into the read-only path, which
* threw and exited 1 before any work ran. Pure; exported for the
* regression test.
*/
export function resolveReadOnlyDispatchTimeoutMs(
command: string,
subArgs: string[],
userTimeoutMs: number | null,
): number | null {
if (command !== 'search' && command !== 'sources') return null;
const defaultMs =
command === 'search' ? 30_000 :
(subArgs[0] === 'list' || subArgs[0] === undefined) ? 10_000 :
null;
return userTimeoutMs ?? defaultMs;
}
/**
* v0.41.6.0 D3: dispatch helper for the read-only commands that take a
* default wallclock timeout (`gbrain search`, `gbrain sources list`).
+34 -1
View File
@@ -1,9 +1,42 @@
import { defaultTimeoutMsFor } from '../core/minions/handler-timeouts.ts';
// #2781: the full-cycle floor used to be a literal `1_800_000` that merely
// HAPPENED to match the 'autopilot-cycle' / 'autopilot-global-maintenance'
// handler anchors (`HANDLER_DEFAULT_TIMEOUT_MS`, #1737) instead of being
// derived from them. A duplicated literal can silently drift from the
// handler default it's supposed to track — which is exactly the bug class
// #2781 reported (an explicit `timeout_ms` stamp permanently overrides the
// handler default per `queue.ts`'s `opts?.timeout_ms ?? defaultTimeoutMsFor`,
// so a stale/lower literal here would starve a phase the handler default
// was sized for). Deriving the floor from `defaultTimeoutMsFor` for both
// full-cycle job names keeps the stamp coupled to its anchor by construction.
// Fail fast (not `?? 0`) if either handler ever loses its entry in
// HANDLER_DEFAULT_TIMEOUT_MS — silently falling back to "no floor" would
// reintroduce #2781 rather than surface the drift.
function requireHandlerAnchorMs(jobName: string): number {
const ms = defaultTimeoutMsFor(jobName);
if (ms === null) {
throw new Error(
`resolveAutopilotDispatchTimeoutMs: '${jobName}' has no entry in HANDLER_DEFAULT_TIMEOUT_MS ` +
'(handler-timeouts.ts) — the full-cycle timeout floor can no longer be derived from it. ' +
'See #2781: a missing/removed anchor here silently reintroduces the interval-derived stamp ' +
'permanently overriding the handler default.',
);
}
return ms;
}
const FULL_CYCLE_TIMEOUT_FLOOR_MS = Math.max(
requireHandlerAnchorMs('autopilot-cycle'),
requireHandlerAnchorMs('autopilot-global-maintenance'),
);
export function resolveAutopilotDispatchTimeoutMs(
baseIntervalSeconds: number,
fullCycle: boolean,
): number {
const intervalDerivedTimeoutMs = Math.max(baseIntervalSeconds * 2 * 1000, 300_000);
return fullCycle
? Math.max(intervalDerivedTimeoutMs, 1_800_000)
? Math.max(intervalDerivedTimeoutMs, FULL_CYCLE_TIMEOUT_FLOOR_MS)
: intervalDerivedTimeoutMs;
}
+13 -4
View File
@@ -981,12 +981,21 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// can't shrink throughput (codex #9/D5). autopilot-cycle jobs run on
// the 'default' queue, so that's the concurrency we compare against.
const fanoutMax = await resolveEffectiveFanoutMax(engine, 'default');
// #2781: both 'autopilot-cycle' (per-source) and 'autopilot-global-
// maintenance' carry a 30-min handler anchor (handler-timeouts.ts)
// because a full cycle can outlive short daemon intervals — unlike
// the lighter interval-derived `timeoutMs` above (sync/freshness,
// extract-atoms-drain, targeted small-plan steps), which have no
// such anchor and are meant to stay interval-derived. Naming this
// separately (rather than reusing the outer `timeoutMs`) avoids
// the #2781 bug class: dispatchGlobalMaintenance previously reused
// the outer non-full-cycle `timeoutMs` by shorthand, silently
// dropping its own handler anchor.
const fullCycleTimeoutMs = resolveAutopilotDispatchTimeoutMs(baseInterval, true);
const result = await dispatchPerSource(engine, queue, {
repoPath,
slot,
// Full cycles can outlive short daemon intervals. Keep lighter dispatches
// interval-derived while giving per-source consolidation enough time.
timeoutMs: resolveAutopilotDispatchTimeoutMs(baseInterval, true),
timeoutMs: fullCycleTimeoutMs,
fanoutMax,
jsonMode,
});
@@ -997,7 +1006,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// the per-source path (legacy single-source still runs everything).
if (!result.legacy_fallback) {
try {
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs, jsonMode });
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs: fullCycleTimeoutMs, jsonMode });
} catch (e) {
if (jsonMode) process.stderr.write(JSON.stringify({ event: 'global_maintenance_dispatch_failed', error: e instanceof Error ? e.message : String(e) }) + '\n');
}
+9 -13
View File
@@ -16,10 +16,10 @@
* always reserving 1 connection for HNSW + heartbeat + doctor probes.
*/
import type { BrainEngine } from '../core/engine.ts';
import { resolveDirectPoolSize } from '../core/connection-manager.ts';
import { listBackfills, getBackfill } from '../core/backfill-registry.ts';
import { runBackfill, clearBackfillCheckpoint } from '../core/backfill-base.ts';
import { loadConfig, toEngineConfig } from '../core/config.ts';
interface BackfillArgs {
kind?: string;
@@ -114,7 +114,14 @@ function clampConcurrency(requested: number | undefined): { effective: number; w
return { effective: requested };
}
export async function runBackfillCommand(args: string[]): Promise<void> {
/**
* #1963 (same class as reindex-frontmatter): takes the ALREADY-CONNECTED
* engine from cli.ts's dispatch. Building a second engine here deadlocked on
* the PGLite data-dir lock (cli.ts's `connectEngine()` already holds it in
* this same process) every `gbrain backfill <kind>` on PGLite timed out
* after 30s. Engine lifecycle belongs to cli.ts's connect + teardown.
*/
export async function runBackfillCommand(engine: BrainEngine, args: string[]): Promise<void> {
const cli = parseArgs(args);
if (cli.help) { printHelp(); return; }
@@ -144,20 +151,10 @@ export async function runBackfillCommand(args: string[]): Promise<void> {
process.exit(2);
}
const config = loadConfig();
if (!config) {
console.error('No brain configured. Run: gbrain init');
process.exit(2);
}
// X5 admission control — clamp concurrency to direct-pool capacity.
const { effective: concurrency, warning } = clampConcurrency(cli.concurrency);
if (warning) console.warn(warning);
const { createEngine } = await import('../core/engine-factory.ts');
const engine = await createEngine(toEngineConfig(config));
await engine.connect(toEngineConfig(config));
if (cli.fresh) {
await clearBackfillCheckpoint(engine, reg.spec.name);
console.log(`Cleared checkpoint for backfill.${reg.spec.name}`);
@@ -192,7 +189,6 @@ export async function runBackfillCommand(args: string[]): Promise<void> {
if (result.cappedByMaxRows) console.log(` ⚠️ Capped by --max-rows; more remain.`);
if (result.cappedByErrors) console.log(` ⚠️ Capped by --max-errors at ${result.errors}.`);
await engine.disconnect();
if (result.cappedByErrors) process.exit(1);
}
+8 -1
View File
@@ -46,7 +46,14 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
}
console.log('GBrain config:');
for (const [k, v] of Object.entries(config)) {
const display = typeof v === 'string' ? redactConfigValue(k, v) : v;
// #575: objects interpolated into the template literal printed
// `[object Object]` — render them as JSON instead. Sensitive keys
// stay redacted whether the value is a string or an object.
const display = typeof v === 'string'
? redactConfigValue(k, v)
: v !== null && typeof v === 'object'
? (isSensitiveConfigKey(k) ? '***' : JSON.stringify(v))
: v;
console.log(` ${k}: ${display}`);
}
return;
+108
View File
@@ -0,0 +1,108 @@
/**
* #1835 storage_path resolution for the doctor `image_assets` check.
*
* `files.storage_path` rows written by a Windows gbrain install carry Windows
* drive paths (`D:/foo/img.jpg`, `D:\foo\img.jpg`). On POSIX,
* `path.isAbsolute()` is false for those, so the old code joined them onto the
* repo root and produced a path that can never exist a false-positive
* "missing from disk, restore from git" WARN under WSL and macOS.
*
* Policy:
* - win32: drive paths are absolute; stat them as-is.
* - WSL (linux + "microsoft" in /proc/version): translate `D:/x` to
* `<automount root>/d/x` (automount root read from /etc/wsl.conf
* `[automount] root`, default `/mnt`) and stat that.
* - any other POSIX host (macOS, plain Linux): the path is unresolvable on
* this platform report it as foreign so the caller SKIPS the stat
* instead of inventing a path that will never exist.
*
* Kept in its own module (not doctor.ts) so the pure tests don't pull the
* 7k-line doctor dep graph, and so open PRs rewriting the image_assets block
* (e.g. a `resolveImageAssetPath` helper) can adopt it with a one-line call.
*/
import { readFileSync } from 'node:fs';
import { join, posix, win32 } from 'node:path';
const WINDOWS_DRIVE_RE = /^([A-Za-z]):[\\/](.*)$/;
export interface AssetPathResolution {
/** Absolute path to stat, or null when the path is unresolvable here. */
abs: string | null;
/** True when storage_path is a Windows drive path this host cannot stat. */
foreign: boolean;
}
/**
* Resolve a files.storage_path to a stat-able absolute path.
* `opts.platform` / `opts.wslMountRoot` exist for tests; production callers
* pass neither (process.platform + detected WSL automount root).
* `wslMountRoot: null` means "not under WSL".
*/
export function resolveAssetPath(
storagePath: string,
repoRoot: string,
opts: { platform?: NodeJS.Platform; wslMountRoot?: string | null } = {},
): AssetPathResolution {
const platform = opts.platform ?? process.platform;
if (platform !== 'win32') {
const m = WINDOWS_DRIVE_RE.exec(storagePath);
if (m) {
const root = opts.wslMountRoot !== undefined ? opts.wslMountRoot : detectWslMountRoot();
if (root === null) return { abs: null, foreign: true };
const abs = `${root.replace(/\/+$/, '')}/${m[1].toLowerCase()}/${m[2].replace(/\\/g, '/')}`;
return { abs, foreign: false };
}
}
// Platform-appropriate absoluteness (not the host's) so injected-platform
// tests behave identically everywhere; in production platform === host.
const isAbs = platform === 'win32' ? win32.isAbsolute(storagePath) : posix.isAbsolute(storagePath);
return {
abs: isAbs ? storagePath : join(repoRoot, storagePath),
foreign: false,
};
}
/**
* Extract the `[automount] root` value from /etc/wsl.conf content.
* Defaults to `/mnt` (WSL's own default) when absent/unparseable.
*/
export function parseWslAutomountRoot(conf: string): string {
let inAutomount = false;
for (const raw of conf.split(/\r?\n/)) {
const line = raw.replace(/[#;].*$/, '').trim();
if (line.startsWith('[')) {
inAutomount = /^\[automount\]$/i.test(line);
continue;
}
if (!inAutomount) continue;
const m = /^root\s*=\s*"?([^"]+?)"?\s*$/.exec(line);
if (m) return m[1];
}
return '/mnt';
}
let cachedWslMountRoot: string | null | undefined;
/**
* Detect the WSL Windows-drive automount root. Returns null when not running
* under WSL (including macOS and plain Linux). Memoized per process.
*/
export function detectWslMountRoot(): string | null {
if (cachedWslMountRoot === undefined) cachedWslMountRoot = computeWslMountRoot();
return cachedWslMountRoot;
}
function computeWslMountRoot(): string | null {
if (process.platform !== 'linux') return null;
try {
// The standard WSL tell: kernel version string names Microsoft.
if (!/microsoft/i.test(readFileSync('/proc/version', 'utf8'))) return null;
} catch {
return null;
}
try {
return parseWslAutomountRoot(readFileSync('/etc/wsl.conf', 'utf8'));
} catch {
return '/mnt'; // WSL default when wsl.conf is absent.
}
}
+66 -9
View File
@@ -4349,8 +4349,18 @@ export async function checkCycleFreshness(
: `'${source.id}'`;
const raw = source.config?.last_full_cycle_at;
if (typeof raw !== 'string') {
// #2540: WARN, not FAIL. This check iterates EVERY local_path source,
// so on a multi-source install where only some vaults are cycled
// (e.g. one nightly `gbrain dream --dir <vault>`), a never-cycled
// sibling source turned doctor permanently red — which erodes the
// check's signal until real staleness hides inside the noise (the
// reporter's install masked genuinely stale sources for weeks this
// way). "Never cycled" also fires on a source added minutes ago.
// A source that HAS cycled and then went stale still escalates
// through the warn/fail age thresholds below — that is the
// regression signal this check exists for.
issues.push(`Source ${display} has never completed a full cycle`);
hasFailures = true;
hasWarnings = true;
continue;
}
const last = new Date(raw).getTime();
@@ -4386,7 +4396,7 @@ export async function checkCycleFreshness(
return {
name: 'cycle_freshness',
status: 'warn',
message: `${issues.join('; ')}.`,
message: `${issues.join('; ')}. Run \`gbrain dream --source <id>\` to cycle a source, or start \`gbrain autopilot\`.`,
};
}
return {
@@ -5592,6 +5602,42 @@ export async function buildChecks(
// Best-effort filesystem-hygiene check; never block doctor.
}
// 3f. npm_squat (#505). The npm registry name `gbrain` belongs to an
// unrelated third-party package — this project is NOT distributed on npm.
// A reflexive `npm i -g gbrain` / `bun add -g gbrain` installs something
// unrelated that can shadow the real binary on PATH. Classify every
// `gbrain` that `which -a` finds (pure helpers in
// src/core/npm-squat-check.ts) and warn when an unrelated install wins on
// PATH or the entry is broken. Skips silently when gbrain isn't on PATH
// at all (e.g. running via `bun src/cli.ts`).
try {
const { execSync } = await import('node:child_process');
let candidates: string[] = [];
try {
candidates = execSync('which -a gbrain', {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'ignore'],
})
.split('\n')
.map((s) => s.trim())
.filter(Boolean);
} catch {
// `which` exits non-zero when gbrain isn't on PATH (or is missing
// entirely on this platform) — nothing to check.
}
const { assessGbrainBinaries } = await import('../core/npm-squat-check.ts');
const assessment = assessGbrainBinaries(candidates);
if (assessment.status !== 'skip') {
checks.push({
name: 'npm_squat',
status: assessment.status,
message: assessment.message,
});
}
} catch {
// Best-effort environment check; never block doctor.
}
// 3b-multi-source. Multi-source drift (v0.31.8 — D8 + D17 + OV12 + OV13).
// Pre-v0.30.3 putPage misrouted multi-source writes to (default, slug).
// For each non-default source with local_path set, walk the FS and surface
@@ -7536,33 +7582,44 @@ export async function buildChecks(
`SELECT storage_path FROM files WHERE mime_type LIKE 'image/%' LIMIT 1000`
);
let vanished = 0;
let foreign = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
const { resolveAssetPath } = await import('./doctor-asset-paths.ts');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
// #1835: Windows drive paths (D:/…) translate to the WSL automount
// (/mnt/d/…) under WSL, and are SKIPPED (not "missing") on hosts
// where they cannot exist (macOS / plain Linux) — never joined onto
// repoRoot, which produced a false "restore from git" WARN.
const resolved = resolveAssetPath(r.storage_path, repoRoot);
if (resolved.abs === null) {
foreign++;
continue;
}
try {
fs.statSync(abs);
fs.statSync(resolved.abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
}
}
const checked = rows.length - foreign;
const foreignNote = foreign > 0
? ` (${foreign} Windows-drive path(s) skipped — not resolvable on this platform)`
: '';
if (rows.length === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: 'No image assets indexed yet' });
} else if (vanished === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: `${rows.length} image(s) all present on disk` });
checks.push({ name: 'image_assets', status: 'ok', message: `${checked} image(s) all present on disk${foreignNote}` });
} else {
checks.push({
name: 'image_assets',
status: 'warn',
message: `${vanished} of ${rows.length} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')}). ` +
message: `${vanished} of ${checked} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')})${foreignNote}. ` +
`Fix: restore from git, or \`gbrain sync --skip-failed\` to acknowledge.`,
});
}
+1 -1
View File
@@ -78,7 +78,7 @@ FLAGS:
cycle is 3 model calls; verdict aggregates over them.
--slot-a-model <id> Override default 'openai:gpt-5.2'.
--slot-b-model <id> Override default 'anthropic:claude-opus-4-7'.
--slot-c-model <id> Override default 'google:gemini-1.5-pro'.
--slot-c-model <id> Override default 'deepseek:deepseek-v4-pro'.
--receipt-dir <path> Default: gbrainPath('eval-receipts').
--max-tokens N Output token budget per call. Default: 4000.
--json Emit final aggregate as JSON to stdout (progress to stderr).
+7 -1
View File
@@ -349,7 +349,13 @@ function inferTypeByDir(fromDir: string, toDir: string, frontmatter?: Record<str
const to = toDir.split('/')[0];
if (from === 'people' && to === 'companies') {
if (Array.isArray(frontmatter?.founded)) return 'founded';
return 'works_at';
// #3466: bare people/ -> companies/ adjacency is not evidence of
// employment, so it gets the neutral 'mentions' verb instead of
// 'works_at'. Real works_at edges still come from the two paths that
// read actual evidence: the company:/companies: frontmatter fields
// (FRONTMATTER_LINK_MAP) and employment phrasing in prose
// (inferLinkType in link-extraction.ts).
return 'mentions';
}
if (from === 'people' && to === 'deals') return 'involved_in';
if (from === 'deals' && to === 'companies') return 'deal_for';
+1 -1
View File
@@ -42,7 +42,7 @@ interface FeatureScanResult {
const RECIPE_META = [
{ id: 'email-to-brain', name: 'Email to Brain', secrets: ['GMAIL_APP_PASSWORD'] },
{ id: 'calendar-to-brain', name: 'Calendar Sync', secrets: ['GOOGLE_CALENDAR_API_KEY'] },
{ id: 'x-to-brain', name: 'X/Twitter to Brain', secrets: ['X_BEARER_TOKEN'] },
{ id: 'x-to-brain', name: 'X/Twitter to Brain', secrets: ['X_API_BEARER_TOKEN'] },
{ id: 'twilio-voice-brain', name: 'Voice to Brain', secrets: ['TWILIO_AUTH_TOKEN'] },
{ id: 'meeting-sync', name: 'Meeting Sync', secrets: ['CIRCLEBACK_API_KEY'] },
{ id: 'credential-gateway', name: 'Credential Gateway', secrets: ['OAUTH_CLIENT_SECRET'] },
+10 -2
View File
@@ -16,7 +16,7 @@ interface FileRecord {
filename: string;
storage_path: string;
mime_type: string | null;
size_bytes: number;
size_bytes: number | bigint | string | null;
content_hash: string;
metadata: Record<string, unknown>;
created_at: string;
@@ -42,6 +42,14 @@ function fileHash(filePath: string): string {
return createHash('sha256').update(content).digest('hex');
}
export function formatFileSizeKb(rawSizeBytes: number | bigint | string | null): string {
if (rawSizeBytes == null) return '?';
const sizeBytes = Number(rawSizeBytes);
return Number.isFinite(sizeBytes) && sizeBytes >= 0
? `${Math.round(sizeBytes / 1024)}KB`
: '?';
}
export async function runFiles(engine: BrainEngine, args: string[]) {
const subcommand = args[0];
@@ -116,7 +124,7 @@ async function listFiles(engine: BrainEngine, slug?: string) {
console.log(`${rows.length} file(s):`);
for (const row of rows) {
const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?';
const size = formatFileSizeKb(row.size_bytes as FileRecord['size_bytes']);
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
}
}
+29 -7
View File
@@ -23,7 +23,8 @@ import matter from 'gray-matter';
import { readFileSync, existsSync, writeFileSync, mkdirSync, readdirSync } from 'fs';
import { join, basename } from 'path';
import { homedir } from 'os';
import { gbrainPath } from '../core/config.ts';
import { gbrainPath, loadConfig } from '../core/config.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { execSync } from 'child_process';
// --- Types ---
@@ -122,9 +123,28 @@ export function isUnsafeHealthCheck(check: string): boolean {
return /[;&|`$(){}\\<>\n]/.test(check);
}
/** Expand $VAR references with process.env values */
/**
* Env view for secret resolution (#2789): apply the same config.jsonenv
* folding the runtime applies via buildGatewayConfig, so a credential stored
* only in ~/.gbrain/config.json which powers a perfectly healthy
* integration is not reported [missing] by show/status. process.env still
* wins for non-empty values (buildGatewayConfig spreads it last, dropping
* only ''/undefined entries). Falls back to bare process.env before
* `gbrain init` (no config file yet). Mirrors the #2728 fix on the
* providers command.
*/
export function secretEnv(): Record<string, string | undefined> {
try {
const cfg = loadConfig();
if (cfg) return buildGatewayConfig(cfg).env;
} catch { /* integrations must keep working pre-init — fall through */ }
return process.env;
}
/** Expand $VAR references with gateway-env (config-folded) values */
export function expandVars(s: string): string {
return s.replace(/\$([A-Z_][A-Z0-9_]*)/g, (_, name) => process.env[name] || '');
const env = secretEnv();
return s.replace(/\$([A-Z_][A-Z0-9_]*)/g, (_, name) => env[name] || '');
}
// --- SSRF Protection ---
@@ -249,7 +269,7 @@ export async function executeHealthCheck(
}
case 'env_exists': {
const val = process.env[check.name];
const val = secretEnv()[check.name];
return {
...base,
status: val ? 'ok' : 'fail',
@@ -457,11 +477,12 @@ function readHeartbeat(id: string): HeartbeatEntry[] {
// --- Secret Checking ---
function checkSecrets(secrets: RecipeSecret[]): { set: string[]; missing: RecipeSecret[] } {
export function checkSecrets(secrets: RecipeSecret[]): { set: string[]; missing: RecipeSecret[] } {
const set: string[] = [];
const missing: RecipeSecret[] = [];
const env = secretEnv();
for (const s of secrets) {
if (process.env[s.name]) {
if (env[s.name]) {
set.push(s.name);
} else {
missing.push(s);
@@ -607,8 +628,9 @@ function cmdShow(args: string[]): void {
if (f.requires.length > 0) console.log(`Requires: ${f.requires.join(', ')}`);
console.log('\nSecrets needed:');
const env = secretEnv();
for (const s of f.secrets) {
const isSet = process.env[s.name] ? ' [set]' : ' [missing]';
const isSet = env[s.name] ? ' [set]' : ' [missing]';
console.log(` ${s.name}${isSet}`);
console.log(` ${s.description}`);
console.log(` Get it: ${s.where}`);
+21 -34
View File
@@ -151,8 +151,17 @@ export async function runReindexFrontmatter(
};
}
/** CLI entrypoint. Argv shape matches reindex-code for consistency. */
export async function reindexFrontmatterCli(args: string[]): Promise<void> {
/**
* CLI entrypoint. Argv shape matches reindex-code for consistency.
*
* #1963: takes the ALREADY-CONNECTED engine from cli.ts's dispatch instead of
* building its own. The old self-managed `createEngine()+connect()` here was a
* same-process double-connect: cli.ts's `connectEngine()` already held the
* PGLite data-dir lock, so the second `connect()` spun the full 30s lock
* timeout waiting on its own process and the command always exited 1 on
* PGLite. The engine lifecycle (connect + teardown) belongs to cli.ts.
*/
export async function reindexFrontmatterCli(engine: BrainEngine, args: string[]): Promise<void> {
const opts: ReindexFrontmatterOpts = {};
for (let i = 0; i < args.length; i++) {
const a = args[i];
@@ -173,37 +182,15 @@ export async function reindexFrontmatterCli(args: string[]): Promise<void> {
}
}
const { createEngine } = await import('../core/engine-factory.ts');
const { loadConfig, toEngineConfig } = await import('../core/config.ts');
const cfg = loadConfig();
if (!cfg) {
console.error('No gbrain config; run `gbrain init` first.');
process.exit(1);
}
const engineConfig = toEngineConfig(cfg);
const engine = await createEngine(engineConfig);
// v0.37.7.0 #1225: createEngine() only constructs; callers MUST connect
// before any executeRaw call. Pre-fix, the first query in countAffected
// crashed with "PGLite not connected. Call connect() first." even on
// --dry-run. initSchema is idempotent on a current schema, costs ~1ms.
await engine.connect(engineConfig);
await engine.initSchema();
try {
const result = await runReindexFrontmatter(engine, opts);
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
const noun = result.status === 'dry_run' ? 'would update' : 'updated';
console.error(
`\nReindex ${result.status}: examined=${result.examined} ${noun}=${result.updated} ` +
`fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`,
);
}
if (result.status === 'cancelled') process.exit(1);
} finally {
if ('disconnect' in engine && typeof engine.disconnect === 'function') {
await engine.disconnect();
}
const result = await runReindexFrontmatter(engine, opts);
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
const noun = result.status === 'dry_run' ? 'would update' : 'updated';
console.error(
`\nReindex ${result.status}: examined=${result.examined} ${noun}=${result.updated} ` +
`fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`,
);
}
if (result.status === 'cancelled') process.exit(1);
}
+90 -5
View File
@@ -12,6 +12,7 @@
import express from 'express';
import type { Request, Response, NextFunction } from 'express';
import type { Server as HttpServer } from 'http';
import cookieParser from 'cookie-parser';
import cors from 'cors';
import rateLimit from 'express-rate-limit';
@@ -46,6 +47,7 @@ import {
type IngestionEvent,
} from '../core/ingestion/types.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
import { registerCleanup } from '../core/process-cleanup.ts';
/**
* /health endpoint timeout. 3s rather than 5s: Fly.io's default
@@ -55,6 +57,71 @@ import { resolveOwnerHolder } from '../core/owner-holder.ts';
*/
export const HEALTH_TIMEOUT_MS = 3000;
/** Exported so tests can type their structural fakes exactly (#3599). */
export type HttpServerLifecycle = Pick<HttpServer, 'listening' | 'once' | 'off' | 'close'>;
/** Exported so tests can type their structural fakes exactly (#3599). */
export type SignalSource = Pick<NodeJS.Process, 'once' | 'off'>;
type CleanupRegistrar = typeof registerCleanup;
/**
* Keep the HTTP server strongly referenced and make the daemon lifetime
* explicit instead of relying on runtime-specific event-loop behavior for an
* unobserved `app.listen()` return value. The shared abnormal-termination
* cleanup pass closes it before process exit.
*/
export function waitForHttpServerLifecycle(
server: HttpServerLifecycle,
options: {
signals?: SignalSource;
register?: CleanupRegistrar;
} = {},
): Promise<void> {
const signals = options.signals ?? process;
const register = options.register ?? registerCleanup;
return new Promise<void>((resolve, reject) => {
let settled = false;
let closePromise: Promise<void> | null = null;
const closeServer = (): Promise<void> => {
if (closePromise) return closePromise;
closePromise = new Promise<void>((closeResolve, closeReject) => {
if (!server.listening) {
closeResolve();
return;
}
server.close((error?: Error) => {
if (error) closeReject(error);
else closeResolve();
});
});
return closePromise;
};
const deregister = register('http-server', closeServer);
const finish = (error?: Error) => {
if (settled) return;
settled = true;
server.off('close', onClose);
server.off('error', onError);
signals.off('SIGINT', onSigint);
deregister();
if (error) reject(error);
else resolve();
};
const onClose = () => finish();
const onError = (error: Error) => finish(error);
const onSigint = () => {
void closeServer().catch(onError);
};
server.once('close', onClose);
server.once('error', onError);
signals.once('SIGINT', onSigint);
});
}
/**
* v0.36.1.x #1024: bootstrap token resolution.
*
@@ -135,6 +202,25 @@ export type ProbeHealthResult =
| { ok: true; status: 200; body: { status: 'ok'; version: string; engine: string; [k: string]: unknown } }
| { ok: false; status: 503; body: { error: 'service_unavailable'; error_description: string } };
/** Exported so tests can type their structural fakes exactly (#3598). */
export type AdminSseResponse = Pick<Response, 'setHeader' | 'flushHeaders' | 'write'>;
/**
* Complete the admin EventSource handshake immediately.
*
* `flushHeaders()` alone can leave reverse proxies and browsers waiting for
* the first response body bytes. An SSE comment is protocol-valid, ignored by
* EventSource consumers, and makes the stream observable end-to-end without
* fabricating an application event.
*/
export function openAdminSseStream(res: AdminSseResponse): void {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
res.write(': connected\n\n');
}
/**
* Pure async health probe. Races `engine.getStats()` against a timeout,
* returns a tagged result. No Express coupling easy to unit-test with a
@@ -1632,10 +1718,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// SSE live activity feed
// ---------------------------------------------------------------------------
app.get('/admin/events', requireAdmin, (req: Request, res: Response) => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
openAdminSseStream(res);
sseClients.add(res);
req.on('close', () => sseClients.delete(res));
@@ -2410,7 +2493,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// ---------------------------------------------------------------------------
const clientCount = await sql`SELECT count(*)::int as count FROM oauth_clients`;
app.listen(port, bind, () => {
const httpServer = app.listen(port, bind, () => {
console.error(`
GBrain MCP Server v${VERSION.padEnd(37)}
@@ -2435,4 +2518,6 @@ ${bootstrapFromEnv
: `║ Admin Token (paste into /admin login): ║\n║ ${bootstrapToken.substring(0, 50)}\n║ ${bootstrapToken.substring(50).padEnd(50)}\n╚══════════════════════════════════════════════════════╝`}
`);
});
await waitForHttpServerLifecycle(httpServer);
}
+81 -4
View File
@@ -2874,10 +2874,17 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
: await resolveSlugByPathOrSourcePath(engine, from, undefined);
// The new path doesn't yet have a row, so resolve from path only.
const newSlug = resolveSlugForPath(to);
// #3056: the cheap rename is OBSERVED, not assumed. A zero-row UPDATE
// doesn't throw, and a thrown collision used to be swallowed by an
// empty catch — both fell through to importFile, which created/updated
// the row at the new path while the old row stayed behind live. Both
// shapes now fall through to the reconcile below.
let renameApplied = false;
try {
await engine.updateSlug(oldSlug, newSlug, renameOpts);
renameApplied = (await engine.updateSlug(oldSlug, newSlug, renameOpts)) > 0;
} catch {
// Slug doesn't exist or collision, treat as add
// Destination slug occupied or invalid — treat as add; the reconcile
// below removes the stale old row once the destination materialized.
}
// Reimport at new path (picks up content changes). Wrapped to match the
// deletes/adds loops: a malformed renamed file is recorded to failedFiles
@@ -2890,9 +2897,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// NAV-1 TOCTOU: refuse a destination that realpath-resolves outside the
// repo (committed symlink pointing out).
const filePath = join(gitContextRoot, to);
let importResult: Awaited<ReturnType<typeof importFile>> | undefined;
if (existsSync(filePath) && isPathSafe(filePath, gitContextRoot)) {
try {
const result = await importFile(engine, filePath, to, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack });
importResult = result;
if (result.status === 'imported') chunksCreated += result.chunks;
else if (result.status === 'skipped' && (result as { error?: string }).error) {
failedFiles.push({ path: to, error: String((result as { error?: string }).error) });
@@ -2901,9 +2910,68 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
failedFiles.push({ path: to, error: e instanceof Error ? e.message : String(e) });
}
}
// #3056 reconcile: the rename fell back to add semantics, so the row
// that still represents the OLD path is the stale half of the rename
// (git reported the old path gone; a plain delete of that path would
// remove this row). Two safety rails, both from the #3252 review:
//
// 1. Delete only after the destination demonstrably materialized —
// `imported`, or an errorless `skipped` AT the new slug. Identity
// dedup can skip against the OLD row (result.slug === oldSlug),
// in which case nothing landed at newSlug and deleting the old
// row would destroy the only copy.
// 2. Locate the stale row POSITIVELY by `source_path = from`, never
// by the oldSlug guess — after a collision, a path-derived
// fallback slug could name an unrelated (e.g. manually curated)
// row. No source_path match → nothing is deleted (this also means
// code-strategy imports, which don't populate source_path, fall
// back safely to leaving the old row rather than guessing).
//
// A failed delete records a `<rename:…>` SENTINEL (not an ordinary
// path failure): the gate hard-blocks the bookmark, and — unlike a
// plain path row — the auto-skip valve can never chronic-skip it after
// N attempts, which would advance the bookmark and make a transient
// delete outage a permanent duplicate. The sentinel clears through the
// ordinary success path once the rename converges on a later run.
let reconcileFailed = false;
if (!renameApplied && importResult !== undefined) {
const destMaterialized = importResult.status === 'imported' ||
(importResult.status === 'skipped' && !importResult.error && importResult.slug === newSlug);
if (destMaterialized) {
try {
const staleMap = await engine.resolveSlugsByPaths([from], { sourceId: opts.sourceId ?? DEFAULT_SOURCE_ID });
const staleSlug = staleMap.get(from);
if (staleSlug !== undefined && staleSlug !== newSlug) {
await engine.deletePage(staleSlug, renameOpts);
deletedSlugs.add(staleSlug); // never hand a deleted slug to auto-embed
serr(` [sync] rename reconciled: removed stale row ${staleSlug} (${from} -> ${to} fell back to add).`);
} else if (staleSlug === undefined) {
serr(` [sync] rename fallback: no row has source_path ${from}; stale row (if any) left in place.`);
}
} catch (e: unknown) {
reconcileFailed = true;
failedFiles.push({
path: `<rename:${to}>`,
error: `rename reconcile failed (stale row for ${from} not removed): ` +
`${e instanceof Error ? e.message : String(e)}`,
});
}
} else {
serr(
` [sync] rename fallback: ${from} -> ${to} did not materialize at ${newSlug} ` +
`(import ${importResult.status}); old row left in place.`,
);
}
}
// Converged (cheap rename, clean reconcile, or nothing to reconcile):
// clear any `<rename:…>` sentinel a previous failing run recorded.
if (!reconcileFailed) succeededPaths.push(`<rename:${to}>`);
pagesAffected.push(newSlug);
deletedSlugs.delete(newSlug); // #1284: rename landed on a previously-deleted slug → embeddable again
await markCompleted(to);
// A failed reconcile must NOT checkpoint: banking `to` would make the
// resume filter skip this rename on the retry run, turning a transient
// delete failure into a permanent duplicate — the exact bug being fixed.
if (!reconcileFailed) await markCompleted(to);
progress.tick(1, newSlug);
}
progress.finish();
@@ -3362,7 +3430,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
if (!gate.advanced) {
const codeBreakdown = formatCodeBreakdown(failedFiles);
if (gate.sentinelBlocked) {
// Two sentinel classes block here: `<head>` (pin ancestry broken) and
// `<rename:…>` (#3056 — a rename-reconcile delete failed and advancing
// would permanently bank the duplicate). Pick the message by which fired.
if (gate.sentinelBlocked && failedFiles.some(f => f.path === '<head>')) {
serr(
`\nSync blocked: repository history changed during sync (force-push / reset).\n` +
`${codeBreakdown}\n\n` +
@@ -3370,6 +3441,12 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
`a commit that doesn't match the indexed tree. Re-run sync to re-pin against ` +
`current HEAD.`,
);
} else if (gate.sentinelBlocked) {
serr(
`\nSync blocked: a rename left a stale duplicate that could not be removed:\n` +
`${codeBreakdown}\n\n` +
`The next 'gbrain sync' retries the reconcile from the same diff.`,
);
} else {
const fileFailCount = failedFiles.filter(f => isSkippablePath(f.path)).length;
serr(
+34 -7
View File
@@ -9,7 +9,7 @@
* import it from `../../src/cli.ts`.
*
* The single ownership site for: (a) folding file-plane API keys
* (openai/anthropic/zeroentropy/openrouter/voyage) into the gateway env, and (b) threading
* (openai/anthropic/zeroentropy/openrouter/voyage/dashscope/google) into the gateway env, and (b) threading
* local-server `*_BASE_URL` env vars into base_urls. Both matter for the
* init-time embedding-key probe without (a) it would false-warn on
* config.json-keyed users, and without (b) a live probe could hit the wrong
@@ -44,6 +44,18 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
// multimodal/image embeds despite config.json looking complete. process.env
// still wins via the later spread.
if (c.voyage_api_key) envFromConfig.VOYAGE_API_KEY = c.voyage_api_key;
// #3500: same seam for DashScope. The dashscope + dashscope-rerank recipes
// require DASHSCOPE_API_KEY, but the config-plane key was never folded, so
// daemon/launchd/MCP contexts with no process-env export failed auth
// despite config.json looking complete. process.env still wins via the
// later spread.
if (c.dashscope_api_key) envFromConfig.DASHSCOPE_API_KEY = c.dashscope_api_key;
// #3500: same seam for Google Gemini. The google recipe reads
// GOOGLE_GENERATIVE_AI_API_KEY; before this fold, the ONLY way to
// configure Gemini was exporting that exact env var. (This closes the
// deferral noted in src/core/brain-score-recommendations.ts, whose
// HOSTED_EMBED_KEY_CONFIG entry lands in the same change.)
if (c.google_api_key) envFromConfig.GOOGLE_GENERATIVE_AI_API_KEY = c.google_api_key;
// Azure OpenAI (keyless/Entra): fold the non-secret endpoint/deployment + the
// Entra opt-in into the gateway env so the azure-openai recipe works in any
// shell (incl. non-interactive agent shells). The bearer token is minted at
@@ -86,11 +98,26 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
// every gateway op then throws NO_ANTHROPIC_API_KEY. Drop empty-string /
// undefined entries before the merge. Only '' and undefined are dropped —
// '0' and 'false' are legitimate values and survive.
env: {
...envFromConfig,
...Object.fromEntries(
Object.entries(process.env).filter(([, v]) => v !== undefined && v !== ''),
),
},
env: buildEnv(envFromConfig),
};
}
/**
* Merge config-plane fallbacks with process.env (env wins for keys carrying a
* real value see #1249 note above), then apply the GEMINI_API_KEY alias:
* Google's own docs/SDKs export GEMINI_API_KEY, but the google recipe (and
* every gateway read site) uses GOOGLE_GENERATIVE_AI_API_KEY. Precedence:
* env GOOGLE_GENERATIVE_AI_API_KEY > env GEMINI_API_KEY > config
* google_api_key i.e. the alias is still process-env, so it beats the
* config-plane fallback, but never the canonical env name.
*/
function buildEnv(envFromConfig: Record<string, string>): Record<string, string> {
const envReal = Object.fromEntries(
Object.entries(process.env).filter(([, v]) => v !== undefined && v !== ''),
) as Record<string, string>;
const merged = { ...envFromConfig, ...envReal };
if (!envReal.GOOGLE_GENERATIVE_AI_API_KEY && envReal.GEMINI_API_KEY) {
merged.GOOGLE_GENERATIVE_AI_API_KEY = envReal.GEMINI_API_KEY;
}
return merged;
}
+16 -2
View File
@@ -16,6 +16,17 @@ export const google: Recipe = {
dims_options: [768, 1536, 3072],
cost_per_1m_tokens_usd: 0.15,
price_last_verified: '2026-04-20',
// Gemini's embedding endpoint has a low per-request cap relative to
// Voyage. Declaring max_batch_tokens makes the gateway pre-split bulk
// batches proactively (splitByTokenBudget) instead of relying solely on
// the recursive-halving retry on a token-limit rejection. Conservative
// value: each gemini-embedding-001 input tops out at 2048 tokens, so a
// 20k budget × 0.8 safety keeps a batch well within request limits while
// staying efficient. chars_per_token ~4 matches Gemini's SentencePiece
// density on English. Tunable; recursion stays the backstop.
max_batch_tokens: 20_000,
chars_per_token: 4,
safety_factor: 0.8,
},
expansion: {
models: ['gemini-2.0-flash', 'gemini-2.0-flash-lite'],
@@ -23,11 +34,14 @@ export const google: Recipe = {
price_last_verified: '2026-04-20',
},
chat: {
models: ['gemini-2.0-flash-exp', 'gemini-2.0-flash', 'gemini-1.5-pro'],
// gemini-1.5-pro was retired by Google (#3510) — deliberately NOT
// listed. Default-slot guard tests validate hardcoded defaults against
// this list, so re-adding a dead model here masks dead defaults.
models: ['gemini-2.0-flash-exp', 'gemini-2.0-flash'],
supports_tools: true,
supports_subagent_loop: true,
supports_prompt_cache: false,
max_context_tokens: 1000000, // Gemini 1.5 Pro
max_context_tokens: 1000000, // Gemini 2.0 Flash
cost_per_1m_input_usd: 0.30,
cost_per_1m_output_usd: 1.20,
price_last_verified: '2026-04-20',
+16 -2
View File
@@ -59,10 +59,24 @@ const ALL: Recipe[] = [
/** Map from `provider:id` key to recipe. */
export const RECIPES: Map<string, Recipe> = new Map(ALL.map(r => [r.id, r]));
/**
* Test-only seam. Synthetic recipes appended to the registry so tests can
* exercise registry-walking logic notably gateway.ts's missing-batch-cap
* startup warning against a recipe that intentionally omits a field,
* without editing the shipped `ALL` array. Every real embedding recipe now
* declares a cap (token budget, `no_batch_cap`, or item cap), so a synthetic
* cap-less recipe is the only way to cover the warn-fires path. Empty in
* production (nothing in `src/` calls the setter); pass `[]` to reset.
*/
let _testRecipes: Recipe[] = [];
export function __setTestRecipesForTests(recipes: Recipe[]): void {
_testRecipes = recipes;
}
export function getRecipe(id: string): Recipe | undefined {
return RECIPES.get(id);
return RECIPES.get(id) ?? _testRecipes.find(r => r.id === id);
}
export function listRecipes(): Recipe[] {
return [...ALL];
return _testRecipes.length > 0 ? [...ALL, ..._testRecipes] : [...ALL];
}
+4 -2
View File
@@ -37,8 +37,10 @@ export const voyage: Recipe = {
'voyage-multimodal-3',
],
default_dims: 1024,
cost_per_1m_tokens_usd: 0.18,
price_last_verified: '2026-04-20',
// Display hint for `gbrain providers` only (billing math goes through
// src/core/embedding-pricing.ts). Rate for the default voyage-4-large.
cost_per_1m_tokens_usd: 0.12,
price_last_verified: '2026-07-28',
// Voyage enforces 120K tokens per batch. Voyage's tokenizer runs
// ~3-4× denser than OpenAI tiktoken on mixed content (code/JSON/CJK),
// so the per-recipe pre-split uses 1 char ≈ 1 token at 0.5 utilization
+7 -1
View File
@@ -37,7 +37,13 @@ export function readRecentParserProbeEvents(
days = 7,
now: Date = new Date(),
): ParserProbeAuditEvent[] {
return writer.readRecent(days, now);
// Chronological order (oldest → newest). The shared reader walks the
// CURRENT week's file first, then the previous week's, so without sorting
// the array tail is the OLDEST in-window event whenever last week's file
// has entries — and doctor's "latest" (which reads the tail) reported a
// days-old run while counts included the newest one.
return writer.readRecent(days, now)
.sort((a, b) => Date.parse(a.ts) - Date.parse(b.ts));
}
/** Exposed for tests pinning the rotation edge cases. */
+6 -1
View File
@@ -118,5 +118,10 @@ export function readRecentQualityProbeEvents(
}
}
}
return out;
// Chronological order (oldest → newest). Events accumulate across two
// week files read current-week-FIRST, so without sorting the array tail
// is the OLDEST in-window event whenever last week's file has entries —
// and doctor's "Latest:" (which reads the tail) reported a days-old run
// while the counts included the newest one.
return out.sort((a, b) => Date.parse(a.ts) - Date.parse(b.ts));
}
+6 -8
View File
@@ -13,17 +13,13 @@ import { parseModelId } from './ai/model-resolver.ts';
*
* Only keys that `buildGatewayConfig` (src/core/ai/build-gateway-config.ts)
* actually folds from config into the gateway env may appear here.
* GOOGLE_GENERATIVE_AI_API_KEY is deliberately absent: its config field is NOT
* threaded to the gateway today, so the producer closures fall through to
* checking `process.env` ONLY for it. That matches what the gateway can
* actually use (the recipe reads that key from env). Counting a config-plane
* google_api_key here would be a false positive: doctor/autopilot would call
* the provider "configured" and dispatch an embed.stale job that then fails
* auth at the gateway. When a future change threads google_api_key into
* buildGatewayConfig, re-add the matching entry here in the same change.
*
* VOYAGE_API_KEY voyage_api_key was the same kind of gap (#2662) until
* buildGatewayConfig started folding it now safe to list here too.
* GOOGLE_GENERATIVE_AI_API_KEY google_api_key and DASHSCOPE_API_KEY
* dashscope_api_key joined for the same reason (#3500): both are folded by
* buildGatewayConfig now, so a config-plane key is genuinely usable by the
* gateway and counting it here is no longer a false positive.
*
* Caveat inherited from the existing OPENAI_API_KEY/ZEROENTROPY_API_KEY
* entries (unchanged by #2662, noted here for anyone extending this map):
@@ -40,6 +36,8 @@ export const HOSTED_EMBED_KEY_CONFIG: Record<string, string> = {
OPENAI_API_KEY: 'openai_api_key',
ZEROENTROPY_API_KEY: 'zeroentropy_api_key',
VOYAGE_API_KEY: 'voyage_api_key',
GOOGLE_GENERATIVE_AI_API_KEY: 'google_api_key',
DASHSCOPE_API_KEY: 'dashscope_api_key',
};
/**
+72 -20
View File
@@ -51,9 +51,29 @@ export const DEFAULT_CLI_OPTIONS: CliOptions = {
*
* Unknown flags are passed through unchanged per-command parsers see them.
*/
/**
* #3013: commands that parse their own `--timeout` flag out of argv.
* `sync` reads a seconds-based graceful-abort budget (src/commands/sync.ts +
* resolveSyncHardDeadline); `remote` reads a ms-based request budget
* (src/commands/remote.ts). For these commands the global parser must hand
* the flag back: claiming it stripped the flag before the per-command parser
* could read it, and for `sync` a non-null global timeoutMs flipped the
* read-only dispatch gate in cli.ts, rerouting a write command into
* dispatchReadOnlyCommand (exit 1 before any work ran).
*/
export const TIMEOUT_OWNING_COMMANDS = new Set(['sync', 'remote']);
export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: string[] } {
const cliOpts: CliOptions = { ...DEFAULT_CLI_OPTIONS };
const rest: string[] = [];
// #3013: --timeout can't be resolved inline — whether the GLOBAL parser
// claims it depends on which command is running, and the command token is
// only known once the whole argv has been scanned (global flags may precede
// it). The scan collects positional slots; --timeout slots are resolved in
// a second pass below.
type Slot =
| { plain: string }
| { timeoutValue: string; equalsForm: boolean };
const slots: Slot[] = [];
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
@@ -74,7 +94,7 @@ export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: s
continue;
}
// not a number — let per-command parser handle; pass through
rest.push(a);
slots.push({ plain: a });
continue;
}
if (a.startsWith('--progress-interval=')) {
@@ -84,29 +104,20 @@ export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: s
cliOpts.progressInterval = parsed;
continue;
}
rest.push(a);
slots.push({ plain: a });
continue;
}
// v0.31.1: --timeout=Ns or --timeout Ns. Accepts plain ms, "30s", "2m".
if (a === '--timeout' && i + 1 < argv.length) {
const next = argv[i + 1];
const parsed = parseTimeout(next);
if (parsed !== null) {
cliOpts.timeoutMs = parsed;
i++;
continue;
}
rest.push(a);
// A following token that is itself a flag is NOT a value — leave it for
// its own iteration (pre-#3013 behavior: an unparseable next token was
// never consumed).
if (a === '--timeout' && i + 1 < argv.length && !argv[i + 1].startsWith('-')) {
slots.push({ timeoutValue: argv[i + 1], equalsForm: false });
i++;
continue;
}
if (a.startsWith('--timeout=')) {
const val = a.slice('--timeout='.length);
const parsed = parseTimeout(val);
if (parsed !== null) {
cliOpts.timeoutMs = parsed;
continue;
}
rest.push(a);
slots.push({ timeoutValue: a.slice('--timeout='.length), equalsForm: true });
continue;
}
// v0.40.4 — --explain for `gbrain search/query` per-stage attribution.
@@ -114,9 +125,50 @@ export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: s
cliOpts.explain = true;
continue;
}
rest.push(a);
slots.push({ plain: a });
}
// The command is the first plain token (matches `command = rest[0]` in
// cli.ts). If it owns --timeout, every --timeout is handed back in the
// space-separated spelling (the only form the owning parsers read; this
// also normalizes `--timeout=60s`), value verbatim so the owning command
// applies its own unit + validity rules (`sync`: bare integers are
// SECONDS, `ms`/fractional rejected loudly; `remote` accepts `h`).
// Handed-back flags are APPENDED after every other token: both owning
// commands treat leading args as positional subcommands (`sync trigger`,
// `remote ping`) and locate --timeout by scanning args, so appending can't
// shadow a subcommand while duplicate flags keep their argv order (the
// owning parsers' first-occurrence-wins precedence matches what the user
// typed). Non-owning commands keep the pre-#3013 global behavior:
// parseable values are claimed into cliOpts.timeoutMs (last one wins),
// unparseable ones pass through in their original spelling for the
// per-command parser.
const commandSlot = slots.find((s): s is { plain: string } => 'plain' in s);
const commandOwnsTimeout =
commandSlot !== undefined && TIMEOUT_OWNING_COMMANDS.has(commandSlot.plain);
const rest: string[] = [];
const handback: string[] = [];
for (const s of slots) {
if ('plain' in s) {
rest.push(s.plain);
continue;
}
if (commandOwnsTimeout) {
handback.push('--timeout', s.timeoutValue);
continue;
}
const parsed = parseTimeout(s.timeoutValue);
if (parsed !== null) {
cliOpts.timeoutMs = parsed;
} else if (s.equalsForm) {
rest.push(`--timeout=${s.timeoutValue}`);
} else {
rest.push('--timeout', s.timeoutValue);
}
}
rest.push(...handback);
return { cliOpts, rest };
}
+19
View File
@@ -63,6 +63,23 @@ export interface GBrainConfig {
* config.json file-plane route is wired through today.
*/
voyage_api_key?: string;
/**
* Alibaba DashScope API key (#3500). File-plane slot so config.json's
* `dashscope_api_key` reaches the dashscope / dashscope-rerank recipes:
* file plane buildGatewayConfig env dict recipe reads
* DASHSCOPE_API_KEY. Same fold pattern (and same DB-plane caveat) as
* voyage_api_key above.
*/
dashscope_api_key?: string;
/**
* Google Gemini API key (#3500). File-plane slot folded into the gateway
* env as GOOGLE_GENERATIVE_AI_API_KEY (the name the google recipe reads).
* buildGatewayConfig also accepts process-env GEMINI_API_KEY the name
* Google's own docs/SDKs use as an alias for
* GOOGLE_GENERATIVE_AI_API_KEY. Same fold pattern (and same DB-plane
* caveat) as voyage_api_key above.
*/
google_api_key?: string;
/** Azure OpenAI (keyless/Entra). Non-secret endpoint + deployment + Entra opt-in,
* folded into the gateway env so the azure-openai recipe works in any shell.
* The bearer token is minted at request time via `az` no secret stored here. */
@@ -919,6 +936,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'zeroentropy_api_key',
'openrouter_api_key',
'voyage_api_key',
'dashscope_api_key',
'google_api_key',
'azure_openai_endpoint',
'azure_openai_deployment',
'azure_openai_use_entra',
+5 -1
View File
@@ -51,7 +51,11 @@ export const DEFAULT_SLOTS: SlotConfig[] = [
// 2-model quorum without a Google key (verdict: permanently inconclusive).
{ id: 'A', model: 'openai:gpt-5.2' },
{ id: 'B', model: 'anthropic:claude-opus-4-7' },
{ id: 'C', model: 'google:gemini-1.5-pro' },
// gemini-1.5-pro was retired by Google (#3510), so slot C failed even with
// a Google key configured. deepseek:deepseek-v4-pro preserves the
// three-distinct-provider contract with a model registered in both the
// recipe and canonical pricing tables (same replacement as PR #3501).
{ id: 'C', model: 'deepseek:deepseek-v4-pro' },
];
export interface SlotConfig {
+11 -2
View File
@@ -895,8 +895,17 @@ export async function resolveSourceForDir(
// (the cycleSourceId precedence) or 'default'.
if (brainDir === null) return undefined;
try {
// #2540: exclude archived rows (dream's --source guard refuses to stamp
// them, so an archived alias winning here means the stamp silently never
// lands and doctor's cycle_freshness stays red on a healthy install) and
// order deterministically so a duplicate registration of the same path
// can't shadow the active source on whichever row the engine scans first.
// Ordering matches listAllSources/sources-ops for operator-output parity.
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE local_path = $1 LIMIT 1`,
`SELECT id FROM sources
WHERE local_path = $1 AND archived = false
ORDER BY (id = 'default') DESC, id
LIMIT 1`,
[brainDir],
);
if (rows[0]) return rows[0].id;
@@ -2432,7 +2441,7 @@ export async function runCycle(
try {
const { runSchemaSuggestPhase } = await import('./cycle/schema-suggest.ts');
const { result, duration_ms } = await timePhase(async () => {
const r = await runSchemaSuggestPhase(engine, { dryRun: !!opts.dryRun });
const r = await runSchemaSuggestPhase(engine, { sourceId: cycleSourceId, dryRun: !!opts.dryRun });
return {
phase: 'schema-suggest' as const,
status: (r.skipped ? 'skipped' : 'ok') as PhaseStatus,
+34 -6
View File
@@ -482,24 +482,52 @@ export async function runPhaseExtractAtoms(
}
// 3. Dual-source merge: transcripts + pages, dedup by contentHash.
// Transcripts win on collision (origin attribution stays with the
// raw transcript file even if the same content was later imported
// as a brain page).
// Transcripts win on COLLISION (origin attribution stays with the raw
// transcript file even if the same content was later imported as a
// brain page) — that's decided by the two loops below, which register
// every transcript hash into `seenHashes` before any page is checked,
// same as before this fix. It's independent of the FINAL work-item
// ORDER built after them.
//
// Order is page-item-first, interleaved 1-for-1 with transcripts (NOT
// concatenated transcripts-then-pages). The per-call budget cap (step
// 4 below) stops processing `work` in list order once
// budgetTracker.totalSpent >= budgetCap, skipping everything after
// that point. Two failure modes this avoids:
// - Concatenation (old code): a transcript corpus that alone
// exceeds the budget cap starves the page pool completely, no
// matter how many drain batches run.
// - Interleaving with transcripts first: still starves ALL pages
// whenever the budget only covers exactly one call (item 0 is a
// transcript, item 1 — the first page — never gets attempted).
// Pages are the ONLY pool `countExtractAtomsBacklog`/doctor's
// extract_atoms_backlog check measures (see that function's
// docstring), so page-first guarantees the doctor-visible backlog
// makes forward progress on every budget-capped call, however tight
// the cap — `--drain` can no longer report the same backlog number
// forever while atoms keep getting extracted from transcripts.
type WorkItem =
| { kind: 'transcript'; filePath: string; content: string; contentHash: string }
| { kind: 'page'; slug: string; content: string; contentHash: string };
const seenHashes = new Set<string>();
const work: WorkItem[] = [];
const transcriptItems: WorkItem[] = [];
for (const t of transcriptsLive) {
if (seenHashes.has(t.contentHash)) { duplicatesSkipped++; continue; }
seenHashes.add(t.contentHash);
work.push({ kind: 'transcript', ...t });
transcriptItems.push({ kind: 'transcript', ...t });
}
const pageItems: WorkItem[] = [];
for (const p of pages) {
if (seenHashes.has(p.contentHash)) { duplicatesSkipped++; continue; }
seenHashes.add(p.contentHash);
work.push({ kind: 'page', ...p });
pageItems.push({ kind: 'page', ...p });
}
const work: WorkItem[] = [];
const maxPoolLen = Math.max(transcriptItems.length, pageItems.length);
for (let i = 0; i < maxPoolLen; i++) {
if (i < pageItems.length) work.push(pageItems[i]);
if (i < transcriptItems.length) work.push(transcriptItems[i]);
}
// Phase-level no-op: nothing to extract today.
+1 -1
View File
@@ -219,7 +219,7 @@ export interface GradeTakesOpts extends BasePhaseOpts {
/**
* E2 ensemble judges. When useEnsemble=true and the single-model verdict
* is borderline, all three judges are called in parallel via Promise.allSettled.
* Defaults to [openai:gpt-4o, anthropic:claude-sonnet-4-6, google:gemini-1.5-pro]
* Defaults to [openai:gpt-5.2, anthropic:claude-sonnet-4-6, google:gemini-2.0-flash]
* via defaultJudge with model-string overrides. Tests inject deterministic
* judges.
*/
+1
View File
@@ -145,6 +145,7 @@ export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
'federation_health',
'home_dir_in_worktree',
'index_audit',
'npm_squat',
'oauth_confidential_client_health',
'orphan_clones',
'pgbouncer_prepare',
+22 -11
View File
@@ -5,12 +5,15 @@
* cost-estimate prompt so users with large brains see a dollar figure
* before the chunker-version sweep re-embeds.
*
* Prices in USD per 1M tokens. Numbers as of 2026-05-11. Verify alongside
* the Anthropic-pricing refresh cycle; drift here produces estimates
* that mislead operators.
* Prices in USD per 1M tokens. Every entry carries the official page it came
* from plus the date it was last read against that page re-verify alongside
* the Anthropic-pricing refresh cycle; drift here produces estimates that
* mislead operators. This table is for EMBEDDINGS only; chat/completion
* pricing lives in `model-pricing.ts` (different unit) and must never be
* mixed in here.
*
* Codex outside-voice C3 fold: non-OpenAI embedding providers (Voyage,
* Hunyuan, Dashscope, etc.) return UNKNOWN_PROVIDER from `lookupPrice`
* Codex outside-voice C3 fold: embedding providers with no entry below
* (Hunyuan, Dashscope, etc.) return UNKNOWN_PROVIDER from `lookupPrice`
* so the cost-estimate prompt can fall back to a "estimate unavailable
* for <provider>; press Ctrl-C in 10s to abort" message rather than
* fabricate numbers.
@@ -26,25 +29,33 @@ export interface EmbeddingPricing {
* gateway model strings (e.g. 'openai:text-embedding-3-large').
*/
export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
// OpenAI (https://openai.com/api/pricing/, verified 2026-05-11)
// OpenAI (https://developers.openai.com/api/docs/pricing, verified 2026-07-28)
'openai:text-embedding-3-large': { pricePerMTok: 0.13 },
'openai:text-embedding-3-small': { pricePerMTok: 0.02 },
// Legacy OpenAI ada (still common in older brains)
'openai:text-embedding-ada-002': { pricePerMTok: 0.10 },
// Voyage (https://www.voyageai.com/pricing)
// Voyage (https://docs.voyageai.com/docs/pricing, verified 2026-07-28)
'voyage:voyage-4-large': { pricePerMTok: 0.12 },
'voyage:voyage-4': { pricePerMTok: 0.06 },
'voyage:voyage-4-lite': { pricePerMTok: 0.02 },
// voyage-4-nano is deliberately absent: it's the open-weight variant (see
// src/core/ai/recipes/voyage.ts) and Voyage's pricing page lists no hosted
// rate for it. A 0 entry would under-estimate anyone paying for it via the
// hosted API; no entry means lookupEmbeddingPrice returns `unknown` and the
// caller prints "estimate unavailable" instead of a wrong number.
// Legacy Voyage models (same page, "older models" section — no free tokens):
'voyage:voyage-3-large': { pricePerMTok: 0.18 },
'voyage:voyage-3': { pricePerMTok: 0.06 },
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
// ZeroEntropy (https://www.zeroentropy.dev/pricing, verified 2026-07-28)
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
// ZeroEntropy reranker (docs/ai-providers/zeroentropy.md — $0.025/1M tokens).
// Reused here (not a separate rerank table) because budget-tracker.ts's
// rerank-kind lookup falls back to this same table for paid providers.
'zeroentropyai:zerank-2': { pricePerMTok: 0.025 },
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-19)
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-28)
'mistral:mistral-embed': { pricePerMTok: 0.10 },
'mistral:mistral-embed-2312': { pricePerMTok: 0.10 },
// Perplexity (https://docs.perplexity.ai/getting-started/pricing, verified 2026-07-21)
// Perplexity (https://docs.perplexity.ai/getting-started/pricing, verified 2026-07-28)
'perplexity:pplx-embed-v1-0.6b': { pricePerMTok: 0.004 },
'perplexity:pplx-embed-v1-4b': { pricePerMTok: 0.03 },
};
+6 -1
View File
@@ -1951,8 +1951,13 @@ export interface BrainEngine {
* preserved via stable page_id). `opts.sourceId` scopes the UPDATE without
* it, the bare `WHERE slug = old` matches every row across every source and
* would either rename them all OR violate the (source_id, slug) UNIQUE.
*
* Returns the number of rows moved. 0 means the old slug had no row in the
* scoped source an UPDATE that matches nothing does NOT throw, so callers
* that need to know whether the rename actually happened (the sync rename
* path, #3056) must check the return value rather than rely on the catch.
*/
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void>;
updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number>;
rewriteLinks(oldSlug: string, newSlug: string): Promise<void>;
/**
+49 -3
View File
@@ -39,6 +39,7 @@ import { normalizeAliasList } from './search/alias-normalize.ts';
import { isUndefinedTableError, warnOncePerProcess, validateSlug } from './utils.ts';
import { computeCorpusGeneration } from './contextual-retrieval-service.ts';
import { runGuardrails } from './guardrails.ts';
import { FACTS_FENCE_BEGIN, FACTS_FENCE_END, parseFactsFence } from './facts-fence.ts';
/**
* v0.20.0 Cathedral II Layer 8 D2 markdown fence extraction helper.
@@ -104,6 +105,27 @@ function fenceTagToPseudoPath(lang: string | undefined): string | null {
*/
const MAX_FENCES_PER_PAGE = Number.parseInt(process.env.GBRAIN_MAX_FENCES_PER_PAGE || '100', 10);
function extractFactsFenceBlock(body: string): string | null {
const beginIdx = body.indexOf(FACTS_FENCE_BEGIN);
if (beginIdx === -1) return null;
const endIdx = body.indexOf(FACTS_FENCE_END, beginIdx + FACTS_FENCE_BEGIN.length);
if (endIdx === -1) return null;
return body.slice(beginIdx, endIdx + FACTS_FENCE_END.length);
}
function replaceOrAppendFactsFence(body: string, fenceBlock: string): string {
const beginIdx = body.indexOf(FACTS_FENCE_BEGIN);
if (beginIdx !== -1) {
const endIdx = body.indexOf(FACTS_FENCE_END, beginIdx + FACTS_FENCE_BEGIN.length);
if (endIdx !== -1) {
return body.slice(0, beginIdx) + fenceBlock + body.slice(endIdx + FACTS_FENCE_END.length);
}
}
const sep = body.endsWith('\n') ? '\n' : '\n\n';
return `${body}${sep}## Facts\n\n${fenceBlock}\n`;
}
/**
* Walk the marked lexer output and extract recognizable code fences.
* Returns one ChunkInput per fence whose language tag maps to a grammar
@@ -548,6 +570,26 @@ export async function importFromContent(
// hash-match skip) and (b) the hash short-circuit below reuses this row.
const existing = await engine.getPage(slug, sourceId ? { sourceId } : undefined);
// #2044: remote get_page intentionally strips private facts rows. A
// documented get_page -> edit -> put_page round-trip can therefore arrive
// with an empty/missing Facts fence even though the existing page still has
// canonical fence rows. Preserve the old fence in that narrow case so the
// system-of-record markdown is not truncated by the privacy boundary.
if (opts.remote === true && existing?.compiled_truth) {
const incomingFacts = parseFactsFence(parsed.compiled_truth);
const existingFacts = parseFactsFence(existing.compiled_truth);
const existingFenceBlock = extractFactsFenceBlock(existing.compiled_truth);
if (
incomingFacts.facts.length === 0 &&
incomingFacts.warnings.length === 0 &&
existingFacts.warnings.length === 0 &&
existingFacts.facts.length > 0 &&
existingFenceBlock
) {
parsed.compiled_truth = replaceOrAppendFactsFence(parsed.compiled_truth, existingFenceBlock);
}
}
// #1035: absence of an explicit frontmatter `type:` on an EXISTING page
// means "preserve the stored type", not "re-infer". Pre-fix, a round-trip
// put (get_page → edit body → put_page without `type:`) silently regressed
@@ -1161,6 +1203,10 @@ export async function importCodeFile(
const title = `${relativePath} (${lang})`;
const sourceId = opts.sourceId;
const txOpts = sourceId ? { sourceId } : undefined;
// PostgreSQL text columns reject U+0000 even though source files may
// legitimately contain it inside string/regex fixtures. Preserve a visible,
// searchable representation instead of dropping the entire code page.
const storageContent = content.replaceAll('\0', '\\0');
const byteLength = Buffer.byteLength(content, 'utf-8');
if (byteLength > MAX_FILE_SIZE) {
@@ -1202,7 +1248,7 @@ export async function importCodeFile(
// from the chunker (nested methods carry ['ClassName'] etc.) so the
// chunk-grain FTS trigger picks up scope for ranking and downstream
// Layer 5 edge resolution can use scope-qualified identity.
const { chunks: codeChunks, edges: extractedEdges } = await chunkCodeTextFull(content, relativePath);
const { chunks: codeChunks, edges: extractedEdges } = await chunkCodeTextFull(storageContent, relativePath);
const chunks: ChunkInput[] = codeChunks.map((c, i) => ({
chunk_index: i,
chunk_text: c.text,
@@ -1270,7 +1316,7 @@ export async function importCodeFile(
type: 'code' as string,
page_kind: 'code',
title,
compiled_truth: content,
compiled_truth: storageContent,
timeline: '',
frontmatter: { language: lang, file: relativePath },
content_hash: hash,
@@ -1342,7 +1388,7 @@ export async function importCodeFile(
const edgeInputs: import('./types.ts').CodeEdgeInput[] = [];
for (const e of extractedEdges) {
const idx = findChunkForOffset(e.callSiteByteOffset, content, rangeList);
const idx = findChunkForOffset(e.callSiteByteOffset, storageContent, rangeList);
if (idx == null) continue;
const from = rangeList[idx]!;
if (!from.id || !from.symbol_name_qualified) continue;
+5 -4
View File
@@ -28,10 +28,11 @@ import { ensureWellFormed } from './text-safe.ts';
* OR updated_at > links_extracted_at`. It is an ISO-8601 string (NOT a number) —
* the column is TIMESTAMPTZ and the predicate binds it as `::timestamptz`.
*/
// 2026-07-10: bumped for the #2576 --stale nullResolver fix — sweeps before it
// stamped pages with their bare wikilinks silently dropped; the bump re-flags
// them so the fixed sweep re-extracts.
export const LINK_EXTRACTOR_VERSION_TS = '2026-07-10T00:00:00Z';
// 2026-07-30: bumped for the #3466 inferTypeByDir fix — unevidenced
// people/ -> companies/ adjacency now infers 'mentions' instead of
// 'works_at'; the bump re-flags stamped pages so the next --stale sweep
// re-extracts them under the corrected inference.
export const LINK_EXTRACTOR_VERSION_TS = '2026-07-30T00:00:00Z';
// ─── Entity references ──────────────────────────────────────────
+17 -125
View File
@@ -2,6 +2,13 @@ import type { BrainEngine } from './engine.ts';
import { slugifyPath } from './sync.ts';
import { getFtsLanguage } from './fts-language.ts';
import { hnswMaxDimsForType } from './vector-index.ts';
// runMigrations executes while an initialized engine is live. Keep its helper
// modules in the static graph rather than importing them from async handlers.
import {
isStatementTimeoutError,
isRetryableConnError,
} from './retry-matcher.ts';
import { repairTimelineDedupIndex } from './timeline-dedup-repair.ts';
/**
* Schema migrations run automatically on initSchema().
@@ -539,18 +546,7 @@ export const MIGRATIONS: Migration[] = [
sql: '',
handler: async (engine) => {
if (engine.kind === 'postgres') {
await engine.runMigration(
14,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'idx_pages_updated_at_desc' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS idx_pages_updated_at_desc';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 14, 'idx_pages_updated_at_desc');
await engine.runMigration(
14,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_updated_at_desc
@@ -1656,18 +1652,7 @@ export const MIGRATIONS: Migration[] = [
// 3. Partial index for the autopilot purge sweep. Postgres CONCURRENTLY
// avoids the SHARE lock on `pages`; PGLite has no concurrent writers.
if (engine.kind === 'postgres') {
// Pre-drop any invalid index from a prior CONCURRENTLY failure (matches v14 pattern).
await engine.runMigration(34, `
DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_deleted_at_purge_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_deleted_at_purge_idx';
END IF;
END $$;
`);
await dropInvalidConcurrentIndex(engine, 34, 'pages_deleted_at_purge_idx');
await engine.runMigration(34, `
CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
@@ -2004,18 +1989,7 @@ export const MIGRATIONS: Migration[] = [
// 2. Expression index for since/until date-range filters.
if (engine.kind === 'postgres') {
// Pre-drop any invalid index from a prior CONCURRENTLY failure.
await engine.runMigration(38, `
DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_coalesce_date_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_coalesce_date_idx';
END IF;
END $$;
`);
await dropInvalidConcurrentIndex(engine, 38, 'pages_coalesce_date_idx');
await engine.runMigration(38, `
CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
@@ -3577,19 +3551,7 @@ export const MIGRATIONS: Migration[] = [
sql: '',
handler: async (engine) => {
if (engine.kind === 'postgres') {
// Pre-drop invalid remnant from a failed CONCURRENTLY attempt.
await engine.runMigration(
71,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'takes_resolved_at_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS takes_resolved_at_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 71, 'takes_resolved_at_idx');
await engine.runMigration(
71,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS takes_resolved_at_idx
@@ -4249,20 +4211,7 @@ export const MIGRATIONS: Migration[] = [
await engine.runMigration(91, columnsAndTrigger);
if (engine.kind === 'postgres') {
// Pre-drop any invalid index from a prior CONCURRENTLY failure
// (matches v14 pattern).
await engine.runMigration(
91,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_generation_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_generation_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 91, 'pages_generation_idx');
await engine.runMigration(
91,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_generation_idx ON pages (generation);`
@@ -4516,18 +4465,7 @@ export const MIGRATIONS: Migration[] = [
sql: '',
handler: async (engine) => {
if (engine.kind === 'postgres') {
await engine.runMigration(
96,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'idx_facts_extract_conversation_session' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS idx_facts_extract_conversation_session';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 96, 'idx_facts_extract_conversation_session');
await engine.runMigration(
96,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_facts_extract_conversation_session
@@ -4569,18 +4507,7 @@ export const MIGRATIONS: Migration[] = [
transaction: false,
handler: async (engine) => {
if (engine.kind === 'postgres') {
await engine.runMigration(
97,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_dedup_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_dedup_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 97, 'pages_dedup_idx');
await engine.runMigration(
97,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_dedup_idx
@@ -4748,18 +4675,7 @@ export const MIGRATIONS: Migration[] = [
);
if (engine.kind === 'postgres') {
await engine.runMigration(
103,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'content_chunks_stale_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS content_chunks_stale_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 103, 'content_chunks_stale_idx');
await engine.runMigration(
103,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS content_chunks_stale_idx
@@ -4793,18 +4709,7 @@ export const MIGRATIONS: Migration[] = [
sql: '',
handler: async (engine) => {
if (engine.kind === 'postgres') {
await engine.runMigration(
104,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_atom_source_hash_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_atom_source_hash_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 104, 'pages_atom_source_hash_idx');
await engine.runMigration(
104,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_atom_source_hash_idx
@@ -5105,18 +5010,7 @@ export const MIGRATIONS: Migration[] = [
`ALTER TABLE pages ADD COLUMN IF NOT EXISTS links_extracted_at TIMESTAMPTZ;`
);
if (engine.kind === 'postgres') {
await engine.runMigration(
112,
`DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_links_extracted_at_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_links_extracted_at_idx';
END IF;
END $$;`
);
await dropInvalidConcurrentIndex(engine, 112, 'pages_links_extracted_at_idx');
await engine.runMigration(
112,
`CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_links_extracted_at_idx
@@ -5801,7 +5695,6 @@ async function runMigrationSQLWithRetry(
m: Migration,
sql: string,
): Promise<void> {
const { isStatementTimeoutError, isRetryableConnError } = await import('./retry-matcher.ts');
// GBRAIN_MIGRATE_BACKOFF_MS lets tests skip the 5s/15s/45s backoff. In
// production the env var is unset and the default cadence applies.
const fastBackoff = process.env.GBRAIN_MIGRATE_BACKOFF_MS;
@@ -6071,7 +5964,6 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num
// reach the loop below). Best-effort + idempotent: a no-op on a healthy
// index; `doctor` surfaces it independently if this ever fails.
try {
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
const r = await repairTimelineDedupIndex(engine);
if (r.repaired) {
console.error(
+3
View File
@@ -84,6 +84,9 @@ export const CANONICAL_PRICING: Record<string, ModelPricing> = {
'openai:gpt-5.5': { input: 4.00, output: 16.00 },
// ── Google ─────────────────────────────────────────────────────────────
// `gemini-1.5-pro` was retired by Google (#3510); kept so historical
// usage/audit rows still price. Not a valid default — it's deliberately
// absent from the google recipe's chat list.
'google:gemini-1.5-pro': { input: 1.25, output: 5.00 },
// Gemini 2.0 Flash: $0.10 in / $0.40 out (verified 2026-06-03). Reconciled
// from a stale $0.30/$1.20 entry that had drifted in takes-quality-eval.
+191
View File
@@ -0,0 +1,191 @@
/**
* npm-squat-check classify `gbrain` binaries found on PATH (#505).
*
* The npm registry name `gbrain` belongs to an unrelated third-party package;
* this project is NOT distributed on npm. A reflexive `npm i -g gbrain` /
* `bun add -g gbrain` therefore installs something that is not this project
* and can shadow the real binary on PATH.
*
* Pure classification helpers (filesystem-only, no network, no shelling out)
* so `gbrain doctor` can warn with receipts. The caller supplies the candidate
* paths (typically the output of `which -a gbrain`).
*/
import { closeSync, openSync, readFileSync, readSync, realpathSync } from 'node:fs';
import { dirname, join } from 'node:path';
export type GbrainBinaryKind = 'real' | 'foreign' | 'broken' | 'unknown';
export interface ClassifiedGbrainBinary {
/** The candidate path as given (PATH entry / symlink). */
path: string;
kind: GbrainBinaryKind;
/** Human-readable evidence for the classification. */
detail: string;
}
export interface NpmSquatAssessment {
status: 'ok' | 'warn' | 'skip';
message: string;
binaries: ClassifiedGbrainBinary[];
}
/** Repository marker identifying this project's package.json. */
const REAL_REPO_MARKER = 'garrytan/gbrain';
/** The documented install/remediation path, reused in doctor output. */
export const NPM_SQUAT_REMEDIATION =
`Remove the unrelated package (\`bun remove -g gbrain\` or \`npm uninstall -g gbrain\`) ` +
`and install/upgrade only via the documented path: \`bun install -g github:${REAL_REPO_MARKER}\` ` +
`(or \`git clone https://github.com/${REAL_REPO_MARKER}.git && bun install && bun link\`).`;
/**
* A `bun build --compile` gbrain binary is a native executable, not a script.
* Sniff the magic bytes: ELF, Mach-O (thin + fat), PE.
*/
function isNativeExecutable(path: string): boolean {
let fd: number | undefined;
try {
fd = openSync(path, 'r');
const buf = Buffer.alloc(4);
if (readSync(fd, buf, 0, 4, 0) < 4) return false;
const be = buf.readUInt32BE(0);
const le = buf.readUInt32LE(0);
return (
be === 0x7f454c46 || // ELF
be === 0xcafebabe || be === 0xcafebabf || // fat Mach-O
le === 0xfeedface || le === 0xfeedfacf || // Mach-O 32/64
(buf[0] === 0x4d && buf[1] === 0x5a) // PE ("MZ")
);
} catch {
return false;
} finally {
if (fd !== undefined) closeSync(fd);
}
}
/** Walk up from `start` to the nearest parseable package.json. */
function nearestPackageJson(start: string): { dir: string; pkg: Record<string, any> } | null {
let cur = start;
for (let depth = 0; depth < 64; depth++) {
try {
const pkg = JSON.parse(readFileSync(join(cur, 'package.json'), 'utf8'));
if (pkg && typeof pkg === 'object') return { dir: cur, pkg };
} catch {
// Missing or unparseable at this level; keep walking.
}
const parent = dirname(cur);
if (parent === cur) break;
cur = parent;
}
return null;
}
/**
* Is this package.json THIS project? Two markers, either suffices:
* - repository field pointing at garrytan/gbrain (string or { url }), or
* - this repo's known bin shape (`"bin": { "gbrain": "src/cli.ts" }` a
* git checkout / `bun install -g github:...` install carries it verbatim;
* a registry-published package ships built JS, not a bare .ts bin).
*/
function isRealGbrainPackage(pkg: Record<string, any>): boolean {
const repo = typeof pkg.repository === 'string' ? pkg.repository : pkg.repository?.url;
if (typeof repo === 'string' && repo.includes(REAL_REPO_MARKER)) return true;
if (pkg.bin && typeof pkg.bin === 'object' && pkg.bin.gbrain === 'src/cli.ts') return true;
return false;
}
/**
* Classify one candidate `gbrain` path:
* - 'broken' : symlink that doesn't resolve / unreadable path.
* - 'real' : compiled gbrain binary, or a script whose nearest
* package.json is this project's (repo checkout / bun link /
* `bun install -g github:garrytan/gbrain`).
* - 'foreign' : nearest package.json is named "gbrain" but is NOT this
* project an unrelated registry install.
* - 'unknown' : can't tell (no gbrain package.json above the resolved file).
*/
export function classifyGbrainBinary(path: string): ClassifiedGbrainBinary {
let resolved: string;
try {
resolved = realpathSync(path);
} catch {
return { path, kind: 'broken', detail: 'broken symlink or unreadable path' };
}
if (isNativeExecutable(resolved)) {
return { path, kind: 'real', detail: `compiled gbrain binary at ${resolved}` };
}
const found = nearestPackageJson(dirname(resolved));
if (!found || found.pkg.name !== 'gbrain') {
return { path, kind: 'unknown', detail: `no gbrain package.json found above ${resolved}` };
}
if (isRealGbrainPackage(found.pkg)) {
return { path, kind: 'real', detail: `this project's install at ${found.dir}` };
}
return {
path,
kind: 'foreign',
detail: `unrelated npm package named "gbrain" at ${found.dir}`,
};
}
/**
* Assess candidate paths in PATH precedence order (first entry wins when the
* shell runs `gbrain`).
*
* - skip : no candidates (gbrain not on PATH nothing to check).
* - warn : the winning entry is broken, or an unrelated npm package shadows
* (appears before) the real binary including when no real binary
* is on PATH at all.
* - ok : the winning entry is the real binary (an unrelated install
* sitting BEHIND it is noted but not a warn).
*/
export function assessGbrainBinaries(candidates: string[]): NpmSquatAssessment {
const unique = [...new Set(candidates.map((c) => c.trim()).filter(Boolean))];
if (unique.length === 0) {
return { status: 'skip', message: 'gbrain not found on PATH', binaries: [] };
}
const binaries = unique.map(classifyGbrainBinary);
const first = binaries[0]!;
const realIdx = binaries.findIndex((b) => b.kind === 'real');
const foreignIdx = binaries.findIndex((b) => b.kind === 'foreign');
if (first.kind === 'broken') {
return {
status: 'warn',
message:
`\`gbrain\` on PATH is a broken link (${first.path}). ` +
`Note: gbrain is NOT distributed on npm — the npm package named "gbrain" is unrelated. ` +
NPM_SQUAT_REMEDIATION,
binaries,
};
}
if (foreignIdx !== -1 && (realIdx === -1 || foreignIdx < realIdx)) {
const foreign = binaries[foreignIdx]!;
return {
status: 'warn',
message:
`\`gbrain\` on PATH resolves to an unrelated npm package, not this project ` +
`(${foreign.path}${foreign.detail}). gbrain is NOT distributed on npm. ` +
NPM_SQUAT_REMEDIATION,
binaries,
};
}
if (foreignIdx !== -1) {
return {
status: 'ok',
message:
`real gbrain wins on PATH (${first.path}), but an unrelated npm package named ` +
`"gbrain" is also installed (${binaries[foreignIdx]!.path}). Consider removing it: ` +
`\`bun remove -g gbrain\` / \`npm uninstall -g gbrain\`.`,
binaries,
};
}
return {
status: 'ok',
message:
first.kind === 'real'
? `gbrain on PATH is the real binary (${first.path}).`
: `no unrelated npm "gbrain" install detected on PATH (${first.path}).`,
binaries,
};
}
+27 -93
View File
@@ -1196,7 +1196,9 @@ const put_page: Operation = {
let writerLint: { error_count: number; warning_count: number } | { skipped: string } | undefined;
try {
const { runPostWriteLint } = await import('./output/post-write.ts');
const lint = await runPostWriteLint(ctx.engine, result.slug);
const lint = await runPostWriteLint(ctx.engine, result.slug, {
sourceId: ctx.sourceId ?? 'default',
});
if (lint.ran) {
writerLint = {
error_count: lint.findings.filter(f => f.severity === 'error').length,
@@ -5041,7 +5043,7 @@ const schema_review_orphans: Operation = {
const schema_apply_mutations: Operation = {
name: 'schema_apply_mutations',
description: 'v0.40.7.0: batched schema pack mutation. ATOMIC: all mutations succeed or all roll back. Audit log records one batch_id. Admin scope; NOT localOnly so remote agents (your OpenClaw, etc.) can author packs over normal MCP. Mutation shape per ApplyMutationsRequest type — supports add_type / remove_type / update_type / add_alias / remove_alias / add_prefix / remove_prefix / add_link_type / remove_link_type / set_extractable / set_expert_routing.',
description: 'v0.40.7.0: batched schema pack mutation. ATOMIC: every mutation is validated against an in-memory manifest first, and the pack file is written to disk at most once, after the FULL batch has proven valid — so a failure at any point leaves the pack file byte-identical to its pre-batch state (never a partial write). Audit log records one batch_id. Admin scope; NOT localOnly so remote agents (your OpenClaw, etc.) can author packs over normal MCP. Mutation shape per ApplyMutationsRequest type — supports add_type / remove_type / update_type / add_alias / remove_alias / add_prefix / remove_prefix / add_link_type / remove_link_type / set_extractable / set_expert_routing.',
params: {
pack: { type: 'string', required: true, description: 'Pack to mutate (must not be bundled)' },
mutations: {
@@ -5064,92 +5066,20 @@ const schema_apply_mutations: Operation = {
const batchId = `batch-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
const actor = ctx.auth?.clientId ? `mcp:${ctx.auth.clientId.slice(0, 8)}` : 'cli';
const sourceId = ctx.sourceId; // codex C5: write-side scoping
// Compose every mutation inside ONE withPackLock so the batch is
// truly atomic. The withMutation skeleton handles audit / cache
// invalidation per operation; we orchestrate the lock + iteration.
const { withPackLock } = await import('./schema-pack/pack-lock.ts');
const {
addTypeToPack, removeTypeFromPack, updateTypeOnPack,
addAliasToType, removeAliasFromType, addPrefixToType, removePrefixFromType,
addLinkTypeToPack, removeLinkTypeFromPack,
setExtractableOnType, setExpertRoutingOnType,
SchemaPackMutationError,
} = await import('./schema-pack/mutate.ts');
const baseMutateOpts = {
actor: actor as 'cli' | `mcp:${string}`,
batchId,
engine: ctx.engine,
...(sourceId ? { sourceId } : {}),
...(force ? { force: true } : {}),
};
const results: unknown[] = [];
// `applyMutationsAtomic` (issue #2581) owns the lock + single read +
// single write for the whole batch: every mutation is validated
// in-memory first, and the pack file is written at most once, only
// after the FULL batch checks out. That is what makes this actually
// atomic (a failure at any index can never leave earlier mutations on
// disk), vs. the old per-mutation-writes-as-it-goes shape.
const { applyMutationsAtomic } = await import('./schema-pack/mutate.ts');
try {
// Outer lock: hold the pack for the whole batch so other writers
// can't slip in between mutations.
await withPackLock(pack, { force, lockDir: undefined }, async () => {
for (let i = 0; i < mutations.length; i++) {
const m = mutations[i]!;
// Each primitive acquires the lock internally; the outer
// withPackLock makes that re-entrant via fast-stale-detect
// (--force option for the inner call). To keep semantics
// simple, we pass {force:true} to the inner calls because
// they're nested inside our outer lock — we already own it.
const innerOpts = { ...baseMutateOpts, force: true };
let r: unknown;
switch (m.op) {
case 'add_type':
r = await addTypeToPack(pack, {
name: m.name as string,
primitive: m.primitive as never,
prefix: m.prefix as string,
extractable: m.extractable as boolean | undefined,
expertRouting: m.expert_routing as boolean | undefined,
aliases: m.aliases as string[] | undefined,
}, innerOpts);
break;
case 'remove_type':
r = await removeTypeFromPack(pack, m.name as string, innerOpts);
break;
case 'update_type':
r = await updateTypeOnPack(pack, { name: m.name as string, patch: (m.patch as object) ?? {} }, innerOpts);
break;
case 'add_alias':
r = await addAliasToType(pack, m.type as string, m.alias as string, innerOpts);
break;
case 'remove_alias':
r = await removeAliasFromType(pack, m.type as string, m.alias as string, innerOpts);
break;
case 'add_prefix':
r = await addPrefixToType(pack, m.type as string, m.prefix as string, innerOpts);
break;
case 'remove_prefix':
r = await removePrefixFromType(pack, m.type as string, m.prefix as string, innerOpts);
break;
case 'add_link_type':
r = await addLinkTypeToPack(pack, {
name: m.name as string,
inverse: m.inverse as string | undefined,
inference: m.inference as { regex?: string; page_type?: string; target_type?: string } | undefined,
}, innerOpts);
break;
case 'remove_link_type':
r = await removeLinkTypeFromPack(pack, m.name as string, innerOpts);
break;
case 'set_extractable':
r = await setExtractableOnType(pack, m.type as string, m.value as boolean, innerOpts);
break;
case 'set_expert_routing':
r = await setExpertRoutingOnType(pack, m.type as string, m.value as boolean, innerOpts);
break;
default:
throw new SchemaPackMutationError(
'INVALID_RESULT',
`unknown mutation op: '${m.op}' at index ${i}`,
{ index: i, op: m.op },
);
}
results.push({ index: i, op: m.op, ...(r as object) });
}
const results = await applyMutationsAtomic(pack, mutations, {
actor: actor as 'cli' | `mcp:${string}`,
batchId,
engine: ctx.engine,
...(sourceId ? { sourceId } : {}),
...(force ? { force: true } : {}),
});
return {
schema_version: 1,
@@ -5160,17 +5090,21 @@ const schema_apply_mutations: Operation = {
};
} catch (e) {
const code = (e as { code?: string }).code ?? 'UNKNOWN';
const failedAtIndex = (e as { details?: { index?: number } }).details?.index;
return {
error: 'mutation_failed',
code,
message: (e as Error).message,
batch_id: batchId,
// Partial results recorded so the agent can inspect which
// mutations landed before the failure (the atomic guarantee
// is at the LOCK level — individual mutations are sequential
// and each is atomic; pack state reflects everything up to the
// failed mutation).
partial_results: results,
// Nothing was written to disk — applyMutationsAtomic only writes
// once, after every mutation in the batch has validated cleanly.
// (Pre-fix, this field was `partial_results` and listed mutations
// that HAD already landed on disk, because the old implementation
// wrote as it went — that shape is gone; a failed batch can no
// longer imply partial application.)
mutations_applied: 0,
pack_unchanged: true,
...(failedAtIndex !== undefined ? { failed_at_index: failedAtIndex } : {}),
};
}
},
+12 -1
View File
@@ -38,6 +38,10 @@ export interface PostWriteLintOpts {
force?: boolean;
/** Skip file writes; used by tests. */
noLog?: boolean;
/** Exact scalar source for the page and nested validation reads. */
sourceId?: string;
/** Federated read scope; when non-empty, takes precedence over sourceId. */
sourceIds?: string[];
}
export interface PostWriteLintResult {
@@ -80,7 +84,12 @@ export async function runPostWriteLint(
return { ran: false, slug, findings: [], skippedReason: 'flag_disabled' };
}
const page = await engine.getPage(slug);
const sourceOpts = opts.sourceIds && opts.sourceIds.length > 0
? { sourceIds: opts.sourceIds }
: opts.sourceId
? { sourceId: opts.sourceId }
: undefined;
const page = await engine.getPage(slug, sourceOpts);
if (!page) {
return { ran: false, slug, findings: [], skippedReason: 'page_not_found' };
}
@@ -97,6 +106,8 @@ export async function runPostWriteLint(
timeline: page.timeline,
frontmatter: page.frontmatter ?? {},
engine,
sourceId: opts.sourceId,
sourceIds: opts.sourceIds,
};
const findings: ValidationFinding[] = [];
+31 -10
View File
@@ -23,25 +23,46 @@ export const backLinkValidator: PageValidator = {
async validate(ctx: PageValidationContext): Promise<ValidationFinding[]> {
const findings: ValidationFinding[] = [];
const federatedSourceIds = ctx.sourceIds && ctx.sourceIds.length > 0
? ctx.sourceIds
: undefined;
const outboundOpts = federatedSourceIds
? { sourceIds: federatedSourceIds }
: ctx.sourceId
? { sourceId: ctx.sourceId }
: undefined;
const outbound = await ctx.engine.getLinks(ctx.slug);
const outbound = await ctx.engine.getLinks(ctx.slug, outboundOpts);
if (outbound.length === 0) return findings;
// Iron Law: if ctx.slug → target, target must ALSO link back to ctx.slug.
// We check target's outbound links; if none of them point at ctx.slug,
// the back-link is missing.
const uniqueTargets = new Set<string>();
for (const link of outbound) uniqueTargets.add(link.to_slug);
// A federated lookup can return same-slug origins and targets from several
// sources. Deduplicate only identical endpoint pairs; every distinct origin
// still needs its own exact reverse.
const uniqueEdges = new Map<string, typeof outbound[number]>();
for (const link of outbound) {
uniqueEdges.set(
`${link.from_source_id}\0${link.from_slug}\0${link.to_source_id}\0${link.to_slug}`,
link,
);
}
for (const target of uniqueTargets) {
const targetOutbound = await ctx.engine.getLinks(target);
const hasReverse = targetOutbound.some(l => l.to_slug === ctx.slug);
for (const target of uniqueEdges.values()) {
const targetOpts = federatedSourceIds
? { sourceIds: federatedSourceIds }
: { sourceId: target.to_source_id };
const targetOutbound = await ctx.engine.getLinks(target.to_slug, targetOpts);
const hasReverse = targetOutbound.some(link =>
link.from_source_id === target.to_source_id
&& link.from_slug === target.to_slug
&& link.to_source_id === target.from_source_id
&& link.to_slug === target.from_slug
);
if (!hasReverse) {
findings.push({
slug: ctx.slug,
validator: 'back-link',
severity: 'warning',
message: `Outbound link to ${target} has no back-link (${target} does not reference ${ctx.slug}). runAutoLink should reconcile this on next put_page; flag for inspection.`,
message: `Outbound link to ${target.to_slug} has no back-link (${target.to_slug} does not reference ${ctx.slug}). runAutoLink should reconcile this on next put_page; flag for inspection.`,
});
}
}
+7 -2
View File
@@ -62,9 +62,14 @@ export const linkValidator: PageValidator = {
linkPositions.set(slug, list);
}
// Batch-check which targets exist.
// Batch-check which targets exist within the validation read scope.
const sourceOpts = ctx.sourceIds && ctx.sourceIds.length > 0
? { sourceIds: ctx.sourceIds }
: ctx.sourceId
? { sourceId: ctx.sourceId }
: undefined;
for (const slug of internalTargets) {
const page = await ctx.engine.getPage(slug);
const page = await ctx.engine.getPage(slug, sourceOpts);
if (page) continue;
const positions = linkPositions.get(slug) ?? [];
for (const pos of positions) {
+16 -2
View File
@@ -93,6 +93,10 @@ export interface PageValidationContext {
timeline: string;
frontmatter: Record<string, unknown>;
engine: BrainEngine;
/** Exact scalar source for source-qualified validation reads. */
sourceId?: string;
/** Federated read scope; when non-empty, takes precedence over sourceId. */
sourceIds?: string[];
}
// ---------------------------------------------------------------------------
@@ -249,7 +253,9 @@ export class BrainWriter {
// Validators run before the outer transaction commits.
if (strict !== 'off') {
report = await runValidators(txEngine, validators, tx.touchedSlugs);
report = await runValidators(txEngine, validators, tx.touchedSlugs, {
sourceId: 'default',
});
// `ctx.logger.info` would be nice but keep validator behavior uniform
// regardless of strict/lint mode. Caller inspects the report.
if (strict === 'strict' && report.errorCount > 0) {
@@ -281,11 +287,17 @@ async function runValidators(
engine: BrainEngine,
validators: PageValidator[],
touchedSlugs: Set<string>,
scope: { sourceId?: string; sourceIds?: string[] } = {},
): Promise<ValidationReport> {
const findings: ValidationFinding[] = [];
const sourceOpts = scope.sourceIds && scope.sourceIds.length > 0
? { sourceIds: scope.sourceIds }
: scope.sourceId
? { sourceId: scope.sourceId }
: undefined;
for (const slug of touchedSlugs) {
const page = await engine.getPage(slug);
const page = await engine.getPage(slug, sourceOpts);
if (!page) continue; // could have been deleted in this tx
// Grandfather opt-out
@@ -298,6 +310,8 @@ async function runValidators(
timeline: page.timeline,
frontmatter: page.frontmatter ?? {},
engine,
sourceId: scope.sourceId,
sourceIds: scope.sourceIds,
};
for (const v of validators) {
+74 -26
View File
@@ -17,7 +17,26 @@ import type {
SourceRow,
} from './engine.ts';
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay, type BatchAuditSite } from './retry.ts';
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatchExhausted } from './audit/batch-retry-audit.ts';
import { runMigrations } from './migrate.ts';
import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
@@ -419,7 +438,9 @@ export class PGLiteEngine implements BrainEngine {
let dims: number = DEFAULT_EMBEDDING_DIMENSIONS;
let model: string = DEFAULT_EMBEDDING_MODEL;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
@@ -2264,7 +2285,6 @@ export class PGLiteEngine implements BrainEngine {
});
} catch (err) {
if (err instanceof Error && err.name === 'RetryAbortError') throw err;
const { isRetryableConnError } = await import('./retry.ts');
if (isRetryableConnError(err)) {
auditLogBatchExhausted(auditSite, batchSize, opts.maxRetries + 1, err);
}
@@ -2330,7 +2350,9 @@ export class PGLiteEngine implements BrainEngine {
// rationale — pglite mirrors it for parity.
let resolvedModel: string | null = null;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
try {
@@ -2879,9 +2901,11 @@ export class PGLiteEngine implements BrainEngine {
// Remote MCP clients always land here.
if (opts?.sourceIds && opts.sourceIds.length > 0) {
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -2897,9 +2921,11 @@ export class PGLiteEngine implements BrainEngine {
// opts.sourceId, scope to that source (D20).
if (opts?.sourceId) {
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -2910,9 +2936,11 @@ export class PGLiteEngine implements BrainEngine {
return rows as unknown as Link[];
}
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -2929,9 +2957,11 @@ export class PGLiteEngine implements BrainEngine {
// foreign referrer nor a foreign origin slug is disclosed to the caller.
if (opts?.sourceIds && opts.sourceIds.length > 0) {
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -2944,9 +2974,11 @@ export class PGLiteEngine implements BrainEngine {
// v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks.
if (opts?.sourceId) {
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -2957,9 +2989,11 @@ export class PGLiteEngine implements BrainEngine {
return rows as unknown as Link[];
}
const { rows } = await this.db.query(
`SELECT f.slug as from_slug, t.slug as to_slug,
`SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3842,7 +3876,6 @@ export class PGLiteEngine implements BrainEngine {
async mergeOntologyFact(obs: OntologyObservationInput): Promise<OntologyMergeResult> {
const sourceId = obs.sourceId ?? 'default';
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const dimension = normalizeDimension(obs.dimension);
const vh = valueHash(obs.value);
const conf = obs.confidence ?? 0.7;
@@ -4276,7 +4309,11 @@ export class PGLiteEngine implements BrainEngine {
$14, $15,
$16, $17, $18, $19,
$20
) RETURNING id`
)
ON CONFLICT (source_id, source_markdown_slug, row_num)
WHERE row_num IS NOT NULL
DO NOTHING
RETURNING id`
: `INSERT INTO facts (
source_id, entity_slug, fact, kind, visibility, notability, context,
valid_from, valid_until, source, source_session, confidence,
@@ -4290,12 +4327,16 @@ export class PGLiteEngine implements BrainEngine {
$15, $16,
$17, $18, $19, $20,
$21
) RETURNING id`,
)
ON CONFLICT (source_id, source_markdown_slug, row_num)
WHERE row_num IS NOT NULL
DO NOTHING
RETURNING id`,
embedStr === null
? [ctx.source_id, entitySlug, input.fact, kind, visibility, notability, context, validFrom, validUntil, input.source, sourceSession, confidence, embeddedAt, input.row_num, input.source_markdown_slug, claimMetric, claimValue, claimUnit, claimPeriod, eventType]
: [ctx.source_id, entitySlug, input.fact, kind, visibility, notability, context, validFrom, validUntil, input.source, sourceSession, confidence, embedStr, embeddedAt, input.row_num, input.source_markdown_slug, claimMetric, claimValue, claimUnit, claimPeriod, eventType],
);
out.push(ins.rows[0].id);
if (ins.rows[0]) out.push(ins.rows[0].id);
}
return out;
});
@@ -5332,12 +5373,16 @@ export class PGLiteEngine implements BrainEngine {
// pages_with_timeline) and v0.10.3 graph layer (link_coverage, timeline_coverage,
// most_connected). Both coexist: master's brain_score is the composite
// dashboard, v0.10.3 metrics give entity-page-level granularity.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const { rows: [h] } = await this.db.query(`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5362,7 +5407,7 @@ export class PGLiteEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company')
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
ORDER BY link_count DESC
LIMIT 5
`);
@@ -5381,6 +5426,7 @@ export class PGLiteEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`);
const r = h as Record<string, unknown>;
@@ -5475,15 +5521,18 @@ export class PGLiteEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
newSlug = validateSlug(newSlug);
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (mirrors postgres-engine.ts).
await this.db.query(
const result = await this.db.query(
`UPDATE pages SET slug = $1, updated_at = now() WHERE slug = $2 AND source_id = $3`,
[newSlug, oldSlug, sourceId]
);
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.affectedRows ?? 0;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
@@ -5997,7 +6046,6 @@ export class PGLiteEngine implements BrainEngine {
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
+74 -31
View File
@@ -13,7 +13,29 @@ import type {
NewFact, FactListOpts, FactsHealth,
SourceRow,
} from './engine.ts';
import { withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay, type BatchAuditSite } from './retry.ts';
// Engine-path imports stay static unless a call site carries an explicit
// engine-dynamic-import-ok justification. The gateway is the only current
// exception because its local try/catch preserves a soft fallback.
import {
withRetry,
BULK_RETRY_OPTS,
resolveBulkRetryOpts,
computeNextDelay,
isRetryableConnError,
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import {
valueHash,
normalizeDimension,
isNovelDimension,
} from './chronicle/ontology.ts';
import {
resolveRecencyDecayMap,
DEFAULT_FALLBACK,
} from './search/recency-decay.ts';
import { logDbDisconnect } from './audit/db-disconnect-audit.ts';
import { logPoolRecovery } from './audit/pool-recovery-audit.ts';
import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatchExhausted } from './audit/batch-retry-audit.ts';
import type {
DomainBankSampleOpts, CorpusSampleOpts, DomainBankRow,
@@ -331,7 +353,6 @@ export class PostgresEngine implements BrainEngine {
// even a no-op disconnect (engine that was never connected) is
// recorded — that case may itself be a caller-side bug worth seeing.
try {
const { logDbDisconnect } = await import('./audit/db-disconnect-audit.ts');
logDbDisconnect('postgres', this._connectionStyle ?? 'unknown');
} catch { /* best-effort; never block disconnect on audit failure */ }
// v0.30.1: tear down the direct pool first if the manager owns one.
@@ -381,7 +402,9 @@ export class PostgresEngine implements BrainEngine {
let dims: number = DEFAULT_EMBEDDING_DIMENSIONS;
let model: string = DEFAULT_EMBEDDING_MODEL;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy: its static closure is large, and evaluation inside
// this try/catch preserves the unconfigured-gateway default fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
// Both accessors THROW when the gateway is unconfigured (they never
// return falsy), so the catch below is the only fallback path (#3461).
dims = gw.getEmbeddingDimensions();
@@ -2381,8 +2404,8 @@ export class PostgresEngine implements BrainEngine {
if (err instanceof Error && err.name === 'RetryAbortError') throw err;
// Best-effort exhausted-retry log. If the error wasn't retryable in
// the first place, isRetryableConnError(err) is false and we skip.
// Lazy-import to avoid a circular dep concern.
const { isRetryableConnError } = await import('./retry.ts');
// retry.ts is already in this module's static graph through withRetry, so
// classifying the exhausted error does not need a second runtime import.
if (isRetryableConnError(err)) {
auditLogBatchExhausted(auditSite, batchSize, opts.maxRetries + 1, err);
}
@@ -2451,7 +2474,9 @@ export class PostgresEngine implements BrainEngine {
// is the LAST resort (fresh brain whose config row doesn't exist yet).
let resolvedModel: string | null = null;
try {
const gw = await import('./ai/gateway.ts');
// Keep the gateway lazy so module-load failure remains inside this soft
// fallback boundary; eager evaluation would bypass the config-row fallback.
const gw = await import('./ai/gateway.ts'); // engine-dynamic-import-ok
resolvedModel = gw.getEmbeddingModel();
} catch {
try {
@@ -3027,9 +3052,11 @@ export class PostgresEngine implements BrainEngine {
if (opts?.sourceIds && opts.sourceIds.length > 0) {
const ids = opts.sourceIds;
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3044,9 +3071,11 @@ export class PostgresEngine implements BrainEngine {
// opts.sourceId, scope the from-page lookup.
if (opts?.sourceId) {
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3056,9 +3085,11 @@ export class PostgresEngine implements BrainEngine {
return rows as unknown as Link[];
}
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3080,9 +3111,11 @@ export class PostgresEngine implements BrainEngine {
if (opts?.sourceIds && opts.sourceIds.length > 0) {
const ids = opts.sourceIds;
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3094,9 +3127,11 @@ export class PostgresEngine implements BrainEngine {
// v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks.
if (opts?.sourceId) {
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3106,9 +3141,11 @@ export class PostgresEngine implements BrainEngine {
return rows as unknown as Link[];
}
const rows = await tx`
SELECT f.slug as from_slug, t.slug as to_slug,
SELECT f.slug as from_slug, f.source_id as from_source_id,
t.slug as to_slug, t.source_id as to_source_id,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
o.slug as origin_slug, o.source_id as origin_source_id,
l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
@@ -3983,7 +4020,6 @@ export class PostgresEngine implements BrainEngine {
async mergeOntologyFact(obs: OntologyObservationInput): Promise<OntologyMergeResult> {
const sql = this.sql;
const sourceId = obs.sourceId ?? 'default';
const { valueHash, normalizeDimension, isNovelDimension } = await import('./chronicle/ontology.ts');
const dimension = normalizeDimension(obs.dimension);
const vh = valueHash(obs.value);
const conf = obs.confidence ?? 0.7;
@@ -4455,9 +4491,13 @@ export class PostgresEngine implements BrainEngine {
${input.row_num}, ${input.source_markdown_slug},
${claimMetric}, ${claimValue}, ${claimUnit}, ${claimPeriod},
${eventType}
) RETURNING id
)
ON CONFLICT (source_id, source_markdown_slug, row_num)
WHERE row_num IS NOT NULL
DO NOTHING
RETURNING id
`;
out.push(Number(ins[0].id));
if (ins[0]) out.push(Number(ins[0].id));
}
return out;
});
@@ -5432,12 +5472,16 @@ export class PostgresEngine implements BrainEngine {
// no outbound links). The raw islanded list is filtered through the same
// policy as `gbrain orphans` so convention pages do not count against
// dashboard health.
// #1305: every page-scoped count here excludes soft-deleted rows — same
// posture as getStats — so brain_score moves when the user deletes pages.
// Chunk/link counts stay raw (storage until the purge phase), matching
// getStats, and destructive-removal counts elsewhere deliberately stay raw.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company')
SELECT id, slug FROM pages WHERE type IN ('entity', 'person', 'company') AND deleted_at IS NULL
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
0 as stale_pages,
@@ -5459,7 +5503,7 @@ export class PostgresEngine implements BrainEngine {
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('entity', 'person', 'company')
WHERE p.type IN ('entity', 'person', 'company') AND p.deleted_at IS NULL
ORDER BY link_count DESC
LIMIT 5
`;
@@ -5478,6 +5522,7 @@ export class PostgresEngine implements BrainEngine {
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)) as islanded,
EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = p.id) as has_timeline
FROM pages p
WHERE p.deleted_at IS NULL
`;
const pageCount = Number(h.page_count);
@@ -5569,14 +5614,17 @@ export class PostgresEngine implements BrainEngine {
}
// Sync
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<void> {
async updateSlug(oldSlug: string, newSlug: string, opts?: { sourceId?: string }): Promise<number> {
newSlug = validateSlug(newSlug);
const sql = this.sql;
const sourceId = opts?.sourceId ?? 'default';
// Source-qualify so a rename in source A doesn't sweep up same-slug rows
// in sources B/C/D (which would either rename them all OR fail the
// (source_id, slug) UNIQUE if the new slug already exists in another source).
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
const result = await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug} AND source_id = ${sourceId}`;
// #3056: rows moved — a zero-row UPDATE does not throw, so the count is
// the only way callers can see the no-op.
return result.count ?? 0;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
@@ -5815,12 +5863,10 @@ export class PostgresEngine implements BrainEngine {
let isReap = false;
if (ctx?.error !== undefined) {
try {
const { isConnectionEndedError } = await import('./retry-matcher.ts');
isReap = isConnectionEndedError(ctx.error);
} catch { /* classification is best-effort */ }
}
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery(isReap ? 'reap_detected' : 'reconnect_other', ctx?.error);
} catch { /* audit is best-effort */ }
@@ -5844,7 +5890,6 @@ export class PostgresEngine implements BrainEngine {
// New pool is live — discard the old one best-effort.
if (oldSql) { try { await oldSql.end({ timeout: 5 }); } catch { /* swallow */ } }
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery('reconnect_succeeded');
} catch { /* best-effort */ }
} catch (err) {
@@ -5856,7 +5901,6 @@ export class PostgresEngine implements BrainEngine {
this._sql = oldSql;
this.connectionManager = oldManager;
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
logPoolRecovery('reconnect_failed', err);
} catch { /* best-effort */ }
throw err; // let batchRetry's backoff handle the retry
@@ -6293,7 +6337,6 @@ export class PostgresEngine implements BrainEngine {
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
+3
View File
@@ -186,6 +186,9 @@ export {
removeLinkTypeFromPack,
setExtractableOnType,
setExpertRoutingOnType,
type BatchMutationRequest,
type BatchMutationResult,
applyMutationsAtomic,
} from './mutate.ts';
export { invalidateQueryCache } from './query-cache-invalidator.ts';
+286 -27
View File
@@ -497,11 +497,18 @@ export interface AddTypeOpts {
aliases?: string[];
}
export async function addTypeToPack(packName: string, opts: AddTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
// Each `build*Mutator` below does the primitive's up-front (file-free,
// lock-free) shape validation and returns the pure `(current) => next`
// transform. The public async functions wrap the builder with
// `withMutation` for the single-mutation (CLI) path; `applyMutationsAtomic`
// (batch path, below) reuses the SAME builders so single-call and batched
// mutations can never drift in what they accept or reject.
function buildAddTypeMutator(opts: AddTypeOpts): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(opts.name);
validatePrimitive(opts.primitive);
validatePrefix(opts.prefix);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
if (m.page_types.some((pt) => pt.name === opts.name)) {
throw new SchemaPackMutationError(
'TYPE_EXISTS',
@@ -518,16 +525,24 @@ export async function addTypeToPack(packName: string, opts: AddTypeOpts, mutateO
expert_routing: opts.expertRouting ?? false,
};
return { ...m, page_types: [...m.page_types, newType] };
}, 'add_type', { type: opts.name, prefix: opts.prefix });
};
}
export async function removeTypeFromPack(packName: string, name: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
export async function addTypeToPack(packName: string, opts: AddTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildAddTypeMutator(opts), 'add_type', { type: opts.name, prefix: opts.prefix });
}
function buildRemoveTypeMutator(name: string): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(name);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
findType(m, name); // throws TYPE_NOT_FOUND if missing
checkNoReferences(m, name); // codex C14
return { ...m, page_types: m.page_types.filter((t) => t.name !== name) };
}, 'remove_type', { type: name });
};
}
export async function removeTypeFromPack(packName: string, name: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildRemoveTypeMutator(name), 'remove_type', { type: name });
}
export interface UpdateTypeOpts {
@@ -535,56 +550,76 @@ export interface UpdateTypeOpts {
patch: Partial<Omit<PackPageType, 'name'>>;
}
export async function updateTypeOnPack(packName: string, opts: UpdateTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
function buildUpdateTypeMutator(opts: UpdateTypeOpts): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(opts.name);
if (opts.patch.primitive !== undefined) validatePrimitive(opts.patch.primitive);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
const existing = findType(m, opts.name);
const updated: PackPageType = { ...existing, ...opts.patch, name: existing.name };
return { ...m, page_types: m.page_types.map((t) => (t.name === opts.name ? updated : t)) };
}, 'update_type', { type: opts.name });
};
}
export async function addAliasToType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
export async function updateTypeOnPack(packName: string, opts: UpdateTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildUpdateTypeMutator(opts), 'update_type', { type: opts.name });
}
function buildAddAliasMutator(typeName: string, alias: string): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(typeName);
validateTypeName(alias);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
const t = findType(m, typeName);
if (t.aliases.includes(alias)) return m; // idempotent
const next: PackPageType = { ...t, aliases: [...t.aliases, alias] };
return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) };
}, 'add_alias', { type: typeName });
};
}
export async function removeAliasFromType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
export async function addAliasToType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildAddAliasMutator(typeName, alias), 'add_alias', { type: typeName });
}
function buildRemoveAliasMutator(typeName: string, alias: string): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(typeName);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
const t = findType(m, typeName);
if (!t.aliases.includes(alias)) return m; // idempotent
const next: PackPageType = { ...t, aliases: t.aliases.filter((a) => a !== alias) };
return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) };
}, 'remove_alias', { type: typeName });
};
}
export async function addPrefixToType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
export async function removeAliasFromType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildRemoveAliasMutator(typeName, alias), 'remove_alias', { type: typeName });
}
function buildAddPrefixMutator(typeName: string, prefix: string): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(typeName);
validatePrefix(prefix);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
const t = findType(m, typeName);
if (t.path_prefixes.includes(prefix)) return m;
const next: PackPageType = { ...t, path_prefixes: [...t.path_prefixes, prefix] };
return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) };
}, 'add_prefix', { type: typeName, prefix });
};
}
export async function removePrefixFromType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
export async function addPrefixToType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildAddPrefixMutator(typeName, prefix), 'add_prefix', { type: typeName, prefix });
}
function buildRemovePrefixMutator(typeName: string, prefix: string): (m: SchemaPackManifest) => SchemaPackManifest {
validateTypeName(typeName);
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
const t = findType(m, typeName);
if (!t.path_prefixes.includes(prefix)) return m;
const next: PackPageType = { ...t, path_prefixes: t.path_prefixes.filter((p) => p !== prefix) };
return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) };
}, 'remove_prefix', { type: typeName, prefix });
};
}
export async function removePrefixFromType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildRemovePrefixMutator(typeName, prefix), 'remove_prefix', { type: typeName, prefix });
}
export interface AddLinkTypeOpts {
@@ -593,11 +628,11 @@ export interface AddLinkTypeOpts {
inference?: { regex?: string; page_type?: string; target_type?: string };
}
export async function addLinkTypeToPack(packName: string, opts: AddLinkTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
function buildAddLinkTypeMutator(opts: AddLinkTypeOpts): (m: SchemaPackManifest) => SchemaPackManifest {
if (typeof opts.name !== 'string' || opts.name.length === 0) {
throw new SchemaPackMutationError('INVALID_RESULT', `link_type.name is required`);
}
return withMutation(packName, mutateOpts, (m) => {
return (m) => {
if (m.link_types.some((lt) => lt.name === opts.name)) {
throw new SchemaPackMutationError(
'TYPE_EXISTS',
@@ -611,11 +646,15 @@ export async function addLinkTypeToPack(packName: string, opts: AddLinkTypeOpts,
...(opts.inference ? { inference: opts.inference } : {}),
} as PackLinkType;
return { ...m, link_types: [...m.link_types, newLink] };
}, 'add_link_type', { type: opts.name });
};
}
export async function removeLinkTypeFromPack(packName: string, linkName: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, (m) => {
export async function addLinkTypeToPack(packName: string, opts: AddLinkTypeOpts, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildAddLinkTypeMutator(opts), 'add_link_type', { type: opts.name });
}
function buildRemoveLinkTypeMutator(linkName: string): (m: SchemaPackManifest) => SchemaPackManifest {
return (m) => {
if (!m.link_types.some((lt) => lt.name === linkName)) {
throw new SchemaPackMutationError(
'TYPE_NOT_FOUND',
@@ -633,7 +672,11 @@ export async function removeLinkTypeFromPack(packName: string, linkName: string,
);
}
return { ...m, link_types: m.link_types.filter((lt) => lt.name !== linkName) };
}, 'remove_link_type', { type: linkName });
};
}
export async function removeLinkTypeFromPack(packName: string, linkName: string, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return withMutation(packName, mutateOpts, buildRemoveLinkTypeMutator(linkName), 'remove_link_type', { type: linkName });
}
export async function setExtractableOnType(packName: string, typeName: string, value: boolean, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
@@ -643,3 +686,219 @@ export async function setExtractableOnType(packName: string, typeName: string, v
export async function setExpertRoutingOnType(packName: string, typeName: string, value: boolean, mutateOpts: MutateOpts = {}): Promise<MutateResult> {
return updateTypeOnPack(packName, { name: typeName, patch: { expert_routing: value } }, { ...mutateOpts });
}
// ────────────────────────────────────────────────────────────────────────
// Atomic batch application (issue #2581) — one lock, one file read, one
// write. `schema_apply_mutations` used to loop over these same primitives
// and let each one independently read/validate/WRITE the pack file, so a
// batch that failed partway left every earlier mutation permanently on
// disk even though the op is documented as all-or-nothing. Here every
// mutation in the batch is applied + lint-validated against an IN-MEMORY
// manifest only; `writePackManifest` is called at most once, after every
// mutation in the batch has been proven valid. A failure at any index
// therefore leaves the pack file byte-identical to its pre-batch state —
// partial application is structurally impossible, not just cleaned up
// after the fact.
// ────────────────────────────────────────────────────────────────────────
export interface BatchMutationRequest {
op: string;
[key: string]: unknown;
}
export interface BatchMutationResult {
index: number;
op: string;
pack: string;
path: string;
format: PackFileFormat;
/** sha8 of the manifest immediately before this mutation (chained). */
prev_sha8: string;
/** sha8 of the manifest immediately after this mutation (chained). */
new_sha8: string;
}
/**
* Resolve one batch entry to its pure mutator + audit context, reusing the
* exact same `build*Mutator` a single-mutation call would use. Throws
* `SchemaPackMutationError('INVALID_RESULT', ...)` for an unrecognized
* `op`, matching the pre-existing single-mutation shape-validation
* contract: this runs before the file is touched, so it is deliberately
* NOT audit-logged here (mirrors `addTypeToPack` etc. throwing from their
* own up-front `validate*` calls, before `withMutation` ever starts).
*/
function buildBatchMutator(
m: BatchMutationRequest,
index: number,
): { mutate: (current: SchemaPackManifest) => SchemaPackManifest; auditContext: { type?: string; prefix?: string } } {
switch (m.op) {
case 'add_type':
return {
mutate: buildAddTypeMutator({
name: m.name as string,
primitive: m.primitive as never,
prefix: m.prefix as string,
extractable: m.extractable as boolean | undefined,
expertRouting: m.expert_routing as boolean | undefined,
aliases: m.aliases as string[] | undefined,
}),
auditContext: { type: m.name as string, prefix: m.prefix as string },
};
case 'remove_type':
return { mutate: buildRemoveTypeMutator(m.name as string), auditContext: { type: m.name as string } };
case 'update_type':
return {
mutate: buildUpdateTypeMutator({ name: m.name as string, patch: (m.patch as object) ?? {} }),
auditContext: { type: m.name as string },
};
case 'add_alias':
return { mutate: buildAddAliasMutator(m.type as string, m.alias as string), auditContext: { type: m.type as string } };
case 'remove_alias':
return { mutate: buildRemoveAliasMutator(m.type as string, m.alias as string), auditContext: { type: m.type as string } };
case 'add_prefix':
return {
mutate: buildAddPrefixMutator(m.type as string, m.prefix as string),
auditContext: { type: m.type as string, prefix: m.prefix as string },
};
case 'remove_prefix':
return {
mutate: buildRemovePrefixMutator(m.type as string, m.prefix as string),
auditContext: { type: m.type as string, prefix: m.prefix as string },
};
case 'add_link_type':
return {
mutate: buildAddLinkTypeMutator({
name: m.name as string,
inverse: m.inverse as string | undefined,
inference: m.inference as { regex?: string; page_type?: string; target_type?: string } | undefined,
}),
auditContext: { type: m.name as string },
};
case 'remove_link_type':
return { mutate: buildRemoveLinkTypeMutator(m.name as string), auditContext: { type: m.name as string } };
case 'set_extractable':
return {
mutate: buildUpdateTypeMutator({ name: m.type as string, patch: { extractable: m.value as boolean } }),
auditContext: { type: m.type as string },
};
case 'set_expert_routing':
return {
mutate: buildUpdateTypeMutator({ name: m.type as string, patch: { expert_routing: m.value as boolean } }),
auditContext: { type: m.type as string },
};
default:
throw new SchemaPackMutationError('INVALID_RESULT', `unknown mutation op: '${m.op}' at index ${index}`, { index, op: m.op });
}
}
export async function applyMutationsAtomic(
packName: string,
mutations: BatchMutationRequest[],
opts: MutateOpts,
): Promise<BatchMutationResult[]> {
const actor: MutationActor = opts.actor ?? 'cli';
const firstOp = (mutations[0]?.op as MutationOp) ?? 'add_type';
// Bundled-pack guard, same as withMutation step 1 — happens once for
// the whole batch since `pack` is constant across mutations.
let path: string;
let format: PackFileFormat;
try {
({ path, format } = locateMutablePackFile(packName));
} catch (e) {
if (e instanceof SchemaPackMutationError) {
await logMutationFailure({ op: firstOp, pack: packName, actor, reason: e.code, batch_id: opts.batchId });
}
throw e;
}
return withPackLock(packName, opts, async () => {
let current: SchemaPackManifest;
let batchPrevSha8: string;
try {
current = loadPackFromFile(path);
batchPrevSha8 = await computeManifestSha8(current);
} catch (e) {
const err = new SchemaPackMutationError(
'PACK_CORRUPT',
`cannot read or parse pack file at ${path}: ${(e as Error).message}`,
{ path },
);
await logMutationFailure({ op: firstOp, pack: packName, actor, reason: err.code, batch_id: opts.batchId });
throw err;
}
// Phase 1: apply + lint-validate every mutation against the IN-MEMORY
// manifest only. Nothing here touches disk — a throw at any index
// propagates straight out (lock released by withPackLock's finally)
// and `path` is left completely untouched.
const pending: Array<{ index: number; op: string; auditContext: { type?: string; prefix?: string }; prevSha8: string; newSha8: string }> = [];
let runningPrevSha8 = batchPrevSha8;
for (let i = 0; i < mutations.length; i++) {
const m = mutations[i]!;
const opForAudit = (m.op as MutationOp) ?? firstOp;
const built = buildBatchMutator(m, i); // shape validation — unaudited, matches single-mutation contract
let next: SchemaPackManifest;
try {
next = built.mutate(current);
} catch (e) {
const base = e instanceof SchemaPackMutationError ? e : new SchemaPackMutationError('INVALID_RESULT', (e as Error).message);
// Re-wrap so `details.index` is always present for the batch
// caller (operations.ts) to report which mutation failed,
// without losing the primitive's own code/message/details.
const wrapped = new SchemaPackMutationError(base.code, base.message, { ...base.details, index: i });
await logMutationFailure({
op: opForAudit, pack: packName, actor, ...built.auditContext,
reason: wrapped.code, prev_sha8: runningPrevSha8, batch_id: opts.batchId,
});
throw wrapped;
}
const lintReport = await runFilePlaneLintRules(next);
if (!lintReport.ok) {
const msg = lintReport.errors.map((iss) => `${iss.rule}: ${iss.message}`).join('; ');
const err = new SchemaPackMutationError('INVALID_RESULT', `mutation would produce invalid pack: ${msg}`, { index: i, errors: lintReport.errors });
await logMutationFailure({
op: opForAudit, pack: packName, actor, ...built.auditContext,
reason: err.code, prev_sha8: runningPrevSha8, batch_id: opts.batchId,
});
throw err;
}
const newSha8 = await computeManifestSha8(next);
pending.push({ index: i, op: m.op, auditContext: built.auditContext, prevSha8: runningPrevSha8, newSha8 });
current = next;
runningPrevSha8 = newSha8;
}
// Phase 2: every mutation validated clean — write ONCE.
try {
writePackManifest(path, current, format);
} catch (e) {
const err = e instanceof SchemaPackMutationError ? e : new SchemaPackMutationError('IO_ERROR', (e as Error).message, { path });
const last = pending[pending.length - 1];
await logMutationFailure({
op: (last?.op as MutationOp) ?? firstOp, pack: packName, actor, ...(last?.auditContext ?? {}),
reason: err.code, prev_sha8: batchPrevSha8, batch_id: opts.batchId,
});
throw err;
}
// Step 7 equivalent: best-effort post-hooks, once for the whole batch.
try { invalidatePackCache(packName); } catch { /* swallow — cache invalidation must not block mutation success */ }
if (opts.engine) {
try { await invalidateQueryCache(opts.engine, opts.sourceId); } catch { /* swallow */ }
}
// Only now — after the single write has actually landed on disk — do
// we log success and report results. Nothing above this point may
// ever be reported as applied.
const results: BatchMutationResult[] = [];
for (const p of pending) {
await logMutationSuccess({
op: p.op as MutationOp, pack: packName, actor, ...p.auditContext,
prev_sha8: p.prevSha8, new_sha8: p.newSha8, batch_id: opts.batchId,
});
results.push({ index: p.index, op: p.op, pack: packName, path, format, prev_sha8: p.prevSha8, new_sha8: p.newSha8 });
}
return results;
});
}
+32 -5
View File
@@ -48,6 +48,32 @@ import {
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
/**
* Which detail levels get the compiled_truth boost (#3430).
*
* ONLY `low`. The documented contract (`src/core/operations.ts`) is
* "low (compiled truth only), medium (default, all with dedup), high (all
* chunks)" so `low` is the level that privileges compiled truth, and both
* `medium` and `high` are supposed to see everything on equal footing.
*
* This was previously spelled `detail !== 'high'`, i.e. written as though
* `high` were the special case. Because COMPILED_TRUTH_BOOST is applied AFTER
* RRF normalization, and RRF's whole range over a 100-deep pool is 1/60 1/160,
* a 2.0x multiplier is not a tilt break-even is `2/(60+r) >= 1/60`, so any
* boosted chunk inside the first 60 ranks outranks an unboosted rank-1 chunk.
* At the default detail that made search categorically compiled-truth-only:
* a page whose answer lived in a `fenced_code` chunk returned the prose chunk,
* and the code chunk fell out of the window entirely.
*
* Extracted as a named predicate rather than left inline at three call sites so
* the detailboost mapping is directly testable. An inline expression can only
* be covered through a full `hybridSearch` round trip, which is why the
* original inversion went unnoticed.
*/
export function shouldBoostCompiledTruth(detail: string | null | undefined): boolean {
return detail === 'low';
}
const pendingCacheWrites = new Set<Promise<unknown>>();
/**
@@ -841,11 +867,12 @@ export async function embedQueryBounded(
embedOpts: { embeddingModel?: string; dimensions?: number } | undefined,
dl: QueryEmbedDeadline,
): Promise<Float32Array> {
const p = embedQuery(text, { ...(embedOpts ?? {}), abortSignal: dl.signal });
p.catch(() => { /* swallow the loser's late rejection */ });
// Floor the budget so a healthy embed isn't starved when the shared absolute
// deadline was mostly consumed by prior work (codex). Still bounded overall.
const remaining = Math.max(MIN_QUERY_EMBED_BUDGET_MS, dl.deadlineAt - Date.now());
const signal = AbortSignal.timeout(remaining);
const p = embedQuery(text, { ...(embedOpts ?? {}), abortSignal: signal });
p.catch(() => { /* swallow the loser's late rejection */ });
let timer: ReturnType<typeof setTimeout> | undefined;
const deadline = new Promise<never>((_, reject) => {
timer = setTimeout(
@@ -1169,7 +1196,7 @@ export async function hybridSearch(
const noEmbedLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
noEmbedResults = rrfFusionWeighted(noEmbedLists, shouldBoostCompiledTruth(detailResolved));
}
if (noEmbedResults.length > 0) {
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
@@ -1413,7 +1440,7 @@ export async function hybridSearch(
const fallbackLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
fallbackResults = rrfFusionWeighted(fallbackLists, shouldBoostCompiledTruth(detail));
}
if (fallbackResults.length > 0) {
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
@@ -1500,7 +1527,7 @@ export async function hybridSearch(
// arms BEFORE fusion so the compiled-truth authority boost skips them.
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
let fused = rrfFusionWeighted(allLists, detail !== 'high');
let fused = rrfFusionWeighted(allLists, shouldBoostCompiledTruth(detail));
// Cosine re-scoring before dedup so semantically better chunks survive.
// v0.36 (D9): hydrate from the active embedding column so rescore happens
+24 -1
View File
@@ -25,6 +25,7 @@
import { createHash } from 'crypto';
import { CR_MODES, type CRMode } from '../types.ts';
import { getFtsLanguage } from '../fts-language.ts';
import { getRecipe } from '../ai/recipes/index.ts';
/**
@@ -766,7 +767,19 @@ export function attributeKnob<K extends keyof ModeBundle>(
// written between the #3391 stale-fix (which changes which chunks count as
// current) and the operator's migration run. Same one-time global cold-miss
// pattern as the bumps above.
export const KNOBS_HASH_VERSION = 13;
//
// bump 14→15: the FTS configuration name (GBRAIN_FTS_LANGUAGE, resolved by
// getFtsLanguage()) folds into the key via the `fts=` part. It reaches BOTH
// engines' keyword SQL (websearch_to_tsquery/to_tsvector in postgres-engine
// and pglite-engine) and the two search_vector trigger functions, so it
// changes which rows the keyword arm returns — but it only applied at
// DB-query build time (cache miss). Switching language and running
// `gbrain reindex-search-vector` therefore left every pre-switch query_cache
// row reachable: the freshly retokenized index was silently bypassed for up
// to cache.ttl_seconds, with no warning and no way for an operator to tell.
// Same one-time global cold-miss pattern as the bumps above; refills within
// cache.ttl_seconds (3600s default).
export const KNOBS_HASH_VERSION = 15;
/**
* v0.36 (D8 / CDX-2) second-arg context for the cache key. The
@@ -898,6 +911,16 @@ export function knobsHash(
// across processes. Sorted copy so ['a/','b/'] and ['b/','a/'] hash
// identically; undefined falls back to 'none' for legacy callers.
`hx=${ctx?.hardExcludes ? [...ctx.hardExcludes].sort().join(',') : 'none'}`,
// v=15 addition (append-only): the resolved FTS configuration name. Read
// from getFtsLanguage() rather than threaded through KnobsHashContext on
// purpose — the language is a process-global env read with no per-call
// dimension, and the `prov=` bump note above records what threading costs:
// a ctx field only isolates callers that pass it, so legacy callers keep
// hashing the fallback literal on both sides of a switch. Reading it here
// covers every knobsHash() caller, present and future. getFtsLanguage()
// memoizes and validates against /^[a-z][a-z0-9_]*$/, so this stays a
// cheap, bounded string.
`fts=${getFtsLanguage()}`,
];
const h = createHash('sha256');
h.update(parts.join('|'));
+1 -1
View File
@@ -96,7 +96,7 @@ const ENTITY_PATTERNS = [
/\boverview\b/i,
/\bbackground\b/i,
/\bprofile\b/i,
/\bwhat\s+do\s+(you|we)\s+know\b/i,
/\bwhat\s+do\s+(i|you|we)\s+know\b/i,
];
const FULL_CONTEXT_PATTERNS = [
+4 -1
View File
@@ -34,6 +34,7 @@ export interface ModelPricing {
const SUPPORTED_MODELS = [
'openai:gpt-4o',
'openai:gpt-5',
'openai:gpt-5.2',
'openai:gpt-5.5',
'anthropic:claude-opus-5',
'anthropic:claude-opus-4-8',
@@ -41,7 +42,9 @@ const SUPPORTED_MODELS = [
'anthropic:claude-sonnet-5',
'anthropic:claude-sonnet-4-6',
'anthropic:claude-haiku-4-5',
'google:gemini-1.5-pro',
// gemini-1.5-pro was retired by Google (#3510); gemini-2.0-flash replaces
// it in DEFAULT_MODEL_PANEL. `gemini-2-flash` stays as the legacy alias.
'google:gemini-2.0-flash',
'google:gemini-2-flash',
] as const;
+9 -2
View File
@@ -33,10 +33,17 @@ import type { TakesQualityReceipt } from './receipt.ts';
import { estimateCost, getPricing, PricingNotFoundError } from './pricing.ts';
import { DEFAULT_CYCLES_NONTTY } from '../eval/cycle-default.ts';
/**
* Three distinct providers (uncorrelated judge blind spots). Every entry MUST
* be listed in its recipe's chat touchpoint AND in the SUPPORTED_MODELS
* pricing allowlist pinned by test/default-model-panels.test.ts.
* google:gemini-1.5-pro (retired by Google) and openai:gpt-4o (dropped from
* the OpenAI recipe's chat list) sat here dead until #3510.
*/
export const DEFAULT_MODEL_PANEL = [
'openai:gpt-4o',
'openai:gpt-5.2',
'anthropic:claude-opus-4-7',
'google:gemini-1.5-pro',
'google:gemini-2.0-flash',
] as const;
export interface RunOpts {
+6
View File
@@ -1203,7 +1203,11 @@ export interface CodeEdgeResult {
// Links
export interface Link {
from_slug: string;
/** Exact source identity of the from-page joined by from_page_id. */
from_source_id: string;
to_slug: string;
/** Exact source identity of the to-page joined by to_page_id. */
to_source_id: string;
link_type: string;
context: string;
/**
@@ -1221,6 +1225,8 @@ export interface Link {
* multiple pages reference the same (from, to, type) tuple.
*/
origin_slug?: string | null;
/** Exact source identity of origin_slug; null when absent or grant-redacted. */
origin_source_id?: string | null;
/**
* The frontmatter field name that created this edge (e.g. 'key_people',
* 'investors'). Used for debug output and the `unresolved` response list.
+37
View File
@@ -0,0 +1,37 @@
import { describe, expect, test } from 'bun:test';
import { openAdminSseStream, type AdminSseResponse } from '../src/commands/serve-http.ts';
describe('admin SSE handshake', () => {
test('flushes a protocol-valid comment immediately after the headers', () => {
const calls: string[] = [];
const headers = new Map<string, string>();
openAdminSseStream({
setHeader(name: string, value: string | number | readonly string[]) {
headers.set(name, String(value));
calls.push(`header:${name}`);
return this;
},
flushHeaders() {
calls.push('flush');
},
write(chunk: unknown) {
calls.push(`write:${String(chunk)}`);
return true;
},
} as unknown as AdminSseResponse);
expect(headers).toEqual(new Map([
['Content-Type', 'text/event-stream'],
['Cache-Control', 'no-cache'],
['Connection', 'keep-alive'],
]));
expect(calls).toEqual([
'header:Content-Type',
'header:Cache-Control',
'header:Connection',
'flush',
'write:: connected\n\n',
]);
});
});
+37 -7
View File
@@ -40,6 +40,8 @@ import {
__getShrinkStateForTests,
} from '../../src/core/ai/gateway.ts';
import { AIConfigError, AITransientError } from '../../src/core/ai/errors.ts';
import { __setTestRecipesForTests } from '../../src/core/ai/recipes/index.ts';
import type { Recipe } from '../../src/core/ai/types.ts';
// The last test in this file leaves the gateway configured with a remote
// provider + fake key and a REAL embed transport. Without a final reset,
@@ -94,6 +96,31 @@ function configureGoogle(): void {
});
}
// A recipe that declares an embedding touchpoint but omits every batch cap.
// Every shipped recipe now declares one (google gained max_batch_tokens), so
// the startup warning is exercised against this synthetic cap-less recipe —
// injected into the registry only for the duration of the test that needs it.
const CAPLESS_RECIPE: Recipe = {
id: 'synthetic-capless',
name: 'Synthetic cap-less (test fixture)',
tier: 'openai-compat',
implementation: 'openai-compatible',
touchpoints: {
embedding: {
models: ['synthetic-embed-1'],
default_dims: 768,
},
},
};
function configureCapless(): void {
configureGateway({
embedding_model: 'synthetic-capless:synthetic-embed-1',
embedding_dimensions: 768,
env: {},
});
}
// --------- 1. Pure helpers ---------
describe('splitByTokenBudget (pure helper)', () => {
@@ -429,20 +456,22 @@ describe('startup warning for recipes missing max_batch_tokens', () => {
beforeEach(() => resetGateway());
test('configured missing-cap recipe warns once; unrelated recipes stay quiet', () => {
__setTestRecipesForTests([CAPLESS_RECIPE]);
const warnings: string[] = [];
const original = console.warn;
console.warn = (msg: string) => warnings.push(String(msg));
try {
configureOpenAI();
expect(warnings.length).toBe(0);
configureGoogle();
configureCapless();
const firstCallCount = warnings.length;
// Reconfigure: the warning should NOT re-fire for the same recipes
// Reconfigure: the warning should NOT re-fire for the same recipe
// within one process (we already told the operator).
configureGoogle();
configureCapless();
expect(warnings.length).toBe(firstCallCount);
} finally {
console.warn = original;
__setTestRecipesForTests([]);
}
// The warning text should match the documented contract.
@@ -451,11 +480,12 @@ describe('startup warning for recipes missing max_batch_tokens', () => {
);
expect(contractMatch.length).toBe(1);
// Voyage declares max_batch_tokens → suppressed. OpenAI is the
// canonical fast-path recipe → also suppressed by id. Both must be
// absent from the warnings.
// Voyage + google declare max_batch_tokens → suppressed. OpenAI is the
// canonical fast-path recipe → also suppressed by id. Only the synthetic
// cap-less recipe warns.
expect(warnings.find(w => w.includes('"voyage"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"openai"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"google"'))).toBeDefined();
expect(warnings.find(w => w.includes('"google"'))).toBeUndefined();
expect(warnings.find(w => w.includes('"synthetic-capless"'))).toBeDefined();
});
});
+100 -1
View File
@@ -20,7 +20,7 @@
import { describe, expect, test } from 'bun:test';
import { buildGatewayConfig } from '../../src/cli.ts';
import type { GBrainConfig } from '../../src/core/config.ts';
import { KNOWN_CONFIG_KEYS, type GBrainConfig } from '../../src/core/config.ts';
import { withEnv } from '../helpers/with-env.ts';
const PASSTHROUGHS: Array<{ envVar: string; recipeId: string }> = [
@@ -139,6 +139,105 @@ describe('buildGatewayConfig config-plane API-key folding', () => {
expect(cfg.env.VOYAGE_API_KEY).toBe('pa-env-plane');
});
});
// #3500: dashscope_api_key was accepted at the file plane but never folded,
// so the dashscope/dashscope-rerank recipes (required: DASHSCOPE_API_KEY)
// could only be keyed via a process-env export.
test('dashscope_api_key folds into gateway env as DASHSCOPE_API_KEY', async () => {
await withEnv({ DASHSCOPE_API_KEY: undefined }, async () => {
const cfg = buildGatewayConfig({
dashscope_api_key: 'sk-ds-config-plane',
} as unknown as GBrainConfig);
expect(cfg.env.DASHSCOPE_API_KEY).toBe('sk-ds-config-plane');
});
});
test('a real DASHSCOPE_API_KEY process.env value wins over the config-plane fallback', async () => {
await withEnv({ DASHSCOPE_API_KEY: 'sk-ds-env-plane' }, async () => {
const cfg = buildGatewayConfig({
dashscope_api_key: 'sk-ds-config-plane',
} as unknown as GBrainConfig);
expect(cfg.env.DASHSCOPE_API_KEY).toBe('sk-ds-env-plane');
});
});
// #3500: the google recipe reads GOOGLE_GENERATIVE_AI_API_KEY; before this
// fold the ONLY configuration route was exporting that exact env name.
test('google_api_key folds into gateway env as GOOGLE_GENERATIVE_AI_API_KEY', async () => {
await withEnv(
{ GOOGLE_GENERATIVE_AI_API_KEY: undefined, GEMINI_API_KEY: undefined },
async () => {
const cfg = buildGatewayConfig({
google_api_key: 'AIza-config-plane',
} as unknown as GBrainConfig);
expect(cfg.env.GOOGLE_GENERATIVE_AI_API_KEY).toBe('AIza-config-plane');
},
);
});
// Recurring-class guard: EVERY *_api_key field declared in
// KNOWN_CONFIG_KEYS must reach the gateway env dict. Adding a new
// provider key field to GBrainConfig without folding it in
// buildGatewayConfig fails here — the #121/#2662/#3500 bug class.
test('every KNOWN_CONFIG_KEYS *_api_key field reaches the gateway env', async () => {
const keyFields = KNOWN_CONFIG_KEYS.filter((k) => k.endsWith('_api_key'));
expect(keyFields.length).toBeGreaterThanOrEqual(7);
for (const field of keyFields) {
const sentinel = `sentinel-${field}`;
// Clear the two env names the field could map to so config must win.
await withEnv(
{
[field.replace(/_api_key$/, '').toUpperCase() + '_API_KEY']: undefined,
GOOGLE_GENERATIVE_AI_API_KEY: undefined,
GEMINI_API_KEY: undefined,
},
async () => {
const cfg = buildGatewayConfig({ [field]: sentinel } as unknown as GBrainConfig);
expect(
Object.values(cfg.env).includes(sentinel),
`config field "${field}" never reaches the gateway env — add a fold in buildGatewayConfig`,
).toBe(true);
},
);
}
});
});
describe('buildGatewayConfig GEMINI_API_KEY alias (#3500)', () => {
// GEMINI_API_KEY is the env name Google's own docs and SDKs use; the
// recipe/gateway read GOOGLE_GENERATIVE_AI_API_KEY. Precedence:
// env GOOGLE_GENERATIVE_AI_API_KEY > env GEMINI_API_KEY > config google_api_key.
test('GEMINI_API_KEY aliases to GOOGLE_GENERATIVE_AI_API_KEY', async () => {
await withEnv(
{ GOOGLE_GENERATIVE_AI_API_KEY: undefined, GEMINI_API_KEY: 'AIza-gemini-env' },
async () => {
const cfg = buildGatewayConfig(baseConfig);
expect(cfg.env.GOOGLE_GENERATIVE_AI_API_KEY).toBe('AIza-gemini-env');
},
);
});
test('canonical GOOGLE_GENERATIVE_AI_API_KEY env wins over the GEMINI_API_KEY alias', async () => {
await withEnv(
{ GOOGLE_GENERATIVE_AI_API_KEY: 'AIza-canonical', GEMINI_API_KEY: 'AIza-alias' },
async () => {
const cfg = buildGatewayConfig(baseConfig);
expect(cfg.env.GOOGLE_GENERATIVE_AI_API_KEY).toBe('AIza-canonical');
},
);
});
test('GEMINI_API_KEY (process env) wins over the config-plane google_api_key', async () => {
await withEnv(
{ GOOGLE_GENERATIVE_AI_API_KEY: undefined, GEMINI_API_KEY: 'AIza-gemini-env' },
async () => {
const cfg = buildGatewayConfig({
google_api_key: 'AIza-config-plane',
} as unknown as GBrainConfig);
expect(cfg.env.GOOGLE_GENERATIVE_AI_API_KEY).toBe('AIza-gemini-env');
},
);
});
});
describe('buildGatewayConfig env empty-string clobber guard (#1249)', () => {
@@ -11,7 +11,28 @@
import { afterAll, beforeAll, describe, expect, mock, test } from 'bun:test';
import { capBatchItems, configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
import { listRecipes, getRecipe } from '../../src/core/ai/recipes/index.ts';
import { listRecipes, getRecipe, __setTestRecipesForTests } from '../../src/core/ai/recipes/index.ts';
import type { Recipe } from '../../src/core/ai/types.ts';
/**
* A recipe that declares an embedding touchpoint but omits every batch cap
* (no max_batch_tokens, no no_batch_cap, no max_batch_items). This is the
* exact shape a future provider PR might forget the case the startup
* warning exists to catch. Kept synthetic because every shipped recipe now
* declares a cap, so no real recipe can play this role anymore.
*/
const CAPLESS_RECIPE: Recipe = {
id: 'synthetic-capless',
name: 'Synthetic cap-less (test fixture)',
tier: 'openai-compat',
implementation: 'openai-compatible',
touchpoints: {
embedding: {
models: ['synthetic-embed-1'],
default_dims: 768,
},
},
};
describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warning', () => {
let warnSpy: ReturnType<typeof mock>;
@@ -75,7 +96,12 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
}
});
test('configureGateway warns for google only when google embedding is configured', () => {
test('google no longer warns — it now declares max_batch_tokens', () => {
// google's gemini-embedding endpoint ships a declared batch-token budget,
// so configuring it must NOT trip the missing-cap warning.
const r = getRecipe('google');
expect(r?.touchpoints.embedding?.max_batch_tokens).toBeGreaterThan(0);
warnSpy.mockClear();
resetGateway();
configureGateway({ env: {} });
@@ -95,8 +121,33 @@ describe('v0.32 #779: no_batch_cap suppresses the missing-max_batch_tokens warni
messages = warnSpy.mock.calls.map(c => String(c[0] ?? ''));
expect(
messages.some(m => m.includes('"google"') && m.includes('without max_batch_tokens')),
'google should warn when configured because it has fixed-cap models',
).toBe(true);
'google now declares a cap and must stay quiet even when configured',
).toBe(false);
});
test('a configured recipe that omits every batch cap still warns', () => {
// Regression guard the google fixture used to provide. Every shipped
// embedding recipe now declares a cap, so the warn-fires path is exercised
// with a synthetic cap-less recipe injected into the registry.
__setTestRecipesForTests([CAPLESS_RECIPE]);
try {
warnSpy.mockClear();
resetGateway();
configureGateway({
embedding_model: 'synthetic-capless:synthetic-embed-1',
embedding_dimensions: 768,
env: {},
});
const messages = warnSpy.mock.calls.map(c => String(c[0] ?? ''));
expect(
messages.some(
m => m.includes('"synthetic-capless"') && m.includes('without max_batch_tokens'),
),
'a configured recipe missing every batch cap must warn',
).toBe(true);
} finally {
__setTestRecipesForTests([]);
}
});
test('every recipe with empty models[] declares user_provided_models OR has openai-fast-path', () => {
+28 -1
View File
@@ -6,7 +6,7 @@
* the env override is process-global.
*/
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import { mkdtempSync, rmSync, readdirSync } from 'node:fs';
import { mkdtempSync, rmSync, readdirSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
@@ -73,6 +73,33 @@ describe('parser-probe audit trail', () => {
logParserProbeEvent(makeEvent({ ts: old }));
expect(readRecentParserProbeEvents(7).length).toBe(0);
});
test('cross-week ordering: events come back chronological so the tail is the newest run', () => {
// Regression: the shared week-file reader walks the current week's file
// first, then the previous week's. Without sorting, the array tail —
// which doctor's conversation_parser_probe_health reports as "latest" —
// was the OLDEST in-window event whenever last week's file had entries.
// Write the two week files directly so the cross-file case is genuinely
// exercised (the writer routes by write time, not event ts).
const now = new Date('2026-07-23T12:00:00Z');
const thisWeekFile = computeParserProbeAuditFilename(now);
const prevWeekFile = computeParserProbeAuditFilename(new Date(now.getTime() - 7 * 86400000));
writeFileSync(join(auditDir, thisWeekFile), [
JSON.stringify(makeEvent({ ts: '2026-07-22T08:00:00Z' })),
JSON.stringify(makeEvent({ ts: '2026-07-23T08:00:00Z' })),
].join('\n') + '\n');
writeFileSync(join(auditDir, prevWeekFile), [
JSON.stringify(makeEvent({ ts: '2026-07-17T08:00:00Z' })),
JSON.stringify(makeEvent({ ts: '2026-07-18T08:00:00Z' })),
].join('\n') + '\n');
const events = readRecentParserProbeEvents(7, now);
expect(events.map(e => e.ts)).toEqual([
'2026-07-17T08:00:00Z',
'2026-07-18T08:00:00Z',
'2026-07-22T08:00:00Z',
'2026-07-23T08:00:00Z',
]);
});
});
describe('parserProbeRanWithin — 24h rate-limit gate', () => {
+85 -2
View File
@@ -16,12 +16,18 @@ import { describe, expect, test } from 'bun:test';
import { readFileSync } from 'fs';
import { join } from 'path';
import { resolveAutopilotDispatchTimeoutMs } from '../src/commands/autopilot-timeout.ts';
import { defaultTimeoutMsFor } from '../src/core/minions/handler-timeouts.ts';
const AUTOPILOT_SRC = readFileSync(
join(import.meta.dir, '..', 'src', 'commands', 'autopilot.ts'),
'utf8',
);
const AUTOPILOT_TIMEOUT_SRC = readFileSync(
join(import.meta.dir, '..', 'src', 'commands', 'autopilot-timeout.ts'),
'utf8',
);
describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
test('imports dispatchPerSource from the fan-out helper', () => {
expect(AUTOPILOT_SRC).toMatch(
@@ -59,9 +65,33 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
expect(AUTOPILOT_SRC).toContain(
'const timeoutMs = resolveAutopilotDispatchTimeoutMs(baseInterval, false);',
);
expect(AUTOPILOT_SRC).toMatch(
/dispatchPerSource\(engine, queue, \{[\s\S]{0,300}timeoutMs: resolveAutopilotDispatchTimeoutMs\(baseInterval, true\)/,
expect(AUTOPILOT_SRC).toContain(
'const fullCycleTimeoutMs = resolveAutopilotDispatchTimeoutMs(baseInterval, true);',
);
expect(AUTOPILOT_SRC).toMatch(
/dispatchPerSource\(engine, queue, \{[\s\S]{0,300}timeoutMs: fullCycleTimeoutMs/,
);
});
test('#2781: dispatchGlobalMaintenance gets the full-cycle floor, not the outer (non-full-cycle) timeoutMs', () => {
// Live #2781 regression, found in review: dispatchGlobalMaintenance's
// call used the object-shorthand `timeoutMs`, which resolved to the
// OUTER `const timeoutMs = resolveAutopilotDispatchTimeoutMs(baseInterval, false)`
// declared earlier in the same function for the sync/freshness dispatch
// — not to the full-cycle value computed for dispatchPerSource a few
// lines above it. 'autopilot-global-maintenance' carries the same
// 30-min handler anchor as 'autopilot-cycle' (handler-timeouts.ts), so
// this silently starved brain-wide maintenance (embed/orphans/purge/…)
// at exactly the #2781 symptom (600s budget at the default 300s
// interval) even after the per-source path was fixed. Pin the correct
// wiring by source-shape: the call must pass the *full-cycle* variable.
const dispatchGlobalIdx = AUTOPILOT_SRC.indexOf('dispatchGlobalMaintenance(engine, queue');
expect(dispatchGlobalIdx).toBeGreaterThan(-1);
const dispatchGlobalCall = AUTOPILOT_SRC.slice(dispatchGlobalIdx, dispatchGlobalIdx + 200);
expect(dispatchGlobalCall).toContain('timeoutMs: fullCycleTimeoutMs');
// Guard against the exact regression: the shorthand `timeoutMs` (bare,
// no colon) resolving to the non-full-cycle outer const.
expect(dispatchGlobalCall).not.toMatch(/\{\s*repoPath,\s*slot,\s*timeoutMs,/);
});
test('updates lastFullCycleAt on dispatch (so the 60-min floor is honored)', () => {
@@ -70,6 +100,59 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
expect(AUTOPILOT_SRC).toMatch(/lastFullCycleAt\s*=\s*Date\.now\(\)/);
});
test('#2781: full-cycle floor is derived from BOTH handler anchors, not a duplicated literal', () => {
// #2781's root cause: autopilot stamped an explicit `timeout_ms` that was
// only a `Math.max(interval-derived, 300_000)`-shaped literal, so it
// silently overrode the 'autopilot-cycle' handler's own #1737 anchor
// (`queue.ts`: an explicit stamp always wins over `defaultTimeoutMsFor`).
// A prior fix (#2852) hardcoded a matching `1_800_000` floor for
// full-cycle dispatch, but a literal that merely happens to equal the
// handler anchor can drift from it again if the anchor is ever retuned
// in handler-timeouts.ts without a matching edit here — reintroducing
// the exact #2781 bug class.
//
// Prove the floor is *derived* (not just numerically coincidental) two
// ways: (a) it equals Math.max of BOTH job names' anchors — a bare
// duplicated literal could accidentally match a SINGLE anchor (as the
// prior #2852 fix did) but wiring `Math.max(cycle, global)` is what
// actually protects a future divergence between the two anchors; (b) a
// source-shape check that both job-name string literals reach
// `defaultTimeoutMsFor` (directly or via a thin wrapper), and that no
// bare numeric literal sits in the full-cycle branch.
const cycleAnchorMs = defaultTimeoutMsFor('autopilot-cycle');
const globalAnchorMs = defaultTimeoutMsFor('autopilot-global-maintenance');
if (cycleAnchorMs === null) throw new Error("expected a handler anchor for 'autopilot-cycle'");
if (globalAnchorMs === null) throw new Error("expected a handler anchor for 'autopilot-global-maintenance'");
const expectedFloorMs = Math.max(cycleAnchorMs, globalAnchorMs);
// A short interval collapses the interval-derived component to its
// 300_000ms minimum, so the full-cycle result must equal the derived
// floor exactly.
expect(resolveAutopilotDispatchTimeoutMs(1, true)).toBe(expectedFloorMs);
// A regular (non-full-cycle) dispatch — e.g. the 'sync' freshness job,
// which has no long-job handler anchor — must NOT pick up the floor.
expect(resolveAutopilotDispatchTimeoutMs(1, false)).toBe(300_000);
// Guard against reintroducing a hardcoded literal floor directly in the
// full-cycle branch instead of the derived FULL_CYCLE_TIMEOUT_FLOOR_MS.
expect(AUTOPILOT_TIMEOUT_SRC).not.toMatch(/fullCycle\s*\?\s*Math\.max\([^)]*,\s*1_?800_?000\)/);
// Pin the derivation END TO END in source shape (codex round-2): both
// job-name anchor lookups must participate in the floor's Math.max, and
// the full-cycle branch must consume that derived const — otherwise the
// floor could be swapped back to a bare literal while the wrapper,
// import, and job-name strings survive as dead code and the assertions
// above still pass.
expect(AUTOPILOT_TIMEOUT_SRC).toMatch(
/FULL_CYCLE_TIMEOUT_FLOOR_MS\s*=\s*Math\.max\(\s*requireHandlerAnchorMs\('autopilot-cycle'\),\s*requireHandlerAnchorMs\('autopilot-global-maintenance'\),?\s*\)/,
);
expect(AUTOPILOT_TIMEOUT_SRC).toMatch(
/fullCycle\s*\?\s*Math\.max\(intervalDerivedTimeoutMs,\s*FULL_CYCLE_TIMEOUT_FLOOR_MS\)/,
);
// The wrapper itself must consult defaultTimeoutMsFor (fail-loud on a
// missing anchor, never a numeric fallback).
expect(AUTOPILOT_TIMEOUT_SRC).toMatch(/requireHandlerAnchorMs[\s\S]{0,200}defaultTimeoutMsFor\(jobName\)/);
});
test('does NOT regress to the single-job dispatch on the full-cycle path', () => {
// Pre-PR: the shouldFullCycle branch did:
// const job = await queue.add('autopilot-cycle', { repoPath }, {
+5 -3
View File
@@ -63,9 +63,11 @@ describe('embeddingProviderConfigured (recipe-aware helper)', () => {
// #2662: buildGatewayConfig now folds voyage_api_key → VOYAGE_API_KEY,
// so this producer-facing map must recognize it as gateway-propagated.
expect(HOSTED_EMBED_KEY_CONFIG.VOYAGE_API_KEY).toBe('voyage_api_key');
// Not propagated to the gateway today → must NOT be backed by a config field
// (producer closures fall through to process.env only for this one).
expect(HOSTED_EMBED_KEY_CONFIG.GOOGLE_GENERATIVE_AI_API_KEY).toBeUndefined();
// #3500: buildGatewayConfig now folds google_api_key and
// dashscope_api_key, so both are gateway-propagated and must be mapped
// (a config-plane key is genuinely usable by the gateway).
expect(HOSTED_EMBED_KEY_CONFIG.GOOGLE_GENERATIVE_AI_API_KEY).toBe('google_api_key');
expect(HOSTED_EMBED_KEY_CONFIG.DASHSCOPE_API_KEY).toBe('dashscope_api_key');
});
// #2662: end-to-end regression through the REAL file-plane loader
+53
View File
@@ -0,0 +1,53 @@
/**
* #575 `gbrain config show` printed `[object Object]` for object-valued
* config fields (e.g. `provider_base_urls`) because non-string values were
* interpolated straight into a template literal.
*
* Behavioral pin: object values render as JSON; object values under a
* sensitive key stay redacted; scalars keep their existing rendering.
*/
import { describe, test, expect } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { withEnv } from './helpers/with-env.ts';
import { runConfig } from '../src/commands/config.ts';
import type { BrainEngine } from '../src/core/engine.ts';
describe('config show object-valued fields (#575)', () => {
test('provider_base_urls renders as JSON, not [object Object]', async () => {
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-cfgshow-'));
try {
mkdirSync(join(tmpHome, '.gbrain'), { recursive: true });
writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify({
engine: 'pglite',
database_path: join(tmpHome, '.gbrain', 'brain'),
provider_base_urls: { ollama: 'http://localhost:11434' },
some_nested_secret: { api_key: 'sk-super-secret' },
}, null, 2));
const outLines: string[] = [];
const origLog = console.log;
console.log = (...args: unknown[]) => { outLines.push(args.map(String).join(' ')); };
try {
await withEnv({ GBRAIN_HOME: tmpHome, DATABASE_URL: undefined }, async () => {
await runConfig({} as unknown as BrainEngine, ['show']);
});
} finally {
console.log = origLog;
}
const out = outLines.join('\n');
expect(out).not.toContain('[object Object]');
const urlLine = outLines.find(l => l.includes('provider_base_urls'));
expect(urlLine).toBeDefined();
expect(urlLine!).toContain('http://localhost:11434');
// Objects under a sensitive key must NOT leak their contents.
const secretLine = outLines.find(l => l.includes('some_nested_secret'));
expect(secretLine).toBeDefined();
expect(secretLine!).not.toContain('sk-super-secret');
} finally {
rmSync(tmpHome, { recursive: true, force: true });
}
});
});
+24
View File
@@ -22,6 +22,7 @@ let extractCalls: Array<{ mode: string; dir: string; slugs: string[] | undefined
let embedCalls: Array<{ stale: boolean | undefined; dryRun: boolean | undefined }> = [];
let orphansCalls: number = 0;
let orphansOpts: Array<{ sourceId?: string } | undefined> = [];
let schemaSuggestOpts: Array<{ sourceId?: string; dryRun?: boolean } | undefined> = [];
// Mock lint
mock.module('../../src/commands/lint.ts', () => ({
@@ -116,6 +117,14 @@ mock.module('../../src/commands/orphans.ts', () => ({
formatOrphansText: () => '',
}));
// Mock schema-suggest
mock.module('../../src/core/cycle/schema-suggest.ts', () => ({
runSchemaSuggestPhase: async (_engine: any, opts?: { sourceId?: string; dryRun?: boolean }) => {
schemaSuggestOpts.push(opts);
return { suggestions_emitted: 0, source_id: opts?.sourceId ?? 'default', skipped: false };
},
}));
// Import after mocks.
const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts');
const { PGLiteEngine } = await import('../../src/core/pglite-engine.ts');
@@ -151,6 +160,7 @@ beforeEach(() => {
embedCalls = [];
orphansCalls = 0;
orphansOpts = [];
schemaSuggestOpts = [];
});
// ─── dryRun propagation (regression guards) ────────────────────────
@@ -519,6 +529,20 @@ describe('runCycle — sourceId resolution (regression #475)', () => {
expect(orphansOpts.at(-1)).toEqual({ sourceId: 'alpha' });
});
// schema-suggest (T12 cathedral phase) was never threaded through
// cycleSourceId — it silently fell back to 'default' for every source,
// the same bug class as #1586 (synthesize) and #2666 (patterns), just
// undiscovered for this phase. Pins the fix: the resolved per-source id
// must reach runSchemaSuggestPhase the same way it reaches orphans/sync.
test('seeded sources row → schema-suggest phase receives matching sourceId (not "default")', async () => {
await (sharedEngine as any).db.query(
`INSERT INTO sources (id, name, local_path) VALUES ($1, $2, $3)`,
['bravo', 'bravo', '/tmp/brain-schema-suggest-bravo'],
);
await runCycle(sharedEngine, { brainDir: '/tmp/brain-schema-suggest-bravo', phases: ['schema-suggest'] });
expect(schemaSuggestOpts.at(-1)?.sourceId).toBe('bravo');
});
test('forceGlobalOrphans keeps orphans brain-wide even when brainDir maps to a source', async () => {
await (sharedEngine as any).db.query(
`INSERT INTO sources (id, name, local_path) VALUES ($1, $2, $3)`,
+5 -2
View File
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
return resolveSearchMode({ mode: 'balanced' });
}
test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => {
test('KNOBS_HASH_VERSION is 15 (cross-modal still appended; 14→15 FTS language fold)', () => {
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
@@ -146,7 +146,10 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
// finally reaches asymmetric providers — pre-fix rows were keyed on
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
expect(KNOBS_HASH_VERSION).toBe(13);
// #3430: 13→14 compiled_truth boost no longer applies at detail=medium.
// 14→15: the resolved FTS configuration name (fts=) — a language switch
// plus `reindex-search-vector` must not keep serving pre-switch rows.
expect(KNOBS_HASH_VERSION).toBe(15);
});
test('flipping unified_multimodal changes the hash', () => {
+14 -8
View File
@@ -38,7 +38,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv, emptyHome } from './helpers/with-env.ts';
import { runCycle, ALL_PHASES } from '../src/core/cycle.ts';
import { mkdtempSync, writeFileSync } from 'fs';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
@@ -139,19 +139,25 @@ describe('#2540 (i) — pack omitting optional phases, all enabled phases comple
describe('#2540 (ii) — an enabled phase that never completes still prevents the stamp', () => {
test('every selected phase failing reports status=failed and does NOT stamp last_full_cycle_at', async () => {
await withEnv({ GBRAIN_HOME: gbrainHome, OPENAI_API_KEY: undefined, ANTHROPIC_API_KEY: undefined }, async () => {
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
await seedSource('always-fails');
expect(await readLastFullCycleAt('always-fails')).toBeNull();
// embed is a real, always-enabled phase (no pack gate, no config
// .enabled toggle). With no embedding provider key configured it
// deterministically fails — this is NOT the fix under test, it's
// the pre-existing "an enabled phase genuinely never completes"
// case the issue says must keep failing doctor's check.
// Deterministic, environment-independent failure: run the sync phase
// against a brain directory that no longer exists. The previous shape
// ('embed' with OPENAI_API_KEY/ANTHROPIC_API_KEY unset) was
// environment-sensitive — on a machine where any OTHER embedding
// provider resolves (Voyage, ZeroEntropy, a local endpoint, …), embed
// with zero stale chunks succeeds and the cycle reports 'clean',
// flipping this test's expectation. A vanished checkout fails the
// sync phase on every machine. This is NOT the fix under test; it's
// the pre-existing "an enabled phase genuinely never completes" case
// the issue says must keep failing doctor's check.
rmSync(brainDir, { recursive: true, force: true });
const report = await runCycle(engine, {
brainDir,
sourceId: 'always-fails',
phases: ['embed'],
phases: ['sync'],
});
expect(report.status).toBe('failed');
@@ -0,0 +1,203 @@
// Regression guard: extract_atoms merges the transcript pool and the
// DB-page pool into one `work[]` list before applying the per-call budget
// cap. When transcripts are concatenated ahead of pages, a transcript
// corpus alone can exhaust the (default $0.30) budget every single call,
// so the page pool — the ONLY pool `countExtractAtomsBacklog` /
// doctor's extract_atoms_backlog check measures — never gets processed.
// `gbrain dream --phase extract_atoms --drain` then reports forward
// progress (atoms extracted) while the doctor-visible backlog number
// never moves, because it was all coming from transcripts. Real-world
// case: a brain with a growing transcript corpus and a stagnant
// page-backlog warning that the doctor's own suggested fix
// (`--drain --window 120`) can't clear.
//
// Fixed by interleaving the two pools 1-for-1 instead of concatenating
// transcripts-then-pages, so both pools make forward progress within a
// single budget-capped call.
//
// The budget cap in production is enforced entirely inside the real
// gatewayChat (AsyncLocalStorage-scoped BudgetTracker — see
// `withBudgetTracker` in ai/gateway.ts); the `_chat` test seam bypasses
// gatewayChat, so `budgetTracker.totalSpent` never advances from a plain
// stub. The loop's OWN budget-exhaustion path is driven by catching a
// thrown `BudgetExhausted` from `chat()` (extract-atoms.ts's
// `if (err instanceof BudgetExhausted) { budgetExhausted = true; ... }`).
// These tests throw that exact error from `_chat` after N successful
// calls, which is the same mechanism a real exhausted budget triggers.
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runPhaseExtractAtoms, countExtractAtomsBacklog } from '../../src/core/cycle/extract-atoms.ts';
import { BudgetExhausted } from '../../src/core/budget/budget-tracker.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
import type { ChatResult, ChatOpts } from '../../src/core/ai/gateway.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
function okChatResult(text: string): ChatResult {
return {
text,
blocks: [{ type: 'text', text }],
stopReason: 'end',
usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 },
model: 'anthropic:claude-haiku-4-5',
providerId: 'anthropic',
};
}
/** N successful calls, then every further call throws BudgetExhausted the
* same shape the real budget-tracker throws mid-loop once the cap is hit. */
function chatExhaustingAfter(n: number, text = '[]'): (o: ChatOpts) => Promise<ChatResult> {
let calls = 0;
return async (_o: ChatOpts) => {
calls++;
if (calls > n) {
throw new BudgetExhausted('budget cap exceeded', {
reason: 'cost',
spent: 999,
cap: 0.0005,
modelId: 'anthropic:claude-haiku-4-5',
});
}
return okChatResult(text);
};
}
describe('extract_atoms work-list interleave (budget-starvation regression)', () => {
// Codex review flagged that a transcript-first interleave ([t1, p1, ...])
// still starves EVERY page when the budget only covers exactly one call —
// item 0 (a transcript) succeeds, item 1 (the first page) never gets
// attempted. That reproduces the original symptom exactly: `--drain`
// extracts atoms from transcripts forever while the doctor-visible page
// backlog never moves. Page-first interleave guarantees the FIRST work
// item is always a page (when any exist), so even a budget-for-one call
// makes forward progress on the backlog doctor actually measures.
test('a budget that fits exactly 1 call processes a page, not a transcript', async () => {
const result = await runPhaseExtractAtoms(engine, {
sourceId: 'default',
_transcripts: [
{ filePath: '/tmp/t1.txt', content: 'transcript one', contentHash: '1'.repeat(16) },
{ filePath: '/tmp/t2.txt', content: 'transcript two', contentHash: '2'.repeat(16) },
{ filePath: '/tmp/t3.txt', content: 'transcript three', contentHash: '3'.repeat(16) },
],
_pages: [
{ slug: 'note/a', content: 'page a', contentHash: 'a'.repeat(16) },
],
_chat: chatExhaustingAfter(1),
});
expect(result.details.pages_processed).toBe(1);
expect(result.details.transcripts_processed).toBe(0);
expect(result.details.budget_exhausted).toBe(true);
});
test('a budget that fits exactly 2 calls processes one of each pool, not 2 transcripts', async () => {
const result = await runPhaseExtractAtoms(engine, {
sourceId: 'default',
_transcripts: [
{ filePath: '/tmp/t1.txt', content: 'transcript one', contentHash: '1'.repeat(16) },
{ filePath: '/tmp/t2.txt', content: 'transcript two', contentHash: '2'.repeat(16) },
{ filePath: '/tmp/t3.txt', content: 'transcript three', contentHash: '3'.repeat(16) },
],
_pages: [
{ slug: 'note/a', content: 'page a', contentHash: 'a'.repeat(16) },
{ slug: 'note/b', content: 'page b', contentHash: 'b'.repeat(16) },
{ slug: 'note/c', content: 'page c', contentHash: 'c'.repeat(16) },
],
_chat: chatExhaustingAfter(2),
});
// The regression: pre-fix, transcripts-then-pages concatenation means
// the first 2 calls both land on transcripts — pagesProcessed stays 0
// no matter how many batches run, as long as the transcript pool keeps
// outrunning the budget. Interleaving guarantees the page pool gets a
// turn within the same call.
expect(result.details.transcripts_processed).toBe(1);
expect(result.details.pages_processed).toBe(1);
expect(result.details.budget_exhausted).toBe(true);
});
test('when only transcripts exist, all budget still goes to transcripts (no pages to starve)', async () => {
const result = await runPhaseExtractAtoms(engine, {
sourceId: 'default',
_transcripts: [
{ filePath: '/tmp/t1.txt', content: 'transcript one', contentHash: '1'.repeat(16) },
{ filePath: '/tmp/t2.txt', content: 'transcript two', contentHash: '2'.repeat(16) },
],
_pages: [],
_chat: chatExhaustingAfter(2),
});
expect(result.details.transcripts_processed).toBe(2);
expect(result.details.pages_processed).toBe(0);
});
test('a lopsided pool (many transcripts, one page) still gives the page its turn before the budget runs out', async () => {
const result = await runPhaseExtractAtoms(engine, {
sourceId: 'default',
_transcripts: [
{ filePath: '/tmp/t1.txt', content: 'transcript one', contentHash: '1'.repeat(16) },
{ filePath: '/tmp/t2.txt', content: 'transcript two', contentHash: '2'.repeat(16) },
{ filePath: '/tmp/t3.txt', content: 'transcript three', contentHash: '3'.repeat(16) },
{ filePath: '/tmp/t4.txt', content: 'transcript four', contentHash: '4'.repeat(16) },
{ filePath: '/tmp/t5.txt', content: 'transcript five', contentHash: '5'.repeat(16) },
],
_pages: [
{ slug: 'note/a', content: 'page a', contentHash: 'a'.repeat(16) },
],
_chat: chatExhaustingAfter(2),
});
// Interleaved order is [a, t1, t2, t3, t4, t5] (page-first). The first 2
// calls land on item 0 (a) and item 1 (t1) — the single page is NOT
// starved just because 5 transcripts exist.
expect(result.details.transcripts_processed).toBe(1);
expect(result.details.pages_processed).toBe(1);
});
// Codex review (Minor): the tests above pin work-item ORDER via the
// details counters, but not the actual user-facing consequence — that
// `countExtractAtomsBacklog` (what doctor's extract_atoms_backlog check
// reads) really drops. Seeds a real DB page (no `_pages` test seam, so
// production `discoverExtractablePages` finds it) alongside a transcript
// corpus that would have starved it pre-fix, and asserts the backlog
// count goes 1 -> 0 across the call.
test('a real DB page backlog count drops to 0 even with a starving transcript corpus', async () => {
const BODY = 'x'.repeat(600); // >= MIN_PAGE_CHARS_FOR_EXTRACTION (500)
await engine.putPage('article/real-page', {
type: 'article',
title: 'real-page',
compiled_truth: BODY,
});
expect(await countExtractAtomsBacklog(engine, 'default')).toBe(1);
const validAtomJson = JSON.stringify([
{ title: 'A', atom_type: 'insight', body: 'body a' },
]);
await runPhaseExtractAtoms(engine, {
sourceId: 'default',
_transcripts: [
{ filePath: '/tmp/t1.txt', content: 'transcript one', contentHash: '1'.repeat(16) },
{ filePath: '/tmp/t2.txt', content: 'transcript two', contentHash: '2'.repeat(16) },
{ filePath: '/tmp/t3.txt', content: 'transcript three', contentHash: '3'.repeat(16) },
],
_chat: chatExhaustingAfter(1, validAtomJson),
});
expect(await countExtractAtomsBacklog(engine, 'default')).toBe(0);
});
});
+48
View File
@@ -0,0 +1,48 @@
/**
* Consistency guard for the takes-quality DEFAULT_MODEL_PANEL the sibling
* of test/cross-modal-default-slots.test.ts (#3510).
*
* `google:gemini-1.5-pro` sat in the panel after Google retired it, and
* `openai:gpt-4o` after the OpenAI recipe dropped it from its chat list
* either way the gateway rejects the slot on every default run. The guard
* only works if recipes list LIVE models: removing a dead model from its
* recipe makes every hardcoded default that still names it fail here at
* once. Do not re-add retired models to a recipe to quiet this test.
*/
import { describe, expect, test } from 'bun:test';
import { DEFAULT_MODEL_PANEL } from '../src/core/takes-quality-eval/runner.ts';
import { getPricing } from '../src/core/takes-quality-eval/pricing.ts';
import { getRecipe } from '../src/core/ai/recipes/index.ts';
import { splitProviderModelId } from '../src/core/model-id.ts';
import { canonicalLookup } from '../src/core/model-pricing.ts';
describe('takes-quality DEFAULT_MODEL_PANEL ↔ recipe consistency', () => {
test('every default panel model is listed in its recipe chat touchpoint', () => {
for (const id of DEFAULT_MODEL_PANEL) {
const { provider, model } = splitProviderModelId(id);
expect(provider).not.toBeNull();
const recipe = getRecipe(provider!);
expect(recipe, `unknown recipe "${provider}"`).toBeDefined();
expect(
recipe!.touchpoints.chat?.models ?? [],
`"${model}" not listed for ${provider} chat — the default panel can never run`,
).toContain(model);
}
});
test('every default panel model prices via canonical AND the takes-quality allowlist', () => {
for (const id of DEFAULT_MODEL_PANEL) {
expect(canonicalLookup(id), `"${id}" missing from CANONICAL_PRICING`).toBeDefined();
// getPricing throws PricingNotFoundError if the model is missing from
// SUPPORTED_MODELS — a default that can't be budget-gated aborts every
// `--budget-usd` run before the first call.
expect(getPricing(id)).toBeDefined();
}
});
test('panel spans three distinct providers (uncorrelated blind spots)', () => {
const providers = new Set(DEFAULT_MODEL_PANEL.map((id) => splitProviderModelId(id).provider));
expect(providers.size).toBe(3);
});
});
+105
View File
@@ -0,0 +1,105 @@
/**
* #1835 pure unit coverage for src/commands/doctor-asset-paths.ts.
*
* Everything here is path/string-based with injected platform + WSL mount
* root, so it runs identically on macOS / Linux / CI. The WSL translation
* itself is UNVERIFIED-ON-PLATFORM (no real WSL host in this environment);
* these tests pin the intended mapping.
*/
import { describe, expect, test } from 'bun:test';
import { resolveAssetPath, parseWslAutomountRoot } from '../src/commands/doctor-asset-paths.ts';
const REPO = '/mnt/d/brain-repo';
describe('resolveAssetPath — Windows drive paths', () => {
test('WSL: D:/ forward-slash path maps to <root>/d/…', () => {
const r = resolveAssetPath('D:/cicada3301/lost9999/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: '/mnt/d/cicada3301/lost9999/img.jpg', foreign: false });
});
test('WSL: D:\\ backslash path maps with separators normalized', () => {
const r = resolveAssetPath('D:\\cicada3301\\lost9999\\img.jpg', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: '/mnt/d/cicada3301/lost9999/img.jpg', foreign: false });
});
test('WSL: drive letter is lowercased, custom automount root honored', () => {
const r = resolveAssetPath('C:/Users/a/img.png', REPO, {
platform: 'linux',
wslMountRoot: '/windir/',
});
expect(r.abs).toBe('/windir/c/Users/a/img.png');
});
test('macOS: drive path is foreign (skip, never joined onto repoRoot)', () => {
const r = resolveAssetPath('D:/cicada3301/img.jpg', REPO, {
platform: 'darwin',
wslMountRoot: null,
});
expect(r).toEqual({ abs: null, foreign: true });
});
test('plain Linux (non-WSL): drive path is foreign', () => {
const r = resolveAssetPath('D:/x/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: null,
});
expect(r).toEqual({ abs: null, foreign: true });
});
test('win32: drive path stats natively, untouched', () => {
const r = resolveAssetPath('D:/x/img.jpg', REPO, { platform: 'win32' });
expect(r).toEqual({ abs: 'D:/x/img.jpg', foreign: false });
});
});
describe('resolveAssetPath — non-drive paths keep pre-#1835 behavior', () => {
test('POSIX absolute path passes through', () => {
const r = resolveAssetPath('/var/data/img.jpg', REPO, {
platform: 'linux',
wslMountRoot: null,
});
expect(r).toEqual({ abs: '/var/data/img.jpg', foreign: false });
});
test('relative path joins onto repoRoot', () => {
const r = resolveAssetPath('assets/img.jpg', REPO, {
platform: 'darwin',
wslMountRoot: null,
});
expect(r).toEqual({ abs: `${REPO}/assets/img.jpg`, foreign: false });
});
test('lookalike without separator after colon is NOT treated as a drive', () => {
const r = resolveAssetPath('notes:draft.md', REPO, {
platform: 'linux',
wslMountRoot: '/mnt',
});
expect(r).toEqual({ abs: `${REPO}/notes:draft.md`, foreign: false });
});
});
describe('parseWslAutomountRoot', () => {
test('defaults to /mnt on empty or unrelated config', () => {
expect(parseWslAutomountRoot('')).toBe('/mnt');
expect(parseWslAutomountRoot('[boot]\nsystemd=true\n')).toBe('/mnt');
});
test('reads [automount] root', () => {
expect(parseWslAutomountRoot('[automount]\nroot = /custom\n')).toBe('/custom');
});
test('ignores root under a different section', () => {
expect(parseWslAutomountRoot('[network]\nroot = /nope\n')).toBe('/mnt');
});
test('handles quotes, comments, and CRLF', () => {
const conf = '[automount]\r\nroot = "/win" # drives here\r\noptions = "metadata"\r\n';
expect(parseWslAutomountRoot(conf)).toBe('/win');
});
});
+28 -2
View File
@@ -79,12 +79,38 @@ describe('doctor checkCycleFreshness', () => {
expect(result.message).toMatch(/gbrain dream --source/);
});
test('source with NO last_full_cycle_at (never cycled) returns fail', async () => {
test('source with NO last_full_cycle_at (never cycled) returns warn, not fail (#2540)', async () => {
// #2540: never-cycled used to FAIL, which turned doctor permanently red
// on any install that doesn't cycle every local_path source (e.g. one
// nightly `dream --dir <vault>` plus other federated sources) — and on
// any source added minutes ago. It surfaces as a warning; only a source
// that HAS cycled and then went stale escalates to fail.
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('virgin');
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('fail');
expect(result.status).toBe('warn');
expect(result.message).toMatch(/never completed a full cycle/);
expect(result.message).toMatch(/gbrain dream --source/);
});
test('reporter case (#2540): one cycled vault + never-cycled siblings is warn, not permanent fail', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('nightly-vault', agoH(2)); // the one vault dreamt via --dir
await seed('federated-a'); // never cycled
await seed('federated-b'); // never cycled
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('warn');
expect(result.message).toMatch(/federated-a/);
expect(result.message).toMatch(/federated-b/);
expect(result.message).not.toMatch(/nightly-vault/);
});
test('a previously-cycled source gone stale still fails even next to never-cycled sources', async () => {
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
await seed('stale', agoH(72)); // real regression signal
await seed('virgin'); // never cycled — warn-only
const result = await checkCycleFreshness(engine, { nowMs: NOW });
expect(result.status).toBe('fail');
});
test('mixed sources: highest severity wins (fail > warn > ok)', async () => {
+104
View File
@@ -0,0 +1,104 @@
/**
* #1835 doctor `image_assets`: Windows drive paths (`D:/…`, `D:\`) written
* by a Windows gbrain install must not be reported as "missing from disk"
* on POSIX hosts that cannot resolve them.
*
* Behavioral test through the master-existing `buildChecks` seam it
* deliberately imports NOTHING introduced by this fix, so running this file
* against an unmodified master demonstrates the bug (image_assets WARNs
* "restore from git" for a drive path that was never lost).
*
* Pure translation-logic coverage (WSL /mnt mapping, wsl.conf parsing) lives
* in test/doctor-asset-paths.test.ts.
*/
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
import { readFileSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { buildChecks, type Check } from '../src/commands/doctor.ts';
let engine: PGLiteEngine;
let repoRoot: string;
// These assertions describe non-WSL POSIX hosts (macOS, plain Linux — every
// dev box + CI runner here). On real WSL the drive path is translated and
// statted instead; on win32 it stats natively. Skip there.
const onWsl = (() => {
try {
return process.platform === 'linux' && /microsoft/i.test(readFileSync('/proc/version', 'utf8'));
} catch {
return false;
}
})();
const skip = onWsl || process.platform === 'win32';
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
repoRoot = mkdtempSync(join(tmpdir(), 'gbrain-1835-'));
await engine.setConfig('sync.repo_path', repoRoot);
});
async function insertImage(storagePath: string, hash: string): Promise<void> {
await engine.executeRaw(
`INSERT INTO files (source_id, filename, storage_path, mime_type, content_hash)
VALUES ('default', 'img.jpg', $1, 'image/jpeg', $2)`,
[storagePath, hash],
);
}
async function imageAssetsCheck(): Promise<Check> {
const checks = await buildChecks(engine, []);
const check = checks.find((c) => c.name === 'image_assets');
expect(check).toBeDefined();
return check!;
}
describe('doctor image_assets — Windows drive paths on POSIX (#1835)', () => {
test.skipIf(skip)('D:/ path is skipped with a note, not reported missing', async () => {
await insertImage('D:/cicada3301/lost9999/img.jpg', 'h1');
const check = await imageAssetsCheck();
// Master joins the drive path onto repoRoot and WARNs "missing from
// disk … restore from git" — a false data-loss report.
expect(check.status).toBe('ok');
expect(check.message).toContain('Windows-drive path(s) skipped');
expect(check.message).not.toContain('restore from git');
});
test.skipIf(skip)('backslash D:\\ path is also skipped', async () => {
await insertImage('D:\\cicada3301\\lost9999\\img.jpg', 'h2');
const check = await imageAssetsCheck();
expect(check.status).toBe('ok');
expect(check.message).toContain('Windows-drive path(s) skipped');
});
test.skipIf(skip)('drive path skip does not mask a genuinely vanished asset', async () => {
await insertImage('D:/cicada3301/lost9999/img.jpg', 'h3');
await insertImage('assets/really-gone.png', 'h4');
const check = await imageAssetsCheck();
expect(check.status).toBe('warn');
expect(check.message).toContain('assets/really-gone.png');
// The unresolvable drive path is excluded from the checked denominator.
expect(check.message).toContain('1 of 1 image(s) missing');
expect(check.message).toContain('Windows-drive path(s) skipped');
});
test('present relative asset still resolves against repoRoot (regression guard)', async () => {
writeFileSync(join(repoRoot, 'here.png'), 'x');
await insertImage('here.png', 'h5');
const check = await imageAssetsCheck();
expect(check.status).toBe('ok');
expect(check.message).toContain('all present on disk');
});
});
+22
View File
@@ -96,6 +96,28 @@ describe('gbrain dream --dir <path> freshness stamp (#1869)', () => {
expect(await readLastFullCycleAt('mothballed')).toBeNull();
});
}, 60_000);
test('an ARCHIVED alias of the same path does not shadow the active source (#2540)', async () => {
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
// Ordinary shape: a source was archived and re-added under a new id
// pointing at the same checkout. Seed the archived twin FIRST so a
// filterless `LIMIT 1` scan finds it first.
await seedSource('retired-twin', true);
await seedSource('active-twin', false);
const report = await runDream(engine, ['--dir', brainDir, '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
// Pre-fix, resolveSourceForDir's exact match had no `archived = false`
// filter and no ORDER BY, so the archived twin won the lookup; dream's
// archived guard then (correctly) refused to stamp it — and the ACTIVE
// source silently never got its stamp, leaving doctor's cycle_freshness
// permanently stale on a healthy install.
expect(await readLastFullCycleAt('active-twin')).not.toBeNull();
expect(await readLastFullCycleAt('retired-twin')).toBeNull();
});
}, 60_000);
});
/**
+8 -8
View File
@@ -45,7 +45,7 @@ afterEach(() => {
function makeChatStub(scoresBySlot: Record<string, number[]>) {
let callIdx = 0;
const order = ['openai:gpt-4o', 'anthropic:claude-opus-4-7', 'google:gemini-1.5-pro'];
const order = ['openai:gpt-5.2', 'anthropic:claude-opus-4-7', 'deepseek:deepseek-v4-pro'];
return mock(async (opts: { model?: string }) => {
const model = opts.model ?? '';
callIdx++;
@@ -74,9 +74,9 @@ function makeChatStub(scoresBySlot: Record<string, number[]>) {
describe('gbrain eval cross-modal — runner verdict contract', () => {
test('PASS: 3 happy responses, all dims >=7', async () => {
const chatStub = makeChatStub({
'openai:gpt-4o': [9, 8],
'openai:gpt-5.2': [9, 8],
'anthropic:claude-opus-4-7': [8, 7],
'google:gemini-1.5-pro': [8, 8],
'deepseek:deepseek-v4-pro': [8, 8],
});
mock.module('../../src/core/ai/gateway.ts', () => ({
chat: chatStub,
@@ -105,9 +105,9 @@ describe('gbrain eval cross-modal — runner verdict contract', () => {
test('FAIL: one dim mean below 7', async () => {
const chatStub = makeChatStub({
'openai:gpt-4o': [9, 6],
'openai:gpt-5.2': [9, 6],
'anthropic:claude-opus-4-7': [8, 6],
'google:gemini-1.5-pro': [8, 6],
'deepseek:deepseek-v4-pro': [8, 6],
});
mock.module('../../src/core/ai/gateway.ts', () => ({
chat: chatStub,
@@ -130,9 +130,9 @@ describe('gbrain eval cross-modal — runner verdict contract', () => {
test('FAIL: min-score floor caught when one model scores <5 (Q2)', async () => {
const chatStub = makeChatStub({
'openai:gpt-4o': [9, 8],
'openai:gpt-5.2': [9, 8],
'anthropic:claude-opus-4-7': [8, 8],
'google:gemini-1.5-pro': [4, 8], // goal=4 trips the floor
'deepseek:deepseek-v4-pro': [4, 8], // goal=4 trips the floor
});
mock.module('../../src/core/ai/gateway.ts', () => ({
chat: chatStub,
@@ -155,7 +155,7 @@ describe('gbrain eval cross-modal — runner verdict contract', () => {
test('INCONCLUSIVE: 2 of 3 mock 5xx -> exit 2 contract (Q3)', async () => {
const chatStub = mock(async (opts: { model?: string }) => {
if (opts.model === 'openai:gpt-4o') {
if (opts.model === 'openai:gpt-5.2') {
return {
text: JSON.stringify({
scores: { goal: { score: 8 } },
+50 -12
View File
@@ -855,23 +855,61 @@ describeBoth('Engine parity — federated sourceIds[] secondary reads (#2200)',
expect(pg).toEqual(['beta-tag']); // default decoy excluded
});
function exactLinkShape(links: Awaited<ReturnType<BrainEngine['getLinks']>>): string[] {
return links.map(link => [
link.from_source_id,
link.from_slug,
link.to_source_id,
link.to_slug,
link.origin_source_id ?? null,
link.origin_slug ?? null,
link.link_type,
].join('::')).sort();
}
test('getLinks identical under sourceIds[] (all three endpoints scoped)', async () => {
const pg = (await pgEngine.getLinks('fed/doc', grant)).map(l => l.to_slug).sort();
const pglite = (await pgliteEngine.getLinks('fed/doc', grant)).map(l => l.to_slug).sort();
expect(pg).toEqual(pglite);
expect([...new Set(pg)]).toEqual(['fed/target']); // far-endpoint 'fed/outside' excluded
// F1: origin_slug nulled identically on both engines when origin is out-of-grant.
const pgOrigins = (await pgEngine.getLinks('fed/doc', grant)).map(l => l.origin_slug ?? null);
const pgliteOrigins = (await pgliteEngine.getLinks('fed/doc', grant)).map(l => l.origin_slug ?? null);
const pgLinks = await pgEngine.getLinks('fed/doc', grant);
const pgliteLinks = await pgliteEngine.getLinks('fed/doc', grant);
expect(exactLinkShape(pgLinks)).toEqual(exactLinkShape(pgliteLinks));
expect([...new Set(pgLinks.map(l => `${l.to_source_id}:${l.to_slug}`))])
.toEqual(['beta:fed/target']); // far-endpoint 'fed/outside' excluded
// F1: origin identity nulls identically when origin is out-of-grant.
const pgOrigins = pgLinks.map(l => [l.origin_source_id ?? null, l.origin_slug ?? null]);
const pgliteOrigins = pgliteLinks.map(l => [l.origin_source_id ?? null, l.origin_slug ?? null]);
expect(pgOrigins.sort()).toEqual(pgliteOrigins.sort());
expect(pgOrigins).not.toContain('fed/outside');
expect(pgOrigins).not.toContainEqual(['default', 'fed/outside']);
});
test('scalar getLinks preserves cross-source destination identity across engines', async () => {
const scalar = { sourceId: 'beta' };
const pg = await pgEngine.getLinks('fed/doc', scalar);
const pglite = await pgliteEngine.getLinks('fed/doc', scalar);
expect(exactLinkShape(pg)).toEqual(exactLinkShape(pglite));
expect(pg).toContainEqual(expect.objectContaining({
from_source_id: 'beta',
from_slug: 'fed/doc',
to_source_id: 'default',
to_slug: 'fed/outside',
}));
});
test('unscoped link reads expose exact endpoint identity across engines', async () => {
const pgLinks = await pgEngine.getLinks('fed/doc');
const pgliteLinks = await pgliteEngine.getLinks('fed/doc');
expect(exactLinkShape(pgLinks)).toEqual(exactLinkShape(pgliteLinks));
expect(pgLinks.every(link => link.from_source_id && link.to_source_id)).toBe(true);
const pgBacklinks = await pgEngine.getBacklinks('fed/doc');
const pgliteBacklinks = await pgliteEngine.getBacklinks('fed/doc');
expect(exactLinkShape(pgBacklinks)).toEqual(exactLinkShape(pgliteBacklinks));
expect(pgBacklinks.every(link => link.from_source_id && link.to_source_id)).toBe(true);
});
test('getBacklinks identical under sourceIds[] (both endpoints scoped)', async () => {
const pg = (await pgEngine.getBacklinks('fed/doc', grant)).map(l => l.from_slug).sort();
const pglite = (await pgliteEngine.getBacklinks('fed/doc', grant)).map(l => l.from_slug).sort();
expect(pg).toEqual(pglite);
expect(pg).toEqual(['fed/target']);
const pg = await pgEngine.getBacklinks('fed/doc', grant);
const pglite = await pgliteEngine.getBacklinks('fed/doc', grant);
expect(exactLinkShape(pg)).toEqual(exactLinkShape(pglite));
expect(pg.map(l => `${l.from_source_id}:${l.from_slug}`)).toEqual(['beta:fed/target']);
});
test('getTimeline identical under sourceIds[]', async () => {
+6 -2
View File
@@ -26,8 +26,12 @@ if (skip) {
}
describeE2E('E2E: JSONB roundtrip — v0.12.1 reliability wave', () => {
beforeAll(async () => { await setupDB(); });
afterAll(async () => { await teardownDB(); });
// 60s hook budget: setupDB runs connect + the full migration chain, which
// exceeds bun's default 5s hook timeout on loaded CI runners. Hooks do NOT
// inherit a test's third-arg timeout (verified on bun 1.3.14) — they need
// their own second-arg budget. Same pattern as op-checkpoint-jsonb-parity.
beforeAll(async () => { await setupDB(); }, 60_000);
afterAll(async () => { await teardownDB(); }, 60_000);
test('putPage writes frontmatter as object, not double-encoded string', async () => {
const engine = getEngine();
@@ -1,18 +1,25 @@
/**
* E2E regression for #1178: migration v66 (`embed_stale_partial_index`)
* pre-drops an invalid CONCURRENTLY-build remnant using
* `DO $$ BEGIN ... EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS <name>'; END IF;
* END $$;`. Postgres rejects CONCURRENTLY from any function/EXECUTE context,
* so the guard's EXISTS check passed but the EXECUTE inside it always threw
* "DROP INDEX CONCURRENTLY cannot be executed from a function" the
* migration only failed on brains carrying an invalid-index leftover from a
* prior interrupted CREATE INDEX CONCURRENTLY.
* E2E regression for #1178: 11 historical migrations (v14, v34, v38, v66, v71,
* v91, v96, v97, v103, v104, v112) pre-drop an invalid CONCURRENTLY-build
* remnant using `DO $$ BEGIN ... EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS
* <name>'; END IF; END $$;`. Postgres rejects CONCURRENTLY from any
* function/EXECUTE context, so the guard's EXISTS check passed but the
* EXECUTE inside it always threw "DROP INDEX CONCURRENTLY cannot be executed
* from a function" each migration only failed on brains carrying an
* invalid-index leftover from a prior interrupted CREATE INDEX CONCURRENTLY.
*
* The fix replaces the DO block with dropInvalidConcurrentIndex(): the
* validity probe runs as a plain application-level SELECT, and the DROP (when
* needed) runs as its own top-level statement. This test reproduces the
* issue's exact repro steps against real Postgres and confirms the migration
* now recovers instead of throwing.
* The fix (introduced for the issue-reported migration, v66, in a prior PR)
* replaces the DO block with dropInvalidConcurrentIndex(): the validity probe
* runs as a plain application-level SELECT, and the DROP (when needed) runs
* as its own top-level statement. This PR applies the same helper to the
* remaining 10 historical sites the issue's own re-scan found the
* "recurrence" half of #1178 (5 new copies had shipped since the bug was
* first reported, via copy-paste of the nearest similar migration).
*
* This test reproduces the issue's exact repro steps against real Postgres
* for the two most structurally distinct sites (v66, already covered by the
* prior PR, kept here for full-suite context; v112, a second independent
* site) and confirms both migrations now recover instead of throwing.
*
* Real Postgres only gated by DATABASE_URL, skips otherwise.
*
@@ -96,4 +103,19 @@ describeE2E('migration invalid-remnant recovery (#1178)', () => {
// wouldn't catch a spurious drop+recreate (codex review, #1178).
expect(await indexOid('idx_chunks_embedding_null')).toBe(oidBefore);
});
test('v112 (pages_links_extracted_at_idx, a second independent site from this batch): same recovery', async () => {
await plantInvalidIndex(
'pages_links_extracted_at_idx',
`CREATE INDEX pages_links_extracted_at_idx ON pages (source_id, links_extracted_at)`,
);
expect(await isIndexValid('pages_links_extracted_at_idx')).toBe(false);
const v112 = MIGRATIONS.find(m => m.version === 112);
expect(v112?.handler).toBeDefined();
await expect(v112!.handler!(getEngine())).resolves.toBeUndefined();
expect(await isIndexValid('pages_links_extracted_at_idx')).toBe(true);
});
});
+15 -2
View File
@@ -274,10 +274,23 @@ describe('openclaw-plugin-load-real (Tier 2 e2e)', () => {
// level fixture proved openclaw is installed and reachable.
let registerContextEngine: ((id: string, factory: () => unknown) => void) | undefined;
// Minimal structural shape of the openclaw plugin SDK surface this
// test uses. `openclaw` is deliberately NOT a declared dependency —
// the test probes whatever install is present at runtime — so the
// import result is cast to this local interface instead of letting
// TypeScript type-check against whichever openclaw version happens
// to be resolvable from an ancestor node_modules. That keeps
// `bunx tsc --noEmit` hermetic on a clean checkout (#2729); the
// export's presence/shape is still verified at runtime below.
interface OpenclawPluginSdk {
registerContextEngine?: (id: string, factory: () => unknown) => void;
}
const importErrors: string[] = [];
try {
// @ts-ignore — bare specifier resolution depends on node_modules.
const sdk = await import('openclaw/plugin-sdk');
// @ts-ignore — bare specifier; whether this resolves (TS2307 or not)
// depends on ambient node_modules, so @ts-expect-error would flip.
const sdk = (await import('openclaw/plugin-sdk')) as unknown as OpenclawPluginSdk;
registerContextEngine = sdk.registerContextEngine;
} catch (err) {
importErrors.push(`bare 'openclaw/plugin-sdk': ${err instanceof Error ? err.message : String(err)}`);
+42 -2
View File
@@ -38,11 +38,11 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { importFromFile } from '../../src/core/import-file.ts';
import { importFromContent, importFromFile } from '../../src/core/import-file.ts';
import { runExtractCore } from '../../src/commands/extract.ts';
import { extractTakes } from '../../src/core/cycle/extract-takes.ts';
import { runExtractFacts } from '../../src/core/cycle/extract-facts.ts';
import { stripFactsFence } from '../../src/core/facts-fence.ts';
import { parseFactsFence, stripFactsFence } from '../../src/core/facts-fence.ts';
let engine: PGLiteEngine;
let brainDir: string;
@@ -305,6 +305,46 @@ describe('get_page privacy strip via stripFactsFence({keepVisibility:["world"]})
expect(remoteBody).not.toContain('PRIVATE_DETAIL_PROOF'); // remote MCP strips
expect(remoteBody).toContain('Founded Acme in 2017'); // world fact retained
});
test('remote put_page round-trip preserves an existing private-only facts fence', async () => {
const slug = 'people/private-only-facts';
await importFromContent(engine, slug, `---
type: person
title: Private Only Facts
slug: ${slug}
---
# Private Only Facts
## Facts
<!--- gbrain:facts:begin -->
| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |
|---|-------|------|------------|------------|------------|------------|-------------|--------|---------|
| 1 | PRIVATE_ONLY_FACT | preference | 0.9 | private | medium | 2026-07-30 | | meeting | |
<!--- gbrain:facts:end -->
`, { noEmbed: true, sourceId: 'default' });
const trusted = await engine.getPage(slug, { sourceId: 'default' });
expect(trusted).not.toBeNull();
if (!trusted) return;
const remoteBody = stripFactsFence(trusted.compiled_truth ?? '', { keepVisibility: ['world'] });
expect(remoteBody).toContain('gbrain:facts:begin');
expect(remoteBody).not.toContain('PRIVATE_ONLY_FACT');
await importFromContent(engine, slug, `---
type: person
title: Private Only Facts
slug: ${slug}
---
${remoteBody}`, { noEmbed: true, sourceId: 'default', remote: true });
const after = await engine.getPage(slug, { sourceId: 'default' });
expect(after?.compiled_truth).toContain('PRIVATE_ONLY_FACT');
expect(parseFactsFence(after?.compiled_truth ?? '').facts).toHaveLength(1);
});
});
afterAll(() => {
+14 -3
View File
@@ -26,10 +26,21 @@ describe('lookupEmbeddingPrice — first-class providers', () => {
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
});
test('Voyage voyage-4-large at $0.18/MTok (v0.35.1.0+)', () => {
const r = lookupEmbeddingPrice('voyage:voyage-4-large');
// Voyage v4 family, verified against docs.voyageai.com/docs/pricing 2026-07-28.
test.each([
['voyage:voyage-4-large', 0.12],
['voyage:voyage-4', 0.06],
['voyage:voyage-4-lite', 0.02],
])('Voyage %s at $%d/MTok', (model, expected) => {
const r = lookupEmbeddingPrice(model);
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
if (r.kind === 'known') expect(r.pricePerMTok).toBe(expected);
});
// voyage-4-nano is the open-weight variant with no hosted rate published;
// it must stay unpriced so callers say "estimate unavailable" (see table comment).
test('voyage-4-nano is deliberately unpriced', () => {
expect(lookupEmbeddingPrice('voyage:voyage-4-nano').kind).toBe('unknown');
});
test('ZeroEntropy zembed-1 at $0.05/MTok (v0.35.1.0+)', () => {

Some files were not shown because too many files have changed in this diff Show More