Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 fcbe037bbc test(e2e): 60s hook timeout for jsonb-parity setup/teardown
The #2339 parity guard's beforeAll runs setupDB (full migration chain)
under bun's default 5s hook timeout, which flaked on a slow CI runner
(setupDB hit 5001ms). Other e2e suites already pass explicit hook
timeouts; bring this file in line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 11:07:08 -07:00
Garry Tan 4e0487ba32 Merge remote-tracking branch 'origin/master' into fix/backlog-ff2 2026-07-24 11:06:23 -07:00
1f319e6d5a fix(scripts): capture check/shard rc before watchdog teardown in no-timeout fallback (#2864) (#3340)
On machines with neither gtimeout nor timeout on PATH, run-verify-parallel.sh
and run-unit-parallel.sh fall back to a bg-pid + sleep-watchdog cap. Both
read $? only after tearing the watchdog down (kill + wait on cap_pid), so the
sentinel .exit files recorded the killed watchdog's status — 143 — instead of
the check/shard's own exit code. Every run reported total failure (verify:
pass=0 fail=31; unit: rc=143 per shard) while every per-check/shard log
showed success.

Capture rc immediately after `wait $pid` in both scripts, and reap the
watchdog's sleep child (pkill -P, children-first — the same orphan quirk the
heartbeat cleanup documents) so the fallback stops leaking one sleep per
check/shard.

Regression tests force the fallback branch hermetically on any host via a
curated PATH with no timeout binaries: the verify dispatcher runs from a
tempdir copy with a stubbed `bun`, pinning exit 0 + all-zero sentinels when
checks pass and the check's own rc (not 143) when one fails; the unit wrapper
runs real two-shard fixture passes, pinning rc=0 sentinels and a real
failure's rc=1.

Co-authored-by: paul-0320 <paul@ymyd.co.kr>
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:48:34 -07:00
Time AttakcandSong d15e2ab8cf fix(webhook): extract links for incremental push syncs (#2850) (#3337)
* test(webhook): pin sync extraction contract (#2849)

* test(webhook): target the submitted sync payload (#2849)

* fix(webhook): run extraction in sync job (#2849)

* fix(sync): align push trigger extraction (#2849)

Co-authored-by: Song <patentsong@gmail.com>
2026-07-23 18:48:27 -07:00
e1919fab9f reland: fix(embed): stamp gateway-resolved model in content_chunks.model, not compiled default (#2846) (#3343)
* fix(embed): stamp gateway-resolved model in content_chunks.model, not compiled default (#2846)

upsertChunks fell back to the compile-time DEFAULT_EMBEDDING_MODEL
('zeroentropyai:zembed-1') when a ChunkInput carried no explicit `model`.
The embed pipeline (src/commands/embed.ts) builds ChunkInputs without a
`model` field, so rows whose vectors were produced by the config-resolved
model (e.g. openai:text-embedding-3-large) were mislabeled with the
hardcoded default — corrupting the provenance that signature-drift
staleness and dimension-migration logic depend on.

Both engines now resolve the gateway's runtime embedding model once per
upsert and use it as the fallback, mirroring the existing resolve-then-
default pattern used for schema sizing. Regression test added (pglite);
verified via negative control that it fails against the old fallback.

This is a write-path change (upsertChunks), not a search-path change, so
retrieval eval replay is not applicable.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* test: Lane A.7 pins gateway-resolved chunk model, not compiled default

#2846 changed upsertChunks' fallback from DEFAULT_EMBEDDING_MODEL to the
gateway-resolved runtime model. Lane A.7 still pinned the old fallback,
and the test preload (test/helpers/legacy-embedding-preload.ts) pins the
gateway to openai:text-embedding-3-large for every test process — so the
original #2846 landing failed this test deterministically and got batch-
reverted. The test now asserts the resolved model (the intended #2846
semantics) while keeping the CDX2-4 bare-literal regression guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: SailorJoe6 <SailorJoe6@Gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-23 18:48:22 -07:00
Time Attakcandzay 5e665c1c06 fix: clarify PGLite data-dir lock contention (#2658) (#3336)
Co-authored-by: zay <richardicruz25@gmail.com>
2026-07-23 18:35:22 -07:00
f5e5736f09 feat(ai): dashscope-rerank recipe (DashScope serves PLURAL /reranks under compatible-api) (#2644) (#3328)
DashScope's OpenAI-compatible rerank endpoint lives at
{base}/compatible-api/v1/reranks — PLURAL leaf, different base path from
the embedding surface (compatible-mode). Reusing llama-server-reranker
against DashScope forces users to hand-patch the recipe's '/rerank' leaf
in node_modules, which every upgrade silently reverts (and llama.cpp
genuinely serves singular /rerank, so changing that recipe would break
real llama.cpp users).

New dedicated recipe rides the v0.40.6.1 recipe-pluggable reranker path:
- id dashscope-rerank, base_url_default compatible-api/v1 (intl), ZE wire
- path '/reranks', default_timeout_ms 30s, 5MB payload ceiling
- models: only qwen3-rerank (live-verified 200; gte-rerank-v2 is rejected
  by the compat surface with 'Unsupported model for OpenAI compatibility
  mode', so it is deliberately not listed)
- separate recipe (not a reranker touchpoint on dashscope) because
  provider_base_urls is keyed by recipe id and the two capabilities need
  different prefixes — same topology as llama-server vs
  llama-server-reranker

Tests: recipe shape smoke mirroring recipe-llama-server-reranker.test.ts
(path/timeout/payload pins, /v1/v1 concat guard, auth resolve, sibling
recipe isolation). bun test test/ai/: 322 pass / 0 fail.

Co-authored-by: Yicon <charlieyiconghuang@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 18:35:18 -07:00
cd252b080b fix(test): isolate GBRAIN_HOME in hybrid-reranker integration test (#1527) (#2640) (#3327)
The four `hybridSearch — reranker enabled (reorder)` cases stub the gateway
at 1536 dims (DIMS). Since v0.36.3.0 hybridSearch resolves the embedding
column via loadConfig(), whose precedence is
cfg.embedding_dimensions > gateway dims > default. On any machine whose
~/.gbrain/config.json sets embedding_dimensions to something other than 1536
(e.g. text-embedding-3-small at 1280), the real config outranks the stub: the
1536-d stub vector fails the gateway dim check, the error is swallowed, search
falls back to keyword-only, and the reranker never runs (rerankerFn gets 0
docs, rerank_score undefined). Green in CI only because a fresh runner has no
config file — deterministic red on a contributor's machine.

Fix (test-only): isolate GBRAIN_HOME to an empty tmpdir in beforeAll so
loadConfig() returns null and the stub's dims win, then restore it and clean
up in afterAll. Same idiom as emptyHome() in
test/ai/gateway-probe-chat-model.test.ts.

Verified with a planted ~/.gbrain/config.json at 1280 dims: 2 pass / 4 fail
before, 6 pass / 0 fail after; still green with no config file. typecheck clean.

Fixes #1527

Co-authored-by: Willisbest <132954469+Willisbest@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 18:35:13 -07:00
97bdf6acc1 fix(health): count 'entity' pages in graph health metrics (#2639) (#3330)
Reland of #2639, reverted with its batch in 68e4cebd. getHealth's
entity_pages CTE and the top-linked-pages query only match the legacy
'person' and 'company' types, so brains using the gbrain-base-v2 pack's
'entity' type report 0% entity link/timeline coverage in `gbrain health`.
Add 'entity' to both queries in both engines (PGLite + Postgres, in
lockstep per the engine-parity rule).

Reland fix (the batch-red root cause): the original PR's test expected
the entities/project-x page to appear in orphan_pages, but #3023's shared
orphan-reporting policy (landed before #2639 merged) excludes the
'entities' first segment from orphan reporting, so the test failed on
master. Orphan expectations now account for the policy exclusion.

Co-authored-by: Tyler Robinson <tylr.rob@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:35:09 -07:00
Time Attakcandspiky02plateau 900ee3c678 perf(contextual-retrieval): bound per-chunk synopsis concurrency (#2628) (#3326)
Replace the strictly sequential per-chunk synopsis loop with a bounded
sliding worker pool (existing runSlidingPool helper). Results land in
chunk order via index-addressed writes; code chunks still bypass the
wrapper; embedding remains one page-level batch after all synopses.

New knob GBRAIN_CONTEXTUAL_CHUNK_CONCURRENCY, default 4, clamped to
[1,16]; 1 reproduces the prior sequential behavior exactly. Each chunk
task still acquires/releases the global synopsis rate-lease, which
remains the cross-worker governor; the lease id now travels from
acquire to release instead of shared mutable state, and lease waits
are abort-responsive.

At 20-45s per synopsis call, a 120-chunk transcript page previously
needed 60-90+ min wall time and routinely outlived job timeouts.

Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
2026-07-23 18:35:05 -07:00
Time Attakcandspiky02plateau 96465d8c35 fix(migrations): let force-retry escape completed ledger entries (#2616) (#3325)
statusForVersion short-circuited on any 'complete' entry before checking
the trailing 'retry' marker, so --force-retry appended an inert row and a
version marked complete with zero work done could never be re-run without
hand-editing completed.jsonl. Check retry-latest first: an explicit
--force-retry now yields 'pending' even past an earlier 'complete', while
a stray 'partial' after 'complete' still cannot regress the version.

Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
2026-07-23 18:34:59 -07:00
Time Attakcandspiky02plateau 7fdecd5c01 fix(minions): default timeout for contextual reindex (#2611) (#3323)
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
2026-07-23 18:17:14 -07:00
06001248ef fix(storage): Supabase signed URLs — prepend /storage/v1 (#2565) (#3320)
SupabaseStorage.getSignedUrl built the download URL as `${projectUrl}${signedURL}`,
but Supabase's sign API returns `signedURL` relative to the Storage API root
(/object/sign/<bucket>/<path>?token=...), so the generated link dropped /storage/v1
and returned 404. Now prepends `${projectUrl}/storage/v1`, tolerating an
already-absolute URL or a value that already carries the prefix. `gbrain files
signed-url` links resolve again.

Co-authored-by: FloridaStyle <daniel.wiggins@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 18:16:21 -07:00
9e28379038 fix: handle <think> reasoning tags in parseExtractorOutput (#2559) (#3318)
Reasoning models (MiniMax-M3, DeepSeek-R1, etc.) return <think>...</think>
tags in the content field before the actual JSON output. This caused
parseExtractorOutput to fail in two ways:

1. The fence regex /^\`\`\`(json)?...$/ requires the fence at text start;
   <think> preceding it prevents matching, so the raw text (with trailing
   fences) hits JSON.parse and throws.

2. When think tags contain [ or { characters, indexOf finds them inside
   the reasoning block instead of the actual JSON array.

Changes:
- Strip <think>...</think> tags before any parsing (covers all reasoning models)
- Add JSON.parse fallback: truncate at last ] or } to handle trailing
  noise (leftover markdown fences after stripping)

Tests: 28/28 pass (3 new cases for think tags + trailing noise).

Co-authored-by: qaz8545355 <603191978@qq.com>
Co-authored-by: qaz8545355 <junjun@openclaw.local>
2026-07-23 18:16:17 -07:00
48d83bd200 fix(cycle): extract_facts guard requires live backing page, not just non-NULL entity_slug (#2497) (#3321)
The empty-fence guard counted every `row_num IS NULL AND entity_slug IS NOT NULL`
row as a pending v0_32_2 backfill, but the inline facts writer keeps producing
rows of exactly that shape post-migration: when a resolved slug has no fenceable
page (slugify-floor / stub-guard-blocked unprefixed slugs like `wingman`,
`people-jane-doe`), backstop.ts falls through to a DB-only insert with row_num
NULL. Those rows are structurally unfenceable — no page to fence onto, and the
ledger-complete migration won't re-run — so they jammed the phase forever
(~16/day observed) and the warning advised a no-op `apply-migrations --yes`.

Discriminator: a row is a genuine backfill candidate only if its entity_slug
resolves to a LIVE page in the same source (EXISTS in `pages` with deleted_at
NULL) — mirroring the migration's Phase B, which only fences slugs that map to
a writable page. Genuine pre-v0.32.2 rows (their entity page exists) still gate;
inline-writer unfenceable rows no longer do. Warning text updated to name the
"entity page present, not yet fenced" condition.

Regression tests pin both sides: unfenceable rows (no page / soft-deleted page)
do NOT gate and the phase converges; a legacy row WITH a backing page still
gates. Fails pre-fix, passes post-fix.

(#2484)




Reland note: original merge (53c90869) was batch-reverted (4b6cf32c) —
the guard-semantics change broke test/phantom-redirect.test.ts
'round 2 P1: legacy-row guard fires BEFORE phantom-redirect pass',
which seeded a legacy row WITHOUT a backing page and expected the
guard to fire. Under the new (intended) semantics such a row is
structurally unfenceable and must NOT gate. Fixed by seeding a live
backing page for the legacy row, preserving what the test pins
(guard fires before the phantom-redirect pass).

Co-authored-by: Javier Aldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:15:24 -07:00
Time AttakcandSean Gearin 454c26ab56 fix(init): point soul-audit hint at the conversational skill, not a nonexistent CLI verb (#2486) (#3314)
Co-authored-by: Sean Gearin <sean@indistinct.ai>
2026-07-23 18:15:20 -07:00
b3891fa7fc fix(chunkers/code): tolerate tiktoken special tokens in estimateTokens (#2453) (#3315)
Code legitimately contains tiktoken special-token strings (e.g. CLIP/GPT tokenizers embed the literal <|endoftext|>). The default encode() uses disallowed_special='all' and THROWS on those, crashing reindex-code on valid source files. Re-encode treating them as ordinary text (allowed=[], disallowed=[]); heuristic fallback if even that fails. A token COUNT needs no special-token semantics.

Co-authored-by: Jim Tang <jimruitang@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 18:15:15 -07:00
ca04874c8f fix: Bun+Windows write-through EEXIST, non-Anthropic --max-cost pricing, dream-page exclusion in enrich (#2407) (#3316)
* fix(write-through): guard mkdir against EEXIST on Bun+Windows



* fix(budget): resolve non-Anthropic model pricing via canonical table under --max-cost



* fix(enrich): exclude dream-generated pages from thin candidates



---------

Co-authored-by: nguyenchiviet <40517873+nguyenchiviet@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 18:02:03 -07:00
Time AttakcandHaoqian 9b3f1c6786 fix dream orphan source scope (#2368) (#3344)
Co-authored-by: Haoqian <snvtac@qq.com>
2026-07-23 18:01:07 -07:00
Time AttakcandNoetherly 2944d9b7ae fix(dims): handle prefixed model IDs on openai-compatible path (#2325) (#3309)
OpenRouter (and potentially other proxy providers) expose OpenAI's
text-embedding-3 models with a provider prefix in the model ID, e.g.
`openai/text-embedding-3-large` rather than bare `text-embedding-3-large`.

`dimsProviderOptions()` checks `modelId.startsWith('text-embedding-3')`
which fails for the prefixed form, so the `dimensions` parameter is never
sent. The upstream provider returns its native dimensionality (3072 for
-large) instead of the configured value (e.g. 1536), causing an immediate
"dim mismatch" error on first embed.

The default OpenRouter embedding (`text-embedding-3-small` at 1536d)
masked this because its native size happens to match the default config.
The bug surfaces when using `-large`, or `-small` with a non-1536 dim
(512, 768, 1024 — all listed in the recipe's `dims_options`).

Fix: strip the provider prefix before the `startsWith` check. The full
prefixed ID is preserved in the error message for user clarity.

Co-authored-by: Noetherly <280958447+noetherly@users.noreply.github.com>
2026-07-23 18:01:01 -07:00
eba9680775 feat(ai): claude-cli recipe for native gateway-based subagent dispatch (#2277) (#3310)
* feat(subagent): claude-cli MessagesClient adapter (baseline, no tool use)

Closes #334 (partially — text-only baseline; tool use lands in the next
commit on this branch).

Adds a MessagesClient adapter that shells out to `claude --print
--output-format json --model <model>` instead of the Anthropic SDK. When
`GBRAIN_USE_CLAUDE_CLI=1` is set, the subagent worker registers the adapter
in place of the SDK client; the default path (Anthropic SDK with
ANTHROPIC_API_KEY) is unchanged when the env var is unset or set to
anything else.

The benefit is that Claude Max subscribers can run Minions subagents
against their existing OAuth subscription, no ANTHROPIC_API_KEY needed.

New: src/core/minions/handlers/claude-cli-adapter.ts
- Implements the MessagesClient interface exported from subagent.ts.
- Strips provider prefixes (`anthropic:`, `litellm:`) from the model id
  because `claude --print` only accepts CLI-native aliases (`sonnet`,
  `opus`, `haiku`, or the bare `claude-*-N-M` form).
- Flattens the Anthropic messages array into a single text prompt for
  claude-cli stdin. Tool blocks (tool_use / tool_result) are stringified
  as placeholders so multi-turn conversations stay coherent in this
  baseline; native tool_use round-tripping is the follow-up commit.
- Spawns claude with stdio piped, captures stdout, parses the
  `{type:"result", subtype:"success", result, usage, ...}` JSON envelope,
  and returns it as a properly shaped Anthropic.Message with
  `stop_reason: 'end_turn'`.
- Token totals propagate from the claude usage block so the subagent
  handler's `ctx.updateTokens()` reports usable numbers.
- AbortSignal is wired through to SIGTERM the child so the subagent loop's
  cancellation path stays correct.

Modified: src/commands/jobs.ts (worker registration)
- Conditionally constructs a MessagesClient via the new adapter when
  GBRAIN_USE_CLAUDE_CLI=1.
- Passes it into makeSubagentHandler({ engine, client: subagentClient }).
- Logs `[minion worker] subagent routing via claude-cli (GBRAIN_USE_CLAUDE_CLI=1)`
  on startup so the env var status is operator-visible.

Limitations of this commit (addressed in the follow-up):
- Tool use is not yet supported. Tools in params.tools are ignored; the
  adapter returns a single text block with stop_reason='end_turn'.
- Token counts come from claude-cli's reporting and may not match the
  Anthropic API's accounting precisely (especially for cache tiers).

Original design from #334; this commit preserves that author's attribution.
The follow-up commits on this branch carry the tool-use implementation.

* feat(subagent): tool use + context isolation + convention rename on top of jarvisdoes baseline

Builds on the previous commit (jarvisdoes's #334 baseline) by adding three
things the upstream issue called out as gaps or that surfaced during review:

1. Tool use support via system-prompt-instructed JSON emission.
2. Context isolation flags so claude-cli does not load operator-level
   CLAUDE.md, skills, and local project context into every subagent call.
3. Env var rename from GBRAIN_USE_CLAUDE_CLI=1 to
   GBRAIN_SUBAGENT_PROVIDER=claude-cli to match the existing
   GBRAIN_<noun>_<role>=<value> convention used by GBRAIN_CHAT_MODEL,
   GBRAIN_EMBEDDING_MODEL, GBRAIN_EXPANSION_MODEL.

## Tool use

The MessagesClient interface returns Anthropic.Message objects whose
content array may include tool_use blocks. The subagent handler filters
those blocks and dispatches each tool, so any backend that produces
correctly shaped tool_use blocks gets the same loop behavior as the
Anthropic SDK.

The adapter injects a system-prompt addendum describing the tool registry
plus an emission protocol:

  <use_tools>
  [{"id": "...", "name": "...", "input": {...}}, ...]
  </use_tools>

After the response comes back, extractToolCalls() scans for the block,
parses the JSON (tolerant of optional ```json fencing), and converts each
entry into a tool_use content block. Multiple parallel tool calls in one
turn are supported via the array shape; this is the exact case that
breaks today on the codex-proxy / litellm GPT-5.x bridge where parallel
tool-call response IDs get dropped.

Defensive fallbacks:
 - Malformed JSON inside the block: drop to text-only, stop_reason='end_turn'.
 - Unterminated <use_tools> (no close tag): drop to text-only.
 - Model omits id field: adapter synthesizes a toolu_claude_cli_<rand> id.
 - Empty response: still hand the subagent loop a well-formed content
   array so the .filter chain does not crash.

## Context isolation

claude-cli auto-discovers CLAUDE.md from cwd upward and injects the
operator's skills + plugins + auto-memory into the default system prompt.
On a real install that is ~42-65k tokens of contamination per subagent
call, with both cost and behavioral consequences (the subagent picks up
the operator's coding conventions, opinions, and preferences).

The maximum suppression that still preserves OAuth / Claude Max
subscription auth is:
 - Spawn from a dedicated clean cwd (tmpdir-based) so LOCAL CLAUDE.md
   auto-discovery has nothing to find. -13k tokens on a real gbrain
   install where CLAUDE.md is substantial.
 - --disable-slash-commands so skill resolution does not pull in
   /skill-name handlers.
 - --system-prompt <gbrain prompt> so the default system prompt is
   replaced rather than appended to.

The --bare flag would also strip user-level ~/.claude/CLAUDE.md but it
forces ANTHROPIC_API_KEY auth, defeating the whole point of this adapter.
The remaining ~42k cached tokens from user-level instructions are
accepted as a cost-trivial trade-off because the Max subscription absorbs
the per-call cost. Behavioral contamination is mitigated by gbrain's
strong per-call system prompt overriding any operator-level drift.

## Env var rename

Surveyed all ~140 GBRAIN_* env vars in src/. The codebase uses three
patterns: GBRAIN_NO_<feature> (negative toggles), GBRAIN_<noun>_<role>
=<value> (routing keys), GBRAIN_ALLOW_<feature> (permissive toggles).
GBRAIN_USE_* does not appear anywhere except jarvisdoes's original
commit; it would introduce a fourth pattern.

GBRAIN_SUBAGENT_PROVIDER=claude-cli aligns with the routing-keys family
and is value-extensible — adding codex-cli / meridian-proxy / etc. later
means a new value, not a new env var. The scope ('SUBAGENT_*') is also
unambiguous about which calls the toggle covers; GBRAIN_USE_CLAUDE_CLI
was silent on whether it applied to all gbrain LLM calls or only the
subagent path.

Unknown values are rejected with a fail-fast error message naming the
two valid values rather than silently falling through to the default.

## Tests

New file: test/claude-cli-adapter.test.ts — 12 tests, 33 assertions:
 - Text-only round trip (single text block, usage propagation, end_turn).
 - Provider prefix stripping ('anthropic:claude-sonnet-4-6' -> 'claude-sonnet-4-6').
 - Single tool_use parsing.
 - Multiple parallel tool calls in one block (the case that triggered
   the codex-proxy regression).
 - Fenced JSON inside <use_tools> block.
 - Model-omitted id gets synthesized to toolu_claude_cli_<rand>.
 - Malformed JSON falls back to text.
 - Unterminated block falls back to text.
 - AbortSignal SIGTERMs the child.
 - Error envelope rejected with informative message.
 - Non-JSON output rejected with raw-output excerpt in the error.
 - argv + cwd assertion: --disable-slash-commands + --system-prompt are
   present and cwd is the dedicated tmpdir.

Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN that emits a
scripted --output-format json envelope, so the suite runs without
claude-cli installed and without API credits.

* feat(ai): claude-cli recipe with native gateway integration (supersedes #334 baseline)

Replaces the MessagesClient adapter + GBRAIN_USE_CLAUDE_CLI=1 env-var
gate from the previous commit on this branch with a proper gateway recipe.
The recipe path gives per-call routing as a native capability: a model
string like `claude-cli:claude-sonnet-4-6` lands here while a sibling
`litellm:gpt-5.4` continues through the litellm-proxy / codex-proxy path
in the same worker. No global env-var switch, no agent.use_gateway_loop
bypass, no MessagesClient injection at jobs.ts worker startup.

The previous commit on this branch (jarvisdoes baseline) is preserved
in the history for #334 authorship attribution. Its functional changes
are backed out here because the recipe pattern is gbrain's established
integration seam; introducing a parallel MessagesClient + env-var path
would have created two routing mechanisms competing for the same job.

New: src/core/ai/recipes/claude-cli.ts
- Recipe declaration: id 'claude-cli', tier 'native', implementation
  'claude-cli', chat-only (no embedding or expansion touchpoints).
- Models: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001.
- supports_tools and supports_subagent_loop both true.
- supports_prompt_cache false because the CLI handles caching internally
  and does not surface cache_control via the standard control plane.
- auth_env.required is the empty array because the CLI owns auth (OAuth
  session managed by `claude login`).
- Friendly aliases mirror the `anthropic` recipe: `sonnet`, `haiku`,
  `opus` and the same legacy-id rewrites for back-compat with stale
  config strings.

New: src/core/ai/providers/claude-cli-language-model.ts
- ClaudeCliLanguageModel class implementing the ai-sdk LanguageModelV2
  interface.
- doGenerate: renders the ai-sdk prompt array into a system text + user
  text, injects the use_tools protocol instructions when tools are
  present, spawns `claude --print --output-format json --model <X>
  --disable-slash-commands --system-prompt <gbrain prompt>` from a
  dedicated tmpdir (contamination suppression: no local CLAUDE.md
  auto-discovery), parses the JSON envelope, extracts <use_tools>
  blocks, and returns ai-sdk-shaped LanguageModelV2Content (text +
  tool-call parts with stringified-JSON input matching the V2 contract).
- Tolerates fenced JSON inside use_tools blocks, malformed JSON
  (falls back to text), missing close tag (falls back to text),
  model-omitted ids (synthesizes toolu_claude_cli_<rand>).
- Parallel tool calls in one block round-trip cleanly: this is the
  case that drops IDs on the litellm + codex-proxy bridge today.
- AbortSignal SIGTERMs the child for proper cancellation.
- doStream throws not-supported (gateway.toolLoop is non-streaming).

Modified: src/core/ai/gateway.ts
- Adds case 'claude-cli' to instantiateChat (returns ClaudeCliLanguageModel).
- Adds case 'claude-cli' to instantiateExpansion (same wrapper, reserved
  for a future expansion touchpoint declaration).
- Adds case 'claude-cli' to instantiateEmbedding (throws, no embedding
  model, mirrors the native-anthropic path).
- Lazy require() at the call site keeps the gateway module load cheap
  for users who never use the claude-cli path.

Modified: src/core/ai/recipes/index.ts
- Registers `claudeCli` in the ALL[] array next to `anthropic`.

Modified: src/core/ai/types.ts
- Adds 'claude-cli' to the Implementation union so the gateway switch
  is exhaustive at compile time.

Reverted: src/commands/jobs.ts
- Drops the GBRAIN_USE_CLAUDE_CLI=1 env-var gate the prior commit
  added. Routing now happens at the gateway based on the model string.

Deleted: src/core/minions/handlers/claude-cli-adapter.ts
- The MessagesClient adapter is superseded by the recipe + LanguageModelV2
  path. Two routing mechanisms competing for the same job would have
  forced users to reason about which one wins; the recipe is the single
  source of truth.

New file: test/claude-cli-recipe.test.ts (16 tests, 46 assertions):
- Recipe registration: getRecipe returns chat-only Recipe; aliases map
  short names (sonnet/haiku/opus) to canonical model ids.
- Text round trip: single text content block, usage propagation, stop
  finish reason.
- Provider prefix stripping.
- Single tool-call parsing.
- Multiple parallel tool calls in one block.
- Fenced JSON inside the block.
- Model-omitted id synthesizes toolu_claude_cli_<rand>.
- Malformed JSON falls back to text + stop reason.
- Unterminated block falls back to text + stop reason.
- Tools offered but model declines: returns text-only with stop reason
  so the gateway-loop treats it as a final answer rather than wedging
  for tool calls that never come.
- AbortSignal SIGTERMs the child.
- is_error envelope rejected.
- Non-JSON output rejected.
- doStream throws.
- argv + cwd assertion: --print, --disable-slash-commands,
  --system-prompt are present and cwd is the dedicated tmpdir.

Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN so the suite runs
without claude-cli installed and without API credits.

End-to-end smoke verified against a real `claude --print --model haiku`
invocation: model emitted `<use_tools>` block with toolu_add_001 +
{"a":12,"b":30}, adapter parsed back into a `tool-call` content block,
finishReason 'tool-calls'.

* feat(ai/claude-cli): harden subagent isolation, env scrub, verbose + stdin robustness

Four defensive fixes to the claude-cli provider so a subagent call behaves
identically regardless of the host's ambient Claude Code config:

- Agent isolation: pass `--tools ''` and `--strict-mcp-config` so the subprocess
  runs as a raw LLM with no built-in tools and no inherited user MCP servers.
  Without `--strict-mcp-config`, each call boots the user's MCP servers (including
  gbrain's own), causing recursion plus PGLite single-writer lock contention.
- Env scrub: drop ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL
  from the child env so the CLI authenticates via its own OAuth subscription
  session. An inherited API key silently flips billing to per-token API usage,
  the exact setup this recipe exists to replace.
- Verbose-mode compat: with `"verbose": true` in ~/.claude/settings.json,
  `--print --output-format json` emits an event array instead of a bare result
  object. Tolerate both shapes and select the result event.
- stdin robustness: handle the child stdin 'error' event and wrap write/end so a
  missing binary (ENOENT) or early child death (EPIPE) rejects cleanly instead of
  crashing the worker with an unhandled error.

Adds unit coverage for the env scrub, the isolation argv, and the verbose event
array. Verified against claude CLI 2.1.x.

* test(ai/claude-cli): cover verbose-array no-result + missing-binary reject paths

Two error branches in the hardened claude-cli provider had no coverage: the
verbose event-array path when no result event is present, and a missing binary
surfacing as a clean spawn-failed rejection. The missing-binary case is the
deterministic form of the stdin/EPIPE robustness; a synchronous stdin-write
throw is not reliably triggerable in a unit test, so the real ENOENT path the
handlers defend is exercised instead. Both reuse the existing shell-stub harness.

---------

Co-authored-by: Brett <brettdavies@users.noreply.github.com>
Co-authored-by: jarvisdoes <258486803+jarvisdoes@users.noreply.github.com>
Co-authored-by: Marco Maldonado <34176133+loweaxerium@users.noreply.github.com>
2026-07-23 18:00:56 -07:00
b313938e86 fix(queue): dead/cancelled jobs no longer block idempotency re-submission (#2253) (#3306)
queue.add() with an idempotency_key returns any existing row regardless
of status. This means dead jobs (exhausted retries from a transient
provider outage) permanently block re-submission of the same work —
even after the underlying issue is fixed.

Fix: when the existing row is dead or cancelled, NULL its
idempotency_key (preserving the row for audit) and fall through to the
INSERT path so a fresh job can be created.

Affects dream synthesize children that died during provider migrations
(429 rate-limit on old Anthropic proxy, tool-results-missing on old
OpenRouter). 45 dead children were blocking re-synthesis of transcripts
in production.

Includes 4 new tests covering dead, cancelled, completed, and active
status interactions with idempotency dedup.

Co-authored-by: Rafael Reis <57492577+rafaelreis-r@users.noreply.github.com>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
2026-07-23 17:43:11 -07:00
Time Attakcandcaioribeiroclw-pixel 26c6bad445 Reject unknown init flags before migrations (#2201) (#3307)
Co-authored-by: caioribeiroclw-pixel <caio.ribeiro.clw@gmail.com>
2026-07-23 17:43:06 -07:00
Time AttakcandBrett 38b8b1e41e fix(doctor): stop claiming "Brain is at target" when the target is unreachable (#2151) (#3339)
`gbrain doctor --remediation-plan` printed two consecutive lines that
contradicted each other when the brain was below target AND the target
was unreachable with autonomous remediation:

    Brain score: 45/100 → target 90
    Target unreachable: max with autonomous remediation is 70/100.
    No remediations needed. Brain is at target.

The second sentence hid the real next step (configure the prereqs that
would lift `max_reachable_score`) and made the brain look healthy when it
was not.

Fix: gate the "Brain is at target" line on `brain_score_current >=
targetScore`. When the plan is empty AND the brain is below target, the
"Target unreachable" line above is already the user-facing explanation;
the `Blocked checks` block below surfaces the manual gap.

Extracted `renderRemediationPlanLines(plan, targetScore): string[]` as a
pure helper alongside `runRemediationPlan` so the regression coverage
asserts on the rendered output directly rather than mocking
`console.log`. `runRemediationPlan` now joins the lines verbatim through
console.log; behavior is byte-identical for every case other than the
fixed contradiction.

Five regression tests cover: unreachable-and-below-target (the bug
case), reachable-and-at-target, exact-target, below-target-with-plan,
unreachable-with-partial-plan. 38 tests across the adjacent doctor test
files stay green; `bun run typecheck` clean.

Co-authored-by: Brett <brettdavies@users.noreply.github.com>
2026-07-23 17:43:00 -07:00
eb6cb4a16f fix(cycle): extract_atoms stamps concepts so synthesize_concepts has material (#2123) (#2124) (#3308)
synthesize-concepts.ts's design comment says extract_atoms stamps a
`concepts:` frontmatter field on each atom and :92 consumes ONLY that
field — but the extractor never wrote it, so the atoms → concepts
pipeline was dead end-to-end: every cycle reported "synthesize_concepts:
skipped — no atoms with concept refs" no matter how many atoms
accumulated (696 page-derived atoms / 0 with concepts on our production
brain before an external backfill).

Fix, all on the extractor side (no synthesize change needed):
- EXTRACT_PROMPT asks for `concepts` (1-3 kebab-case TOPIC labels) with
  an explicit reuse-over-coinage instruction — labels must cluster,
  since synthesize_concepts only materializes groups of >=2.
- parseAtomsResponse validates labels (kebab regex, max 3, drop
  invalid; empty -> undefined).
- The putPage frontmatter write stamps `concepts` alongside lesson /
  source_quote.

Tests: 4 parse cases + an end-to-end regression that goes extractor ->
real frontmatter -> synthesize_concepts' OWN DB query path -> concept
page. The existing tests fed synthesize via the `_atoms` seam, which is
exactly how this gap survived.

Validated in production ahead of this PR by stamping the same shape
externally: the next synthesize_concepts run wrote 33 concept pages
(T2=7/T3=26) from 60 stamped atoms, zero failures.

Co-authored-by: 陈源泉 <84364275+ChenyqThu@users.noreply.github.com>
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:42:55 -07:00
aae1a5107e fix(doctor): raw-source persistence guarantee for synthesized pages — warn-only v1 (#3300)
* feat(doctor): raw-source persistence guarantee — warn-only v1 (#1978)

Every synthesized/derived page (dream_generated:true or type:synthesis)
must carry a raw trace or an explicit exemption. v1 is warn-only:

- New doctor check `raw_provenance` (brain category) flags synthesized
  pages with none of: raw_trace/raw_source/source_uri/raw_trace_exempt
  frontmatter, an attached raw_data row, or synthesis_evidence rows.
- Dream synthesize now stamps `raw_source: <transcript path>` into each
  written page's frontmatter via the existing #2569 provenance stamp.
- Dream-cycle summary index pages and extract receipts carry an explicit
  `raw_trace_exempt: true` + reason (no source document of their own).

No write path is blocked; fail-closed enforcement is the v2 escalation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(doctor): exclude soft-deleted pages from raw_provenance check

Sibling frontmatter checks (quarantined_pages, flagged_pages) filter
deleted_at IS NULL; without it a deleted synthesized page keeps warning
(and its slug keeps being named) through the 72h recovery window with
no way to clear the warn.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:51 -07:00
40d9b83d5c fix(cycle): wire drift detection into the dream cycle — report-only v1 (#2653) (#3317)
dream.drift.enabled has gated an unwired scaffold since v0.28: runPhaseDrift
had zero call sites, the resolved model + BudgetMeter were discarded
(void modelId; void meter), and no operator-readable output existed.

- Wire 'drift' as a CyclePhase (default OFF via dream.drift.enabled),
  ordered after the calibration trio and before embed so the report page
  gets embedded same-cycle. PHASE_SCOPE=global, cycle-lock coordinated,
  --once (--phase drift --once) bypasses the gate for one run.
- Implement the LLM judge: soft-band candidates (weight 0.3-0.85, active,
  unresolved, fresh timeline evidence) are judged against their page's
  recent timeline entries via gateway chat; BudgetMeter-gated
  (dream.drift.budget, default $1), capped by dream.drift.max_per_cycle
  (default 20). Judge model resolves models.drift -> reasoning tier ->
  sonnet fallback (unchanged from scaffold).
- Report-only v1: judged candidates land on a reports/drift-<date> page.
  dream.drift.auto_update mutates NOTHING; the flag state is recorded in
  the report for operators.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:47 -07:00
69bc37f745 fix(auth): admin-gated rescope surface for DCR clients stuck on default scope (#3299)
* fix(auth): admin-gated rescope surface for DCR clients stuck on default scope (#1914)

DCR clients self-register with source_id='default' + federated_read=['default']
and the registration comment promised 'rescope via the CLI later' — but no
rescope surface existed. Adds:

- GBrainOAuthProvider.rescopeClient(clientId, { sourceId?, federatedRead? }):
  single-statement COALESCE update, canonical source-id validation
  (assertValidSourceId), FK-backed existence check on the write source,
  friendly errors for pre-v60/v61 schemas and unknown clients. Takes effect
  for already-issued tokens because verifyAccessToken re-reads oauth_clients.
- gbrain auth rescope-client <client_id> [--source S] [--federated-read a,b]
  (trusted local CLI).
- POST /admin/api/rescope-client (requireAdmin), mirroring the existing
  register-client / revoke-client admin endpoints.

Deliberately does NOT let clients self-widen scope (options a/b from the
issue) — fail-closed trust invariant.

Fixes #1914

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): rescope-client admin endpoint returns 400 (not 500) for nonexistent write source

The FK-translated 'Source "x" does not exist' error is a client error;
map it to 400 like the sibling validation failures. ('No OAuth client
found' is matched first, so the 404 path is unaffected.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:42 -07:00
4b38724aa2 fix(sources): federated-source pages visible to get_page/list_pages/resolve_slugs and no-grant MCP callers (#3242) (#3301)
Pages ingested into a config.federated=true source were invisible to
normal reads: get_page/list_pages scoped to the scalar resolved source
('default'), while the fully UNSCOPED resolve_slugs leaked every
source's slugs — the reporter's exact observation matrix.

- federatedSearchScope now backs get_page, list_pages and resolve_slugs
  (not just search/query), so the unqualified read surface shares one
  visibility set: grant > federated set > scalar source. resolve_slugs
  gains the missing sourceScopeOpts-family scoping (leak sealed).
- The widening gate is now field-presence instead of ctx.remote:
  localFederatedSourceIds is populated only by server-side transports
  (never from caller params), so trust stays fail-closed while the
  stdio MCP transport (no GBRAIN_SOURCE) and the legacy HTTP token
  path (no operator-set permissions.source_id grant) can opt their
  unqualified callers into the operator-configured federated set.
  Tokens WITH a grant, per-call source_id, and OAuth allowedSources
  all still win and never widen.
- gbrain sync now attributes its ingest-log row to the synced source
  instead of the shared 'default' bucket (attribution sub-bug).

No engine SQL changes: getPage/listPages/resolveSlugs already accept
sourceIds[] in both engines (#1393/#876).

Fixes #3242

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:38 -07:00
cd18081f4a fix(takes): keyword search matches words in long claims via word_similarity (#3267) (#3333)
Both engines' searchTakes used whole-string trigram similarity
(claim % query), which structurally cannot pass the 0.3 threshold for a
short keyword against a 100-200 char claim — keyword search returned
zero results on real brains. Switch the predicate to word similarity
(query <% claim) and rank by word_similarity(query, claim), in both
postgres-engine and pglite-engine per the engine-parity invariant.
Holder allow-list and source-scope filters unchanged.

Regression test: single-word query must match a long claim containing
it (fails under the old predicate).

Fixes #3267

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:33 -07:00
ef1133df3e docs(security): Docker network isolation for co-located self-hosted Postgres (#3270) (#3331)
OAuth/source scoping only guards the serve --http path; a container
sharing Docker's default bridge with the brain's Postgres can open a
direct DB session without a token. Adds a 'Co-located Docker workloads'
subsection to docs/mcp/DEPLOY.md with the operator checklist, a
trust-boundary paragraph in SECURITY.md, and an ops note + cross-link
in the company-brain tutorial.

Fixes #3270

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:29 -07:00
26b938c37d fix(auth): expose OAuth source grants in whoami (takeover of #3279) (#3332)
Rebase of #3279 onto current master: the whoami oauth shape gains
source_id (AuthInfo.sourceId, null when absent) and federated_read
(AuthInfo.allowedSources, [] when absent) — read-only self-introspection
that widens no grant. Re-applied against the post-#3091 description
string (stdio transport shape preserved) and merged the grant tests
into the current whoami.test.ts alongside the stdio cases.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: boundless-forest <boundless-forest@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:28:22 -07:00
Igby 5d7a8d7d4b chore(gitignore): ignore CLAUDE.local.md / AGENTS.local.md (#3290)
Agent tools (Claude Code, Codex, …) support a *.local.md counterpart to the
committed CLAUDE.md/AGENTS.md for personal, per-clone instruction overrides that
load after the committed file and are meant to stay uncommitted.

gbrain already ignores other workspace-local agent artifacts (.context/,
.claude/), so this fills the remaining gap for the two root-level override files.
Explicit filenames rather than a *.local.md glob, matching the existing
commented, specific style.
2026-07-23 17:13:07 -07:00
arisgysel-designandarisgysel-design be722ee5f7 fix(takes): query active row before superseding (#3275)
Fixes #2663.

Co-authored-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
2026-07-23 17:13:03 -07:00
RP-AGENT-BOTandDavid Guidry ca4cf2a0c6 fix(cycle): preserve per-page multi-claim proposals (#3297)
Co-authored-by: David Guidry <hairpie@mac.com>
2026-07-23 17:12:57 -07:00
d89d6ea293 fix(doctor): scope timeline labels to disambiguate entity coverage vs brain-score component (#2298) (#3073)
doctor and the get_health CLI surface printed two different timeline
metrics under one ambiguous 'timeline' label: the entity-scoped
timeline_coverage fraction (eligible entity pages with a timeline entry)
and the whole-brain timeline_coverage_score brain-score component (all
pages with a timeline entry, 0-15). Different numerators AND
denominators, indistinguishable in output.

Label-only fix, scoring unchanged:
- graph_coverage check: 'entity timeline coverage N%'
- brain_score breakdown: 'timeline density (all pages) N/15'
- get_health CLI: 'Timeline coverage (entity pages)' plus a new
  'Timeline density (all pages): N/15' line when the score is present

Adds test/doctor-timeline-metric-labels-2298.test.ts pinning the
denominator semantics, the rendered doctor messages, and the CLI
guard matrix (fails on master, passes here).

Takeover of #2761.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: TurgutKural <TurgutKural@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:05:12 -07:00
03cd52631b fix(thin-client): map --source scope onto source_id for remote-routed ops (#3086)
* fix(thin-client): map --source/GBRAIN_SOURCE/.gbrain-source onto source_id for routed ops (#2098)

The thin-client route short-circuits before makeContext, so the 6-tier
source resolution never ran and `gbrain query --source X` against a
remote brain sent the unknown `source` key verbatim — the server op
ignored it and searched unscoped.

applyThinClientSourceScope now runs the engine-free tiers (flag → env →
dotfile; DB-backed tiers need an engine, and the server's grant scoping
covers the rest) and sets the op's source_id wire param. Ops declaring
their own `source` param are untouched; an explicit --source on an op
with no source_id wire param errors loudly instead of silently dropping;
explicit --source-id/--all-sources on the wire win over ambient tiers.

Fixes #2098

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(thin-client): use withEnv() instead of direct process.env mutation (test-isolation R1)

CI verify failed on check:test-isolation — thin-client-source-scope.test.ts
mutated process.env.GBRAIN_SOURCE via beforeEach/afterEach. Wrapped each
test body in withEnv() from test/helpers/with-env.ts instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(thin-client): keep ambient scope out of get_skill's non-scope source_id param

get_skill's source_id is a mode switch (host catalog vs brain-resident-pack
lookup), not a read-scope filter. Ambient GBRAIN_SOURCE / .gbrain-source
injection would silently reroute 'gbrain skill <name>' on thin clients to
getResidentSkillDetail. Exclude it via NON_SCOPE_SOURCE_ID_OPS; explicit
--source-id still passes through, explicit --source errors with a hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-23 16:45:36 -07:00
800108e014 v0.42.65.0 chore(release): 93 verified fixes since v0.42.64.0 — changelog + version bump (#3346)
* chore(ci): refresh GitHub Actions SHA pins (checkout v4, action-gh-release v2)

Pre-ship pin staleness check per docs/RELEASING.md: both floating major
tags moved upstream; pins updated to the current tag commits.

* v0.42.65.0 chore(release): 92 verified fixes since v0.42.64.0 — changelog + version bump

Aggregates everything merged to master since the v0.42.64.0 bump commit:
community fixes, credited takeovers, batch re-lands, CI hardening, and
maintainer-approved features. Net commit list excludes revert pairs.
No new schema migrations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(deps): clear OSV-flagged transitive dependencies via override floors

Raise the existing security-floor overrides so the lockfile resolves
patched versions of three transitive packages flagged by the OSV scan
(@hono/node-server, fast-uri, body-parser). None are on gbrain's own
runtime path (@hono/node-server is only referenced by the MCP SDK's
optional hono transport, which gbrain does not load); the floors keep
the dependency scan green. MCP/OAuth unit tests pass against the
resolved versions.

* chore(release): fold #3110 into the v0.42.65.0 entry (93 net changes)

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 16:45:29 -07:00
5e8816e7d6 fix(links): resolve [[wikilink]] + slug-path frontmatter values; frontmatter-fresh incremental extract (#3087)
* fix(links): resolve [[wikilink]] + slug-path frontmatter values; keep frontmatter links fresh on the incremental cycle

Takeover/rebase of two community PRs:

PR #1983 — frontmatter link fields never resolved Obsidian-style values:
- makeResolver step 1's strict slug regex rejected digit-leading folders
  (90-people/nicolai) and nested paths (a/b/c); broadened to any slug-shaped
  value with an EXACT getPage match only (no fuzzy, no false positives).
- extractFrontmatterLinks resolved "[[dir/slug]]" verbatim; new anchored
  unwrapWikilink() strips wholly-wrapped [[...]] (and |alias/#heading/^block)
  before resolution. Bare values pass through unchanged.
- Same broadened slug-shape applied to the fs-path synthetic resolver in
  extractLinksFromFile (exact Set membership guards it), so the fs
  frontmatter path resolves PARA-numbered slugs too.

PR #2434 — the cycle's incremental extract (extractForSlugs) extracted body
links only, so externally-edited YAML (sources:/related:) edges drifted
stale. Adds an includeFrontmatter opt (threaded as a param after sourceId,
which master added in #1747/#1503 after the PR was cut), gated by the new
config key autopilot.incremental_extract_include_frontmatter (default off,
preserves body-only behavior).

Tests: unwrapWikilink unit coverage, broadened-resolver + end-to-end
frontmatter cases in test/link-extraction.test.ts; fs-resolver digit-leading
case in test/extract.test.ts; incremental gate off/on cases in
test/extract-incremental.test.ts.

Co-authored-by: spiky02plateau <spiky02plateau@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cycle): honor DB-plane config for incremental_extract_include_frontmatter

The gate read loadConfig() (file/env plane) only, but the documented enable
command — gbrain config set autopilot.incremental_extract_include_frontmatter
true — writes the DB plane via engine.setConfig, so the feature could never be
turned on the documented way (silent no-op, #2120 class). Now the file plane
wins when the key is present there; otherwise the DB plane is consulted,
matching the autopilot.auto_drain.* read pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: spiky02plateau <spiky02plateau@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-23 16:45:27 -07:00
593ba16535 fix(embed): support hosted Perplexity embeddings (pplx-embed-v1-*) (#1046) (#3099)
Adds a `perplexity` embedding recipe (OpenAI-compatible at
https://api.perplexity.ai/v1, auth via PERPLEXITY_API_KEY only — never an
OPENAI_API_KEY fallback) covering pplx-embed-v1-0.6b and pplx-embed-v1-4b.

Perplexity's /embeddings endpoint diverges from OpenAI's wire shape in two
places that break the AI SDK adapter, handled by a new perplexityCompatFetch
shim (mirrors the Voyage/ZeroEntropy pattern incl. the two-layer OOM caps):
- encoding_format only accepts base64_int8/base64_binary; the SDK's 'float'
  default is forced to 'base64_int8' outbound.
- The response embedding is base64-encoded signed int8 components (natively
  quantized); decoded to number[] inbound so the SDK's Zod schema validates.
  Cosine similarity is scale-invariant, so raw int8 components rank correctly.

Flexible dims (Matryoshka-style 128..native max: 1024 for 0.6b, 2560 for 4b)
validate fail-loud in dims.ts + the init preflight; `dimensions` is
Perplexity's native field so no wire translation is needed. default_dims is
1024 (works on a plain vector column for both models); the 4b model's full
2560 width rides the existing halfvec (>2000 dims) storage/ANN path. Pricing
entries land in embedding-pricing.ts.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 16:43:59 -07:00
ea6cb025be fix(schema,minions): truthful bundled pack inspection + config-aware subagent auth (#3110)
* fix(schema): make bundled pack inspection truthful (#2029)

Two live bugs:
- parseYamlMini had no block-scalar support, so a 'description: |' swallowed
  every following top-level key — the active gbrain-recommended pack loaded
  with 0 page types. Add parseBlockScalar for |/|-/|+ and >/>-/>+ in both
  mapping and sequence-sibling positions.
- The bundled-pack list was hand-copied in three places (operations.ts had 2
  names, mutate.ts had 3, load-active.ts had 7). New single registry
  src/core/schema-pack/bundled.ts carries all 7 shipped packs; every
  consumer derives from it.

Takeover of #2029, rebased onto master (schema.ts hunks already landed).

Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(minions): subagent default client resolves config-stored Anthropic key (#2048)

The legacy subagent path constructed a bare new Anthropic() (env-only), so
launchd/MCP workers whose key lives in gbrain config (anthropic_api_key)
failed auth. anthropic-key.ts now exports resolveAnthropicKey() (env first,
then config; hasAnthropicKey delegates) and makeSubagentHandler passes it as
apiKey.

Partial takeover of #2048 — only the auth patch; the path patches were
superseded by the outputRoot mechanism (#2415).

Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(schema): point bundled-registry test at bundled.ts source of truth

The bundled pack list moved from load-active.ts to bundled.ts in the
truthful-inspection refactor; the T4 registry test still grepped
load-active.ts source. Assert BUNDLED_PACK_NAMES directly instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(schema): block scalars keep '#' as literal content

Inside a YAML block scalar '#' is content, not a comment; parseBlockScalar
was routing lines through stripComment/isBlank, truncating descriptions
like 'see issue #2029' and blanking comment-looking lines. Use the raw
line inside the scalar. Adds a pinning test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: JiraiyaETH <JiraiyaETH@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 16:13:51 -07:00
Garry TanandClaude Fable 5 f1f6fcbeb6 fix(serve): boot-readiness deadline releases PGLite lock on wedged boot (#3273)
A serve process that wedges mid-boot (e.g. a boot step blocked on an
unreachable upstream) held the PGLite write lock indefinitely — the
post-#2348 lock discipline never steals from a live holder, so every CLI
consumer timed out until the serve PID was manually killed.

runServe (stdio path) now arms a boot-readiness deadline around
startMcpServer: if the transport hasn't connected within
GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS (default 60, 0 disables), it logs the
condition, awaits engine.disconnect() (raced against the existing
5s cleanup deadline so a wedged WASM close can't trap it either), and
exits non-zero so supervisors restart with backoff. A completed boot
clears the timer; the HTTP path is untouched (own lifecycle).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:39:39 -07:00
ca47c054b8 fix(gateway): brainstorm/propose_takes model-config takeovers — configured-model cost preview, judge config key, provider-probe skip, narrow page projection (#3120)
* fix(cycle): propose_takes skips cleanly when the chat provider is unavailable + narrow page projection

Takeover of PR #1979 by @shawnduggan. The original PR gated on a
hardcoded ANTHROPIC_API_KEY heuristic (modelNeedsAnthropicKey defaulting
to true), which master deliberately removed elsewhere — it misclassified
non-Anthropic stacks and fought the tier-config model resolution. This
lands the intent the master-blessed way: probe the RESOLVED chat model
(opts.model ?? getChatModel()) via probeChatModel — same semantics as
patterns.ts / think/index.ts — and skip the phase cheaply when the
provider can't run. Injected extractors are never gated.

Also keeps the PR's uncontested half: load proposal candidates with a
narrow projection (slug, source_id, compiled_truth) instead of
listPages' SELECT p.*, preserving sourceIds > sourceId scope precedence
and updated_desc ordering.

Co-authored-by: shawnduggan <shawnduggan@users.noreply.github.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(brainstorm): price cost preview against the configured chat model + models.brainstorm.judge config key

Takeover of PR #1855 by @starm2010, shrunk to the brainstorm-only
portion (the cycle-phase hunks are superseded by the resolveModel-in-
phase approach already on master). The cost preview + hard cost ceiling
previously always priced anthropic:claude-sonnet-4-6 even when the
configured chat_model (which the gateway actually runs) was something
else; modelStr now resolves override → config.chat_model → fallback.
The judge phase honors a new models.brainstorm.judge config key when no
--judge-model flag is passed, resolved in the orchestrator so every
caller (brainstorm, lsd, eval-brainstorm) benefits.

Co-authored-by: starm2010 <starm2010@users.noreply.github.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): use withEnv()/emptyHome() in propose-takes no-key tests

check:test-isolation R1 flagged direct process.env mutation in the two
new no-key tests. Swap the hand-rolled save/mutate/restore for the
canonical withEnv() helper (+ emptyHome() for the hermetic GBRAIN_HOME).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:29:09 -07:00
Garry Tan 97df1e78b7 Revert "fix(init): explicit --embedding-model overrides persisted --no-embedding sentinel (#3138)"
This reverts commit df22c81996.
2026-07-23 15:26:33 -07:00
Garry Tan 1392243d3b Revert "fix(jobs/autopilot): --install interval persistence, --lock-duration flag, dead-jobs doctor check, deployment-shape docs (#3129)"
This reverts commit 8345abce42.
2026-07-23 15:26:33 -07:00
8345abce42 fix(jobs/autopilot): --install interval persistence, --lock-duration flag, dead-jobs doctor check, deployment-shape docs (#3129)
* fix(jobs/autopilot): interval persistence, --lock-duration flag, dead-jobs doctor check, deployment-shape docs

Four backlog items in the jobs/autopilot workers, locks & installers area:

- #2794: `gbrain autopilot --install` silently dropped `--interval`. The
  installer now parses + validates it, persists it to config
  (autopilot.interval), and threads it into the wrapper's exec line; a
  later flag-less --install regenerates the wrapper from the persisted
  value, and the daemon run path falls back to the same config key.

- #1014: new `--lock-duration MS` flag (env: GBRAIN_LOCK_DURATION) on
  `gbrain jobs work` and `gbrain jobs supervisor` to tune the worker
  stall-lock window (and so the lockDuration x max_stalled wall-clock
  dead-letter cap). Validated like --health-interval (integer >= 1000ms);
  the supervisor propagates it to the spawned worker via buildWorkerArgs;
  shown in the worker startup banner.

- Takeover of PR #1185 (@ethanbeard): `gbrain integrations doctor` now
  surfaces dead minion jobs as a cross-cutting [queue] check. Reworked
  from the original: consumes a new machine-readable `gbrain jobs list
  --json` surface instead of screen-scraping the human table (long job
  names shift the columns), and scopes to a 24h finished_at window so one
  ancient dead job can't flag ISSUES forever (parity with main doctor's
  queue checks).

- #631: documented the production deployment shape for autopilot vs jobs
  supervisor in docs/guides/minions-deployment.md — recommend the
  `autopilot --no-worker` + `jobs supervisor` split, warn against running
  both worker lanes, and cross-link the --no-worker liveness probe.

Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(autopilot): make #2794 wrapper-script tests hermetic — fake gbrain on PATH

writeWrapperScript calls resolveGbrainCliPath(), which shells out to
`which gbrain` and throws on CI runners where no gbrain binary is
installed. The two new --interval threading tests failed only in CI
(dev machines have gbrain on PATH). Prepend a fake executable to PATH
for the describe block so resolution is deterministic everywhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:06:17 -07:00
df22c81996 fix(init): explicit --embedding-model overrides persisted --no-embedding sentinel (#3138)
* fix(init): explicit --embedding-model overrides the persisted --no-embedding sentinel (#2301)

Pre-fix, once ~/.gbrain/config.json carried embedding_disabled: true (the
--no-embedding deferred-setup sentinel), every re-init silently re-deferred
embedding: resolveAIOptions honored the sentinel BEFORE the explicit
--embedding-model flag and never cleared noEmbedding, and the persistence
merge carried the sentinel forward via ...existingFile. Both recovery paths
were dead ends — `gbrain config set embedding_model` is hard-refused
(schema-sizing field), and re-init hit the sentinel.

Fix:
- resolveAIOptions: an explicit --embedding-model / --model flag clears the
  sentinel-derived noEmbedding (explicit --no-embedding on the same
  invocation still wins — that branch runs after).
- initPGLite + initPostgres persistence: a resolved (model, dims) tuple
  drops the stale embedding_disabled key instead of inheriting it.
- assertEmbeddingEnabled message no longer recommends the refused
  `gbrain config set embedding_model` command; the working re-init recipe
  leads.

Test: test/e2e/init-reinit-after-deferred.test.ts — deferred init then
re-init with an explicit model recovers (sentinel gone, model persisted);
bare re-init still honors the sentinel.

Fixes #2301

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: scrub remaining hard-refused `config set embedding_model` advice from init deferred-setup messages

The PR fixed the recovery recipe in assertEmbeddingEnabled but the
deferred-setup lines in initPGLite/initPostgres and the fail-loud
defer hint still pointed users at the Lane C.2 hard-refused command.
Point all three at the working re-init recipe instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:03:46 -07:00
d2fd1f297c fix(cycle): stamp path-derived dream sources; close engine on autopilot shutdown (#3178)
* fix(dream): stamp path-derived sources so --dir runs land cycle freshness (#1869)

gbrain dream --dir <path> (and the configured sync.repo_path fallback)
never wrote last_source_cycle_at / last_full_cycle_at because runCycle's
stamp gate reads opts.sourceId and dream only set it from --source.
Doctor's cycle_freshness stayed perpetually stale on path-scoped brains.

Fix at the command level: dream derives the source id from the resolved
brain dir via resolveSourceForDir (now exported from cycle.ts) and passes
it as opts.sourceId. runCycle's stamp/lock semantics are untouched, so
legacy global callers (autopilot-global-maintenance runs GLOBAL_PHASES
with a brainDir and no sourceId) cannot falsely stamp per-source
freshness — the flaw that sank the runCycle-wide variant in PR #2549.
A derived match on an archived source is skipped (mirrors the explicit
--source archived guard).

Takeover of #2549.

Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(autopilot): close the engine on SIGTERM/SIGINT instead of hard-exiting (#1872)

systemctl stop (SIGTERM) previously hard-exited autopilot without ever
closing the engine. On PGLite the cycle steps run INLINE in the autopilot
process, so a mid-write exit kills WASM Postgres with the WAL dirty and
can corrupt the brain.

Now both exit paths close the engine first:
- autopilot's own shutdown() (SIGINT + internal stops like max_crashes /
  cycle-failure-cap) aborts the in-flight inline cycle via an
  AbortController threaded into runCycle, drains it briefly, and awaits
  engine.disconnect() before process.exit(0).
- process-cleanup's SIGTERM handler (installed at cli.ts module load,
  exits within its 3s cleanup deadline) reaches the same closeEngine via
  a registered 'autopilot-engine-close' cleanup callback.

PGLite's disconnect() drains the pending query and checkpoints before
closing; a second call is a no-op, so both paths firing is safe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(dream): conform dream-dir-source-stamp to canonical PGLite isolation pattern

check:test-isolation R3/R4 flagged the new test file: engine was created
in beforeEach (outside beforeAll) and never disconnected in afterAll.
Switch to the canonical shared-engine pattern (beforeAll create,
beforeEach resetPgliteState, afterAll disconnect) per
test/helpers/reset-pglite.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:47:11 -07:00
Anton Senkovskiy 0a4f062cac fix(orphans): exclude life/events/ chronicle volume from orphan_ratio (#2264) (#3214)
orphan_ratio's denominator is swamped on auto_chronicle brains by the
machine-generated chronicle events (life/events/<day>-<hash>, written
per eligible event) — no inbound links by design. The shipped policy
already excludes raw/atoms/skills/dreaming/daily and extracts/, but
life/events/ was still counted; on a 1,657-page auto_chronicle brain it
was ~72% of the orphan mass, enough to pin the ratio red.

Add 'life/events/' to DENY_PREFIXES — a scoped prefix, NOT the whole
`life/` first-segment, so human-authored life/diary/ (gbrain capture
--type diary) stays IN the denominator. Same shipped hardcoded-class
mechanism as the existing entries; not the #2215 user-config route
(closed not_planned). Knowledge classes (concepts/people/notes/projects)
also stay in, so genuine graph decay still trips.

Regression in test/orphans-pure-fn.test.ts: life/events/ now excluded
(fails before, passes after); life/diary/ and concepts//notes//projects/
pinned as still-counted. doctor's orphan_ratio uses the same shouldExclude
path (getOrphansData; local + doctor-remote MCP), covered transitively.
2026-07-23 14:33:02 -07:00
MasaandClaude Fable 5 2f4ad2c0a4 fix(pricing): add the zeroentropyai:zerank-2 reranker entry the budget tracker needs (#3223) (#3233)
zerank-2 is the default reranker under search_mode: tokenmax, but had no
pricing entry — any --max-cost-capped rerank call TX2 hard-failed in
BudgetTracker.reserve() with "no pricing entry".

Adding the entry to EMBEDDING_PRICING alone (the issue's suggested fix)
does not resolve this: lookupPricing()'s rerank branch in
budget-tracker.ts never consulted that table at all, only
ANTHROPIC_PRICING and the FREE_LOCAL_RERANK_PROVIDERS zero-price set.
Verified by reproducing the hard-fail with only the pricing-table entry
added and confirming it still threw.

Fix: add the $0.025/1M-token entry (docs/ai-providers/zeroentropy.md)
and wire the rerank branch to fall back to lookupEmbeddingPrice, reusing
the existing provider:model-keyed table instead of duplicating a third
pricing surface.

Addresses the report in #3223.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:22:01 -07:00
MasaandClaude Fable 5 526c597ccf fix(migrate): count and surface per-page copy failures instead of silently advancing (#3241)
gbrain migrate's per-page copy loop had no failure handling at all: a page
write that threw (e.g. a NOT-NULL column with no protection against the
JS `undefined` postgres.js's UNDEFINED_VALUE guard rejects) crashed the
whole command outright, with no per-page accounting and no way to tell
which page caused it.

Two changes:

- Normalize `undefined` column values to explicit `null` at the migrate
  copy boundary before calling putPage. PGLite can hand back `undefined`
  for a column that is legitimately NULL/empty; postgres.js rejects a raw
  `undefined` bound parameter but accepts `null` fine. This is the root
  cause behind the report: a page whose title/compiled_truth/type came
  back `undefined` threw mid-insert.

- Wrap the per-page copy in try/catch: failures are tracked (slug +
  reason), excluded from the resume manifest's completed_slugs (so a
  retry picks them back up), and the run ends with a non-zero exit
  verdict + an honest "N copied, M failed" summary instead of a bare
  crash or a false "N/N copied" success.

Fixing this properly also required making the pre-existing resume
manifest actually usable without --force (a matching manifest now
bypasses the non-empty-target guard instead of demanding a wipe that
would orphan already-copied pages), always resetting the manifest on
--force regardless of whether the target looked empty, persisting the
manifest before the copy loop starts (so a run where every page fails
after its row lands still leaves a resumable manifest on disk), only
flipping the active config to the target once the migration is fully
clean, and skipping link-copy for slugs known to have failed above
(avoiding an FK-violation crash on the next phase).

Addresses the report in #3194 (reported by @hbohlen).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:21:56 -07:00
MasaandClaude Fable 5 2a0c51d093 fix(gateway): fold config-plane voyage_api_key into VOYAGE_API_KEY like the other hosted keys (#3236)
Addresses the report in #2662: buildGatewayConfig folded openai_api_key,
anthropic_api_key, zeroentropy_api_key and openrouter_api_key from
~/.gbrain/config.json into the gateway env, but not voyage_api_key. In
launchd/daemon/MCP contexts (no process-env export), multimodal/image
embeds with Voyage failed silently even though config.json looked complete.

- build-gateway-config.ts: fold voyage_api_key -> VOYAGE_API_KEY, mirroring
  the existing zeroentropy/openrouter fold (process.env still wins).
- config.ts: add the voyage_api_key file-plane field to GBrainConfig and
  KNOWN_CONFIG_KEYS.
- brain-score-recommendations.ts: HOSTED_EMBED_KEY_CONFIG now maps
  VOYAGE_API_KEY -> voyage_api_key so doctor/autopilot judge a config-keyed
  Voyage brain as usable instead of dispatching a doomed embed job.
- autopilot.ts: the HOSTED_EMBED_KEY_CONFIG producer closure now resolves
  hosted keys via the same file-plane source (loadConfigFileOnly) doctor
  already uses, instead of the DB plane (engine.getConfig) - the DB plane
  is never threaded into buildGatewayConfig for these fields, so reading it
  here would let a DB-only key report "configured" while the gateway still
  has no key. This also tightens the pre-existing openai/zeroentropy path,
  not just voyage.
- Tests: fold + env-precedence tests in build-gateway-config.test.ts,
  HOSTED_EMBED_KEY_CONFIG map test, and a real end-to-end regression in
  brain-score-recommendations.test.ts through loadConfigFileOnly() and
  buildGatewayConfig() with an actual temp config.json.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:21:51 -07:00
Francois de FitteandFrancois de Fitte 920aea5eb8 Notify about the conflict between gbrain serve (MCP) and CLI commands (#3243)
* Notify about gbrain serve and CLI conflict

* Handle serve flags in PGLite lock notice

---------

Co-authored-by: Francois de Fitte <4712833+fdefitte@users.noreply.github.com>
2026-07-23 14:21:46 -07:00
MasaandClaude Fable 5 4be9d112cb fix(frontmatter): stop treating YAML comments inside the fence as markdown headings (#3225) (#3247)
* fix(frontmatter): stop treating YAML comments inside the fence as markdown headings

autoFixFrontmatter's MISSING_CLOSE repair walked lines from the opening
`---` and broke out of the scan on the first `#`-prefixed line, treating
it as a markdown heading before it ever reached the real closing fence.
A `#` line inside a closed YAML block is a comment, not a heading — but
the scan never got that far, so it inserted a spurious `---` right
before the comment and split valid frontmatter in two, pushing the real
keys (title, pubDate, ...) into the document body.

This is the same bug PR #2153 fixed in the parseMarkdown validator, but
autoFixFrontmatter in brain-writer.ts is a separate reimplementation of
the same MISSING_CLOSE logic that PR never touched. Because
parseMarkdown's validator now parses this shape cleanly, autoFixFrontmatter
is only reachable when some other fixable error (SLUG_MISMATCH, NULL_BYTES,
etc.) also fires on the same file — a common real-world case (e.g. a
renamed file with a stale slug: field) that still corrupts otherwise-valid
frontmatter today.

Fix: scan the full zone for the closing `---` first; only fall back to
the heading-shaped-line heuristic when no closer is found at all.

Addresses the report in #3225. Thanks to @WilliamCourterWelch for the
clear repro and for catching this via git diff before it reached a live
site.

Tests: 4 new regression cases in test/brain-writer.test.ts covering a
YAML comment before the close, comment-only frontmatter, a `#` inside a
quoted string value, and a comment co-occurring with an unrelated real
fix (SLUG_MISMATCH) — confirmed all 3 corruption-covering cases fail
against the pre-fix code (stash/red/restore) and pass after the fix.
The pre-existing genuinely-missing-closer case is unchanged.

bun test test/brain-writer.test.ts test/markdown-validation.test.ts
test/markdown.test.ts test/lint-frontmatter.test.ts
test/doctor-frontmatter-partial.test.ts test/frontmatter-cli.test.ts
-> 122 pass / 0 fail. bun run typecheck -> clean. Full suite intentionally
not run locally (targeted scope per contribution norms); CI covers it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(frontmatter): swap non-exercising regression case per codex review

The quoted-string test (title: "Chapter #1 recap") never exercised the
fixed branch — the heading regex is line-anchored on the trimmed line,
so a `#` mid-string never matched before or after the fix. Replace it
with an indented `#` line inside a YAML block scalar, which does hit
the same closer-first-scan code path as the other regression cases
with a different real-world shape.

bun test test/brain-writer.test.ts -> 27 pass / 0 fail. bun run
typecheck -> clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:21:39 -07:00
MasaandClaude Fable 5 b82f520314 fix(cycle): propagate all-provider-failed atom drains so durable jobs retry (#3218) (#3248)
extract-atoms-drain's runBatch discarded runPhaseExtractAtoms's per-item
failures/status, so a batch where EVERY provider call errored collapsed to
{extracted: 0, skipped: 0} — indistinguishable from a legitimate no-op. The
drain loop reported status: 'ok' regardless, the Minion handler returned
normally, and the worker marked the durable job complete while the backlog
sat untouched with no retry ever applied.

- runBatch now derives providerFailure from the same counts the phase
  already returns (failures.length > 0 && transcripts_processed +
  pages_processed === 0 — every attempted item errored, zero succeeded).
  Partial success (>=1 item processed) is unaffected.
- The pure loop surfaces this as status/stopped = 'provider_failure',
  breaking immediately (same hot-loop guard as no_progress) instead of
  letting a final remaining===0 recount silently overwrite it to 'drained'.
- The extract-atoms-drain Minion handler throws when it sees
  status === 'provider_failure', so the worker's ordinary failJob path
  (attempt+backoff, dead-letter on exhaustion) takes over. The
  LockUnavailableError -> deferred path is unchanged.
- autopilot's auto-drain submission bumps max_attempts from 1 to 3 (queue
  default) — with the handler now actually throwing, max_attempts:1 meant
  the first provider blip dead-lettered instantly with no backoff attempt.

Tests: pure-loop provider_failure propagation (incl. the remaining===0
precedence case), runPhaseExtractAtoms's all-items-fail counts contract,
and source-shape guards on the handler throw + autopilot max_attempts.
Full suite deferred to CI per repo convention (targeted run: 104 pass / 0
fail across the touched + adjacent extract-atoms/drain/autopilot files;
`bun run typecheck` clean).

Two rounds of codex review (gpt-5.6-sol, high effort): round 1 flagged
autopilot's max_attempts:1 and the stopped-precedence bug (both fixed
above); round 2 confirmed no new issues.

Thanks to @aaronkhawkins for the detailed report. Addresses the report in
#3218.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:21:34 -07:00
MasaandClaude Fable 5 00bcd66c3e fix(sync): failed git pull with zero imports reports partial (pull_failed) instead of up_to_date (#3068) (#3253)
* fix(sync): report partial (pull_failed) instead of up_to_date when git pull fails with zero imports (#3068)

A warn-and-continue internal git pull failure (e.g. a local-path origin
rejected by protocol.file.allow=never) combined with a zero-import run
previously reported `up_to_date`, exited 0, and bumped the last_sync_at
freshness heartbeat. A permanently-failing pull was therefore invisible
forever: doctor's sync_freshness never fired and every scheduled sync
looked clean while the source silently went stale.

Now, when the pull failed and the run imported nothing, sync returns
`partial` with the new reason `pull_failed`, leaves last_commit AND
last_sync_at untouched (so staleness monitoring fires), and prints a
dedicated non-success message. The fall-through-to-working-tree design
is unchanged: local commits still import when the remote is unreachable,
and the anchor still advances over commits that were actually imported.

Addresses the report in #3068.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): surface pull_failed to CLI exit codes, sync --all JSON, and the cycle phase (#3068 review round)

Codex review round 1 follow-ups:

- Single-source `gbrain sync` sets exit code 1 on partial/pull_failed
  (timeout-class partials keep exit 0 — they converge on retry; a failing
  pull does not).
- `sync --all` exits 1 when any source reports pull_failed, and the
  --json envelope carries the per-source partial `reason`.
- The autopilot cycle's sync phase maps partial/pull_failed to `warn`
  with a dedicated summary and a `syncReason` detail, so a scheduled
  cycle no longer reports a clean run over a wedged source.
- The regression test now isolates GBRAIN_HOME to a temp dir so the
  first full sync cannot touch the real sync-failure ledger.
- Current-state docs: KEY_FILES.md sync.ts entry + TESTING.md inventory
  describe the pull_failed contract and the new test.

Addresses the report in #3068.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): route pull_failed exit through the owned verdict channel; make the regression test serial (#3068 review round 2)

Codex review round 2 follow-ups:

- Single-source exit now uses setCliExitVerdict(1) instead of a raw
  process.exitCode assignment, which the CLI teardown deliberately
  ignores (PGLite's Emscripten runtime clobbers process.exitCode
  mid-run; the owned channel in src/core/cli-force-exit.ts is the only
  trusted verdict). Pinned by test/cli-exit-verdict-pin.test.ts.
- The regression test is renamed to *.serial.test.ts because it pins
  GBRAIN_HOME for the whole file (scripts/check-test-isolation.sh R1);
  docs updated to the new name.

Addresses the report in #3068.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:09:30 -07:00
22fca8f891 fix(schema-pack): merge extends chain + borrow_from into the resolved manifest (#1749) (#3181)
Takeover of #2856 (fork-head PR). Applied cleanly onto origin/master;
llms bundles regenerated (byte-identical — touched docs are not inlined).

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: coder8080 <67740875+coder8080@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:09:23 -07:00
caterpillarC15andcaterpillarC15 178d3404a4 fix(ci): normalize scanner roots on macOS (#3198)
Co-authored-by: caterpillarC15 <caterpillarC15@users.noreply.github.com>
2026-07-23 13:54:14 -07:00
Yolan Maldonado 1cc17f014d fix: select query-relevant think excerpts (#3197)
Keep each page excerpt within the existing fixed budget while selecting the window with the strongest query-term coverage. Preserve leading truncation for callers without a matching question.
2026-07-23 13:54:09 -07:00
Masa 2b00b7abeb fix(migrate): drop invalid CONCURRENTLY-build remnants without a DO block (#3191)
* fix(migrate): drop invalid CONCURRENTLY-build remnants without a DO block

Migration v66 (embed_stale_partial_index) pre-drops an invalid index left
over from a previously interrupted CREATE INDEX CONCURRENTLY using
DO $$ BEGIN ... EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS ...'; END $$.
Postgres rejects CONCURRENTLY from any function/EXECUTE context, so the
guard's EXISTS check passes but the EXECUTE inside it always throws
"DROP INDEX CONCURRENTLY cannot be executed from a function" -- the
migration only fails on brains carrying an invalid-index leftover.

Add dropInvalidConcurrentIndex(): the validity probe runs as a plain
application-level SELECT, and the DROP runs as its own top-level
runMigration call instead of inside a DO block. Fixes #1178.

* fix(migrate): address codex review — schema-safe index resolution + OID-based no-op assertion

- dropInvalidConcurrentIndex(): resolve indexName via to_regclass() (search_path
  resolution, same as the unqualified DROP that follows) instead of matching
  pg_class.relname bare, which could hit a same-named index in a different
  schema on a non-default search_path.
- e2e test: the no-op re-run case now compares index OID before/after, not
  just validity -- validity alone wouldn't catch a spurious drop+recreate.
2026-07-23 13:53:15 -07:00
caterpillarC15andcaterpillarC15 18513c65be fix(budget): make paid MCP spend atomic and fail closed (#3203)
Co-authored-by: caterpillarC15 <caterpillarC15@users.noreply.github.com>
2026-07-23 13:53:11 -07:00
caterpillarC15andcaterpillarC15 f70c3fe9d8 fix(sync): report pinned commit after resumed sync (#3202)
Co-authored-by: caterpillarC15 <caterpillarC15@users.noreply.github.com>
2026-07-23 13:38:57 -07:00
caterpillarC15andcaterpillarC15 c7dd0fa64b fix(budget): record actual resolver spend before cap error (#3204)
Co-authored-by: caterpillarC15 <caterpillarC15@users.noreply.github.com>
2026-07-23 13:38:52 -07:00
alexey-metaengage 9fb046a110 feat(operations): include source_id in list_pages rows (#3209) 2026-07-23 13:38:47 -07:00
alexey-metaengage 581a1eed29 fix(eval): raise contradiction judge token cap for thinking models (#3210) 2026-07-23 13:38:42 -07:00
alexey-metaengage 19c6b6ef67 fix(cycle): raise atom maxTokens + case-normalize atom_type for Gemini (#3211) 2026-07-23 13:38:37 -07:00
Anton Senkovskiy 4213ac8da8 fix(facts): gate anonymous-speaker self-attribution in conversation extractor (#3228)
The conversation-fact extractor renders turns as `${speaker} (${ts}): ${text}`
and its `confidence` field scores confidence-in-the-CLAIM, not confidence-in-
WHO-said-it. So a first-person self-assertion from an anonymous speaker
("Speaker A: I'm joining Acme") could come back with the anonymous label echoed
as `entity` — a confident attribution to a person we cannot identify. That
label is then stored verbatim as the fact's `entity_slug` (the batch insert
path does no canonicalization), polluting entity-scoped queries and the
top_entities aggregation, or misattributing the claim.

Add a deterministic gate (`isUnknownSpeakerLabel`) at the single candidate-loop
choke point that nulls ONLY that self-referential attribution, plus one
EXTRACTOR_SYSTEM rule telling the model not to guess a name for anonymous
first-person turns. Third-person entities from the same turn ("Acme raised $5M"
-> entity=acme) and named-speaker attributions are untouched. The fact itself
is always preserved; only the bad attribution is dropped.
2026-07-23 13:38:33 -07:00
MasaandClaude Fable 5 6aa055024c docs(todos): drop the completed #2684-residual entry — landed via #2973 (#3229)
The P1 entry asked for fail-closed semantics in resolveTakesSourceId
(src/commands/takes.ts). That landed in #2973 (merged 2026-07-20):
the function now delegates straight to resolveSourceId with no
catch-and-fallback, so an unresolvable explicit source throws instead
of silently restoring the pre-#2698 unscoped cross-source write path.
Regression tests for the invalid-source path shipped in the same PR.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 13:38:27 -07:00
MasaandClaude Fable 5 cce774c904 fix(sync): keep the expected discover_git_root probe failure off stderr (#3232)
discoverGitRoot() probes `git rev-parse --show-toplevel` to locate the repo
root; a miss is expected/routine (a non-git-yet brain dir, a scratch dir) and
is either self-healed via auto git-init or surfaced as a friendlier Error.
Node's execFileSync writes the child's stderr straight to the parent's real
stderr by default unless an explicit `stdio` array is given, so every routine
probe miss dumped git's raw "fatal: not a git repository ..." line into
gbrain's operator logs -- indistinguishable from an actual crash to an
operator grepping logs for "fatal:" as a crash signature.

Add an opt-in `silenceStderr` param to the shared `git()` helper (sets
`stdio: ['ignore', 'pipe', 'pipe']`, which disables the implicit
passthrough-to-parent-stderr behavior) and pass it only from
discoverGitRoot's internal probe call. Every other `git()` call site is
unchanged, so unexpected-failure visibility elsewhere is preserved.

Related to #2964, which added the auto-recovery this probe feeds.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 13:38:23 -07:00
chengzehsuandKevin Hsu b78fc56e98 fix(throttle): use /proc/meminfo MemAvailable on Linux (#556)
`getMemoryUsage()` in src/core/backoff.ts computes 1 - freemem()/totalmem(),
where Node's `os.freemem()` returns Linux's `MemFree`. `MemFree` excludes the
page cache, which the kernel grows aggressively in any environment that reads
files (i.e. essentially all containers). On a healthy 4 GB Linux container with
~1.4 GB MemAvailable, MemFree is routinely ~100 MB, so `getMemoryUsage()`
reports 96-97% used and `waitForCapacity()` rejects every job with:

  Throttle timeout: system overloaded after 20 attempts (~600s).
  Load: ..%, Memory: 97%

even though the host has plenty of usable memory.

Linux exposes `MemAvailable` in `/proc/meminfo` precisely as the kernel's
estimate of memory available for new allocations without swapping (it factors
in reclaimable page cache). This is what `htop` and `free -h` show as
"available". Using it removes the false positive entirely.

Behaviour:
- On Linux (when /proc/meminfo is readable): use 1 - MemAvailable/MemTotal.
- Anywhere else (macOS, Windows, sandboxed envs without /proc): unchanged
  fallback to 1 - freemem()/totalmem().

Scope is intentionally minimal — data correctness only. An env override like
GBRAIN_MEMORY_STOP_PCT would also be reasonable but is out of scope here.

Co-authored-by: Kevin Hsu <kevinhsu.ecofirst@gmail.com>
2026-07-23 13:24:46 -07:00
38f446bb6f fix(test): isolate $HOME in mechanical.test.ts so E2E suite stops clobbering user config (#434)
mechanical.test.ts shells out to `gbrain init --non-interactive`,
`gbrain import`, and similar commands via Bun.spawnSync. The four
`cliEnv()` helpers in this file forward `process.env` unchanged, so
`gbrain init` ends up calling saveConfig() against the developer's real
$HOME/.gbrain/config.json, overwriting their production database_url
with the test container's URL on every `bun run test:e2e` invocation.

Sibling test/e2e/migration-flow.test.ts already solved this with a
module-level temp HOME and an afterAll restore. Mirror that pattern in
mechanical.test.ts.

Verified by md5'ing ~/.gbrain/config.json before and after running the
Setup Journey, Init Edge Cases, Schema Idempotency, RLS Verification,
Doctor Command, and Parallel Import describe blocks — config hash is
identical pre and post (26 passing tests, 0 failures, 0 mutations to
the user's real config).

Co-authored-by: Seth Armbrust <setharmbrust@seth.local>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-07-23 13:24:40 -07:00
8901dc0f45 fix(backlog): x-to-brain health check, propose_takes deadlines, capture title truncation, extract_atoms backlog + pooler direct-URL (part4-6) (#3165)
* fix(recipes/x-to-brain): use /users/by/username for app-only bearer health check

Takeover of #2343. /users/me requires user-context OAuth and always fails
under the app-only bearer the recipe collects. Health check + setup curls
now use /users/by/username/$X_HANDLE, with X_HANDLE declared in secrets
so the installer prompts for it. Recipe version 0.8.1 -> 0.8.2.

Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cycle): bound propose_takes with per-call timeout + phase deadline

Takeover of #2262. The extractor's gateway.chat call had no abortSignal, so
one stalled provider socket could pin the phase for the 300s gateway default
per page; the nightly wrapper then SIGTERMed the whole phase mid-run. Each
extractor call is now bounded at 90s (per-page failure already logs a warning
and continues), and the page loop carries a 30-min wall-clock deadline that
breaks cleanly into a partial result with deadline_hit:true + warn status.

Unlike the original PR, the default pageLimit stays at 100 — shrinking it to
30 was an unrelated product-knob change that would permanently cut nightly
take coverage.

Co-authored-by: tschew72 <tschew72@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(capture): make fallback title truncation explicit and astral-safe

Takeover of #2310. deriveTitle's silent .slice(0, 80) could split an astral
surrogate pair mid-character and gave no signal the title was cut. Truncation
is now codepoint-aware and appends an ellipsis (still capped at 80 codepoints).

Unlike the original PR, this stays a three-line change: no whitespace
normalization of every derived title, no word-boundary heuristics.

Co-authored-by: xd-Neji <xd-Neji@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(doctor): clear extract_atoms raw source-holder backlog + normalize pooler direct-URL overrides

Takeover of #2242, split to the two concerns that survive review:

- extract_atoms: exclude source pages whose frontmatter declares a raw
  payload pointer from discovery AND the doctor backlog count (shared SQL
  fragment so they can't drift). Extraction on these yields zero atoms, so
  no atom row is ever written and they re-enter the backlog every cycle —
  a permanent no-progress doctor blocker.
- connection-manager: a direct-URL override (opts/env) that still points at
  the Supavisor TRANSACTION pooler (port 6543, usually a copy-paste of the
  primary URL) is normalized to the real direct host via deriveDirectUrl.
  Session-mode pooler overrides (port 5432) pass through — they are a
  legitimate direct-ish target, which the original PR would have nulled out.

Dropped from the original PR: orphan-reporting atom exclusions (master
already excludes atoms/ and raw/ first segments plus /raw/ segments in
src/commands/orphans.ts) and the drain dry-run status tweak.

Co-authored-by: benjonp <benjonp@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cycle): record propose_takes deadline break as a halt in the extract rollup

A deadline-hit run breaks the page loop mid-list — same posture as budget
exhaustion — but the rollup still counted it as a completed round with no
halt, hiding chronic never-finishing nightly runs from extract-status/
doctor. Treat deadline_hit like budget_exhausted in the rollup deltas;
deadline test now pins halt=1 / completed=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: ethanbeard <ethanbeard@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: tschew72 <tschew72@users.noreply.github.com>
Co-authored-by: xd-Neji <xd-Neji@users.noreply.github.com>
Co-authored-by: benjonp <benjonp@users.noreply.github.com>
2026-07-23 13:23:36 -07:00
04e6b3af14 fix(cycle,lint): PGLite inline synth subagent drain + lint --exclude (takeover of #2699, #2649) (#3162)
* fix(cycle): drain PGLite synth subagents inline (takeover of #2699)

PGLite holds an exclusive file lock on its embedded data-dir, so no
separate Minions worker can serve the subagent children the synthesize
phase enqueues — they sat in 'waiting' until waitForCompletion timed
out. Drain a private per-run child queue inline (claim → run →
complete/fail, plus the promote/stall/timeout housekeeping a worker
would perform). No-op on Postgres, where children stay on the shared
'default' queue.

Rebased onto the reworked synthesize (config.subagentTimeoutMs, #1586
source scoping): the inline job context now carries deadlineAtMs from
the claim-time timeout_at stamp, and opts.yieldDuringPhase is ticked on
a 60s keepalive while each child runs so the 5-min cycle lock TTL
refreshes across long (up to 30-min) children.

Co-authored-by: TheRealMrSystem <TheRealMrSystem@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(lint): --exclude flag for mixed-content repos (takeover of #2649)

Adds --exclude=a,b (and LintOpts.exclude) so mixed-content repos can
skip software trees and repo metafiles by basename when collecting
pages. The only built-in default is node_modules — vendored dependency
trees are never knowledge pages; dot/underscore entries were already
skipped by the walk.

Diverges from #2649 deliberately: the original hardcoded an opinionated
default list (README.md, CHANGELOG.md, CLAUDE.md, test/ dirs at any
depth, plus fork-specific filenames), which silently changed lint
counts for every existing repo. Those are repo policy — pass --exclude.

Co-authored-by: ryangu00 <ryangu00@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cycle): enforce per-job timeout_ms in the PGLite inline subagent drain

The inline drain claimed children with deadlineAtMs derived from timeout_at
but never armed the worker's timeout timer — and the handleTimeouts sweep
only runs between jobs, so nothing could stop a child that blew past its
30-min timeout_ms. A hung LLM call wedged the drain loop (and the whole
cycle) indefinitely, with the 60s keepalive refreshing the cycle lock
forever. Worker.ts parity: arm a timer from the claim-time timeout_at
stamp, abort ctx.signal on fire, and dead-letter (never delayed-retry)
timed-out children, mirroring handleTimeouts' stall→retry / timeout→dead
split.

Regression test: a child with timeout_ms=100 whose handler only ends on
ctx.signal abort is dead-lettered with 'timeout exceeded'; pre-fix the
test hangs to its 30s timeout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: TheRealMrSystem <TheRealMrSystem@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: ryangu00 <ryangu00@users.noreply.github.com>
2026-07-23 13:09:42 -07:00
080b64e052 fix(doctor): brain_score orphan/timeline components use the orphans-audit linkable scope (#3155)
Takeover of #2525, rebased onto current master. getHealth() in both engines
now computes orphan_pages and the timeline component over a linkable_pages
CTE driven by the same constants the orphans audit uses
(src/core/linkable-scope.ts), so one doctor report can no longer show a 19%
orphan_ratio next to a no-orphans score implying ~70%. Master's newer
first-segment exclusions (raw, atoms, skills) are folded into the shared
scope so the orphans audit loses nothing in the move.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: pabloglzg <pabloglzg@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 13:07:08 -07:00
7e4094b2cd fix(ai): OpenRouter family-scoped prompt caching + expansion on chat-capable openai-compat recipes (#3152)
Takeover of #1988 (OpenRouter prompt caching), reimplemented on current
master: supports_prompt_cache may now be a per-model-id predicate; the
OpenRouter recipe marks openai/* chat and anthropic/claude-* routes
cacheable. Claude routes get an explicit cache_control on the system
content block via the recipe compat fetch shim (OpenRouter's documented
per-block format, not a top-level body field), signaled through a private
in-process marker header instead of the promptCacheKey sentinel that now
collides with the real OpenAI prompt_cache_key derivation. Cache reads on
OpenAI-compatible routes surface via the SDK's cachedInputTokens.

Root fix for #1135: deepseek, groq, and together now declare expansion
touchpoints (their expansion path is the same plain OpenAI-compatible
languageModel call as chat), so an explicit expansion_model pointed at
them no longer silently yields zero expansion.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: tmchow <tmchow@users.noreply.github.com>
Co-authored-by: warkcod <warkcod@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 13:07:00 -07:00
b5675437c0 fix(import): normalize mixed-case slugs before chunk upsert (#430) (#3143)
putPage lowercases slugs via validateSlug, but upsertChunks queried
pages by the caller's raw slug — so a mixed-case slug through
importFromContent created the page row, then failed the chunk upsert
with 'Page not found' and rolled back the whole import.

Normalize via validateSlug at importFromContent entry and inside
_upsertChunksOnce on BOTH engines (postgres + pglite parity).

Takeover of #855, rebased onto current master shapes (batchRetry
wrapper / _upsertChunksOnce, rewritten importFromContent opts block).

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:57:46 -07:00
8160236ade fix(search): honor sources.config.federated in unqualified local CLI search/query (#2561) (#3141)
A source registered with `gbrain sources add --federated` was invisible to
an unqualified `gbrain search`/`gbrain query`: the local CLI always emitted
a scalar {sourceId} scope, and nothing on the read path ever consulted
sources.config.federated — contradicting docs/guides/multi-source-brains.md
('Source participates in unqualified gbrain search results').

Fix, at the trusted-local boundary only:
- src/cli.ts makeContext resolves the source WITH its tier and, when the
  tier is non-explicit (local_path / brain_default / sole_non_default /
  seed_default), computes ctx.localFederatedSourceIds = [resolved source,
  ...other config.federated=true sources] (archived excluded).
- New federatedSearchScope (operations.ts) delegates to
  resolveRequestedScope, then widens an unqualified trusted-local scalar
  scope to that set. Used by the search + query handlers only.
- Expansion NEVER applies when ctx.remote !== false (fail-closed source
  isolation), when a per-call source_id/__all__ is passed, when an OAuth
  grant (allowedSources) is present, or when --source/GBRAIN_SOURCE/dotfile
  named the source explicitly.

Deliberately NOT inside sourceScopeOpts: code-intel ops reject multi-source
scopes (resolveCodeIntelScope) and non-search reads keep their scalar
behavior. Cache contamination is already handled — cacheScopeKey folds
sourceIds sets into the query-cache key.

Fixes #2561

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:57:42 -07:00
69e7e79a1f fix(ingest,sync,serve): three singleton P0s — type round-trip, deleted-slug embed noise, stateless width guard (#3140)
- #1035: importFromContent preserves an existing page's type when incoming
  frontmatter omits an explicit type: field. Explicit type stays an override;
  absence means preserve; new pages still path-infer. The existing-page fetch
  moved above the content-hash compute so a no-op re-put stays a hash-match
  skip. Root-cause fix covers put_page, sync, capture — every caller.
- #1284: sync's end-of-run auto-embed no longer receives slugs deleted in the
  same run (embedPage threw 'Page not found' per deleted slug and serr-logged
  noise on every rename/delete sync). pagesAffected stays the full manifest
  for extract/report paths; a slug deleted then re-imported in the same run
  stays embeddable.
- #1196: gbrain serve --http now runs doctor's embedding_width_consistency
  check at startup and prints a loud stderr banner (with the paste-ready
  recipe + GBRAIN_EMBEDDING_MODEL/DIMENSIONS hint) when the resolved width
  diverges from the brain's vector(N) column — the stateless-container
  fallthrough that broke every write. Fail-open; reads unaffected.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:57:37 -07:00
3594c316b5 fix(rerank): classify missing auth before fallback (#2059) (#3139)
Missing ZEROENTROPY_API_KEY threw AIConfigError from auth resolution, which
rerank.ts recorded as reason 'unknown' — and doctor's reranker_health had no
unknown bucket, so it reported ok while every rerank silently failed open.

- gateway.rerank wraps AIConfigError from applyResolveAuth as
  RerankError(reason: 'auth') before any HTTP call.
- checkRerankerHealth warns on >=3 'unknown' failures in the 7-day window
  (covers historical pre-fix audit rows), with a ZEROENTROPY_API_KEY setup
  hint when the error summary points at a missing key.
- Tests: RerankError(auth) classification, applyReranker fail-open + audit
  reason, doctor warn on repeated unknowns.

Takeover of #2070.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: maxpetrusenkoagent <maxpetrusenkoagent@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:57:31 -07:00
0853491eb2 fix(schema-pack): narrow stats catch-all so masked errors surface, not fake 0 pages (#2466) (#3133)
fetchCountRows and detectDeadPrefixes in src/core/schema-pack/stats.ts
swallowed EVERY engine error into empty results, so any real failure
printed 'Total pages: 0' + a vacuous 100% coverage on a populated brain.
Both catches now swallow only isUndefinedTableError (pre-init brain,
missing pages table) and rethrow everything else. Four regression tests:
real non-zero count on a populated PGLite brain, rethrow on non-missing-
table errors in both catch sites, and the missing-table degrade path.

Takeover of #2493.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:55:00 -07:00
2283269932 fix(context): read documented '## P1 — Today' plain tasks in live context (#2186) (#3124)
resolveTodayTasks only matched a bare '## Today' heading and bold-prefixed
'- [ ] **task**' lines, while the daily-task-manager skill's documented
Output Format writes '## P1 — Today' with plain '- [ ] task' lines — so
documented writes surfaced zero tasks in live context.

Reader now accepts both heading forms and both line forms, two-step: the
legacy bold prefix extracts just the task name (dropping trailing metadata),
falling back to the plain full-line form.

Salvaged from PR #2188 (reader-side half). The skill-doc rewrites in that PR
are dropped: master #2938 kept ops/ synced and made put_page write-through
durable, so the 'gbrain get/put ops/tasks' docs are correct as-is. The PR's
single-regex line matcher is replaced with the two-step match because its
alternation captured '**name** — metadata' verbatim for bold lines.

Takeover of #2188. Fixes #2186.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: caioribeiroclw-pixel <caioribeiroclw-pixel@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:42:56 -07:00
c571bf82de fix(write-through): guard case-insensitive filesystem collisions before atomic write (#2831) (#3119)
On macOS/Windows (case-folding filesystems), the write-through rename
silently clobbered a differently-cased file already occupying the target
path (uncontrolled repo files like README.md vs slug readme, or unicode
normalization variants between slugs). Refuse with
skipped: 'case_insensitive_collision' when the path exists on disk but no
exactly-named directory entry does; exact-case updates fall through and
case-sensitive filesystems are unaffected.

Fixes #2831

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:42:50 -07:00
8dc3310483 fix(dream): stamp incremental extraction watermark (#2636) (#3115)
The Dream cycle disables sync's inline extraction and routes changed
slugs through extractForSlugs, which flushed link/timeline batches but
never stamped links_extracted_at — so incrementally extracted pages
stayed permanently visible to `extract --stale` / doctor.

Collect processedRefs per successfully processed page and stamp them
via stampExtracted (best-effort) after both batch flushes, non-dry-run
mode 'all' only. Source-id threading from the original PR #2637 already
landed on master via #1503/#1747, so this rebase carries only the
missing watermark stamp plus regression tests.

Takeover of #2637.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: JavanC <JavanC@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:42:40 -07:00
58606cc924 fix(serve-http): make OAuth /token rate limit configurable via env (#3114)
Adds GBRAIN_OAUTH_TOKEN_RATE_LIMIT_MAX and
GBRAIN_OAUTH_TOKEN_RATE_LIMIT_WINDOW_MS to tune the /token
client_credentials limiter (default unchanged: 50 req / 15 min).
Invalid, zero, or negative values fall back to the default.

Takeover of #2501 (mechanical rebase onto master after #2625 shifted
the surrounding context in serve-http.ts). Fixes #2463.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: techtony2018 <techtony2018@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:29:24 -07:00
66f4cb6d82 fix(dream): keep dream --dry-run --json stdout clean of embed summaries (#394) (#3109)
The cycle's embed phase called runEmbedCore with no output suppression, so
the '[dry-run] Would embed ...' / 'Embedded N chunks ...' slog summaries
landed on stdout ahead of the JSON CycleReport, breaking the documented
stdout-clean-for-JSON contract (docs/progress-events.md).

Adds EmbedOpts.quiet gating the human stdout summary slog sites in
embed.ts (embedPage, embedAll, embedAllStale); the cycle's runPhaseEmbed
sets quiet: true since it reports counts via its own PhaseResult. Errors
and warnings still go to stderr regardless.

Takeover of #854 (same approach, reimplemented on current master — the
original patch predates the slog migration and the widened
embedAll/embedAllStale signatures). Regression test ported from #854.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:28:27 -07:00
c852abfcb6 fix(onboard): stop dropping onboard-check remediations on the --apply --auto path (#3097)
Takeover of #2161. runRemediation ignored onboard-check extras in three
places: the pre-flight plan, the initial recommendation build, and the D7
mid-run recheck that rebuilds recs after every completed step. The --check
path threaded extras correctly, so `gbrain onboard --apply --auto` reported
"Nothing to do" when the only remediable work came from onboard checks —
and even with the first two sites fixed (the original PR diff), any plan
with 2+ steps dropped all remaining extras after step 1 via the recheck.

- Add RemediationOpts.extraRemediations; thread it through the pre-flight
  plan, initial recs, and the mid-run recheck.
- Recheck filters extras to ids not already processed this run: extras
  carry static status:'remediable', so unfiltered threading would resubmit
  completed extras forever.
- Wire the CLI --auto path (onboard.ts) AND the MCP run_onboard auto path
  (operations.ts), which already computed the scope-filtered allowedExtras
  and then dropped it.
- Regression test: extras-only plan on an empty brain runs BOTH extras
  exactly once and terminates (serial file: mock.module queue stub +
  GBRAIN_HOME tmpdir).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:26:43 -07:00
b91350d778 fix(autopilot,eval): nightly quality probe enable path + conversation-parser probe wire-up (takeover of #2629, #2630) (#3094)
* fix(autopilot,eval): nightly quality probe enable path works end-to-end + wire conversation-parser probe

Takeover of #2629 and #2630 (rebased onto master; dropped the
test/engine-find-trajectory.test.ts hunk both PRs carried — master
already ships the equivalent gateway-dims fix).

#2629 — nightly quality probe enable path:
- autopilot + doctor read the probe flag dual-plane (DB config row from
  'gbrain config set' wins, ~/.gbrain/config.json fallback) via new
  resolveProbeEnabled/resolveProbeMaxUsd helpers
- resolveRepoRoot prefers the gbrain package root where the committed
  fixture lives, not the brain repoPath
- rate_limited skips no longer write an audit row every autopilot cycle
- eval-longmemeval strips 'provider:' recipe ids before raw Anthropic SDK
  calls and emits the gold answer for downstream judges
- cross-modal batch folds the gold answer into the judge task; probe
  passes QA-shaped dimensions instead of the agent-response rubric
- DEFAULT_SLOTS slot A moves to openai:gpt-5.2 (gpt-4o left the recipe);
  new consistency test pins every default slot to its recipe

#2630 — conversation-parser nightly probe wire-up:
- autopilot step 4.6 invokes runConversationParserNightlyProbe (dual-plane
  flag + D10 tokenmax mode-gate, package-root fixtures, 24h gate, audit
  trail via new src/core/audit-parser-probe.ts)
- doctor's conversation_parser_probe_health replaces the hardcoded
  'Skipped' stub with a real pure-function check over the audit trail

Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pricing): add openai:gpt-5.2 canonical entry for the new default slot A

DEFAULT_SLOTS slot A moved to openai:gpt-5.2, which had no CANONICAL_PRICING
entry — estimateCost silently dropped slot A from the --max-usd pre-flight
and est_cost_usd audit rows (~1/3 under-count on the default panel). Rates
from the OpenAI recipe chat touchpoint (verified 2026-04-20). Also refresh
the --slot-a-model help text default and pin a pricing-presence assertion
in the DEFAULT_SLOTS consistency test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-23 12:26:33 -07:00
e1156a5642 fix(migrations): scope v0.32.2 dirty-check to targeted sources; surface failed phase detail (#3093)
- phaseBFenceFacts now queries legacy rows FIRST and dirty-checks only
  the source_ids it will actually write into. Zero fenceable rows (or
  rows scoped to clean sources) no longer fail on an unrelated dirty
  source. Targeted-dirty-source refusal unchanged. Fixes #927.
- apply-migrations now prints each failed phase's name + detail to
  stderr alongside 'reported status=failed', instead of burying the
  actionable message in the ledger. Fixes #921.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:26:28 -07:00
872d4eebb5 fix(init,mcp): seed init AI options from env on cold install; whoami reports stdio transport (#3091)
Two backlog fixes:

- init (#1058): loadConfig() returns null on a cold install (no config.json
  AND no DATABASE_URL), short-circuiting before its env merge — so
  GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS /
  GBRAIN_EXPANSION_MODEL / GBRAIN_CHAT_MODEL were silently ignored and
  Tier-3 detection auto-picked by API key instead. resolveAIOptions' config
  seed now falls back to those env vars directly when loadConfig() is null
  (new exported helper seedAIOptionsFromConfig, env-injectable for tests).

- whoami (#1061): the stdio MCP dispatch is remote/untrusted by design but
  has no per-token auth (local pipe), so whoami threw unknown_transport on
  the primary stdio surface. The stdio dispatch now marks
  ctx.transport = 'stdio' and whoami returns {transport: 'stdio', scopes: []}
  for it. Trust posture unchanged: remote stays true, the marker is never
  used for trust decisions, and an unmarked auth-less remote context still
  throws (fail-closed preserved).

Co-authored-by: Garry Tan <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:14:14 -07:00
fecd331f02 fix(recipes/minimax): embedding wire-shape compat fetch + chat touchpoint (#1977) (#3089)
MiniMax's /v1/embeddings endpoint is not OpenAI-compatible: it requires
texts (not input) plus a type field and returns {vectors} instead of
{data:[{embedding}]}. The recipe shipped no transport shim, so every
embed call failed with an invalid-params error, and it declared no chat
touchpoint, so assertTouchpoint blocked gbrain think even though
MiniMax chat is genuinely OpenAI-compatible.

Fix (takeover of #2882, corrected):
- minimaxCompatFetch via the DeepSeek-style compat.fetch seam (keeps
  cfg.base_urls overrides working; no new env var), gated on the
  /embeddings path so chat requests/responses pass through untouched.
- Response rewrite parses via resp.clone() and rebuilds with fresh
  headers — never returns a body-consumed Response (the flaw in #2882's
  wrapper, which broke every non-streaming chat completion).
- chat touchpoint with the /v1/models list from #1977.

Fixes #1977

Co-authored-by: Garry Tan <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ArthurHeung <ArthurHeung@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:14:07 -07:00
79f6d1bfee fix(gateway): fall back to the pooler when the derived direct host is unreachable (#1641) (#3088)
deriveDirectUrl() swaps the Supabase pooler host to db.<ref>.supabase.co:5432,
which is IPv6-only without the paid IPv4 add-on. On IPv4-only networks the
direct pool could never connect, and initDirectPool()'s throw killed
'gbrain init --url' and migrations with ENOTFOUND/ECONNREFUSED.

getDirectPool() now classifies network-unreachable errors (ENOTFOUND,
ECONNREFUSED, ENETUNREACH, EHOSTUNREACH, ETIMEDOUT, CONNECT_TIMEOUT) via the
new isNetworkUnreachableError(), self-activates the kill-switch, logs one
stderr line pointing at GBRAIN_DIRECT_DATABASE_URL / GBRAIN_DISABLE_DIRECT_POOL,
and returns the read pool. Auth/SQL errors still throw (misconfig, not
unreachability). The failed pool is ended via endPoolBounded so it can't
leak sockets into the now-continuing process.

Also surfaces the kill-switch + override envs in the init.ts IPv6 warnings
and docs/guides/live-sync.md (they were previously undocumented outside
connection-manager.ts).

Fixes #1641

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:14:02 -07:00
ef840d9561 fix(gateway): add chat touchpoint to zhipu recipe so GLM subagents work (#1157) (#3084)
The zhipu recipe was embedding-only, so models.tier.subagent=zhipu:glm-5.1
threw "does not offer a chat touchpoint" — while the error hint falsely
listed zhipu (and dashscope/minimax, also embedding-only) among providers
with chat.

- zhipu recipe: add a chat touchpoint (glm-5.1 family, supports_tools +
  supports_subagent_loop; no Anthropic-style prompt cache on the
  OpenAI-compat path, so the loop runs with the degraded:no_caching warn).
  openai-compat tier means newer GLM ids pass without a recipe edit.
- capabilities.ts: compute the "Known providers with chat" hint from the
  recipe registry instead of a hardcoded list, so it can never drift into
  naming chat-less providers again.
- Declines the originally requested models.anthropic_compatible_prefixes
  config: v0.38's recipe-driven capability gate already replaced the
  Anthropic-only enforcement, so a recipe chat touchpoint is the whole fix.

Fixes #1157

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:13:57 -07:00
b139602119 fix(slugs): CJK slug support in SlugRegistry and dream-cycle summary slug (takeover of #782, #738) (#3083)
Master already widened slugifySegment (sync.ts) and validatePageSlug
(operations.ts) to CJK in v0.32.7, but the other two validators #782
targeted stayed ASCII-only: SlugRegistry's SLUG_RE rejected any CJK
desiredSlug from BrainWriter, and synthesize.ts's SUMMARY_SLUG_RE (whose
comment claimed it was kept in sync with validatePageSlug) rejected CJK
output roots.

Hoist the segment grammar into cjk.ts as PAGE_SLUG_SEG and compose all
three regex sites from it, so the four slug validators share one grammar.
Each site keeps its own shape (SlugRegistry's >=2-segment dir/name form,
validatePageSlug's case-insensitive flag).

Scope stays CJK (matching v0.32.7), not full \p{L} Unicode as #782
proposed — all-scripts slugs (lookalike/RTL spoofing) is a maintainer
policy call.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: tamagodo-fu <tamagodo-fu@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:13:50 -07:00
7421efc41e fix(schema): skip unsupported large-dim HNSW indexes (#1734) (#3080)
Takeover of #2510: migrations v40 (facts) and v55 (query_cache)
unconditionally created HNSW indexes with the configured embedding
dimension, so `gbrain init` with embedding_dimensions above pgvector's
per-type HNSW caps (vector 2000 / halfvec 4000) failed with
"column cannot have more than 4000 dimensions for hnsw index".

- vector-index.ts: add PGVECTOR_HNSW_HALFVEC_MAX_DIMS + hnswMaxDimsForType
- migrate.ts v40/v55: emit the HNSW index only when dims fit the cap,
  otherwise a comment noting exact scans remain available
- embedding-dim-check.ts: buildFactsAlterRecipe skips the reindex step
  above the cap for the same reason
- tests: 4096d init round-trip on PGLite (columns exist, indexes
  skipped) + recipe-skip unit test

Drops the unrelated context-engine.ts interface change and the
tsconfig.json strictFunctionTypes=false hunk from #2510; typecheck is
clean without them.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:13:42 -07:00
b0f74017d7 fix(calibration): resolve owner holder via config (default 'self') (#3077)
* fix(calibration): resolve owner holder via config (default 'self'), fixes #2464

Takeover of #2467 (rebased onto master). consolidate writes owner takes
with holder='self' while calibration-profile, calibration CLI/op, think's
calibration block, emotional-weight, and doctor's calibration_freshness
all defaulted to a hardcoded 'garry' — so getScorecard returned 0
resolved and the calibration profile never built on non-upstream brains.

New src/core/owner-holder.ts is the single source of truth:
resolveOwnerHolder({override, configValue}) = override >
emotional_weight.user_holder config > 'self'. All six call sites route
through it; doctor's freshness SQL is parameterized ().

Upgrade note: upstream-owner brains with historical holder='garry'
profiles should `gbrain config set emotional_weight.user_holder garry`
to keep reading them.

Co-authored-by: devty <devty@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(calibration): replace real-name holder fixture with charlie-example placeholder

Privacy iron rule: no real people's names in checked-in code. The sanctioned
placeholder mapping uses people/charlie-example.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: devty <devty@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-23 12:03:26 -07:00
941e7746d4 fix(backlinks): honor positional check-backlinks directory argument (#3076)
The help text (gbrain check-backlinks <check|fix> [dir]) promised a
positional directory argument, but runBacklinks only parsed --dir and
defaulted to cwd, so the walker ran from the wrong root and could hit
EPERM on unreadable sibling dirs.

Extract parseBacklinksArgs: positional [dir] is now honored, --dir still
overrides it, --dry-run preserved, and a --dir flag missing its value
falls back to the positional dir instead of picking up undefined.

Takeover of #852 (rebased onto master past the findBacklinkGaps dedupe
test block). Fixes #485.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Kage18 <Kage18@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:03:16 -07:00
d6fe486370 fix(doctor): register onboard check names in doctor-categories to stop unknown-check warnings (#3075)
doctor.ts pushes runAllOnboardChecks results into the checks list, but the
7 onboard check names (embed_staleness, entity_link_coverage,
timeline_coverage, takes_count, dangling_aliases, pack_upgrade_available,
type_proliferation) were never added to doctor-categories.ts, so every
doctor run emitted an 'unknown check name' stderr warn per onboard check.

Registers the 5 data-quality names under BRAIN and the 2 schema-pack names
under META (alphabetical order preserved), and widens the drift-guard test
to scan src/core/onboard/checks.ts alongside src/commands/doctor.ts so
future onboard checks can't drift uncategorized.

Takeover of #1839, rebased onto master (keeps master's timeline_dedup_index).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: mvanhorn <mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:03:06 -07:00
080692fa47 fix(import): fall back to body H1 for title when frontmatter lacks title: (#2446) (#3072)
Title precedence is now frontmatter title: > body's first ATX H1 > the
slug/filename-humanized fallback. Slug-based imports (contacts, calendar)
carry a correct # Heading but no frontmatter title; without the H1 fallback
they got junk titles humanized from the slug. The H1 scan skips h2+ and
lines inside fenced code blocks, and strips closed-ATX trailing hashes.

Takeover of #2495.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:03:01 -07:00
Spinsirr d574e843a8 fix(mcp): source-scope hardening for remote callers (#2881)
Three fixes in the same leak class (a remote caller reading or writing
outside its granted sources), for multi-source / multi-tenant brains:

1. dispatchToolCall now refuses remote calls that arrive without a
   resolved sourceId (missing_source_scope error envelope) instead of
   silently falling back to the shared 'default' source. Every shipped
   transport already passes sourceId explicitly (serve-http from the
   OAuth client row, http-transport from the legacy token grant, stdio
   from GBRAIN_SOURCE); reaching the fallback remotely always meant a
   programmatic caller skipped scope resolution — the bug class behind
   #1924 / #1371. Trusted local callers (remote === false) keep the
   historical fallback. Direct-dispatch tests updated to carry an
   explicit sourceId, matching the real transport contract.

2. log_ingest threads ctx.sourceId (same pattern as get_chunks /
   get_page), so ingest events are attributed to the caller's source
   instead of piling into 'default'. Engines already accept
   entry.source_id (v0.31.2).

3. get_ingest_log is source-scoped for remote callers via the
   linkReadScopeOpts collapse rule (scalar grant → [scalar]; federated
   grant → granted array); it previously returned the whole brain's
   ingest log to any read-scoped remote client, and ingest summaries can
   carry another source's private context. Trusted local callers keep
   the whole-brain view.

Tests: dispatch guard (refuse remote-without-source, keep local
fallback, guard ordering after op lookup) and end-to-end ingest-log
attribution + scoping over the real dispatch path, on PGLite.
2026-07-23 12:02:56 -07:00
Gawie van BlerkandGawie van Blerk 22cb074943 fix(sync): honor the embedding_disabled sentinel as implicit --no-embed (#2879)
gbrain init --no-embedding writes embedding_disabled: true as a
deferred-setup sentinel, and init/import/embed honor it via
assertEmbeddingEnabled. sync's embed credential preflight (v0.41.6.0 D1)
only checked the --no-embed CLI flag, so every gbrain sync on a keyless
deferred-setup brain exited 1 demanding <PROVIDER>_API_KEY — including
orchestrated callers (gstack /sync-gbrain) that never pass --no-embed.

embed-preflight.ts's own skip protocol documents that the sentinel is
owned upstream of the credential check; this wires that contract into
sync by deriving noEmbed from CLI args + config in one exported pure
helper (resolveNoEmbed), covered by test/sync-no-embed-sentinel.test.ts.

Co-authored-by: Gawie van Blerk <gawie.vanblerk@emeraldlife.co.za>
2026-07-23 12:02:50 -07:00
Amit AgarwalandAmit Agarwal d21f34e96d fix(search): project email citation metadata (#2873)
Co-authored-by: Amit Agarwal <5302320+amtagrwl@users.noreply.github.com>
2026-07-23 12:02:46 -07:00
Andreandmerlin-drizzyenterprises[bot] <144527811+merlin-drizzyenterprises[bot]@users.noreply.github.com> fa43907df4 fix(import): post-write read-back verification with durable ingest-log record (#2869)
A page write is not 'done' until it is readable back. After the import
transaction commits, verify the page resolves via getPage and its
content_hash matches what was just written. On mismatch or miss, fail
LOUDLY instead of reporting success, and record the failure in
ingest_log (best-effort) so it is durable and agent-inspectable rather
than a transient stderr message.

This catches the silent-desync class: the page file exists on disk (or
the git commit landed) but the DB index never picked the write up —
the operation previously reported success while the page stayed
invisible to every read path (get_page, search, query) until someone
noticed the gap manually.

Guard applies to both importFromContent (markdown) and importCodeFile.

Tests: new write-verify-guard suite (hermetic PGLite) covering the
happy path, index-miss, stale-hash, ingest_log record, and the put_page
operation surface; import-file.test.ts mock upgraded to simulate a
readable DB (writes are read-backable), matching the new guard.

PRJ-2026-032

Co-authored-by: merlin-drizzyenterprises[bot] <144527811+merlin-drizzyenterprises[bot]@users.noreply.github.com>
2026-07-23 12:02:41 -07:00
Garry Tan e0a208d7b7 Revert "fix(sources): stop source config re-wrapping into a growing JSON string scalar (#2829) (#2837)"
This reverts commit e36251c023.
2026-07-23 12:02:36 -07:00
Garry Tan 418357332f Revert "fix(embed): stamp gateway-resolved model in content_chunks.model, not compiled default (#2846)"
This reverts commit 5aa4795c04.
2026-07-23 12:02:36 -07:00
Garry Tan 45f85df8f4 Revert "fix(webhook): extract links for incremental push syncs (#2850)"
This reverts commit 11659743a2.
2026-07-23 12:02:36 -07:00
Garry Tan 9a70945152 Revert "fix(autopilot): give full-cycle dispatch a 30-minute timeout floor (#2852)"
This reverts commit b98fae9b61.
2026-07-23 12:02:36 -07:00
Garry Tan aea6df3da7 Revert "fix(onboard): stop repeating the same auto-remediation within a run (#2854)"
This reverts commit 054badbe60.
2026-07-23 12:02:36 -07:00
Garry Tan 35edd0e2d5 Revert "fix(scripts): capture check/shard rc before watchdog teardown in no-timeout fallback (#2864)"
This reverts commit e9a4fee97f.
2026-07-23 12:02:36 -07:00
Garry Tan 6388be2088 Revert "fix(list_pages): surface truncation instead of silently capping enumeration (#2865)"
This reverts commit 323610ecd7.
2026-07-23 12:02:36 -07:00
323610ecd7 fix(list_pages): surface truncation instead of silently capping enumeration (#2865)
list_pages clamps limit to max 100 (default 50) — deliberate server
protection, pinned in test/search-limit.test.ts. But the clamp was
SILENT: a caller whose limit was defaulted or clamped got a
full-looking array with no signal that rows were dropped, and with the
default updated_desc sort the dropped rows are always the OLDEST —
precisely what exhaustive consumers (audits, scans, backfills) exist
to find. Observed in the field: a source with 212 pages enumerated as
80 visible rows, hiding 26 pages from a compliance scan for days.

Fix, with no response-shape change (MCP consumers still get an array)
and no engine surface change (handler probes limit+1):

- handler probes one row past the effective limit; when the caller's
  limit was NOT honored (unset -> default, or clamped to cap) and rows
  were dropped, it warns on stderr for local (CLI) callers — same
  operator-facing channel as the put_page unknown-type hint, but
  without the isTTY gate: scripted callers are exactly the consumers
  that cannot detect truncation any other way, and stderr keeps stdout
  parseable. An explicit honored limit stays silent (ordinary
  pagination), as does a clamped-but-complete result. Remote (MCP)
  ctx never writes to stderr.
- LIST_PAGES_DESCRIPTION documents the cap and the exhaustive-listing
  recipe (sort=updated_asc + updated_after cursor) — the description
  is the signal channel MCP clients actually read.
- regression suite: default-limit truncation warns, honored limit
  silent, clamped-but-complete silent, remote silent, and the
  documented cursor recipe enumerates a corpus to completion.

Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:38:20 -07:00
e9a4fee97f fix(scripts): capture check/shard rc before watchdog teardown in no-timeout fallback (#2864)
On machines with neither gtimeout nor timeout on PATH, run-verify-parallel.sh
and run-unit-parallel.sh fall back to a bg-pid + sleep-watchdog cap. Both
read $? only after tearing the watchdog down (kill + wait on cap_pid), so the
sentinel .exit files recorded the killed watchdog's status — 143 — instead of
the check/shard's own exit code. Every run reported total failure (verify:
pass=0 fail=31; unit: rc=143 per shard) while every per-check/shard log
showed success.

Capture rc immediately after `wait $pid` in both scripts, and reap the
watchdog's sleep child (pkill -P, children-first — the same orphan quirk the
heartbeat cleanup documents) so the fallback stops leaking one sleep per
check/shard.

Regression tests force the fallback branch hermetically on any host via a
curated PATH with no timeout binaries: the verify dispatcher runs from a
tempdir copy with a stubbed `bun`, pinning exit 0 + all-zero sentinels when
checks pass and the check's own rc (not 143) when one fails; the unit wrapper
runs real two-shard fixture passes, pinning rc=0 sentinels and a real
failure's rc=1.

Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:38:15 -07:00
Sanchal Ranjan 054badbe60 fix(onboard): stop repeating the same auto-remediation within a run (#2854)
When the recommendation list is refreshed between remediation steps, a
remediation that doesn't clear its own health signal is reintroduced
under its stable id and attempted again, indefinitely on long runs.
Track attempted recommendation ids for the run and skip re-attempts.

Includes a behavioral regression test: a persistently-stuck signal is
attempted once, the loop terminates, and other remediations still run.
2026-07-23 11:38:10 -07:00
Sanchal Ranjan b98fae9b61 fix(autopilot): give full-cycle dispatch a 30-minute timeout floor (#2852)
Dispatch timeout was derived as interval*2 with a 5-minute floor, tuned
for light per-interval work. A full autopilot cycle routinely needs more
than 10 minutes at common intervals, so healthy full cycles were killed
mid-run. Full-cycle dispatch now gets a 30-minute floor; lighter
dispatches keep the interval-derived budget.

Adds a regression test for the full-cycle floor.
2026-07-23 11:38:05 -07:00
Song 11659743a2 fix(webhook): extract links for incremental push syncs (#2850)
* test(webhook): pin sync extraction contract (#2849)

* test(webhook): target the submitted sync payload (#2849)

* fix(webhook): run extraction in sync job (#2849)

* fix(sync): align push trigger extraction (#2849)
2026-07-23 11:38:00 -07:00
SailorJoe6andClaude Opus 4.8 5aa4795c04 fix(embed): stamp gateway-resolved model in content_chunks.model, not compiled default (#2846)
upsertChunks fell back to the compile-time DEFAULT_EMBEDDING_MODEL
('zeroentropyai:zembed-1') when a ChunkInput carried no explicit `model`.
The embed pipeline (src/commands/embed.ts) builds ChunkInputs without a
`model` field, so rows whose vectors were produced by the config-resolved
model (e.g. openai:text-embedding-3-large) were mislabeled with the
hardcoded default — corrupting the provenance that signature-drift
staleness and dimension-migration logic depend on.

Both engines now resolve the gateway's runtime embedding model once per
upsert and use it as the fallback, mirroring the existing resolve-then-
default pattern used for schema sizing. Regression test added (pglite);
verified via negative control that it fails against the old fallback.

This is a write-path change (upsertChunks), not a search-path change, so
retrieval eval replay is not applicable.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 11:37:06 -07:00
1alessioandClaude Fable 5 e36251c023 fix(sources): stop source config re-wrapping into a growing JSON string scalar (#2829) (#2837)
`sources.config` is a jsonb OBJECT column, but a read→write cycle that
JSON.stringify'd an already-stringified value re-wrapped it into a JSON string
scalar ("{}", "\"{}\"", ...) that grew one layer per write. parseSourceConfig
only unwrapped one layer, so the corruption never healed and federation/ACL
reads saw a string instead of the settings object.

- Add normalizeSourceConfig: a bounded (10-iteration) loop that JSON.parses
  while the value is a string and returns {} (with a console.warn) when the
  result is not a plain object. All six `UPDATE sources SET config` writers run
  their config through it before stringify, converging the stored value back to
  a jsonb object on the next write.
- parseSourceConfig now does the same bounded unwrap and warns once when more
  than one layer was found (one layer is the normal PGLite path).
- Add a `source_config_shape` doctor check that flags any sources row where
  jsonb_typeof(config) <> 'object', with the repair path.
- Unit-test the helper (object passthrough, 1-layer, 5-layer nested, garbage
  and over-bound inputs) and the doctor check (mock engine).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:37:02 -07:00
Ziyang Guo 7bbd087cb7 fix(pages): restore soft-deleted rows on putPage (#2779) 2026-07-23 11:25:00 -07:00
TurgutKural e0d2cbf353 fix(doctor): distinguish entity timeline coverage from whole-brain density (#2761)
Issue #2298: 'gbrain doctor' surfaced two distinct timeline
metrics under the same user-facing 'timeline' label:

1. Entity timeline coverage (graph_coverage metric)
   - numerator: eligible entity pages WITH a timeline entry
   - denominator: eligible entity pages
   - 0-1 fraction, surfaced by graph_coverage check
2. Whole-brain timeline density (brain-score 0-15 component)
   - numerator: all pages WITH a timeline entry
   - denominator: all pages
   - 0-15 scale, surfaced by brain_score breakdown

These have DIFFERENT numerators/denominators. The old single
'timeline X%' label let a reader mistake the entity-scoped
percentage for whole-brain density.

Presentation/contract clarity only — scoring formula,
health weights, takes, source routing, extraction UNCHANGED.

- doctor.ts graph_coverage: 'timeline X%' -> 'entity timeline coverage X%'
- doctor.ts brain_score: 'timeline X/15' -> 'timeline density (all pages) X/15'
- cli.ts get_health: 'Timeline coverage (entity pages)' ->
  'Timeline density (all pages): X/15 (whole-brain brain-score component)'

Test: test/doctor-timeline-metric-labels-2298.test.ts uses a
synthetic in-memory PGLite fixture (NO private EriadorMu data):
4 total pages, 2 eligible entity pages, 1 entity page with a
timeline entry, 1 total page with a timeline entry.
Expected: entity coverage 1/2 = 50%; whole-brain density
1/4 -> round(25% * 15) = 4/15. Asserts the two metrics
render with distinct scoped labels and the brain-score component
is explicitly whole-brain (no 'entity' in its label). 5/5 pass.

Addresses #2298
2026-07-23 11:24:55 -07:00
symmetric-matthewandMatthew Thompson 7a1f61a31a fix: clear verified sync head sentinels (#2734)
Co-authored-by: Matthew Thompson <matthew@symmetric-consulting.com>
2026-07-23 11:24:50 -07:00
9b8b829ca5 fix(extract): --stale sweep runs the real resolver — basename resolution reaches stale pages (#2576) (#2717)
extractStaleFromDB still used the pre-#972 `includeFrontmatter ? resolver :
nullResolver` ternary. The synthetic resolver has no resolveBasenameMatches,
so the gate in extractPageLinks skipped the issue-#972 bare-wikilink pass
regardless of link_resolution.global_basename — the sweep stamped every page
as extracted while silently dropping its [[bare-name]] links. Same brain,
same pages: `extract --stale` created 0 links where `extract links --source
db` created 218.

- Always pass the real batch resolver; gate passes via extractPageLinks opts
  ({ skipFrontmatter: !includeFrontmatter, globalBasename }), mirroring
  extractLinksFromDB — including the codex-[P1] sourceId scoping.
- Bump LINK_EXTRACTOR_VERSION_TS (documented protocol) so pages stamped by
  the broken sweep re-flag stale and re-extract under the fixed logic.
- Regression tests: bare wikilink resolves on --stale with the flag ON;
  still drops with the flag OFF (back-compat). The #1768 fixture now derives
  its updated_at from LINK_EXTRACTOR_VERSION_TS instead of a hardcoded date,
  so future version bumps can't silently flip its version arm.

Fixes bug 1 + bug 3 of #2576. Bug 2 (DIR_PATTERN gaps) is a separate
whitelist design call, intentionally not addressed here.

Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:24:45 -07:00
c27b2e4b0f fix(put): refuse to overwrite a non-empty page with empty content (#2708)
An empty --content (most commonly a non-interactive caller that meant
file input — put has no --file flag — so the missing --content fell
back to reading empty stdin) silently blanked existing pages. put_page
now rejects an empty/whitespace-only body over an existing non-empty
page with invalid_params, pointing at `gbrain capture --file PATH
--slug SLUG` for file input; allow_empty: true (CLI: --allow-empty)
opts into an intentional blank. The guard read is scoped to the exact
(source_id, slug) row the write targets; new-slug creates and
soft-deleted-page overwrites stay allowed.

Co-authored-by: Matthew Thompson <matthew@symmetric-consulting.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 11:24:40 -07:00
Ziyang Guo 9bcfa67748 fix(schema): count dead prefixes by slug (#2697) 2026-07-23 11:24:35 -07:00
Javier Aldapeandgbrain-contrib 16eb8cd06c fix(doctor): flag embed backfills without a worker (#2696)
Co-authored-by: gbrain-contrib <gbrain-contrib@example.com>
2026-07-23 11:24:30 -07:00
Garry Tan 92a3202198 Revert "fix(trajectory): stop negative metrics from inverting regression signals (#2621)"
This reverts commit 5dcf3e7b2f.
2026-07-23 11:24:25 -07:00
Garry Tan 8b7e30afcd Revert "perf(contextual-retrieval): bound per-chunk synopsis concurrency (#2628)"
This reverts commit 3454dca0b4.
2026-07-23 11:24:25 -07:00
Garry Tan 68e4cebd1a Revert "fix(health): count 'entity' pages in graph health metrics (#2639)"
This reverts commit 8fc93c8fac.
2026-07-23 11:24:25 -07:00
Garry Tan 8bbb19102c Revert "fix(test): isolate GBRAIN_HOME in hybrid-reranker integration test (#1527) (#2640)"
This reverts commit fe6850b067.
2026-07-23 11:24:25 -07:00
Garry Tan 9ae4e04d22 Revert "feat(ai): dashscope-rerank recipe (DashScope serves PLURAL /reranks under compatible-api) (#2644)"
This reverts commit 220af4b2d0.
2026-07-23 11:24:25 -07:00
Garry Tan fe2f2f6b2a Revert "fix: clarify PGLite data-dir lock contention (#2658)"
This reverts commit 0556dbdc2c.
2026-07-23 11:24:25 -07:00
Garry Tan fc169d9770 Revert "fix(import): normalize mixed-case slugs (#2695)"
This reverts commit 50406fc212.
2026-07-23 11:24:25 -07:00
Ziyang Guo 50406fc212 fix(import): normalize mixed-case slugs (#2695) 2026-07-23 11:03:09 -07:00
zay 0556dbdc2c fix: clarify PGLite data-dir lock contention (#2658) 2026-07-23 11:03:03 -07:00
YiconandClaude Opus 4.8 220af4b2d0 feat(ai): dashscope-rerank recipe (DashScope serves PLURAL /reranks under compatible-api) (#2644)
DashScope's OpenAI-compatible rerank endpoint lives at
{base}/compatible-api/v1/reranks — PLURAL leaf, different base path from
the embedding surface (compatible-mode). Reusing llama-server-reranker
against DashScope forces users to hand-patch the recipe's '/rerank' leaf
in node_modules, which every upgrade silently reverts (and llama.cpp
genuinely serves singular /rerank, so changing that recipe would break
real llama.cpp users).

New dedicated recipe rides the v0.40.6.1 recipe-pluggable reranker path:
- id dashscope-rerank, base_url_default compatible-api/v1 (intl), ZE wire
- path '/reranks', default_timeout_ms 30s, 5MB payload ceiling
- models: only qwen3-rerank (live-verified 200; gte-rerank-v2 is rejected
  by the compat surface with 'Unsupported model for OpenAI compatibility
  mode', so it is deliberately not listed)
- separate recipe (not a reranker touchpoint on dashscope) because
  provider_base_urls is keyed by recipe id and the two capabilities need
  different prefixes — same topology as llama-server vs
  llama-server-reranker

Tests: recipe shape smoke mirroring recipe-llama-server-reranker.test.ts
(path/timeout/payload pins, /v1/v1 concat guard, auth resolve, sibling
recipe isolation). bun test test/ai/: 322 pass / 0 fail.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 11:02:53 -07:00
WillisbestandClaude Opus 4.8 fe6850b067 fix(test): isolate GBRAIN_HOME in hybrid-reranker integration test (#1527) (#2640)
The four `hybridSearch — reranker enabled (reorder)` cases stub the gateway
at 1536 dims (DIMS). Since v0.36.3.0 hybridSearch resolves the embedding
column via loadConfig(), whose precedence is
cfg.embedding_dimensions > gateway dims > default. On any machine whose
~/.gbrain/config.json sets embedding_dimensions to something other than 1536
(e.g. text-embedding-3-small at 1280), the real config outranks the stub: the
1536-d stub vector fails the gateway dim check, the error is swallowed, search
falls back to keyword-only, and the reranker never runs (rerankerFn gets 0
docs, rerank_score undefined). Green in CI only because a fresh runner has no
config file — deterministic red on a contributor's machine.

Fix (test-only): isolate GBRAIN_HOME to an empty tmpdir in beforeAll so
loadConfig() returns null and the stub's dims win, then restore it and clean
up in afterAll. Same idiom as emptyHome() in
test/ai/gateway-probe-chat-model.test.ts.

Verified with a planted ~/.gbrain/config.json at 1280 dims: 2 pass / 4 fail
before, 6 pass / 0 fail after; still green with no config file. typecheck clean.

Fixes #1527

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 11:01:51 -07:00
Tyler Robinson 8fc93c8fac fix(health): count 'entity' pages in graph health metrics (#2639)
getHealth's entity_pages CTE and the top-linked-pages query only match the
legacy 'person' and 'company' types, so brains using the gbrain-base-v2
pack's 'entity' type report 0% entity link/timeline coverage in `gbrain
health` even when doctor's graph_coverage shows real coverage. Add 'entity'
to both queries in both engines (PGLite + Postgres, in lockstep per the
engine-parity rule) and extend the getHealth graph-metrics test with an
entity-typed page.

Validation: bun test test/pglite-engine.test.ts --test-name-pattern 'getHealth graph metrics' (5 pass).
2026-07-23 11:01:46 -07:00
spiky02plateau 3454dca0b4 perf(contextual-retrieval): bound per-chunk synopsis concurrency (#2628)
Replace the strictly sequential per-chunk synopsis loop with a bounded
sliding worker pool (existing runSlidingPool helper). Results land in
chunk order via index-addressed writes; code chunks still bypass the
wrapper; embedding remains one page-level batch after all synopses.

New knob GBRAIN_CONTEXTUAL_CHUNK_CONCURRENCY, default 4, clamped to
[1,16]; 1 reproduces the prior sequential behavior exactly. Each chunk
task still acquires/releases the global synopsis rate-lease, which
remains the cross-worker governor; the lease id now travels from
acquire to release instead of shared mutable state, and lease waits
are abort-responsive.

At 20-45s per synopsis call, a 120-chunk transcript page previously
needed 60-90+ min wall time and routinely outlived job timeouts.
2026-07-23 11:01:39 -07:00
morluto 5dcf3e7b2f fix(trajectory): stop negative metrics from inverting regression signals (#2621) 2026-07-23 11:01:35 -07:00
Garry Tan dbca701008 Revert "fix(import): fall back to body H1 for title when frontmatter lacks title: instead of slug-derived junk (#2446) (#2495)"
This reverts commit 033fd24fe8.
2026-07-23 11:01:29 -07:00
Garry Tan 4b6cf32c9f Revert "fix(cycle): extract_facts guard requires live backing page, not just non-NULL entity_slug (#2497)"
This reverts commit 53c9086945.
2026-07-23 11:01:29 -07:00
Garry Tan 0c66715f90 Revert "fix(propose_takes): memoize empty extractions so zero-claim pages don't re-spend every cycle (#2514)"
This reverts commit 1233051a20.
2026-07-23 11:01:29 -07:00
Garry Tan 55af5fc091 Revert "fix: handle <think> reasoning tags in parseExtractorOutput (#2559)"
This reverts commit 2724c3b6c9.
2026-07-23 11:01:29 -07:00
Garry Tan 10b5746053 Revert "fix(storage): Supabase signed URLs — prepend /storage/v1 (#2565)"
This reverts commit 5a295bc293.
2026-07-23 11:01:29 -07:00
Garry Tan 5bee08c3c4 Revert "fix: honor explicit list_pages limit for local callers, warn on remote clamp, thread offset (#2591)"
This reverts commit 70ffe4a2a2.
2026-07-23 11:01:29 -07:00
Garry Tan f02919c041 Revert "fix(minions): default timeout for contextual reindex (#2611)"
This reverts commit fc1f88cdcb.
2026-07-23 11:01:29 -07:00
Garry Tan 66fa5fba22 Revert "fix(migrations): let force-retry escape completed ledger entries (#2616)"
This reverts commit e79b8d5780.
2026-07-23 11:01:29 -07:00
281 changed files with 15574 additions and 1644 deletions
+1 -1
View File
@@ -28,5 +28,5 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: rhysd/actionlint@393031adb9afb225ee52ae2ccd7a5af5525e03e8 # v1.7.11
+3 -3
View File
@@ -45,7 +45,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -82,7 +82,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -116,7 +116,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
+1 -1
View File
@@ -55,7 +55,7 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
+2 -2
View File
@@ -24,7 +24,7 @@ jobs:
id-token: write # for attest-build-provenance (Sigstore OIDC)
attestations: write # for attest-build-provenance
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -49,7 +49,7 @@ jobs:
with:
path: artifacts
- name: Create release
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
with:
files: |
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
container:
image: semgrep/semgrep:1.170.0@sha256:c98f8829eea377274ee4b10656458b078b88232469b2ff913f091c2317347c9d
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
# Non-blocking initially (continue-on-error): the first runs establish a
# baseline without failing unrelated PRs. Graduation path: once the
# baseline findings are triaged (fixed or `# nosemgrep`'d), remove
+7 -7
View File
@@ -43,7 +43,7 @@ jobs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Compute content hash
id: compute
run: |
@@ -84,7 +84,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
@@ -103,7 +103,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -124,7 +124,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -149,7 +149,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -172,7 +172,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
@@ -216,7 +216,7 @@ jobs:
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
+5
View File
@@ -35,6 +35,11 @@ export/
# .context/test-shards/. Workspace-local by design — never committed.
.context/
# Local agent instruction overrides (CLAUDE.local.md / AGENTS.local.md) — personal,
# per-clone, loaded after the committed CLAUDE.md/AGENTS.md. Never committed.
CLAUDE.local.md
AGENTS.local.md
# Tier 3 PGLite snapshot fixture (built on demand by build:pglite-snapshot)
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
+138
View File
@@ -2,6 +2,144 @@
All notable changes to GBrain will be documented in this file.
## [0.42.65.0] - 2026-07-23
**A large maintenance release: 93 verified fixes and small features merged since v0.42.64.0, most of them community contributions.**
If you use gbrain day to day, this release makes the boring parts trustworthy. Importing and syncing notes is safer: a failed pull no longer pretends everything is up to date, imported pages are read back after writing to confirm they landed, and a page with real content can no longer be silently overwritten by an empty one. Search answers get better inputs: the think command now picks excerpts that actually match your question, and results respect your federated source settings. Background enrichment (the "dream" cycle) wastes less money and retries properly when an AI provider is down. Spending caps now fail closed, so a billing hiccup can never turn into an uncapped spend. And `gbrain doctor` is quieter, with several false alarms removed and real problems (like an embedding backlog with no worker running) now flagged.
More AI providers work out of the box, including OpenRouter prompt caching, MiniMax and Zhipu GLM recipes, Ollama Matryoshka embedding dimensions, and llama-server batch limits.
## To take advantage of v0.42.65.0
`gbrain upgrade` should do this automatically. No new schema migrations ship in this release.
1. **Upgrade and verify:**
```bash
gbrain upgrade
gbrain doctor
gbrain stats
```
2. **If `gbrain doctor` reports new findings after upgrading,** that is the quieter, more accurate check set working as intended. Each finding names its fix.
3. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
#### Security
- MCP source scoping for remote callers got a hardening pass, so agent-facing connections stay confined to the sources they were granted. (#2881, contributed by @spinsirr)
- Paid MCP spend accounting is now atomic and fails closed, and resolver spend is recorded before a cap error is raised, so caps cannot be raced past or undercounted. (#3203, #3204, contributed by @caterpillarC15)
- The OAuth token endpoint rate limit on the HTTP server is now configurable via env for deployments behind shared IPs. (#3114, contributed by @time-attack)
- `WWW-Authenticate` responses now carry `resource_metadata` per the MCP spec and RFC 9728, so conforming clients can discover the auth server. (#1410, contributed by @rayers)
#### Search, retrieval, and think
- `think` selects query-relevant excerpts instead of generic ones. (#3197, contributed by @Y0lan)
- Unqualified local CLI `search`/`query` now honors `sources.config.federated` read visibility. (#2561, #3141, contributed by @time-attack)
- Email citation metadata is projected into search results. (#2873, contributed by @amtagrwl)
- The `think` Gaps section renders once instead of twice. (#1662, contributed by @howwohmm)
- Fuzzy entity lookup threads the caller's source scope and skips soft-deleted entities. (#1508, contributed by @tim404x)
- `code-def` surfaces method, constructor, field, and struct definitions, not just top-level symbols. (#1628, contributed by @rayers)
- Briefing pages are excluded from their own Brain Pulse salience. (#1202, contributed by @rwbaker)
- Reranker calls with missing auth are classified as configuration errors before falling back. (#2059, #3139, contributed by @time-attack)
#### Import, sync, and ingestion
- A failed git pull with zero imports reports `partial (pull_failed)` instead of `up_to_date`. (#3068, #3253, contributed by @Masashi-Ono0611)
- Imports run a post-write read-back verification with a durable ingest-log record. (#2869, contributed by @Andredsouza1984)
- `put` refuses to overwrite a non-empty page with empty content. (#2708, contributed by @symmetric-matthew)
- `putPage` restores soft-deleted rows instead of colliding with them. (#2779, contributed by @RerankerGuo)
- Mixed-case slugs are normalized before chunk upsert, ending duplicate-chunk churn. (#430, #3143, contributed by @time-attack)
- Imports fall back to the body H1 for the title when frontmatter lacks `title:`. (#2446, #3072, contributed by @time-attack)
- YAML comments inside the frontmatter fence are no longer treated as markdown headings. (#3225, #3247, contributed by @Masashi-Ono0611)
- Write-through guards case-insensitive filesystem collisions before the atomic write. (#2831, #3119, contributed by @time-attack)
- Path-qualified wikilinks outside the known directory pattern resolve on the DB/put_page path. (#2866, contributed by @paul-0320)
- CJK slugs are supported in the slug registry and dream-cycle summary slugs. (#782, #738, #3083, contributed by @time-attack)
- Three ingest/sync/serve singleton fixes: page-type round-trip, deleted-slug embed noise, and a stateless width guard. (#3140, contributed by @time-attack)
- Sync honors the `embedding_disabled` sentinel as an implicit `--no-embed`. (#2879, contributed by @gawievanblerk)
- Verified sync head sentinels are cleared correctly. (#2734, contributed by @symmetric-matthew)
- Resumed syncs report the pinned commit they actually landed on. (#3202, contributed by @caterpillarC15)
- The expected `discover_git_root` probe failure stays off stderr. (#3232, contributed by @Masashi-Ono0611)
- `extract --stale` runs the real resolver so basename resolution reaches stale pages, and clears pre-version-bump pages. (#2576, #2717, contributed by @paul-0320; #1791, contributed by @Nazim22)
- Oversized code chunks are capped so they stay embeddable, and code-chunk metadata survives re-embeds. (#1675, contributed by @lubosxyz; #769, #1232, contributed by @rayers)
#### Background cycle, dream, and facts
- Path-derived dream sources are stamped, and the engine closes cleanly on autopilot shutdown. (#3178, contributed by @time-attack)
- All-provider-failed atom drains propagate so durable jobs retry instead of silently dropping work. (#3218, #3248, contributed by @Masashi-Ono0611)
- Atom extraction raises `maxTokens` and case-normalizes `atom_type` for Gemini models. (#3211, contributed by @alexey-metaengage)
- The conversation extractor gates anonymous-speaker self-attribution instead of guessing. (#3228, contributed by @asenkovskiy)
- Incremental dream extraction stamps its watermark so re-runs stop reprocessing. (#2636, #3115, contributed by @time-attack)
- `dream --dry-run --json` keeps stdout clean of embed summaries. (#394, #3109, contributed by @time-attack)
- Synthesized dream pages require a self-contained opening summary. (#2770, contributed by @Masashi-Ono0611)
- PGLite inline synth subagent drains complete, and `lint` gains `--exclude`. (#2699, #2649, #3162, contributed by @time-attack)
- Live context reads the documented "P1 Today" heading form with plain checkbox tasks, matching the daily-task-manager skill's output format. (#2186, #3124, contributed by @time-attack)
- Queued AI jobs refresh gateway config at execution time instead of using a stale snapshot. (#2125, contributed by @maxpetrusenkoagent)
- `brainstorm`/`propose_takes` honor configured models: cost preview uses the configured model, the judge reads its config key, provider probes are skipped when unneeded, and page projection is narrowed. (#3120, contributed by @time-attack)
- Backlog hardening wave: x-to-brain health check, propose_takes deadlines, capture title truncation, extract_atoms backlog handling, and pooler direct-URL routing. (#3165, contributed by @time-attack)
- `skillopt` emits `proposed.md` in no-mutate mode. (#2635, #3182, contributed by @time-attack)
- Nightly quality probe enable path and conversation-parser probe are wired up. (#2629, #2630, #3094, contributed by @time-attack)
#### Doctor, health, and maintenance
- New safe maintenance automation with a shared orphan-exclusion policy, so routine cleanup runs without risking linked content. (#3015, #3023, contributed by @time-attack)
- `orphan_ratio` excludes the chronicle volume under `life/events/`. (#2264, #3214, contributed by @asenkovskiy)
- `brain_score` orphan/timeline components use the orphans-audit linkable scope. (#3155, contributed by @time-attack)
- Entity timeline coverage is measured separately from whole-brain density. (#2761, contributed by @TurgutKural)
- Doctor flags embed backfills queued with no worker running. (#2696, contributed by @javieraldape)
- Two doctor false-positive/timeout fixes: the drift walk skips `node_modules`, and the bare-tweet check skips inline code and cited lines. (#1772, contributed by @sonlndv)
- A dead `llm_fallback_enabled` recommendation is dropped from conversation format coverage. (#1903, contributed by @ElliotDrel)
- Skill triggers with CRLF line endings parse on Windows. (#1149, contributed by @samporter-31)
- Onboard check names are registered in doctor categories, ending unknown-check warnings, and onboard-check remediations survive the `--apply --auto` path. (#3075, #3097, contributed by @time-attack)
- Dead slug prefixes are counted by slug. (#2697, contributed by @RerankerGuo)
- The backlinks worker defaults to check, not fix, and `check-backlinks` honors its positional directory argument. (#1853, contributed by @choomz; #3076, contributed by @time-attack)
- Calibration resolves the owner holder via config, defaulting to `self`. (#3077, contributed by @time-attack)
- Memory throttling on Linux reads `/proc/meminfo` MemAvailable. (#556, contributed by @chengzehsu)
#### AI providers and gateway
- OpenRouter gets family-scoped prompt caching, and query expansion works on chat-capable openai-compat recipes. (#3152, contributed by @time-attack)
- MiniMax recipe: embedding wire-shape compat fetch plus a chat touchpoint. (#1977, #3089, contributed by @time-attack)
- The Zhipu recipe gains a chat touchpoint so GLM subagents work. (#1157, #3084, contributed by @time-attack)
- Tier-configured models reach the recipe allowlist, Anthropic model lists are refreshed, tier resolutions are registered, and probe labels are honest. (#2800, contributed by @p3ob7o)
- Provider base URL config merges from the DB. (#1676, contributed by @TheLordArgus)
- The gateway falls back to the pooler when the derived direct host is unreachable. (#1641, #3088, contributed by @time-attack)
- Config-plane `voyage_api_key` folds into `VOYAGE_API_KEY` like the other hosted keys. (#3236, contributed by @Masashi-Ono0611)
- The `zeroentropyai:zerank-2` reranker has a pricing entry so the budget tracker can meter it. (#3223, #3233, contributed by @Masashi-Ono0611)
- llama-server embedding batches are capped at its 32-input request limit. (#1281, contributed by @mmekkaoui)
- Matryoshka dimensions thread through for Qwen3-Embedding on Ollama. (#1072, contributed by @mgandal)
- `init` seeds AI options from env on cold install, and `whoami` reports the stdio transport. (#3091, contributed by @time-attack)
- The `models` dispatch subcommand reads its first argument correctly. (#1428, contributed by @BenjaminDSmithy)
- Synopsis generation tail-truncates document text for small-model chat handlers. (#1427, contributed by @BenjaminDSmithy)
- The contradiction judge token cap is raised for thinking models. (#3210, contributed by @alexey-metaengage)
#### Schema, migrations, and storage engines
- Engine migration counts and surfaces per-page copy failures instead of silently advancing. (#3241, contributed by @Masashi-Ono0611)
- Invalid `CONCURRENTLY`-build index remnants are dropped without a DO block. (#3191, contributed by @Masashi-Ono0611)
- Unsupported large-dimension HNSW indexes are skipped instead of failing schema setup. (#1734, #3080, contributed by @time-attack)
- The v0.32.2 migration dirty-check scopes to targeted sources and surfaces failed phase detail. (#3093, contributed by @time-attack)
- Schema packs merge the full `extends` chain and `borrow_from` into the resolved manifest. (#1749, #3181, contributed by @time-attack)
- The schema-pack stats catch-all is narrowed so masked errors surface instead of fake zero-page counts. (#2466, #3133, contributed by @time-attack)
- Bundled schema-pack inspection reports the pack actually shipped in the binary, and minion subagent auth resolves through config. (#3110, contributed by @time-attack)
- PGLite `putPage` guards against zero-row RETURNING. (#1649, contributed by @alexhawkins)
#### MCP server and CLI surface
- `list_pages` rows include `source_id`. (#3209, contributed by @alexey-metaengage)
- Running CLI commands while `gbrain serve` (MCP) holds the brain now notifies about the conflict instead of failing confusingly. (#3243, contributed by @fdefitte)
- The OpenClaw plugin manifest entry is declared so the plugin loads. (#2551, #3185, contributed by @time-attack)
#### For contributors
- CI scanner roots are normalized on macOS. (#3198, contributed by @caterpillarC15)
- CI shard timeout raised to 22 minutes plus a delta-assert reporter leak test. (#3231, contributed by @time-attack)
- E2E suite hardening: flaky tests, no-op assertions, and cross-test coupling removed. (#1704, contributed by @auroracapital)
- `mechanical.test.ts` isolates `$HOME` so the E2E suite stops clobbering user config. (#434, contributed by @lloydarmbrust)
- The lint code-fence-wrap detector and fixer regex now agree. (#1597, contributed by @chungty)
- README project links for OpenClaw and Hermes are corrected. (#1961, #3179, contributed by @time-attack)
- A completed TODOS entry is dropped. (#3229, contributed by @Masashi-Ono0611)
## [0.42.64.0] - 2026-07-20
### Fixed
+12
View File
@@ -135,6 +135,18 @@ the PGLite schema. Local agents continue to use stdio (`gbrain serve`).
Running `--http` against a PGLite-backed install fails fast with a clear
error message at startup.
### Docker network isolation (self-hosted Postgres)
OAuth and source scoping enforce isolation on the `serve --http` path only.
Raw Postgres reachability bypasses both: a container that shares Docker's
default `bridge` network with the brain's Postgres can open a direct DB
session without any token and read every source. Put the brain's Postgres on
a user-defined Docker network with nothing untrusted on it, publish its port
loopback-only (if at all), and never put `DATABASE_URL` or a Postgres
password in untrusted agent containers — those should reach the brain
exclusively via OAuth against `serve --http`. Full operator checklist:
[docs/mcp/DEPLOY.md — Co-located Docker workloads](docs/mcp/DEPLOY.md#co-located-docker-workloads-self-hosted-postgres).
### CORS
Default-deny: no `Access-Control-Allow-Origin` header is sent unless an
+19 -12
View File
@@ -2,14 +2,6 @@
## community fix-wave follow-ups (filed v0.42.60.0)
- [ ] **P1 — take-writes source scoping fails open when source resolution errors (#2684 residual).**
`resolveTakesSourceId` (src/commands/takes.ts) swallows resolution errors and returns
`undefined`, which falls back to the unscoped slug-only page lookup — so an invalid
`GBRAIN_SOURCE` (or a broken dotfile chain) silently restores the pre-#2698 cross-source
write behavior on multi-source brains. Decide fail-closed semantics: error out when a
source was explicitly requested but doesn't resolve; keep the unscoped fallback only for
brains with no source configuration at all. Add a regression test for the invalid-source
path. Found by cross-model adversarial review during the v0.42.60.0 release ship.
- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent`
before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered.
@@ -2287,10 +2279,25 @@ at plan time and got carved out:
via `buildPerSourceBindings`. Document workaround: register
source-scoped OAuth clients.
- [ ] **v0.41+: T20 — extends-chain merging in registry.ts.**
`registry.ts:167` documents the gap. Implementing full child-wins
merge cascades through every consumer of `manifest.page_types`. ~1
day CC.
- [x] **v0.41+: T20 — extends-chain merging in registry.ts.** DONE (#1749).
`resolvePack` now merges parent → child (child-wins) for the six
ingest/query-shaping fields (`page_types`, `link_types`,
`frontmatter_links`, `enrichable_types`, `filing_rules`, `takes_kinds`)
plus `borrow_from` materialization, in `src/core/schema-pack/merge.ts`.
The cascade was transparent (consumers already read `resolved.manifest`),
not per-consumer. `phases`/`calibration_domains` deliberately excluded —
see the P3 follow-up below.
- [ ] **P3: explicit opt-in to inherit `phases` / `calibration_domains`.**
T20 excludes these two from the child-wins merge because they gate real
cycle execution (`cycle.ts` `packDeclaresPhase`) and the manifest
contract says each pack declares its own participation explicitly —
auto-inheriting would silently make a child run cycle phases it never
requested. Multi-level lens packs (`gbrain-everything`) therefore still
re-declare them by hand. If that redeclaration becomes painful, add an
explicit manifest flag (e.g. `inherit_phases: true`) so a pack author
opts in consciously. Depends on: T20 (landed). Start in
`src/core/schema-pack/merge.ts` (`mergeInheritedManifest`).
- [ ] **v0.41+: T21 — comment-preserving YAML emitter.**
v0.40.7.0 emitter does NOT preserve comments. Authors who care
+1 -1
View File
@@ -1 +1 @@
0.42.64.0
0.42.65.0
+10 -5
View File
@@ -51,8 +51,9 @@
"@electric-sql/pglite",
],
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"@hono/node-server": "^2.0.5",
"body-parser": "^2.3.0",
"fast-uri": "^3.1.4",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
@@ -162,7 +163,7 @@
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
"@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="],
"@hono/node-server": ["@hono/node-server@2.0.11", "", { "peerDependencies": { "hono": "^4" } }, "sha512-bjD221KPLoJTWUwso1J6fGKiTXEUFedG/s0visavY4zakFPkeGURMRNly+FhBHs7T8Dz4qHaZIMX9ZoJHSJtKA=="],
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
@@ -326,7 +327,7 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"body-parser": ["body-parser@2.2.2", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^1.0.5", "debug": "^4.4.3", "http-errors": "^2.0.0", "iconv-lite": "^0.7.0", "on-finished": "^2.4.1", "qs": "^6.14.1", "raw-body": "^3.0.1", "type-is": "^2.0.1" } }, "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA=="],
"body-parser": ["body-parser@2.3.0", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^2.0.0", "debug": "^4.4.3", "http-errors": "^2.0.1", "iconv-lite": "^0.7.2", "on-finished": "^2.4.1", "qs": "^6.15.2", "raw-body": "^3.0.2", "type-is": "^2.1.0" } }, "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw=="],
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
@@ -400,7 +401,7 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.3", "", {}, "sha512-i70LwGWUduXqzicKXWshooq+sWL1K3WUU5rKZNG/0i3a1OSoX3HqhH5WbWwTmqWfor4urUakGPiRQcleRZTwOg=="],
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
@@ -614,6 +615,10 @@
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"body-parser/content-type": ["content-type@2.0.0", "", {}, "sha512-j/O/d7GcZCyNl7/hwZAb606rzqkyvaDctLmckbxLzHvFBzTJHuGEdodATcP3yIRoDrLHkIATJuvzbFlp/ki2cQ=="],
"body-parser/type-is": ["type-is@2.1.0", "", { "dependencies": { "content-type": "^2.0.0", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA=="],
"es-set-tostringtag/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"eventsource/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
+1
View File
@@ -189,6 +189,7 @@ Unit tests and what they cover:
- `test/orphans.test.ts` — orphans command: detection, pseudo filtering, text/json/count outputs, MCP op.
- `test/postgres-engine.test.ts``statement_timeout` scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against a reintroduced bare `SET statement_timeout`.
- `test/sync.test.ts` — sync logic + regression guard asserting top-level `engine.transaction` is not called.
- `test/sync-pull-failed-anchor.serial.test.ts`#3068 regression: a failed internal `git pull` (local-path origin vs `protocol.file.allow=never`) with zero imports returns `partial`/`pull_failed` (not `up_to_date`), freezes `last_commit` + `last_sync_at`, recovers after a manual pull; fall-through import of local commits preserved. Serial: pins `GBRAIN_HOME` to a temp dir for the whole file.
- `test/sync-concurrency.test.ts``autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping; `shouldRunParallel()` explicit-bypasses-floor contract; `parseWorkers()` validation rejecting `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars.
- `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract.
- `test/sync-failures.test.ts``classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` `AcknowledgeResult` shape + backfill on legacy entries.
File diff suppressed because one or more lines are too long
+9
View File
@@ -75,6 +75,15 @@ Meta-pack stacking creator + investor + engineer via the v0.38
preserved — this IS the active pack; the registry walks extends +
borrow to materialize the merged view.
**Merge contract (T20 / #1749).** `resolvePack` merges parent → child
(child-wins) for the six ingest/query-shaping fields: `page_types`,
`link_types`, `frontmatter_links`, `enrichable_types`, `filing_rules`,
and `takes_kinds` (unioned — a child cannot narrow it). `phases` and
`calibration_domains` are **NOT** inherited: they gate cycle execution,
so each pack must declare its own participation explicitly. That is why
`gbrain-everything` re-declares all its phases and all 7
`calibration_domains` — inheritance does not carry them.
Activate via `gbrain config set schema_pack gbrain-everything` and
calibration_profile produces all 7 domain scorecards in one JSONB.
+29 -1
View File
@@ -145,7 +145,7 @@ api_version: gbrain-schema-pack-v1
name: my-pack
version: 0.0.1
gbrain_min_version: 0.39.0
extends: gbrain-base # inherits everything from base; add overrides below
extends: gbrain-base # inherits base's TYPES (see Merge contract below); add overrides
description: |
My personal pack.
@@ -170,6 +170,34 @@ enrichable_types: []
filing_rules: []
```
## Merge contract (`extends` + `borrow_from`)
`resolvePack` composes a pack against its `extends` chain (and any
`borrow_from` targets) into the `resolved.manifest` every consumer reads
(T20 / #1749). The rules:
- **Six fields inherit, child-wins:** `page_types`, `link_types`,
`frontmatter_links`, `enrichable_types`, `filing_rules`, and `takes_kinds`.
A child value with the same key (type name, link name, etc.) overrides the
parent's; keys the child doesn't declare come through from the parent.
- **`page_types` ordering:** overrides of a base type keep the base's declared
position (base's `inferType` prefix priority is authoritative); a genuinely
new type — from the child, a `borrow_from`, or a middle pack in the chain —
is prepended nearest-first, so a more-derived type's `path_prefix` wins
regardless of how deep the chain is.
- **`takes_kinds` is UNION, not replace** — it carries a Zod default, so an
omitted field is indistinguishable from an explicit one. A child can ADD
kinds but **cannot narrow** `takes_kinds` below base parent. If you need a
smaller set, don't `extends` a pack that declares the larger one.
- **`phases` and `calibration_domains` are NOT inherited** (child-only). They
gate real cycle execution, so each pack must declare its own participation
explicitly — inheriting them would silently make a child run phases it never
requested. This is why `gbrain-everything` re-declares all its phases and
calibration domains by hand. See `lens-packs.md` for the worked example.
- **`borrow_from` is selective + non-transitive + fail-closed:** it pulls only
the named `types`/`link_types` from the target's OWN declarations (omitting a
category borrows none of it); a missing target throws `UnknownPackError`.
## Recovery + revert
The single-PR cathedral is hard to revert atomically. Per codex finding
+8 -5
View File
@@ -21,14 +21,17 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
### The Primitives
+37
View File
@@ -258,6 +258,43 @@ the user owns the machine.
See [ALTERNATIVES.md](ALTERNATIVES.md) for a comparison of ngrok, Tailscale
Funnel, and cloud hosts (Fly.io, Railway).
### Co-located Docker workloads (self-hosted Postgres)
OAuth scopes and source scoping guard the `gbrain serve --http` path. They do
NOT guard raw Postgres. If the brain's Postgres runs as a container on the same
Docker host as other workloads (agent runtimes, n8n, staging fixtures), any
container sharing Docker's default `bridge` network can open a direct DB
session — no OAuth token required — and read every source. That silently
recreates a privileged path underneath the isolation you configured at the MCP
layer.
Network-zone the host so untrusted containers can never reach Postgres:
```
Docker host
├── gbrain-net ← ONLY the brain's Postgres (+ gbrain serve, if containerized)
├── agent-<id>-net ← each untrusted agent runtime, isolated
└── default bridge ← no secret-bearing databases
```
Operator checklist:
```text
[ ] Postgres is on a user-defined Docker network, not the default bridge
(or nothing else runs on that bridge)
[ ] If Postgres publishes a host port at all, it binds loopback only
(`-p 127.0.0.1:5432:5432`, never `0.0.0.0`)
[ ] Untrusted agent containers have no DATABASE_URL or Postgres password
[ ] Untrusted agents reach the brain via OAuth/Bearer against serve --http only
(host loopback via host.docker.internal / host gateway — never gbrain-net)
[ ] OAuth clients are least-privilege: scoped --source / --federated-read,
pre-minted short-lived tokens preferred over long-lived client secrets
[ ] Isolation verified: a team-scoped client cannot read internal-only sources
```
Optional defense-in-depth: a dedicated Postgres role (or RLS) limited to the
allowed `source_id`s, so even a leaked connection string can't read everything.
## Troubleshooting
**"missing_auth" error**
+15
View File
@@ -91,3 +91,18 @@ First full takes extraction run on a ~100K-page brain:
4. **Self-reported ≠ verified.** "Reports 7 figures" → holder=person, weight=0.75, NOT world/1.0
5. **No false precision.** Use 0.05 increments (0.35, 0.55, 0.75), not 0.74 or 0.82
6. **"So what" test.** Skip Twitter handles, follower counts, obvious metadata
## Owner-holder canonicalization
"The brain owner" is, by convention, the holder string **`self`** — the value the
dream `consolidate` phase stamps when it promotes the owner's hot facts into cold
takes. Calibration, `think`, and the `doctor` calibration check resolve the owner
holder through `resolveOwnerHolder` (`src/core/owner-holder.ts`): explicit override
> `emotional_weight.user_holder` config > `self`.
Known limitation (tracked in garrytan/gbrain#2465): the owner can also
appear under `brain` (a take the owner asserts, via `propose_takes`) and
`people/<owner>` (extraction that names the owner). The resolver selects the
*default* canonical owner string for reads; it does not merge those other
strings. Per-take attribution for other people (e.g. `people/george`) is
unaffected and correct.
+4
View File
@@ -484,6 +484,10 @@ Returns a per-source dashboard: when each source last synced, how many pages, ho
The admin dashboard at `https://brain.acme-co.com/admin` shows live request volume, registered OAuth clients, recent activity, and brain stats. Use the admin bootstrap token from Part 4 to log in the first time, then register additional admin users from inside the dashboard.
### If agents run as containers on the same Docker host
OAuth source scoping only guards the HTTP MCP path. If the brain's Postgres and your teammates' agent runtimes are containers on the same Docker host, make sure the agents can't reach Postgres directly over Docker's default bridge network — a direct DB session skips OAuth entirely. Put Postgres on its own user-defined network, publish it loopback-only if at all, and never hand agent containers a `DATABASE_URL`. The copy-paste operator checklist lives in [docs/mcp/DEPLOY.md — Co-located Docker workloads](../mcp/DEPLOY.md#co-located-docker-workloads-self-hosted-postgres).
---
## Part 13: Cost and speed expectations
+45 -5
View File
@@ -2720,14 +2720,17 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
### The Primitives
@@ -3902,6 +3905,43 @@ the user owns the machine.
See [ALTERNATIVES.md](ALTERNATIVES.md) for a comparison of ngrok, Tailscale
Funnel, and cloud hosts (Fly.io, Railway).
### Co-located Docker workloads (self-hosted Postgres)
OAuth scopes and source scoping guard the `gbrain serve --http` path. They do
NOT guard raw Postgres. If the brain's Postgres runs as a container on the same
Docker host as other workloads (agent runtimes, n8n, staging fixtures), any
container sharing Docker's default `bridge` network can open a direct DB
session — no OAuth token required — and read every source. That silently
recreates a privileged path underneath the isolation you configured at the MCP
layer.
Network-zone the host so untrusted containers can never reach Postgres:
```
Docker host
├── gbrain-net ← ONLY the brain's Postgres (+ gbrain serve, if containerized)
├── agent-<id>-net ← each untrusted agent runtime, isolated
└── default bridge ← no secret-bearing databases
```
Operator checklist:
```text
[ ] Postgres is on a user-defined Docker network, not the default bridge
(or nothing else runs on that bridge)
[ ] If Postgres publishes a host port at all, it binds loopback only
(`-p 127.0.0.1:5432:5432`, never `0.0.0.0`)
[ ] Untrusted agent containers have no DATABASE_URL or Postgres password
[ ] Untrusted agents reach the brain via OAuth/Bearer against serve --http only
(host loopback via host.docker.internal / host gateway — never gbrain-net)
[ ] OAuth clients are least-privilege: scoped --source / --federated-read,
pre-minted short-lived tokens preferred over long-lived client secrets
[ ] Isolation verified: a team-scoped client cannot read internal-only sources
```
Optional defense-in-depth: a dedicated Postgres role (or RLS) limited to the
allowed `source_id`s, so even a leaked connection string can't read everything.
## Troubleshooting
**"missing_auth" error**
+4 -3
View File
@@ -144,10 +144,11 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.64.0",
"version": "0.42.65.0",
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
"body-parser": "^2.3.0",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
+13 -8
View File
@@ -1,7 +1,7 @@
---
id: x-to-brain
name: X-to-Brain
version: 0.8.1
version: 0.8.2
description: Twitter timeline, mentions, and keyword monitoring flow into brain pages. Tracks deletions, engagement velocity, OCR on images, and real-time alerts.
category: sense
requires: []
@@ -9,9 +9,12 @@ secrets:
- name: X_BEARER_TOKEN
description: X API v2 Bearer token (Basic tier minimum, $200/mo for full archive search)
where: https://developer.x.com/en/portal/dashboard — create a project + app, copy the Bearer Token from "Keys and tokens"
- name: X_HANDLE
description: Your X username without the @ (used for the app-only health check — /users/me requires user-context OAuth, which app-only bearer tokens don't have)
where: Your X profile — the handle in your profile URL, e.g. x.com/yourhandle → yourhandle
health_checks:
- type: http
url: "https://api.x.com/2/users/me"
url: "https://api.x.com/2/users/by/username/$X_HANDLE"
auth: bearer
auth_token: "$X_BEARER_TOKEN"
label: "X API"
@@ -110,15 +113,17 @@ Tell the user:
4. Inside the project, create a new App
5. Go to the app's 'Keys and tokens' tab
6. Under 'Bearer Token', click 'Generate' (or 'Regenerate')
7. Copy the Bearer Token and paste it to me
7. Copy the Bearer Token and paste it to me, along with your X handle (without the @)
Note: Free tier gives read-only access with low limits. Basic tier ($200/mo)
gives search/recent endpoint and higher limits. Pro tier gets full archive search."
Validate immediately:
Set both `X_BEARER_TOKEN` and `X_HANDLE` in the environment. Validate immediately
(app-only bearer tokens cannot call `/users/me` — that endpoint requires
user-context OAuth — so validation uses the by-username lookup):
```bash
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
"https://api.x.com/2/users/me" \
"https://api.x.com/2/users/by/username/$X_HANDLE" \
&& echo "PASS: X API connected" \
|| echo "FAIL: X API token invalid"
```
@@ -134,10 +139,10 @@ starting with 'AAA...', (3) if you just created the app, the token is valid imme
```bash
# Look up the user's X user ID from their handle
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/USERNAME" | grep -o '"id":"[^"]*"'
"https://api.x.com/2/users/by/username/$X_HANDLE" | grep -o '"id":"[^"]*"'
```
Ask the user for their X handle (e.g., @yourhandle). Look up their user ID.
Look up the user ID from the handle collected in Step 1.
Save it — the collector needs the numeric ID, not the handle.
### Step 3: Configure the Collector
@@ -205,7 +210,7 @@ The agent should review collected data 2-3x daily and run enrichment.
```bash
mkdir -p ~/.gbrain/integrations/x-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.1","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.8.2","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
```
## Production Patterns (v0.8.1)
+1 -1
View File
@@ -70,7 +70,7 @@ PATTERN='import[[:space:]]+(\*[[:space:]]+as[[:space:]]+[a-zA-Z_$][a-zA-Z0-9_$]*
FOUND_FILES=""
while IFS= read -r f; do
[ -n "$f" ] && FOUND_FILES="$FOUND_FILES$f"$'\n'
done < <(grep -rlE --include='*.ts' "$PATTERN" src/ 2>/dev/null | sort -u || true)
done < <(grep -rlE --include='*.ts' "$PATTERN" src 2>/dev/null | sort -u || true)
FAIL=0
+2 -2
View File
@@ -100,9 +100,9 @@ IFS='|' eval 'PATTERN="${PATTERN_PARTS[*]}"'
# Find tool.
if command -v rg >/dev/null 2>&1; then
matches="$(rg -niH --no-heading -t ts "$PATTERN" test/ 2>/dev/null || true)"
matches="$(rg -niH --no-heading -t ts "$PATTERN" test 2>/dev/null || true)"
elif command -v grep >/dev/null 2>&1; then
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test/ 2>/dev/null || true)"
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test 2>/dev/null || true)"
else
echo "check-test-real-names: ERROR: neither rg nor grep available." >&2
exit 2
+12 -1
View File
@@ -133,6 +133,7 @@ for i in $(seq 1 "$N"); do
env SHARD="$i/$N" \
bash scripts/run-unit-shard.sh --max-concurrency="$INTRA_CONC" \
> "$SHARD_LOG" 2>&1
rc=$?
else
env SHARD="$i/$N" \
bash scripts/run-unit-shard.sh --max-concurrency="$INTRA_CONC" \
@@ -142,10 +143,20 @@ for i in $(seq 1 "$N"); do
sleep 5 && kill -KILL "$pid" 2>/dev/null ) &
cap_pid=$!
wait "$pid" 2>/dev/null
# Capture the shard's exit code from ITS `wait`, before any watchdog
# teardown runs. The teardown commands below overwrite $? — the killed
# watchdog reports 143 — which used to get stamped into every shard's
# sentinel on machines with no gtimeout/timeout: every run "failed"
# with rc=143 summaries even when all tests passed.
rc=$?
# Reap the watchdog's `sleep` child too (pkill -P), then the watchdog.
# Killing only the subshell leaves the sleep orphaned until
# $SHARD_TIMEOUT elapses — same quirk the heartbeat cleanup below works
# around; CI's orphan-process sweep flags those.
pkill -P "$cap_pid" 2>/dev/null
kill "$cap_pid" 2>/dev/null
wait "$cap_pid" 2>/dev/null
fi
rc=$?
echo "$rc" > "$LOG_DIR/shard-$i.exit"
[ "$rc" = "124" ] && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
) &
+12 -1
View File
@@ -126,6 +126,7 @@ for c in "${CHECKS[@]}"; do
(
if [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" "${TIMEOUT}s" bun run "$c" > "$LOG_FILE" 2>&1
rc=$?
else
bun run "$c" > "$LOG_FILE" 2>&1 &
pid=$!
@@ -133,10 +134,20 @@ for c in "${CHECKS[@]}"; do
sleep 5 && kill -KILL "$pid" 2>/dev/null ) &
cap_pid=$!
wait "$pid" 2>/dev/null
# Capture the check's exit code from ITS `wait`, before any watchdog
# teardown runs. The teardown commands below overwrite $? — the killed
# watchdog reports 143 — which used to get stamped into every sentinel
# on machines with no gtimeout/timeout: verify reported pass=0
# fail=<all> while every per-check log said OK.
rc=$?
# Reap the watchdog's `sleep` child too (pkill -P), then the watchdog.
# Killing only the subshell leaves the sleep orphaned until $TIMEOUT
# elapses — same quirk the heartbeat cleanup in run-unit-parallel.sh
# works around; CI's orphan-process sweep flags those.
pkill -P "$cap_pid" 2>/dev/null
kill "$cap_pid" 2>/dev/null
wait "$cap_pid" 2>/dev/null
fi
rc=$?
echo "$rc" > "$EXIT_FILE"
) &
PIDS+=($!)
+1 -1
View File
@@ -62,7 +62,7 @@ gbrain capture "..." --json # structured output for agents
- **Slug:** `inbox/YYYY-MM-DD-<hash8>` (stable for same content; the daemon's 24h dedup catches re-captures).
- **Type:** `note` (override with `--type idea` etc.).
- **Frontmatter stamps:** `captured_via: capture-cli`, `captured_at: <ISO>`.
- **Title:** first non-empty line of the body, capped at 80 chars.
- **Title:** first non-empty line of the body, capped at 80 chars (truncation appends `…`).
## Output Format
+79 -3
View File
@@ -24,6 +24,7 @@ import type { GBrainConfig } from './core/config.ts';
import type { AIGatewayConfig } from './core/ai/types.ts';
import type { BrainEngine } from './core/engine.ts';
import { operations, OperationError } from './core/operations.ts';
import { resolveSourceIdEngineFree } from './core/source-resolver.ts';
import { formatVolunteeredPage } from './core/context/volunteer.ts';
import type { Operation, OperationContext } from './core/operations.ts';
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
@@ -384,6 +385,15 @@ async function main() {
if (op.localOnly) {
refuseThinClient(command, cfgPre!.remote_mcp!.mcp_url);
}
// #2098: the local path resolves --source / GBRAIN_SOURCE / .gbrain-source
// inside makeContext (ctx.sourceId), which this route never reaches — so
// scope must be mapped onto the op's source_id wire param before the call.
try {
applyThinClientSourceScope(op, params);
} catch (e: unknown) {
console.error(e instanceof Error ? e.message : String(e));
process.exit(1);
}
await runThinClientRouted(op, params, cfgPre!, cliOpts);
return;
}
@@ -804,18 +814,80 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
return params;
}
/**
* #2098: thin-client source scoping. Locally, --source / GBRAIN_SOURCE /
* .gbrain-source resolve to ctx.sourceId in makeContext; the thin-client
* route short-circuits before that, so `gbrain query --source X` against a
* remote brain silently searched unscoped. This runs the engine-free tiers
* (flag env dotfile; the DB-backed tiers can't run without an engine
* the server's grant scoping covers the rest) and maps the result onto the
* op's `source_id` wire param.
*
* Ops that declare their OWN `source` param (facts add, etc.) are left
* untouched their --source is an op param, not scope. An explicit --source
* on an op with no source_id wire param throws (loud beats silent drop);
* ambient env/dotfile scope with nowhere to send it is ignored, matching the
* pre-fix behavior for non-scopeable ops. Exported for tests.
*/
// Ops whose `source_id` wire param is NOT read-scope semantics: get_skill's
// source_id flips the lookup from host catalog to brain-resident-pack
// (getResidentSkillDetail). Ambient env/dotfile scope must never leak into
// these; an explicit --source-id still passes through untouched above.
const NON_SCOPE_SOURCE_ID_OPS = new Set(['get_skill']);
export function applyThinClientSourceScope(
op: Operation,
params: Record<string, unknown>,
cwd?: string,
): void {
if ('source' in op.params) return; // the op owns --source; not a scope flag
const explicit = typeof params.source === 'string' && params.source.length > 0
? (params.source as string)
: null;
delete params.source; // never a wire param on these ops — don't leak it
// Explicit per-call scope already on the wire wins over ambient tiers.
if (params.source_id !== undefined || params.all_sources === true) {
if (explicit) {
throw new Error('Pass either --source or --source-id/--all-sources, not both.');
}
return;
}
const resolved = resolveSourceIdEngineFree(explicit, cwd);
if (!resolved) return;
if (!('source_id' in op.params) || NON_SCOPE_SOURCE_ID_OPS.has(op.name)) {
if (explicit) {
const hint = NON_SCOPE_SOURCE_ID_OPS.has(op.name)
? `(its source_id parameter is not a scope filter; pass --source-id explicitly if you mean it)`
: `(the remote op has no source_id parameter; the server scopes it to your grant)`;
throw new Error(
`gbrain ${op.cliHints?.name || op.name} does not accept --source on a thin-client install ${hint}.`,
);
}
return; // ambient env/dotfile scope with nowhere to send it
}
params.source_id = resolved;
}
async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
// 'default'. Wrapped in try/catch so a doctor / single-source brain that
// never set up sources still returns 'default' silently.
let sourceId: string | undefined;
// #2561: when the source resolved via a NON-explicit tier (path-match /
// brain default / sole-non-default / seed default), unqualified search-shaped
// reads span every `config.federated = true` source. Computed here (the
// trusted local boundary) and consumed by federatedSearchScope in
// operations.ts, which additionally gates on ctx.remote === false.
let localFederated: string[] | undefined;
try {
const { resolveSourceId } = await import('./core/source-resolver.ts');
const { resolveSourceWithTier, localFederatedSourceIds } = await import('./core/source-resolver.ts');
// params.source is set when a CLI flag was parsed for the op (rare; most
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
const explicit = (params.source as string | undefined) ?? null;
sourceId = await resolveSourceId(engine, explicit);
const resolved = await resolveSourceWithTier(engine, explicit);
sourceId = resolved.source_id;
localFederated = await localFederatedSourceIds(engine, resolved.source_id, resolved.tier);
} catch {
// Source resolution failed (e.g. sources table doesn't exist on a fresh
// pre-init brain). Leave sourceId unset; engine read methods fall through
@@ -836,6 +908,7 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
// table). Matches dispatch.ts's auto-fill so the contract holds across
// every transport.
sourceId: sourceId ?? 'default',
...(localFederated ? { localFederatedSourceIds: localFederated } : {}),
};
}
@@ -937,7 +1010,10 @@ export function formatResult(opName: string, result: unknown): string {
lines.push(`Link coverage (entities): ${(h.link_coverage * 100).toFixed(1)}%`);
}
if (h.timeline_coverage !== undefined) {
lines.push(`Timeline coverage (entities): ${(h.timeline_coverage * 100).toFixed(1)}%`);
lines.push(`Timeline coverage (entity pages): ${(h.timeline_coverage * 100).toFixed(1)}%`);
}
if (h.timeline_coverage_score !== undefined) {
lines.push(`Timeline density (all pages): ${h.timeline_coverage_score}/15 (whole-brain brain-score component)`);
}
if (Array.isArray(h.most_connected) && h.most_connected.length > 0) {
lines.push('Most connected entities:');
+7
View File
@@ -439,6 +439,13 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
const result = await m.orchestrator(orchestratorOptsFrom(cli));
if (result.status === 'failed') {
console.error(`Migration v${m.version} reported status=failed.`);
// Surface each failed phase's detail — the ledger records it, but
// the operator needs it on stderr to act (#921).
for (const p of result.phases) {
if (p.status === 'failed') {
console.error(` phase ${p.name}: ${p.detail ?? '(no detail)'}`);
}
}
// Record the attempt as 'partial' (not 'complete') so the cap counts
// it. Don't let a failed orchestrator look like it never ran.
try {
+60
View File
@@ -515,6 +515,60 @@ async function registerClient(name: string, args: string[]) {
}
}
/**
* v0.42.x (#1914): rescope an existing OAuth client's write source and/or
* federated read scope. This is the operator surface the DCR registration
* comment promised ("rescope via the CLI later") DCR clients land with
* source_id='default' / federated_read=['default'] and must not self-widen,
* so widening happens here (trusted local CLI) or via the requireAdmin
* /admin/api/rescope-client endpoint.
*/
async function rescopeClient(clientId: string, args: string[]) {
const usage = 'Usage: auth rescope-client <client_id> [--source SOURCE] [--federated-read SRC1,SRC2,...]';
if (!clientId) {
console.error(usage);
process.exit(1);
}
let sourceId: string | undefined;
let federatedRead: string[] | undefined;
for (let i = 0; i < args.length; i += 2) {
const flag = args[i];
const value = args[i + 1];
if (value === undefined || value.startsWith('--')) {
console.error(`Error: ${flag} requires a value`);
console.error(usage);
process.exit(1);
}
if (flag === '--source') sourceId = value;
else if (flag === '--federated-read') {
federatedRead = value.split(',').map(s => s.trim()).filter(Boolean);
} else {
console.error(`Error: Unknown flag: ${flag}`);
console.error(usage);
process.exit(1);
}
}
if (sourceId === undefined && federatedRead === undefined) {
console.error('Error: pass --source and/or --federated-read');
console.error(usage);
process.exit(1);
}
try {
await withConfiguredSql(async (sql) => {
const { GBrainOAuthProvider } = await import('../core/oauth-provider.ts');
const provider = new GBrainOAuthProvider({ sql });
const result = await provider.rescopeClient(clientId, { sourceId, federatedRead });
console.log(`OAuth client rescoped: "${result.clientName}" (${result.clientId})\n`);
console.log(` Write source: ${result.sourceId}`);
console.log(` Federated reads: ${result.federatedRead.join(', ') || '<none>'}`);
console.log('\nTakes effect on the client\'s next request (existing tokens included).');
});
} catch (e: any) {
console.error('Error:', e.message);
process.exit(1);
}
}
/**
* Entry point for the `gbrain auth` CLI subcommand. Also reused by the
* direct-script path (see bottom of file) so `bun run src/commands/auth.ts`
@@ -556,6 +610,7 @@ export async function runAuth(args: string[]): Promise<void> {
return;
}
case 'register-client': await registerClient(rest[0], rest.slice(1)); return;
case 'rescope-client': await rescopeClient(rest[0], rest.slice(1)); return;
case 'revoke-client': await revokeClient(rest[0]); return;
case 'test': {
const tokenIdx = rest.indexOf('--token');
@@ -593,6 +648,11 @@ Usage:
--bound-slug-prefixes <prefix1,prefix2> Bind submit_agent writes to slug prefixes
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
gbrain auth rescope-client <client_id> [options] Change an existing client's source scope (e.g. a DCR
client stuck on the 'default' source). Only the flags
you pass change; the other axis is left as-is.
--source <id> New write source
--federated-read <id1,id2,...> New read-scope source list
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
`);
+140 -8
View File
@@ -38,6 +38,7 @@ import { logSelfUpgrade } from '../core/audit/self-upgrade-audit.ts';
import { detectInstallMethod } from './upgrade.ts';
import { evaluateQuietHours } from '../core/minions/quiet-hours.ts';
import { inspectLock } from '../core/db-lock.ts';
import { registerCleanup } from '../core/process-cleanup.ts';
/**
* v0.37.7.0 #1162 classify autopilot reconnect-loop errors.
@@ -433,6 +434,37 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
let stopping = false;
let childSupervisor: ChildWorkerSupervisor | null = null;
// #1872: graceful engine shutdown. On PGLite the cycle steps run INLINE in
// this process, so a hard `process.exit` mid-write (systemctl stop →
// SIGTERM) kills WASM Postgres with the WAL dirty and can corrupt the
// brain. Two exit paths must both close the engine:
// - autopilot's own shutdown() below (owns SIGINT + internal stops like
// max_crashes / cycle-failure-cap), and
// - process-cleanup's SIGTERM handler (installed at cli.ts module load;
// it runs the cleanup registry with a 3s deadline and then exits) —
// which is why closeEngine is ALSO registered there.
// closeEngine aborts the in-flight inline cycle (runCycle checks the
// signal between phases and threads it into phase sub-work), gives it a
// short bounded window to wind down, then disconnects. PGLite's
// disconnect() drains the pending query and checkpoints before closing;
// a second call is a no-op (disconnect snapshots + nulls the handle), so
// both paths firing is safe.
const shutdownAbort = new AbortController();
let inflightInlineCycle: Promise<unknown> | null = null;
const closeEngine = async () => {
shutdownAbort.abort(new Error('autopilot shutdown'));
if (inflightInlineCycle) {
// ponytail: 2s cap keeps us inside process-cleanup's 3s deadline; a
// between-phase abort resolves instantly, a mid-phase one may not.
await Promise.race([
inflightInlineCycle.catch(() => { /* cycle errors already logged by the loop */ }),
new Promise((r) => setTimeout(r, 2_000)),
]);
}
try { await engine.disconnect(); } catch { /* best-effort */ }
};
const deregisterEngineClose = registerCleanup('autopilot-engine-close', closeEngine);
if (spawnManagedWorker) {
const cliPath = resolveGbrainCliPath();
// Cgroup-aware auto-sized RSS watchdog cap (issue #1678). The old flat
@@ -520,6 +552,10 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
childSupervisor.killChild('SIGKILL');
}
}
// #1872: abort the in-flight inline cycle and close the engine BEFORE
// process.exit — a hard exit mid-write corrupts PGLite's WASM Postgres.
await closeEngine();
deregisterEngineClose();
try { unlinkSync(lockPath); } catch { /* already gone */ }
process.exit(0);
};
@@ -527,6 +563,9 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
process.on('SIGINT', () => { void shutdown('SIGINT'); });
let consecutiveErrors = 0;
// Parser-probe fixture warning is once-per-process, not once-per-cycle
// (compiled-binary installs have no source tree; don't spam the log).
let parserProbeFixtureWarned = false;
// v0.37.7.0 #1162 — counter for consecutive reconnect failures.
// Reset on every successful health probe or reconnect. Threshold
// controlled by GBRAIN_AUTOPILOT_MAX_RECONNECT_FAILS env (default 30).
@@ -825,7 +864,10 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
{
queue: 'default',
idempotency_key: idemKey,
max_attempts: 1,
// issue #3218: the handler now throws on an
// all-provider-failed batch, so give the queue's
// backoff a chance (was 1 — dead-lettered instantly).
max_attempts: 3,
timeout_ms: timeoutMs,
},
{ allowProtectedSubmit: true },
@@ -865,9 +907,19 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
} catch {
embeddingModel = (await engine.getConfig('embedding_model')) ?? undefined;
}
const embedKeyCfg: Record<string, string | null> = {};
// #2662 (codex round-3): HOSTED_EMBED_KEY_CONFIG entries are keys
// buildGatewayConfig folds from the FILE plane only — `gbrain config
// set <key> X` writes the DB plane, which never reaches the gateway
// for these fields. Reading via engine.getConfig() here (DB plane)
// would report a provider "configured" from a DB-only key that the
// gateway can never actually use, dispatching a doomed embed job.
// Read the same file-plane source context.ts (doctor) reads instead,
// so autopilot and doctor agree with what the gateway can see.
const { loadConfigFileOnly } = await import('../core/config.ts');
const fileCfg = loadConfigFileOnly() as Record<string, unknown> | null;
const embedKeyCfg: Record<string, unknown> = {};
for (const field of Object.values(HOSTED_EMBED_KEY_CONFIG)) {
embedKeyCfg[field] = await engine.getConfig(field);
embedKeyCfg[field] = fileCfg?.[field];
}
const ctx = {
repoPath,
@@ -1008,16 +1060,21 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// path's phase set). Now both converge on the same primitive.
try {
const { runCycle } = await import('../core/cycle.ts');
const report = await runCycle(engine, {
// #1872: track the promise so closeEngine can drain it on shutdown,
// and pass the abort signal so the cycle winds down between phases.
const cyclePromise = runCycle(engine, {
brainDir: repoPath,
// Autopilot daemon path: pulls by default (matches
// pre-v0.17 autopilot behavior). CLI dream defaults false
// for cron safety; that choice is scoped to dream only.
pull: true,
signal: shutdownAbort.signal,
yieldBetweenPhases: async () => {
await new Promise(r => setImmediate(r));
},
});
inflightInlineCycle = cyclePromise;
const report = await cyclePromise.finally(() => { inflightInlineCycle = null; });
// Only 'failed' (every attempted phase failed) trips the autopilot
// circuit breaker. 'partial' means at least one phase warned or
// failed while others ran — that's a soft signal, not a fatal
@@ -1073,17 +1130,36 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// loop. Probe runs even when cycleOk=false (probe may surface signal
// explaining why the cycle is failing).
try {
const probeEnabled = cfg?.autopilot?.nightly_quality_probe?.enabled === true;
const { resolveProbeEnabled, resolveProbeMaxUsd, runNightlyQualityProbe } = await import('../core/cycle/nightly-quality-probe.ts');
// Dual-plane read: `gbrain config set` (what the doctor enable hint
// prints) writes the DB plane; ~/.gbrain/config.json is the fallback.
let dbEnabled: string | null = null;
let dbMaxUsd: string | null = null;
try {
dbEnabled = await engine.getConfig('autopilot.nightly_quality_probe.enabled');
dbMaxUsd = await engine.getConfig('autopilot.nightly_quality_probe.max_usd');
} catch { /* DB unavailable → file plane only */ }
const probeEnabled = resolveProbeEnabled(dbEnabled, cfg?.autopilot?.nightly_quality_probe?.enabled);
if (probeEnabled) {
const { runNightlyQualityProbe } = await import('../core/cycle/nightly-quality-probe.ts');
const { runLongMemEvalForProbe, runCrossModalBatchForProbe } = await import('../core/cycle/nightly-probe-adapters.ts');
const { isAvailable } = await import('../core/ai/gateway.ts');
const maxUsd = Number(cfg?.autopilot?.nightly_quality_probe?.max_usd ?? 5);
const { existsSync } = await import('node:fs');
const { fileURLToPath } = await import('node:url');
const { join } = await import('node:path');
const maxUsd = resolveProbeMaxUsd(dbMaxUsd, cfg?.autopilot?.nightly_quality_probe?.max_usd);
// The committed fixture (test/fixtures/longmemeval-nightly.jsonl)
// lives in the gbrain PACKAGE, not the brain repo — repoPath is
// sync.repo_path (the user's brain), where the fixture never
// exists, so the probe error'd on every real install. Resolve the
// package root from the module location; keep repoPath as the
// fallback for setups that vendor the fixture into the brain repo.
const pkgRoot = fileURLToPath(new URL('../..', import.meta.url));
const fixtureAtPkgRoot = existsSync(join(pkgRoot, 'test', 'fixtures', 'longmemeval-nightly.jsonl'));
await runNightlyQualityProbe({
isEnabled: () => true, // already gated above; phase re-checks for defense-in-depth
hasEmbeddingProvider: () => isAvailable('embedding'),
resolveMaxUsd: () => maxUsd,
resolveRepoRoot: () => repoPath ?? gbrainHomePath('.'),
resolveRepoRoot: () => (fixtureAtPkgRoot ? pkgRoot : repoPath ?? gbrainHomePath('.')),
runLongMemEval: runLongMemEvalForProbe,
runCrossModalBatch: runCrossModalBatchForProbe,
now: () => new Date(),
@@ -1095,6 +1171,62 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// informational; autopilot loop continues.
}
// 4.6 — Nightly conversation-parser probe (v0.41.16.0 phase module;
// the scheduler wire-up was deferred at ship and is added here). Same
// posture as 4.5: the phase owns its gates (enabled/mode-gate, LLM
// key), the wiring owns invocation + the audit row, and a probe
// failure NEVER crashes the autopilot loop. Per D10 the probe is
// default-ON for search.mode=tokenmax, opt-in otherwise.
try {
const { runConversationParserNightlyProbe } = await import('../core/conversation-parser/nightly-probe.ts');
const { logParserProbeEvent, parserProbeRanWithin } = await import('../core/audit-parser-probe.ts');
const { isAvailable } = await import('../core/ai/gateway.ts');
const { existsSync } = await import('node:fs');
const { fileURLToPath } = await import('node:url');
const { join } = await import('node:path');
// Flag reads dual-plane: the DB row (`gbrain config set …`) wins,
// ~/.gbrain/config.json is the fallback. search.mode lives on the
// DB plane only (mode.ts owns it).
let parserDbEnabled: string | null = null;
let dbSearchMode: string | null = null;
try {
parserDbEnabled = await engine.getConfig('autopilot.conversation_parser_probe.enabled');
dbSearchMode = await engine.getConfig('search.mode');
} catch { /* DB unavailable → file plane only */ }
const parserEnabled = parserDbEnabled != null
? parserDbEnabled === 'true'
: cfg?.autopilot?.conversation_parser_probe?.enabled === true;
const searchMode = dbSearchMode ?? '';
// Fixtures are committed in the gbrain package (test/fixtures/…),
// NOT the brain repo — resolve from the module location. Compiled
// binaries carry no source tree: skip quietly instead of writing
// failure rows that would flip doctor to WARN on every binary install.
const pkgRoot = fileURLToPath(new URL('../..', import.meta.url));
const fixturePath = join(pkgRoot, 'test', 'fixtures', 'conversation-formats', 'all.jsonl');
const adversarialPath = join(pkgRoot, 'test', 'fixtures', 'conversation-formats', 'adversarial.jsonl');
const shouldInvoke = parserEnabled || searchMode === 'tokenmax';
if (shouldInvoke && existsSync(fixturePath) && existsSync(adversarialPath)) {
const result = await runConversationParserNightlyProbe({
isEnabled: () => parserEnabled,
searchMode: () => searchMode,
hasLlmKey: () => isAvailable('chat'),
resolveFixturePath: () => fixturePath,
resolveAdversarialPath: () => adversarialPath,
now: () => new Date(),
shouldSkipForRateLimit: () => parserProbeRanWithin(24 * 60 * 60 * 1000),
});
// rate_limited is a non-run: the loop ticks every few minutes, so
// logging every skip would flood the audit file with no-signal rows.
if (result.outcome !== 'rate_limited') logParserProbeEvent(result);
} else if (shouldInvoke && !parserProbeFixtureWarned) {
parserProbeFixtureWarned = true;
console.error(`[parser-probe] fixtures not found under ${pkgRoot}; skipping (probe needs a source-checkout install)`);
}
} catch (e) {
logError('autopilot.parser_probe', e);
// Informational, like 4.5: do NOT bump consecutiveErrors.
}
// Wait for next cycle
await new Promise(r => setTimeout(r, interval * 1000));
}
+40 -8
View File
@@ -5,8 +5,8 @@
* checks if back-links exist, and optionally creates them.
*
* Usage:
* gbrain check-backlinks check [--dir <brain-dir>] # report missing back-links
* gbrain check-backlinks fix [--dir <brain-dir>] # create missing back-links
* gbrain check-backlinks check [dir] [--dir <brain-dir>] # report missing back-links
* gbrain check-backlinks fix [dir] [--dir <brain-dir>] # create missing back-links
* gbrain check-backlinks fix --dry-run # preview fixes
*/
@@ -201,6 +201,40 @@ export interface BacklinksResult {
dryRun: boolean;
}
export interface ParsedBacklinksArgs {
subcommand: string | undefined;
brainDir: string;
dryRun: boolean;
}
export function parseBacklinksArgs(args: string[]): ParsedBacklinksArgs {
const subcommand = args[0];
const dryRun = args.includes('--dry-run');
const dirIdx = args.indexOf('--dir');
const flagDir = dirIdx >= 0 && args[dirIdx + 1] && !args[dirIdx + 1].startsWith('--')
? args[dirIdx + 1]
: undefined;
let positionalDir: string | undefined;
for (let i = 1; i < args.length; i++) {
const arg = args[i];
if (arg === '--dir') {
i++;
continue;
}
if (arg === '--dry-run') continue;
if (arg.startsWith('--')) continue;
positionalDir = arg;
break;
}
return {
subcommand,
brainDir: flagDir ?? positionalDir ?? '.',
dryRun,
};
}
/**
* Library-level backlinks check/fix. Throws on validation errors; returns a
* structured result so Minions handlers + autopilot-cycle can surface counts.
@@ -236,16 +270,14 @@ export async function runBacklinksCore(opts: BacklinksOpts): Promise<BacklinksRe
}
export async function runBacklinks(args: string[]) {
const subcommand = args[0];
const dirIdx = args.indexOf('--dir');
const brainDir = dirIdx >= 0 ? args[dirIdx + 1] : '.';
const dryRun = args.includes('--dry-run');
const { subcommand, brainDir, dryRun } = parseBacklinksArgs(args);
if (!subcommand || !['check', 'fix'].includes(subcommand)) {
console.error('Usage: gbrain check-backlinks <check|fix> [--dir <brain-dir>] [--dry-run]');
console.error('Usage: gbrain check-backlinks <check|fix> [dir] [--dir <brain-dir>] [--dry-run]');
console.error(' check Report missing back-links');
console.error(' fix Create missing back-links (appends to Timeline)');
console.error(' --dir Brain directory (default: current directory)');
console.error(' dir Brain directory (default: current directory)');
console.error(' --dir Brain directory override');
console.error(' --dry-run Preview fixes without writing');
process.exit(1);
}
+10 -3
View File
@@ -23,6 +23,7 @@ import { runPhaseCalibrationProfile } from '../core/cycle/calibration-profile.ts
import { sourceScopeOpts, type OperationContext } from '../core/operations.ts';
import type { GBrainConfig } from '../core/config.ts';
import { GBrainError } from '../core/types.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
export interface CalibrationProfileRow {
/** BIGSERIAL string (postgres.js int8 wire shape; never Number() int8
@@ -167,7 +168,10 @@ export async function runCalibration(
config: GBrainConfig,
): Promise<void> {
const { opts } = parseArgs(args);
const holder = opts.holder ?? 'garry';
const holder = resolveOwnerHolder({
override: opts.holder,
configValue: await engine.getConfig('emotional_weight.user_holder'),
});
// Resolve --source / GBRAIN_SOURCE / .gbrain-source so the (now reachable, #2035)
// calibration command targets the right source in a multi-source brain instead
// of always reading `default`. No signal → 'default' (prior behavior).
@@ -253,12 +257,15 @@ export async function getCalibrationProfileOp(
ctx: OperationContext,
params: { holder?: string },
): Promise<CalibrationProfileRow | null> {
const holder = params.holder ?? 'garry';
const holder = resolveOwnerHolder({
override: params.holder,
configValue: await ctx.engine.getConfig('emotional_weight.user_holder'),
});
if (typeof holder !== 'string' || holder.length === 0) {
throw new GBrainError(
'INVALID_HOLDER',
'get_calibration_profile.holder must be a non-empty string',
'pass holder="<slug>" or omit to default to "garry"',
'pass holder="<slug>" or omit to default to the owner holder (config emotional_weight.user_holder, else "self")',
);
}
const scope = sourceScopeOpts(ctx);
+6 -2
View File
@@ -233,14 +233,18 @@ export function maybeRewriteSourceFkError(err: unknown, sourceId: string | undef
/**
* Derive a title from the first non-empty, non-`---` line of the body,
* stripping leading markdown heading marks, capped at 80 chars.
* stripping leading markdown heading marks, capped at 80 chars. Truncation
* is codepoint-aware (never splits an astral surrogate pair) and appends an
* ellipsis so a cut title is visibly cut.
* Falls back to 'Capture' when no usable line exists.
*/
function deriveTitle(rawBody: string): string {
const firstLine = rawBody
.split('\n')
.find((l) => l.trim().length > 0 && l.trim() !== '---') ?? '';
return firstLine.replace(/^#+\s*/, '').slice(0, 80) || 'Capture';
const stripped = firstLine.replace(/^#+\s*/, '');
const cps = [...stripped];
return (cps.length > 80 ? cps.slice(0, 79).join('') + '…' : stripped) || 'Capture';
}
/**
+395 -204
View File
@@ -28,6 +28,7 @@ import type { DbUrlSource } from '../core/config.ts';
import { gbrainPath, loadConfig } from '../core/config.ts';
import { reflexEnabled } from '../core/context/reflex.ts';
import { resolveSocketPath } from '../core/context/resolve-ipc.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
import { homedir } from 'os';
import { dirname, isAbsolute, join, resolve as resolvePath } from 'path';
import { fileURLToPath } from 'url';
@@ -528,6 +529,61 @@ export async function childTableOrphansCheck(engine: BrainEngine): Promise<Check
};
}
/**
* Raw-source persistence guarantee (#1978, warn-only v1).
*
* Invariant: every synthesized/derived page (dream_generated:true frontmatter
* or type:synthesis) must either carry a raw trace or declare an explicit
* exemption. Accepted traces:
* - frontmatter key `raw_trace` / `raw_source` / `source_uri`
* - an attached `raw_data` row
* - `synthesis_evidence` rows (think-op citations)
* - explicit `raw_trace_exempt: true` (reason in `raw_trace_exempt_reason`)
*
* v1 is deliberately warn-only no write path is blocked. Escalation to
* fail-closed enforcement in the synthesis/import write paths is the v2
* follow-up once real brains run clean.
*
* Pure helper (engine.executeRaw only) for parity with
* childTableOrphansCheck so tests can target it directly.
*/
export async function rawProvenanceCheck(engine: BrainEngine): Promise<Check> {
const where = `
p.deleted_at IS NULL
AND (COALESCE(p.frontmatter->>'dream_generated', '') = 'true' OR p.type = 'synthesis')
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ?| ARRAY['raw_trace', 'raw_source', 'source_uri', 'raw_trace_exempt'])
AND NOT EXISTS (SELECT 1 FROM raw_data rd WHERE rd.page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM synthesis_evidence se WHERE se.synthesis_page_id = p.id)`;
try {
const rows = await engine.executeRaw<{ n: string | number }>(
`SELECT COUNT(*)::int AS n FROM pages p WHERE ${where}`,
);
const n = Number(rows[0]?.n ?? 0);
if (n === 0) {
return {
name: 'raw_provenance',
status: 'ok',
message: 'All synthesized pages carry a raw trace or explicit exemption',
};
}
const sample = await engine.executeRaw<{ slug: string }>(
`SELECT p.slug FROM pages p WHERE ${where} ORDER BY p.slug LIMIT 5`,
);
const slugs = sample.map(r => r.slug).join(', ');
return {
name: 'raw_provenance',
status: 'warn',
message:
`${n} synthesized page(s) lack a raw trace (no raw_trace/raw_source/source_uri frontmatter, ` +
`raw_data row, or synthesis evidence) and carry no raw_trace_exempt marker. e.g. ${slugs}. ` +
`Fix: stamp raw_source (path/URI of the source material) or raw_trace_exempt: true + ` +
`raw_trace_exempt_reason in frontmatter. Warn-only (#1978).`,
};
} catch {
return { name: 'raw_provenance', status: 'warn', message: 'Could not check raw provenance (older schema?)' };
}
}
export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorReport> {
const checks: Check[] = [];
@@ -757,31 +813,7 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
// 5. Queue health (Postgres-only). PGLite has no minion_jobs in the same
// shape; skip the check there with an informational message.
if (engine.kind === 'postgres') {
try {
// issue #1801: column is `status`, not `state` (schema.sql:780). The
// pre-fix query errored every run and the catch silently returned "No
// queue activity," so this remote/thin-client check was a no-op.
const rows = await engine.executeRaw<{ stalled: string | number }>(
`SELECT COUNT(*) AS stalled FROM minion_jobs
WHERE status = 'active'
AND started_at IS NOT NULL
AND started_at < NOW() - INTERVAL '1 hour'`,
);
const stalled = Number(rows[0]?.stalled ?? 0);
checks.push({
name: 'queue_health',
status: stalled === 0 ? 'ok' : 'warn',
message: stalled === 0
? 'No stalled active jobs'
: `${stalled} active job(s) stalled > 1h — \`gbrain jobs cancel <id>\` or \`gbrain jobs retry <id>\` on the host`,
});
} catch {
checks.push({ name: 'queue_health', status: 'ok', message: 'No queue activity' });
}
} else {
checks.push({ name: 'queue_health', status: 'ok', message: 'PGLite — no queue to check' });
}
checks.push(await computeQueueHealthCheck(engine));
// issue #1801 — wedged_queue (cross-surface parity with buildChecks).
checks.push(await computeWedgedQueueCheck(engine));
@@ -1287,14 +1319,19 @@ export async function checkAbandonedThreads(engine: BrainEngine): Promise<Check>
/**
* calibration_freshness: warns when the active calibration profile is
* older than 7 days (configurable). Default holder 'garry'. Multi-source
* older than 7 days (configurable). Default holder resolves via resolveOwnerHolder
* (config emotional_weight.user_holder, else 'self'). Multi-source
* brains see one row per source; this check uses the most recent across
* all sources.
*/
export async function checkCalibrationFreshness(engine: BrainEngine): Promise<Check> {
try {
const ownerHolder = resolveOwnerHolder({
configValue: await engine.getConfig('emotional_weight.user_holder'),
});
const rows = await engine.executeRaw<{ generated_at: Date | null }>(
`SELECT MAX(generated_at) AS generated_at FROM calibration_profiles WHERE holder = 'garry'`,
`SELECT MAX(generated_at) AS generated_at FROM calibration_profiles WHERE holder = $1`,
[ownerHolder],
);
const generated = rows[0]?.generated_at;
if (!generated) {
@@ -1551,6 +1588,24 @@ export async function checkRerankerHealth(engine: BrainEngine): Promise<Check> {
};
}
// Historical #2059 rows were logged as `unknown` before missing reranker
// auth was classified at the gateway. Surface repeated unknowns instead of
// reporting "ok" while every rerank fails open.
const unknownFails = failures.filter((f) => f.reason === 'unknown');
if (unknownFails.length >= 3) {
const setupHint = unknownFails.some((f) => {
const summary = String(f.error_summary ?? '');
return summary.includes('ZEROENTROPY_API_KEY') || summary.toLowerCase().includes('api key');
})
? ' Fix: verify ZEROENTROPY_API_KEY and run `gbrain models doctor`.'
: '';
return {
name: 'reranker_health',
status: 'warn',
message: `${unknownFails.length} unknown reranker failure(s) in last 7 days.${setupHint}`,
};
}
return {
name: 'reranker_health',
status: 'ok',
@@ -1585,6 +1640,174 @@ export async function checkRerankerHealth(engine: BrainEngine): Promise<Check> {
* Also surfaces (codex M-10): runs resolveBulkRetryOpts(process.env) at
* startup so bad GBRAIN_BULK_* config fails at doctor time, not first-retry.
*/
/**
* queue_health: Postgres Minion queue diagnostics.
*
* Includes the original stalled/depth/memory/prompt checks plus the #2557
* no-worker signal: old `embed-backfill` jobs waiting on a queue with no live
* registered worker for that queue. That catches the default deployment shape
* where `sync` enqueues deferred embedding work but the operator never started
* `gbrain jobs work` or a supervisor.
*/
export async function computeQueueHealthCheck(
engine: BrainEngine,
opts: {
waitingDepthThreshold?: number;
oldWaitingHours?: number;
readWorkers?: () => Array<{ queue: string }>;
} = {},
): Promise<Check> {
if (engine.kind === 'pglite') {
return {
name: 'queue_health',
status: 'ok',
message: 'Skipped (PGLite — no multi-process worker surface)',
};
}
try {
// issue #1801: column is `status`, not `state` (schema.sql:780).
const stalledRows: Array<{ id: number; name: string; started_at: string }> =
await engine.executeRaw(
`SELECT id, name, started_at::text AS started_at
FROM minion_jobs
WHERE status = 'active'
AND started_at IS NOT NULL
AND started_at < now() - interval '1 hour'
ORDER BY started_at ASC
LIMIT 5`,
);
const threshold = opts.waitingDepthThreshold
?? _resolveEnvNumber('GBRAIN_QUEUE_WAITING_THRESHOLD', 10);
const depthRows: Array<{ name: string; queue: string; depth: number }> =
await engine.executeRaw(
`SELECT name, queue, count(*)::int AS depth
FROM minion_jobs
WHERE status = 'waiting'
GROUP BY name, queue
HAVING count(*) > $1
ORDER BY depth DESC
LIMIT 5`,
[threshold],
);
const rssKillRows: Array<{ cnt: number }> = await engine.executeRaw(
`SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE status IN ('dead', 'failed')
AND finished_at > now() - interval '24 hours'
AND error_text = 'aborted: watchdog'`,
);
const rssKillCount = Number(rssKillRows[0]?.cnt ?? 0);
const promptTooLongRows: Array<{ cnt: number }> = await engine.executeRaw(
`SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE name = 'subagent'
AND status = 'dead'
AND finished_at > now() - interval '24 hours'
AND error_text LIKE 'prompt_too_long:%'`,
);
const promptTooLongCount = Number(promptTooLongRows[0]?.cnt ?? 0);
const oldWaitingHours = opts.oldWaitingHours
?? _resolveEnvNumber('GBRAIN_QUEUE_NO_WORKER_WARN_HOURS', 1);
const oldWaitingRows: Array<{
name: string;
queue: string;
depth: number;
oldest_age_seconds: number;
}> = await engine.executeRaw(
`SELECT name,
queue,
count(*)::int AS depth,
EXTRACT(EPOCH FROM (now() - min(created_at)))::int AS oldest_age_seconds
FROM minion_jobs
WHERE status = 'waiting'
AND name = 'embed-backfill'
GROUP BY name, queue
HAVING min(created_at) < now() - ($1::text::interval)
ORDER BY oldest_age_seconds DESC
LIMIT 5`,
[`${oldWaitingHours} hours`],
);
let liveWorkerQueues = new Set<string>();
if (oldWaitingRows.length > 0) {
const workers = opts.readWorkers
? opts.readWorkers()
: (await import('../core/minions/worker-registry.ts')).readWorkers();
liveWorkerQueues = new Set(workers.map((w) => w.queue));
}
const problems: string[] = [];
if (stalledRows.length > 0) {
const sample = stalledRows
.map(r => `#${r.id}(${r.name})`)
.join(', ');
problems.push(
`${stalledRows.length} stalled-forever job(s): ${sample}. ` +
`Fix: gbrain jobs get <id> to inspect; gbrain jobs cancel <id> to force-kill.`
);
}
if (depthRows.length > 0) {
const sample = depthRows
.map(r => `${r.name}@${r.queue}=${r.depth}`)
.join(', ');
problems.push(
`waiting-queue depth exceeds ${threshold} for: ${sample}. ` +
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
for (const row of oldWaitingRows) {
if (liveWorkerQueues.has(row.queue)) continue;
const hours = Math.max(1, Math.round(Number(row.oldest_age_seconds ?? 0) / 3600));
problems.push(
`${row.depth} ${row.name} job(s) have waited on queue '${row.queue}' for up to ${hours}h ` +
`and no live worker is registered for that queue. ` +
`Start one with \`gbrain jobs work --queue ${row.queue}\` or ` +
`\`gbrain jobs supervisor start --queue ${row.queue}\`.`
);
}
if (rssKillCount > 0) {
problems.push(
`${rssKillCount} job(s) dead-lettered for RSS-watchdog memory-limit kills in last 24h. ` +
`Fix: raise the limit (e.g. \`gbrain jobs work --max-rss 4096\`) or opt out (\`--max-rss 0\`). ` +
`→ see worker_oom_loop for the cap + fix (the authoritative OOM-loop signal).`
);
}
if (promptTooLongCount > 0) {
problems.push(
`${promptTooLongCount} subagent job(s) dead-lettered with prompt_too_long in last 24h. ` +
`Dream/synthesize transcripts exceeded the model's input context. ` +
`Fix: \`gbrain dream --phase synthesize --dry-run --json\` to identify fat transcripts; ` +
`set \`dream.synthesize.max_prompt_tokens\` to bound the per-chunk budget, or use a ` +
`larger-context model (Opus 4.7 = 1M tokens vs Sonnet 4.6 = 200K).`
);
}
if (problems.length === 0) {
return {
name: 'queue_health',
status: 'ok',
message: `No stalled-forever jobs; no queue over depth ${threshold}; no old embed-backfill jobs without a worker.`,
};
}
return {
name: 'queue_health',
status: 'warn',
message: problems.join(' '),
};
} catch (e) {
return {
name: 'queue_health',
status: 'warn',
message: `queue_health scan skipped: ${e instanceof Error ? e.message : String(e)}`,
};
}
}
/**
* issue #1801 `wedged_queue` check. Surfaces the alive-but-wedged-worker
* signature (a queue with claimable work waiting, zero live-lock active jobs,
@@ -2960,6 +3183,54 @@ function _resolveSyncFreshnessHours(varName: string, fallback: number): number {
* branch (disabled / enabled-no-events / enabled-all-pass / enabled-with-failures)
* without spinning up the audit JSONL or a real config file.
*/
/**
* Pure function form of the conversation_parser_probe_health check.
* Mirrors computeNightlyQualityProbeHealthCheck: skip-with-hint when the
* probe is off and silent, surface the last 7 days of audit events when
* it has run, WARN on any non-pass outcome.
*
* `effectiveEnabled` folds the D10 mode-gate in: explicitly enabled OR
* search.mode=tokenmax (where the probe is default-on).
*/
export function computeConversationParserProbeHealthCheck(
effectiveEnabled: boolean,
events: ReadonlyArray<{ outcome: string; ts: string; reason?: string }>,
): Check {
const name = 'conversation_parser_probe_health';
if (!effectiveEnabled && events.length === 0) {
return {
name,
status: 'ok',
message:
'disabled (opt-in; default-on only for search.mode=tokenmax). Enable with: ' +
'`gbrain config set autopilot.conversation_parser_probe.enabled true`',
};
}
if (events.length === 0) {
return {
name,
status: 'ok',
message: 'enabled but no probe events in the last 7 days (next run by autopilot; fixtures require a source-checkout install).',
};
}
const bad = events.filter(e => e.outcome !== 'pass');
const latest = events[events.length - 1]!;
if (bad.length > 0) {
return {
name,
status: 'warn',
message:
`${bad.length}/${events.length} probe run(s) in the last 7 days did not pass; ` +
`latest: ${latest.outcome}${latest.reason ? ` (${latest.reason})` : ''}`,
};
}
return {
name,
status: 'ok',
message: `${events.length} probe run(s) in the last 7 days, all pass (latest ${latest.ts}).`,
};
}
export function computeNightlyQualityProbeHealthCheck(
probeEnabled: boolean,
events: ReadonlyArray<{ outcome: string; ts: string; detail?: string }>,
@@ -4843,10 +5114,17 @@ export async function buildChecks(
try {
const { readRecentQualityProbeEvents } = await import('../core/audit-quality-probe.ts');
const { loadConfig } = await import('../core/config.ts');
const { resolveProbeEnabled } = await import('../core/cycle/nightly-quality-probe.ts');
let probeEnabled = false;
try {
// Dual-plane read, matching the autopilot gate: the DB row (what the
// enable hint's `gbrain config set` writes) wins; file plane fallback.
let dbVal: string | null = null;
try {
dbVal = engine ? await engine.getConfig('autopilot.nightly_quality_probe.enabled') : null;
} catch { /* DB unavailable → file plane only */ }
const cfg = loadConfig();
probeEnabled = Boolean((cfg as any)?.autopilot?.nightly_quality_probe?.enabled);
probeEnabled = resolveProbeEnabled(dbVal, (cfg as any)?.autopilot?.nightly_quality_probe?.enabled);
} catch { /* config unavailable → treat as disabled */ }
const events = readRecentQualityProbeEvents(7);
const check = computeNightlyQualityProbeHealthCheck(probeEnabled, events);
@@ -5029,19 +5307,29 @@ export async function buildChecks(
// 3d.5 v0.41.13.0 — conversation_parser_probe_health. Mode-gated
// per D10: ON when search.mode=tokenmax, opt-in for other modes.
// Surface the last 7 days of nightly-probe events; warn on FAIL /
// BUDGET_EXCEEDED / adversarial_false_positive.
//
// v0.41.13.0 ships the probe as opt-in (autopilot wiring deferred
// to T7 in the cathedral plan); this check skips with an enable
// hint until the probe has at least one audit event written.
checks.push({
name: 'conversation_parser_probe_health',
status: 'ok',
message:
'Skipped (nightly probe is opt-in; enable with ' +
'`gbrain config set autopilot.conversation_parser_probe.enabled true`)',
});
// Surfaces the last 7 days of nightly-probe audit events; warn on any
// non-pass outcome (fail / budget_exceeded / adversarial_false_positive).
// (Until the autopilot wire-up this was a hardcoded "Skipped" stub.)
try {
const { readRecentParserProbeEvents } = await import('../core/audit-parser-probe.ts');
let parserProbeEnabled = false;
try {
let dbVal: string | null = null;
let dbMode: string | null = null;
try {
dbVal = engine ? await engine.getConfig('autopilot.conversation_parser_probe.enabled') : null;
dbMode = engine ? await engine.getConfig('search.mode') : null;
} catch { /* DB unavailable → file plane only */ }
const { loadConfig } = await import('../core/config.ts');
const fileVal = (loadConfig() as any)?.autopilot?.conversation_parser_probe?.enabled;
const flagOn = dbVal != null ? dbVal === 'true' : fileVal === true;
parserProbeEnabled = flagOn || dbMode === 'tokenmax';
} catch { /* config unavailable → treat as disabled */ }
const parserEvents = readRecentParserProbeEvents(7);
checks.push(computeConversationParserProbeHealthCheck(parserProbeEnabled, parserEvents));
} catch {
// Best-effort; audit-log read failure shouldn't stop doctor.
}
// 3e. home_dir_in_worktree (v0.35.8.0). Walks up from `gbrainPath()`
// looking for a `.git` directory OR file. If found, warns: `~/.gbrain/`
@@ -5867,12 +6155,12 @@ export async function buildChecks(
message: `Only code/test fixture entity pages found (${entityCount}); graph_coverage not applicable`,
});
} else if (linkCoverage >= 0.5 && timelineCoverage >= 0.5) {
checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%` });
checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, entity timeline coverage ${timelinePct}%` });
} else {
checks.push({
name: 'graph_coverage',
status: 'warn',
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`,
message: `Entity link coverage ${linkPct}%, entity timeline coverage ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`,
});
}
@@ -5884,7 +6172,7 @@ export async function buildChecks(
const parts = [
`embed ${health.embed_coverage_score}/35`,
`links ${health.link_density_score}/25`,
`timeline ${health.timeline_coverage_score}/15`,
`timeline density (all pages) ${health.timeline_coverage_score}/15`,
`orphans ${health.no_orphans_score}/15`,
`dead-links ${health.no_dead_links_score}/10`,
];
@@ -6057,6 +6345,12 @@ export async function buildChecks(
progress.heartbeat('child_table_orphans');
checks.push(await childTableOrphansCheck(engine));
// 10d. Raw-source persistence guarantee (#1978, warn-only v1).
// Every synthesized/derived page must carry a raw trace or an explicit
// exemption. Warn-only in v1 — surfaces violations, blocks nothing.
progress.heartbeat('raw_provenance');
checks.push(await rawProvenanceCheck(engine));
// v0.33: whoknows_health — fixture presence + row count. The eval
// gate itself runs via `gbrain eval whoknows`; this check is the
// "did you do the assignment?" signal.
@@ -6915,159 +7209,12 @@ export async function buildChecks(
}
}
// 11b. Queue health (v0.19.1 queue-resilience wave).
// Postgres-only because PGLite has no multi-process worker surface. Two
// subchecks, both cheap (single SELECT each, status-index-covered):
//
// 1. stalled-forever: any active job whose started_at is > 1h old. The
// incident that motivated this release ran 90+ min before surfacing.
// Surface the ID so the operator can `gbrain jobs get <id>` to inspect
// or `gbrain jobs cancel <id>` to force-kill.
//
// 2. backpressure-missed: per-name waiting depth exceeds the threshold
// (default 10, override via GBRAIN_QUEUE_WAITING_THRESHOLD env). Signal
// that a submitter probably needs maxWaiting set. Bounded by per-name
// aggregation so a single name's pile shows up clearly instead of
// getting lost in the total.
//
// Not included in v0.19.1 (tracked as B7 follow-up): worker-heartbeat
// staleness. It needs a minion_workers table; the lock_until-on-active-jobs
// proxy can't distinguish "no worker" from "worker idle," and a check that
// cries wolf erodes trust in every other doctor check.
progress.heartbeat('queue_health');
if (engine.kind === 'pglite') {
checks.push({
name: 'queue_health',
status: 'ok',
message: 'Skipped (PGLite — no multi-process worker surface)',
});
} else {
const queueHealthHb = startHeartbeat(progress, 'scanning queue health…');
try {
const sql = db.getConnection();
// Subcheck 1: stalled-forever active jobs (>1h wall-clock).
const stalledRows: Array<{ id: number; name: string; started_at: string }> = await sql`
SELECT id, name, started_at::text AS started_at
FROM minion_jobs
WHERE status = 'active'
AND started_at IS NOT NULL
AND started_at < now() - interval '1 hour'
ORDER BY started_at ASC
LIMIT 5
`;
// Subcheck 2: per-name waiting depth exceeds threshold.
const rawThreshold = process.env.GBRAIN_QUEUE_WAITING_THRESHOLD;
const parsedThreshold = rawThreshold ? parseInt(rawThreshold, 10) : 10;
const threshold = Number.isFinite(parsedThreshold) && parsedThreshold >= 1
? parsedThreshold
: 10;
const depthRows: Array<{ name: string; queue: string; depth: number }> = await sql`
SELECT name, queue, count(*)::int AS depth
FROM minion_jobs
WHERE status = 'waiting'
GROUP BY name, queue
HAVING count(*) > ${threshold}
ORDER BY depth DESC
LIMIT 5
`;
// Subcheck 3 (v0.22.14): RSS-watchdog kills in the last 24h. Bare workers
// newly default to --max-rss 2048 (was 0); operators who run large embed
// or import jobs may see kills that didn't happen pre-v0.22.14. We surface
// a hint when this signature appears so the upgrade path is obvious.
// Signature: when the watchdog trips, gracefulShutdown('watchdog') aborts
// in-flight jobs with `new Error('watchdog')`. The worker's failJob path
// (worker.ts:660-664) writes `error_text = 'aborted: watchdog'` for any
// job in-flight at the moment of the kill.
//
// We deliberately DO NOT do a loose `ILIKE '%watchdog%'`:
// 1. Parent jobs that inherit `on_child_fail='fail_parent'` get
// `"child job N failed: aborted: watchdog"` — counting that
// double-counts (child + parent) for one watchdog event.
// 2. Any user error_text containing the word "watchdog" matches.
// Match the exact prefix `'aborted: watchdog'` to scope this purely to
// the worker's own kill signature.
const rssKillRows: Array<{ cnt: number }> = await sql`
SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE status IN ('dead', 'failed')
AND finished_at > now() - interval '24 hours'
AND error_text = 'aborted: watchdog'
`;
const rssKillCount = rssKillRows[0]?.cnt ?? 0;
// Subcheck 4 (v0.30.2): prompt_too_long terminal failures on subagent
// jobs in the last 24h. The dream/synthesize phase classifies Anthropic
// 400 "prompt is too long" responses as UnrecoverableError so they
// dead-letter on first attempt instead of clogging the queue with
// max_stalled retries. Surface count + fix hint when present.
const promptTooLongRows: Array<{ cnt: number }> = await sql`
SELECT count(*)::int AS cnt
FROM minion_jobs
WHERE name = 'subagent'
AND status = 'dead'
AND finished_at > now() - interval '24 hours'
AND error_text LIKE 'prompt_too_long:%'
`;
const promptTooLongCount = promptTooLongRows[0]?.cnt ?? 0;
const problems: string[] = [];
if (stalledRows.length > 0) {
const sample = stalledRows
.map(r => `#${r.id}(${r.name})`)
.join(', ');
problems.push(
`${stalledRows.length} stalled-forever job(s): ${sample}. ` +
`Fix: gbrain jobs get <id> to inspect; gbrain jobs cancel <id> to force-kill.`
);
}
if (depthRows.length > 0) {
const sample = depthRows
.map(r => `${r.name}@${r.queue}=${r.depth}`)
.join(', ');
problems.push(
`waiting-queue depth exceeds ${threshold} for: ${sample}. ` +
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
if (rssKillCount > 0) {
problems.push(
`${rssKillCount} job(s) dead-lettered for RSS-watchdog memory-limit kills in last 24h. ` +
`Fix: raise the limit (e.g. \`gbrain jobs work --max-rss 4096\`) or opt out (\`--max-rss 0\`). ` +
`→ see worker_oom_loop for the cap + fix (the authoritative OOM-loop signal).`
);
}
if (promptTooLongCount > 0) {
problems.push(
`${promptTooLongCount} subagent job(s) dead-lettered with prompt_too_long in last 24h. ` +
`Dream/synthesize transcripts exceeded the model's input context. ` +
`Fix: \`gbrain dream --phase synthesize --dry-run --json\` to identify fat transcripts; ` +
`set \`dream.synthesize.max_prompt_tokens\` to bound the per-chunk budget, or use a ` +
`larger-context model (Opus 4.7 = 1M tokens vs Sonnet 4.6 = 200K).`
);
}
if (problems.length === 0) {
checks.push({
name: 'queue_health',
status: 'ok',
message: `No stalled-forever jobs; no queue over depth ${threshold}.`,
});
} else {
checks.push({
name: 'queue_health',
status: 'warn',
message: problems.join(' '),
});
}
} catch (e) {
checks.push({
name: 'queue_health',
status: 'warn',
message: `queue_health scan skipped: ${e instanceof Error ? e.message : String(e)}`,
});
} finally {
queueHealthHb();
}
const queueHealthHb = startHeartbeat(progress, 'scanning queue health…');
try {
checks.push(await computeQueueHealthCheck(engine));
} finally {
queueHealthHb();
}
// 11.4 subagent_capability (v0.38 — D7; was subagent_provider in v0.31.12). Surfaces a
@@ -7777,27 +7924,71 @@ export async function runRemediationPlan(
return;
}
// Human output
console.log(`Brain score: ${plan.brain_score_current}/100 → target ${targetScore}`);
for (const line of renderRemediationPlanLines(plan, targetScore)) {
console.log(line);
}
}
/**
* Human-render the remediation plan into a sequence of console lines.
* Exported for unit-test access `runRemediationPlan` consumes it
* verbatim and only adds the JSON-mode short-circuit.
*
* Gating the "at target" line on `brain_score_current >= targetScore`
* is load-bearing: when the plan is empty AND the target is unreachable,
* the prior shape printed both "Target unreachable: …" and "Brain is at
* target" back-to-back, which contradicted itself and hid the real next
* step (manual prereq config to lift `max_reachable_score`).
*/
export function renderRemediationPlanLines(
plan: RemediationPlanShape,
targetScore: number,
): string[] {
const lines: string[] = [];
lines.push(`Brain score: ${plan.brain_score_current}/100 → target ${targetScore}`);
if (plan.target_unreachable) {
console.log(`Target unreachable: max with autonomous remediation is ${plan.max_reachable_score}/100.`);
lines.push(`Target unreachable: max with autonomous remediation is ${plan.max_reachable_score}/100.`);
}
if (plan.plan.length === 0) {
console.log('No remediations needed. Brain is at target.');
if (plan.brain_score_current >= targetScore) {
lines.push('No remediations needed. Brain is at target.');
}
// When brain_score < targetScore and plan is empty, the unreachable
// line (if applicable) is the user-facing explanation; the blocked-
// checks block below surfaces the manual gap. Don't follow with a
// misleading "at target" claim.
} else {
console.log(`Plan: ${plan.plan.length} step(s), est ${plan.est_total_seconds}s, est $${plan.est_total_usd_cost.toFixed(2)}`);
lines.push(`Plan: ${plan.plan.length} step(s), est ${plan.est_total_seconds}s, est $${plan.est_total_usd_cost.toFixed(2)}`);
for (const step of plan.plan) {
const protectedMark = step.protected ? ' [PROTECTED]' : '';
const costMark = step.est_usd_cost ? ` ($${step.est_usd_cost.toFixed(2)})` : '';
console.log(` ${step.step}. [${step.severity}] ${step.job}${protectedMark}${step.rationale}${costMark}`);
lines.push(` ${step.step}. [${step.severity}] ${step.job}${protectedMark}${step.rationale}${costMark}`);
}
}
if (plan.blocked.length > 0) {
console.log(`\nBlocked checks (prereq missing):`);
lines.push(`\nBlocked checks (prereq missing):`);
for (const b of plan.blocked) {
console.log(` - ${b.check}: ${b.reason}`);
lines.push(` - ${b.check}: ${b.reason}`);
}
}
return lines;
}
interface RemediationPlanShape {
brain_score_current: number;
target_unreachable: boolean;
max_reachable_score: number;
plan: Array<{
step: number;
severity: string;
job: string;
protected?: boolean;
est_usd_cost?: number;
rationale: string;
}>;
est_total_seconds: number;
est_total_usd_cost: number;
blocked: Array<{ check: string; reason: string }>;
}
/**
+25 -5
View File
@@ -26,6 +26,7 @@
import type { BrainEngine } from '../core/engine.ts';
import {
runCycle,
resolveSourceForDir,
ALL_PHASES,
type CyclePhase,
type CycleReport,
@@ -85,7 +86,7 @@ interface DreamArgs {
* `--phase <name>`; bare `--once` is a usage error (there'd be no single
* phase to target). Applies only to phases with a config `.enabled` gate
* (patterns, synthesize, conversation_facts_backfill, enrich_thin,
* skillopt) a no-op for phases that always run when named directly.
* skillopt, drift) a no-op for phases that always run when named directly.
*/
once: boolean;
}
@@ -366,7 +367,7 @@ Options:
unlike toggling the flag on/off around the run, a
crash mid-invocation can't leave it stuck. Applies to
patterns, synthesize, conversation_facts_backfill,
enrich_thin, skillopt; no-op on phases with no such
enrich_thin, skillopt, drift; no-op on phases with no such
gate. Requires an EXPLICIT --phase <name> a phase
implied by --input or --drain does not count (bare
--once, or --once with --input/--drain and no
@@ -380,9 +381,9 @@ Options:
--source <id> Scope the cycle to one source so doctor's
cycle_freshness check sees a fresh stamp on
completion. Without this, gbrain dream's
timestamp never lands and federated brains
see "stale cycle" forever.
completion. When omitted, gbrain derives the
source from --dir / the configured checkout
when it matches a source's local_path (#1869).
--source-id <id> Alias for --source. Matches the v0.37.7.0+
naming used by import/extract/graph-query.
@@ -634,6 +635,25 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
);
process.exit(1);
}
// #1869: a path-scoped run (--dir, or the configured sync.repo_path) whose
// directory matches a registered source's local_path IS that source's cycle
// — derive the source id so runCycle writes last_source_cycle_at /
// last_full_cycle_at on success and doctor's cycle_freshness check stops
// reading perpetually stale. Explicit --source still wins (resolved above).
// Fixed here at the command level, NOT in runCycle's stamp gate, so legacy
// global callers (autopilot-global-maintenance runs GLOBAL_PHASES with a
// brainDir and no sourceId) can't falsely stamp per-source freshness.
// A derived match on an archived source is skipped silently (falls back to
// legacy unscoped behavior) — stamping it would mask staleness on restore,
// mirroring the explicit --source archived guard above.
if (resolvedSourceId === undefined && engine !== null && brainDir !== null) {
const derived = await resolveSourceForDir(engine, brainDir);
if (derived !== undefined) {
const src = await fetchSource(engine, derived);
if (src?.archived !== true) resolvedSourceId = derived;
}
}
// ─── issue #1678: bounded single-hold extract_atoms drain ──────────
if (opts.drain) {
if (engine === null) {
+33 -15
View File
@@ -107,6 +107,14 @@ export interface EmbedOpts {
* runs lock every source in sorted order. dryRun skips it.
*/
singleFlight?: boolean;
/**
* #394: suppress human stdout summaries (the `[dry-run] Would embed ...` /
* `Embedded N chunks ...` slog lines). Set by structured-output callers
* the cycle's embed phase (dream --json must keep stdout JSON-clean per
* docs/progress-events.md) reports counts via its own PhaseResult instead.
* Errors/warnings still go to stderr regardless.
*/
quiet?: boolean;
}
/**
@@ -253,7 +261,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
for (const s of opts.slugs) {
if (isAborted(opts.signal)) break; // #1737: stop the per-slug loop on abort
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal);
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal, opts.quiet);
} catch (e: unknown) {
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
@@ -347,6 +355,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
catchUp: opts.catchUp,
pacer,
paceMaxConcurrency,
quiet: opts.quiet,
}, opts.signal);
} finally {
// E1: surface pacing telemetry (human + structured) when pacing was on.
@@ -376,7 +385,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
return result;
}
if (opts.slug) {
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId, opts.signal);
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId, opts.signal, opts.quiet);
return result;
}
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
@@ -521,6 +530,7 @@ async function embedPage(
result: EmbedResult,
sourceId?: string,
signal?: AbortSignal,
quiet?: boolean,
) {
const opts = sourceId ? { sourceId } : undefined;
const page = await engine.getPage(slug, opts);
@@ -565,7 +575,7 @@ async function embedPage(
result.skipped += chunks.length - toEmbed.length;
if (toEmbed.length === 0) {
slog(`${slug}: all ${chunks.length} chunks already embedded`);
if (!quiet) slog(`${slug}: all ${chunks.length} chunks already embedded`);
result.pages_processed++;
return;
}
@@ -602,7 +612,7 @@ async function embedPage(
}
result.embedded += toEmbed.length;
result.pages_processed++;
slog(`${slug}: embedded ${toEmbed.length} chunks`);
if (!quiet) slog(`${slug}: embedded ${toEmbed.length} chunks`);
}
/**
@@ -645,6 +655,8 @@ async function embedAll(
pacer?: DbPacer;
/** Resolved concurrency cap (E-1: the worker count, no separate permit). */
paceMaxConcurrency?: number;
/** #394: suppress human stdout summaries (structured-output callers). */
quiet?: boolean;
},
signal?: AbortSignal,
) {
@@ -790,10 +802,12 @@ async function embedAll(
});
// Stdout summary preserved for scripts/tests that grep for counts.
if (dryRun) {
slog(`[dry-run] Would embed ${result.would_embed} chunks across ${pages.length} pages`);
} else {
slog(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
if (!staleOpts?.quiet) {
if (dryRun) {
slog(`[dry-run] Would embed ${result.would_embed} chunks across ${pages.length} pages`);
} else {
slog(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
}
}
}
@@ -829,6 +843,8 @@ async function embedAllStale(
pacer?: DbPacer;
/** Resolved concurrency cap (E-1: the worker count, no separate permit). */
paceMaxConcurrency?: number;
/** #394: suppress human stdout summaries (structured-output callers). */
quiet?: boolean;
},
signature?: string,
externalSignal?: AbortSignal,
@@ -846,7 +862,7 @@ async function embedAllStale(
signature,
...(sourceId && { sourceId }),
});
if (invalidated > 0) {
if (invalidated > 0 && !staleOpts?.quiet) {
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
}
}
@@ -857,10 +873,12 @@ async function embedAllStale(
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
);
if (staleCount === 0) {
if (dryRun) {
slog('[dry-run] Would embed 0 chunks (0 stale found)');
} else {
slog('Embedded 0 chunks (0 stale found)');
if (!staleOpts?.quiet) {
if (dryRun) {
slog('[dry-run] Would embed 0 chunks (0 stale found)');
} else {
slog('Embedded 0 chunks (0 stale found)');
}
}
return;
}
@@ -869,7 +887,7 @@ async function embedAllStale(
result.would_embed += staleCount;
result.total_chunks += staleCount;
if (onProgress) onProgress(1, 1, 0);
slog(`[dry-run] Would embed ${staleCount} stale chunks`);
if (!staleOpts?.quiet) slog(`[dry-run] Would embed ${staleCount} stale chunks`);
return;
}
@@ -1112,7 +1130,7 @@ async function embedAllStale(
if (budgetTimer) clearTimeout(budgetTimer);
}
slog(`Embedded ${result.embedded} chunks across ${totalProcessedPages} pages`);
if (!staleOpts?.quiet) slog(`Embedded ${result.embedded} chunks across ${totalProcessedPages} pages`);
// #1946 (OV2a): a catch-up pass that completed without being aborted but left
// chunks unembedded means those chunks are stuck (a non-transient embed
+15 -2
View File
@@ -76,7 +76,7 @@ FLAGS:
dimensions (goal, depth, sourcing, specificity, useful).
--cycles N 1-3. Default: 3 in TTY, 1 in non-TTY (T11). Each
cycle is 3 model calls; verdict aggregates over them.
--slot-a-model <id> Override default 'openai:gpt-4o'.
--slot-a-model <id> Override default 'openai:gpt-5.2'.
--slot-b-model <id> Override default 'anthropic:claude-opus-4-7'.
--slot-c-model <id> Override default 'google:gemini-1.5-pro'.
--receipt-dir <path> Default: gbrainPath('eval-receipts').
@@ -468,6 +468,14 @@ interface BatchRow {
question_id: string;
question: string;
hypothesis: string;
/**
* Gold answer from the benchmark dataset, when the upstream eval emits
* it (eval-longmemeval does). Folded into the judge task so CORRECTNESS
* is verifiable without it a judge panel that sees only
* {question, hypothesis} cannot validate a terse factual answer against
* a haystack it never saw.
*/
answer?: string;
}
/**
@@ -581,6 +589,7 @@ function readBatchRows(path: string): BatchReadResult {
question_id: typeof obj.question_id === 'string' ? obj.question_id : `line-${lineNo}`,
question: obj.question,
hypothesis: obj.hypothesis,
...(typeof obj.answer === 'string' && obj.answer.length > 0 ? { answer: obj.answer } : {}),
});
}
if (summarySkipped > 0) {
@@ -697,7 +706,11 @@ async function runBatchMode(parsed: ParsedArgs, opts: RunCrossModalOpts): Promis
fn: async (row, idx) => {
process.stderr.write(`[eval cross-modal batch] ${idx + 1}/${rows.length} ${row.question_id} starting...\n`);
return await runEvalFn({
task: row.question,
// With a gold answer the judges can actually verify correctness;
// without one they see only {question, hypothesis} and cannot.
task: row.answer
? `${row.question}\n\nExpected answer (gold label from the benchmark dataset): ${row.answer}`
: row.question,
output: row.hypothesis,
slug: row.question_id,
dimensions,
+17 -3
View File
@@ -33,6 +33,7 @@ import {
type AliasMap,
} from '../eval/longmemeval/extract.ts';
import { extractCandidateEntities } from '../core/think/entity-extract.ts';
import { splitProviderModelId } from '../core/model-id.ts';
import { resolveEntitySlugWithSource, type ResolutionSource } from '../core/entities/resolve.ts';
import { formatTrajectoryBlock } from '../core/trajectory-format.ts';
@@ -469,14 +470,22 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
});
// Wrap Anthropic SDK so its `.messages.create` shape matches ThinkLLMClient.
// Same pattern as src/core/think/index.ts:247-249.
// Same pattern as src/core/think/index.ts:247-249 — EXCEPT think's default
// client routes through the gateway, which parses `provider:model` recipe
// ids. This eval's client is a raw SDK by design (hermetic, no gateway
// dependency), and resolveModel returns RECIPE ids (`anthropic:claude-…`);
// passing one through unstripped 404s every answer/extractor call, which
// surfaces downstream as all-upstream_error batches in the nightly probe.
const toSdkModel = (m: string): string => splitProviderModelId(m).model || m;
const realClient = new Anthropic();
const client: ThinkLLMClient = runOpts.client ?? {
create: (params, callOpts) => realClient.messages.create(params, callOpts),
create: (params, callOpts) =>
realClient.messages.create({ ...params, model: toSdkModel(params.model) }, callOpts),
};
// v0.40.2.0 — separate extractor client (defaults to same SDK).
const extractorClient: ThinkLLMClient = runOpts.extractorClient ?? {
create: (params, callOpts) => realClient.messages.create(params, callOpts),
create: (params, callOpts) =>
realClient.messages.create({ ...params, model: toSdkModel(params.model) }, callOpts),
};
const trajectoryEnabled = !opts.noTrajectory;
const extractorModel = trajectoryEnabled
@@ -751,6 +760,11 @@ async function runOneQuestion(
// v0.40.1.0 (Track D / T2) — copy question_type into the row so the
// by_type_summary can be rebuilt from the file on resume runs.
question_type: q.question_type,
// Gold answer for downstream consumers that verify correctness (the
// cross-modal --batch judge folds it into the task; evaluate_qa.py
// ignores unknown fields). Without it a judge can't validate a terse
// factual hypothesis against a haystack it never saw.
...(q.answer !== undefined ? { answer: q.answer } : {}),
hypothesis,
retrieved_session_ids: retrievedSessionIds,
...(recallHit !== undefined ? { recall_hit: recallHit } : {}),
+47 -7
View File
@@ -433,7 +433,10 @@ export async function extractLinksFromFile(
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
if (!name) return null;
const trimmed = name.trim();
if (/^[a-z][a-z0-9-]*\/[a-z0-9][a-z0-9-]*$/.test(trimmed) && allSlugs.has(trimmed)) {
// Same broadened slug-shape as makeResolver step 1: accepts
// digit-leading folders (`90-people/nicolai`) and nested paths.
// Exact Set membership guards it — no false positives.
if (/\//.test(trimmed) && /^[a-z0-9][a-z0-9/_-]*$/.test(trimmed) && allSlugs.has(trimmed)) {
return trimmed;
}
const hints = Array.isArray(dirHint) ? dirHint : (dirHint ? [dirHint] : []);
@@ -582,6 +585,17 @@ export interface ExtractOpts {
* before (single-'default'-source brains unaffected).
*/
sourceId?: string;
/**
* v0.42 also extract frontmatter links on the incremental (slugs) path.
* `extractForSlugs` extracts BODY links only by default; set this true to also
* parse each changed page's frontmatter so `sources:`/`related:` edges stay fresh
* when YAML is edited externally and synced in. Applied PER changed page, so the
* incremental walk stays bounded (no switch to a full DB scan). Only honored on
* the incremental path (`slugs` defined); the full-walk path already covers
* frontmatter via its own dispatch. Gated upstream by the config key
* `autopilot.incremental_extract_include_frontmatter` (default off).
*/
includeFrontmatter?: boolean;
}
/**
@@ -620,7 +634,7 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
// Nothing changed — skip entirely.
return result;
}
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers, opts.signal, opts.sourceId, opts.includeFrontmatter);
result.links_created = r.links_created;
result.timeline_entries_created = r.timeline_created;
result.pages_processed = r.pages;
@@ -1011,6 +1025,11 @@ async function extractForSlugs(
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id on batch rows (see ExtractOpts.sourceId).
sourceId?: string,
// v0.42: when true, also extract frontmatter links per changed page so
// externally-edited YAML (`sources:`/`related:`) stays fresh on the cycle.
// Default false preserves the body-only incremental behavior. Gated upstream
// by `autopilot.incremental_extract_include_frontmatter`.
includeFrontmatter: boolean = false,
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
// Build the full slug set for link resolution (fast: just readdir, no file reads)
const allFiles = walkMarkdownFiles(brainDir);
@@ -1025,6 +1044,10 @@ async function extractForSlugs(
let linksCreated = 0;
let timelineCreated = 0;
let pagesProcessed = 0;
// #2636: successfully processed pages get their extraction watermark
// stamped after the final flush (mode 'all' only — a partial-mode run
// hasn't done the full extraction the watermark asserts).
const processedRefs: Array<{ slug: string; source_id: string }> = [];
// Issue #972: read the basename flag once per extract run.
const globalBasename = await isGlobalBasenameEnabled(engine);
@@ -1085,7 +1108,7 @@ async function extractForSlugs(
const content = readFileSync(fullPath, 'utf-8');
if (doLinks) {
const links = await extractLinksFromFile(content, relPath, allSlugs, { globalBasename });
const links = await extractLinksFromFile(content, relPath, allSlugs, { globalBasename, includeFrontmatter });
for (const link of links) {
if (dryRun) {
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
@@ -1113,6 +1136,7 @@ async function extractForSlugs(
}
pagesProcessed++;
if (!dryRun) processedRefs.push({ slug, source_id: sourceId ?? 'default' });
} catch { /* skip unreadable */ }
progress.tick(1);
},
@@ -1120,6 +1144,13 @@ async function extractForSlugs(
await flushLinks();
await flushTimeline();
// #2636: the Dream cycle disables sync's inline extraction and routes
// changed slugs through this incremental path — without a stamp here,
// those pages never get links_extracted_at and stay permanently visible
// to `extract --stale` / doctor. Stamp only after BOTH batches flushed.
if (!dryRun && mode === 'all') {
await stampExtracted(engine, processedRefs);
}
progress.finish();
if (!jsonMode) {
@@ -1684,9 +1715,17 @@ export async function extractStaleFromDB(
// Batch mode = pg_trgm + exact only, NO per-name search fallback. The
// resolution map sees ALL sources so qualified cross-source wikilinks resolve
// even when --source-id scopes the stale SCAN.
const resolver = makeResolver(engine, { mode: 'batch' });
const nullResolver = { resolve: async () => null as string | null };
const activeResolver = includeFrontmatter ? resolver : nullResolver;
//
// #2576 bug 1: ALWAYS the real resolver — extractPageLinks's opts gate which
// pass runs (`skipFrontmatter` for the frontmatter pass, `globalBasename` for
// the issue-#972 bare-wikilink pass). The former `includeFrontmatter ?
// resolver : nullResolver` ternary predates #972; the synthetic resolver has
// no `resolveBasenameMatches`, so the --stale sweep silently skipped basename
// resolution even with `link_resolution.global_basename` enabled, stamping
// pages as extracted with their bare wikilinks dropped. Mirrors
// extractLinksFromDB (including the codex-[P1] `sourceId` scoping).
const resolver = makeResolver(engine, { mode: 'batch', sourceId: sourceIdFilter });
const globalBasename = await isGlobalBasenameEnabled(engine);
const allRefs = await engine.listAllPageRefs();
const allSlugs = new Set<string>();
const slugToSources = new Map<string, string[]>();
@@ -1718,7 +1757,8 @@ export async function extractStaleFromDB(
for (const page of rows) {
const fullContent = page.compiled_truth + '\n' + page.timeline;
const extracted = await extractPageLinks(
page.slug, fullContent, page.frontmatter, page.type, activeResolver,
page.slug, fullContent, page.frontmatter, page.type, resolver,
{ skipFrontmatter: !includeFrontmatter, globalBasename },
);
for (const c of extracted.candidates) {
const r = resolveCandidateSources(c, page.slug, page.source_id, allSlugs, slugToSources);
+110 -13
View File
@@ -26,6 +26,8 @@ export async function runInit(args: string[]) {
return;
}
validateInitFlags(args);
const isSupabase = args.includes('--supabase');
const isPGLite = args.includes('--pglite');
const isMcpOnly = args.includes('--mcp-only');
@@ -151,6 +153,65 @@ export async function runInit(args: string[]) {
return initPostgres({ databaseUrl, jsonOutput, apiKey, aiOpts, schemaPack, skipEmbedCheck });
}
const INIT_BOOLEAN_FLAGS = new Set([
'--pglite',
'--supabase',
'--mcp-only',
'--force',
'--non-interactive',
'--migrate-only',
'--json',
'--no-embedding',
'--skip-embed-check',
]);
const INIT_VALUE_FLAGS = new Set([
'--url',
'--key',
'--path',
'--schema-pack',
'--embedding-model',
'--model',
'--embedding-dimensions',
'--expansion-model',
'--chat-model',
'--mcp-url',
'--issuer-url',
'--oauth-client-id',
'--oauth-client-secret',
]);
function validateInitFlags(args: string[]) {
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (!arg.startsWith('-')) continue;
if (INIT_BOOLEAN_FLAGS.has(arg)) continue;
if (INIT_VALUE_FLAGS.has(arg)) {
if (i + 1 >= args.length || args[i + 1].startsWith('-')) {
failInitFlag(`gbrain init: ${arg} requires a value`, args.includes('--json'));
}
i += 1;
continue;
}
if (arg.startsWith('--')) {
failInitFlag(`gbrain init: unknown flag ${arg}`, args.includes('--json'));
}
}
}
function failInitFlag(message: string, jsonOutput: boolean): never {
if (jsonOutput) {
console.log(JSON.stringify({ status: 'error', reason: 'invalid_flag', message }));
} else {
console.error(message);
console.error('Run `gbrain init --help` for supported flags.');
}
process.exit(1);
}
interface ResolveAIOptionsArgs {
verbose: string | null; // --embedding-model
shorthand: string | null; // --model
@@ -161,7 +222,7 @@ interface ResolveAIOptionsArgs {
nonInteractive: boolean; // --non-interactive (forces D3 fail-loud, no picker)
}
interface ResolvedAIOptions {
export interface ResolvedAIOptions {
embedding_model?: string;
embedding_dimensions?: number;
expansion_model?: string;
@@ -170,6 +231,41 @@ interface ResolvedAIOptions {
noEmbedding?: boolean;
}
/**
* Seed init's AI options from persisted config, falling back to the raw env
* vars when loadConfig() returned null (#1058). On a cold install (no
* config.json AND no DATABASE_URL) loadConfig short-circuits BEFORE its env
* merge, so GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS /
* GBRAIN_EXPANSION_MODEL / GBRAIN_CHAT_MODEL were silently ignored by init
* and Tier-3 detection auto-picked by API key instead. Exported for unit
* tests (env injectable).
*/
export function seedAIOptionsFromConfig(
cfg: GBrainConfig | null,
env: NodeJS.ProcessEnv = process.env,
): ResolvedAIOptions {
const envDims = env.GBRAIN_EMBEDDING_DIMENSIONS
? parseInt(env.GBRAIN_EMBEDDING_DIMENSIONS, 10)
: NaN;
const seed = cfg ?? {
embedding_disabled: undefined,
embedding_model: env.GBRAIN_EMBEDDING_MODEL,
embedding_dimensions: Number.isFinite(envDims) ? envDims : undefined,
expansion_model: env.GBRAIN_EXPANSION_MODEL,
chat_model: env.GBRAIN_CHAT_MODEL,
};
const out: ResolvedAIOptions = {};
if (seed.embedding_disabled) {
out.noEmbedding = true;
} else if (seed.embedding_model) {
out.embedding_model = seed.embedding_model;
if (seed.embedding_dimensions) out.embedding_dimensions = seed.embedding_dimensions;
}
if (seed.expansion_model) out.expansion_model = seed.expansion_model;
if (seed.chat_model) out.chat_model = seed.chat_model;
return out;
}
/**
* Resolve AI provider options for `gbrain init`.
*
@@ -203,18 +299,13 @@ async function resolveAIOptions(opts: ResolveAIOptionsArgs): Promise<ResolvedAIO
// user already opted into deferred mode.
try {
const { loadConfig } = await import('../core/config.ts');
const cfg = loadConfig();
if (cfg?.embedding_disabled) {
out.noEmbedding = true;
} else if (cfg?.embedding_model) {
out.embedding_model = cfg.embedding_model;
if (cfg.embedding_dimensions) out.embedding_dimensions = cfg.embedding_dimensions;
}
if (cfg?.expansion_model) out.expansion_model = cfg.expansion_model;
if (cfg?.chat_model) out.chat_model = cfg.chat_model;
// #1058: loadConfig() returns null on a cold install (no config.json AND
// no DATABASE_URL) — before it ever reaches its env merge. The seed helper
// falls back to the same GBRAIN_* env vars directly in that case.
Object.assign(out, seedAIOptionsFromConfig(loadConfig()));
} catch {
// loadConfig throws when no brain configured — first-time install, fall
// through to env detection.
// loadConfig threw — treat as first-time install, fall through to env
// detection.
}
// --- Tier 1+2: explicit flags ---------------------------------------------
@@ -1078,6 +1169,9 @@ async function initPostgres(opts: {
console.warn(' Direct connections are IPv6 only and fail in many environments.');
console.warn(' Use the Transaction pooler connection string instead (port 6543):');
console.warn(' Supabase Dashboard > Connect (top bar) > Connection String > Transaction pooler');
console.warn(' (With a pooler URL, gbrain derives a direct connection for DDL and falls back');
console.warn(' to the pooler automatically if that host is unreachable. Power users:');
console.warn(' GBRAIN_DIRECT_DATABASE_URL overrides the derived URL; GBRAIN_DISABLE_DIRECT_POOL=1 disables it.)');
console.warn('');
}
@@ -1091,6 +1185,9 @@ async function initPostgres(opts: {
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Transaction pooler connection string instead (port 6543).');
console.error('(gbrain derives its own direct connection from pooler URLs for DDL; if that host is');
console.error('unreachable it falls back to the pooler. GBRAIN_DIRECT_DATABASE_URL overrides the');
console.error('derived URL; GBRAIN_DISABLE_DIRECT_POOL=1 disables the direct pool entirely.)');
}
throw e;
}
@@ -1455,7 +1552,7 @@ export function reportModStatus(): void {
console.log(' cd ~/.claude/skills/gstack && ./setup');
}
console.log('Resolver: skills/RESOLVER.md');
console.log('Soul audit: run `gbrain soul-audit` to customize agent identity');
console.log('Soul audit: ask your agent to "run a soul audit" to customize its identity (see skills/soul-audit)');
// Retrieval Reflex (#1981): the deterministic pointer layer is ON by default
// (no action needed). The policy skill is installed into the HOST repo on
// request — we PRINT the command rather than silently mutating the host repo.
+17 -1
View File
@@ -1885,6 +1885,7 @@ export async function registerBuiltinHandlers(
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
phases,
forceGlobalOrphans: true,
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
});
@@ -2061,11 +2062,26 @@ export async function registerBuiltinHandlers(
? job.data.repoPath
: ((await engine.getConfig('sync.repo_path')) ?? undefined);
try {
return await runExtractAtomsDrainForSource(engine, {
const result = await runExtractAtomsDrainForSource(engine, {
sourceId,
windowSeconds,
brainDir: repoPath,
});
// issue #3218: every item the drain attempted failed (0 succeeded, >=1
// provider error) — completing this job normally would mark the
// durable job done while the backlog sits untouched, and no retry
// policy would ever fire on it again. Throw so the worker's ordinary
// failJob path (attempt+backoff, or dead-letter once exhausted) takes
// over instead — matching the existing behavior for every other
// handler failure. Partial success (>=1 item extracted) keeps
// completing normally, unchanged.
if (result.status === 'provider_failure') {
throw new Error(
`extract-atoms-drain: all provider calls failed this batch ` +
`(batches=${result.batches}, remaining=${result.remaining ?? '?'}) — retrying`,
);
}
return result;
} catch (e) {
if (e instanceof LockUnavailableError) {
return { phase: 'extract_atoms', status: 'skipped', deferred: true, reason: 'cycle_already_running' };
+43 -8
View File
@@ -383,15 +383,30 @@ async function resolveLintContentSanity(
};
}
/**
* Directories never containing knowledge pages, skipped by default.
* Deliberately tiny: only vendored dependency trees qualify. Anything
* more opinionated (README.md, CHANGELOG.md, test/) is repo policy
* callers opt in via `--exclude` / `LintOpts.exclude`. Dot- and
* underscore-prefixed entries are already skipped by the walk.
*/
const DEFAULT_LINT_EXCLUDE_DIRS = new Set(['node_modules']);
/** Collect markdown files from a directory */
function collectPages(dir: string): string[] {
function collectPages(dir: string, extraExcludes: string[] = []): string[] {
const extra = new Set(extraExcludes);
const pages: string[] = [];
function walk(d: string) {
for (const entry of readdirSync(d)) {
if (entry.startsWith('.') || entry.startsWith('_')) continue;
const full = join(d, entry);
if (lstatSync(full).isDirectory()) walk(full);
else if (entry.endsWith('.md')) pages.push(full);
if (lstatSync(full).isDirectory()) {
if (DEFAULT_LINT_EXCLUDE_DIRS.has(entry) || extra.has(entry)) continue;
walk(full);
} else if (entry.endsWith('.md')) {
if (extra.has(entry)) continue;
pages.push(full);
}
}
}
walk(dir);
@@ -419,6 +434,13 @@ export interface LintOpts {
* yields + checks this every 200 pages.
*/
signal?: AbortSignal;
/**
* #2649: extra dir/file basenames to skip while collecting pages, in
* addition to node_modules and dot/underscore entries. For mixed-content
* repos (knowledge pages alongside software trees). Ignored for
* single-file targets.
*/
exclude?: string[];
}
export interface LintResult {
@@ -445,7 +467,7 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
}
const isSingleFile = statSync(opts.target).isFile();
const pages = isSingleFile ? [opts.target] : collectPages(opts.target);
const pages = isSingleFile ? [opts.target] : collectPages(opts.target, opts.exclude ?? []);
// Resolve content-sanity config once for this lint run (D1: lift DB
// config when reachable). Caller can pre-pass via opts.contentSanity
@@ -496,14 +518,27 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
}
export async function runLint(args: string[]) {
const target = args.find(a => !a.startsWith('--'));
// #2649: --exclude=a,b or --exclude a,b — extra basenames to skip.
const extraExcludes: string[] = [];
const skipIdx = new Set<number>();
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a.startsWith('--exclude=')) {
extraExcludes.push(...a.slice('--exclude='.length).split(',').map(s => s.trim()).filter(Boolean));
} else if (a === '--exclude' && i + 1 < args.length) {
extraExcludes.push(...args[i + 1].split(',').map(s => s.trim()).filter(Boolean));
skipIdx.add(i + 1);
}
}
const target = args.find((a, i) => !a.startsWith('--') && !skipIdx.has(i));
const doFix = args.includes('--fix');
const dryRun = args.includes('--dry-run');
if (!target) {
console.error('Usage: gbrain lint <dir|file.md> [--fix] [--dry-run]');
console.error('Usage: gbrain lint <dir|file.md> [--fix] [--dry-run] [--exclude a,b]');
console.error(' --fix Auto-fix fixable issues (LLM preambles, code fences)');
console.error(' --dry-run Preview fixes without writing');
console.error(' --exclude Comma-separated dir/file basenames to skip (in addition to node_modules)');
process.exit(1);
}
@@ -515,7 +550,7 @@ export async function runLint(args: string[]) {
// Single file or directory — print human detail as we go, then rely on
// Core for the aggregate numbers at the end.
const isSingleFile = statSync(target).isFile();
const pages = isSingleFile ? [target] : collectPages(target);
const pages = isSingleFile ? [target] : collectPages(target, extraExcludes);
// Progress on stderr. Stdout keeps the per-issue human output it always had.
const { createProgress } = await import('../core/progress.ts');
@@ -562,7 +597,7 @@ export async function runLint(args: string[]) {
// produces canonical numbers for the summary line).
// Pass contentSanity through so runLintCore skips its own resolve
// (we already resolved once for the human-detail loop above).
const result = await runLintCore({ target, fix: doFix, dryRun, contentSanity });
const result = await runLintCore({ target, fix: doFix, dryRun, contentSanity, exclude: extraExcludes });
console.log(`\n${result.pages_scanned} pages scanned. ${result.total_issues} issue(s) in ${result.pages_with_issues} page(s).`);
if (doFix) {
console.log(`${dryRun ? '(dry run) ' : ''}${result.total_fixed} auto-fixed.`);
+216 -98
View File
@@ -10,12 +10,13 @@
import { createEngine } from '../core/engine-factory.ts';
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
import type { BrainEngine } from '../core/engine.ts';
import type { EngineConfig } from '../core/types.ts';
import type { EngineConfig, Page } from '../core/types.ts';
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
import { createHash } from 'crypto';
import { resolve } from 'path';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
interface MigrateOpts {
targetEngine: 'postgres' | 'pglite';
@@ -143,6 +144,99 @@ export async function copyMigrationSources(source: BrainEngine, target: BrainEng
}
}
/**
* postgres.js's UNDEFINED_VALUE guard rejects any bound parameter that is JS
* `undefined` unlike PGLite, it will not silently treat it as SQL NULL.
* A page read back from a PGLite source can carry `undefined` for a column
* that is legitimately empty/NULL (a read-side driver-shape difference, not
* a data problem), and passing that value straight into a Postgres
* `putPage` throws mid-insert (#3194). Normalizing at this migrate-only
* boundary rather than inside `putPage` itself, which many non-migrate
* callers also use turns that driver-shape difference into an explicit
* SQL NULL, so only a genuine NOT-NULL constraint violation (an actual data
* problem) still surfaces as a page-copy failure.
*/
function nullifyUndefinedColumns<T extends Record<string, unknown>>(row: T): T {
const normalized = { ...row };
for (const key of Object.keys(normalized) as (keyof T)[]) {
if (normalized[key] === undefined) normalized[key] = null as T[typeof key];
}
return normalized;
}
/**
* Copy one page's full row (page body, chunks, tags, timeline, raw data)
* from source to target. Throws on any failure the caller (the per-page
* loop in runMigrateEngine) decides how to account for that: track it as a
* failed page and keep going, rather than letting one bad row silently
* disappear from the progress count (#3194). Exported so unit tests can
* inject fake engines and exercise the failure path without a live
* DATABASE_URL.
*/
export async function copyPageToTarget(
source: BrainEngine,
target: BrainEngine,
page: Page,
): Promise<void> {
const sourceOpts = { sourceId: page.source_id };
// Copy page (preserve source_id). v0.32.8 F8: thread source_id end-to-end
// so multi-source pages migrate intact.
await target.putPage(page.slug, nullifyUndefinedColumns({
type: page.type,
title: page.title,
compiled_truth: page.compiled_truth,
timeline: page.timeline,
frontmatter: page.frontmatter,
content_hash: page.content_hash,
}), sourceOpts);
// Copy chunks with embeddings.
const chunks = await source.getChunksWithEmbeddings(page.slug, sourceOpts);
if (chunks.length > 0) {
await target.upsertChunks(page.slug, chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
embedding: c.embedding || undefined,
model: c.model,
token_count: c.token_count || undefined,
})), sourceOpts);
}
// Copy tags
const tags = await source.getTags(page.slug, sourceOpts);
for (const tag of tags) {
await target.addTag(page.slug, tag, sourceOpts);
}
// Copy timeline
const timeline = await source.getTimeline(page.slug, sourceOpts);
for (const entry of timeline) {
await target.addTimelineEntry(page.slug, {
date: entry.date,
source: entry.source,
summary: entry.summary,
detail: entry.detail,
}, sourceOpts);
}
// Copy raw data
const rawData = await source.getRawData(page.slug, undefined, sourceOpts);
for (const rd of rawData) {
await target.putRawData(page.slug, rd.source, rd.data, sourceOpts);
}
}
/** A page that failed to copy during migrate tracked so the run's final
* summary reports it honestly instead of letting the "N copied" counter
* imply every page landed (#3194). */
export interface MigratePageFailure {
source_id: string;
slug: string;
reason: string;
}
export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]): Promise<void> {
const opts = parseArgs(args);
const config = loadConfig();
@@ -177,32 +271,47 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
await targetEngine.connect(targetConfig);
await targetEngine.initSchema();
// Check if target has data
const targetStats = await targetEngine.getStats();
if (targetStats.page_count > 0 && !opts.force) {
console.error(`Target brain is not empty (${targetStats.page_count} pages).`);
console.error('Run with --force to overwrite, or migrate to an empty brain.');
await targetEngine.disconnect();
process.exit(1);
}
if (targetStats.page_count > 0 && opts.force) {
console.log('--force: wiping target brain...');
// v0.18.0+ multi-source: deletePage(slug) is now source-scoped (defaults
// to 'default'), so per-page iteration would skip non-default-source
// rows. migrate-engine --force is a destructive wipe across the entire
// brain — all sources, all pages — so we issue a raw DELETE that matches
// the original semantic. Cascades through content_chunks / page_links /
// tags / timeline_entries / page_versions via existing FKs.
await targetEngine.executeRaw('DELETE FROM pages');
}
// Load or create manifest for resume
// Load or create manifest for resume. Checked BEFORE the non-empty-target
// guard below: a manifest matching this exact target means the target's
// existing rows came from OUR OWN in-progress migration (#3194's per-page
// failures now leave the target non-empty by design instead of crashing),
// so a resume must not be treated as "attempting to migrate into a
// foreign non-empty brain".
let manifest = loadManifest();
if (manifest && !manifestMatchesTarget(manifest, targetId)) {
console.log('Previous migration was to a different target. Starting fresh.');
manifest = null;
}
const resumingMatchingManifest = manifest !== null;
// Check if target has data
const targetStats = await targetEngine.getStats();
if (opts.force) {
if (targetStats.page_count > 0) {
console.log('--force: wiping target brain...');
// v0.18.0+ multi-source: deletePage(slug) is now source-scoped (defaults
// to 'default'), so per-page iteration would skip non-default-source
// rows. migrate-engine --force is a destructive wipe across the entire
// brain — all sources, all pages — so we issue a raw DELETE that matches
// the original semantic. Cascades through content_chunks / page_links /
// tags / timeline_entries / page_versions via existing FKs.
await targetEngine.executeRaw('DELETE FROM pages');
}
// --force always starts this exact migration fresh against this target:
// a manifest tracking a previous attempt must not be trusted to skip
// pages, regardless of whether the target LOOKED non-empty just now
// (e.g. the target DB file was recreated out-of-band but
// ~/.gbrain/migrate-manifest.json survived) — round 2 of #3194.
manifest = null;
} else if (targetStats.page_count > 0 && !resumingMatchingManifest) {
console.error(`Target brain is not empty (${targetStats.page_count} pages).`);
console.error('Run with --force to overwrite, or migrate to an empty brain.');
await targetEngine.disconnect();
process.exit(1);
} else if (targetStats.page_count > 0 && resumingMatchingManifest) {
console.log(`Resuming previous migration: ${manifest!.completed_slugs.length} page(s) already copied.`);
}
// v0.32.8 F8: manifest keys are now `${source_id}::${slug}` so multi-source
// migrations don't collide on same-slug-different-source pages. Pre-v0.32.8
// entries were bare slugs; we keep treating those as default-source for
@@ -219,6 +328,13 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
started_at: new Date().toISOString(),
};
}
// Persist immediately, before any page copy runs. Otherwise a run where
// EVERY page fails after its putPage lands (but before completed_slugs
// ever gets a successful entry) leaves the target non-empty with no
// manifest file on disk at all — the next invocation can't tell this
// was a resumable in-progress migration and hits the non-empty guard
// above requiring --force (round 2 of #3194).
saveManifest(manifest);
// Pages.source_id is a foreign key. Copy the complete source catalog first,
// including archived rows and sync/routing metadata, so every page write has
@@ -235,82 +351,68 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start('migrate.copy_pages', pagesToMigrate.length);
// v0.32.8 F8: thread source_id end-to-end so multi-source pages migrate
// intact. Pre-fix: putPage / getTags / getTimeline / getRawData / getLinks
// all silently defaulted to source_id='default', so non-default-source
// tags / timeline / raw / links were either dropped or attached to the
// wrong row.
let migrated = 0;
const failures: MigratePageFailure[] = [];
for (const page of pagesToMigrate) {
// v0.32.8 F8: thread source_id end-to-end so multi-source pages migrate
// intact. Pre-fix: putPage / getTags / getTimeline / getRawData / getLinks
// all silently defaulted to source_id='default', so non-default-source
// tags / timeline / raw / links were either dropped or attached to the
// wrong row.
const sourceOpts = { sourceId: page.source_id };
// Copy page (preserve source_id)
await targetEngine.putPage(page.slug, {
type: page.type,
title: page.title,
compiled_truth: page.compiled_truth,
timeline: page.timeline,
frontmatter: page.frontmatter,
content_hash: page.content_hash,
}, sourceOpts);
// Copy chunks with embeddings.
const chunks = await sourceEngine.getChunksWithEmbeddings(page.slug, sourceOpts);
if (chunks.length > 0) {
await targetEngine.upsertChunks(page.slug, chunks.map(c => ({
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
embedding: c.embedding || undefined,
model: c.model,
token_count: c.token_count || undefined,
})), sourceOpts);
try {
await copyPageToTarget(sourceEngine, targetEngine, page);
// Track progress with composite key so multi-source resume is correct.
manifest!.completed_slugs.push(makeManifestKey(page.source_id, page.slug));
saveManifest(manifest!);
migrated++;
} catch (e) {
// #3194: a per-page write failure must never be swallowed into the
// success count. Leave it OUT of completed_slugs (a resume retries
// it — putPage/upsertChunks/etc. are all upserts, so re-running the
// whole page copy is safe) and surface it in the final summary below
// instead of letting "N pages copied" imply everything landed.
failures.push({
source_id: page.source_id,
slug: page.slug,
reason: e instanceof Error ? e.message : String(e),
});
}
// Copy tags
const tags = await sourceEngine.getTags(page.slug, sourceOpts);
for (const tag of tags) {
await targetEngine.addTag(page.slug, tag, sourceOpts);
}
// Copy timeline
const timeline = await sourceEngine.getTimeline(page.slug, sourceOpts);
for (const entry of timeline) {
await targetEngine.addTimelineEntry(page.slug, {
date: entry.date,
source: entry.source,
summary: entry.summary,
detail: entry.detail,
}, sourceOpts);
}
// Copy raw data
const rawData = await sourceEngine.getRawData(page.slug, undefined, sourceOpts);
for (const rd of rawData) {
await targetEngine.putRawData(page.slug, rd.source, rd.data, sourceOpts);
}
// Copy versions
const versions = await sourceEngine.getVersions(page.slug, sourceOpts);
// Versions are snapshots, we recreate them on the target
// (createVersion takes a snapshot of current state, which we just set)
// Track progress with composite key so multi-source resume is correct.
manifest!.completed_slugs.push(makeManifestKey(page.source_id, page.slug));
saveManifest(manifest!);
migrated++;
progress.tick(1, page.slug);
}
progress.finish();
if (failures.length > 0) {
console.error(`\n${failures.length} of ${pagesToMigrate.length} page(s) FAILED to copy and were NOT migrated:`);
for (const f of failures) {
const key = f.source_id === 'default' ? f.slug : `${f.source_id}::${f.slug}`;
console.error(` - ${key}: ${f.reason}`);
}
console.error('Re-run `gbrain migrate` to retry the failed pages (already-copied pages resume via the manifest).');
// Non-fatal so the run still copies links + config for everything that
// DID land, but the process must exit non-zero — a partial migration
// must never look identical to a clean one.
setCliExitVerdict(1);
}
// Copy links (after all pages exist in target).
// v0.32.8 F8: thread source_id so cross-source links migrate correctly.
// #3194: a page that failed to copy above does NOT exist on the target,
// so any link touching it would violate the target's FK and abort this
// whole phase (the exact "addLink failed: page ... not found" crash from
// the original report). Skip links on either end of a known-failed page —
// a retry that successfully copies the page also re-copies its links.
const failedKeys = new Set(failures.map(f => makeManifestKey(f.source_id, f.slug)));
console.log('Copying links...');
progress.start('migrate.copy_links', allPages.length);
for (const page of allPages) {
if (failedKeys.has(makeManifestKey(page.source_id, page.slug))) {
progress.tick(1);
continue;
}
const sourceOpts = { sourceId: page.source_id };
const links = await sourceEngine.getLinks(page.slug, sourceOpts);
for (const link of links) {
if (failedKeys.has(makeManifestKey(page.source_id, link.to_slug))) continue;
await targetEngine.addLink(
link.from_slug, link.to_slug,
link.context, link.link_type,
@@ -342,22 +444,38 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
// Update local config. v0.37 fix wave: preserve existing file-plane
// embedding/expansion/chat config across the engine migration; only
// the engine + connection target should change.
const existingFile = (await import('../core/config.ts')).loadConfigFileOnly() ?? ({} as GBrainConfig);
const newConfig: GBrainConfig = {
...existingFile,
engine: opts.targetEngine,
...(opts.targetEngine === 'postgres'
? { database_url: targetConfig.database_url, database_path: undefined }
: { database_path: targetConfig.database_path, database_url: undefined }),
};
saveConfig(newConfig);
//
// #3194: only flip the ACTIVE config when the migration is fully clean.
// A partial migration leaves the target's data incomplete; auto-switching
// every subsequent `gbrain` invocation onto that incomplete target would
// (a) make the failure invisible behind otherwise-normal usage and (b)
// break the natural retry — `gbrain migrate --to X` again would hit the
// "Already using X engine" guard even though the migration never actually
// finished. Leaving the file-plane config untouched keeps the source the
// active engine, so a retry (which resumes via the still-intact manifest)
// is a same-shaped command, not a special case.
if (failures.length === 0) {
const existingFile = (await import('../core/config.ts')).loadConfigFileOnly() ?? ({} as GBrainConfig);
const newConfig: GBrainConfig = {
...existingFile,
engine: opts.targetEngine,
...(opts.targetEngine === 'postgres'
? { database_url: targetConfig.database_url, database_path: undefined }
: { database_path: targetConfig.database_path, database_url: undefined }),
};
saveConfig(newConfig);
// Clean up the resume manifest — only safe once nothing is left pending.
clearManifest();
}
// Clean up
clearManifest();
console.log(`\nMigration complete. ${migrated} pages transferred.`);
console.log(`Config updated to engine: ${opts.targetEngine}`);
if (config.engine === 'pglite' && config.database_path) {
if (failures.length > 0) {
console.log(`\nMigration completed with errors. ${migrated} of ${pagesToMigrate.length} pages copied, ${failures.length} failed (${completedSet.size} already done from a prior run). See failure list above.`);
console.log(`Config NOT switched — still using engine: ${config.engine}. Re-run \`gbrain migrate --to ${opts.targetEngine}\` to retry; already-copied pages resume via the manifest.`);
} else {
console.log(`\nMigration complete. ${migrated} pages transferred.`);
console.log(`Config updated to engine: ${opts.targetEngine}`);
}
if (failures.length === 0 && config.engine === 'pglite' && config.database_path) {
console.log(`Original PGLite brain preserved at ${config.database_path} (backup).`);
}
+15 -11
View File
@@ -186,17 +186,6 @@ async function phaseBFenceFacts(
const localPathById = new Map<string, string | null>();
for (const s of sources) localPathById.set(s.id, s.local_path);
// Dirty-tree refusal: check every source's local_path before writing.
for (const [id, localPath] of localPathById) {
if (localPath && isLocalPathDirty(localPath)) {
return {
name: 'fence_facts',
status: 'failed',
detail: `source "${id}" has uncommitted changes in ${localPath}. Commit or stash, then re-run.`,
};
}
}
// Walk legacy rows in (source_id, entity_slug) groups for per-page
// atomic writes.
const legacy = await engine.executeRaw<LegacyFactRow>(
@@ -235,6 +224,21 @@ async function phaseBFenceFacts(
groups.set(key, list);
}
// Dirty-tree refusal: check ONLY the sources we are about to write
// into. A dirty tree in an unrelated source (or zero fenceable rows
// at all) must not block a no-op or a targeted backfill (#927).
const targetSourceIds = new Set([...groups.keys()].map(k => k.split('\0')[0]));
for (const id of targetSourceIds) {
const localPath = localPathById.get(id);
if (localPath && isLocalPathDirty(localPath)) {
return {
name: 'fence_facts',
status: 'failed',
detail: `source "${id}" has uncommitted changes in ${localPath}. Commit or stash, then re-run.`,
};
}
}
for (const [key, group] of groups) {
const [sourceId, entitySlug] = key.split('\0');
const localPath = localPathById.get(sourceId)!;
+5 -1
View File
@@ -142,12 +142,16 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
// --auto path: runs through the T2 library orchestrator. Hooks emit CLI
// progress to stderr; the final result lands as JSON on stdout (or human
// summary).
// summary). extraRemediations (gathered above from runAllOnboardChecks)
// is threaded into the runner so the onboard-check remediations
// (extract-ner, extract-timeline-from-meetings, etc.) reach the planner
// — the same wiring the --check path uses above.
const result = await runRemediation(
engine,
{
targetScore,
maxUsd,
extraRemediations,
// --auto --yes opts into the prompt_required tier too; library
// doesn't distinguish auto_apply vs prompt_required, it just runs
// every remediation in the plan. The plan-building side (T12 render)
+99 -8
View File
@@ -45,6 +45,7 @@ import {
type IngestionContentType,
type IngestionEvent,
} from '../core/ingestion/types.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
/**
* /health endpoint timeout. 3s rather than 5s: Fly.io's default
@@ -112,6 +113,24 @@ export function shouldSuppressBootstrapPrint(opts: {
return !opts.isTty;
}
export type OAuthTokenRateLimitConfig = {
windowMs: number;
max: number;
};
function parsePositiveIntEnv(value: string | undefined, fallback: number): number {
if (value === undefined) return fallback;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}
export function resolveOAuthTokenRateLimit(env: NodeJS.ProcessEnv = process.env): OAuthTokenRateLimitConfig {
return {
windowMs: parsePositiveIntEnv(env.GBRAIN_OAUTH_TOKEN_RATE_LIMIT_WINDOW_MS, 15 * 60 * 1000),
max: parsePositiveIntEnv(env.GBRAIN_OAUTH_TOKEN_RATE_LIMIT_MAX, 50),
};
}
export type ProbeHealthResult =
| { ok: true; status: 200; body: { status: 'ok'; version: string; engine: string; [k: string]: unknown } }
| { ok: false; status: 503; body: { error: 'service_unavailable'; error_description: string } };
@@ -430,6 +449,34 @@ export function skillPublishStatus(publishSkills: boolean): { bannerValue: strin
};
}
/**
* #1196: startup embedding-width guard for stateless host deployments.
*
* `embedding_model` / `embedding_dimensions` are file/env-plane only, so a
* container booted WITHOUT a config.json (stateless host) resolves the
* compiled-in default embedding width. Against an existing brain whose
* `content_chunks.embedding` is a different `vector(N)`, every write then
* fails with an opaque dim mismatch. Run doctor's existing
* embedding_width_consistency check at serve startup and return a loud
* banner (with the paste-ready recipe) when it isn't ok. Fail-open: a check
* error never blocks serving read traffic.
*/
export async function embeddingWidthStartupWarning(engine: BrainEngine): Promise<string | null> {
try {
const { checkEmbeddingWidthConsistency } = await import('./doctor.ts');
const check = await checkEmbeddingWidthConsistency(engine);
if (check.status === 'ok') return null;
return (
`[serve-http] WARNING: embedding width check failed — writes that embed will fail until fixed.\n` +
`${check.message}\n` +
`Stateless hosts: embedding_model/embedding_dimensions resolve from env/config.json only — ` +
`set GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS (or mount config.json) to match the brain's schema.`
);
} catch {
return null;
}
}
export async function runServeHttp(engine: BrainEngine, options: ServeHttpOptions) {
const { port, tokenTtl, enableDcr, enableDcrInsecure, publicUrl, logFullParams } = options;
// v0.34.1 (#864, D11): default bind flipped from 0.0.0.0 to 127.0.0.1.
@@ -454,6 +501,14 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
);
}
// #1196: fail-loud at startup when the resolved embedding width diverges
// from the brain's actual vector(N) column (stateless containers falling
// through to the compiled-in default). Non-fatal: reads still work.
{
const widthWarn = await embeddingWidthStartupWarning(engine);
if (widthWarn) console.error(widthWarn);
}
// Skill-publishing status for the banner + nudge. Mirrors readMcpPublishSkills
// (skill-catalog.ts): the DB plane (`gbrain config set`) wins over the file
// plane. When OFF, a connected coding agent can't see the host's skill
@@ -632,12 +687,13 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// Custom client_credentials handler (before mcpAuthRouter)
// SDK's token handler only supports authorization_code and refresh_token
// ---------------------------------------------------------------------------
const oauthTokenRateLimit = resolveOAuthTokenRateLimit();
const ccRateLimiter = rateLimit({
windowMs: 15 * 60 * 1000,
max: 50,
windowMs: oauthTokenRateLimit.windowMs,
max: oauthTokenRateLimit.max,
standardHeaders: true,
legacyHeaders: false,
message: { error: 'too_many_requests', error_description: 'Rate limit exceeded. Try again in 15 minutes.' },
message: { error: 'too_many_requests', error_description: 'Rate limit exceeded. Try again later.' },
});
// Magic-link rate limiter: 10 requests/min/IP. The bootstrap token is
@@ -1205,7 +1261,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
app.get('/admin/api/calibration/pattern/:id', requireAdmin, async (req: Request, res: Response) => {
try {
const { getLatestProfile } = await import('./calibration.ts');
const holder = (req.query.holder as string) || 'garry';
const holder = resolveOwnerHolder({ override: (req.query.holder as string) || undefined, configValue: await engine.getConfig('emotional_weight.user_holder') });
const profile = await getLatestProfile(engine, { holder });
if (!profile) {
res.status(404).json({ error: 'no_profile' });
@@ -1255,7 +1311,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
app.get('/admin/api/calibration/profile', requireAdmin, async (req: Request, res: Response) => {
try {
const { getLatestProfile } = await import('./calibration.ts');
const holder = (req.query.holder as string) || 'garry';
const holder = resolveOwnerHolder({ override: (req.query.holder as string) || undefined, configValue: await engine.getConfig('emotional_weight.user_holder') });
const profile = await getLatestProfile(engine, { holder });
res.json(profile);
} catch (err) {
@@ -1272,7 +1328,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
renderAbandonedThreadsCard,
renderPatternStatementsCard,
} = await import('../core/calibration/svg-renderer.ts');
const holder = (req.query.holder as string) || 'garry';
const holder = resolveOwnerHolder({ override: (req.query.holder as string) || undefined, configValue: await engine.getConfig('emotional_weight.user_holder') });
const type = req.params.type;
const profile = await getLatestProfile(engine, { holder });
@@ -1511,6 +1567,38 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
}
});
// v0.42.x (#1914): rescope an OAuth client's write source / federated read
// scope. Admin-gated on purpose — DCR clients must never self-widen their
// scope (fail-closed trust); only the operator rescopes, here or via
// `gbrain auth rescope-client`. Source ids are validated by the canonical
// validator inside rescopeClient.
app.post('/admin/api/rescope-client', requireAdmin, express.json(), async (req: Request, res: Response) => {
try {
const { clientId, sourceId, federatedRead } = req.body ?? {};
if (!clientId || typeof clientId !== 'string') {
res.status(400).json({ error: 'clientId required' });
return;
}
if (federatedRead !== undefined &&
!(Array.isArray(federatedRead) && federatedRead.every((s: unknown) => typeof s === 'string'))) {
res.status(400).json({ error: 'federatedRead must be an array of source id strings' });
return;
}
if (sourceId !== undefined && typeof sourceId !== 'string') {
res.status(400).json({ error: 'sourceId must be a string' });
return;
}
const result = await oauthProvider.rescopeClient(clientId, { sourceId, federatedRead });
res.json(result);
} catch (e) {
const message = e instanceof Error ? e.message : 'Rescope failed';
const status = /No OAuth client found/.test(message) ? 404
: /Invalid source_id|requires --source|cannot be empty|does not exist/.test(message) ? 400
: 500;
res.status(status).json({ error: message });
}
});
// Revoke OAuth client
app.post('/admin/api/revoke-client', requireAdmin, express.json(), async (req: Request, res: Response) => {
try {
@@ -2161,8 +2249,10 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
// Other event types (ping, pull_request, etc.) return 202 'ignored'
// so GitHub doesn't retry.
// D15.5: HMAC compare uses the shared safeHexEqual helper.
// D18: submits 'sync' job with auto_embed_backfill=true and priority -10
// (above autopilot's 0).
// D18: submits 'sync' job with extraction + auto_embed_backfill enabled and
// priority -10 (above autopilot's 0). This opts normal incremental pushes
// into sync's inline extraction while pagesAffected still identifies the
// changed pages. The sync core can still defer large (>100) changes.
// ---------------------------------------------------------------------------
const githubWebhookLimiter = rateLimit({
windowMs: 60_000,
@@ -2282,6 +2372,7 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
'sync',
{
sourceId: source.id,
noExtract: false,
auto_embed_backfill: true,
embed_reason: 'webhook',
},
+68 -1
View File
@@ -9,6 +9,17 @@ import { startMcpServer } from '../mcp/server.ts';
// the dir, sees a dead PID, and removes it).
const CLEANUP_DEADLINE_MS = 5_000;
// Boot-readiness deadline (#3273). A serve process that wedges mid-boot
// (e.g. an MCP boot step that never completes because a configured
// upstream is unreachable) holds the PGLite write lock indefinitely: the
// post-#2348 lock discipline never steals from a live holder, so every
// CLI consumer times out until someone hunts down and kills the PID. If
// startMcpServer hasn't finished connecting the transport within this
// window, we release the engine (dropping the lock) and exit non-zero so
// a supervisor can restart with backoff. Env-tunable via
// GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS; 0 disables.
const DEFAULT_BOOT_TIMEOUT_SECONDS = 60;
// How often the parent-process watchdog polls the live kernel parent PID
// (via `readLiveParentPid`, NOT the cached `process.ppid` — see that
// helper's comment). We don't receive a signal when our parent dies (the
@@ -67,6 +78,10 @@ export interface ServeOptions {
// transport.onclose still cover legitimate shutdown.
// Defaults to `process.env.MCP_STDIO === '1'` when omitted.
mcpStdio?: boolean;
// Test seam for the boot-readiness deadline (#3273). Milliseconds.
// Defaults to GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS (seconds; 60 when
// unset, 0 disables) when omitted.
bootTimeoutMs?: number;
}
export async function runServe(
@@ -142,7 +157,43 @@ export async function runServe(
installStdioLifecycle(engine, args, opts);
const start = opts.startMcpServer ?? startMcpServer;
await start(engine);
// Boot-readiness deadline (#3273): never sit on the PGLite write lock
// forever with a boot that never completes. On expiry: log, release the
// engine (drops the lock), exit non-zero so supervisors restart with
// backoff. The disconnect itself is raced against CLEANUP_DEADLINE_MS,
// same as the graceful-shutdown path, so a wedged WASM close can't trap
// us either.
const bootTimeoutMs = opts.bootTimeoutMs ?? resolveBootTimeoutMs();
let bootDeadline: ReturnType<typeof setTimeout> | null = null;
if (bootTimeoutMs > 0) {
const log = opts.log ?? ((msg: string) => console.error(msg));
const exit = opts.exit ?? ((code?: number) => { process.exit(code); });
bootDeadline = setTimeout(() => {
log(
`GBrain MCP server: boot did not complete within ${bootTimeoutMs}ms — releasing DB lock and exiting so other consumers unblock (check configured provider endpoints; tune via GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS, 0 disables)`,
);
const cleanup = setTimeout(() => { exit(1); }, CLEANUP_DEADLINE_MS);
cleanup.unref?.();
Promise.resolve()
.then(() => engine.disconnect())
.catch((err: unknown) => {
const msg = err instanceof Error ? err.message : String(err);
log(`GBrain MCP server: boot-deadline cleanup error: ${msg}`);
})
.finally(() => {
clearTimeout(cleanup);
exit(1);
});
}, bootTimeoutMs);
bootDeadline.unref?.();
}
try {
await start(engine);
} finally {
if (bootDeadline) clearTimeout(bootDeadline);
}
// startMcpServer's `await server.connect(transport)` resolves once the
// SDK has wired up its stdin 'data' listener; that listener keeps the
// event loop alive. We deliberately do NOT add `await new Promise(() =>
@@ -150,6 +201,22 @@ export async function runServe(
// hooks from being able to call process.exit() cleanly.
}
// Env resolution for the boot deadline. Lenient (warn + default) rather
// than throw: this is an incident-time escape hatch, and a typo'd env var
// must not turn a boot-safety net into a boot failure of its own.
function resolveBootTimeoutMs(): number {
const raw = process.env.GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS;
if (raw === undefined || raw.trim() === '') return DEFAULT_BOOT_TIMEOUT_SECONDS * 1000;
const n = Number(raw);
if (!Number.isFinite(n) || n < 0) {
console.error(
`[gbrain serve] ignoring invalid GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS=${JSON.stringify(raw)} — using default ${DEFAULT_BOOT_TIMEOUT_SECONDS}s`,
);
return DEFAULT_BOOT_TIMEOUT_SECONDS * 1000;
}
return n * 1000;
}
interface StdioLifecycleDeps {
stdin: NodeJS.ReadableStream & { isTTY?: boolean };
signals: Pick<NodeJS.Process, 'on'>;
+170 -11
View File
@@ -213,7 +213,7 @@ export interface SyncResult {
* cron operators can disambiguate timeout vs pull-timeout in monitoring.
*/
filesImported?: number;
reason?: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
reason?: 'timeout' | 'pull_timeout' | 'pull_failed' | 'stall_timeout' | 'checkpoint_unavailable';
/**
* v0.42.x (#1794): cumulative file paths durably banked to the checkpoint
* across THIS run + prior resumed runs. Surfaced on every partial/blocked
@@ -909,6 +909,25 @@ export function buildAutoEmbedArgs(slugs: string[], sourceId?: string): string[]
return sourceId ? ['--source', sourceId, '--slugs', ...slugs] : ['--slugs', ...slugs];
}
/**
* Resolve sync's effective no-embed mode from CLI args + config.
*
* The deferred-setup sentinel (`embedding_disabled: true`, written by
* `gbrain init --no-embedding`) is an implicit `--no-embed`: without this,
* the embed credential preflight demands provider credentials the user
* deliberately deferred at init, and every `gbrain sync` on a keyless
* brain exits 1. See embed-preflight.ts's skip protocol the sentinel is
* meant to be honored before the credential check ever runs.
*
* Exported for `test/sync-no-embed-sentinel.test.ts`.
*/
export function resolveNoEmbed(
args: string[],
cfg: { embedding_disabled?: boolean } | null,
): boolean {
return args.includes('--no-embed') || cfg?.embedding_disabled === true;
}
/**
* Shell out to git with a generous maxBuffer.
*
@@ -918,12 +937,28 @@ export function buildAutoEmbedArgs(slugs: string[], sourceId?: string): string[]
*
* 100 MiB is generous but still bounded a 100K-file diff with long
* paths tops out around 1020 MiB in practice.
*
* `silenceStderr`: Node's `execFileSync` writes the child's stderr straight
* through to the parent's real stderr by default (in addition to attaching
* it to the thrown error's `.stderr`) *unless* an explicit `stdio` array is
* given. Callers that treat a failure as an expected, self-handled outcome
* (rather than a crash to surface) pass `silenceStderr: true` so git's raw
* `fatal: ...` line never reaches the process's own stderr only the
* caller's own (usually friendlier) handling of the caught error does.
* Default `false` preserves today's passthrough for every other call site.
*/
function git(repoPath: string, args: string[], configs: string[] = [], timeoutMs = 30000): string {
function git(
repoPath: string,
args: string[],
configs: string[] = [],
timeoutMs = 30000,
{ silenceStderr = false }: { silenceStderr?: boolean } = {},
): string {
return execFileSync('git', buildGitInvocation(repoPath, args, configs), {
encoding: 'utf-8',
timeout: timeoutMs,
maxBuffer: 100 * 1024 * 1024,
...(silenceStderr ? { stdio: ['ignore', 'pipe', 'pipe'] as const } : {}),
}).trim();
}
@@ -932,10 +967,19 @@ function git(repoPath: string, args: string[], configs: string[] = [], timeoutMs
* `git -C <path> rev-parse --show-toplevel`. Handles worktrees and submodules
* natively (git itself resolves them). Throws a user-friendly error when no
* git repo is found.
*
* The probe's failure is expected and routine (a non-git-yet brain dir, a
* scratch dir, a caller checking "is this a repo?") `sync.ts` self-heals
* it (git-init) or surfaces the message below, never the raw git stderr.
* `silenceStderr: true` keeps git's own `fatal: not a git repository ...`
* off the process's real stderr so operator log-scanning for `fatal:` as a
* crash signature doesn't false-alarm on every routine probe miss (#2964
* auto-recovery made the *outcome* self-healing; this keeps the *log* quiet
* about the expected miss that triggered it).
*/
export function discoverGitRoot(inputPath: string): string {
try {
return git(inputPath, ['rev-parse', '--show-toplevel']);
return git(inputPath, ['rev-parse', '--show-toplevel'], [], 30000, { silenceStderr: true });
} catch {
throw new Error(
`Not inside a git repository: ${inputPath}. GBrain sync requires a git-initialized repo (or a subdirectory of one).`,
@@ -1379,6 +1423,7 @@ See also:
{
sourceId: sourceIdArg,
repoPath: source.local_path,
noExtract: false,
auto_embed_backfill: true,
embed_reason: 'sync_trigger',
},
@@ -1717,7 +1762,7 @@ function buildPartialResult(opts: {
modified: number;
deleted: number;
renamed: number;
reason: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
reason: 'timeout' | 'pull_timeout' | 'pull_failed' | 'stall_timeout' | 'checkpoint_unavailable';
bankedFiles?: number;
}): SyncResult {
return {
@@ -1948,6 +1993,15 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
});
}
// #3068: remember a warn-and-continue pull failure. The fall-through-to-
// working-tree design stays (local commits still import when the remote is
// unreachable), but a ZERO-import sync after a failed pull must not report
// `up_to_date` / bump the freshness heartbeat — that is what made a
// permanently-failing pull (e.g. a local-path origin rejected by
// protocol.file.allow=never, #1315) invisible forever: every nightly run
// exited 0 with "Already up to date" and doctor's sync_freshness never
// fired because last_sync_at kept advancing.
let pullFailed = false;
if (!opts.noPull && !detachedHead && originRemotePresent) {
const _t0 = Date.now();
serr(`[gbrain phase] sync.git_pull start`);
@@ -1990,6 +2044,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
reason: 'pull_timeout',
});
}
pullFailed = true;
if (msg.includes('non-fast-forward') || msg.includes('diverged')) {
serr(`Warning: git pull failed (remote diverged). Syncing from local state.`);
} else {
@@ -2164,6 +2219,29 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
detachedWorkingTreeManifest.renamed.length > 0);
if (lastCommit === headCommit && !versionMismatch && !versionNeverSet && !hasDetachedWorkingTreeChanges) {
// #3068: the pull failed and nothing local advanced — this run imported
// NOTHING and the remote may hold commits we could not fetch. Reporting
// `up_to_date` here (and bumping the heartbeat below) is exactly the
// silent-wedge from the issue: every scheduled sync exits 0 forever while
// the source is stale. Return `partial` instead (not a clean status, and
// last_sync_at stays frozen so doctor/sources-status staleness fires).
// The anchor is untouched; the next sync retries the pull from the same
// bookmark.
if (pullFailed) {
serr(
`[sync] git pull failed and no local changes imported — reporting partial ` +
`(not up_to_date); sync anchor unchanged at ${lastCommit.slice(0, 8)}.`,
);
return buildPartialResult({
fromCommit: lastCommit,
toCommit: lastCommit,
filesImported: 0,
pagesAffected: [],
chunksCreated: 0,
added: 0, modified: 0, deleted: 0, renamed: 0,
reason: 'pull_failed',
});
}
// v0.42.52.0 (PR #22xx): bump last_sync_at as a heartbeat on every successful
// 0-changes sync. D4 invariant ("never advance last_commit on partial") is
// preserved: last_sync_at is a monitoring signal (doctor sync_freshness
@@ -2348,6 +2426,27 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
}
if (totalChanges === 0) {
// #3068: same guard as the git-HEAD-equality gate above — a failed pull
// plus zero imports must not produce a clean `up_to_date` (and must not
// advance the anchor past commits this run never looked at remotely).
// Reached when local-only commits landed with no syncable content while
// the pull kept failing. Nothing is written; the next sync re-diffs the
// same trivial range and retries the pull.
if (pullFailed) {
serr(
`[sync] git pull failed and no syncable changes imported — reporting partial ` +
`(not up_to_date); sync anchor unchanged at ${lastCommit.slice(0, 8)}.`,
);
return buildPartialResult({
fromCommit: lastCommit,
toCommit: lastCommit,
filesImported: 0,
pagesAffected: [],
chunksCreated: 0,
added: 0, modified: 0, deleted: 0, renamed: 0,
reason: 'pull_failed',
});
}
// Update sync state even with no syncable changes (git advanced). v0.42.x
// (#1794): advance to the PINNED target, and clear any checkpoint (a resume
// whose remaining range turned out to have no syncable changes still
@@ -2456,6 +2555,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
};
const pagesAffected: string[] = [];
// #1284: slugs deleted this run (delete loop, or renamed-away old slugs are
// NOT pushed — only confirmed deletes land here). pagesAffected stays the
// full manifest for extract/report paths, but the auto-embed at the end
// must NOT be handed deleted slugs: embedPage throws 'Page not found' for
// each one and serr-logs noise. A slug re-imported later in the same run
// (delete + re-add) is removed from this set at its push site.
const deletedSlugs = new Set<string>();
// issue #1939: file paths that imported cleanly this run. The failure-ledger
// gate clears these so a previously-failing file's `attempts` streak resets
// on success (consecutive-failure semantics for the auto-skip valve).
@@ -2616,6 +2722,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// slugs (paths in filtered.deleted but with no DB row) so
// downstream extract/embed don't waste lookups.
pagesAffected.push(...deleted);
for (const s of deleted) deletedSlugs.add(s);
// v0.42.x (#1794): the whole batch is handled (deleted or already
// gone); checkpoint every path so a resume skips it.
for (const p of batch) await markCompleted(p);
@@ -2628,6 +2735,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
try {
await engine.deletePage(slugs[j], deleteScopedOpts);
pagesAffected.push(slugs[j]);
deletedSlugs.add(slugs[j]);
await markCompleted(batch[j]);
} catch (perSlugErr) {
failedFiles.push({
@@ -2655,6 +2763,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
try {
await engine.deletePage(slug, deleteOpts);
pagesAffected.push(slug);
deletedSlugs.add(slug);
await markCompleted(path);
} catch (err) {
failedFiles.push({
@@ -2755,6 +2864,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
}
}
pagesAffected.push(newSlug);
deletedSlugs.delete(newSlug); // #1284: rename landed on a previously-deleted slug → embeddable again
await markCompleted(to);
progress.tick(1, newSlug);
}
@@ -2955,6 +3065,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
deletedSlugs.delete(result.slug); // #1284: deleted-then-re-added in the same run → embeddable again
// issue #1939: record the file path (not slug) so the gate clears any
// prior failure-ledger row — success resets the auto-skip attempt streak.
succeededPaths.push(path);
@@ -3129,6 +3240,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// pin..HEAD diff. Advance to pin.
// - pin NOT an ancestor of HEAD (history REWRITE / reset / force-push) →
// the tree we imported against is gone. Block; do not advance.
let headVerificationSucceeded = false;
try {
const currentHead = git(gitContextRoot, ['rev-parse', 'HEAD']);
if (currentHead !== pin) {
@@ -3144,8 +3256,12 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
path: '<head>',
error: `git history rewritten during sync: pinned target ${pin.slice(0, 8)} is no longer an ancestor of HEAD ${currentHead.slice(0, 8)}`,
});
} else {
headVerificationSucceeded = true;
}
// else: forward progress (enrich committed on top) — safe, advance to pin.
} else {
headVerificationSucceeded = true;
}
} catch (e) {
// rev-parse failure is itself a drift signal (worktree disappeared).
@@ -3191,6 +3307,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
...succeededPaths,
...filtered.deleted,
...filtered.renamed.map(r => r.from),
// A prior transient rev-parse timeout records a hard-blocking sentinel that
// operators cannot acknowledge manually. Once pin ancestry is verified on
// a later run, clear that stale sentinel through the ordinary success path.
...(headVerificationSucceeded ? ['<head>'] : []),
];
const gate = await applySyncFailureGate({
@@ -3265,6 +3385,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// Log ingest
await engine.logIngest({
// #3242 (attribution sub-bug): credit the sync to the source it wrote
// to, not the shared 'default' bucket.
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
source_type: 'git_sync',
source_ref: `${repoPath} @ ${headCommit.slice(0, 8)}`,
pages_updated: pagesAffected,
@@ -3368,14 +3491,19 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// sync. Non-mismatch errors stay best-effort (rate limits, transient
// network) — those shouldn't break sync.
let embedded = 0;
if (!noEmbed && pagesAffected.length > 0 && pagesAffected.length <= 100) {
// #1284: never hand deleted slugs to the embedder — embedPage throws
// 'Page not found' per deleted slug and logs one error line each. Filter
// against this run's confirmed-deleted set (slugs re-imported later in the
// run were removed from it at their push sites).
const embedSlugs = pagesAffected.filter((s) => !deletedSlugs.has(s));
if (!noEmbed && embedSlugs.length > 0 && pagesAffected.length <= 100) {
try {
const { runEmbedCore } = await import('./embed.ts');
const embedOpts = opts.sourceId
? { slugs: pagesAffected, sourceId: opts.sourceId }
: { slugs: pagesAffected };
? { slugs: embedSlugs, sourceId: opts.sourceId }
: { slugs: embedSlugs };
await runEmbedCore(engine, embedOpts);
embedded = pagesAffected.length;
embedded = embedSlugs.length;
} catch (e: unknown) {
const { EmbeddingDimMismatchError } = await import('./embed.ts');
if (e instanceof EmbeddingDimMismatchError) {
@@ -3392,7 +3520,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
return {
status: 'synced',
fromCommit: lastCommit,
toCommit: headCommit,
toCommit: pin,
added: filtered.added.length,
modified: filtered.modified.length,
deleted: filtered.deleted.length,
@@ -4014,7 +4142,7 @@ See also:
const dryRun = args.includes('--dry-run');
const full = args.includes('--full');
const noPull = args.includes('--no-pull');
const noEmbed = args.includes('--no-embed');
const noEmbed = resolveNoEmbed(args, loadConfig());
const noExtract = args.includes('--no-extract'); // v0.42.7 #1696
const skipFailed = args.includes('--skip-failed');
const retryFailed = args.includes('--retry-failed');
@@ -4546,6 +4674,9 @@ See also:
status: r.status,
...(r.result ? {
sync_status: r.result.status,
// #3068: surface the partial reason (e.g. pull_failed) so JSON
// consumers can distinguish a self-healing timeout from a wedge.
...(r.result.reason ? { reason: r.result.reason } : {}),
added: r.result.added,
modified: r.result.modified,
deleted: r.result.deleted,
@@ -4567,7 +4698,14 @@ See also:
// Best-effort, stderr-only; skipped on dry-run.
if (!dryRun) await maybeExtractionNudge(engine);
if (errCount > 0) process.exit(1);
// #3068: any source wedged on a failed pull (partial/pull_failed) makes
// the whole --all run non-zero — it will not self-heal on retry, so a
// green exit would hide it from cron/monitoring. Timeout-class partials
// keep the pre-existing exit-0 behavior (they converge on retry).
const pullFailedCount = perSourceResults.filter(
(r) => r.status === 'ok' && r.result?.status === 'partial' && r.result.reason === 'pull_failed',
).length;
if (errCount > 0 || pullFailedCount > 0) process.exit(1);
return;
}
@@ -4655,6 +4793,16 @@ See also:
process.off('SIGINT', onSingleSourceSigint);
}
printSyncResult(result);
// #3068: a pull_failed partial is NOT a success — unlike timeout-class
// partials (which converge on retry), a failing pull will not self-heal.
// Exit non-zero so cron/monitoring sees the wedge instead of a green run.
// Routed through the owned verdict channel (NOT bare `process.exitCode`,
// which PGLite's Emscripten runtime clobbers mid-run — see
// src/core/cli-force-exit.ts).
if (result.status === 'partial' && result.reason === 'pull_failed') {
const { setCliExitVerdict } = await import('../core/cli-force-exit.ts');
setCliExitVerdict(1);
}
// v0.42.7 (#1696, D5): extraction-lag nudge after a completed single-source
// sync. Fire on every non-error completion (synced | first_sync | up_to_date)
// — NOT just 'synced'; a fresh/--full import (`first_sync`) is the biggest
@@ -5360,6 +5508,17 @@ function printSyncResult(result: SyncResult, sink: NodeJS.WriteStream = process.
write(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`);
break;
case 'partial':
// #3068: a failed (non-timeout) pull with zero imports gets its own
// message — "imported 0 of 0" reads like success, but the local
// checkout may be behind a remote we could not fetch.
if (result.reason === 'pull_failed') {
write(
`Sync INCOMPLETE at ${result.fromCommit?.slice(0, 8) ?? '<initial>'}: ` +
`git pull failed — the local checkout may be behind its remote.`,
);
write(` Fix the pull (see the warning above), then re-run 'gbrain sync' (last_commit unchanged; safe to retry).`);
break;
}
// v0.41.13.0 (T7 / D-V3-5): --timeout fired before the bookmark write
// so last_commit is UNCHANGED. The next sync re-walks the same diff
// and content_hash short-circuits already-imported files at ~10ms each.
+3 -2
View File
@@ -29,6 +29,7 @@ import {
} from '../core/takes-fence.ts';
import { withPageLock } from '../core/page-lock.ts';
import { resolveSourceId } from '../core/source-resolver.ts';
import { resolveOwnerHolder } from '../core/owner-holder.ts';
// --- Helpers ---
@@ -291,7 +292,7 @@ async function cmdSupersede(engine: BrainEngine, args: string[], sourceId?: stri
const pageId = await getPageId(engine, slug, sourceId);
// Read existing row to inherit kind/holder unless overridden
const existing = await engine.listTakes({ page_id: pageId, active: false, limit: 500 });
const existing = await engine.listTakes({ page_id: pageId, active: true, limit: 500 });
const target = existing.find(t => t.row_num === rowNum);
if (!target) {
console.error(`Row #${rowNum} not found on ${slug}.`);
@@ -364,7 +365,7 @@ async function cmdResolve(engine: BrainEngine, args: string[], sourceId?: string
// --evidence is the v0.30.0 alias for --source on the resolve subcommand
// (semantic clarity: "what evidence resolved this bet?").
const source = flagValue(args, '--evidence') ?? flagValue(args, '--source');
const resolvedBy = flagValue(args, '--by') ?? 'garry';
const resolvedBy = flagValue(args, '--by') ?? resolveOwnerHolder({ configValue: await engine.getConfig('emotional_weight.user_holder') });
const dirArg = flagValue(args, '--dir');
const pageId = await getPageId(engine, slug, sourceId);
+13 -3
View File
@@ -18,12 +18,22 @@
import { loadConfig } from '../config.ts';
export function hasAnthropicKey(): boolean {
if (process.env.ANTHROPIC_API_KEY) return true;
return resolveAnthropicKey() !== undefined;
}
/**
* Resolve the actual key value: env first, then the gbrain config file.
* Callers constructing an Anthropic client directly (e.g. the legacy
* subagent path) must pass this as `apiKey` a bare `new Anthropic()`
* only sees env, so launchd/MCP workers with config-stored keys fail.
*/
export function resolveAnthropicKey(): string | undefined {
if (process.env.ANTHROPIC_API_KEY) return process.env.ANTHROPIC_API_KEY;
try {
const cfg = loadConfig();
if (cfg?.anthropic_api_key) return true;
if (cfg?.anthropic_api_key) return cfg.anthropic_api_key;
} catch {
// loadConfig may throw on first-run installs; treat as no key available.
}
return false;
return undefined;
}
+7 -1
View File
@@ -9,7 +9,7 @@
* import it from `../../src/cli.ts`.
*
* The single ownership site for: (a) folding file-plane API keys
* (openai/anthropic/zeroentropy) into the gateway env, and (b) threading
* (openai/anthropic/zeroentropy/openrouter/voyage) into the gateway env, and (b) threading
* local-server `*_BASE_URL` env vars into base_urls. Both matter for the
* init-time embedding-key probe without (a) it would false-warn on
* config.json-keyed users, and without (b) a live probe could hit the wrong
@@ -38,6 +38,12 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
// config.json) must reach the openrouter recipe's OPENROUTER_API_KEY.
// process.env still wins via the later spread.
if (c.openrouter_api_key) envFromConfig.OPENROUTER_API_KEY = c.openrouter_api_key;
// #2662: same seam for Voyage. Before this, config.json's voyage_api_key
// was accepted at the file plane but never threaded into the gateway env,
// so launchd/daemon/MCP contexts (no process-env export) silently failed
// multimodal/image embeds despite config.json looking complete. process.env
// still wins via the later spread.
if (c.voyage_api_key) envFromConfig.VOYAGE_API_KEY = c.voyage_api_key;
// v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars
// into base_urls so the gateway hits the user's configured port. Without
+10 -7
View File
@@ -22,6 +22,7 @@
*/
import { resolveRecipe } from './model-resolver.ts';
import { listRecipes } from './recipes/index.ts';
import { AIConfigError } from './errors.ts';
export interface ProviderCapabilities {
@@ -77,7 +78,10 @@ export function getProviderCapabilities(modelString: string): ProviderCapabiliti
if (!chat) {
throw new AIConfigError(
`Provider "${recipe.id}" does not offer a chat touchpoint.`,
`Known providers with chat: openai, anthropic, google, openrouter, litellm-proxy, deepseek, groq, together, azure-openai, dashscope, minimax, zhipu, ollama, llama-server. Pick one for models.tier.subagent.`,
// Computed from the registry so the hint can't drift into listing
// chat-less providers (the pre-fix list falsely included embedding-only
// recipes, sending users in circles — #1157).
`Known providers with chat: ${listRecipes().filter(r => r.touchpoints.chat).map(r => r.id).join(', ')}. Pick one for models.tier.subagent.`,
);
}
@@ -88,9 +92,13 @@ export function getProviderCapabilities(modelString: string): ProviderCapabiliti
// boundary; this function returns capabilities for whatever the user asked
// for, on the assumption it'll be validated elsewhere.
const promptCache = chat.supports_prompt_cache;
return {
supportsToolCalling: chat.supports_tools === true,
supportsPromptCaching: chat.supports_prompt_cache === true,
supportsPromptCaching: typeof promptCache === 'function'
? promptCache(parsed.modelId)
: promptCache === true,
// No recipe exposes parallel-tools-specifically yet; gate on supports_tools.
// Subsequent waves can split this into its own recipe field if a provider
// ever supports tools without parallel dispatch.
@@ -101,11 +109,6 @@ export function getProviderCapabilities(modelString: string): ProviderCapabiliti
supportsThinking: false,
maxContext: chat.max_context_tokens ?? 128_000,
};
// The `parsed` binding is intentionally unused — `resolveRecipe` is called
// here for its validation side-effects (throws on unknown provider). Keeping
// the destructure makes future per-model capability overrides cheap.
void parsed;
}
/**
+45 -3
View File
@@ -90,6 +90,30 @@ export function isValidOpenAITextEmbedding3Dim(modelId: string, dims: number): b
return Number.isInteger(dims) && dims >= 1 && dims <= max;
}
// Perplexity hosted embeddings (#1046): Matryoshka-style flexible dims,
// any integer from 128 up to the model's native size. `dimensions` is the
// native wire field (no translation needed); output encoding divergence
// (base64 int8) is handled by perplexityCompatFetch in gateway.ts.
const PERPLEXITY_EMBEDDING_MAX_DIMS: Record<string, number> = {
'pplx-embed-v1-0.6b': 1024,
'pplx-embed-v1-4b': 2560,
};
export const PERPLEXITY_MIN_DIMS = 128;
export function isPerplexityEmbeddingModel(modelId: string): boolean {
return modelId in PERPLEXITY_EMBEDDING_MAX_DIMS;
}
export function maxPerplexityEmbeddingDim(modelId: string): number | undefined {
return PERPLEXITY_EMBEDDING_MAX_DIMS[modelId];
}
export function isValidPerplexityDim(modelId: string, dims: number): boolean {
const max = PERPLEXITY_EMBEDDING_MAX_DIMS[modelId];
if (max === undefined) return false;
return Number.isInteger(dims) && dims >= PERPLEXITY_MIN_DIMS && dims <= max;
}
// NVIDIA NIM hosted embedding models use asymmetric input_type values. Most
// emit fixed natural dimensions, but llama-nemotron-embed-1b-v2 accepts
// Matryoshka-style dimension overrides (e.g. matching an existing 1280d
@@ -226,6 +250,23 @@ export function dimsProviderOptions(
},
};
}
// Perplexity pplx-embed-v1-* — flexible dims via the native
// `dimensions` field. Fail-loud when the configured dim is outside
// the model's range (same rationale as the Voyage/ZE guards: the
// upstream HTTP 400 misroutes as a transient network error).
// Symmetric retrieval — inputType is never emitted.
if (isPerplexityEmbeddingModel(modelId)) {
if (!isValidPerplexityDim(modelId, dims)) {
const max = maxPerplexityEmbeddingDim(modelId)!;
throw new AIConfigError(
`Perplexity model "${modelId}" supports embedding_dimensions in ` +
`${PERPLEXITY_MIN_DIMS}..${max}, got ${dims}.`,
`Set \`embedding_dimensions\` to a value between ${PERPLEXITY_MIN_DIMS} and ${max} ` +
`in your gbrain config.`,
);
}
return { openaiCompatible: { dimensions: dims } };
}
// NVIDIA NIM hosted embeddings are OpenAI-compatible but require
// asymmetric input_type. Use passage for indexing/document-side vectors
// and query for search-side vectors. Only llama-nemotron-embed-1b-v2
@@ -244,9 +285,10 @@ export function dimsProviderOptions(
// configured for a smaller width (e.g. 1536) hard-fail at first embed.
// Azure/OpenAI-compat embeddings are symmetric — inputType ignored.
// v0.36.0.0 (D13): same range validation as native-openai path.
if (modelId.startsWith('text-embedding-3')) {
if (isOpenAITextEmbedding3Model(modelId) && !isValidOpenAITextEmbedding3Dim(modelId, dims)) {
const max = maxOpenAITextEmbedding3Dim(modelId)!;
const bareModelId = modelId.includes('/') ? modelId.split('/').pop()! : modelId;
if (bareModelId.startsWith('text-embedding-3')) {
if (isOpenAITextEmbedding3Model(bareModelId) && !isValidOpenAITextEmbedding3Dim(bareModelId, dims)) {
const max = maxOpenAITextEmbedding3Dim(bareModelId)!;
throw new AIConfigError(
`OpenAI model "${modelId}" supports embedding_dimensions in 1..${max}, got ${dims}.`,
`Set \`embedding_dimensions\` to a value between 1 and ${max} ` +
+173 -3
View File
@@ -47,6 +47,10 @@ import type {
TouchpointKind,
} from './types.ts';
import { resolveRecipe, assertTouchpoint, parseModelId } from './model-resolver.ts';
import {
OPENROUTER_CACHE_HEADER,
openrouterRequiresExplicitPromptCache,
} from './recipes/openrouter.ts';
import { resolveModel, TIER_DEFAULTS } from '../model-config.ts';
import type { BrainEngine } from '../engine.ts';
import { dimsProviderOptions } from './dims.ts';
@@ -263,6 +267,18 @@ export class ZeroEntropyResponseTooLargeError extends Error {
}
}
/** Perplexity twin of the Voyage/ZE OOM caps (#1046). Int8 components are
* 1 byte each, so a real response (512 texts × 2560 dims) is ~1.3 MB
* anything near this cap is unambiguously not legitimate. */
const MAX_PERPLEXITY_RESPONSE_BYTES = 256 * 1024 * 1024;
export class PerplexityResponseTooLargeError extends Error {
constructor(message: string) {
super(message);
this.name = 'PerplexityResponseTooLargeError';
}
}
// ---- Unified auth resolution (D12=A) ----
//
// Pre-v0.32, openai-compatible auth was duplicated across instantiateEmbedding,
@@ -1294,6 +1310,103 @@ const openAICompatAsymmetricFetch = (async (input: RequestInfo | URL, init?: Req
return fetch(typeof input === 'string' ? input : input.toString(), baseInit);
}) as unknown as typeof fetch;
/**
* Perplexity compatibility shim (#1046). Perplexity's `/v1/embeddings`
* endpoint is OpenAI-shaped but diverges on two points that break the AI
* SDK's openai-compatible adapter:
* - `encoding_format` only accepts 'base64_int8' (default) or
* 'base64_binary'; the SDK sends 'float', which Perplexity rejects.
* Force 'base64_int8' on the wire.
* - The response `embedding` is a base64 string encoding SIGNED INT8
* components (natively quantized output). The SDK schema expects
* `number[]` decode Int8Array number[] here. Cosine similarity is
* scale-invariant, so the raw int8 components rank correctly.
* `dimensions` is Perplexity's native field name no translation needed
* (dims.ts emits it directly). Layer 1/Layer 2 OOM caps mirror the Voyage
* pattern.
*
* Exported for tests (behavioral coverage of the int8 decode); not part of
* the public gateway API.
*/
export const perplexityCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
// OUTBOUND: force the encoding Perplexity actually accepts.
if (init?.body && typeof init.body === 'string') {
try {
const parsed = JSON.parse(init.body);
if (parsed && typeof parsed === 'object' && parsed.encoding_format !== 'base64_int8') {
parsed.encoding_format = 'base64_int8';
// Drop Content-Length so fetch recomputes from the new body.
const headers = new Headers(init.headers ?? {});
headers.delete('content-length');
init = { ...init, body: JSON.stringify(parsed), headers };
}
} catch {
// Body wasn't JSON — pass through untouched.
}
}
const resp = await fetch(input as any, init);
if (!resp.ok) return resp;
const ct = resp.headers.get('content-type') ?? '';
if (!ct.toLowerCase().includes('application/json')) return resp;
// Layer 1: Content-Length pre-check BEFORE the body is parsed.
const contentLengthHeader = resp.headers.get('content-length');
if (contentLengthHeader) {
const len = parseInt(contentLengthHeader, 10);
if (Number.isFinite(len) && len > MAX_PERPLEXITY_RESPONSE_BYTES) {
throw new PerplexityResponseTooLargeError(
`Perplexity response Content-Length=${len} exceeds ${MAX_PERPLEXITY_RESPONSE_BYTES} bytes — ` +
`likely compromised endpoint or misconfiguration`,
);
}
}
// INBOUND: decode base64 int8 embeddings to number[] so the SDK's Zod
// schema validates.
try {
const json: any = await resp.clone().json();
if (!json || typeof json !== 'object') return resp;
let modified = false;
if (Array.isArray(json.data)) {
for (const item of json.data) {
if (item && typeof item.embedding === 'string') {
// Layer 2: per-embedding cap for chunked responses that skipped
// Layer 1. base64 → bytes is the canonical 0.75 ratio.
const estDecoded = Math.ceil(item.embedding.length * 0.75);
if (estDecoded > MAX_PERPLEXITY_RESPONSE_BYTES) {
throw new PerplexityResponseTooLargeError(
`Perplexity embedding base64 exceeds ${MAX_PERPLEXITY_RESPONSE_BYTES} bytes ` +
`(estimated ${estDecoded} bytes from ${item.embedding.length} base64 chars)`,
);
}
// base64_int8: one signed int8 per component.
const bytes = Buffer.from(item.embedding, 'base64');
item.embedding = Array.from(new Int8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength));
modified = true;
}
}
}
if (json.usage && typeof json.usage === 'object' && json.usage.prompt_tokens === undefined) {
json.usage.prompt_tokens = typeof json.usage.total_tokens === 'number'
? json.usage.total_tokens
: 0;
modified = true;
}
if (!modified) return resp;
return new Response(JSON.stringify(json), {
status: resp.status,
statusText: resp.statusText,
headers: resp.headers,
});
} catch (err) {
// OOM-cap throws MUST propagate; anything else falls back to the
// original response (same contract as voyageCompatFetch).
if (err instanceof PerplexityResponseTooLargeError) throw err;
return resp;
}
}) as unknown as typeof fetch;
async function resolveEmbeddingProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
const { parsed, recipe } = resolveRecipe(modelStr);
assertTouchpoint(recipe, 'embedding', parsed.modelId, getExtendedModelsForProvider(parsed.providerId));
@@ -1338,6 +1451,10 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon
throw new AIConfigError(
`Anthropic has no embedding model. Use openai or google for embeddings.`,
);
case 'claude-cli':
throw new AIConfigError(
`claude-cli has no embedding model. Use openai or google for embeddings.`,
);
case 'openai-compatible': {
// D12=A: unified auth via Recipe.resolveAuth (or default).
const auth = applyResolveAuth(recipe, cfg, 'embedding');
@@ -1362,6 +1479,8 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon
? zeroEntropyCompatFetch
: recipe.id === 'nvidia'
? nvidiaCompatFetch
: recipe.id === 'perplexity'
? perplexityCompatFetch
: openAICompatAsymmetricFetch);
const client = createOpenAICompatible({
name: recipe.id,
@@ -2280,6 +2399,15 @@ function instantiateExpansion(recipe: Recipe, modelId: string, cfg: AIGatewayCon
const baseURL = resolveNativeBaseUrl('anthropic', cfg);
return createAnthropic({ apiKey, ...(baseURL ? { baseURL } : {}) }).languageModel(modelId);
}
case 'claude-cli': {
// The CLI handles its own auth (OAuth session); spawn the subprocess
// directly via the same LanguageModelV2 implementation chat uses. There
// is no separate expansion path because claude-cli does not declare a
// separate expansion touchpoint — but routing here keeps the switch
// exhaustive and lets a future expansion touchpoint use the same code.
const { ClaudeCliLanguageModel } = require('./providers/claude-cli-language-model.ts');
return new ClaudeCliLanguageModel(modelId);
}
case 'openai-compatible': {
// D12=A: unified auth via Recipe.resolveAuth (or default).
const auth = applyResolveAuth(recipe, cfg, 'expansion');
@@ -2735,6 +2863,17 @@ export function probeChatModel(modelStr: string): ChatModelProbe {
return { ok: true };
}
/**
* Per-model prompt-cache capability: `supports_prompt_cache` may be a static
* boolean (native providers) or a per-model-id predicate (OpenRouter's
* family-scoped caching).
*/
function chatSupportsPromptCache(recipe: Recipe, modelId: string): boolean {
const support = recipe.touchpoints.chat?.supports_prompt_cache;
if (typeof support === 'function') return support(modelId);
return support === true;
}
async function resolveChatProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
const { parsed, recipe } = resolveRecipe(modelStr);
assertTouchpoint(recipe, 'chat', parsed.modelId, getExtendedModelsForProvider(parsed.providerId));
@@ -2768,6 +2907,15 @@ function instantiateChat(recipe: Recipe, modelId: string, cfg: AIGatewayConfig):
const baseURL = resolveNativeBaseUrl('anthropic', cfg);
return createAnthropic({ apiKey, ...(baseURL ? { baseURL } : {}) }).languageModel(modelId);
}
case 'claude-cli': {
// The CLI handles its own auth (OAuth session managed by `claude`
// login). Subprocess-based LanguageModelV2 dispatches via the recipe
// path so per-call routing works: `claude-cli:claude-sonnet-4-6` lands
// here, while sibling `litellm:gpt-5.4` continues through the
// openai-compatible path below. No env-var switch, no global flag.
const { ClaudeCliLanguageModel } = require('./providers/claude-cli-language-model.ts');
return new ClaudeCliLanguageModel(modelId);
}
case 'openai-compatible': {
// D12=A: unified auth via Recipe.resolveAuth (or default).
const auth = applyResolveAuth(recipe, cfg, 'chat');
@@ -3056,9 +3204,19 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
const { model, recipe, modelId } = await resolveChatProvider(modelStr);
const cfg = requireConfig();
const supportsCache = recipe.touchpoints.chat?.supports_prompt_cache === true;
const supportsCache = chatSupportsPromptCache(recipe, modelId);
const useCache = !!opts.cacheSystem && supportsCache;
// OpenRouter Claude routes need an explicit `cache_control` on the system
// content block, but the openai-compatible adapter drops anthropic-namespace
// providerOptions before building the wire body. Signal intent via a private
// header; the recipe's compat fetch shim rewrites the body and strips the
// header before the request leaves the process. OpenAI routes through
// OpenRouter cache automatically — no marker needed.
const requestHeaders = useCache && recipe.id === 'openrouter' && openrouterRequiresExplicitPromptCache(modelId)
? { [OPENROUTER_CACHE_HEADER]: '1' }
: undefined;
const tools = toAISDKTools(opts.tools);
const providerOptions: Record<string, any> = {};
@@ -3170,6 +3328,7 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
// shorter wins). Covers native-anthropic (the default provider + facts Haiku).
abortSignal: withDefaultTimeout(opts.abortSignal, AI_CHAT_TIMEOUT_MS),
providerOptions: Object.keys(providerOptions).length > 0 ? providerOptions : undefined,
...(requestHeaders ? { headers: requestHeaders } : {}),
});
// Normalize blocks. Vercel SDK gives us `result.content` (an array of typed
@@ -3218,7 +3377,10 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
usage: {
input_tokens: inTok,
output_tokens: outTok,
cache_read_tokens: Number(anthropicCache.cacheReadInputTokens ?? anthropicCache.cache_read_input_tokens ?? 0),
// `usage.cachedInputTokens` is the AI SDK's provider-neutral cache-read
// count — it's how OpenAI-compatible routes (OpenRouter's
// prompt_tokens_details.cached_tokens) surface cache hits.
cache_read_tokens: Number(anthropicCache.cacheReadInputTokens ?? anthropicCache.cache_read_input_tokens ?? usage.cachedInputTokens ?? 0),
cache_creation_tokens: Number(anthropicCache.cacheCreationInputTokens ?? anthropicCache.cache_creation_input_tokens ?? 0),
},
model: `${recipe.id}:${modelId}`,
@@ -3682,7 +3844,15 @@ export async function rerank(input: RerankInput): Promise<RerankResult[]> {
// whose request/response shape differs from ZE/llama.cpp (e.g. Voyage with
// `top_k` / `data[]`) needs separate adapter hooks in a follow-up plan.
const url = `${compat.baseURL.replace(/\/$/, '')}${tp.path ?? '/models/rerank'}`;
const auth = applyResolveAuth(recipe, cfg, 'reranker');
let auth: { apiKey?: string; headers?: Record<string, string> };
try {
auth = applyResolveAuth(recipe, cfg, 'reranker');
} catch (err) {
if (err instanceof AIConfigError) {
throw new RerankError(err.message, 'auth');
}
throw err;
}
// applyResolveAuth returns { apiKey } for Bearer-style auth (SDK's native
// path) or { headers } for custom-header providers (Azure). v0.37.6.0:
// recipes can ALSO declare default_headers (attribution etc.) which flow
@@ -0,0 +1,444 @@
/**
* ai-sdk LanguageModelV2 implementation that dispatches via the `claude --print`
* CLI subprocess. Used by the `claude-cli` recipe to route gateway.toolLoop /
* gateway.chat calls through Claude Code's OAuth session instead of the
* Anthropic SDK + ANTHROPIC_API_KEY.
*
* Per-call routing is the contract: the gateway resolves the model string
* to this recipe based on the `claude-cli:` prefix, instantiates one of
* these objects per modelId, and dispatches doGenerate. Sibling subagent
* jobs with `litellm:gpt-5.4` continue routing through litellm-proxy in
* the same worker; no env-var switch, no global state.
*
* Tool use is supported via system-prompt-instructed JSON emission:
* The recipe injects a fenced instruction block into the system prompt
* that teaches the model the `<use_tools>[{id,name,input}, ...]</use_tools>`
* emission format. The adapter parses those blocks back into ai-sdk
* `tool-call` content parts. Parallel tool calls (multiple entries in
* the JSON array) round-trip cleanly this is the case that breaks
* on the codex-proxy / litellm GPT-5.x bridge today.
*
* Context isolation:
* The subprocess is spawned from a dedicated tmpdir so claude-cli's
* CLAUDE.md auto-discovery has no local files to find. `--system-prompt`
* replaces the default system prompt; `--disable-slash-commands` skips
* skill resolution. User-level ~/.claude/CLAUDE.md still loads because
* the only way to skip it is `--bare`, which forces ANTHROPIC_API_KEY
* auth and defeats the whole point of this provider. The ~42k cached
* tokens from user-level instructions are accepted as a cost-trivial
* trade-off on the subscription path.
*
* doStream is not yet implemented; the model declares no streaming. Callers
* (gateway.toolLoop primarily) use doGenerate.
*/
import { spawn } from 'node:child_process';
import { mkdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import type {
LanguageModelV2,
LanguageModelV2CallOptions,
LanguageModelV2Content,
LanguageModelV2FunctionTool,
LanguageModelV2Prompt,
LanguageModelV2Message,
LanguageModelV2ProviderDefinedTool,
} from '@ai-sdk/provider';
function claudeBin(): string {
return process.env.GBRAIN_CLAUDE_CLI_BIN ?? 'claude';
}
const CLAUDE_CWD = join(tmpdir(), `gbrain-claude-cli-cwd-${process.pid}`);
let cwdEnsured = false;
function ensureCleanCwd(): string {
if (!cwdEnsured) {
mkdirSync(CLAUDE_CWD, { recursive: true });
cwdEnsured = true;
}
return CLAUDE_CWD;
}
/** Parsed shape of `claude --print --output-format json`. */
interface ClaudeJsonResult {
type: 'result';
subtype: 'success' | string;
is_error: boolean;
result: string;
stop_reason: string | null;
session_id: string;
num_turns: number;
usage?: {
input_tokens?: number;
output_tokens?: number;
cache_read_input_tokens?: number;
cache_creation_input_tokens?: number;
};
}
/**
* Build the system-prompt addendum that teaches the model the
* `<use_tools>...</use_tools>` emission format. Returns the empty string
* when no tools are registered for this turn so the model gets a normal
* text-completion prompt without protocol noise.
*/
function buildToolUseInstructions(
tools: ReadonlyArray<LanguageModelV2FunctionTool | LanguageModelV2ProviderDefinedTool> | undefined,
): string {
if (!tools || tools.length === 0) return '';
const functionTools = tools.filter((t): t is LanguageModelV2FunctionTool => t.type === 'function');
if (functionTools.length === 0) return '';
const toolSpecs = functionTools.map(t => ({
name: t.name,
description: t.description ?? '',
input_schema: t.inputSchema ?? { type: 'object', properties: {} },
}));
return [
'',
'## Tool Use Protocol',
'',
'You have access to these tools:',
'',
'```json',
JSON.stringify(toolSpecs, null, 2),
'```',
'',
'To call one or more tools in this turn, emit EXACTLY ONE block of this form, ' +
'with no other text outside the block on its own lines:',
'',
'<use_tools>',
'[',
' {"id": "<unique tool call id, like toolu_01ABC>", "name": "<tool name>", "input": <input object matching the tool\'s input_schema>}',
']',
'</use_tools>',
'',
'Multiple tool calls go in the array. Tool results are returned to you on the ' +
'next turn as [tool_result <text>] entries. You may then call more tools or emit a final response.',
'',
'When you are ready to give a final answer instead of calling tools, respond with prose text only — ' +
'do not include a <use_tools> block in that case.',
'',
].join('\n');
}
/**
* Render the ai-sdk message array into a single text prompt for `claude --print`
* stdin. System messages are extracted up-front and concatenated into the
* `--system-prompt` flag value. Tool calls and tool results are rendered as
* placeholders so the model sees the conversation in a coherent shape even
* though the adapter does not natively round-trip tool calls through claude-cli.
*/
function renderPrompt(prompt: LanguageModelV2Prompt): { systemText: string; userPrompt: string } {
const systemParts: string[] = [];
const convo: string[] = [];
for (const msg of prompt as ReadonlyArray<LanguageModelV2Message>) {
if (msg.role === 'system') {
systemParts.push(msg.content);
continue;
}
if (msg.role === 'user') {
const text = msg.content
.map(p => {
if (p.type === 'text') return p.text;
// File parts get a stub — multimodal is not supported via subprocess yet.
if (p.type === 'file') return `[file ${p.mediaType ?? 'unknown'}]`;
return '';
})
.filter(s => s.length > 0)
.join('\n');
if (text) convo.push(`User: ${text}`);
continue;
}
if (msg.role === 'assistant') {
const rendered = msg.content
.map(p => {
if (p.type === 'text') return p.text;
if (p.type === 'reasoning') return ''; // dropped on replay
if (p.type === 'tool-call') {
return `[tool_use ${p.toolName}(${p.input})]`;
}
if (p.type === 'tool-result') {
const out = typeof p.output === 'string' ? p.output : JSON.stringify(p.output);
return `[tool_result ${out}]`;
}
return '';
})
.filter(s => s.length > 0)
.join('\n');
if (rendered) convo.push(`Assistant: ${rendered}`);
continue;
}
if (msg.role === 'tool') {
const rendered = msg.content
.map(p => {
const out = typeof p.output === 'string' ? p.output : JSON.stringify(p.output);
return `[tool_result ${out}]`;
})
.join('\n');
if (rendered) convo.push(`User: ${rendered}`);
continue;
}
}
return { systemText: systemParts.join('\n'), userPrompt: convo.join('\n\n') };
}
/**
* Spawn `claude --print` with the contamination-suppression flags and return
* the parsed `--output-format json` envelope. Aborts propagate to SIGTERM on
* the child.
*/
function runClaude(
systemPrompt: string,
userPrompt: string,
model: string,
signal?: AbortSignal,
): Promise<ClaudeJsonResult> {
return new Promise((resolve, reject) => {
const args = [
'--print',
'--output-format', 'json',
'--model', model,
'--disable-slash-commands',
// Agent isolation: this subprocess must behave like a raw LLM, not a
// full Claude Code agent. `--tools ""` disables every built-in tool
// (Bash/Read/WebSearch/...); `--strict-mcp-config` ignores all user-level
// MCP servers (without it, each call would boot the user's MCP servers —
// including gbrain's own MCP → recursion + PGLite single-writer lock
// contention). Verified against claude CLI 2.1.145 --help.
'--tools', '',
'--strict-mcp-config',
];
if (systemPrompt) {
args.push('--system-prompt', systemPrompt);
}
// Env scrub: guarantee the CLI authenticates via its own OAuth session
// (subscription), never via an inherited API key. Without this, an
// ANTHROPIC_API_KEY in gbrain's env (the exact setup this recipe is meant
// to replace) silently flips billing to per-token API usage.
const env = { ...process.env };
delete env.ANTHROPIC_API_KEY;
delete env.ANTHROPIC_AUTH_TOKEN;
delete env.ANTHROPIC_BASE_URL;
const child = spawn(claudeBin(), args, {
stdio: ['pipe', 'pipe', 'pipe'],
cwd: ensureCleanCwd(),
env,
});
let stdout = '';
let stderr = '';
child.stdout.on('data', chunk => { stdout += String(chunk); });
child.stderr.on('data', chunk => { stderr += String(chunk); });
const onAbort = () => {
child.kill('SIGTERM');
reject(new Error('claude-cli adapter aborted'));
};
if (signal) {
if (signal.aborted) {
onAbort();
return;
}
signal.addEventListener('abort', onAbort, { once: true });
}
child.on('error', err => {
if (signal) signal.removeEventListener('abort', onAbort);
reject(new Error(`claude-cli spawn failed: ${err instanceof Error ? err.message : String(err)}`));
});
child.on('close', code => {
if (signal) signal.removeEventListener('abort', onAbort);
if (code !== 0) {
reject(new Error(`claude-cli exited ${code}: ${stderr.trim() || stdout.trim()}`));
return;
}
try {
let parsed = JSON.parse(stdout) as unknown;
// Compat: when the user has `"verbose": true` in ~/.claude/settings.json,
// `--print --output-format json` emits an ARRAY of events
// ([{type:"system",subtype:"init",...}, ..., {type:"result",...}])
// instead of the bare result object. There is no CLI flag to force it
// off (no --no-verbose; --settings '{}' merges, does not replace), so
// tolerate both shapes and pick the result event. Verified on CLI 2.1.145.
if (Array.isArray(parsed)) {
const resultEvent = parsed.find(
(ev): ev is ClaudeJsonResult =>
!!ev && typeof ev === 'object' && (ev as { type?: unknown }).type === 'result',
);
if (!resultEvent) {
reject(new Error(`claude-cli JSON event array had no "result" event\n--- raw ---\n${stdout.slice(0, 500)}`));
return;
}
parsed = resultEvent;
}
const envelope = parsed as ClaudeJsonResult;
if (envelope.is_error) {
reject(new Error(`claude-cli reported error: ${envelope.result || envelope.subtype}`));
return;
}
resolve(envelope);
} catch (e) {
reject(new Error(`claude-cli output not JSON: ${e instanceof Error ? e.message : String(e)}\n--- raw ---\n${stdout.slice(0, 500)}`));
}
});
// stdin error handler: if the binary does not exist (ENOENT) or the child
// dies before draining stdin, write/end can emit an unhandled 'error'
// (EPIPE) that would crash the worker. The spawn-level 'error' / non-zero
// 'close' handlers above already surface the real failure, so the stdin
// error itself is safe to swallow.
child.stdin.on('error', () => { /* surfaced via child 'error'/'close' */ });
try {
child.stdin.write(userPrompt);
child.stdin.end();
} catch (e) {
if (signal) signal.removeEventListener('abort', onAbort);
reject(new Error(`claude-cli stdin write failed (is the claude binary installed?): ${e instanceof Error ? e.message : String(e)}`));
}
});
}
interface ParsedToolCall {
id: string;
name: string;
/** Stringified JSON, matching the ai-sdk LanguageModelV2ToolCall.input contract. */
input: string;
}
/**
* Locate and parse the `<use_tools>...</use_tools>` block in the assistant's
* raw text response. Returns the parsed tool calls plus whatever prose
* surrounded the block. Returns an empty `toolCalls` array when no block is
* present, malformed, or unterminated the caller then treats the full
* raw text as a final text response.
*/
function extractToolCalls(raw: string): {
toolCalls: ParsedToolCall[];
beforeText: string;
afterText: string;
} {
const openTag = '<use_tools>';
const closeTag = '</use_tools>';
const openIdx = raw.indexOf(openTag);
if (openIdx === -1) {
return { toolCalls: [], beforeText: raw.trim(), afterText: '' };
}
const closeIdx = raw.indexOf(closeTag, openIdx + openTag.length);
if (closeIdx === -1) {
// Unterminated block — recover gracefully.
return { toolCalls: [], beforeText: raw.trim(), afterText: '' };
}
const beforeText = raw.slice(0, openIdx).trim();
const afterText = raw.slice(closeIdx + closeTag.length).trim();
let inner = raw.slice(openIdx + openTag.length, closeIdx).trim();
if (inner.startsWith('```')) {
inner = inner.replace(/^```(?:json|JSON)?\s*\n?/, '').replace(/\n?```$/, '').trim();
}
let parsed: unknown;
try {
parsed = JSON.parse(inner);
} catch {
return { toolCalls: [], beforeText: raw.trim(), afterText: '' };
}
if (!Array.isArray(parsed)) {
return { toolCalls: [], beforeText: raw.trim(), afterText: '' };
}
const toolCalls: ParsedToolCall[] = [];
for (const entry of parsed) {
if (!entry || typeof entry !== 'object') continue;
const e = entry as Record<string, unknown>;
const name = typeof e.name === 'string' ? e.name : null;
if (!name) continue;
const id = typeof e.id === 'string' && e.id.length > 0
? e.id
: `toolu_claude_cli_${Math.random().toString(36).slice(2, 12)}`;
const inputJson = JSON.stringify(e.input ?? {});
toolCalls.push({ id, name, input: inputJson });
}
return { toolCalls, beforeText, afterText };
}
/**
* Strip provider prefixes (`anthropic:`, `litellm:`, `claude-cli:`) that the
* underlying CLI does not understand. The gateway hands us a bare model id
* via `recipe.aliases` resolution, but defensive normalization here keeps
* direct LanguageModelV2 construction (in tests, for example) ergonomic.
*/
function normalizeModel(model: string): string {
const idx = model.indexOf(':');
return idx >= 0 ? model.slice(idx + 1) : model;
}
export class ClaudeCliLanguageModel implements LanguageModelV2 {
readonly specificationVersion = 'v2' as const;
readonly provider = 'claude-cli';
readonly modelId: string;
readonly supportedUrls = {};
constructor(modelId: string) {
this.modelId = normalizeModel(modelId);
}
async doGenerate(options: LanguageModelV2CallOptions): Promise<{
content: LanguageModelV2Content[];
finishReason: 'stop' | 'length' | 'content-filter' | 'tool-calls' | 'error' | 'other' | 'unknown';
usage: { inputTokens: number | undefined; outputTokens: number | undefined; totalTokens: number | undefined };
warnings: never[];
}> {
const { systemText, userPrompt } = renderPrompt(options.prompt);
const toolInstructions = buildToolUseInstructions(options.tools);
const systemPrompt = [systemText, toolInstructions].filter(s => s.length > 0).join('\n');
const result = await runClaude(systemPrompt, userPrompt, this.modelId, options.abortSignal);
const { toolCalls, beforeText, afterText } = extractToolCalls(result.result);
const content: LanguageModelV2Content[] = [];
if (beforeText) content.push({ type: 'text', text: beforeText });
for (const call of toolCalls) {
content.push({
type: 'tool-call',
toolCallId: call.id,
toolName: call.name,
input: call.input,
});
}
if (afterText) content.push({ type: 'text', text: afterText });
if (content.length === 0) {
// Empty response — still hand the caller a well-formed content array.
content.push({ type: 'text', text: result.result ?? '' });
}
const finishReason = toolCalls.length > 0 ? 'tool-calls' as const : 'stop' as const;
const inputTokens = result.usage?.input_tokens;
const outputTokens = result.usage?.output_tokens;
const totalTokens = (inputTokens ?? 0) + (outputTokens ?? 0);
return {
content,
finishReason,
usage: {
inputTokens,
outputTokens,
totalTokens: inputTokens !== undefined && outputTokens !== undefined ? totalTokens : undefined,
},
warnings: [],
};
}
async doStream(): Promise<never> {
throw new Error(
'claude-cli LanguageModel does not support streaming. Use doGenerate or set ' +
'the model on a non-streaming chat surface (gateway.toolLoop is non-streaming).',
);
}
}
+71
View File
@@ -0,0 +1,71 @@
import type { Recipe } from '../types.ts';
/**
* Claude via the local `claude` CLI binary, using its built-in OAuth session
* (Claude Code / Claude Max subscription). No ANTHROPIC_API_KEY needed the
* CLI manages its own auth state and the gateway dispatches via subprocess.
*
* Solves the #334 case where Max subscribers want Minions subagent dispatch
* to run against their existing subscription instead of paying per-token API
* charges. The recipe sits alongside the existing `anthropic` recipe so users
* pick per call: `anthropic:claude-sonnet-4-6` (API key + per-token billing)
* vs `claude-cli:claude-sonnet-4-6` (OAuth subscription, no API key).
*
* Chat-only. Claude has no first-party embedding model; users wanting an
* Anthropic chat path with embeddings still combine this with openai/google/
* voyage for embedding the way the existing `anthropic` recipe documents.
*
* Auth: `auth_env.required: []` because the CLI handles auth itself. The
* `claude` binary on PATH (or `GBRAIN_CLAUDE_CLI_BIN`) IS the auth surface;
* there is nothing for the gateway to forward.
*
* Setup expectation: `claude` CLI installed and logged in (Claude Code
* onboarding does this), or `GBRAIN_CLAUDE_CLI_BIN` pointing at the binary.
*/
export const claudeCli: Recipe = {
id: 'claude-cli',
name: 'Claude (via CLI)',
tier: 'native',
implementation: 'claude-cli',
// The CLI owns auth; no env vars are required from the gateway side.
auth_env: {
required: [],
},
touchpoints: {
// No embedding or expansion touchpoints — chat-only.
chat: {
models: [
'claude-opus-4-7',
'claude-sonnet-4-6',
'claude-haiku-4-5-20251001',
],
supports_tools: true,
supports_subagent_loop: true,
// The CLI handles caching internally and does not surface it via the
// standard cache_control control plane. From the gateway's POV the
// model does not support prompt caching.
supports_prompt_cache: false,
max_context_tokens: 200000,
// Cost figures match the underlying Claude API tier, but the actual
// bill is borne by the subscription. We report them for the budget
// ledger's per-call accounting; operators on flat-rate subscriptions
// can treat the numbers as nominal.
cost_per_1m_input_usd: 3.0,
cost_per_1m_output_usd: 15.0,
price_last_verified: '2026-06-17',
},
},
// Friendly aliases mirror the `anthropic` recipe so config strings stay
// portable: switching `anthropic:claude-sonnet-4-6` to `claude-cli:claude-sonnet-4-6`
// is a one-token edit. Reverse aliases rewrite legacy IDs back to canonical.
aliases: {
'claude-haiku-4-5': 'claude-haiku-4-5-20251001',
'claude-sonnet-4-6-20250929': 'claude-sonnet-4-6',
'sonnet': 'claude-sonnet-4-6',
'haiku': 'claude-haiku-4-5-20251001',
'opus': 'claude-opus-4-7',
},
setup_hint:
'Install Claude Code (`claude` CLI) and run `claude` once to log in. ' +
'Set GBRAIN_CLAUDE_CLI_BIN if the binary is not on PATH.',
};
+61
View File
@@ -0,0 +1,61 @@
import type { Recipe } from '../types.ts';
/**
* Alibaba DashScope () reranker. DashScope's OpenAI-compatible surface
* splits by capability: embeddings live under `/compatible-mode/v1` (see the
* sibling `dashscope` recipe) while rerank lives under `/compatible-api/v1`
* with a PLURAL leaf `POST {base}/reranks`. Wire shape matches ZeroEntropy:
* request `{model, query, documents, top_n?}`, response
* `{results: [{index, relevance_score}]}` so it rides gateway.rerank()'s
* native path with only the recipe-pluggable `path` override (v0.40.6.1).
*
* This is a SEPARATE recipe rather than a reranker touchpoint on `dashscope`
* because the two capabilities need different base URLs (`compatible-mode`
* vs `compatible-api`) and `provider_base_urls` is keyed by recipe id one
* recipe can't point embeddings and rerank at different prefixes. Same
* topology precedent as llama-server vs llama-server-reranker.
*
* Live-verified against the China endpoint (2026-07): `/reranks` with
* `qwen3-rerank` 200 `results[].relevance_score`; `/rerank` (singular)
* 404; `gte-rerank-v2` 404 "Unsupported model for OpenAI compatibility
* mode" (native-API only, so it is deliberately NOT listed here).
*
* Note: the international endpoint requires a region-aware DASHSCOPE_API_KEY.
* China-region users point at https://dashscope.aliyuncs.com/compatible-api/v1
* via `provider_base_urls['dashscope-rerank']`, mirroring the embedding
* recipe's convention.
*/
export const dashscopeRerank: Recipe = {
id: 'dashscope-rerank',
name: 'Alibaba DashScope (灵积, reranker)',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://dashscope-intl.aliyuncs.com/compatible-api/v1',
auth_env: {
required: ['DASHSCOPE_API_KEY'],
setup_url: 'https://help.aliyun.com/zh/model-studio/getting-started/',
},
touchpoints: {
reranker: {
// Only the model verified live on the OpenAI-compat /reranks surface.
// gte-rerank-v2 exists on DashScope's native API but the compat path
// rejects it ("Unsupported model for OpenAI compatibility mode").
models: ['qwen3-rerank'],
default_model: 'qwen3-rerank',
// Mirror ZE's defensive per-request ceiling; gateway.rerank()
// pre-flights body size and fails open.
max_payload_bytes: 5_000_000,
// PLURAL leaf under compatible-api — the whole reason this recipe
// exists. `${base_url}${path}` → `…/compatible-api/v1/reranks`.
path: '/reranks',
// Hosted API: no local warmup, but cross-region latency can exceed
// the 5s gateway default (same rationale as llama-server-reranker).
default_timeout_ms: 30_000,
},
},
setup_hint:
'Get an API key at https://help.aliyun.com/zh/model-studio/getting-started/, then ' +
'`export DASHSCOPE_API_KEY=...` and `gbrain config set search.reranker.model ' +
'dashscope-rerank:qwen3-rerank`. China-region accounts: `gbrain config set ' +
'provider_base_urls.dashscope-rerank https://dashscope.aliyuncs.com/compatible-api/v1`.',
};
+9
View File
@@ -76,6 +76,15 @@ export const deepseek: Recipe = {
setup_url: 'https://platform.deepseek.com/api_keys',
},
touchpoints: {
// Query expansion reuses the same OpenAI-compatible chat endpoint (the
// gateway's expansion path is a plain languageModel call). Without this
// declaration an explicit `expansion_model: deepseek:...` silently
// yields no expansion (#1135).
expansion: {
models: ['deepseek-chat'],
cost_per_1m_tokens_usd: 0.14,
price_last_verified: '2026-04-20',
},
chat: {
models: ['deepseek-chat', 'deepseek-reasoner'],
supports_tools: true,
+8
View File
@@ -16,6 +16,14 @@ export const groq: Recipe = {
setup_url: 'https://console.groq.com/keys',
},
touchpoints: {
// Same OpenAI-compatible endpoint as chat; declared so an explicit
// `expansion_model: groq:...` resolves instead of silently dropping
// expansion (#1135). 8b-instant is the natural expansion pick (cheap,
// fast, no tool-calling needed for multi-query rewrites).
expansion: {
models: ['llama-3.1-8b-instant', 'llama-3.3-70b-versatile'],
price_last_verified: '2026-04-20',
},
chat: {
models: [
'llama-3.3-70b-versatile',
+6
View File
@@ -9,6 +9,7 @@ import type { Recipe } from '../types.ts';
import { openai } from './openai.ts';
import { google } from './google.ts';
import { anthropic } from './anthropic.ts';
import { claudeCli } from './claude-cli.ts';
import { ollama } from './ollama.ts';
import { openrouter } from './openrouter.ts';
import { voyage } from './voyage.ts';
@@ -19,6 +20,7 @@ import { together } from './together.ts';
import { llamaServer } from './llama-server.ts';
import { minimax } from './minimax.ts';
import { dashscope } from './dashscope.ts';
import { dashscopeRerank } from './dashscope-rerank.ts';
import { zhipu } from './zhipu.ts';
import { azureOpenAI } from './azure-openai.ts';
import { zeroentropyai } from './zeroentropyai.ts';
@@ -26,11 +28,13 @@ import { llamaServerReranker } from './llama-server-reranker.ts';
import { moonshot } from './moonshot.ts';
import { mistral } from './mistral.ts';
import { nvidia } from './nvidia.ts';
import { perplexity } from './perplexity.ts';
const ALL: Recipe[] = [
openai,
google,
anthropic,
claudeCli,
ollama,
openrouter,
voyage,
@@ -42,12 +46,14 @@ const ALL: Recipe[] = [
llamaServerReranker,
minimax,
dashscope,
dashscopeRerank,
zhipu,
azureOpenAI,
zeroentropyai,
moonshot,
mistral,
nvidia,
perplexity,
];
/** Map from `provider:id` key to recipe. */
+112 -2
View File
@@ -1,8 +1,100 @@
import type { Recipe } from '../types.ts';
/**
* MiniMax (AI). OpenAI-compatible /embeddings endpoint at
* api.minimax.chat. The flagship embedding model is `embo-01` (1536 dims).
* MiniMax transport shim (#1977). MiniMax's `/v1/embeddings` endpoint is NOT
* OpenAI-compatible at the wire level despite the recipe's
* `implementation: 'openai-compatible'`:
* - Request: requires `texts` (the AI SDK sends `input`) plus an optional
* `type: 'db' | 'query'` asymmetric-retrieval field, and rejects OpenAI's
* `encoding_format`.
* - Response: returns `{vectors: number[][], total_tokens}` where the AI
* SDK's Zod schema expects `{data: [{embedding, index}], usage}`.
*
* Chat (`/chat/completions`) IS OpenAI-compatible, and this same fetch is
* applied to every openai-compatible touchpoint by `applyOpenAICompatConfig`,
* so everything outside the embeddings path passes through untouched and
* the response rewrite parses via `resp.clone()` only (never consume the
* body of a response we return as-is; the DeepSeek shim rule). Fail-open:
* any rewrite error returns the original request/response.
*
* @internal exported for tests.
*/
// Cast through `unknown` because Bun's `typeof fetch` carries a `preconnect`
// member the arrow function does not implement (matches deepseek.ts).
export const minimaxCompatFetch = (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === 'string' ? input : input instanceof URL ? input.href : input.url;
const isEmbeddings = url.includes('/embeddings');
// OUTBOUND (embeddings only): `input` → `texts`, default `type: 'db'`
// (the recipe's documented symmetric default — the AI SDK adapter strips
// the `type` threaded via providerOptions before it reaches the wire,
// same class as #1400), and drop `encoding_format` (not a MiniMax param).
if (isEmbeddings && init?.body && typeof init.body === 'string') {
try {
const parsed = JSON.parse(init.body);
if (
parsed && typeof parsed === 'object' &&
parsed.input !== undefined && parsed.texts === undefined
) {
parsed.texts = Array.isArray(parsed.input) ? parsed.input : [parsed.input];
delete parsed.input;
delete parsed.encoding_format;
if (parsed.type === undefined) parsed.type = 'db';
// Drop Content-Length so fetch recomputes from the new body.
const headers = new Headers(init.headers ?? {});
headers.delete('content-length');
init = { ...init, body: JSON.stringify(parsed), headers };
}
} catch {
// Body wasn't JSON — pass through untouched.
}
}
const res = await fetch(input as any, init as any);
// INBOUND (embeddings only): `{vectors: [[...]]}` → `{data: [{embedding}]}`.
// Anything else (chat completions, MiniMax base_resp errors, non-JSON)
// returns the ORIGINAL response with its body unread.
if (!isEmbeddings || !res.ok) return res;
const ctype = res.headers.get('content-type') ?? '';
if (!ctype.toLowerCase().includes('application/json')) return res;
try {
const json = await res.clone().json();
if (!json || typeof json !== 'object' || !Array.isArray(json.vectors)) return res;
const totalTokens = typeof json.total_tokens === 'number' ? json.total_tokens : 0;
const rewritten = {
object: 'list',
data: (json.vectors as number[][]).map((embedding, index) => ({
object: 'embedding',
embedding,
index,
})),
model: typeof json.model === 'string' ? json.model : 'embo-01',
usage: { prompt_tokens: totalTokens, total_tokens: totalTokens },
};
// Fresh header set: the body changed, so upstream content-length /
// content-encoding would now be wrong.
const headers = new Headers(res.headers);
headers.delete('content-length');
headers.delete('content-encoding');
return new Response(JSON.stringify(rewritten), {
status: res.status,
statusText: res.statusText,
headers,
});
} catch {
return res;
}
}) as unknown as typeof fetch;
/**
* MiniMax (AI). `/embeddings` endpoint at api.minimaxi.com (wire shape
* normalized by `minimaxCompatFetch` above); OpenAI-compatible
* `/chat/completions`. The flagship embedding model is `embo-01` (1536 dims).
*
* MiniMax's API takes an extra `type: 'db' | 'query'` field for asymmetric
* retrieval. gbrain currently has no notion of "this is a document vs a
@@ -38,7 +130,25 @@ export const minimax: Recipe = {
// halving in the gateway catches token-limit errors at runtime.
max_batch_tokens: 4096,
},
chat: {
// Model list from MiniMax's /v1/models (#1977). Chat is genuinely
// OpenAI-compatible — no wire rewrite needed (minimaxCompatFetch
// passes non-embedding requests through untouched).
models: [
'MiniMax-M3',
'MiniMax-M2.7',
'MiniMax-M2.7-highspeed',
'MiniMax-M2.5',
'MiniMax-M2.5-highspeed',
'MiniMax-M2.1',
'MiniMax-M2.1-highspeed',
'MiniMax-M2',
],
supports_tools: false,
supports_subagent_loop: false,
},
},
setup_hint:
'Get an API key at https://www.minimaxi.com, then `export MINIMAX_API_KEY=...`',
compat: { fetch: minimaxCompatFetch },
};
+100 -1
View File
@@ -1,5 +1,101 @@
import type { Recipe } from '../types.ts';
/**
* Private in-process marker header. `gateway.chat()` sets it when the caller
* asked for prompt caching (`cacheSystem`) on an OpenRouter route that needs
* an explicit `cache_control` (Anthropic Claude). The compat fetch shim below
* strips it and rewrites the body; the header NEVER leaves the process.
*
* Why a header and not providerOptions: the AI SDK's openai-compatible
* adapter validates providerOptions against a fixed schema and silently
* drops anthropic-namespace fields before building the wire body (same class
* of problem as the embedding `input_type` ALS in gateway.ts). Headers pass
* through untouched.
*/
export const OPENROUTER_CACHE_HEADER = 'x-gbrain-anthropic-prompt-cache';
/**
* Family-scoped prompt-cache capability (per OpenRouter docs):
* - OpenAI chat routes cache automatically (no request mutation needed).
* - Anthropic Claude routes cache when the request carries `cache_control`
* on a content block (applied by the fetch shim below).
* Everything else is not marked cacheable deliberately narrow rather than
* blessing every routed model family forever.
*/
export function openrouterSupportsPromptCache(modelId: string): boolean {
const normalized = modelId.trim().toLowerCase();
if (normalized.startsWith('openai/gpt-') || /^openai\/o\d/.test(normalized)) return true;
if (normalized.startsWith('anthropic/claude-')) return true;
return false;
}
/** Only Anthropic Claude routes need an explicit cache_control block. */
export function openrouterRequiresExplicitPromptCache(modelId: string): boolean {
return modelId.trim().toLowerCase().startsWith('anthropic/claude-');
}
/**
* Rewrite the last system message's string content into OpenRouter's
* documented Anthropic caching shape: a content-part array carrying
* `cache_control: { type: 'ephemeral' }` on the text block. (A top-level
* body `cache_control` is NOT the OpenRouter format OR forwards per-block
* markers only.) Returns the input unchanged when it doesn't apply.
*/
function withSystemCacheControl(body: unknown): unknown {
if (!body || typeof body !== 'object' || Array.isArray(body)) return body;
const record = body as Record<string, unknown>;
const model = typeof record.model === 'string' ? record.model : '';
if (!openrouterRequiresExplicitPromptCache(model)) return body;
const messages = Array.isArray(record.messages) ? record.messages : undefined;
if (!messages) return body;
let idx = -1;
for (let i = 0; i < messages.length; i++) {
const m = messages[i];
if (m && typeof m === 'object' && (m as Record<string, unknown>).role === 'system') idx = i;
}
if (idx === -1) return body;
const sys = messages[idx] as Record<string, unknown>;
if (typeof sys.content !== 'string' || sys.content.length === 0) return body;
const next = messages.slice();
next[idx] = {
...sys,
content: [{ type: 'text', text: sys.content, cache_control: { type: 'ephemeral' } }],
};
return { ...record, messages: next };
}
/**
* Compat fetch: honors the OPENROUTER_CACHE_HEADER marker by splicing an
* Anthropic cache_control breakpoint onto the system block, then strips the
* marker. Fail-open: any parse problem sends the original body unchanged.
*
* @internal exported for tests. Cast through `unknown` because TS's
* `typeof fetch` includes a `preconnect` member (matches azure-openai.ts).
*/
export const openrouterCompatFetch = (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
if (!init?.headers) return fetch(input as any, init as any);
const headers = new Headers(init.headers as any);
if (!headers.has(OPENROUTER_CACHE_HEADER)) return fetch(input as any, init as any);
headers.delete(OPENROUTER_CACHE_HEADER);
let body = init.body;
if (typeof body === 'string') {
try {
const parsed = JSON.parse(body);
const rewritten = withSystemCacheControl(parsed);
if (rewritten !== parsed) {
body = JSON.stringify(rewritten);
headers.delete('content-length');
}
} catch {
// Non-JSON body: let the provider surface the original problem.
}
}
return fetch(input as any, { ...init, headers, body } as any);
}) as unknown as typeof fetch;
/**
* OpenRouter single-key fan-out to OpenAI, Anthropic, Google, DeepSeek, and
* dozens of other providers via a single OpenAI-compatible endpoint at
@@ -93,7 +189,9 @@ export const openrouter: Recipe = {
supports_tools: true,
// Informational only — real gate is isAnthropicProvider() upstream.
supports_subagent_loop: false,
supports_prompt_cache: false,
// Family-scoped: OpenAI routes cache automatically; Anthropic routes
// cache via the compat fetch shim's cache_control rewrite.
supports_prompt_cache: openrouterSupportsPromptCache,
// No max_context_tokens: catalog spans 128K to 1M+; a single recipe-wide
// value is either unsafe for smaller models or wasteful for larger ones.
// Let upstream errors surface per-model.
@@ -102,4 +200,5 @@ export const openrouter: Recipe = {
},
setup_hint:
'Get an API key at https://openrouter.ai/settings/keys, then `export OPENROUTER_API_KEY=...` and use `openrouter:<provider>/<model>`. Optional overrides: OPENROUTER_BASE_URL (proxy), OPENROUTER_REFERER (attribution URL), OPENROUTER_TITLE (attribution name).',
compat: { fetch: openrouterCompatFetch },
};
+54
View File
@@ -0,0 +1,54 @@
import type { Recipe } from '../types.ts';
/**
* Perplexity's hosted embeddings API (#1046). OpenAI-shaped at
* `POST {base}/embeddings` but diverges on the wire:
* - `encoding_format` only accepts 'base64_int8' (default) or
* 'base64_binary' the AI SDK's 'float' default is rejected.
* - The response `embedding` is a base64 string encoding SIGNED INT8
* components (natively quantized output), not a float array.
* Both divergences are handled by perplexityCompatFetch in gateway.ts
* (force 'base64_int8' outbound; decode Int8Array number[] inbound).
* Cosine similarity is scale-invariant, so the raw int8 components store
* and rank correctly as floats.
*
* Models (per docs.perplexity.ai/api-reference/embeddings-post, 2026-07):
* - pplx-embed-v1-0.6b: dims 128..1024 (default 1024)
* - pplx-embed-v1-4b: dims 128..2560 (default 2560)
* The flexible-dim range validation lives in src/core/ai/dims.ts
* (PERPLEXITY_EMBEDDING_MAX_DIMS). default_dims is pinned at 1024 so both
* models work out of the box on a plain vector(N) column; users who want
* the 4b model's full 2560 width set `embedding_dimensions: 2560` and the
* existing halfvec path (dims > 2000) covers storage + ANN.
*
* Auth is PERPLEXITY_API_KEY only deliberately NO OPENAI_API_KEY
* fallback (a Perplexity brain must never silently bill/route through
* OpenAI). If your key lives in PPLX_API_KEY, re-export it.
*/
export const perplexity: Recipe = {
id: 'perplexity',
name: 'Perplexity',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://api.perplexity.ai/v1',
auth_env: {
required: ['PERPLEXITY_API_KEY'],
setup_url: 'https://www.perplexity.ai/settings/api',
},
touchpoints: {
embedding: {
models: ['pplx-embed-v1-0.6b', 'pplx-embed-v1-4b'],
default_dims: 1024,
cost_per_1m_tokens_usd: 0.03, // pplx-embed-v1-4b; 0.6b is $0.004/M
price_last_verified: '2026-07-21',
// Perplexity enforces 120K combined tokens (and 512 texts) per
// request. Same pre-split posture as Voyage: assume a dense
// tokenizer (1 char ≈ 1 token) at 0.5 utilization; the gateway's
// recursive halving is the runtime safety net.
max_batch_tokens: 120_000,
chars_per_token: 1,
safety_factor: 0.5,
},
},
setup_hint: 'Get an API key at https://www.perplexity.ai/settings/api, then `export PERPLEXITY_API_KEY=...` (re-export PPLX_API_KEY if that is where your key lives).',
};
+7
View File
@@ -16,6 +16,13 @@ export const together: Recipe = {
setup_url: 'https://api.together.ai/settings/api-keys',
},
touchpoints: {
// Same OpenAI-compatible endpoint as chat; declared so an explicit
// `expansion_model: together:...` resolves instead of silently dropping
// expansion (#1135).
expansion: {
models: ['meta-llama/Llama-3.3-70B-Instruct-Turbo', 'Qwen/Qwen2.5-72B-Instruct-Turbo'],
price_last_verified: '2026-04-20',
},
chat: {
models: [
'Qwen/Qwen2.5-72B-Instruct-Turbo',
+19 -4
View File
@@ -1,9 +1,10 @@
import type { Recipe } from '../types.ts';
/**
* Zhipu AI (AI) BigModel Open Platform. OpenAI-compatible /embeddings
* endpoint at open.bigmodel.cn. Hosts embedding-2 (1024d) and embedding-3
* (Matryoshka up to 2048d).
* Zhipu AI (AI) BigModel Open Platform. OpenAI-compatible /embeddings and
* /chat/completions endpoints at open.bigmodel.cn. Hosts embedding-2 (1024d),
* embedding-3 (Matryoshka up to 2048d), and the GLM chat family (glm-5.1 etc.)
* with native tool calling usable for models.tier.subagent (#1157).
*
* embedding-3 at 2048 dims exceeds pgvector's HNSW cap of 2000 those
* brains fall back to exact vector scans (see
@@ -25,6 +26,20 @@ export const zhipu: Recipe = {
setup_url: 'https://open.bigmodel.cn/',
},
touchpoints: {
chat: {
// Informational list (openai-compat tier: assertTouchpoint doesn't
// enforce it), so newer GLM ids pass without a recipe edit.
models: ['glm-5.1', 'glm-4.6', 'glm-4.5'],
supports_tools: true,
// gbrain-side stable tool ids (v0.38 D11) decoupled the loop from
// Anthropic response formats; GLM tool calling is stable through the
// OpenAI-compat path, same as deepseek/groq.
supports_subagent_loop: true,
// Anthropic-style cache_control markers are not honored on the
// OpenAI-compat path — the loop runs hot (degraded:no_caching warn).
supports_prompt_cache: false,
max_context_tokens: 128000,
},
embedding: {
models: ['embedding-3', 'embedding-2'],
default_dims: 1024,
@@ -36,5 +51,5 @@ export const zhipu: Recipe = {
},
},
setup_hint:
'Get an API key at https://open.bigmodel.cn/, then `export ZHIPUAI_API_KEY=...`',
'Get an API key at https://open.bigmodel.cn/, then `export ZHIPUAI_API_KEY=...`. Chat/subagent: use `zhipu:glm-5.1`.',
};
+9 -3
View File
@@ -22,7 +22,8 @@ export type Implementation =
| 'native-openai'
| 'native-google'
| 'native-anthropic'
| 'openai-compatible';
| 'openai-compatible'
| 'claude-cli';
export interface EmbeddingTouchpoint {
models: string[];
@@ -232,8 +233,13 @@ export interface ChatTouchpoint {
* Strictly stronger than supports_tools.
*/
supports_subagent_loop: boolean;
/** Anthropic-style ephemeral prompt cache markers honored. */
supports_prompt_cache?: boolean;
/**
* Prompt caching honored for this chat touchpoint. Static booleans cover
* native providers; openai-compatible aggregators may decide per model id
* (e.g. OpenRouter caches OpenAI and Anthropic routes but not every routed
* model family).
*/
supports_prompt_cache?: boolean | ((modelId: string) => boolean);
max_context_tokens?: number;
cost_per_1m_input_usd?: number;
cost_per_1m_output_usd?: number;
+63
View File
@@ -0,0 +1,63 @@
/**
* Nightly conversation-parser probe audit trail.
*
* One event per REAL probe run lands in
* `~/.gbrain/audit/parser-probe-YYYY-Www.jsonl` (ISO-week rotation via the
* shared audit-writer primitive; honors `GBRAIN_AUDIT_DIR`).
* Scheduler-cadence skips (`rate_limited`) are NOT logged the autopilot
* loop ticks every few minutes, so logging every skip would flood the
* audit file with rows that carry no signal.
*
* Read by `gbrain doctor`'s `conversation_parser_probe_health` check and
* by the autopilot wiring's 24h rate-limit gate (`parserProbeRanWithin`).
*/
import { createAuditWriter } from './audit/audit-writer.ts';
import type { NightlyProbeResult } from './conversation-parser/nightly-probe.ts';
export type ParserProbeAuditEvent = NightlyProbeResult;
const writer = createAuditWriter<ParserProbeAuditEvent>({
featureName: 'parser-probe',
errorLabel: 'gbrain',
errorMessagePrefix: 'parser-probe audit ',
errorTrailer: '; probe continues',
});
/** Append one parser-probe event. Best-effort; never throws. */
export function logParserProbeEvent(event: ParserProbeAuditEvent): void {
writer.log(event);
}
/**
* Read recent parser-probe events (current + previous ISO week, filtered
* to the window). Missing files and corrupt rows are skipped silently.
*/
export function readRecentParserProbeEvents(
days = 7,
now: Date = new Date(),
): ParserProbeAuditEvent[] {
return writer.readRecent(days, now);
}
/** Exposed for tests pinning the rotation edge cases. */
export function computeParserProbeAuditFilename(now: Date = new Date()): string {
return writer.computeFilename(now);
}
/**
* 24h rate-limit gate for the autopilot wiring: true when any audited run
* happened within `windowMs` of `now`. Only REAL outcomes are audited (see
* module header), so a pass/fail today blocks re-runs until tomorrow while
* scheduler-cadence skips never extend the window.
*/
export function parserProbeRanWithin(
windowMs: number,
now: Date = new Date(),
): boolean {
const cutoff = now.getTime() - windowMs;
return readRecentParserProbeEvents(2, now).some((ev) => {
const ts = Date.parse(ev.ts);
return Number.isFinite(ts) && ts >= cutoff;
});
}
+14
View File
@@ -81,6 +81,20 @@ function getLoad(): number {
/** Get memory usage fraction (0-1) */
function getMemoryUsage(): number {
// Prefer /proc/meminfo MemAvailable on Linux — os.freemem() returns
// MemFree which excludes page cache, falsely reading "high pressure"
// in any container where the kernel caches files.
try {
// eslint-disable-next-line @typescript-eslint/no-var-requires
const fs = require('fs');
const meminfo: string = fs.readFileSync('/proc/meminfo', 'utf8');
const totalKb = Number(meminfo.match(/MemTotal:\s+(\d+)/)?.[1]);
const availKb = Number(meminfo.match(/MemAvailable:\s+(\d+)/)?.[1]);
if (totalKb > 0 && availKb >= 0) return 1 - availKb / totalKb;
} catch {
/* fall through to os.freemem() (non-Linux or /proc unavailable) */
}
// Non-Linux fallback (macOS, Windows, or any host without /proc/meminfo)
const total = totalmem();
if (total === 0) return 0;
return 1 - (freemem() / total);
+25 -11
View File
@@ -11,21 +11,35 @@ import { parseModelId } from './ai/model-resolver.ts';
* RecommendationContext (doctor + autopilot) use this to build a sync
* `resolveKey` closure without re-parsing recipes.
*
* Only OPENAI_API_KEY and ZEROENTROPY_API_KEY appear here because those are the
* only embedding keys `buildGatewayConfig` (src/cli.ts) folds from config into
* the gateway env. VOYAGE_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY are deliberately
* absent: their config fields are NOT threaded to the gateway today, so the
* producer closures fall through to checking `process.env` ONLY for them. That
* matches what the gateway can actually use (the recipes read those keys from
* env). Counting a config-plane voyage_api_key/google_api_key here would be a
* false positive: doctor/autopilot would call the provider "configured" and
* dispatch an embed.stale job that then fails auth at the gateway. When a future
* change threads voyage_api_key/google_api_key into buildGatewayConfig (the open
* voyage-config-mapping work), re-add the matching entry here in the same change.
* Only keys that `buildGatewayConfig` (src/core/ai/build-gateway-config.ts)
* actually folds from config into the gateway env may appear here.
* GOOGLE_GENERATIVE_AI_API_KEY is deliberately absent: its config field is NOT
* threaded to the gateway today, so the producer closures fall through to
* checking `process.env` ONLY for it. That matches what the gateway can
* actually use (the recipe reads that key from env). Counting a config-plane
* google_api_key here would be a false positive: doctor/autopilot would call
* the provider "configured" and dispatch an embed.stale job that then fails
* auth at the gateway. When a future change threads google_api_key into
* buildGatewayConfig, re-add the matching entry here in the same change.
*
* VOYAGE_API_KEY voyage_api_key was the same kind of gap (#2662) until
* buildGatewayConfig started folding it now safe to list here too.
*
* Caveat inherited from the existing OPENAI_API_KEY/ZEROENTROPY_API_KEY
* entries (unchanged by #2662, noted here for anyone extending this map):
* autopilot's resolveKey resolves these fields via `engine.getConfig()`
* (DB plane), while `buildGatewayConfig` only folds the FILE-plane
* (config.json) value. A `gbrain config set voyage_api_key X` with no
* matching config.json entry can therefore still read "configured" here
* while the gateway has no key a pre-existing false-positive class, not
* introduced or fixed by this change. Closing it requires threading
* `*_api_key` DB values through `loadConfigWithEngine()` before
* `buildGatewayConfig`, which is a separate, larger change.
*/
export const HOSTED_EMBED_KEY_CONFIG: Record<string, string> = {
OPENAI_API_KEY: 'openai_api_key',
ZEROENTROPY_API_KEY: 'zeroentropy_api_key',
VOYAGE_API_KEY: 'voyage_api_key',
};
/**
+34 -18
View File
@@ -139,8 +139,21 @@ export function autoFixFrontmatter(
fixes.push({ code: 'NULL_BYTES', description: 'Stripped null bytes' });
}
// 2. MISSING_CLOSE — if there's an opener but no closer before a heading,
// insert `---` immediately before the heading. Walk lines once.
// 2. MISSING_CLOSE — if there's an opener but no closer at all, insert
// `---` immediately before the first heading-shaped line (best-effort
// guess at where the frontmatter was meant to end).
//
// Find the closer FIRST, scanning the full zone — do not stop at the
// first `#`-prefixed line. A `#` line between the opening and closing
// `---` is a YAML comment (comments are valid anywhere in a YAML
// document), not a markdown heading; only the genuine absence of a
// closing `---` counts as MISSING_CLOSE. Mirrors the fix applied to
// the parseMarkdown validator in #2153 — this is the sibling
// reimplementation in the auto-fixer and had the same bug (it broke
// out of the scan on the first heading-shaped line, so a `#` comment
// appearing before a real closing fence was misdetected as
// MISSING_CLOSE and the fix inserted a spurious `---` that split
// valid frontmatter in two, pushing the real keys into the body).
{
const lines = working.split('\n');
let firstNonEmpty = -1;
@@ -149,24 +162,27 @@ export function autoFixFrontmatter(
}
if (firstNonEmpty >= 0 && lines[firstNonEmpty].trim() === '---') {
let closeIdx = -1;
let headingIdx = -1;
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
const t = lines[i].trim();
if (t === '---') { closeIdx = i; break; }
if (/^#{1,6}\s/.test(t)) { headingIdx = i; break; }
if (lines[i].trim() === '---') { closeIdx = i; break; }
}
if (closeIdx === -1 && headingIdx >= 0) {
const fixed = [
...lines.slice(0, headingIdx),
'---',
'',
...lines.slice(headingIdx),
];
working = fixed.join('\n');
fixes.push({
code: 'MISSING_CLOSE',
description: `Inserted closing --- before heading at line ${headingIdx + 1}`,
});
if (closeIdx === -1) {
let headingIdx = -1;
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
if (/^#{1,6}\s/.test(lines[i].trim())) { headingIdx = i; break; }
}
if (headingIdx >= 0) {
const fixed = [
...lines.slice(0, headingIdx),
'---',
'',
...lines.slice(headingIdx),
];
working = fixed.join('\n');
fixes.push({
code: 'MISSING_CLOSE',
description: `Inserted closing --- before heading at line ${headingIdx + 1}`,
});
}
}
}
}
+43 -7
View File
@@ -68,6 +68,7 @@ import {
type BrainstormCheckpoint,
type CheckpointCross,
} from './checkpoint.ts';
import { resolveOwnerHolder } from '../owner-holder.ts';
export { BudgetExhausted };
@@ -139,7 +140,7 @@ export interface BrainstormOptions {
modelOverride?: string;
/** Skip the cost-preview TTY grace window. Required for non-interactive callers. */
skipCostPreview?: boolean;
/** When set, force the user holder for calibration profile lookup. Falls back to config (`emotional_weight.user_holder`) then `'garry'`. */
/** When set, force the user holder for calibration profile lookup. Falls back to config (`emotional_weight.user_holder`) then `'self'`. */
holderOverride?: string;
/** Source scope. */
sourceId?: string;
@@ -485,9 +486,44 @@ const DEFAULT_PARALLELISM = 4;
* src/core/errors.ts (the v0.19.0 envelope every new agent-facing
* surface uses) rather than introducing a new BrainstormError class.
*/
/** File-config slice the orchestrator reads (see loadConfig in core/config.ts). */
export interface BrainstormRunConfig {
embedding_model?: string;
chat_model?: string;
emotional_weight?: { user_holder?: string };
}
/**
* Model used for the cost preview + hard cost ceiling. Mirrors what the
* gateway will actually run: explicit --model override, else the configured
* chat_model (gateway default), else the hardcoded gateway fallback. Before
* this resolved through config, a non-Sonnet chat_model got its preview
* priced against the wrong model. (Takeover of PR #1855 by @starm2010.)
*/
export function resolveBrainstormChatModel(
config: { chat_model?: string },
modelOverride?: string,
): string {
return modelOverride ?? config.chat_model ?? 'anthropic:claude-sonnet-4-6';
}
/**
* Judge-phase model precedence: --judge-model flag, else the
* `models.brainstorm.judge` config key, else undefined (falls back to
* `modelOverride` then the gateway default at the runJudge callsite).
*/
export async function resolveBrainstormJudgeModel(
engine: BrainEngine,
judgeModelFlag?: string,
): Promise<string | undefined> {
if (judgeModelFlag) return judgeModelFlag;
const configured = await engine.getConfig('models.brainstorm.judge');
return configured ?? undefined;
}
export async function runBrainstorm(
engine: BrainEngine,
config: { embedding_model?: string; emotional_weight?: { user_holder?: string } },
config: BrainstormRunConfig,
opts: BrainstormOptions
): Promise<BrainstormResult> {
// v0.39.3.0 (Phase 5, CV11+T4): outer try/catch around the orchestrator
@@ -509,7 +545,7 @@ export async function runBrainstorm(
async function runBrainstormImpl(
engine: BrainEngine,
config: { embedding_model?: string; emotional_weight?: { user_holder?: string } },
config: BrainstormRunConfig,
opts: BrainstormOptions,
): Promise<BrainstormResult> {
// v0.39.0.0 T10: install a gateway-layer BudgetTracker scope around the
@@ -529,7 +565,7 @@ async function runBrainstormImpl(
async function _runBrainstormInner(
engine: BrainEngine,
config: { embedding_model?: string; emotional_weight?: { user_holder?: string } },
config: BrainstormRunConfig,
opts: BrainstormOptions,
): Promise<BrainstormResult> {
const profile = opts.profile ?? BRAINSTORM_PROFILE;
@@ -538,7 +574,7 @@ async function _runBrainstormInner(
const embedFn = opts.embedQueryFn ?? embedQuery;
// ---- Phase 0: cost preview + TTY grace ----
const modelStr = opts.modelOverride ?? 'anthropic:claude-sonnet-4-6';
const modelStr = resolveBrainstormChatModel(config, opts.modelOverride);
const { aborted, estimate } = await previewCostAndWait({
profile,
model: modelStr,
@@ -623,7 +659,7 @@ async function _runBrainstormInner(
}
// ---- Phase 3: calibration context (cold-start fallback) ----
const holder = opts.holderOverride ?? config.emotional_weight?.user_holder ?? 'garry';
const holder = resolveOwnerHolder({ override: opts.holderOverride, configValue: config.emotional_weight?.user_holder });
const calibContext = await loadCalibrationContext(engine, {
holder,
sourceId: opts.sourceId,
@@ -847,7 +883,7 @@ async function _runBrainstormInner(
far_slug: i.far_slug,
}));
const judgeResult = await runJudge(profile.judge_config, judgeInput, {
modelOverride: opts.judgeModel ?? opts.modelOverride,
modelOverride: (await resolveBrainstormJudgeModel(engine, opts.judgeModel)) ?? opts.modelOverride,
chatFn: opts.chatFn,
activeBiasTags: activeBiasTags ?? undefined,
abortSignal: opts.abortSignal,
+22 -3
View File
@@ -32,6 +32,7 @@ import { mkdirSync, appendFileSync } from 'node:fs';
import { dirname } from 'node:path';
import { gbrainPath } from '../config.ts';
import { ANTHROPIC_PRICING, type ModelPricing } from '../anthropic-pricing.ts';
import { canonicalLookup } from '../model-pricing.ts';
import { EMBEDDING_PRICING, lookupEmbeddingPrice } from '../embedding-pricing.ts';
import { splitProviderModelId } from '../model-id.ts';
import { isoWeekFilename, resolveAuditDir } from '../audit-week-file.ts';
@@ -166,9 +167,13 @@ const FREE_LOCAL_EMBED_PROVIDERS: ReadonlySet<string> = new Set([
* local-inference providers (FREE_LOCAL_EMBED_PROVIDERS) price at $0 so
* `--max-cost` callers don't hard-fail.
* - Rerank: try ANTHROPIC_PRICING (legacy path for any Claude-priced
* rerank); else if the provider half is in FREE_LOCAL_RERANK_PROVIDERS,
* return zero pricing so `--max-cost` callers don't TX2 hard-fail on
* local inference recipes (electricity, not tokens); else unknown.
* rerank); else try lookupEmbeddingPrice paid rerank providers (e.g.
* ZeroEntropy's zerank-2) share the same provider:model-keyed,
* $/1M-token table as their embedding siblings, so it's reused here
* rather than duplicated into a third table; else if the provider half
* is in FREE_LOCAL_RERANK_PROVIDERS, return zero pricing so `--max-cost`
* callers don't TX2 hard-fail on local inference recipes (electricity,
* not tokens); else unknown.
*/
function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null {
if (kind === 'embed') {
@@ -194,6 +199,14 @@ function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null {
const tailHit = ANTHROPIC_PRICING[modelTail];
if (tailHit) return tailHit;
}
// Paid rerank providers (e.g. ZeroEntropy's zerank-2) aren't Claude-priced,
// so they miss the ANTHROPIC_PRICING checks above. Reuse the embedding
// pricing table (issue #3223) — same provider:model key shape, same
// $/1M-token unit — instead of hand-copying a third pricing surface.
if (kind === 'rerank') {
const hit = lookupEmbeddingPrice(modelId);
if (hit.kind === 'known') return { input: hit.pricePerMTok, output: 0 };
}
// v0.40.6.1: zero-price local-inference rerank providers so the budget
// tracker's TX2 hard-fail doesn't trip on `llama-server-reranker:<model>`
// under `--max-cost`. Only the rerank kind — chat/embed already have
@@ -201,6 +214,12 @@ function lookupPricing(modelId: string, kind: BudgetKind): ModelPricing | null {
if (kind === 'rerank' && providerId && FREE_LOCAL_RERANK_PROVIDERS.has(providerId)) {
return { input: 0, output: 0 };
}
// Fall back to the full canonical pricing table so non-Anthropic chat
// models with a known price (openai:*, google:*, deepseek:*) resolve under
// --max-cost instead of TX2 no_pricing hard-failing at $0. ANTHROPIC_PRICING
// above is only the bare-keyed Claude view.
const canon = canonicalLookup(modelId);
if (canon) return canon;
return null;
}
+1 -1
View File
@@ -25,7 +25,7 @@ import type { BrainEngine } from '../engine.ts';
export interface ABRunInput {
question: string;
/** Holder context for calibration. Default 'garry'. */
/** Holder context for calibration. Resolves via resolveOwnerHolder (config emotional_weight.user_holder, else 'self'). */
holder?: string;
/** Engine for DB write. */
engine: BrainEngine;
+19 -1
View File
@@ -1235,7 +1235,25 @@ export function estimateTokens(text: string): number {
tiktokenInitialized = true;
}
if (tiktokenEncoder) {
return tiktokenEncoder.encode(text).length;
try {
return tiktokenEncoder.encode(text).length;
} catch {
// Code legitimately contains tiktoken special-token strings (e.g. CLIP/GPT
// tokenizers embed the literal "<|endoftext|>"). The default encode() uses
// disallowed_special='all' and THROWS on those, crashing reindex-code on
// valid source files. For a token COUNT we don't need special-token
// semantics: re-encode treating them as ordinary text (never throws),
// heuristic only if even that fails.
try {
return (
tiktokenEncoder as unknown as {
encode: (s: string, allowed: string[], disallowed: string[]) => Uint32Array;
}
).encode(text, [], []).length;
} catch {
return Math.max(1, Math.ceil(text.length / 4));
}
}
}
return Math.max(1, Math.ceil(text.length / 4));
}
+8
View File
@@ -20,6 +20,14 @@ export const CJK_SLUG_CHARS = '一-鿿぀-ゟ゠-ヿ가-힯';
export const CJK_RANGES_REGEX = new RegExp(`[${CJK_SLUG_CHARS}]`);
/**
* Page-slug segment grammar (no anchors): alnum-or-CJK lead char, then
* alnum/CJK/hyphen continuation. Single source for validatePageSlug
* (operations.ts), SlugRegistry's SLUG_RE, and the dream-cycle
* SUMMARY_SLUG_RE so every slug validator shares one grammar (#738).
*/
export const PAGE_SLUG_SEG = `[a-z0-9${CJK_SLUG_CHARS}][a-z0-9${CJK_SLUG_CHARS}\\-]*`;
export const CJK_SENTENCE_DELIMITERS = ['。', '', '']; // 。!?
export const CJK_CLAUSE_DELIMITERS = ['', '', '', '、']; // ;:,、
+39
View File
@@ -48,6 +48,21 @@ export interface GBrainConfig {
* reads OPENROUTER_API_KEY.
*/
openrouter_api_key?: string;
/**
* Voyage AI API key (#2662). File-plane slot so `~/.gbrain/config.json`'s
* `voyage_api_key` reaches the voyage recipe the same way
* zeroentropy_api_key/openrouter_api_key do: file plane
* buildGatewayConfig env dict recipe reads VOYAGE_API_KEY. Before this,
* launchd/daemon/MCP contexts without a process-env export silently
* failed multimodal embeds despite config.json looking complete.
*
* NOTE (scoped to what this fix covers): `gbrain config set
* voyage_api_key X` writes the DB plane, which `loadConfigWithEngine()`
* does NOT merge for any `*_api_key` field (zeroentropy_api_key /
* openrouter_api_key have the same pre-existing gap) only the
* config.json file-plane route is wired through today.
*/
voyage_api_key?: string;
/** AI gateway config (v0.14+). v0.36+ default: "zeroentropyai:zembed-1" / 1280 / "anthropic:claude-haiku-4-5-20251001". */
embedding_model?: string;
embedding_dimensions?: number;
@@ -105,6 +120,16 @@ export interface GBrainConfig {
*/
max_usd?: number;
};
/**
* v0.41.16.0 nightly conversation-parser probe. Per D10: default ON
* for `search.mode=tokenmax` brains, opt-in for conservative/balanced.
* ~$0.05/night with the committed fixtures × Haiku polish. Gated
* INSIDE the autopilot tick body, like nightly_quality_probe.
*/
conversation_parser_probe?: {
/** Enable for non-tokenmax modes. Defaults to false. */
enabled?: boolean;
};
/**
* v0.42.x (#1685 GAP D) extract_atoms backlog auto-drain. Default ON so a
* pack-gated silent backlog never piles up unseen; daily-spend-capped so the
@@ -121,6 +146,18 @@ export interface GBrainConfig {
/** Daily spend cap (USD); bounds drains/day = floor(cap / ~$0.30). Default 2.0. */
max_usd_per_day?: number;
};
/**
* v0.42 keep frontmatter links fresh on the incremental cycle. The cycle's
* extract phase re-extracts only the slugs a sync changed, but `extractForSlugs`
* extracts BODY links only frontmatter (`sources:`/`related:` etc.) link edges
* silently drift stale when a page's YAML is edited externally and synced in.
* Set true to also extract frontmatter links per changed page each cycle, keeping
* externally-edited YAML edges fresh without a full rescan. Default false
* (preserves current behavior). Read via the file/env/DB plane in the cycle's
* extract dispatch. Disable/enable with
* `gbrain config set autopilot.incremental_extract_include_frontmatter <bool>`.
*/
incremental_extract_include_frontmatter?: boolean;
};
eval?: {
/** false disables capture entirely. Defaults to true. */
@@ -875,6 +912,7 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'anthropic_api_key',
'zeroentropy_api_key',
'openrouter_api_key',
'voyage_api_key',
'embedding_model',
'embedding_dimensions',
'embedding_disabled',
@@ -936,6 +974,7 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'models.subagent',
'models.expansion',
'models.chat',
'models.brainstorm.judge',
'models.eval.longmemeval',
'facts.extraction_model',
// #2113: output-token cap for the per-turn facts extractor (default 4000).
+78 -4
View File
@@ -167,6 +167,52 @@ export function deriveDirectUrl(url: string): string | null {
}
}
/** True when the URL targets Supavisor transaction mode (port 6543). */
function isTransactionPoolerUrl(url: string): boolean {
try {
return new URL(url.replace(/^postgres(ql)?:\/\//, 'http://')).port === '6543';
} catch {
return false;
}
}
/**
* Resolve the direct-pool URL from an explicit/env override + the primary URL.
*
* A direct override still pointing at the TRANSACTION-mode pooler (port 6543,
* usually a copy-paste of the primary URL) is a misconfiguration: the manager
* would believe DDL/bulk work runs on a long-timeout direct connection while
* still routing through Supavisor transaction mode (short timeouts, no
* prepared statements). Normalize it via deriveDirectUrl. Session-mode pooler
* URLs (pooler host, port 5432) pass through they are a legitimate
* direct-ish target when the db.<ref> host is unreachable.
*/
export function normalizeDirectUrl(primaryUrl: string, override?: string | null): string | null {
const candidate = override ?? deriveDirectUrl(primaryUrl);
if (!candidate) return null;
if (!isTransactionPoolerUrl(candidate)) return candidate;
return deriveDirectUrl(candidate) ?? deriveDirectUrl(primaryUrl);
}
/**
* Error codes that mean "the direct host is unreachable from this network"
* (#1641). The auto-derived db.<ref>.supabase.co host is IPv6-only without
* the paid IPv4 add-on, so ENOTFOUND/ECONNREFUSED here is expected on
* IPv4-only networks we fall back to the pooler instead of failing init.
*/
const NETWORK_UNREACHABLE_CODES = [
'ENOTFOUND', 'ECONNREFUSED', 'ENETUNREACH', 'EHOSTUNREACH',
'ETIMEDOUT', 'CONNECT_TIMEOUT',
];
/** True when err looks like a network-unreachable failure (not auth/SQL). */
export function isNetworkUnreachableError(err: unknown): boolean {
const code = (err as { code?: unknown } | null)?.code;
if (typeof code === 'string' && NETWORK_UNREACHABLE_CODES.includes(code)) return true;
const msg = err instanceof Error ? err.message : String(err);
return NETWORK_UNREACHABLE_CODES.some(c => msg.includes(c));
}
/**
* Read kill-switch state from env. Subordinate to parent manager's state
* when present (A2 inheritance).
@@ -213,9 +259,10 @@ export class ConnectionManager {
} else {
this._killSwitch = readKillSwitchEnv();
this._isSupabase = isSupabasePoolerUrl(opts.url);
// Direct URL: explicit override > env > derive > null
// Direct URL: explicit override > env > derive > null. Pooler-shaped
// overrides are normalized to a real direct host (or dropped).
const envOverride = process.env.GBRAIN_DIRECT_DATABASE_URL;
this._directUrl = opts.directUrl ?? envOverride ?? deriveDirectUrl(opts.url);
this._directUrl = normalizeDirectUrl(opts.url, opts.directUrl ?? envOverride);
}
}
@@ -319,7 +366,30 @@ export class ConnectionManager {
throw err;
});
}
const pool = await this._directInit;
let pool: Sql | null;
try {
pool = await this._directInit;
} catch (err) {
// #1641: the derived direct host (db.<ref>.supabase.co) is IPv6-only
// without Supabase's IPv4 add-on. On IPv4-only networks the direct
// pool can never connect — permanently fall back to the read pool
// (self-activating kill-switch) instead of failing init/migrations.
// Non-network errors (auth, SQL) still throw: they mean misconfig,
// not unreachability.
if (isNetworkUnreachableError(err)) {
const alreadyWarned = this._killSwitch;
this._killSwitch = true;
const msg = err instanceof Error ? err.message : String(err);
if (!alreadyWarned) console.error(
`gbrain: direct connection to ${this._directUrl ? this.hostOnly(this._directUrl) : 'unknown host'} unreachable (${msg}); ` +
'falling back to the pooler for DDL/bulk (long migrations may hit the pooler statement timeout). ' +
'Set GBRAIN_DIRECT_DATABASE_URL to a reachable direct URL (e.g. the Session pooler, port 5432) or enable the Supabase IPv4 add-on; ' +
'GBRAIN_DISABLE_DIRECT_POOL=1 silences this.',
);
return this.getReadPool();
}
throw err;
}
if (!pool) {
// Defensive — initDirectPool should have thrown.
throw new Error('connection-manager: direct pool init returned null');
@@ -350,8 +420,9 @@ export class ConnectionManager {
},
};
const t0 = Date.now();
let pool: Sql | null = null;
try {
const pool = postgres(this._directUrl, opts);
pool = postgres(this._directUrl, opts);
// Probe to validate connectivity early.
await pool`SELECT 1`;
logConnectionEvent({
@@ -362,6 +433,9 @@ export class ConnectionManager {
});
return pool;
} catch (err) {
// Don't leak the failed pool's sockets/timers (#1641 fallback keeps
// the process running afterward).
if (pool) await endPoolBounded(pool);
logConnectionEvent({
pool: 'ddl',
op: 'error',
+15 -4
View File
@@ -453,7 +453,14 @@ function resolveActivity(
* every `assemble()` call. 1 MB is generous for a human-edited task list. */
const MAX_TASKS_MD_BYTES = 1_000_000;
/** Extract open tasks from ops/tasks.md "## Today" section. */
/** Extract open tasks from ops/tasks.md Today section.
*
* The daily-task-manager skill's documented Output Format uses priority
* headings (`## P1 — Today`) with plain `- [ ] task` lines; older fixtures
* used a bare `## Today` heading with bold task names. Accept both so the
* live-context reader matches the documented writer contract instead of
* silently surfacing no tasks (#2186).
*/
function resolveTodayTasks(workspaceDir: string): string[] {
try {
const path = join(workspaceDir, 'ops', 'tasks.md');
@@ -461,14 +468,18 @@ function resolveTodayTasks(workspaceDir: string): string[] {
// statSync throws if the file doesn't exist; that lands in the outer catch.
if (statSync(path).size > MAX_TASKS_MD_BYTES) return [];
const raw = readFileSync(path, 'utf8');
const todayMatch = raw.match(/## Today[\s\S]*?(?=\n## |$)/);
const todayMatch = raw.match(/^##\s+(?:P\d\s*[—–-]\s*)?Today\b[\s\S]*?(?=\n##\s|$(?![\s\S]))/m);
if (!todayMatch) return [];
const lines = todayMatch[0].split('\n');
const open: string[] = [];
for (const line of lines) {
// Match unchecked task lines: - [ ] **task name** ...
const m = line.match(/^\s*-\s*\[ \]\s*\*\*(.+?)\*\*/);
// Match unchecked task lines. Legacy bold form first (extracts just
// the task name, dropping trailing metadata), then the documented
// plain form (whole line body is the task).
const m =
line.match(/^\s*-\s*\[ \]\s*\*\*(.+?)\*\*/) ??
line.match(/^\s*-\s*\[ \]\s*(.+?)\s*$/);
if (m) open.push(sanitizeForPrompt(m[1].trim()));
}
return open.slice(0, 5); // cap at 5 to keep prompt lean
+164 -79
View File
@@ -45,7 +45,6 @@ import { embedBatch } from './embedding.ts';
import { resolveContextualRetrievalMode } from './contextual-retrieval-resolver.ts';
import {
buildContextualPrefix,
extractFirstTwoSentences,
modeRequiresHaiku,
modeRequiresWrapper,
sanitizeTitle,
@@ -57,10 +56,8 @@ import {
SYNOPSIS_DOC_MAX_CHARS,
type GeneratePerChunkSynopsisResult,
} from './page-summary.ts';
import {
logSynopsisFailure,
type SynopsisFailureKind,
} from './audit-synopsis.ts';
import type { SynopsisFailureKind } from './audit-synopsis.ts';
import { runSlidingPool } from './worker-pool.ts';
import type { BrainEngine } from './engine.ts';
import type { ChunkInput, CRMode, Page } from './types.ts';
import type { SourceRow } from './sources-ops.ts';
@@ -73,6 +70,24 @@ import type { SourceRow } from './sources-ops.ts';
* corpus_generation hash.
*/
export const TITLE_WRAPPER_VERSION = 1;
const DEFAULT_HAIKU_MODEL = 'anthropic:claude-haiku-4-5-20251001';
export const DEFAULT_CONTEXTUAL_CHUNK_CONCURRENCY = 4;
export const MAX_CONTEXTUAL_CHUNK_CONCURRENCY = 16;
export function resolveContextualChunkConcurrency(
env: Record<string, string | undefined> = process.env,
): number {
const raw = env.GBRAIN_CONTEXTUAL_CHUNK_CONCURRENCY;
if (raw === undefined || raw.trim() === '') return DEFAULT_CONTEXTUAL_CHUNK_CONCURRENCY;
const n = Number(raw);
if (!Number.isFinite(n)) return DEFAULT_CONTEXTUAL_CHUNK_CONCURRENCY;
return clampContextualChunkConcurrency(n);
}
function clampContextualChunkConcurrency(n: number): number {
if (!Number.isFinite(n)) return DEFAULT_CONTEXTUAL_CHUNK_CONCURRENCY;
return Math.max(1, Math.min(MAX_CONTEXTUAL_CHUNK_CONCURRENCY, Math.trunc(n)));
}
/**
* Embedding model placeholder. The actual model name lands here from
@@ -208,12 +223,16 @@ export interface ReembedPageArgs {
* src/core/minions/rate-leases.ts here; inline callers (import-file,
* reindex command) pass undefined and rely on gateway-level retry.
*/
acquireSynopsisLease?: () => Promise<void>;
releaseSynopsisLease?: () => Promise<void>;
acquireSynopsisLease?: () => Promise<unknown>;
releaseSynopsisLease?: (lease?: unknown) => Promise<void>;
/**
* Intra-page per-chunk synopsis concurrency. 1 preserves the legacy
* sequential loop exactly; higher values only parallelize Haiku synopsis
* calls. Embedding remains one batch after all synopses succeed.
*/
chunkConcurrency?: number;
}
const DEFAULT_HAIKU_MODEL = 'anthropic:claude-haiku-4-5-20251001';
/**
* Re-embed one page through the active CR mode. Implements the D26 P0-2
* two-phase build pattern.
@@ -432,82 +451,41 @@ async function tryBuildPhase1(opts: {
}
// per_chunk_synopsis path. Read source text via fallback chain,
// generate synopsis per chunk sequentially within this page (D10),
// generate synopsis per chunk through a bounded sliding pool, then
// batch embed at the end (D27 P2-2).
const sourceText = readSourceTextWithFallback(page, chunks);
const wrappedTexts: string[] = [];
const wrappedTexts: string[] = new Array(chunks.length);
const chunkConcurrency = clampContextualChunkConcurrency(
args.chunkConcurrency ?? resolveContextualChunkConcurrency(),
);
for (let i = 0; i < chunks.length; i++) {
const c = chunks[i];
// Code chunks always bypass the wrapper (D20-T4) — pass through.
if (c.chunk_source === 'fenced_code') {
wrappedTexts.push(c.chunk_text);
continue;
}
// Acquire rate-lease per chunk (D26 P0-3). Inline callers pass no
// hooks; only the Minion handler wires through rate-leases.ts.
if (args.acquireSynopsisLease) {
await args.acquireSynopsisLease();
}
let synopsisResult: GeneratePerChunkSynopsisResult;
try {
synopsisResult = await generatePerChunkSynopsis({
documentText: sourceText,
chunkText: c.chunk_text,
pageTitle: page.title,
pageSlug: args.pageSlug,
sourceId: args.sourceId,
chunkIndex: c.chunk_index,
model: haikuModel,
abortSignal: args.abortSignal,
const poolResult = await runSlidingPool({
items: chunks,
workers: chunkConcurrency,
signal: args.abortSignal,
onError: 'abort',
failureLabel: (c) => String(c.chunk_index),
onItem: async (c, i) => {
wrappedTexts[i] = await buildWrappedChunkText({
chunk: c,
sourceText,
safeTitle,
page,
args,
haikuModel,
});
} finally {
if (args.releaseSynopsisLease) {
try {
await args.releaseSynopsisLease();
} catch {
// Lease release failure shouldn't abort the page; surfacing it
// would race with the synopsis result. Audit-only.
}
}
}
},
});
if (synopsisResult.kind === 'success') {
const prefix = buildContextualPrefix(safeTitle, synopsisResult.synopsis);
wrappedTexts.push(
wrapChunkForEmbedding(c.chunk_text, prefix, c.chunk_source),
);
continue;
if (poolResult.failures.length > 0) {
const failure = [...poolResult.failures].sort((a, b) => a.idx - b.idx)[0].error;
if (failure instanceof ChunkSynopsisPhase1Error) {
return failure.result;
}
// Failure classification per D27 P1-2:
// refusal | empty | malformed → page-level fall-back to title-only
// auth_failure → permanent (won't fix with retry)
// rate_limit | timeout | network | provider_5xx → transient
// source_missing → walked into fallback already; would be 'malformed'
// from generatePerChunkSynopsis if we ever propagated it here
if (
synopsisResult.kind === 'refusal' ||
synopsisResult.kind === 'empty' ||
synopsisResult.kind === 'malformed'
) {
return { kind: 'page_level_fallback_requested', cause: synopsisResult.kind };
}
if (synopsisResult.kind === 'auth_failure') {
return {
kind: 'permanent',
cause: synopsisResult.kind,
detail: synopsisResult.detail ?? 'auth failure',
};
}
return {
kind: 'transient',
cause: synopsisResult.kind,
detail: synopsisResult.detail ?? 'transient',
};
throw failure;
}
if (poolResult.aborted || args.abortSignal?.aborted) {
return { kind: 'transient', cause: 'timeout', detail: 'aborted' };
}
// All chunks synthesized successfully. Single batch embed (D27 P2-2).
@@ -528,6 +506,113 @@ async function tryBuildPhase1(opts: {
}
}
class ChunkSynopsisPhase1Error extends Error {
constructor(readonly result: Exclude<Phase1Result, Phase1Success>) {
super(`chunk synopsis failed: ${result.kind}`);
this.name = 'ChunkSynopsisPhase1Error';
}
}
async function buildWrappedChunkText(opts: {
chunk: ChunkInput;
sourceText: string;
safeTitle: string;
page: Page;
args: ReembedPageArgs;
haikuModel: string;
}): Promise<string> {
const { chunk: c, sourceText, safeTitle, page, args, haikuModel } = opts;
// Code chunks always bypass the wrapper (D20-T4) — pass through.
if (c.chunk_source === 'fenced_code') {
return c.chunk_text;
}
// Acquire rate-lease per chunk (D26 P0-3). Inline callers pass no
// hooks; only the Minion handler wires through rate-leases.ts.
let lease: unknown;
let leaseAcquired = false;
let synopsisResult: GeneratePerChunkSynopsisResult;
try {
if (args.acquireSynopsisLease) {
try {
lease = await args.acquireSynopsisLease();
} catch (err) {
if (args.abortSignal?.aborted || isAbortError(err)) {
throw new ChunkSynopsisPhase1Error({
kind: 'transient',
cause: 'timeout',
detail: 'aborted',
});
}
throw err;
}
leaseAcquired = true;
}
synopsisResult = await generatePerChunkSynopsis({
documentText: sourceText,
chunkText: c.chunk_text,
pageTitle: page.title,
pageSlug: args.pageSlug,
sourceId: args.sourceId,
chunkIndex: c.chunk_index,
model: haikuModel,
abortSignal: args.abortSignal,
});
} finally {
if (leaseAcquired && args.releaseSynopsisLease) {
try {
await args.releaseSynopsisLease(lease);
} catch {
// Lease release failure shouldn't abort the page; surfacing it
// would race with the synopsis result. Audit-only.
}
}
}
if (synopsisResult.kind === 'success') {
const prefix = buildContextualPrefix(safeTitle, synopsisResult.synopsis);
return wrapChunkForEmbedding(c.chunk_text, prefix, c.chunk_source);
}
// Failure classification per D27 P1-2:
// refusal | empty | malformed → page-level fall-back to title-only
// auth_failure → permanent (won't fix with retry)
// rate_limit | timeout | network | provider_5xx → transient
// source_missing → walked into fallback already; would be 'malformed'
// from generatePerChunkSynopsis if we ever propagated it here
if (
synopsisResult.kind === 'refusal' ||
synopsisResult.kind === 'empty' ||
synopsisResult.kind === 'malformed'
) {
throw new ChunkSynopsisPhase1Error({
kind: 'page_level_fallback_requested',
cause: synopsisResult.kind,
});
}
if (synopsisResult.kind === 'auth_failure') {
throw new ChunkSynopsisPhase1Error({
kind: 'permanent',
cause: synopsisResult.kind,
detail: synopsisResult.detail ?? 'auth failure',
});
}
throw new ChunkSynopsisPhase1Error({
kind: 'transient',
cause: synopsisResult.kind,
detail: synopsisResult.detail ?? 'transient',
});
}
function isAbortError(err: unknown): boolean {
return (
typeof err === 'object' &&
err !== null &&
(err as { name?: unknown }).name === 'AbortError'
);
}
/**
* Source-text fallback chain per D11:
* 1. read page.source_path from disk (truest "document")
@@ -17,11 +17,11 @@
* Cost: ~$0.05/night with default fixtures × Haiku polish. Bounded
* by the active BudgetTracker the autopilot loop creates per-tick.
*
* **Wiring into the autopilot loop is deferred to a follow-up**
* (filed in TODOS.md). v0.41.16.0 ships the phase as a callable
* module so doctor + future cron drivers can invoke it; the
* scheduler wire-up follows the same shape as
* `src/core/cycle/nightly-quality-probe.ts` (v0.40.1.0 Track D / T6).
* Wired into the autopilot loop (step 4.6 in autopilot.ts), following
* the same shape as `src/core/cycle/nightly-quality-probe.ts`
* (v0.40.1.0 Track D / T6): the wiring resolves fixtures from the
* gbrain package root, writes real outcomes to the parser-probe audit
* trail (`audit-parser-probe.ts`), and never crashes the loop.
*
* Test seam: all dependencies are injected via NightlyProbeDeps so
* unit tests don't touch real LLMs or real fixtures.
+6 -1
View File
@@ -44,7 +44,12 @@ export const DEFAULT_DIMENSIONS: string[] = [
* `--slot-a-model`, `--slot-b-model`, `--slot-c-model` on the CLI.
*/
export const DEFAULT_SLOTS: SlotConfig[] = [
{ id: 'A', model: 'openai:gpt-4o' },
// Every default MUST be listed in its recipe's chat touchpoint (pinned by
// test/cross-modal-default-slots.test.ts) — `openai:gpt-4o` sat here after
// the OpenAI recipe dropped it, so slot A errored "not listed for OpenAI
// chat" on every install and the 3-slot panel could never reach its
// 2-model quorum without a Google key (verdict: permanently inconclusive).
{ id: 'A', model: 'openai:gpt-5.2' },
{ id: 'B', model: 'anthropic:claude-opus-4-7' },
{ id: 'C', model: 'google:gemini-1.5-pro' },
];
+119 -15
View File
@@ -66,6 +66,10 @@ export type CyclePhase =
// - calibration_profile: aggregates the resolved subset into 2-4
// narrative pattern statements + active bias tags. Voice-gated.
| 'propose_takes' | 'grade_takes' | 'calibration_profile'
// #2653 — drift detection (default OFF; dream.drift.enabled). LLM-judges
// soft-band takes against recent timeline evidence; report-only in v1
// (writes reports/drift-<date>; auto_update mutates nothing).
| 'drift'
| 'embed' | 'orphans' | 'purge'
// v0.39 T12: schema-suggest passive trigger (D3 + D4 plan-eng-review).
// Wraps runSuggest() — same library the CLI verb + EIIRP call.
@@ -151,6 +155,10 @@ export const ALL_PHASES: CyclePhase[] = [
'propose_takes',
'grade_takes',
'calibration_profile',
// #2653 — drift detection. Default OFF (dream.drift.enabled). Runs AFTER
// the calibration trio (fresh take resolutions) and BEFORE embed so the
// drift report page gets embedded same-cycle. Report-only in v1.
'drift',
// v0.41.11.0 — opt-in conversation-facts backfill. Default OFF; reads
// cycle.conversation_facts_backfill.enabled gate inside the wrapper.
// Ordered AFTER calibration_profile (matches the runCycle dispatch
@@ -195,8 +203,10 @@ export const ALL_PHASES: CyclePhase[] = [
* - `source`: safe to parallelize per source. Sync reads/writes the
* one source's rows; extract walks changed slugs.
* - `global`: must serialize across the brain. Embed walks all stale
* chunks; orphans/purge sweep brain-wide; grade_takes + calibration
* aggregate across sources; resolve_symbol_edges walks every chunk.
* chunks; purge sweeps brain-wide; orphans can report a single
* resolved source but still belongs in the serialized global lane;
* grade_takes + calibration aggregate across sources;
* resolve_symbol_edges walks every chunk.
* - `mixed`: per-phase decomposition needed before parallelizing.
* Synthesize reads the brain-global transcripts dir but writes to
* per-source slugs (via subagent allowlist). Patterns reads
@@ -221,6 +231,9 @@ export const PHASE_SCOPE: Record<CyclePhase, PhaseScope> = {
propose_takes: 'source',
grade_takes: 'global',
calibration_profile: 'global',
// #2653 — drift walks takes brain-wide (same posture as grade_takes) and
// writes one brain-global report page.
drift: 'global',
embed: 'global',
orphans: 'global',
purge: 'global',
@@ -289,6 +302,8 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'propose_takes',
'grade_takes',
'calibration_profile',
// #2653 — writes the reports/drift-<date> page.
'drift',
// v0.41 T9 — extract_atoms writes atom-typed pages via put_page;
// synthesize_concepts writes concept-typed pages + tier updates. Both
// mutate DB state and need the lock.
@@ -453,6 +468,12 @@ export interface CycleOpts {
* loop bug (codex finding #3).
*/
synthBypassDreamGuard?: boolean;
/**
* Force the orphans phase to scan brain-wide even when `brainDir` resolves to
* a source. Used by autopilot global maintenance, whose phase set is
* intentionally brain-wide.
*/
forceGlobalOrphans?: boolean;
/**
* AbortSignal from the Minions worker (v0.22.1, #403). When aborted
* (timeout, cancel, lock-loss), runCycle bails between phases and
@@ -469,12 +490,14 @@ export interface CycleOpts {
* + every existing caller).
*
* **Note for follow-up waves:** this only scopes the LOCK. Several
* cycle phases (`embed`, `orphans`, `purge`, `resolve_symbol_edges`,
* `grade_takes`, `calibration_profile`) still operate brain-wide
* regardless of sourceId see the `PHASE_SCOPE` taxonomy. Per-source
* cycle locks let two cycles RUN, but the global-scoped phases
* inside each will still touch the same rows. Genuine per-source
* fan-out requires the deferred TODOs in the plan.
* cycle phases (`embed`, `purge`, `resolve_symbol_edges`, `grade_takes`,
* `calibration_profile`) still operate brain-wide regardless of sourceId
* see the `PHASE_SCOPE` taxonomy. `orphans` uses the resolved source
* for its candidate set when one exists, but it remains in the serialized
* global lane for autopilot scheduling. Per-source cycle locks let two
* cycles RUN, but the global-scoped phases inside each will still touch
* the same rows. Genuine per-source fan-out requires the deferred TODOs
* in the plan.
*
* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).
*/
@@ -855,8 +878,16 @@ interface SyncPhaseResult extends PhaseResult {
* Resolve the source id for a brain directory by looking up the sources
* table. Returns undefined when no registered source matches (falls back
* to pre-v0.18 global config.sync.* keys).
*
* Exported for dream.ts (#1869): a `gbrain dream --dir <path>` run whose
* path matches a registered source's local_path is a per-source cycle in
* everything but name, so dream derives the source id up front and passes
* it as opts.sourceId landing the freshness stamp without changing
* runCycle's stamp/lock semantics for legacy global callers (the
* autopilot-global-maintenance handler runs GLOBAL_PHASES with a brainDir
* and MUST NOT stamp per-source freshness; see rejected PR #2549).
*/
async function resolveSourceForDir(
export async function resolveSourceForDir(
engine: BrainEngine,
brainDir: string | null,
): Promise<string | undefined> {
@@ -935,13 +966,21 @@ async function runPhaseSync(
// sync's inline extract still runs to preserve prior behavior.
});
const syncedCount = result.added + result.modified;
// #3068: a pull_failed partial means the internal git pull failed and the
// run imported nothing — the source may be silently behind its remote and
// will not self-heal. Surface it as 'warn' (not 'ok') so a scheduled cycle
// doesn't report a clean run over a wedged source. Timeout-class partials
// keep the pre-existing 'ok' mapping (they converge on retry by design).
const pullFailedPartial = result.status === 'partial' && result.reason === 'pull_failed';
return {
phase: 'sync',
status: result.status === 'blocked_by_failures' ? 'warn' : 'ok',
status: result.status === 'blocked_by_failures' || pullFailedPartial ? 'warn' : 'ok',
duration_ms: 0,
summary: dryRun
? `${syncedCount} page(s) would sync, ${result.deleted} would delete`
: `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`,
: pullFailedPartial
? `git pull failed, nothing imported — source may be behind its remote (sync anchor unchanged)`
: `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`,
details: {
added: result.added,
modified: result.modified,
@@ -950,6 +989,7 @@ async function runPhaseSync(
chunksCreated: result.chunksCreated,
failedFiles: result.failedFiles ?? 0,
syncStatus: result.status,
...(result.reason ? { syncReason: result.reason } : {}),
dryRun,
},
pagesAffected: result.pagesAffected,
@@ -996,6 +1036,21 @@ async function runPhaseExtract(
): Promise<PhaseResult> {
try {
const { runExtractCore } = await import('../commands/extract.ts');
const { loadConfig } = await import('./config.ts');
// Default off: the incremental cycle extracts body links only unless the
// operator opts in to keeping externally-edited frontmatter links fresh too.
// Both planes, file wins (env > file > DB precedence, per loadConfigWithEngine):
// `gbrain config set autopilot.incremental_extract_include_frontmatter true`
// writes the DB plane (engine.setConfig), so a file-plane-only read here
// would make the documented enable command a silent no-op (#2120 class).
const fileVal = loadConfig()?.autopilot?.incremental_extract_include_frontmatter;
let includeFrontmatter = fileVal === true;
if (fileVal === undefined) {
try {
includeFrontmatter =
(await engine.getConfig('autopilot.incremental_extract_include_frontmatter')) === 'true';
} catch { /* config table unreadable → default off */ }
}
// Extract is read-mostly against the filesystem + write to links table.
// Honor dryRun by skipping with a 'skipped' entry: extract doesn't have
// a clean dry-run mode today and runCycle should be honest about it.
@@ -1016,6 +1071,7 @@ async function runPhaseExtract(
slugs: changedSlugs, // undefined = full walk (first run / manual)
signal,
sourceId,
includeFrontmatter, // honored on the incremental (slugs) path only
});
const linksCreated = result?.links_created ?? 0;
const timelineCreated = result?.timeline_entries_created ?? 0;
@@ -1214,7 +1270,9 @@ async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean, signal?: Abor
// 10-15 min one) bails within a batch instead of running to completion
// after the job was killed — which left gbrain_cycle_locks held and
// wedged every subsequent autopilot cycle.
const result = await runEmbedCore(engine, { stale: true, dryRun, signal });
// #394: quiet — the cycle reports embed counts via its own PhaseResult;
// raw `[dry-run] Would embed ...` stdout lines would corrupt `dream --json`.
const result = await runEmbedCore(engine, { stale: true, dryRun, signal, quiet: true });
const embeddedCount = dryRun ? result.would_embed : result.embedded;
return {
phase: 'embed',
@@ -1391,10 +1449,10 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
* to avoid a static import (purge phase is only loaded in the autopilot path). */
const SOFT_DELETE_TTL_HOURS_FOR_PURGE = 72;
async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
async function runPhaseOrphans(engine: BrainEngine, sourceId?: string): Promise<PhaseResult> {
try {
const { findOrphans } = await import('../commands/orphans.ts');
const result = await findOrphans(engine);
const result = await findOrphans(engine, sourceId !== undefined ? { sourceId } : {});
const count = result.total_orphans;
// Orphans are a code-smell signal, not a fatal condition. The
// original `count > 20` cutoff was tuned for small dev brains; on
@@ -1413,8 +1471,10 @@ async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
summary: `${count} orphan page(s) out of ${result.total_pages} total`,
details: {
total_orphans: count,
total_linkable: result.total_linkable,
total_pages: result.total_pages,
excluded: result.excluded,
...(sourceId !== undefined ? { source_id: sourceId } : {}),
},
};
} catch (e) {
@@ -1478,6 +1538,7 @@ export async function runCycle(
const cycleSourceId: string | undefined = engine
? (opts.sourceId ?? (await resolveSourceForDir(engine, brainDir)))
: opts.sourceId;
const orphansSourceId = opts.forceGlobalOrphans ? undefined : cycleSourceId;
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
@@ -2116,6 +2177,49 @@ export async function runCycle(
}
}
// ── #2653: drift detection ──────────────────────────────────
// Default OFF (dream.drift.enabled). LLM-judges soft-band takes
// against recent timeline evidence; report-only in v1 — writes one
// reports/drift-<date> page, mutates no takes regardless of
// dream.drift.auto_update.
if (phases.includes('drift')) {
checkAborted(opts.signal);
if (!engine) {
phaseResults.push({
phase: 'drift',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.drift');
const { runPhaseDrift } = await import('./cycle/drift.ts');
const { result, duration_ms } = await timePhase(async (): Promise<PhaseResult> => {
const r = await runPhaseDrift(engine, {
dryRun,
brainDir: brainDir ?? undefined,
forceEnabled: opts.onceForPhase === 'drift',
});
const status: PhaseStatus =
r.status === 'complete' ? 'ok' :
r.status === 'partial' ? 'warn' :
r.status === 'failed' ? 'fail' : 'skipped';
return {
phase: 'drift',
status,
duration_ms: 0,
summary: r.detail,
details: { ...(r.totals ?? {}) },
};
});
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── v0.41.11.0: conversation_facts_backfill ─────────────────
// Opt-in (default OFF). Walks long-form conversation/meeting/slack/
// email pages, segments by 30-min gap, runs facts extractor with a
@@ -2249,7 +2353,7 @@ export async function runCycle(
});
} else {
progress.start('cycle.orphans');
const { result, duration_ms } = await timePhase(() => runPhaseOrphans(engine));
const { result, duration_ms } = await timePhase(() => runPhaseOrphans(engine, orphansSourceId));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
+6 -2
View File
@@ -26,6 +26,7 @@
*/
import { BaseCyclePhase, type ScopedReadOpts, type BasePhaseOpts } from './base-phase.ts';
import { resolveOwnerHolder } from '../owner-holder.ts';
import { chat as gatewayChat } from '../ai/gateway.ts';
import { TIER_DEFAULTS } from '../model-config.ts';
import { gateVoice, type VoiceGateGenerator, type VoiceGateJudge } from '../calibration/voice-gate.ts';
@@ -96,7 +97,7 @@ export type PatternStatementsGenerator = (input: {
export type BiasTagsGenerator = (patterns: string[]) => Promise<string[]>;
export interface CalibrationProfileOpts extends BasePhaseOpts {
/** Holder to generate the profile for. Default 'garry'. */
/** Holder to generate the profile for. Default resolves via resolveOwnerHolder (config emotional_weight.user_holder, else 'self'). */
holder?: string;
/** Inject the patterns generator (tests). */
patternsGenerator?: PatternStatementsGenerator;
@@ -227,7 +228,10 @@ class CalibrationProfilePhase extends BaseCyclePhase {
_ctx: OperationContext,
opts: CalibrationProfileOpts,
): Promise<{ summary: string; details: Record<string, unknown>; status?: PhaseStatus }> {
const holder = opts.holder ?? 'garry';
const holder = resolveOwnerHolder({
override: opts.holder,
configValue: await engine.getConfig('emotional_weight.user_holder'),
});
const promptVersion = opts.promptVersion ?? CALIBRATION_PROFILE_PROMPT_VERSION;
const modelId = opts.model ?? TIER_DEFAULTS.reasoning;
const gradeCompletion = opts.gradeCompletion ?? 1.0;
+250 -35
View File
@@ -1,18 +1,25 @@
/**
* v0.28: drift dream phase.
* Drift dream phase (#2653 wired v0.42.x; scaffold shipped v0.28).
*
* Detects takes where the underlying evidence has shifted since the take
* was made. v0.28 ships the SCAFFOLD: the phase iterates active takes,
* runs a lightweight check against recent timeline entries on the same
* page, and writes a drift-report-<date>.md if any takes look stale.
* Detects takes whose underlying evidence has shifted since the take was
* made. Two stages:
* 1. Cheap SQL heuristic (findDriftCandidates): soft-band takes
* (weight 0.30.85, active, unresolved) on pages with fresh
* timeline_entries evidence inside the lookback window.
* 2. LLM judge: each candidate's claim is compared against the recent
* timeline evidence; the judge returns {drifted, confidence,
* reasoning, suggested_weight}. BudgetMeter-gated.
*
* The full LLM-driven drift detection (compare each take's claim to recent
* page evidence and propose a weight adjustment) is the v0.29 follow-up.
* v0.28 lays the phase orchestration so the contract is stable.
* Output is REPORT-ONLY (v1 conservative posture): judged candidates land
* on a `reports/drift-<date>` page. `dream.drift.auto_update` mutates
* NOTHING in v1 it is recorded in the report so operators can see the
* flag state, and a future wave may wire weight adjustment behind it.
*
* Default-disabled. Operator opts in:
* gbrain config set dream.drift.enabled true
* gbrain config set dream.drift.lookback_days 30
* gbrain config set dream.drift.max_per_cycle 20
* gbrain config set dream.drift.budget 1.0
*/
import type { BrainEngine } from '../engine.ts';
@@ -25,6 +32,10 @@ export interface DriftPhaseOpts {
dryRun: boolean;
/** Override the audit ledger path (tests). */
auditPath?: string;
/** issue #2860 --once: bypass the dream.drift.enabled gate for this run only. */
forceEnabled?: boolean;
/** Inject the judge model call (tests). Defaults to gateway chat. */
judge?: DriftJudgeFn;
}
export interface DriftConfig {
@@ -32,6 +43,7 @@ export interface DriftConfig {
lookbackDays: number;
budgetUsd: number;
autoUpdate: boolean;
maxPerCycle: number;
}
async function loadDriftConfig(engine: BrainEngine): Promise<DriftConfig> {
@@ -39,16 +51,19 @@ async function loadDriftConfig(engine: BrainEngine): Promise<DriftConfig> {
const lookbackStr = await engine.getConfig('dream.drift.lookback_days');
const budgetStr = await engine.getConfig('dream.drift.budget');
const autoStr = await engine.getConfig('dream.drift.auto_update');
const maxPerStr = await engine.getConfig('dream.drift.max_per_cycle');
return {
enabled: enabledStr === 'true',
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
budgetUsd: budgetStr ? Math.max(0, parseFloat(budgetStr) || 1.0) : 1.0,
autoUpdate: autoStr === 'true',
maxPerCycle: maxPerStr ? Math.max(1, parseInt(maxPerStr, 10) || 20) : 20,
};
}
interface DriftCandidate {
export interface DriftCandidate {
takeId: number;
pageId: number;
pageSlug: string;
rowNum: number;
claim: string;
@@ -57,24 +72,121 @@ interface DriftCandidate {
recentEvidenceCount: number;
}
/** Judge verdict: has the claim drifted relative to the evidence? */
export interface DriftVerdict {
drifted: boolean;
confidence: number;
reasoning: string;
/** Judge's suggested new weight (advisory only — v1 never applies it). */
suggested_weight?: number;
}
/** Judge function signature — injected for tests. */
export type DriftJudgeFn = (input: {
candidate: DriftCandidate;
evidence: string;
modelHint?: string;
}) => Promise<DriftVerdict>;
export const DRIFT_JUDGE_PROMPT = `You are auditing a knowledge-base "take" (a weighted claim) for drift:
has newer evidence shifted the ground under this claim since it was made?
Output ONLY one JSON object with these fields:
- drifted (boolean) true when the evidence meaningfully contradicts,
supersedes, or reframes the claim; false when it is
consistent or merely adjacent.
- confidence (number in [0,1]) your confidence in the drifted verdict.
- reasoning (string, <=300 chars) what in the evidence drove the verdict.
- suggested_weight (number in [0,1], optional) where the take's weight should
move if drifted. Advisory only.
If the evidence is sparse or unrelated to the claim, return drifted=false with
low confidence.
TAKE:
Claim: {CLAIM}
Weight: {WEIGHT}
Page: {PAGE}
RECENT EVIDENCE (timeline entries on the same page):
{EVIDENCE_BLOCK}
`;
/**
* Parse the judge model's JSON output. Tolerant of fence wrapping and
* leading prose; returns null on unrecoverable parse failure.
*/
export function parseDriftOutput(raw: string): DriftVerdict | null {
if (!raw || raw.trim().length === 0) return null;
let text = raw.trim();
const fenced = text.match(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/);
if (fenced) text = (fenced[1] ?? '').trim();
const firstObj = text.indexOf('{');
if (firstObj === -1) return null;
let parsed: unknown;
try {
parsed = JSON.parse(text.slice(firstObj));
} catch {
return null;
}
if (typeof parsed !== 'object' || parsed === null) return null;
const r = parsed as Record<string, unknown>;
if (typeof r.drifted !== 'boolean') return null;
const confRaw = typeof r.confidence === 'number' ? r.confidence : Number.parseFloat(String(r.confidence ?? ''));
if (!Number.isFinite(confRaw)) return null;
const verdict: DriftVerdict = {
drifted: r.drifted,
confidence: Math.max(0, Math.min(1, confRaw)),
reasoning: typeof r.reasoning === 'string' ? r.reasoning.slice(0, 300) : '',
};
const sw = typeof r.suggested_weight === 'number' ? r.suggested_weight : NaN;
if (Number.isFinite(sw)) verdict.suggested_weight = Math.max(0, Math.min(1, sw));
return verdict;
}
/** Production judge — calls gateway.chat with the DRIFT_JUDGE_PROMPT. */
export async function defaultDriftJudge(input: {
candidate: DriftCandidate;
evidence: string;
modelHint?: string;
}): Promise<DriftVerdict> {
const { chat } = await import('../ai/gateway.ts');
const prompt = DRIFT_JUDGE_PROMPT
.replace('{CLAIM}', input.candidate.claim)
.replace('{WEIGHT}', String(input.candidate.weight))
.replace('{PAGE}', input.candidate.pageSlug)
.replace('{EVIDENCE_BLOCK}', input.evidence);
const result = await chat({
messages: [{ role: 'user', content: prompt }],
...(input.modelHint ? { model: input.modelHint } : {}),
maxTokens: 400,
});
const parsed = parseDriftOutput(result.text);
if (!parsed) {
// Failed parse — conservative no-drift at zero confidence so the row
// still surfaces in the report instead of disappearing silently.
return { drifted: false, confidence: 0, reasoning: 'judge_output_parse_failed' };
}
return parsed;
}
/**
* Cheap pre-LLM heuristic: takes that have substantial recent timeline
* evidence on the same page MAY have drifted. Surface them; the v0.29
* LLM judge will decide if the weight should move.
* evidence on the same page MAY have drifted. Surface them; the LLM judge
* decides.
*/
async function findDriftCandidates(
engine: BrainEngine,
lookbackDays: number,
): Promise<DriftCandidate[]> {
const cutoffMs = Date.now() - lookbackDays * 86_400_000;
const cutoffIso = new Date(cutoffMs).toISOString().slice(0, 10);
const cutoffIso = lookbackCutoffIso(lookbackDays);
// Only consider takes with weight in the "soft" middle band (0.3..0.85)
// — facts (1.0) don't drift, very-low hunches (<0.3) aren't actionable yet.
const rows = await engine.executeRaw<{
take_id: number; page_slug: string; row_num: number;
take_id: number; page_id: number; page_slug: string; row_num: number;
claim: string; weight: number; recent_evidence: number;
}>(`
SELECT t.id AS take_id, p.slug AS page_slug, t.row_num,
SELECT t.id AS take_id, p.id AS page_id, p.slug AS page_slug, t.row_num,
t.claim, t.weight,
(SELECT count(*)::int FROM timeline_entries te
WHERE te.page_id = p.id
@@ -92,6 +204,7 @@ async function findDriftCandidates(
.filter(r => Number(r.recent_evidence) >= 1)
.map(r => ({
takeId: Number(r.take_id),
pageId: Number(r.page_id),
pageSlug: String(r.page_slug),
rowNum: Number(r.row_num),
claim: String(r.claim),
@@ -100,6 +213,58 @@ async function findDriftCandidates(
}));
}
function lookbackCutoffIso(lookbackDays: number): string {
return new Date(Date.now() - lookbackDays * 86_400_000).toISOString().slice(0, 10);
}
/** Format the candidate page's recent timeline entries as judge evidence. */
async function loadEvidence(
engine: BrainEngine,
pageId: number,
cutoffIso: string,
): Promise<string> {
const rows = await engine.executeRaw<{ date: string; source: string; summary: string }>(
`SELECT date::text AS date, source, summary FROM timeline_entries
WHERE page_id = $1 AND date >= $2::date
ORDER BY date DESC LIMIT 12`,
[pageId, cutoffIso],
);
if (rows.length === 0) return '(no timeline entries in window)';
return rows.map(r => `- ${r.date} [${r.source}] ${r.summary}`).join('\n');
}
interface JudgedCandidate {
candidate: DriftCandidate;
verdict: DriftVerdict;
}
function buildReportBody(
judged: JudgedCandidate[],
cfg: DriftConfig,
modelId: string,
): string {
const drifted = judged.filter(j => j.verdict.drifted);
const lines: string[] = [
`Drift check over active soft-band takes (weight 0.30.85) with timeline`,
`evidence from the last ${cfg.lookbackDays} day(s). Judge model: ${modelId}.`,
``,
`**Report-only:** no takes were modified. \`dream.drift.auto_update\` is`,
`${cfg.autoUpdate ? 'set but ignored in v1 (report-only posture)' : 'off'}; review and adjust weights manually.`,
``,
`${drifted.length} of ${judged.length} judged take(s) look drifted.`,
``,
];
for (const { candidate: c, verdict: v } of judged) {
lines.push(`## ${v.drifted ? 'DRIFTED' : 'stable'}${c.pageSlug} (take #${c.rowNum})`);
lines.push(`- Claim: ${c.claim}`);
lines.push(`- Weight: ${c.weight}${v.suggested_weight !== undefined ? ` → suggested ${v.suggested_weight}` : ''}`);
lines.push(`- Confidence: ${v.confidence.toFixed(2)}`);
if (v.reasoning) lines.push(`- Reasoning: ${v.reasoning}`);
lines.push('');
}
return lines.join('\n');
}
function skipped(_reason: string, detail: string): DreamPhaseResult {
return { name: 'drift', status: 'skipped', detail, duration_ms: 0 };
}
@@ -110,7 +275,7 @@ export async function runPhaseDrift(
): Promise<DreamPhaseResult> {
const start = Date.now();
const config = await loadDriftConfig(engine);
if (!config.enabled) {
if (!config.enabled && !opts.forceEnabled) {
return skipped('not_configured', 'dream.drift.enabled is false');
}
@@ -125,9 +290,16 @@ export async function runPhaseDrift(
};
}
// Resolve model for the (future v0.29) LLM judge. For v0.28 we just
// surface the candidates — the meter call is a no-op when we don't actually
// submit, but resolveModel sets the right pricing key when v0.29 ships.
if (opts.dryRun) {
return {
name: 'drift',
status: 'skipped',
detail: `dry-run: ${Math.min(candidates.length, config.maxPerCycle)} of ${candidates.length} candidates would be evaluated`,
totals: { candidates: candidates.length },
duration_ms: Date.now() - start,
};
}
const modelId = await resolveModel(engine, {
configKey: 'models.drift',
deprecatedConfigKey: 'dream.drift.model',
@@ -139,27 +311,70 @@ export async function runPhaseDrift(
phase: 'drift',
auditPath: opts.auditPath,
});
const judge = opts.judge ?? defaultDriftJudge;
const cutoffIso = lookbackCutoffIso(config.lookbackDays);
// v0.28 scaffold: write a candidate report. v0.29 wires LLM-driven weight
// adjustment through autoUpdate. modelId + meter are wired now so the
// ledger captures the gate state even when we don't submit.
void modelId; void meter;
if (opts.dryRun) {
return {
name: 'drift',
status: 'skipped',
detail: `dry-run: ${candidates.length} candidates would be evaluated`,
totals: { candidates: candidates.length },
duration_ms: Date.now() - start,
};
const judged: JudgedCandidate[] = [];
let budgetExhausted = false;
let failed = 0;
for (const candidate of candidates.slice(0, config.maxPerCycle)) {
const check = meter.check({
modelId,
estimatedInputTokens: 1500,
maxOutputTokens: 400,
label: `drift:${candidate.pageSlug}#${candidate.rowNum}`,
});
if (!check.allowed) {
budgetExhausted = true;
break;
}
const evidence = await loadEvidence(engine, candidate.pageId, cutoffIso);
try {
const verdict = await judge({ candidate, evidence, modelHint: modelId });
judged.push({ candidate, verdict });
} catch (e) {
failed += 1;
process.stderr.write(`[drift] judge failed on take ${candidate.takeId}: ${(e as Error).message}\n`);
}
}
const driftedCount = judged.filter(j => j.verdict.drifted).length;
let reportSlug: string | undefined;
if (judged.length > 0) {
const date = new Date().toISOString().slice(0, 10);
reportSlug = `reports/drift-${date}`;
// Report-only v1: the report page is the ONLY write this phase makes.
// Lands in the default source (brain-global artifact, same-day re-runs
// upsert the same slug).
await engine.putPage(reportSlug, {
type: 'report',
title: `Drift report ${date}`,
compiled_truth: buildReportBody(judged, config, modelId),
});
}
const detail =
`judged ${judged.length}/${candidates.length} candidates: ${driftedCount} drifted` +
(reportSlug ? `${reportSlug}` : '') +
(budgetExhausted ? ' (budget exhausted)' : '') +
(failed > 0 ? ` (${failed} judge failure(s))` : '') +
`. Cumulative cost: $${meter.totalSpent.toFixed(4)} / $${config.budgetUsd.toFixed(2)}` +
`. Report-only: auto_update=${config.autoUpdate} mutates nothing in v1.`;
return {
name: 'drift',
status: 'complete',
detail: `surfaced ${candidates.length} drift candidates (LLM judge: v0.29 follow-up). autoUpdate=${config.autoUpdate}`,
totals: { candidates: candidates.length },
status: judged.length > 0
? (budgetExhausted || failed > 0 ? 'partial' : 'complete')
// Zero judged: budget capped before any judge ran → partial (capped,
// not broken); otherwise every judge call failed → failed.
: (budgetExhausted ? 'partial' : 'failed'),
detail,
totals: {
candidates: candidates.length,
judged: judged.length,
drifted: driftedCount,
failed,
},
duration_ms: Date.now() - start,
};
}
+7 -4
View File
@@ -14,6 +14,8 @@
* See `loadHighEmotionTags` for the resolution path.
*/
import { DEFAULT_OWNER_HOLDER } from '../owner-holder.ts';
/**
* Default high-emotion tag seed list. Pages with any tag in this set get the
* tag-emotion boost in the formula below. Override via config key
@@ -43,11 +45,12 @@ export const HIGH_EMOTION_TAGS: ReadonlySet<string> = new Set([
]);
/**
* Holder name treated as "the user" for the Garry-as-holder ratio. Configurable
* via the `emotional_weight.user_holder` config key (defaults to 'garry' to
* match the v0.28 schema's takes table convention).
* Holder name treated as "the user" for the user-as-holder ratio. Configurable
* via the `emotional_weight.user_holder` config key; defaults to the canonical
* owner holder ('self', DEFAULT_OWNER_HOLDER) so it matches the consolidate
* factstakes writer instead of a hardcoded name.
*/
export const DEFAULT_USER_HOLDER = 'garry';
export const DEFAULT_USER_HOLDER = DEFAULT_OWNER_HOLDER;
export interface EmotionalWeightTake {
holder: string;
+62 -7
View File
@@ -33,8 +33,14 @@ export interface ExtractAtomsDrainDeps {
* routine cycle's skip contract.
*/
withLock: <T>(work: () => Promise<T>) => Promise<T>;
/** Process one bounded batch (rediscovers eligibility). Returns counts. */
runBatch: () => Promise<{ extracted: number; skipped: number }>;
/**
* Process one bounded batch (rediscovers eligibility). Returns counts, plus
* `providerFailure` (issue #3218) when EVERY item the batch attempted threw
* (zero items succeeded, at least one failure) i.e. the batch's warning
* result was actually a total provider outage, not a partial/no-op batch.
* Omit/false for the ordinary partial-success or nothing-to-do cases.
*/
runBatch: () => Promise<{ extracted: number; skipped: number; providerFailure?: boolean }>;
/** Count remaining eligible-but-unextracted pages, or null on query error. */
countRemaining: () => Promise<number | null>;
/** Injectable clock. Production: Date.now. */
@@ -52,15 +58,22 @@ export interface ExtractAtomsDrainOpts {
export interface ExtractAtomsDrainResult {
phase: 'extract_atoms';
status: 'ok';
/**
* issue #3218: 'provider_failure' when any batch reported `providerFailure`
* (every item it attempted errored). The Minion handler throws on this
* status so the durable job retries instead of completing over a backlog
* that made zero forward progress. Partial-success batches (>=1 item
* succeeded) always report 'ok', unchanged from before.
*/
status: 'ok' | 'provider_failure';
extracted: number;
skipped: number;
/** Eligible pages still pending after the window. null if the count errored. */
remaining: number | null;
/** Batches actually processed. */
batches: number;
/** Why the loop stopped: drained | window | no_progress | max_batches. */
stopped: 'drained' | 'window' | 'no_progress' | 'max_batches';
/** Why the loop stopped: drained | window | no_progress | max_batches | provider_failure. */
stopped: 'drained' | 'window' | 'no_progress' | 'max_batches' | 'provider_failure';
}
export async function runExtractAtomsDrain(
@@ -74,6 +87,10 @@ export async function runExtractAtomsDrain(
let skipped = 0;
let batches = 0;
let stopped: ExtractAtomsDrainResult['stopped'] = 'window';
// issue #3218: latched once any batch reports providerFailure — drives
// the returned `status`, independent of how `stopped` reads after the
// final (possibly overriding) remaining-count check below.
let providerFailure = false;
while (deps.now() < deadline) {
if (batches >= maxBatches) { stopped = 'max_batches'; break; }
@@ -87,6 +104,17 @@ export async function runExtractAtomsDrain(
batches++;
deps.onBatch?.({ batch: batches, extracted: r.extracted, remaining: before });
// issue #3218: every item this batch attempted failed (0 succeeded, >=1
// error) — a total provider outage, not ordinary no-op/partial progress.
// Stop immediately (same hot-loop guard as no_progress below) and flag
// it so the caller can retry via its own policy instead of treating the
// drain as a clean completion.
if (r.providerFailure) {
providerFailure = true;
stopped = 'provider_failure';
break;
}
// Stop if a batch made zero forward progress — extraction is failing or
// everything left is ineligible (e.g. all skipped). Prevents a hot loop
// that spends budget without draining.
@@ -94,8 +122,22 @@ export async function runExtractAtomsDrain(
}
const remaining = await deps.countRemaining();
if (remaining === 0) stopped = 'drained';
return { phase: 'extract_atoms', status: 'ok', extracted, skipped, remaining, batches, stopped };
// issue #3218 (codex P2): don't let a final remaining===0 recount
// overwrite 'provider_failure' back to 'drained' — that would report the
// contradictory {status: 'provider_failure', stopped: 'drained'} and
// mislead the CLI/JSON consumer (dream.ts prints both fields verbatim).
// status already takes precedence for the Minion handler's retry
// decision; keep `stopped` consistent with it once a failure latched.
if (!providerFailure && remaining === 0) stopped = 'drained';
return {
phase: 'extract_atoms',
status: providerFailure ? 'provider_failure' : 'ok',
extracted,
skipped,
remaining,
batches,
stopped,
};
});
}
@@ -157,9 +199,22 @@ export async function runExtractAtomsDrainForSource(
brainDir: opts.brainDir,
});
const d = (r.details ?? {}) as Record<string, unknown>;
// issue #3218: `r.status` collapses to 'warn' whether ONE item failed
// (partial success — leave the drain's existing ok/no_progress path
// alone) or EVERY item failed (a total provider outage the drain
// adapter was silently swallowing). Re-derive the total-failure case
// from the per-item counts `runPhaseExtractAtoms` already returns:
// >=1 failure AND zero items successfully processed (transcripts_processed
// + pages_processed both 0 means every attempted `chat()` call threw —
// items that succeed with 0 atoms still count as processed, so this
// does not fire on "provider fine, nothing extractable").
const failures = Array.isArray(d.failures) ? d.failures : [];
const itemsSucceeded =
Number(d.transcripts_processed ?? 0) + Number(d.pages_processed ?? 0);
return {
extracted: Number(d.atoms_extracted ?? 0),
skipped: Number(d.duplicates_skipped ?? 0),
providerFailure: failures.length > 0 && itemsSucceeded === 0,
};
},
countRemaining: () => countExtractAtomsBacklog(engine, extractionSourceId),
+43 -5
View File
@@ -83,6 +83,14 @@ const SYNTHESIS_OUTPUT_TYPES = new Set<string>(['atom', 'concept']);
const PAGE_DISCOVERY_BUDGET = 50;
const MIN_PAGE_CHARS_FOR_EXTRACTION = 500;
// Source pages whose frontmatter declares a `raw` payload pointer hold raw
// import data, not extractable prose. Extraction on them yields zero atoms,
// so no atom row is ever written and they re-enter discovery + the doctor
// backlog count on every cycle — a permanent no-progress loop. Shared by
// discoverExtractablePages and countExtractAtomsBacklog so the phase and the
// doctor check can't drift.
const RAW_SOURCE_HOLDER_EXCLUSION_SQL =
`AND NOT (p.type = 'source' AND COALESCE(p.frontmatter ? 'raw', false))`;
/**
* Pure allowlist policy: the legacy floor UNION the pack's `extractable: true`
@@ -163,10 +171,20 @@ interface ExtractedAtom {
body: string;
source_quote?: string;
lesson?: string;
/**
* 1-3 kebab-case topic labels for concept clustering. Consumed by
* synthesize_concepts (groups atoms by `frontmatter.concepts`; only
* labels shared by >=2 atoms materialize a concept page, so the prompt
* biases reuse-over-coinage). #2123.
*/
concepts?: string[];
virality_score?: number;
emotional_register?: string;
}
/** kebab-case validator for concept labels ("captive-portal", "channel-pricing"). */
const CONCEPT_LABEL_RE = /^[a-z0-9]+(-[a-z0-9]+)*$/;
const EXTRACT_PROMPT = `You extract atomic content nuggets from a transcript.
An atom is a single-source, self-contained idea that could become a tweet,
@@ -177,12 +195,17 @@ quote, or short essay angle. Each atom must:
Output a JSON array of atoms (1-3 per transcript, never more than 3).
Each atom: {title (80 chars), atom_type, body (2-4 sentences),
source_quote (verbatim 200 chars), lesson (one sentence), virality_score
(0-100), emotional_register (one of: shocking, inspiring, funny, sobering,
practical, controversial)}.
source_quote (verbatim 200 chars), lesson (one sentence), concepts
(1-3 topic labels), virality_score (0-100), emotional_register (one of:
shocking, inspiring, funny, sobering, practical, controversial)}.
atom_type MUST be one of: ${ATOM_TYPES.join(', ')}.
concepts are kebab-case English TOPIC labels used to cluster atoms into
concept pages (e.g. "captive-portal", "channel-pricing-strategy") never
entity or brand names. Use the same label for the same topic across atoms;
prefer a label you already used over coining a near-synonym.
Output ONLY the JSON array, no prose.`;
interface DiscoveredPage {
@@ -207,6 +230,10 @@ interface DiscoveredPage {
* participate in the NOT EXISTS check anyway.
* #4 dream_generated exclusion prevents the phase from chewing
* its own output (e.g. dream-generated originals).
* #5 raw source-holder exclusion source pages that only point at a raw
* import payload are not extractable prose; counting them creates a
* permanent backlog/no-progress loop (see
* RAW_SOURCE_HOLDER_EXCLUSION_SQL).
*/
export async function discoverExtractablePages(
engine: BrainEngine,
@@ -225,6 +252,7 @@ export async function discoverExtractablePages(
AND p.content_hash IS NOT NULL
AND COALESCE(p.frontmatter->>'imported_from', '') <> 'markdown-greenfield'
AND COALESCE(p.frontmatter->>'dream_generated', '') <> 'true'
${RAW_SOURCE_HOLDER_EXCLUSION_SQL}
AND length(COALESCE(p.compiled_truth, '')) >= $3
${hasFilter ? "AND p.slug = ANY($5::text[])" : ''}
AND NOT EXISTS (
@@ -297,6 +325,7 @@ export async function countExtractAtomsBacklog(
AND p.content_hash IS NOT NULL
AND COALESCE(p.frontmatter->>'imported_from', '') <> 'markdown-greenfield'
AND COALESCE(p.frontmatter->>'dream_generated', '') <> 'true'
${RAW_SOURCE_HOLDER_EXCLUSION_SQL}
AND length(COALESCE(p.compiled_truth, '')) >= $3
AND NOT EXISTS (
SELECT 1 FROM pages atom
@@ -310,6 +339,7 @@ export async function countExtractAtomsBacklog(
AND p.content_hash IS NOT NULL
AND COALESCE(p.frontmatter->>'imported_from', '') <> 'markdown-greenfield'
AND COALESCE(p.frontmatter->>'dream_generated', '') <> 'true'
${RAW_SOURCE_HOLDER_EXCLUSION_SQL}
AND length(COALESCE(p.compiled_truth, '')) >= $2
AND NOT EXISTS (
SELECT 1 FROM pages atom
@@ -543,7 +573,7 @@ export async function runPhaseExtractAtoms(
content: `Source: ${originLabel}\n\n---\n\n${item.content.slice(0, 50_000)}`,
},
],
maxTokens: 2000,
maxTokens: 4096,
});
// Post-await yield: closes the "long LLM call past TTL" hazard
// codex flagged. The 30s throttle inside maybeYield bounds the
@@ -585,6 +615,7 @@ export async function runPhaseExtractAtoms(
source_hash: item.contentHash.slice(0, 16),
...(atom.source_quote && { source_quote: atom.source_quote }),
...(atom.lesson && { lesson: atom.lesson }),
...(atom.concepts && atom.concepts.length > 0 && { concepts: atom.concepts }),
...(atom.virality_score !== undefined && { virality_score: atom.virality_score }),
...(atom.emotional_register && { emotional_register: atom.emotional_register }),
extracted_at: new Date().toISOString(),
@@ -711,7 +742,7 @@ export function parseAtomsResponse(raw: string): ExtractedAtom[] {
if (typeof item !== 'object' || item === null) continue;
const obj = item as Record<string, unknown>;
const title = typeof obj.title === 'string' ? obj.title.slice(0, 200) : null;
const atomType = typeof obj.atom_type === 'string' ? obj.atom_type : null;
const atomType = typeof obj.atom_type === 'string' ? obj.atom_type.trim().toLowerCase() : null;
const body = typeof obj.body === 'string' ? obj.body : null;
if (!title || !atomType || !body) continue;
if (!ATOM_TYPES.includes(atomType as typeof ATOM_TYPES[number])) continue;
@@ -721,6 +752,13 @@ export function parseAtomsResponse(raw: string): ExtractedAtom[] {
body,
source_quote: typeof obj.source_quote === 'string' ? obj.source_quote.slice(0, 500) : undefined,
lesson: typeof obj.lesson === 'string' ? obj.lesson : undefined,
concepts: (() => {
if (!Array.isArray(obj.concepts)) return undefined;
const labels = obj.concepts
.filter((c): c is string => typeof c === 'string' && CONCEPT_LABEL_RE.test(c))
.slice(0, 3);
return labels.length > 0 ? labels : undefined;
})(),
virality_score:
typeof obj.virality_score === 'number' &&
obj.virality_score >= 0 &&
+24
View File
@@ -70,6 +70,28 @@ export async function runLongMemEvalForProbe(args: LongMemEvalProbeArgs): Promis
* the batch input) or unparseable (cross-modal wrote garbage). Both
* cases are paste-ready in the error message.
*/
/**
* QA-shaped judge dimensions for the nightly probe. The batch judge's
* DEFAULT_DIMENSIONS rubric (DEPTH / SOURCING / SPECIFICITY / ) is built
* for rich agent responses; LongMemEval hypotheses are deliberately terse
* factual answers ("in widget-co") that can never score 7 on DEPTH or
* SOURCING so with the default rubric the probe FAILs every night even
* when retrieval + answering are perfectly healthy. The probe owns its
* invocation of the eval tool and passes dimensions matching the
* fixture's QA shape instead.
*
* NOTE: the `--dimensions` CLI flag splits on commas, so these dimension
* descriptions must stay comma-free.
*/
export const PROBE_QA_DIMENSIONS: string[] = [
// No faithfulness/grounding dimension on purpose: the judge never sees
// the haystack, so any accurate detail beyond the terse gold label reads
// as "invented" and correct answers fail (verified empirically — a
// correct "before + dates" answer scored 4/10 on such a dimension).
'CORRECTNESS — Does the hypothesis state the same fact as the expected answer? A terse direct answer is ideal.',
'DIRECTNESS — Does it answer THIS question without hedging or padding or answering something else?',
];
export async function runCrossModalBatchForProbe(
args: CrossModalProbeArgs,
): Promise<{ exitCode: number; summary: CrossModalBatchSummary }> {
@@ -81,6 +103,8 @@ export async function runCrossModalBatchForProbe(
args.summaryPath,
'--max-usd',
String(args.maxUsd),
'--dimensions',
PROBE_QA_DIMENSIONS.join(','),
'--yes',
'--json',
]);
+43 -11
View File
@@ -62,6 +62,42 @@ export interface NightlyProbeDeps {
now: () => Date;
}
/**
* Dual-plane flag resolution (same precedent as `mcp.publish_skills` in
* serve-http.ts): the DB config row what `gbrain config set` writes
* wins when present; the file plane (~/.gbrain/config.json) is the
* fallback. Doctor's paste-ready enable hint says `gbrain config set
* autopilot.nightly_quality_probe.enabled true`, so the gate MUST read
* the DB plane a file-only read turns that hint into a silent no-op.
*/
export function resolveProbeEnabled(
dbVal: string | null | undefined,
fileVal: unknown,
): boolean {
if (dbVal != null) return dbVal === 'true';
return fileVal === true;
}
/**
* Same dual-plane rule for the per-run cost cap. Malformed or negative
* values on either plane fall through to the next plane / the default.
*/
export function resolveProbeMaxUsd(
dbVal: string | null | undefined,
fileVal: unknown,
fallback: number = DEFAULT_MAX_USD,
): number {
if (dbVal != null) {
const n = Number(dbVal);
if (Number.isFinite(n) && n >= 0) return n;
}
if (fileVal != null) {
const n = Number(fileVal);
if (Number.isFinite(n) && n >= 0) return n;
}
return fallback;
}
/**
* Pure function: decide whether the probe should run given the audit
* history. Returns reason when skipping.
@@ -101,21 +137,17 @@ export async function runNightlyQualityProbe(deps: NightlyProbeDeps): Promise<Ni
return { outcome: 'disabled', exit_code: 0, detail: 'feature flag off' };
}
// 24h rate limit — skip + audit "rate_limited".
// 24h rate limit — skip WITHOUT an audit row. The autopilot loop invokes
// the probe every cycle (~5-10 min), so all but one invocation per day
// lands here; logging each skip floods the audit file (~hundreds of
// rows/day) and — because doctor treats any non-pass outcome as bad
// signal — flips nightly_quality_probe_health to a permanent WARN the
// moment the probe is enabled. A skip is a non-event: the real runs are
// the signal, and their rows are what gates the next 24h window.
const now = deps.now();
const recent = readRecentQualityProbeEvents(2, now); // 2-day window is enough for 24h check
const decision = shouldRunNightly(now, recent);
if (!decision.run) {
logQualityProbeEvent({
outcome: 'rate_limited',
exit_code: 0,
pass_count: 0,
fail_count: 0,
inconclusive_count: 0,
error_count: 0,
est_cost_usd: 0,
detail: 'already ran within 24h window',
});
return { outcome: 'rate_limited', exit_code: 0, detail: 'already ran within 24h' };
}
+123 -105
View File
@@ -39,11 +39,11 @@
import { randomUUID, createHash } from 'node:crypto';
import { BaseCyclePhase, type ScopedReadOpts, type BasePhaseOpts } from './base-phase.ts';
import { chat as gatewayChat, getChatModel } from '../ai/gateway.ts';
import { chat as gatewayChat, getChatModel, probeChatModel } from '../ai/gateway.ts';
import { normalizeModelId } from '../model-id.ts';
import { writeReceipt } from '../extract/receipt-writer.ts';
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
import { GBrainError } from '../types.ts';
import type { Page, PageFilters } from '../types.ts';
import type { OperationContext } from '../operations.ts';
import type { BrainEngine } from '../engine.ts';
import type { PhaseStatus, CyclePhase } from '../cycle.ts';
@@ -55,17 +55,6 @@ import type { PhaseStatus, CyclePhase } from '../cycle.ts';
*/
export const PROPOSE_TAKES_PROMPT_VERSION = 'v0.36.1.0-tuned-cat15';
/**
* Sentinel claim_text for the tombstone row written when a page extracts
* ZERO gradeable claims. Without a tombstone the idempotency tuple is never
* recorded, so every cycle re-spends an LLM call on unchanged zero-claim
* prose the "unchanged page never re-spends tokens" contract only held
* for pages that produced >=1 claim. The tombstone is inserted with
* status='rejected' so no pending-review query surfaces it as a live
* proposal; its only job is to make the next cycle a cache hit.
*/
export const EMPTY_EXTRACTION_TOMBSTONE_TEXT = '(no gradeable claims)';
/**
* Tuned extractor prompt, validated against the hand-labeled synthetic
* corpus at test/fixtures/calibration/. Measured F1 on first live run
@@ -156,6 +145,8 @@ export interface ProposeTakesOpts extends BasePhaseOpts {
model?: string;
/** Skip pages that already have a complete takes fence. Default: true. */
skipPagesWithFence?: boolean;
/** Override the phase wall-clock deadline (tests). Default: 30 min. */
deadlineMs?: number;
}
export interface ProposeTakesResult {
@@ -163,12 +154,54 @@ export interface ProposeTakesResult {
cache_hits: number;
cache_misses: number;
proposals_inserted: number;
/** Idempotency rows written for pages that extracted zero claims. */
tombstones_written: number;
budget_exhausted: boolean;
/** True when the phase deadline fired before the page loop completed (partial result). */
deadline_hit?: boolean;
warnings: string[];
}
/** Narrow projection of `pages` — the only columns this phase reads. */
interface ProposeTakesPageRow {
slug: string;
source_id: string;
compiled_truth: string | null;
}
/**
* Load proposal candidates with a narrow projection instead of
* `engine.listPages` (`SELECT p.*`). The phase only reads slug, source_id
* and compiled_truth skipping timeline/frontmatter/title keeps large
* toasted columns out of the hot path. Scope precedence mirrors
* `sourceScopeOpts`: federated array (`sourceIds`) beats scalar
* (`sourceId`); ordering matches `PAGE_SORT_SQL.updated_desc` with an id
* tiebreak for determinism. (Takeover of PR #1979's projection by
* @shawnduggan.)
*/
async function listCandidatePages(
engine: BrainEngine,
scope: ScopedReadOpts,
limit: number,
): Promise<ProposeTakesPageRow[]> {
const where = ['deleted_at IS NULL'];
const params: unknown[] = [];
if (scope.sourceIds && scope.sourceIds.length > 0) {
params.push(scope.sourceIds);
where.push(`source_id = ANY($${params.length}::text[])`);
} else if (scope.sourceId) {
params.push(scope.sourceId);
where.push(`source_id = $${params.length}`);
}
params.push(limit);
return engine.executeRaw<ProposeTakesPageRow>(
`SELECT slug, source_id, compiled_truth
FROM pages
WHERE ${where.join(' AND ')}
ORDER BY updated_at DESC, id DESC
LIMIT $${params.length}`,
params,
);
}
/**
* Compute the content_hash key for the idempotency cache. SHA-256 of the
* page body suffices page slug + prompt_version are separate columns in
@@ -223,6 +256,9 @@ export function extractExistingTakesForDedup(pageBody: string): Array<{
return rows;
}
/** Per-call wall-clock timeout for the extractor LLM call. */
const EXTRACTOR_CALL_TIMEOUT_MS = 90_000;
/**
* Production extractor calls gateway.chat with the EXTRACT_TAKES_PROMPT
* and parses the JSON array output. Returns [] on parse failure (logged as
@@ -240,50 +276,18 @@ export async function defaultExtractor(
.replace('{EXISTING_TAKES_JSON}', JSON.stringify(input.existingTakes, null, 2))
.replace('{PAGE_BODY}', input.pageBody);
// Bound each call so one stalled provider socket can't pin the phase for the
// full gateway default (GBRAIN_AI_CHAT_TIMEOUT_MS, 300s) x pageLimit. The
// caller already catches per-page errors, logs a warning, and continues.
const result = await gatewayChat({
messages: [{ role: 'user', content: prompt }],
...(input.modelHint ? { model: input.modelHint } : {}),
maxTokens: 2048,
abortSignal: AbortSignal.timeout(EXTRACTOR_CALL_TIMEOUT_MS),
});
// ChatResult.text is already the concatenated text content.
const takes = parseExtractorOutput(result.text);
// A parse-level `[]` is AMBIGUOUS: it means either "the model genuinely
// found no gradeable claims" OR "the model returned malformed/prose/
// truncated output we couldn't parse." The caller memoizes empty
// extractions with a tombstone, so a transient parse failure would
// PERMANENTLY suppress a page that actually has claims. Only a cleanly
// parsed empty array is a real "no claims" result worth memoizing; treat
// anything else as a transient error and throw, so the phase's catch
// retries the page next cycle (writing no tombstone).
if (takes.length === 0 && !isWellFormedEmptyExtraction(result.text)) {
throw new Error('propose_takes extractor: no parseable takes JSON (transient — retry)');
}
return takes;
}
/**
* True only when `raw` is a cleanly-parseable EMPTY JSON array the
* well-behaved "no gradeable claims" response (the prompt instructs the model
* to return `[]`). Distinguishes a genuine empty extraction (safe to memoize
* via a tombstone) from malformed / prose / truncated output (transient
* must be retried, never tombstoned). Mirrors parseExtractorOutput's
* fence-strip + first-array handling so both agree on what "the model
* returned []" means.
*/
export function isWellFormedEmptyExtraction(raw: string): boolean {
if (!raw || raw.trim().length === 0) return false;
let text = raw.trim();
const fenced = text.match(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/);
if (fenced) text = (fenced[1] ?? '').trim();
const arrStart = text.indexOf('[');
if (arrStart === -1) return false;
try {
const parsed = JSON.parse(text.slice(arrStart));
return Array.isArray(parsed) && parsed.length === 0;
} catch {
return false;
}
return parseExtractorOutput(result.text);
}
/**
@@ -352,6 +356,14 @@ class ProposeTakesPhase extends BaseCyclePhase {
readonly name = 'propose_takes' as CyclePhase;
protected readonly budgetUsdKey = 'cycle.propose_takes.budget_usd';
protected readonly budgetUsdDefault = 5.0;
/**
* Hard wall-clock deadline for the phase. Even with the per-call timeout in
* defaultExtractor, a long tail of slow-but-completing calls can accumulate.
* The phase breaks cleanly and returns a partial result with
* `deadline_hit: true` instead of being killed mid-write by an outer
* `timeout` wrapper (the recurring SIGTERM in nightly dream runs).
*/
private static readonly PHASE_DEADLINE_MS = 30 * 60 * 1000;
protected override mapErrorCode(err: unknown): string {
if (err instanceof GBrainError) return err.problem;
@@ -372,33 +384,67 @@ class ProposeTakesPhase extends BaseCyclePhase {
const promptVersion = opts.promptVersion ?? PROPOSE_TAKES_PROMPT_VERSION;
const pageLimit = opts.pageLimit ?? 100;
const skipPagesWithFence = opts.skipPagesWithFence ?? false;
const deadlineMs = opts.deadlineMs ?? ProposeTakesPhase.PHASE_DEADLINE_MS;
const phaseStartMs = Date.now();
const proposalRunId = `propose-${new Date().toISOString().slice(0, 19).replace(/[-:T]/g, '')}-${randomUUID().slice(0, 8)}`;
const modelId = opts.model ?? getChatModel();
// With the default (gateway) extractor, skip cheaply when the resolved
// model's provider can't run — same probe semantics as patterns.ts /
// think/index.ts: unknown provider/model or Anthropic-without-key skips;
// other providers' auth surfaces lazily at chat() time. An injected
// extractor bypasses the gateway, so it is never gated. (Takeover of
// PR #1979's intent by @shawnduggan.)
if (!opts.extractor) {
const probe = probeChatModel(normalizeModelId(modelId));
if (!probe.ok) {
return {
summary: `propose_takes skipped: ${probe.detail}`,
details: {
reason: 'no_provider',
model: modelId,
pages_scanned: 0,
cache_hits: 0,
cache_misses: 0,
proposals_inserted: 0,
budget_exhausted: false,
warnings: [],
},
status: 'skipped',
};
}
}
const result: ProposeTakesResult = {
pages_scanned: 0,
cache_hits: 0,
cache_misses: 0,
proposals_inserted: 0,
tombstones_written: 0,
budget_exhausted: false,
warnings: [],
};
// Load pages eligible for proposal. Source-scoped per BaseCyclePhase.
const pageFilters: PageFilters = {
...scope,
limit: pageLimit,
sort: 'updated_desc',
};
const pages: Page[] = await engine.listPages(pageFilters);
const pages = await listCandidatePages(engine, scope, pageLimit);
if (opts.reporter) {
opts.reporter.start('propose_takes.pages' as never, pages.length);
}
const modelId = opts.model ?? getChatModel();
for (const page of pages) {
// Phase deadline check. Break (not throw) so the phase returns a
// partial result with deadline_hit:true; work already banked stays.
const elapsedMs = Date.now() - phaseStartMs;
if (elapsedMs > deadlineMs) {
result.warnings.push(
`phase deadline hit at page ${result.pages_scanned}/${pages.length} ` +
`after ${(elapsedMs / 1000).toFixed(0)}s (cap ${(deadlineMs / 1000).toFixed(0)}s); partial completion`,
);
result.deadline_hit = true;
break;
}
result.pages_scanned += 1;
this.tick(opts);
@@ -454,16 +500,18 @@ class ProposeTakesPhase extends BaseCyclePhase {
continue;
}
// Write proposals to take_proposals. Each row is a separate INSERT
// because the composite idempotency key is on the per-page tuple — a
// bulk UPSERT would collapse a same-page-multi-claim run into one row.
// Write proposals to take_proposals. #2138: the idempotency key is
// per-CLAIM — take_proposals_idempotency_idx folds md5(claim_text) into
// the per-page tuple (migration v125), so a multi-claim page keeps every
// claim. RETURNING id prevents a repeated claim from inflating the count.
for (const p of proposals) {
await engine.executeRaw(
const inserted = await engine.executeRaw<{ id: number }>(
`INSERT INTO take_proposals
(source_id, page_slug, content_hash, prompt_version, proposal_run_id,
claim_text, kind, holder, weight, domain, dedup_against_fence_rows, model_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
ON CONFLICT (source_id, page_slug, content_hash, prompt_version) DO NOTHING`,
ON CONFLICT (source_id, page_slug, content_hash, prompt_version, md5(claim_text)) DO NOTHING
RETURNING id`,
[
sourceId,
page.slug,
@@ -479,41 +527,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
modelId,
],
);
result.proposals_inserted += 1;
}
// Memoize the empty case too. A page that extracted zero claims gets
// NO row from the loop above, so without this its idempotency tuple is
// never recorded and the next cycle re-spends an LLM call on unchanged
// prose (the idle-cost bug). Write one tombstone row keyed by the same
// (source, slug, content_hash, prompt_version) tuple. status='rejected'
// keeps it out of any pending-review query; its sole purpose is to make
// the next cycle a cache hit. Only reached on a SUCCESSFUL empty extract
// — the extractor-throw path `continue`s above, so failed pages are
// retried rather than tombstoned.
if (proposals.length === 0) {
await engine.executeRaw(
`INSERT INTO take_proposals
(source_id, page_slug, content_hash, prompt_version, proposal_run_id,
claim_text, kind, holder, weight, domain, dedup_against_fence_rows, model_id, status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, 'rejected')
ON CONFLICT (source_id, page_slug, content_hash, prompt_version) DO NOTHING`,
[
sourceId,
page.slug,
ch,
promptVersion,
proposalRunId,
EMPTY_EXTRACTION_TOMBSTONE_TEXT,
'fact',
'brain',
0,
null,
JSON.stringify(existingTakes),
opts.model ?? 'claude-sonnet-4-6',
],
);
result.tombstones_written += 1;
result.proposals_inserted += inserted.length;
}
}
@@ -540,17 +554,20 @@ class ProposeTakesPhase extends BaseCyclePhase {
console.error(`[propose_takes] receipt write failed: ${(err as Error).message}`);
}
}
// A deadline-hit run halted mid-list the same way a budget-exhausted one
// does — record it as a halt, not a completed round.
const halted = result.budget_exhausted || result.deadline_hit === true;
await upsertExtractRollup(engine, {
kind: 'takes.proposed',
source_id: sourceIdForReceipt,
round_completed_delta: result.budget_exhausted ? 0 : 1,
halt_delta: result.budget_exhausted ? 1 : 0,
round_completed_delta: halted ? 0 : 1,
halt_delta: halted ? 1 : 0,
});
return {
summary: `propose_takes: scanned ${result.pages_scanned} pages, ${result.cache_hits} cached, ${result.proposals_inserted} new proposals, ${result.tombstones_written} empty (run ${proposalRunId})`,
summary: `propose_takes: scanned ${result.pages_scanned} pages, ${result.cache_hits} cached, ${result.proposals_inserted} new proposals (run ${proposalRunId})`,
details: { ...result, proposal_run_id: proposalRunId, prompt_version: promptVersion },
status: result.budget_exhausted ? 'warn' : 'ok',
status: result.budget_exhausted || result.deadline_hit ? 'warn' : 'ok',
};
}
}
@@ -573,4 +590,5 @@ export const __testing = {
contentHash,
hasCompleteFence,
extractExistingTakesForDedup,
listCandidatePages,
};
+171 -12
View File
@@ -28,6 +28,7 @@
import type Anthropic from '@anthropic-ai/sdk';
import { readFileSync, existsSync, writeFileSync, mkdirSync } from 'node:fs';
import { randomUUID } from 'node:crypto';
import { chat as gatewayChat, validateModelId, type ChatResult } from '../ai/gateway.ts';
import { AIConfigError } from '../ai/errors.ts';
import { normalizeModelId } from '../model-id.ts';
@@ -37,16 +38,18 @@ import type { BrainEngine } from '../engine.ts';
import type { PhaseResult, PhaseError } from '../cycle.ts';
import { MinionQueue } from '../minions/queue.ts';
import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts';
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
import { makeSubagentHandler } from '../minions/handlers/subagent.ts';
import type { MinionJobInput, MinionJobContext, MinionHandler, SubagentHandlerData } from '../minions/types.ts';
import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts';
import { serializeMarkdown, serializePageToMarkdown } from '../markdown.ts';
import type { Page, PageType } from '../types.ts';
import { validateSourceId } from '../utils.ts';
import { safeSplitIndex } from '../text-safe.ts';
import { PAGE_SLUG_SEG } from '../cjk.ts';
// Slug regex from validatePageSlug — kept in sync.
// Slug grammar from validatePageSlug — shared via PAGE_SLUG_SEG (#738).
// Used for the orchestrator-written summary index slug.
const SUMMARY_SLUG_RE = /^[a-z0-9][a-z0-9\-]*(\/[a-z0-9][a-z0-9\-]*)*$/;
const SUMMARY_SLUG_RE = new RegExp(`^${PAGE_SLUG_SEG}(\\/${PAGE_SLUG_SEG})*$`);
// ── Model context budget (D1, D5, D7, D9) ─────────────────────────────
@@ -261,6 +264,121 @@ export interface SynthesizePhaseOpts {
once?: boolean;
}
const INLINE_PGLITE_LOCK_MS = 30_000;
/**
* PGLite cannot be served by a separate Minions worker process: the embedded
* data-dir holds an exclusive file lock, so subagent children enqueued by the
* synth parent would sit in 'waiting' until waitForCompletion times out.
* Drive the same claim run complete/fail loop a worker would perform,
* inline, against this phase's private child queue.
*
* `yieldDuringPhase` is ticked on a 60s interval while a child runs so the
* 5-min cycle lock TTL keeps refreshing during long (up to 30-min) children.
*/
async function runPgliteSubagentsInline(
engine: BrainEngine,
queue: MinionQueue,
queueName: string,
yieldDuringPhase?: () => Promise<void>,
handler: MinionHandler = makeSubagentHandler({ engine }),
): Promise<void> {
if (engine.kind !== 'pglite') return;
while (true) {
// Housekeeping a worker would normally perform, so child rows can reach
// terminal states (delayed retries promoted, timeouts dead-lettered)
// before the synth parent enters waitForCompletion polling.
await queue.promoteDelayed();
await queue.handleStalled();
await queue.handleTimeouts();
await queue.handleWallClockTimeouts(INLINE_PGLITE_LOCK_MS);
const lockToken = randomUUID();
const job = await queue.claim(lockToken, INLINE_PGLITE_LOCK_MS, queueName, ['subagent']);
if (!job) return;
const abort = new AbortController();
const shutdown = new AbortController();
const context: MinionJobContext = {
id: job.id,
name: job.name,
data: job.data,
attempts_made: job.attempts_made,
signal: abort.signal,
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
shutdownSignal: shutdown.signal,
updateProgress: async (progress: unknown) => {
await queue.updateProgress(job.id, lockToken, progress);
},
updateTokens: async (tokens) => {
await queue.updateTokens(job.id, lockToken, tokens);
},
log: async (message) => {
const value = typeof message === 'string' ? message : JSON.stringify(message);
await engine.executeRaw(
`UPDATE minion_jobs SET stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
updated_at = now()
WHERE id = $2 AND status = 'active' AND lock_token = $3`,
[value, job.id, lockToken],
);
},
isActive: async () => {
const rows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM minion_jobs WHERE id = $1 AND status = 'active' AND lock_token = $2`,
[job.id, lockToken],
);
return rows.length > 0;
},
readInbox: async () => queue.readInbox(job.id, lockToken),
};
// Per-job deadline enforcement (worker.ts parity). While the drain loop
// awaits the handler, the handleTimeouts sweep above can't run, so nothing
// else can stop a child that blows past timeout_ms — the handler only
// stops when ctx.signal fires. Derive the delay from the claim-time
// timeout_at stamp so timer, DB sweeper, and deadlineAtMs agree.
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
if (job.timeout_ms != null) {
const delayMs = job.timeout_at != null
? Math.max(0, job.timeout_at.getTime() - Date.now())
: job.timeout_ms;
timeoutTimer = setTimeout(() => {
if (!abort.signal.aborted) abort.abort(new Error('timeout'));
}, delayMs);
}
// Cycle-lock keepalive while the child runs (best-effort, never throws).
const keepalive = yieldDuringPhase
? setInterval(() => { yieldDuringPhase().catch(() => { /* best-effort */ }); }, 60_000)
: null;
try {
const result = await handler(context);
await queue.completeJob(
job.id,
lockToken,
result != null ? (typeof result === 'object' ? result as Record<string, unknown> : { value: result }) : undefined,
);
} catch (e) {
// Timeout is terminal (handleTimeouts parity: stall → retry,
// timeout → dead), never a delayed retry.
const timedOut = abort.signal.aborted;
const errorText = timedOut ? 'timeout exceeded' : (e instanceof Error ? e.message : String(e));
const attemptsExhausted = job.attempts_made + 1 >= job.max_attempts;
await queue.failJob(
job.id,
lockToken,
errorText,
timedOut || attemptsExhausted ? 'dead' : 'delayed',
0,
);
} finally {
if (timeoutTimer) clearTimeout(timeoutTimer);
if (keepalive) clearInterval(keepalive);
}
}
}
export async function runPhaseSynthesize(
engine: BrainEngine,
opts: SynthesizePhaseOpts,
@@ -427,9 +545,17 @@ export async function runPhaseSynthesize(
}
const queue = new MinionQueue(engine);
// PGLite children drain inline (no separate worker can open the embedded
// data-dir), so give them a private per-run queue: the inline drain must
// never claim unrelated 'default'-queue jobs a Postgres worker owns.
const childQueueName = engine.kind === 'pglite'
? `dream-inline-${Date.now()}-${randomUUID().slice(0, 8)}`
: 'default';
const childIds: number[] = [];
/** Map child job_id → chunk metadata for D6 orchestrator-side slug rewrite. */
const chunkInfo = new Map<number, { idx: number; hash6: string }>();
/** #1978: map child job_id → source transcript path so written pages get a raw_source stamp. */
const jobRawSource = new Map<number, string>();
/** Skip reasons for the cycle report (D5 cap hits, D8 legacy-key skips). */
const skipReports: Array<{ filePath: string; reason: string }> = [];
@@ -505,6 +631,7 @@ export async function runPhaseSynthesize(
on_child_fail: 'continue',
idempotency_key,
timeout_ms: config.subagentTimeoutMs,
queue: childQueueName,
};
const child = await queue.add(
'subagent',
@@ -513,12 +640,19 @@ export async function runPhaseSynthesize(
{ allowProtectedSubmit: true },
);
childIds.push(child.id);
jobRawSource.set(child.id, t.filePath);
if (isChunked) {
chunkInfo.set(child.id, { idx: i, hash6 });
}
}
}
// PGLite cannot run a separate Minions worker because the embedded DB
// holds an exclusive file lock. Drain this phase's private child queue
// inline so the parent observes terminal child states instead of polling
// waiters until subagentWaitTimeoutMs expires. No-op on Postgres.
await runPgliteSubagentsInline(engine, queue, childQueueName, opts.yieldDuringPhase);
// Wait for every child to reach a terminal state. Tick yieldDuringPhase
// every 5 min so the cycle lock TTL refreshes.
const childOutcomes: Array<{ jobId: number; status: string }> = [];
@@ -551,7 +685,7 @@ export async function runPhaseSynthesize(
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
// source (children write there via SubagentHandlerData.source_id).
const cycleSourceId = opts.sourceId ?? 'default';
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId, jobRawSource);
const summaryDate = opts.date ?? today();
@@ -1103,7 +1237,8 @@ async function collectChildPutPageSlugs(
childIds: number[],
chunkInfo: Map<number, { idx: number; hash6: string }>,
sourceId = 'default',
): Promise<Array<{ slug: string; source_id: string }>> {
jobRawSource?: Map<number, string>,
): Promise<Array<{ slug: string; source_id: string; raw_source?: string }>> {
if (childIds.length === 0) return [];
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
// the orchestrator sees what each child wrote. COALESCE handles both
@@ -1125,13 +1260,21 @@ async function collectChildPutPageSlugs(
AND status = 'complete'`,
[childIds],
);
const rewritten = new Set<string>();
// #1978: slug → source transcript path (first writer wins) so the
// provenance stamp can record WHERE the synthesized content came from.
const rewritten = new Map<string, string | undefined>();
for (const r of rows) {
if (typeof r.slug !== 'string' || r.slug.length === 0) continue;
const ci = chunkInfo.get(r.job_id);
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
const slug = ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug;
if (!rewritten.has(slug) || rewritten.get(slug) === undefined) {
rewritten.set(slug, jobRawSource?.get(r.job_id));
}
}
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
return Array.from(rewritten.keys()).sort().map(slug => {
const raw_source = rewritten.get(slug);
return { slug, source_id: sourceId, ...(raw_source ? { raw_source } : {}) };
});
}
/**
@@ -1177,12 +1320,12 @@ async function hasLegacySingleChunkCompletion(
*/
async function stampDreamProvenance(
engine: BrainEngine,
refs: Array<{ slug: string; source_id: string }>,
refs: Array<{ slug: string; source_id: string; raw_source?: string }>,
cycleDate: string,
): Promise<void> {
if (refs.length === 0) return;
const { executeRawJsonb } = await import('../sql-query.ts');
for (const { slug, source_id } of refs) {
for (const { slug, source_id, raw_source } of refs) {
try {
await executeRawJsonb(
engine,
@@ -1190,7 +1333,14 @@ async function stampDreamProvenance(
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
WHERE slug = $1 AND source_id = $2`,
[slug, source_id],
[{ dream_generated: true, dream_cycle_date: cycleDate }],
// #1978 raw-source persistence: record the transcript path the
// synthesis was derived from, so `gbrain doctor` (raw_provenance
// check) can verify every generated page carries a raw trace.
[{
dream_generated: true,
dream_cycle_date: cycleDate,
...(raw_source ? { raw_source } : {}),
}],
);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
@@ -1292,7 +1442,15 @@ async function writeSummaryPage(
// parseMarkdown below round-trips it into the DB-stored frontmatter, so the
// marker survives any later reverse-render of the summary page.
const fullMarkdown = serializeMarkdown(
{ dream_generated: true, dream_cycle_date: summaryDate } as Record<string, unknown>,
{
dream_generated: true,
dream_cycle_date: summaryDate,
// #1978: deterministic index page — no source document of its own;
// raw traces live on the listed pages. Explicit exemption keeps the
// doctor raw_provenance check quiet.
raw_trace_exempt: true,
raw_trace_exempt_reason: 'deterministic dream-cycle index; raw traces live on listed pages',
} as Record<string, unknown>,
body,
'',
{ type: 'note' as string, title: `Dream cycle ${summaryDate}`, tags: ['dream-cycle'] },
@@ -1381,4 +1539,5 @@ export const __testing = {
buildSynthesisPrompt,
stampDreamProvenance,
reverseWriteRefs,
runPgliteSubagentsInline,
};
+12 -3
View File
@@ -35,10 +35,11 @@
*
* The doctor renders both side by side.
*
* Drift contract: every check name that ships in doctor.ts MUST appear in
* Drift contract: every check name that ships through doctor MUST appear in
* exactly one set below. The drift-guard test in
* `test/doctor-categories.test.ts` enforces this by reading doctor.ts source
* via a tagged-string scan and asserting set membership exactly.
* `test/doctor-categories.test.ts` enforces this by reading doctor check
* emitter sources via a tagged-string scan and asserting set membership
* exactly.
*
* If you add a new doctor check, you MUST add its name to the appropriate
* set here. The categorize step in `src/commands/doctor.ts` falls through
@@ -67,12 +68,15 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'conversation_parser_probe_health',
'cross_modal_modality_backfill',
'cycle_freshness',
'dangling_aliases',
'effective_date_health',
'embed_staleness',
'embedding_column_registry',
'embedding_env_override',
'embedding_provider',
'embedding_width_consistency',
'embeddings',
'entity_link_coverage',
'eval_drift',
'extract_atoms_backlog',
'extract_health',
@@ -95,6 +99,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'orphan_ratio',
'oversized_pages',
'quarantined_pages',
'raw_provenance',
'flagged_pages',
'salience_health',
'scraper_junk_pages',
@@ -102,7 +107,9 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'stub_guard_24h',
'sync_failures',
'sync_freshness',
'takes_count',
'takes_weight_grid',
'timeline_coverage',
'unified_multimodal_coverage',
'voice_gate_health',
]);
@@ -170,12 +177,14 @@ export const META_CHECK_NAMES: ReadonlySet<string> = new Set([
'eval_capture',
'minions_migration',
'multi_source_drift',
'pack_upgrade_available',
'schema_pack_active',
'schema_pack_consistency',
'schema_pack_source_drift',
'schema_version',
'slug_fallback_audit',
'timeline_dedup_index',
'type_proliferation',
'upgrade_errors',
]);
+26 -4
View File
@@ -14,7 +14,7 @@
*/
import type { BrainEngine } from './engine.ts';
import { PGVECTOR_HNSW_VECTOR_MAX_DIMS } from './vector-index.ts';
import { PGVECTOR_HNSW_VECTOR_MAX_DIMS, hnswMaxDimsForType } from './vector-index.ts';
import { gbrainPath } from './config.ts';
import { resolveRecipe } from './ai/model-resolver.ts';
import type { Recipe } from './ai/types.ts';
@@ -32,6 +32,10 @@ import {
nvidiaEmbeddingDim,
nvidiaEmbeddingDimOptions,
supportsNvidiaEmbeddingDimension,
isPerplexityEmbeddingModel,
isValidPerplexityDim,
maxPerplexityEmbeddingDim,
PERPLEXITY_MIN_DIMS,
} from './ai/dims.ts';
/**
@@ -462,6 +466,15 @@ function isCustomDimValidForProvider(
`(allowed: ${ZEROENTROPY_VALID_DIMS.join(', ')}).`,
};
}
if (recipe.id === 'perplexity' && isPerplexityEmbeddingModel(modelId)) {
if (isValidPerplexityDim(modelId, requestedDims)) return { valid: true, error: '' };
return {
valid: false,
error:
`Perplexity ${modelId} accepts dimensions ${PERPLEXITY_MIN_DIMS}..${maxPerplexityEmbeddingDim(modelId)}, ` +
`got ${requestedDims}.`,
};
}
if (recipe.id === 'openai' && isOpenAITextEmbedding3Model(modelId)) {
if (isValidOpenAITextEmbedding3Dim(modelId, requestedDims)) return { valid: true, error: '' };
const maxDim = maxOpenAITextEmbedding3Dim(modelId);
@@ -609,6 +622,17 @@ export function buildFactsAlterRecipe(
const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
const targetType = columnType === 'halfvec' ? `halfvec(${configuredDims})` : `vector(${configuredDims})`;
const dimsChanged = columnDims !== configuredDims;
const hnswMaxDims = hnswMaxDimsForType(columnType);
const indexLines = configuredDims <= hnswMaxDims
? [
`CREATE INDEX idx_facts_embedding_hnsw`,
` ON facts USING hnsw (embedding ${opclass})`,
` WHERE embedding IS NOT NULL AND expired_at IS NULL;`,
]
: [
`-- Skip reindex. ${columnType}(${configuredDims}) exceeds pgvector's HNSW cap of ${hnswMaxDims};`,
`-- fact similarity falls back to exact scans.`,
];
return [
`-- ALTER ${columnType}(${columnDims}) → ${columnType}(${configuredDims}) on indexed column.`,
`-- HOLD a maintenance window: this rewrites every row's embedding.`,
@@ -629,9 +653,7 @@ export function buildFactsAlterRecipe(
: []),
`ALTER TABLE facts ALTER COLUMN embedding TYPE ${targetType}`,
` USING embedding::${targetType};`,
`CREATE INDEX idx_facts_embedding_hnsw`,
` ON facts USING hnsw (embedding ${opclass})`,
` WHERE embedding IS NOT NULL AND expired_at IS NULL;`,
...indexLines,
].join('\n');
}
+7
View File
@@ -37,9 +37,16 @@ export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
// ZeroEntropy reranker (docs/ai-providers/zeroentropy.md — $0.025/1M tokens).
// Reused here (not a separate rerank table) because budget-tracker.ts's
// rerank-kind lookup falls back to this same table for paid providers.
'zeroentropyai:zerank-2': { pricePerMTok: 0.025 },
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-19)
'mistral:mistral-embed': { pricePerMTok: 0.10 },
'mistral:mistral-embed-2312': { pricePerMTok: 0.10 },
// Perplexity (https://docs.perplexity.ai/getting-started/pricing, verified 2026-07-21)
'perplexity:pplx-embed-v1-0.6b': { pricePerMTok: 0.004 },
'perplexity:pplx-embed-v1-4b': { pricePerMTok: 0.03 },
};
export type PriceLookupResult =
+5 -1
View File
@@ -1904,7 +1904,11 @@ export interface BrainEngine {
// Ingest log
logIngest(entry: IngestLogInput): Promise<void>;
getIngestLog(opts?: { limit?: number }): Promise<IngestLogEntry[]>;
/**
* `opts.sourceIds` scopes the log to those sources (federated read grant /
* remote caller scope). Omitted whole brain (trusted local callers).
*/
getIngestLog(opts?: { limit?: number; sourceIds?: string[] }): Promise<IngestLogEntry[]>;
// Sync
/**
+26 -21
View File
@@ -58,7 +58,11 @@ export interface BudgetStateRow {
// Errors
// ---------------------------------------------------------------------------
export type BudgetErrorCode = 'reservation_not_found' | 'already_finalized' | 'invalid_input';
export type BudgetErrorCode =
| 'reservation_not_found'
| 'already_finalized'
| 'invalid_input'
| 'cap_exceeded';
export class BudgetError extends Error {
constructor(public code: BudgetErrorCode, message: string, public reservationId?: string) {
@@ -169,12 +173,11 @@ export class BudgetLedger {
* committed_usd up by the actual.
*
* Re-checks the cap against the post-commit total: reserving $0.01 then
* committing $100 against a $1 cap must not silently blow through. When
* actualUsd would exceed the effective cap, the commit clamps to (cap -
* other_committed - other_reserved) and throws. The reservation is still
* marked committed (the API call already happened and we don't want
* retry loops), but the excess is attributed as a cap-exhaustion error
* the caller can log.
* committing $100 against a $1 cap must not silently blow through. The API
* call has already happened, so actualUsd is recorded in full (truthful
* accounting), the reservation is finalized, and a cap_exceeded error is
* thrown only AFTER the transaction commits. Throwing inside the transaction
* would roll back both writes and leave a paid call looking pending.
*
* Negative actuals are rejected refunds should be a separate operation,
* not a side-channel on commit().
@@ -187,7 +190,7 @@ export class BudgetLedger {
throw new BudgetError('invalid_input', `commit: actualUsd must be non-negative (got ${actualUsd}). Use a dedicated refund API instead.`);
}
return await this.engine.transaction(async (tx) => {
const capError = await this.engine.transaction(async (tx) => {
const rows = await tx.executeRaw<{ scope: string; resolver_id: string; local_date: string; estimate_usd: string | number; status: string }>(
`SELECT scope, resolver_id, local_date, estimate_usd, status
FROM budget_reservations
@@ -215,16 +218,14 @@ export class BudgetLedger {
const committedSoFar = ledger ? toNum(ledger.committed_usd) : 0;
const reservedSoFar = ledger ? toNum(ledger.reserved_usd) : 0;
let chargedAmount = actualUsd;
let overage: number | null = null;
let capExceededBy: number | null = null;
if (cap != null) {
// Available headroom = cap - already-committed (exclude this reservation
// from reserved pool since we're about to finalize it).
// Project against committed spend plus every OTHER held reservation.
// This reservation leaves the held pool as part of this transaction.
const otherReserved = Math.max(0, reservedSoFar - estimate);
const available = Math.max(0, cap - committedSoFar - otherReserved);
if (actualUsd > available + 1e-9) {
chargedAmount = Math.max(0, available);
overage = actualUsd - chargedAmount;
const projected = committedSoFar + otherReserved + actualUsd;
if (projected > cap + 1e-9) {
capExceededBy = projected - cap;
}
}
@@ -239,17 +240,21 @@ export class BudgetLedger {
committed_usd = committed_usd + $2,
updated_at = now()
WHERE scope = $3 AND resolver_id = $4 AND local_date = $5`,
[estimate, chargedAmount, r.scope, r.resolver_id, r.local_date],
[estimate, actualUsd, r.scope, r.resolver_id, r.local_date],
);
if (overage !== null && overage > 0) {
throw new BudgetError(
'invalid_input',
`commit: actualUsd ${actualUsd.toFixed(4)} exceeds cap. Charged ${chargedAmount.toFixed(4)}, overage ${overage.toFixed(4)} was NOT recorded. Cap enforcement prevented double-charge but the API call already happened.`,
if (capExceededBy !== null && capExceededBy > 0) {
return new BudgetError(
'cap_exceeded',
`commit: actualUsd ${actualUsd.toFixed(4)} exceeded the available cap by ${capExceededBy.toFixed(4)}. ` +
'The provider call already happened, so actual spend was recorded and future reservations remain blocked.',
reservationId,
);
}
return null;
});
if (capError) throw capError;
}
/** Cancel a held reservation; reserved_usd drops back. Idempotent-ish. */
+1 -1
View File
@@ -348,7 +348,7 @@ export async function judgeContradiction(input: JudgeInput): Promise<JudgeOutput
const result = await callFn({
model: input.model,
messages: [{ role: 'user', content: prompt }],
maxTokens: 200,
maxTokens: 1024,
abortSignal: input.abortSignal,
});
if (isRefusalResponse(result)) {
+5
View File
@@ -157,6 +157,11 @@ function buildReceiptFrontmatter(input: ExtractReceiptInput): Record<string, unk
const fm: Record<string, unknown> = {
type: 'extract_receipt',
dream_generated: true,
// #1978: receipts record an operation, not a source document — the
// run_id/round fields ARE the provenance. Explicit exemption keeps the
// doctor raw_provenance check quiet.
raw_trace_exempt: true,
raw_trace_exempt_reason: 'operation receipt; provenance is run_id + round',
kind: input.kind,
source_id: input.source_id,
run_id: input.run_id,

Some files were not shown because too many files have changed in this diff Show More