Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 35ec84c1ae fix(ci): delta-assert reporter leak test + raise shard timeout to 22min
The signal-handler test asserted an absolute liveReporters===0 on a
module-global set, so any other test file in the shard holding a live
reporter flaked it — it red-flagged ~12 unrelated PR runs and one master
push in two days, purely as a function of shard composition. The delta
form pins the same claim (50 reporter lifecycles leak nothing).

The 15-minute shard timeout cancelled 13 fully-passing runs under
parallel PR load (PGLite WASM cold-starts stretch shards); the
test-status gate then reported the cancellations as failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 13:26:25 -07:00
02ba4b4fc2 dims: thread Matryoshka dimensions for Qwen3-Embedding on Ollama (#1072)
Qwen3-Embedding family on Ollama supports Matryoshka truncation via the
'dimensions' field on /v1/embeddings. Without this passthrough, gbrain
ignores user-selected reduced dims and the provider returns its native
size, causing dim-mismatch errors against brains configured for narrower
widths (e.g. existing 1536-dim brains).

Matches by bare name 'qwen3-embedding' or any tag variant
'qwen3-embedding:0.6b' / ':4b' / ':8b'.

Native dims: 0.6B=1024, 4B=2560, 8B=4096. All MRL-truncatable.

5 new tests; full AI suite 137/137 green.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-22 12:38:30 -07:00
d9eb027bdd fix(openclaw): declare gbrain plugin manifest entry (takeover of #2551) (#3185)
Add the OpenClaw-required top-level id to openclaw.plugin.json, export a
direct register(api) entrypoint from src/openclaw-context-engine.ts, add a
manifest regression test, and document that skillpack harvest must preserve
OpenClaw-native manifest fields (id, configSchema, contracts).

llms bundles regenerated (bun run build:llms) — no content drift.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Filip <FilipHarald@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 01:32:11 -07:00
62e009d192 fix(skillopt): emit proposed.md in no-mutate mode (#2635) (#3182)
Takeover of #2719 (fork head; rebased onto origin/master).

- writeProposed now writes both best.md (current-best pointer) and
  proposed.md (stable human-review artifact); returns the proposal path.
- Orchestrator reports the real proposed.md path for accepted --no-mutate runs.
- Tutorial updated; llms bundles regenerated (no content drift — tutorial
  is not inlined in the bundle).

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Ziyang Guo <121015044+RerankerGuo@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 00:23:10 -07:00
314fefa560 fix(readme): correct broken OpenClaw and Hermes project links (#1961) (#3179)
Point the OpenClaw and Hermes anchors in the "Have your agent install
it" section at their real upstream repos; the previous openclawagents
org URLs 404. Regenerated llms-full.txt to match.

Takeover of #1961 (fork branch) rebased onto current master.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jessems <jessems@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 20:22:27 -07:00
7f841fae7f feat(maintain): safe maintenance automation + shared orphan-exclusion policy (#3015) (#3023)
Ports #3015 by @jdewoski-cmd onto current master:

- src/core/orphan-policy.ts centralizes the orphan-reporting exclusion
  convention so `gbrain orphans`, doctor's orphan_ratio, and both engines'
  getHealth orphan_pages can no longer drift.
- getHealth stale_pages now uses the link-extractor stale watermark
  (countStalePagesForExtraction) so health agrees with what `gbrain extract
  --stale` will actually process.
- New `gbrain maintain` command: dry-run by default, `--safe` applies only
  the conservative runbook actions (DB-backed stale extraction + source-scoped
  dream cycles for doctor cycle_freshness findings), `--json` for structured
  before/action/after reports. Frontmatter mutations, schema-pack upgrades,
  and semantic hub links stay review-only by design.

Changed from the original PR: the shared defaults carried slugs specific to
the contributor's own brain ('josa-secrets/', '*-ga4-property-id.md',
'*-josa-test', literal 'welcome'/'untitled' fixtures). Global defaults now
carry only GBrain-wide conventions; brain-specific exclusions move to a new
per-brain config plane the policy reads through loadOrphanPolicyOverrides:

    gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
    gbrain config set orphans.exclude_slugs "some-one-off-page"

Both engines' getHealth and the orphans command thread the overrides;
tests cover the neutral defaults, the override plane, and health parity.
Also registered `maintain` in CLI_ONLY_SELF_HELP so `gbrain maintain --help`
reaches the command's own usage block.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jdewoski-cmd <jdewoski@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 19:52:18 -07:00
1fabbb9849 fix(links): resolve path-qualified wikilinks outside DIR_PATTERN in the DB/put_page path (#2866)
The generic wikilink pass (issue #972) forwarded the raw literal to
resolveBasenameMatches, whose index is keyed by final path segments —
so [[notes/struktura]] (any dir outside DIR_PATTERN) silently produced
zero edges from `extract links --source db` and put_page auto-link,
while the FS extractor resolves the identical content (resolveSlugAll
strips the dirname before its basename lookup).

Query by the literal's final segment, then keep only matches whose slug
ends with the written path — [[notes/struktura]] can resolve to
vault/notes/struktura but never attach to wiki/struktura. Bare literals
are untouched. Flag-gated by link_resolution.global_basename as before.

Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 18:45:23 -07:00
64920f83c9 fix(embed): preserve code-chunk metadata across re-embed (#769) (#1232)
Closes #769. Every re-embed pass clobbered code-chunk metadata
(language, symbol_name, symbol_type, start_line, end_line,
parent_symbol_path, doc_comment, symbol_name_qualified) to NULL,
disabling code-def queries across thousands of indexed chunks.

Two complementary fixes:

embed.ts — three re-upsert call sites (embedPage, embedAll
non-stale, embedAllStale autopilot path) build ChunkInputs from
loaded chunks; they were stripping the 8 metadata fields. New
preserveCodeMetadata helper threads those fields through
consistently. Integrated cleanly with v0.34.4.0's cursor-paginated
--stale hardening — the wrap sits inside the worker function
between embedBatchWithBackoff and engine.upsertChunks.

postgres-engine.ts + pglite-engine.ts — upsertChunks ON CONFLICT
clause OVERWROTE metadata columns from EXCLUDED. Asymmetric vs the
embedding/embedded_at columns which already used a chunk_text-gated
CASE pattern (re-chunk → trust EXCLUDED, re-embed → COALESCE
preserve). Applied the same pattern to all 8 metadata columns.

Three regression tests in test/embed.serial.test.ts cover --stale
(autopilot), --all, and --slugs paths. Each loads a chunk with
full metadata, runs runEmbed, and asserts engine.upsertChunks
receives the metadata round-tripped. Coexists with master's D5
embedBatchWithBackoff test block.

Backfill required after deploy: \`gbrain sync --strategy code
--force --source <id>\` per code source to re-populate metadata via
the chunker. Without backfill, existing NULL columns stay NULL —
re-embed alone never produces metadata, only the chunker does.

Originally landed as part of PR #768 (the wave that bundled #767 +
fix; this PR carries the #769 fix alone with no scope overlap.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 18:13:45 -07:00
Benjamin D. SmithandTime Attakc e861b92da7 feat(synopsis): tail-truncate documentText for small-model chat handlers (#1427)
* feat(synopsis): tail-truncate documentText for small-model chat handlers

Small local chat models (Gemma 4 E2B, Qwen3 4B) get dramatically
slower on long contexts even at 131K declared windows. A 73K-char
page synopsis on Gemma 4 E2B takes 60-120s, exceeding the worker's
default 30s `lockDuration` and tripping `lock-lost` errors.

Add `SYNOPSIS_DOC_MAX_CHARS` env-overridable cap (default 32768
chars, ~8K tokens) applied in `buildUserPrompt`. Truncate the TAIL
so the head (title, frontmatter, intro) preserves the document-level
anchor the synopsis needs.

Anthropic Haiku is unaffected at this cap; bump via
`GBRAIN_SYNOPSIS_DOC_MAX_CHARS` for frontier models that want
richer document anchoring.

Belt-and-suspenders companion to commit 0aaff691 (--lock-duration
flag on the worker). Combined: bumping lock TTL gives the handler
more time, AND truncating doc cap makes the handler complete faster.
Either alone helps; both together get the synopsis backfill running
reliably on small local LLMs.

Verified: real 1383-chunk personal brain backfill at
GBRAIN_SYNOPSIS_MODEL=lmstudio:google/gemma-4-e2b +
GBRAIN_SYNOPSIS_DOC_MAX_CHARS=16384 +
`gbrain jobs work --concurrency 4 --lock-duration 300000`
transitions from "lock-lost on every transcript page" to "no
deaths, no stalls, steady throughput."

RECOVERY REBUILD 2026-05-26 of original ac213aa6.

* fix: fold SYNOPSIS_DOC_MAX_CHARS into corpus_generation hash

Codex review of #1427 flagged that changing GBRAIN_SYNOPSIS_DOC_MAX_CHARS
shifts the synopsis prompt + downstream embeddings for long documents
but was NOT folded into the computeCorpusGeneration hash. Pages
re-embedded with a different cap would retain the same
corpus_generation, defeating the v0.40.3.0 D27 P1-5 cache invalidation
contract.

Three changes:

1. Export SYNOPSIS_DOC_MAX_CHARS from src/core/page-summary.ts
2. computeCorpusGeneration accepts optional synopsisDocMaxChars param.
   When set, folded into hash via '|doc_cap=<N>'. Omitted for
   non-synopsis modes (title / none don't consult the cap) so existing
   pre-PR caches stay valid for those.
3. Service-layer call sites (2 in contextual-retrieval-service.ts)
   pass SYNOPSIS_DOC_MAX_CHARS when attemptMode/resolution.mode is
   per_chunk_synopsis, undefined otherwise.
4. import-file.ts inline path passes undefined (per_chunk_synopsis
   refused upstream there).

One-time effect: per_chunk_synopsis pages re-embedded post-PR get a
NEW corpus_generation including the cap. v0.40.3.0 query_cache.page_generations
contract auto-invalidates cached query results on first re-embed.
Future cap changes track correctly.

Addresses codex review P2 on PR #1427.

---------

Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-21 16:50:43 -07:00
Masa 0612b0daa8 fix(dream): require self-contained opening summary in synthesized pages (#2770)
Synth pages written by dream synthesize currently open straight into
detail (quotes, cross-references) with no framing, so a reader who
lands on the page later — without the source transcript in front of
them — has no way to tell what it's about without reading the whole
thing.

Add OUTPUT POLICY item 5: every new page's body must open with a 2-3
sentence self-contained summary a reader unfamiliar with the source
conversation could understand on its own, before any quotes or detail.
2026-07-21 13:40:41 -07:00
maxpetrusenkoagentandmaxpetrusenkoagent <[REDACTED EMAIL]> f529eaa231 fix(jobs): refresh gateway config for queued AI work (#2125)
Long-lived minion workers can outlive DB-backed model config changes. Refresh the AI gateway before gateway-backed handlers run so queued cycle/propose_takes work does not fall back to a stale Anthropic default when the operator configured another provider.

Also record the active gateway chat model in propose_takes budget/proposal metadata instead of hardcoding claude-sonnet-4-6, and keep provider:model IDs intact for budget pricing.

Regression coverage verifies queued worker refresh, propose_takes model metadata, nested provider IDs, skipFence threading, and the updated autopilot signal source guard.

Co-authored-by: maxpetrusenkoagent <[REDACTED EMAIL]>
2026-07-21 13:21:51 -07:00
447e57ec41 fix(ai): tier-configured models reach the recipe allowlist — refresh Anthropic models, register tier resolutions, honest probe labels (#2800)
Setting `models.tier.deep anthropic:claude-opus-4-8` silently disabled
think and auto_think: the Anthropic recipe's chat allowlist stopped at
Opus 4.7, the tier-resolved model never joined the extended set that
assertTouchpoint's contract promises for config-chosen models, and the
resulting probe failure was stamped NO_ANTHROPIC_API_KEY — sending the
operator to debug env/keychain when the fix was the model id. Three
fixes, one per layer:

- recipes/anthropic.ts: add claude-fable-5, claude-opus-4-8, and
  claude-sonnet-5 to chat models; claude-sonnet-5 to expansion models.
- gateway.ts reconfigureGatewayWithEngine: resolve all four tiers and
  register the results as extended models, honoring the documented
  contract for models.default / models.tier.* (model-resolver.ts
  docstring). A tier-only model now validates like a chat/expansion one.
- think/index.ts: when the gateway client can't be built, re-probe and
  label honestly — MODEL_NOT_USABLE:<reason> for unknown_model /
  unknown_provider, NO_ANTHROPIC_API_KEY only for the actual missing-key
  case; the stub answer carries the probe detail and fix hint.

Tests: recipe-list presence pins; a new gateway-tier-extended-models
suite proving a fictional tier model validates post-reconfigure (and an
unconfigured one still doesn't); think-pipeline coverage for the honest
label (unknown_model beats missing-key even keyless); the existing
non-explicit bogus-provider test updated from the old catch-all label to
the honest one (no-throw contract unchanged).

Verified live: a brain with tier.deep=claude-opus-4-8 had think degrade
to gather-only with the misleading key warning; with this change the
probe passes and synthesis runs.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 13:06:32 -07:00
d61808d806 v0.42.64.0 fix: harden confidential OAuth token revocation (takeover of #3017) (#3032)
* fix(oauth): validate confidential revoke secrets

* fix(oauth): harden confidential token revocation

* chore: bump version and changelog (v0.42.64.0)

Co-Authored-By: OpenAI Codex <noreply@openai.com>

---------

Co-authored-by: Robin <rayme@boltdsolutions.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:44:46 -07:00
60125ee626 feat(dream): --once for one-shot phase runs without toggling config gates (takeover of #2983) (#3031)
* feat(dream): --once for one-shot phase runs without toggling config gates

Fixes the "toggle enabled true, run, toggle back to false" workaround
that gbrain doctor's extract_atoms_backlog message implicitly
recommends and that #2860's reporter had to script around: with an
external orchestrator running `gbrain dream --phase patterns` on a
cadence outside the autopilot, the only way to run patterns once was
`config set dream.patterns.enabled true` -> run -> `config set ...
false`. A crash between steps left the flag stuck true, and the
autopilot (which polls the same flag) re-enqueued patterns every
cycle -- 119 LLM jobs / ~$400 over 24h before it was caught.

Root cause: `--phase X` only controls which phase FUNCTION cycle.ts
calls; it does not bypass that phase's own `dream.<phase>.enabled` /
`cycle.<phase>.enabled` config read. Each gated phase (patterns,
synthesize, conversation_facts_backfill, enrich_thin, skillopt) reads
its enabled flag internally and skips regardless of how the phase was
selected -- confirmed by reading each phase module, not assumed.
extract_atoms/synthesize_concepts are a DIFFERENT mechanism entirely
(pack-declaration via packDeclaresPhase, not a config .enabled read)
and already have a working one-shot escape hatch: `--drain`. The
existing doctor message for extract_atoms already says `--phase
extract_atoms --drain --window 120`, so no doctor text needed
updating there -- verified by reading src/commands/doctor.ts directly
rather than assuming the paraphrase in the issue was literal.

Design: `gbrain dream --phase <name> --once`. Requires an explicit
--phase (bare --once is a usage error, exit 2) so it can never
force-enable every disabled phase at once in a full/default cycle --
that would recreate the same unbounded-spend risk the flag exists to
prevent. Threaded through CycleOpts as `onceForPhase?: CyclePhase`
(the literal phase name, not a boolean) so the bypass can never leak
to a phase other than the one named, even if a future programmatic
caller passes a wider `phases` array than the CLI does. Never reads
or writes config -- the phase still evaluates its .enabled gate every
call; --once only overrides the boolean OUTCOME for that one
invocation, mirroring the existing --unsafe-bypass-dream-guard /
--input precedents (stderr warning at the bypass point, no new
config-touching code path).

Rejected alternatives (documented per task instructions):
- Making explicit --phase X always bypass .enabled: breaking change
  for existing crons that rely on the disabled flag as a cheap no-op;
  an upgrade would silently start running LLM/write phases.
- A new subcommand: adds a whole dispatch/help/arg surface that
  internally routes through the same override anyway.
- Extending --once to also bypass packDeclaresPhase for
  extract_atoms/synthesize_concepts: conflates two different gating
  mechanisms (config toggle vs. pack membership) under one flag;
  extract_atoms already has --drain, which is purpose-built for its
  batched/windowed execution model.

Design was cross-validated by an independent second-model review
(external design consultation) before implementation; its
recommendation to also update the extract_atoms doctor message to
`--once` was NOT adopted because that phase has no .enabled gate to
bypass -- doing so would be a documented no-op, contradicted by
reading src/commands/doctor.ts:3264 directly.

Tests: 9 new (structural CLI-flag wiring in dream-cli-flags.test.ts;
a real PGLite E2E test in dream-patterns-pglite.test.ts proving the
bypass fires AND that dream.patterns.enabled is never written; 4
runCycle-level tests in cycle.serial.test.ts proving onceForPhase
does not leak across phases). Verified 6 of 9 fail against the
pre-fix source (via git stash of source-only changes) to confirm
they're meaningful regressions, not tautologies.

Closes #2860

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dream): --help short-circuits before --once usage validation

Codex review finding (P2): `gbrain dream --help --once` (no --phase)
called process.exit(2) from the new --once usage-error check inside
parseArgs before runDream's documented IRON RULE ("--help
short-circuits BEFORE any engine-bearing work") ever got a chance to
run -- parseArgs computes ALL its validations unconditionally before
runDream checks opts.help. Repo precedent for this ordering already
exists as a pinned regression test (test/dream.test.ts's "--help
--source whatever prints help and exits 0").

Fix: compute wantsHelp once in parseArgs and exempt the --once
validation when it's set, mirroring that precedent. Added the same
class of pinned tests here: bare `--once` still exits 2 with the
usage hint, `--help --once` prints help and exits 0, and a real
--phase patterns --once run against a PGLite engine proves the
bypass actually fires (falls through to insufficient_evidence
instead of disabled) without writing dream.patterns.enabled. Also
fixed the structural test in dream-cli-flags.test.ts that asserted
the exact pre-fix guard-condition source text.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dream): --once must require an EXPLICIT --phase, not a derived one

Codex review finding (P3): the --once validation checked the derived
`phase` value, but `phase` gets defaulted implicitly by --input
(implies --phase synthesize) and --drain (implies --phase
extract_atoms) BEFORE that check ran. So `gbrain dream --input <f>
--once` and `gbrain dream --drain --once` both slipped past the
"explicit --phase required" contract silently -- and --once became a
true no-op in both cases: --drain returns from runDream before
onceForPhase is ever read (the drain path doesn't call runCycle at
all), and --input already bypasses the synthesize enabled-gate on its
own via the existing opts.inputFile check, so onceForPhase would
never even be consulted.

Fix: capture `phaseWasExplicit = phaseIdx !== -1` at the very top of
parseArgs, before the --input/--drain defaulting blocks run, and
validate --once against that instead of the derived `phase`. Updated
the usage-error message and --help text to say "an explicit --phase"
so a user hitting this understands why `--input ... --once` doesn't
count.

Tests: 2 new pins in test/dream.test.ts exercising runDream directly
(--input <file> --once exits 2; --drain --once exits 2), plus a
structural test in dream-cli-flags.test.ts pinning that
phaseWasExplicit is captured before both implicit-defaulting blocks.
Updated the two existing structural/behavioral tests whose literal
guard-condition / error-message assertions changed shape.

Verified: dream-cli-flags.test.ts 27/27, dream.test.ts 31/31,
typecheck clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:29:35 -07:00
e320ad71b3 fix(providers): reuse buildGatewayConfig for --model test override (takeover of #2980) (#3029)
* fix(providers): reuse buildGatewayConfig for --model test override (#2863)

`gbrain providers test --model <id>` overrode the gateway with only
embedding_model/chat_model + env, dropping config.provider_base_urls
entirely. A brain configured with a custom endpoint (e.g. a China-region
DashScope base URL) would pass the bare `providers test` (which goes
through configureFromEnv() and does forward base_urls) but fail the
`--model`-scoped probe with a misleading "Incorrect API key" error, even
though the key was valid for the configured endpoint — the probe silently
fell back to the recipe's hardcoded default endpoint instead.

Root cause: two independent, drifted resolvers. The production path
(src/cli.ts#connectEngine, src/core/init-embed-check.ts) builds its
AIGatewayConfig via buildGatewayConfig(), which folds provider_base_urls,
env-sourced local-server base URLs, provider_chat_options, and file-plane
API keys. The --model override branch in runTest() hand-rolled a second,
narrower config object that only carried the overridden model + raw env.

Fix: lift `cfg` out of the existing try/catch (it was already loaded there
for the isolation-warning message) and spread `buildGatewayConfig(cfg)`
into both configureGateway() calls before overriding embedding_model/
chat_model. The isolated --model probe now resolves its endpoint exactly
the way the brain's real import/query path would; only the requested
model is overridden, so the probe still targets exactly the model the
user asked for. Falls back to bare env when no brain is configured yet
(cfg is null), matching prior first-time-install behavior.

Confirmed chat_fallback_chain (also threaded through by buildGatewayConfig)
has no runtime retry effect — it's only consumed to pre-register extended
model ids — so spreading the full production config does not mask an
isolated model's own failures behind a silent fallback.

Other diagnostic surfaces (providers list/env/explain) were checked and
are unaffected: `runProviders()` already calls configureFromEnv() (which
forwards base_urls correctly) before dispatch, and none of them accept
--model, so they never hit the broken override branch.

Adds test/providers-test-model-base-url.test.ts: drives runProviders('test',
...) end-to-end against a mocked fetch + temp GBRAIN_HOME/config.json with
provider_base_urls set for the dashscope recipe (the exact recipe named in
the bug report), asserting the outbound request hits the configured base
URL rather than the recipe default. Verified red on pre-fix code via
git stash, green after.

Closes #2863

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix: drop duplicate buildGatewayConfig import after master merge

Master's f3e78fd2 added the same import the PR carried; the textual
merge was clean but the result failed typecheck (TS2300 duplicate
identifier).

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:17:18 -07:00
b6dd3e1121 fix(test): reset AI gateway after adaptive-embed-batch suite — cross-file config leak (#3065)
The file's final test configures the gateway with a remote provider and a
fake key, and its afterEach only clears the mock transport. With no
afterAll, the poisoned global config survives the file boundary; the next
test file in the shard that triggers an embed makes a real HTTP call and
fails. Surfaced on master when #3022's new test file reshuffled shard
composition (shard 6: synthesize-concepts-progress failed twice with a
live Google embed rejection). The legacy-embedding preload can't catch
this: it only re-applies defaults when the gateway slot is empty.

One-line root-cause fix at the leaker. A repo-wide guard for the class
(~70 files call configureGateway without a final reset) is filed as a
follow-up.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 12:05:31 -07:00
3cc34c92ee feat(ai): add NVIDIA NIM provider recipe (#2965) (#3022)
Adds NVIDIA NIM / API Catalog as a first-class OpenAI-compatible AI recipe:

- chat via nvidia/nemotron-3-super-120b-a12b (conservative capability
  claims: no tools, no subagent loop until proven)
- hosted embedding models incl. nvidia/llama-nemotron-embed-1b-v2 with
  Matryoshka-style dimension overrides (1024/1280/1536/2048) and fixed
  natural dims for the other catalog models
- asymmetric input_type mapping (document -> passage, query -> query) via a
  gateway compat fetch shim, since the generic openai-compatible recipe
  cannot infer that provider-specific requirement
- base URL https://integrate.api.nvidia.com/v1 verified live (OpenAI-shaped
  /v1/models, all five recipe model ids present in the catalog)

Changed from the original PR: dropped the recipe's custom resolveAuth — it
duplicated defaultResolveAuth's Authorization-Bearer behavior exactly and
violated the IRON RULE that only Azure overrides resolveAuth
(test/ai/recipes-existing-regression.test.ts). NVIDIA_API_KEY now flows
through defaultResolveAuth via auth_env.required, and the recipe test pins
resolveAuth === undefined + the default Bearer resolution + the missing-key
AIConfigError. Also scrubbed a private downstream-agent name from ported
comments per the repo privacy rule.

Takeover of #2965 by @ravehorn.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: SAGE Codex <codex@sage.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 00:17:26 -07:00
Anton Senkovskiy a93fcf504f fix(chronicle): bound last-seen to <= asof/today so future events do not read as "seen today" (#2993)
getLastSeen had no upper date bound, so a future-dated chronicle event (a
scheduled calendar-event, a planned milestone) became the entity's "last
seen" date; finalizeLastSeen's Math.max(0, ...) then clamped the negative
delta, reporting days_ago: 0 -- the entity reads as seen-today. Recording
future events is intended (eligibility ELIGIBLE_TYPES includes calendar-event);
the reader just needs to stop counting them as "seen".

Bound the query to te.date <= COALESCE(asof, current_date) in both engines,
mirroring getOnThisDay's existing te.date < target bound. asof now reaches
the WHERE clause (previously it only reached finalizeLastSeen), so as-of
time-travel is honored for the date filter too.

Regression test added: an entity with past events plus a future event -> last
seen returns the most recent PAST event, not the future one; and as-of after
the future date lets it through. Fails before, passes after.
2026-07-21 00:04:41 -07:00
Hanchen Qiu 84fad4738d fix(pages): default chunker_version to MARKDOWN_CHUNKER_VERSION on INSERT (#2807) (#2988)
putPage's INSERT used COALESCE(<chunkerVersion>, 1), so callers that don't
supply chunker_version (no MCP/subagent caller does — it's internal metadata)
landed new pages at version 1. Dream subagents write through putPage directly,
so their pages got v1 and doctor's contextual_retrieval_coverage check flagged
them as "older chunker_version" forever, even though they were chunked and
embedded with the current chunker.

Default the INSERT to MARKDOWN_CHUNKER_VERSION on both engines. The ON CONFLICT
UPDATE still COALESCE-preserves an explicitly supplied version. Add an
engine-level regression test.
2026-07-20 23:47:09 -07:00
Hanchen Qiu f815246eef fix(cli): register reconcile-links in CLI_ONLY so dispatch reaches its handler (#2900) (#2987)
reconcile-links is advertised in `gbrain --help` and implemented with a
`case 'reconcile-links'` block in handleCliOnly, but it was missing from the
CLI_ONLY Set. Dispatch only enters handleCliOnly when the command is in
CLI_ONLY, so every invocation fell through to the shared-operations lookup and
hit the generic "Unknown command" branch — leaving the documented doc↔impl
edge-rebuild tool silently unreachable via the CLI.

Add 'reconcile-links' to CLI_ONLY, plus a reachability regression test
mirroring the #2035 (`calibration`) guard.
2026-07-20 23:23:03 -07:00
Vishnu JandClaude Fable 5 42c4ea929f fix(test): isolate audit writes to a per-run scratch dir in the shared bootstrap (#2823) (#2966)
The content-sanity gate's audit logger (logContentSanityAssessment)
defaults, via audit-writer.ts::resolveAuditDir(), to writing
~/.gbrain/audit/content-sanity-YYYY-Www.jsonl on disk. A GBRAIN_AUDIT_DIR
env override exists, but nothing in the shared test bootstrap ever set
it, so any test that exercised an audit-emitting code path without
wrapping the call in its own withEnv() fell through to the operator's
real audit trail. test/import-file.test.ts's oversize-boundary fixture
('borderline-slug', content just under MAX_FILE_SIZE but over
DEFAULT_BYTES_BLOCK) fired a real soft_block event into the developer's
live ~/.gbrain/audit on every run — which doctor's
content_sanity_audit_recent check then reported as production signal.

Fix: add a bootstrap preload (test/helpers/audit-dir-preload.ts, wired
via bunfig.toml) that points GBRAIN_AUDIT_DIR at a fresh per-process
mkdtemp dir before any test file loads. Each run-unit-shard.sh shard is
its own bun process, so each shard gets its own scratch dir with no
cross-shard collision. This closes the leak for every audit-emitting
test, not just this fixture. It respects a developer-exported override
(only sets the var when unset), and files that manage their own
per-test GBRAIN_AUDIT_DIR via withEnv() are unaffected.

Also fix a latent isolation bug this surfaced: gbrain-home-isolation.test.ts
unconditionally deleted GBRAIN_AUDIT_DIR in a finally block instead of
restoring the prior value, which clobbered the bootstrap's scratch dir
for every test file that ran after it in the same shard process.

Adds test/audit/audit-dir-preload.test.ts to pin the behavior: it
reproduces the exact soft_block event shape and asserts it lands in the
scratch dir, never in ~/.gbrain/audit.

Reported by @paul-0320.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:07:58 -07:00
Benjamin D. Smith 6370ce3d7e fix(infrastructure): chmod 644 autopilot supervisor files on install (#2963)
📝 Summary:
• launchd rejects group/world-writable agent plists — when the installer
  runs under a umask-0 parent shell, `writeFileSync(plistPath(), plist)`
  produces a 0666 plist that makes `launchctl load`/`bootstrap` fail with
  the opaque `Bootstrap failed: 5: Input/output error` and the login-time
  LaunchAgents scan skip the file silently
• on an affected machine the daemon never registers while everything
  looks installed — the plist exists, launchd's disabled-table says
  enabled, and no log file is ever created

🔧 Technical Improvements:
• `installLaunchd`: write plist with `{ mode: 0o644 }` AND
  `chmodSync(0o644)` — writeFileSync mode applies only on create, so a
  reinstall over an existing 0666 plist must normalize explicitly
• `installSystemd`: same hardening on the unit file (systemd warns on
  world-writable units); symmetric with the launchd path
• Restart-policy rewrite path (`generateSystemdUnit` rewrite of an
  existing unit): chmod is load-bearing here — the file always exists,
  so the write mode never applies
• `chmodSync` added to the fs import

📊 Code Changes: 18 insertions, 4 deletions (net +14)

📦 Files Modified:
• src/commands/autopilot.ts (minor updates) — mode + chmod on the three
  supervisor-file writers; comments carry the launchd failure signature
  so the next EIO hunt greps straight to it
2026-07-20 22:48:16 -07:00
levineam c21d7b253a fix(doctor): derive host skill manifests (SUP-3488) (#2961) 2026-07-20 22:38:25 -07:00
zsimovanforgeopsandForge 4df7796061 Preserve modality and symbol metadata in embed-stale merge (#2969)
embedStaleForSource rebuilds each page's chunks as a merged ChunkInput[]
carrying only five fields (chunk_index, chunk_text, chunk_source,
embedding, token_count), while upsertChunks writes the metadata columns
as EXCLUDED.<col>. Any page containing at least one stale chunk
therefore has ALL its chunks' metadata reset on the next embed-stale
pass:

- image chunks flip modality 'image' -> 'text' and disappear from the
  cross-modal image search arm permanently (it filters
  modality='image'), while keeping their embedding_image vector — the
  data looks intact but is unreachable;
- code chunks lose language, symbol_name, symbol_type,
  symbol_name_qualified.

The read side compounds this: rowToChunk never returned modality, so a
correct merge was impossible without also extending the Chunk shape.

Fix: expose modality on Chunk/rowToChunk and carry modality, language,
and the symbol fields through the merge. embedding_image is deliberately
not carried — upsertChunks already COALESCEs it server-side.

Repair for affected brains: UPDATE content_chunks SET modality='image'
WHERE chunk_source='image_asset' AND embedding_image IS NOT NULL.

The new test seeds a mixed page (settled image chunk + stale text chunk
with symbol metadata) and asserts both survive an embedStaleForSource
pass; it fails on master.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 22:14:46 -07:00
zsimovanforgeopsandForge 6db4cea2e4 Resolve relative storage paths in doctor image_assets check (#2971)
The image_assets check statSyncs files.storage_path directly, but
sync-ingested assets store repo-relative paths. Run doctor from any
directory other than the brain repo and every image is reported
'missing from disk' — a persistent false WARN with a suggested fix
(gbrain sync --skip-failed) that does nothing.

Resolve relative paths against sync.repo_path before statting; absolute
paths are untouched. Falls back to cwd when the config key is unset,
preserving the old behavior for brains without a configured repo.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 21:59:24 -07:00
sameerbopardikarandSameer Bopardikar 11eebc3605 fix(conversation): extract iMessage facts with real timestamps (#2756) (#2958)
Co-authored-by: Sameer Bopardikar <203024074+sameerbopardikar@users.noreply.github.com>
2026-07-20 16:57:53 -07:00
Eddie AshandEddie Ash ee45653a02 fix: initialize foreground chat gateway (#2590) (#3003)
Co-authored-by: Eddie Ash <119880+cazador481@users.noreply.github.com>
2026-07-20 16:47:12 -07:00
zsimovanforgeopsandForge 706d3cea3d Scope maxWaiting backpressure by data.sourceId (#2970)
The submission-time backpressure cap counts all waiting (name, queue)
rows regardless of which source a job targets. On a multi-source brain
this makes per-source submissions with maxWaiting: 1 mutually exclusive:
while one source's sync sits waiting, every other source's freshness
sync coalesces into that row and never runs. The dispatch log shows the
starved source 'dispatched' each interval (queue.add returns the other
source's waiting row), so the starvation is invisible unless you notice
sources.last_sync_at falling behind — we found a secondary source 29
hours stale on a 5-minute freshness interval.

Fix: when the submitted data carries a string sourceId, key the advisory
lock, the waiting count, and the coalesce target on it. Submissions
without sourceId keep the existing single-scope behavior, so
single-source brains and non-sync jobs are unchanged.

The new test asserts same-source submissions still coalesce while a
different source gets its own row and its own cap; it fails on master.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 16:38:03 -07:00
MasaandClaude Sonnet 5 2934c53c1d fix(sources): validate --path is a git repo with committed content at registration (#2707) (#2975)
* fix(sources): validate --path is a git repo at registration time (#2707)

`sources add --path <dir>` accepted any existing non-git directory with
zero validation, deferring the failure to the first `gbrain sync` ("Not
inside a git repository: ..."). By the time that surfaces, the source
has been silently stale for however long nobody read the sync logs.

Add a registration-time check (git-remote.ts:isInsideGitRepo, mirroring
sync.ts's discoverGitRoot walk-up so subdir-of-git-repo sources still
pass) that rejects an existing-but-non-git --path directory with an
actionable error pointing at `git init && git add -A && git commit`.
Non-existent paths are unaffected (out of scope — different, pre-existing
failure mode) and `--force` opts out for callers who want to register
before git-init exists.

This is registration-time validation ONLY — it never auto-`git init`s
the directory, preserving the consent boundary #2967 established for
sync-time self-heal (a --path source is the user's own external
directory; gbrain must not mutate it without explicit ask).

Also documents the git requirement (docs/guides/multi-source-brains.md),
including the "files must be committed, not just present" gotcha and
that a stale/unreachable sync anchor already self-heals on plain
`gbrain sync` (verified manually against HEAD — no reset-anchor command
needed).

* fix(sources): require a committed HEAD + shell-quote remediation cmd (codex round 1)

Codex review round 1 on #2707 found two real gaps:

1. isInsideGitRepo alone accepts a `git init`ed-but-never-committed
   directory (rev-parse --show-toplevel succeeds with no HEAD), so
   registration would still pass a source that fails sync's own
   "No commits in repo ... Make at least one commit before syncing."
   Add hasGitCommits (git rev-parse HEAD) as a second required check.

2. The remediation command in the error message interpolated the raw
   path unquoted — spaces, $(), backticks, etc. would break or, worse,
   execute unintended shell syntax if pasted. POSIX single-quote it
   (mirrors src/commands/connect.ts:shellQuote; duplicated locally
   rather than imported, since commands/ depends on core/ not the
   reverse).

* fix(sources): require tracked content in HEAD, not just a resolvable HEAD (codex round 2)

Codex review round 2 P1: hasGitCommits (rev-parse HEAD) accepted a repo
with an empty commit (git commit --allow-empty) followed by untracked
files — HEAD resolves fine (to git's well-known empty-tree object), so
registration passed, but the first sync would "succeed" importing
nothing and then silently never notice the untracked files change. The
exact same gap applied to an untracked subdirectory of an otherwise-
real git repo (monorepo case).

Replace hasGitCommits with hasTrackedContent (`git ls-tree HEAD -- .`,
non-recursive — one entry is enough, no need to walk the whole
subtree). `-C path` + pathspec `.` scopes correctly to both a repo
toplevel and a subdirectory-of-a-repo source, and an empty tree lists
zero entries where a bare `rev-parse HEAD` would still succeed. Also
subsumes the "no commits at all" case hasGitCommits covered (ls-tree
on an unborn repo fails the same way), so this is one check instead
of two.

Updated the error copy and docs/guides/multi-source-brains.md to match
what's actually verified now.

* fix(sources): O(1)-output tree-emptiness probe, avoid maxBuffer overflow (codex round 3)

Codex review round 3 found the round-2 `git ls-tree HEAD -- .` listing
buffers the whole (non-recursive) tree — a real repo with ~17-20K
directly-tracked entries exceeds execFileSync's default 1 MiB
maxBuffer, throws ENOBUFS, and the catch-all incorrectly rejects a
perfectly valid registration.

Replace the listing with `git rev-parse --verify HEAD:./` (resolves
the tree object for `path` specifically, correct for both toplevel and
subdirectory sources same as before) compared against git's canonical
empty-tree SHA-1 (4b825dc6...) — a fixed ~40-byte read regardless of
how many entries the tree has, structurally immune to this class of
bug rather than just raising the threshold. Added a 300-file
regression test locking this in.

Declined a second round-3 finding (P1: reject a tree if ANY untracked
file exists anywhere under the path, not just when the tree is
entirely empty) — untracked files never being synced is standard,
existing git-source behavior throughout this codebase (identical for
--url managed clones), not a bug specific to this validation. Enforcing
zero-untracked-files at registration would reject ordinary repos with
gitignored build output, .DS_Store, editor swapfiles, etc. Out of
scope relative to what #2707 actually asks for (a directory with real,
committed content that will sync) and how every other git source in
this system already behaves.

* fix(sources): derive empty-tree OID per repo instead of hardcoding SHA-1 (codex round 4)

Codex review round 4 P2, confirmed by directly testing against a
`git init --object-format=sha256` repo: the hardcoded SHA-1 empty-tree
constant only matches SHA-1 repositories. An empty SHA-256 repo's real
empty-tree OID is a different (64-char) hash, so the SHA-1 comparison
silently mismatched and let an empty/untracked SHA-256 source through
— exactly the case this validation exists to catch.

Replace the constant with `emptyTreeOid()`: `git hash-object -t tree
--stdin < /dev/null` computed in the target repo's own context, so it
returns the correct empty-tree OID for whichever object format that
repo actually uses, without gbrain needing to know or care which one.
Added gated regression tests (git 2.29+ / --object-format=sha256,
test.skipIf on older git) for both the empty-repo-rejected and
real-content-registers-fine cases.

Converging here (4 review rounds; this is the last outstanding
finding from round 4, and round 4 raised only this one issue).

* fix(test): --force the incidental non-git second source in #1434 routing test (#2707)

CI caught a real regression from this PR's registration-time git
validation: test/sync-sole-non-default-routing.test.ts's "2+ non-default
sources" case registers a bare mkdtempSync temp dir (no git init) as a
second source purely to have 2 sources present — the directory's content
was never meant to be exercised, only its existence as a distinct
local_path. #2707's new validation correctly rejects that dir at
registration time, since nothing else in the test suite told it
otherwise.

--force is the right fix, not adding unnecessary git-init/commit
boilerplate to secondRepo: it documents that this specific registration
intentionally doesn't care about git-validity, matching what a real
caller opting into the legacy lenient behavior would do.

Verified: the specific test (3/3 pass), plus every other test file in
the repo using `sources add --path` (sources.test.ts, sources-ops.test.ts
already covered by the PR's own commits; repos-alias.test.ts,
sync-cost-gate.serial.test.ts — 11/11 pass, no similar fixture gap).
typecheck clean, verify 31/31.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:28:53 -07:00
b60656245f fix(migrate): v124 notice to stderr — the stdout-cleanliness guard caught it on master (#3035)
Same class as #3019's v123 fix; the guard test added there flagged this
within one push. Route the notice through process.stderr.write.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 16:19:40 -07:00
MasaandClaude Sonnet 5 d698b44438 fix(search): drop compiled_truth from pages.search_vector — was overflowing tsvector on large pages (#2704) (#2977)
A single markdown page whose compiled_truth exceeds Postgres's hard
1,048,575-byte tsvector cap made update_page_search_vector() throw
"string is too long for tsvector" INSIDE the pages UPSERT transaction.
Not a per-file ledger entry — a transaction abort. The whole source's
sync checkpoint stayed pinned (Sync BLOCKED) until the oversized file was
fixed or manually skipped, even though every other file in the run
imported fine. --retry-failed re-failed the same files every run; the
3-consecutive-failure auto-skip eventually moved past them, but for a
scheduled collector that meant hours of blocked cycles per oversized
file, per source.

Root cause: pages.search_vector indexed compiled_truth — the unbounded
whole-page body — even though it's write-only dead weight for actual
search. searchKeyword() (postgres-engine.ts / pglite-engine.ts) ranks and
queries content_chunks.search_vector exclusively (Cathedral II Layer 3,
chunk-grain, already populated separately from compiled_truth via
chunking at import time, and well under the tsvector cap since
chunkText() targets embedding-sized pieces). Verified directly: no
pages.search_vector / bare search_vector read appears anywhere outside
this trigger's own definition and the reindex/backfill machinery that
maintains it.

Fix: v124 migration recreates update_page_search_vector() without
compiled_truth — title + timeline stay (both naturally small), so the
column keeps carrying some signal rather than going fully inert. Updated
in lockstep (documented contract, see reindex-search-vector.ts's own
comment): migrate.ts's new v124, reindex-search-vector.ts's
recreatePagesFn, and the fresh-install baselines in pglite-schema.ts +
src/schema.sql (regenerates schema-embedded.ts via `bun run
build:schema`). No backfill: existing rows keep whatever search_vector
they already computed until their next UPDATE — harmless, since nothing
reads this column, and the brains that actually hit this bug never
successfully wrote a value for the oversized page in the first place.

Considered (from the issue) and rejected: truncating compiled_truth to
fit under the cap. Silent, position-dependent recall loss, and the byte
cap doesn't line up cleanly with any natural character/token boundary
for UTF-8 content. content_chunks.search_vector already gives full,
untruncated chunk-grain coverage for large pages — truncating a
now-redundant whole-page vector would trade a real bug for a subtler one.

## Test plan

- New test/page-search-vector-overflow.test.ts: a >1MB page (genuinely
  diverse tokens — a repetitive lorem-ipsum-style fixture does NOT
  reproduce this bug, since to_tsvector's cap is on its DEDUPLICATED
  output size, not raw input length) now imports successfully instead of
  throwing; remains keyword-searchable via the chunk-grain path; a normal
  page's search_vector still carries title signal (not fully inert).
  Verified the test is meaningful both directions: fails with the exact
  reported error on the pre-fix trigger (git-stashed the fix, reran,
  confirmed byte-for-byte match: "string is too long for tsvector
  (2684620 bytes, max 1048575 bytes)"), passes with the fix restored.
- Updated fts-language-migration.serial.test.ts: removed an assertion
  that configurable_fts_language (v123) is LATEST_VERSION — that was only
  ever true until the next migration landed; the codebase's own pattern
  elsewhere for this (migrate.test.ts) uses toBeGreaterThanOrEqual, not
  exact-match, for exactly this reason.
- bun run typecheck clean, bun run verify 31/31 green.
- test/page-search-vector-overflow.test.ts (3/3),
  test/reindex-search-vector.serial.test.ts,
  test/fts-language-migration.serial.test.ts, test/migration-v120.test.ts,
  test/sync.test.ts, test/bootstrap.test.ts, test/migrate.test.ts — 254
  total, 0 fail, no regressions.

## Design consultation

Investigated jointly with masa-codex (async design review) before
implementing — their read of the codebase (content_chunks.search_vector
already covers keyword search; pages.search_vector's compiled_truth feed
is the only overflow-prone, effectively-dead write) matched independent
verification and shaped the "remove from trigger" fix over the issue's
alternative options (chunk-grain rebuild — largely already exists; input
truncation — rejected above; ledger-entry-only — insufficient alone,
since the page upsert failing in the same transaction also loses the
chunk write, so checkpoint advancing without this fix would make the
content permanently unsearchable, not just delayed).


Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:51:16 -07:00
MasaandClaude Sonnet 5 1d0b5ed816 fix(autopilot): forward process.env to execSync('which gbrain') under Bun (#2747) (#2976)
resolveGbrainCliPath() (both copies — src/commands/autopilot.ts and the
inlined duplicate in src/core/brain-repo-durability.ts) called `execSync`
without an explicit `env`, relying on default inheritance. Under Bun,
execSync/execFileSync snapshot process.env at BUN'S OWN STARTUP, not at
call time — a runtime PATH mutation (dotenv/config loading, wrapper-script
env sourcing, etc.) happening after Bun boots but before this call is
invisible to `which gbrain` unless the current env is forwarded
explicitly.

This is a known, already-precedented Bun quirk in this exact codebase:
spawn-helpers.ts's detectTini() was already fixed for the identical
symptom with the identical one-line fix (`env: process.env`), with a
comment explaining the mechanism — this call site was simply missed.

Matches the reported symptom precisely: "which gbrain" resolves fine when
run standalone (a fresh Bun process, no prior env mutation to hide), but
throws specifically from inside autopilot's managed-worker spawn path
(src/commands/autopilot.ts:416, guarded by `spawnManagedWorker` — Postgres
engine + minion_mode enabled), which fires after config/dotenv loading has
already run in that process. Impact per the report: this silently
degrades to no worker ever picking up queued jobs (including embed jobs),
with `gbrain doctor` only showing a growing "N stale chunks" warning that
reads like an ordinary backlog rather than a broken worker.

Also improved the throw-path error message to include the actual
PATH/execPath/argv[1] values observed at failure time, so a future report
doesn't require guessing at what the process actually saw.

Not fixed here (documented as a separate, smaller finding): a third call
site with the identical missing-env pattern exists in
src/core/claw-test/runners/openclaw.ts ('which openclaw'). Left out of
scope for this PR, which is specifically about #2747's reported symptom;
worth a small follow-up.

Verification: bun run typecheck clean, bun run verify 31/31 green,
test/autopilot-resolve-cli.test.ts 4/4 pass (existing coverage, no
regressions — a genuinely-simulated Bun-env-snapshot race isn't
reproducible in a same-process unit test, and this codebase's own
convention explicitly avoids mock.module for child_process per
doctor-orphan-ratio.test.ts's stated test-isolation rule, so this PR
relies on the fix's precedent-match + existing coverage rather than a new
mock-based regression test).


Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:39:04 -07:00
MasaandClaude Sonnet 5 4c71a76c0a fix(jobs): retry resets started_at/attempts/stalled_counter (#2783) (#2974)
* fix(jobs): retry resets started_at/attempts_made/attempts_started (#2783)

`gbrain jobs retry` re-queued a dead job by resetting status/error_text/
locks/delay/finished_at, but left started_at, attempts_made, and
attempts_started untouched. On re-claim, claim()'s
`started_at = COALESCE(started_at, now())` preserved the ORIGINAL
first-claim timestamp instead of re-stamping it. handleWallClockTimeouts()
anchors on `now() - started_at`: a retry issued more than timeout_ms * 2
after the original claim was immediately dead-lettered again in under a
second, with attempts_made already past max_attempts — making retry
useless for exactly the case it exists for (recovering work after an
outage that outlasted the job's timeout).

An explicit `jobs retry` is an operator asserting "run this fresh", so
retryJob now also clears started_at (NULL, re-stamped on next claim) and
resets attempts_made/attempts_started to 0.

Two new tests: direct assertion that retry resets all three columns, and
a full repro of the reported bug (wall-clock-killed job retried long
after the original claim now survives re-claim instead of being
immediately re-killed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(jobs): also reset stalled_counter on retry (#2783)

Codex review round 1 found the fix was incomplete: a job dead-lettered by
stall exhaustion (handleStalled() at stalled_counter + 1 >= max_stalled)
retained its exhausted stalled_counter across retry. The retried job's
very first lock expiry after re-claim would immediately re-satisfy the
dead-letter threshold, contradicting the same "run this fresh" intent the
started_at/attempts reset already established.

New test mirrors the existing wall-clock repro: exhaust the stall budget
via two real handleStalled() calls, retry, confirm one more stall now
requeues instead of dead-lettering again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:26:52 -07:00
Masa 354c8c36a9 fix(takes): fail closed when takes-write source resolution errors (#2698 follow-up) (#2973)
resolveTakesSourceId() caught every error from resolveSourceId() and fell
back to undefined, which restores the pre-#2698 unscoped (cross-source)
slug lookup for takes add/update/supersede/resolve. resolveSourceId()
only ever throws when a source was explicitly in play (an invalid or
unregistered GBRAIN_SOURCE, a .gbrain-source dotfile pointing at a
source that doesn't exist, or a genuine DB error) — it never throws for
"nothing configured," which resolves cleanly to the seeded 'default'
source. So swallowing the error had no legitimate case to protect and
only reintroduced the cross-source write bug on any resolution failure.
Let it propagate so the write is blocked instead.

Adds regression coverage for both the unchanged happy path (no source
configured resolves cleanly) and the newly fail-closed path (an
unregistered GBRAIN_SOURCE blocks the write instead of falling back to
an unscoped lookup).
2026-07-20 15:17:43 -07:00
dbf2b3f562 fix(takes): default takes extraction to the configured chat_model instead of hardcoded cloud Haiku (#2997) (#3021)
extractTakesFromPages hardcoded anthropic:claude-haiku-4-5 as the classifier
model. On an OAuth/local-only install (no ANTHROPIC_API_KEY; chat routed
through a gateway model) every takes extraction died with llm_unavailable —
the takes layer silently never populated and the takes_count health check
stayed red despite a working configured chat_model.

Resolution is now `opts.model || getChatModel()` — the same file-plane
gateway-config idiom enrich.ts uses — NOT engine.getConfig('chat_model')
(the DB config plane), keeping model routing on the single config plane the
rest of the codebase reads. Explicit opts.model still wins; unconfigured
installs fall through to the gateway's DEFAULT_CHAT_MODEL.

Adds a regression test that pins the file-plane read: a conflicting DB-plane
config.chat_model row is ignored, the gateway-configured chat_model is used
when opts.model is unset, and explicit opts.model wins. Verified the
file-plane test fails against the pre-fix code.

Takeover of #2997 by @Nazim22 with the model read moved from the DB config
plane to the file-plane gateway idiom.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Nazz <nazim.mj@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 15:05:32 -07:00
9ed53e4e1c fix(code-edges): per-row $n::text::jsonb binds in addCodeEdges — Bun SQL mis-encodes jsonb[] arrays (#2968) (#3020)
Bun SQL double-encodes ::jsonb[] array binds on the Postgres engine: every
edge_metadata element landed as a jsonb string scalar instead of an object,
so the resolver's `edge_metadata || jsonb_build_object(...)` UPDATE produced
a jsonb array and resolved_chunk_id was never readable — code_callers /
code_callees / code_blast returned nothing on Postgres-engine brains while
the resolver logged edges_resolved > 0. PGLite was unaffected (per-row
placeholders already).

Rewrites both inserts (code_edges_chunk, code_edges_symbol) to per-row
$n::text::jsonb placeholders via sql.unsafe — the same shape executeRawJsonb
and the PGLite engine use.

Adds the DATABASE_URL-gated Postgres regression test this class requires
(PGLite cannot reproduce it): asserts jsonb_typeof(edge_metadata) = 'object'
for resolved + unresolved inserts and that the resolver-style || UPDATE
keeps object shape. Verified the test fails 3/3 against the pre-fix code
and passes with the fix.

Takeover of #2968 by @zsimovanforgeops with the missing regression test added.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 14:51:47 -07:00
Masa 9f7244a77f fix(cli): remove sync --install-cron help text — no handler ever existed (#2795) (#2972)
The top-level `gbrain --help` advertised `sync --install-cron` since the
line was first added, but `src/commands/sync.ts` never parsed or handled
the flag — `gbrain sync --install-cron` silently ran an ordinary one-off
sync instead of installing anything, manufacturing false confidence in
the exact durability layer operators reach for it to secure.

git blame shows the line was introduced once (v0.42.29.0 help-text
scaffold) and never touched again — no design intent to recover.
Implementing it would also compete with autopilot, which already owns
this job: `gbrain autopilot --install` runs a self-maintaining daemon
(sync+extract+embed) on a schedule, including a per-source freshness
check that submits `sync` jobs on its own interval. A second, separate
sync-only cron would be a competing scheduler outside the D10
cycle-lock invariant that already keeps autopilot's own targeted-submit
and full-cycle paths from double-processing.

Removed the misleading line and pointed sync's --watch entry at
`autopilot --install`, mirroring the existing `dream` command's
"See also: autopilot --install (continuous daemon)." pattern one
section below. Added regression coverage to
test/cli-help-discoverability.test.ts asserting the help text no
longer promises install-cron and does point at autopilot.
2026-07-20 14:27:50 -07:00
MasaandClaude Sonnet 5 6ec3dd410e fix(gateway): land Anthropic cache_control breakpoints on the system block, not just the call-level auto marker (#2490) (#2981)
gateway.chat() requested cacheSystem:true but never got a system-prompt
cache hit on single-turn callers (page-summary, skillopt, enrich): the
call-level providerOptions.anthropic.cacheControl is real (it becomes
Anthropic's documented top-level "auto-cache the last cacheable block"
shorthand via @ai-sdk/anthropic 3.0.47+), but for a stable system prompt
paired with a different user message every call, "the last cacheable
block" is that ever-varying tail -- every call writes a fresh cache
entry there and never reads a prior one.

Fix: pass system as a SystemModelMessage object (ai's documented shape
for attaching provider options to the system block) carrying its own
providerOptions.anthropic.cacheControl when cacheSystem is requested,
and mirror the same marker onto the last tool def (Anthropic caches
everything up to and including the last cache_control block it sees).
The call-level marker is kept, not removed -- it still gives toolLoop()'s
growing multi-turn conversation a rolling cache breakpoint on each
turn's tail. All three markers now derive from one canonical
cacheControlValue computed after provider_chat_options config merging,
so a configured TTL override (e.g. ttl: '1h') applies consistently
instead of only reaching the call-level marker.

Verified red-before-fix by stashing the gateway.ts diff and confirming
the new assertions fail on unfixed code, then restoring.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:16:00 -07:00
MasaandClaude Sonnet 5 bcf3b73dcf fix(sync): self-heal a never-git-initialized default brain dir (#2964) (#2967)
* fix(sync): self-heal a never-git-initialized default brain dir (#2964)

The dream cycle's sync phase throws unconditionally on a legacy
sync.repo_path-anchored default brain dir that was never git init-ed
(predates git-backed sync, or was rsync'd without its .git), failing
every nightly run with no recovery. doctor's sync_freshness/
sync_consolidation checks report "ok" for this exact brain, but only
because they query the sources table (0 rows for a legacy default
brain) — a coincidental false-negative, not a real diagnosis.

Self-heal by git-initializing the dir and capturing the current
on-disk state as the sync baseline, scoped to !opts.sourceId only —
gbrain owns this directory outright, unlike a registered local source
(sources add --path, no --url) which is the user's own external
directory and should keep failing loudly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): dry-run no-write contract, unborn-HEAD recovery, no-gpg-sign (#2964)

Codex review on the initial self-heal patch (b1671ee) found 3 real gaps:

- P1: the self-heal ran even under --dry-run, mutating the filesystem
  during what's documented as a preview-only command. Gated the whole
  self-heal (both discoverGitRoot and the headCommit read) on
  !opts.dryRun, same as the existing !opts.sourceId ownership check.
- P2: if `git init` succeeded but the process died before the baseline
  commit landed, the next run's discoverGitRoot would succeed (`.git`
  exists) and skip recovery entirely, permanently wedging on "No
  commits in repo" forever. Added the same self-heal at the
  `git rev-parse HEAD` catch site, sharing a new createSyncBaselineCommit
  helper with the discoverGitRoot catch.
- P2: the baseline commit inherited the operator's global
  commit.gpgSign, which can block headless cron/launchd runs on an
  unavailable signing agent/pinentry. Added --no-gpg-sign.

Two new tests cover dry-run no-mutation and unborn-HEAD recovery.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): restrict git auto-init self-heal to the anchor-resolved path only (#2964)

Second Codex review round (a10eeab) found the ownership check still too
loose:

- P1 (security): !opts.sourceId alone isn't proof gbrain owns repoPath.
  jobs.ts's `sync` job handler leaves sourceId undefined whenever
  job.data.repoPath doesn't match a registered source's local_path, so
  an admin-scope submit_job({name:'sync', data:{repoPath}}) MCP call
  could point the self-heal at an arbitrary directory and have it
  silently git-init + commit + ingest it. Gated both self-heal sites on
  !opts.repoPath too — only the path resolved from gbrain's own
  sync.repo_path anchor (never a caller-supplied one) is eligible.

- P2: the unborn-HEAD recovery site calls discoverGitRoot, which walks
  UP from repoPath and can resolve to an ANCESTOR repo for a
  --src-subpath/subdir-as-repoPath sync with an unborn HEAD. Committing
  there would `git add -A` sibling files well outside the sync scope.
  Added a check that gitContextRoot === realpathSync(repoPath) before
  self-healing; refuses (falls through to the original error) otherwise.

Tests rewritten to exercise the true self-heal-eligible path (anchor
config via engine.setConfig('sync.repo_path', dir), no repoPath/sourceId
passed) instead of an explicit repoPath, plus a new test asserting a
caller-supplied repoPath with no sourceId still throws.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): prove self-heal ownership by anchor VALUE, not field presence (#2964)

Third Codex review round (613ad5b) found the previous round's fix broke
the very call site it was meant to repair, plus a second scope gap:

- P1 (critical): !opts.repoPath rejected self-heal on the REAL production
  callers too. runPhaseSync (dream cycle's sync phase, cycle.ts) always
  passes `repoPath: brainDir` explicitly after resolving it upstream, and
  the CLI's bare `gbrain sync` resolves sourceId='default'. Both made the
  ownership gate rethrow, leaving `gbrain dream` and `gbrain sync`
  wedged on the exact non-git legacy brain this fix targets — only
  synthetic callers that omitted both fields ever healed.

  Fixed by proving ownership by VALUE instead of by field absence: a new
  isAnchorOwnedSyncPath() re-reads gbrain's own persisted
  sync.repo_path config and requires the resolved repoPath to equal it
  exactly, regardless of whether the caller passed it explicitly or let
  it default. An attacker-supplied arbitrary path (e.g. via
  submit_job({name:'sync', data:{repoPath}})) only self-heals if it
  happens to already equal gbrain's own anchor — which is the
  legitimate case, not an escalation. opts.sourceId and opts.srcSubpath
  still disqualify unconditionally (registered/subpath-scoped syncs are
  a different ownership context).

- P2: a --src-subpath sync with an unborn parent-repo HEAD would commit
  the whole ancestor root, capturing sibling files outside the scope.
  isAnchorOwnedSyncPath's opts.srcSubpath check closes this; the
  existing gitContextRoot === repoPath check stays as defense in depth.

- P2: manageGitignore's "warn and return" contract (a deliberate side-
  effect that must never kill the sync job for its OTHER callers) meant
  a broken gbrain.yml or unwritable .gitignore would silently let the
  baseline `git add -A` commit db_only content. createSyncBaselineCommit
  now recomputes db_only exclusion directly from loadStorageConfig and
  passes it to `git add` as pathspecs, independent of the .gitignore
  write's success — true fail-closed. (A redundant pathspec exclude for
  a path .gitignore ALREADY covers makes git's -A bail with "paths
  ignored, use -f" even though the negation is correct, so each dir is
  check-ignore'd first and only pathspec-excluded when NOT already
  covered.)

Tests rewritten around the anchor-VALUE model: the critical regression
case (explicit repoPath matching the anchor still heals — the exact
scenario Codex proved was broken) plus a true negative (a caller-supplied
path that does NOT match the anchor still throws), --src-subpath refusal,
and db_only fail-closed exclusion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): allow default-source ownership, realpath compare, generous timeout, --no-verify (#2964)

Fourth Codex review round (9c1d461) found the previous round's ownership
gate still didn't match the REAL installed-brain shape, plus 3 more gaps:

- P1 (critical): rejecting all non-empty opts.sourceId meant self-heal
  still never fired on a real brain. Migration sources_table_additive
  seeds a 'default' source row whose local_path mirrors sync.repo_path
  on every brain that's run it (virtually all of them), so
  resolveSourceForDir (dream cycle) and bare `gbrain sync` both resolve
  sourceId:'default' in practice, never undefined. isAnchorOwnedSyncPath
  now permits sourceId undefined OR exactly 'default' (gbrain's own
  bootstrap identity, never something a caller names) and proves
  ownership by rereading the LIVE anchor for that same identity
  (sources.default.local_path vs config.sync.repo_path).

- P2: compared raw anchor/repoPath strings, so a cosmetic difference
  (trailing slash, ..) between the stored anchor and dream.ts's
  path.resolve()-normalized brainDir would defeat the match. Now
  realpath-compares both sides (fail-closed on ENOENT/dangling).

- P2: the shared git() helper's 30s timeout could abort the baseline
  `git add -A` on a large legacy brain mid-way, after `git init` already
  created `.git` — leaving an unborn repo every subsequent sync would
  retry and time out identically forever. Added an optional timeoutMs
  param (default unchanged at 30s); the baseline add call uses 10min.

- P2: the baseline commit could trigger an operator's global
  core.hooksPath/init.templateDir hooks (pre-commit/commit-msg),
  breaking headless recovery if those hooks need project tooling or
  prompt. Added --no-verify.

Tests: rewrote the mis-scoped "registered source" test (it used
sourceId:'default', which is now correctly permitted) into two — a new
regression test proving sourceId='default' + matching local_path heals
(the actual production shape), and a corrected non-default-sourceId
refusal test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): defer .gitignore write past first import, rebuild index, fail closed on unparseable db_only (#2964)

Fifth Codex review round (c17cd23) found the baseline-commit helper
interacting badly with the pre-existing db_only storage-tiering feature:

- P1 (data loss): createSyncBaselineCommit called manageGitignore BEFORE
  performFullSync's collectSyncableFiles ran. collectSyncableFiles
  enumerates via `git ls-files --cached --others --exclude-standard`, so
  writing db_only entries into .gitignore first would silently exclude
  those pages from the DATABASE, not just from git — on a brain's very
  first sync. This is the exact bug class runSync's existing "manage
  .gitignore ONLY on successful sync" ordering (this file, ~line 4540,
  itself a prior Codex P1 fix, comment literally says so) was written to
  prevent — my new code reintroduced it in a different spot. Fix: stopped
  calling manageGitignore inside the self-heal at all. db_only exclusion
  for the COMMIT still happens via the existing pathspec computation
  (independent of .gitignore); .gitignore itself gets written by the
  already-existing post-sync flow once this sync completes, same as any
  other sync.

- P1 (data leak): the unborn-HEAD recovery site can reach
  createSyncBaselineCommit with a repo whose INDEX already has entries
  staged from some prior operation (manual `git add`, interrupted
  workflow) before gbrain's self-heal ever touched it. `git add -A`
  only adds/updates — it doesn't drop an already-staged path our
  exclusion pathspecs now want excluded. Added `git read-tree --empty`
  to reset the index before staging (no-op on a freshly-`git init`-ed
  repo, whose index is already empty).

- P2: loadStorageConfig warns-and-returns an EMPTY config (not a throw)
  for syntactically-valid-but-unsupported YAML (e.g. flow-style
  `db_only: [dir/]` — the narrow custom parser only handles block-style
  lists), which would silently resolve zero exclusions from a gbrain.yml
  that clearly intended some. Added a sniff-test: if gbrain.yml exists
  and mentions db_only but nothing resolved from it, refuse the baseline
  commit rather than guess "genuinely empty" vs "syntax silently
  ignored" (git init may already have run by this point — same "unborn,
  retry on next sync" recovery path handles it, and will hit this same
  guard again until the user fixes gbrain.yml).

Tests: a positive regression proving db_only markdown IS imported into
the DB on first sync (the actual data-loss scenario), the sniff-test
refusal, and the stale-staged-content-gets-dropped case for the index
rebuild.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): post-heal .gitignore write, supabase_only alias, honor abort signal (#2964)

Sixth Codex review round (e687913), 3 P2s:

- Dream-cycle callers (cycle.ts:runPhaseSync) invoke performSync directly
  and never run runSync's CLI-only post-success manageGitignoreAtGitRoot.
  A brain self-healed only via the dream cycle would have db_only content
  correctly excluded from the baseline commit (createSyncBaselineCommit's
  pathspec exclusion) but no .gitignore ever written, leaving the user's
  own future manual git add/commit unprotected. Added
  performFullSyncAndMaybeGitignore, a thin wrapper around the 3
  post-self-heal performFullSync call sites that writes .gitignore
  (same success-status gate runSync already uses) only when didSelfHeal
  is true — a no-op for the normal path, which still relies on runSync
  exactly as before.

- The fail-closed sniff-test only checked the canonical `db_only` key;
  the deprecated-but-still-supported `supabase_only` alias (same
  keep-out-of-git semantics) could silently bypass it. Now checks both.

- Self-heal didn't check opts.signal?.aborted before starting the
  (now up to 10-minute) git init + baseline commit, so a cancelled sync
  could still mutate disk and overrun its budget instead of returning
  partial. Added the check at both self-heal sites, before any git
  operation runs.

New test proves .gitignore gets written after a bare performSync call
(no runSync wrapper) — the actual dream-cycle shape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): neutralize a leftover .gitignore during the self-heal first sync (#2964)

Seventh Codex review round (27337b0) ran an actual repro and caught the
primary motivating scenario still broken: a brain rsync'd from another
machine without its .git can retain that machine's old auto-managed
.gitignore. collectSyncableFiles (inside performFullSync) enumerates via
`git ls-files --exclude-standard`, so a leftover db_only ignore rule
would silently omit those pages from THIS first sync's DATABASE import —
the same bug class the round-6 ordering fix prevented for a .gitignore
gbrain would have written itself, just triggered by a pre-existing file
this time.

Fix: performFullSyncAndMaybeGitignore now neutralizes any existing
.gitignore for the duration of the one first-sync call — read, delete,
restore byte-for-byte immediately after (even on error) — before
manageGitignore re-merges the managed db_only block onto the restored
original content. This matches exactly what a truly fresh brain with no
.gitignore at all already does on its first sync (nothing to suppress
collection there either); db_only content stays out of the git COMMIT
independently via createSyncBaselineCommit's pathspec exclusion, which
never depended on .gitignore.

Test proves both halves: db_only markdown IS imported despite a leftover
ignore rule, AND the user's own unrelated .gitignore lines (e.g.
.DS_Store) survive the restore intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): simplify — drop db_only-import machinery, isolate hooks fully (#2964)

Eighth Codex review round (54c1e6f) found MORE problems with round 7's
.gitignore-neutralization fix (deleting the whole file loses the user's
own unrelated ignore rules; a multi-sync retry scenario could silently
skip a still-broken db_only file while advancing the bookmark) plus 2
more issues in existing code. Rather than patch those too, stepped back
and checked the actual documented semantics of db_only
(docs/storage-tiering.md): it's for "bulk machine-generated content...
written to disk as a local cache" — DB is the source of truth, disk is a
cache populated FROM the DB (`export --restore-only` restores it), never
the other way. Nothing in the docs says `gbrain sync`'s git-diff-based
file collection is how db_only content is supposed to reach the
database — that's ingest-specific tooling's job. Confirmed directly:
`loadStorageConfig` returns the byte-identical `{db_tracked:[],
db_only:[]}` for a malformed flow-style array AND a literal empty
`db_only: []`, so rounds 6-7's "ensure db_only markdown gets imported on
this first sync" chase was solving a problem outside sync's actual scope
in the first place, on an increasingly complex, adversarially-discovered-
edge-case foundation.

Reverted: performFullSyncAndMaybeGitignore (the wrapper + didSelfHeal
tracking + .gitignore neutralize/restore dance + post-success
manageGitignore call). After self-heal, import and any subsequent
.gitignore management now behave EXACTLY like any other brain,
self-healed or not — runSync's existing post-success
manageGitignoreAtGitRoot covers the CLI path identically either way; the
dream cycle not calling it is a separate, pre-existing characteristic of
the dream cycle in general (applies equally to an already-git-initialized
brain going through the same path), not something this fix introduces.

Kept (still correct, self-contained, don't depend on the reverted
machinery): createSyncBaselineCommit's pathspec-based db_only exclusion
for the COMMIT itself (matches the documented "not committed to git"
requirement), the fail-closed sniff-test guard (now documents its
known, structurally-unavoidable false-positive on a genuinely-empty
`db_only: []` — the trade-off is deliberate: low-cost, self-resolving
false positive vs. high-cost, hard-to-undo false negative), the index
rebuild, and the 600s add timeout.

Improved (round 8, P2): hooks isolation. --no-verify only skips
pre-commit/commit-msg; added `-c core.hooksPath=/dev/null` for the
baseline commit, which disables prepare-commit-msg and post-commit too
(the latter runs synchronously inside the same git invocation and could
otherwise hang past the timeout without even being the slow step).

Tests: removed the 3 that exercised the reverted db_only-import
machinery; the remaining 12 (ownership, dry-run, index rebuild, sniff
test, commit-exclusion, unborn-HEAD recovery) are unaffected by the
simplification.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): unconditional db_only exclusion, literal pathspecs, precise sniff test (#2964)

Ninth Codex review round (a6e07f6):

- P1: the check-ignore pre-filter (skip pathspec-excluding a dir already
  covered by .gitignore) could be defeated by a pre-existing .gitignore
  that ignores a db_only tree with a wildcard but re-includes a child via
  negation (e.g. `private-cache/*` + `!private-cache/index.md`) —
  check-ignore on the directory still reports "ignored", so the filter
  skipped the pathspec exclusion, and `git add -A` staged the re-included
  child anyway. Fixed by making exclusion unconditional: every db_only
  dir is always pathspec-excluded now, never pre-filtered against
  .gitignore state at all — our own pathspec doesn't consult .gitignore,
  so no .gitignore content (negated or not) can defeat it. The advisory
  "paths ignored... use -f" error this can now trigger when a dir IS also
  already .gitignore'd (verified: git still stages everything else
  correctly despite the nonzero exit) is caught and swallowed by matching
  its exact stderr text; anything else rethrows.

- P2: `:!dir` pathspec shorthand reinterprets a dir name that itself
  starts with a pathspec magic character (e.g. `:private/`) instead of
  excluding it literally. Switched to `:(exclude,literal)dir`.

- P2: the fail-closed sniff-test's bare substring search on gbrain.yml's
  raw content could trip on a comment or unrelated prose mentioning
  "db_only" even when there's no real storage section at all, refusing
  self-heal forever on an unrelated false positive. Now requires an
  actual YAML key line (`db_only:`/`supabase_only:`, trimmed, ignoring
  `#` comments) — the round-8-documented "genuinely empty db_only: []"
  false positive is unchanged and remains an accepted trade-off (still
  structurally indistinguishable from unsupported syntax at the
  loadStorageConfig API boundary), but comment/prose mentions no longer
  false-positive.

Not fixed (deliberately, documented trade-off — see PR description):
Codex's other P1 this round (refuse baselining when other git refs/
history exist alongside an unborn HEAD) is a narrow, non-destructive
scenario — self-heal only ever acts on the current branch ref when it's
provably commit-less, never touches or deletes any other ref (remote-
tracking, other branches), so at worst it creates a possibly-unexpected
extra commit on an otherwise-empty branch the user hadn't checked out
yet. Chasing it further trades diminishing real-world risk reduction
against unbounded scope growth in what's fundamentally still the
self-heal fix from round 1.

Two new tests: unconditional exclusion despite a matching pre-existing
.gitignore (proves the advisory-swallow path), and a comment-only
gbrain.yml no longer false-positives the sniff test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:06:21 -07:00
MasaandClaude Fable 5 23e0541d9b fix(cycle): budget the patterns subagent from remaining job time — one phase's fixed 35-min worst case defeats any interval-derived cycle budget (#2781) (#2959)
* fix(cycle): budget the patterns subagent from remaining job time, not a fixed constant (#2781)

The autopilot-cycle job gets an interval-derived timeout stamped at submit,
but the patterns phase submits its subagent with a fixed 30-min job timeout
and waits up to 35 min — one phase's worst case exceeds ANY interval-derived
budget <= 35 min, so the parent job dead-letters mid-patterns and the tail
phases (consolidate -> schema-suggest) starve for days (#2781, the deeper
half left open by the #2852 dispatch-floor fix).

- MinionJobContext.deadlineAtMs: absolute deadline from the claim-time
  timeout_at stamp (the DB ground truth handleTimeouts() sweeps against;
  re-stamped on every claim so retries get a fresh budget). Null when the
  job has no per-job timeout.
- worker: the per-job abort timer now derives its delay from timeout_at
  when present, so the in-process timer, the DB sweeper, and the
  handler-visible deadline agree on ONE absolute instant.
- autopilot-cycle + autopilot-global-maintenance handlers thread
  deadlineAtMs into runCycle; CycleOpts carries it to the patterns phase.
- patterns: clampSubagentBudgets() derives BOTH the child job timeout and
  the wait timeout from the same child deadline (parent deadline minus a
  60s stop-margin reserve — enough for the wait poll + force-evict grace
  + cleanup, deliberately NOT a promise that tail phases complete). Under
  a 2-min minimum the phase skips honestly (insufficient_cycle_budget)
  instead of submitting a guaranteed-kill LLM call; the next cycle
  retries with a fresh budget.
- Direct callers (gbrain dream) pass no deadline and keep the configured
  timeouts unchanged.

Follow-up (separate PR): synthesize has the same shape plus sequential
per-child waits that accumulate N x subagent_wait_timeout_ms past any
parent budget; it needs per-wait remaining-time recomputation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

* fix(cycle): address review — cancel timed-out patterns child; thread deadline through phase-wrapper handlers

- P1: the child's timeout_ms clock starts at ITS claim, so a queued child
  could outlive the parent deadline the wait was clamped to. On wait
  timeout, cancelJob strips it (waiting -> cancelled; active -> lock
  stripped, worker abort fires next renew tick).
- P2: makePhaseHandler (standalone patterns/synthesize/... minion jobs)
  now threads job.deadlineAtMs into runCycle like the autopilot handlers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:56:00 -07:00
bo-developingandClaude Opus 4.8 c873ce3014 feat(ai): add Mistral provider recipe (#3001)
Adds an EU-hosted provider covering embedding, expansion and chat on one
OpenAI-compatible endpoint (https://api.mistral.ai/v1), so a brain that must
stay inside EU jurisdiction does not need a US hop for any AI touchpoint.

Every field is measured against the live API, not copied from docs:

- mistral-embed is fixed 1024 dims and accepts no dimension parameter.
  Both spellings are rejected: {"dimensions": N} returns 400 extra_forbidden,
  {"output_dimension": N} returns 400 "does not support output_dimension".
  The generic openai-compatible branch of dimsProviderOptions() already falls
  through to `return undefined` for these model ids, so nothing is emitted.
  Same contract as voyage-4-nano, pinned by a negative assertion in the test.
- max_batch_tokens 65536: a 65,286-token batch is accepted, 66,960 returns
  400 code 3210 "Too many tokens overall, split into more batches."
- chars_per_token 2: the value is a DIVISOR in splitByTokenBudget()
  (estTokens = text.length / charsPerToken), so lower is the conservative
  direction. The module default of 4 assumes English prose; a German-language
  corpus measured 3.58 chars/token, which the default overshoots toward
  overflow.

codestral-embed is deliberately left out: it returns 1536 dims, and a
touchpoint carries a single default_dims. Listing it under a 1024 declaration
is the mixed-dim case embedding-dim-check.ts exists to catch.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 13:42:41 -07:00
hhamilton-fv f3e78fd2fb fix(providers): route diagnostics through buildGatewayConfig so file-plane keys are visible (#3000)
configureFromEnv() hand-assembled its own AIGatewayConfig instead of calling
buildGatewayConfig(), the single seam that folds file-plane API keys
(openrouter_api_key, zeroentropy_api_key, ...) into the gateway env. That let
`gbrain providers list`/`test` report a provider as missing env even when it
was correctly set in ~/.gbrain/config.json and the real gateway path resolved
it fine. Both configureFromEnv() and runList() now build their env through
buildGatewayConfig() (falling back to a bare process.env passthrough
pre-init), matching what init-provider-picker.ts already does.
2026-07-20 13:07:27 -07:00
Nazim22andClaude Fable 5 4528bfa79c feat(cli): GBRAIN_DRAIN_TIMEOUT_MS env override for the per-sink teardown drain budget (#2996)
The one-shot CLI teardown drains fire-and-forget background sinks with a
hardcoded 2s per-sink budget (DEFAULT_DRAIN_TIMEOUT_MS). That budget
assumes a sub-second cloud chat provider; on a self-hosted provider (e.g.
an ollama model at 10-20s per completion) a facts:absorb extraction can
never finish inside it, so every one-shot CLI exit — sync timers
especially — aborts the in-flight chat with
'pipeline_error: The operation was aborted', and the same touched pages
retry-and-abort on every subsequent sync. Facts from those pages silently
never land, and doctor's facts_extraction_health warns permanently.

Fix: resolveDrainTimeoutMs() — GBRAIN_DRAIN_TIMEOUT_MS env override
(same env-only escape-hatch pattern as GBRAIN_TEARDOWN_DEADLINE_MS and
GBRAIN_FLUSH_GRACE_MS) over the 2000ms default. Explicit drainTimeoutMs
from a call site still wins; computeTeardownDeadlineMs already computes
the backstop from the resolved value, so the deadline scales with it.
Garbage/zero/negative env values fall back to the default.

Tests: default, env override, garbage/zero/negative fallback,
finishCliTeardown drains with the env-resolved budget, explicit opts
still win over env.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:00:16 -07:00
Anton Senkovskiy 6498b872ea fix(extract): --dry-run must not write extract_rollup_7d (#2994)
`extract-conversation-facts --dry-run` promises "no DB writes, no
checkpoint advance" (help text) and correctly skips the fact INSERTs,
orphan delete, checkpoint advance, and receipt page. But
writeRunReceiptAndRollup was called unconditionally at both exit paths,
and its upsertExtractRollup always UPSERTs a row into extract_rollup_7d
("ALWAYS fire so doctor's extract_health sees the cycle ran") — so a
dry run mutates the DB.

Gate both writeRunReceiptAndRollup call sites on !dryRun. The writer
returns void and its only non-rollup action (the receipt page) is
already suppressed in dry-run via facts_inserted > 0, so gating at the
call site skips nothing else. Mirrors the existing !dryRun guards on
the fact-insert / checkpoint / audit paths.

Regression test: a dry run leaves extract_rollup_7d empty. Fails before
(row count 1), passes after. Hermetic PGLite + stubbed transports, no
live LLM.

Note: --dry-run still calls the extractor (LLM) by design — facts_extracted
is the reported "would extract N" preview count; left unchanged.
2026-07-20 12:16:57 -07:00
324c355318 test(e2e): production guard — setupDB refuses non-test databases (#2957)
setupDB() TRUNCATEs every data table on whatever DATABASE_URL points at,
and run-e2e.sh deliberately preserves an exported DATABASE_URL — one
stray environment variable away from wiping a production brain, with no
guard of any kind (found during an independent review, 2026-07-18).

assertSafeE2eDatabaseUrl (pure, unit-tested) now runs before any
connection: allowed when the database name carries "test" as a word
segment (the gbrain_test convention used by CI and
.env.testing.example), or when GBRAIN_E2E_ALLOW_DB names the exact
database intentionally. Refusal is loud and actionable. 7 unit tests,
no DB required.

Co-authored-by: Aleksei Razsadin <razsoc.01@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 12:05:04 -07:00
184b6cb8a1 fix search: title candidate arm + gated OR fallback for lexical recall (#2956)
Pages were unreachable by their own exact titles: FTS indexed only chunk
body text while the title-weighted pages.search_vector (GIN-indexed since
its introduction) was never queried by any search path, and
websearch_to_tsquery AND-at-chunk-grain semantics meant one non-matching
token zeroed keyword recall with no fallback — long or acronym-bearing
titles (e.g. "IAWG ... AAR-LL deck") fell through to the vector arm alone
and missed.

- searchTitles (both engines): page-grain candidate arm over
  pages.search_vector (title 'A' + compiled_truth 'B' + timeline 'C'),
  ts_rank_cd ranked, representative-chunk LATERAL join, full filter
  parity with searchKeyword (visibility, soft-delete, source grants,
  hard-excludes, dates, types); fused as a weighted RRF list at the
  keyword arm's intent-effective k on all three hybrid return paths;
  fail-open with warnOncePerProcess. No schema changes — the index
  already existed, dark.
- AND->OR one-retry fallback for the keyword arm, gated behind
  SearchOpts.orFallback (only hybridSearch opts in; countMentions, link
  resolution, eval, and keyword-only MCP callers keep the strict-AND
  contract). Refused for queries carrying websearch operators (negation,
  quoted phrases). searchTitles carries its own page-grain fallback.
- Lexical arms parallelized (Promise.all) on the main path.

Verified: typecheck clean; 18 hermetic PGLite tests + 2 engine-parity e2e
cases (CI Postgres); consumer regression enrichment 18/0 +
link-extraction 127/0; independent live QA on a 10,664-page brain —
exact-title target miss -> rank 1 (exact_title_match), controls held,
negation/quoted guards proven, strict-consumer contract pinned.

Diagnosed from a 3-lane read-only diagnostic; adversarial review round
closed findings on fallback scope, Postgres test coverage, and operator
handling before this commit.

Co-authored-by: Aleksei Razsadin <razsoc.01@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:57:31 -07:00
MasaandClaude Fable 5 912407bef1 fix(search): per-call token-budget meta masked the real cut on both cache paths; restore vacuous search-lite coverage (#2954)
* fix(search): per-call token-budget meta no longer masks the real cut; restore vacuous search-lite coverage

The search-lite integration tests were structurally vacuous: putPage never
creates chunks and searchKeyword joins content_chunks, so every fixture
query returned zero rows on every machine. The tight-budget cut test's
defensive skip ('keyword search may dedupe by page') silently returned
before its assertions had ever executed anywhere, and the two budget-meta
tests ran against empty result sets.

Restoring the fixture (upsertChunks per page) and hardening the
assertions immediately exposed a real meta bug: with a per-call
tokenBudget, the inner hybridSearch enforces the same resolved budget
(per-call wins in resolveSearchMode) and its meta carries the true
dropped count — but hybridSearchCached re-applies the budget to the
already-cut set and published THAT pass's meta, which always reads
dropped=0. onMeta consumers saw a budget record claiming nothing was
dropped while rows were; telemetry (recorded from the inner meta)
disagreed with the caller-visible meta.

- hybrid.ts finalMeta: prefer innerMeta.token_budget when a per-call
  budget is set (outer budgetMeta stays as the fallback and remains the
  enforcement for the cache-HIT path, where no inner run exists)
- test fixture: chunk each page (pattern: chunk-grain-fts.test.ts)
- cut test: defensive skip replaced with a hard >=2 precondition;
  results non-empty + strictly-fewer-than-unbounded + dropped>0 now
  actually execute (revert-checked red on the unfixed meta)
- budget-meta tests: assert non-empty result sets so kept=results.length
  can no longer pass vacuously at 0=0

The unbounded 'builder' query returns 2 of 3 fixture pages by design —
dedup Layer 3 caps any single page type at 60% of results and the
fixture is all-person — which the >=2 precondition accommodates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

* fix(search): cache-HIT budget meta prefers the stored cut record per review; exact-count assertions, mixed-type fixture

- hit path had the symmetric masking (codex P2): the re-application runs
  on the already-trimmed stored set and read dropped=0 while the miss
  that produced the same result set reported the real cut. Prefer
  hit.meta.token_budget unconditionally — tokenBudget is folded into
  knobsHash ('tb='), so a hit only ever serves a lookup with the
  identical resolved budget as the write and the outer pass can never
  cut further (verified against mode.ts; this is why the reviewer's
  'outer wins when it drops' branch is unreachable). budgetMeta remains
  the fallback for legacy rows stored without a budget record
- new serial test drives a real store-then-hit roundtrip (mocked
  embedQuery, real PGLite cache) and pins hit token_budget == miss
  token_budget; revert-checked red (dropped=0) on the unfixed path
- lite test: dropped asserted as the exact unbounded-minus-kept count
  and used>0 (dropped>0 alone accepts any wrong positive; used<=250
  alone accepts a bogus zero), fixture types mixed (person/company/note)
  so dedup Layer-3 type-diversity policy no longer shapes the test

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:49:47 -07:00
MasaandClaude Fable 5 89f226eb38 fix(search): classify cache hit/miss in telemetry — hits were invisible, misses unclassified (#2953)
* fix(search): classify cache hit/miss in telemetry — hits were invisible, misses unclassified (#2952)

search stats reported 0 hit / 0 miss forever: recordSearchTelemetry fired
only from bare hybridSearch, whose meta never carries a cache field, and a
cache HIT returned from hybridSearchCached before any record at all — so
hit searches also vanished from count/results/tokens/rank-1.

- HybridSearchOpts: internal _telemetryCacheStatus ('miss' | 'disabled')
  threaded from hybridSearchCached into the inner hybridSearch (same
  pattern as _queryEmbedDeadline), folded into the RECORDED meta only —
  onMeta payloads unchanged, count/sum_tokens/budget_dropped/rank-1
  behavior byte-identical for the miss/disabled paths
- hit path: record once from hybridSearchCached with the already-built
  cachedMeta (cache.status='hit'), post-slice/budget result count, tokens
  from the budget pass, and the same rank-1 rule as the inner paths
- bare hybridSearch direct callers (think/gather, brainstorm, enrich,
  evals, ...) keep recording exactly as before, with no cache field
- test: serial wiring test drives a real store-then-hit roundtrip through
  hybridSearchCached (mocked embedQuery, real PGLite SemanticQueryCache)
  and pins the decision matrix (miss / hit / consult-skipped / bare);
  revert-checked red on pre-fix source at the miss classification

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

* fix(search): harden cache-hit telemetry per review — mode-gated tokens, embed-failure coverage

- hit-path tokens_estimate now gated on the MODE-resolved budget,
  mirroring the inner paths' resolvedMode.tokenBudget > 0 meta condition
  (a tokenmax budget-off brain would otherwise record real tokens on hits
  but 0 on misses, inflating avg-tokens as the hit rate rises)
- test: exact token-delta parity assertion (hit contributes the same
  tokens as the miss that stored the served set) — catches the class of
  hit/miss accounting asymmetry the increase-only check accepted
- test: embed-provider-failure flavor of the disabled path (consult
  degrades via catch, keyword fallback serves, neither counter bumps)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:35:02 -07:00
Sailesh SivakumarandClaude Fable 5 f1031d5a0b fix(cli): stop thin-client jobs/config from fabricating a scratch PGLite (#2951)
`jobs list|get` have had remote MCP routing since v0.32, but the CLI
shell still ran connectEngine() before dispatch — on a thin-client
install that fabricates an empty scratch PGLite in the thin-client
GBRAIN_HOME and replays the entire migration chain on every invocation,
before the remote call even runs. Host-only jobs subcommands (work,
supervisor, submit, ...) and `config` did the same instead of refusing.

- cli.ts: dispatch thin-client `jobs list|get` engine-free
  (runJobs(null, ...)); refuse the other jobs subcommands with a
  pinpoint hint; add `config` to THIN_CLIENT_REFUSED_COMMANDS with a
  hint (it reads/writes the host brain's config plane).
- jobs.ts: widen runJobs to accept a null engine, guarded so null can
  only reach the MCP-routed list/get branches.
- tests: behavioral (no scratch store created, no migration replay,
  refusals carry hints) + source-audit pins in the existing idioms.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:25:46 -07:00
Sailesh SivakumarandClaude Fable 5 3a5c4c194c fix(remote): poll MinionJob.status, not .state — ping now sees completion (#2950)
submit_job/get_job return the MinionJob row verbatim; its lifecycle
field is `status` (src/core/minions/types.ts), not `state`. remote ping
typed and read `state`, so every poll saw undefined, the terminal check
never matched, and ping always burned its full --timeout and exited 1
even when the autopilot-cycle had completed — printing
"Job #N is still undefined." on the way out.

Reads fixed to `status`; the ping's own JSON output keys (`state`,
`last_state`) are unchanged for consumers. Source-audit regression test
pins the field reads.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:16:21 -07:00
MasaandClaude Fable 5 d165e99f0b fix(brain-writer): deadline race verdict from the sentinel, not the clock the timer raced (#2947)
Part of #2946. The hung-COUNT deadline race derived its verdict from a
post-await wall-clock re-check, which races the timer's own drift: on
loaded CI runners setTimeout callbacks fire measurably EARLY relative to
Date.now(), so each padding/boundary adjustment (>=, +1ms) only moved
which wrong status the partial-scan test received ('scanned', then
'partial').

The race's timeout arm now resolves a module-private sentinel; the
sentinel winning IS the deadline verdict (deadlineHit), consulted by the
post-await check without re-reading the clock. A COUNT that resolves
null (failed/absent count) stays distinguishable and does not skip the
scan; a COUNT that resolves slowly without the timer winning is still
caught by the retained wall-clock re-check. The +1ms pad is gone — the
sentinel makes timer drift irrelevant for the hung path.

Verified: partial-scan suite green 8 consecutive runs incl. the new
null-vs-sentinel distinction test.


Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:08:41 -07:00
raymeboltdandOpenAI Codex 8b325041ee v0.42.63.0 fix: preserve configured PGLite schema database path (#3016)
* fix(schema): preserve configured PGLite database path

* chore: bump version and changelog (v0.42.63.0)

Co-Authored-By: OpenAI Codex <noreply@openai.com>

---------

Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-20 10:46:46 -07:00
a46f28a63e fix(cli): keep doctor --json stdout clean — v123 migration handler printed to stdout (#3019)
The v123 configurable-FTS migration (#2941) logged its completion notice
via console.log. Migrations run lazily inside any command's first DB
connect, so on the nightly heavy run (fresh Postgres service DB) the
line landed as the first line of `gbrain doctor --json` stdout and broke
the fm_wallclock jq parse ("Invalid numeric literal at line 1, column 7",
run 29731426470). runMigrations' contract routes all migration noise to
stderr; move the v123 prints (and the pre-existing v2 slug-rename print)
there.

Also un-vacuous the fm_wallclock harness: its register-source step used
`bun run -e` (bun dumps usage with exit 0 instead of running the code)
and `connect({})` (in-memory), so the source was never registered and
doctor scanned nothing. It now resolves the engine the way the CLI does
and registers the source in the DB doctor actually reads.

Regression test: test/migrate-stdout-clean.test.ts re-runs migrations
from v122 asserting zero stdout writes, plus a source-level guard that
migrate.ts contains no console.log.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 10:33:27 -07:00
f72de97943 feat(sync): --src-subpath + --exclude for monorepo subdir-source support (#753, supersedes #774) (#2942)
* feat(sync): --src-subpath + --exclude for monorepo subdir-source support (#753)

Rebased port of PR #774 onto current master. A single git repo can hold N
logical sources at subdirectories: git operations run at the discovered
repo root (git rev-parse --show-toplevel — worktrees and submodules
resolve natively) while file walking, imports, deletes and renames are
scoped to the subpath. Passing the subdirectory directly as the repo path
(auto-discovery) works through the same code path.

Path-containment guards (the point of the feature):
- NAV-1/NAV-2: the realpath-resolved scope must live inside the
  realpath-resolved git root — ../-traversal and symlinked subdirs
  pointing outside the repo are rejected before any git op.
- NAV-1 TOCTOU: per-file realpath re-validation during the incremental
  import drain and rename reimport; symlink-escape files are recorded as
  failures (fail-closed — the bookmark cannot advance past an escape).
- NAV-4: an --exclude set that filters out every candidate warns loudly.

Scoped syncs use git-root-relative slugs + source_path in BOTH the full
and incremental paths (runImport gains slugRoot), fixing the original
PR's full/incremental slug divergence in the auto-discovery flow.
--exclude matches scope-relative paths in both paths; exclusion never
deletes previously-imported pages. The full-sync delete reconcile is
scope-restricted and relativizes against the slug base so a healthy
scoped source can't trip the #2828 mass-delete valve. .gitignore
management resolves to the git root at every call site.

Preserves all master-side sync work since the original branch: the #2828
mass-delete safety valve, #1794 resumable checkpoints + pinned targets,
#1950 stall watchdog, #2335 heartbeat bump, #1970 bookmark reachability,
and the git-ls-files walker (#2315/#2462/#2678, whose symlink/cycle
hardening is untouched).

Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: listEverCommittedPaths uses gitContextRoot after #753 root-triple refactor

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:43:09 -07:00
f8d11f67a3 feat(search): configurable FTS language + reindex command (lands #580/#581/#582) (#2941)
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:

- Migration renumbered v116 -> v123 (master's v116 was already claimed by
  code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
  REPLACE trigger-function bodies (migration handler + reindex command)
  now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
  resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
  reindex_search_vector.pages/.chunks), id-keyset batched backfill
  (5000 rows/UPDATE) instead of single whole-table statements, and
  --json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
  written (README referenced it but no PR added it); llms bundles
  regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
  check-test-isolation).

Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:23:47 -07:00
93cfb37540 feat(engine): opt-in Postgres RLS source-scope binding (lands #2387) (#2940)
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.

With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.

Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
  (the #1794 PgBouncer pool-exhaustion class). Only the three search
  methods keep a transaction when off — exactly the sql.begin() +
  SET LOCAL statement_timeout wrap they already had on master — via the
  helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
  setseed() + SELECT to one connection when seeded (alwaysTransaction
  gated on opts.seed), so the deterministic path can't split across
  pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
  (search methods route through withScopedReadTransaction with
  alwaysTransaction; helper owns the sql.begin(); flag-off path is
  callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
  pass-through, flag-off alwaysTransaction, flag-on set_config emission,
  federated > scalar > '*' precedence, and the CSV as a bound parameter
  (never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
  ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
  SECURITY for owner roles, and the honest caveat about unwrapped paths.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:15:49 -07:00
9fe4628d02 feat(engine): opt-in Postgres RLS source-scope binding (lands #2387) (#2940)
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.

With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.

Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
  (the #1794 PgBouncer pool-exhaustion class). Only the three search
  methods keep a transaction when off — exactly the sql.begin() +
  SET LOCAL statement_timeout wrap they already had on master — via the
  helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
  setseed() + SELECT to one connection when seeded (alwaysTransaction
  gated on opts.seed), so the deterministic path can't split across
  pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
  (search methods route through withScopedReadTransaction with
  alwaysTransaction; helper owns the sql.begin(); flag-off path is
  callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
  pass-through, flag-off alwaysTransaction, flag-on set_config emission,
  federated > scalar > '*' precedence, and the CSV as a bound parameter
  (never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
  ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
  SECURITY for owner roles, and the honest caveat about unwrapped paths.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:15:10 -07:00
1833d95896 fix(dream,chronicle): synthesize/concepts output family — source scope, output root, retrieval reach, durable provenance, honest judge failures (#1586 #2415 #2163 #2569 #2606) (#2939)
Five verified-open fixes to the dream/synthesize output family:

- #1586: thread the cycle's resolved sourceId (cycleSourceId) through
  runPhaseSynthesize -> SubagentHandlerData.source_id -> subagent tool
  OperationContext, and stamp collected refs + summary page with the same
  source, so synthesized pages stop landing in 'default'.
- #2415: new config knob dream.synthesize.output_root (default 'wiki',
  zero behavior change unless set) drives the synthesize prompt slug
  templates, the patterns reflection lookup + prompt, and remaps the
  filing-rule allow-list globs. Registered in KNOWN_CONFIG_KEYS.
- #2163: synthesize_concepts writes concept pages through
  importFromContent (put_page's parse->chunk->embed pipeline) instead of
  bare engine.putPage, so concepts/ pages are chunked + embedded and
  reachable by retrieval.
- #2569: stampDreamProvenance persists dream_generated + dream_cycle_date
  into pages.frontmatter (JSONB merge via executeRawJsonb) at write time,
  so generated pages are DB-queryable and put_page write-through can't
  erase the marker.
- #2606: chronicle judge detects stopReason 'length' truncation and
  no-JSON-array parse failures as distinct skipped reasons
  (judge_truncated / judge_parse_failed) instead of a false terminal
  no_events; output cap raised to 4000 and configurable via
  chronicle.judge_max_tokens.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-17 15:01:27 -07:00
42375bded5 fix(sync): stop the sync data-loss family — ops/ prune, DB-only write-through, full-sync gate drift (#2404, #2426, #2607) (#2938)
Three verified-open defects, one family: sync silently destroying or
diverging on content it should preserve.

#2404 (P0) — 'ops' was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era
carve-out), so any path with an ops segment was 'pruned-dir': committed
ops/*.md never imported, and modified ops/* files hit the
unsyncableModified delete loop (whose #1433 guard only spared
'metafile'), silently deleting put-created pages like the bundled
daily-task-manager's canonical ops/tasks on every sync. Fix: remove
'ops' from the prune list (ordinary user content; the vendor/generated
entries stay), and harden the delete loop to also skip 'pruned-dir' —
a page under a pruned dir can only exist via a deliberate put_page.

#2426 (P0) — write-through content stayed DB-only and was deleted by
sync --full. All three compounding bugs fixed:
 1. writePageThrough now best-effort commits the artifact (path-limited
    git commit) on durability-hardened repos, so the post-commit hook
    can push it; result carries committed?: boolean.
 2. scripts/brain-commit-push.sh stages+commits BEFORE any pull — the
    old fetch+pull-rebase-first order aborted on any dirty tree, so the
    helper could never commit a MODIFIED page; brain_push's
    rebase-on-reject already handles an advanced remote.
 3. The full-sync delete-reconcile partitions stale pages by git
    history (listEverCommittedPaths): never-committed source_paths are
    DB-only write-through — pages are KEPT and re-exported to the
    working tree instead of soft-deleted. Builds on the #2828
    mass-delete valve (covers the below-valve cases).

#2607 — the sync --full git ls-files fast path bypassed pruneDir, so a
full pass imported (and resurrected soft-deleted) pages under dot-dirs
and vendored trees that incremental sync excludes. Fix:
isCollectibleForWalker applies the same segment-level pruneDir gate as
classifySync, so full and incremental enumeration agree.

One regression test per defect (all verified failing against master
src): test/sync-ops-pages.serial.test.ts,
test/write-through-commit.serial.test.ts, the #2426 helper-order test
in test/brain-durability-hook.serial.test.ts,
test/sync-reconcile-db-only.serial.test.ts,
test/import-git-fastpath-prune.test.ts.

Fixes #2404
Fixes #2426
Fixes #2607

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:41:25 -07:00
Konradopenclaw a8e6b1d177 feat(ai): add Moonshot Kimi provider recipe (#2378) 2026-07-17 14:31:59 -07:00
54a8070640 fix(facts): make dream extract_facts idempotent so fence rows don't duplicate each cycle (#2932)
Port of #1837 (mvanhorn) onto current master. The extract_facts cycle
phase unconditionally wipe-and-reinserted every page's fence-owned DB
rows, so re-running a cycle on unchanged content churned rows and — on
the Postgres engine reported in #1781 — accumulated duplicates each run.

The phase now de-dupes extracted facts by the canonical (claim, source)
content key and reconciles the page-scoped DB index: no-op when already
in sync, insert-only for new keys, wipe/reinsert only when stale rows
need cleanup.

Adjustments over the original PR to fit current master:
- preserve #1972's abortSignal threading into the batch embed call
- preserve #1928's excludeSourcePrefixes: ['cli:'] on every wipe, and
  exclude cli:-origin conversation facts from the existing-row set so
  they neither count as stale (which would force a wipe every cycle)
  nor get compared against the fence

Fixes #1781

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:29:07 -07:00
e1cefd0654 fix(subagent): orchestration fix-wave G — configurable timeouts/caps, honest child outcomes, fenced timeline writes (#2937)
Four verified-open issues in the subagent-orchestration family, one PR:

- #1594: dream synthesize subagent job/wait timeouts promoted from
  hardcoded 30/35-min constants to config keys
  dream.synthesize.subagent_timeout_ms / subagent_wait_timeout_ms.
  Approach ported from stale PR #1596 (credit @ai920wisco).
- #2778: add_timeline_entry joins the subagent brain-tool allowlist,
  fenced fail-closed by the shared enforceSubagentSlugFence (extracted
  from put_page's inline check — same trusted-workspace allow-list /
  wiki/agents/<id>/ namespace policy). The per-turn output cap is now
  resolveMaxOutputTokens (data.max_tokens → agent.max_output_tokens →
  8192, was hardcoded 4096); a max_tokens stop surfaces as
  stop_reason 'max_tokens' instead of a silent end_turn, and a
  mid-tool-round cap hit injects a truncation note so the model
  re-issues the dropped call.
- #2782: patterns phase status now reflects the child outcome —
  non-complete outcome with zero writes → fail (PATTERNS_CHILD_<OUTCOME>),
  partial writes → warn. Patterns timeouts get the same config-key pair
  (dream.patterns.subagent_timeout_ms / subagent_wait_timeout_ms).
- #2113: facts extraction cap is config facts.extraction_max_tokens
  (default 4000, was hardcoded 1500); stopReason 'length' is checked,
  retried once at 2x the cap, and surfaced on stderr instead of
  silently extracting zero facts.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ai920wisco <ai920wisco@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:25:48 -07:00
a0ef951586 fix(dream): gate patterns phase on gateway provider reachability, not ANTHROPIC_API_KEY (takeover of #2279) (#2936)
Absorbs PR #2279's intent (drop the hardcoded ANTHROPIC_API_KEY gate) with
the end-state the gateway world actually wants: the patterns phase now
probes the RESOLVED patterns model through probeChatModel(normalizeModelId)
— the same semantics as think/index.ts and synthesize's makeJudgeClient.

Fixes two misclassifications of the old env gate:
- Non-Anthropic stacks (litellm, deepseek, openrouter, ...) were skipped as
  "no upstream" even though the subagent routes them through the gateway
  (agent.use_gateway_loop). They now pass the gate; their auth is checked
  lazily at dispatch and surfaces in the job outcome.
- Anthropic keys set via `gbrain config set anthropic_api_key` (stdio MCP
  launches without shell env) were treated as missing. hasAnthropicKey
  inside probeChatModel reads both sources.

Skip reason renames no_api_key → no_provider (carrying the probe's detail).
Both pinning tests updated: the structural test asserts the probe wiring;
the PGLite E2E swaps its env-only helper for the shared hermetic
withoutAnthropicKey (env + config file) so it can't flip to a live LLM call
on a dev machine with a config-file key.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:56 -07:00
bd2ba46a61 fix(retry): reconnect on null instance pool in ALL non-batch config accessors (takeover of #1891) (#2934)
Ports the still-unmerged remnant of PR #1891 (#1593 follow-up). Master's
getConfig gained retry-with-reconnect in #1603, but its siblings —
setConfig, unsetConfig, listConfigKeys — still touched `this.sql` bare, so
the first config write/list after a mid-cycle instance-pool teardown threw
the retryable "No database connection" (issue #1678) unhandled instead of
rebuilding the pool.

Adds the connRetry helper from #1891 (same retry+reconnect posture as
batchRetry, but no batch audit JSONL — a config accessor is not a sized
batch), refactors getConfig onto it, and wraps the other three. Writes are
safe to retry: withRetry only retries connection-class failures and both
writes are idempotent (upsert / delete).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: jalagrange <jalagrange@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:52 -07:00
2df41a84c9 feat(ai-gateway): derive OpenAI prompt_cache_key for native-OpenAI chat models (takeover of #2442) (#2933)
Ports the still-unmerged half of PR #2442. OpenAI caches prompt prefixes
automatically, but a stable prompt_cache_key keeps requests that share a
prefix on the same inference engine, lifting the automatic-cache hit rate.
chat() now derives a stable key from the system prompt + sorted tool names
for native-openai models and passes it via providerOptions.openai.
promptCacheKey. Config provider_chat_options still overrides the derived
key; anthropic/google/openai-compatible providers are untouched.

The Anthropic half of #2442 (cache_control "silent no-op") is superseded:
@ai-sdk/anthropic 3.0.74 forwards call-level providerOptions.anthropic.
cacheControl as the Messages API's request-level cache_control (automatic
prefix caching), so master's existing marker is live on current deps.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: CoachRyanNguyen <CoachRyanNguyen@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:48 -07:00
da1bab532a fix(import): make checkpoints staging-first — canonical dir identity + self-describing metadata (#2935)
Port of #1731 (diazMelgarejo) onto current master. gbrain import wrote
~/.gbrain/import-checkpoint.json with the caller's raw dir argument, so
a checkpoint left behind by an interrupted run (e.g. SIGTERM) could carry
"." or a symlinked spelling — an identity that resolves to whatever CWD
the next consumer happens to run from. Downstream tooling that treated
the checkpoint dir as an owned staging boundary could then act on the
wrong directory.

- runImport captures the import target ONCE via resolveImportTargetDir
  (resolve + realpathSync) and threads that canonical value through
  collection, checkpoint load/save, and resume filtering
- checkpoints are self-describing (schema_version: 1, owner: "gbrain",
  kind: "import"); loadCheckpoint tolerates absent metadata (legacy
  path-based files) but rejects present-and-wrong metadata and any
  relative dir
- checkpoint contract documented in docs/guides/live-sync.md (llms
  bundle regenerated)
- test/import-resume.test.ts fixture now realpaths its tmpdir so planted
  checkpoints match the canonicalized dir (macOS /var -> /private/var)

Fixes #1728

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Lawrence Melgarejo <Lawrence@cyre.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:17:37 -07:00
3aeb622dc7 v0.42.62.0 chore(release): thirty verified fixes — changelog + version bump (#2924)
Source-provenance wave, reconnect resilience, SSE proxy fix, rolling
prompt-cache, security CI, three consolidated fix-waves, and twenty more
individually verified community fixes.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:48:18 -07:00
3253fb824c fix(deps): resolve all OSV-flagged dependency vulnerabilities (same-major bumps) (#2927)
* fix(deps): resolve all 34 OSV-flagged vulnerabilities with same-major patch bumps

Direct deps: js-yaml ^3.15.0, marked ^18.0.2 (resolves 18.0.6),
admin vite ^6.4.3. Transitive deps pinned via overrides at their
minimum fixed versions (same major, no promotion to direct):
@hono/node-server, fast-uri, fast-xml-builder, fast-xml-parser,
form-data, hono, ip-address, qs; admin: @babel/core, postcss.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(deps): force js-yaml >=3.15.0 for all resolutions via override

A transitive consumer held a second js-yaml@3.14.2 resolution the
direct-dep range bump did not move.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:38:05 -07:00
b075a9c8d4 test+ci: unbreak master — symlink-walker probe files + OSV caller permissions (#2926)
* test: symlink-walker tests use a non-metafile probe (README now skipped by design, #2315)

The import walker deliberately skips README/metafiles since #2315 (closing
#345); the symlink-hardening tests used README.md as their probe file and
went red on the intersection. Probe with notes.md instead; test intent
(cycle hardening + strategy filter) unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: grant security-events write to the OSV caller job — reusable workflow requires it at startup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:10:47 -07:00
Brett a31f16f471 fix(markdown): treat # lines inside closed frontmatter as YAML comments, not headings (#2153)
`parseMarkdown` previously walked the lines after the opening `---` and
recorded the first `^#{1,6}\s`-shaped line as a `headingBeforeClose`,
then flagged MISSING_CLOSE when that index came before the actual closing
fence. YAML allows `#` comment lines anywhere inside the document, so a
template that leads with annotation comments inside the fence (e.g. a
`# Research Template` header before the keys) hit a false-positive
MISSING_CLOSE even though the closing `---` was present.

Fix: only walk for the closing `---`. When it is found, content between
the fences is YAML; `#` lines are comments, not headings. When the close
is genuinely missing, surface the first heading-shaped line as a
where-it-went-off-the-rails hint (this path was already correct; we keep
it for the genuine missing-close case).

Two regression tests added to `test/markdown-validation.test.ts`:
- `#` comment lines at the top of a closed frontmatter
- `#` comment lines interleaved with keys

All 68 tests across the four markdown/frontmatter test files stay green.
2026-07-17 11:39:15 -07:00
7ffac65c62 fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522 #1747 #1503) (#2920)
* fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522 #1747 #1503)

Three fixes in the same invariant class (source identity silently dropped
on the write path, collapsing to the 'default' source):

- #1522: the ingest_capture Minion handler validated IngestionEvent
  provenance (source_id/source_kind/source_uri) then dropped it on the
  importFromContent call. Now threads source_kind/source_uri +
  ingested_via='ingest_capture' into the page write, and routes the write
  under event.source_id when it names a registered source AND the event
  is trusted (fail-closed: untrusted webhook payloads carry a
  caller-controlled x-gbrain-source-id header and must not choose their
  write source; unregistered emitter ids keep default routing so the
  webhook path can't FK-fail).

- #1747: the fs-walk extractors (extractLinksFromDir /
  extractTimelineFromDir / extractForSlugs) built batch rows with no
  source_id, so addLinksBatch/addTimelineEntriesBatch mapped missing →
  'default' and the pages JOIN dropped every row on a non-default source
  ("Links: created 0 from N pages", no error). ExtractOpts gains
  sourceId; the CLI fs path resolves it via resolveSourceId
  (--source-id > env > dotfile > registered path > sole-non-default)
  and rows are stamped from/to/origin_source_id + timeline source_id.

- #1503: the cycle's extract phase (runPhaseExtract) never passed a
  sourceId, so federated-brain dream/autopilot cycles persisted nothing
  every night. It now threads cycleSourceId (explicit --source or
  resolveSourceForDir(brainDir) — the same seam runPhaseSync uses) into
  runExtractCore for both the incremental and full-walk paths.

Regression tests: handler provenance write-through (registered/
unregistered/untrusted routing), fs-walk + CLI resolution + incremental
cycle route landing edges/timeline in the right source (all red on
unfixed master), and a negative control pinning the pre-fix JOIN-drop
shape.

Reuses the ExtractOpts.sourceId threading approach from PR #1719,
rebased onto the current signal-aware signatures and extended to the
cycle + timeline paths.

Fixes #1522
Fixes #1747
Fixes #1503

Co-authored-by: seungsu <kss530c@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: update cycle source pin for cycleSourceId threading

The #1972 source-pin test asserts the literal runPhaseExtract call site in
cycle.ts. The #1503 fix appended cycleSourceId after opts.signal; signal
threading is unchanged (still the 5th arg, forwarded to runExtractCore).
Pin updated to the new literal — invariant intact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: seungsu <kss530c@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:36:43 -07:00
maxpetrusenkoagent b263d9bc20 fix(minions): reconnect worker after promote connection loss (#2025)
Recover the worker-owned Postgres pool when promoteDelayed escapes a retryable connection error, preventing the repeated Promotion error: No database connection loop from issue #1491.\n\nAdds a regression test proving reconnect happens before the worker continues to claim work.
2026-07-17 11:33:04 -07:00
kubi 73bbbde01d fix frontmatter scans to respect git excludes (#2462) 2026-07-17 11:32:48 -07:00
ff8ce4d764 fix(import): walker skips SYNC_SKIP_FILES metafiles so import and sync agree (#2315)
Closes #345.

The bulk-import walker isCollectibleForWalker filtered admitted files by
extension only, while incremental sync excludes README/index/log/schema via
isSyncable -> SYNC_SKIP_FILES. A directory import therefore ingested every
directory README as a folder-titled ghost page that trigram-corrupts fuzzy
entity resolution and inflates orphan count. Apply SYNC_SKIP_FILES (basename
guard) at the top of isCollectibleForWalker so both the FS-walk and the
git-fast-path collection routes agree with sync.

Also add RESOLVER.md to SYNC_SKIP_FILES: a structural routing metafile
(docs-aligned with schema.md/index.md/log.md/README.md), not indexable content.

Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:32:42 -07:00
ad1fe25e61 fix(sources): audit walker inverted pruneDir — nested sources scanned 0 files (#2678)
The audit walk in `gbrain sources audit` used `if (pruneDir(entry, dir)) continue;`
but pruneDir() returns true = descend, false = prune (core/sync.ts). The
inversion made the walker skip every legitimate subdirectory (any source with
nested content reports 'Files scanned: 0 markdown files') while descending
into exactly the trees it should skip (node_modules/, .git/, vendor/, .raw/).

The walker's own comment says 'Mirror gbrain sync's descent rules' — this
makes it actually do so.

Repro on a real brain (v0.42.56/57): a comms source with per-contact
subdirectories audits 0 files; a flat source audits correctly.

Co-authored-by: Idrees Kamal <idreeskamal@MacBook-Air.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:32:37 -07:00
Masa c4ff8b63a8 fix(minions): restore rolling conversation prompt-cache on the direct SDK path (#2771)
* fix(minions): restore rolling conversation prompt-cache on the direct SDK path

Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.

Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.

Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).

Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.

Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.

* fix(minions): normalize seed message content before caching it

Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.

Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.

* fix(minions): retain the prior rolling cache breakpoint, not just the newest

Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.

Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
2026-07-17 11:32:31 -07:00
Ryan AyersandClaude Opus 4.8 9eac872136 fix(postgres-engine): build-then-swap reconnect() so a failed rebuild can't brick the engine (#1593) (#1906)
The instance-pool reconnect() did disconnect() (nulling _sql) BEFORE connect(),
so a connect() failure during a transient Postgres blip left _sql null for the
rest of the process — every subsequent non-retry-wrapped call then fell through
to the never-connected module singleton and threw 'No database connection',
crashing the autopilot worker into a respawn loop. Build-then-swap: snapshot the
live pool, build a fresh one, end the old only once the new validates, restore
on failure. Keeps upstream's reap-detection + pool-recovery audit. Confirmed by
jalagrange on closed PR #1593; this is the Layer-1 fix he left to us.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 11:32:23 -07:00
xd-Neji cfc120fcb3 fix(stats): exclude soft-deleted pages from visible counts (#2235) 2026-07-17 11:32:17 -07:00
Mersad Ajanovic 0a021f6f6b fix: send admin SSE cookies through reverse proxies (#1560) 2026-07-17 11:32:12 -07:00
mzkarami cd9bd3f731 fix(auth): add register-client agent binding flags (#1976) 2026-07-17 11:32:06 -07:00
Brett 00523b8412 feat(ai/recipes/litellm): declare chat + expansion touchpoints (#2208)
The litellm recipe shipped only an `embedding` touchpoint. `getProviderCapabilities()` in `src/core/ai/capabilities.ts` throws when `recipe.touchpoints.chat` is missing, `classifyCapabilities()` returns `'unknown'`, and `enforceSubagentCapable()` in `src/core/model-config.ts` silently falls back to `TIER_DEFAULTS.subagent` (anthropic). Any brain that routes paid traffic through a litellm-style proxy AND has no `ANTHROPIC_API_KEY` then sees every subagent loop dispatch throw `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`. The user's explicit `models.tier.subagent = litellm:*` choice is overridden without their knowledge.

Declare `chat` and `expansion` touchpoints mirroring the openai recipe's shape: `models: []` (litellm proxies arbitrary backends; allowlist is intentionally empty, since `assertTouchpoint` already skips allowlist checks for `tier: 'openai-compat'`), `supports_tools: true`, `supports_subagent_loop: true`, `supports_prompt_cache: false` (OpenAI-compat backends don't honor Anthropic-style `cache_control`), `max_context_tokens: 200_000` (conservative GPT-5-family default; per-deployment override needed for smaller-context backends), costs `undefined` (varies by proxied provider).

Reproduction (deterministic):

1. Fresh brain with no `ANTHROPIC_API_KEY` in env.
2. `gbrain config set models.tier.subagent litellm:gpt-5.4` (or any `litellm:*` string).
3. `gbrain models` warns and falls back to `anthropic:claude-sonnet-4-6`.
4. Submit any subagent job; throws `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`.

After the patch, `classifyCapabilities('litellm:gpt-5.4', recipe)` returns `degraded:no_caching` (chat-capable, no Anthropic-style prompt cache). `enforceSubagentCapable` no longer steals the model choice. The subagent loop emits a one-time `degraded:no_caching` warn about prompt-cache absence; cost scales linearly with conversation length, accepted trade for the proxy path.
2026-07-17 11:32:01 -07:00
lost9999andlost9999 3a2033e8e2 v0.42.52.0 fix(sync): bump last_sync_at heartbeat on 0-changes sync (#2335)
D4 invariant ("never advance last_commit on partial", sync.ts comment)
preserved. last_sync_at is a monitoring signal read by doctor
sync_freshness (warn 24h / fail 72h), separate from the import-converged
bookmark. Without this heartbeat write, a cron-driven */15 sync over a
quiet vault pins last_sync_at to the last real commit, so doctor
falsely flags the source as stale for as long as the vault is quiet.

Reproduction (5 lines, no gbrain install required):
  1. Setup a fresh obsidian source + commit a single .md file
  2. gbrain sync --source obsidian   # first_sync, last_sync_at = NOW
  3. (do nothing) gbrain sync --source obsidian   # up_to_date
  4. SELECT last_sync_at FROM sources WHERE id = 'obsidian';
  5. Observed: still pinned to step 2. Expected: bumped to step 3.

Fix: in the up_to_date early-return (sync.ts line ~1786), execute a
single `UPDATE sources SET last_sync_at = now() WHERE id = $1` before
returning. The D4-protected writeSyncAnchor path is untouched.

Test: test/sync.test.ts adds a describe block that runs two
consecutive syncs against a quiet vault and asserts last_sync_at
advances while last_commit is unchanged. PGLite + executeRaw pattern
matches the existing #1970 test scaffold.

Workaround: hourly psql touch in WSL crontab documented at
https://github.com/garrytan/gbrain/issues/[link-to-issue]

Co-authored-by: lost9999 <lost9999@users.noreply.github.com>
2026-07-17 11:31:55 -07:00
supportswiftandjaxlewis-swift 74bc8f8cd1 fix(sync): crash-safe renames loop — record per-file failures instead of throwing (#2402)
The renames loop reimports each renamed file via importFile() but, unlike the
deletes and adds/mods loops, does not wrap the call. importFile() still throws
on content sanity-block, duplicate-slug, and missing-link endpoints, so a single
malformed renamed file throws uncaught and crashes the whole sync mid-run —
freezing the checkpoint and defeating --skip-failed. A 'skipped' result carrying
an error was also silently dropped (never recorded to failedFiles).

This wraps the reimport in try/catch and records both the throw and the
skipped-with-error case to failedFiles, matching the existing deletes/adds loop
pattern. Surfaced in the wild by a tree-wide rename (a shared/ -> system/ vault
migration, ~10.8k renames) where one malformed-YAML renamed file crashed the
entire incremental sync at rename ~4900/10857.

Co-authored-by: jaxlewis-swift <jaxlewis@swiftsolutions.ai>
2026-07-17 11:31:49 -07:00
78bc2fef09 fix(cli,config,doctor): CLI/config UX wave — config-get file plane, idempotent archive, honest help + doctor text (#2120 #2792 #1175 #1123 #2451) (#2918)
* fix(cli,config,doctor): CLI/config UX wave — config-get file plane, idempotent archive, honest help + doctor text, prefixed model defaults (#2120 #2792 #1175 #1123 #2451)

- config get resolves the file/env plane before the DB plane (runtime
  precedence) and reports provenance on stderr; stdout stays a bare value.
- sources archive distinguishes already-archived (friendly no-op, exit 0)
  from not-found (clear exit-4 error).
- gbrain --help SOURCES block now lists archive/restore/archived/purge/status
  plus a pointer at `sources --help` for the long tail.
- multi_source_drift doctor advice references only real CLI surfaces
  (drops the never-built 'sources rehome'; pins delete to GBRAIN_SOURCE=default).
- #2451 (bare model ids in calibration defaults) verified already fixed +
  tested on master by #2892 — no change needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: satisfy test-isolation gate for wave-B tests

check-test-isolation R1 forbids direct process.env mutation in non-serial
unit tests (env leaks across files sharing a shard process). Route the
GBRAIN_HOME / GBRAIN_CHAT_MODEL / GBRAIN_PGLITE_SNAPSHOT overrides through
the canonical withEnv() helper instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:27:34 -07:00
9aaa3be05f ci(security): OSV dependency scan, release artifact attestations, Semgrep CE SAST (#2182 #2142 #2272) (#2917)
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:05:13 -07:00
245 changed files with 14894 additions and 1212 deletions
+33
View File
@@ -0,0 +1,33 @@
name: OSV-Scanner
# Dependency vulnerability scan (#2182) via Google's official reusable
# workflow. Runs weekly and on any PR that touches the dependency manifests.
# Tokenless: needs zero secrets. Findings are reported in the job log and as
# a SARIF artifact on the run; code-scanning upload is deliberately disabled
# so the workflow stays read-only (no security-events: write).
on:
pull_request:
branches: [master]
paths:
- 'bun.lock'
- 'package.json'
schedule:
- cron: '30 6 * * 1' # weekly, Monday 06:30 UTC
workflow_dispatch:
permissions:
contents: read
jobs:
osv-scan:
permissions:
actions: read
contents: read
# Required by the reusable workflow's own top-level permissions block —
# GitHub validates the caller grants a superset AT STARTUP, even with
# upload-sarif: false (nothing is actually uploaded; see #2117 upstream).
security-events: write
uses: google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2 # v2.3.8
with:
upload-sarif: false
+8
View File
@@ -19,6 +19,10 @@ jobs:
target: bun-linux-x64
artifact: gbrain-linux-x64
runs-on: ${{ matrix.os }}
permissions:
contents: read
id-token: write # for attest-build-provenance (Sigstore OIDC)
attestations: write # for attest-build-provenance
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -28,6 +32,10 @@ jobs:
- run: bun test
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Attest build provenance
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
subject-path: bin/${{ matrix.artifact }}
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: ${{ matrix.artifact }}
+36
View File
@@ -0,0 +1,36 @@
name: Semgrep
# Static analysis (SAST) with Semgrep Community Edition (#2272). Tokenless:
# uses the public registry rulesets, needs zero secrets. Findings print in
# the job log; no code-scanning/SARIF upload by design (keeps permissions
# read-only, no security-events: write).
on:
pull_request:
branches: [master]
schedule:
- cron: '30 7 * * 1' # weekly, Monday 07:30 UTC
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
semgrep:
runs-on: ubuntu-latest
timeout-minutes: 20
container:
image: semgrep/semgrep:1.170.0@sha256:c98f8829eea377274ee4b10656458b078b88232469b2ff913f091c2317347c9d
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
# Non-blocking initially (continue-on-error): the first runs establish a
# baseline without failing unrelated PRs. Graduation path: once the
# baseline findings are triaged (fixed or `# nosemgrep`'d), remove
# continue-on-error so new findings block PRs.
- name: Semgrep scan (report-only)
run: semgrep scan --config p/default --config p/typescript --error
continue-on-error: true
+5 -1
View File
@@ -206,7 +206,11 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
# 22, not 15: under parallel PR load the PGLite WASM cold-starts stretch a
# shard past 15 min while every test is still passing — the timeout then
# cancels the job and the test-status gate reads it as a failure. 13 runs
# died this way on 2026-07-21/22 alone.
timeout-minutes: 22
strategy:
fail-fast: false
matrix:
+82
View File
@@ -2,6 +2,88 @@
All notable changes to GBrain will be documented in this file.
## [0.42.64.0] - 2026-07-20
### Fixed
- Confidential OAuth clients can now revoke access tokens through the standard revocation endpoint when client secrets are stored as hashes. Invalid credentials fail closed, malformed or mixed authentication is rejected, backend failures remain retryable, and discovery metadata accurately advertises supported authentication methods.
No schema migrations.
## [0.42.63.0] - 2026-07-20
**Schema commands now open the local brain you actually configured.**
If your PGLite brain lives at a custom path, commands such as `gbrain schema stats` previously ignored that path and could inspect the default brain instead. That made a healthy configured brain look empty or report the wrong schema counts. Schema commands now use the same complete database configuration as the rest of GBrain. PostgreSQL behavior is unchanged, and no migration is required.
### How to use it
Upgrade, then run the schema command normally:
```bash
gbrain upgrade
gbrain schema stats --json
```
The reported page and type counts now come from the `database_path` in `~/.gbrain/config.json` when the engine is PGLite.
### Itemized changes
#### Fixed
- **Schema CLI commands preserve configured PGLite paths.** Engine construction and connection now receive the canonical complete engine configuration, including both `database_path` and `database_url` where applicable.
- **CLI tests are isolated from ambient database URLs.** Schema subprocess tests explicitly clear inherited PostgreSQL URL variables, and a persistent-PGLite regression test proves `schema stats` reads the configured database rather than the default brain.
## [0.42.62.0] - 2026-07-17
**If your brain holds more than one source, everything now lands in the right one. Link extraction, timeline extraction, background cycles, and webhook captures used to quietly file some of their output under the default source; all of those paths now carry the correct source identity. Background agent jobs got tougher too: a failed database reconnect can no longer wedge the engine, and workers recover from dropped connections instead of crash-looping. If you run the admin dashboard behind a reverse proxy, the live activity panel finally connects. Long agent conversations cost less because repeated context is reused between turns on Anthropic calls. Local LiteLLM proxies work out of the box. Nested sources scan correctly again instead of reporting zero files. And the project's automated checks now include dependency vulnerability scanning, static code-security analysis, and signed provenance for release builds. Thirty merged changes in all, the largest batch to date, each one reviewed and verified against the live codebase before landing.**
## To take advantage of v0.42.62.0
`gbrain upgrade`. No new schema migrations.
1. **Multi-source brains:** run `gbrain extract all` once (or let the next cycle do it) so previously mis-scoped link and timeline rows are regenerated under the right source.
2. **If you serve the admin dashboard behind a reverse proxy,** hard-refresh it once after upgrading; Live Activity should connect.
3. **Verify:**
```bash
gbrain doctor
gbrain stats
```
4. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
### Itemized changes
#### Fixed
- **Source identity threaded through write paths.** Filesystem link/timeline extraction (`src/commands/extract.ts`), the cycle extract phase, and ingest capture now stamp the resolved source id instead of defaulting to `default`, with fail-closed validation on externally supplied ids. (#1522, #1747, #1503 via #2920; absorbs #1719, contributed by @seungsu)
- **`reconnect()` is build-then-swap.** The new pool is validated before replacing the old one, so a failed rebuild restores the previous connection instead of leaving `_sql` null. (#1593 follow-up via #1906, contributed by @rayers)
- **Minion worker reconnects after promote-time connection loss** instead of crash-looping. (#1491 class via #2025, contributed by @maxpetrusenkoagent)
- **Admin Live Activity works behind reverse proxies.** The EventSource now sends credentials so strict-cookie sessions survive the proxy hop. (#912 via #1560, contributed by @flamerged)
- **Stats exclude soft-deleted pages** from visible counts on both engines; destructive-removal counts stay all-inclusive. (#2235, contributed by @xd-Neji)
- **LiteLLM recipes declare chat and expansion touchpoints,** so the subagent loop no longer swaps to Anthropic and fails without an Anthropic key. (#2207 via #2208, contributed by @brettdavies)
- **Rolling prompt-cache on the direct SDK path.** Growing conversations place rolling cache breakpoints (two, within the four-marker budget), cutting repeat-token cost on multi-turn Anthropic tool loops. (#2740 via #2771, contributed by @Masashi-Ono0611)
- **Nested sources scan again.** `sources audit` had one inverted prune check (descending into node_modules while reporting 0 files). (#2678, contributed by @ikamal97)
- **Import and sync agree on metafiles.** The import walker now skips the same structural metafiles sync skips. (#345 via #2315, contributed by @ElliotDrel)
- **Frontmatter scans respect git excludes** via a shared git-visible-files helper. (#2462, contributed by @kubi-dev)
- **Sync renames are crash-safe** (per-file failures recorded instead of aborting the run) and **zero-change syncs still bump `last_sync_at`** so freshness reporting stops lying. (#2402, contributed by @supportswift; #2335, contributed by @lost9999)
- **Facts survive one-shot CLI runs.** Facts-absorb work is enqueued as durable minion jobs instead of dying with the process exit drain; fence paths are source-scoped. (#2104, contributed by @reghar-bot)
- **Takes reads are source-scoped, `gbrain calibration` is reachable, outputs are BigInt-safe.** (#2035 and the takes slice of #2200 via #2892, takeover of #2452, contributed by @spinsirr)
- **CLI answers honestly.** `config get` reads both config planes with provenance, `sources archive` is idempotent, help text matches real subcommands, doctor recommendations name commands that exist. (#2120, #2792, #1175, #1123, #2451 via #2918)
- **PGLite init failures name plausible causes for your platform** instead of blaming a macOS-specific bug everywhere, and non-Error crashes print their message instead of `[object Object]`. (#2674 class via #2891)
- **YAML comments inside frontmatter parse.** `#` lines inside a closed fence are comments, not headings; no more false MISSING_CLOSE. (#2152 via #2153, contributed by @brettdavies)
- **Conversation facts read the raw transcript sidecar** and recognize plain `Speaker A:` lines. (#1897 via #1898, contributed by @ElliotDrel)
- **`get_timeline` exposes date-window filters** (#2604 via #2694, contributed by @RerankerGuo) and **`query` since/until filter on effective date,** not updated_at (#1520 via #1706, contributed by @mvanhorn).
- **Windows serve watchdog works** via a signal-0 liveness probe instead of a POSIX-only process listing. (#2049, contributed by @abyss-node)
- **Doctor probes route through the active engine** (no false pgvector/jsonb warnings on PGLite; #1513 via #1183, contributed by @duncanclaw) and **a disabled retrieval reflex reads as intentional** (#2459, contributed by @eloe).
- **Cross-platform installs.** The postinstall hook is a real bun script, not POSIX shell that failed on Windows. (#1486 via #1554, contributed by @Sanjays2402)
- **Agent-bound auth clients.** `auth register-client` gains the `--bound-*` flags the submit_agent gate requires. (#1945, #1971 via #1976, contributed by @mzkarami)
#### Added
- **Security automation in the project's checks:** scheduled OSV dependency scanning, Semgrep static analysis on every PR (non-blocking initially), and build-provenance attestations wired into the release workflow. (#2182, #2142, #2272 via #2917)
- **`provider_chat_options` config passthrough** to the gateway, e.g. disabling thinking mode per provider or model. (#2577 via #2857)
- **Docs:** macOS 26.x PGLite workaround and native Postgres setup guide. (#1671, contributed by @roysaurav)
#### Internal
- release.yml runs `verify` before building. (#2222 via #2243, contributed by @mzkarami)
- Regenerated llms bundle after the docs merge. (#2893)
## [0.42.61.0] - 2026-07-16
**If gbrain's background daemon dies hard, a restart now takes over right away instead of waiting minutes for a stale lock to expire. Re-processing the same content no longer piles up near-duplicate knowledge atoms. On large brains, the takes bootstrap finally works through the whole corpus instead of re-scanning the same newest pages every run. And `gbrain schema use` can now activate the schema packs gbrain actually ships — including the install default — instead of just one hardcoded name. Cost tracking also learns the newest Claude models, so spend on them is metered instead of invisible.**
+20 -2
View File
@@ -71,8 +71,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
+1 -1
View File
@@ -1 +1 @@
0.42.61.0
0.42.64.0
+52 -20
View File
@@ -13,48 +13,52 @@
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"typescript": "^5.8.3",
"vite": "^6.3.3",
"vite": "^6.4.3",
},
},
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10",
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.0", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.28.5", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw=="],
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
"@babel/compat-data": ["@babel/compat-data@7.29.0", "", {}, "sha512-T1NCJqT/j9+cn8fvkt7jtwbLBfLC/1y1c7NtCeXFRgzGTsafi68MRv8yzkYSapBnFA6L3U2VSc02ciDzoAJhJg=="],
"@babel/compat-data": ["@babel/compat-data@7.29.7", "", {}, "sha512-locTkQyKvwIEgBzVrn8693ebc97F2U8ZHjbXwDXJ5Fn2TCpNwTlKcaKLkdHop5c/icOFE7qt7Q9JC5hnKNa6Gg=="],
"@babel/core": ["@babel/core@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-compilation-targets": "^7.28.6", "@babel/helper-module-transforms": "^7.28.6", "@babel/helpers": "^7.28.6", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/traverse": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA=="],
"@babel/core": ["@babel/core@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.7", "@babel/helper-compilation-targets": "^7.29.7", "@babel/helper-module-transforms": "^7.29.7", "@babel/helpers": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/template": "^7.29.7", "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-RgHBCvtjbOK2gXSNBNIkNoEc9qoVEtau3hj8gEqKQuL3HZAibKarWFEI3Lfm6EYKkLalOh8eSrj9b+ch9H/VBA=="],
"@babel/generator": ["@babel/generator@7.29.1", "", { "dependencies": { "@babel/parser": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw=="],
"@babel/generator": ["@babel/generator@7.29.7", "", { "dependencies": { "@babel/parser": "^7.29.7", "@babel/types": "^7.29.7", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ=="],
"@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.28.6", "", { "dependencies": { "@babel/compat-data": "^7.28.6", "@babel/helper-validator-option": "^7.27.1", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-JYtls3hqi15fcx5GaSNL7SCTJ2MNmjrkHXg4FSpOA/grxK8KwyZ5bubHsCq8FXCkua6xhuaaBit+3b7+VZRfcA=="],
"@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.29.7", "", { "dependencies": { "@babel/compat-data": "^7.29.7", "@babel/helper-validator-option": "^7.29.7", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-wem6WaBj4NaVYVdNhLPPVacES6ZJ+KBBfSkTMD3YZxbP3rm3Di85tJU5ljaUNhaOynt+Aj0xruhYuzQBt8n71g=="],
"@babel/helper-globals": ["@babel/helper-globals@7.28.0", "", {}, "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw=="],
"@babel/helper-globals": ["@babel/helper-globals@7.29.7", "", {}, "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA=="],
"@babel/helper-module-imports": ["@babel/helper-module-imports@7.28.6", "", { "dependencies": { "@babel/traverse": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw=="],
"@babel/helper-module-imports": ["@babel/helper-module-imports@7.29.7", "", { "dependencies": { "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g=="],
"@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.28.6", "", { "dependencies": { "@babel/helper-module-imports": "^7.28.6", "@babel/helper-validator-identifier": "^7.28.5", "@babel/traverse": "^7.28.6" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA=="],
"@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.29.7", "", { "dependencies": { "@babel/helper-module-imports": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7", "@babel/traverse": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg=="],
"@babel/helper-plugin-utils": ["@babel/helper-plugin-utils@7.28.6", "", {}, "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug=="],
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.29.7", "", {}, "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.29.7", "", {}, "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg=="],
"@babel/helper-validator-option": ["@babel/helper-validator-option@7.27.1", "", {}, "sha512-YvjJow9FxbhFFKDSuFnVCe2WxXk1zWc22fFePVNEaWJEu8IrZVlda6N0uHwzZrUM1il7NC9Mlp4MaJYbYd9JSg=="],
"@babel/helper-validator-option": ["@babel/helper-validator-option@7.29.7", "", {}, "sha512-N9ZErrD+yW5geCDtBqnOoxmR8+tNKiGuxKlDpuJxfsqpa2dFcexaziGAE/qoHLiDDreVNMupxGmSoNlyvsA3gw=="],
"@babel/helpers": ["@babel/helpers@7.29.2", "", { "dependencies": { "@babel/template": "^7.28.6", "@babel/types": "^7.29.0" } }, "sha512-HoGuUs4sCZNezVEKdVcwqmZN8GoHirLUcLaYVNBK2J0DadGtdcqgr3BCbvH8+XUo4NGjNl3VOtSjEKNzqfFgKw=="],
"@babel/helpers": ["@babel/helpers@7.29.7", "", { "dependencies": { "@babel/template": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-1k2lAGRMfHTcwuNYcCNUmaUffmQv8KWMfh2iJUUeRlwlwH4FdNG7mfPI10NPfLHJFThE4Tyr4mv7kTNZOiPuBg=="],
"@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@babel/parser": ["@babel/parser@7.29.7", "", { "dependencies": { "@babel/types": "^7.29.7" }, "bin": "./bin/babel-parser.js" }, "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg=="],
"@babel/plugin-transform-react-jsx-self": ["@babel/plugin-transform-react-jsx-self@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-6UzkCs+ejGdZ5mFFC/OCUrv028ab2fp1znZmCZjAOBKiBK2jXD1O+BPSfX8X2qjJ75fZBMSnQn3Rq2mrBJK2mw=="],
"@babel/plugin-transform-react-jsx-source": ["@babel/plugin-transform-react-jsx-source@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-zbwoTsBruTeKB9hSq73ha66iFeJHuaFkUbwvqElnygoNbj/jHRsSeokowZFN3CZ64IvEqcmmkVe89OPXc7ldAw=="],
"@babel/template": ["@babel/template@7.28.6", "", { "dependencies": { "@babel/code-frame": "^7.28.6", "@babel/parser": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ=="],
"@babel/template": ["@babel/template@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg=="],
"@babel/traverse": ["@babel/traverse@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-globals": "^7.28.0", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/types": "^7.29.0", "debug": "^4.3.1" } }, "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA=="],
"@babel/traverse": ["@babel/traverse@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.7", "@babel/helper-globals": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/template": "^7.29.7", "@babel/types": "^7.29.7", "debug": "^4.3.1" } }, "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw=="],
"@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@babel/types": ["@babel/types@7.29.7", "", { "dependencies": { "@babel/helper-string-parser": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7" } }, "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA=="],
"@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.25.12", "", { "os": "aix", "cpu": "ppc64" }, "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA=="],
@@ -220,7 +224,7 @@
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nanoid": ["nanoid@3.3.11", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="],
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
@@ -228,7 +232,7 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.9", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-7a70Nsot+EMX9fFU3064K/kdHWZqGVY+BADLyXc8Dfv+mTLLVl6JzJpPaCZ2kQL9gIJvKXSLMHhqdRRjwQeFtw=="],
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
@@ -250,8 +254,36 @@
"update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "^3.2.0", "picocolors": "^1.1.1" }, "peerDependencies": { "browserslist": ">= 4.21.0" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="],
"vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="],
"vite": ["vite@6.4.3", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-NTKlcQjlAK7MlQoyb6LgaqHc8sso/pVyUJYWMws3jg21uTJw/LddqIFPcPqP6PzpgbIcZyKI85sFE4HBrQDA8A=="],
"yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
"@types/babel__core/@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@types/babel__core/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__generator/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__template/@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@types/babel__template/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__traverse/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__core/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__core/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__generator/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__generator/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__template/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__template/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__traverse/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__traverse/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
}
}
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -7,7 +7,7 @@
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<script type="module" crossorigin src="/admin/assets/index-DqP-zmqH.js"></script>
<script type="module" crossorigin src="/admin/assets/index-CoGEje3-.js"></script>
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
</head>
<body>
+5 -1
View File
@@ -15,7 +15,11 @@
"@types/react": "^19.1.2",
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"vite": "^6.3.3",
"vite": "^6.4.3",
"typescript": "^5.8.3"
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10"
}
}
+1 -1
View File
@@ -21,7 +21,7 @@ export function DashboardPage() {
api.stats().then(setStats).catch(() => {});
api.health().then(setHealth).catch(() => {});
const es = new EventSource('/admin/events');
const es = new EventSource('/admin/events', { withCredentials: true });
eventSourceRef.current = es;
es.onopen = () => setSseStatus('connected');
es.onmessage = (e) => {
+40 -17
View File
@@ -26,8 +26,8 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"js-yaml": "^3.15.0",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
@@ -50,6 +50,17 @@
"trustedDependencies": [
"@electric-sql/pglite",
],
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"js-yaml": "^3.15.0",
"qs": "^6.15.2",
},
"packages": {
"@ai-sdk/anthropic": ["@ai-sdk/anthropic@3.0.74", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-Xew9rfz9WWhDSyF8rNhjT/XWOWelNfJrMlmG0Ahw210hStisRpQZ1s+7VeI9JTJOZ5y5tXqBi5kfPwYnCfyRTA=="],
@@ -151,7 +162,7 @@
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
"@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="],
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
@@ -159,6 +170,8 @@
"@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="],
"@nodable/entities": ["@nodable/entities@3.0.0", "", {}, "sha512-8L9xFeTYKhm49xfIypoe2W5wV1m/3Z58kT+7kR9A8OyFxcPduI4VmxaUMQyKYrRjUoLLSXv6EKKID5Tvj9cUVw=="],
"@opentelemetry/api": ["@opentelemetry/api@1.9.0", "", {}, "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg=="],
"@smithy/chunked-blob-reader": ["@smithy/chunked-blob-reader@5.2.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-St+kVicSyayWQca+I1rGitaOEH6uKgE8IUWoYnnEX26SWdWQcL6LvMSD19Lg+vYHKdT9B2Zuu7rd3i6Wnyb/iw=="],
@@ -307,6 +320,8 @@
"ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],
"anynum": ["anynum@1.0.1", "", {}, "sha512-N6//FLET/tXYNM/F6ABca1oH6fWB+KlTt909Le28WMDBk8oaT4vY17DCrwg2MvmuqUKt3Ni4N5dGJ/EoBgcO6A=="],
"argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="],
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
@@ -385,15 +400,15 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
"fast-uri": ["fast-uri@3.1.3", "", {}, "sha512-i70LwGWUduXqzicKXWshooq+sWL1K3WUU5rKZNG/0i3a1OSoX3HqhH5WbWwTmqWfor4urUakGPiRQcleRZTwOg=="],
"fast-xml-builder": ["fast-xml-builder@1.1.4", "", { "dependencies": { "path-expression-matcher": "^1.1.3" } }, "sha512-f2jhpN4Eccy0/Uz9csxh3Nu6q4ErKxf0XIsasomfOihuSUa3/xw6w8dnOtCDgEItQFJG8KyXPzQXzcODDrrbOg=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
"fast-xml-parser": ["fast-xml-parser@5.5.8", "", { "dependencies": { "fast-xml-builder": "^1.1.4", "path-expression-matcher": "^1.2.0", "strnum": "^2.2.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-Z7Fh2nVQSb2d+poDViM063ix2ZGt9jmY1nWhPfHBOK2Hgnb/OW3P4Et3P/81SEej0J7QbWtJqxO05h8QYfK7LQ=="],
"fast-xml-parser": ["fast-xml-parser@5.10.1", "", { "dependencies": { "@nodable/entities": "^3.0.0", "fast-xml-builder": "^1.2.0", "is-unsafe": "^2.0.0", "path-expression-matcher": "^1.6.2", "strnum": "^2.4.1", "xml-naming": "^0.3.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-IEMIf7298kXuZSRFoGfMYrl7is8LpavODgbNz1cwIudv7KwVFnuU+UsMporfq6PD6aXSlawZlARiA3UywCTfMw=="],
"finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "^4.4.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "on-finished": "^2.4.1", "parseurl": "^1.3.3", "statuses": "^2.0.1" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="],
"form-data": ["form-data@4.0.5", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.2", "mime-types": "^2.1.12" } }, "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w=="],
"form-data": ["form-data@4.0.6", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.4", "mime-types": "^2.1.35" } }, "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ=="],
"form-data-encoder": ["form-data-encoder@1.7.2", "", {}, "sha512-qfqtYan3rxrnCk1VYaA4H+Ms9xdpPqvLZa6xmMgFvhO32x7/3J/ExcTd6qpxM0vH2GdMI+poehyBZvqfMTto8A=="],
@@ -417,11 +432,11 @@
"has-tostringtag": ["has-tostringtag@1.0.2", "", { "dependencies": { "has-symbols": "^1.0.3" } }, "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw=="],
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"hasown": ["hasown@2.0.4", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A=="],
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.12.10", "", {}, "sha512-mx/p18PLy5og9ufies2GOSUqep98Td9q4i/EF6X7yJgAiIopxqdfIO3jbqsi3jRgTgw88jMDEzVKi+V2EF+27w=="],
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -431,7 +446,7 @@
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="],
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
@@ -439,11 +454,13 @@
"is-promise": ["is-promise@4.0.0", "", {}, "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ=="],
"is-unsafe": ["is-unsafe@2.0.0", "", {}, "sha512-2LdV822R+wmI86unXA93WCFpL6g+av8ynWk0nrHyJqGop5VoocYsSLFgN8jrfalT6iGeLNM4KXuVSsULP53kEA=="],
"isexe": ["isexe@2.0.0", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="],
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
"js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
@@ -455,7 +472,7 @@
"libheif-js": ["libheif-js@1.19.8", "", {}, "sha512-vQJWusIxO7wavpON1dusciL8Go9jsIQ+EUrckauFYAiSTjcmLAsuJh3SszLpvkwPci3JcL41ek2n+LUZGFpPIQ=="],
"marked": ["marked@18.0.0", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-2e7Qiv/HJSXj8rDEpgTvGKsP8yYtI9xXHKDnrftrmnrJPaFNM7VRb2YCzWaX4BP1iCJ/XPduzDJZMFoqTCcIMA=="],
"marked": ["marked@18.0.6", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-MrV5puXBfuiy6wl6DLaq3BtIJQAJToAd5zt/ZKhRfGRAuFPALE7/4Y7jnxRQoEgK/pBgurGqLyAuRgZ2xOjr6w=="],
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
@@ -487,7 +504,7 @@
"parseurl": ["parseurl@1.3.3", "", {}, "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="],
"path-expression-matcher": ["path-expression-matcher@1.5.0", "", {}, "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ=="],
"path-expression-matcher": ["path-expression-matcher@1.6.2", "", {}, "sha512-enSlaiat05iasnzmgNxRj8reFdj3puY2QpNgP1aPIaVfT6nn9ICuPoFlKHk8EN22HcwewshO+mN2DGbkCEOtqQ=="],
"path-key": ["path-key@3.1.1", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="],
@@ -503,7 +520,7 @@
"pure-rand": ["pure-rand@8.4.0", "", {}, "sha512-IoM8YF/jY0hiugFo/wOWqfmarlE6J0wc6fDK1PhftMk7MGhVZl88sZimmqBBFomLOCSmcCCpsfj7wXASCpvK9A=="],
"qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="],
"qs": ["qs@6.15.3", "", { "dependencies": { "es-define-property": "^1.0.1", "side-channel": "^1.1.1" } }, "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A=="],
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],
@@ -529,9 +546,9 @@
"shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="],
"side-channel": ["side-channel@1.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3", "side-channel-list": "^1.0.0", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw=="],
"side-channel": ["side-channel@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4", "side-channel-list": "^1.0.1", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ=="],
"side-channel-list": ["side-channel-list@1.0.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3" } }, "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA=="],
"side-channel-list": ["side-channel-list@1.0.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4" } }, "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w=="],
"side-channel-map": ["side-channel-map@1.0.1", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3" } }, "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA=="],
@@ -543,7 +560,7 @@
"strip-bom-string": ["strip-bom-string@1.0.0", "", {}, "sha512-uCC2VHvQRYu+lMh4My/sFNmF2klFymLX1wHJeXnbEJERpV/ZsVuonzerjfrGpIGF7LBVa1O7i9kjiWvJiFck8g=="],
"strnum": ["strnum@2.2.3", "", {}, "sha512-oKx6RUCuHfT3oyVjtnrmn19H1SiCqgJSg+54XqURKp5aCMbrXrhLjRN9TjuwMjiYstZ0MzDrHqkGZ5dFTKd+zg=="],
"strnum": ["strnum@2.4.1", "", { "dependencies": { "anynum": "^1.0.1" } }, "sha512-M9eUSMT2dCB2cTNPG7UYj6KuK7RJR2SN2+yCV/fTW3xzTCS6EaGZ5pSMgDIjB7r8zSfTGk+dvvn9rTjpVS9Mwg=="],
"toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="],
@@ -577,6 +594,8 @@
"wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="],
"xml-naming": ["xml-naming@0.3.0", "", {}, "sha512-ghig2TBE/H11aOVgmahA3MhimvkBr6JIYknH/Dhdk10nXwdbIqBJsbfMxpvFPG8bAw77gN29aQWvKpmVoPlvPQ=="],
"zod": ["zod@4.3.6", "", {}, "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg=="],
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
@@ -595,12 +614,16 @@
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"es-set-tostringtag/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"eventsource/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
"express/cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"get-intrinsic/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
+6 -1
View File
@@ -13,4 +13,9 @@ timeout = 60_000
# fixtures still match the schema. v0.37's production default is ZE/1280;
# tests that want the new default call configureGateway() explicitly in
# their own beforeAll.
preload = ["./test/helpers/legacy-embedding-preload.ts"]
#
# #2823: redirect GBRAIN_AUDIT_DIR to a per-run scratch dir BEFORE any test
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
# test/helpers/audit-dir-preload.ts for the full rationale.
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
+45
View File
@@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
Defense-in-depth layer for Postgres deployments that want the database itself
to enforce source isolation, in addition to the mandatory app-layer filters
(`sourceScopeOpts` — layer 1, always on).
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
source-scoped read methods wrap their queries in a transaction that first runs
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
bound params). An RLS policy can then filter rows by
`current_setting('app.scopes', true)`.
**Default off.** With the env var unset, reads call through on the shared pool
exactly as before — no per-read transaction, no pool-slot hold (the search
methods keep the transaction they always had for their `SET LOCAL
statement_timeout`). Existing operators see zero behavior change.
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
```sql
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
CREATE POLICY pages_scope_filter ON pages
USING (current_setting('app.scopes', true) = '*'
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
-- Required: connections that don't run through the scoped read helper
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
-- see zero rows once the policy exists:
ALTER ROLE <runtime-role> SET app.scopes = '*';
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
```
Safe to enable in either order: the env var without a policy is a no-op
setting; a policy without the env var is enforced only via the role default.
**Honest caveat:** only read paths routed through the scoped helper carry a
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine (v0.7, ships)
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
+2
View File
@@ -3,6 +3,8 @@
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
only.
`test/e2e/serve-http-oauth.test.ts` additionally pins confidential POST/Basic revocation, public-client SDK fallthrough, malformed/mixed authentication rejection, cross-client isolation, unknown-token opacity, metadata auth methods, no-store responses, strict post-revoke `401`, and retryable backend `503` semantics.
### Test command tiers
Seven test command tiers, each with a clear scope:
File diff suppressed because one or more lines are too long
+10
View File
@@ -131,6 +131,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
history rewrite still hard-blocks even with `--skip-failed`. Run
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
5. **Import checkpoints name the import target, not the caller's CWD.**
Interrupted `gbrain import <dir>` runs may leave
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
checkpoint `dir` is the absolute, resolved import target captured when
import starts. It is not a cleanup instruction and it must not be
re-derived from the process working directory. Checkpoints written by
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
`kind: "import"` so downstream tools can validate the contract before
deciding whether to resume.
## How to Verify
1. **Edit a file and search for the change.** Edit a brain markdown file,
+97
View File
@@ -0,0 +1,97 @@
# Multi-language full-text search
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
environment variable. Default: `english`.
## How it works
Postgres text-search configurations control stemming and stop-word removal.
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
both sides of the search:
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
(Postgres and PGLite).
- **Write side** — the `update_page_search_vector` and
`update_chunk_search_vector` trigger functions that populate
`pages.search_vector` and `content_chunks.search_vector`.
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
interpolated into SQL (tsvector functions don't accept parameterized config
names). Invalid values fall back to `english` with a warning.
## Built-in languages
Set the env var to any configuration your Postgres instance ships:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german
```
List what's available:
```sql
SELECT cfgname FROM pg_ts_config;
```
PGLite (the embedded default engine) ships the same built-in snowball
configurations as stock Postgres.
## First install vs. changing language later
On first install (or upgrade), the `configurable_fts_language` schema
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
that language. After the migration has run, changing the env var alone does
NOT retokenize existing rows — the migration shows as applied and is skipped.
Use the explicit command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview: language + row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command recreates both trigger functions under the new language and
backfills every existing `pages` and `content_chunks` row in batches,
streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
the `unaccent` extension, then stems with the portuguese snowball dictionary:
```sql
CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
ALTER TEXT SEARCH CONFIGURATION pt_br
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, portuguese_stem;
```
Then point GBrain at it:
```bash
export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes
```
Note: custom configurations require a real Postgres instance (e.g. the
Supabase engine). The config must exist BEFORE the migration or the reindex
command runs, or Postgres will reject the trigger recreation with
`text search configuration "pt_br" does not exist`.
## Caveats
- One language per brain: the setting is global to the database, not
per-source. Mixed-language brains should pick the dominant language (the
vector-search arm is language-agnostic and covers the rest).
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
the env var tokenizes new rows in `english` until the next reindex.
+45 -1
View File
@@ -114,8 +114,11 @@ Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
Full subcommand reference:
```
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
--path must be a git repo (or a subdirectory of one) — see
"The git requirement for --path sources" below. --force
skips that check to register before git-init exists.
gbrain sources list [--json] List all sources with page counts + federation state.
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
Cascade-delete a source (pages, chunks, timeline).
@@ -128,6 +131,47 @@ gbrain sources federate <id>
gbrain sources unfederate <id>
```
## The git requirement for --path sources
Every `--path` source must be a git repository (or live inside one — a
subdirectory of a git repo works too) with at least one committed, tracked
file under that path. `gbrain sources add` validates this at registration
time and refuses a directory that doesn't qualify — no `.git` at all, a
`git init` with no commit yet, or a commit made before `git add` — with an
actionable error instead of silently registering a source that will fail
(or worse, "succeed" while importing nothing) on its first `gbrain sync`.
Fix it with:
```bash
git -C <path> init
git -C <path> add -A
git -C <path> commit -m "initial import"
gbrain sources add <id> --path <path>
```
Two details that are easy to miss:
- **Files must actually be committed, not just present.** The sync walker
reads files through git objects, so `git init` alone — even followed by an
empty commit (`git commit --allow-empty`) — isn't enough. Registration
checks for real tracked content (`git ls-tree HEAD` scoped to the path),
not just a resolvable `HEAD`, so this footgun is caught immediately
instead of surfacing later as a sync that imports nothing.
- **`--force` registers the source anyway**, skipping the check. Use this if
you're registering a path before an automated pipeline gets around to
`git init`-ing it. GBrain never auto-`git init`s a `--path` source for
you — it's your directory, not a gbrain-managed clone (same consent
boundary as sync-time self-heal, which also never mutates a `--path`
source without an explicit ask).
**If sync ever reports a problem with the sync anchor** (`last_commit`) —
after a force-push, a history rewrite, or a from-scratch `git init` on a
directory that was synced before — you do not need to reset anything by
hand. `gbrain sync` detects an unreachable or non-ancestor anchor
automatically and recovers: either a full reimport (anchor object missing)
or a direct tree-to-tree diff against the orphaned bookmark (anchor present
but rewritten), advancing the anchor to the new HEAD when it completes.
## Citation format for agents
When agents receive multi-source results they MUST cite pages in
+3 -1
View File
@@ -131,7 +131,9 @@ into gbrain so other clients can scaffold it. Default behavior:
`~/.gbrain/harvest-private-patterns.txt` plus built-in defaults
(canonical private fork name, common email regex, Slack channel pattern). Any
match → rollback (delete the harvested files) and exit non-zero.
- `openclaw.plugin.json` updated with the new slug, sorted.
- `openclaw.plugin.json` updated with the new slug, sorted. Harvest must preserve
the top-level OpenClaw-native plugin fields (`id`, `configSchema`, `contracts`)
because OpenClaw validates those before it can install the package.
- `--no-lint` bypasses the linter (after a manual editorial scrub).
Use the `skillpack-harvest` skill (its companion editorial workflow)
@@ -233,13 +233,14 @@ keep it or `git checkout` to throw it away. Nothing is committed for you.
**For a skill that ships with gbrain** (anything under the gbrain repo's own
`skills/`): SkillOpt refuses to overwrite it by default and writes the winner to
`skills/<name>/skillopt/best.md` instead, so an optimization pass can never
silently mutate a skill other people depend on. Two ways to handle that:
`skills/<name>/skillopt/proposed.md` instead (while keeping `best.md` as the
optimizer's current-best pointer), so an optimization pass can never silently
mutate a skill other people depend on. Two ways to handle that:
```bash
# See the proposed improvement without touching SKILL.md (works for ANY skill):
gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
# → writes skills/meeting-prep/skillopt/proposed.md, updates best.md, and prints the proposal path.
# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+75 -2
View File
@@ -1565,8 +1565,8 @@ GBrain is designed to be installed and operated by an AI agent. The fastest path
If you don't already have an AI agent platform running, start with one of these. Both are designed to read GBrain's install protocol and execute it:
- **[OpenClaw](https://github.com/openclawagents/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/openclawagents/hermes)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
- **[OpenClaw](https://github.com/openclaw/openclaw)** — deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM)
- **[Hermes](https://github.com/NousResearch/hermes-agent)** — deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click)
Then paste this into your agent:
@@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
@@ -2095,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
Defense-in-depth layer for Postgres deployments that want the database itself
to enforce source isolation, in addition to the mandatory app-layer filters
(`sourceScopeOpts` — layer 1, always on).
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
source-scoped read methods wrap their queries in a transaction that first runs
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
bound params). An RLS policy can then filter rows by
`current_setting('app.scopes', true)`.
**Default off.** With the env var unset, reads call through on the shared pool
exactly as before — no per-read transaction, no pool-slot hold (the search
methods keep the transaction they always had for their `SET LOCAL
statement_timeout`). Existing operators see zero behavior change.
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
```sql
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
CREATE POLICY pages_scope_filter ON pages
USING (current_setting('app.scopes', true) = '*'
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
-- Required: connections that don't run through the scoped read helper
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
-- see zero rows once the policy exists:
ALTER ROLE <runtime-role> SET app.scopes = '*';
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
```
Safe to enable in either order: the env var without a policy is a no-op
setting; a policy without the env var is enforced only via the role default.
**Honest caveat:** only read paths routed through the scoped helper carry a
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine (v0.7, ships)
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
@@ -2767,6 +2830,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
history rewrite still hard-blocks even with `--skip-failed`. Run
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
5. **Import checkpoints name the import target, not the caller's CWD.**
Interrupted `gbrain import <dir>` runs may leave
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
checkpoint `dir` is the absolute, resolved import target captured when
import starts. It is not a cleanup instruction and it must not be
re-derived from the process working directory. Checkpoints written by
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
`kind: "import"` so downstream tools can validate the contract before
deciding whether to resume.
## How to Verify
1. **Edit a file and search for the change.** Edit a brain markdown file,
+1
View File
@@ -1,4 +1,5 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.32.3.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
+14 -3
View File
@@ -118,8 +118,8 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"js-yaml": "^3.15.0",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
@@ -144,5 +144,16 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.61.0"
"version": "0.42.64.0",
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"qs": "^6.15.2",
"js-yaml": "^3.15.0"
}
}
+2 -1
View File
@@ -266,4 +266,5 @@ editorial pass.
(e.g. `src/commands/<slug>.ts` if the host SKILL.md declares it
in frontmatter)
- gbrain's `openclaw.plugin.json` — adds the slug to `skills:`
array, sorted alphabetically
array, sorted alphabetically, without removing OpenClaw-native plugin fields
like `id`, `configSchema`, or `contracts`
+3 -1
View File
@@ -57,6 +57,8 @@ This mode guarantees:
- `skills/manifest.json` lists every skill directory
- `skills/RESOLVER.md` references every skill in the manifest
- `openclaw.plugin.json` `skills[]` round-trips with both
- `openclaw.plugin.json` keeps OpenClaw install-required native plugin fields
(`id`, object `configSchema`, and `contracts.contextEngines` when applicable)
- No MECE violations (duplicate triggers across skills)
### Phases
@@ -72,7 +74,7 @@ This mode guarantees:
### Automation
```bash
bun test test/skills-conformance.test.ts test/resolver.test.ts
bun test test/skills-conformance.test.ts test/resolver.test.ts test/openclaw-plugin-manifest.test.ts
```
The CI-gated check is the package.json `test` script.
+3 -3
View File
@@ -1,13 +1,13 @@
// AUTO-GENERATED — do not edit by hand.
// Run `bun run scripts/build-admin-embedded.ts` to regenerate.
// Source: admin/dist/ at 2026-05-24.
// Source: admin/dist/ at 2026-05-27.
//
// Bun resolves the file: imports to a path that works at runtime even
// inside a compiled binary (`bun build --compile`). The manifest maps
// the request path the express handler sees to (resolved-path, mime).
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
import A_0_assets_index_DqP_zmqH_js from '../admin/dist/assets/index-DqP-zmqH.js' with { type: 'file' };
import A_0_assets_index_CoGEje3__js from '../admin/dist/assets/index-CoGEje3-.js' with { type: 'file' };
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
import A_1_assets_index_GxkWX7v3_css from '../admin/dist/assets/index-GxkWX7v3.css' with { type: 'file' };
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
@@ -19,7 +19,7 @@ export interface AdminAsset {
}
export const ADMIN_ASSETS: Record<string, AdminAsset> = {
"/admin/assets/index-DqP-zmqH.js": { path: A_0_assets_index_DqP_zmqH_js as unknown as string, mime: "application/javascript; charset=utf-8" },
"/admin/assets/index-CoGEje3-.js": { path: A_0_assets_index_CoGEje3__js as unknown as string, mime: "application/javascript; charset=utf-8" },
"/admin/assets/index-GxkWX7v3.css": { path: A_1_assets_index_GxkWX7v3_css as unknown as string, mime: "text/css; charset=utf-8" },
"/admin/index.html": { path: A_2_index_html as unknown as string, mime: "text/html; charset=utf-8" },
};
+60 -3
View File
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -78,6 +78,8 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// maintain (#3015) prints its own usage block (modes + not-auto-applied list).
'maintain',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
@@ -998,6 +1000,13 @@ const THIN_CLIENT_REFUSED_COMMANDS = new Set([
// - `code-def`/`code-refs`/`code-callers`/`code-callees` have NO MCP ops
// in operations.ts:2630-2671; cannot be "fixed by routing" yet
'pages', 'files', 'eval', 'code-def', 'code-refs', 'code-callers', 'code-callees',
// scratch-DB audit: `config` get/set operate on the host brain's config
// plane (DB rows / host file-plane). On a thin client they fabricated an
// ephemeral local PGLite (full migration replay per call) and read/wrote
// config nobody would ever see. NOTE: `jobs` is deliberately NOT here —
// it gets a partial dispatch (list/get route over MCP engine-free, the
// rest refuse) in the main dispatch before connectEngine().
'config',
]);
/**
@@ -1035,6 +1044,9 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
'code-refs': '`code-refs` has no MCP op yet. Run on the host.',
'code-callers': '`code-callers` has no MCP op yet. Run on the host.',
'code-callees': '`code-callees` has no MCP op yet. Run on the host.',
// scratch-DB audit additions
config: "config reads/writes the host brain's config plane. Edit the host's .gbrain/config.json (file-plane keys) or run on the host with GBRAIN_HOME set.",
jobs: '`jobs list` and `jobs get <id>` are thin-client routable; this subcommand runs against the host queue. Use the submit_job / list_jobs / get_job MCP tools from your agent, or run on the host with GBRAIN_HOME set.',
};
/**
@@ -1593,6 +1605,27 @@ async function handleCliOnly(command: string, args: string[]) {
}
}
// Thin-client `jobs` dispatch: `list` and `get` route over MCP (v0.32
// routing branches in commands/jobs.ts) and never touch a local engine —
// but falling through to connectEngine() below fabricates an empty
// scratch PGLite in the thin-client GBRAIN_HOME and replays the entire
// migration chain on every invocation before the remote call even runs.
// Dispatch them engine-free here; every other jobs subcommand is
// host-queue-bound, so refuse with a pinpoint hint instead of building
// the scratch store.
if (command === 'jobs') {
const cfgJobs = loadConfig();
if (isThinClient(cfgJobs)) {
const jobsSub = args[0];
if (jobsSub === 'list' || jobsSub === 'get') {
const { runJobs } = await import('./commands/jobs.ts');
await runJobs(null, args);
return;
}
refuseThinClient('jobs', cfgJobs!.remote_mcp!.mcp_url);
}
}
// All remaining CLI-only commands need a DB connection
const engine = await connectEngine();
try {
@@ -1726,6 +1759,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runOrphans(engine, args);
break;
}
case 'maintain': {
const { runMaintain } = await import('./commands/maintain.ts');
await runMaintain(engine, args);
break;
}
// v0.32.7 CJK wave — post-upgrade markdown re-chunk sweep.
// v0.36 Phase 3 wave — `gbrain reindex --multimodal` re-embeds content_chunks
// into the unified Voyage multimodal-3 column.
@@ -2001,6 +2039,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runReindexCodeCli(engine, args);
break;
}
case 'reindex-search-vector': {
// Explicit recreate of FTS trigger functions + batched backfill,
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
// env var on a brain that already ran the configurable_fts_language
// migration.
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
await runReindexSearchVectorCli(engine, args);
break;
}
case 'reindex-frontmatter': {
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
// Mirror of reindex-code shape. Wraps the shared library function in
@@ -2249,7 +2296,7 @@ IMPORT/EXPORT
import <dir> [--no-embed] Import markdown directory
sync [--repo <path>] [flags] Git-to-brain incremental sync
sync --watch [--interval N] Continuous sync (loops until stopped)
sync --install-cron Install persistent sync daemon
See also: autopilot --install (continuous daemon).
export [--dir ./out/] Export to markdown
export --restore-only [--repo <p>] Restore missing supabase-only files
[--type T] [--slug-prefix S] With optional filters
@@ -2315,7 +2362,14 @@ BRAIN (capture / ideate / explore — v0.37/v0.38)
SOURCES (multi-repo / multi-brain)
sources list Show registered sources
sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki')
sources remove <id> Remove a source + its pages
sources remove <id> Remove a source + its pages (--confirm-destructive)
sources archive <id> Soft-delete: hide from search, recoverable for 72h
sources restore <id> Un-archive a soft-deleted source
sources archived List soft-deleted sources and their purge expiry
sources purge [<id>] Permanently delete archived sources
sources status Per-source dashboard (sync lag, embed coverage)
sources --help Full subcommand list (rename, default, attach,
current, federate, set-cr-mode, webhook, harden, ...)
sync --all Sync all sources with a local_path
sync --source <id> Sync one specific source
repos ... DEPRECATED alias for 'sources' (v0.19.0)
@@ -2329,6 +2383,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
reindex-search-vector [--dry-run] [--yes] [--json]
Recreate FTS triggers + backfill under
$GBRAIN_FTS_LANGUAGE (default 'english')
sync --strategy code Sync code files into the brain
JOBS (Minions)
+70 -4
View File
@@ -346,6 +346,12 @@ interface RegisterClientArgs {
federatedRead: string[] | undefined;
redirectUris: string[];
tokenEndpointAuthMethod: string | undefined;
boundTools: string[] | undefined;
boundSourceId: string | undefined;
boundBrainId: string | undefined;
boundSlugPrefixes: string[] | undefined;
boundMaxConcurrent: number | undefined;
budgetUsdPerDay: string | undefined;
}
export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
@@ -356,6 +362,12 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
federatedRead: undefined,
redirectUris: [],
tokenEndpointAuthMethod: undefined,
boundTools: undefined,
boundSourceId: undefined,
boundBrainId: undefined,
boundSlugPrefixes: undefined,
boundMaxConcurrent: undefined,
budgetUsdPerDay: undefined,
};
let i = 0;
let grantTypesSet = false;
@@ -389,6 +401,34 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
case '--token-endpoint-auth-method':
out.tokenEndpointAuthMethod = requireValue();
i += 2; break;
case '--bound-tools': {
const v = requireValue();
out.boundTools = v.split(',').map(s => s.trim()).filter(Boolean);
i += 2; break;
}
case '--bound-source': out.boundSourceId = requireValue(); i += 2; break;
case '--bound-brain': out.boundBrainId = requireValue(); i += 2; break;
case '--bound-slug-prefixes': {
const v = requireValue();
out.boundSlugPrefixes = v.split(',').map(s => s.trim()).filter(Boolean);
i += 2; break;
}
case '--bound-max-concurrent': {
const v = Number(requireValue());
if (!Number.isInteger(v) || v < 1) {
throw new Error('--bound-max-concurrent must be a positive integer');
}
out.boundMaxConcurrent = v;
i += 2; break;
}
case '--budget-usd-per-day': {
const v = requireValue();
if (!/^\d+(?:\.\d{1,2})?$/.test(v)) {
throw new Error('--budget-usd-per-day must be a non-negative decimal with at most 2 decimal places');
}
out.budgetUsdPerDay = v;
i += 2; break;
}
default:
throw new Error(`Unknown flag: ${flag}`);
}
@@ -405,7 +445,7 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
async function registerClient(name: string, args: string[]) {
if (!name) {
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none]');
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none] [--bound-tools T1,T2] [--bound-source SOURCE] [--bound-brain BRAIN] [--bound-slug-prefixes P1,P2] [--bound-max-concurrent N] [--budget-usd-per-day USD]');
process.exit(1);
}
let parsed: RegisterClientArgs;
@@ -413,17 +453,28 @@ async function registerClient(name: string, args: string[]) {
parsed = parseRegisterClientArgs(args);
} catch (e: any) {
console.error(`Error: ${e.message}`);
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none]');
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none] [--bound-tools T1,T2] [--bound-source SOURCE] [--bound-brain BRAIN] [--bound-slug-prefixes P1,P2] [--bound-max-concurrent N] [--budget-usd-per-day USD]');
process.exit(1);
}
const { grantTypes, scopes, sourceId, federatedRead, redirectUris, tokenEndpointAuthMethod } = parsed;
const agentBindings = parsed.boundTools || parsed.boundSourceId || parsed.boundBrainId ||
parsed.boundSlugPrefixes || parsed.boundMaxConcurrent !== undefined || parsed.budgetUsdPerDay !== undefined
? {
boundTools: parsed.boundTools,
boundSourceId: parsed.boundSourceId,
boundBrainId: parsed.boundBrainId,
boundSlugPrefixes: parsed.boundSlugPrefixes,
boundMaxConcurrent: parsed.boundMaxConcurrent,
budgetUsdPerDay: parsed.budgetUsdPerDay,
}
: undefined;
try {
await withConfiguredSql(async (sql) => {
const { GBrainOAuthProvider } = await import('../core/oauth-provider.ts');
const provider = new GBrainOAuthProvider({ sql });
const { clientId, clientSecret } = await provider.registerClientManual(
name, grantTypes, scopes, redirectUris, sourceId, federatedRead, tokenEndpointAuthMethod,
name, grantTypes, scopes, redirectUris, sourceId, federatedRead, tokenEndpointAuthMethod, agentBindings,
);
const effectiveFederated = federatedRead && federatedRead.length > 0 ? federatedRead : [sourceId];
const effectiveAuthMethod = tokenEndpointAuthMethod || 'client_secret_post';
@@ -441,7 +492,16 @@ async function registerClient(name: string, args: string[]) {
console.log(` Redirect URIs: ${redirectUris.join(', ')}`);
}
console.log(` Write source: ${sourceId}`);
console.log(` Federated reads: ${effectiveFederated.join(', ')}\n`);
console.log(` Federated reads: ${effectiveFederated.join(', ')}`);
if (agentBindings) {
console.log(` Bound tools: ${(parsed.boundTools ?? []).join(', ') || '<none>'}`);
console.log(` Bound source: ${parsed.boundSourceId ?? '<none>'}`);
console.log(` Bound brain: ${parsed.boundBrainId ?? '<none>'}`);
console.log(` Bound slug prefixes:${parsed.boundSlugPrefixes ? ' ' + parsed.boundSlugPrefixes.join(', ') : ' <none>'}`);
console.log(` Max concurrency: ${parsed.boundMaxConcurrent ?? 1}`);
console.log(` Daily budget USD: ${parsed.budgetUsdPerDay ?? '<none>'}`);
}
console.log('');
if (clientSecret) {
console.log('Save the client secret — it will not be shown again.');
} else {
@@ -527,6 +587,12 @@ Usage:
--redirect-uri <https://...> (v0.41.3+; repeatable; required for authorization_code)
--token-endpoint-auth-method <method> (v0.41.3+; client_secret_post | client_secret_basic | none;
'none' = public PKCE-only client, no secret minted)
--bound-tools <tool1,tool2> Bind submit_agent to an allow-list of tools
--bound-source <id> Bind submit_agent jobs to a source id
--bound-brain <id> Bind submit_agent jobs to a brain id
--bound-slug-prefixes <prefix1,prefix2> Bind submit_agent writes to slug prefixes
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
`);
+41 -6
View File
@@ -17,7 +17,7 @@
* gbrain autopilot --status [--json]
*/
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join } from 'path';
import { execSync } from 'child_process';
@@ -109,7 +109,21 @@ function logError(phase: string, e: unknown) {
*/
export function resolveGbrainCliPath(): string {
try {
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
// #2747: `env: process.env` is required under Bun. Bun's execSync
// snapshots process.env at Bun's OWN startup, not at call time — a
// runtime PATH mutation (dotenv/config loading, shell-profile sourcing
// in a wrapper, etc.) happening between Bun boot and this call is
// invisible to `which` without explicitly forwarding the current env.
// This is why "which gbrain" succeeds when run standalone (fresh Bun
// process, no prior mutation) but can fail from inside autopilot's own
// process at this exact call site. Same fix already applied to
// detectTini() in spawn-helpers.ts (see its comment) — this call site
// was missed.
const which = execSync('which gbrain', {
encoding: 'utf-8',
stdio: ['ignore', 'pipe', 'ignore'],
env: process.env,
}).trim();
if (which) return which;
} catch { /* not on $PATH — fall through */ }
@@ -123,7 +137,14 @@ export function resolveGbrainCliPath(): string {
return arg1;
}
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
// #2747: include what we actually saw so an operator (or a future bug
// report) doesn't have to guess whether PATH/execPath/argv[1] looked
// sane at the moment of failure.
throw new Error(
'Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH ' +
'(e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly. ' +
`Debug: PATH=${JSON.stringify(process.env.PATH ?? '')} execPath=${JSON.stringify(exec)} argv1=${JSON.stringify(arg1)}`,
);
}
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
@@ -1242,7 +1263,14 @@ function installLaunchd(wrapperPath: string, home: string, repoPath: string) {
try {
const agentsDir = join(home, 'Library', 'LaunchAgents');
mkdirSync(agentsDir, { recursive: true });
writeFileSync(plistPath(), plist);
writeFileSync(plistPath(), plist, { mode: 0o644 });
// launchd rejects group/world-writable agent plists: bootstrap/load fails
// with the opaque "Bootstrap failed: 5: Input/output error" and the login
// scan skips the file silently. writeFileSync's mode only applies on
// create — a reinstall over an existing plist keeps the old bits (a 0666
// plist written under an umask-0 parent stays 0666 forever) — so
// normalize unconditionally.
chmodSync(plistPath(), 0o644);
execSync(`launchctl load "${plistPath()}"`, { stdio: 'pipe' });
console.log('Installed launchd service: com.gbrain.autopilot');
console.log(` Repo: ${repoPath}`);
@@ -1332,7 +1360,11 @@ export function migrateSystemdUnitToRestartAlways(): { rewritten: boolean; reaso
return { rewritten: false, reason: 'hand-edited' };
}
try {
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]));
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]), { mode: 0o644 });
// This path always rewrites an EXISTING unit, so writeFileSync's mode
// never applies — chmod is the only thing that normalizes a unit born
// 0666 under a umask-0 parent (systemd warns on world-writable units).
chmodSync(unitPath, 0o644);
try {
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
} catch {
@@ -1349,7 +1381,10 @@ function installSystemd(wrapperPath: string, repoPath: string) {
try {
const unitPath = systemdUnitPath();
mkdirSync(join(process.env.HOME || '', '.config', 'systemd', 'user'), { recursive: true });
writeFileSync(unitPath, unit);
writeFileSync(unitPath, unit, { mode: 0o644 });
// Same umask-0 hardening as the launchd path (systemd warns on
// world-writable units); mode only applies on create, so normalize.
chmodSync(unitPath, 0o644);
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
execSync('systemctl --user enable --now gbrain-autopilot.service', { stdio: 'pipe', timeout: 15_000 });
console.log('Installed systemd user service: gbrain-autopilot.service');
+19 -3
View File
@@ -98,9 +98,25 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
const value = args[2];
if (action === 'get' && key) {
const val = await engine.getConfig(key);
if (val !== null) {
console.log(val);
// #2120: `get` used to read only the DB plane, so a runtime-effective key
// in ~/.gbrain/config.json (or env) reported not-found. Resolve the way
// the runtime does — env/file plane wins over DB (loadConfig() already
// overlays env onto the file) — and report which plane answered on
// stderr, keeping stdout a bare value for scripts.
const filePlane = loadConfig() as Record<string, unknown> | null;
const fileVal = filePlane?.[key];
const dbVal = await engine.getConfig(key);
const val = fileVal !== undefined && fileVal !== null ? fileVal : dbVal;
if (val !== null && val !== undefined) {
console.log(typeof val === 'string' ? val : JSON.stringify(val));
if (fileVal !== undefined && fileVal !== null) {
const shadow = dbVal !== null && dbVal !== undefined
? ' — a DB-plane value also exists and is shadowed at runtime'
: '';
console.error(`[config] source: file/env plane (~/.gbrain/config.json or env)${shadow}`);
} else {
console.error(`[config] source: db plane`);
}
} else {
console.error(`Config key not found: ${key}`);
process.exit(1);
+44 -22
View File
@@ -3056,7 +3056,7 @@ export async function computeConversationFactsBacklogCheck(
const typesRaw = await engine.getConfig(
'cycle.conversation_facts_backfill.types',
);
let types = ['conversation', 'meeting', 'slack', 'email'];
let types = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'];
if (typesRaw) {
try {
const parsed = JSON.parse(typesRaw);
@@ -4345,7 +4345,7 @@ export async function buildChecks(
// 2. Skill conformance (SKILL group — gated)
if (scope === 'all' && skillsDir) {
const conformanceResult = checkSkillConformance(skillsDir);
const conformanceResult = skillConformanceCheck(skillsDir);
checks.push(conformanceResult);
}
@@ -4927,8 +4927,8 @@ export async function buildChecks(
try {
const { readConversationBodyForParsing } = await import('../core/conversation-parser/body.ts');
const { parseConversation } = await import('../core/conversation-parser/parse.ts');
const allowedTypes = ['conversation', 'meeting', 'slack', 'email'] as const;
// PageFilters supports singular `type` only; iterate the 4 types
const allowedTypes = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'] as const;
// PageFilters supports singular `type` only; iterate the allowed types
// and cap at ~50/each to land at ~200 total max.
const sample: import('../core/types.ts').Page[] = [];
for (const t of allowedTypes) {
@@ -5138,13 +5138,7 @@ export async function buildChecks(
checks.push({
name: 'multi_source_drift',
status: 'warn',
message:
`${result.count} page slug(s) appear at 'default' but NOT at the intended source ` +
`(e.g., ${sampleStr}). Two possible causes: (1) pre-v0.30.3 putPage misroutes; ` +
`(2) source X never completed initial sync and the default page is unrelated. ` +
`Verify with 'gbrain sources status', then either re-sync with ` +
`'gbrain sync --source <id> --full' or 'gbrain delete <slug>' if the default-source ` +
`row is the misroute. (A 'gbrain sources rehome' cleanup command is tracked for v0.32.0.)`,
message: multiSourceDriftAdvice(result.count, sampleStr),
});
} else {
checks.push({
@@ -7184,9 +7178,17 @@ export async function buildChecks(
let vanished = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
try {
fs.statSync(r.storage_path);
fs.statSync(abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
@@ -7430,15 +7432,13 @@ function printAutoFixReport(report: AutoFixReport, dryRun: boolean, jsonOutput:
/** Quick skill conformance check — frontmatter + required sections */
function checkSkillConformance(skillsDir: string): Check {
const manifestPath = join(skillsDir, 'manifest.json');
if (!existsSync(manifestPath)) {
return { name: 'skill_conformance', status: 'warn', message: 'manifest.json not found' };
}
export function skillConformanceCheck(skillsDir: string): Check {
try {
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8'));
const skills = manifest.skills || [];
// Host workspaces are allowed to omit a gbrain-specific manifest. Keep
// conformance aligned with resolver_health and skill_brain_first by using
// the canonical fallback that derives entries from direct SKILL.md files.
const manifest = loadOrDeriveManifest(skillsDir);
const skills = manifest.skills;
let passing = 0;
const failing: string[] = [];
@@ -7458,7 +7458,8 @@ function checkSkillConformance(skillsDir: string): Check {
}
if (failing.length === 0) {
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass` };
const derivedNote = manifest.derived ? ' (derived from SKILL.md files)' : '';
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass${derivedNote}` };
}
return {
name: 'skill_conformance',
@@ -7466,7 +7467,7 @@ function checkSkillConformance(skillsDir: string): Check {
message: `${passing}/${skills.length} pass. Failing: ${failing.join(', ')}`,
};
} catch {
return { name: 'skill_conformance', status: 'warn', message: 'Could not parse manifest.json' };
return { name: 'skill_conformance', status: 'warn', message: 'Could not load or derive skills manifest' };
}
}
@@ -8074,3 +8075,24 @@ async function checkSchemaPackSourceDrift(engine: BrainEngine): Promise<Check> {
};
}
}
/**
* #1123 multi_source_drift remediation advice. Exported so the regression
* test can pin that it only references CLI surfaces that actually exist
* (the pre-fix text pointed at 'gbrain sources rehome', which was never
* built, and at 'gbrain delete <slug>' without explaining that delete
* targets the ACTIVE source following it literally on a multi-source
* brain deletes the correctly-routed row).
*/
export function multiSourceDriftAdvice(count: number, sampleStr: string): string {
return (
`${count} page slug(s) appear at 'default' but NOT at the intended source ` +
`(e.g., ${sampleStr}). Two possible causes: (1) pre-v0.30.3 putPage misroutes; ` +
`(2) the intended source never completed initial sync and the default page is unrelated. ` +
`Verify with 'gbrain sources status', then re-sync with ` +
`'gbrain sync --source <id> --full' (reconciles drift without deleting data). ` +
`If a misrouted default-source row remains after re-sync, remove it with ` +
`'GBRAIN_SOURCE=default gbrain delete <slug>' — delete targets the active source, ` +
`so pin it to 'default' explicitly.`
);
}
+65
View File
@@ -76,6 +76,18 @@ interface DreamArgs {
drain: boolean;
/** Drain wallclock budget in seconds. Default 300 (5 min). */
windowSeconds: number;
/**
* issue #2860 `--once`. One-shot bypass of the named `--phase`'s own
* `dream.<phase>.enabled` / `cycle.<phase>.enabled` config gate, for this
* invocation only. Never reads or writes config unlike the old
* "toggle enabled true, run, toggle back to false" workaround, a crash
* mid-run can't leave any global state stuck. Requires an explicit
* `--phase <name>`; bare `--once` is a usage error (there'd be no single
* phase to target). Applies only to phases with a config `.enabled` gate
* (patterns, synthesize, conversation_facts_backfill, enrich_thin,
* skillopt) a no-op for phases that always run when named directly.
*/
once: boolean;
}
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
@@ -105,6 +117,14 @@ function collectFlagValues(args: string[], flag: string): string[] | null {
function parseArgs(args: string[]): DreamArgs {
const phaseIdx = args.indexOf('--phase');
// issue #2860 (Codex P3): captured BEFORE --input/--drain get a chance to
// implicitly default `phase` below, so --once's validation can require
// the user actually TYPED --phase, not merely that some phase ended up
// resolved. Without this, `--input <f> --once` and `--drain --once`
// slip past the "explicit --phase required" contract (the derived
// `phase` value is already non-null by the time that check runs) and
// --once becomes silently ineffective for both.
const phaseWasExplicit = phaseIdx !== -1;
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
let phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
? (rawPhase as CyclePhase)
@@ -214,6 +234,35 @@ function parseArgs(args: string[]): DreamArgs {
}
}
// issue #2860: --once requires an EXPLICIT single --phase target (typed
// by the user, not merely implied by --input/--drain — see
// `phaseWasExplicit` above). Bare `--once` (full/default cycle) has no
// single phase to bypass the gate for, and force-enabling EVERY
// currently-disabled phase at once would be exactly the kind of
// surprise-spend risk the flag exists to prevent. An implicit phase
// (from --input or --drain) is rejected too: --drain returns before
// onceForPhase is ever read, and --input already bypasses the
// synthesize gate on its own, so --once would silently do nothing in
// either case — reject loudly instead of pretending it worked (Codex
// review finding).
//
// Codex review finding: `--help` must short-circuit BEFORE this exits(2),
// matching the "IRON RULE" pinned by test/dream.test.ts's
// "--help --source whatever prints help and exits 0" case — `gbrain
// dream --help --once` (no --phase) must show help, not a usage error.
const once = args.includes('--once');
const wantsHelp = args.includes('--help') || args.includes('-h');
if (once && !phaseWasExplicit && !wantsHelp) {
console.error(
'--once requires an explicit --phase <name> (bypasses that one ' +
'phase\'s dream.<phase>.enabled / cycle.<phase>.enabled gate for ' +
'this run only; never touches config). A phase implied by --input ' +
'or --drain does not count — --once would silently do nothing for ' +
'those. Usage: gbrain dream --phase <name> --once',
);
process.exit(2);
}
return {
json: args.includes('--json'),
dryRun: args.includes('--dry-run'),
@@ -229,6 +278,7 @@ function parseArgs(args: string[]): DreamArgs {
source,
drain,
windowSeconds,
once,
};
}
@@ -310,6 +360,17 @@ Options:
"--dry-run" does NOT mean "zero LLM calls."
--json Emit the CycleReport as JSON (agent-readable)
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
--once With --phase <name>: run that phase once even if its
own dream.<phase>.enabled / cycle.<phase>.enabled
config gate is false. Never reads or writes config
unlike toggling the flag on/off around the run, a
crash mid-invocation can't leave it stuck. Applies to
patterns, synthesize, conversation_facts_backfill,
enrich_thin, skillopt; no-op on phases with no such
gate. Requires an EXPLICIT --phase <name> a phase
implied by --input or --drain does not count (bare
--once, or --once with --input/--drain and no
explicit --phase, is a usage error).
--pull git pull the brain repo before syncing (default: no pull)
--dir <path> Brain directory (default: configured brain). On a
postgres/remote brain with no local checkout, the
@@ -353,6 +414,7 @@ Examples:
gbrain dream
gbrain dream --dry-run --json
gbrain dream --phase lint
gbrain dream --phase patterns --once # run once, ignore dream.patterns.enabled=false
gbrain dream --phase synthesize --input ~/transcripts/2026-04-25.txt
gbrain dream --phase synthesize --from 2026-04-01 --to 2026-04-25
0 2 * * * gbrain dream --json # nightly via cron
@@ -594,6 +656,9 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
synthFrom: opts.from ?? undefined,
synthTo: opts.to ?? undefined,
synthBypassDreamGuard: opts.bypassDreamGuard,
// issue #2860: opts.phase is guaranteed non-null here when opts.once is
// set (parseArgs enforces --once requires --phase).
onceForPhase: opts.once ? opts.phase! : undefined,
});
if (opts.json) {
+34 -4
View File
@@ -581,7 +581,7 @@ async function embedPage(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
const updated: ChunkInput[] = chunks.map(c => ({
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -605,6 +605,31 @@ async function embedPage(
slog(`${slug}: embedded ${toEmbed.length} chunks`);
}
/**
* Carry code-chunk metadata (language, symbol_name, symbol_type, line range,
* parent scope, doc comment, qualified name) from a loaded Chunk back into a
* ChunkInput destined for upsertChunks.
*
* Issue #769: every re-embed used to strip these fields, and upsertChunks
* overwrites (does not COALESCE) the metadata columns from EXCLUDED, so
* each pass clobbered code-def's primary index to NULL. Pulling the
* preservation into one helper keeps the three re-embed call sites
* (embedPage, embedAll non-stale, embedAllStale) in lock-step.
*/
function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
return {
...base,
language: loaded.language ?? undefined,
symbol_name: loaded.symbol_name ?? undefined,
symbol_type: loaded.symbol_type ?? undefined,
start_line: loaded.start_line ?? undefined,
end_line: loaded.end_line ?? undefined,
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
doc_comment: loaded.doc_comment ?? undefined,
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
};
}
async function embedAll(
engine: BrainEngine,
staleOnly: boolean,
@@ -717,8 +742,10 @@ async function embedAll(
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
}
// Preserve ALL chunks, only update embeddings for stale ones
const updated: ChunkInput[] = chunks.map(c => ({
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
// doesn't clobber language/symbol_name/symbol_type to NULL.
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
@@ -1012,7 +1039,10 @@ async function embedAllStale(
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
}
const merged: ChunkInput[] = existing.map(c => ({
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
// to NULL on every cycle.
const merged: ChunkInput[] = existing.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
chunk_text: c.chunk_text,
chunk_source: c.chunk_source,
+4 -2
View File
@@ -33,7 +33,7 @@ import type { BrainEngine } from '../core/engine.ts';
import type { EnrichCandidate, PageType } from '../core/types.ts';
import { operations } from '../core/operations.ts';
import type { OperationContext } from '../core/operations.ts';
import { isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
import { configureGatewayIfUninitialized, isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
import { hybridSearch } from '../core/search/hybrid.ts';
import { serializeMarkdown } from '../core/markdown.ts';
@@ -807,7 +807,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
process.exit(1);
}
// Chat gateway required for non-dry-run.
// Chat gateway is required for non-dry-run. Recover a cold singleton before
// reporting an availability error (#2590).
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
if (!parsed.dryRun && !isAvailable('chat')) {
console.error('Chat gateway unavailable. Configure a chat model (e.g. `gbrain config set chat_model anthropic:claude-haiku-4-5`), or pass --dry-run to preview candidates.');
process.exit(1);
+23 -6
View File
@@ -71,7 +71,7 @@ import {
extractFactsFromTurn,
isFactsExtractionEnabled,
} from '../core/facts/extract.ts';
import { isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
import { configureGatewayIfUninitialized, isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
import { listSources } from '../core/sources-ops.ts';
import {
@@ -81,7 +81,6 @@ import {
} from '../core/op-checkpoint.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions, maybeBackground } from '../core/cli-options.ts';
import { loadConfig } from '../core/config.ts';
import { createHash } from 'crypto';
// v0.41.15.0 (T5): worker-pool primitive + per-source-clamp wrapper +
// per-page advisory lock + delete-orphans-first replay safety. See plan
@@ -141,7 +140,14 @@ export const DEFAULT_MAX_COST_USD = 5.0;
* `--types` flag is an explicit per-run override; cycle config is
* the single source of truth.
*/
export const ALLOWED_TYPES = ['conversation', 'meeting', 'slack', 'email'] as const;
export const ALLOWED_TYPES = [
'conversation',
'meeting',
'slack',
'email',
'imessage',
'imessage-daily',
] as const;
export type AllowedType = (typeof ALLOWED_TYPES)[number];
/**
@@ -757,6 +763,12 @@ async function processPage(
source_markdown_slug: page.slug,
source: PER_SEGMENT_SOURCE_PREFIX,
source_session: sessionId,
// Preserve the conversation's valid time instead of defaulting every
// extracted fact to extraction time. Epoch-anchored parses have no
// trustworthy date, so they retain the existing now() fallback.
...(seg.startIso && !seg.startIso.startsWith('1970-')
? { valid_from: new Date(seg.startIso) }
: {}),
context:
fact.context ?? `from ${page.slug} segment ${seg.startIso}..${seg.endIso}`,
}));
@@ -1069,7 +1081,8 @@ export async function runExtractConversationFactsCore(
}
// Fall through to receipt+rollup write so the partial run is
// still observable in extract_health doctor + extracts/ pages.
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
// ...but not under --dry-run: a preview must not persist cache state.
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
// Return partial result — caller (CLI / Minion) decides how to
// surface. NOT a thrown failure.
return result;
@@ -1081,7 +1094,9 @@ export async function runExtractConversationFactsCore(
// (queryable + citable per D-EXTRACT-17/19) AND UPSERTs the per-day
// rollup row (best-effort cache per F-OUT-19). Both are best-effort —
// failures stderr-warn but never fail the parent operation.
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
// --dry-run must not persist cache/knowledge state: skip the rollup UPSERT +
// receipt-page write so a preview leaves no extract cache row behind.
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
return result;
}
@@ -1351,7 +1366,9 @@ export async function runExtractConversationFacts(
process.exit(1);
}
// Chat gateway is required for non-dry-run.
// Chat gateway is required for non-dry-run. Recover a cold singleton before
// reporting an availability error (#2590).
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
if (!parsed.dryRun && !isAvailable('chat')) {
console.error('Chat gateway unavailable. Configure an Anthropic or compatible chat model, or pass --dry-run to preview segmentation.');
process.exit(1);
+45 -9
View File
@@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
// call isSyncable at all — so it descended into `node_modules/`, emitted
// markdown files from there, AND ignored the canonical exclusion list
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
// (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor
// subtrees before recursion (saving IO), and isSyncable filters the emit
// set against the canonical markdown-strategy rules.
const files: { path: string; relPath: string }[] = [];
@@ -568,6 +568,20 @@ export interface ExtractOpts {
* paths: extractForSlugs, extractLinksFromDir, extractTimelineFromDir.
*/
signal?: AbortSignal;
/**
* Brain source id to stamp on extracted fs-walk rows (#1747 / #1503).
*
* The fs-walk extractors build LinkBatchInput / TimelineBatchInput rows
* with no source_id, so addLinksBatch / addTimelineEntriesBatch map
* missing literal 'default'. On a brain whose content lives in a
* non-'default' source (e.g. 'wiki'), the batch INSERT's
* `JOIN pages ON (slug, source_id='default')` drops EVERY row 0
* inserted, no error (the "created 0 from N pages" silent no-op).
* Threading the resolved source id here stamps from/to/origin_source_id
* so the JOIN matches. When undefined, rows fall back to 'default' as
* before (single-'default'-source brains unaffected).
*/
sourceId?: string;
}
/**
@@ -606,7 +620,7 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
// Nothing changed — skip entirely.
return result;
}
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers, opts.signal);
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.links_created = r.links_created;
result.timeline_entries_created = r.timeline_created;
result.pages_processed = r.pages;
@@ -615,12 +629,12 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
// Full walk path: CLI `gbrain extract` or first-run.
if (opts.mode === 'links' || opts.mode === 'all') {
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal);
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.links_created = r.created;
result.pages_processed = r.pages;
}
if (opts.mode === 'timeline' || opts.mode === 'all') {
const r = await extractTimelineFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal);
const r = await extractTimelineFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.timeline_entries_created = r.created;
result.pages_processed = Math.max(result.pages_processed, r.pages);
}
@@ -941,11 +955,21 @@ Status (v0.42):
}
}
} else {
// #1747: resolve the brain source id and thread it into the fs-walk
// extractors so batch rows carry from/to_source_id. Without this they
// default to 'default' and addLinksBatch's JOIN drops every row on a
// non-'default' brain → silent "created 0 from N pages". Resolution
// honors --source-id, then GBRAIN_SOURCE / .gbrain-source /
// registered-path / sole-non-default, mirroring the source-aware
// inline hooks (extractLinksForSlugs) that #1204 confirmed correct.
const { resolveSourceId } = await import('../core/source-resolver.ts');
const resolvedSourceId = await resolveSourceId(engine, sourceIdFilter, brainDir);
result = await runExtractCore(engine, {
mode: subcommand as 'links' | 'timeline' | 'all',
dir: brainDir,
dryRun,
jsonMode,
sourceId: resolvedSourceId,
workers,
});
}
@@ -985,6 +1009,8 @@ async function extractForSlugs(
// shared counter increments atomic.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id on batch rows (see ExtractOpts.sourceId).
sourceId?: string,
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
// Build the full slug set for link resolution (fast: just readdir, no file reads)
const allFiles = walkMarkdownFiles(brainDir);
@@ -1065,7 +1091,9 @@ async function extractForSlugs(
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
linksCreated++;
} else {
linkBatch.push(link);
linkBatch.push(sourceId
? { ...link, from_source_id: sourceId, to_source_id: sourceId, origin_source_id: sourceId }
: link);
if (linkBatch.length >= BATCH_SIZE) await flushLinks();
}
}
@@ -1078,7 +1106,7 @@ async function extractForSlugs(
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date}${entry.summary}`);
timelineCreated++;
} else {
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail, ...(sourceId ? { source_id: sourceId } : {}) });
if (timelineBatch.length >= BATCH_SIZE) await flushTimeline();
}
}
@@ -1107,6 +1135,9 @@ async function extractLinksFromDir(
// v0.41.15.0 (T7): in-process worker count. Default 1.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id on batch rows so the
// addLinksBatch JOIN matches non-'default' source pages.
sourceId?: string,
): Promise<{ created: number; pages: number }> {
const files = walkMarkdownFiles(brainDir);
const allSlugs = new Set(files.map(f => pathToSlug(f.relPath)));
@@ -1163,7 +1194,9 @@ async function extractLinksFromDir(
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
created++;
} else {
batch.push(link);
batch.push(sourceId
? { ...link, from_source_id: sourceId, to_source_id: sourceId, origin_source_id: sourceId }
: link);
if (batch.length >= BATCH_SIZE) await flush();
}
}
@@ -1186,6 +1219,9 @@ async function extractTimelineFromDir(
// v0.41.15.0 (T7): in-process worker count. Default 1.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id so addTimelineEntriesBatch
// matches non-'default' source pages.
sourceId?: string,
): Promise<{ created: number; pages: number }> {
const files = walkMarkdownFiles(brainDir);
@@ -1232,7 +1268,7 @@ async function extractTimelineFromDir(
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date}${entry.summary}`);
created++;
} else {
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail, ...(sourceId ? { source_id: sourceId } : {}) });
if (batch.length >= BATCH_SIZE) await flush();
}
}
@@ -1615,7 +1651,7 @@ async function extractTimelineFromDB(
* make re-extraction idempotent). EVERY processed page is stamped, including
* zero-link pages they WERE processed.
*/
async function extractStaleFromDB(
export async function extractStaleFromDB(
engine: BrainEngine,
opts: {
dryRun: boolean;
+8
View File
@@ -30,6 +30,7 @@ import {
type AuditReport,
type AuditFix,
} from '../core/brain-writer.ts';
import { collectGitVisibleFiles } from '../core/git-visible-files.ts';
import { isSyncable, pruneDir, slugifyPath } from '../core/sync.ts';
export async function runFrontmatter(args: string[]): Promise<void> {
@@ -272,6 +273,13 @@ export function collectFiles(
if (st.isFile()) {
return [target];
}
const gitFiles = collectGitVisibleFiles(target, (rel) => isSyncable(rel, { strategy: 'markdown' }));
if (gitFiles) {
if (visitDir) visitDir(target);
return gitFiles;
}
const out: string[] = [];
const stack = [target];
if (visitDir) visitDir(target);
+95 -8
View File
@@ -11,7 +11,9 @@ import {
isCodeFilePath,
isMarkdownFilePath,
isImageFilePath as isImageFilePathFromSync,
matchesAnyGlob,
pruneDir,
SYNC_SKIP_FILES,
type SyncStrategy,
} from '../core/sync.ts';
import { sortNewestFirst } from '../core/sort-newest-first.ts';
@@ -19,6 +21,7 @@ import {
loadCheckpoint,
saveCheckpoint,
clearCheckpoint,
resolveImportTargetDir,
resumeFilter,
} from '../core/import-checkpoint.ts';
@@ -45,7 +48,25 @@ export interface RunImportResult {
export async function runImport(
engine: BrainEngine,
args: string[],
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string; managedBookmark?: boolean } = {},
opts: {
commit?: string;
strategy?: SyncStrategy;
sourceId?: string;
managedBookmark?: boolean;
/**
* #753/#774: glob patterns to exclude from the import (same semantics as
* `isSyncable`'s `exclude` matched against the dir-relative path).
* Threaded by performFullSync for `gbrain sync --exclude`.
*/
exclude?: string[];
/**
* #753/#774 monorepo subdir-source support: when set, slugs and
* `source_path` are computed relative to this root (the git repo root)
* instead of `dir` (the sync scope), so `wiki/page1.md` lands as slug
* `wiki/page1` consistently across full and incremental sync.
*/
slugRoot?: string;
} = {},
): Promise<RunImportResult> {
const noEmbed = args.includes('--no-embed');
const fresh = args.includes('--fresh');
@@ -167,7 +188,19 @@ export async function runImport(
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--json]');
process.exit(1);
}
const dir: string = dirArg; // narrowed; survives closure capture
// #1728: capture the import target ONCE as an absolute real path. Every
// downstream consumer of `dir` (collection, checkpoint load/save, resume
// filtering) sees the same canonical identity — never the caller's `.`/
// relative spelling, which would make the persisted checkpoint `dir`
// resolve against whatever CWD a later process happens to run from.
let dir: string;
try {
dir = resolveImportTargetDir(dirArg);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Import target is not readable: ${dirArg} (${msg})`);
process.exit(1);
}
// v0.31.2: collect under the right strategy. Pre-fix this called
// collectMarkdownFiles unconditionally — code-strategy first sync
@@ -176,13 +209,30 @@ export async function runImport(
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
const _walkT0 = Date.now();
console.error(`[gbrain phase] import.collect_files start dir=${dir} strategy=${strategy}`);
const allFiles = collectSyncableFiles(dir, { strategy });
let allFiles = collectSyncableFiles(dir, { strategy });
console.error(
`[gbrain phase] import.collect_files done ${Date.now() - _walkT0}ms files=${allFiles.length}`,
);
const fileTypeLabel = strategy === 'code' ? 'code'
: strategy === 'auto' ? 'syncable' : 'markdown';
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
// #753/#774: apply --exclude glob patterns (threaded by performFullSync).
if (opts.exclude && opts.exclude.length > 0) {
const beforeExclude = allFiles.length;
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
console.log(
`Found ${allFiles.length} ${fileTypeLabel} files ` +
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
);
// NAV-4: everything excluded is almost always a mistyped pattern — warn.
if (beforeExclude > 0 && allFiles.length === 0) {
console.warn(
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
);
}
} else {
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
}
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
// See src/core/sort-newest-first.ts for the policy.
@@ -228,6 +278,11 @@ export async function runImport(
async function processFile(eng: BrainEngine, filePath: string) {
const relativePath = relative(dir, filePath);
// #753/#774: slug + source_path base. When performFullSync syncs a
// monorepo subdir, slugRoot is the git root so slugs stay git-root-
// relative (matching the incremental path's git-diff paths). The
// checkpoint (`completed`) stays dir-relative — resumeFilter's contract.
const importRelPath = opts.slugRoot ? relative(opts.slugRoot, filePath) : relativePath;
// v0.31.2 (D5): per-file slow-path log. Fires only when a single
// file takes >5s. The user's hang surfaces as one file taking
// forever — without this, the agent can't see which file.
@@ -238,8 +293,8 @@ export async function runImport(
// up images when GBRAIN_EMBEDDING_MULTIMODAL=true so this branch is
// unreachable when the gate is off; defense-in-depth check anyway.
const result = isImageFilePath(relativePath) && process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'
? await importImageFile(eng, filePath, relativePath, { noEmbed, sourceId })
: await importFile(eng, filePath, relativePath, { noEmbed, sourceId, activePack: importActivePack });
? await importImageFile(eng, filePath, importRelPath, { noEmbed, sourceId })
: await importFile(eng, filePath, importRelPath, { noEmbed, sourceId, activePack: importActivePack });
const _fileMs = Date.now() - _fileT0;
if (_fileMs > 5000) {
console.error(`[gbrain phase] import.process_file slow ${_fileMs}ms ${relativePath}`);
@@ -255,7 +310,9 @@ export async function runImport(
if (result.error && result.error !== 'unchanged') {
console.error(` Skipped ${relativePath}: ${result.error}`);
// Bug 9 — non-"unchanged" skips carry a real error reason.
failures.push({ path: relativePath, error: result.error });
// #774: ledger paths use the slug base so an incremental sync's
// success at the same (git-root-relative) path clears the row.
failures.push({ path: importRelPath, error: result.error });
} else {
// 'unchanged' or no-error skip: content_hash matched a prior
// successful import, so this file IS done for checkpoint purposes.
@@ -273,7 +330,7 @@ export async function runImport(
}
errors++;
skipped++;
failures.push({ path: relativePath, error: msg });
failures.push({ path: importRelPath, error: msg });
}
processed++;
tickProgress();
@@ -287,6 +344,9 @@ export async function runImport(
catch { /* non-fatal */ }
}
saveCheckpoint(checkpointPath, {
schema_version: 1,
owner: 'gbrain',
kind: 'import',
dir,
completedPaths: Array.from(completed),
timestamp: new Date().toISOString(),
@@ -493,12 +553,39 @@ interface CollectOpts {
* The first-sync walker historically admitted them on markdown too when
* `GBRAIN_EMBEDDING_MULTIMODAL=true`. Codex (C5) flagged the contradiction
* preserve the walker semantic explicitly.
*
* Closes #345: exclude `SYNC_SKIP_FILES` metafiles
* (`README.md` / `index.md` / `log.md` / `schema.md` / `RESOLVER.md`).
* Incremental `sync` skips these via `isSyncable`, but the bulk-import
* walker only filtered by extension so a directory import imported every
* directory README as a page, titled by its folder ("People", "Companies",
* ). Those index-titled pages then trigram-corrupt fuzzy entity resolution
* (any `people/X` slug matches the "People" page) and inflate orphan count.
* Funnel both admission paths through the same metafile exclusion so import
* and sync agree on what is a page.
*/
function isCollectibleForWalker(
path: string,
strategy: SyncStrategy,
multimodalOn: boolean,
): boolean {
// #2607: apply the SAME segment-level prune gate as incremental sync's
// `classifySync` (core/sync.ts). The FS walk below prunes at descent time,
// but the git fast path enumerates via `git ls-files` and historically
// filtered only by extension — so `sync --full` imported (and resurrected
// previously-deleted) pages under dot-dirs / vendored trees that incremental
// sync excludes. Full and incremental must agree on the exclusion set.
// (In the FS-walk route `path` is a basename, so this is the same dot-file
// check pruneDir already applied there — no behavior change on that route.)
const segments = path.split('/');
if (segments.some((seg) => !pruneDir(seg))) return false;
// Metafiles are directory scaffolding (READMEs / index / log / schema /
// resolver), not typed brain pages — same exclusion `sync`'s `isSyncable`
// applies. Guards both the FS-walk and the git-fast-path collection routes.
const basename = segments[segments.length - 1] || '';
if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false;
switch (strategy) {
case 'code':
return isCodeFilePath(path);
+78 -16
View File
@@ -7,7 +7,7 @@ import type { BrainEngine } from '../core/engine.ts';
import { MinionQueue } from '../core/minions/queue.ts';
import { MinionWorker } from '../core/minions/worker.ts';
import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts';
import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts';
import type { MinionHandler, MinionJob, MinionJobStatus } from '../core/minions/types.ts';
import type { PaceKeyOverrides } from '../core/pace-mode.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
@@ -22,6 +22,49 @@ function hasFlag(args: string[], flag: string): boolean {
return args.includes(flag);
}
/**
* Long-lived workers outlive operator config changes. Re-stamp the AI gateway
* from DB-backed model config immediately before queued jobs enter gateway-backed
* paths, so a stale process-level default cannot route new work to the wrong
* provider.
*/
async function refreshGatewayForJob(engine: BrainEngine): Promise<void> {
const { reconfigureGatewayWithEngine } = await import('../core/ai/gateway.ts');
await reconfigureGatewayWithEngine(engine);
}
const GATEWAY_REFRESH_JOB_NAMES = new Set([
'embed',
'extract-conversation-facts',
'enrich',
'contextual_reindex_per_chunk',
'autopilot-cycle',
'synthesize',
'patterns',
'consolidate',
'extract_facts',
'extract-atoms-drain',
'embed-backfill',
'extract-takes-from-pages',
'embed-catch-up',
]);
function registerBuiltinJob(
worker: MinionWorker,
engine: BrainEngine,
name: string,
handler: MinionHandler,
): void {
if (!GATEWAY_REFRESH_JOB_NAMES.has(name)) {
worker.register(name, handler);
return;
}
worker.register(name, async (job) => {
await refreshGatewayForJob(engine);
return await handler(job);
});
}
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
* Throws on malformed input (caller should surface the error and exit).
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
@@ -132,9 +175,23 @@ function formatJobDetail(job: MinionJob): string {
return lines.join('\n');
}
export async function runJobs(engine: BrainEngine, args: string[]): Promise<void> {
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
const sub = args[0];
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
// with remote MCP routing (`list`, `get`) so no scratch local engine is
// ever built. Any other subcommand arriving with a null engine is a
// routing bug upstream of this function — refuse instead of crashing
// inside MinionQueue.
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
process.exit(1);
}
// Null only ever reaches the MCP-routed `list`/`get` branches, which
// never touch the engine — narrowed once here so the host-only cases
// below typecheck unchanged.
const engine = engineOrNull as BrainEngine;
if (!sub || sub === '--help' || sub === '-h') {
console.log(`gbrain jobs — Minions job queue
@@ -217,6 +274,8 @@ HANDLER TYPES (built in)
return;
}
// The constructor just stores the reference; on the null (thin-client
// list/get) paths no queue method is ever reached.
const queue = new MinionQueue(engine);
switch (sub) {
@@ -1423,7 +1482,7 @@ export async function registerBuiltinHandlers(
return { ...result, embed_job_id: embedJobId, embed_skip_reason: embedSkipReason };
});
worker.register('embed', async (job) => {
registerBuiltinJob(worker, engine, 'embed', async (job) => {
const { runEmbedCore } = await import('./embed.ts');
// Primary Minion progress channel is job.updateProgress (DB-backed,
// readable via `gbrain jobs get <id>`). Stderr from the worker daemon
@@ -1470,7 +1529,7 @@ export async function registerBuiltinHandlers(
// BudgetTracker inside its own process. BudgetExhausted is caught at
// the core level and returned as `result.budget_exhausted: true` (NOT
// a job failure) so the user can resume with a higher cap.
worker.register('extract-conversation-facts', async (job) => {
registerBuiltinJob(worker, engine, 'extract-conversation-facts', async (job) => {
const { runExtractConversationFactsCore } = await import('./extract-conversation-facts.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId) {
@@ -1481,7 +1540,7 @@ export async function registerBuiltinHandlers(
}
const types = Array.isArray(job.data.types)
? (job.data.types as string[]).filter((t) =>
['conversation', 'meeting', 'slack', 'email'].includes(t),
['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'].includes(t),
)
: undefined;
const result = await runExtractConversationFactsCore(engine, {
@@ -1529,7 +1588,7 @@ export async function registerBuiltinHandlers(
// at the core level and returned as result.budget_exhausted (NOT a failure).
// Strict per-source: the CLI fans out one job per source when --source is
// omitted, so a job ALWAYS carries data.sourceId.
worker.register('enrich', async (job) => {
registerBuiltinJob(worker, engine, 'enrich', async (job) => {
const { runEnrichCore } = await import('./enrich.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId) {
@@ -1669,13 +1728,13 @@ export async function registerBuiltinHandlers(
const { makeContextualReindexHandler } = await import(
'../core/minions/handlers/contextual-reindex-per-chunk.ts'
);
worker.register('contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
registerBuiltinJob(worker, engine, 'contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
}
// derivation); the handler returns { partial, status, report } so
// `gbrain jobs get <id>` shows the full structured report. Does NOT
// throw on partial: a flaky phase must not block every future cycle.
worker.register('autopilot-cycle', async (job) => {
registerBuiltinJob(worker, engine, 'autopilot-cycle', async (job) => {
const { runCycle } = await import('../core/cycle.ts');
// v0.41.30 (T2): fall back to null (NOT cwd '.') when no repo is configured.
// The queued cycle is the same primitive `gbrain dream` uses; a checkout-less
@@ -1780,6 +1839,7 @@ export async function registerBuiltinHandlers(
brainDir: effectiveBrainDir,
pull,
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
...(sourceId ? { sourceId } : {}),
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
yieldBetweenPhases: async () => {
@@ -1817,6 +1877,7 @@ export async function registerBuiltinHandlers(
brainDir: repoPath,
pull: false, // brain-wide DB/maintenance work never git-pulls
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
phases,
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
});
@@ -1962,17 +2023,18 @@ export async function registerBuiltinHandlers(
brainDir: repoPath,
phases: [phase as any],
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
});
return { phase, status: report.status, report };
};
// PROTECTED — internally spawn subagent children
worker.register('synthesize', makePhaseHandler('synthesize'));
worker.register('patterns', makePhaseHandler('patterns'));
worker.register('consolidate', makePhaseHandler('consolidate'));
registerBuiltinJob(worker, engine, 'synthesize', makePhaseHandler('synthesize'));
registerBuiltinJob(worker, engine, 'patterns', makePhaseHandler('patterns'));
registerBuiltinJob(worker, engine, 'consolidate', makePhaseHandler('consolidate'));
// Open — DB writes only, no LLM spend
worker.register('extract_facts', makePhaseHandler('extract_facts'));
registerBuiltinJob(worker, engine, 'extract_facts', makePhaseHandler('extract_facts'));
worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges'));
worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight'));
@@ -1982,7 +2044,7 @@ export async function registerBuiltinHandlers(
// window / defer behavior. On LockUnavailableError (the routine cycle holds
// the per-source lock) the job completes `{ deferred: true }` and retries
// next tick instead of failing — cooperative interleave (CODEX accepted).
worker.register('extract-atoms-drain', async (job) => {
registerBuiltinJob(worker, engine, 'extract-atoms-drain', async (job) => {
const { runExtractAtomsDrainForSource } = await import('../core/cycle/extract-atoms-drain.ts');
const { LockUnavailableError } = await import('../core/db-lock.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
@@ -2010,7 +2072,7 @@ export async function registerBuiltinHandlers(
// Cost-bounded via D6 ($10/job BudgetTracker) + D19 (source-level cooldown
// + 24h rolling cap, gated at submit time). NOT in PROTECTED_JOB_NAMES —
// embedding-only spend, no API-by-the-minute risk like subagent.
worker.register('embed-backfill', async (job) => {
registerBuiltinJob(worker, engine, 'embed-backfill', async (job) => {
const { makeEmbedBackfillHandler } = await import('../core/minions/handlers/embed-backfill.ts');
return await makeEmbedBackfillHandler(engine)(job);
});
@@ -2031,7 +2093,7 @@ export async function registerBuiltinHandlers(
// (LLM-bearing). Two-gate consent enforced at the handler boundary:
// refuses to run unless takes.bootstrap_enabled config is true, even
// when allowProtectedSubmit was set at queue.add time.
worker.register('extract-takes-from-pages', async (job) => {
registerBuiltinJob(worker, engine, 'extract-takes-from-pages', async (job) => {
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
const data = (job.data ?? {}) as { sourceId?: string; maxPages?: number };
const bootstrapCfg = await engine.getConfig('takes.bootstrap_enabled');
@@ -2058,7 +2120,7 @@ export async function registerBuiltinHandlers(
// remediation pipeline. Wraps runEmbedCore with stale + catchUp + the
// priority/batchSize the recommendation supplies. NOT in
// PROTECTED_JOB_NAMES (embedding spend only).
worker.register('embed-catch-up', async (job) => {
registerBuiltinJob(worker, engine, 'embed-catch-up', async (job) => {
const { runEmbedCore } = await import('./embed.ts');
const data = (job.data ?? {}) as {
sourceId?: string;
+224
View File
@@ -0,0 +1,224 @@
/**
* gbrain maintain conservative self-healing maintenance.
*
* This command automates the safe parts of the operator runbook:
* - stale link/timeline extraction
* - stale per-source dream cycles when doctor reports cycle_freshness
*
* It deliberately does NOT mutate source files, apply schema-pack upgrades, or
* invent semantic hub links. Those need review or a separate command with an
* auditable proposal surface.
*/
import { existsSync } from 'fs';
import type { BrainEngine } from '../core/engine.ts';
import type { BrainHealth } from '../core/types.ts';
import { buildChecks, computeDoctorReport, type DoctorReport, type Check } from './doctor.ts';
import { extractStaleFromDB } from './extract.ts';
import { runCycle, type CycleReport } from '../core/cycle.ts';
type ActionStatus = 'ok' | 'would_apply' | 'applied' | 'blocked' | 'skipped';
export interface MaintenanceAction {
name: string;
status: ActionStatus;
message: string;
details?: Record<string, unknown>;
}
export interface MaintainOptions {
json: boolean;
safe: boolean;
dryRun: boolean;
help: boolean;
}
export interface MaintainReport {
mode: 'dry-run' | 'safe';
before: {
health: BrainHealth;
doctor: DoctorReport;
};
actions: MaintenanceAction[];
after: {
health: BrainHealth;
doctor: DoctorReport;
};
}
export function parseMaintainArgs(args: string[]): MaintainOptions {
const safe = args.includes('--safe');
return {
json: args.includes('--json'),
safe,
dryRun: args.includes('--dry-run') || !safe,
help: args.includes('--help') || args.includes('-h'),
};
}
export function extractCycleFreshnessSourceIds(checks: Check[]): string[] {
const ids = new Set<string>();
for (const check of checks) {
if (check.name !== 'cycle_freshness' || check.status === 'ok') continue;
const re = /Source '([^']+)' last cycled/g;
for (const match of check.message.matchAll(re)) {
const id = match[1]?.trim();
if (id) ids.add(id);
}
}
return [...ids].sort();
}
async function buildDoctorReport(engine: BrainEngine): Promise<DoctorReport> {
const checks = await buildChecks(engine, ['--json', '--scope=brain']);
return computeDoctorReport(checks);
}
async function runStaleExtraction(
engine: BrainEngine,
beforeHealth: BrainHealth,
dryRun: boolean,
): Promise<MaintenanceAction> {
if (beforeHealth.stale_pages <= 0) {
return { name: 'extract_stale', status: 'ok', message: 'No stale pages.' };
}
if (dryRun) {
return {
name: 'extract_stale',
status: 'would_apply',
message: `Would run DB-backed stale extraction for ${beforeHealth.stale_pages} page(s).`,
details: { stale_pages: beforeHealth.stale_pages },
};
}
const result = await extractStaleFromDB(engine, {
dryRun: false,
jsonMode: false,
includeFrontmatter: false,
catchUp: false,
});
return {
name: 'extract_stale',
status: 'applied',
message: `Processed ${result.pagesProcessed} stale page(s); ${result.staleRemaining} remain.`,
details: {
links_created: result.linksCreated,
timeline_created: result.timelineCreated,
pages_processed: result.pagesProcessed,
stale_remaining: result.staleRemaining,
},
};
}
async function runCycleFreshnessMaintenance(
engine: BrainEngine,
beforeDoctor: DoctorReport,
dryRun: boolean,
): Promise<MaintenanceAction[]> {
const sourceIds = extractCycleFreshnessSourceIds(beforeDoctor.checks);
if (sourceIds.length === 0) {
return [{ name: 'cycle_freshness', status: 'ok', message: 'All sources cycled recently.' }];
}
if (dryRun) {
return sourceIds.map((sourceId) => ({
name: 'cycle_freshness',
status: 'would_apply',
message: `Would run source-scoped dream cycle for ${sourceId}.`,
details: { source_id: sourceId },
}));
}
const sources = await engine.listAllSources();
const actions: MaintenanceAction[] = [];
for (const sourceId of sourceIds) {
const source = sources.find((s) => s.id === sourceId);
const localPath = source?.local_path ?? null;
const brainDir = localPath && existsSync(localPath) ? localPath : null;
const report: CycleReport = await runCycle(engine, {
brainDir,
dryRun: false,
pull: false,
sourceId,
});
actions.push({
name: 'cycle_freshness',
status: report.status === 'failed' ? 'blocked' : 'applied',
message: `Ran source-scoped dream cycle for ${sourceId}: ${report.status}.`,
details: {
source_id: sourceId,
brain_dir: brainDir,
cycle_status: report.status,
phases: report.phases.map((p) => ({ phase: p.phase, status: p.status })),
},
});
}
return actions;
}
export async function runMaintain(engine: BrainEngine, args: string[]): Promise<MaintainReport | void> {
const opts = parseMaintainArgs(args);
if (opts.help) {
console.log(`Usage: gbrain maintain [--safe] [--dry-run] [--json]
Conservative self-healing maintenance.
Modes:
--dry-run Preview safe actions without writes. Default when --safe is absent.
--safe Apply safe actions: stale extraction and source cycle freshness.
--json Emit a structured before/action/after report.
Not auto-applied:
source-file frontmatter fixes, schema-pack upgrades, atom-pack changes,
semantic hub-link guesses, and destructive cleanup.
`);
return;
}
const beforeHealth = await engine.getHealth();
const beforeDoctor = await buildDoctorReport(engine);
const actions: MaintenanceAction[] = [];
actions.push(await runStaleExtraction(engine, beforeHealth, opts.dryRun));
actions.push(...await runCycleFreshnessMaintenance(engine, beforeDoctor, opts.dryRun));
const afterHealth = await engine.getHealth();
const afterDoctor = await buildDoctorReport(engine);
const report: MaintainReport = {
mode: opts.dryRun ? 'dry-run' : 'safe',
before: { health: beforeHealth, doctor: beforeDoctor },
actions,
after: { health: afterHealth, doctor: afterDoctor },
};
if (opts.json) {
console.log(JSON.stringify(report, null, 2));
} else {
printMaintainReport(report);
}
return report;
}
function printMaintainReport(report: MaintainReport): void {
console.log(`GBrain maintain (${report.mode})`);
console.log(
`Before: brain_score=${Math.round(report.before.health.brain_score)}/100 ` +
`stale=${report.before.health.stale_pages} islands=${report.before.health.orphan_pages} ` +
`doctor=${report.before.doctor.status}`,
);
for (const action of report.actions) {
console.log(` ${action.status}: ${action.name}${action.message}`);
}
console.log(
`After: brain_score=${Math.round(report.after.health.brain_score)}/100 ` +
`stale=${report.after.health.stale_pages} islands=${report.after.health.orphan_pages} ` +
`doctor=${report.after.doctor.status}`,
);
if (report.mode === 'dry-run') {
console.log('Run `gbrain maintain --safe` to apply safe actions.');
}
}
+10 -55
View File
@@ -15,6 +15,11 @@
import type { BrainEngine } from '../core/engine.ts';
import { createProgress, startHeartbeat } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import {
shouldExcludeFromOrphanReporting,
loadOrphanPolicyOverrides,
type OrphanPolicyOverrides,
} from '../core/orphan-policy.ts';
// --- Types ---
@@ -32,65 +37,14 @@ export interface OrphanResult {
excluded: number;
}
// --- Filter constants ---
/** Slug suffixes that are always auto-generated root files */
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
/** Page slugs that are pseudo-pages by convention */
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
/** Slug segment that marks raw sources */
const RAW_SEGMENT = '/raw/';
/** Slug prefixes where no inbound links is expected */
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'openclaw/config/',
];
/** First slug segments where no inbound links is expected */
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
]);
// --- Filter logic ---
/**
* Returns true if a slug should be excluded from orphan reporting by default.
* These are pages where having no inbound links is expected / not a content problem.
*/
export function shouldExclude(slug: string): boolean {
// Pseudo-pages (exact match)
if (PSEUDO_SLUGS.has(slug)) return true;
// Auto-generated suffix patterns
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
// Raw source slugs
if (slug.includes(RAW_SEGMENT)) return true;
// Deny-prefix slugs
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
// First-segment exclusions
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
return false;
export function shouldExclude(slug: string, overrides?: OrphanPolicyOverrides): boolean {
return shouldExcludeFromOrphanReporting(slug, overrides);
}
/**
@@ -156,6 +110,7 @@ export async function findOrphans(
let allOrphans: { slug: string; title: string; domain: string | null }[];
let total: number;
let excludedAll: number;
const overrides = includePseudo ? undefined : await loadOrphanPolicyOverrides(engine);
try {
allOrphans = await engine.findOrphanPages(
sourceIds ? { sourceIds } : sourceId ? { sourceId } : undefined,
@@ -184,7 +139,7 @@ export async function findOrphans(
total = liveRows.length;
excludedAll = includePseudo
? 0
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug) ? 1 : 0), 0);
: liveRows.reduce((n, r) => n + (shouldExclude(r.slug, overrides) ? 1 : 0), 0);
} finally {
stopHb();
progress.finish();
@@ -192,7 +147,7 @@ export async function findOrphans(
const filtered = includePseudo
? allOrphans
: allOrphans.filter(row => !shouldExclude(row.slug));
: allOrphans.filter(row => !shouldExclude(row.slug, overrides));
const orphans: OrphanPage[] = filtered.map(row => ({
slug: row.slug,
+43 -14
View File
@@ -7,6 +7,7 @@
import { listRecipes, getRecipe } from '../core/ai/recipes/index.ts';
import { configureGateway, embedOne, isAvailable as gwIsAvailable, chat as gwChat } from '../core/ai/gateway.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { probeOllama, probeLMStudio } from '../core/ai/probes.ts';
import { loadConfig } from '../core/config.ts';
import { AIConfigError, AITransientError } from '../core/ai/errors.ts';
@@ -33,16 +34,19 @@ interface ProviderOption {
function configureFromEnv(): void {
const config = loadConfig();
configureGateway({
embedding_model: config?.embedding_model,
embedding_dimensions: config?.embedding_dimensions,
expansion_model: config?.expansion_model,
chat_model: config?.chat_model,
chat_fallback_chain: config?.chat_fallback_chain,
base_urls: config?.provider_base_urls,
provider_chat_options: config?.provider_chat_options,
env: { ...process.env },
});
// Route through buildGatewayConfig — the single ownership seam that folds
// file-plane API keys (openrouter_api_key, zeroentropy_api_key, ...) into
// the gateway env — instead of hand-assembling AIGatewayConfig field by
// field. Hand-building it here let this diagnostic report a provider as
// missing env even when ~/.gbrain/config.json had it and the real gateway
// path resolved it fine (#2728). Pre-init (no file-plane config yet) falls
// back to a bare env passthrough so the command still works before
// `gbrain init`.
if (config) {
configureGateway(buildGatewayConfig(config));
return;
}
configureGateway({ env: { ...process.env } });
}
export function envReady(recipe: Recipe, env: NodeJS.ProcessEnv = process.env): boolean {
@@ -137,7 +141,12 @@ EXAMPLES
}
function runList(_args: string[]): void {
console.log(formatRecipeTable(listRecipes()));
// Same env the gateway actually sees (file-plane keys folded in), not bare
// process.env — keeps this table's STATUS column honest with what
// `providers test` (and the real init/gateway path) would report.
const cfg = loadConfig();
const env = cfg ? buildGatewayConfig(cfg).env : process.env;
console.log(formatRecipeTable(listRecipes(), env));
}
async function runTest(args: string[]): Promise<void> {
@@ -164,8 +173,18 @@ async function runTest(args: string[]): Promise<void> {
// the divergence at the top of the test so the recovery experience
// doesn't repeat the bug-reporter's "providers test ✓ but import still
// broken" trap.
//
// #2863: `cfg` is lifted out of the try block (not just used for the
// warning) so the configureGateway calls below can reuse it. Before this
// fix, the --model override only forwarded embedding_model/chat_model +
// env, dropping config.provider_base_urls entirely — a probe against a
// custom endpoint (e.g. a regional DashScope base URL) would silently
// fall back to the recipe's hardcoded default endpoint and fail with a
// misleading "Incorrect API key" error even though the key was valid for
// the configured endpoint.
let cfg: ReturnType<typeof loadConfig> | null = null;
try {
const cfg = loadConfig();
cfg = loadConfig();
const configuredModel = tpArg === 'embedding' ? cfg?.embedding_model : cfg?.chat_model;
if (!configuredModel) {
console.error(
@@ -181,17 +200,27 @@ async function runTest(args: string[]): Promise<void> {
}
} catch { /* loadConfig throws when no brain configured — first-time install path; the no-config branch above handles it. */ }
// Reuse the SAME resolver the production path uses (buildGatewayConfig —
// also used by cli.ts#connectEngine and init-embed-check.ts) so the probe
// sees the identical base_urls / provider_chat_options / folded API keys
// that a real `gbrain import`/`gbrain query` call would. Only the
// touchpoint's model (+ embedding dims) is overridden on top, so an
// isolated `--model` probe still targets exactly the requested model —
// it just resolves that model's endpoint the way the brain actually
// would. Falls back to bare env when no brain is configured yet (cfg is
// null on first-time install, matching the old behavior for that case).
const baseGatewayConfig = cfg ? buildGatewayConfig(cfg) : { env: { ...process.env } };
if (tpArg === 'embedding') {
const dims = recipe?.touchpoints.embedding?.default_dims ?? 1536;
configureGateway({
...baseGatewayConfig,
embedding_model: modelArg,
embedding_dimensions: dims,
env: { ...process.env },
});
} else {
configureGateway({
...baseGatewayConfig,
chat_model: modelArg,
env: { ...process.env },
});
}
void modelId; // intentionally unused but preserved for readability
+282
View File
@@ -0,0 +1,282 @@
/**
* `gbrain reindex-search-vector` recreate FTS trigger functions and
* backfill existing rows under the language configured via
* GBRAIN_FTS_LANGUAGE.
*
* Why this command exists: schema migration v123 (configurable_fts_language)
* stamps the trigger functions with the configured language at first apply.
* After that, changing the env var has no effect on the write side because
* v123 already shows as "applied" the migrations runner will skip it.
* This command is the documented escape hatch: it re-runs the same
* recreate-and-backfill logic v123 uses, gated on an explicit user
* action so the operation is intentional and visible (writes touch
* every row in pages and content_chunks).
*
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
* the same trigger function bodies and the same tokenized vectors.
*
* Flags:
* --dry-run Show what would happen, exit 0 without touching DB.
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
* --json Machine-readable result envelope. Does NOT imply --yes.
*
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
* so a large brain never holds one giant row lock, and streams progress
* through the shared reporter (stderr; stdout stays clean for --json).
*
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
* expect ~5-15s depending on Postgres CPU and content size.
*/
import type { BrainEngine } from '../core/engine.ts';
import { getFtsLanguage } from '../core/fts-language.ts';
import { createInterface } from 'readline';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
export interface ReindexSearchVectorOpts {
dryRun?: boolean;
yes?: boolean;
json?: boolean;
}
export interface ReindexSearchVectorResult {
status: 'ok' | 'dry_run' | 'cancelled';
language: string;
pagesUpdated: number;
chunksUpdated: number;
triggersRecreated: number;
durationMs: number;
}
interface CountRow {
pages: number;
chunks: number;
}
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
export const BACKFILL_BATCH_SIZE = 5000;
/**
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
* the shared progress reporter after each batch. Terminates when a batch
* returns fewer rows than the batch size (or none).
*/
async function batchedBackfill(
engine: BrainEngine,
table: 'pages' | 'content_chunks',
setClause: string,
tick: (n: number) => void
): Promise<void> {
let cursor = 0;
for (;;) {
const rows = await engine.executeRaw<{ id: number }>(`
UPDATE ${table} SET ${setClause}
WHERE id IN (
SELECT id FROM ${table}
WHERE search_vector IS NOT NULL AND id > ${cursor}
ORDER BY id
LIMIT ${BACKFILL_BATCH_SIZE}
)
RETURNING id
`);
if (rows.length === 0) break;
tick(rows.length);
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
if (rows.length < BACKFILL_BATCH_SIZE) break;
}
}
/**
* Programmatic entrypoint takes a typed opts object. Used by tests and
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
* defined at the bottom of this file.
*/
export async function runReindexSearchVector(
engine: BrainEngine,
opts: ReindexSearchVectorOpts
): Promise<ReindexSearchVectorResult> {
const lang = getFtsLanguage();
const startedAt = Date.now();
// Inventory: how many rows will the backfill touch?
const counts = await engine.executeRaw<CountRow>(
`SELECT
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
);
const pagesCount = counts[0]?.pages ?? 0;
const chunksCount = counts[0]?.chunks ?? 0;
if (opts.dryRun) {
const result: ReindexSearchVectorResult = {
status: 'dry_run',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
}
return result;
}
// Confirm unless --yes. --json does NOT bypass the gate — a machine
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
if (!opts.yes) {
if (!process.stdin.isTTY) {
if (opts.json) {
console.log(JSON.stringify({
error: {
class: 'ConfirmationRequired',
code: 'reindex_requires_yes',
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
hint: 'Pass --yes to proceed, or --dry-run to preview.',
},
language: lang,
pages: pagesCount,
chunks: chunksCount,
}));
} else {
console.error('Refusing to run without --yes in non-TTY environment.');
}
process.exit(2);
}
const rl = createInterface({ input: process.stdin, output: process.stdout });
const answer = await new Promise<string>(resolve => {
rl.question(
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
resolve
);
});
rl.close();
if (!/^y(es)?$/i.test(answer.trim())) {
const result: ReindexSearchVectorResult = {
status: 'cancelled',
language: lang,
pagesUpdated: 0,
chunksUpdated: 0,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log('Cancelled.');
}
return result;
}
}
// Recreate trigger functions. The strings are intentionally identical to
// the v124 migration body — keeping them in lockstep is the contract.
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
// hardening from every brain that runs this command.
//
// #2704: compiled_truth (the unbounded whole-page body) is deliberately
// NOT indexed here — it overflows Postgres's 1MB tsvector cap on large
// pages, and content_chunks.search_vector (populated separately, chunk-
// grain, well under the cap) is what searchKeyword() actually queries.
// See migrate.ts's v124 for the full rationale; keep this copy in sync.
const recreatePagesFn = `
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
const recreateChunksFn = `
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
await engine.executeRaw(recreatePagesFn);
await engine.executeRaw(recreateChunksFn);
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
// direct vector compute since the column itself is what we want.
progress.start('reindex_search_vector.pages', pagesCount);
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
progress.finish();
progress.start('reindex_search_vector.chunks', chunksCount);
await batchedBackfill(
engine,
'content_chunks',
`search_vector =
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
n => progress.tick(n)
);
progress.finish();
const result: ReindexSearchVectorResult = {
status: 'ok',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 2,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
}
return result;
}
/**
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
*
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
*/
export async function runReindexSearchVectorCli(
engine: BrainEngine,
args: string[]
): Promise<void> {
const dryRun = args.includes('--dry-run');
const yes = args.includes('--yes') || args.includes('-y');
const json = args.includes('--json');
await runReindexSearchVector(engine, { dryRun, yes, json });
}
+18 -12
View File
@@ -105,13 +105,19 @@ function printHelp(): void {
async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>, args: string[]): Promise<void> {
const { json, timeoutMs } = parseFlags(args);
let submitted: { id: number; name: string; state: string };
// submit_job / get_job return the MinionJob row verbatim — the lifecycle
// field is `status` (src/core/minions/types.ts), not `state`. Reading
// `state` here made every poll see `undefined`, so the terminal check
// never matched and ping always exhausted its timeout (exit 1) even when
// the cycle completed. The ping's own JSON *output* keys (`state`,
// `last_state`) are kept as-is for consumers.
let submitted: { id: number; name: string; status: string };
try {
const res = await callRemoteTool(config, 'submit_job', {
name: 'autopilot-cycle',
data: { phases: ['sync', 'extract', 'embed'] },
});
submitted = unpackToolResult<{ id: number; name: string; state: string }>(res);
submitted = unpackToolResult<{ id: number; name: string; status: string }>(res);
} catch (e) {
return failPing(e, json);
}
@@ -122,43 +128,43 @@ async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>,
const startMs = Date.now();
let attempt = 0;
let lastState = submitted.state;
let lastState = submitted.status;
while (Date.now() - startMs < timeoutMs) {
const elapsed = Date.now() - startMs;
const intervalMs = elapsed < 30_000 ? 1_000 : elapsed < 5 * 60_000 + 30_000 ? 5_000 : 10_000;
await sleep(intervalMs);
attempt++;
let job: { id: number; state: string; failed_reason?: string };
let job: { id: number; status: string; failed_reason?: string };
try {
const res = await callRemoteTool(config, 'get_job', { id: submitted.id });
job = unpackToolResult<{ id: number; state: string; failed_reason?: string }>(res);
job = unpackToolResult<{ id: number; status: string; failed_reason?: string }>(res);
} catch (e) {
// Network blip mid-poll: log and keep going. Surface only if persistent.
if (!json) console.error(` poll #${attempt} failed (${e instanceof Error ? e.message : String(e)}); continuing...`);
continue;
}
if (job.state !== lastState) {
lastState = job.state;
if (!json) console.error(` job #${submitted.id}${job.state}`);
if (job.status !== lastState) {
lastState = job.status;
if (!json) console.error(` job #${submitted.id}${job.status}`);
}
const terminal = ['completed', 'failed', 'dead', 'cancelled'];
if (terminal.includes(job.state)) {
const ok = job.state === 'completed';
if (terminal.includes(job.status)) {
const ok = job.status === 'completed';
if (json) {
console.log(JSON.stringify({
status: ok ? 'success' : 'error',
job_id: submitted.id,
state: job.state,
state: job.status,
...(job.failed_reason ? { failed_reason: job.failed_reason } : {}),
elapsed_ms: Date.now() - startMs,
}));
} else {
console.log(ok
? `\nautopilot-cycle complete (${Math.round((Date.now() - startMs) / 1000)}s).`
: `\nautopilot-cycle ended ${job.state}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
: `\nautopilot-cycle ended ${job.status}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
}
process.exit(ok ? 0 : 1);
}
+3 -7
View File
@@ -48,7 +48,7 @@ import {
} from '../core/schema-pack/index.ts';
import type { SchemaPackManifest, PackPrimitive } from '../core/schema-pack/manifest-v1.ts';
import { PACK_PRIMITIVES } from '../core/schema-pack/manifest-v1.ts';
import { gbrainPath, loadConfig, configPath } from '../core/config.ts';
import { gbrainPath, loadConfig, configPath, toEngineConfig } from '../core/config.ts';
export async function runSchema(args: string[]): Promise<void> {
const sub = args[0];
@@ -434,16 +434,12 @@ function parseFlags(args: string[]): ParsedFlags {
async function withConnectedEngine<T>(fn: (engine: import('../core/engine.ts').BrainEngine) => Promise<T>): Promise<T> {
const { createEngine } = await import('../core/engine-factory.ts');
const cfg = loadConfig() ?? {};
const engineKind = (cfg as { engine?: string }).engine === 'postgres' ? 'postgres' : 'pglite';
const cfg = loadConfig() ?? { engine: 'pglite' as const };
// PR #1321 (closed) defensive fix retained: build the EngineConfig once and
// pass it to BOTH createEngine and engine.connect. The factory captures
// config at construction; explicit re-pass at connect() is defense in depth
// against future engine implementations that read URL from connect-time.
const connectConfig: import('../core/types.ts').EngineConfig = {
engine: engineKind,
database_url: (cfg as { database_url?: string }).database_url,
};
const connectConfig = toEngineConfig(cfg);
const engine = await createEngine(connectConfig);
await engine.connect(connectConfig);
try {
+99
View File
@@ -22,6 +22,7 @@ import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import { mcpAuthRouter } from '@modelcontextprotocol/sdk/server/auth/router.js';
import { requireBearerAuth } from '@modelcontextprotocol/sdk/server/auth/middleware/bearerAuth.js';
import { OAuthTokenRevocationRequestSchema } from '@modelcontextprotocol/sdk/shared/auth.js';
import type { BrainEngine } from '../core/engine.ts';
import { operations, OperationError } from '../core/operations.ts';
import type { OperationContext, AuthInfo } from '../core/operations.ts';
@@ -37,6 +38,7 @@ import { VERSION } from '../version.ts';
import * as db from '../core/db.ts';
import { sqlQueryForEngine, executeRawJsonb } from '../core/sql-query.ts';
import { MinionQueue } from '../core/minions/queue.ts';
import { isRetryableError } from '../core/retry-matcher.ts';
import {
computeContentHash,
validateIngestionEvent,
@@ -745,6 +747,93 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
}
});
// The SDK's /revoke handler compares the presented secret with
// client.client_secret as plaintext. GBrain stores only a SHA-256 hash, so
// confidential clients need the same hash-aware validation used above for
// authorization_code and refresh_token exchanges. Public clients present no
// secret and continue through to the SDK's PKCE-compatible handler.
app.post('/revoke', ccRateLimiter, express.urlencoded({ extended: false }), async (req, res, next) => {
res.setHeader('Cache-Control', 'no-store');
const rawClientId: unknown = req.body?.client_id;
const rawBodySecret: unknown = req.body?.client_secret;
const authHeader = (req.headers.authorization ?? '').toString();
// RFC 6749 §2.3: one client-authentication method per request. Reject
// duplicates/arrays from express.urlencoded rather than letting them reach
// hashToken() as non-strings and become a misleading invalid_client error.
const hasBasicAuth = /^Basic\b/i.test(authHeader);
if (
(rawClientId !== undefined && typeof rawClientId !== 'string') ||
(rawBodySecret !== undefined && typeof rawBodySecret !== 'string') ||
(hasBasicAuth && (rawClientId !== undefined || rawBodySecret !== undefined))
) {
res.status(400).json({ error: 'invalid_request', error_description: 'Malformed or mixed client authentication' });
return;
}
let clientId = typeof rawClientId === 'string' ? rawClientId : undefined;
let presentedSecret = typeof rawBodySecret === 'string' && rawBodySecret.length > 0
? rawBodySecret
: undefined;
if (hasBasicAuth) {
try {
const match = authHeader.match(/^Basic\s+([^\s]+)$/i);
if (!match) throw new Error('Malformed Basic authentication');
const decoded = Buffer.from(match[1], 'base64').toString('utf8');
const idx = decoded.indexOf(':');
if (idx < 1) throw new Error('Malformed Basic authentication');
clientId = decodeURIComponent(decoded.slice(0, idx).replace(/\+/g, ' '));
presentedSecret = decodeURIComponent(decoded.slice(idx + 1).replace(/\+/g, ' '));
if (!presentedSecret) throw new Error('Malformed Basic authentication');
} catch {
res.setHeader('WWW-Authenticate', 'Basic realm="gbrain"');
res.status(401).json({ error: 'invalid_client', error_description: 'Invalid client' });
return;
}
}
if (!clientId || !presentedSecret) return next();
const parsedRequest = OAuthTokenRevocationRequestSchema.safeParse(req.body);
if (!parsedRequest.success || parsedRequest.data.token.length === 0) {
res.status(400).json({ error: 'invalid_request', error_description: 'Valid token required' });
return;
}
let client;
try {
client = await oauthProvider.verifyConfidentialClientSecret(clientId, presentedSecret);
} catch (e) {
const msg = e instanceof Error ? e.message : '';
if (msg === 'Invalid client' || msg === 'Client has been revoked') {
if (hasBasicAuth) res.setHeader('WWW-Authenticate', 'Basic realm="gbrain"');
res.status(401).json({ error: 'invalid_client', error_description: 'Invalid client' });
return;
}
console.error('[serve-http] revoke client verification failed:', msg || 'Unknown error');
const retryable = isRetryableError(e);
res.status(retryable ? 503 : 500).json({
error: retryable ? 'temporarily_unavailable' : 'server_error',
error_description: retryable ? 'Token revocation temporarily unavailable' : 'Token revocation failed',
});
return;
}
try {
await oauthProvider.revokeToken(client, parsedRequest.data);
// RFC 7009 §2.2: successful revocation, including an unknown token, is 200.
res.status(200).end();
} catch (e) {
const msg = e instanceof Error ? e.message : 'Unknown error';
console.error('[serve-http] token revocation failed:', msg);
const retryable = isRetryableError(e);
res.status(retryable ? 503 : 500).json({
error: retryable ? 'temporarily_unavailable' : 'server_error',
error_description: retryable ? 'Token revocation temporarily unavailable' : 'Token revocation failed',
});
}
});
// ---------------------------------------------------------------------------
// MCP SDK Auth Router (OAuth endpoints)
// ---------------------------------------------------------------------------
@@ -796,6 +885,16 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
if (body?.grant_types_supported && !body.grant_types_supported.includes('client_credentials')) {
body.grant_types_supported.push('client_credentials');
}
if (body?.token_endpoint_auth_methods_supported) {
for (const method of ['client_secret_basic', 'none']) {
if (!body.token_endpoint_auth_methods_supported.includes(method)) {
body.token_endpoint_auth_methods_supported.push(method);
}
}
}
if (body?.revocation_endpoint_auth_methods_supported && !body.revocation_endpoint_auth_methods_supported.includes('client_secret_basic')) {
body.revocation_endpoint_auth_methods_supported.push('client_secret_basic');
}
return origJson(body);
};
}
+34 -7
View File
@@ -7,7 +7,9 @@
* full story.
*
* Subcommands:
* gbrain sources add <id> --path <path> [--name <display>] [--federated|--no-federated]
* gbrain sources add <id> --path <path> [--name <display>] [--federated|--no-federated] [--force]
* --path must be a git-initialized repo (files committed,
* not just present) #2707. --force skips the check.
* gbrain sources list [--json]
* gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
* gbrain sources rename <id> <new-name>
@@ -107,7 +109,7 @@ async function fetchSource(engine: BrainEngine, id: string): Promise<SourceRow |
async function countPages(engine: BrainEngine, sourceId: string): Promise<number> {
const rows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1 AND deleted_at IS NULL`,
[sourceId],
);
return rows[0]?.n ?? 0;
@@ -120,7 +122,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
if (!id) {
console.error(
'Usage: gbrain sources add <id> [--path <path> | --url <https-url>] ' +
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>]',
'[--name <display>] [--federated|--no-federated] [--clone-dir <path>] [--force]',
);
process.exit(2);
}
@@ -132,6 +134,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
let cloneDir: string | undefined;
let patFile: string | undefined;
let noHarden = false;
let force = false;
for (let i = 1; i < args.length; i++) {
const a = args[i];
@@ -143,6 +146,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
if (a === '--clone-dir') { cloneDir = args[++i]; continue; }
if (a === '--pat-file') { patFile = args[++i]; continue; }
if (a === '--no-harden') { noHarden = true; continue; }
if (a === '--force') { force = true; continue; }
console.error(`Unknown flag: ${a}`);
process.exit(2);
}
@@ -162,6 +166,7 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
remoteUrl,
federated,
cloneDir,
force,
});
// Topology A discovery: if the just-added source carries a brain-resident
@@ -502,6 +507,19 @@ async function runArchive(engine: BrainEngine, args: string[]): Promise<void> {
const result = await softDeleteSource(engine, id);
if (!result) {
// #2792: softDeleteSource returns null both for "not found" (handled by
// the impact check above) and for "already archived" (UPDATE matched no
// `archived = false` row). Distinguish them: already-archived is a
// friendly idempotent no-op, not a reasonless failure.
const rows = await engine.executeRaw<{ archived: boolean }>(
`SELECT archived FROM sources WHERE id = $1`,
[id],
);
if (rows[0]?.archived) {
console.log(`Source "${id}" is already archived — nothing to do.`);
console.log(` 'gbrain sources archived' shows its purge expiry; 'gbrain sources restore ${id}' un-archives it.`);
return;
}
console.error(`Failed to archive source "${id}".`);
process.exit(4);
}
@@ -1134,7 +1152,8 @@ async function runAudit(engine: BrainEngine, args: string[]): Promise<void> {
continue;
}
if (stat.isDirectory()) {
if (pruneDir(entry, dir)) continue;
// pruneDir returns true = descend, false = prune (see core/sync.ts).
if (!pruneDir(entry, dir)) continue;
walk(full);
} else if (entry.endsWith('.md')) {
files.push(full);
@@ -1164,7 +1183,14 @@ async function runAudit(engine: BrainEngine, args: string[]): Promise<void> {
// frontmatter.type and estimates per-page segment count from body
// bytes. Estimated per-segment Sonnet cost is a rough heuristic
// (~2000 in + 500 out tokens at $3/MTok in + $15/MTok out ≈ $0.013).
const FACTS_BACKFILL_ALLOWED = ['conversation', 'meeting', 'slack', 'email'];
const FACTS_BACKFILL_ALLOWED = [
'conversation',
'meeting',
'slack',
'email',
'imessage',
'imessage-daily',
];
const FACTS_BACKFILL_CHARS_PER_SEGMENT = 6500; // matches SEGMENT_TEXT_CHAR_LIMIT
const FACTS_BACKFILL_USD_PER_SEGMENT = 0.013;
let factsBackfillPages = 0;
@@ -1354,8 +1380,9 @@ function printHelp(): void {
console.log(`gbrain sources — manage multi-source brain configuration (v0.26.5)
Subcommands:
add <id> --path <p> [--name <n>] [--federated|--no-federated]
Register a new source.
add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
Register a new source. --path must be a git repo
with committed files; --force skips that check.
list [--json] List registered sources with page counts.
remove <id> [--confirm-destructive] [--dry-run]
Permanently delete a source and all its data.
+639 -61
View File
File diff suppressed because it is too large Load Diff
+12 -6
View File
@@ -101,12 +101,18 @@ async function getPageId(engine: BrainEngine, slug: string, sourceId?: string):
return rows[0].id;
}
async function resolveTakesSourceId(engine: BrainEngine): Promise<string | undefined> {
try {
return await resolveSourceId(engine, null);
} catch {
return undefined;
}
// Fail-closed (#2698 residual, TODOS.md): `resolveSourceId` only ever
// throws when a source WAS explicitly in play — an invalid or
// unregistered `GBRAIN_SOURCE`, a `.gbrain-source` dotfile pointing at a
// source that doesn't exist, or a genuine DB error — never for "nothing
// configured" (that path resolves cleanly to the seeded `'default'`
// source, tier 6 of resolveSourceId). Swallowing those errors here used
// to fall back to the unscoped slug-only page lookup, silently
// reintroducing the pre-#2698 cross-source write bug whenever resolution
// merely errored instead of resolving cleanly. Let it propagate so the
// write is blocked instead of silently unscoped.
async function resolveTakesSourceId(engine: BrainEngine): Promise<string> {
return resolveSourceId(engine, null);
}
function readBodyOrEmpty(path: string): string {
+52
View File
@@ -90,6 +90,38 @@ export function isValidOpenAITextEmbedding3Dim(modelId: string, dims: number): b
return Number.isInteger(dims) && dims >= 1 && dims <= max;
}
// NVIDIA NIM hosted embedding models use asymmetric input_type values. Most
// emit fixed natural dimensions, but llama-nemotron-embed-1b-v2 accepts
// Matryoshka-style dimension overrides (e.g. matching an existing 1280d
// brain column without re-embedding through another provider).
const NVIDIA_EMBEDDING_DIMS: Record<string, number> = {
'nvidia/nv-embedqa-e5-v5': 1024,
'nvidia/llama-nemotron-embed-1b-v2': 2048,
'nvidia/nv-embed-v1': 4096,
'nvidia/nv-embedcode-7b-v1': 4096,
};
const NVIDIA_EMBEDDING_DIM_OPTIONS: Record<string, number[]> = {
'nvidia/llama-nemotron-embed-1b-v2': [1024, 1280, 1536, 2048],
};
export function isNvidiaEmbeddingModel(modelId: string): boolean {
return modelId in NVIDIA_EMBEDDING_DIMS;
}
export function nvidiaEmbeddingDim(modelId: string): number | undefined {
return NVIDIA_EMBEDDING_DIMS[modelId];
}
export function nvidiaEmbeddingDimOptions(modelId: string): number[] | undefined {
return NVIDIA_EMBEDDING_DIM_OPTIONS[modelId];
}
export function supportsNvidiaEmbeddingDimension(modelId: string, dims: number): boolean {
const options = nvidiaEmbeddingDimOptions(modelId);
return !!options && options.includes(dims);
}
/**
* Build the providerOptions blob for embedMany() that pins output dimensions.
*
@@ -194,6 +226,17 @@ export function dimsProviderOptions(
},
};
}
// NVIDIA NIM hosted embeddings are OpenAI-compatible but require
// asymmetric input_type. Use passage for indexing/document-side vectors
// and query for search-side vectors. Only llama-nemotron-embed-1b-v2
// supports a dimensions override; fixed-dim models reject it.
if (isNvidiaEmbeddingModel(modelId)) {
const opts: Record<string, any> = {
input_type: inputType === 'query' ? 'query' : 'passage',
};
if (supportsNvidiaEmbeddingDimension(modelId, dims)) opts.dimensions = dims;
return { openaiCompatible: opts };
}
// OpenAI text-embedding-3 family on the openai-compatible adapter
// (Azure OpenAI hosts these via its OpenAI-compatible /embeddings
// endpoint). The provider defaults to the model's native size (3072
@@ -220,6 +263,15 @@ export function dimsProviderOptions(
if (modelId === 'text-embedding-v3' || modelId === 'embedding-3') {
return { openaiCompatible: { dimensions: dims } };
}
// Qwen3-Embedding family on Ollama (and any other openai-compatible
// provider serving it) supports Matryoshka truncation via `dimensions`.
// Native sizes: 0.6B=1024, 4B=2560, 8B=4096. Without `dimensions`,
// Ollama returns the native size and brains configured for narrower
// widths hard-fail with a dim-mismatch error. Pattern match the bare
// model name + any `:tag` (e.g. `qwen3-embedding:4b`, `qwen3-embedding:0.6b`).
if (modelId === 'qwen3-embedding' || modelId.startsWith('qwen3-embedding:')) {
return { openaiCompatible: { dimensions: dims } };
}
// MiniMax embo-01 takes a `type: 'db' | 'query'` field for asymmetric
// retrieval. Today still hardcoded to 'db' for back-compat — opting
// into the new inputType seam is a follow-up (see plan's deferred
+157 -1
View File
@@ -23,6 +23,7 @@
import { embed as aiEmbed, embedMany, generateObject, generateText, jsonSchema } from 'ai';
import { AsyncLocalStorage } from 'node:async_hooks';
import { createHash } from 'node:crypto';
import { listRecipes } from './recipes/index.ts';
import { createOpenAI } from '@ai-sdk/openai';
import { createGoogleGenerativeAI } from '@ai-sdk/google';
@@ -52,6 +53,8 @@ import { dimsProviderOptions } from './dims.ts';
import { hasAnthropicKey } from './anthropic-key.ts';
import { AIConfigError, AITransientError, normalizeAIError } from './errors.ts';
import { runGuardrails, hasGuardrails, type GuardrailHook } from '../guardrails.ts';
import { loadConfig } from '../config.ts';
import { buildGatewayConfig } from './build-gateway-config.ts';
// ---- Gateway-wide AI-HTTP timeout (v0.42.20.0, #1762/#1775) ----
//
@@ -116,6 +119,18 @@ const DEFAULT_RERANKER_MODEL = 'zeroentropyai:zerank-2';
let _config: AIGatewayConfig | null = null;
const _modelCache = new Map<string, any>();
/**
* Recover the process-global gateway for foreground command entrypoints that
* were reached without cli.ts's normal engine-connect initialization (#2590).
* Existing configured gateways, including their DB-resolved model overrides,
* are deliberately left unchanged.
*/
export function configureGatewayIfUninitialized(): void {
if (_config) return;
const config = loadConfig();
if (config) configureGateway(buildGatewayConfig(config));
}
/**
* v0.31.12 recipe-models merge: per-gateway-instance set of model ids the
* user opted into via config. Keyed by provider id (`anthropic`, `openai`,
@@ -506,6 +521,20 @@ export async function reconfigureGatewayWithEngine(engine: BrainEngine): Promise
const expansionFull = newExpansion.includes(':') ? newExpansion : prefixWithProviderFrom(cfg.expansion_model ?? DEFAULT_EXPANSION_MODEL, newExpansion);
const chatFull = newChat.includes(':') ? newChat : prefixWithProviderFrom(cfg.chat_model ?? DEFAULT_CHAT_MODEL, newChat);
// ALSO resolve the four tier models and register them as extended models.
// assertTouchpoint's contract (model-resolver.ts) says config-chosen models —
// `models.default` and `models.tier.*` included — bypass the native recipe
// allowlist, but pre-fix only chat/expansion/embedding/reranker were
// registered. A model reachable ONLY through a tier (e.g. `models.tier.deep`
// set to an Opus newer than the recipe list) failed `probeChatModel` at call
// time and silently degraded think/auto_think to the gather-only stub.
// Resolving per-tier also honors `models.default` (it sits above tiers in
// the resolveModel chain).
const tierModels: string[] = [];
for (const tier of ['utility', 'reasoning', 'deep', 'subagent'] as const) {
tierModels.push(await resolveModel(engine, { tier, fallback: TIER_DEFAULTS[tier] }));
}
_config = { ...cfg, expansion_model: expansionFull, chat_model: chatFull };
_modelCache.clear();
_shrinkState.clear();
@@ -517,6 +546,7 @@ export async function reconfigureGatewayWithEngine(engine: BrainEngine): Promise
_config.chat_model,
_config.reranker_model,
...(_config.chat_fallback_chain ?? []),
...tierModels,
]) {
if (m) registerExtendedModel(m);
}
@@ -1044,6 +1074,30 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
* float[] (not base64), so the Layer 2 cap compares against the JSON
* payload size of each embedding rather than a base64 string length.
*/
/**
* NVIDIA NIM compatibility shim. NVIDIA uses the OpenAI embeddings wire
* shape but requires asymmetric input_type values: query for retrieval and
* passage for indexed documents. The generic gateway store carries
* query/document across the AI SDK boundary; map document to passage here.
*/
const nvidiaCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
let baseInit: RequestInit = init ?? {};
if (baseInit.body && typeof baseInit.body === 'string') {
try {
const parsed = JSON.parse(baseInit.body);
if (parsed && typeof parsed === 'object' && parsed.input_type === undefined) {
parsed.input_type = __embedInputTypeStore.getStore() === 'query' ? 'query' : 'passage';
const headers = new Headers(baseInit.headers ?? {});
headers.delete('content-length');
baseInit = { ...baseInit, body: JSON.stringify(parsed), headers };
}
} catch {
// Preserve the provider response when the SDK body is unexpectedly non-JSON.
}
}
return fetch(input as any, baseInit);
}) as unknown as typeof fetch;
const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
// OUTBOUND: normalize URL, rewrite path /embeddings → /models/embed, then
// rewrite body. fetch accepts RequestInfo (string | Request) | URL; we
@@ -1304,6 +1358,8 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon
? voyageCompatFetch
: recipe.id === 'zeroentropyai'
? zeroEntropyCompatFetch
: recipe.id === 'nvidia'
? nvidiaCompatFetch
: openAICompatAsymmetricFetch);
const client = createOpenAICompatible({
name: recipe.id,
@@ -2849,6 +2905,30 @@ async function classifyGatewayGuardrail(input: {
}
}
/**
* Derive OpenAI's `prompt_cache_key` (the AI SDK's `providerOptions.openai.
* promptCacheKey`). It's a ROUTING hint, not a cache breakpoint: OpenAI caches
* prefixes automatically, and a stable key makes requests sharing a prefix
* land on the same engine, raising the hit rate (OpenAI cites 60%87%).
*
* Hash the system prompt + sorted tool names that's the stable prefix
* gbrain's repeated loops (enrich, page-summary, skillopt, subagent) actually
* share. Returns undefined when there's no system prompt (nothing stable to
* key on), so one-off requests don't get pinned to a single engine. An
* explicit key can still be set per provider/model via
* `provider_chat_options` config, which overrides the derived key.
*
* @internal exported for tests; not part of the public gateway API.
*/
export function openAIPromptCacheKey(args: {
system?: string;
toolNames?: string[];
}): string | undefined {
if (!args.system) return undefined;
const basis = `${args.system} ${(args.toolNames ?? []).slice().sort().join(',')}`;
return `gbrain:${createHash('sha256').update(basis).digest('hex').slice(0, 32)}`;
}
export function toAISDKTools(tools: ChatToolDef[] | undefined): Record<string, any> | undefined {
if (!tools || tools.length === 0) return undefined;
return tools.reduce((acc, t) => {
@@ -2957,10 +3037,68 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
const providerOptions: Record<string, any> = {};
if (useCache) {
// Call-level `providerOptions.anthropic.cacheControl` is NOT a no-op:
// @ai-sdk/anthropic 3.0.47+ passes it through as a top-level
// `cache_control` field on the Anthropic request body, which the
// Messages API resolves as its documented "auto-cache the last
// cacheable block in the request" shorthand (see Anthropic's
// prompt-caching docs — "top-level auto-caching ... is the simplest
// option when you don't need fine-grained placement"). Keep it: it's
// what gives a growing multi-turn conversation (toolLoop()) a rolling
// cache breakpoint on each turn's tail for free, without us having to
// hand-roll the marker-walking logic subagent.ts's raw-SDK path uses.
//
// But "last cacheable block" is the wrong block for gbrain#2490's
// actual callers (page-summary, skillopt, enrich): those are
// single-turn calls with a STABLE system prompt and a DIFFERENT user
// message every time, so the auto-marker lands on the ever-varying
// tail — every call WRITES a fresh cache entry and never READS a prior
// one (cache_read_input_tokens stays 0 forever). Caching the stable
// prefix needs an EXPLICIT breakpoint on the system block itself,
// which is applied below via a `SystemModelMessage` (round-trips its
// own `providerOptions`) instead of a bare string.
providerOptions.anthropic = { cacheControl: { type: 'ephemeral' } };
}
// OpenAI prompt_cache_key (native-openai only): a stable per-prefix routing
// hint that keeps requests sharing a system prompt + tool set on the same
// inference engine, lifting OpenAI's automatic prefix-cache hit rate. The
// openai-compatible path (litellm/azure/groq/...) ignores
// providerOptions.openai, so it gets nothing. Applied BEFORE the configured
// provider options so `provider_chat_options.openai.promptCacheKey` from
// config still overrides the derived key.
if (recipe.implementation === 'native-openai') {
const promptCacheKey = openAIPromptCacheKey({
system: opts.system,
toolNames: (opts.tools ?? []).map(t => t.name),
});
if (promptCacheKey) providerOptions.openai = { promptCacheKey };
}
applyConfiguredChatProviderOptions(providerOptions, cfg, recipe.id, modelId);
// Derive ONE canonical cache-control value AFTER config merging and reuse
// it for every breakpoint (system block, last tool def, call-level). If
// `provider_chat_options.anthropic.cacheControl` overrides the TTL (e.g.
// `{ type: 'ephemeral', ttl: '1h' }`), that override lands in
// `providerOptions.anthropic.cacheControl` via the deep-merge above —
// reusing it here (instead of hardcoding `{ type: 'ephemeral' }` per
// breakpoint) keeps every marker in the request on the same TTL.
const cacheControlValue: { type: 'ephemeral'; ttl?: '5m' | '1h' } | undefined = useCache
? (providerOptions.anthropic?.cacheControl ?? { type: 'ephemeral' })
: undefined;
// Anthropic-only secondary breakpoint: mark the LAST tool def too (mirrors
// subagent.ts's raw-SDK path — Anthropic caches everything up to and
// including the last `cache_control` block it sees in the request, so
// marking the last tool extends the cached prefix through the whole tool
// list). `tool.providerOptions.anthropic.cacheControl` is the shape
// @ai-sdk/anthropic 3.x reads for tool-def breakpoints.
if (cacheControlValue && opts.tools && opts.tools.length > 0 && tools) {
const lastTool = tools[opts.tools[opts.tools.length - 1]!.name];
if (lastTool) {
lastTool.providerOptions = { anthropic: { cacheControl: cacheControlValue } };
}
}
let _budgetRecorded = false;
const _recordBudget = (modelLabel: string, inputTokens: number, outputTokens: number): void => {
if (!tracker || _budgetRecorded) return;
@@ -2977,10 +3115,28 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
}
};
// The actual Anthropic system-prompt cache breakpoint. A bare string
// `system` produces `{ role: 'system', content }` with no `providerOptions`
// field (ai@6's convertToLanguageModelPrompt), so @ai-sdk/anthropic's
// getCacheControl(providerOptions) on that block always resolves to
// nothing. Passing a `SystemModelMessage` object instead — the shape `ai`
// documents specifically for "additional provider options (e.g. for
// caching)" — round-trips `providerOptions` onto that block. Byte-identical
// to the old bare-string form when useCache is false. Reuses
// `cacheControlValue` (the config-merged value) so this breakpoint's TTL
// always matches the last-tool and call-level breakpoints.
const systemParam = cacheControlValue && opts.system
? {
role: 'system' as const,
content: opts.system,
providerOptions: { anthropic: { cacheControl: cacheControlValue } },
}
: opts.system;
try {
const result = await _generateTextTransport({
model,
system: opts.system,
system: systemParam,
messages: toModelMessages(repairToolPairing(opts.messages)) as any,
tools: opts.tools && opts.tools.length > 0 ? tools : undefined,
maxOutputTokens: opts.maxTokens ?? defaultMaxOutputTokens(modelStr),
+4 -1
View File
@@ -17,13 +17,16 @@ export const anthropic: Recipe = {
touchpoints: {
// No embedding model available.
expansion: {
models: ['claude-haiku-4-5-20251001', 'claude-sonnet-4-6'],
models: ['claude-haiku-4-5-20251001', 'claude-sonnet-5', 'claude-sonnet-4-6'],
cost_per_1m_tokens_usd: 0.25,
price_last_verified: '2026-05-10',
},
chat: {
models: [
'claude-fable-5',
'claude-opus-4-8',
'claude-opus-4-7',
'claude-sonnet-5',
'claude-sonnet-4-6',
'claude-haiku-4-5-20251001',
],
+6
View File
@@ -23,6 +23,9 @@ import { zhipu } from './zhipu.ts';
import { azureOpenAI } from './azure-openai.ts';
import { zeroentropyai } from './zeroentropyai.ts';
import { llamaServerReranker } from './llama-server-reranker.ts';
import { moonshot } from './moonshot.ts';
import { mistral } from './mistral.ts';
import { nvidia } from './nvidia.ts';
const ALL: Recipe[] = [
openai,
@@ -42,6 +45,9 @@ const ALL: Recipe[] = [
zhipu,
azureOpenAI,
zeroentropyai,
moonshot,
mistral,
nvidia,
];
/** Map from `provider:id` key to recipe. */
+15
View File
@@ -41,6 +41,21 @@ export const litellmProxy: Recipe = {
// mismatched-dim responses pre-storage).
supports_multimodal: true,
},
expansion: {
models: [],
cost_per_1m_tokens_usd: undefined,
price_last_verified: '2026-06-14',
},
chat: {
models: [],
supports_tools: true,
supports_subagent_loop: true,
supports_prompt_cache: false,
max_context_tokens: 200_000,
cost_per_1m_input_usd: undefined,
cost_per_1m_output_usd: undefined,
price_last_verified: '2026-06-14',
},
},
setup_hint: 'Run LiteLLM (https://docs.litellm.ai) in front of any provider; set LITELLM_BASE_URL (include the /v1 suffix if your proxy serves the OpenAI route there, e.g. http://localhost:4000/v1) + pass --embedding-model litellm:<model> and --embedding-dimensions <N>.',
};
+84
View File
@@ -0,0 +1,84 @@
import type { Recipe } from '../types.ts';
/**
* Mistral AI exposes an OpenAI-compatible API at https://api.mistral.ai/v1
* (/embeddings + /chat/completions). EU-hosted the reason this recipe
* exists: a brain that must stay inside EU jurisdiction can run embed +
* expansion + chat on a single provider without a US hop.
*
* Verified against the live API on 2026-07-19 (model catalog, embedding
* dimensions, dimension-parameter rejection, and the batch ceiling see
* the notes on each field below).
*
* DIMENSIONS mistral-embed is FIXED 1024 and accepts NO dimension
* parameter at all. Both spellings are rejected upstream:
* {"dimensions": 512} -> 400 extra_forbidden (not in the API schema)
* {"output_dimension": 512} -> 400 "This model does not support output_dimension"
* The generic `openai-compatible` branch of dims.ts:dimsProviderOptions()
* already falls through to `return undefined` for these model ids, so no
* dimension field is emitted. Do NOT add mistral-embed to any of the
* flexible-dim allowlists there it would 400 every embed call. Same
* contract as voyage-4-nano, for the same reason.
*
* codestral-embed / codestral-embed-2505 are deliberately NOT listed: they
* return 1536 dims, and a touchpoint carries a single `default_dims`.
* Mixing them under a 1024 declaration is the mixed-dim footgun
* embedding-dim-check.ts exists to catch. They are code-retrieval models
* anyway; a prose brain wants mistral-embed.
*/
export const mistral: Recipe = {
id: 'mistral',
name: 'Mistral AI',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://api.mistral.ai/v1',
auth_env: {
required: ['MISTRAL_API_KEY'],
setup_url: 'https://console.mistral.ai/api-keys',
},
touchpoints: {
embedding: {
models: ['mistral-embed', 'mistral-embed-2312'],
default_dims: 1024,
// Mistral's published list price. Advisory only — canonical embedding
// spend accounting lives in src/core/embedding-pricing.ts.
cost_per_1m_tokens_usd: 0.1,
price_last_verified: '2026-07-19',
// Measured ceiling, not a doc guess: the /embeddings endpoint accepts a
// 65,286-token batch and rejects 66,960 with
// 400 code 3210 "Too many tokens overall, split into more batches."
// -> the real cap is 65,536 (64K) tokens per request.
max_batch_tokens: 65_536,
// chars_per_token is a DIVISOR in splitByTokenBudget()
// (estTokens = text.length / charsPerToken), so a LOWER value is the
// conservative direction. The module default of 4 is an English-prose
// assumption; German prose measured 3.58 here, and code/JSON/CJK runs
// denser still. 2 keeps the estimate above the real token count for
// every content shape we see.
chars_per_token: 2,
// With safety_factor 0.5 the pre-split budget is 32,768 estimated
// tokens = 65,536 chars. Worst realistic density (~1.5 chars/token)
// puts that at ~43.7K real tokens — still clear of the 64K ceiling.
safety_factor: 0.5,
},
expansion: {
models: ['ministral-3b-latest', 'mistral-small-latest'],
price_last_verified: '2026-07-19',
},
chat: {
models: [
'mistral-small-latest', 'mistral-medium-latest', 'mistral-large-latest',
'ministral-3b-latest', 'ministral-8b-latest', 'magistral-small-latest',
],
supports_tools: true,
// Same call as the Moonshot recipe: ordinary tool calls are fine, but
// gbrain's subagent loop stays Anthropic-pinned for stable tool_use_id
// behavior across crashes/replays.
supports_subagent_loop: false,
supports_prompt_cache: false,
max_context_tokens: 262144,
price_last_verified: '2026-07-19',
},
},
setup_hint: 'Get an API key at https://console.mistral.ai/api-keys, then `export MISTRAL_API_KEY=...` and use `mistral:mistral-embed` (1024 dims) for embeddings.',
};
+42
View File
@@ -0,0 +1,42 @@
import type { Recipe } from '../types.ts';
/**
* Moonshot AI / Kimi Open Platform. Kimi exposes an OpenAI-compatible
* /v1/chat/completions API at https://api.moonshot.ai/v1.
*
* Verified against Kimi API docs and live /v1/models on 2026-06-23.
* The recipe is local-production glue until upstream GBrain carries a native
* Moonshot recipe; keep it registered in the local patch registry.
*/
export const moonshot: Recipe = {
id: 'moonshot',
name: 'Moonshot AI / Kimi',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://api.moonshot.ai/v1',
auth_env: {
required: ['MOONSHOT_API_KEY'],
setup_url: 'https://platform.kimi.ai/console/api-keys',
},
touchpoints: {
expansion: {
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
// Kimi pricing varies by current promotional/account terms; do not use
// this advisory field for budget enforcement. Canonical budget pricing
// belongs in src/core/model-pricing.ts when verified for the account.
price_last_verified: '2026-06-23',
},
chat: {
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
supports_tools: true,
// Kimi tool calling is enough for ordinary chat/tool calls. GBrain's
// subagent loop remains Anthropic-pinned because upstream requires stable
// Anthropic-style tool_use_id behavior across crashes/replays.
supports_subagent_loop: false,
supports_prompt_cache: false,
max_context_tokens: 256000,
price_last_verified: '2026-06-23',
},
},
setup_hint: 'Get an API key at https://platform.kimi.ai/console/api-keys, then `export MOONSHOT_API_KEY=...` and use `moonshot:kimi-k2.7-code`.',
};
+71
View File
@@ -0,0 +1,71 @@
import type { Recipe } from '../types.ts';
/**
* NVIDIA NIM / API Catalog exposes OpenAI-compatible /v1/chat/completions
* and /v1/embeddings APIs.
*
* Retrieval models use asymmetric encoding. The gateway maps gbrain's
* document/query distinction to NVIDIA's wire values:
* document -> input_type: passage
* query -> input_type: query
*
* The model ids below intentionally keep NVIDIA's full catalog ids because
* the hosted endpoint expects values like `nvidia/nv-embedqa-e5-v5` in the
* request body. Short aliases are provided for CLI ergonomics.
*/
export const nvidia: Recipe = {
id: 'nvidia',
name: 'NVIDIA NIM',
tier: 'openai-compat',
implementation: 'openai-compatible',
base_url_default: 'https://integrate.api.nvidia.com/v1',
auth_env: {
required: ['NVIDIA_API_KEY'],
setup_url: 'https://build.nvidia.com',
},
aliases: {
'nv-embedqa-e5-v5': 'nvidia/nv-embedqa-e5-v5',
'llama-nemotron-embed-1b-v2': 'nvidia/llama-nemotron-embed-1b-v2',
'nemotron-3-super': 'nvidia/nemotron-3-super-120b-a12b',
'nemotron-3-super-120b-a12b': 'nvidia/nemotron-3-super-120b-a12b',
'nv-embed-v1': 'nvidia/nv-embed-v1',
'nv-embedcode-7b-v1': 'nvidia/nv-embedcode-7b-v1',
},
// No resolveAuth override: NVIDIA is plain `Authorization: Bearer <key>`,
// which defaultResolveAuth derives from auth_env.required. IRON RULE
// (test/ai/recipes-existing-regression.test.ts): only Azure overrides
// resolveAuth.
touchpoints: {
chat: {
models: [
'nvidia/nemotron-3-super-120b-a12b',
],
supports_tools: false,
supports_subagent_loop: false,
// Do not treat Nemotron as a Minions subagent driver until tool-calling
// and replay stability are proven through a separate adapter test.
max_context_tokens: 128000,
price_last_verified: '2026-05-24',
},
embedding: {
models: [
'nvidia/nv-embedqa-e5-v5',
'nvidia/llama-nemotron-embed-1b-v2',
'nvidia/nv-embed-v1',
'nvidia/nv-embedcode-7b-v1',
],
// Default to the lightest tested hosted model. Larger NVIDIA models are
// supported via explicit embedding_dimensions (2048 or 4096).
default_dims: 1024,
dims_options: [1024, 2048, 4096],
// Conservative split; hosted NVIDIA embedding endpoints require
// input_type and may reject large payloads before tokenizing.
max_batch_tokens: 8192,
chars_per_token: 4,
safety_factor: 0.75,
cost_per_1m_tokens_usd: undefined,
price_last_verified: '2026-05-24',
},
},
setup_hint: 'Get an API key at https://build.nvidia.com, then `export NVIDIA_API_KEY=...`.',
};
+55 -4
View File
@@ -100,7 +100,15 @@ function gbrainHome(): string {
* corecommands import). which gbrain process.execPath argv[1] "gbrain". */
function resolveGbrainCliPath(): string {
try {
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
// #2747: `env: process.env` required under Bun — see the sibling copy
// of this function in commands/autopilot.ts for the full explanation
// (Bun snapshots process.env at its own startup; execSync without an
// explicit env is blind to any PATH mutation since then).
const which = execSync('which gbrain', {
encoding: 'utf-8',
stdio: ['ignore', 'pipe', 'ignore'],
env: process.env,
}).trim();
if (which) return which;
} catch { /* not on PATH */ }
const exec = process.execPath ?? '';
@@ -183,15 +191,17 @@ if [ "\${1:-}" = "--push-only" ]; then
fi
_msg="\${1:?usage: brain-commit-push.sh <message> <path> [paths...]}"; shift || true
# Pull first so the local tree is current before we stage.
git fetch origin >/dev/null 2>&1 || true
git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; }
# EXPLICIT paths only never a blind 'git add -A' (would risk committing
# secrets, temp files, or unrelated edits).
if [ "$#" -eq 0 ]; then
echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2
fi
# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage)
# aborted on any dirty tree 'cannot pull with rebase: You have unstaged
# changes' so the helper could never commit a MODIFIED page (exactly the
# write-through case). Stage + commit first; brain_push below already handles
# a remote that advanced (push -> rejected -> pull --rebase -> push).
git add -- "$@"
if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi
git commit -m "$_msg"
@@ -337,6 +347,47 @@ function uninstallLocalHook(repoPath: string): boolean {
return true;
}
/**
* True when the gbrain durability post-commit hook is installed i.e. the
* user opted this repo into push-durability via `gbrain sources harden`.
* Cheap (one git-config read + one file read); used as the gate for
* write-through auto-commit (#2426).
*/
export function isDurabilityHardened(repoPath: string): boolean {
try {
const { dir } = resolveHooksDir(repoPath);
const hookPath = join(dir, 'post-commit');
return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER);
} catch {
return false;
}
}
/**
* #2426: best-effort commit of a single write-through artifact so DB writes
* reach git (the post-commit hook then background-pushes). Pre-fix,
* write-through `.md` accumulated uncommitted forever: it never reached the
* remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full`
* delete-reconcile treated the never-committed pages as disposable.
*
* Path-limited (`git commit -- <path>`) so unrelated staged/dirty edits are
* never swept into the commit. Never throws; returns false on any failure
* (index.lock contention, nothing changed, detached states) the DB row and
* the on-disk file remain the durable sinks either way.
*/
export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean {
try {
const rel = relative(repoPath, absPath);
if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false;
const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const;
execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts);
execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts);
return true;
} catch {
return false;
}
}
// ── Committed helper ────────────────────────────────────────────────────────
function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } {
+46 -28
View File
@@ -22,6 +22,7 @@ import { join, relative, resolve, dirname, basename, isAbsolute } from 'path';
import type { BrainEngine } from './engine.ts';
import type { ProgressReporter } from './progress.ts';
import { gbrainPath } from './config.ts';
import { collectGitVisibleFiles } from './git-visible-files.ts';
import {
parseMarkdown,
type ParseValidationCode,
@@ -408,6 +409,11 @@ export interface ScanOpts {
visitDir?: (dirPath: string) => void;
}
/** Timeout-arm winner for the COUNT-vs-deadline race in scanBrainSources.
* A unique object so it can never collide with a legitimate COUNT result
* (number | null). Module-private. */
const DEADLINE_SENTINEL: unique symbol = Symbol('gbrain.scan.deadline');
export async function scanBrainSources(
engine: BrainEngine,
opts: ScanOpts = {},
@@ -479,41 +485,43 @@ export async function scanBrainSources(
// pool can make this await hang past the budget. Without the race, we'd
// wait indefinitely AND defeat the wall-clock guarantee.
let dbPageCount: number | null = null;
// Set when the deadline race's timeout arm wins: the verdict that the
// budget is spent, independent of any later Date.now() reading. Timer
// callbacks on loaded runners can fire measurably EARLY relative to the
// wall clock (a +1ms pad was drifted past in practice — see the flake
// lineage in test/brain-writer-partial-scan.test.ts and issue #2946), so
// the hung-COUNT path must not re-derive "did the deadline fire?" from
// the clock the timer just raced against.
let deadlineHit = false;
if (opts.dbPageCountForSource) {
try {
if (opts.deadline) {
const remainingMs = opts.deadline - Date.now();
if (remainingMs <= 0) {
dbPageCount = null;
deadlineHit = true;
} else {
// Race COUNT against the deadline so a hung query can't eat the budget.
//
// Boundary overshoot (+1ms): the post-await deadline check at line
// ~512 uses `Date.now() >= deadline`. setTimeout fires AT OR AFTER
// the requested delay, so in theory the check always passes. In
// practice on heavily-loaded CI runners (8 parallel shards × 4
// concurrent test files = ~32 concurrent bun processes) we saw
// intermittent failures where the timer callback resolved
// microseconds BEFORE the wall-clock boundary, leaving Date.now()
// a tick below deadline and the skip-check evaluating false. The
// src-a scan then ran on a populated dir before src-b's
// between-source check caught up — causing
// `firstSource.status === 'skipped'` to receive 'scanned'.
//
// Adding 1ms guarantees the timer fires past the deadline by at
// least one millisecond regardless of runner timer drift. Cost is
// 1ms additional wall-clock latency on hung COUNT queries, which
// is operationally negligible. Flake repro:
// https://github.com/garrytan/gbrain/actions/runs/77611667786
dbPageCount = await Promise.race([
// Race COUNT against the deadline so a hung query can't eat the
// budget. The timeout arm resolves a private sentinel — NOT null —
// so a deadline win is distinguishable from a COUNT that resolved
// null (failed/absent count keeps its existing semantics).
const raced = await Promise.race([
opts.dbPageCountForSource(src.id),
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs + 1)),
new Promise<typeof DEADLINE_SENTINEL>(resolve =>
setTimeout(() => resolve(DEADLINE_SENTINEL), remainingMs)),
]);
if (raced === DEADLINE_SENTINEL) {
dbPageCount = null;
deadlineHit = true;
} else {
dbPageCount = raced;
}
}
} else {
dbPageCount = await opts.dbPageCountForSource(src.id);
}
} catch {
// A throwing COUNT is a failed count, not a deadline verdict.
dbPageCount = null;
}
}
@@ -523,11 +531,11 @@ export async function scanBrainSources(
// status='partial' with files_scanned=0, which is misleading ("partial
// scan" when actually nothing was scanned). Mark this source + remainder
// as 'skipped' so the doctor message is honest.
// `>=` matches the between-source check above (line 445). The Promise.race
// setTimeout resolves null at exactly `remainingMs` from now, so post-await
// Date.now() often equals deadline within integer-ms precision — strict `>`
// missed those landings on CI and let the next scanOneSource run anyway.
if (opts.signal?.aborted || (opts.deadline && Date.now() >= opts.deadline)) {
// `deadlineHit` is the authoritative verdict for the hung-COUNT path (the
// sentinel above); the wall-clock re-check (`>=`, matching the
// between-source check at line ~445) still covers a COUNT that RESOLVED
// slowly enough to eat the budget without the timer winning.
if (opts.signal?.aborted || deadlineHit || (opts.deadline && Date.now() >= opts.deadline)) {
if (abortedAtSource === null) {
abortedAtSource = src.id;
}
@@ -579,7 +587,7 @@ function scanOneSource(
let ignoredMissingOpen = 0;
let interrupted = false;
walkDir(rootResolved, (absPath) => {
const visitFile = (absPath: string): boolean | void => {
// Per-file deadline + abort gate. Deadline is the load-bearing
// wall-clock bound (sync I/O blocks the event loop so timer-based
// AbortSignal.timeout can't fire mid-walk — codex C1).
@@ -625,7 +633,17 @@ function scanOneSource(
opts.onProgress.tick(50);
}
return true;
}, opts.visitDir);
};
const gitFiles = collectGitVisibleFiles(rootResolved, (rel) => isSyncable(rel, { strategy: 'markdown' }));
if (gitFiles) {
if (opts.visitDir) opts.visitDir(rootResolved);
for (const absPath of gitFiles) {
if (visitFile(absPath) === false) break;
}
} else {
walkDir(rootResolved, visitFile, opts.visitDir);
}
if (opts.onProgress) {
opts.onProgress.heartbeat(`scanned ${scanned} pages in ${sourceId}`);
+50 -8
View File
@@ -26,7 +26,17 @@ export interface ChronicleJudgeInput {
effectiveDate: string | null; // depth page effective_date (deterministic when)
attendees: string[]; // deterministic who from frontmatter
}
export interface ChronicleJudgeResult { events: ChronicleEventProposal[] }
export interface ChronicleJudgeResult {
events: ChronicleEventProposal[];
/**
* #2606 distinct judge-failure signal so an unusable response is never
* recorded as a legitimate `no_events`:
* - 'truncated': the model hit the output-token cap (stopReason 'length');
* the JSON array was cut mid-stream and must not be parsed as complete.
* - 'parse_failed': the model returned text but no valid JSON array.
*/
failure?: 'truncated' | 'parse_failed';
}
export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise<ChronicleJudgeResult>;
export interface ChronicleExtractResult {
@@ -126,6 +136,12 @@ export async function runChronicleExtract(
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' };
}
// #2606: a truncated or unparseable judge response is a FAILURE, not an
// empty page. Record it as a distinct skipped reason so operators (and
// retries) can tell it apart from a genuine no_events.
if (result?.failure) {
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` };
}
const proposals = Array.isArray(result?.events) ? result.events : [];
if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 };
// PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes.
@@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli
Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}.
Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown just the JSON array.`;
/**
* #2606: default output-token cap for the judge. Raised from the original
* 1500 (which event-dense pages overflowed, silently truncating the JSON
* array). Override via `chronicle.judge_max_tokens`.
*/
const DEFAULT_JUDGE_MAX_TOKENS = 4000;
function defaultJudge(engine: BrainEngine): ChronicleJudge {
return async (input) => {
const { isAvailable, chat } = await import('../ai/gateway.ts');
if (!isAvailable('chat')) return { events: [] };
const body = (input.body || '').slice(0, 12_000);
// #2606: configurable cap so event-dense pages have headroom.
let maxTokens = DEFAULT_JUDGE_MAX_TOKENS;
const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null);
if (capRaw) {
const n = parseInt(capRaw, 10);
if (Number.isFinite(n) && n > 0) maxTokens = n;
}
let text: string;
try {
const res = await chat({
@@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge {
`${input.title}\n\n${body}\n</page>\n\n` +
`Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`,
}],
maxTokens: 1500,
maxTokens,
});
if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] };
// #2606: output hit the token cap — the JSON array is cut mid-stream.
// Do NOT feed it to the parser as if complete; surface the truncation.
if (res.stopReason === 'length') return { events: [], failure: 'truncated' };
text = res.text;
} catch (err) {
if ((err as Error)?.name === 'AbortError') throw err;
return { events: [] };
}
const parsed = parseJudgeJson(text);
// #2606: non-empty model text with no parseable JSON array is a parse
// failure, distinct from the model legitimately answering `[]`.
if (parsed === null) return { events: [], failure: 'parse_failed' };
return { events: parsed };
};
}
/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */
export function parseJudgeJson(text: string): ChronicleEventProposal[] {
if (!text) return [];
/**
* Tolerant JSON-array extraction from a model response (mirrors facts parser).
*
* #2606: returns `null` on parse FAILURE (empty text, no `[...]` found,
* JSON.parse throw, non-array result) so callers can distinguish "the model
* said no events" (a legitimate `[]`) from "the response was unusable".
*/
export function parseJudgeJson(text: string): ChronicleEventProposal[] | null {
if (!text) return null;
let s = text.trim();
const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i);
if (fence) s = fence[1].trim();
const start = s.indexOf('[');
const end = s.lastIndexOf(']');
if (start === -1 || end === -1 || end < start) return [];
if (start === -1 || end === -1 || end < start) return null;
try {
const arr = JSON.parse(s.slice(start, end + 1));
return Array.isArray(arr) ? arr : [];
return Array.isArray(arr) ? arr : null;
} catch {
return [];
return null;
}
}
+25 -2
View File
@@ -114,6 +114,26 @@ function resolveFlushGraceMs(): number {
/** Default per-sink drain budget (matches drainAllBackgroundWorkForCliExit). */
const DEFAULT_DRAIN_TIMEOUT_MS = 2_000;
/**
* Resolve the per-sink drain budget: `GBRAIN_DRAIN_TIMEOUT_MS` env override
* (slow-provider escape hatch, same env-only pattern as
* GBRAIN_TEARDOWN_DEADLINE_MS) over the 2000ms default. An explicit
* `drainTimeoutMs` from a call site still wins the env replaces only the
* DEFAULT. The 2s default assumes a sub-second cloud chat provider; a
* self-hosted model (e.g. ollama at 10-20s per completion) can never finish a
* fire-and-forget facts:absorb extraction inside it, so every one-shot CLI
* exit sync timers especially aborts the in-flight chat and the
* extraction never lands, retrying (and re-aborting) on each subsequent sync
* of the same page. Raising the budget via env lets those installs drain
* instead of abort; computeTeardownDeadlineMs already scales the backstop
* from the resolved value, so the deadline widens with it.
*/
export function resolveDrainTimeoutMs(): number {
const env = Number(process.env.GBRAIN_DRAIN_TIMEOUT_MS);
if (Number.isFinite(env) && env > 0) return env;
return DEFAULT_DRAIN_TIMEOUT_MS;
}
/**
* Backstop deadline for drain + disconnect COMBINED, computed from the bounds
* it guards so it fires only when a component violated its own bound (#2084
@@ -262,7 +282,10 @@ export function flushThenExit(code: number, opts: FlushThenExitOpts = {}): void
export interface FinishCliTeardownOpts {
/** Engine to disconnect. A disconnect throw is warned + swallowed (D3). */
engine: { disconnect(): Promise<void> };
/** Per-sink drain budget. Default 2000 (the registry default). */
/**
* Per-sink drain budget. Default: `GBRAIN_DRAIN_TIMEOUT_MS` env override,
* else 2000 (the registry default).
*/
drainTimeoutMs?: number;
/** Test seam — wins over the env override and the computed formula. */
deadlineMs?: number;
@@ -284,7 +307,7 @@ export interface FinishCliTeardownOpts {
* exit in here, and it means a component violated its own bound.
*/
export async function finishCliTeardown(opts: FinishCliTeardownOpts): Promise<void> {
const drainTimeoutMs = opts.drainTimeoutMs ?? DEFAULT_DRAIN_TIMEOUT_MS;
const drainTimeoutMs = opts.drainTimeoutMs ?? resolveDrainTimeoutMs();
const warn = opts.warn ?? ((m: string) => console.warn(m));
const drain = opts.drain ?? drainAllBackgroundWorkForCliExit;
const deadlineMs =
+32
View File
@@ -271,6 +271,8 @@ export interface GBrainConfig {
verdict_model?: string;
max_prompt_tokens?: number;
max_chunks_per_transcript?: number;
subagent_timeout_ms?: number;
subagent_wait_timeout_ms?: number;
};
patterns?: {
lookback_days?: number;
@@ -710,6 +712,12 @@ export async function loadConfigWithEngine(
const n = parseInt(v, 10);
return Number.isFinite(n) && n > 0 ? n : undefined;
}
async function dbNum(key: string): Promise<number | undefined> {
const v = await dbStr(key);
if (v === undefined) return undefined;
const n = Number(v);
return Number.isNaN(n) ? undefined : n;
}
const dbWarnBytes = await dbInt('content_sanity.bytes_warn');
const dbBlockBytes = await dbInt('content_sanity.bytes_block');
const dbJunkEnabled = await dbBool('content_sanity.junk_patterns_enabled');
@@ -759,6 +767,8 @@ export async function loadConfigWithEngine(
const dbVerdictModel = await dbStr('dream.synthesize.verdict_model');
const dbMaxPromptTokens = await dbInt('dream.synthesize.max_prompt_tokens');
const dbMaxChunksPerTranscript = await dbInt('dream.synthesize.max_chunks_per_transcript');
const dbSubagentTimeoutMs = await dbNum('dream.synthesize.subagent_timeout_ms');
const dbSubagentWaitTimeoutMs = await dbNum('dream.synthesize.subagent_wait_timeout_ms');
const dbLookbackDays = await dbInt('dream.patterns.lookback_days');
const dbMinEvidence = await dbInt('dream.patterns.min_evidence');
@@ -783,6 +793,12 @@ export async function loadConfigWithEngine(
if (mergedSynth.max_chunks_per_transcript === undefined && dbMaxChunksPerTranscript !== undefined) {
mergedSynth.max_chunks_per_transcript = dbMaxChunksPerTranscript;
}
if (mergedSynth.subagent_timeout_ms === undefined && dbSubagentTimeoutMs !== undefined) {
mergedSynth.subagent_timeout_ms = dbSubagentTimeoutMs;
}
if (mergedSynth.subagent_wait_timeout_ms === undefined && dbSubagentWaitTimeoutMs !== undefined) {
mergedSynth.subagent_wait_timeout_ms = dbSubagentWaitTimeoutMs;
}
if (mergedPatterns.lookback_days === undefined && dbLookbackDays !== undefined) {
mergedPatterns.lookback_days = dbLookbackDays;
}
@@ -854,6 +870,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
// subagent handler's error message tells users to `config set` this, so it
// must be a known key or `config set` rejects it without --force.
'agent.use_gateway_loop',
// #2778: per-turn output-token cap for the subagent loop (default 8192).
'agent.max_output_tokens',
// DB-plane (v0.32.3 search modes + related)
'search.mode',
'search.cache.enabled',
@@ -888,6 +906,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'models.chat',
'models.eval.longmemeval',
'facts.extraction_model',
// #2113: output-token cap for the per-turn facts extractor (default 4000).
'facts.extraction_max_tokens',
// Dream cycle config
'dream.synthesize.session_corpus_dir',
'dream.synthesize.meeting_transcripts_dir',
@@ -895,8 +915,16 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'dream.synthesize.verdict_model',
'dream.synthesize.max_prompt_tokens',
'dream.synthesize.max_chunks_per_transcript',
// #2415: top-level namespace for synthesize/patterns output (default 'wiki').
'dream.synthesize.output_root',
'dream.synthesize.subagent_timeout_ms',
'dream.synthesize.subagent_wait_timeout_ms',
'dream.patterns.lookback_days',
'dream.patterns.min_evidence',
// #2782-family: patterns-phase subagent timeouts (mirror of the
// dream.synthesize.* pair from #1594).
'dream.patterns.subagent_timeout_ms',
'dream.patterns.subagent_wait_timeout_ms',
// Emotional weight (v0.29)
'emotional_weight.high_tags',
'emotional_weight.user_holder',
@@ -945,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
// operator had to discover --force by reading source. Same class as the
// spend-controls registration above.
'auto_chronicle',
// #2606: chronicle judge output-token cap (default 4000). Event-dense
// pages overflowed the old hardcoded 1500 and were misrecorded as
// no_events; the cap is now configurable and truncation is surfaced.
'chronicle.judge_max_tokens',
// Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate
// consent reads this key, and enabling it is the documented path to
// `gbrain takes extract --from-pages` — same unregistered-key class.
+22 -5
View File
@@ -54,6 +54,7 @@ import {
import {
generatePerChunkSynopsis,
SYNOPSIS_PROMPT_VERSION,
SYNOPSIS_DOC_MAX_CHARS,
type GeneratePerChunkSynopsisResult,
} from './page-summary.ts';
import {
@@ -103,8 +104,17 @@ function getEmbeddingModelTag(): string {
export function computeCorpusGeneration(args: {
crMode: CRMode;
haikuModel: string;
/**
* Resolved `SYNOPSIS_DOC_MAX_CHARS` for per_chunk_synopsis runs. When
* present, folded into the hash so changes to
* `GBRAIN_SYNOPSIS_DOC_MAX_CHARS` invalidate the prior cache cleanly.
* Omit for `crMode !== 'per_chunk_synopsis'` title / none modes
* don't consult the cap and the field stays out of the hash for
* back-compat with pre-cap embeddings.
*/
synopsisDocMaxChars?: number;
}): string {
return createHash('sha256')
const h = createHash('sha256')
.update(args.crMode)
.update('|')
.update(String(SYNOPSIS_PROMPT_VERSION))
@@ -113,9 +123,11 @@ export function computeCorpusGeneration(args: {
.update('|')
.update(String(TITLE_WRAPPER_VERSION))
.update('|')
.update(getEmbeddingModelTag())
.digest('hex')
.slice(0, 16);
.update(getEmbeddingModelTag());
if (args.synopsisDocMaxChars !== undefined) {
h.update('|doc_cap=').update(String(args.synopsisDocMaxChars));
}
return h.digest('hex').slice(0, 16);
}
/**
@@ -253,7 +265,11 @@ export async function reembedPageWithContextualRetrieval(
args.pageSlug,
args.sourceId,
resolution.mode,
computeCorpusGeneration({ crMode: resolution.mode, haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL }),
computeCorpusGeneration({
crMode: resolution.mode,
haikuModel: args.haikuModel ?? DEFAULT_HAIKU_MODEL,
synopsisDocMaxChars: resolution.mode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
}),
);
return { kind: 'skipped', reason: 'no_chunks' };
}
@@ -282,6 +298,7 @@ export async function reembedPageWithContextualRetrieval(
const corpus_generation = computeCorpusGeneration({
crMode: attemptMode,
haikuModel,
synopsisDocMaxChars: attemptMode === 'per_chunk_synopsis' ? SYNOPSIS_DOC_MAX_CHARS : undefined,
});
// ── PHASE 2: single DB transaction ───────────────────────────
+37 -2
View File
@@ -1,7 +1,7 @@
/**
* v0.41.16.0 Built-in conversation parser pattern registry.
*
* Fourteen hand-vetted patterns covering the chat-export formats this
* Fifteen hand-vetted patterns covering the chat-export formats this
* codebase is most likely to encounter. Each pattern's regex was
* derived from a public format reference (source_doc field) so future
* maintainers can verify against the wild shape.
@@ -50,7 +50,7 @@ export function cleanSpeaker(raw: string, override?: RegExp): string {
return stripped || raw.trim();
}
/** The 14 hand-vetted built-in patterns. */
/** The 15 hand-vetted built-in patterns. */
export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
// -------------------------------------------------------------------
// INLINE-DATE patterns (date in every line; less ambiguous; tried first).
@@ -178,6 +178,41 @@ export const BUILTIN_PATTERNS: readonly PatternEntry[] = [
'OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts (see your OpenClaw skills/meeting-ingestion/SKILL.md)',
},
{
// iMessage sync's time-only 12-hour shape. AM/PM is required so this
// cannot shadow bold-paren-time's 24-hour form or imessage-slack's
// full-date form.
id: 'bold-paren-time-12h',
origin: 'builtin',
regex: /^\*\*(.+?)\*\*\s*\((\d{1,2}):(\d{2})\s*(AM|PM|am|pm)\)\s*:\s*(.*)$/,
captures: {
speaker_group: 1,
hour_group: 2,
minute_group: 3,
ampm_group: 4,
text_group: 5,
},
date_source: 'frontmatter',
time_format: '12h_ampm',
timezone_policy: 'utc_assumed_with_warn',
multi_line: false,
quick_reject: /^\*\*/,
test_positive: [
'**Me** (9:04 AM): sounds good, see you then',
'**+155****0135** (9:39 AM): Will do',
'**Alice Example** (12:00 PM): noon message',
'**Bob Example** (5:38 pm): lowercase ampm',
],
test_negative: [
'**Alice** (00:00): 24h shape',
'**Alice Example** (2024-03-15 9:00 AM): full-date iMessage shape',
'**[18:37] G T:** telegram bracket',
'Alice (9:00 AM): missing the bold',
],
source_doc:
'Time-only 12h AM/PM iMessage export shape: `**Speaker** (H:MM AM): text`',
},
{
// Fathom/phone-call raw transcripts in this workspace use a plain
// `Speaker A: ...` / `Speaker B: ...` shape with no per-line time.
+13 -2
View File
@@ -321,11 +321,22 @@ export function applyPattern(
if (!body) return [];
const out: MatchedMessage[] = [];
const lines = body.split(/\r?\n/);
// Some multi-day conversation exports use markdown date headings instead
// of repeating a date on every message. Keep the caller's context immutable
// while advancing a local date anchor as those headings are encountered.
const runningCtx: DateContext = { ...dateCtx };
const dateHeaderRe = /^#{1,4}\s+(\d{4}-\d{2}-\d{2})\s*$/;
for (let i = 0; i < lines.length; i++) {
const rawLine = lines[i];
const line = rawLine.trim();
if (!line) continue;
const dateHeader = dateHeaderRe.exec(line);
if (dateHeader) {
runningCtx.fallbackDate = dateHeader[1];
continue;
}
// Quick-reject fast path.
if (entry.quick_reject && !entry.quick_reject.test(line)) {
// Continuation handling for orphan lines.
@@ -339,7 +350,7 @@ export function applyPattern(
const m = entry.regex.exec(line);
if (m) {
const iso = buildIso(m, entry, dateCtx);
const iso = buildIso(m, entry, runningCtx);
if (iso === null) continue; // reconstruction failed; skip line
const rawSpeaker = m[entry.captures.speaker_group] ?? '';
const speaker = cleanSpeaker(rawSpeaker, entry.speaker_clean);
@@ -380,7 +391,7 @@ function getNonBlankLines(body: string, headCap?: number): string[] {
* window) and `scorePatternFull` (whole body) delegate here so the
* quick_reject + regex loop lives in one place. Reused by
* `parseConversation`'s fallback path which pre-splits ONCE and
* passes the array to all 12 candidates (saves 11 redundant body
* passes the array to all 15 candidates (saves 14 redundant body
* splits per fallback pass).
*/
function scoreFromLines(
+56 -3
View File
@@ -479,6 +479,37 @@ export interface CycleOpts {
* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).
*/
sourceId?: string;
/**
* issue #2860 one-shot per-invocation bypass of a phase's own
* `dream.<phase>.enabled` / `cycle.<phase>.enabled` config gate. Wired
* from `gbrain dream --phase <name> --once`.
*
* Deliberately typed as the SINGLE named `CyclePhase`, not a boolean
* each gated phase's dispatch block below only honors the override when
* `onceForPhase` matches ITS OWN phase name, so the bypass can never leak
* to a different phase even if a caller passes a wider `phases` array
* than the CLI does (the CLI always restricts to `phases: [phase]`).
*
* Never reads or writes config the phase still evaluates its config
* gate every call; this only overrides the boolean OUTCOME for that one
* call. Applies to: patterns, synthesize, conversation_facts_backfill,
* enrich_thin, skillopt (the phases that gate on a `.enabled` config
* key read inside the phase's own module). Does NOT apply to
* extract_atoms / synthesize_concepts those are pack-gated via
* `packDeclaresPhase`, a different mechanism with its own existing
* one-shot escape hatch (`--drain` for extract_atoms).
*/
onceForPhase?: CyclePhase;
/**
* Absolute wall-clock deadline (epoch ms) of the enclosing minion job,
* from `MinionJobContext.deadlineAtMs` (the claim-time `timeout_at`
* stamp). Phases that spawn bounded sub-work (patterns' subagent) clamp
* their own timeouts to the REMAINING time so one phase's fixed
* worst-case can't blow past the job budget and dead-letter the whole
* cycle mid-phase (#2781). Unset for direct callers (`gbrain dream`)
* phases then use their configured timeouts unchanged.
*/
deadlineAtMs?: number | null;
}
// ─── Lock primitives ───────────────────────────────────────────────
@@ -956,6 +987,12 @@ async function runPhaseExtract(
dryRun: boolean,
changedSlugs?: string[],
signal?: AbortSignal,
// #1503: the brain source the cycle is scoped to (cycleSourceId — explicit
// --source or resolved from brainDir). Threaded to runExtractCore so
// fs-walk link/timeline rows carry source_id; without it addLinksBatch maps
// missing → 'default' and its pages JOIN drops every row on a federated
// brain ("Links: created 0 from N pages" every cycle).
sourceId?: string,
): Promise<PhaseResult> {
try {
const { runExtractCore } = await import('../commands/extract.ts');
@@ -978,6 +1015,7 @@ async function runPhaseExtract(
dir: brainDir,
slugs: changedSlugs, // undefined = full walk (first run / manual)
signal,
sourceId,
});
const linksCreated = result?.links_created ?? 0;
const timelineCreated = result?.timeline_entries_created ?? 0;
@@ -1675,6 +1713,10 @@ export async function runCycle(
from: opts.synthFrom,
to: opts.synthTo,
bypassDreamGuard: opts.synthBypassDreamGuard,
// #1586: scope synthesized writes to the cycle's resolved source
// (explicit --source wins, else derived from the checkout dir).
sourceId: cycleSourceId,
once: opts.onceForPhase === 'synthesize',
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
@@ -1706,7 +1748,7 @@ export async function runCycle(
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
// is undefined → extract falls back to full walk (safe default).
progress.start('cycle.extract');
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, opts.signal));
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, opts.signal, cycleSourceId));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
@@ -1878,6 +1920,8 @@ export async function runCycle(
brainDir,
dryRun,
yieldDuringPhase: opts.yieldDuringPhase,
once: opts.onceForPhase === 'patterns',
deadlineAtMs: opts.deadlineAtMs ?? null,
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
@@ -2093,7 +2137,11 @@ export async function runCycle(
progress.start('cycle.conversation_facts_backfill');
const { runPhaseConversationFactsBackfill } = await import('./cycle/conversation-facts-backfill.ts');
const { result, duration_ms } = await timePhase(() =>
runPhaseConversationFactsBackfill(engine, { dryRun, signal: opts.signal }),
runPhaseConversationFactsBackfill(engine, {
dryRun,
signal: opts.signal,
once: opts.onceForPhase === 'conversation_facts_backfill',
}),
);
result.duration_ms = duration_ms;
phaseResults.push(result);
@@ -2121,7 +2169,11 @@ export async function runCycle(
progress.start('cycle.enrich_thin');
const { runPhaseEnrichThin } = await import('./cycle/enrich-thin.ts');
const { result, duration_ms } = await timePhase(() =>
runPhaseEnrichThin(engine, { dryRun, signal: opts.signal }),
runPhaseEnrichThin(engine, {
dryRun,
signal: opts.signal,
once: opts.onceForPhase === 'enrich_thin',
}),
);
result.duration_ms = duration_ms;
phaseResults.push(result);
@@ -2152,6 +2204,7 @@ export async function runCycle(
runPhaseSkillopt({
engine,
dryRun,
once: opts.onceForPhase === 'skillopt',
...(opts.signal ? { signal: opts.signal } : {}),
}),
);
+25 -11
View File
@@ -57,6 +57,14 @@ import {
export interface ConversationFactsBackfillPhaseOpts {
dryRun?: boolean;
signal?: AbortSignal;
/**
* issue #2860 `gbrain dream --phase conversation_facts_backfill --once`.
* Bypasses the `cycle.conversation_facts_backfill.enabled` gate for THIS
* call only; never reads or writes config. Per-source + brain-wide cost/
* walltime caps still apply the override lifts the on/off switch, not
* the spend guards.
*/
once?: boolean;
}
/** Phase return shape (matches PhaseResult contract from cycle.ts). */
@@ -155,17 +163,23 @@ export async function runPhaseConversationFactsBackfill(
const cfg = await loadCfg(engine);
if (!cfg.enabled) {
return {
phase: 'conversation_facts_backfill',
status: 'skipped',
duration_ms: 0,
summary: 'cycle.conversation_facts_backfill.enabled=false (default OFF)',
details: {
reason: 'disabled',
enable_hint:
'gbrain config set cycle.conversation_facts_backfill.enabled true',
},
};
if (!opts.once) {
return {
phase: 'conversation_facts_backfill',
status: 'skipped',
duration_ms: 0,
summary: 'cycle.conversation_facts_backfill.enabled=false (default OFF)',
details: {
reason: 'disabled',
enable_hint:
'gbrain config set cycle.conversation_facts_backfill.enabled true',
},
};
}
process.stderr.write(
'[dream] --once: cycle.conversation_facts_backfill.enabled is false but ' +
'--phase conversation_facts_backfill --once forces this run (config untouched)\n',
);
}
const startedAt = Date.now();
+22 -10
View File
@@ -45,6 +45,12 @@ import {
export interface EnrichThinPhaseOpts {
dryRun?: boolean;
signal?: AbortSignal;
/**
* issue #2860 `gbrain dream --phase enrich_thin --once`. Bypasses the
* `cycle.enrich_thin.enabled` gate for THIS call only; never reads or
* writes config. Per-source + brain-wide cost/walltime caps still apply.
*/
once?: boolean;
}
export interface EnrichThinPhaseResult {
@@ -139,16 +145,22 @@ export async function runPhaseEnrichThin(
const cfg = await loadCfg(engine);
if (!cfg.enabled) {
return {
phase: 'enrich_thin',
status: 'skipped',
duration_ms: 0,
summary: 'cycle.enrich_thin.enabled=false (default OFF)',
details: {
reason: 'disabled',
enable_hint: 'gbrain config set cycle.enrich_thin.enabled true',
},
};
if (!opts.once) {
return {
phase: 'enrich_thin',
status: 'skipped',
duration_ms: 0,
summary: 'cycle.enrich_thin.enabled=false (default OFF)',
details: {
reason: 'disabled',
enable_hint: 'gbrain config set cycle.enrich_thin.enabled true',
},
};
}
process.stderr.write(
'[dream] --once: cycle.enrich_thin.enabled is false but ' +
'--phase enrich_thin --once forces this run (config untouched)\n',
);
}
const startedAt = Date.now();
+125 -30
View File
@@ -11,15 +11,17 @@
* 1. Reads the markdown body (DB-side fetch via engine.getPage).
* 2. Parses the `## Facts` fence with parseFactsFence.
* 3. Maps ParsedFact FenceExtractedFact via extractFactsFromFenceText.
* 4. Wipes the page's DB index via deleteFactsForPage.
* 5. Re-inserts via engine.insertFacts batch.
* 4. De-dupes rows by canonical (claim, source) content key.
* 5. Reconciles the page-scoped DB index: no-op when already in sync,
* insert only missing keys when possible, or wipe/reinsert when stale
* DB rows need cleanup (#1781 the unconditional wipe-and-reinsert
* made every cycle non-idempotent, re-appending duplicate rows).
*
* After the phase, the DB index for every affected page byte-matches
* the fence (modulo embeddings + runtime-derived fields). Pages with
* no fence go through delete-then-empty-insert DB rows for that
* page coordinate are wiped; legacy NULL-source_markdown_slug rows
* survive because deleteFactsForPage targets source_markdown_slug =
* slug only.
* After the phase, the DB index for every affected page matches the
* fence's canonical (claim, source) row set (modulo embeddings +
* runtime-derived fields). Pages with no fence wipe DB rows for that
* page coordinate only; legacy NULL-source_markdown_slug rows survive
* because deleteFactsForPage targets source_markdown_slug = slug only.
*
* Empty-fence guard (Codex R2-#7): the phase refuses to do its
* destructive reconciliation pass when legacy rows (row_num IS NULL,
@@ -35,7 +37,11 @@ import type { BrainEngine } from '../engine.ts';
import { writeReceipt } from '../extract/receipt-writer.ts';
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
import { parseFactsFence } from '../facts-fence.ts';
import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts';
import {
extractFactsFromFenceText,
FENCE_SOURCE_DEFAULT,
type FenceExtractedFact,
} from '../facts/extract-from-fence.ts';
import {
runPhantomRedirectPass,
emptyPhantomPassResult,
@@ -44,6 +50,51 @@ import {
import { embed, isAvailable } from '../ai/gateway.ts';
import { isAborted } from '../abort-check.ts';
interface ExistingPageFact {
fact: string;
source: string | null;
row_num: number | string | null;
}
function factContentKey(fact: string, source: string | null | undefined): string {
return `${fact}\u0000${source ?? FENCE_SOURCE_DEFAULT}`;
}
function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFact[] {
const seen = new Set<string>();
const deduped: FenceExtractedFact[] = [];
for (const fact of facts) {
const key = factContentKey(fact.fact, fact.source);
if (seen.has(key)) continue;
seen.add(key);
deduped.push(fact);
}
return deduped;
}
/**
* Fence-owned DB rows for one page coordinate. Excludes `cli:`-origin
* conversation facts (#1928) they are not fence-owned, so they must
* neither count as "stale" (which would force a wipe every cycle) nor
* be compared against the fence's row set. Mirrors the
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
*/
async function listExistingFactsForPage(
engine: BrainEngine,
slug: string,
sourceId: string,
): Promise<ExistingPageFact[]> {
return engine.executeRaw<ExistingPageFact>(
`SELECT fact, source, row_num
FROM facts
WHERE source_id = $1
AND source_markdown_slug = $2
AND COALESCE(source, '') NOT LIKE 'cli:%'
ORDER BY row_num ASC, id ASC`,
[sourceId, slug],
);
}
export interface ExtractFactsOpts {
/** Subset of slugs to reconcile. undefined = walk every page in the brain. */
slugs?: string[];
@@ -220,28 +271,70 @@ export async function runExtractFacts(
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
if (opts.dryRun) continue;
// Wipe-and-reinsert per page. The delete targets source_markdown_slug =
// slug only, so NULL-source_markdown_slug legacy rows survive (the
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation
// facts from extract-conversation-facts) are NOT fence-owned — the page
// carries no `## Facts` fence to recreate them — so they MUST survive this
// reconcile. Exclude them from the wipe.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
});
result.factsDeleted += deleted.deleted;
if (parsed.facts.length === 0) continue;
// v0.35.4 (D-ENG-1) — thread page.effective_date as the fallback
// valid_from. Without this, fence rows without explicit `validFrom:`
// land with `valid_from = now()` (import timestamp) and every
// trajectory query against the page returns import dates instead of
// claim dates.
const pageEffectiveDate = page.effective_date ? new Date(page.effective_date) : null;
const extracted = extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate });
const extracted = dedupeFactsByContentKey(
extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate }),
);
if (opts.dryRun) continue;
// #1781 — reconcile instead of unconditional wipe-and-reinsert. Compare
// the fence's canonical (claim, source) row set against the page's
// fence-owned DB rows: no-op when already in sync, insert only missing
// keys when possible, wipe/reinsert only when stale rows need cleanup.
const existing = await listExistingFactsForPage(engine, slug, sourceId);
const existingKeys = new Set(existing.map(f => factContentKey(f.fact, f.source)));
const desiredByKey = new Map(extracted.map(f => [factContentKey(f.fact, f.source), f]));
if (extracted.length === 0) {
if (existing.length > 0) {
// The delete targets source_markdown_slug = slug only, so
// NULL-source_markdown_slug legacy rows survive (the
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
// (conversation facts from extract-conversation-facts) are NOT
// fence-owned — the page carries no `## Facts` fence to recreate
// them — so they MUST survive this reconcile.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
});
result.factsDeleted += deleted.deleted;
}
continue;
}
const hasStaleExisting = existing.some(f => !desiredByKey.has(factContentKey(f.fact, f.source)));
const hasDuplicateExisting = existing.length !== existingKeys.size;
const hasRowNumDrift = existing.some(f => {
const desired = desiredByKey.get(factContentKey(f.fact, f.source));
return desired !== undefined && Number(f.row_num) !== desired.row_num;
});
if (
existing.length === extracted.length &&
!hasStaleExisting &&
!hasDuplicateExisting &&
!hasRowNumDrift
) {
continue;
}
let toInsert = extracted.filter(f => !existingKeys.has(factContentKey(f.fact, f.source)));
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
// Fall back to the legacy page-level reconcile when old DB rows must
// be removed. Same delete scoping as above: legacy
// NULL-source_markdown_slug rows and `cli:`-origin conversation
// facts (#1928) survive.
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
excludeSourcePrefixes: ['cli:'],
});
result.factsDeleted += deleted.deleted;
toInsert = extracted;
}
// v0.35.4 (D-CDX-3) — batch-embed before insert. Without this,
// cycle-inserted facts land with `embedding = NULL`, which breaks
@@ -250,17 +343,17 @@ export async function runExtractFacts(
// unavailable (no API key configured), facts still insert with
// NULL embeddings — drift_score gracefully returns null and
// clustering falls back to recency.
if (isAvailable('embedding') && extracted.length > 0) {
if (isAvailable('embedding') && toInsert.length > 0) {
try {
const texts = extracted.map(e => e.fact);
const texts = toInsert.map(e => e.fact);
// #1972: forward the abort signal so a cancelled cycle's in-flight
// batch embed (a network call) is itself abortable, not just the loop.
const embeddings = await embed(texts, { abortSignal: opts.signal });
// Defensive: embed should return one vector per input; if the
// gateway returns a partial array (provider partial-batch retry
// returning fewer than requested), only fill what we have.
for (let i = 0; i < extracted.length && i < embeddings.length; i++) {
extracted[i].embedding = embeddings[i];
for (let i = 0; i < toInsert.length && i < embeddings.length; i++) {
toInsert[i].embedding = embeddings[i];
}
} catch (err) {
// Embedding failure is non-fatal — facts still get inserted, just
@@ -271,7 +364,9 @@ export async function runExtractFacts(
}
}
const inserted = await engine.insertFacts(extracted, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
if (toInsert.length === 0) continue;
const inserted = await engine.insertFacts(toInsert, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
result.factsInserted += inserted.inserted;
}
+182 -41
View File
@@ -19,7 +19,7 @@
*/
import { join, dirname } from 'node:path';
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
import { mkdirSync, writeFileSync } from 'node:fs';
import type { BrainEngine } from '../engine.ts';
import type { PhaseResult, PhaseError } from '../cycle.ts';
import { MinionQueue } from '../minions/queue.ts';
@@ -27,11 +27,73 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
import { serializeMarkdown } from '../markdown.ts';
import type { Page, PageType } from '../types.ts';
// #2415: allow-list + output-root resolution shared with the synthesize
// phase — both phases must agree on the configured namespace.
import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts';
import { probeChatModel } from '../ai/gateway.ts';
import { normalizeModelId } from '../model-id.ts';
export interface PatternsPhaseOpts {
brainDir: string;
dryRun: boolean;
yieldDuringPhase?: () => Promise<void>;
/**
* issue #2860 `gbrain dream --phase patterns --once`. Bypasses the
* `dream.patterns.enabled` gate for THIS call only; never reads or
* writes config.
*/
once?: boolean;
/**
* Absolute deadline (epoch ms) of the enclosing minion job, or null for
* direct callers (`gbrain dream`). When set, the subagent's job timeout
* and the wait timeout are clamped so the phase finishes (or times out)
* BEFORE the parent job's budget expires a fixed 30/35-min default
* inside an interval-derived cycle budget dead-letters the whole cycle
* mid-phase and starves every tail phase (#2781).
*/
deadlineAtMs?: number | null;
}
/**
* Stop-margin reserved under the parent deadline when clamping subagent
* budgets. NOT a promise that tail phases complete the cycle is allowed
* to go partial and resume next tick. This only guarantees the phase's
* wait returns and the handler unwinds cleanly before the worker's abort
* fires: wait poll interval (5s) + worker force-evict grace (30s) + lock
* and DB cleanup headroom.
*/
export const CYCLE_DEADLINE_RESERVE_MS = 60 * 1000;
/**
* Smallest remaining budget worth submitting a subagent for. Below this,
* the LLM call is near-certain to be killed mid-flight wasted spend and
* a guaranteed-timeout child so the phase skips honestly instead
* (`insufficient_cycle_budget`) and the next cycle retries with a fresh
* budget.
*/
export const MIN_PATTERNS_SUBAGENT_BUDGET_MS = 2 * 60 * 1000;
/**
* Clamp the configured subagent budgets to the remaining parent-job time.
* Both timeouts derive from the SAME absolute child deadline
* (`deadlineAtMs - reserve`) so the child job's kill switch and our wait
* agree. Returns null when the remaining budget is below the minimum
* caller should skip the phase without submitting.
*/
export function clampSubagentBudgets(
config: { subagentTimeoutMs: number; subagentWaitTimeoutMs: number },
deadlineAtMs: number | null | undefined,
nowMs: number,
): { timeoutMs: number; waitTimeoutMs: number } | null {
if (deadlineAtMs == null) {
return { timeoutMs: config.subagentTimeoutMs, waitTimeoutMs: config.subagentWaitTimeoutMs };
}
const childBudgetMs = deadlineAtMs - CYCLE_DEADLINE_RESERVE_MS - nowMs;
if (childBudgetMs < MIN_PATTERNS_SUBAGENT_BUDGET_MS) return null;
return {
timeoutMs: Math.min(config.subagentTimeoutMs, childBudgetMs),
waitTimeoutMs: Math.min(config.subagentWaitTimeoutMs, childBudgetMs),
};
}
export async function runPhasePatterns(
@@ -43,11 +105,17 @@ export async function runPhasePatterns(
const config = await loadPatternsConfig(engine);
if (!config.enabled) {
return skipped('disabled', 'dream.patterns.enabled is false');
if (!opts.once) {
return skipped('disabled', 'dream.patterns.enabled is false');
}
process.stderr.write(
'[dream] --once: dream.patterns.enabled is false but ' +
'--phase patterns --once forces this run (config untouched)\n',
);
}
// Gather reflections within lookback window.
const reflections = await gatherReflections(engine, config.lookbackDays);
const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot);
if (reflections.length < config.minEvidence) {
return skipped(
'insufficient_evidence',
@@ -63,27 +131,51 @@ export async function runPhasePatterns(
});
}
// Submit one subagent for pattern detection.
if (!process.env.ANTHROPIC_API_KEY) {
return skipped('no_api_key', 'ANTHROPIC_API_KEY unset; pattern detection skipped');
// Submit one subagent for pattern detection. The subagent dispatches via
// the gateway model-tier resolver, so gate on "is the resolved model's
// provider reachable" rather than ANTHROPIC_API_KEY specifically — a
// hardcoded env gate misclassified non-Anthropic stacks (litellm,
// deepseek, openrouter, ...) as "no upstream" even though the subagent
// routes them through the gateway (agent.use_gateway_loop), and it missed
// Anthropic keys set via `gbrain config set anthropic_api_key`. Same
// probe semantics as think/index.ts + synthesize's makeJudgeClient:
// unknown provider/model or Anthropic-without-key skips cheaply; other
// providers' auth is checked lazily at dispatch and surfaces in the job
// outcome. (Takeover of PR #2279's intent by @brettdavies.)
const probe = probeChatModel(normalizeModelId(config.model));
if (!probe.ok) {
return skipped('no_provider', `pattern detection skipped: ${probe.detail}`);
}
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
if (allowedSlugPrefixes.length === 0) {
return failed(makeError('InternalError', 'NO_ALLOWLIST',
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
}
// #2781: budget the subagent from the REMAINING parent-job time, not
// the fixed config default. Checked after the cheap gates (disabled /
// insufficient_evidence / no_provider) so a skip for budget reasons
// only fires when the phase would otherwise have submitted.
const budgets = clampSubagentBudgets(config, opts.deadlineAtMs, Date.now());
if (budgets === null) {
return skipped(
'insufficient_cycle_budget',
`remaining cycle budget under ${Math.round(MIN_PATTERNS_SUBAGENT_BUDGET_MS / 1000)}s ` +
`(reserve ${Math.round(CYCLE_DEADLINE_RESERVE_MS / 1000)}s); next cycle retries with a fresh budget`,
);
}
const queue = new MinionQueue(engine);
const data: SubagentHandlerData = {
prompt: buildPatternsPrompt(reflections, config.minEvidence),
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
model: config.model,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
};
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
timeout_ms: 30 * 60 * 1000,
timeout_ms: budgets.timeoutMs,
};
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
allowProtectedSubmit: true,
@@ -92,13 +184,23 @@ export async function runPhasePatterns(
let outcome: string;
try {
const final = await waitForCompletion(queue, job.id, {
timeoutMs: 35 * 60 * 1000,
timeoutMs: budgets.waitTimeoutMs,
pollMs: 5 * 1000,
});
outcome = final.status;
} catch (e) {
if (e instanceof TimeoutError) outcome = 'timeout';
else throw e;
if (e instanceof TimeoutError) {
outcome = 'timeout';
// The child's own timeout_ms clock starts at ITS claim, not at
// submit — a child that sat queued behind other work can outlive
// the parent deadline this wait was clamped to. Cancel it so the
// subagent can't keep spending/writing after the phase gave up
// (waiting child → cancelled immediately; active child → lock
// stripped, worker abort fires on next renew tick).
try { await queue.cancelJob(job.id); } catch { /* best-effort */ }
} else {
throw e;
}
}
if (opts.yieldDuringPhase) {
@@ -113,13 +215,47 @@ export async function runPhasePatterns(
// Reverse-write to fs.
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
return ok(`${writtenRefs.length} pattern page(s) written/updated (${outcome})`, {
const details = {
reflections_considered: reflections.length,
patterns_written: writtenRefs.length,
reverse_write_count: reverseWriteCount,
child_outcome: outcome,
job_id: job.id,
});
};
// #2782: the phase status must reflect the child outcome. Pre-fix this
// returned status:ok even when the subagent timed out (e.g. no
// subagent-capable worker slot free for the whole wait window) and zero
// pattern pages were written — a silent no-op for days.
if (outcome !== 'complete') {
if (writtenRefs.length === 0) {
return {
phase: 'patterns',
status: 'fail',
duration_ms: 0,
summary: `pattern-detection subagent job ${job.id} ended '${outcome}'; nothing was written`,
details,
error: makeError(
outcome === 'timeout' ? 'Timeout' : 'InternalError',
`PATTERNS_CHILD_${outcome.toUpperCase()}`,
`subagent job ${job.id} outcome '${outcome}' with zero pattern pages written`,
outcome === 'timeout'
? 'A timeout with zero writes usually means no subagent-capable worker claimed the job. Check `gbrain jobs list` and worker capacity.'
: undefined,
),
};
}
// Partial: the child died/timed out but some pages landed first.
return {
phase: 'patterns',
status: 'warn',
duration_ms: 0,
summary: `${writtenRefs.length} pattern page(s) written but subagent job ${job.id} ended '${outcome}'`,
details,
};
}
return ok(`${writtenRefs.length} pattern page(s) written/updated (${outcome})`, details);
} catch (e) {
return failed(makeError('InternalError', 'PATTERNS_PHASE_FAIL',
e instanceof Error ? (e.message || 'patterns phase threw') : String(e)));
@@ -135,6 +271,22 @@ interface PatternsConfig {
lookbackDays: number;
minEvidence: number;
model: string;
/** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */
outputRoot: string;
/** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */
subagentTimeoutMs: number;
/** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */
subagentWaitTimeoutMs: number;
}
const DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS = 30 * 60 * 1000;
const DEFAULT_PATTERNS_SUBAGENT_WAIT_TIMEOUT_MS = 35 * 60 * 1000;
async function getNumberConfig(engine: BrainEngine, key: string, fallback: number): Promise<number> {
const raw = await engine.getConfig(key);
if (raw === undefined || raw === null) return fallback;
const value = Number(raw);
return Number.isNaN(value) ? fallback : value;
}
async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig> {
@@ -155,6 +307,13 @@ async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig>
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
model,
outputRoot: await loadOutputRoot(engine),
subagentTimeoutMs: await getNumberConfig(
engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS,
),
subagentWaitTimeoutMs: await getNumberConfig(
engine, 'dream.patterns.subagent_wait_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_WAIT_TIMEOUT_MS,
),
};
}
@@ -169,16 +328,19 @@ interface ReflectionRef {
async function gatherReflections(
engine: BrainEngine,
lookbackDays: number,
outputRoot = 'wiki',
): Promise<ReflectionRef[]> {
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
// #2415: reflections live under the configured output root (bound as a
// parameter; outputRoot is slug-grammar-validated by loadOutputRoot).
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
`SELECT slug, title, compiled_truth
FROM pages
WHERE slug LIKE 'wiki/personal/reflections/%'
WHERE slug LIKE $2
AND updated_at >= $1::timestamptz
ORDER BY updated_at DESC
LIMIT 100`,
[since],
[since, `${outputRoot}/personal/reflections/%`],
);
return rows.map(r => ({
slug: r.slug,
@@ -189,7 +351,7 @@ async function gatherReflections(
// ── Prompt ────────────────────────────────────────────────────────────
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string {
const today = new Date().toISOString().slice(0, 10);
const corpus = reflections
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
@@ -199,15 +361,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number):
OUTPUT POLICY
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks).
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
- Pattern slug format: \`${outputRoot}/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
DO NOT WRITE
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
- Patterns with <${minEvidence} reflections cited.
- Anything outside wiki/personal/patterns/.
- Anything outside ${outputRoot}/personal/patterns/.
CONTEXT
- Today: ${today}
@@ -298,27 +460,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string {
);
}
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
async function loadAllowedSlugPrefixes(): Promise<string[]> {
const candidates = [
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
];
for (const path of candidates) {
if (!existsSync(path)) continue;
try {
const raw = readFileSync(path, 'utf8');
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
const globs = parsed?.dream_synthesize_paths?.globs;
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
return globs as string[];
}
} catch { /* try next */ }
}
return [];
}
// ── Status helpers ───────────────────────────────────────────────────
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
+5 -3
View File
@@ -39,7 +39,7 @@
import { randomUUID, createHash } from 'node:crypto';
import { BaseCyclePhase, type ScopedReadOpts, type BasePhaseOpts } from './base-phase.ts';
import { chat as gatewayChat } from '../ai/gateway.ts';
import { chat as gatewayChat, getChatModel } from '../ai/gateway.ts';
import { writeReceipt } from '../extract/receipt-writer.ts';
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
import { GBrainError } from '../types.ts';
@@ -330,6 +330,8 @@ class ProposeTakesPhase extends BaseCyclePhase {
opts.reporter.start('propose_takes.pages' as never, pages.length);
}
const modelId = opts.model ?? getChatModel();
for (const page of pages) {
result.pages_scanned += 1;
this.tick(opts);
@@ -359,7 +361,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
// Budget pre-check before the LLM call. Estimate: ~1500 input tokens + 500 output.
const budget = this.checkBudget({
modelId: opts.model ?? 'claude-sonnet-4-6',
modelId,
estimatedInputTokens: 1500,
maxOutputTokens: 500,
});
@@ -408,7 +410,7 @@ class ProposeTakesPhase extends BaseCyclePhase {
p.weight,
p.domain ?? null,
JSON.stringify(existingTakes),
opts.model ?? 'claude-sonnet-4-6',
modelId,
],
);
result.proposals_inserted += 1;
+18 -8
View File
@@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts';
import type { ProgressReporter } from '../progress.ts';
import { writeReceipt } from '../extract/receipt-writer.ts';
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
import { chat as gatewayChat } from '../ai/gateway.ts';
import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts';
// #2163: concept pages route through importFromContent (the same
// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage,
// so they land in the retrieval surface (content_chunks + embeddings) where
// source-boost's 1.3× 'concepts/' weighting can actually reach them.
import { importFromContent } from '../import-file.ts';
import { serializeMarkdown } from '../markdown.ts';
const DEFAULT_BUDGET_USD = 1.5;
const TIER_T1_MIN = 10;
@@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts(
if (!opts.dryRun) {
const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug;
await engine.putPage(`concepts/${title}`, {
title: title.replace(/-/g, ' '),
type: 'concept',
compiled_truth: narrative,
frontmatter: {
type: 'concept',
// #2163: serialize to markdown and import via the canonical pipeline so
// the page is chunked (+ embedded when a provider is configured) —
// mirrors put_page's isAvailable('embedding') → noEmbed gate.
const md = serializeMarkdown(
{
tier: group.tier,
mention_count: group.atomTitles.length,
composite_score: group.atomTitles.length,
synthesized_at: new Date().toISOString(),
synthesized_by: 'synthesize_concepts-v0.41',
},
timeline: '',
narrative,
'',
{ type: 'concept', title: title.replace(/-/g, ' '), tags: [] },
);
await importFromContent(engine, `concepts/${title}`, md, {
noEmbed: !isAvailable('embedding'),
});
}
conceptsWritten++;
+164 -25
View File
@@ -75,6 +75,8 @@ const MIN_PROMPT_TOKENS = 100_000;
const DEFAULT_MAX_CHUNKS = 24;
/** Conservative default budget when model is unknown (200K × HEADROOM_RATIO). */
const UNKNOWN_MODEL_BUDGET_TOKENS = 180_000;
const DEFAULT_SUBAGENT_TIMEOUT_MS = 30 * 60 * 1000;
const DEFAULT_SUBAGENT_WAIT_TIMEOUT_MS = 35 * 60 * 1000;
/**
* Compute per-chunk character budget for the resolved model + config override.
@@ -242,6 +244,21 @@ export interface SynthesizePhaseOpts {
* the synthesize loop. Caller must opt in explicitly.
*/
bypassDreamGuard?: boolean;
/**
* #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts
* explicit --source wins, else derived from the checkout dir). Threaded to
* every subagent child as `source_id` so put_page writes land in this
* source, and stamped onto collected refs so reverse-writes read the
* correct (source_id, slug) row. Unset legacy 'default'.
*/
sourceId?: string;
/**
* issue #2860 `gbrain dream --phase synthesize --once`. Bypasses the
* `dream.synthesize.enabled` gate for THIS call only (does NOT bypass
* the `session_corpus_dir` not-configured check there's nothing to
* run without a corpus). Never reads or writes config.
*/
once?: boolean;
}
export async function runPhaseSynthesize(
@@ -275,8 +292,14 @@ export async function runPhaseSynthesize(
'dream.synthesize.session_corpus_dir is unset');
}
if (!opts.inputFile && !config.enabled) {
return skipped('not_configured',
'dream.synthesize.enabled is explicitly false');
if (!opts.once) {
return skipped('not_configured',
'dream.synthesize.enabled is explicitly false');
}
process.stderr.write(
'[dream] --once: dream.synthesize.enabled is false but ' +
'--phase synthesize --once forces this run (config untouched)\n',
);
}
// Cooldown check (skipped for explicit --input / --date / --from / --to runs).
@@ -397,7 +420,7 @@ export async function runPhaseSynthesize(
// Fan-out: submit one subagent per worth-processing transcript (or one
// per chunk for transcripts that exceed the model's per-prompt budget).
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
if (allowedSlugPrefixes.length === 0) {
return failed(makeError('InternalError', 'NO_ALLOWLIST',
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
@@ -460,10 +483,13 @@ export async function runPhaseSynthesize(
: config.model;
for (let i = 0; i < chunks.length; i++) {
const childData: SubagentHandlerData = {
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot),
model: subagentModel,
max_turns: 30,
allowed_slug_prefixes: allowedSlugPrefixes,
// #1586: scope every child tool call to the cycle's resolved source
// so put_page writes land there instead of the hardcoded 'default'.
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
};
// Idempotency key parity:
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
@@ -478,7 +504,7 @@ export async function runPhaseSynthesize(
max_stalled: 3,
on_child_fail: 'continue',
idempotency_key,
timeout_ms: 30 * 60 * 1000, // 30 min per chunk
timeout_ms: config.subagentTimeoutMs,
};
const child = await queue.add(
'subagent',
@@ -499,7 +525,7 @@ export async function runPhaseSynthesize(
for (const jobId of childIds) {
try {
const job = await waitForCompletion(queue, jobId, {
timeoutMs: 35 * 60 * 1000,
timeoutMs: config.subagentWaitTimeoutMs,
pollMs: 5 * 1000,
});
childOutcomes.push({ jobId, status: job.status });
@@ -522,20 +548,29 @@ export async function runPhaseSynthesize(
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
// even if Sonnet drops the chunk suffix.
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
// (source, slug) row (currently always 'default' from subagent put_page).
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
// source (children write there via SubagentHandlerData.source_id).
const cycleSourceId = opts.sourceId ?? 'default';
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
const summaryDate = opts.date ?? today();
// #2569: persist the dream-output identity marker into the DB frontmatter
// of every child-written page BEFORE reverse-rendering, so generated pages
// are queryable (`frontmatter->>'dream_generated'`) and a later put_page
// write-through (which re-renders from the DB row) can't erase the stamp.
await stampDreamProvenance(engine, writtenRefs, summaryDate);
// Dual-write: reverse-render each DB row → markdown file.
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId);
// Summary index page (deterministic; orchestrator-written via direct
// engine.putPage so no allow-list path needed).
const summaryDate = opts.date ?? today();
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
const writtenSlugs = writtenRefs.map(r => r.slug);
if (SUMMARY_SLUG_RE.test(summarySlug)) {
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId);
}
// Write completion timestamp ON SUCCESS only.
@@ -593,6 +628,27 @@ interface SynthConfig {
* `dream.synthesize.max_chunks_per_transcript`.
*/
maxChunksPerTranscript: number;
/**
* #2415: top-level namespace for synthesized output (reflections, originals,
* patterns). Config key `dream.synthesize.output_root`; default 'wiki'
* zero behavior change unless set. No trailing slash. Must satisfy the slug
* grammar; invalid values fall back to 'wiki' with a stderr warning.
*/
outputRoot: string;
subagentTimeoutMs: number;
subagentWaitTimeoutMs: number;
}
/** #2415: shared output-root resolution (synthesize + patterns phases). */
export async function loadOutputRoot(engine: BrainEngine): Promise<string> {
const raw = await engine.getConfig('dream.synthesize.output_root');
if (!raw) return 'wiki';
const trimmed = raw.trim().replace(/^\/+|\/+$/g, '');
if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed;
process.stderr.write(
`[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`,
);
return 'wiki';
}
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
@@ -621,6 +677,16 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
const cooldownHoursStr = await engine.getConfig('dream.synthesize.cooldown_hours');
const maxPromptTokensStr = await engine.getConfig('dream.synthesize.max_prompt_tokens');
const maxChunksStr = await engine.getConfig('dream.synthesize.max_chunks_per_transcript');
const subagentTimeoutMs = await getNumberConfig(
engine,
'dream.synthesize.subagent_timeout_ms',
DEFAULT_SUBAGENT_TIMEOUT_MS,
);
const subagentWaitTimeoutMs = await getNumberConfig(
engine,
'dream.synthesize.subagent_wait_timeout_ms',
DEFAULT_SUBAGENT_WAIT_TIMEOUT_MS,
);
let excludePatterns: string[] = ['medical', 'therapy'];
if (excludeStr) {
@@ -658,9 +724,23 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
maxPromptTokens,
maxChunksPerTranscript,
outputRoot: await loadOutputRoot(engine),
subagentTimeoutMs,
subagentWaitTimeoutMs,
};
}
async function getNumberConfig(
engine: BrainEngine,
key: string,
fallback: number,
): Promise<number> {
const raw = await engine.getConfig(key);
if (raw === undefined || raw === null) return fallback;
const value = Number(raw);
return Number.isNaN(value) ? fallback : value;
}
async function checkCooldown(
engine: BrainEngine,
hours: number,
@@ -677,7 +757,13 @@ async function checkCooldown(
// ── Allow-list source of truth ───────────────────────────────────────
async function loadAllowedSlugPrefixes(): Promise<string[]> {
/**
* #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the
* configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki'
* returns the globs verbatim. Shared by the patterns phase (imported there
* the two phases must enforce the same allow-list).
*/
export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise<string[]> {
// Search a few known locations relative to the binary / repo. The first
// hit wins; if none found, return [].
const candidates = [
@@ -691,7 +777,10 @@ async function loadAllowedSlugPrefixes(): Promise<string[]> {
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
const globs = parsed?.dream_synthesize_paths?.globs;
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
return globs as string[];
if (outputRoot === 'wiki') return globs as string[];
return (globs as string[]).map(g =>
g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g,
);
}
} catch { /* try next */ }
}
@@ -939,6 +1028,7 @@ function buildSynthesisPrompt(
chunkIdx: number,
chunkTotal: number,
priorContradictionsBlock = '',
outputRoot = 'wiki',
): string {
const dateHint = t.inferredDate ?? today();
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
@@ -964,13 +1054,14 @@ OUTPUT POLICY (ALL of these are required)
2. Cross-reference compulsively: every new page MUST contain at least one wikilink (e.g., \`[ref](people/jane-doe)\` or \`[[people/jane-doe]]\`) to existing brain content. Use the search tool to find existing pages first.
3. Do NOT write to any path outside the allow-list shown in the put_page schema.
4. Slug discipline: lowercase alphanumeric and hyphens only, slash-separated segments. NO underscores, NO file extensions.
5. Self-contained opening: begin every new page's body with a 2-3 sentence summary that a reader unfamiliar with this transcript could understand on its own, before any quotes or detail. Do not assume the reader has the source conversation for context.
TASKS
A. Reflections (self-knowledge, pattern recognition, emotional processing):
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
slug: \`${outputRoot}/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
B. Originals (new ideas, frames, theses, mental models):
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
slug: \`${outputRoot}/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries your job is the reflection/original synthesis, NOT modifying existing person pages).
@@ -1011,6 +1102,7 @@ async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
chunkInfo: Map<number, { idx: number; hash6: string }>,
sourceId = 'default',
): Promise<Array<{ slug: string; source_id: string }>> {
if (childIds.length === 0) return [];
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
@@ -1020,10 +1112,10 @@ async function collectChildPutPageSlugs(
//
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
// put_page tool schema doesn't expose source_id (subagents are scoped to
// a single source); default to 'default' for the current dream-cycle
// product behavior. Threading the source_id through reverseWriteRefs
// guarantees getPage targets the correct (source, slug) row instead of
// the first DB match.
// a single source). #1586: the orchestrator scopes each child to the
// cycle's resolved source via SubagentHandlerData.source_id, and stamps
// the SAME source here so reverseWriteRefs / provenance reads target the
// correct (source_id, slug) row. Unset → legacy 'default'.
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
`SELECT job_id,
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
@@ -1039,7 +1131,7 @@ async function collectChildPutPageSlugs(
const ci = chunkInfo.get(r.job_id);
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
}
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
}
/**
@@ -1068,12 +1160,52 @@ async function hasLegacySingleChunkCompletion(
return rows.length > 0;
}
// ── Dream-provenance DB stamp (#2569) ────────────────────────────────
/**
* Persist the dream-output identity marker (`dream_generated: true` +
* `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page
* a synthesize child wrote. Render-time `frontmatterOverrides` alone only
* reach the markdown FILE the DB row stayed unstamped, so DB consumers
* couldn't enumerate generated pages and a later put_page write-through
* (which re-renders from the DB row) silently erased the marker.
*
* Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb
* never JSON.stringify into a ::jsonb cast; engine-parity safe, no new
* engine method). Best-effort per row: a stamp failure never kills the
* phase (the render-time override still covers the file).
*/
async function stampDreamProvenance(
engine: BrainEngine,
refs: Array<{ slug: string; source_id: string }>,
cycleDate: string,
): Promise<void> {
if (refs.length === 0) return;
const { executeRawJsonb } = await import('../sql-query.ts');
for (const { slug, source_id } of refs) {
try {
await executeRawJsonb(
engine,
`UPDATE pages
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
WHERE slug = $1 AND source_id = $2`,
[slug, source_id],
[{ dream_generated: true, dream_cycle_date: cycleDate }],
);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`);
}
}
}
// ── Reverse-write DB rows → markdown files ───────────────────────────
async function reverseWriteRefs(
engine: BrainEngine,
brainDir: string,
refs: Array<{ slug: string; source_id: string }>,
nativeSourceId = 'default',
): Promise<number> {
let count = 0;
for (const { slug, source_id } of refs) {
@@ -1084,10 +1216,11 @@ async function reverseWriteRefs(
const tags = await engine.getTags(slug, { sourceId: source_id });
try {
const md = renderPageToMarkdown(page, tags);
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
// so same-slug-different-source pages don't collide. Default-source
// pages stay at brainDir/<slug>.md so single-source brains see no change.
const filePath = source_id === 'default'
// v0.32.8 F6: foreign-source pages land at brainDir/.sources/<id>/<slug>.md
// so same-slug-different-source pages don't collide. Pages belonging to
// the cycle's own source (#1586: brainDir IS that source's checkout —
// legacy 'default' when unscoped) stay at brainDir/<slug>.md.
const filePath = source_id === nativeSourceId
? join(brainDir, `${slug}.md`)
: join(brainDir, '.sources', source_id, `${slug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
@@ -1134,6 +1267,7 @@ async function writeSummaryPage(
summaryDate: string,
writtenSlugs: string[],
childOutcomes: Array<{ jobId: number; status: string }>,
sourceId = 'default',
): Promise<void> {
const completed = childOutcomes.filter(c => c.status === 'completed').length;
const failed = childOutcomes.length - completed;
@@ -1171,13 +1305,15 @@ async function writeSummaryPage(
// unnecessarily; we go straight to the engine.
const { parseMarkdown } = await import('../markdown.ts');
const parsed = parseMarkdown(fullMarkdown);
// #1586: summary lands in the cycle's resolved source too — otherwise the
// children live in the named source while the index drifts to 'default'.
await engine.putPage(summarySlug, {
type: parsed.type,
title: parsed.title,
compiled_truth: parsed.compiled_truth,
timeline: parsed.timeline,
frontmatter: parsed.frontmatter,
});
}, { sourceId });
// Also write to disk (orchestrator dual-write).
try {
@@ -1242,4 +1378,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P
// double-encoded jsonb regression). Not part of the runtime contract.
export const __testing = {
collectChildPutPageSlugs,
buildSynthesisPrompt,
stampDreamProvenance,
reverseWriteRefs,
};
+16
View File
@@ -206,6 +206,22 @@ export async function embedStaleForSource(
chunk_source: c.chunk_source,
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
// Carry through per-chunk metadata. upsertChunks writes these as
// EXCLUDED.<col> (not COALESCE), so omitting them here resets image
// rows to modality='text' (breaking the image search arm's
// modality='image' filter) and wipes code-chunk symbol metadata on
// every embed-stale pass. embedding_image is deliberately NOT
// carried: the upsert COALESCEs it, and getChunks returns the
// pgvector as a string which upsertChunks would mis-serialize.
modality: c.modality ?? undefined,
language: c.language ?? undefined,
symbol_name: c.symbol_name ?? undefined,
symbol_type: c.symbol_type ?? undefined,
start_line: c.start_line ?? undefined,
end_line: c.end_line ?? undefined,
parent_symbol_path: c.parent_symbol_path ?? undefined,
doc_comment: c.doc_comment ?? undefined,
symbol_name_qualified: c.symbol_name_qualified ?? undefined,
}));
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
+23 -2
View File
@@ -29,6 +29,9 @@ import {
isOpenAITextEmbedding3Model,
isValidOpenAITextEmbedding3Dim,
maxOpenAITextEmbedding3Dim,
nvidiaEmbeddingDim,
nvidiaEmbeddingDimOptions,
supportsNvidiaEmbeddingDimension,
} from './ai/dims.ts';
/**
@@ -366,7 +369,9 @@ function validateDimAgainstTouchpoint(
dimsOptions: number[] | undefined,
requestedDims: number | undefined,
): ResolveSchemaDimResult {
const dim = requestedDims ?? defaultDims;
const nvidiaNaturalDims = recipe.id === 'nvidia' ? nvidiaEmbeddingDim(modelId) : undefined;
const effectiveDefaultDims = nvidiaNaturalDims ?? defaultDims;
const dim = requestedDims ?? effectiveDefaultDims;
if (!Number.isInteger(dim) || dim <= 0) {
return {
@@ -396,7 +401,7 @@ function validateDimAgainstTouchpoint(
dim,
model: `${recipe.id}:${modelId}`,
provider: recipe.id,
recipeDefault: defaultDims,
recipeDefault: effectiveDefaultDims,
};
}
@@ -411,6 +416,22 @@ function isCustomDimValidForProvider(
requestedDims: number,
dimsOptions: number[] | undefined,
): CustomDimCheck {
// NVIDIA models are mixed: some fixed-dim, one Matryoshka-style. Handle
// them before generic recipe dims_options so llama-nemotron can use 1280d.
if (recipe.id === 'nvidia') {
const naturalDims = nvidiaEmbeddingDim(modelId);
if (naturalDims !== undefined && requestedDims === naturalDims) return { valid: true, error: '' };
if (supportsNvidiaEmbeddingDimension(modelId, requestedDims)) return { valid: true, error: '' };
const options = nvidiaEmbeddingDimOptions(modelId);
return {
valid: false,
error:
`NVIDIA model "${modelId}" does not support dimensions ${requestedDims}. ` +
`Natural dimensions: ${naturalDims ?? 'unknown'}. ` +
(options ? `Supported overrides: ${options.join(', ')}.` : 'No dimension overrides are supported for this NVIDIA model.'),
};
}
// Tier 1: recipe-declared dims_options.
if (dimsOptions && dimsOptions.length > 0) {
if (dimsOptions.includes(requestedDims)) return { valid: true, error: '' };
+3
View File
@@ -37,6 +37,9 @@ export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
// Mistral (https://mistral.ai/pricing/api/, verified 2026-07-19)
'mistral:mistral-embed': { pricePerMTok: 0.10 },
'mistral:mistral-embed-2312': { pricePerMTok: 0.10 },
};
export type PriceLookupResult =
+21
View File
@@ -936,6 +936,27 @@ export interface BrainEngine {
// Search
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
/**
* fix/title-retrieval-arm (D1): page-grain title candidate arm.
*
* content_chunks.search_vector never includes the page TITLE (it is
* doc_comment + symbol_name_qualified + chunk_text), so a page whose
* title tokens are absent from its body is unreachable by searchKeyword.
* This arm queries the PAGE-GRAIN DOCUMENT vector pages.search_vector
* NOT titles alone: per trg_pages_search_vector it is title (weight 'A')
* + compiled_truth ('B') + timeline text ('C'). Ranked by ts_rank_cd,
* the 'A'-weighted title dominates, but body/timeline matches also
* produce (lower-ranked) candidates. Returns page-grain hits joined to
* ONE representative chunk per page (compiled_truth preferred, else
* lowest chunk_index) so rows are shaped like searchKeyword's output and
* can enter RRF fusion in hybridSearch.
*
* Deliberately NO query-length gating unlike the alias hop (6-token
* guard) and the title-phrase re-rank boost, this arm must GENERATE
* candidates for long exact-title queries, which is exactly where
* chunk-grain AND FTS is weakest.
*/
searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
/**
* Hydrate embeddings for chunks already known by id. v0.36 (D9):
+6 -2
View File
@@ -15,7 +15,7 @@
import type { BrainEngine } from './engine.ts';
import type { TakeBatchInput, TakeKind } from './engine.ts';
import { chat, isAvailable } from './ai/gateway.ts';
import { chat, getChatModel, isAvailable } from './ai/gateway.ts';
export const ALLOWED_PAGE_TYPES = [
'concept', 'atom', 'lore', 'briefing', 'writing', 'originals',
@@ -190,7 +190,11 @@ export async function extractTakesFromPages(
let response: { text: string };
try {
response = await chat({
model: opts.model ?? 'anthropic:claude-haiku-4-5',
// #2997 — default to the configured chat model (file-plane gateway
// config, same idiom as enrich.ts) instead of hardcoded cloud Haiku.
// On OAuth/local-only installs the hardcoded model made every takes
// extraction die with llm_unavailable despite a working chat_model.
model: opts.model || getChatModel(),
system: CLASSIFIER_SYSTEM,
messages: [
{
+51 -12
View File
@@ -67,6 +67,23 @@ export async function getFactsExtractionModel(engine?: BrainEngine): Promise<str
return normalizeModelId(resolved);
}
/**
* #2113: output-token cap for the extractor call. The pre-fix hardcoded 1500
* silently truncated output on mandatory-reasoning models (thinking tokens
* count toward the cap), so the JSON never parsed and extraction returned
* zero facts with no signal. Configurable via
* `gbrain config set facts.extraction_max_tokens <n>`; default 4000.
*/
export const DEFAULT_EXTRACTION_MAX_TOKENS = 4000;
export async function getFactsExtractionMaxTokens(engine?: BrainEngine): Promise<number> {
if (!engine) return DEFAULT_EXTRACTION_MAX_TOKENS;
const raw = await engine.getConfig('facts.extraction_max_tokens').catch(() => null);
if (raw == null || raw.trim() === '') return DEFAULT_EXTRACTION_MAX_TOKENS;
const n = Number(raw);
return Number.isFinite(n) && n > 0 ? Math.floor(n) : DEFAULT_EXTRACTION_MAX_TOKENS;
}
export const ALL_EXTRACT_KINDS: readonly FactKind[] = [
'event', 'preference', 'commitment', 'belief', 'fact',
] as const;
@@ -164,24 +181,46 @@ export async function extractFactsFromTurn(input: ExtractInput): Promise<Extract
const cap = Math.max(1, Math.min(input.maxFactsPerTurn ?? 10, 25));
const defaultModel = await getFactsExtractionModel(input.engine);
const maxTokens = await getFactsExtractionMaxTokens(input.engine);
const model = input.model ?? defaultModel;
const userContent = `<turn>\n${cleaned}\n</turn>\n\nExtract up to ${cap} facts.${
input.entityHints && input.entityHints.length
? ` Known entity slugs the user already mentioned: ${input.entityHints.slice(0, 5).join(', ')}.`
: ''
}`;
let result: ChatResult;
try {
result = await chat({
model: input.model ?? defaultModel,
model,
system: EXTRACTOR_SYSTEM,
messages: [
{
role: 'user',
content: `<turn>\n${cleaned}\n</turn>\n\nExtract up to ${cap} facts.${
input.entityHints && input.entityHints.length
? ` Known entity slugs the user already mentioned: ${input.entityHints.slice(0, 5).join(', ')}.`
: ''
}`,
},
],
maxTokens: 1500,
messages: [{ role: 'user', content: userContent }],
maxTokens,
abortSignal: input.abortSignal,
});
// #2113: never checked pre-fix — a truncated response (stopReason
// 'length', e.g. reasoning tokens eating the cap on mandatory-reasoning
// models) produced unparseable JSON and silently extracted zero facts.
// Retry ONCE at double the cap, then surface the truncation loudly.
if (result.stopReason === 'length') {
process.stderr.write(
`[facts-extract] WARN: extractor output truncated at maxTokens=${maxTokens} ` +
`(model=${model}); retrying once at ${maxTokens * 2}\n`,
);
result = await chat({
model,
system: EXTRACTOR_SYSTEM,
messages: [{ role: 'user', content: userContent }],
maxTokens: maxTokens * 2,
abortSignal: input.abortSignal,
});
if (result.stopReason === 'length') {
process.stderr.write(
`[facts-extract] WARN: extractor output STILL truncated at maxTokens=${maxTokens * 2} ` +
`(model=${model}); facts for this turn are likely lost. ` +
`Raise the cap: gbrain config set facts.extraction_max_tokens <n>\n`,
);
}
}
} catch (err) {
// Re-throw aborts; absorb other errors as "no extraction" — caller's
// `put_page` backstop will still record the page itself.
+69
View File
@@ -0,0 +1,69 @@
/**
* Full-text search language configuration.
*
* Postgres tsvector/tsquery require a text search configuration name (e.g.
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
* 'english' across engines and trigger functions, which broke search
* quality for non-English brains (no stemming, no stop-word removal).
*
* This helper centralizes the choice. Default stays 'english' for backward
* compatibility only users who set GBRAIN_FTS_LANGUAGE see different
* behavior.
*
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
* portuguese stemmer) are supported as long as the configuration exists
* in the target Postgres instance. See docs/guides/multi-language-fts.md
* for setup instructions.
*
* Validation: only allow lowercase letters, digits, and underscores. This
* prevents SQL injection when the value is interpolated into queries
* (Postgres tsvector functions don't accept parameterized config names
* they must be literals or identifiers).
*/
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
const DEFAULT_LANGUAGE = 'english';
let cachedLanguage: string | null = null;
/**
* Returns the configured Postgres text search configuration name.
*
* Resolution order:
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
* 2. 'english' (default preserves existing behavior)
*
* The return value is safe to interpolate directly into SQL because it
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
* the default and emits a one-time warning.
*
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
*/
export function getFtsLanguage(): string {
if (cachedLanguage !== null) return cachedLanguage;
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
if (!raw) {
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
if (!VALID_CONFIG_NAME.test(raw)) {
console.warn(
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
`Falling back to '${DEFAULT_LANGUAGE}'.`
);
cachedLanguage = DEFAULT_LANGUAGE;
return cachedLanguage;
}
cachedLanguage = raw;
return cachedLanguage;
}
/**
* Resets the cached language. Tests only don't use in production code.
*/
export function resetFtsLanguageCache(): void {
cachedLanguage = null;
}
+77
View File
@@ -303,6 +303,83 @@ export function validateRepoState(
return 'healthy';
}
/**
* True if `path` is itself a git repo OR a subdirectory of one, per
* `git rev-parse --show-toplevel`. Mirrors the walk-up discovery
* `sync.ts:discoverGitRoot` performs at sync time (#753/#774 subdir-of-git
* sources are valid), so a directory that passes this check is guaranteed
* not to hit sync's "Not inside a git repository" error later. Used by
* `addSource` (#2707) to validate `--path` at registration time instead of
* deferring the failure to the first sync.
*/
export function isInsideGitRepo(path: string): boolean {
try {
execFileSync('git', ['-C', path, 'rev-parse', '--show-toplevel'], {
stdio: ['ignore', 'pipe', 'pipe'],
timeout: 10_000,
env: { ...process.env, ...GIT_ENV },
});
return true;
} catch {
return false;
}
}
/**
* The empty-tree object ID for `path`'s repo, derived (not hardcoded) so
* this works for both the default SHA-1 object format and the opt-in
* `--object-format=sha256` one (git 2.29+) each has its own empty-tree
* OID. `git hash-object -t tree --stdin < /dev/null` computes the hash of
* a zero-entry tree using whatever hash algorithm `path`'s repo is
* configured for, without needing to know which one that is. #2707 codex
* round 4 (P2): an earlier version hardcoded the well-known SHA-1 constant
* (`4b825dc6...`), which silently mismatched and so let an empty
* SHA-256 repo through on a SHA-256 repo's real (different) empty-tree
* OID.
*/
function emptyTreeOid(path: string): string {
return execFileSync('git', ['-C', path, 'hash-object', '-t', 'tree', '--stdin'], {
input: '',
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 10_000,
env: { ...process.env, ...GIT_ENV },
}).toString().trim();
}
/**
* True if `path`'s HEAD tree has at least one tracked entry scoped to
* `path` itself. `-C path` + the `HEAD:./` revision syntax resolves the
* tree object for `path` specifically (not the whole repo root), so this
* is correct for both a repo's toplevel AND a subdirectory-of-a-repo
* source then a single OID comparison against that repo's empty-tree
* object (see `emptyTreeOid`) tells us whether that tree is empty. #2707
* codex round 3 (P2): unlike listing (`ls-tree`), this is O(1) output no
* `maxBuffer` exposure on a repo with a very large number of entries.
*
* Subsumes "no commits at all" (`HEAD:./` on an unborn repo fails to
* resolve there's no HEAD) AND "has a HEAD commit but it's empty"
* (#2707 codex round 2): `git commit --allow-empty` followed by creating
* untracked files resolves `HEAD:./` successfully (to the empty-tree OID)
* but that tree has zero entries a directory that would pass a bare
* `rev-parse HEAD` check yet still can't sync (or worse, "succeeds"
* importing nothing and then never notices the untracked files change
* the silent-staleness class #2707 exists to prevent). A directory
* that's `git init`ed but never committed, or where this specific path
* was never `git add`ed, fails this check either way.
*/
export function hasTrackedContent(path: string): boolean {
try {
const out = execFileSync('git', ['-C', path, 'rev-parse', '--verify', 'HEAD:./'], {
stdio: ['ignore', 'pipe', 'pipe'],
timeout: 10_000,
env: { ...process.env, ...GIT_ENV },
});
return out.toString().trim() !== emptyTreeOid(path);
} catch {
return false;
}
}
// ── Durability helpers (v0.42.44) ───────────────────────────────────────────
// Used by the brain-repo durability feature (`gbrain sources harden/pull`) and
// the DB-free pull cron. These are the auth-capable, rebase-aware counterparts
+59
View File
@@ -0,0 +1,59 @@
import { execFileSync } from 'child_process';
import { lstatSync } from 'fs';
import { join } from 'path';
/**
* Return files visible to git from `dir`, respecting .gitignore,
* .git/info/exclude, and global git excludes. Returns null when `dir` is not
* inside a git work tree or git is unavailable, so callers can keep their
* existing filesystem-walk fallback.
*/
export function collectGitVisibleFiles(
dir: string,
acceptRelPath: (relPath: string) => boolean,
): string[] | null {
let stdout: string;
try {
stdout = execFileSync(
'git',
['-C', dir, 'ls-files', '--cached', '--others', '--exclude-standard', '-z'],
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
);
} catch {
return null;
}
const ignoredTracked = new Set<string>();
try {
const ignoredStdout = execFileSync(
'git',
['-C', dir, 'ls-files', '-ci', '--exclude-standard', '-z'],
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
);
for (const rel of ignoredStdout.split('\0')) {
if (rel) ignoredTracked.add(rel);
}
} catch {
// Best effort: older Git or unusual worktrees still get the standard list.
}
const files: string[] = [];
for (const rel of stdout.split('\0')) {
if (!rel) continue;
if (ignoredTracked.has(rel)) continue;
const normalizedRel = rel.replace(/\\/g, '/');
if (!acceptRelPath(normalizedRel)) continue;
const full = join(dir, rel);
let st: ReturnType<typeof lstatSync>;
try {
st = lstatSync(full);
} catch {
continue;
}
if (st.isSymbolicLink() || !st.isFile()) continue;
files.push(full);
}
return files.sort();
}
+36 -2
View File
@@ -1,5 +1,5 @@
import { existsSync, readFileSync, writeFileSync, renameSync, unlinkSync } from 'fs';
import { relative, isAbsolute } from 'path';
import { existsSync, readFileSync, writeFileSync, renameSync, unlinkSync, realpathSync } from 'fs';
import { relative, isAbsolute, resolve } from 'path';
/**
* Path-based import checkpoint.
@@ -25,6 +25,12 @@ import { relative, isAbsolute } from 'path';
* enter the set.
*/
export interface ImportCheckpoint {
/** Checkpoint payload schema. v1 is path-based with explicit producer metadata. */
schema_version: 1;
/** Producer marker for downstream consumers that validate before acting. */
owner: 'gbrain';
/** Checkpoint kind. Prevents unrelated checkpoint files from being treated as import state. */
kind: 'import';
/** Absolute brain directory the checkpoint was created against. Mismatch on resume → discard. */
dir: string;
/**
@@ -37,6 +43,21 @@ export interface ImportCheckpoint {
}
const OLD_FORMAT_LOG = 'Older checkpoint format detected — re-walking (cheap via content_hash)';
export const IMPORT_CHECKPOINT_SCHEMA_VERSION = 1;
export const IMPORT_CHECKPOINT_OWNER = 'gbrain';
export const IMPORT_CHECKPOINT_KIND = 'import';
/**
* Capture the import target once at run start. `resolve()` removes caller
* spelling such as `.` or `../staging`; `realpathSync()` collapses symlinks
* and proves the target exists. The returned value is the only directory
* identity import checkpoints should ever persist (#1728 a raw `.` here
* made the checkpoint `dir` resolve to whatever CWD the NEXT consumer ran
* from, which downstream tooling treated as an owned staging directory).
*/
export function resolveImportTargetDir(dir: string): string {
return realpathSync(resolve(dir));
}
/**
* Load a checkpoint and verify it's compatible with the current run.
@@ -72,11 +93,21 @@ export function loadCheckpoint(path: string, currentDir: string): ImportCheckpoi
}
if (typeof obj.dir !== 'string') return null;
if (!isAbsolute(obj.dir)) return null;
if (obj.dir !== currentDir) return null;
// Self-describing metadata (#1728): absent fields are tolerated (legacy
// path-based checkpoints predate them), but present-and-wrong means the
// file was written by something else — don't resume from it.
if (obj.schema_version !== undefined && obj.schema_version !== IMPORT_CHECKPOINT_SCHEMA_VERSION) return null;
if (obj.owner !== undefined && obj.owner !== IMPORT_CHECKPOINT_OWNER) return null;
if (obj.kind !== undefined && obj.kind !== IMPORT_CHECKPOINT_KIND) return null;
if (typeof obj.timestamp !== 'string') return null;
if (!obj.completedPaths.every((p): p is string => typeof p === 'string')) return null;
return {
schema_version: IMPORT_CHECKPOINT_SCHEMA_VERSION,
owner: IMPORT_CHECKPOINT_OWNER,
kind: IMPORT_CHECKPOINT_KIND,
dir: obj.dir,
completedPaths: obj.completedPaths,
timestamp: obj.timestamp,
@@ -98,6 +129,9 @@ export function saveCheckpoint(path: string, cp: ImportCheckpoint): void {
// Sort for stable serialization — keeps diffs across snapshots minimal
// and tests deterministic.
const payload: ImportCheckpoint = {
schema_version: IMPORT_CHECKPOINT_SCHEMA_VERSION,
owner: IMPORT_CHECKPOINT_OWNER,
kind: IMPORT_CHECKPOINT_KIND,
dir: cp.dir,
completedPaths: [...cp.completedPaths].sort(),
timestamp: cp.timestamp,
+5
View File
@@ -733,6 +733,11 @@ export async function importFromContent(
: computeCorpusGeneration({
crMode: effectiveCRMode,
haikuModel: 'anthropic:claude-haiku-4-5-20251001',
// Inline import-file path never uses per_chunk_synopsis (refuses
// upstream); pass undefined so the doc-cap field stays out of
// the hash here. Per_chunk_synopsis runs through the Minion
// backfill handler which threads SYNOPSIS_DOC_MAX_CHARS through
// the service layer.
});
// Transaction wraps all DB writes. Every per-page tx call carries the
+16 -1
View File
@@ -489,7 +489,22 @@ export async function extractPageLinks(
// text inside `[[...]]` before any `|`), NOT the display alias
// (ref.name = match[2]). `[[struktura|the project]]` must resolve
// `struktura`, not "the project". The display text is for context only.
const matches = await resolver.resolveBasenameMatches(ref.slug);
//
// The literal may be path-qualified (`[[notes/struktura]]`). The FS
// path (resolveSlugAll) strips the dirname before its basename lookup,
// but this path passed the raw literal to an index keyed by final
// segments only — so every slash-containing wikilink outside
// DIR_PATTERN silently resolved to nothing. Query by the final
// segment, then use the written path as a disambiguation filter
// (the analogue of the FS ancestor walk honoring the written path):
// a match must end with the literal, so `[[notes/struktura]]` can
// resolve to `vault/notes/struktura` but never to `wiki/struktura`.
const slashIdx = ref.slug.lastIndexOf('/');
const basename = slashIdx === -1 ? ref.slug : ref.slug.slice(slashIdx + 1);
let matches = await resolver.resolveBasenameMatches(basename);
if (slashIdx !== -1) {
matches = matches.filter(m => m === ref.slug || m.endsWith(`/${ref.slug}`));
}
if (matches.length === 0) continue;
const idx = content.indexOf(ref.slug);
const context = idx >= 0 ? excerpt(content, idx, 240) : ref.name;
+19 -19
View File
@@ -213,39 +213,39 @@ function collectValidationErrors(
return;
}
// 3. MISSING_CLOSE — find the next `---` after the opener. If a markdown
// heading appears before it, that's a strong signal the closing
// delimiter is missing (the heading was meant to be in the body).
// 3. MISSING_CLOSE — find the next `---` after the opener.
let closeLine = -1;
let headingBeforeClose = -1;
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
const t = lines[i].trim();
if (t === '---') {
if (lines[i].trim() === '---') {
closeLine = i;
break;
}
if (/^#{1,6}\s/.test(t) && headingBeforeClose === -1) {
headingBeforeClose = i;
}
}
if (closeLine === -1) {
// No closing fence found. Surface the first heading-shaped line as a
// hint for where the parser thinks the frontmatter went off the rails —
// only useful when the close is genuinely missing, since YAML allows
// `#` comment lines inside a closed fence (see comment below).
let headingHint = -1;
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
if (/^#{1,6}\s/.test(lines[i].trim())) {
headingHint = i;
break;
}
}
errors.push({
code: 'MISSING_CLOSE',
message:
headingBeforeClose >= 0
? `No closing --- before heading at line ${headingBeforeClose + 1}`
headingHint >= 0
? `No closing --- before heading at line ${headingHint + 1}`
: 'No closing --- delimiter found',
line: headingBeforeClose >= 0 ? headingBeforeClose + 1 : firstNonEmpty + 1,
line: headingHint >= 0 ? headingHint + 1 : firstNonEmpty + 1,
});
return;
}
if (headingBeforeClose >= 0 && headingBeforeClose < closeLine) {
errors.push({
code: 'MISSING_CLOSE',
message: `Heading at line ${headingBeforeClose + 1} found inside frontmatter zone (closing --- comes after)`,
line: headingBeforeClose + 1,
});
}
// Closing fence found. Content between opening and closing is YAML, which
// permits `#` comment lines anywhere — those are not markdown headings
// and must not raise MISSING_CLOSE.
// 4. EMPTY_FRONTMATTER — open and close present but nothing meaningful between.
const fmBody = lines.slice(firstNonEmpty + 1, closeLine).join('\n').trim();
+167 -1
View File
@@ -1,5 +1,6 @@
import type { BrainEngine } from './engine.ts';
import { slugifyPath } from './sync.ts';
import { getFtsLanguage } from './fts-language.ts';
/**
* Schema migrations run automatically on initSchema().
@@ -134,7 +135,10 @@ export const MIGRATIONS: Migration[] = [
}
}
}
if (renamed > 0) console.log(` Renamed ${renamed} slugs`);
// Migration progress goes to stderr — stdout must stay clean for
// callers parsing JSON (e.g. `gbrain doctor --json | jq`); migrations
// can run lazily inside ANY command's first DB connect.
if (renamed > 0) process.stderr.write(` Renamed ${renamed} slugs\n`);
},
},
{
@@ -5505,6 +5509,168 @@ export const MIGRATIONS: Migration[] = [
WHERE dimension IS NOT NULL;
`,
},
{
version: 123,
name: 'configurable_fts_language',
// Recreate the two search_vector trigger functions using the language
// configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent:
// CREATE OR REPLACE swaps the function body atomically; no trigger
// recreation needed since the trigger references the function by name.
//
// Why a handler instead of a static SQL string: Postgres tsvector
// functions don't accept parameterized config names — the language
// must be a literal in the SQL. getFtsLanguage() validates the value
// (lowercase letters/digits/underscores only) before interpolation.
//
// Function bodies mirror schema.sql / pglite-schema.ts exactly —
// INCLUDING the `SET search_path = pg_catalog, public` hardening from
// v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here
// would silently strip the hardening on every upgraded brain). Only
// the text-search config name is parameterized. Keep all copies in
// sync when the trigger logic changes.
//
// Backfill: after recreating the functions, re-tokenize existing rows
// under the new language. Skipped when the configured language is
// 'english' (trigger output identical — re-tokenizing is wasted I/O).
// To change language after this migration has run, use
// `gbrain reindex-search-vector`.
sql: '',
handler: async (engine) => {
const lang = getFtsLanguage();
const recreatePagesFn = `
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
const recreateChunksFn = `
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
await engine.executeRaw(recreatePagesFn);
await engine.executeRaw(recreateChunksFn);
if (lang === 'english') {
// stderr, NOT stdout: migrations run lazily inside any command's
// first DB connect — a console.log here polluted `doctor --json`
// stdout and broke jq consumers (heavy-tests fm_wallclock).
process.stderr.write(` v123: trigger functions recreated with language='english' (default — no backfill needed)\n`);
return;
}
// Backfill existing rows under the new tokenizer. UPDATE-to-same-value
// re-fires the pages trigger; chunks are rewritten directly with the
// same expression as the trigger.
await engine.executeRaw(`
UPDATE pages SET id = id
WHERE search_vector IS NOT NULL;
`);
await engine.executeRaw(`
UPDATE content_chunks
SET search_vector =
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')
WHERE search_vector IS NOT NULL;
`);
process.stderr.write(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows\n`);
},
},
{
version: 124,
name: 'page_search_vector_drop_compiled_truth',
// #2704: a single markdown page whose compiled_truth exceeds Postgres's
// hard 1,048,575-byte tsvector cap made update_page_search_vector()
// throw "string is too long for tsvector" INSIDE the pages UPSERT
// transaction — not a per-file ledger entry, a transaction abort. The
// whole source's sync checkpoint stayed pinned (Sync BLOCKED) until the
// oversized file was fixed or manually skipped, even though every
// OTHER file in the run imported fine.
//
// Fix: drop compiled_truth (the unbounded whole-page body) from this
// trigger. It was already redundant — content_chunks.search_vector
// (Cathedral II Layer 3, v0.20.0) is the ACTUAL keyword-search source:
// searchKeyword() in postgres-engine.ts/pglite-engine.ts ranks and
// queries `cc.search_vector` exclusively; `pages.search_vector` is
// written by this trigger but never read by any query in this
// codebase (verified: no `pages.search_vector`/bare `search_vector`
// appears on either side of a WHERE/ts_rank anywhere outside this
// trigger's own definition and the reindex/backfill machinery that
// maintains it). And chunking already bounds each chunk_text well
// under the tsvector limit (chunkText() targets embedding-sized
// pieces, several orders of magnitude smaller than 1MB) — the overflow
// was specific to the whole-page grain this trigger no longer builds.
//
// title + timeline (both naturally small — a compiled_truth-sized
// title or timeline field would be its own bug) stay, so
// pages.search_vector keeps carrying SOME signal rather than going
// fully inert; a future PR can drop the column outright once its
// last non-search consumer (if any turns up) is confirmed gone.
//
// No backfill: existing rows keep whatever search_vector they already
// computed until their next UPDATE (harmless — nothing reads this
// column, so staleness has zero behavioral effect). The brains that
// actually hit this bug never successfully wrote a value for the
// oversized page in the first place, so there's nothing stale to fix
// for them specifically — the NEXT sync of that exact file is what
// proves the fix, not a backfill of already-working rows.
//
// Function body mirrors reindex-search-vector.ts's recreatePagesFn
// (documented contract there: keep both in lockstep) and the fresh-
// install baselines in pglite-schema.ts / schema-embedded.ts — all
// four updated in the same commit as this migration.
sql: '',
handler: async (engine) => {
const lang = getFtsLanguage();
await engine.executeRaw(`
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`);
process.stderr.write(` v124: update_page_search_vector() no longer indexes compiled_truth (was overflowing tsvector on large pages, #2704)
`);
},
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+30 -1
View File
@@ -113,7 +113,36 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {
// by passing { noEmbed: false } in job.data.
const noEmbed = (data as { noEmbed?: unknown }).noEmbed !== false;
const result = await importFromContent(engine, slug, event.content, { noEmbed });
// #1522: thread the validated event's provenance into the page write
// instead of dropping it on the floor. source_kind / source_uri are
// pure provenance strings (no scoping power) and persist
// unconditionally via importFromContent's putPage write-through.
//
// event.source_id is the emitter's IngestionSource instance id, NOT
// necessarily a registered brain source (the webhook path fabricates
// `webhook-<clientId>`, which pages.source_id's FK would reject). It
// routes the page write only when BOTH hold:
// - the event is trusted (fail-closed: an untrusted webhook payload
// carries a caller-controlled x-gbrain-source-id header and must
// not get to choose its write source), AND
// - the id names a registered source row.
// Otherwise the write keeps the pre-fix default-source routing.
let sourceId: string | undefined;
if (!untrustedPayload) {
const rows = await engine.executeRaw<{ id: string }>(
`SELECT id FROM sources WHERE id = $1`,
[event.source_id],
);
if (rows.length > 0) sourceId = event.source_id;
}
const result = await importFromContent(engine, slug, event.content, {
noEmbed,
sourceId,
source_kind: event.source_kind,
source_uri: event.source_uri,
ingested_via: 'ingest_capture',
});
return {
slug,
+111 -3
View File
@@ -58,8 +58,29 @@ import { randomUUIDv7 } from 'bun';
const DEFAULT_MODEL = 'claude-sonnet-4-6';
const DEFAULT_MAX_TURNS = 20;
const DEFAULT_MAX_OUTPUT_TOKENS = 8192;
const DEFAULT_RATE_KEY = 'anthropic:messages';
/**
* Resolve the per-turn output-token cap (#2778). Per-job data wins, then the
* `agent.max_output_tokens` config row, then the 8192 default (was a
* hardcoded 4096 that made pages >~12KB unwritable via put_page). Invalid
* values (NaN / zero / negative) fall through to the next tier.
*/
export function resolveMaxOutputTokens(
perJob: number | undefined,
configRaw: string | null | undefined,
): number {
if (typeof perJob === 'number' && Number.isFinite(perJob) && perJob > 0) {
return Math.floor(perJob);
}
if (typeof configRaw === 'string' && configRaw.trim() !== '') {
const n = Number(configRaw);
if (Number.isFinite(n) && n > 0) return Math.floor(n);
}
return DEFAULT_MAX_OUTPUT_TOKENS;
}
/**
* Resolve the rate-lease cap from the env var.
*
@@ -212,6 +233,11 @@ export function makeSubagentHandler(deps: SubagentDeps) {
fallback: TIER_DEFAULTS.subagent,
});
const maxTurns = data.max_turns ?? DEFAULT_MAX_TURNS;
// #2778: per-turn output cap — data.max_tokens → config → 8192 default.
const maxOutputTokens = resolveMaxOutputTokens(
data.max_tokens,
await engine.getConfig('agent.max_output_tokens').catch(() => null),
);
// v0.41 Approach C: systemPrompt is now built AFTER toolDefs (a few
// lines below) so the renderer can splice a tool-usage preamble
// listing each available tool's usage_hint. The renderer is
@@ -246,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) {
config,
brainId: data.brain_id,
allowedSlugPrefixes: data.allowed_slug_prefixes,
// #1586: cycle-resolved source scope for tool-call OperationContexts.
sourceId: data.source_id,
});
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
? filterAllowedTools(registry, data.allowed_tools)
@@ -277,6 +305,7 @@ export function makeSubagentHandler(deps: SubagentDeps) {
systemPrompt,
toolDefs,
maxTurns,
maxOutputTokens,
});
}
@@ -475,12 +504,67 @@ export function makeSubagentHandler(deps: SubagentDeps) {
// covers the whole request. A mid-call renewal loop would add
// complexity; for v0.15 we lean on the 120s TTL + abort-on-signal.
try {
// --- Patch B (borrow-ahead, hand-authored): rolling conversation prompt-cache ---
// Direct path marks cache_control only on static system(485)+last-tool(498) ~5.2K;
// the growing anthroMessages conversation is re-billed every turn (6/18 v0.42.51
// regression). Anthropic caches up to the last cache_control block, so mark the last
// content block of the last message and keep the 4-breakpoint API limit
// (system + last-tool + 2 rolling = 4).
//
// Two rolling markers, not one: Anthropic's automatic cache lookup only walks
// back up to 20 content blocks from a breakpoint to find a prior cached prefix
// (see prompt-caching docs, "20-block lookback window"). A turn that adds more
// than 20 blocks since the last marker (e.g. a large parallel tool_use/tool_result
// round) would make a freshly-placed single marker miss the previous cache
// entirely. Keeping the immediately-preceding rolling marker in place — and only
// evicting anything older than that — guarantees at least that marker's prefix is
// still a valid, already-written cache read even when this turn's new marker's
// lookback comes up empty.
if (anthroMessages.length > 0) {
const markerIndices: number[] = [];
for (let i = 0; i < anthroMessages.length; i++) {
const m = anthroMessages[i] as any;
if (Array.isArray(m.content)) {
for (const b of m.content) {
if (b && typeof b === 'object' && 'cache_control' in b) {
markerIndices.push(i);
break;
}
}
}
}
const keepIdx = markerIndices.length > 0 ? markerIndices[markerIndices.length - 1] : -1;
for (const i of markerIndices) {
if (i === keepIdx) continue;
const m = anthroMessages[i] as any;
for (const b of m.content) {
if (b && typeof b === 'object' && 'cache_control' in b) delete b.cache_control;
}
}
const lastMsg = anthroMessages[anthroMessages.length - 1] as any;
// A fresh job's seed message has string content (see the
// `[{ role: 'user', content: data.prompt }]` init above), which
// the array-only check below would silently skip — leaving the
// very first call, and thus the common single-tool round-trip,
// with no rolling breakpoint at all. Normalize to a one-block
// array first so it gets the marker like every later turn.
if (typeof lastMsg.content === 'string') {
lastMsg.content = [{ type: 'text', text: lastMsg.content }];
}
if (Array.isArray(lastMsg.content) && lastMsg.content.length > 0) {
const lastBlock = lastMsg.content[lastMsg.content.length - 1];
if (lastBlock && typeof lastBlock === 'object') {
lastBlock.cache_control = { type: 'ephemeral' };
}
}
}
// --- end Patch B ---
const params: Anthropic.MessageCreateParamsNonStreaming = {
// v0.41 Bug 3: strip `provider:` prefix at the SDK call site only.
// `model` stays qualified everywhere else (persistence, recipe
// lookup at recipeIdFromModel(), capability gate).
model: stripProviderPrefix(model),
max_tokens: 4096,
max_tokens: maxOutputTokens,
system: [
{ type: 'text', text: systemPrompt, cache_control: { type: 'ephemeral' } },
] as any,
@@ -573,7 +657,10 @@ export function makeSubagentHandler(deps: SubagentDeps) {
b.type === 'tool_use',
);
if (toolUses.length === 0) {
stopReason = 'end_turn';
// #2778: an output-cap hit is NOT end_turn — the text (and possibly a
// dropped trailing tool_use block) is truncated. Surface it as its own
// stop_reason instead of silently reporting a clean end_turn.
stopReason = assistantMsg.stop_reason === 'max_tokens' ? 'max_tokens' : 'end_turn';
// Concatenate text blocks as the final answer.
finalText = blocks
.filter(b => b.type === 'text' && typeof b.text === 'string')
@@ -686,6 +773,24 @@ export function makeSubagentHandler(deps: SubagentDeps) {
}
}
// #2778: a max_tokens stop with tool_use blocks means the API dropped an
// incomplete trailing block (e.g. a large put_page body that overflowed
// the cap). Tell the model so it re-issues the cut-off call (split, or
// smaller pages) instead of assuming the write happened.
if (assistantMsg.stop_reason === 'max_tokens') {
toolResults.push({
type: 'text',
text: `[system] Your previous response hit the ${maxOutputTokens}-token output cap and was truncated; ` +
`any tool call cut off by the cap was DROPPED and did not execute. Re-issue it, splitting large content if needed.`,
} as ContentBlock);
logSubagentHeartbeat({
job_id: ctx.id,
event: 'llm_call_completed',
turn_idx: turnIdx,
error: `stop_reason=max_tokens at cap ${maxOutputTokens}; truncation note injected`,
});
}
// 6. Append the synthesized user turn (tool_result wrappers) to the
// conversation and persist it so replay picks it up.
const userIdx = nextMessageIdx++;
@@ -721,6 +826,8 @@ interface GatewayRunArgs {
systemPrompt: string;
toolDefs: ToolDef[];
maxTurns: number;
/** #2778: per-turn output-token cap (resolved by resolveMaxOutputTokens). */
maxOutputTokens: number;
}
/**
@@ -738,7 +845,7 @@ interface GatewayRunArgs {
* reconciler sees both shapes uniformly.
*/
async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResult> {
const { engine, ctx, data, model, systemPrompt, toolDefs, maxTurns } = args;
const { engine, ctx, data, model, systemPrompt, toolDefs, maxTurns, maxOutputTokens } = args;
// Map ToolDef → ChatToolDef (gateway shape). The gateway's chat() bridges
// this to provider-specific tool definitions via the Vercel AI SDK.
@@ -862,6 +969,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
tools: chatTools,
toolHandlers,
maxTurns,
maxTokens: maxOutputTokens,
abortSignal: ctx.signal,
cacheSystem,
// ALWAYS pass replayState (even on fresh runs) so the gateway loop's
+41 -7
View File
@@ -163,24 +163,36 @@ export class MinionQueue {
if (opts?.maxWaiting !== undefined) {
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
const backpressureQueue = opts?.queue ?? 'default';
// Multi-source scope: jobs of the same (name, queue) but different
// data.sourceId are independent workstreams (per-source sync/cycle).
// Counting them together made a waiting default-source sync swallow
// every other source's freshness sync — a secondary source sat 29h stale
// while dispatch logs showed its syncs "dispatched" (coalesced into
// the default row). Key the lock and the count on sourceId when the
// submission carries one; NULL keeps legacy single-scope behavior.
const bpSourceId = typeof (data as Record<string, unknown> | undefined)?.sourceId === 'string'
? (data as Record<string, unknown>).sourceId as string
: null;
await tx.executeRaw(
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2))`,
[jobName, backpressureQueue]
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2 || ':' || coalesce($3, '')))`,
[jobName, backpressureQueue, bpSourceId]
);
const waitingCountRows = await tx.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count
FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'`,
[jobName, backpressureQueue]
WHERE name = $1 AND queue = $2 AND status = 'waiting'
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)`,
[jobName, backpressureQueue, bpSourceId]
);
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
if (waitingCount >= maxWaiting) {
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
`SELECT * FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)
ORDER BY created_at DESC, id DESC
LIMIT 1`,
[jobName, backpressureQueue]
[jobName, backpressureQueue, bpSourceId]
);
if (existingWaiting.length > 0) {
const coalesced = rowToMinionJob(existingWaiting[0]);
@@ -471,12 +483,34 @@ export class MinionQueue {
});
}
/** Re-queue a failed or dead job for retry. */
/**
* Re-queue a failed or dead job for retry.
*
* #2783: an explicit `jobs retry` is an operator asserting "run this
* fresh" so it clears `started_at` (re-stamped on re-claim via
* `claim()`'s `COALESCE(started_at, now())`, `queue.ts:620`) and resets
* `attempts_made`/`attempts_started` to 0. Without this, `started_at`
* kept the ORIGINAL first-claim time, so `handleWallClockTimeouts()`
* (anchored on `now() - started_at`, `queue.ts:729-749`) could measure
* from long before the retry a retry issued more than `timeout_ms * 2`
* after the original claim was dead-lettered again in under a second,
* with `attempts_made` already past `max_attempts`. This made retry
* useless for exactly the case it exists for: recovering work after an
* outage that outlasted the job's timeout.
*
* Also resets `stalled_counter` (Codex review): `handleStalled()`
* dead-letters once `stalled_counter + 1 >= max_stalled` (`queue.ts:1190`).
* A job dead-lettered BY stall exhaustion, left un-reset, would hit that
* same threshold on its very first lock expiry after retry a job
* killed by 3 stalls doesn't get a fresh stall budget, contradicting
* "run this fresh" the same way the unreset attempt counters did.
*/
async retryJob(id: number): Promise<MinionJob | null> {
const rows = await this.engine.executeRaw<Record<string, unknown>>(
`UPDATE minion_jobs SET status = 'waiting', error_text = NULL,
lock_token = NULL, lock_until = NULL, delay_until = NULL,
finished_at = NULL, updated_at = now()
finished_at = NULL, started_at = NULL, attempts_made = 0,
attempts_started = 0, stalled_counter = 0, updated_at = now()
WHERE id = $1 AND status IN ('failed', 'dead')
RETURNING *`,
[id]
+24 -1
View File
@@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts';
import { operations } from '../../operations.ts';
import type { Operation, OperationContext } from '../../operations.ts';
import { paramDefToSchema } from '../../../mcp/tool-defs.ts';
import { validateSourceId } from '../../utils.ts';
import type { ToolCtx, ToolDef } from '../types.ts';
/**
@@ -61,6 +62,12 @@ export const BRAIN_TOOL_ALLOWLIST: ReadonlySet<string> = new Set([
'resolve_slugs',
'get_ingest_log',
'put_page',
// #2778: the canonical timeline-write op. Fenced exactly like put_page —
// operations.ts:enforceSubagentSlugFence confines the target slug to the
// trusted-workspace allow-list (or the wiki/agents/<id>/ namespace) when
// ctx.viaSubagent=true, so a subagent can only append timeline entries to
// pages it could have written anyway.
'add_timeline_entry',
// v0.29 — Salience + Anomaly Detection. Both read-only. `get_recent_transcripts`
// is intentionally NOT included: subagent calls always have ctx.remote=true,
// and the v0.29 trust gate rejects remote callers — adding it here would be
@@ -97,6 +104,7 @@ export const BRAIN_TOOL_USAGE_HINTS: Readonly<Record<string, string>> = {
resolve_slugs: 'Resolve free-form entity names to canonical slugs (e.g. "Alice" → `people/alice-example`). Use before any tool that takes a slug if the user gave a name not a slug.',
get_ingest_log: 'Read the brain ingestion log for diagnostic / verification queries.',
put_page: 'Write a markdown page to the gbrain DATABASE (NOT the local filesystem). Page becomes searchable + linkable. Slug must match the agent\'s allowed namespace.',
add_timeline_entry: 'Append a dated timeline entry to an existing page (the canonical timeline write). Use over rewriting the page body when recording a dated event. Slug must match the agent\'s allowed namespace.',
get_recent_salience: 'Read pages ranked by emotional + activity salience over a recency window. Use for "what\'s been on my mind lately".',
find_anomalies: 'Read cohort-level activity outliers (e.g. tag-cohort or type-cohort with unusual recent volume). Use for "what\'s unusual lately".',
};
@@ -194,6 +202,13 @@ export interface BuildBrainToolsOpts {
* SubagentHandlerData.allowed_slug_prefixes via the handler.
*/
allowedSlugPrefixes?: readonly string[];
/**
* Brain source every tool-call OperationContext is scoped to (#1586).
* Trusted (flows from SubagentHandlerData.source_id, which only
* PROTECTED_JOB_NAMES-gated submitters can set); validated at build time.
* Unset legacy 'default'.
*/
sourceId?: string;
}
interface OpContextDeps {
@@ -204,6 +219,7 @@ interface OpContextDeps {
signal?: AbortSignal;
brainId?: string;
allowedSlugPrefixes?: readonly string[];
sourceId?: string;
}
function buildOpContext(deps: OpContextDeps): OperationContext {
@@ -217,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
},
dryRun: false,
remote: true, // match MCP trust boundary for auto-link skip
sourceId: 'default', // v0.34 D4: required; subagent tools default to host source
// #1586: cycle-resolved source when provided; legacy host default else.
sourceId: deps.sourceId ?? 'default',
jobId: deps.jobId,
subagentId: deps.subagentId,
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
@@ -241,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name),
);
// #1586: fail fast on a malformed source id before any tool executes
// (defense-in-depth — the seam is trusted, but the value round-trips
// through the job payload).
if (opts.sourceId !== undefined) validateSourceId(opts.sourceId);
return picked.map<ToolDef>(op => {
const schema = op.name === 'put_page'
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
@@ -270,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
signal: ctx.signal,
brainId: opts.brainId,
allowedSlugPrefixes: opts.allowedSlugPrefixes,
sourceId: opts.sourceId,
});
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
return op.handler(opCtx, params);
+24
View File
@@ -200,6 +200,12 @@ export interface MinionJobContext {
attempts_made: number;
/** AbortSignal for cooperative cancellation (fires on timeout, cancel, pause, or lock loss). */
signal: AbortSignal;
/** Absolute wall-clock deadline (epoch ms) from the claim-time `timeout_at` stamp,
* or null when the job has no per-job timeout. This is the DB's ground truth
* the same instant handleTimeouts() dead-letters against so handlers that
* spawn bounded sub-work (e.g. autopilot-cycle's subagent phases) can budget
* from the REMAINING time instead of a fixed constant that may exceed it. */
deadlineAtMs: number | null;
/** AbortSignal that fires only on worker process SIGTERM/SIGINT. Handlers sensitive
* to deploy restarts (e.g. the shell handler, which must run a SIGTERM 5s SIGKILL
* sequence on its child) listen to this in addition to `signal`. Most handlers can
@@ -411,6 +417,12 @@ export interface SubagentHandlerData {
model?: string;
/** Max assistant turns before the loop fails with stop_reason='max_turns'. */
max_turns?: number;
/**
* Per-turn max output tokens (#2778). Resolution: this field
* `agent.max_output_tokens` config 8192 default. The pre-#2778
* hardcoded 4096 made pages >~12KB unwritable via put_page.
*/
max_tokens?: number;
/**
* Whitelist of tool names the agent may call. MUST be a subset of the
* derived registry names invalid entries are rejected at tool-dispatch
@@ -449,6 +461,17 @@ export interface SubagentHandlerData {
* and direct CLI submitters set it.
*/
allowed_slug_prefixes?: string[];
/**
* Brain source the subagent's tool calls are scoped to (#1586).
*
* When set, every tool-call `OperationContext.sourceId` uses this value
* instead of the legacy 'default', so put_page writes land in the cycle's
* resolved source. Same trust story as `allowed_slug_prefixes`:
* PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and
* direct CLI submitters can set it. Validated via `validateSourceId` at
* tool-registry build time.
*/
source_id?: string;
/**
* v0.41 Approach C: opt out of the auto-generated tool-usage preamble
* that `buildSystemPrompt()` splices into `system`. Default behavior
@@ -562,6 +585,7 @@ export type ContentBlock =
export type SubagentStopReason =
| 'end_turn' // Anthropic says end_turn and last message has no tool_use
| 'max_turns' // hit max_turns budget before end_turn
| 'max_tokens' // final turn hit the output-token cap — result text is TRUNCATED (#2778)
| 'refusal' // detected via stop_reason + content shape
| 'error'; // unrecoverable (empty response retry exhausted, etc.)
+38 -10
View File
@@ -507,7 +507,17 @@ export class MinionWorker extends EventEmitter {
try {
await this.queue.promoteDelayed();
} catch (e) {
console.error('Promotion error:', e instanceof Error ? e.message : String(e));
const msg = e instanceof Error ? e.message : String(e);
console.error('Promotion error:', msg);
// issue #1491: a retryable pool/connection loss during promotion used
// to be logged and ignored, leaving the worker in a repeated
// "Promotion error: No database connection" loop until a later path
// happened to reconnect or crash. Promotion is a standalone UPDATE
// from delayed→waiting, so after a connection failure we can safely
// rebuild the worker-owned pool before continuing to claim work.
if (isRetryableConnError(e)) {
await this.reconnectAfterConnectionError('promoteDelayed', e);
}
}
// Claim jobs up to concurrency limit
@@ -532,13 +542,7 @@ export class MinionWorker extends EventEmitter {
if (!isRetryableConnError(e)) throw e;
const msg = e instanceof Error ? e.message : String(e);
console.error(`[worker] claim hit a connection error; reconnecting, retry on next tick: ${msg}`);
const reconnect = (this.engine as { reconnect?: () => Promise<void> }).reconnect;
if (reconnect) {
try { await reconnect.call(this.engine); }
catch (re) {
console.error(`[worker] reconnect after claim error failed: ${re instanceof Error ? re.message : String(re)}`);
}
}
await this.reconnectAfterConnectionError('claim', e);
await new Promise(resolve => setTimeout(resolve, this.opts.pollInterval));
continue;
}
@@ -658,6 +662,22 @@ export class MinionWorker extends EventEmitter {
this.running = false;
}
/**
* Rebuild the worker-owned DB pool after a retryable connection failure.
*
* PostgresEngine exposes reconnect(); PGLite and test doubles may not. Absence
* is a no-op so non-Postgres workers preserve their legacy behavior.
*/
private async reconnectAfterConnectionError(site: string, error: unknown): Promise<void> {
const reconnect = (this.engine as { reconnect?: (ctx?: { error?: unknown }) => Promise<void> }).reconnect;
if (!reconnect) return;
try {
await reconnect.call(this.engine, { error });
} catch (re) {
console.error(`[worker] reconnect after ${site} error failed: ${re instanceof Error ? re.message : String(re)}`);
}
}
/** RSS watchdog. Called from the per-job finally and the periodic timer.
* Idempotent: returns early if already not running or already shut down.
* When threshold is exceeded, hands off to gracefulShutdown(). */
@@ -880,15 +900,22 @@ export class MinionWorker extends EventEmitter {
// Per-job wall-clock timeout (timer-armed only if `timeout_ms` was
// set on the job; the grace-evict pattern above now lives outside
// this branch).
// this branch). The delay derives from the claim-time `timeout_at`
// stamp when present so this timer, the DB sweeper (handleTimeouts),
// and the handler-visible `deadlineAtMs` all agree on ONE absolute
// deadline instead of three clocks started at slightly different
// instants.
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
if (job.timeout_ms != null) {
const delayMs = job.timeout_at != null
? Math.max(0, job.timeout_at.getTime() - Date.now())
: job.timeout_ms;
timeoutTimer = setTimeout(() => {
if (!abort.signal.aborted) {
console.warn(`Job ${job.id} (${job.name}) hit per-job timeout (${job.timeout_ms}ms), aborting`);
abort.abort(new Error('timeout'));
}
}, job.timeout_ms);
}, delayMs);
}
const promise = this.executeJob(job, lockToken, abort, lockTimer)
@@ -944,6 +971,7 @@ export class MinionWorker extends EventEmitter {
data: job.data,
attempts_made: job.attempts_made,
signal: abort.signal,
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
shutdownSignal: this.shutdownAbort.signal,
updateProgress: async (progress: unknown) => {
await this.queue.updateProgress(job.id, lockToken, progress);
+47 -9
View File
@@ -30,6 +30,15 @@ import { parseLegacyTokenScope } from './legacy-token-scope.ts';
import type { SqlQuery, SqlValue } from './sql-query.ts';
export type { SqlQuery, SqlValue };
export interface AgentClientBindings {
boundTools?: string[];
boundSourceId?: string;
boundBrainId?: string;
boundSlugPrefixes?: string[];
boundMaxConcurrent?: number;
budgetUsdPerDay?: string;
}
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
@@ -885,6 +894,7 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
sourceId: string = 'default',
federatedRead?: string[],
tokenEndpointAuthMethod?: string,
agentBindings?: AgentClientBindings,
): Promise<{ clientId: string; clientSecret?: string }> {
// v0.28: ALLOWED_SCOPES allowlist. Reject `--scopes "read flying-unicorn"`
// at registration so meaningless scope strings can't pile up in the DB.
@@ -917,16 +927,44 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
// has read scope == write scope, the v0.33 default)
const federated = federatedRead && federatedRead.length > 0 ? federatedRead : [sourceId];
try {
await this.sql`
INSERT INTO oauth_clients (client_id, client_secret_hash, client_name, redirect_uris,
grant_types, scope, token_endpoint_auth_method,
client_id_issued_at,
source_id, federated_read)
VALUES (${clientId}, ${secretHash}, ${name},
${pgArray(redirectUris)}, ${pgArray(grantTypes)}, ${scopes}, ${authMethod}, ${now},
${sourceId}, ${pgArray(federated)})
`;
if (agentBindings) {
await this.sql`
INSERT INTO oauth_clients (client_id, client_secret_hash, client_name, redirect_uris,
grant_types, scope, token_endpoint_auth_method,
client_id_issued_at,
source_id, federated_read,
bound_tools, bound_source_id, bound_brain_id,
bound_slug_prefixes, bound_max_concurrent, budget_usd_per_day)
VALUES (${clientId}, ${secretHash}, ${name},
${pgArray(redirectUris)}, ${pgArray(grantTypes)}, ${scopes}, ${authMethod}, ${now},
${sourceId}, ${pgArray(federated)},
${agentBindings.boundTools ? pgArray(agentBindings.boundTools) : null},
${agentBindings.boundSourceId ?? null}, ${agentBindings.boundBrainId ?? null},
${agentBindings.boundSlugPrefixes ? pgArray(agentBindings.boundSlugPrefixes) : null},
${agentBindings.boundMaxConcurrent ?? 1}, ${agentBindings.budgetUsdPerDay ?? null})
`;
} else {
await this.sql`
INSERT INTO oauth_clients (client_id, client_secret_hash, client_name, redirect_uris,
grant_types, scope, token_endpoint_auth_method,
client_id_issued_at,
source_id, federated_read)
VALUES (${clientId}, ${secretHash}, ${name},
${pgArray(redirectUris)}, ${pgArray(grantTypes)}, ${scopes}, ${authMethod}, ${now},
${sourceId}, ${pgArray(federated)})
`;
}
} catch (err) {
if (agentBindings && (
isUndefinedColumnError(err, 'bound_tools') ||
isUndefinedColumnError(err, 'bound_source_id') ||
isUndefinedColumnError(err, 'bound_brain_id') ||
isUndefinedColumnError(err, 'bound_slug_prefixes') ||
isUndefinedColumnError(err, 'bound_max_concurrent') ||
isUndefinedColumnError(err, 'budget_usd_per_day')
)) {
throw new Error('register-client --bound-* flags require an up-to-date OAuth schema; run `gbrain apply-migrations --yes` and retry.');
}
// Pre-v60 / pre-v61 brain: column missing. Fall back through both
// projections so registration still works until apply-migrations.
if (isUndefinedColumnError(err, 'federated_read')) {
+41 -31
View File
@@ -193,6 +193,39 @@ export function matchesSlugAllowList(slug: string, prefixes: readonly string[]):
return false;
}
/**
* Subagent slug-fence enforcement, shared by every mutating op a subagent
* can reach (put_page, add_timeline_entry). FAIL-CLOSED: `viaSubagent=true`
* enforces the check even if the dispatcher forgot to populate `subagentId`.
*
* - Trusted-workspace path (ctx.allowedSlugPrefixes set by cycle.ts under
* PROTECTED_JOB_NAMES \u2014 MCP cannot reach it): slug must match the
* allow-list globs.
* - Legacy default: slug must live under `wiki/agents/<subagentId>/...`
* (anchored, slash-boundary \u2014 `wiki/agents/12evil/*` can't impersonate
* subagent 12).
*/
function enforceSubagentSlugFence(ctx: OperationContext, slug: string, opName: string): void {
if (ctx.viaSubagent !== true) return;
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
throw new OperationError('permission_denied', `${opName} via subagent requires ctx.subagentId`);
}
const allowList = ctx.allowedSlugPrefixes;
if (allowList && allowList.length > 0) {
if (!matchesSlugAllowList(slug, allowList)) {
throw new OperationError(
'permission_denied',
`${opName} slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
);
}
} else {
const prefix = `wiki/agents/${ctx.subagentId}/`;
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
throw new OperationError('permission_denied', `${opName} via subagent must write under '${prefix}...'`);
}
}
}
/**
* Allowlist validator for uploaded file basenames. Rejects control chars, backslashes,
* RTL overrides (\u202E), leading dot (hidden files) and leading dash (CLI flag confusion).
@@ -784,37 +817,9 @@ const put_page: Operation = {
}
// Subagent namespace enforcement (v0.15+). Runs BEFORE the dry-run
// short-circuit so preview calls surface the same rejection. Confines
// LLM-driven writes to wiki/agents/<subagentId>/... — no leading slash
// (slug grammar rejects that), anchored, slash-boundary to defeat prefix
// collisions like `wiki/agents/12evil/*` impersonating subagent 12.
//
// FAIL-CLOSED: `viaSubagent=true` enforces the check even if the
// dispatcher forgot to populate `subagentId`. Agent-originated writes
// without an owning subagent id are rejected outright.
if (ctx.viaSubagent === true) {
if (typeof ctx.subagentId !== 'number' || Number.isNaN(ctx.subagentId)) {
throw new OperationError('permission_denied', 'put_page via subagent requires ctx.subagentId');
}
const allowList = ctx.allowedSlugPrefixes;
if (allowList && allowList.length > 0) {
// Trusted-workspace path: explicit allow-list bounds writes.
// Set only by cycle.ts (synthesize/patterns) which submits subagent
// jobs under PROTECTED_JOB_NAMES — MCP cannot reach this branch.
if (!matchesSlugAllowList(slug, allowList)) {
throw new OperationError(
'permission_denied',
`put_page slug '${slug}' is not within the trusted-workspace allow-list (${allowList.join(', ')})`
);
}
} else {
// Legacy default: agent-namespace confinement.
const prefix = `wiki/agents/${ctx.subagentId}/`;
if (!slug.startsWith(prefix) || slug.length === prefix.length) {
throw new OperationError('permission_denied', `put_page via subagent must write under '${prefix}...'`);
}
}
}
// short-circuit so preview calls surface the same rejection. See
// enforceSubagentSlugFence for the fail-closed policy.
enforceSubagentSlugFence(ctx, slug, 'put_page');
if (ctx.dryRun) return { dry_run: true, action: 'put_page', slug: p.slug };
// Skip embedding when the AI gateway has no embedding provider configured.
@@ -2149,6 +2154,11 @@ const add_timeline_entry: Operation = {
mutating: true,
scope: 'write',
handler: async (ctx, p) => {
// #2778: same fail-closed slug fence as put_page. add_timeline_entry is
// subagent-allowlisted (brain-allowlist.ts), so timeline writes must be
// confined to the same namespace/allow-list as page writes. Runs before
// the dry-run short-circuit so preview calls surface the same rejection.
enforceSubagentSlugFence(ctx, p.slug as string, 'add_timeline_entry');
if (ctx.dryRun) return { dry_run: true, action: 'add_timeline_entry', slug: p.slug };
const date = p.date as string;
// Reject anything that isn't a strict YYYY-MM-DD with year 1900-2199 and
+116
View File
@@ -0,0 +1,116 @@
/**
* Shared orphan-reporting exclusion policy.
*
* These are pages where "no inbound links" is expected and should not count
* against health. Keep this in core so the CLI orphan report and engine health
* dashboard cannot drift.
*
* Defaults are GBrain-wide conventions only. Brain-specific exclusions
* (private folder names, one-off fixture slugs) belong in the brain's own
* config, not here:
*
* gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
* gbrain config set orphans.exclude_slugs "some-one-off-page"
*/
const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
const RAW_SEGMENT = '/raw/';
const DENY_PREFIXES = [
'output/',
'dashboards/',
'scripts/',
'templates/',
'_templates/',
'openclaw/config/',
'extracts/',
];
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
'dreaming',
'daily',
]);
const ROOT_DATE_SLUG = /^\d{4}-\d{2}-\d{2}(?:-.+)?$/;
function isAgentWorkspaceConvention(slug: string): boolean {
if (!slug.startsWith('agents/')) return false;
if (slug.includes('/memory/dreaming/')) return true;
return /^agents\/[^/]+\/(?:agents|identity|soul|tools|user|heartbeat|dreams|dormant)$/.test(slug);
}
/** Per-brain additions to the convention defaults (from config). */
export interface OrphanPolicyOverrides {
excludePrefixes?: string[];
excludeSlugs?: string[];
}
/** Config keys for per-brain orphan exclusions (comma-separated values). */
export const ORPHAN_EXCLUDE_PREFIXES_KEY = 'orphans.exclude_prefixes';
export const ORPHAN_EXCLUDE_SLUGS_KEY = 'orphans.exclude_slugs';
function parseList(value: string | null): string[] {
if (!value) return [];
return value.split(',').map(s => s.trim()).filter(Boolean);
}
/**
* Load per-brain orphan exclusions from the brain config table. Callers with
* an engine in hand (getHealth, `gbrain orphans`) pass the result as the
* second argument to shouldExcludeFromOrphanReporting.
*/
export async function loadOrphanPolicyOverrides(
engine: { getConfig(key: string): Promise<string | null> },
): Promise<OrphanPolicyOverrides> {
const [prefixes, slugs] = await Promise.all([
engine.getConfig(ORPHAN_EXCLUDE_PREFIXES_KEY),
engine.getConfig(ORPHAN_EXCLUDE_SLUGS_KEY),
]);
return { excludePrefixes: parseList(prefixes), excludeSlugs: parseList(slugs) };
}
export function shouldExcludeFromOrphanReporting(
slug: string,
overrides?: OrphanPolicyOverrides,
): boolean {
if (PSEUDO_SLUGS.has(slug)) return true;
for (const suffix of AUTO_SUFFIX_PATTERNS) {
if (slug.endsWith(suffix)) return true;
}
if (slug.includes(RAW_SEGMENT)) return true;
if (slug.includes('/daily/')) return true;
for (const prefix of DENY_PREFIXES) {
if (slug.startsWith(prefix)) return true;
}
const firstSegment = slug.split('/')[0];
if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
if (ROOT_DATE_SLUG.test(slug)) return true;
if (slug.startsWith('_brain-')) return true;
if (isAgentWorkspaceConvention(slug)) return true;
if (overrides) {
if (overrides.excludeSlugs?.includes(slug)) return true;
for (const prefix of overrides.excludePrefixes ?? []) {
if (slug.startsWith(prefix)) return true;
}
}
return false;
}
+36 -1
View File
@@ -44,6 +44,33 @@ const HAIKU_MAX_TOKENS = 200;
/** Default model when caller doesn't override. Resolves through the gateway. */
const DEFAULT_SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';
/**
* Hard cap on `documentText` length (chars) before send.
*
* 2026-05-25 fix wave: small local chat models (Gemma 4 E2B, Qwen3 4B) get
* dramatically slower on long contexts even with 131K-token windows declared.
* A 73K-char page synopsis on Gemma 4 E2B takes 60-120s, exceeding the
* worker's default 30s `lockDuration` and tripping `lock-lost` errors.
*
* Truncate to a budget that fits a small model's effective throughput while
* preserving enough document context for the synopsis to be useful. Truncates
* the TAIL because the head (title, frontmatter, intro) carries the
* document-level anchor the synopsis needs.
*
* Override per workload via `GBRAIN_SYNOPSIS_DOC_MAX_CHARS`. Default 32768
* (~8K tokens at 4 chars/tok) keeps small-model synopsis under ~30s.
* Anthropic Haiku is unaffected at this cap; bump higher when running
* frontier models if you want richer document anchoring.
*/
export const SYNOPSIS_DOC_MAX_CHARS = (() => {
const env = process.env.GBRAIN_SYNOPSIS_DOC_MAX_CHARS;
if (env && /^\d+$/.test(env)) {
const n = parseInt(env, 10);
if (n >= 512 && n <= 1_048_576) return n;
}
return 32768;
})();
/**
* Synopsis prompt version. Folded into corpus_generation so prompt edits
* invalidate prior embeddings via the v0.40.3.0 query_cache.page_generations
@@ -188,11 +215,19 @@ function buildUserPrompt(
documentText: string,
chunkText: string,
): string {
// Tail-truncate `documentText` to `SYNOPSIS_DOC_MAX_CHARS` so small local
// chat models don't stall on >100KB pages. Head preserved (title block,
// frontmatter, intro paragraphs carry the document-level anchor).
let trimmedDoc = documentText;
if (documentText.length > SYNOPSIS_DOC_MAX_CHARS) {
trimmedDoc = documentText.slice(0, SYNOPSIS_DOC_MAX_CHARS) +
`\n\n[... ${documentText.length - SYNOPSIS_DOC_MAX_CHARS} chars truncated for synopsis budget ...]`;
}
return [
`<page_title>${pageTitle}</page_title>`,
'',
'<full_document>',
documentText,
trimmedDoc,
'</full_document>',
'',
'<chunk>',
+187 -30
View File
@@ -23,7 +23,9 @@ import { runMigrations } from './migrate.ts';
import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
import { getFtsLanguage } from './fts-language.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
@@ -54,7 +56,9 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { shouldExcludeFromOrphanReporting, loadOrphanPolicyOverrides } from './orphan-policy.ts';
import { LINK_EXTRACTOR_VERSION_TS } from './link-extraction.ts';
import {
normalizeEngineColumn,
buildVectorCastFragment,
@@ -1034,7 +1038,7 @@ export class PGLiteEngine implements BrainEngine {
const ingestedAt = (sourceKind || sourceUri || ingestedVia) ? new Date().toISOString() : null;
const { rows } = await this.db.query(
`INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path, source_kind, source_uri, ingested_via, ingested_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, 1), $14, $15, $16, $17, $18::timestamptz)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, ${MARKDOWN_CHUNKER_VERSION}), $14, $15, $16, $17, $18::timestamptz)
ON CONFLICT (source_id, slug) DO UPDATE SET
type = EXCLUDED.type,
page_kind = EXCLUDED.page_kind,
@@ -1625,20 +1629,24 @@ export class PGLiteEngine implements BrainEngine {
extraFilter += ` AND p.source_id = $${params.length}`;
}
const { rows } = await this.db.query(
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const keywordSql =
`WITH ranked AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
-- v0.27.1: hide image rows from default text-keyword search so
-- OCR text doesn't drown text-page hits. Image-similarity queries
-- run a separate vector path on embedding_image.
@@ -1649,10 +1657,140 @@ export class PGLiteEngine implements BrainEngine {
${buildBestPerPagePoolCte('ranked')}
SELECT * FROM best_per_page
ORDER BY score DESC, page_id ASC, chunk_id ASC
LIMIT $3 OFFSET $4`,
params
);
LIMIT $3 OFFSET $4`;
let { rows } = await this.db.query(keywordSql, params);
// D2 fix (fix/title-retrieval-arm): websearch AND semantics at chunk
// grain mean one non-co-occurring token zeroes keyword recall. When the
// strict query returns nothing, retry ONCE with OR-of-terms. Strict-AND
// results always win when non-empty (no change for working queries).
// Opt-in via SearchOpts.orFallback (Reviewer F1): only hybridSearch's
// recall arm relaxes; precision consumers (countMentions,
// link-extraction, eval) keep the strict-AND contract.
if (rows.length === 0 && opts?.orFallback) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) {
const fallbackParams = [...params];
fallbackParams[0] = orQuery;
({ rows } = await this.db.query(keywordSql, fallbackParams));
}
}
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
}
/**
* fix/title-retrieval-arm (D1): page-grain title candidate arm. See the
* BrainEngine interface doc for the full contract. Queries
* pages.search_vector (title weight 'A' dominates ts_rank_cd by
* construction) with the same page-grain filters the keyword arm applies
* (type/types/excludeSlugs/date/source scoping, hard-excludes,
* visibility), joined to one representative chunk per page. Applies the
* same ANDOR recall fallback as searchKeyword. NO query-length gate
* long exact-title queries are the case this arm exists for.
*
* CJK queries fall through to websearch FTS here (a single-token CJK
* query CAN exact-match a single-token CJK title); the richer CJK ILIKE
* fallback stays keyword-arm-only.
*/
async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
// language/symbolKind are chunk-grain code filters with no page-grain
// meaning; a code-scoped query gets no title candidates rather than
// rows that silently violate the caller's filter.
if (opts?.language || opts?.symbolKind) return [];
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const detailLow = opts?.detail === 'low';
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
const boostMap = resolveBoostMap();
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const visibilityClause = buildVisibilityClause('p', 's');
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const params: unknown[] = [query, limit, offset];
let extraFilter = '';
if (opts?.type) {
params.push(opts.type);
extraFilter += ` AND p.type = $${params.length}`;
}
if (opts?.types && opts.types.length > 0) {
params.push(opts.types);
extraFilter += ` AND p.type = ANY($${params.length}::text[])`;
}
if (opts?.exclude_slugs?.length) {
params.push(opts.exclude_slugs);
extraFilter += ` AND p.slug != ALL($${params.length}::text[])`;
}
if (opts?.afterDate) {
params.push(opts.afterDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
if (opts?.beforeDate) {
params.push(opts.beforeDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
if (opts?.sourceIds && opts.sourceIds.length > 0) {
params.push(opts.sourceIds);
extraFilter += ` AND p.source_id = ANY($${params.length}::text[])`;
} else if (opts?.sourceId) {
params.push(opts.sourceId);
extraFilter += ` AND p.source_id = $${params.length}`;
}
// Page grain — one row per page by construction, so no best_per_page
// pooling CTE is needed. The LEFT JOIN LATERAL picks the representative
// chunk (compiled_truth first, then lowest chunk_index); COALESCEs keep
// chunkless pages retrievable (the extreme D1 case: a title with no
// body) with the alias-hop row shape (chunk_id 0, empty chunk_text).
// Accepted limitations (Reviewer F5/F6): the synthetic chunkless row
// inherits the compiled-truth RRF boost and dedups on empty chunk_text;
// and detail='low' filters only the REPRESENTATIVE — pages without a
// compiled_truth chunk still surface (unlike the keyword arm's filter).
const titlesSql =
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
COALESCE(rep.id, 0) as chunk_id,
COALESCE(rep.chunk_index, 0) as chunk_index,
COALESCE(rep.chunk_text, '') as chunk_text,
COALESCE(rep.chunk_source, 'compiled_truth') as chunk_source,
ts_rank_cd(p.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM pages p
JOIN sources s ON s.id = p.source_id
LEFT JOIN LATERAL (
SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_source
FROM content_chunks cc
WHERE cc.page_id = p.id
AND cc.modality = 'text'
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
ORDER BY (cc.chunk_source = 'compiled_truth') DESC, cc.chunk_index ASC
LIMIT 1
) rep ON true
WHERE p.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC, p.id ASC
LIMIT $2 OFFSET $3`;
let { rows } = await this.db.query(titlesSql, params);
if (rows.length === 0) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) {
const fallbackParams = [...params];
fallbackParams[0] = orQuery;
({ rows } = await this.db.query(titlesSql, fallbackParams));
}
}
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
}
@@ -1857,20 +1995,23 @@ export class PGLiteEngine implements BrainEngine {
}
// visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse).
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const { rows } = await this.db.query(
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC
LIMIT $2 OFFSET $3`,
params
@@ -2183,6 +2324,10 @@ export class PGLiteEngine implements BrainEngine {
// v0.40.3.0 D24 NULL→non-NULL race fix mirrors postgres-engine.ts. Two writers
// racing on the same chunk previously raced last-write-wins; the fix lets the
// fresher `embedded_at` win in the text-unchanged branch.
//
// Code-chunk metadata columns follow the same chunk_text-gated CASE pattern as `embedding`
// (#769). Re-chunk trusts EXCLUDED outright; pure re-embed COALESCEs so a caller carrying
// only embedding-shaped fields doesn't clobber metadata to NULL.
await this.db.query(
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
@@ -2206,14 +2351,14 @@ export class PGLiteEngine implements BrainEngine {
THEN EXCLUDED.embedded_at
ELSE content_chunks.embedded_at
END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified,
language = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.language ELSE COALESCE(EXCLUDED.language, content_chunks.language) END,
symbol_name = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name ELSE COALESCE(EXCLUDED.symbol_name, content_chunks.symbol_name) END,
symbol_type = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_type ELSE COALESCE(EXCLUDED.symbol_type, content_chunks.symbol_type) END,
start_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.start_line ELSE COALESCE(EXCLUDED.start_line, content_chunks.start_line) END,
end_line = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.end_line ELSE COALESCE(EXCLUDED.end_line, content_chunks.end_line) END,
parent_symbol_path = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.parent_symbol_path ELSE COALESCE(EXCLUDED.parent_symbol_path, content_chunks.parent_symbol_path) END,
doc_comment = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.doc_comment ELSE COALESCE(EXCLUDED.doc_comment, content_chunks.doc_comment) END,
symbol_name_qualified = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.symbol_name_qualified ELSE COALESCE(EXCLUDED.symbol_name_qualified, content_chunks.symbol_name_qualified) END,
modality = EXCLUDED.modality,
embedding_image = COALESCE(EXCLUDED.embedding_image, content_chunks.embedding_image)`,
params
@@ -3536,6 +3681,13 @@ export class PGLiteEngine implements BrainEngine {
THEN ep.frontmatter->'event'->'who' ELSE '[]'::jsonb END
) AS w(name) WHERE w.name = $1 OR w.name LIKE $2)))`,
];
// "Last seen" is a PAST relation: chronicle stores future events
// (calendar-event is eligible), which must not read as "last seen".
// Bound to <= asof/today, mirroring getOnThisDay's `te.date < target`.
let seenThrough: string;
if (opts?.asof) { params.push(opts.asof); seenThrough = `$${params.length}::date`; }
else { seenThrough = `current_date`; }
where.push(`te.date <= ${seenThrough}`);
this.pushChronicleSource(where, params, opts);
const result = await this.db.query(
`SELECT te.date::text AS last_date, ep.slug AS last_event_slug
@@ -5029,7 +5181,7 @@ export class PGLiteEngine implements BrainEngine {
`);
const { rows: types } = await this.db.query(
`SELECT type, count(*)::int as count FROM pages GROUP BY type ORDER BY count DESC`
`SELECT type, count(*)::int as count FROM pages WHERE deleted_at IS NULL GROUP BY type ORDER BY count DESC`
);
const pages_by_type: Record<string, number> = {};
for (const t of types as { type: string; count: number }[]) {
@@ -5061,15 +5213,10 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
0 as stale_pages,
-- Bug 11 orphan = islanded (no inbound AND no outbound). The raw
-- list is filtered in TS using the shared orphan-reporting policy.
0 as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
@@ -5094,10 +5241,20 @@ export class PGLiteEngine implements BrainEngine {
LIMIT 5
`);
const { rows: islandedRows } = await this.db.query(`
SELECT p.slug
FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
`);
const r = h as Record<string, unknown>;
const pageCount = Number(r.page_count);
const embedCoverage = Number(r.embed_coverage);
const orphanPages = Number(r.orphan_pages);
const stalePages = await this.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS });
const orphanOverrides = await loadOrphanPolicyOverrides(this);
const orphanPages = (islandedRows as { slug: string }[])
.filter(row => !shouldExcludeFromOrphanReporting(row.slug, orphanOverrides)).length;
const deadLinks = Number(r.dead_links);
const linkCount = Number(r.link_count);
const pagesWithTimeline = Number(r.pages_with_timeline);
@@ -5125,7 +5282,7 @@ export class PGLiteEngine implements BrainEngine {
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: Number(r.stale_pages),
stale_pages: stalePages,
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,

Some files were not shown because too many files have changed in this diff Show More