Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 6548e5cffc fix(autopilot): add --target to value-flag set so install targets survive positional translation
Review finding on #3103: --target is installDaemon's value flag
(macos | linux-systemd | ephemeral-container | linux-cron). The
translator only knew --repo/--interval, so `gbrain autopilot
--install --target linux-cron` misread the target value as an
unknown positional subcommand and exited 2 before installDaemon ran.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 12:05:46 -07:00
0ac3d4da9c fix(autopilot): translate positional subcommands so autopilot status doesn't start the daemon
Takeover/rebase of #1529 onto current master. `gbrain autopilot status`
(and install/uninstall/start) previously fell through the flag-only
branches in runAutopilot and silently started the daemon (lockfile +
worker spawn + sync dispatch). A pure translatePositionalSubcommands()
now maps known positionals to their flag form before any side effect,
is value-flag aware (--repo/--interval), and fails loud (exit 2) on
unknown positionals. 22 tests including the exact #1525 repro.

Fixes #1525

Co-authored-by: Oszkar <Oszkar@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:28:16 -07:00
Masa 0612b0daa8 fix(dream): require self-contained opening summary in synthesized pages (#2770)
Synth pages written by dream synthesize currently open straight into
detail (quotes, cross-references) with no framing, so a reader who
lands on the page later — without the source transcript in front of
them — has no way to tell what it's about without reading the whole
thing.

Add OUTPUT POLICY item 5: every new page's body must open with a 2-3
sentence self-contained summary a reader unfamiliar with the source
conversation could understand on its own, before any quotes or detail.
2026-07-21 13:40:41 -07:00
maxpetrusenkoagentandmaxpetrusenkoagent <[REDACTED EMAIL]> f529eaa231 fix(jobs): refresh gateway config for queued AI work (#2125)
Long-lived minion workers can outlive DB-backed model config changes. Refresh the AI gateway before gateway-backed handlers run so queued cycle/propose_takes work does not fall back to a stale Anthropic default when the operator configured another provider.

Also record the active gateway chat model in propose_takes budget/proposal metadata instead of hardcoding claude-sonnet-4-6, and keep provider:model IDs intact for budget pricing.

Regression coverage verifies queued worker refresh, propose_takes model metadata, nested provider IDs, skipFence threading, and the updated autopilot signal source guard.

Co-authored-by: maxpetrusenkoagent <[REDACTED EMAIL]>
2026-07-21 13:21:51 -07:00
447e57ec41 fix(ai): tier-configured models reach the recipe allowlist — refresh Anthropic models, register tier resolutions, honest probe labels (#2800)
Setting `models.tier.deep anthropic:claude-opus-4-8` silently disabled
think and auto_think: the Anthropic recipe's chat allowlist stopped at
Opus 4.7, the tier-resolved model never joined the extended set that
assertTouchpoint's contract promises for config-chosen models, and the
resulting probe failure was stamped NO_ANTHROPIC_API_KEY — sending the
operator to debug env/keychain when the fix was the model id. Three
fixes, one per layer:

- recipes/anthropic.ts: add claude-fable-5, claude-opus-4-8, and
  claude-sonnet-5 to chat models; claude-sonnet-5 to expansion models.
- gateway.ts reconfigureGatewayWithEngine: resolve all four tiers and
  register the results as extended models, honoring the documented
  contract for models.default / models.tier.* (model-resolver.ts
  docstring). A tier-only model now validates like a chat/expansion one.
- think/index.ts: when the gateway client can't be built, re-probe and
  label honestly — MODEL_NOT_USABLE:<reason> for unknown_model /
  unknown_provider, NO_ANTHROPIC_API_KEY only for the actual missing-key
  case; the stub answer carries the probe detail and fix hint.

Tests: recipe-list presence pins; a new gateway-tier-extended-models
suite proving a fictional tier model validates post-reconfigure (and an
unconfigured one still doesn't); think-pipeline coverage for the honest
label (unknown_model beats missing-key even keyless); the existing
non-explicit bogus-provider test updated from the old catch-all label to
the honest one (no-throw contract unchanged).

Verified live: a brain with tier.deep=claude-opus-4-8 had think degrade
to gather-only with the misleading key warning; with this change the
probe passes and synthesis runs.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 13:06:32 -07:00
d61808d806 v0.42.64.0 fix: harden confidential OAuth token revocation (takeover of #3017) (#3032)
* fix(oauth): validate confidential revoke secrets

* fix(oauth): harden confidential token revocation

* chore: bump version and changelog (v0.42.64.0)

Co-Authored-By: OpenAI Codex <noreply@openai.com>

---------

Co-authored-by: Robin <rayme@boltdsolutions.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:44:46 -07:00
60125ee626 feat(dream): --once for one-shot phase runs without toggling config gates (takeover of #2983) (#3031)
* feat(dream): --once for one-shot phase runs without toggling config gates

Fixes the "toggle enabled true, run, toggle back to false" workaround
that gbrain doctor's extract_atoms_backlog message implicitly
recommends and that #2860's reporter had to script around: with an
external orchestrator running `gbrain dream --phase patterns` on a
cadence outside the autopilot, the only way to run patterns once was
`config set dream.patterns.enabled true` -> run -> `config set ...
false`. A crash between steps left the flag stuck true, and the
autopilot (which polls the same flag) re-enqueued patterns every
cycle -- 119 LLM jobs / ~$400 over 24h before it was caught.

Root cause: `--phase X` only controls which phase FUNCTION cycle.ts
calls; it does not bypass that phase's own `dream.<phase>.enabled` /
`cycle.<phase>.enabled` config read. Each gated phase (patterns,
synthesize, conversation_facts_backfill, enrich_thin, skillopt) reads
its enabled flag internally and skips regardless of how the phase was
selected -- confirmed by reading each phase module, not assumed.
extract_atoms/synthesize_concepts are a DIFFERENT mechanism entirely
(pack-declaration via packDeclaresPhase, not a config .enabled read)
and already have a working one-shot escape hatch: `--drain`. The
existing doctor message for extract_atoms already says `--phase
extract_atoms --drain --window 120`, so no doctor text needed
updating there -- verified by reading src/commands/doctor.ts directly
rather than assuming the paraphrase in the issue was literal.

Design: `gbrain dream --phase <name> --once`. Requires an explicit
--phase (bare --once is a usage error, exit 2) so it can never
force-enable every disabled phase at once in a full/default cycle --
that would recreate the same unbounded-spend risk the flag exists to
prevent. Threaded through CycleOpts as `onceForPhase?: CyclePhase`
(the literal phase name, not a boolean) so the bypass can never leak
to a phase other than the one named, even if a future programmatic
caller passes a wider `phases` array than the CLI does. Never reads
or writes config -- the phase still evaluates its .enabled gate every
call; --once only overrides the boolean OUTCOME for that one
invocation, mirroring the existing --unsafe-bypass-dream-guard /
--input precedents (stderr warning at the bypass point, no new
config-touching code path).

Rejected alternatives (documented per task instructions):
- Making explicit --phase X always bypass .enabled: breaking change
  for existing crons that rely on the disabled flag as a cheap no-op;
  an upgrade would silently start running LLM/write phases.
- A new subcommand: adds a whole dispatch/help/arg surface that
  internally routes through the same override anyway.
- Extending --once to also bypass packDeclaresPhase for
  extract_atoms/synthesize_concepts: conflates two different gating
  mechanisms (config toggle vs. pack membership) under one flag;
  extract_atoms already has --drain, which is purpose-built for its
  batched/windowed execution model.

Design was cross-validated by an independent second-model review
(external design consultation) before implementation; its
recommendation to also update the extract_atoms doctor message to
`--once` was NOT adopted because that phase has no .enabled gate to
bypass -- doing so would be a documented no-op, contradicted by
reading src/commands/doctor.ts:3264 directly.

Tests: 9 new (structural CLI-flag wiring in dream-cli-flags.test.ts;
a real PGLite E2E test in dream-patterns-pglite.test.ts proving the
bypass fires AND that dream.patterns.enabled is never written; 4
runCycle-level tests in cycle.serial.test.ts proving onceForPhase
does not leak across phases). Verified 6 of 9 fail against the
pre-fix source (via git stash of source-only changes) to confirm
they're meaningful regressions, not tautologies.

Closes #2860

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dream): --help short-circuits before --once usage validation

Codex review finding (P2): `gbrain dream --help --once` (no --phase)
called process.exit(2) from the new --once usage-error check inside
parseArgs before runDream's documented IRON RULE ("--help
short-circuits BEFORE any engine-bearing work") ever got a chance to
run -- parseArgs computes ALL its validations unconditionally before
runDream checks opts.help. Repo precedent for this ordering already
exists as a pinned regression test (test/dream.test.ts's "--help
--source whatever prints help and exits 0").

Fix: compute wantsHelp once in parseArgs and exempt the --once
validation when it's set, mirroring that precedent. Added the same
class of pinned tests here: bare `--once` still exits 2 with the
usage hint, `--help --once` prints help and exits 0, and a real
--phase patterns --once run against a PGLite engine proves the
bypass actually fires (falls through to insufficient_evidence
instead of disabled) without writing dream.patterns.enabled. Also
fixed the structural test in dream-cli-flags.test.ts that asserted
the exact pre-fix guard-condition source text.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dream): --once must require an EXPLICIT --phase, not a derived one

Codex review finding (P3): the --once validation checked the derived
`phase` value, but `phase` gets defaulted implicitly by --input
(implies --phase synthesize) and --drain (implies --phase
extract_atoms) BEFORE that check ran. So `gbrain dream --input <f>
--once` and `gbrain dream --drain --once` both slipped past the
"explicit --phase required" contract silently -- and --once became a
true no-op in both cases: --drain returns from runDream before
onceForPhase is ever read (the drain path doesn't call runCycle at
all), and --input already bypasses the synthesize enabled-gate on its
own via the existing opts.inputFile check, so onceForPhase would
never even be consulted.

Fix: capture `phaseWasExplicit = phaseIdx !== -1` at the very top of
parseArgs, before the --input/--drain defaulting blocks run, and
validate --once against that instead of the derived `phase`. Updated
the usage-error message and --help text to say "an explicit --phase"
so a user hitting this understands why `--input ... --once` doesn't
count.

Tests: 2 new pins in test/dream.test.ts exercising runDream directly
(--input <file> --once exits 2; --drain --once exits 2), plus a
structural test in dream-cli-flags.test.ts pinning that
phaseWasExplicit is captured before both implicit-defaulting blocks.
Updated the two existing structural/behavioral tests whose literal
guard-condition / error-message assertions changed shape.

Verified: dream-cli-flags.test.ts 27/27, dream.test.ts 31/31,
typecheck clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:29:35 -07:00
e320ad71b3 fix(providers): reuse buildGatewayConfig for --model test override (takeover of #2980) (#3029)
* fix(providers): reuse buildGatewayConfig for --model test override (#2863)

`gbrain providers test --model <id>` overrode the gateway with only
embedding_model/chat_model + env, dropping config.provider_base_urls
entirely. A brain configured with a custom endpoint (e.g. a China-region
DashScope base URL) would pass the bare `providers test` (which goes
through configureFromEnv() and does forward base_urls) but fail the
`--model`-scoped probe with a misleading "Incorrect API key" error, even
though the key was valid for the configured endpoint — the probe silently
fell back to the recipe's hardcoded default endpoint instead.

Root cause: two independent, drifted resolvers. The production path
(src/cli.ts#connectEngine, src/core/init-embed-check.ts) builds its
AIGatewayConfig via buildGatewayConfig(), which folds provider_base_urls,
env-sourced local-server base URLs, provider_chat_options, and file-plane
API keys. The --model override branch in runTest() hand-rolled a second,
narrower config object that only carried the overridden model + raw env.

Fix: lift `cfg` out of the existing try/catch (it was already loaded there
for the isolation-warning message) and spread `buildGatewayConfig(cfg)`
into both configureGateway() calls before overriding embedding_model/
chat_model. The isolated --model probe now resolves its endpoint exactly
the way the brain's real import/query path would; only the requested
model is overridden, so the probe still targets exactly the model the
user asked for. Falls back to bare env when no brain is configured yet
(cfg is null), matching prior first-time-install behavior.

Confirmed chat_fallback_chain (also threaded through by buildGatewayConfig)
has no runtime retry effect — it's only consumed to pre-register extended
model ids — so spreading the full production config does not mask an
isolated model's own failures behind a silent fallback.

Other diagnostic surfaces (providers list/env/explain) were checked and
are unaffected: `runProviders()` already calls configureFromEnv() (which
forwards base_urls correctly) before dispatch, and none of them accept
--model, so they never hit the broken override branch.

Adds test/providers-test-model-base-url.test.ts: drives runProviders('test',
...) end-to-end against a mocked fetch + temp GBRAIN_HOME/config.json with
provider_base_urls set for the dashscope recipe (the exact recipe named in
the bug report), asserting the outbound request hits the configured base
URL rather than the recipe default. Verified red on pre-fix code via
git stash, green after.

Closes #2863

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix: drop duplicate buildGatewayConfig import after master merge

Master's f3e78fd2 added the same import the PR carried; the textual
merge was clean but the result failed typecheck (TS2300 duplicate
identifier).

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-21 12:17:18 -07:00
b6dd3e1121 fix(test): reset AI gateway after adaptive-embed-batch suite — cross-file config leak (#3065)
The file's final test configures the gateway with a remote provider and a
fake key, and its afterEach only clears the mock transport. With no
afterAll, the poisoned global config survives the file boundary; the next
test file in the shard that triggers an embed makes a real HTTP call and
fails. Surfaced on master when #3022's new test file reshuffled shard
composition (shard 6: synthesize-concepts-progress failed twice with a
live Google embed rejection). The legacy-embedding preload can't catch
this: it only re-applies defaults when the gateway slot is empty.

One-line root-cause fix at the leaker. A repo-wide guard for the class
(~70 files call configureGateway without a final reset) is filed as a
follow-up.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 12:05:31 -07:00
3cc34c92ee feat(ai): add NVIDIA NIM provider recipe (#2965) (#3022)
Adds NVIDIA NIM / API Catalog as a first-class OpenAI-compatible AI recipe:

- chat via nvidia/nemotron-3-super-120b-a12b (conservative capability
  claims: no tools, no subagent loop until proven)
- hosted embedding models incl. nvidia/llama-nemotron-embed-1b-v2 with
  Matryoshka-style dimension overrides (1024/1280/1536/2048) and fixed
  natural dims for the other catalog models
- asymmetric input_type mapping (document -> passage, query -> query) via a
  gateway compat fetch shim, since the generic openai-compatible recipe
  cannot infer that provider-specific requirement
- base URL https://integrate.api.nvidia.com/v1 verified live (OpenAI-shaped
  /v1/models, all five recipe model ids present in the catalog)

Changed from the original PR: dropped the recipe's custom resolveAuth — it
duplicated defaultResolveAuth's Authorization-Bearer behavior exactly and
violated the IRON RULE that only Azure overrides resolveAuth
(test/ai/recipes-existing-regression.test.ts). NVIDIA_API_KEY now flows
through defaultResolveAuth via auth_env.required, and the recipe test pins
resolveAuth === undefined + the default Bearer resolution + the missing-key
AIConfigError. Also scrubbed a private downstream-agent name from ported
comments per the repo privacy rule.

Takeover of #2965 by @ravehorn.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: SAGE Codex <codex@sage.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 00:17:26 -07:00
Anton Senkovskiy a93fcf504f fix(chronicle): bound last-seen to <= asof/today so future events do not read as "seen today" (#2993)
getLastSeen had no upper date bound, so a future-dated chronicle event (a
scheduled calendar-event, a planned milestone) became the entity's "last
seen" date; finalizeLastSeen's Math.max(0, ...) then clamped the negative
delta, reporting days_ago: 0 -- the entity reads as seen-today. Recording
future events is intended (eligibility ELIGIBLE_TYPES includes calendar-event);
the reader just needs to stop counting them as "seen".

Bound the query to te.date <= COALESCE(asof, current_date) in both engines,
mirroring getOnThisDay's existing te.date < target bound. asof now reaches
the WHERE clause (previously it only reached finalizeLastSeen), so as-of
time-travel is honored for the date filter too.

Regression test added: an entity with past events plus a future event -> last
seen returns the most recent PAST event, not the future one; and as-of after
the future date lets it through. Fails before, passes after.
2026-07-21 00:04:41 -07:00
Hanchen Qiu 84fad4738d fix(pages): default chunker_version to MARKDOWN_CHUNKER_VERSION on INSERT (#2807) (#2988)
putPage's INSERT used COALESCE(<chunkerVersion>, 1), so callers that don't
supply chunker_version (no MCP/subagent caller does — it's internal metadata)
landed new pages at version 1. Dream subagents write through putPage directly,
so their pages got v1 and doctor's contextual_retrieval_coverage check flagged
them as "older chunker_version" forever, even though they were chunked and
embedded with the current chunker.

Default the INSERT to MARKDOWN_CHUNKER_VERSION on both engines. The ON CONFLICT
UPDATE still COALESCE-preserves an explicitly supplied version. Add an
engine-level regression test.
2026-07-20 23:47:09 -07:00
Hanchen Qiu f815246eef fix(cli): register reconcile-links in CLI_ONLY so dispatch reaches its handler (#2900) (#2987)
reconcile-links is advertised in `gbrain --help` and implemented with a
`case 'reconcile-links'` block in handleCliOnly, but it was missing from the
CLI_ONLY Set. Dispatch only enters handleCliOnly when the command is in
CLI_ONLY, so every invocation fell through to the shared-operations lookup and
hit the generic "Unknown command" branch — leaving the documented doc↔impl
edge-rebuild tool silently unreachable via the CLI.

Add 'reconcile-links' to CLI_ONLY, plus a reachability regression test
mirroring the #2035 (`calibration`) guard.
2026-07-20 23:23:03 -07:00
Vishnu JandClaude Fable 5 42c4ea929f fix(test): isolate audit writes to a per-run scratch dir in the shared bootstrap (#2823) (#2966)
The content-sanity gate's audit logger (logContentSanityAssessment)
defaults, via audit-writer.ts::resolveAuditDir(), to writing
~/.gbrain/audit/content-sanity-YYYY-Www.jsonl on disk. A GBRAIN_AUDIT_DIR
env override exists, but nothing in the shared test bootstrap ever set
it, so any test that exercised an audit-emitting code path without
wrapping the call in its own withEnv() fell through to the operator's
real audit trail. test/import-file.test.ts's oversize-boundary fixture
('borderline-slug', content just under MAX_FILE_SIZE but over
DEFAULT_BYTES_BLOCK) fired a real soft_block event into the developer's
live ~/.gbrain/audit on every run — which doctor's
content_sanity_audit_recent check then reported as production signal.

Fix: add a bootstrap preload (test/helpers/audit-dir-preload.ts, wired
via bunfig.toml) that points GBRAIN_AUDIT_DIR at a fresh per-process
mkdtemp dir before any test file loads. Each run-unit-shard.sh shard is
its own bun process, so each shard gets its own scratch dir with no
cross-shard collision. This closes the leak for every audit-emitting
test, not just this fixture. It respects a developer-exported override
(only sets the var when unset), and files that manage their own
per-test GBRAIN_AUDIT_DIR via withEnv() are unaffected.

Also fix a latent isolation bug this surfaced: gbrain-home-isolation.test.ts
unconditionally deleted GBRAIN_AUDIT_DIR in a finally block instead of
restoring the prior value, which clobbered the bootstrap's scratch dir
for every test file that ran after it in the same shard process.

Adds test/audit/audit-dir-preload.test.ts to pin the behavior: it
reproduces the exact soft_block event shape and asserts it lands in the
scratch dir, never in ~/.gbrain/audit.

Reported by @paul-0320.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 23:07:58 -07:00
Benjamin D. Smith 6370ce3d7e fix(infrastructure): chmod 644 autopilot supervisor files on install (#2963)
📝 Summary:
• launchd rejects group/world-writable agent plists — when the installer
  runs under a umask-0 parent shell, `writeFileSync(plistPath(), plist)`
  produces a 0666 plist that makes `launchctl load`/`bootstrap` fail with
  the opaque `Bootstrap failed: 5: Input/output error` and the login-time
  LaunchAgents scan skip the file silently
• on an affected machine the daemon never registers while everything
  looks installed — the plist exists, launchd's disabled-table says
  enabled, and no log file is ever created

🔧 Technical Improvements:
• `installLaunchd`: write plist with `{ mode: 0o644 }` AND
  `chmodSync(0o644)` — writeFileSync mode applies only on create, so a
  reinstall over an existing 0666 plist must normalize explicitly
• `installSystemd`: same hardening on the unit file (systemd warns on
  world-writable units); symmetric with the launchd path
• Restart-policy rewrite path (`generateSystemdUnit` rewrite of an
  existing unit): chmod is load-bearing here — the file always exists,
  so the write mode never applies
• `chmodSync` added to the fs import

📊 Code Changes: 18 insertions, 4 deletions (net +14)

📦 Files Modified:
• src/commands/autopilot.ts (minor updates) — mode + chmod on the three
  supervisor-file writers; comments carry the launchd failure signature
  so the next EIO hunt greps straight to it
2026-07-20 22:48:16 -07:00
levineam c21d7b253a fix(doctor): derive host skill manifests (SUP-3488) (#2961) 2026-07-20 22:38:25 -07:00
zsimovanforgeopsandForge 4df7796061 Preserve modality and symbol metadata in embed-stale merge (#2969)
embedStaleForSource rebuilds each page's chunks as a merged ChunkInput[]
carrying only five fields (chunk_index, chunk_text, chunk_source,
embedding, token_count), while upsertChunks writes the metadata columns
as EXCLUDED.<col>. Any page containing at least one stale chunk
therefore has ALL its chunks' metadata reset on the next embed-stale
pass:

- image chunks flip modality 'image' -> 'text' and disappear from the
  cross-modal image search arm permanently (it filters
  modality='image'), while keeping their embedding_image vector — the
  data looks intact but is unreachable;
- code chunks lose language, symbol_name, symbol_type,
  symbol_name_qualified.

The read side compounds this: rowToChunk never returned modality, so a
correct merge was impossible without also extending the Chunk shape.

Fix: expose modality on Chunk/rowToChunk and carry modality, language,
and the symbol fields through the merge. embedding_image is deliberately
not carried — upsertChunks already COALESCEs it server-side.

Repair for affected brains: UPDATE content_chunks SET modality='image'
WHERE chunk_source='image_asset' AND embedding_image IS NOT NULL.

The new test seeds a mixed page (settled image chunk + stale text chunk
with symbol metadata) and asserts both survive an embedStaleForSource
pass; it fails on master.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 22:14:46 -07:00
zsimovanforgeopsandForge 6db4cea2e4 Resolve relative storage paths in doctor image_assets check (#2971)
The image_assets check statSyncs files.storage_path directly, but
sync-ingested assets store repo-relative paths. Run doctor from any
directory other than the brain repo and every image is reported
'missing from disk' — a persistent false WARN with a suggested fix
(gbrain sync --skip-failed) that does nothing.

Resolve relative paths against sync.repo_path before statting; absolute
paths are untouched. Falls back to cwd when the config key is unset,
preserving the old behavior for brains without a configured repo.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 21:59:24 -07:00
sameerbopardikarandSameer Bopardikar 11eebc3605 fix(conversation): extract iMessage facts with real timestamps (#2756) (#2958)
Co-authored-by: Sameer Bopardikar <203024074+sameerbopardikar@users.noreply.github.com>
2026-07-20 16:57:53 -07:00
Eddie AshandEddie Ash ee45653a02 fix: initialize foreground chat gateway (#2590) (#3003)
Co-authored-by: Eddie Ash <119880+cazador481@users.noreply.github.com>
2026-07-20 16:47:12 -07:00
zsimovanforgeopsandForge 706d3cea3d Scope maxWaiting backpressure by data.sourceId (#2970)
The submission-time backpressure cap counts all waiting (name, queue)
rows regardless of which source a job targets. On a multi-source brain
this makes per-source submissions with maxWaiting: 1 mutually exclusive:
while one source's sync sits waiting, every other source's freshness
sync coalesces into that row and never runs. The dispatch log shows the
starved source 'dispatched' each interval (queue.add returns the other
source's waiting row), so the starvation is invisible unless you notice
sources.last_sync_at falling behind — we found a secondary source 29
hours stale on a 5-minute freshness interval.

Fix: when the submitted data carries a string sourceId, key the advisory
lock, the waiting count, and the coalesce target on it. Submissions
without sourceId keep the existing single-scope behavior, so
single-source brains and non-sync jobs are unchanged.

The new test asserts same-source submissions still coalesce while a
different source gets its own row and its own cap; it fails on master.

Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
2026-07-20 16:38:03 -07:00
MasaandClaude Sonnet 5 2934c53c1d fix(sources): validate --path is a git repo with committed content at registration (#2707) (#2975)
* fix(sources): validate --path is a git repo at registration time (#2707)

`sources add --path <dir>` accepted any existing non-git directory with
zero validation, deferring the failure to the first `gbrain sync` ("Not
inside a git repository: ..."). By the time that surfaces, the source
has been silently stale for however long nobody read the sync logs.

Add a registration-time check (git-remote.ts:isInsideGitRepo, mirroring
sync.ts's discoverGitRoot walk-up so subdir-of-git-repo sources still
pass) that rejects an existing-but-non-git --path directory with an
actionable error pointing at `git init && git add -A && git commit`.
Non-existent paths are unaffected (out of scope — different, pre-existing
failure mode) and `--force` opts out for callers who want to register
before git-init exists.

This is registration-time validation ONLY — it never auto-`git init`s
the directory, preserving the consent boundary #2967 established for
sync-time self-heal (a --path source is the user's own external
directory; gbrain must not mutate it without explicit ask).

Also documents the git requirement (docs/guides/multi-source-brains.md),
including the "files must be committed, not just present" gotcha and
that a stale/unreachable sync anchor already self-heals on plain
`gbrain sync` (verified manually against HEAD — no reset-anchor command
needed).

* fix(sources): require a committed HEAD + shell-quote remediation cmd (codex round 1)

Codex review round 1 on #2707 found two real gaps:

1. isInsideGitRepo alone accepts a `git init`ed-but-never-committed
   directory (rev-parse --show-toplevel succeeds with no HEAD), so
   registration would still pass a source that fails sync's own
   "No commits in repo ... Make at least one commit before syncing."
   Add hasGitCommits (git rev-parse HEAD) as a second required check.

2. The remediation command in the error message interpolated the raw
   path unquoted — spaces, $(), backticks, etc. would break or, worse,
   execute unintended shell syntax if pasted. POSIX single-quote it
   (mirrors src/commands/connect.ts:shellQuote; duplicated locally
   rather than imported, since commands/ depends on core/ not the
   reverse).

* fix(sources): require tracked content in HEAD, not just a resolvable HEAD (codex round 2)

Codex review round 2 P1: hasGitCommits (rev-parse HEAD) accepted a repo
with an empty commit (git commit --allow-empty) followed by untracked
files — HEAD resolves fine (to git's well-known empty-tree object), so
registration passed, but the first sync would "succeed" importing
nothing and then silently never notice the untracked files change. The
exact same gap applied to an untracked subdirectory of an otherwise-
real git repo (monorepo case).

Replace hasGitCommits with hasTrackedContent (`git ls-tree HEAD -- .`,
non-recursive — one entry is enough, no need to walk the whole
subtree). `-C path` + pathspec `.` scopes correctly to both a repo
toplevel and a subdirectory-of-a-repo source, and an empty tree lists
zero entries where a bare `rev-parse HEAD` would still succeed. Also
subsumes the "no commits at all" case hasGitCommits covered (ls-tree
on an unborn repo fails the same way), so this is one check instead
of two.

Updated the error copy and docs/guides/multi-source-brains.md to match
what's actually verified now.

* fix(sources): O(1)-output tree-emptiness probe, avoid maxBuffer overflow (codex round 3)

Codex review round 3 found the round-2 `git ls-tree HEAD -- .` listing
buffers the whole (non-recursive) tree — a real repo with ~17-20K
directly-tracked entries exceeds execFileSync's default 1 MiB
maxBuffer, throws ENOBUFS, and the catch-all incorrectly rejects a
perfectly valid registration.

Replace the listing with `git rev-parse --verify HEAD:./` (resolves
the tree object for `path` specifically, correct for both toplevel and
subdirectory sources same as before) compared against git's canonical
empty-tree SHA-1 (4b825dc6...) — a fixed ~40-byte read regardless of
how many entries the tree has, structurally immune to this class of
bug rather than just raising the threshold. Added a 300-file
regression test locking this in.

Declined a second round-3 finding (P1: reject a tree if ANY untracked
file exists anywhere under the path, not just when the tree is
entirely empty) — untracked files never being synced is standard,
existing git-source behavior throughout this codebase (identical for
--url managed clones), not a bug specific to this validation. Enforcing
zero-untracked-files at registration would reject ordinary repos with
gitignored build output, .DS_Store, editor swapfiles, etc. Out of
scope relative to what #2707 actually asks for (a directory with real,
committed content that will sync) and how every other git source in
this system already behaves.

* fix(sources): derive empty-tree OID per repo instead of hardcoding SHA-1 (codex round 4)

Codex review round 4 P2, confirmed by directly testing against a
`git init --object-format=sha256` repo: the hardcoded SHA-1 empty-tree
constant only matches SHA-1 repositories. An empty SHA-256 repo's real
empty-tree OID is a different (64-char) hash, so the SHA-1 comparison
silently mismatched and let an empty/untracked SHA-256 source through
— exactly the case this validation exists to catch.

Replace the constant with `emptyTreeOid()`: `git hash-object -t tree
--stdin < /dev/null` computed in the target repo's own context, so it
returns the correct empty-tree OID for whichever object format that
repo actually uses, without gbrain needing to know or care which one.
Added gated regression tests (git 2.29+ / --object-format=sha256,
test.skipIf on older git) for both the empty-repo-rejected and
real-content-registers-fine cases.

Converging here (4 review rounds; this is the last outstanding
finding from round 4, and round 4 raised only this one issue).

* fix(test): --force the incidental non-git second source in #1434 routing test (#2707)

CI caught a real regression from this PR's registration-time git
validation: test/sync-sole-non-default-routing.test.ts's "2+ non-default
sources" case registers a bare mkdtempSync temp dir (no git init) as a
second source purely to have 2 sources present — the directory's content
was never meant to be exercised, only its existence as a distinct
local_path. #2707's new validation correctly rejects that dir at
registration time, since nothing else in the test suite told it
otherwise.

--force is the right fix, not adding unnecessary git-init/commit
boilerplate to secondRepo: it documents that this specific registration
intentionally doesn't care about git-validity, matching what a real
caller opting into the legacy lenient behavior would do.

Verified: the specific test (3/3 pass), plus every other test file in
the repo using `sources add --path` (sources.test.ts, sources-ops.test.ts
already covered by the PR's own commits; repos-alias.test.ts,
sync-cost-gate.serial.test.ts — 11/11 pass, no similar fixture gap).
typecheck clean, verify 31/31.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:28:53 -07:00
b60656245f fix(migrate): v124 notice to stderr — the stdout-cleanliness guard caught it on master (#3035)
Same class as #3019's v123 fix; the guard test added there flagged this
within one push. Route the notice through process.stderr.write.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 16:19:40 -07:00
MasaandClaude Sonnet 5 d698b44438 fix(search): drop compiled_truth from pages.search_vector — was overflowing tsvector on large pages (#2704) (#2977)
A single markdown page whose compiled_truth exceeds Postgres's hard
1,048,575-byte tsvector cap made update_page_search_vector() throw
"string is too long for tsvector" INSIDE the pages UPSERT transaction.
Not a per-file ledger entry — a transaction abort. The whole source's
sync checkpoint stayed pinned (Sync BLOCKED) until the oversized file was
fixed or manually skipped, even though every other file in the run
imported fine. --retry-failed re-failed the same files every run; the
3-consecutive-failure auto-skip eventually moved past them, but for a
scheduled collector that meant hours of blocked cycles per oversized
file, per source.

Root cause: pages.search_vector indexed compiled_truth — the unbounded
whole-page body — even though it's write-only dead weight for actual
search. searchKeyword() (postgres-engine.ts / pglite-engine.ts) ranks and
queries content_chunks.search_vector exclusively (Cathedral II Layer 3,
chunk-grain, already populated separately from compiled_truth via
chunking at import time, and well under the tsvector cap since
chunkText() targets embedding-sized pieces). Verified directly: no
pages.search_vector / bare search_vector read appears anywhere outside
this trigger's own definition and the reindex/backfill machinery that
maintains it.

Fix: v124 migration recreates update_page_search_vector() without
compiled_truth — title + timeline stay (both naturally small), so the
column keeps carrying some signal rather than going fully inert. Updated
in lockstep (documented contract, see reindex-search-vector.ts's own
comment): migrate.ts's new v124, reindex-search-vector.ts's
recreatePagesFn, and the fresh-install baselines in pglite-schema.ts +
src/schema.sql (regenerates schema-embedded.ts via `bun run
build:schema`). No backfill: existing rows keep whatever search_vector
they already computed until their next UPDATE — harmless, since nothing
reads this column, and the brains that actually hit this bug never
successfully wrote a value for the oversized page in the first place.

Considered (from the issue) and rejected: truncating compiled_truth to
fit under the cap. Silent, position-dependent recall loss, and the byte
cap doesn't line up cleanly with any natural character/token boundary
for UTF-8 content. content_chunks.search_vector already gives full,
untruncated chunk-grain coverage for large pages — truncating a
now-redundant whole-page vector would trade a real bug for a subtler one.

## Test plan

- New test/page-search-vector-overflow.test.ts: a >1MB page (genuinely
  diverse tokens — a repetitive lorem-ipsum-style fixture does NOT
  reproduce this bug, since to_tsvector's cap is on its DEDUPLICATED
  output size, not raw input length) now imports successfully instead of
  throwing; remains keyword-searchable via the chunk-grain path; a normal
  page's search_vector still carries title signal (not fully inert).
  Verified the test is meaningful both directions: fails with the exact
  reported error on the pre-fix trigger (git-stashed the fix, reran,
  confirmed byte-for-byte match: "string is too long for tsvector
  (2684620 bytes, max 1048575 bytes)"), passes with the fix restored.
- Updated fts-language-migration.serial.test.ts: removed an assertion
  that configurable_fts_language (v123) is LATEST_VERSION — that was only
  ever true until the next migration landed; the codebase's own pattern
  elsewhere for this (migrate.test.ts) uses toBeGreaterThanOrEqual, not
  exact-match, for exactly this reason.
- bun run typecheck clean, bun run verify 31/31 green.
- test/page-search-vector-overflow.test.ts (3/3),
  test/reindex-search-vector.serial.test.ts,
  test/fts-language-migration.serial.test.ts, test/migration-v120.test.ts,
  test/sync.test.ts, test/bootstrap.test.ts, test/migrate.test.ts — 254
  total, 0 fail, no regressions.

## Design consultation

Investigated jointly with masa-codex (async design review) before
implementing — their read of the codebase (content_chunks.search_vector
already covers keyword search; pages.search_vector's compiled_truth feed
is the only overflow-prone, effectively-dead write) matched independent
verification and shaped the "remove from trigger" fix over the issue's
alternative options (chunk-grain rebuild — largely already exists; input
truncation — rejected above; ledger-entry-only — insufficient alone,
since the page upsert failing in the same transaction also loses the
chunk write, so checkpoint advancing without this fix would make the
content permanently unsearchable, not just delayed).


Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:51:16 -07:00
MasaandClaude Sonnet 5 1d0b5ed816 fix(autopilot): forward process.env to execSync('which gbrain') under Bun (#2747) (#2976)
resolveGbrainCliPath() (both copies — src/commands/autopilot.ts and the
inlined duplicate in src/core/brain-repo-durability.ts) called `execSync`
without an explicit `env`, relying on default inheritance. Under Bun,
execSync/execFileSync snapshot process.env at BUN'S OWN STARTUP, not at
call time — a runtime PATH mutation (dotenv/config loading, wrapper-script
env sourcing, etc.) happening after Bun boots but before this call is
invisible to `which gbrain` unless the current env is forwarded
explicitly.

This is a known, already-precedented Bun quirk in this exact codebase:
spawn-helpers.ts's detectTini() was already fixed for the identical
symptom with the identical one-line fix (`env: process.env`), with a
comment explaining the mechanism — this call site was simply missed.

Matches the reported symptom precisely: "which gbrain" resolves fine when
run standalone (a fresh Bun process, no prior env mutation to hide), but
throws specifically from inside autopilot's managed-worker spawn path
(src/commands/autopilot.ts:416, guarded by `spawnManagedWorker` — Postgres
engine + minion_mode enabled), which fires after config/dotenv loading has
already run in that process. Impact per the report: this silently
degrades to no worker ever picking up queued jobs (including embed jobs),
with `gbrain doctor` only showing a growing "N stale chunks" warning that
reads like an ordinary backlog rather than a broken worker.

Also improved the throw-path error message to include the actual
PATH/execPath/argv[1] values observed at failure time, so a future report
doesn't require guessing at what the process actually saw.

Not fixed here (documented as a separate, smaller finding): a third call
site with the identical missing-env pattern exists in
src/core/claw-test/runners/openclaw.ts ('which openclaw'). Left out of
scope for this PR, which is specifically about #2747's reported symptom;
worth a small follow-up.

Verification: bun run typecheck clean, bun run verify 31/31 green,
test/autopilot-resolve-cli.test.ts 4/4 pass (existing coverage, no
regressions — a genuinely-simulated Bun-env-snapshot race isn't
reproducible in a same-process unit test, and this codebase's own
convention explicitly avoids mock.module for child_process per
doctor-orphan-ratio.test.ts's stated test-isolation rule, so this PR
relies on the fix's precedent-match + existing coverage rather than a new
mock-based regression test).


Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:39:04 -07:00
MasaandClaude Sonnet 5 4c71a76c0a fix(jobs): retry resets started_at/attempts/stalled_counter (#2783) (#2974)
* fix(jobs): retry resets started_at/attempts_made/attempts_started (#2783)

`gbrain jobs retry` re-queued a dead job by resetting status/error_text/
locks/delay/finished_at, but left started_at, attempts_made, and
attempts_started untouched. On re-claim, claim()'s
`started_at = COALESCE(started_at, now())` preserved the ORIGINAL
first-claim timestamp instead of re-stamping it. handleWallClockTimeouts()
anchors on `now() - started_at`: a retry issued more than timeout_ms * 2
after the original claim was immediately dead-lettered again in under a
second, with attempts_made already past max_attempts — making retry
useless for exactly the case it exists for (recovering work after an
outage that outlasted the job's timeout).

An explicit `jobs retry` is an operator asserting "run this fresh", so
retryJob now also clears started_at (NULL, re-stamped on next claim) and
resets attempts_made/attempts_started to 0.

Two new tests: direct assertion that retry resets all three columns, and
a full repro of the reported bug (wall-clock-killed job retried long
after the original claim now survives re-claim instead of being
immediately re-killed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(jobs): also reset stalled_counter on retry (#2783)

Codex review round 1 found the fix was incomplete: a job dead-lettered by
stall exhaustion (handleStalled() at stalled_counter + 1 >= max_stalled)
retained its exhausted stalled_counter across retry. The retried job's
very first lock expiry after re-claim would immediately re-satisfy the
dead-letter threshold, contradicting the same "run this fresh" intent the
started_at/attempts reset already established.

New test mirrors the existing wall-clock repro: exhaust the stall budget
via two real handleStalled() calls, retry, confirm one more stall now
requeues instead of dead-lettering again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:26:52 -07:00
Masa 354c8c36a9 fix(takes): fail closed when takes-write source resolution errors (#2698 follow-up) (#2973)
resolveTakesSourceId() caught every error from resolveSourceId() and fell
back to undefined, which restores the pre-#2698 unscoped (cross-source)
slug lookup for takes add/update/supersede/resolve. resolveSourceId()
only ever throws when a source was explicitly in play (an invalid or
unregistered GBRAIN_SOURCE, a .gbrain-source dotfile pointing at a
source that doesn't exist, or a genuine DB error) — it never throws for
"nothing configured," which resolves cleanly to the seeded 'default'
source. So swallowing the error had no legitimate case to protect and
only reintroduced the cross-source write bug on any resolution failure.
Let it propagate so the write is blocked instead.

Adds regression coverage for both the unchanged happy path (no source
configured resolves cleanly) and the newly fail-closed path (an
unregistered GBRAIN_SOURCE blocks the write instead of falling back to
an unscoped lookup).
2026-07-20 15:17:43 -07:00
dbf2b3f562 fix(takes): default takes extraction to the configured chat_model instead of hardcoded cloud Haiku (#2997) (#3021)
extractTakesFromPages hardcoded anthropic:claude-haiku-4-5 as the classifier
model. On an OAuth/local-only install (no ANTHROPIC_API_KEY; chat routed
through a gateway model) every takes extraction died with llm_unavailable —
the takes layer silently never populated and the takes_count health check
stayed red despite a working configured chat_model.

Resolution is now `opts.model || getChatModel()` — the same file-plane
gateway-config idiom enrich.ts uses — NOT engine.getConfig('chat_model')
(the DB config plane), keeping model routing on the single config plane the
rest of the codebase reads. Explicit opts.model still wins; unconfigured
installs fall through to the gateway's DEFAULT_CHAT_MODEL.

Adds a regression test that pins the file-plane read: a conflicting DB-plane
config.chat_model row is ignored, the gateway-configured chat_model is used
when opts.model is unset, and explicit opts.model wins. Verified the
file-plane test fails against the pre-fix code.

Takeover of #2997 by @Nazim22 with the model read moved from the DB config
plane to the file-plane gateway idiom.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Nazz <nazim.mj@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 15:05:32 -07:00
9ed53e4e1c fix(code-edges): per-row $n::text::jsonb binds in addCodeEdges — Bun SQL mis-encodes jsonb[] arrays (#2968) (#3020)
Bun SQL double-encodes ::jsonb[] array binds on the Postgres engine: every
edge_metadata element landed as a jsonb string scalar instead of an object,
so the resolver's `edge_metadata || jsonb_build_object(...)` UPDATE produced
a jsonb array and resolved_chunk_id was never readable — code_callers /
code_callees / code_blast returned nothing on Postgres-engine brains while
the resolver logged edges_resolved > 0. PGLite was unaffected (per-row
placeholders already).

Rewrites both inserts (code_edges_chunk, code_edges_symbol) to per-row
$n::text::jsonb placeholders via sql.unsafe — the same shape executeRawJsonb
and the PGLite engine use.

Adds the DATABASE_URL-gated Postgres regression test this class requires
(PGLite cannot reproduce it): asserts jsonb_typeof(edge_metadata) = 'object'
for resolved + unresolved inserts and that the resolver-style || UPDATE
keeps object shape. Verified the test fails 3/3 against the pre-fix code
and passes with the fix.

Takeover of #2968 by @zsimovanforgeops with the missing regression test added.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 14:51:47 -07:00
Masa 9f7244a77f fix(cli): remove sync --install-cron help text — no handler ever existed (#2795) (#2972)
The top-level `gbrain --help` advertised `sync --install-cron` since the
line was first added, but `src/commands/sync.ts` never parsed or handled
the flag — `gbrain sync --install-cron` silently ran an ordinary one-off
sync instead of installing anything, manufacturing false confidence in
the exact durability layer operators reach for it to secure.

git blame shows the line was introduced once (v0.42.29.0 help-text
scaffold) and never touched again — no design intent to recover.
Implementing it would also compete with autopilot, which already owns
this job: `gbrain autopilot --install` runs a self-maintaining daemon
(sync+extract+embed) on a schedule, including a per-source freshness
check that submits `sync` jobs on its own interval. A second, separate
sync-only cron would be a competing scheduler outside the D10
cycle-lock invariant that already keeps autopilot's own targeted-submit
and full-cycle paths from double-processing.

Removed the misleading line and pointed sync's --watch entry at
`autopilot --install`, mirroring the existing `dream` command's
"See also: autopilot --install (continuous daemon)." pattern one
section below. Added regression coverage to
test/cli-help-discoverability.test.ts asserting the help text no
longer promises install-cron and does point at autopilot.
2026-07-20 14:27:50 -07:00
MasaandClaude Sonnet 5 6ec3dd410e fix(gateway): land Anthropic cache_control breakpoints on the system block, not just the call-level auto marker (#2490) (#2981)
gateway.chat() requested cacheSystem:true but never got a system-prompt
cache hit on single-turn callers (page-summary, skillopt, enrich): the
call-level providerOptions.anthropic.cacheControl is real (it becomes
Anthropic's documented top-level "auto-cache the last cacheable block"
shorthand via @ai-sdk/anthropic 3.0.47+), but for a stable system prompt
paired with a different user message every call, "the last cacheable
block" is that ever-varying tail -- every call writes a fresh cache
entry there and never reads a prior one.

Fix: pass system as a SystemModelMessage object (ai's documented shape
for attaching provider options to the system block) carrying its own
providerOptions.anthropic.cacheControl when cacheSystem is requested,
and mirror the same marker onto the last tool def (Anthropic caches
everything up to and including the last cache_control block it sees).
The call-level marker is kept, not removed -- it still gives toolLoop()'s
growing multi-turn conversation a rolling cache breakpoint on each
turn's tail. All three markers now derive from one canonical
cacheControlValue computed after provider_chat_options config merging,
so a configured TTL override (e.g. ttl: '1h') applies consistently
instead of only reaching the call-level marker.

Verified red-before-fix by stashing the gateway.ts diff and confirming
the new assertions fail on unfixed code, then restoring.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:16:00 -07:00
MasaandClaude Sonnet 5 bcf3b73dcf fix(sync): self-heal a never-git-initialized default brain dir (#2964) (#2967)
* fix(sync): self-heal a never-git-initialized default brain dir (#2964)

The dream cycle's sync phase throws unconditionally on a legacy
sync.repo_path-anchored default brain dir that was never git init-ed
(predates git-backed sync, or was rsync'd without its .git), failing
every nightly run with no recovery. doctor's sync_freshness/
sync_consolidation checks report "ok" for this exact brain, but only
because they query the sources table (0 rows for a legacy default
brain) — a coincidental false-negative, not a real diagnosis.

Self-heal by git-initializing the dir and capturing the current
on-disk state as the sync baseline, scoped to !opts.sourceId only —
gbrain owns this directory outright, unlike a registered local source
(sources add --path, no --url) which is the user's own external
directory and should keep failing loudly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): dry-run no-write contract, unborn-HEAD recovery, no-gpg-sign (#2964)

Codex review on the initial self-heal patch (b1671ee) found 3 real gaps:

- P1: the self-heal ran even under --dry-run, mutating the filesystem
  during what's documented as a preview-only command. Gated the whole
  self-heal (both discoverGitRoot and the headCommit read) on
  !opts.dryRun, same as the existing !opts.sourceId ownership check.
- P2: if `git init` succeeded but the process died before the baseline
  commit landed, the next run's discoverGitRoot would succeed (`.git`
  exists) and skip recovery entirely, permanently wedging on "No
  commits in repo" forever. Added the same self-heal at the
  `git rev-parse HEAD` catch site, sharing a new createSyncBaselineCommit
  helper with the discoverGitRoot catch.
- P2: the baseline commit inherited the operator's global
  commit.gpgSign, which can block headless cron/launchd runs on an
  unavailable signing agent/pinentry. Added --no-gpg-sign.

Two new tests cover dry-run no-mutation and unborn-HEAD recovery.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): restrict git auto-init self-heal to the anchor-resolved path only (#2964)

Second Codex review round (a10eeab) found the ownership check still too
loose:

- P1 (security): !opts.sourceId alone isn't proof gbrain owns repoPath.
  jobs.ts's `sync` job handler leaves sourceId undefined whenever
  job.data.repoPath doesn't match a registered source's local_path, so
  an admin-scope submit_job({name:'sync', data:{repoPath}}) MCP call
  could point the self-heal at an arbitrary directory and have it
  silently git-init + commit + ingest it. Gated both self-heal sites on
  !opts.repoPath too — only the path resolved from gbrain's own
  sync.repo_path anchor (never a caller-supplied one) is eligible.

- P2: the unborn-HEAD recovery site calls discoverGitRoot, which walks
  UP from repoPath and can resolve to an ANCESTOR repo for a
  --src-subpath/subdir-as-repoPath sync with an unborn HEAD. Committing
  there would `git add -A` sibling files well outside the sync scope.
  Added a check that gitContextRoot === realpathSync(repoPath) before
  self-healing; refuses (falls through to the original error) otherwise.

Tests rewritten to exercise the true self-heal-eligible path (anchor
config via engine.setConfig('sync.repo_path', dir), no repoPath/sourceId
passed) instead of an explicit repoPath, plus a new test asserting a
caller-supplied repoPath with no sourceId still throws.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): prove self-heal ownership by anchor VALUE, not field presence (#2964)

Third Codex review round (613ad5b) found the previous round's fix broke
the very call site it was meant to repair, plus a second scope gap:

- P1 (critical): !opts.repoPath rejected self-heal on the REAL production
  callers too. runPhaseSync (dream cycle's sync phase, cycle.ts) always
  passes `repoPath: brainDir` explicitly after resolving it upstream, and
  the CLI's bare `gbrain sync` resolves sourceId='default'. Both made the
  ownership gate rethrow, leaving `gbrain dream` and `gbrain sync`
  wedged on the exact non-git legacy brain this fix targets — only
  synthetic callers that omitted both fields ever healed.

  Fixed by proving ownership by VALUE instead of by field absence: a new
  isAnchorOwnedSyncPath() re-reads gbrain's own persisted
  sync.repo_path config and requires the resolved repoPath to equal it
  exactly, regardless of whether the caller passed it explicitly or let
  it default. An attacker-supplied arbitrary path (e.g. via
  submit_job({name:'sync', data:{repoPath}})) only self-heals if it
  happens to already equal gbrain's own anchor — which is the
  legitimate case, not an escalation. opts.sourceId and opts.srcSubpath
  still disqualify unconditionally (registered/subpath-scoped syncs are
  a different ownership context).

- P2: a --src-subpath sync with an unborn parent-repo HEAD would commit
  the whole ancestor root, capturing sibling files outside the scope.
  isAnchorOwnedSyncPath's opts.srcSubpath check closes this; the
  existing gitContextRoot === repoPath check stays as defense in depth.

- P2: manageGitignore's "warn and return" contract (a deliberate side-
  effect that must never kill the sync job for its OTHER callers) meant
  a broken gbrain.yml or unwritable .gitignore would silently let the
  baseline `git add -A` commit db_only content. createSyncBaselineCommit
  now recomputes db_only exclusion directly from loadStorageConfig and
  passes it to `git add` as pathspecs, independent of the .gitignore
  write's success — true fail-closed. (A redundant pathspec exclude for
  a path .gitignore ALREADY covers makes git's -A bail with "paths
  ignored, use -f" even though the negation is correct, so each dir is
  check-ignore'd first and only pathspec-excluded when NOT already
  covered.)

Tests rewritten around the anchor-VALUE model: the critical regression
case (explicit repoPath matching the anchor still heals — the exact
scenario Codex proved was broken) plus a true negative (a caller-supplied
path that does NOT match the anchor still throws), --src-subpath refusal,
and db_only fail-closed exclusion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): allow default-source ownership, realpath compare, generous timeout, --no-verify (#2964)

Fourth Codex review round (9c1d461) found the previous round's ownership
gate still didn't match the REAL installed-brain shape, plus 3 more gaps:

- P1 (critical): rejecting all non-empty opts.sourceId meant self-heal
  still never fired on a real brain. Migration sources_table_additive
  seeds a 'default' source row whose local_path mirrors sync.repo_path
  on every brain that's run it (virtually all of them), so
  resolveSourceForDir (dream cycle) and bare `gbrain sync` both resolve
  sourceId:'default' in practice, never undefined. isAnchorOwnedSyncPath
  now permits sourceId undefined OR exactly 'default' (gbrain's own
  bootstrap identity, never something a caller names) and proves
  ownership by rereading the LIVE anchor for that same identity
  (sources.default.local_path vs config.sync.repo_path).

- P2: compared raw anchor/repoPath strings, so a cosmetic difference
  (trailing slash, ..) between the stored anchor and dream.ts's
  path.resolve()-normalized brainDir would defeat the match. Now
  realpath-compares both sides (fail-closed on ENOENT/dangling).

- P2: the shared git() helper's 30s timeout could abort the baseline
  `git add -A` on a large legacy brain mid-way, after `git init` already
  created `.git` — leaving an unborn repo every subsequent sync would
  retry and time out identically forever. Added an optional timeoutMs
  param (default unchanged at 30s); the baseline add call uses 10min.

- P2: the baseline commit could trigger an operator's global
  core.hooksPath/init.templateDir hooks (pre-commit/commit-msg),
  breaking headless recovery if those hooks need project tooling or
  prompt. Added --no-verify.

Tests: rewrote the mis-scoped "registered source" test (it used
sourceId:'default', which is now correctly permitted) into two — a new
regression test proving sourceId='default' + matching local_path heals
(the actual production shape), and a corrected non-default-sourceId
refusal test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): defer .gitignore write past first import, rebuild index, fail closed on unparseable db_only (#2964)

Fifth Codex review round (c17cd23) found the baseline-commit helper
interacting badly with the pre-existing db_only storage-tiering feature:

- P1 (data loss): createSyncBaselineCommit called manageGitignore BEFORE
  performFullSync's collectSyncableFiles ran. collectSyncableFiles
  enumerates via `git ls-files --cached --others --exclude-standard`, so
  writing db_only entries into .gitignore first would silently exclude
  those pages from the DATABASE, not just from git — on a brain's very
  first sync. This is the exact bug class runSync's existing "manage
  .gitignore ONLY on successful sync" ordering (this file, ~line 4540,
  itself a prior Codex P1 fix, comment literally says so) was written to
  prevent — my new code reintroduced it in a different spot. Fix: stopped
  calling manageGitignore inside the self-heal at all. db_only exclusion
  for the COMMIT still happens via the existing pathspec computation
  (independent of .gitignore); .gitignore itself gets written by the
  already-existing post-sync flow once this sync completes, same as any
  other sync.

- P1 (data leak): the unborn-HEAD recovery site can reach
  createSyncBaselineCommit with a repo whose INDEX already has entries
  staged from some prior operation (manual `git add`, interrupted
  workflow) before gbrain's self-heal ever touched it. `git add -A`
  only adds/updates — it doesn't drop an already-staged path our
  exclusion pathspecs now want excluded. Added `git read-tree --empty`
  to reset the index before staging (no-op on a freshly-`git init`-ed
  repo, whose index is already empty).

- P2: loadStorageConfig warns-and-returns an EMPTY config (not a throw)
  for syntactically-valid-but-unsupported YAML (e.g. flow-style
  `db_only: [dir/]` — the narrow custom parser only handles block-style
  lists), which would silently resolve zero exclusions from a gbrain.yml
  that clearly intended some. Added a sniff-test: if gbrain.yml exists
  and mentions db_only but nothing resolved from it, refuse the baseline
  commit rather than guess "genuinely empty" vs "syntax silently
  ignored" (git init may already have run by this point — same "unborn,
  retry on next sync" recovery path handles it, and will hit this same
  guard again until the user fixes gbrain.yml).

Tests: a positive regression proving db_only markdown IS imported into
the DB on first sync (the actual data-loss scenario), the sniff-test
refusal, and the stale-staged-content-gets-dropped case for the index
rebuild.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): post-heal .gitignore write, supabase_only alias, honor abort signal (#2964)

Sixth Codex review round (e687913), 3 P2s:

- Dream-cycle callers (cycle.ts:runPhaseSync) invoke performSync directly
  and never run runSync's CLI-only post-success manageGitignoreAtGitRoot.
  A brain self-healed only via the dream cycle would have db_only content
  correctly excluded from the baseline commit (createSyncBaselineCommit's
  pathspec exclusion) but no .gitignore ever written, leaving the user's
  own future manual git add/commit unprotected. Added
  performFullSyncAndMaybeGitignore, a thin wrapper around the 3
  post-self-heal performFullSync call sites that writes .gitignore
  (same success-status gate runSync already uses) only when didSelfHeal
  is true — a no-op for the normal path, which still relies on runSync
  exactly as before.

- The fail-closed sniff-test only checked the canonical `db_only` key;
  the deprecated-but-still-supported `supabase_only` alias (same
  keep-out-of-git semantics) could silently bypass it. Now checks both.

- Self-heal didn't check opts.signal?.aborted before starting the
  (now up to 10-minute) git init + baseline commit, so a cancelled sync
  could still mutate disk and overrun its budget instead of returning
  partial. Added the check at both self-heal sites, before any git
  operation runs.

New test proves .gitignore gets written after a bare performSync call
(no runSync wrapper) — the actual dream-cycle shape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): neutralize a leftover .gitignore during the self-heal first sync (#2964)

Seventh Codex review round (27337b0) ran an actual repro and caught the
primary motivating scenario still broken: a brain rsync'd from another
machine without its .git can retain that machine's old auto-managed
.gitignore. collectSyncableFiles (inside performFullSync) enumerates via
`git ls-files --exclude-standard`, so a leftover db_only ignore rule
would silently omit those pages from THIS first sync's DATABASE import —
the same bug class the round-6 ordering fix prevented for a .gitignore
gbrain would have written itself, just triggered by a pre-existing file
this time.

Fix: performFullSyncAndMaybeGitignore now neutralizes any existing
.gitignore for the duration of the one first-sync call — read, delete,
restore byte-for-byte immediately after (even on error) — before
manageGitignore re-merges the managed db_only block onto the restored
original content. This matches exactly what a truly fresh brain with no
.gitignore at all already does on its first sync (nothing to suppress
collection there either); db_only content stays out of the git COMMIT
independently via createSyncBaselineCommit's pathspec exclusion, which
never depended on .gitignore.

Test proves both halves: db_only markdown IS imported despite a leftover
ignore rule, AND the user's own unrelated .gitignore lines (e.g.
.DS_Store) survive the restore intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): simplify — drop db_only-import machinery, isolate hooks fully (#2964)

Eighth Codex review round (54c1e6f) found MORE problems with round 7's
.gitignore-neutralization fix (deleting the whole file loses the user's
own unrelated ignore rules; a multi-sync retry scenario could silently
skip a still-broken db_only file while advancing the bookmark) plus 2
more issues in existing code. Rather than patch those too, stepped back
and checked the actual documented semantics of db_only
(docs/storage-tiering.md): it's for "bulk machine-generated content...
written to disk as a local cache" — DB is the source of truth, disk is a
cache populated FROM the DB (`export --restore-only` restores it), never
the other way. Nothing in the docs says `gbrain sync`'s git-diff-based
file collection is how db_only content is supposed to reach the
database — that's ingest-specific tooling's job. Confirmed directly:
`loadStorageConfig` returns the byte-identical `{db_tracked:[],
db_only:[]}` for a malformed flow-style array AND a literal empty
`db_only: []`, so rounds 6-7's "ensure db_only markdown gets imported on
this first sync" chase was solving a problem outside sync's actual scope
in the first place, on an increasingly complex, adversarially-discovered-
edge-case foundation.

Reverted: performFullSyncAndMaybeGitignore (the wrapper + didSelfHeal
tracking + .gitignore neutralize/restore dance + post-success
manageGitignore call). After self-heal, import and any subsequent
.gitignore management now behave EXACTLY like any other brain,
self-healed or not — runSync's existing post-success
manageGitignoreAtGitRoot covers the CLI path identically either way; the
dream cycle not calling it is a separate, pre-existing characteristic of
the dream cycle in general (applies equally to an already-git-initialized
brain going through the same path), not something this fix introduces.

Kept (still correct, self-contained, don't depend on the reverted
machinery): createSyncBaselineCommit's pathspec-based db_only exclusion
for the COMMIT itself (matches the documented "not committed to git"
requirement), the fail-closed sniff-test guard (now documents its
known, structurally-unavoidable false-positive on a genuinely-empty
`db_only: []` — the trade-off is deliberate: low-cost, self-resolving
false positive vs. high-cost, hard-to-undo false negative), the index
rebuild, and the 600s add timeout.

Improved (round 8, P2): hooks isolation. --no-verify only skips
pre-commit/commit-msg; added `-c core.hooksPath=/dev/null` for the
baseline commit, which disables prepare-commit-msg and post-commit too
(the latter runs synchronously inside the same git invocation and could
otherwise hang past the timeout without even being the slow step).

Tests: removed the 3 that exercised the reverted db_only-import
machinery; the remaining 12 (ownership, dry-run, index rebuild, sniff
test, commit-exclusion, unborn-HEAD recovery) are unaffected by the
simplification.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

* fix(sync): unconditional db_only exclusion, literal pathspecs, precise sniff test (#2964)

Ninth Codex review round (a6e07f6):

- P1: the check-ignore pre-filter (skip pathspec-excluding a dir already
  covered by .gitignore) could be defeated by a pre-existing .gitignore
  that ignores a db_only tree with a wildcard but re-includes a child via
  negation (e.g. `private-cache/*` + `!private-cache/index.md`) —
  check-ignore on the directory still reports "ignored", so the filter
  skipped the pathspec exclusion, and `git add -A` staged the re-included
  child anyway. Fixed by making exclusion unconditional: every db_only
  dir is always pathspec-excluded now, never pre-filtered against
  .gitignore state at all — our own pathspec doesn't consult .gitignore,
  so no .gitignore content (negated or not) can defeat it. The advisory
  "paths ignored... use -f" error this can now trigger when a dir IS also
  already .gitignore'd (verified: git still stages everything else
  correctly despite the nonzero exit) is caught and swallowed by matching
  its exact stderr text; anything else rethrows.

- P2: `:!dir` pathspec shorthand reinterprets a dir name that itself
  starts with a pathspec magic character (e.g. `:private/`) instead of
  excluding it literally. Switched to `:(exclude,literal)dir`.

- P2: the fail-closed sniff-test's bare substring search on gbrain.yml's
  raw content could trip on a comment or unrelated prose mentioning
  "db_only" even when there's no real storage section at all, refusing
  self-heal forever on an unrelated false positive. Now requires an
  actual YAML key line (`db_only:`/`supabase_only:`, trimmed, ignoring
  `#` comments) — the round-8-documented "genuinely empty db_only: []"
  false positive is unchanged and remains an accepted trade-off (still
  structurally indistinguishable from unsupported syntax at the
  loadStorageConfig API boundary), but comment/prose mentions no longer
  false-positive.

Not fixed (deliberately, documented trade-off — see PR description):
Codex's other P1 this round (refuse baselining when other git refs/
history exist alongside an unborn HEAD) is a narrow, non-destructive
scenario — self-heal only ever acts on the current branch ref when it's
provably commit-less, never touches or deletes any other ref (remote-
tracking, other branches), so at worst it creates a possibly-unexpected
extra commit on an otherwise-empty branch the user hadn't checked out
yet. Chasing it further trades diminishing real-world risk reduction
against unbounded scope growth in what's fundamentally still the
self-heal fix from round 1.

Two new tests: unconditional exclusion despite a matching pre-existing
.gitignore (proves the advisory-swallow path), and a comment-only
gbrain.yml no longer false-positives the sniff test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:06:21 -07:00
MasaandClaude Fable 5 23e0541d9b fix(cycle): budget the patterns subagent from remaining job time — one phase's fixed 35-min worst case defeats any interval-derived cycle budget (#2781) (#2959)
* fix(cycle): budget the patterns subagent from remaining job time, not a fixed constant (#2781)

The autopilot-cycle job gets an interval-derived timeout stamped at submit,
but the patterns phase submits its subagent with a fixed 30-min job timeout
and waits up to 35 min — one phase's worst case exceeds ANY interval-derived
budget <= 35 min, so the parent job dead-letters mid-patterns and the tail
phases (consolidate -> schema-suggest) starve for days (#2781, the deeper
half left open by the #2852 dispatch-floor fix).

- MinionJobContext.deadlineAtMs: absolute deadline from the claim-time
  timeout_at stamp (the DB ground truth handleTimeouts() sweeps against;
  re-stamped on every claim so retries get a fresh budget). Null when the
  job has no per-job timeout.
- worker: the per-job abort timer now derives its delay from timeout_at
  when present, so the in-process timer, the DB sweeper, and the
  handler-visible deadline agree on ONE absolute instant.
- autopilot-cycle + autopilot-global-maintenance handlers thread
  deadlineAtMs into runCycle; CycleOpts carries it to the patterns phase.
- patterns: clampSubagentBudgets() derives BOTH the child job timeout and
  the wait timeout from the same child deadline (parent deadline minus a
  60s stop-margin reserve — enough for the wait poll + force-evict grace
  + cleanup, deliberately NOT a promise that tail phases complete). Under
  a 2-min minimum the phase skips honestly (insufficient_cycle_budget)
  instead of submitting a guaranteed-kill LLM call; the next cycle
  retries with a fresh budget.
- Direct callers (gbrain dream) pass no deadline and keep the configured
  timeouts unchanged.

Follow-up (separate PR): synthesize has the same shape plus sequential
per-child waits that accumulate N x subagent_wait_timeout_ms past any
parent budget; it needs per-wait remaining-time recomputation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

* fix(cycle): address review — cancel timed-out patterns child; thread deadline through phase-wrapper handlers

- P1: the child's timeout_ms clock starts at ITS claim, so a queued child
  could outlive the parent deadline the wait was clamped to. On wait
  timeout, cancelJob strips it (waiting -> cancelled; active -> lock
  stripped, worker abort fires next renew tick).
- P2: makePhaseHandler (standalone patterns/synthesize/... minion jobs)
  now threads job.deadlineAtMs into runCycle like the autopilot handlers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:56:00 -07:00
bo-developingandClaude Opus 4.8 c873ce3014 feat(ai): add Mistral provider recipe (#3001)
Adds an EU-hosted provider covering embedding, expansion and chat on one
OpenAI-compatible endpoint (https://api.mistral.ai/v1), so a brain that must
stay inside EU jurisdiction does not need a US hop for any AI touchpoint.

Every field is measured against the live API, not copied from docs:

- mistral-embed is fixed 1024 dims and accepts no dimension parameter.
  Both spellings are rejected: {"dimensions": N} returns 400 extra_forbidden,
  {"output_dimension": N} returns 400 "does not support output_dimension".
  The generic openai-compatible branch of dimsProviderOptions() already falls
  through to `return undefined` for these model ids, so nothing is emitted.
  Same contract as voyage-4-nano, pinned by a negative assertion in the test.
- max_batch_tokens 65536: a 65,286-token batch is accepted, 66,960 returns
  400 code 3210 "Too many tokens overall, split into more batches."
- chars_per_token 2: the value is a DIVISOR in splitByTokenBudget()
  (estTokens = text.length / charsPerToken), so lower is the conservative
  direction. The module default of 4 assumes English prose; a German-language
  corpus measured 3.58 chars/token, which the default overshoots toward
  overflow.

codestral-embed is deliberately left out: it returns 1536 dims, and a
touchpoint carries a single default_dims. Listing it under a 1024 declaration
is the mixed-dim case embedding-dim-check.ts exists to catch.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 13:42:41 -07:00
hhamilton-fv f3e78fd2fb fix(providers): route diagnostics through buildGatewayConfig so file-plane keys are visible (#3000)
configureFromEnv() hand-assembled its own AIGatewayConfig instead of calling
buildGatewayConfig(), the single seam that folds file-plane API keys
(openrouter_api_key, zeroentropy_api_key, ...) into the gateway env. That let
`gbrain providers list`/`test` report a provider as missing env even when it
was correctly set in ~/.gbrain/config.json and the real gateway path resolved
it fine. Both configureFromEnv() and runList() now build their env through
buildGatewayConfig() (falling back to a bare process.env passthrough
pre-init), matching what init-provider-picker.ts already does.
2026-07-20 13:07:27 -07:00
Nazim22andClaude Fable 5 4528bfa79c feat(cli): GBRAIN_DRAIN_TIMEOUT_MS env override for the per-sink teardown drain budget (#2996)
The one-shot CLI teardown drains fire-and-forget background sinks with a
hardcoded 2s per-sink budget (DEFAULT_DRAIN_TIMEOUT_MS). That budget
assumes a sub-second cloud chat provider; on a self-hosted provider (e.g.
an ollama model at 10-20s per completion) a facts:absorb extraction can
never finish inside it, so every one-shot CLI exit — sync timers
especially — aborts the in-flight chat with
'pipeline_error: The operation was aborted', and the same touched pages
retry-and-abort on every subsequent sync. Facts from those pages silently
never land, and doctor's facts_extraction_health warns permanently.

Fix: resolveDrainTimeoutMs() — GBRAIN_DRAIN_TIMEOUT_MS env override
(same env-only escape-hatch pattern as GBRAIN_TEARDOWN_DEADLINE_MS and
GBRAIN_FLUSH_GRACE_MS) over the 2000ms default. Explicit drainTimeoutMs
from a call site still wins; computeTeardownDeadlineMs already computes
the backstop from the resolved value, so the deadline scales with it.
Garbage/zero/negative env values fall back to the default.

Tests: default, env override, garbage/zero/negative fallback,
finishCliTeardown drains with the env-resolved budget, explicit opts
still win over env.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:00:16 -07:00
Anton Senkovskiy 6498b872ea fix(extract): --dry-run must not write extract_rollup_7d (#2994)
`extract-conversation-facts --dry-run` promises "no DB writes, no
checkpoint advance" (help text) and correctly skips the fact INSERTs,
orphan delete, checkpoint advance, and receipt page. But
writeRunReceiptAndRollup was called unconditionally at both exit paths,
and its upsertExtractRollup always UPSERTs a row into extract_rollup_7d
("ALWAYS fire so doctor's extract_health sees the cycle ran") — so a
dry run mutates the DB.

Gate both writeRunReceiptAndRollup call sites on !dryRun. The writer
returns void and its only non-rollup action (the receipt page) is
already suppressed in dry-run via facts_inserted > 0, so gating at the
call site skips nothing else. Mirrors the existing !dryRun guards on
the fact-insert / checkpoint / audit paths.

Regression test: a dry run leaves extract_rollup_7d empty. Fails before
(row count 1), passes after. Hermetic PGLite + stubbed transports, no
live LLM.

Note: --dry-run still calls the extractor (LLM) by design — facts_extracted
is the reported "would extract N" preview count; left unchanged.
2026-07-20 12:16:57 -07:00
324c355318 test(e2e): production guard — setupDB refuses non-test databases (#2957)
setupDB() TRUNCATEs every data table on whatever DATABASE_URL points at,
and run-e2e.sh deliberately preserves an exported DATABASE_URL — one
stray environment variable away from wiping a production brain, with no
guard of any kind (found during an independent review, 2026-07-18).

assertSafeE2eDatabaseUrl (pure, unit-tested) now runs before any
connection: allowed when the database name carries "test" as a word
segment (the gbrain_test convention used by CI and
.env.testing.example), or when GBRAIN_E2E_ALLOW_DB names the exact
database intentionally. Refusal is loud and actionable. 7 unit tests,
no DB required.

Co-authored-by: Aleksei Razsadin <razsoc.01@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 12:05:04 -07:00
184b6cb8a1 fix search: title candidate arm + gated OR fallback for lexical recall (#2956)
Pages were unreachable by their own exact titles: FTS indexed only chunk
body text while the title-weighted pages.search_vector (GIN-indexed since
its introduction) was never queried by any search path, and
websearch_to_tsquery AND-at-chunk-grain semantics meant one non-matching
token zeroed keyword recall with no fallback — long or acronym-bearing
titles (e.g. "IAWG ... AAR-LL deck") fell through to the vector arm alone
and missed.

- searchTitles (both engines): page-grain candidate arm over
  pages.search_vector (title 'A' + compiled_truth 'B' + timeline 'C'),
  ts_rank_cd ranked, representative-chunk LATERAL join, full filter
  parity with searchKeyword (visibility, soft-delete, source grants,
  hard-excludes, dates, types); fused as a weighted RRF list at the
  keyword arm's intent-effective k on all three hybrid return paths;
  fail-open with warnOncePerProcess. No schema changes — the index
  already existed, dark.
- AND->OR one-retry fallback for the keyword arm, gated behind
  SearchOpts.orFallback (only hybridSearch opts in; countMentions, link
  resolution, eval, and keyword-only MCP callers keep the strict-AND
  contract). Refused for queries carrying websearch operators (negation,
  quoted phrases). searchTitles carries its own page-grain fallback.
- Lexical arms parallelized (Promise.all) on the main path.

Verified: typecheck clean; 18 hermetic PGLite tests + 2 engine-parity e2e
cases (CI Postgres); consumer regression enrichment 18/0 +
link-extraction 127/0; independent live QA on a 10,664-page brain —
exact-title target miss -> rank 1 (exact_title_match), controls held,
negation/quoted guards proven, strict-consumer contract pinned.

Diagnosed from a 3-lane read-only diagnostic; adversarial review round
closed findings on fallback scope, Postgres test coverage, and operator
handling before this commit.

Co-authored-by: Aleksei Razsadin <razsoc.01@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:57:31 -07:00
MasaandClaude Fable 5 912407bef1 fix(search): per-call token-budget meta masked the real cut on both cache paths; restore vacuous search-lite coverage (#2954)
* fix(search): per-call token-budget meta no longer masks the real cut; restore vacuous search-lite coverage

The search-lite integration tests were structurally vacuous: putPage never
creates chunks and searchKeyword joins content_chunks, so every fixture
query returned zero rows on every machine. The tight-budget cut test's
defensive skip ('keyword search may dedupe by page') silently returned
before its assertions had ever executed anywhere, and the two budget-meta
tests ran against empty result sets.

Restoring the fixture (upsertChunks per page) and hardening the
assertions immediately exposed a real meta bug: with a per-call
tokenBudget, the inner hybridSearch enforces the same resolved budget
(per-call wins in resolveSearchMode) and its meta carries the true
dropped count — but hybridSearchCached re-applies the budget to the
already-cut set and published THAT pass's meta, which always reads
dropped=0. onMeta consumers saw a budget record claiming nothing was
dropped while rows were; telemetry (recorded from the inner meta)
disagreed with the caller-visible meta.

- hybrid.ts finalMeta: prefer innerMeta.token_budget when a per-call
  budget is set (outer budgetMeta stays as the fallback and remains the
  enforcement for the cache-HIT path, where no inner run exists)
- test fixture: chunk each page (pattern: chunk-grain-fts.test.ts)
- cut test: defensive skip replaced with a hard >=2 precondition;
  results non-empty + strictly-fewer-than-unbounded + dropped>0 now
  actually execute (revert-checked red on the unfixed meta)
- budget-meta tests: assert non-empty result sets so kept=results.length
  can no longer pass vacuously at 0=0

The unbounded 'builder' query returns 2 of 3 fixture pages by design —
dedup Layer 3 caps any single page type at 60% of results and the
fixture is all-person — which the >=2 precondition accommodates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

* fix(search): cache-HIT budget meta prefers the stored cut record per review; exact-count assertions, mixed-type fixture

- hit path had the symmetric masking (codex P2): the re-application runs
  on the already-trimmed stored set and read dropped=0 while the miss
  that produced the same result set reported the real cut. Prefer
  hit.meta.token_budget unconditionally — tokenBudget is folded into
  knobsHash ('tb='), so a hit only ever serves a lookup with the
  identical resolved budget as the write and the outer pass can never
  cut further (verified against mode.ts; this is why the reviewer's
  'outer wins when it drops' branch is unreachable). budgetMeta remains
  the fallback for legacy rows stored without a budget record
- new serial test drives a real store-then-hit roundtrip (mocked
  embedQuery, real PGLite cache) and pins hit token_budget == miss
  token_budget; revert-checked red (dropped=0) on the unfixed path
- lite test: dropped asserted as the exact unbounded-minus-kept count
  and used>0 (dropped>0 alone accepts any wrong positive; used<=250
  alone accepts a bogus zero), fixture types mixed (person/company/note)
  so dedup Layer-3 type-diversity policy no longer shapes the test

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:49:47 -07:00
MasaandClaude Fable 5 89f226eb38 fix(search): classify cache hit/miss in telemetry — hits were invisible, misses unclassified (#2953)
* fix(search): classify cache hit/miss in telemetry — hits were invisible, misses unclassified (#2952)

search stats reported 0 hit / 0 miss forever: recordSearchTelemetry fired
only from bare hybridSearch, whose meta never carries a cache field, and a
cache HIT returned from hybridSearchCached before any record at all — so
hit searches also vanished from count/results/tokens/rank-1.

- HybridSearchOpts: internal _telemetryCacheStatus ('miss' | 'disabled')
  threaded from hybridSearchCached into the inner hybridSearch (same
  pattern as _queryEmbedDeadline), folded into the RECORDED meta only —
  onMeta payloads unchanged, count/sum_tokens/budget_dropped/rank-1
  behavior byte-identical for the miss/disabled paths
- hit path: record once from hybridSearchCached with the already-built
  cachedMeta (cache.status='hit'), post-slice/budget result count, tokens
  from the budget pass, and the same rank-1 rule as the inner paths
- bare hybridSearch direct callers (think/gather, brainstorm, enrich,
  evals, ...) keep recording exactly as before, with no cache field
- test: serial wiring test drives a real store-then-hit roundtrip through
  hybridSearchCached (mocked embedQuery, real PGLite SemanticQueryCache)
  and pins the decision matrix (miss / hit / consult-skipped / bare);
  revert-checked red on pre-fix source at the miss classification

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

* fix(search): harden cache-hit telemetry per review — mode-gated tokens, embed-failure coverage

- hit-path tokens_estimate now gated on the MODE-resolved budget,
  mirroring the inner paths' resolvedMode.tokenBudget > 0 meta condition
  (a tokenmax budget-off brain would otherwise record real tokens on hits
  but 0 on misses, inflating avg-tokens as the hit rate rises)
- test: exact token-delta parity assertion (hit contributes the same
  tokens as the miss that stored the served set) — catches the class of
  hit/miss accounting asymmetry the increase-only check accepted
- test: embed-provider-failure flavor of the disabled path (consult
  degrades via catch, keyword fallback serves, neither counter bumps)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:35:02 -07:00
Sailesh SivakumarandClaude Fable 5 f1031d5a0b fix(cli): stop thin-client jobs/config from fabricating a scratch PGLite (#2951)
`jobs list|get` have had remote MCP routing since v0.32, but the CLI
shell still ran connectEngine() before dispatch — on a thin-client
install that fabricates an empty scratch PGLite in the thin-client
GBRAIN_HOME and replays the entire migration chain on every invocation,
before the remote call even runs. Host-only jobs subcommands (work,
supervisor, submit, ...) and `config` did the same instead of refusing.

- cli.ts: dispatch thin-client `jobs list|get` engine-free
  (runJobs(null, ...)); refuse the other jobs subcommands with a
  pinpoint hint; add `config` to THIN_CLIENT_REFUSED_COMMANDS with a
  hint (it reads/writes the host brain's config plane).
- jobs.ts: widen runJobs to accept a null engine, guarded so null can
  only reach the MCP-routed list/get branches.
- tests: behavioral (no scratch store created, no migration replay,
  refusals carry hints) + source-audit pins in the existing idioms.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:25:46 -07:00
Sailesh SivakumarandClaude Fable 5 3a5c4c194c fix(remote): poll MinionJob.status, not .state — ping now sees completion (#2950)
submit_job/get_job return the MinionJob row verbatim; its lifecycle
field is `status` (src/core/minions/types.ts), not `state`. remote ping
typed and read `state`, so every poll saw undefined, the terminal check
never matched, and ping always burned its full --timeout and exited 1
even when the autopilot-cycle had completed — printing
"Job #N is still undefined." on the way out.

Reads fixed to `status`; the ping's own JSON output keys (`state`,
`last_state`) are unchanged for consumers. Source-audit regression test
pins the field reads.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:16:21 -07:00
MasaandClaude Fable 5 d165e99f0b fix(brain-writer): deadline race verdict from the sentinel, not the clock the timer raced (#2947)
Part of #2946. The hung-COUNT deadline race derived its verdict from a
post-await wall-clock re-check, which races the timer's own drift: on
loaded CI runners setTimeout callbacks fire measurably EARLY relative to
Date.now(), so each padding/boundary adjustment (>=, +1ms) only moved
which wrong status the partial-scan test received ('scanned', then
'partial').

The race's timeout arm now resolves a module-private sentinel; the
sentinel winning IS the deadline verdict (deadlineHit), consulted by the
post-await check without re-reading the clock. A COUNT that resolves
null (failed/absent count) stays distinguishable and does not skip the
scan; a COUNT that resolves slowly without the timer winning is still
caught by the retained wall-clock re-check. The +1ms pad is gone — the
sentinel makes timer drift irrelevant for the hung path.

Verified: partial-scan suite green 8 consecutive runs incl. the new
null-vs-sentinel distinction test.


Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:08:41 -07:00
raymeboltdandOpenAI Codex 8b325041ee v0.42.63.0 fix: preserve configured PGLite schema database path (#3016)
* fix(schema): preserve configured PGLite database path

* chore: bump version and changelog (v0.42.63.0)

Co-Authored-By: OpenAI Codex <noreply@openai.com>

---------

Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-20 10:46:46 -07:00
a46f28a63e fix(cli): keep doctor --json stdout clean — v123 migration handler printed to stdout (#3019)
The v123 configurable-FTS migration (#2941) logged its completion notice
via console.log. Migrations run lazily inside any command's first DB
connect, so on the nightly heavy run (fresh Postgres service DB) the
line landed as the first line of `gbrain doctor --json` stdout and broke
the fm_wallclock jq parse ("Invalid numeric literal at line 1, column 7",
run 29731426470). runMigrations' contract routes all migration noise to
stderr; move the v123 prints (and the pre-existing v2 slug-rename print)
there.

Also un-vacuous the fm_wallclock harness: its register-source step used
`bun run -e` (bun dumps usage with exit 0 instead of running the code)
and `connect({})` (in-memory), so the source was never registered and
doctor scanned nothing. It now resolves the engine the way the CLI does
and registers the source in the DB doctor actually reads.

Regression test: test/migrate-stdout-clean.test.ts re-runs migrations
from v122 asserting zero stdout writes, plus a source-level guard that
migrate.ts contains no console.log.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 10:33:27 -07:00
f72de97943 feat(sync): --src-subpath + --exclude for monorepo subdir-source support (#753, supersedes #774) (#2942)
* feat(sync): --src-subpath + --exclude for monorepo subdir-source support (#753)

Rebased port of PR #774 onto current master. A single git repo can hold N
logical sources at subdirectories: git operations run at the discovered
repo root (git rev-parse --show-toplevel — worktrees and submodules
resolve natively) while file walking, imports, deletes and renames are
scoped to the subpath. Passing the subdirectory directly as the repo path
(auto-discovery) works through the same code path.

Path-containment guards (the point of the feature):
- NAV-1/NAV-2: the realpath-resolved scope must live inside the
  realpath-resolved git root — ../-traversal and symlinked subdirs
  pointing outside the repo are rejected before any git op.
- NAV-1 TOCTOU: per-file realpath re-validation during the incremental
  import drain and rename reimport; symlink-escape files are recorded as
  failures (fail-closed — the bookmark cannot advance past an escape).
- NAV-4: an --exclude set that filters out every candidate warns loudly.

Scoped syncs use git-root-relative slugs + source_path in BOTH the full
and incremental paths (runImport gains slugRoot), fixing the original
PR's full/incremental slug divergence in the auto-discovery flow.
--exclude matches scope-relative paths in both paths; exclusion never
deletes previously-imported pages. The full-sync delete reconcile is
scope-restricted and relativizes against the slug base so a healthy
scoped source can't trip the #2828 mass-delete valve. .gitignore
management resolves to the git root at every call site.

Preserves all master-side sync work since the original branch: the #2828
mass-delete safety valve, #1794 resumable checkpoints + pinned targets,
#1950 stall watchdog, #2335 heartbeat bump, #1970 bookmark reachability,
and the git-ls-files walker (#2315/#2462/#2678, whose symlink/cycle
hardening is untouched).

Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: listEverCommittedPaths uses gitContextRoot after #753 root-triple refactor

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:43:09 -07:00
f8d11f67a3 feat(search): configurable FTS language + reindex command (lands #580/#581/#582) (#2941)
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:

- Migration renumbered v116 -> v123 (master's v116 was already claimed by
  code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
  REPLACE trigger-function bodies (migration handler + reindex command)
  now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
  resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
  reindex_search_vector.pages/.chunks), id-keyset batched backfill
  (5000 rows/UPDATE) instead of single whole-table statements, and
  --json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
  written (README referenced it but no PR added it); llms bundles
  regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
  check-test-isolation).

Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:23:47 -07:00
93cfb37540 feat(engine): opt-in Postgres RLS source-scope binding (lands #2387) (#2940)
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.

With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.

Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
  (the #1794 PgBouncer pool-exhaustion class). Only the three search
  methods keep a transaction when off — exactly the sql.begin() +
  SET LOCAL statement_timeout wrap they already had on master — via the
  helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
  setseed() + SELECT to one connection when seeded (alwaysTransaction
  gated on opts.seed), so the deterministic path can't split across
  pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
  (search methods route through withScopedReadTransaction with
  alwaysTransaction; helper owns the sql.begin(); flag-off path is
  callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
  pass-through, flag-off alwaysTransaction, flag-on set_config emission,
  federated > scalar > '*' precedence, and the CSV as a bound parameter
  (never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
  ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
  SECURITY for owner roles, and the honest caveat about unwrapped paths.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:15:49 -07:00
9fe4628d02 feat(engine): opt-in Postgres RLS source-scope binding (lands #2387) (#2940)
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.

With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.

Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
  (the #1794 PgBouncer pool-exhaustion class). Only the three search
  methods keep a transaction when off — exactly the sql.begin() +
  SET LOCAL statement_timeout wrap they already had on master — via the
  helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
  setseed() + SELECT to one connection when seeded (alwaysTransaction
  gated on opts.seed), so the deterministic path can't split across
  pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
  (search methods route through withScopedReadTransaction with
  alwaysTransaction; helper owns the sql.begin(); flag-off path is
  callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
  pass-through, flag-off alwaysTransaction, flag-on set_config emission,
  federated > scalar > '*' precedence, and the CSV as a bound parameter
  (never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
  ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
  SECURITY for owner roles, and the honest caveat about unwrapped paths.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 15:15:10 -07:00
1833d95896 fix(dream,chronicle): synthesize/concepts output family — source scope, output root, retrieval reach, durable provenance, honest judge failures (#1586 #2415 #2163 #2569 #2606) (#2939)
Five verified-open fixes to the dream/synthesize output family:

- #1586: thread the cycle's resolved sourceId (cycleSourceId) through
  runPhaseSynthesize -> SubagentHandlerData.source_id -> subagent tool
  OperationContext, and stamp collected refs + summary page with the same
  source, so synthesized pages stop landing in 'default'.
- #2415: new config knob dream.synthesize.output_root (default 'wiki',
  zero behavior change unless set) drives the synthesize prompt slug
  templates, the patterns reflection lookup + prompt, and remaps the
  filing-rule allow-list globs. Registered in KNOWN_CONFIG_KEYS.
- #2163: synthesize_concepts writes concept pages through
  importFromContent (put_page's parse->chunk->embed pipeline) instead of
  bare engine.putPage, so concepts/ pages are chunked + embedded and
  reachable by retrieval.
- #2569: stampDreamProvenance persists dream_generated + dream_cycle_date
  into pages.frontmatter (JSONB merge via executeRawJsonb) at write time,
  so generated pages are DB-queryable and put_page write-through can't
  erase the marker.
- #2606: chronicle judge detects stopReason 'length' truncation and
  no-JSON-array parse failures as distinct skipped reasons
  (judge_truncated / judge_parse_failed) instead of a false terminal
  no_events; output cap raised to 4000 and configurable via
  chronicle.judge_max_tokens.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
2026-07-17 15:01:27 -07:00
42375bded5 fix(sync): stop the sync data-loss family — ops/ prune, DB-only write-through, full-sync gate drift (#2404, #2426, #2607) (#2938)
Three verified-open defects, one family: sync silently destroying or
diverging on content it should preserve.

#2404 (P0) — 'ops' was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era
carve-out), so any path with an ops segment was 'pruned-dir': committed
ops/*.md never imported, and modified ops/* files hit the
unsyncableModified delete loop (whose #1433 guard only spared
'metafile'), silently deleting put-created pages like the bundled
daily-task-manager's canonical ops/tasks on every sync. Fix: remove
'ops' from the prune list (ordinary user content; the vendor/generated
entries stay), and harden the delete loop to also skip 'pruned-dir' —
a page under a pruned dir can only exist via a deliberate put_page.

#2426 (P0) — write-through content stayed DB-only and was deleted by
sync --full. All three compounding bugs fixed:
 1. writePageThrough now best-effort commits the artifact (path-limited
    git commit) on durability-hardened repos, so the post-commit hook
    can push it; result carries committed?: boolean.
 2. scripts/brain-commit-push.sh stages+commits BEFORE any pull — the
    old fetch+pull-rebase-first order aborted on any dirty tree, so the
    helper could never commit a MODIFIED page; brain_push's
    rebase-on-reject already handles an advanced remote.
 3. The full-sync delete-reconcile partitions stale pages by git
    history (listEverCommittedPaths): never-committed source_paths are
    DB-only write-through — pages are KEPT and re-exported to the
    working tree instead of soft-deleted. Builds on the #2828
    mass-delete valve (covers the below-valve cases).

#2607 — the sync --full git ls-files fast path bypassed pruneDir, so a
full pass imported (and resurrected soft-deleted) pages under dot-dirs
and vendored trees that incremental sync excludes. Fix:
isCollectibleForWalker applies the same segment-level pruneDir gate as
classifySync, so full and incremental enumeration agree.

One regression test per defect (all verified failing against master
src): test/sync-ops-pages.serial.test.ts,
test/write-through-commit.serial.test.ts, the #2426 helper-order test
in test/brain-durability-hook.serial.test.ts,
test/sync-reconcile-db-only.serial.test.ts,
test/import-git-fastpath-prune.test.ts.

Fixes #2404
Fixes #2426
Fixes #2607

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:41:25 -07:00
Konradopenclaw a8e6b1d177 feat(ai): add Moonshot Kimi provider recipe (#2378) 2026-07-17 14:31:59 -07:00
54a8070640 fix(facts): make dream extract_facts idempotent so fence rows don't duplicate each cycle (#2932)
Port of #1837 (mvanhorn) onto current master. The extract_facts cycle
phase unconditionally wipe-and-reinserted every page's fence-owned DB
rows, so re-running a cycle on unchanged content churned rows and — on
the Postgres engine reported in #1781 — accumulated duplicates each run.

The phase now de-dupes extracted facts by the canonical (claim, source)
content key and reconciles the page-scoped DB index: no-op when already
in sync, insert-only for new keys, wipe/reinsert only when stale rows
need cleanup.

Adjustments over the original PR to fit current master:
- preserve #1972's abortSignal threading into the batch embed call
- preserve #1928's excludeSourcePrefixes: ['cli:'] on every wipe, and
  exclude cli:-origin conversation facts from the existing-row set so
  they neither count as stale (which would force a wipe every cycle)
  nor get compared against the fence

Fixes #1781

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:29:07 -07:00
e1cefd0654 fix(subagent): orchestration fix-wave G — configurable timeouts/caps, honest child outcomes, fenced timeline writes (#2937)
Four verified-open issues in the subagent-orchestration family, one PR:

- #1594: dream synthesize subagent job/wait timeouts promoted from
  hardcoded 30/35-min constants to config keys
  dream.synthesize.subagent_timeout_ms / subagent_wait_timeout_ms.
  Approach ported from stale PR #1596 (credit @ai920wisco).
- #2778: add_timeline_entry joins the subagent brain-tool allowlist,
  fenced fail-closed by the shared enforceSubagentSlugFence (extracted
  from put_page's inline check — same trusted-workspace allow-list /
  wiki/agents/<id>/ namespace policy). The per-turn output cap is now
  resolveMaxOutputTokens (data.max_tokens → agent.max_output_tokens →
  8192, was hardcoded 4096); a max_tokens stop surfaces as
  stop_reason 'max_tokens' instead of a silent end_turn, and a
  mid-tool-round cap hit injects a truncation note so the model
  re-issues the dropped call.
- #2782: patterns phase status now reflects the child outcome —
  non-complete outcome with zero writes → fail (PATTERNS_CHILD_<OUTCOME>),
  partial writes → warn. Patterns timeouts get the same config-key pair
  (dream.patterns.subagent_timeout_ms / subagent_wait_timeout_ms).
- #2113: facts extraction cap is config facts.extraction_max_tokens
  (default 4000, was hardcoded 1500); stopReason 'length' is checked,
  retried once at 2x the cap, and surfaced on stderr instead of
  silently extracting zero facts.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ai920wisco <ai920wisco@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:25:48 -07:00
a0ef951586 fix(dream): gate patterns phase on gateway provider reachability, not ANTHROPIC_API_KEY (takeover of #2279) (#2936)
Absorbs PR #2279's intent (drop the hardcoded ANTHROPIC_API_KEY gate) with
the end-state the gateway world actually wants: the patterns phase now
probes the RESOLVED patterns model through probeChatModel(normalizeModelId)
— the same semantics as think/index.ts and synthesize's makeJudgeClient.

Fixes two misclassifications of the old env gate:
- Non-Anthropic stacks (litellm, deepseek, openrouter, ...) were skipped as
  "no upstream" even though the subagent routes them through the gateway
  (agent.use_gateway_loop). They now pass the gate; their auth is checked
  lazily at dispatch and surfaces in the job outcome.
- Anthropic keys set via `gbrain config set anthropic_api_key` (stdio MCP
  launches without shell env) were treated as missing. hasAnthropicKey
  inside probeChatModel reads both sources.

Skip reason renames no_api_key → no_provider (carrying the probe's detail).
Both pinning tests updated: the structural test asserts the probe wiring;
the PGLite E2E swaps its env-only helper for the shared hermetic
withoutAnthropicKey (env + config file) so it can't flip to a live LLM call
on a dev machine with a config-file key.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:56 -07:00
bd2ba46a61 fix(retry): reconnect on null instance pool in ALL non-batch config accessors (takeover of #1891) (#2934)
Ports the still-unmerged remnant of PR #1891 (#1593 follow-up). Master's
getConfig gained retry-with-reconnect in #1603, but its siblings —
setConfig, unsetConfig, listConfigKeys — still touched `this.sql` bare, so
the first config write/list after a mid-cycle instance-pool teardown threw
the retryable "No database connection" (issue #1678) unhandled instead of
rebuilding the pool.

Adds the connRetry helper from #1891 (same retry+reconnect posture as
batchRetry, but no batch audit JSONL — a config accessor is not a sized
batch), refactors getConfig onto it, and wraps the other three. Writes are
safe to retry: withRetry only retries connection-class failures and both
writes are idempotent (upsert / delete).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: jalagrange <jalagrange@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:52 -07:00
2df41a84c9 feat(ai-gateway): derive OpenAI prompt_cache_key for native-OpenAI chat models (takeover of #2442) (#2933)
Ports the still-unmerged half of PR #2442. OpenAI caches prompt prefixes
automatically, but a stable prompt_cache_key keeps requests that share a
prefix on the same inference engine, lifting the automatic-cache hit rate.
chat() now derives a stable key from the system prompt + sorted tool names
for native-openai models and passes it via providerOptions.openai.
promptCacheKey. Config provider_chat_options still overrides the derived
key; anthropic/google/openai-compatible providers are untouched.

The Anthropic half of #2442 (cache_control "silent no-op") is superseded:
@ai-sdk/anthropic 3.0.74 forwards call-level providerOptions.anthropic.
cacheControl as the Messages API's request-level cache_control (automatic
prefix caching), so master's existing marker is live on current deps.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: CoachRyanNguyen <CoachRyanNguyen@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:18:48 -07:00
da1bab532a fix(import): make checkpoints staging-first — canonical dir identity + self-describing metadata (#2935)
Port of #1731 (diazMelgarejo) onto current master. gbrain import wrote
~/.gbrain/import-checkpoint.json with the caller's raw dir argument, so
a checkpoint left behind by an interrupted run (e.g. SIGTERM) could carry
"." or a symlinked spelling — an identity that resolves to whatever CWD
the next consumer happens to run from. Downstream tooling that treated
the checkpoint dir as an owned staging boundary could then act on the
wrong directory.

- runImport captures the import target ONCE via resolveImportTargetDir
  (resolve + realpathSync) and threads that canonical value through
  collection, checkpoint load/save, and resume filtering
- checkpoints are self-describing (schema_version: 1, owner: "gbrain",
  kind: "import"); loadCheckpoint tolerates absent metadata (legacy
  path-based files) but rejects present-and-wrong metadata and any
  relative dir
- checkpoint contract documented in docs/guides/live-sync.md (llms
  bundle regenerated)
- test/import-resume.test.ts fixture now realpaths its tmpdir so planted
  checkpoints match the canonicalized dir (macOS /var -> /private/var)

Fixes #1728

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Lawrence Melgarejo <Lawrence@cyre.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 14:17:37 -07:00
3aeb622dc7 v0.42.62.0 chore(release): thirty verified fixes — changelog + version bump (#2924)
Source-provenance wave, reconnect resilience, SSE proxy fix, rolling
prompt-cache, security CI, three consolidated fix-waves, and twenty more
individually verified community fixes.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:48:18 -07:00
3253fb824c fix(deps): resolve all OSV-flagged dependency vulnerabilities (same-major bumps) (#2927)
* fix(deps): resolve all 34 OSV-flagged vulnerabilities with same-major patch bumps

Direct deps: js-yaml ^3.15.0, marked ^18.0.2 (resolves 18.0.6),
admin vite ^6.4.3. Transitive deps pinned via overrides at their
minimum fixed versions (same major, no promotion to direct):
@hono/node-server, fast-uri, fast-xml-builder, fast-xml-parser,
form-data, hono, ip-address, qs; admin: @babel/core, postcss.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(deps): force js-yaml >=3.15.0 for all resolutions via override

A transitive consumer held a second js-yaml@3.14.2 resolution the
direct-dep range bump did not move.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:38:05 -07:00
b075a9c8d4 test+ci: unbreak master — symlink-walker probe files + OSV caller permissions (#2926)
* test: symlink-walker tests use a non-metafile probe (README now skipped by design, #2315)

The import walker deliberately skips README/metafiles since #2315 (closing
#345); the symlink-hardening tests used README.md as their probe file and
went red on the intersection. Probe with notes.md instead; test intent
(cycle hardening + strategy filter) unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: grant security-events write to the OSV caller job — reusable workflow requires it at startup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 12:10:47 -07:00
Brett a31f16f471 fix(markdown): treat # lines inside closed frontmatter as YAML comments, not headings (#2153)
`parseMarkdown` previously walked the lines after the opening `---` and
recorded the first `^#{1,6}\s`-shaped line as a `headingBeforeClose`,
then flagged MISSING_CLOSE when that index came before the actual closing
fence. YAML allows `#` comment lines anywhere inside the document, so a
template that leads with annotation comments inside the fence (e.g. a
`# Research Template` header before the keys) hit a false-positive
MISSING_CLOSE even though the closing `---` was present.

Fix: only walk for the closing `---`. When it is found, content between
the fences is YAML; `#` lines are comments, not headings. When the close
is genuinely missing, surface the first heading-shaped line as a
where-it-went-off-the-rails hint (this path was already correct; we keep
it for the genuine missing-close case).

Two regression tests added to `test/markdown-validation.test.ts`:
- `#` comment lines at the top of a closed frontmatter
- `#` comment lines interleaved with keys

All 68 tests across the four markdown/frontmatter test files stay green.
2026-07-17 11:39:15 -07:00
7ffac65c62 fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522 #1747 #1503) (#2920)
* fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522 #1747 #1503)

Three fixes in the same invariant class (source identity silently dropped
on the write path, collapsing to the 'default' source):

- #1522: the ingest_capture Minion handler validated IngestionEvent
  provenance (source_id/source_kind/source_uri) then dropped it on the
  importFromContent call. Now threads source_kind/source_uri +
  ingested_via='ingest_capture' into the page write, and routes the write
  under event.source_id when it names a registered source AND the event
  is trusted (fail-closed: untrusted webhook payloads carry a
  caller-controlled x-gbrain-source-id header and must not choose their
  write source; unregistered emitter ids keep default routing so the
  webhook path can't FK-fail).

- #1747: the fs-walk extractors (extractLinksFromDir /
  extractTimelineFromDir / extractForSlugs) built batch rows with no
  source_id, so addLinksBatch/addTimelineEntriesBatch mapped missing →
  'default' and the pages JOIN dropped every row on a non-default source
  ("Links: created 0 from N pages", no error). ExtractOpts gains
  sourceId; the CLI fs path resolves it via resolveSourceId
  (--source-id > env > dotfile > registered path > sole-non-default)
  and rows are stamped from/to/origin_source_id + timeline source_id.

- #1503: the cycle's extract phase (runPhaseExtract) never passed a
  sourceId, so federated-brain dream/autopilot cycles persisted nothing
  every night. It now threads cycleSourceId (explicit --source or
  resolveSourceForDir(brainDir) — the same seam runPhaseSync uses) into
  runExtractCore for both the incremental and full-walk paths.

Regression tests: handler provenance write-through (registered/
unregistered/untrusted routing), fs-walk + CLI resolution + incremental
cycle route landing edges/timeline in the right source (all red on
unfixed master), and a negative control pinning the pre-fix JOIN-drop
shape.

Reuses the ExtractOpts.sourceId threading approach from PR #1719,
rebased onto the current signal-aware signatures and extended to the
cycle + timeline paths.

Fixes #1522
Fixes #1747
Fixes #1503

Co-authored-by: seungsu <kss530c@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: update cycle source pin for cycleSourceId threading

The #1972 source-pin test asserts the literal runPhaseExtract call site in
cycle.ts. The #1503 fix appended cycleSourceId after opts.signal; signal
threading is unchanged (still the 5th arg, forwarded to runExtractCore).
Pin updated to the new literal — invariant intact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: seungsu <kss530c@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:36:43 -07:00
maxpetrusenkoagent b263d9bc20 fix(minions): reconnect worker after promote connection loss (#2025)
Recover the worker-owned Postgres pool when promoteDelayed escapes a retryable connection error, preventing the repeated Promotion error: No database connection loop from issue #1491.\n\nAdds a regression test proving reconnect happens before the worker continues to claim work.
2026-07-17 11:33:04 -07:00
kubi 73bbbde01d fix frontmatter scans to respect git excludes (#2462) 2026-07-17 11:32:48 -07:00
ff8ce4d764 fix(import): walker skips SYNC_SKIP_FILES metafiles so import and sync agree (#2315)
Closes #345.

The bulk-import walker isCollectibleForWalker filtered admitted files by
extension only, while incremental sync excludes README/index/log/schema via
isSyncable -> SYNC_SKIP_FILES. A directory import therefore ingested every
directory README as a folder-titled ghost page that trigram-corrupts fuzzy
entity resolution and inflates orphan count. Apply SYNC_SKIP_FILES (basename
guard) at the top of isCollectibleForWalker so both the FS-walk and the
git-fast-path collection routes agree with sync.

Also add RESOLVER.md to SYNC_SKIP_FILES: a structural routing metafile
(docs-aligned with schema.md/index.md/log.md/README.md), not indexable content.

Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:32:42 -07:00
ad1fe25e61 fix(sources): audit walker inverted pruneDir — nested sources scanned 0 files (#2678)
The audit walk in `gbrain sources audit` used `if (pruneDir(entry, dir)) continue;`
but pruneDir() returns true = descend, false = prune (core/sync.ts). The
inversion made the walker skip every legitimate subdirectory (any source with
nested content reports 'Files scanned: 0 markdown files') while descending
into exactly the trees it should skip (node_modules/, .git/, vendor/, .raw/).

The walker's own comment says 'Mirror gbrain sync's descent rules' — this
makes it actually do so.

Repro on a real brain (v0.42.56/57): a comms source with per-contact
subdirectories audits 0 files; a flat source audits correctly.

Co-authored-by: Idrees Kamal <idreeskamal@MacBook-Air.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:32:37 -07:00
Masa c4ff8b63a8 fix(minions): restore rolling conversation prompt-cache on the direct SDK path (#2771)
* fix(minions): restore rolling conversation prompt-cache on the direct SDK path

Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.

Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.

Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).

Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.

Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.

* fix(minions): normalize seed message content before caching it

Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.

Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.

* fix(minions): retain the prior rolling cache breakpoint, not just the newest

Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.

Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
2026-07-17 11:32:31 -07:00
Ryan AyersandClaude Opus 4.8 9eac872136 fix(postgres-engine): build-then-swap reconnect() so a failed rebuild can't brick the engine (#1593) (#1906)
The instance-pool reconnect() did disconnect() (nulling _sql) BEFORE connect(),
so a connect() failure during a transient Postgres blip left _sql null for the
rest of the process — every subsequent non-retry-wrapped call then fell through
to the never-connected module singleton and threw 'No database connection',
crashing the autopilot worker into a respawn loop. Build-then-swap: snapshot the
live pool, build a fresh one, end the old only once the new validates, restore
on failure. Keeps upstream's reap-detection + pool-recovery audit. Confirmed by
jalagrange on closed PR #1593; this is the Layer-1 fix he left to us.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 11:32:23 -07:00
xd-Neji cfc120fcb3 fix(stats): exclude soft-deleted pages from visible counts (#2235) 2026-07-17 11:32:17 -07:00
Mersad Ajanovic 0a021f6f6b fix: send admin SSE cookies through reverse proxies (#1560) 2026-07-17 11:32:12 -07:00
mzkarami cd9bd3f731 fix(auth): add register-client agent binding flags (#1976) 2026-07-17 11:32:06 -07:00
Brett 00523b8412 feat(ai/recipes/litellm): declare chat + expansion touchpoints (#2208)
The litellm recipe shipped only an `embedding` touchpoint. `getProviderCapabilities()` in `src/core/ai/capabilities.ts` throws when `recipe.touchpoints.chat` is missing, `classifyCapabilities()` returns `'unknown'`, and `enforceSubagentCapable()` in `src/core/model-config.ts` silently falls back to `TIER_DEFAULTS.subagent` (anthropic). Any brain that routes paid traffic through a litellm-style proxy AND has no `ANTHROPIC_API_KEY` then sees every subagent loop dispatch throw `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`. The user's explicit `models.tier.subagent = litellm:*` choice is overridden without their knowledge.

Declare `chat` and `expansion` touchpoints mirroring the openai recipe's shape: `models: []` (litellm proxies arbitrary backends; allowlist is intentionally empty, since `assertTouchpoint` already skips allowlist checks for `tier: 'openai-compat'`), `supports_tools: true`, `supports_subagent_loop: true`, `supports_prompt_cache: false` (OpenAI-compat backends don't honor Anthropic-style `cache_control`), `max_context_tokens: 200_000` (conservative GPT-5-family default; per-deployment override needed for smaller-context backends), costs `undefined` (varies by proxied provider).

Reproduction (deterministic):

1. Fresh brain with no `ANTHROPIC_API_KEY` in env.
2. `gbrain config set models.tier.subagent litellm:gpt-5.4` (or any `litellm:*` string).
3. `gbrain models` warns and falls back to `anthropic:claude-sonnet-4-6`.
4. Submit any subagent job; throws `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`.

After the patch, `classifyCapabilities('litellm:gpt-5.4', recipe)` returns `degraded:no_caching` (chat-capable, no Anthropic-style prompt cache). `enforceSubagentCapable` no longer steals the model choice. The subagent loop emits a one-time `degraded:no_caching` warn about prompt-cache absence; cost scales linearly with conversation length, accepted trade for the proxy path.
2026-07-17 11:32:01 -07:00
lost9999andlost9999 3a2033e8e2 v0.42.52.0 fix(sync): bump last_sync_at heartbeat on 0-changes sync (#2335)
D4 invariant ("never advance last_commit on partial", sync.ts comment)
preserved. last_sync_at is a monitoring signal read by doctor
sync_freshness (warn 24h / fail 72h), separate from the import-converged
bookmark. Without this heartbeat write, a cron-driven */15 sync over a
quiet vault pins last_sync_at to the last real commit, so doctor
falsely flags the source as stale for as long as the vault is quiet.

Reproduction (5 lines, no gbrain install required):
  1. Setup a fresh obsidian source + commit a single .md file
  2. gbrain sync --source obsidian   # first_sync, last_sync_at = NOW
  3. (do nothing) gbrain sync --source obsidian   # up_to_date
  4. SELECT last_sync_at FROM sources WHERE id = 'obsidian';
  5. Observed: still pinned to step 2. Expected: bumped to step 3.

Fix: in the up_to_date early-return (sync.ts line ~1786), execute a
single `UPDATE sources SET last_sync_at = now() WHERE id = $1` before
returning. The D4-protected writeSyncAnchor path is untouched.

Test: test/sync.test.ts adds a describe block that runs two
consecutive syncs against a quiet vault and asserts last_sync_at
advances while last_commit is unchanged. PGLite + executeRaw pattern
matches the existing #1970 test scaffold.

Workaround: hourly psql touch in WSL crontab documented at
https://github.com/garrytan/gbrain/issues/[link-to-issue]

Co-authored-by: lost9999 <lost9999@users.noreply.github.com>
2026-07-17 11:31:55 -07:00
supportswiftandjaxlewis-swift 74bc8f8cd1 fix(sync): crash-safe renames loop — record per-file failures instead of throwing (#2402)
The renames loop reimports each renamed file via importFile() but, unlike the
deletes and adds/mods loops, does not wrap the call. importFile() still throws
on content sanity-block, duplicate-slug, and missing-link endpoints, so a single
malformed renamed file throws uncaught and crashes the whole sync mid-run —
freezing the checkpoint and defeating --skip-failed. A 'skipped' result carrying
an error was also silently dropped (never recorded to failedFiles).

This wraps the reimport in try/catch and records both the throw and the
skipped-with-error case to failedFiles, matching the existing deletes/adds loop
pattern. Surfaced in the wild by a tree-wide rename (a shared/ -> system/ vault
migration, ~10.8k renames) where one malformed-YAML renamed file crashed the
entire incremental sync at rename ~4900/10857.

Co-authored-by: jaxlewis-swift <jaxlewis@swiftsolutions.ai>
2026-07-17 11:31:49 -07:00
78bc2fef09 fix(cli,config,doctor): CLI/config UX wave — config-get file plane, idempotent archive, honest help + doctor text (#2120 #2792 #1175 #1123 #2451) (#2918)
* fix(cli,config,doctor): CLI/config UX wave — config-get file plane, idempotent archive, honest help + doctor text, prefixed model defaults (#2120 #2792 #1175 #1123 #2451)

- config get resolves the file/env plane before the DB plane (runtime
  precedence) and reports provenance on stderr; stdout stays a bare value.
- sources archive distinguishes already-archived (friendly no-op, exit 0)
  from not-found (clear exit-4 error).
- gbrain --help SOURCES block now lists archive/restore/archived/purge/status
  plus a pointer at `sources --help` for the long tail.
- multi_source_drift doctor advice references only real CLI surfaces
  (drops the never-built 'sources rehome'; pins delete to GBRAIN_SOURCE=default).
- #2451 (bare model ids in calibration defaults) verified already fixed +
  tested on master by #2892 — no change needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: satisfy test-isolation gate for wave-B tests

check-test-isolation R1 forbids direct process.env mutation in non-serial
unit tests (env leaks across files sharing a shard process). Route the
GBRAIN_HOME / GBRAIN_CHAT_MODEL / GBRAIN_PGLITE_SNAPSHOT overrides through
the canonical withEnv() helper instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:27:34 -07:00
9aaa3be05f ci(security): OSV dependency scan, release artifact attestations, Semgrep CE SAST (#2182 #2142 #2272) (#2917)
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 11:05:13 -07:00
26d2f8abfc fix(calibration,takes,cli): calibration CLI routing, source-scoped takes reads, BigInt-safe outputs (takeover of #2452) (#2892)
Rebase-port of #2452 (spinsirr:fix/calibration-profile-scope-and-cli) onto
current master after tonight's merges made the fork branch conflict.

- cli: add 'calibration' to CLI_ONLY so dispatch reaches its existing
  handler instead of falling through to "Unknown command" (#2035); honor
  --source / GBRAIN_SOURCE in the calibration CLI.
- takes: route takes_list / takes_search / takes_scorecard /
  takes_calibration through sourceScopeOpts(ctx) (federated array > scalar
  > nothing) and scope engine reads via the take's page.source_id — JOIN
  filter for list/search, EXISTS for scorecard/curve — on both engines
  (#2200-class).
- bigint: shared takeHitRowToHit coercion in searchTakes /
  searchTakesVector (both engines) + bigintToStringReplacer on the cli.ts
  output normalizer and the `gbrain call` exit, so int8/BIGSERIAL columns
  no longer crash JSON.stringify (#2450); calibration profile id
  BIGSERIAL → string.
- calibration: default model ids route through TIER_DEFAULTS
  (provider-prefixed) instead of bare model strings; admin calibration
  chart endpoints fixed (takes has no page_slug column; month-precision
  since_date; Date generated_at; bigint id in drill-down).

The think/gather source-scope slice of the original PR was dropped: it
already landed on master via #2739.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: spinsirr <ID+spinsirr@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 21:20:12 -07:00
2166545849 fix(pglite): platform-gate the init-failure banner — stop blaming the macOS 26.3 bug on every platform (#2674) (#2891)
classifyPgliteInitError() routes bare Emscripten aborts to the 'unknown'
verdict, whose hint unconditionally printed "Most common cause: the macOS
26.3 WASM bug (#223)" — even on Windows and Linux (#2195, #1870).

- buildPgliteInitErrorMessage now takes a platform param (default
  process.platform): darwin keeps the #223 link as a *possible* cause;
  other platforms get the plausible off-macOS causes (lock contention,
  damaged data dir) plus `gbrain doctor` / `gbrain reinit-pglite`.
- New stringifyPgliteInitError(): non-Error rejections (Emscripten
  aborts can throw plain objects) no longer print "[object Object]".
- Regression tests for both branches + the stringifier in
  test/pglite-init-classifier.test.ts.

Canonical for a 7-report class: #2674, #1870, #1195, #1502, #2195,
#939, #391.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 21:20:10 -07:00
10ad7f156a chore(docs): regenerate llms bundle after #1671 docs merge (#2893)
The README/INSTALL updates from #1671 landed without the bundle regen the
freshness gate requires; every branch cut from master since inherits the
failure.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 21:09:49 -07:00
Sanjay Santhanam 1229bec1eb fix(postinstall): cross-platform node shim instead of POSIX shell (#1554)
* fix(postinstall): cross-platform node shim instead of POSIX shell

The postinstall script used POSIX shell syntax ('command -v',
'>/dev/null 2>&1', '1>&2') which Bun's built-in script parser rejects
on Windows. 'bun install' aborted with 'expected a command or
assignment but got: "Redirect"' before gbrain could ever be probed.

Replace with a one-liner 'node -e' shim that:

* uses spawnSync to probe 'gbrain --version' (shell:true on win32 so
  the Windows shim/.cmd resolution works)
* on success: runs 'gbrain apply-migrations --yes --non-interactive'
  and propagates its exit code
* on failure: writes the same skip message to stderr and exits 0, so
  fresh clones (where gbrain isn't on PATH yet) still complete install

POSIX hosts retain the original behavior; Windows hosts now succeed
instead of failing the whole install.

Fixes #1486

* fix(postinstall): move logic to scripts/postinstall.ts to survive Bun Windows script-runner

The `node -e` inline shim still failed on Windows under Bun. Embedding a
program inside the package.json postinstall string lets the lifecycle shell
mangle it: Bun's Windows script-runner expands the `\n` in the hint string
into a REAL newline before node sees it, producing `SyntaxError: Invalid or
unexpected token` and aborting the whole install. `node` is also not
guaranteed present under a Bun install (bun is the guaranteed runtime), and
`shell: win32` re-opened a quoting surface.

Move the logic into a checked-in `scripts/postinstall.ts` run via
`bun run scripts/postinstall.ts`, matching the repo's existing convention of
~19 scripts under scripts/*.ts. This sidesteps all three failure modes:

* `which('gbrain')` from bun does Windows-aware PATH resolution (finds
  gbrain / gbrain.exe / gbrain.cmd) with no shell.
* `Bun.spawnSync` with an argv array invokes apply-migrations directly —
  no shell, nothing to quote, no `\n` expansion.
* No dependency on `node` being present; bun runs the script.

Behavior is preserved exactly: same `apply-migrations --yes
--non-interactive` command, same issue-218 skip hint, and the same
never-fail-the-install guarantee (every path exits 0). Verified on macOS:
`bun run scripts/postinstall.ts` exits 0 on the skip path (gbrain absent),
on a failing migration, and on a successful migration.

Fixes #1486
2026-07-16 20:51:08 -07:00
abyss-nodeandClaude Fable 5 42f3960ba5 fix(serve): enable parent-death watchdog on Windows via signal-0 liveness probe (#2049)
The stdio parent-death watchdog was hard-wired to spawnSync('ps'), which
does not exist on Windows. The startup probe therefore failed on every
Windows host, the watchdog was permanently disabled ("watchdog disabled:
ps unavailable"), and an orphaned `gbrain serve` held the PGLite write
lock until reboot. Orphans are especially easy to produce on Windows
because MCP hosts launch the server through a cmd.exe wrapper (.bat) and
killing the wrapper does not kill the bun child.

Windows never re-parents orphans, so the cached process.ppid stays
correct for the process lifetime and the watchdog question inverts from
"did the live PPID change?" to "is the original parent still alive?" --
answered in-process with a signal-0 existence probe (process.kill(ppid, 0),
OpenProcess under the hood). No external binary needed. EPERM counts as
alive. Parent dead reports PID 0, which differs from initialParentPid and
fires the existing shutdown path.

- readLiveParentPid / probeWatchdogAvailable: platform split (ps on
  POSIX, signal-0 on win32), exported with a platform test seam so CI on
  any OS exercises both branches.
- isPidAlive: shared exported helper.
- Watchdog install guard tightened from `!== 1` to `> 1` so a PID-0
  "parent already gone" report cannot install a phantom interval that
  compares 0 to 0 forever.
- Disabled-mode log generalized (no longer claims ps is the only
  mechanism); existing probe-fail test updated to match.
- New unit suite for the platform defaults (6 tests).

Verified live on Windows 11: serve spawned via a .bat wrapper, wrapper
killed without closing stdin -> bun child exits within ~10s and the
PGLite lock dir is released. Before this change the child survived
indefinitely and every subsequent gbrain invocation timed out waiting
for the lock.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:49:34 -07:00
duncanclaw e78f8a1590 fix(doctor): use active engine for PGLite probes (#1183) 2026-07-16 20:49:31 -07:00
Eric Loes 79d8c6773e fix(doctor): treat disabled retrieval reflex as intentional (#2459) 2026-07-16 20:49:28 -07:00
spiky02plateau 06f58c2b32 feat(gateway): config-driven provider_chat_options passthrough (fixes #2577) (#2857)
Add provider_chat_options alongside provider_base_urls and thread it through
the gateway config path into chat(). The chat request now deep-merges
provider-scoped options and model-scoped overrides into providerOptions keyed
by recipe id, preserving existing gateway-built options such as Anthropic
cacheControl and leaving absent-config behavior unchanged.

This lets operators disable thinking for small-budget hybrid-reasoning utility
calls without hardcoding that behavior for every use of those models.
2026-07-16 20:48:57 -07:00
roysauravandSaurav Roy fe6838ffac docs: add macOS 26.x Tahoe PGLite WASM workaround + native Postgres setup guide (#1671)
PGLite's embedded WASM engine crashes on macOS 26.x (Tahoe) on Apple
Silicon during engine initialization. This adds:

- A Troubleshooting section in docs/INSTALL.md with step-by-step
  instructions for using native Homebrew PostgreSQL 17 + pgvector
  as a workaround
- A callout in README.md's Troubleshooting section pointing users
  to the detailed setup guide

Tested on macOS 26.5 (arm64), Bun 1.3.14, gbrain 0.41.29.0,
PostgreSQL 17.10 (Homebrew), pgvector 0.8.0. All 102 schema
migrations pass. gbrain doctor green.

Co-authored-by: Saurav Roy <roysaurav@users.noreply.github.com>
2026-07-16 20:48:54 -07:00
6abab9d584 fix(facts): durable facts-absorb jobs for one-shot CLI processes + source-scoped fence paths (#2104)
* fix(facts): durable facts-absorb jobs for one-shot CLI processes

Every gbrain capture/put from a short-lived CLI enqueued the facts:absorb
chat into the in-process FactsQueue, then the exit teardown drained for
1-2s and aborted the in-flight call — logging 'pipeline_error: [chat(...)]
The operation was aborted.' on every eligible CLI page write and never
extracting facts.

cli.ts now marks one-shot processes (everything except serve/jobs/
autopilot); runFactsBackstop's queue mode submits a durable facts-absorb
minion job for the long-lived jobs worker instead, with content-hash
idempotency and fallback to the in-process queue if submission fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(facts): fence-write resolves source-scoped page path

writeFactsToFence joined local_path + slug directly, writing main-source
fences to the repo ROOT (the default source's tree) and polluting ~/brain
with stray root-level fence files. Route through resolvePageFilePath —
the same helper the put_page write-through and dream-cycle reverse-render
use — so non-default sources fence into .sources/<id>/<slug>.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Ragnar Åström <reghar@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:47:47 -07:00
Ziyang Guo e0ca74200a fix(timeline): expose date window filters (#2694) 2026-07-16 20:47:44 -07:00
Elliot DrelandClaude Opus 4.8 9315fd0746 fix(conversation-parser): read raw_transcript sidecar + parse plain Speaker A/B lines (#1898)
The parser/doctor/extractor read the polished page body (compiled_truth +
timeline) instead of the raw turn-by-turn transcript that meeting pages store
in a `raw_transcript` frontmatter sidecar, and the brain's actual raw format
(`Speaker A: ...` / `Speaker B: ...`) had no built-in pattern. Result: scan
returned no_match / 0 messages and conversation-fact extraction produced 0
segments / 0 facts.

- new readConversationBodyForParsing(): prefer the raw_transcript sidecar when
  present, fall back to compiled_truth + timeline (src/core/conversation-parser/body.ts)
- wire it into conversation-parser scan, doctor coverage check, and
  extract-conversation-facts (drops the old readPageBody helper)
- add a built-in `speaker-letter-no-time` pattern for plain `Speaker A:` lines

Verified: scan now returns phase=regex_match (speaker-letter-no-time), and the
Ben page extracts 61 facts / 7 segments. Tests added; suite green.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 20:47:41 -07:00
Matt Van HornandMatt Van Horn d33aee843b query: filter since/until on effective date, not updated_at (#1706)
since/until range filters were applied against updated_at, so edited-but-old
entries leaked into time-bounded queries. Filter on the effective date instead.

Closes #1520

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-07-16 20:47:38 -07:00
mzkarami 414940204a ci(release): run verify before build (#2243) 2026-07-16 20:47:35 -07:00
0cf5596c88 v0.42.61.0 chore(release): ten verified community improvements — changelog + version bump (#2890)
Autopilot crash recovery, deterministic atom slugs, takes bootstrap
progression, bundled-pack activation, Sonnet 5/Fable 5 pricing, inline
citation timelines, pack-driven extraction discovery, book-mirror HTML
tables, gateway test-pin, docs sync.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:34:58 -07:00
ff2eb6ff3a docs: post-release reference-doc sync for v0.42.59.0 (#2798)
Cross-referenced the v0.42.59.0 five-fix rollup (#2735-#2739) against the
reference docs and updated every entry that no longer described current
behavior:

- KEY_FILES.md: migrate-engine.ts (source-catalog copy + target-aware resume
  manifest), pglite/postgres bootstrap probe set (timeline_entries.event_page_id),
  searchTakes/searchTakesVector source scope, think op scope threading through
  runGather via thinkSourceScopeOpts, new fence-shared.ts entry (escape-aware
  parseRowCells as escapeFenceCell's inverse).
- TESTING.md: one-liners for the three new e2e suites
  (think-source-isolation-pglite, facts-fence-reconcile-postgres,
  migrate-engine-sources-postgres) + the new multi-source-bug-class case.
- TODOS.md: new v0.42.59.0 follow-ups section (6 items); refreshed the two
  existing items the wave partially resolved (think gather scope plumbing,
  #2200 takes_search engine-layer scope).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 20:10:56 -07:00
maxpetrusenkoagent 323d7d6336 test(ai): pin gateway tool schema conversion (#2063) 2026-07-16 19:54:06 -07:00
garrytan-agentsandgarrytan-agents 9ceca6063b book-mirror: emit HTML <table valign=top> instead of markdown pipe tables (#2270)
Markdown pipe tables have no vertical-align control, so every renderer except
GitHub middle-aligns rows — unreadable when the two columns differ in length.
Switch the per-chapter prompt + frontmatter tag to the HTML <table> form with
valign=top on every cell (matches the 20 hand-built mirror pages that already
render correctly).

Co-authored-by: garrytan-agents <agents@garrytan.dev>
2026-07-16 19:53:43 -07:00
pabloglzgandpabloglzg e538051401 feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries (#2524)
* feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries

gbrain's own quality conventions (skills/conventions/quality.md) require a
dated [Source: ..., YYYY-MM-DD] citation on every brain write, so curated
pages are full of dated evidence — but extractTimelineFromContent only
recognized the timeline-bullet and date-header formats. A page whose dates
all live in citations scored zero timeline coverage in brain_score, and
doctor pointed users at a formatting convention their own citations already
satisfied in spirit.

Format 3 files one entry per citation: date and source from the marker,
summary from the annotated line with citation markers stripped. Lines
already captured by Format 1 are skipped so a timeline bullet carrying its
own citation is not double-filed. Bare citations with no surrounding text
are ignored.

Idempotency is unchanged: persistence already dedupes at the DB layer.

* fix(extract): Format 3 citations also in parseTimelineEntries (db-source + ingest path)

The first commit only taught extractTimelineFromContent (fs-source) the
citation format; the db-source extract and the ingest path parse through
parseTimelineEntries in core/link-extraction.ts, which still could not see
citations. Same rules as the fs parser: bullet-captured lines skipped,
bare citations ignored, invalid calendar dates rejected; the citation
source is preserved in the entry detail.

---------

Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com>
2026-07-16 19:53:40 -07:00
7202ebf3da fix(takes): bootstrap runs progress through the corpus instead of rescanning the newest slice (#2638)
extractTakesFromPages selected pages by updated_at DESC + LIMIT with no
exclusion of pages that already hold takes. The CLI clamps --max-pages to
1000, so on a corpus larger than one run every re-run rescanned the same
most-recent 1000: the older tail could never be bootstrapped, and each
rescan re-spent Haiku budget producing upsert-identical rows. Seen live on
a 2,311-eligible-page brain — a second run would have covered 0 new pages.

Covered pages are now skipped by default (NOT EXISTS on takes.page_id), so
repeat runs sweep a large corpus in slices; --include-covered restores the
full rescan for refresh use. Usage text documents both plus the 1000 clamp.


Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 19:53:16 -07:00
joelwpandClaude Opus 4.8 f981f70a2f fix(extract): deterministic atom slug — stop cross-day + trailing-dash duplicate atoms (#2482)
extract_atoms minted duplicate atom pages two ways:

A. Trailing-dash twins. The local slugger truncated the title at 60 chars
   with no re-strip, so a cut landing on a hyphen left a trailing dash
   (`…would-`). The FS-import normalizer (slugifySegment) strips it
   (`…would`), so the same atom persisted under two slugs and the
   dedup-by-slug check never collapsed them.

B. Cross-day re-mint. The slug used the run date (todayDate()), while the
   idempotency guard keys on the whole-file source_hash. Append-only
   sources (chat/transcript exports) grow daily, so the file hash changes,
   the guard never matches, the source is re-extracted, and each re-mint
   lands under a new date prefix → a new slug → no upsert → a duplicate.

Fix: make the atom slug a deterministic function of stable inputs —
`atoms/<source-date>/<stem>-<title-hash>`:
- source date is parsed from the source ref (transcript filename / page
  slug), not the run date, so re-extraction converges on the same slug and
  putPage upserts instead of duplicating;
- the 6-char title hash keeps two atoms whose titles share the first 60
  chars on distinct slugs (no silent clobber of a different atom);
- the stem routes through the canonical slugifySegment and re-strips a
  trailing dash, so the two write paths can no longer disagree.

The whole-file source_hash batch check is retained only as a cost fast-path
(skip re-running the model on an unchanged source); correctness no longer
depends on it.

Adds a hermetic PGLite regression test (no DATABASE_URL) asserting the
source-dated prefix, title-hash suffix, trailing-dash strip, and upsert on
re-extraction of a grown append-only transcript.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 19:53:13 -07:00
vinsew 34c0ff0c56 fix(autopilot): verify lock holder process before exiting (#477)
A lock file can outlive its autopilot process after a crash or forced termination. The previous mtime-only check treated that file as proof that another instance was running, so supervisor restarts could exit repeatedly until the file aged out.

Read the holder PID and probe it with signal 0. Keep the lock whenever that process is alive; take over only dead, malformed, empty, or self-owned locks. EPERM remains conservative and counts as alive, preventing two autopilot instances from running concurrently.

Add hermetic tests for missing, live, dead, malformed, and empty lock states.
2026-07-16 19:53:10 -07:00
e1e1f3bac2 feat(extract_atoms): honor pack manifest extractable flag in page discovery (#2615)
Atom-extraction page discovery hardcoded EXTRACTABLE_PAGE_TYPES and ignored the
active pack's `extractable: true` flags, so a type declared extractable in the
manifest (e.g. `note`) never actually extracted. Closes the D2 TODO the code
already flagged (extract-atoms.ts: 'future pack-aware refactor ... pull from the
active pack manifest').

Resolve the allowlist as: legacy hardcoded floor UNION the pack's extractable
types, MINUS synthesis outputs (atom, concept — extracting from these would loop,
since concepts are synthesized from atoms). Mirrors facts/eligibility.ts, which
excludes concept the same way. Back-compat: gbrain-base brains keep every legacy
target via the union; fail-soft falls back to the legacy floor if the pack can't
load.

- pure unionExtractableTypes() policy, unit-tested
- discoverExtractablePages + countExtractAtomsBacklog resolve from the pack
- page-discovery fixture updated (note now extracts; concept stays excluded)

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 19:52:57 -07:00
Matt Van HornandMatt Van Horn a12db46350 schema: resolve all bundled packs in schema use, not just gbrain-base (#1707)
`schema use` hardcoded `gbrain-base` (and a fixed bundled list), so other
bundled packs could not be selected by name. Use the shared BUNDLED_PACK_NAMES
set and resolve `<name>.yaml` generically so every bundled pack activates.

Closes #1574

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-07-16 19:52:54 -07:00
9f313db374 fix(pricing): add Sonnet 5 and Fable 5 to the canonical chat-pricing table (#2799)
claude-sonnet-5 and claude-fable-5 are GA Anthropic models, but neither
was in CANONICAL_PRICING. A brain routing a tier to them (e.g.
models.tier.reasoning = anthropic:claude-sonnet-5) ran with cost
telemetry blind on that tier: canonicalLookup missed, the budget meter
logged BUDGET_METER_NO_PRICING and disabled the gate, and cost views
under-reported spend.

- model-pricing.ts: anthropic:claude-sonnet-5 at $3/$15 and
  anthropic:claude-fable-5 at $10/$50. Sonnet 5's launch intro discount
  ($2/$10 through 2026-08-31) is deliberately not modeled — the table
  carries standard rates so estimates stay conservative and the entry
  needs no time-bombed edit when the promo lapses.
- takes-quality-eval/pricing.ts: claude-sonnet-5 added to the curated
  SUPPORTED_MODELS allowlist (a likely judge override). Fable 5 stays
  out — priced for warn-only consumers, not a budgeted-eval panel model.
- model-pricing.test.ts: pin tests for both rows, matching the existing
  Opus 4.8/4.7 pattern. Drift guards iterate the table; no changes.

All derived views (ANTHROPIC_PRICING bare view -> budget-tracker,
batch-projection, budget-meter) pick the rows up automatically.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 19:52:52 -07:00
a7b0ae80a9 v0.42.60.0 chore(release): eleven verified community fixes — changelog + version bump (#2888)
* v0.42.60.0 chore(release): eleven verified community fixes — changelog + version bump

Windows full-sync mass-delete fix, gateway tool-loop resume consolidation
(fix-wave A), two source-isolation closes, search-cache exclude-policy
keying, and six more verified community fixes. Files the take-writes
fail-open source fallback and the #2112 doctor hunk as follow-up TODOs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update reference docs for v0.42.60.0

- KEY_FILES.md: unpin stale KNOBS_HASH_VERSION number in the autocut entry
  (mode.ts is the single source of truth); document the full-sync reconcile
  path-separator normalization + mass-delete safety valve
  (planReconcileDeletes, GBRAIN_ALLOW_MASS_RECONCILE); describe the TTY-gated
  admin bootstrap token banner (--print-admin-token, env-sourced always hidden)
- docs/mcp/DEPLOY.md + docs/tutorials/company-brain.md: bootstrap token is now
  hidden on non-TTY starts; document GBRAIN_ADMIN_BOOTSTRAP_TOKEN and
  --print-admin-token for headless deploys

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(docs): regenerate llms bundle after KEY_FILES/deploy-doc sync

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 19:30:09 -07:00
bb417051fe fix(search): fold hard-exclude/include prefixes into knobs_hash — stop cross-process cache leak of excluded slugs (#2825) (#2885)
resolveHardExcludes() only ran at DB-query build time (cache miss), so
query_cache rows written by a process without GBRAIN_SEARCH_EXCLUDE could
be served to a process with it (and vice versa), leaking excluded slugs.
hybridSearchCached now resolves the effective hard-exclude list exactly
as the engines' query-build path does and folds it (sorted, append-only
hx= part) into knobsHash via a new KnobsHashContext.hardExcludes field.
KNOBS_HASH_VERSION 11 -> 12: one-time global cache cold-miss on upgrade,
refills within cache.ttl_seconds.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:57:55 -07:00
+4 86962b242b fix(gateway): consolidate tool-loop resume + provider fixes (fix-wave A) (#2820)
Collector branch superseding the gateway tool-loop duplicate cluster and
adjacent provider fixes. Re-implemented from the best of each PR (deduped by
content, not file-overlap); every fix carries test coverage.

Gateway tool-loop resume (supersedes #1934 #2062 #2065 #2112 #2274 #2487
#2336 #2257 #2499 #2491, test #2063):
- toolLoop now persists the tool-result user turn per round (onToolResultTurn),
  so a resumed subagent job reloads a balanced transcript instead of dangling
  assistant tool-calls that non-Anthropic providers reject with
  AI_MissingToolResultsError.
- runSubagentViaGateway reconciles an already-corrupted transcript on resume:
  it heals every dangling assistant tool-call turn (not just the tail) from the
  settled subagent_tool_executions rows, re-dispatching idempotent-pending
  tools and throwing on non-idempotent, mirroring the legacy Anthropic path.
  Terminal early-return for a transcript that already reached end_turn.
- repairToolPairing() is a last-resort normalization at the chat() boundary
  (from #2336): back-fills error stubs for any assistant tool-call still
  unanswered (partial turns, provider-duplicated/dropped IDs on local models,
  length-truncated batches). No-op on balanced input.
- toModelMessages is Date-safe (Postgres timestamptz -> ISO via a JSON round
  trip at the SDK boundary, never a ::jsonb cast; degrades bigint/circular to a
  string instead of throwing) and drops non-string text blocks reasoning models
  emit that AI SDK v6 rejects (#2488).

Adjacent provider fixes:
- Model-aware default max output tokens: thinking-by-default Claude 5 models get
  headroom (gateway 32000, think 16000) while everything else stays 4096/4000,
  so DeepSeek/OpenAI subagents don't exceed provider caps (#2614 #2806).
- DeepSeek: promote reasoning_content into content when content is empty, via a
  fail-open recipe fetch shim (#2617).
- OpenRouter: map openrouter_api_key (config + env) into OPENROUTER_API_KEY
  through buildGatewayConfig; register agent.use_gateway_loop, zeroentropy and
  openrouter keys in KNOWN_CONFIG_KEYS so `config set` accepts them (#2572,
  config key from #2112).

Preserves JSONB (no JSON.stringify into ::jsonb), engine parity, source
isolation, and trust-boundary invariants. Verified with the gbrain-pr-test-env
consumer matrix (clawlancer/Postgres, gstack + hivemindos/PGLite) baseline-FAIL
-> candidate-PASS on the same repro, plus bun run verify (31/31) and 439
targeted unit + e2e tests.

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: thomaskong119 <thomaskong119@hotmail.com>
Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: fbal23 <fbal.public@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: David Carolan <david@joyrestart.com>
Co-authored-by: Masashi-Ono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
Co-authored-by: psam-717 <mphilannorbah@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 16:31:53 -07:00
Ziyang Guo ec3910afc4 fix(import): route image pages by source (#2718) 2026-07-16 16:31:50 -07:00
Ziyang Guo 7a275bf0b5 fix(takes): scope page lookup by source (#2698) 2026-07-16 16:31:47 -07:00
285cf39f9a fix(config): register Life Chronicle keys so the documented enable command works (#2632)
* fix(config): register Life Chronicle keys so the documented enable command works

The v0.42.56.0 release notes say `gbrain config set auto_chronicle true`,
but the key was never added to KNOWN_CONFIG_KEYS — the documented command
fails with 'Unknown config key' and the operator has to discover --force by
reading source. Registers 'auto_chronicle' plus the 'chronicle.' prefix
(chronicle.tz and future knobs). Same registration class as the v0.42.42.0
spend-controls fix. Regression test pins both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk

* fix(config): register takes.bootstrap_enabled too — same unregistered-key class

Hit while enabling the takes bootstrap on a live brain: the onboard
remediation's documented enable key fails 'Unknown config key' exactly like
auto_chronicle did. Registered + pinned by the same regression test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk

---------

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:27:22 -07:00
Matt Gunnin 836d83012d fix(orphans): exclude generated corpus roots (#2068) 2026-07-16 16:27:19 -07:00
vinsew d0447a597b fix(files): normalize bigint sizes before JSON serialization (#472)
Postgres returns BIGINT file sizes as native BigInt values. Returning those values directly from file_list makes JSON serialization fail, and using them in CLI arithmetic can also throw.

Convert size_bytes to Number at the operation boundary and in the CLI display path. File sizes remain exact well beyond any practical attachment size.

Add a unit regression that exercises a native BigInt row and proves the operation result is JSON-serializable, plus a real-Postgres E2E assertion for the file_list response.
2026-07-16 16:27:16 -07:00
1e1b9a9441 test(doctor): pin embedding dims in hidden-by-search-policy — kill the shard-order 1280/1536 flake (#2801)
doctor-hidden-by-search-policy.test.ts hardcodes Float32Array(1536)
vectors (basisEmbedding) but lets initSchema size its vector columns from
process-global gateway state (getEmbeddingDimensions(), default 1280).
Whether the file passes depends on which test files run before it in the
shard; adding test files to the repo reshuffles the weight-packed shards,
so unrelated PRs trip it (seen on #2800 CI, test (1): every upsertChunks
died with 'expected 1280 dimensions, not 1536').

Same fix + rationale as engine-find-trajectory.test.ts and
cosine-rescore-column.test.ts, which document this exact class:
configureGateway(1536) in beforeAll BEFORE initSchema, resetGateway in
afterAll. The suite is now self-sufficient regardless of predecessor
state. Not reproducible outside CI's exact shard packing; the pin removes
the order-dependence either way.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:26:46 -07:00
Jaehwan LeeandClaude Opus 4.8 659b6e9b4d fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437) (#2440)
* fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437)

extractFencedChunks() ran marked.lexer() on every page body. The lexer
allocates transient memory proportional to page size even when there is no
code fence to extract — a ~2MB table/doc page spikes ~110MB of heap to
produce zero fenced chunks. During bulk import these per-page spikes stack
on accumulated chunk/embedding memory and can OOM the worker; the existing
try/catch cannot rescue an OOM (process death, not a throw).

On a representative brain ~99% of importable pages have no fence, so the
lexer pass is pure wasted work there. Add a fast-path that returns early
when the body contains no fence marker. Matches both ``` and ~~~ so tilde
fenced code still extracts.

Scope: this removes the fence-less transient-allocation surface (the
observed incident). It does not make marked.lexer safe for pages that DO
contain a fence; an input-size/nesting cap is a sensible follow-up.

Tests: add two regression cases — tilde-fenced code still extracts, and a
large fence-less table page imports with zero fenced chunks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(import): match marked's \r normalization in the fence fast-path (#2437)

Self-review follow-up. marked normalizes `\r\n|\r → \n` before lexing, but the
no-fence fast-path probed the raw body with `(^|\n)`. A CR-only (classic-Mac)
line-ended page with a real fenced block would be skipped by the guard while
marked would have extracted it — a lost fenced_code chunk. Widen the line-start
class to `(^|[\r\n])` so the probe agrees with marked. CRLF was already covered.

Add a regression test (CR-only fenced page still extracts); it fails on the old
`(^|\n)` regex and passes on the fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 16:26:43 -07:00
1alessioandClaude Fable 5 f15163f727 fix(sync): normalize path separators + add mass-delete safety valve to full-sync reconcile (#2828) (#2836)
On a Windows checkout, path.relative yields backslash-separated paths while a
page's stored source_path can hold forward slashes (e.g. git-derived). The
full-sync reconcile compared the two without normalizing separators, so every
file-backed page looked stale and the reconcile deleted the entire source.

- Normalize separators on BOTH sides of the membership test (shared
  .replace(/\/g, '/')), so pages written with either separator match on any OS.
- Add a mass-delete safety valve: when a reconcile would sweep > 50% of the
  file-backed pages a strategy manages on a source with > 20 of them, skip the
  delete and surface a loud warning instead of silently wiping the brain.
  GBRAIN_ALLOW_MASS_RECONCILE=1 restores the old behavior.
- Factor the decision into pure, exported helpers (planReconcileDeletes,
  massReconcileAllowed) and unit-test the separator matching, the valve
  threshold, and the env override without a live engine.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:22:21 -07:00
Jaehwan LeeandClaude Opus 4.8 bb3376e3b0 fix(security): prevent leaking admin bootstrap token to non-TTY stdout (#2625)
* fix(security): #2624 don't print admin bootstrap token on non-TTY (log-leak)

serve --http printed the generated admin token in the startup banner
unconditionally. In containerized deploys stderr ships to centralized log
storage, turning the token into a standing secret in logs.

Fail-safe default: the generated token now prints only when stderr is an
interactive TTY. Non-TTY starts hide it (--print-admin-token forces it;
$GBRAIN_ADMIN_BOOTSTRAP_TOKEN + --suppress-bootstrap-token already existed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): #2624 banner shows 'from env' before non-TTY hidden guard

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 13:34:28 -07:00
5008b287e4 v0.42.59.0 chore(release): five verified community fixes — changelog + version bump (#2797)
Rolls up the five fixes merged as #2735 #2736 #2737 #2738 #2739
(issues #2724 #2677 #2723 #2726, plus the think slice of #2200).

Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 00:19:29 -07:00
Time Attakc 010847c020 fix(think): enforce source scope across gather (#2200) (#2739)
Carry the caller's scalar or federated source scope through every think gather stream: hybrid page retrieval, takes keyword/vector retrieval, and graph traversal. Adds source predicates to the takes retrieval methods in both engines. Part of #2200 (the think slice; #2200 stays open as the tracking issue for the remaining by-slug read ops).
2026-07-13 00:09:27 -07:00
Time Attakc 8e84c5b4a1 fix(facts): parse escaped pipes in facts fence round-trips (#2726) (#2738)
parseRowCells now splits on unescaped pipes only and decodes escaped pipes while preserving ordinary backslashes and empty cells, so facts whose text contains literal | survive the fence->DB reconcile instead of being silently deleted. Fixes #2726.
2026-07-13 00:09:16 -07:00
Time Attakc 68ed7bafa4 fix(facts): quarantine ambiguous entity matches (#2723) (#2737)
Bare names resolve only when prefix expansion finds exactly one canonical candidate; ambiguous collisions and low-specificity multi-token fuzzy matches fall through to the guarded holding path instead of confident wrong attribution. Fixes #2723.
2026-07-13 00:09:04 -07:00
Time Attakc 42ab0956a4 fix(migrate): preserve sources and scope resume targets (#2677) (#2736)
migrate --to now copies the complete source catalog before pages (fixes the pages_source_id_fkey failure on multi-source brains), and resume manifests carry an opaque target identity so a checkpoint from one target is discarded for a different target. Fixes #2677.
2026-07-13 00:08:52 -07:00
Time Attakc 2fca124468 fix(schema): unblock pre-v121 schema replay (#2724) (#2735)
Adds the missing timeline_entries.event_page_id forward-reference bootstrap probe + bare-column repair to both engines so pre-v121 brains can replay the current schema and reach migration v121. Fixes #2724.
2026-07-13 00:08:36 -07:00
Garry TanandClaude Opus 4.8 a25209bbb2 v0.42.58.0 fix(ai): provider-agnostic gateway — env clobber, base-URL /v1, embedding dims (#1249 #1250 #1292 #2271 #2209) (#2627)
* fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249)

Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an
unconditional process.env spread let that empty string override a valid
config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter
'' / undefined before the merge; '0' and 'false' are preserved.

* fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292)

#1250: createAnthropic/createOpenAI were called with no baseURL, so an
env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a
shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic +
openai native sites (google deferred until its suffix is verified).

#1292/D6: the user_provided_model_unset guard was structurally unreachable as
a no-model check (parseModelId throws on a bare provider) and only ever
false-positived for litellm:<model>, silently disabling vector search. Replace
it with a real dims-presence check for user-provided/zero-default recipes and
delete the dead branch in both consumers. Also stop configureGateway from
fabricating a default embedding_dimensions, so 'no dims set' stays honest.

* fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209)

#2271: a new trust_custom_dims flag adds a passthrough tier so ollama /
llama-server / litellm accept a user-supplied --embedding-dimensions instead of
being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/
zeroentropy) and excludes openrouter (declares dims_options). Register modern
ollama embed model names.

#2209: litellm setup_hint now states the /v1 path convention and the docs
pointer is corrected to docs/integrations/embedding-providers.md.

* docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292)

* fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage

Pre-landing adversarial review (P1): the new dims-unset guard told users to run
'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a
schema-sizing field). Both consumer messages now point at 'gbrain init
--embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the
trust_custom_dims passthrough, and a configureGateway backfill-invariant test.

* chore: bump version and changelog (v0.42.57.0)

Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL
normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims
passthrough, #2209 litellm /v1 hint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0)

Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave:
- LiteLLM section now names the /v1 base-URL convention (#2209).
- Ollama section lists the newly-registered modern embedders qwen3-embed-8b
  + snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271).
- llama-server section notes gbrain trusts the user-declared dimension (#2271).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave

- KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims
  (#2271 passthrough tier, runs after dims_options + Matryoshka allowlists)
- ENGINES.md: embedding design-choice note now names the provider-agnostic
  gateway delegation instead of the stale OpenAI-only parenthetical
- embedding-providers.md: drop an exact-duplicate doctor-8c paragraph
- llms-full.txt regenerated (ENGINES.md is inlined in the bundle)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal)

- embedding-providers.md OpenAI section: document OPENAI_BASE_URL /
  ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface)
- TL;DR table: litellm multimodal is backend-permitting (recipe declares
  supports_multimodal: true, routed via the openai-compat multimodal path),
  not "no"

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files

CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not
1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema
from AMBIENT gateway state in beforeAll — which runs before the
legacy-embedding-preload's per-test 1536 restore. A preceding file that ends
with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare
resetGateway poisons the next fresh initSchema down to 1280-d columns. The
new test files in this PR reshuffled shard bin-packing and exposed the trap.

- engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the
  pattern bunfig's preload documents) — deterministic regardless of neighbors
- facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight:
  restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless

Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7
CI failures; with the pin = 0. Verified in-process pair runs both orders.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 10:05:23 +09:00
Garry TanandClaude Opus 4.8 058f448b9a v0.42.57.0 fix(pglite): incident — never steal a live data-dir lock + corrupted-store recovery hint (#2348) (#2400)
* fix(pglite): never steal the data-dir lock from a live holder (#2348)

A busy `gbrain dream`/`embed` holder whose 30s heartbeat lapsed (the JS event
loop is blocked during long synchronous WASM imports/CHECKPOINTs) used to get
its lock reaped past the steal-grace window. PGLite/WASM is strictly
single-writer, so a second OS process then opened the same data dir and
corrupted the catalog + pgvector extension state (58P01 / internal_load_library
/ "type vector does not exist"), recoverable only by wipe+restore. Reap ONLY a
dead PID; a live holder is never stolen — a wedged-but-alive or PID-reused
holder makes the acquire time out with a message naming the PID. Removes the
GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS knob (no longer meaningful).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pglite): point a corrupted store at reinit-pglite recovery (#2348)

classifyPgliteInitError gains a `corrupt` verdict for the 58P01 /
internal_load_library / "vector does not exist" / "content_chunks does not
exist" signature (beating the generic wasm-runtime match), so an already-damaged
store gets actionable recovery (gbrain reinit-pglite / restore a backup) instead
of the wrong macOS-WASM hint. Updates KEY_FILES.md to current lock behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.55.0 fix(pglite): incident — never steal a live lock + corrupted-store recovery hint (#2348)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.56.0 chore: re-bump past #2399 version collision + refresh ownerToken comment

#2399 (security wave) claimed 0.42.55.0; take the next slot. Also updates the
LockHandle.ownerToken JSDoc to current #2348 behavior (live holders are never
reaped, so reap-then-reacquire is dead-holder-only; token guard stays as
defense-in-depth).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 02:38:37 -07:00
Garry TanandClaude Opus 4.8 646179047a v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + thought diary + bi-temporal per-entity ontology (#2390) (#2533)
* feat(chronicle): register event + diary page types (#2390)

Life Chronicle Phase A.1. Adds `event` (timeline atom) and `diary`
(first-person interiority) as temporal-primitive page types under
life/events/ and life/diary/, extractable:false — registered in
ALL_PAGE_TYPES, both base schema packs, and the inferType prefix table,
with parity fixtures. Also lands the chronicle read result types
(ChronicleTimelineRow/ChronicleTimelineOpts/LastSeenResult) consumed by
Phase A.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): event_page_id timeline projection + day/since/last-seen reads (#2390)

Life Chronicle Phase A.2. Adds a nullable event_page_id FK to
timeline_entries (migration v120; mirrored in schema.sql, pglite-schema,
schema-embedded) so a type:event page projects ONE date-index row keyed
to its depth page; a partial UNIQUE(event_page_id, date) makes
re-extraction with a changed summary an update, not a duplicate.

Dual-engine getTimelineForDate / getSince / getLastSeen filter the depth
page on deleted_at, hide soft-deleted event projections at READ time
(not just doctor), order by event effective_date for intra-day sequence,
and honor source scope (sourceIds[] > sourceId). Ops surface as
`gbrain day <date> [--week]`, `gbrain since <date> [--kind]`,
`gbrain last-seen <entity>`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): auto-emit extractor — backstop + chronicle_extract job (#2390)

Life Chronicle Phase A.3. A put_page backstop (gated on status==='imported',
the auto-link/timeline trust gate, and the default-OFF auto_chronicle flag;
diary + event pages never eligible) enqueues a chronicle_extract minion job.
The job runs the extractor off the write path: deterministic when/who, an
injectable LLM judge (default = chat gateway; no-op when no gateway), an
all-or-nothing parse barrier (a malformed proposal writes NOTHING), then
content-addressed event pages + a timeline projection via the new dual-engine
upsertEventProjection (idempotent — re-run yields one event + one projection).

New: src/core/chronicle/{eligibility,config,extract-events,backstop}.ts,
engine.upsertEventProjection (both engines), the jobs.ts handler + a 10-min
timeout. 14 unit tests (eligibility, idempotency, parse barrier, backstop
gating + enqueue) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): quick-capture for diary + manual events (#2390)

Life Chronicle Phase A.5. `gbrain capture` now routes the default slug by
type (diary → life/diary/, event → life/events/, else inbox/) and accepts
--who/--what/--where/--kind/--depth to assemble the `event:` frontmatter
block for `--type event`. User-declared event keys win per-key over the
flags. Goes through the existing put_page → write-through → embed path. 6
new unit tests; existing capture tests stay green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): bi-temporal per-entity ontology on the facts table (#2390)

Life Chronicle Phase B.10 — the feature's differentiator. Rather than a
parallel store, the open-world per-entity ontology EXTENDS the existing
`facts` table (eng-review G1): migration v121 adds dimension/value/value_hash
/dim_status columns + a deterministic partial-UNIQUE dedup key + an asof read
index. facts already supplies bi-temporal validity, supersession
(superseded_by), visibility (remote redaction), confidence, provenance, and
embedding — all inherited.

Dual-engine methods: mergeOntologyFact (deterministic value_hash dedup →
idempotent retry; same value corroborates; a different value forward-closes
the prior row's valid_until + superseded_by; a BACKDATED conflicting value is
recorded WITHOUT rewriting, surfaced by findOntologyConflicts), getOntology
with `--asof` valid-time travel (expired_at + status + validity-window in the
predicate so quarantined/superseded never leak), discoverOntologyDimensions,
findOntologyConflicts. Novel/LLM-proposed dimensions quarantine; a seed lexicon
canonicalizes name drift (job_role → role). 9 unit tests cover the full
lifecycle; typecheck pins both engines to the interface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): ontology ops — get/add/dimensions/contradictions (#2390)

Life Chronicle Phase B.11. Exposes the bi-temporal ontology over CLI + MCP
(contract-first, auto-generated): `gbrain ontology <entity> [--asof]`,
`gbrain ontology-add <entity> <dim> <value>`, `gbrain ontology-dimensions`
(meta-ontology rollup), `gbrain ontology-contradictions`. All reads route
through sourceScopeOpts. Privacy: ontology_get redacts diary-sourced
observations (source under life/diary/) for untrusted (remote) callers. 3 op
tests (incl. the remote-redaction path); 47 op-registry/tool-def/description
tests stay green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): agent-context loader — volunteer_chronicle (#2390)

Life Chronicle Phase B.12. `loadChronicleContext` hands an agent the recent
timeline (last N days) + the validity-resolved current ontology for the
entities in play, in one zero-LLM payload, so it orients before acting — the
exact gap behind fumbled chronology. Pure composition over getSince +
getOntology (no new SQL). Exposed as the `volunteer_chronicle` read op
(`gbrain orient [--days] [--entities a,b]`); diary-sourced ontology is redacted
for remote callers. 2 loader tests + op-registry green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): backfill op — sweep existing meetings into events (#2390)

Life Chronicle Phase A.8. `chronicle_backfill` (local-only admin op;
`gbrain chronicle-backfill [--since] [--limit] [--dry-run]`) lists existing
meeting/conversation/calendar pages (source-scoped via listPages), filters
through the chronicle eligibility predicate, and enqueues one chronicle_extract
job per eligible page so existing brains populate the timeline. --dry-run
counts only; per-page enqueue failures are surfaced in `errors`, never
swallowed. 2 op tests (dry-run count + enqueue).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): delight — on-this-day + narrative rendering (#2390)

Life Chronicle Phase A.6 (delight). Dual-engine getOnThisDay (events from the
same month-day in prior years; `gbrain on-this-day [--date]`) reusing the
chronicle JOIN shape (deleted-event hiding + source scope). A pure
renderTimelineNarrative turns timeline rows into prose; `gbrain day --narrative`
returns it alongside the events. 5 tests. (Coverage gap-detection ships with
the advisor collectors next.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): proactive advisor collector (#2390)

Life Chronicle Phase A.7. A brain-state advisor collector surfaces two
proactive signals in `gbrain advisor`: unresolved ontology conflicts (warn,
→ `gbrain ontology-contradictions`) and recent meetings with no timeline
coverage (info, → `gbrain chronicle-backfill`). Advisory-only (no dispatch_id);
runs over MCP too (not workspace-dependent); tolerant of pre-migration brains.
3 tests + advisor-op-gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): doctor chronicle_projection_health probe (#2390)

Life Chronicle Phase B.13. An always-run doctor check (keyed off the
event_page_id schema, NOT a migration verify-hook) counts timeline
projections whose event page is soft-deleted — hidden at read time, surfaced
here as a cleanup backlog (`gbrain integrity auto`). Tolerant of pre-migration
brains. 1 detection test. (auto_chronicle / chronicle.tz flags already work
via getConfig defaults; their docs land with document-release at ship.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): E1 temporal recall — chronicle-type boost on temporal queries (#2390)

Life Chronicle Phase A.4 (E1, ambient temporality). Rather than a separate RRF
arm (which needs chunk hydration + risks the fusion path), E1 is a bounded
post-fusion boost: applyChronicleTypeBoost lifts `event`/`diary` results on
temporal queries, wired INSIDE runPostFusionStages' `recency !== 'off'` branch
so it fires ONLY on temporal intent — non-temporal search is bit-for-bit
unchanged (proven by 110 passing search-path tests). Bounded ([1.0,1.25]) +
floor-gated like the other metadata stages; attribution via `chronicle_boost`.
3 unit tests. (Empirical precision/negative measurement lands with the eval.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chronicle): feature eval — gbrain eval chronicle (PRIMARY proof) (#2390)

Life Chronicle Phase A.9, the North-Star proof. A deterministic, CI-safe eval
(brings its own in-memory PGLite; no LLM, no gateway) builds a synthetic month
corpus with a known gold chronology + a planted ontology supersession + a
planted conflict, then scores the chronicle layer on six gold tasks: day
reconstruction (intra-day order), last-seen exact date, ontology supersession,
--asof valid-time travel, contradiction surfacing, and source isolation.
`gbrain eval chronicle [--json]` exits 0 iff all pass — currently 6/6. The OFF
baseline (raw meeting pages) structurally can't order intra-day events or
time-travel ontology; the ON path does. (The live-LLM OFF-vs-ON agent arm +
LongMemEval temporal slice are a follow-up; this deterministic bar gates CI.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chronicle): pre-landing review — conflict validity, parse-barrier date, remote conflict redaction (#2390)

Three bugs caught by the codex pre-landing review on the diff:
1. findOntologyConflicts ignored valid_until, so a normal forward supersession
   (founder→advisor from two sources) falsely reported as a live conflict. Now
   restricted to currently-open rows (valid_until IS NULL) in both engines.
2. The extractor parse barrier accepted any when-string >= 4 chars; a non-date
   value slipped past, wrote the event page, then threw on the projection's
   ::date cast (partial write). isValidProposal now requires a real parseable date.
3. The ontology_conflicts op had no remote diary redaction (ontology_get did);
   remote callers now get diary-sourced values filtered, and conflicts that lose
   their disagreement after redaction are dropped.

Three regression tests added; 29 chronicle tests + eval 6/6 green; typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + diary + bi-temporal ontology (#2390)

Bump VERSION + package.json to 0.42.56.0 and add the CHANGELOG entry for the
Life Chronicle feature (#2390, closes duplicate #2388).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chronicle): register #2390 surfaces with the five CI guard suites (#2390)

CI caught five guard tests that pin registration invariants my diff tripped:
- schema-bootstrap-coverage: the four v122 facts ontology columns join
  COLUMN_EXEMPTIONS (facts is migration-created; the partial indexes live
  inside v122; every reader filters dimension IS NOT NULL — same precedent
  as facts.claim_metric et al).
- no-valid-until-write (R8): both engines' mergeOntologyFact forward-
  supersession is a deliberate, documented valid_until write authority
  (engine-layer, dimension IS NOT NULL only; the contradiction probe still
  never mutates).
- doctor-categories: chronicle_projection_health registered under
  BRAIN_CHECK_NAMES (same class as child_table_orphans).
- checkTypeProliferation: the test is now threshold-relative (computes
  declared from the active pack, seeds declared+6) so base-pack growth
  can't silently move the fixed threshold again.
- schema-cli: gbrain-base page-type count 25 → 27 (event + diary), with
  assertions on both new types.

All 41 guard tests green; typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: file Life Chronicle follow-up TODOs (v0.42.56.0, #2390)

Eight deferred items from the CEO/eng review decisions (auto-emit default-flip
fast-follow, live eval arm + LongMemEval slice, passive diary consent,
interval-splitting, federation, place-as-entity, meta-ontology dashboard,
materialized daily pages), each with decision provenance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(architecture): KEY_FILES entry for the Life Chronicle module (#2390)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 10:48:54 -07:00
Garry TanandClaude Opus 4.8 dde1132a29 v0.42.55.0 fix(security): dotfile/skills/slug confinement + DCR consent default + schema-lint migration (#418 #419 #245 #1353 #1647 #171 #1385) (#2399)
* fix(security): confine routing dotfiles, skills dir, slugs, and transcription exec

Shared src/core/path-confine.ts consolidates the realpath-containment idiom
(moved from sources-ops.ts) and adds isTrustedDotfile + isWriteTargetContained.

- .gbrain-source (source-resolver) and .gbrain-mount (brain-resolver) walk-up
  dotfiles are now lstat trust-gated: a symlink, foreign-owned, or world-writable
  file is refused on multi-user hosts (#418), fail-closed on stat error.
- resolveWorkspaceSkillsDir + every skills-dir tier (env, cwd_walk_up, repo_root,
  cwd_skills, install_path) route through realpath containment so a symlinked
  workspace/skills can't escape the declared workspace (#419).
- resolveSourceId/resolveBrainId realpath both sides of the registered local_path
  / mount prefix match so a symlinked cwd can't misattribute source/brain.
- validateSlug rejects NUL/control, bidi/RTL overrides, backslashes, and
  URL-encoded path separators at the shared putPage/updateSlug chokepoint;
  write-through confirms the file path stays within the source tree.
- transcribeLargeFile uses execFileSync arg-arrays + fs.rmSync (no shell), so a
  path with shell metacharacters is never parsed by a shell (#245).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(security): default dynamic-registration clients to authorization_code

Self-registered DCR clients (the unauthenticated network registration path)
previously defaulted to the client_credentials grant, which bypasses the
/authorize consent screen. They now default to authorization_code; an explicit
client_credentials request is rejected with invalid_client_metadata unless the
operator opts in with the new --enable-dcr-insecure flag. A loud stderr WARNING
prints at startup whenever DCR is enabled (#1353). Manual CLI/admin client
registration is unchanged (operator-trusted).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(security): schema-lint hardening migration (search_path + view security_invoker)

Migration v120 brings existing brains to the same posture as fresh installs:

- ALTER VIEW page_links SET (security_invoker = on) on Postgres so the view
  honors the caller's RLS instead of the owner's (the view-through-RLS bypass).
- ALTER FUNCTION ... SET search_path on the gbrain-owned trigger/event functions
  (both engines, IF EXISTS so engine-only functions are skipped; body untouched,
  so the load-bearing auto_enable_rls event trigger is unchanged). Closes #171.
- Broaden the BYPASSRLS preflight in the historical RLS migration gates to honor
  superuser and inherited-role BYPASSRLS, so a superuser-connected fresh install
  no longer aborts (#1385).

Fresh-install function definitions in schema.sql / pglite-schema.ts are
born-correct (regenerated schema-embedded.ts). scripts/check-search-path.sh is a
new CI guard (wired into verify) that fails if a trigger function in the schema
base files is added without SET search_path. Postgres-only assertions live in the
bootstrap E2E; the PGLite path is covered by test/migration-v120.test.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.55.0 fix(security): dotfile/skills/slug confinement, DCR consent default, schema-lint migration

Bump VERSION + package.json to 0.42.55.0 and add the CHANGELOG entry for the
security-hardening wave (#418 #419 #245 #1353 #1647 #171 #1385).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(security): note the DCR consent default in SECURITY.md (#1353)

The "disable client_credentials, only allow authorization_code" guidance is now
the built-in DCR default; document the new --enable-dcr-insecure escape hatch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(todos): add takes_search + code_def to the federated by-slug P1 (#2200)

The v0.42.55.0 eng-review codex pass flagged takes_search (holder-allowlist only)
and code_def (brain-wide raw SQL over content_chunks) as remaining same-class
surfaces. Noted on the existing #2200 P1 follow-up, with the caveat that the
#2399 close-list deliberately keeps #1371/#2200 open until this lands.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(security): correct plpgsql alias collision in #1385 BYPASSRLS gate (real-PG)

The broadened BYPASSRLS preflight aliased `pg_roles r`, but several RLS DO-blocks
already declare `r record` for their backfill FOR loop, so plpgsql resolved
`r.oid`/`r.rolbypassrls` to the unassigned record variable → "record \"r\" is
not assigned yet" on real Postgres (PGLite tolerated it; the DATABASE_URL-gated
e2e jobs are the backstop). Renamed the subquery alias to `pr` at all 10
migrate.ts sites; also broadened the schema.sql base RLS gate the same way (with
the `pr` alias) for #1385 consistency on superuser fresh installs, and
regenerated schema-embedded.ts.

Also fixes a PRE-EXISTING engine-parity bug (confirmed failing on clean
origin/master): the relationalFanout shape compared `canonical_chunk_id`, a
serial id that diverges between a fresh PGLite engine and a shared Postgres DB
(setupDB TRUNCATEs without RESTART IDENTITY). Compare its presence, not the
exact value.

Validated on real Postgres (pgvector/pg16): migration v120 applies, the v35 RLS
backfill runs, and engine-parity + postgres-bootstrap + jsonb-parity are green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 08:00:58 -07:00
Garry TanandClaude Opus 4.8 814258dda6 v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard (#2375)
* fix(sync): op_checkpoints pin write double-encodes jsonb — every sync aborts (#2339)

recordCompleted bound JSON.stringify(array) to a $3::jsonb param via postgres.js
.unsafe(), double-encoding it into a jsonb string scalar that violates the v119
op_checkpoints_completed_keys_array CHECK — aborting every multi-source sync on
real Postgres at the first checkpoint write. PGLite parses the string silently,
which is why unit tests stayed green and it shipped. Cast through $3::text::jsonb
so the text->jsonb cast parses a genuine array. Adds a DATABASE_URL-gated parity
test + a dedicated Postgres CI job so the guard can never silently skip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(db): sweep positional jsonb double-encode sites + AST CI guard (#2324)

Every executeRaw/.unsafe site that bound JSON.stringify(x) to a bare positional
jsonb cast double-encodes on real Postgres (same class as #2339). Sweep them all
to the text::jsonb form across query-cache, sources-ops, llm-base,
calibration-profile, impact-capture, subagent, receipt-write, traversal-cache,
symbol-resolver, and the agent/sources commands. Adds scripts/check-jsonb-params.mjs
(AST-lite scanner for the positional form the legacy template grep misses, incl.
generic-typed calls), wired into check-jsonb-pattern.sh, with a self-test. PGLite's
native db.query is not scanned — it parses text to jsonb natively, so the bug can't
occur there.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(search,eval): alias-hop injected results carry page_id (contradiction-probe crash)

applyAliasHop injected synthetic SearchResults without page_id (the `as
SearchResult` cast hid the missing field), so listActiveTakesForPages bound
undefined/NaN into ANY($1::int[]) and crashed the whole contradiction probe on
real Postgres. Stamp page_id=page.id at the injection site and add a finite-id
filter in generateIntraPagePairs as a defensive backstop (mirrors hybrid.ts:63).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(engines): positional jsonb binding rule (text::jsonb vs the double-encode trap)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard

Bumps VERSION + package.json to 0.42.53.0, adds the CHANGELOG entry, and
regenerates llms-full.txt. Ships the #2339 sync-abort hotfix, the repo-wide
positional jsonb double-encode sweep, the alias-hop contradiction-probe crash
fix, and the new positional-form CI guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: post-ship sync — jsonb invariant now covers the positional form + new guard

CLAUDE.md JSONB invariant + KEY_FILES (sql-query, check-jsonb-pattern, op-checkpoint)
now describe the #2339 positional double-encode class, the $N::text::jsonb fix, and
the new check-jsonb-params.mjs guard. Regenerates llms-full.txt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 06:05:16 -07:00
Garry TanandClaude Opus 4.8 bb2e88c42a v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) (#2287)
* test(supervisor): pin LOCK_HELD fence-exit is never counted as a crash (#2227)

A duplicate supervisor loses the queue-scoped DB singleton lock (#1849) and
exits LOCK_HELD before spawning a worker or emitting 'started'. summarizeCrashes
counts only worker_exited, so the fence path is structurally uncountable. Pin it
so a future refactor that logs worker_exited on the fence path fails here instead
of silently re-introducing the crash-budget breaker-trip loop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autopilot): per-source cycle binds FS phases to source.local_path, not global repo (#2194 #2227)

A per-source autopilot-cycle inherited the global sync.repo_path as brainDir while
stamping DB freshness for source_id — mixed scope. FS phases (sync/lint/extract)
ran against the wrong tree, so the failure-cooldown and freshness gates would
attribute work to the wrong source. Resolve the source's local_path in the handler
(reuse the archive-recheck SELECT) and bind brainDir to it; a pure-DB source gets
null (FS phases skip) instead of falling through to the global checkout. Legacy
no-source dispatch keeps the global repoPath. Prerequisite for the cooldown/split
commits (codex outside-voice #8). Resolves TODOS:634.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(supervisor): detect a live supervisor via the DB lock under split $HOME (#2227)

jobs supervisor status + doctor read the HOME-derived pidfile, so a supervisor
started under a different $HOME (keeper=/root vs ops=/data) read as 'not running'
while healthy — the false signal that drives an operator to spawn a duplicate.
Both surfaces now fall back to the queue-scoped DB singleton lock (#1849), the
HOME-independent authority, when the pidfile shows nothing. New isLockHolderLive
keys on lock freshness (ttl + heartbeat steal-grace), never process.kill, so PID
reuse can't false-positive (pid-liveness-alone-pid-reuse). Status surfaces the
holder host/pid + recorded concurrency/max-rss from the latest started event.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(supervisor): degraded retry instead of permanent give-up on crash storm (#1994 #2227)

max_crashes_exceeded gave up forever, so a transient DB-pooler blip that tripped
the soft budget wedged the queue until a human restart (#2227's breaker-trips tail).
Crossing the soft budget now enters degraded mode: keep respawning with capped
exponential backoff (60s cap — a paced retry, not a hot loop) and emit a loud
crash_budget_degraded health_warn. The existing stable-run reset clears the count
once a respawn survives >5min, so a recovered DB self-heals. Permanent give-up
fires only at a much-higher hard ceiling (maxCrashes × 10), tunable/disablable via
GBRAIN_SUPERVISOR_HARD_STOP_CRASHES (0 = never). Resolves TODOS:92.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): clamp fan-out to worker concurrency + doctor warning (#2194)

Fan-out resolved to 4 (Postgres) regardless of worker --concurrency, so surplus
cycles queued behind the worker and raced the stalled-sweeper. Two fixes for the
same mismatch:
- resolveEffectiveFanoutMax clamps to max(1, concurrency-1) (reserve a slot),
  gated on a LIVE DB-lock holder so a stale started-audit row can't shrink
  throughput (codex #9/D5); no live holder → unknown → unclamped base. Escape
  hatch autopilot.fanout_clamp_to_concurrency.
- doctor's autopilot_fanout_concurrency check warns when fan-out exceeds
  effective slots — the misconfig was silent before. Advisory (started-event
  concurrency), wired into both doctor surfaces.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): per-source failure cooldown — break the dead-job storm (#2194)

Only SUCCESS gated dispatch, so a source whose cycle kept failing/timing-out
re-fanned-out every 5-min tick forever (200+ dead jobs/24h). Now a failed source
backs off with bounded exponential cooldown (10→120min). Read at DISPATCH from
minion_jobs dead/failed rows (timeouts/RSS-kills dead-letter via SQL and never
run handler code, so a write-only hook would miss them) AND re-checked at CLAIM
time in the handler (codex #5: already-queued/retrying jobs). A success clears it
(codex #7); null-source rows excluded (codex #6); engine-parity via executeRaw.
Disable with autopilot.failure_cooldown_min=0. Fail-open if config/history reads
error. Surfaced via fanout_cooldown_skipped + the fanout summary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): split the cycle — per-source phases + one global-maintenance job (#2194 #2227)

N per-source cycles each ran the brain-wide global phases (embed-all/orphans/
purge/…) concurrently, thrashing the same rows and taking the worker 4→10GB in
<60s → RSS-kill → orphaned stalls. Split them: per-source jobs now run only
source-scoped (+ mixed) phases and stamp last_source_cycle_at; a new
autopilot-global-maintenance job runs the global phases ONCE per window
(idempotency_key + maxWaiting:1 = structural single-flight) and stamps
autopilot.last_global_at. This is the codex-endorsed design that replaced the
rejected skip-and-stamp-fresh approach (codex #1/#2): no freshness poisoning, no
starvation — global work always runs as its own job, never marked done when it
wasn't. PHASE_SCOPE is now a runtime partition (GLOBAL ∪ NON_GLOBAL == ALL).
last_full_cycle_at still written for doctor/legacy (no longer a global gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): guard nullable engine in supervisor DB-lock fallback (#2227)

Follow-up to the supervisor-visibility commit: doctor's engine binding is
BrainEngine | null, so the inspectLock fallback must guard on a non-null engine
(tsc TS2345). No behavior change — a null engine simply skips the DB-lock probe
and falls back to the pidfile reading, as before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): categorize autopilot_fanout_concurrency check as ops (#2194)

Follow-up to the fan-out/concurrency commit: the doctor-categories drift guard
requires every check name in doctor.ts to belong to exactly one category set.
Add the new autopilot_fanout_concurrency check to OPS_CHECK_NAMES (infrastructure
liveness, alongside wedged_queue/supervisor).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: update KEY_FILES for the autopilot cycle split + supervisor degraded-retry (#2194 #2227)

Post-ship document-release: refresh the KEY_FILES current-state entries that
drifted — cycle.ts (GLOBAL/NON_GLOBAL phase split + last_source_cycle_at /
autopilot.last_global_at), jobs.ts (per-source local_path brainDir, claim-time
cooldown, autopilot-global-maintenance handler), supervisor.ts + child-worker
(degraded retry instead of permanent give-up; hard ceiling), db-lock.ts
(isLockHolderLive), handler-timeouts (new handler). Regenerated llms bundle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minions): handleTimeouts counts the timed-out run as a spent attempt (#1737)

The per-job timeout_at dead-letter (handleTimeouts) set status='dead' without
incrementing attempts_made, unlike the wall-clock and stall dead-letter siblings.
It is the FIRST killer to fire for the long-lane handlers (subagent / embed-backfill
/ autopilot-cycle) because timeout_ms is stamped at submit, so a timed-out long job
reported `attempts: 0/N (started: N)`. Mirror the siblings with attempts_made + 1
(terminal, no retry). Safe against double-count: the worker sweep runs handleStalled
-> handleTimeouts -> handleWallClockTimeouts sequentially and awaited, each guarded on
status='active', so the first to dead-letter excludes the row from the rest.

Regression assertions added (test/minions.test.ts + e2e/minions-resilience.test.ts)
so the increment can't be silently dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(agent): recognize trailing switches in `agent run`, keep prompts freeform (#1738)

parseRunFlags() broke flag parsing at the first positional token, so any flag
after the prompt (`gbrain agent run "do X" --detach`) was swallowed into the
prompt string and silently ignored. Now the no-value switches --detach/--follow/
--no-follow are hoisted when they trail the prompt, while everything else stays
verbatim: an unknown --word is treated as prompt text (no "unknown flag" throw),
a --switch mid-prompt is preserved, and `--` suppresses hoisting entirely for a
literal escape. Value-flags now reject a missing or flag-shaped value (and
--max-turns/--timeout-ms a non-number) instead of capturing undefined/NaN.

Contract change: a prompt that starts with or trails an unguarded --word no
longer errors; a literal trailing --detach needs `--`. Help text updated; tests
revised + extended (test/agent-cli.test.ts).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): honest live-sync status + progress-aware stall-abort (#1950)

Finishes the #2255 honest-freshness story for two gaps it left.

(a) `gbrain sources status` printed "idle" while a sync proc held the per-source
lock (the reported bug). New shared liveSyncStatus() helper in db-lock.ts reads
the SAME live-lock signal `gbrain doctor` uses; runStatus now shows "running"
(BACKFILL column + a sync_running field in --json) and suppresses the misleading
"never synced" warning while a sync is live. One helper, so the surfaces can't
drift (doctor/status retrofit tracked as a follow-up).

(b) A sync wedged-but-alive kept refreshing its lock heartbeat (it fires on its
own timer) and hadn't hit the wall-clock deadline, so only a manual pkill freed
it. New in-band stall watchdog keys off FORWARD IMPORT PROGRESS (progress.tick),
not the heartbeat: if no file completes for GBRAIN_SYNC_STALL_ABORT_SECONDS
(default 900s), it aborts via a controller composed into opts.signal, so the
drain returns partial() (last_commit unchanged, next run resumes from the
checkpoint) and withRefreshingLock releases the lock. Limits, documented in
code: a single file slower than the window trips it; a fully starved event loop
won't fire the timer (the wall-clock hard deadline is that backstop).

Tests: liveSyncStatus (live/expired/none/per-source) in db-lock-inspect; the
resolveStallAbortSeconds env matrix in sync-hard-deadline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(status): version field + per-section --deadline-ms budget (#1984)

`gbrain status` had no version in its JSON envelope and could hang on a slow
connection with no way to get a partial answer. Two additions:

- version: the StatusReport JSON now carries the local gbrain CLI version so a
  poller can pin behavior to a build. Thin-client also surfaces remote_version
  (the brain server's version), and the get_status_snapshot MCP op reports its
  version for that parity.
- --deadline-ms=N / --fast: a shared wall-clock budget. Each section is raced
  against the REMAINING budget via Promise.race (NOT process-watchdog, which
  SIGKILLs and can't return partial output), so one slow/hung section can't
  strand the snapshot — it's marked stale and the rest still return. The
  envelope gains partial:true + stale_sections[]; exit code stays 0 (a snapshot
  was produced). Invalid --deadline-ms → exit 2.

Tests: parseDeadlineFlag + withSectionDeadline (hermetic), the usage-error exit,
version presence in the PGLite envelope, and the op's version key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): report stall_timeout distinctly + document in-flight limit (#1950)

Pre-landing review (codex + adversarial): the stall watchdog aborted opts.signal
but the per-iteration abort checks returned partial('timeout'), collapsing a
wedge-reap into a user --timeout/SIGINT so JSON consumers couldn't tell them
apart. Add a 'stall_timeout' reason (set via a stallAborted flag) on the three
import-loop abort sites; deletes/renames-phase and checkpoint sites stay 'timeout'.
Sharpen the watchdog comment: the abort is observed BETWEEN files, so a hang
inside a single importFile is not interrupted until it returns (TODO: thread a
cancellation signal through importFile).

* fix(agent): `--` escape suppresses trailing-switch hoisting anywhere (#1738)

Pre-landing review: the leading-flag loop breaks at the first positional, so the
`escaped` flag only fired for a leading `--`. A `--` placed after a positional
left trailing-switch hoisting active, so `agent run note -- body --detach`
silently detached and dropped the `--` as junk. Suppress hoisting whenever a
literal `--` appears in the prompt. Regression test added.

* fix(status): deadline-ms usage-error + scoped stale_sections + cancel losing remote call (#1984)

Pre-landing review (codex): (1) bare `--deadline-ms` with no value silently fell
through to no-budget/--fast instead of a usage error; (2) thin-client timeout
reported both sync+cycle stale even under `--section sync`, naming a section the
caller excluded (local path was already correct); (3) the section race abandoned
the remote promise locally but didn't cancel the in-flight MCP call — pass the
budget as timeoutMs so the losing side actually cancels. Regression test added.

* v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984)

Bundles the already-reviewed autopilot/supervisor stabilization (#2194 #2227
#1994: cycle split, per-source failure cooldown, fan-out clamp, degraded
supervisor retry, DB-lock live-supervisor detection) with four operational
fixes: minion timeout attempt-accounting (#1737), agent-run trailing-flag
parsing (#1738), honest live-sync sources status + progress-aware stall
watchdog (#1950), and status version + --deadline-ms partial result (#1984).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document GBRAIN_SYNC_STALL_ABORT_SECONDS env knob (#1950)

Post-ship doc sync (/document-release): add the sync stall watchdog env var to
the CLAUDE.md sync-tuning table (Five → Six knobs) + regenerate the llms bundle.

* test: quarantine #2249 fanout tests as *.serial (R1 env-isolation) (#2194)

The cherry-picked autopilot-fanout-clamp + doctor-autopilot-fanout-concurrency
tests mutate process.env.GBRAIN_AUDIT_DIR in beforeEach/afterEach, which the
check:test-isolation R1 lint flags (parallel shards load multiple files per
process). Rename to *.serial.test.ts (sanctioned quarantine — they run under
--max-concurrency=1) instead of restructuring the reviewed test bodies. No logic
change; both files stay green (9 tests). Fixes the failing verify CI check.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 06:36:43 -07:00
Garry TanandClaude Opus 4.8 9bf96db807 v0.42.51.0 fix(sync): contention-free clock + checkpoint integrity + honest sync freshness (#2255)
* fix(sync): contention-free page-generation clock — sequence swap

The page-generation clock backed the query-cache Layer-1 bookmark via a
FOR EACH STATEMENT trigger running `UPDATE page_generation_clock SET
value=value+1 WHERE id=1`. That took a transaction-length RowExclusiveLock
on one tuple, so every concurrent page writer serialized on the prior
writer's COMMIT — sync ran at ~0.8 cores regardless of worker count.

Swap to a SEQUENCE bumped by nextval() (a microsecond LWLock, never a row
lock). The clock's only contract is monotonic advancement on any page
INSERT/UPDATE/DELETE; last_value is non-transactional, so rolled-back or
concurrent-uncommitted writers only OVER-invalidate the cache (lose a hit),
never serve stale.

- migration v118: CREATE SEQUENCE + load-bearing 2-arg setval (is_called=
  true, floor 1, seeded >= old clock and MAX(generation)) + repoint the
  trigger function body + DELETE query_cache so no old-clock bookmark
  survives the swap. v107 left immutable.
- query-cache-gate.ts: 3 readers -> SELECT last_value FROM page_generation_clock_seq.
- schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the
  sequence on fresh install; table + trigger names retained.
- tests: clockValue reads last_value; mechanism proof (trigger fn uses
  nextval not the row UPDATE); rollback-advances-clock safety pin; real
  PGLite sequence round-trip (is_called gotcha); shape test requires _seq.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): op_checkpoints array-shape guard — CHECK + repair + defensive loader

completed_keys is JSONB and the checkpoint loader runs
jsonb_array_elements_text over it. A non-array (scalar) value makes that
throw "cannot extract elements from a scalar", which takes down the whole
UNION load — including the valid op_checkpoint_paths child rows — and loses
all checkpoint progress for that key. No current writer produces a scalar,
but an older binary / external script / future bug could.

Make the corruption class structurally impossible and self-healing:

- migration v119: LOCK TABLE (so an out-of-band scalar can't land between
  repair and constrain; no-op on single-connection PGLite), repair any
  pre-existing scalar to '[]' (op_checkpoint_paths child rows are the
  append-only source of truth, so the reset loses nothing), then add the
  named CHECK (jsonb_typeof(completed_keys) = 'array') via a pg_constraint
  IF NOT EXISTS guard. A DB-enforced always-on guard — the correct pattern
  vs a migration verify-hook, which never runs on already-stamped brains.
- schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the
  same NAMED inline CHECK so fresh installs match migrated brains and v119
  skips the duplicate.
- op-checkpoint.ts loader: gate the legacy arm on jsonb_typeof = 'array' so
  a scalar parent is skipped (children still load) instead of throwing the
  whole union, and log a specific corruption warning when one is seen.
- tests: CHECK rejects a scalar (exactly one constraint, no blob+migration
  dupe); loader survives a scalar parent and returns the children; v119
  repair converts a scalar to '[]'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): report actively-running sync via live lock, not stale freshness

A slow source that makes partial progress every cycle but never fully
completes used to read as permanently "stale" / "never synced" because
last_sync_at only advances on a full successful sync. The naive fix
(treat recent checkpoint banking as "in progress") is unsafe: a blocked
sync banks the good files then writes no anchor, so banking can't tell
in-progress from wedged.

Use the only honest signal: a LIVE, non-expired per-source sync lock
(inspectLock + syncLockId against gbrain_cycle_locks). Every non-skipLock
sync holds it and refreshes it; a blocked/failed sync's process has exited
(no lock row) and a wedged holder stops refreshing (TTL lapses), so either
correctly falls through to the stale path and is NEVER masked. An
actively-syncing source (including a never-synced source doing its first
sync) counts as synced_recently, preserving the pinned 3-bucket invariant.
The lock lookup reuses doctor's existing dynamic db-lock import and swallows
any throw (stub engine, pre-lock-table brain) to false, so it can only ADD
an in-progress verdict, never suppress a real stale one.

Tests (real PGLiteEngine + real lock rows): stale+no-lock -> fail;
stale+live-lock -> ok; never-synced+live-lock -> ok; never-synced+no-lock
-> fail; expired-TTL lock -> fail (wedged not masked); blocked source with
banked checkpoint rows but no lock -> still fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): honest --force-break-lock diagnostic when no lock is held

--force-break-lock used to emit the same terse "Lock ... is not held
(nothing to break)" line and exit 0 even when a sync was genuinely wedged,
sending the operator down a dead end — the wedge was not a held lock. Keep
rc=0 (breaking a non-existent lock is idempotently successful; flipping the
exit code would break automation), but under --force say plainly that
nothing was broken and point at the real next step (gbrain sync / gbrain
doctor) plus a `wedge_hint` field in --json output. The non-force path is
byte-for-byte unchanged.

runBreakLock is exported for the test. Tests: force+no-lock -> wedge_hint
JSON + human hint, rc 0; non-force+no-lock -> unchanged terse line, no hint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): surface the in-progress sync holder in the freshness message

Plan-completion follow-up to the BUG 4 live-lock signal: when a source is
actively syncing, name the holder (pid + host) in the check message instead
of silently folding it into synced_recently. The note is appended only when
something is in progress, so steady-state messages stay byte-for-byte
unchanged (the pinned exact-message + 3-bucket-invariant tests still pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): pre-landing review fixes — monotonic clock seed, scoped CHECK guard

Adversarial (codex) review of the implementation diff caught three:

- P1 (correctness): the fresh-schema setval was not monotonic. initSchema
  replays the schema blob, and the unconditional setval(MAX(generation))
  could move page_generation_clock_seq.last_value BACKWARD on an
  already-upgraded brain, letting a stored query_cache bookmark serve stale
  rows. Seed via GREATEST over the sequence's OWN last_value (+ old table
  value + MAX(generation)) in all 3 fresh schemas and migration v118, so a
  replay is idempotent — mirrors the old table's ON CONFLICT DO NOTHING.
  Pinned by a new monotonic regression test.
- P2: v119's CHECK-exists guard keyed on conname only (not globally unique).
  Scope it to conrelid = 'op_checkpoints'::regclass.
- P3: in-progress note ran into the prior sentence in fail/warn doctor
  messages; separate it with '. '.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: make Anthropic/ZE no-key tests hermetic against a dev config key

These "no key" tests cleared only ANTHROPIC_API_KEY / ZEROENTROPY_API_KEY
from the env, but hasAnthropicKey() and checkZeEmbeddingHealth() also read
the key from ~/.gbrain/config.json. On a dev machine whose real config holds
a key, the no-key assertions flipped and the tests failed locally (they
passed only in key-less CI). Add a shared with-env emptyHome() helper and
point GBRAIN_HOME at an empty dir in every no-key path so loadConfig finds
nothing — matching the already-hermetic anthropic-key / gateway-probe tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.44.1.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): sync doctor + op-checkpoint entries to v0.44.1.0 truth

checkSyncFreshness now reports an actively-running sync via the live
per-source lock (names holder pid+host, counts as synced_recently) instead
of flagging it stale; loadOpCheckpoint gates the legacy union arm on
jsonb_typeof = 'array' so a scalar parent can't take down the whole load,
and migration v119's CHECK constraint makes the corruption class
structurally impossible. Reference docs describe current behavior only —
both entries updated in place, no release-clause appends. Guard + llms
freshness test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: re-version to v0.42.51.0 (natural next-off-master)

Maintainer override of the queue allocator's leap to 0.44.1.0 (it jumped past
in-flight sibling PR claims at 0.42.50/0.43.0/0.44.0). Take the natural next
slot in the 0.42.x line above the immediate sibling claim (0.42.50.0); a
merge re-bump resolves any collision if a cathedral PR lands first.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(e2e): bound + retry the OpenClaw install so a transient npm hang can't burn the Tier 2 budget

The Tier 2 (LLM Skills) job failed at 30m16s — the `npm install -g
openclaw@2026.4.9` step hung on a transient npm/registry stall (orphan
`npm install openclaw` was still running at cancel time) and consumed the
entire 30m job budget that v0.42.50.0 (#2254) introduced. The install
normally finishes in under a minute (Tier 2 is ~4m end to end on master),
so this is flaky-install infra, not a test failure.

Wrap the install in `timeout 120` + a 3-attempt retry loop with an 8-minute
step backstop: a hung attempt is killed in 2 min and retried instead of
eating the whole job. Same bound-the-hang philosophy as #2254's job timeouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 14:02:47 -07:00
Garry TanandClaude Opus 4.8 70d5f36db6 v0.42.50.0 ci: reliability hardening — cancel-superseded + per-job timeouts + actionlint + hermetic E2E env (#2254)
* ci: cancel superseded runs + per-job timeouts on test.yml & e2e.yml

Ports the GH-Actions hygiene gbrain already uses in heavy-tests.yml to the two
hot-path workflows. concurrency cancels a superseded run (keyed on PR number for
pull_request events — fork-safe — with github.ref fallback for push/scheduled);
frees runners and stops a stale-SHA run reporting a flaky failure on an obsolete
commit. Per-job timeout-minutes (test matrix 15, verify 12, serial 15, slow-* 12,
e2e tier1 20 / tier2 30, trivial jobs 5-10) convert a wedged job from a 6-hour
zombie (GitHub's default) into a fast legible fail. fail-fast:false already set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: add actionlint workflow (rhysd/actionlint v1.7.11, SHA-pinned)

Lints workflow YAML on .github/workflows/** changes so a malformed workflow /
bad action ref / missing-permission bug is caught before it ships a broken
pipeline. gbrain edits these workflows often; cheap preventive guard. Mirrors
GStack's actionlint job, SHA-pinned to gbrain's convention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(e2e): scrub operator/agent env before E2E (hermetic runner)

A dev or Conductor shell exports CONDUCTOR_*/MCP_*/OPENCLAW_*/GBRAIN_* overrides
that silently change test behavior, making hermetic E2E non-hermetic and its
failures unreproducible across machines. run-e2e.sh drops those prefixes before
bun starts (denylist — PATH/HOME/TMPDIR/DATABASE_URL survive; GBRAIN_HOME kept
for the existing HOME isolation). Adapts GStack's buildHermeticEnv to gbrain's
shell runner. Verified: a 78-test e2e file passes with DATABASE_URL surviving +
a planted GBRAIN_BRAIN_ID scrubbed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.50.0 ci: reliability hardening — cancel-superseded + per-job timeouts + actionlint + hermetic E2E env

Ports GStack's CI-reliability hygiene to gbrain's hot-path workflows: concurrency
cancel-in-progress (PR-number keyed), per-job timeout-minutes (no more 6-hour
zombie jobs), an actionlint workflow, and an operator-env scrub in run-e2e.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 07:44:08 -07:00
Garry TanandClaude Opus 4.8 7968f84077 v0.42.49.0 feat(pace): native DB-contention pacing for embed/sync backfills (#2240)
* feat(pace): composable DB-contention pacer primitive

createDbPacer/createNoopPacer (concurrency permit + in-band EWMA + jittered
cooperative sleep, abort-throws, fail-open) + named pace-mode bundles
(env>config>bundle, default off) + shared embed-backfill lock key.

* feat(embed): wire DB-pacing into embed paths + single-flight + bounded keyset re-entry

embedStaleForSource + CLI embedAllStale/embedAll lower worker count to the
resolved cap and observe()/pace() their DB ops; embed job + embed-backfill
handler resolve env>config>bundle; CLI --pace flags; --background carries
overrides into the job payload; single-flight via shared per-source lock;
budget-timer re-arm around paced sleeps; EmbedResult.pacing telemetry.

* feat(sync): shared DB-pacer permit across parallel worker engines

One pacer spans the per-worker PostgresEngines (the multi-pool permit case);
observe() import writes, pace() between files, dispose on all exit paths.

* docs(pace): CLAUDE.md Pace Mode section + regenerated llms bundle

* fix(pace): pre-landing review fixes (Codex P1/P2)

- never unref() the cooperative-sleep timer (could exit mid-sleep)
- pace() excludes the wall-clock budget + re-arm after each sleep
- pacing only lowers concurrency, never raises above an operator cap
- serialized job pace resolves at config tier so GBRAIN_PACE_* still wins
- --pace-max-concurrency consumes its value token

* chore: bump version and changelog (v0.42.48.0)

Native DB-contention pacing for embed/sync backfills.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version to v0.42.49.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:58:07 -07:00
Garry TanandClaude Opus 4.8 7ea92d602c v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241)
* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability

Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean,
never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated
GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's
--ff-only contract is unchanged.

* feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules)

hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed
brain-commit-push.sh (one shared push-retry template), repo-scoped credential
with existing-helper reuse, push-probe verify, active-resolver-file rules with
taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction
via redactSecretsInText.

* feat(sources): harden/pull/unharden commands + auto-harden on add --url

sources harden/pull/unharden subcommands; --pat-file/--no-harden on add;
auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect
early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite).

* test(durability): unit + integration coverage for brain-repo durability

git helpers, core harden/unharden, hook+helper E2E (real background push),
cron generators. 41 tests across 4 files.

* chore: bump version and changelog (v0.42.48.0)

Brain-repo git durability: auto-harden a brain's working tree (local auto-push
hook, committed commit-push helper, always-on agent rules, DB-free pull cron,
repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sources): route harden exit code through setCliExitVerdict

A raw process.exitCode write is zeroed by the owned-verdict flush-exit
(#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard
caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports
needs-attention to cron/automation.

* docs: document brain-repo durability (KEY_FILES + multi-source guide)

KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe,
detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add
brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md:
add a Durability (auto-harden) how-to covering sources harden/pull/unharden,
--pat-file, the guarantees, and the security posture.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:41:37 -07:00
Garry TanandClaude Opus 4.8 9d88680a51 v0.42.47.0 feat(skillpack,advisor): brain-resident skillpacks + proactive gbrain advisor (#2180) (#2231)
* feat(skillpack): brain_resident manifest fields + init-brain-pack scaffolder + tools version-skew lint

Add optional brain_resident/schema_pack to the v1 manifest (additive,
forward-compatible). New runInitBrainPack scaffolds a brain-resident pack
(brain_resident:true, exact gbrain_min_version, 5-section machine-parseable
README) beside brain content; applyWritePlan factored out of init-scaffold.
brain-pack-lint validates each skill's declared tools: against the serving op
set (E6 version-skew). Wires gbrain skillpack init-brain-pack.

* feat(skillpack): Topology A brain-pack discovery on sources add + bounded nag

After 'gbrain sources add', if the source ships a brain_resident pack, print an
agent-readable advisory (ask the user before scaffolding). nag-state.ts tracks
declines per (source-repo brain_id, source, pack) with escalate-then-suppress;
declines count only on CLI-interactive displays, never cron/MCP. Fail-open: a
malformed/absent pack never breaks sources add.

* feat(advisor,skillpack): list_brain_skillpack MCP tool + gbrain advisor

Topology B: dedicated source-scoped list_brain_skillpack op + get_skill source_id
disambiguation (brain-resident-locate.ts); git scaffold-spec never a server FS
path; source-aware schema match. LEARN_INSTRUCTION + serve-http banner.

gbrain advisor: read-only ranked actions from brain state (8 resilient collectors,
shared renderer, JSONL history, --json severity exit codes, local-only argv
--apply dispatcher). Exposed over MCP behind mcp.publish_advisor (default off,
read-only on remote; workspace collectors no-op remotely). Generalizes
post-install-advisory to a single current-state recommended set (install→scaffold).

* feat(skills): bundle gbrain-advisor skill + weekly cron recipe + ranking eval

skills/gbrain-advisor teaches a harness to run gbrain advisor on a cadence and
ping the user (read-only; ask before fixing). Registered in manifest.json,
RESOLVER.md, openclaw.plugin.json. E4 ranking-precision eval on seeded-defect
fixtures (100%).

* chore: bump version and changelog (v0.42.47.0)

Brain-resident skillpacks + gbrain advisor (#2180).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync CLAUDE.md + KEY_FILES for brain-resident skillpacks + advisor (#2180)

Skill count 29->30, Skills section gains the brain-resident skillpacks + advisor
capability, KEY_FILES gets current-state entries for the new modules. Regenerate
llms bundles.

* test,fix: align stale assertions with generalized advisory + advisor resolver triggers (#2180)

- post-install-advisory.test.ts: install→scaffold wording (the install verb was
  removed); restore two-column in book-mirror copy; drop the removed skillpack-list line.
- RESOLVER.md: gbrain-advisor trigger now fuzzy-matches a declared frontmatter
  trigger (resolver round-trip D5/C).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: regenerate llms bundle for updated advisor resolver row (#2180)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 22:45:22 -07:00
Garry TanandClaude Opus 4.8 c023a6041d v0.42.46.0 fix(engine): federated read scope reaches by-slug reads (#2200) (#2239)
* fix(engine): federated sourceIds[] scope on by-slug secondary reads (#2200)

getTags/getLinks/getBacklinks/getTimeline (both engines) + TimelineOpts now
accept a federated `sourceIds[]` read grant, precedence over scalar sourceId,
filtering `source_id = ANY($::text[])` — mirroring getPage from v0.42.37.0.

- getTags: `page_id = (subquery)` -> `IN (subquery)` + DISTINCT so a slug present
  in >1 granted source unions tags instead of throwing on a multi-row subquery.
- getLinks/getBacklinks: federated branch scopes ALL THREE page endpoints (from,
  to, AND the authoring origin) so a cross-source link can't disclose a foreign
  slug. Scalar/unscoped branches unchanged (trusted internal callers keep the
  cross-source view).
- getTimeline: Postgres 8-branch cartesian tree collapsed to one fragment-composed
  query; PGLite adds the sourceIds branch to its dynamic WHERE.

* fix(ops): route by-slug reads through the federated source scope (#2200)

get_page resolves tags against the concrete page's source; get_tags/get_links/
get_backlinks/get_timeline route through sourceScopeOpts(ctx) (replacing the
copy-pasted scalar `ctx.sourceId ? {sourceId} : {}`). New linkReadScopeOpts
promotes an UNTRUSTED remote scalar scope to sourceIds[] so legacy/pre-federated
tokens also get all-endpoint link scoping; trusted local CLI keeps cross-source.

* test: federated read scope on by-slug reads + engine parity (#2200)

Per-op federated reads, isolation (out-of-grant -> empty), cross-source decoy
guard, far-endpoint + origin leak guards (F1), same-slug union (D3A), empty-array
contract, scalar-remote promotion (D1), getTimeline date-window after the Postgres
fragment refactor (D5A), and engine-parity arms for all four methods.

* chore: bump version and changelog (v0.42.46.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): document federated by-slug read scope + linkReadScopeOpts (#2200)

The #2200 fix routed get_page tags + get_tags/get_links/get_backlinks/
get_timeline through the federated source scope and added sourceIds[] to the
engine read methods + TimelineOpts. Bring KEY_FILES.md to current state: the
operations.ts entry's sourceScopeOpts read-op list now includes the by-slug
reads and documents the linkReadScopeOpts helper (three-endpoint link scoping +
untrusted-remote scalar promotion); the engine.ts entry notes the by-slug read
methods + TimelineOpts carry the same sourceIds[] federated axis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 22:32:38 -07:00
Garry TanandClaude Opus 4.8 5c49225e4b v0.42.45.0 feat(sync): delta-aware cost estimator — stop wedging the daily cron (#2139) (#2224)
* feat(core): shared computeSyncDelta + spend-posture module (#2139)

sync-delta.ts: ONE implementation of "what changed since last_commit",
consumed by both the sync executor and the inline cost estimator so the
gate's dollar figure can't drift from what the sync imports.

spend-posture.ts: spend.posture config + parseUsdLimit/formatUsdLimit
off-switch parsing (off/unlimited/none → Infinity; undefined at the budget
boundary so ledger rows never serialize null).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sync): delta-aware cost estimator + non-TTY auto-defer + per-source failure acks (#2139)

The inline-embed cost gate was a ~400x phantom: it priced the entire tree
whenever the working tree was dirty (always, on an active brain), then blocked
the daily cron with exit 2. Now:

- performSyncInner + the estimator both route through computeSyncDelta, so the
  estimate mirrors execution (fetch-first delta; dirty-but-caught-up tree → $0).
- shouldBlockSync is posture-aware; non-TTY above floor AUTO-DEFERS embeds to
  capped backfill jobs (exit 0) instead of wedging — single shared
  runInlineCostGate on both --all and single-source paths.
- --full prices delta + stale backlog (full sync sweeps it inline).
- off/unlimited on the cost knobs; tokenmax bypasses the backfill cap (still
  ledgered) but never the cooldown.
- --skip-failed/--retry-failed scoped per source; the D15 parallel refusal is
  lifted (the #1939 ledger is per-source + lock-serialized).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(config): register spend-control keys + validate spend.posture (#2139)

Adds spend.posture + the five previously --force-only spend knobs to
KNOWN_CONFIG_KEYS so `config set` accepts them directly (removes the
archaeology the issue complained about), and rejects invalid spend.posture
values at set time with a paste-ready hint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reindex,enrich,onboard): spend.posture across the remaining cost gates (#2139)

reindex-code: tokenmax makes the cost gate informational; --max-cost accepts
off/unlimited. enrich + onboard --auto: tokenmax lifts the refuse-without-cap
guardrail and runs UNCAPPED (spend still ledgered by BudgetTracker). Explicit
--max-usd always wins over posture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: cost-gate, delta estimator, spend-posture, off-switch coverage (#2139)

New sync-delta + sync-cost-estimate unit suites; rewritten cost-gate serial
tests (auto-defer instead of exit 2, posture, off-switch, format split,
single-source); parseUsdLimit/posture-aware shouldBlockSync; backfill cap-off
+ tokenmax-bypass + cooldown-still-refuses; config known-key acceptance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(spend-controls): single spend-control surface + ref-map + follow-up TODOs (#2139)

New docs/operations/spend-controls.md (every gate, key, default, off switch,
posture interaction); CLAUDE.md reference-map row; two P3 follow-up TODOs
(measured chunk-count gating, per-source defer granularity). llms bundles
regenerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(spend): SSRF-harden estimator fetch + complete off/uncapped across reindex/enrich/onboard (#2139)

Ship-stage codex pre-landing review caught four P1s in the secondary cost gates:

- The delta estimator's fetch-first ran `git fetch` through the plain git()
  helper, bypassing the GIT_SSRF_FLAGS + GIT_TERMINAL_PROMPT=0 hardening that
  real sync uses. Added `fetchRemote()` to git-remote.ts (same flags as
  pullRepo) and route the estimator through it — a cost preview / dry-run can
  no longer hit a remote through a less-protected path.
- `reindex --max-cost off`, `enrich --max-usd off`, `onboard --auto --max-usd
  off` were parsed but didn't actually proceed/uncap. Now: explicit off (and
  spend.posture=tokenmax) proceed past the confirmation/missing-cap refusal AND
  run uncapped. enrich threads an Infinity sentinel mapped to "no BudgetTracker
  ceiling" (never raw Infinity → no null in audit rows); reindex/onboard use
  their native undefined=uncapped path. Spend still ledgered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.45.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(KEY_FILES): update sync/embedding/git-remote/reindex entries to post-#2139 truth

document-release pass: the cost-gate entries described the pre-#2139 behavior
(full-tree-ceiling estimator, --skip-failed-rejects-under-parallel, exit-2
confirmation gate). Updated to current truth — delta-aware estimator via the
shared computeSyncDelta, per-source failure acks under parallel, non-TTY
auto-defer (no exit 2), posture-aware shouldBlockSync. Added entries for the
two new core modules (sync-delta.ts, spend-posture.ts) + fetchRemote on
git-remote.ts + reindex --max-cost off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 15:08:47 -07:00
Garry TanandClaude Fable 5 090bb53203 v0.42.44.0 docs(tutorial): point AlphaClaw deploy link at the official site (#2165) (#2171)
* docs(tutorial): point AlphaClaw deploy link at the official site (#2165)

Step 4 of the personal-brain tutorial linked to the wrong top-level
domain for AlphaClaw. Corrected so the deploy step works as written.

* chore: bump version and changelog (v0.42.44.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 11:38:53 -07:00
Garry TanandClaude Fable 5 a81f7e05e8 v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175)
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)

Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.

Three changes, one contract:

- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
  command is not a daemon, exit deliberately — after stdout AND stderr drain
  (writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
  blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
  regression class; pinned by a 256KB real-pipe subprocess test.

- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
  (op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
  x3, ze-switch, dream, read-only timeout path). Drains the background-work
  registry, then disconnect (best-effort), bounded by the 10s unref'd
  hard-deadline — which is now armed around the TEARDOWN window only, not
  before the op handler (the old placement would have force-killed any op
  slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
  previously skipped the drain entirely and had no hang timer at all.

- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
  paths (status, friction, claw-test, smoke-test, eval cross-modal /
  takes-quality replay / conversation-parser / whoknows-thin, status-thin)
  become process.exitCode + return so they flow through the drains and the
  flush-exit. Pre-engine usage/parse/refusal exits stay as-is.

BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).

DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)

Three consecutive waves (#1972#2015#2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:

- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
  postgres-1, mirroring the production split-pool topology (direct :5432 +
  pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
  password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
  statement_timeout/idle_in_transaction_session_timeout startup params
  gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
  into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
  then spawns the real CLI against the POOLED url and asserts: exit 0,
  stdout intact (the #1959 truncation class), and NO
  "did not return within 10000ms — force-exiting" banner (pre-#2084 it
  printed on 100% of query-shaped ops on this topology). Class bound, not
  exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
  GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.

Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)

One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.

- Migration v116 (idempotent) + mirrors in src/schema.sql +
  src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
  folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
  multi-row parameterized INSERT — never per-row awaited round-trips) +
  purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
  brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
  trigger (the same mechanism that covered v110 page_aliases and v115
  op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
  table pre-creation; writers guard with try/catch).

Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)

- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
  per-turn extractor across the last N turns (oldest→newest), merges by the
  normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
  orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
  cap drops stale assistant chatter, not the entity the user just named.
  Closes the filed assistant-introduced-entities recall TODO; true pronoun
  coreference (never-named antecedents) stays out of scope.

- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
  matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
  lives next to the arm definitions so identity and score can't drift.
  Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
  report which predicate matched). Federated sourceIds[] scope (alias arm
  loops per source; arm 2 uses source_id = ANY — no engine-interface
  change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
  windowing): the legacy title-whole-word rule would suppress every entity
  merely MENTIONED in a prior window turn, breaking the feature by
  construction — slugs only enter context when a pointer/page was actually
  surfaced. Default stays 'slug-and-title' for the window=1 legacy path.

- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
  unprefixed → one user turn), volunteerContext (zero-LLM: extract →
  resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
  cap 3/5; deterministic rationale strings, never raw conversation text),
  and volunteerUsageStats (per-arm/channel precision from the
  last_retrieved_at join, labeled approximate — 5-min throttle false
  negatives, unrelated-read false positives; codex D9).

Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)

New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.

Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).

cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).

Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)

The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.

Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).

Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.

Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(cli): gbrain watch — push transport over stdin (#2095)

The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.

Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).

Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.

Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave

- docs/architecture/KEY_FILES.md (current-state): context entries gain the
  window extractor, arm provenance/confidence, suppression modes, volunteer
  + volunteer-events modules; background-work entry now lists FIVE sinks and
  the drainThenDisconnect owner-disconnect contract; new entries for
  src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
  the confidence model, CLI usage, config keys, and the approximate-stats
  caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
  bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
  check, structured messages[] param); the #1981 entity-detection TODO
  narrowed (window extraction covered assistant-introduced entities +
  named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
  #2084 wave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): truncate context_volunteer_events in setupDB (#2095)

The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)

Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.

Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.

Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).

Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: re-pin the teardown-arming invariant at its post-#2084 home

Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: coverage for ambient reflex-channel logging + watch window/cap flags

Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT

formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin

The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.

Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.43.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: quarantine the watch SIGINT subprocess test to the serial lane

The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.42.43.0

Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:

- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
  now cover prior_context/days and watch's flag surface accurately;
  feedback-log writes described as best-effort; synopsis fence-strip
  described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
  release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
  parallel default, four Postgres services, transaction-mode PgBouncer
  + GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
  dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
  (cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
  suite (volunteer-context, watch-command, watch-sigint.serial,
  cli-format-volunteer), migrate v117 coverage, and the two new E2E
  files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
  pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
  sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
  "opened" for the used-signal, pooler scoped to the local CI gate,
  feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
  regenerated (build:llms) and freshness test green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): correct the v116 reference — the table shipped as migration v117

* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)

Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:

Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.

Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).

Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.

Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.

Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp

Four red-team findings on the #2095 push-context surface:

- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
  cap, so a recurring already-pushed entity burned cap slots every turn and
  starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
  the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
  resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
  only after the response write succeeds, and buildReflexAddition logs only
  after the per-turn timeout admits the block. A block the client's 250ms
  budget abandoned was never injected and no longer counts as volunteered.
  (logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
  can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
  push-context guide, TODO to route watch via serve IPC); host-resolver
  suppression contract at ResolveEntitiesFn (TODO for a capability gate).

Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims

Three CI-only check failures, two root causes:

1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
   loadConfig() returned null (no config file AND no DATABASE_URL — a clean
   CI shard with no brain). loadConfig drops its env→config mapping in that
   case, so the documented escape hatch silently died and the window fell
   back to 4 → windowed extraction widened when the test set window=1 →
   prior-turn entity leaked. Fixed: read the env var directly in
   windowTurnCount, mirroring reflexEnabled's direct process.env read. This
   is a real product bug, not just a test artifact — any config-less host
   using the env hatch was affected. Regression test pins it.

2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
   sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
   and never reset it. The legacy-embedding preload only restores the
   OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
   config survives into the next file — and a file's beforeAll runs BEFORE
   the preload's restoring beforeEach, so federation-health built a
   vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
   test files reshuffled the deterministic file→shard assignment, exposing
   this latent ordering bug. Hardened both victims to establish the gateway
   state they assert (sync-cost-preview resets to the unconfigured fallback;
   federation-health pins legacy 1536 before initSchema) so they're
   order-independent. Verified against a simulated leaker run before them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(context): use withEnv() in the window env-hatch test (test-isolation guard)

The regression test added in 82cc7fff mutated process.env directly, which
check:test-isolation (R1) forbids — use the withEnv() helper that restores on
exit, same as the rest of this file. Behavior identical; guard green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 09:32:58 -07:00
Garry TanandClaude Fable 5 4ee530f3c5 v0.42.42.0 fix(cli): bounded teardown + explicit exit — kill the 10s force-exit tax on txn-mode poolers (#2084) (#2141)
* feat(core): finishCliTeardown + flushThenExit — bounded teardown, owned exit verdict (#2084)

cli-force-exit.ts becomes the single owner of one-shot CLI exit + teardown:
- finishCliTeardown: bounded sink drain -> bounded disconnect under a backstop
  whose deadline is COMPUTED from the bounds it guards (floor 10s;
  GBRAIN_TEARDOWN_DEADLINE_MS env override). Arms at teardown start, never
  before the op handler.
- flushThenExit: stdio write-fence (unref'd guard, EPIPE-safe) + REF'D
  aliveness grace for non-TTY stdio — Bun only delivers queued pipe writes
  while the process is alive (no flush API reaches the native queue).
- setCliExitVerdict/currentExitCode: the exit verdict lives in a gbrain-owned
  channel, never read back from process.exitCode (PGLite's Emscripten runtime
  scribbles its own status there mid-run).
- background-work.ts exports backgroundWorkSinkCount() for the deadline formula.

Unit tests + a spawned-Bun harness proving byte-complete piped output.

* fix(cli): route all nine disconnect sites through finishCliTeardown; one exit seam (#2084)

Deletes the pre-handler 10s force-exit timer (it measured handler + teardown
combined: PgBouncer txn-mode deployments paid a flat 10s banner tax on every
query, and any >10s op was killed mid-run with exit 0 and truncated output).
Sweeps op-dispatch, CLI_ONLY fall-through, search dashboard, read-only timeout
path, dream, doctor x3 (fixing a pre-existing pool leak when DB checks throw),
and ze-switch. The ONE process exit lives in main().then/catch via
flushThenExit(currentExitCode()), gated by shouldForceExitAfterMain().
Exit-code writers (op-dispatch catch, reindex, transcripts, brainstorm,
autopilot, frontmatter) now set the verdict through setCliExitVerdict.

* fix(pglite): contain Emscripten's process.exitCode writes at PGlite.create (#2084)

PGLite's WASM runtime writes its own status into process.exitCode (99 at
create; in-memory brains run initdb whose status lands on a later tick; the
exit status at close) — on PGLite every error exit was silently clobbered.
preservingProcessExitCode wraps create() to keep the global tidy; db.close()
stays unwrapped (its 0-write is baseline behavior test runners depend on).
The CLI verdict itself is immune: it lives in the owned channel.

* test: e2e + structural pins for the #2084 teardown contract

E2E: failed op exits 1; every swept command spawned (brain-copy isolation for
mutators, no-network); slow-handler regression via the deadline env knob;
piped --json parses complete; teardown banner absent on every happy path;
daemon survival untouched. Structural: no bare awaited engine disconnects in
cli.ts; DISCONNECT_HARD_DEADLINE_MS gone; >=9 helper call sites; verdict
channel + create-wrap pins.

* test: fix R1 env-isolation violations in retrieval-reflex tests

Pre-existing on master: both files mutated GBRAIN_RETRIEVAL_REFLEX directly,
failing scripts/check-test-isolation.sh (bun run verify). Converted to the
canonical withEnv() pattern; the reflex describe's beforeEach also never
restored the flag, leaking it across the shard.

* docs: KEY_FILES entries for the teardown contract; close + file TODOS (#2084)

KEY_FILES.md: current-state entry for cli-force-exit.ts (helper + central exit
seam pair, verdict channel, cli.ts-scoped claim); background-work.ts and
pglite-engine.ts entries updated. TODOS.md: the drain-before-owner-disconnect
P3 (filed from #1972) is done by this wave; files the trigger-gated
GBRAIN_COMMAND_DEADLINE_MS follow-up (eng-review D2/D14).

* fix: pre-landing review fixes (#2084)

Review army (testing/maintainability/security/performance, 0 critical):
- drain defense-in-depth: a throwing drain warns and still disconnects
  (cannot escape a caller's finally or skip the engine teardown)
- behavioral tests for preservingProcessExitCode (connect pins 0; create-throw
  restores the pre-call verdict)
- D9 widening test (live-registry sink count feeds the deadline formula),
  env 0/negative boundary cases, verdict mirror-write assertion
- stale comments: header diagram backstop line, structural-test 'both
  lifecycle calls' contradiction, KEY_FILES 10s-force-exit clauses, e2e D11
  falsification story corrected
- named the formula's pool-end literals

* fix: adversarial-review hardening — daemon-safe command resolution, flush knob, ref'd backstop (#2084)

Cross-model adversarial review (Claude subagent + Codex, both P1'd it):
- shouldForceExitAfterMain now resolves the command through parseGlobalFlags —
  the old first-non-dash heuristic read `gbrain --timeout 30s serve` as
  command "30s" and the new exit seam would have killed the daemon ~250ms
  after boot with exit 0 (unit-pinned)
- GBRAIN_FLUSH_GRACE_MS env override for the non-TTY aliveness grace (batch
  consumers piping large payloads to slow readers can raise it; agent loops
  can lower it)
- backstop timer is now REF'D: a hung teardown on an otherwise-empty event
  loop previously exited naturally — skipping the flush and surfacing
  PGLite's scribbled process.exitCode
- flushThenExit: real process.exit latched once per process
- doctor-site comment corrected; in-command process.exit teardown-bypass
  class (pre-existing) filed as a P2 TODO

* chore: bump version and changelog (v0.42.42.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: move #2084 exitCode-containment lifecycle tests to the serial quarantine (R3)

* docs: update project documentation for v0.42.42.0

- docs/TESTING.md: replace the stale 4-file serial-quarantine enumeration
  with a current-state description (the quarantine is glob-discovered, now
  several dozen files incl. the #2084 exitCode-containment suite); add unit
  inventory entries for test/cli-finish-teardown.test.ts and
  test/flush-then-exit-harness.test.ts.
- docs/architecture/KEY_FILES.md: rephrase the pglite-engine exitCode
  containment note to current-state wording (clears the
  check-key-files-current-state prose-history warning).

llms bundles regenerated (byte-identical: both docs are link-only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: apply cross-model doc-review findings for v0.42.42.0

Codex review of docs-vs-shipped-code found 9 gaps; all verified against
the code before fixing:

- CHANGELOG.md (0.42.42.0 entry, precision narrowing only — no entries
  touched): "every CLI exit path" -> "every cli.ts disconnect site";
  "on every path" -> "on every routed exit path"; dream/doctor/ze-switch
  claim scoped to dispatcher teardown (command-internal process.exit
  sites are tracked in TODOS as the open P2).
- docs/architecture/KEY_FILES.md: the teardown backstop is REF'D, not
  unref'd (matches the F3 adversarial-review decision in the code).
- src/core/cli-force-exit.ts: header diagram comment had the same stale
  unref'd claim + `process.exitCode ?? 0`; now matches the implementation
  (ref'd timer, `currentExitCode()`). Comment-only change.
- docs/TESTING.md: verify is the 30-check parallel battery via
  run-verify-parallel.sh (was described as 4 checks); CI is 10 weighted
  LPT shards + dedicated verify/serial/slow jobs (was "4-way FNV on
  shard 1"); test:serial runs one bun process per file (not
  --max-concurrency=1); dead "cap: 10" line rewritten as debt guidance;
  inventory entries added for test/cli-should-force-exit.test.ts and
  test/e2e/pglite-cli-exit.serial.test.ts.

bun run verify green (30/30); #2084 test files green; llms bundles
regenerated (byte-identical — reference docs are link-only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: route v0.42.41.0's raw exitCode writers through the verdict channel; reconcile merged structural pins (#2084)

CI fallout from merging the v0.42.41.0 triage wave into the #2084 exit-seam
design — both waves fixed the same timer-placement bug independently:

- doctor.ts + extract.ts set failure exit codes via raw `process.exitCode =`
  writes (v0.42.41.0's process.exit -> exitCode conversion); the #2084 exit
  seam reads only the gbrain-owned verdict channel, so doctor FAILs exited 0
  (Tier 1 RLS e2e + half-migrated-Minions tests). Converted to
  setCliExitVerdict, same as the wallclock-124 site in the merge commit.
- cli-force-exit-teardown-arming.test.ts pinned v0.42.41.0's inline
  finally-armed timer, which the merge replaced with finishCliTeardown;
  rewritten to pin the merged invariant (no pre-try arming in cli.ts; the
  backstop arms inside the helper before the drain).
- eval-capture drain timing bound 1s -> 2s: flaked at 1023ms under CI shard
  load after the new test files shifted LPT shard packing (13x budget slack
  still proves bounded-not-hung).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 07:28:13 -07:00
7c27fa129b v0.42.41.0 fix: triage wave — 6 data-loss/availability fixes + 9 community PRs (#2128)
* fix(oauth): default omitted authorize scope to client's full grant

When a client omits `scope` on /authorize, the authorize() grant computed
`(params.scopes || []).filter(...)` → the empty set. That empty grant was
written to oauth_codes and propagated into the access AND refresh tokens, so
every request failed `insufficient_scope` even though the client was
registered with e.g. `read write`. Because refresh inherits the stored grant,
it never self-healed — reconnecting just minted another empty-scoped token.

Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize,
so they hit this on every connection.

Fix: when no scope is requested, default to the client's full registered scope
(RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials,
which already does `requestedScope ? ... : allowedScopes`. The result is still
clamped to the allowed set, so an explicit over-broad request cannot escalate.

Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty →
inherits full grant; explicit subset honored; clamp preserved (over-broad and
disallowed-only requests cannot escalate or trigger inheritance).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): skip Python venv/ in the code walker

collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES
set skipped node_modules but not Python venv/. On a Python repo the walker
descended into venv/ (thousands of files); the resulting slug collisions
crashed putPage's INSERT ... ON CONFLICT ... RETURNING with
"undefined is not an object (evaluating 'row.deleted_at')".

Add `venv` alongside node_modules in both the import.ts inline skip and
PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400)

dimsProviderOptions() threads input_type ('query' | 'document') into
providerOptions.openaiCompatible for asymmetric models (ZE zembed-1,
Voyage v3+), but the AI SDK's openai-compatible adapter validates
providerOptions against a fixed schema and silently drops the field
before building the HTTP body. Every embedQuery() was therefore encoded
document-side: the ZE shim's hard default fired ('document'), Voyage and
local openai-compat servers got no input_type at all, and asymmetric
retrieval silently collapsed toward surface-token overlap — while the
providerOptions-level contract test stayed green.

Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in
embedSubBatch() only when providerOptions actually threads an
input_type, read at body-rewrite time by the fetch shims:
- zeroEntropyCompatFetch: recovers the threaded value; document default
  preserved for ingest paths.
- voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only
  when threaded; the field stays off the wire otherwise.
- NEW openAICompatAsymmetricFetch: fallthrough default for every other
  openai-compatible recipe (llama-server, litellm, ollama, ...) — the
  canonical local/proxy paths for asymmetric models. Strict pass-through
  when nothing was threaded, so symmetric deployments see zero wire
  change; recipes with their own compat fetch (azure) keep it via the
  compat.fetch ?? precedence.

KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on
document-side query vectors; pre-fix rows must not be served to post-fix
lookups (same convention as the v=3 embedding-provider bump). One-time
global cold-miss on upgrade; refills within cache.ttl_seconds.

Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport
with a mocked global fetch and asserts on the outbound body — the only
layer where this regression is observable. Covers ZE hosted, llama-server,
litellm, ollama (query + document sides) and pins the pass-through for
non-asymmetric models and Voyage's opt-in shape. 4 of the original 7
assertions fail on master, proving the pin. One structural pin in
test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape
(same semantic); KEY_FILES.md gateway.ts entry updated to the new truth.

Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to
Voyage + all openai-compatible recipes. Credit to @billy-armstrong for
the original diagnosis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sync): honor .gitignore in code walk; prune vendor/dist/build

collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its
own directory skip list inline (node_modules || ops), bypassing the canonical
pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this
descended into vendor/ (~50k Composer files), storage/, and public/build/,
trying to import 52k dependency/build files and flooding the index with
library internals (a 35-min sync that never finished, killed by the watchdog
at 3%).

- collectSyncableFiles now enumerates via `git ls-files --cached --others
  --exclude-standard` when dir is a git work tree, so the walk honors
  .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for
  non-git dirs. EroLab: 52164 -> 1028 files.
- The FS fallback now prunes through the canonical pruneDir() instead of a
  drifted inline list, so the two skip lists can't diverge again.
- PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees).

Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the
maintainer's #1942 vendor/dist/build prune. Walker regression suites
(sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule)
green: 90 pass.

* fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427)

Bun merges .env files from the process cwd into process.env before any
user code runs. loadConfig() prefers env DATABASE_URL over
~/.gbrain/config.json, so any gbrain invocation from inside a web-app
checkout silently retargets the brain at that app's database — reads go
to the wrong DB and apply-migrations can write gbrain's schema into a
production app database (#427).

effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from
cwd and treats a DATABASE_URL whose value matches one of them as
file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL
and genuinely exported DATABASE_URLs are honored unchanged, so the
operator escape hatch and the e2e suite's env-provided URL keep working.
Applied at loadConfig, getDbUrlSource (doctor parity), init
--non-interactive, and migrate --to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body

The 10s force-exit timer in the shared-op dispatch was armed BEFORE the
try block, so any op whose handler ran past 10s wall-clock was killed
mid-flight with process.exit(0) and zero stdout. On a slow Postgres
pooler (6-10s per fresh connection) a healthy `gbrain search` was
force-exited every time — an empty 'success' indistinguishable from no
results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires
before any error path sets exitCode.

Move the arming into the finally (teardown entry), matching the
fall-through owner-disconnect site later in main(): the timer still
bounds a hung drain/disconnect (the C13 contract) but can no longer
kill a slow-but-progressing op. Verified on a transaction-pooler
Supabase brain: search went from 0 bytes/exit 0 at 10s to real results
at ~21s.

* fix(import): stamp source_id on extracted call-graph edges

importCodeFile built CodeEdgeInput rows without source_id, so every
edge landed NULL. getCallersOf/getCalleesOf filter
`AND source_id = <scoped>` whenever a worktree pin or --source is in
play — NULL never matches, so scoped call-graph queries silently
returned 0 rows on multi-source brains even though the edges existed
(2,122 edges, 26 targeting the probed symbol, count 0 returned).

One-line fix: carry the sourceId already in scope into the edge input.
Existing NULL rows backfill with:
  UPDATE code_edges_symbol e SET source_id = p.source_id
    FROM content_chunks c JOIN pages p ON p.id = c.page_id
   WHERE c.id = e.from_chunk_id AND e.source_id IS NULL;
(same for code_edges_chunk). Verified: code-callers returns 21 callers
where it returned 0.

* docs(migrations): NULL embeddings BEFORE the column-type alter

The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the
UPDATE that clears stale embeddings. pgvector refuses to cast existing
vectors across dimensions ('expected 1024 dimensions, not 1536'), so
the recipe as written aborts the transaction on any brain that has
embeddings — which is every brain doing this migration. Swap the steps:
NULLs cast fine.

* fix: honor legacy token source grants in oauth

* fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review)

With the hard-deadline timer correctly scoped to teardown, a genuinely
wedged read handler (hung pooler connection mid-query) would hang the
CLI forever — the #1633 zombie class the old pre-try timer accidentally
bounded at 10s. Reads now get a generous withTimeout (180s default, far
above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with
the teardown finally still draining + disconnecting). Writes/admin stay
unbounded: a long import/embed must never be killed by a default.

* fix(import): stamp unscoped edges 'default', matching the pages-table default

Review catch: 'sourceId ?? null' fixed the scoped path but left the
unscoped one (reindex --code without --source, importCodeFile callers
without opts.sourceId) stranding edges at NULL while their pages land
under the schema default (pages.source_id DEFAULT 'default') — so
getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug,
other door. Fallback is now 'default'.

* fix(core): runtime dim-migration recipe NULLs embeddings before the alter

Review catch: the doc fix corrected docs/embedding-migrations.md, but
embeddingMismatchMessage still PRINTED the broken order — ALTER before
UPDATE ... SET embedding = NULL — and linked to the now-contradicting
doc. pgvector refuses to cast existing vectors across dimensions, so
the printed recipe aborted on any brain that has embeddings. Swap the
steps and say why inline.

* feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified

1. Backfill: edges written before the stamping fix sit at source_id=NULL
   and stay invisible to scoped call-graph queries until repaired. Derive
   each edge's source from its own from_chunk's page (pages.source_id is
   NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge
   production brain.
2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified,
   which had no index — every callee lookup was a seq scan, amplified
   per-BFS-node by the recursive code walk. With NULL edges repaired,
   scoped walks actually expand, so the latent cost becomes real.
   Mirrored into src/schema.sql; schema-embedded.ts regenerated.

* docs(migrations): align the rationale list with the corrected recipe order

The 'Why we don't do this automatically' list still said alter-then-wipe;
reorder to wipe-then-alter and replace the fragile 'step 3' numeric
cross-reference with a name-based one.

* test: regression coverage for edge source_id stamping, timer placement, recipe order

- import-code-edges-source-id: scoped import stamps edges + scoped
  getCallersOf/getCalleesOf match (verified failing pre-fix), plus the
  unscoped-import case asserting 'default' stamping.
- cli-force-exit-teardown-arming: structural pin — the hard-deadline
  timer arms inside the finally (teardown entry), never before the op
  body; daemon guard, unref, clearTimeout intact.
- embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so
  the printed SQL can't drift from docs/embedding-migrations.md again.

* fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too

Adversarial review, two findings on the new timeout path:
1. On timeout the finally drained, disconnected, then CLEARED the
   hard-deadline timer — removing the only backstop while the abandoned
   handler (withTimeout races, it does not cancel) can hold ref'd
   sockets/SDK timers that keep Bun's loop alive: 'timed out' printed,
   process immortal — the zombie class this branch exists to kill,
   resurrected through its own fix. The finally now exits explicitly
   after teardown completes on the timeout path.
2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat
   outside any bound — a pooler wedge at context build hung reads,
   writes, and admin alike. It now shares the same wallclock bound.

* fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out

Adversarial review: txOpts used truthiness while the edge stamp used
nullish — sourceId:'' put pages under 'default' but stamped edges '',
FK-violating against sources(id) and silently dropping the file's whole
call graph in the best-effort catch. The unscoped getChunks could also
fan out to same-slug chunks from another source. One normalized
edgeSourceId (sourceId || 'default') now drives both the chunk lookup
and the stamp.

* fix(engine): default edge source_id to 'default' at the insert layer (both engines)

Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any
future caller that forgets the field reintroduces invisible NULL edges
the day after the v116 backfill runs. A NULL source_id is invisible to
every scoped call-graph query; default to the schema-default source the
way the pages table does. Applied to both engines (parity).

* fix(core): facts alter recipe NULLs embeddings before cross-dimension alters

Adversarial review: buildFactsAlterRecipe shipped the same defect class
this branch fixes for content_chunks 350 lines up — a cross-dimension
ALTER ... USING cast that pgvector refuses while rows hold old-width
vectors. Dimension changes now wipe first (the facts pipeline re-embeds
on next write); same-dim type swaps (halfvec <-> vector) keep the
lossless cast and PRESERVE data. Both behaviors pinned by tests.

* v0.42.39.0 chore: version bump + CHANGELOG + TODOS

Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up
complete — this branch ships exactly that decoupling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race

## Problem

`updateSourceConfig` used a read-then-write pattern: read the current
`config` row, normalize it in JavaScript, then write the merged result
back with `SET config = <normalized> || <patch>`.

Under concurrent callers (two background autopilot/cycle paths patching
different keys simultaneously), both callers can read the same stale
row. The later `SET config = ...` then clobbers the earlier patch,
silently dropping whatever keys the first caller wrote. Reproduced
at 21/25 lost-update events under real Postgres with parallel callers.

## Fix

Fold the normalization and merge into a single atomic `UPDATE … SET
config = CASE … END || patch` statement. Because the `SET` expression
evaluates against the row-locked latest version of `config`, there is
no snapshot window between the read and the write. Concurrent callers
now converge correctly (50/50 clean in reproduction test).

The `CASE` also normalizes historical bad JSONB shapes inline:
- `object` — used as-is
- `string` — double-encoded config; inner text parsed with the SQL
  `IS JSON` guard (Postgres 16+) so unparseable strings fall back to
  `{}` instead of raising `invalid input syntax for type json`
- `array` — array of patch objects aggregated into a flat object via
  `jsonb_object_agg`
- anything else — falls back to `{}`

`pglite-engine.updateSourceConfig` already used an atomic `||` merge;
this change brings postgres-engine to parity.

## Test

Added two assertions to `test/list-all-sources.test.ts`:
1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw)
2. JSONB string holding double-encoded valid JSON is parsed then merged

* fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard

## 1. Stale lock break hints cover gbrain-cycle: keys

The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes;
everything else fell back to `gbrain sync --break-lock`, which is wrong for
dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle`
lock now suggests `gbrain dream --break-lock [--source <name>]`, and
unknown lock shapes fall back to `gbrain doctor` instead of a
misleading sync command.

## 2. content_sanity_audit_recent counts reject and quarantine as hard failures

v0.42 renamed the hard disposition path: rejected pages emit a `reject`
event and quarantined junk pages emit `quarantine`; `hard_block` is now
only the pre-v0.42 legacy alias. The status check only counted `hard_block`,
so fresh `reject` / `quarantine` events from the new path cleared as `ok`
whenever fewer than 10 events existed. The check now sums all three for the
hard count, and `soft_block + flag` for the soft count.

## 3. graph_coverage excludes test fixture entity pages from the denominator

Brains seeded with code sources (e.g. a sync of the gbrain repo itself)
could accumulate test fixture pages typed as `entity` / `person`. Including
these in the entity-count denominator diluted coverage and produced spurious
warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains
with no real entity pages. The check now queries a per-entity stats CTE that
excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub,
with an additional guard for the all-fixture case (`eligibleEntityCount = 0`).

## 4. process.exitCode instead of process.exit at doctor main exit point

`process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup
handlers (Bun unload events, open DB connections) from running. Using
`process.exitCode = hasFail ? 1 : 0` defers the actual termination until
the end of the event loop, allowing cleanup to complete.

## 5. checkSubagentCapability exported for test seams + gateway loop guard

The function was private, making it untestable in isolation. It is now
exported. Additionally, users running gbrain with a non-Anthropic chat model
via `agent.use_gateway_loop=true` no longer receive a spurious warning that
`ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in
that configuration and do not need the key directly.

## Tests

Doctor test suite: 77 pass, 0 fail (no regressions).

* fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034)

Engine-layer API for two cycle/availability fixes that share these files:
- deleteFactsForPage gains optional excludeSourcePrefixes so the fence
  reconcile can protect non-fence facts (e.g. cli: conversation facts).
- reconnect(ctx?) is now a first-class BrainEngine method on both engines
  (PostgresEngine already had it; PGLite gains config capture + reconnect)
  so callers stop using disconnect()+bare connect().

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cycle): stop extract_facts from wiping conversation facts (#1928)

The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to
recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0
reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit
the failed-sync->full-walk fallback for this destructive phase, and warn on
net-negative reconcile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034)

The autopilot health-probe recovery called connect() with no args after
disconnect(), losing the startup config (database_url undefined -> FATAL
restart-loop on every DB blip) and opening a null-pool window. Both call sites
now use engine.reconnect(), which restores the captured config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018)

put_page write-through resolved the disk target from the global sync.repo_path,
so a default-source page (local_path NULL) got written into an unrelated
federated source's working tree. Now it uses the assigned source's own
local_path; NULL local_path skips (no leak); the global path is used only as a
sole-source fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058)

A live holder's lock was force-removed after 5min age alone, letting a second
process share the single-writer data dir -> WAL corruption. The lock now
heartbeats while held; a holder is reaped only when its PID is dead OR its
heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat
age to also defeat PID reuse.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038)

A migration renumbered during a merge (v102) could be recorded-as-applied
without its DDL running, leaving the 3-column index so every timeline write
failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed
drift repair (dedupe-then-rebuild) even when no migration is pending, and
doctor surfaces the drift.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057)

The meetings extractor's bare catch {} hid a brain-wide timeline-write failure
(0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing
batch regression test proving the #1861 jsonb_to_recordset refactor already
fixed the original ::text[] cast failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.41.0)

Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018
write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline
migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052
#2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: address adversarial review findings (#1928, #2058, #2038, #2057)

Codex as-built review of the authored fixes surfaced 4 real issues:
- #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced
  past the grace must NOT let its resumed heartbeat refresh, nor releaseLock
  remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat
  and release now verify the on-disk lock is still ours. + regression test.
- #1928: the destructive-full-walk guard keyed off phases.includes('sync'),
  which wrongly suppressed a legitimate full reconcile when sync was SKIPPED
  (no engine / no brainDir). Key off a syncAttempted flag set only when sync
  actually ran.
- #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with
  the existing v-migration lower-id rule.
- #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1)
  instead of printing a clean success over failed inserts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(key-files): sync reference to v0.42.41.0 triage-wave behavior

Update KEY_FILES.md to current-state truth for the shipped fixes (no
release-history clauses, per the reference-doc discipline):

- write-through.ts (#2018): resolves the disk target from the assigned
  source's own local_path; sole-source falls back to sync.repo_path,
  multi-source skips with source_has_no_local_path rather than leak.
- engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on
  both engines; config-restoring, never disconnect()+bare connect().
- migrate.ts (#2073): document v116 edge source_id backfill + callee
  index, and the always-run (version-counter-blind) timeline dedup
  self-heal.
- new entry for timeline-dedup-repair.ts (#2038) + the
  timeline_dedup_index doctor check.
- new entry for pglite-lock.ts (#2058): heartbeat + steal-grace
  (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never
  stolen.
- extract-facts.ts (#1928): cli:-fact protection, no failed-sync
  full-walk inheritance, net_fact_deletion warn floor.

bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged);
freshness + current-state guards green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard

The first #2018 fix skipped any no-local_path source on a multi-source brain,
which broke the legitimate nested layout (a source without its own tree nests
under the host repo at .sources/<id>/ — pinned by put-page-write-through.test).
Narrow the guard: a no-local_path source nests under sync.repo_path as before;
only SKIP when sync.repo_path is literally another source's own local_path
(the actual leak — writing there pollutes that sibling's repo). Caught by the
sharded suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: satisfy test-isolation guard for the new lock/reconnect tests

CI `verify` flagged 3 intra-process isolation violations in the tests added
this wave (the parallel runner shares one process per shard):
- pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now
  goes through withEnv() instead of a raw process.env write (R1).
- pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines
  to exercise the connect/reconnect lifecycle, which doesn't fit the shared
  beforeAll-engine model (R3/R4).
verify is now 30/30; both files green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): reconnect() is a no-op for in-memory engines (#2034)

CI serial-tests + test(5) caught two in-branch regressions from the #2034
PGLite reconnect():
- worker/queue claim-error recovery + their renewLock e2e test assume PGLite
  reconnect is absent/no-op (queue.ts documents it). Making it a real
  disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now
  no-ops for in-memory (no database_path) — file-backed still re-opens the dir
  (state persists on disk). Restores the documented worker assumption.
- connection-resilience 'Supervisor still has the 3-strikes-then-reconnect
  path' pinned the removed unsafe-cast text; updated to assert the direct
  this.engine.reconnect() call.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: quarantine embed-input-type-wire to serial lane (CI test(5) leak)

#2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway;
the active dimension survived into engine-find-trajectory when CI's 10-way
hash-disjoint sharding co-located them (this branch's added files reshuffled the
assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not
1536'. resetGateway() in afterEach clears the gateway but the dimension still
leaked. It mutates global gateway/embedding state, so it belongs in the serial
lane (own bun process, true isolation) by the repo's own definition. Root-caused
by reproducing the exact failing pair locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Austin Arnett <austin@sdsconsultinggroup.org>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Dave MacDonald <djmacdonald@ucdavis.edu>
Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com>
Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com>
Co-authored-by: Garry Tan <bo.m.liu@gmail.com>
Co-authored-by: jbarol <barol.j@gmail.com>
Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com>
Co-authored-by: PAI <pai@scaffolde.ai>
2026-06-12 06:05:34 -07:00
Garry TanandClaude Opus 4.8 ecd6ae8772 v0.42.40.0 fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011) (#2031)
* fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011)

excerpt() in link-extraction.ts sliced the link-context window by raw UTF-16
index, so a boundary landing inside a non-BMP char (emoji, math, CJK) left an
unpaired surrogate half in `context`. Serialized to JSONB for the
jsonb_to_recordset batch insert, Postgres rejects it at the ::jsonb cast and
aborts the whole batch — wedging `extract --stale` because the staleness
bookmark only advances on a clean finish.

- text-safe.ts: new ensureWellFormed() (Bun isWellFormed/toWellFormed) — one
  shared surrogate-cleaning primitive.
- link-extraction.ts: excerpt() well-forms the slice (root-cause fix).
- batch-rows.ts: new sanitizeForJsonb() = ensureWellFormed(stripNul(s)) applied
  to every free-text body field (link context; timeline summary/detail/source;
  take claim/source). Identity/security fields stay un-sanitized and fail closed.
- postgres-engine.ts + pglite-engine.ts: scalar addLink + addTimelineEntry use
  sanitizeForJsonb too, matching the batch path on both engines.
- brainstorm/orchestrator.ts: consolidate hand-rolled sanitizeUnicode onto
  ensureWellFormed (also fixes consecutive-lone-surrogate mishandling).

Tests: ensureWellFormed unit cases (incl. consecutive lone surrogates), an
excerpt window-split regression, PGLite + Postgres-e2e surrogate cases across
all free-text fields and scalar paths, and fail-closed identity-field tests
proving sanitization was NOT extended to slugs/holders.

* v0.42.39.0 chore: bump version and changelog (#2011)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.42.39.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.40.0 chore: re-slot release version (was 0.42.39.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: fix env-mutation isolation violations in retrieval-reflex tests

check:test-isolation (rule R1) flags direct process.env mutation in
non-serial test files — bun's parallel runner loads multiple files into
one process, so a leaked GBRAIN_RETRIEVAL_REFLEX flips reflex behavior
in unrelated tests. Both files landed via the #2019 merge; convert the
beforeEach/afterEach env juggling to the canonical withEnv() wrapper,
which restores the prior value via try/finally even on throw.

Fixes the failing `verify` CI check on #2031 (and the `test-status`
aggregate that inherits it). All 30 verify checks green locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 22:00:36 -07:00
Garry TanandClaude Opus 4.8 8f45624e55 v0.42.39.0 feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) (#2019)
* fix(integrations): parameterize resolver-row fence by recipe id

The install fence was hardcoded gbrain:agent-voice:resolver-rows, so any
second copy-into-host-repo recipe wrote a mislabeled block (and refresh/
uninstall keyed on recipe id would miss it). Derive it from manifest.recipe.

* feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981)

Deterministic per-turn pointer layer in the context engine: a zero-LLM,
precision-biased scan resolves salient entities (names, @handles) to existing
brain pages and injects compact pointers (name → slug → safe synopsis). Detect
+ point, never auto-dump. Fail-open, capped, suppression on prior context only.

Engine-aware resolver ladder (no second DB connection): host ctx.brainQuery →
PGLite serve resolve IPC (unix socket) → Postgres cached direct → disabled.
Synopsis runs through get_page's privacy strip. Plus the retrieval-reflex recipe
+ policy skill, the retrieval_reflex_health doctor check, config gate, and the
init next-step hint.

* chore: bump version and changelog (v0.42.38.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(architecture): KEY_FILES entries for Retrieval Reflex surface (#1981)

Document the new src/core/context/ modules, the context-engine resolver
ladder, the serve resolve IPC, the retrieval_reflex_health doctor check,
and the recipe-id-keyed install fence. Current-state only.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 21:12:33 -07:00
Garry TanandClaude Opus 4.8 03ffc6ebdb v0.42.37.0 fix(jobs): reap stale locks, bound disconnect, complete cooperative-abort (#1972) (#2015)
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972)

A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row
until something contended for it — reclaim was on-contention only. Add a
host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder
PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle*
namespaces only (never elections/supervisor/reindex), with a snapshot-matched
delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse.
Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now
auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now
back inspectLock + listStaleLocks (killed the triplication).

* fix(db): bound pool disconnect so teardown can't eat CLI output (#1972)

pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. #1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.

* fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972)

v0.42.29 made only the embed phase honor the abort signal; a 24h pull still
showed force-evicts from a long non-embed phase ignoring it. Thread the signal
into every cycle-reachable long loop: extract (extractForSlugs + the full-walk
extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop +
embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket
loop. Add a terminal abort check so an aborted cycle never stamps
last_full_cycle_at as a completed run (Codex #9). lint now yields + checks
abort every 200 pages (it's synchronous; the yield is what lets the signal
land). New phase-duration force-evict attribution log names any phase that
crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start.

* chore: bump version and changelog (v0.42.38.0)

#1972 — stale-lock reaper, bounded pool disconnect, and complete
cooperative-abort coverage across cycle phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage

document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY),
db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/
extract_facts/consolidate/lint + terminal guard) entries to current truth.

* test(isolation): fix shard-order flakes exposed by #1972's new test files

Adding 3 new test files reshuffled the hash-based shards, exposing two
pre-existing test-isolation bugs:

- cycle-consolidate.test.ts assumed the global legacy-embedding preload's
  1536-d gateway config still held at initSchema, but a co-sharded test that
  calls resetGateway() in teardown nulls it, so initSchema fell back to the
  1280-d default and built a halfvec(1280) facts column its 1536-d fixtures
  can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern
  legacy-embedding-preload.ts documents for 1536-d fixture tests).
- db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated
  process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation
  rule R1. Convert to withEnv().

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 22:28:44 -07:00
Garry TanandClaude Opus 4.8 1eb430a2df v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (#1999)
* fix(security): scope cross-source reads to the caller grant; close get_page exact-path leak

One shared resolveRequestedScope() routes every source-scoped read op
(query, code_callers/callees, search_by_image, code_blast/flow, get_page)
through a single fail-closed trust+grant ladder: a remote caller's __all__
collapses to its granted sources (never the whole brain) and an explicit
out-of-grant source_id is rejected. get_page's exact-match path now honors a
federated grant via getPage(sourceIds[]) in both engines. Legacy bearer tokens
carry their stored permissions.source_id grant (bounded, never widened). Also
retries getConfig on transient connection loss.

Closes #1924, #1371, #1393, #1336, #1603.

* fix(ingest): non-string frontmatter no longer aborts lint/sync; embed/hook/catalog papercuts

Parser coerces a non-string title to a string and falls back to inference for
slug/type (never fabricating a "123" slug), with a lint NON_STRING_FIELD finding
surfacing the malformed frontmatter; a defensive guard in content-sanity stops a
non-string title from crashing the whole lint/sync run brain-wide. Plus: embed
--catch-up no longer arms the overflowed 32-bit budget timer (and surfaces
unembeddable chunks); the frontmatter pre-commit hook ships a correct .md/.mdx
regex; and the skill catalog parses YAML block-scalar descriptions.

Closes #1883, #1658, #1556, #1948, #1946, #1840, #1711.

* v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: add NON_STRING_FIELD frontmatter validation class to docs for v0.42.37.0

The v0.42.37.0 non-string-frontmatter fix added an eighth validation
class (NON_STRING_FIELD / lint code frontmatter-non-string-field). Update
the two current-state docs that enumerate the validation classes:
- skills/frontmatter-guard/SKILL.md (seven->eight + table row)
- docs/integrations/pre-commit.md (seven->eight + table row)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 21:19:25 -07:00
Garry TanandClaude Opus 4.8 959af1068d v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) (#1980)
* fix(retry): match EMAXCONNSESSION + SQLSTATE 53300 as retryable conn errors (#1794)

* feat(schema): add op_checkpoint_paths append-only delta table (migration v115) (#1794)

* refactor(op-checkpoint): append-only deltas via executeRawDirect + withRetry (#1794)

* fix(sync): resumable-checkpoint durability + lock-thrash fix (#1794)

Durable append-only checkpoint writes (executeRawDirect + retry), fail-loud
consecutive-failure abort, first-file/10s flush cadence, race-safe pending-delta
under parallel workers, guaranteed final flush on every exit path incl. SIGTERM
(no-retry one-shot via registerCleanup), bankedFiles/reason observability,
event-loop yield to keep the lock heartbeat alive, and routing the bare
(no-source) sync through withRefreshingLock.

* fix(db-lock): heartbeat-aware takeover + direct-pool refresh (#1794)

* fix(cycle): treat SyncLockBusyError as skip, not a phase failure (#1794)

* docs(sync): document the 5 checkpoint/lock env knobs (#1794)

* v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): update sync.ts + op-checkpoint.ts entries to resumable-checkpoint current state (#1794)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 06:16:34 -07:00
Garry TanandClaude Opus 4.8 612753f318 v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) (#1975)
* v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970)

When a source's history is rewritten (force-push, master→main consolidation,
squash), the recorded last_commit can fall outside HEAD's history. The old
guard sent both "object missing" and "not an ancestor" to performFullSync — a
full repo re-walk that never advances the bookmark under a cron timeout on a
large cross-region brain, so the source goes silently stale.

Fix: only a truly-absent object forces a full reconcile. A present-but-non-
ancestor bookmark is diffed tree-to-tree directly (git diff A..B needs no
ancestry), importing only the real delta. Adds: oversized-diff fallback to full
reconcile (F-B); performFullSync now purges deleted files, gated to file-backed
pages by source_path so manual/put_page and metafile pages are spared (F-A);
rename-to-unsyncable deletes the stale old page (F-C). 7 new PGLite e2e tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(architecture): record #1970 sync bookmark recovery + full-sync delete reconcile in KEY_FILES

Update the sync.ts entry to current truth: entry-time bookmark-reachability
guard (gc'd anchor → full reconcile; non-ancestor-but-present → direct
tree-to-tree diff), oversized-diff fallback, performFullSync now authoritative
for deletes (file-backed pages by source_path), and rename-to-unsyncable delete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 05:43:23 -07:00
Garry TanandClaude Opus 4.8 099d9a8f55 v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers (#1959)
* feat(search): deterministic relational-query parser

Pure, ReDoS-bounded parser that detects relationship queries ("who invested
in X", "who at X works on Y", "who introduced me to X", "what connects A and
B") and maps them to typed edges. Schema-pack-extensible vocab with subset
validation against the link types ingest produces, so query-side and
ingest-side relation vocabularies can't drift. No-match / pronoun-seed /
adjacency guards keep it precision-first (a candidate only; the arm fires
only when a real seed also resolves).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(engine): relationalFanout typed-edge fan-out (both engines)

Generalizes traversePaths to a SEED ARRAY, aggregating to ranked NODES
(shortest hop, edge-richness count, via-link-types, shortest connecting
path, canonical chunk id) instead of edges. Within-source traversal (never
crosses a source boundary even across a federated scope), link_source=
'mentions' excluded by default, deleted_at filtered at seed + every neighbor
+ every node, bounded depth (<=3) + candidate cap. Adds RelationalFanoutRow
/ RelationalFanoutOpts + the relational SearchResult/SearchOpts fields to
types.

Lands in lockstep in postgres + pglite engines, pinned by a DATABASE_URL-
gated parity block in engine-parity.test.ts; a PGLite unit test exercises
the SQL (typed-edge filter, mentions exclusion, deleted_at, canonical chunk,
multi-seed connects, determinism) in default CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(search): relational recall arm + federation key hardening

Wires edge-derived candidates into bare hybridSearch as a FOURTH RRF arm
(relational-recall.ts): parse the original query -> scope-aware,
confidence-gated seed resolution (drops fallback_slugify; never traverses
from a guess) -> relationalFanout -> batch-hydrate, reinforcing each page's
REAL canonical chunk (page-level key for chunkless entity pages) ->
--explain attribution + fail-open audit row. Text-only (no-op in image
mode); pure no-op for non-relational queries; rides every downstream stage
(cosine, post-fusion boosts, dedup, reranker, autocut, token budget).

Mode wiring: relationalRetrieval + relational_retrieval_depth knobs
(conservative off; balanced/tokenmax on; depth 2), per-call thread-through
in both bare + cached paths, KNOBS_HASH_VERSION 9->10 (rel=/reld=), config
keys, modes-dashboard descriptions, and a `relational` param on the query op.

Federation hardening (structural, engine-wide): the RRF/dedup key now
carries source_id via a shared rrfKey() (fixes a latent cross-source
collapse where same-slug pages in different sources merged), and the query
cache scopes by a canonical source-set key (cacheScopeKey) so a federated
read can't be served a single-source row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(eval): relational benchmark + recall@k harness metrics

NamedThingBench harness gains recall@k / recall@10 (the relational headline
metric) on QuestionResult + FamilyReport, plus typed seed/linkTypes/kind on
NamedThingQuestion so the graph-relationship family is machine-checkable.

Adds the relational benchmark corpus (test/fixtures/retrieval-quality/
relational/): a small entity graph whose answers are LEXICALLY UNRECOVERABLE
— every page body is generic and never names the entity it relates to, so
only the typed edge connects query to answer. corpus.ts is the canonical
source for both the seed loader and the 38-question gold set; relational.jsonl
is generated from it (a drift test pins them equal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(eval): relational A/B proof + arm fires on all retrieval paths

Fixes the integration bug the eval caught: the relational arm was only
injected on the main RRF path, so it silently did nothing whenever vector
was unavailable — no embedding provider configured (the default in many
deployments) OR embed failure. The arm is now built once and fused via RRF
on ALL THREE hybridSearch return paths (no-embedding-provider, embed-failed
keyword fallback, main path). Without this it would have been dead in
exactly the setups that most need it.

Adds `gbrain eval retrieval-quality --ab-relational`: runs the gold set
twice (arm off vs on) in a fixed mode and reports the graph-relationship
recall@10 lift + Hit@3 + latency add. The CI A/B test pins the headline
result — recall@10 jumps from <25% (lexically unrecoverable) to >75% with
the arm on — and a non-relational query returns identical results arm-on vs
off (the no-op / no-regression gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.34.0)

Relational retrieval feature: typed-edge recall arm + federation key
hardening. Also updates the KNOBS_HASH_VERSION 9→10 assertions across the
remaining search test files (the bump invalidates relational-off cache rows).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document typed-edge relational retrieval (v0.42.34.0)

CLAUDE.md Search Mode: add relationalRetrieval to the knob table, the
knobs_hash v=9→10 note, and a relational-retrieval summary. RETRIEVAL.md:
add the relational recall arm to the pipeline diagram. Regenerate llms
bundles (build:llms).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): size relational fixtures to the actual embedding-column dim

CI shard runs with the ZeroEntropy gateway default (1280-d), but the
relational test fixtures hardcoded 1536-d embeddings, so chunk inserts were
rejected with "expected 1280 dimensions, not 1536" (CheckExpectedDim) — the
`test (6)` shard + `test-status` failures. The column width tracks the
configured gateway default and can shift with shard order, so fixtures now
probe `content_chunks.embedding`'s actual `atttypmod` after initSchema and
size embeddings to it (the pglite-engine.test.ts pattern), via a shared
`probeEmbeddingDim` helper. Verified passing at a forced 1280-d column.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 22:05:31 -07:00
Garry TanandClaude Opus 4.8 7cf46230e5 docs(designs): add COMMUNITY_IDEAS ledger from open-PR backlog triage (#1969)
Curated, high-bar diary of the valuable ideas surfaced by the community-PR
wave, grouped into 10 themes with contributor credit and OPEN/CLOSED/HELD
status, so good thinking survives PR closure. Captured during a full triage
+ hygiene pass over the open-PR backlog. Pure docs; no code impact.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:46:03 -07:00
Garry TanandClaude Opus 4.8 b31de6613e v0.42.33.0 fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881) (#1960)
* fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881)

recloneIfMissing deleted local_path whenever a source had a remote_url and a
non-healthy on-disk state, with no check that gbrain actually created the clone.
A source whose local_path was a user's live working tree (remote_url set, no
gbrain-created clone) could have its directory removed and re-cloned over.

- isOwnedClone(): ownership, not path-containment. True only for a config
  .managed_clone marker (written by addSource --url) or exact normalized-path
  equality with defaultCloneDir(id) (back-compat for pre-marker default clones).
- recloneIfMissing: ownership guard aborts before ANY filesystem op; EXDEV-safe
  sibling-temp clone + atomic swap (old aside -> new in -> drop old) with
  best-effort restore + a message naming where the original is preserved;
  symlink-leaf reject before the destructive rename.
- sync.ts validate_repo_state guards reclone on isOwnedClone (no per-sync warn).
- sources restore degrades to a warning for an unowned source instead of the
  misleading "missing clone, try sync" hint.

Tests: #1881 regression (tree survives), isOwnedClone matrix, symlink reject,
EXDEV swap residue-free, --clone-dir owned-via-marker, restore CV3, unownedHint
healthy/degraded, sync-level refusal.

* chore: bump version and changelog (v0.42.33.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document sources-ops reclone-ownership invariant for v0.42.33.0 (#1881)

Add the missing src/core/sources-ops.ts entry to KEY_FILES.md capturing the
must-never-violate reclone-ownership guarantee: gbrain only deletes/re-clones a
clone it created (isOwnedClone), never a user working tree. Covers managed_clone
marker, defaultCloneDir back-compat, EXDEV-safe swap, TOCTOU + symlink-leaf
guards, unmanaged_path SourceOpError, and the read-only sources restore path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:28 -07:00
584 changed files with 50117 additions and 2961 deletions
+32
View File
@@ -0,0 +1,32 @@
name: Actionlint
# Lints the GitHub Actions workflow YAML on every change so a malformed
# workflow / bad action ref / missing-permission bug is caught before it ships
# a broken pipeline. gbrain edits .github/workflows/* often (sharding, cache,
# timeouts); this is the cheap guard that keeps those edits honest.
on:
push:
branches: [master]
paths:
- '.github/workflows/**'
pull_request:
branches: [master]
paths:
- '.github/workflows/**'
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
actionlint:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: rhysd/actionlint@393031adb9afb225ee52ae2ccd7a5af5525e03e8 # v1.7.11
+68 -1
View File
@@ -12,10 +12,61 @@ on:
permissions:
contents: read
# Cancel a superseded run when a newer commit lands on the same PR/branch.
# PR number for pull_request events (fork-safe), github.ref fallback for
# push/scheduled runs.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
jsonb-parity:
# Dedicated required guard for the JSONB double-encode bug-class (#2339).
# PGLite parses a double-encoded jsonb string silently, so this assertion can
# ONLY be made on real Postgres — a normal gated e2e file would skip without
# DATABASE_URL and let the bug ship green (as #2339 did). This job provisions
# Postgres and HARD-FAILS if DATABASE_URL is missing, so the guard can never
# silently skip.
name: JSONB parity (#2339 regression guard)
runs-on: ubuntu-latest
timeout-minutes: 15
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- name: Require DATABASE_URL (no silent skip)
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: |
if [ -z "$DATABASE_URL" ]; then
echo "::error::DATABASE_URL must be set for the jsonb-parity job — the #2339 guard would silently skip (the exact failure PGLite hides). Failing the job." >&2
exit 1
fi
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
runs-on: ubuntu-latest
timeout-minutes: 20
services:
postgres:
image: pgvector/pgvector:pg16
@@ -49,6 +100,7 @@ jobs:
# from repo/org secrets. Nightly + manual triggers still supported via
# the workflow-level `on:` list.
needs: tier1
timeout-minutes: 30
services:
postgres:
image: pgvector/pgvector:pg16
@@ -70,7 +122,22 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Install OpenClaw
run: npm install -g openclaw@2026.4.9
# Bound + retry the install: a transient npm/registry stall here used to
# hang unbounded and (since the v0.42.50.0 job timeout) burn the entire
# 30m Tier 2 budget before failing — even though the install normally
# finishes in well under a minute. `timeout` kills a hung attempt fast;
# up to 3 attempts ride out a flaky registry. Step cap is a backstop.
timeout-minutes: 8
run: |
for attempt in 1 2 3; do
if timeout 120 npm install -g openclaw@2026.4.9; then
exit 0
fi
echo "::warning::openclaw install attempt $attempt failed or timed out; retrying in 10s" >&2
sleep 10
done
echo "::error::openclaw install failed after 3 attempts" >&2
exit 1
- name: Configure OpenClaw MCP
run: |
mkdir -p ~/.openclaw
+33
View File
@@ -0,0 +1,33 @@
name: OSV-Scanner
# Dependency vulnerability scan (#2182) via Google's official reusable
# workflow. Runs weekly and on any PR that touches the dependency manifests.
# Tokenless: needs zero secrets. Findings are reported in the job log and as
# a SARIF artifact on the run; code-scanning upload is deliberately disabled
# so the workflow stays read-only (no security-events: write).
on:
pull_request:
branches: [master]
paths:
- 'bun.lock'
- 'package.json'
schedule:
- cron: '30 6 * * 1' # weekly, Monday 06:30 UTC
workflow_dispatch:
permissions:
contents: read
jobs:
osv-scan:
permissions:
actions: read
contents: read
# Required by the reusable workflow's own top-level permissions block —
# GitHub validates the caller grants a superset AT STARTUP, even with
# upload-sarif: false (nothing is actually uploaded; see #2117 upstream).
security-events: write
uses: google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2 # v2.3.8
with:
upload-sarif: false
+9
View File
@@ -19,6 +19,10 @@ jobs:
target: bun-linux-x64
artifact: gbrain-linux-x64
runs-on: ${{ matrix.os }}
permissions:
contents: read
id-token: write # for attest-build-provenance (Sigstore OIDC)
attestations: write # for attest-build-provenance
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -26,7 +30,12 @@ jobs:
bun-version: 1.3.13
- run: bun install
- run: bun test
- run: bun run verify
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Attest build provenance
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
subject-path: bin/${{ matrix.artifact }}
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: ${{ matrix.artifact }}
+36
View File
@@ -0,0 +1,36 @@
name: Semgrep
# Static analysis (SAST) with Semgrep Community Edition (#2272). Tokenless:
# uses the public registry rulesets, needs zero secrets. Findings print in
# the job log; no code-scanning/SARIF upload by design (keeps permissions
# read-only, no security-events: write).
on:
pull_request:
branches: [master]
schedule:
- cron: '30 7 * * 1' # weekly, Monday 07:30 UTC
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
semgrep:
runs-on: ubuntu-latest
timeout-minutes: 20
container:
image: semgrep/semgrep:1.170.0@sha256:c98f8829eea377274ee4b10656458b078b88232469b2ff913f091c2317347c9d
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
# Non-blocking initially (continue-on-error): the first runs establish a
# baseline without failing unrelated PRs. Graduation path: once the
# baseline findings are triaged (fixed or `# nosemgrep`'d), remove
# continue-on-error so new findings block PRs.
- name: Semgrep scan (report-only)
run: semgrep scan --config p/default --config p/typescript --error
continue-on-error: true
+18
View File
@@ -14,6 +14,15 @@ on:
permissions:
contents: read
# Cancel a superseded run when a newer commit lands on the same PR/branch.
# Keyed on the PR number for pull_request events (unique per PR, so two PRs
# from forks sharing a branch name don't cancel each other) and falls back to
# github.ref for push/scheduled runs. Mirrors heavy-tests.yml; frees runners
# and stops a stale-SHA run from reporting a flaky failure on an obsolete commit.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
# ──────────────────────────────────────────────────────────────────────
# cache-check: runs first, computes the content hash of every tracked
@@ -29,6 +38,7 @@ jobs:
# ──────────────────────────────────────────────────────────────────────
cache-check:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
@@ -72,6 +82,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
@@ -90,6 +101,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -110,6 +122,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -134,6 +147,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -156,6 +170,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 12
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
@@ -191,6 +206,7 @@ jobs:
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
@@ -220,6 +236,7 @@ jobs:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: success() && needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Create cache marker
run: |
@@ -242,6 +259,7 @@ jobs:
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Aggregate result
run: |
+3 -2
View File
@@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
unset) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
+629 -4
View File
@@ -2,6 +2,633 @@
All notable changes to GBrain will be documented in this file.
## [0.42.64.0] - 2026-07-20
### Fixed
- Confidential OAuth clients can now revoke access tokens through the standard revocation endpoint when client secrets are stored as hashes. Invalid credentials fail closed, malformed or mixed authentication is rejected, backend failures remain retryable, and discovery metadata accurately advertises supported authentication methods.
No schema migrations.
## [0.42.63.0] - 2026-07-20
**Schema commands now open the local brain you actually configured.**
If your PGLite brain lives at a custom path, commands such as `gbrain schema stats` previously ignored that path and could inspect the default brain instead. That made a healthy configured brain look empty or report the wrong schema counts. Schema commands now use the same complete database configuration as the rest of GBrain. PostgreSQL behavior is unchanged, and no migration is required.
### How to use it
Upgrade, then run the schema command normally:
```bash
gbrain upgrade
gbrain schema stats --json
```
The reported page and type counts now come from the `database_path` in `~/.gbrain/config.json` when the engine is PGLite.
### Itemized changes
#### Fixed
- **Schema CLI commands preserve configured PGLite paths.** Engine construction and connection now receive the canonical complete engine configuration, including both `database_path` and `database_url` where applicable.
- **CLI tests are isolated from ambient database URLs.** Schema subprocess tests explicitly clear inherited PostgreSQL URL variables, and a persistent-PGLite regression test proves `schema stats` reads the configured database rather than the default brain.
## [0.42.62.0] - 2026-07-17
**If your brain holds more than one source, everything now lands in the right one. Link extraction, timeline extraction, background cycles, and webhook captures used to quietly file some of their output under the default source; all of those paths now carry the correct source identity. Background agent jobs got tougher too: a failed database reconnect can no longer wedge the engine, and workers recover from dropped connections instead of crash-looping. If you run the admin dashboard behind a reverse proxy, the live activity panel finally connects. Long agent conversations cost less because repeated context is reused between turns on Anthropic calls. Local LiteLLM proxies work out of the box. Nested sources scan correctly again instead of reporting zero files. And the project's automated checks now include dependency vulnerability scanning, static code-security analysis, and signed provenance for release builds. Thirty merged changes in all, the largest batch to date, each one reviewed and verified against the live codebase before landing.**
## To take advantage of v0.42.62.0
`gbrain upgrade`. No new schema migrations.
1. **Multi-source brains:** run `gbrain extract all` once (or let the next cycle do it) so previously mis-scoped link and timeline rows are regenerated under the right source.
2. **If you serve the admin dashboard behind a reverse proxy,** hard-refresh it once after upgrading; Live Activity should connect.
3. **Verify:**
```bash
gbrain doctor
gbrain stats
```
4. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
### Itemized changes
#### Fixed
- **Source identity threaded through write paths.** Filesystem link/timeline extraction (`src/commands/extract.ts`), the cycle extract phase, and ingest capture now stamp the resolved source id instead of defaulting to `default`, with fail-closed validation on externally supplied ids. (#1522, #1747, #1503 via #2920; absorbs #1719, contributed by @seungsu)
- **`reconnect()` is build-then-swap.** The new pool is validated before replacing the old one, so a failed rebuild restores the previous connection instead of leaving `_sql` null. (#1593 follow-up via #1906, contributed by @rayers)
- **Minion worker reconnects after promote-time connection loss** instead of crash-looping. (#1491 class via #2025, contributed by @maxpetrusenkoagent)
- **Admin Live Activity works behind reverse proxies.** The EventSource now sends credentials so strict-cookie sessions survive the proxy hop. (#912 via #1560, contributed by @flamerged)
- **Stats exclude soft-deleted pages** from visible counts on both engines; destructive-removal counts stay all-inclusive. (#2235, contributed by @xd-Neji)
- **LiteLLM recipes declare chat and expansion touchpoints,** so the subagent loop no longer swaps to Anthropic and fails without an Anthropic key. (#2207 via #2208, contributed by @brettdavies)
- **Rolling prompt-cache on the direct SDK path.** Growing conversations place rolling cache breakpoints (two, within the four-marker budget), cutting repeat-token cost on multi-turn Anthropic tool loops. (#2740 via #2771, contributed by @Masashi-Ono0611)
- **Nested sources scan again.** `sources audit` had one inverted prune check (descending into node_modules while reporting 0 files). (#2678, contributed by @ikamal97)
- **Import and sync agree on metafiles.** The import walker now skips the same structural metafiles sync skips. (#345 via #2315, contributed by @ElliotDrel)
- **Frontmatter scans respect git excludes** via a shared git-visible-files helper. (#2462, contributed by @kubi-dev)
- **Sync renames are crash-safe** (per-file failures recorded instead of aborting the run) and **zero-change syncs still bump `last_sync_at`** so freshness reporting stops lying. (#2402, contributed by @supportswift; #2335, contributed by @lost9999)
- **Facts survive one-shot CLI runs.** Facts-absorb work is enqueued as durable minion jobs instead of dying with the process exit drain; fence paths are source-scoped. (#2104, contributed by @reghar-bot)
- **Takes reads are source-scoped, `gbrain calibration` is reachable, outputs are BigInt-safe.** (#2035 and the takes slice of #2200 via #2892, takeover of #2452, contributed by @spinsirr)
- **CLI answers honestly.** `config get` reads both config planes with provenance, `sources archive` is idempotent, help text matches real subcommands, doctor recommendations name commands that exist. (#2120, #2792, #1175, #1123, #2451 via #2918)
- **PGLite init failures name plausible causes for your platform** instead of blaming a macOS-specific bug everywhere, and non-Error crashes print their message instead of `[object Object]`. (#2674 class via #2891)
- **YAML comments inside frontmatter parse.** `#` lines inside a closed fence are comments, not headings; no more false MISSING_CLOSE. (#2152 via #2153, contributed by @brettdavies)
- **Conversation facts read the raw transcript sidecar** and recognize plain `Speaker A:` lines. (#1897 via #1898, contributed by @ElliotDrel)
- **`get_timeline` exposes date-window filters** (#2604 via #2694, contributed by @RerankerGuo) and **`query` since/until filter on effective date,** not updated_at (#1520 via #1706, contributed by @mvanhorn).
- **Windows serve watchdog works** via a signal-0 liveness probe instead of a POSIX-only process listing. (#2049, contributed by @abyss-node)
- **Doctor probes route through the active engine** (no false pgvector/jsonb warnings on PGLite; #1513 via #1183, contributed by @duncanclaw) and **a disabled retrieval reflex reads as intentional** (#2459, contributed by @eloe).
- **Cross-platform installs.** The postinstall hook is a real bun script, not POSIX shell that failed on Windows. (#1486 via #1554, contributed by @Sanjays2402)
- **Agent-bound auth clients.** `auth register-client` gains the `--bound-*` flags the submit_agent gate requires. (#1945, #1971 via #1976, contributed by @mzkarami)
#### Added
- **Security automation in the project's checks:** scheduled OSV dependency scanning, Semgrep static analysis on every PR (non-blocking initially), and build-provenance attestations wired into the release workflow. (#2182, #2142, #2272 via #2917)
- **`provider_chat_options` config passthrough** to the gateway, e.g. disabling thinking mode per provider or model. (#2577 via #2857)
- **Docs:** macOS 26.x PGLite workaround and native Postgres setup guide. (#1671, contributed by @roysaurav)
#### Internal
- release.yml runs `verify` before building. (#2222 via #2243, contributed by @mzkarami)
- Regenerated llms bundle after the docs merge. (#2893)
## [0.42.61.0] - 2026-07-16
**If gbrain's background daemon dies hard, a restart now takes over right away instead of waiting minutes for a stale lock to expire. Re-processing the same content no longer piles up near-duplicate knowledge atoms. On large brains, the takes bootstrap finally works through the whole corpus instead of re-scanning the same newest pages every run. And `gbrain schema use` can now activate the schema packs gbrain actually ships — including the install default — instead of just one hardcoded name. Cost tracking also learns the newest Claude models, so spend on them is metered instead of invisible.**
### Itemized changes
#### Fixed
- **Autopilot recovers immediately from a crashed daemon.** The stale-lock check verifies whether the lock-holding process is still alive instead of relying on a fixed age window — a hard-killed autopilot no longer delays restarts, and the age check alone can no longer displace a busy, live one. (#477, contributed by @vinsew)
- **Atom extraction stops minting duplicate atoms across runs.** Atom slugs are now deterministic (source-dated, canonical slugging, content-hashed suffix), so re-extracting the same content upserts instead of creating a near-duplicate under a new run-date path, and titles that truncate mid-word no longer produce trailing-dash slug variants. Pre-existing duplicates are not re-created but remain until cleaned up (an `atoms consolidate` command is tracked as a follow-up). (#2482, contributed by @joelwp)
- **Takes bootstrap works through the whole corpus.** Bootstrap runs skip pages that already have takes, so brains larger than the per-run page cap make forward progress instead of rescanning the newest slice and re-spending extraction budget. `--include-covered` restores the old behavior. (#2638, contributed by @p3ob7o)
- **`gbrain schema use` can activate the core bundled packs.** The command resolved only one hardcoded pack name; it now resolves through the bundled-pack registry, so the recommended and v2 base packs (including the install default) can be selected. (#1707, contributed by @mvanhorn)
- **Budget tracking prices Sonnet 5 and Fable 5.** The canonical chat-pricing table adds the newest Claude models at standard list rates (time-limited introductory discounts are deliberately not modeled, so early Sonnet 5 spend reads slightly conservative), removing the no-pricing blind spot in cost telemetry and budget metering. (#2799, contributed by @p3ob7o)
#### Added
- **Inline `[Source: ..., YYYY-MM-DD]` citations become timeline entries.** Both the filesystem extract path and the auto-timeline write path recognize the citation convention gbrains own quality guidance recommends, with idempotent re-extraction. (#2524, contributed by @pabloglzg)
- **Schema packs extend atom-extraction page discovery.** For packs that declare the `extract_atoms` phase, the manifests `extractable` flag now unions with the legacy page-type list (synthesis outputs stay excluded, so concepts never feed back into atom extraction). (#2615, contributed by @p3ob7o)
- **Book-mirror two-column pages are generated as HTML tables** with top alignment instead of markdown pipe tables, which broke on multi-paragraph cells in most renderers. (#2270)
#### Internal
- Gateway tool-schema conversion extracted into a tested helper so the regression test exercises the exact code path production uses. (#2063, contributed by @maxpetrusenkoagent)
- Reference docs synced for the v0.42.59.0 fixes (engine/testing entries). (#2798, contributed by @time-attack)
### To take advantage of v0.42.61.0
`gbrain upgrade`. No new schema migrations.
1. **Heads-up on extraction scope:** if your active schema pack declares the `extract_atoms` phase, page types the pack marks `extractable` now feed atom extraction alongside the legacy list — the first cycle after upgrading may process page types (notes, emails, slack) it previously skipped. Per-run page and budget caps still apply; check `gbrain search stats` / budget output if you watch spend closely.
2. **If takes bootstrap seemed stuck** on a large brain, re-run it — each run now covers new pages.
3. **Verify:**
```bash
gbrain doctor
gbrain stats
```
4. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
## [0.42.60.0] - 2026-07-16
**Eleven verified community fixes: Windows brains no longer risk losing subdirectory pages on a full sync, agent tool loops on non-Anthropic providers survive interruption instead of dead-lettering, multi-source brains get two source-isolation gaps closed, and the search cache stops leaking results across exclude policies. Every fix was reproduced and reviewed against master before landing.**
### Fixed
- **Windows: `sync --full` no longer deletes subdirectory pages.** A path-separator mismatch made every subdirectory page look stale during full-sync reconcile, so a routine full sync could delete them. Paths are now normalized before comparison, and a mass-delete safety valve blocks any reconcile that would remove most of a source's pages. (#2828, #2836, contributed by @1alessio)
- **Gateway tool loops on non-Anthropic providers are reliable across resume.** Tool-result turns are persisted as they happen, interrupted jobs reconcile dangling tool calls on resume instead of dead-lettering with unbalanced-transcript errors, `Date` values in tool outputs no longer crash serialization, DeepSeek reasoning-only replies are read correctly instead of as empty, and `openrouter_api_key` in config reaches the gateway. (#2820, consolidating community fixes #2062, #2065, #2257, #2274, #2336, #2487, #2491, #2572, #2614, #2617, #2806; contributed by @time-attack and the original PR authors)
- **Claude 5 models get output-token headroom.** Thinking-default models no longer have long answers silently truncated by the old 4096-token default output cap; Claude 5 chat calls now default to 32000 output tokens (16000 for `think`). Other providers keep their existing caps, so smaller-limit providers are unaffected. (#2820)
- **Bulk import survives huge fence-less files.** The markdown lexer is skipped when a page contains no code fences, removing an out-of-memory crash on large tables and notes during bulk import. (#2437, #2440, contributed by @irresi)
- **`file_list` no longer crashes on Postgres brains over MCP.** BIGINT file sizes are normalized before JSON serialization; the CLI files listing gets the same fix. (#472, contributed by @vinsew)
- **`gbrain config set auto_chronicle true` works as documented.** The Life Chronicle config keys (and `takes.bootstrap_enabled`) are registered, so the documented enable commands stop being rejected as unknown keys. (#2632, contributed by @p3ob7o)
- **Orphan reports skip generated corpus roots.** `raw/`, `atoms/`, and `skills/` no longer inflate the orphan ratio by default; `--include-pseudo` still shows everything. (#2068, contributed by @mgunnin)
### Security
- **The search cache honors your hard-exclude policy.** Cached search results are now keyed on the effective hard-exclude/include slug-prefix policy, so a process with `GBRAIN_SEARCH_EXCLUDE` set can never be served cached rows written under a different policy — and vice versa. (#2825, #2885)
- **Take-writes are source-scoped.** When a source resolves (via `--source`, `GBRAIN_SOURCE`, or the dotfile chain), CLI take commands look pages up within that source instead of first-match-by-slug, closing a cross-source write path on brains where the same slug exists in multiple sources. Brains without a resolvable source keep the previous lookup. (#2684, #2698, contributed by @RerankerGuo)
- **Image pages land in the right source.** Imported images are stamped with the syncing source (and their auto-links stay within it) instead of always landing in `default`. (#2706, #2718, contributed by @RerankerGuo)
- **The admin bootstrap token no longer prints to a non-terminal stream.** The one-time token is withheld when its output stream is a pipe, log, or CI capture instead of an interactive terminal. (#2625, contributed by @irresi)
### Internal
- Pinned embedding dimensions in a doctor test to eliminate a shard-order flake in CI. (#2801, contributed by @p3ob7o)
### To take advantage of v0.42.60.0
`gbrain upgrade`. No new schema migrations.
1. **Windows users with git-synced sources:** re-run `gbrain sync --full` once after upgrading — if a pre-upgrade sync deleted subdirectory pages, they re-import from the repo.
2. **Your first search after upgrading may be a cache miss** (the cache key now includes the exclude policy). Speeds return to normal as the cache refills within its TTL.
3. **If agent jobs previously dead-lettered** with unbalanced tool-call transcript errors on OpenAI-compatible providers, retry them with `gbrain jobs retry <id>` — resume now reconciles the transcript.
4. **Verify:**
```bash
gbrain doctor
gbrain stats
```
5. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
## [0.42.59.0] - 2026-07-13
**Five community-reported fixes, each reproduced and verified before/after on both engines (PGLite + real Postgres): an upgrade wedge that locked pre-v121 brains out of migrations, two data-integrity holes in engine migration, silent deletion of facts containing pipe characters, confidently-wrong entity attribution on ambiguous names, and tightened source-scope enforcement in `think`.**
### Fixed
- **Existing brains below schema v121 can upgrade again.** Brains created before v0.42.56.0 could get stuck in a loop where every command (including `apply-migrations`) failed with `column "event_page_id" does not exist` — the migration that adds the column could never run. The startup bootstrap now adds the forward-referenced column first; migration v121 still owns the FK and indexes. Re-running is idempotent, and already-wedged brains heal on the next command. (#2724, #2735, contributed by @time-attack)
- **`gbrain migrate --to` no longer fails on multi-source brains.** The source catalog is copied before pages, so the first page no longer dies on a foreign-key violation. Source rows migrate with full fidelity (paths, sync state, config). (#2677, #2736, contributed by @time-attack)
- **Migration resume checkpoints are target-aware.** An interrupted migration to one target no longer convinces a later migration to a *different* target that most pages are "already done" (which silently shorted the new target). A checkpoint for another destination is discarded and the run starts fresh; no connection strings or credentials are written to manifests or logs. (#2677, #2736, contributed by @time-attack)
- **Facts containing `|` characters survive reconciliation.** The facts fence rendered literal pipes escaped but re-parsed rows by splitting on every pipe, so any fact whose text contained a `|` was silently deleted from the DB on the next extract-facts cycle. Render→parse is now symmetric (pipes, backslashes, and empty cells verified round-trip). The takes fence shares the parser and gets the same fix. (#2726, #2738, contributed by @time-attack)
- **Ambiguous entity names quarantine instead of guessing.** A bare first name shared by two people, or a company name sharing a generic token (e.g. "… Capital") with another company, used to resolve confidently to the wrong entity — misattributed facts are invisible and expensive to repair. Bare names now resolve only when exactly one canonical candidate exists; low-specificity fuzzy matches fall through to the guarded holding path (a held fact is recoverable; a misattributed one isn't). Explicit slugs, full names, unique bare names, and close typos still resolve. Trade-off: heavier typos on short names may now hold instead of resolving. (#2723, #2737, contributed by @time-attack)
### Security
- **`think` now applies the caller's source scope across all of its internal retrieval.** Hybrid page retrieval, takes keyword/vector retrieval, and graph traversal all honor scalar and federated source scope, matching the isolation the rest of the read surface already enforces. Part of the #2200 tracking work. (#2739, contributed by @time-attack)
### To take advantage of v0.42.59.0
`gbrain upgrade` should do this automatically. No new schema migrations ship in this release (v121/v122 shipped with v0.42.56.0).
1. **If your brain was stuck below schema v121** (every command printed a schema-probe warning), just upgrade and run any command — the brain heals and migrates to current on first connect. If `gbrain doctor` still complains:
```bash
gbrain apply-migrations --yes
```
2. **Verify:**
```bash
gbrain doctor
gbrain stats
```
3. **If a previously-resolving shorthand name now files under a holding page**, that's the new ambiguity quarantine working as intended — add an alias or use the full name/slug for entities you want bare shorthand to hit.
4. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
## [0.42.58.0] - 2026-07-06
**gbrain now runs cleanly on the stack you already have — a local Ollama box, a self-hosted LiteLLM proxy, llama.cpp's llama-server, or gbrain running as a Claude Code MCP subprocess — instead of silently degrading or hard-failing when you're not on a raw OpenAI/Anthropic key.** A provider-agnostic plumbing pass across the AI gateway: environment handling, base-URL normalization, and embedding-dimension validation all stop tripping on the non-frontier-vendor setups that used to fail without a clear signal.
### Fixed
- **Running gbrain as a Claude Code MCP subprocess no longer breaks every AI call.** Some hosts inject an empty `ANTHROPIC_API_KEY` into the subprocess environment; that empty value used to override a real key set in your `~/.gbrain/config.json`, so every gateway call failed with a missing-key error. Empty environment values no longer clobber a configured key. (#1249)
- **A custom Anthropic or OpenAI base URL without a `/v1` suffix no longer 404s.** When the base URL comes from the environment as a bare host, gbrain normalizes it before the call instead of posting to a path the provider doesn't serve. Unset base URLs are untouched, so the default hosted endpoints are unaffected. (#1250)
- **Vector search stops silently going dark on a LiteLLM or llama-server embedding model.** A model-availability check was rejecting user-provided embedding recipes even when a model was configured, quietly disabling vector search so results looked empty. The check now validates what actually matters — that a dimension is set — and reports a clear, actionable message when it isn't. (#1292, #2295)
- **Local embedding models with non-standard dimensions are accepted.** Ollama, llama-server, and LiteLLM models (e.g. modern 1024- or 4096-dimension embedders) no longer get hard-rejected by the dimension validator; you declare the dimension and gbrain trusts it. Hosted fixed-dimension providers stay strictly validated. Modern Ollama embed models are recognized. (#2271)
### Changed
- **LiteLLM setup guidance now names the `/v1` path convention** so OpenAI-shaped proxies that only serve the `/v1` route don't fail authentication with no hint. (#2209)
### To take advantage of v0.42.58.0
`gbrain upgrade`. If you run on Ollama, a LiteLLM proxy, llama-server, or as a Claude Code MCP subprocess, the fixes apply automatically — no migration, no config change. If you use a user-provided embedding recipe (LiteLLM / llama-server) and see a "no default embedding dimension" message, set it with `gbrain init --embedding-dimensions <N>`.
## [0.42.57.0] - 2026-07-02
**PGLite incident fix: a busy `gbrain dream` (or `embed`) could have its data-directory lock stolen and get its brain corrupted beyond in-place repair. The lock will no longer be taken from a process that is alive, and an already-corrupted store now tells you exactly how to recover.**
### Fixed
- **A live PGLite holder is never stolen.** The data-directory lock used to be reaped if the holder's heartbeat went stale past a grace window. But the heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/checkpoints, so a genuinely working `gbrain dream`/`embed` could look stale while fully alive. Reaping it let a second process open the same store and corrupt the catalog + pgvector extension (surfacing later as `relation "content_chunks" does not exist` / `type "vector" does not exist`, only recoverable by wipe-and-restore). The lock is now reaped only when the holder process is actually dead; a wedged-but-alive or PID-reused holder makes the acquire time out with a clear message naming the PID, instead of risking corruption.
- **A corrupted PGLite store now explains how to recover.** When the store's catalog or pgvector extension can no longer load, the error names the cause and points at `gbrain reinit-pglite --embedding-model <id> --embedding-dimensions <N>` (or restoring a backup), instead of the unrelated "macOS WASM bug" hint. It also notes that deleting the lock dir or `postmaster.pid` does not fix it.
### To take advantage of v0.42.57.0
`gbrain upgrade`. No migration. New corruption is prevented going forward. A brain already corrupted by a prior concurrent open cannot be repaired in place; the upgraded error message walks you through `gbrain reinit-pglite` or restoring a backup.
## [0.42.56.0] - 2026-07-02
**Life Chronicle: gbrain gains a temporal spine. Meetings and transcripts project into a queryable timeline, entities carry a bi-temporal ontology (sourced, confidence-weighted properties that supersede over time), and a low-friction diary captures interiority — so an agent can reconstruct "what happened the week of X", answer "when did I last interact with Y", and see how an entity's role or stance changed, instead of re-deriving chronology from scratch every session.** Built entirely on existing primitives (pages, the `facts` table, `timeline_entries`) — no new datastore. Auto-emission is off by default; opt in per below.
### Added
- **Timeline events + reads.** Meetings/transcripts auto-emit `type:event` atoms (when·where·who·what) that project into a date index and backlink to the depth page. Query them with `gbrain day <date> [--week] [--narrative]`, `gbrain since <date> [--kind]`, `gbrain last-seen <entity>`, and `gbrain on-this-day`. Intra-day ordering, read-time hiding of deleted events, and source isolation throughout.
- **Bi-temporal per-entity ontology.** An open-world, sourced, confidence-weighted property bag rides the existing `facts` table (new `dimension`/`value` columns). A new value supersedes the prior across a validity window, so `gbrain ontology <entity> [--asof <date>]` can time-travel; genuine two-source disagreement surfaces via `gbrain ontology-contradictions`; `gbrain ontology-dimensions` shows what the brain tracks about entities. Novel LLM-proposed dimensions quarantine until confirmed.
- **Diary capture + agent orientation.** `gbrain capture --type diary` (and `--type event` with `--who/--what/--where/--kind`) for low-friction entries; `gbrain orient` hands an agent the recent timeline plus resolved-entity ontology in one zero-LLM payload. `gbrain chronicle-backfill` sweeps existing meetings into the timeline.
- **Ambient temporal recall + proactive surfacing.** Temporal queries lift chronicle pages in search; `gbrain advisor` flags unresolved ontology conflicts and recent meetings missing from the timeline. A deterministic `gbrain eval chronicle` gates the feature (day-order, last-seen, supersession, contradiction, source isolation).
- **Migrations v121 (event-projection column) + v122 (facts ontology columns).** Additive; legacy rows unchanged.
### Security
- **Interiority stays local.** Diary content and diary-sourced ontology are redacted from untrusted (remote/MCP) readers across reads, search, advisor, and orientation. Auto-emission runs only for trusted local writes.
### To take advantage of v0.42.56.0
`gbrain upgrade`, then `gbrain apply-migrations --yes` (or any command that opens the brain) to pick up v121/v122. Auto-emission is OFF by default: turn it on with `gbrain config set auto_chronicle true`, then `gbrain chronicle-backfill` to populate the timeline from existing meetings. Manual capture (`gbrain capture --type event/diary`) and every read surface work immediately. Closes #2390 (duplicate #2388).
## [0.42.55.0] - 2026-06-24
**A security-hardening pass: routing dotfiles, the skills directory, page slugs, and large-file transcription are confined against multi-user-host and untrusted-input edge cases; dynamic OAuth client registration defaults to a consent-bearing grant; and a schema-lint migration brings existing brains to the fresh-install posture.** Several of these close community-reported gaps. Fresh installs were already covered; existing brains are brought to the same bar automatically on upgrade. If `gbrain doctor` flags anything afterward, its message names the object and the exact fix.
### Fixed
- **Walk-up routing dotfiles are trust-gated.** On a shared multi-user host, a `.gbrain-source` / `.gbrain-mount` found by the ancestor-directory walk is accepted only when it's owned by you (or root) and is neither a symlink nor world-writable — otherwise it's skipped, fail-closed. The working-directory match that routes by registered path now resolves symlinks on both sides, so a redirected directory can't misattribute your source or brain. (#418, contributed by @garagon)
- **The skills directory is confined to its workspace.** Every skills-dir resolution tier now requires the resolved directory to stay within the declared workspace, so a redirected `skills` entry can't point the loader outside it. (#419, contributed by @garagon)
- **Page slugs reject unsafe characters at the write boundary.** The shared slug validator now rejects control bytes, bidirectional/RTL overrides, backslashes, and URL-encoded path separators on top of the existing traversal check, and the file-write path confirms the target stays inside the source's working tree. Ordinary slugs — including non-Latin and CJK — are unaffected.
- **Large-file transcription no longer builds shell command strings.** The segmentation path invokes `ffprobe`/`ffmpeg` with argument arrays and removes its temp directory through the filesystem API, so a media path is never parsed by a shell. (#245, contributed by @aliceagent)
- **Superuser-connected fresh installs no longer abort during migration.** The RLS preflight in the schema migrations recognizes superuser and inherited-role privileges, not just the role's own flag. (#1385)
### Changed
- **Dynamic client registration defaults to the consent-bearing grant.** With Dynamic Client Registration enabled, a self-registered client now defaults to `authorization_code` (which goes through the approval screen) instead of `client_credentials`. Operators who need the machine-to-machine grant opt in with the new `--enable-dcr-insecure` flag, and a startup warning prints whenever registration is open. Registering clients via the CLI or admin API is unchanged. (#1353)
### Added
- **Migration v120 — schema-lint hardening.** Brings existing brains to the fresh-install posture: the `page_links` view runs with the caller's privileges on Postgres, and the gbrain-owned trigger/event functions pin their schema search path on both engines. A new CI guard keeps new trigger functions from regressing. (#1647, #171)
### To take advantage of v0.42.55.0
`gbrain upgrade`, then `gbrain apply-migrations --yes` (or any command that opens the brain) to pick up migration v120 — no manual step, all on by default. Fresh installs already carry every change. The hardening applies automatically; `--enable-dcr-insecure` is the explicit escape hatch if you genuinely need the machine-to-machine OAuth grant.
## [0.42.53.0] - 2026-06-23
**`gbrain sync` works again on managed Postgres brains: the durable-checkpoint pin write was encoding its value the wrong way, so every multi-source sync aborted at the very first checkpoint. Fixed, plus a repo-wide sweep of the same JSONB footgun and a new CI guard so it can't come back.** A recent release added a structural check on the sync checkpoint table; the pin write that runs before every drain bound its value as a string rather than a real array, so the check rejected it and the run bailed before importing anything. The bug was invisible on the embedded engine (its driver parses the value either way) and only bit managed Postgres.
### Fixed
- **Multi-source sync no longer aborts at the first checkpoint.** The sync-target pin write now binds its value so Postgres stores a genuine JSONB array instead of a double-encoded string scalar. A dedicated Postgres CI job exercises this on a real database, because the embedded test engine masks the failure — which is exactly why it shipped.
- **The same JSONB double-encode footgun is swept across the codebase.** Every raw write that serialized a value into a JSONB column the bug-prone way is corrected to the safe form (search cache, source config, calibration profiles, subagent tool records, eval receipts, code-intel cache, symbol resolver, and others). Readers were already defensive, so existing rows self-heal as each is rewritten.
- **`gbrain eval suspected-contradictions` no longer crashes on an exact-alias query.** An alias-matched result was missing its page id, which aborted the whole probe on Postgres; the id is now carried through, with a finite-id filter as a defensive backstop.
### Added
- **A CI guard for the positional JSONB double-encode pattern.** The existing guard caught only the template-string spelling; a new static check (`scripts/check-jsonb-params.mjs`) catches the positional-parameter form — the one behind this wave — across the codebase, with its own self-test. The embedded engine's native path is intentionally not flagged, since the bug can't occur there.
### To take advantage of v0.42.53.0
`gbrain upgrade`. Multi-source Postgres brains that had stopped syncing resume on the next `gbrain sync` — no migration, no manual step. Rows written in the double-encoded form before the fix self-heal as each is rewritten; re-running the affected write (or a sync) repairs them eagerly if you'd rather not wait.
## [0.42.52.0] - 2026-06-18
**Autopilot stops manufacturing dead jobs and wedging its own queue, plus four operational rough edges get fixed: minion attempt-accounting, `agent run` flag parsing, honest `sources status`, and a budgeted `gbrain status`.** On a multi-source Postgres brain, autopilot could fan out a continuous stream of dead `autopilot-cycle` jobs while the supervisor periodically wedged the very queue it exists to keep alive. The root cause was one disease with several interacting parts; this wave addresses all of them, then cleans up four smaller reliability bugs found alongside.
### Changed
- **Autopilot runs one brain-wide maintenance pass, not one per source.** The cycle is split: per-source jobs run only source-scoped phases, and a single `autopilot-global-maintenance` job runs the brain-wide phases once per window. This removes the per-cycle memory blow-up that was the shared root cause of the dead-job storm and the queue wedge. Per-source filesystem phases bind to the source's own path, so the freshness stamp and the work agree on which source ran.
- **The supervisor self-heals instead of giving up.** A transient database blip no longer trips the crash-budget breaker into a permanent stop; the supervisor degrades to capped-backoff retry and recovers, with a hard ceiling as the backstop. It detects a live sibling supervisor through the queue's database lock (not a `$HOME`-derived pidfile), so two supervisors under a split home directory can't both claim the queue.
- **`gbrain sources status` tells a running sync apart from an idle source.** A source holding a live sync lock now reads as actively syncing instead of "idle," matching the honest-freshness signal `gbrain doctor` already shows.
### Added
- **Per-source failure cooldown + fan-out clamp.** A source that fails backs off (bounded exponential) instead of being re-dispatched every tick; per-tick fan-out is clamped to the worker concurrency, with a `gbrain doctor` check that warns on a mismatch.
- **`gbrain status --deadline-ms` / `--fast`.** A budgeted status snapshot returns whatever sections completed within the budget (marked partial) instead of hanging a poller; the JSON envelope also carries the CLI `version`.
- **A sync stall watchdog.** If the import drain makes no forward progress for `GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900), the run aborts and releases its per-source lock so the next `gbrain sync` resumes from the checkpoint — no manual `pkill`. It keys off import progress (not the lock heartbeat) and reports a distinct `stall_timeout` reason. (Limit: a hang inside a single file's import is observed between files, not mid-file; the wall-clock deadline remains the backstop there.)
### Fixed
- **A timed-out minion run counts as a spent attempt.** Wall-clock dead-lettering already did; the per-job timeout path didn't, so long-lane jobs (subagent / embed-backfill / autopilot-cycle) could read `attempts: 0/N (started: N)`. Accounting is now honest across all dead-letter paths.
- **`gbrain agent run` no longer swallows flags after the prompt.** A trailing `--detach` / `--follow` is recognized instead of being captured into the prompt string; a `--word` inside the prompt stays verbatim, and an explicit `--` ends flag parsing anywhere.
### To take advantage of v0.42.52.0
`gbrain upgrade`. Existing brains pick up the cycle split, supervisor backoff, and per-source cooldown on the next autopilot tick (one catch-up global-maintenance pass on the first tick) — no migration, all on by default. Tune the sync stall watchdog with `GBRAIN_SYNC_STALL_ABORT_SECONDS` if 900s doesn't fit your largest files; budget a status poller with `gbrain status --fast` or `--deadline-ms=<n>`.
## [0.42.51.0] - 2026-06-17
**`gbrain sync` stops bottlenecking all its workers on a single database row, a malformed checkpoint can no longer wedge a source, and `gbrain doctor` tells an actively-running sync apart from a stuck one.** A slow source that fell behind HEAD could read as permanently stale even while it imported every cycle: sync was single-core-bound at the database layer, so handing it more workers didn't help, and the freshness check couldn't see that a sync was in fact running.
The root cause was the page-generation clock that backs the search cache. Every page write bumped a single locked counter row, so concurrent sync workers serialized on one another's commits no matter how many you ran. It is now a contention-free sequence: the cache invalidation contract is unchanged (it still over-invalidates rather than ever serving stale), but writers no longer wait in line. The other fixes harden checkpoint state and make the freshness signal honest.
### Changed
- **Sync writes scale across cores.** The page-generation clock moved from a single locked counter row to a contention-free sequence, so parallel sync workers stop serializing on each other. A large `gbrain sync` now uses the workers you give it instead of collapsing to roughly one.
- **`gbrain doctor` distinguishes in-progress from stale.** A source holding a live sync lock is reported as actively syncing (naming the running process), not flagged stale. A genuinely stuck, blocked, or never-completed sync still reports stale — the signal is the live lock, so a stopped sync is never masked.
### Fixed
- **A malformed checkpoint record can no longer wedge a source.** Checkpoint state is structurally constrained, repaired automatically on upgrade, and the loader survives a bad record instead of discarding all banked progress for that source.
- **`gbrain sync --force-break-lock` is honest when there is no lock.** It now says plainly that nothing was held and points at how to inspect a genuinely wedged sync, instead of a terse no-op that read like a successful unwedge.
### To take advantage of v0.42.51.0
`gbrain upgrade`, then `gbrain doctor`. Existing brains pick up the contention-free clock and the checkpoint integrity constraint automatically on the next migration; the search cache rebuilds itself on first query. Nothing to configure.
## [0.42.50.0] - 2026-06-17
**CI reliability hardening — a wedged job can no longer run for six hours, a superseded run no longer reports a stale flaky failure, and broken workflow YAML is caught before it ships.** gbrain's CI already had the deep machinery (content-hash run-skip cache, weight-aware shard balancing, test-isolation guards, hermetic E2E). What it lacked was the cheap GitHub-Actions hygiene that was already wired into `heavy-tests.yml` but never into the two hot-path workflows. This pass closes that gap, porting the patterns from the sibling GStack project's CI-reliability work.
### Changed
- **`test.yml` and `e2e.yml` cancel a superseded run** when a newer commit lands on the same PR (`concurrency` keyed on the PR number — fork-safe — with a `github.ref` fallback so push and scheduled runs always complete). Frees runners and stops a run against an obsolete commit from reporting a flaky failure.
- **Every job in `test.yml`/`e2e.yml` now has a `timeout-minutes` bound** (test matrix 15, verify 12, serial 15, slow jobs 12, E2E tier 1 20 / tier 2 30, trivial jobs 5-10). A wedged job is converted from a six-hour zombie (GitHub's default) into a fast, legible failure.
- **`scripts/run-e2e.sh` scrubs operator/agent environment before E2E.** A dev or Conductor shell exporting `CONDUCTOR_*` / `MCP_*` / `GBRAIN_*` config overrides no longer bleeds into E2E child processes (which made "hermetic" E2E non-hermetic and its failures unreproducible across machines). Denylist scrub — `PATH`/`HOME`/`TMPDIR`/`DATABASE_URL` survive; `GBRAIN_HOME` is preserved for the existing HOME isolation.
### Added
- **`actionlint` workflow** (`rhysd/actionlint`, SHA-pinned) lints all workflow YAML on `.github/workflows/**` changes, catching a malformed workflow / bad action ref / missing-permission bug before it ships a broken pipeline.
### To take advantage of v0.42.50.0
Nothing to do — these are CI/test-infra changes that take effect automatically on the next push. Contributors running the suite locally get the same hermetic-E2E env scrub via `bash scripts/run-e2e.sh` (or `bun run ci:local`).
## [0.42.49.0] - 2026-06-16
**Big embed backfills and syncs now throttle themselves when the database gets busy, so clearing a backlog can't starve the job queue — no more external babysitter scripts.** A naive `gbrain embed --stale` or large `gbrain sync` against a PgBouncer transaction-mode pooler could saturate it and starve the minion supervisor's lock renewals, cascading `lock-renewal-failed` into dead jobs. The field workaround was an external wrapper that SIGSTOP/SIGCONT'd the process off a side-pool latency probe. That approach was blind (the side pool read low latency while the pool that mattered starved), unsafe (SIGSTOP can freeze a process mid-transaction holding locks), and couldn't touch peak pressure. gbrain now does this natively, and better.
Pacing is **opt-in** (default `off`) and built on one composable primitive: it caps simultaneous in-flight DB writes (the real lever against pooler-slot starvation), measures the work's own query latency in-band (so it can never be blind), and sleeps cooperatively between safe points (never mid-transaction, so the lock heartbeat keeps firing). Turn it on per-run with `gbrain embed --stale --pace`, or set `pace.mode` in config to pace every embed path plus the production embed-backfill job automatically. `GBRAIN_PACE_*` env vars override config as an incident escape hatch.
### Added
- **`--pace[=mode]` for `gbrain embed`** — `off`/`gentle`/`balanced`/`aggressive` bundles (bare `--pace` = balanced), plus `--pace-max-concurrency=N`. `--background` carries the explicit override into the queued `embed` job; the handler re-resolves env > config > bundle at execution.
- **`pace.mode` config + `GBRAIN_PACE_*` env** — config paces every `runEmbedCore` caller (cycle embed, catch-up, sync-auto-embed) and the prod `embed-backfill` job automatically; env beats config for incident response.
- **Composable `db-pacer` primitive** (`src/core/db-pacer.ts`) + named bundles (`src/core/pace-mode.ts`) — concurrency permit + in-band EWMA + jittered cooperative sleep, abort-throwing, fail-open. `sync` uses the shared permit across its parallel worker engines.
- **Pacing telemetry**`EmbedResult.pacing` (cap, samples, EWMA latency, slept ms) in `--json`, plus a one-line stderr summary.
### Changed
- **`gbrain embed --stale` now single-flights per source** using the same lock the `embed-backfill` job holds, so a hand-run backfill and a queued job can't grind the same source concurrently. Paced runs add a bounded end-of-run rescan (catches rows that landed behind the cursor during a longer run), and the embed time budget excludes paced-sleep time so a contended DB still converges instead of exiting early.
### To take advantage of v0.42.49.0
`gbrain upgrade`. Pacing is off by default — nothing changes until you opt in. To clear a big embed backlog safely on a busy pooler: `gbrain embed --stale --pace` (or `--pace=gentle` to be extra conservative). To pace the background embed-backfill job and every embed path automatically: `gbrain config set pace.mode balanced`. During an incident you can override without a redeploy: `GBRAIN_PACE_MODE=gentle` or `GBRAIN_PACE_MAX_CONCURRENCY=4`.
## [0.42.48.0] - 2026-06-16
**Brain repos harden themselves for durability the moment gbrain is given a PAT and a GitHub URL.** Fresh agents kept drifting out of sync with their knowledge-wiki git repos: writes sat local-only and never pushed, long-lived sessions edited a stale tree, and scratch output landed outside the repo and vanished. Now `gbrain sources add --url <repo> --pat-file <p>` auto-hardens the managed clone, and `gbrain sources harden <id>` runs the same audit idempotently against any source. Hardening is six always-on guarantees: it pulls current state (divergence-safe rebase that skips a dirty tree and never leaves a half-rebase), installs a local auto-push safety net, ships a committed `scripts/brain-commit-push.sh` that refuses to report success without a confirmed push, writes always-on durability rules into the agent's context file (deterministic filing from the canonical taxonomy, commit-and-push-never-deferred, pull-before-each-write-batch), registers a 30-minute background pull so an idle session can't go stale, and verifies push access up front.
This is gbrain's first push path and first credential storage, built secure by default. The push automation is installed locally per machine rather than committed into the repo, the GitHub token is wired per-repo (least-privilege; an existing credential helper is reused when present rather than writing a new one), and the token never enters the repo, the tracked remote URL, logs, or the run report. Hardening proves push works with a dry-run probe before declaring done, so a read-only token or a protected branch surfaces immediately instead of silently dropping writes later. `gbrain sources unharden <id>` cleanly removes everything it installed and runs automatically before `sources remove`.
### Added
- `gbrain sources harden <id|--all> [--pat-file <p>] [--branch <b>] [--no-cron] [--no-verify] [--dry-run] [--json]` — idempotent brain-repo durability hardening.
- `gbrain sources pull <id> | --path <dir> [--branch <b>]` — divergence-safe rebase-pull; `--path` runs DB-free so the 30-minute cron never contends for the local engine lock.
- `gbrain sources unharden <id>` — remove the durability cron, hook, and credential wiring.
- `--pat-file` / `--no-harden` on `gbrain sources add`; managed clones added with a PAT auto-harden.
- `git-remote.ts`: `divergenceSafePull`, `detectDefaultBranch`, `pushProbe`, and an env-gated `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` escape hatch for self-hosted filesystem remotes.
### To take advantage of v0.42.48.0
`gbrain upgrade`. Add a brain repo with `gbrain sources add <id> --url <https-repo> --pat-file <path-to-token>` and it hardens automatically; or run `gbrain sources harden <id> --pat-file <path>` on an existing source. Use a fine-grained PAT scoped to just that repo. Existing brains are untouched until you opt in.
## [0.42.47.0] - 2026-06-16
**A brain now travels with its own operating manual, and gbrain finally tells you how to run it better (gbrain#2180).** Two long-standing gaps closed. First: a brain repo can carry its own skillpack — skills authored for and versioned with that specific brain — and any harness that connects is offered it. Connect a fresh Claude Code or a thin client to a mature brain and it learns, on the spot, which meeting-ingestion or diligence protocol the brain expects, instead of starting blind. Second: gbrain stops being purely passive. `gbrain advisor` reads the brain's own state and hands back a ranked, read-only list of high-leverage actions — pending migrations, version drift, stalled backfills, low embedding coverage, setup smells — each with the exact command to fix it. It never acts on its own; it shows you and asks.
Discovery works on both connection topologies. Add a federated source that ships a brain-resident pack and gbrain prints what's in it and how to install it (with bounded, escalate-then-suppress nagging while it stays uninstalled — it never nags off a cron or an MCP call, only a real CLI prompt). Over MCP, a connecting agent calls `list_brain_skillpack` (source-scoped, so a multi-source brain attributes each pack to its source) and `get_skill --source_id` to fetch a specific pack skill. The advisor is also available over MCP behind its own gate, read-only, so a thin client can coach you in its own voice without ever exposing a fix it could run itself.
Nothing here forks the manifest, the installer, or the trust model — brain packs go through the same TOFU gate and SSRF allowlist as any third-party pack. Thin-client *binary* install (download-and-unpack) remains the separate, still-deferred PR2 work; today a thin client resolves a pack from its git source on its own machine.
### Added
- **Brain-resident skillpacks** — optional `brain_resident` + `schema_pack` fields on the v1 manifest (additive, forward-compatible); `gbrain skillpack init-brain-pack` scaffolds one (with a machine-parseable README a connecting harness can scan) pinned to the exact serving version so a pack can't install on a binary that lacks its ops.
- **Connect-time discovery**`gbrain sources add` surfaces a brain-resident pack and offers to install it; a new `list_brain_skillpack` MCP tool (source-scoped, `mcp.publish_skills`-gated) plus `get_skill --source_id` let a thin client discover and fetch per-source pack skills. `gbrain connect` now teaches agents to call it.
- **`gbrain advisor`** — ranked, read-only "what to do next" for this brain (version drift, pending migrations, schema-pack issues, stalled jobs/sync, embedding coverage, setup smells, uninstalled skills). `--json` with severity-based exit codes for CI/cron; `--apply <id>` runs one fix locally behind an explicit confirm (structured argv, never a shell). Exposed over MCP behind `mcp.publish_advisor` (default off, read-only).
- **`gbrain-advisor` bundled skill + weekly cron recipe** — teaches a harness to run the advisor on a cadence and ping you with what's new since last run.
- **Brain-pack version-skew lint**`init-brain-pack` validates each pack skill's declared `tools:` against the serving op set, so a pack fails loud on drift instead of silently half-working.
### Changed
- **The post-install/upgrade advisory is now state-aware and current.** It reads a single current-state recommended set (not a version-pinned constant) and the install ledger, and it speaks `gbrain skillpack scaffold` (the removed `install` verb is gone from the copy).
### To take advantage of v0.42.47.0
`gbrain upgrade`, then run `gbrain advisor` to see the top things worth doing on your brain right now. To publish a brain's skills to anyone who connects, run `gbrain skillpack init-brain-pack <name>` in the brain repo, fill in the README's five sections, and commit it. To let connecting agents discover packs over MCP, `gbrain config set mcp.publish_skills true`; to let a thin client run the advisor, `gbrain config set mcp.publish_advisor true` (both default off). Install the `gbrain-advisor` skill and its weekly cron recipe for a standing brain checkup.
## [0.42.46.0] - 2026-06-16
**Federated read scope now reaches every by-slug read, not just search and query (gbrain#2200).** A client that mounts several sources (a `federated_read` grant) could find a page through search and query, but the by-slug reads — `get_page`'s tags, plus `get_tags`, `get_links`, `get_backlinks`, and `get_timeline` — didn't honor the same grant. For a page living outside the default source that meant two wrong outcomes: the read came back empty for content the client was authorized to see, or it resolved against the wrong source. This release routes all of those reads through the same source-scope ladder that already governs search/query, so a federated client reads exactly the sources it's granted — no more, no less. Thanks to @mlobo2012 for the report and the proposed fix.
Link reads are scoped on every endpoint. A link connects up to three pages (the source page, the target, and the page that authored the edge); a federated read now constrains all three to the grant, so a link that crosses out of your granted sources doesn't surface a foreign page's slug. Untrusted remote callers carrying a single-source token get the same all-endpoint scoping; trusted local CLI keeps its cross-source view for link reconciliation and validators.
The semantic query cache was already corrected in v0.42.34.0 (cache rows key on the full source set, so a federated result can't be served to a caller with a different scope); this release closes the read-path half of the same theme.
### Fixed
- **By-slug reads honor the federated read grant.** `get_page` resolves a page's tags against that page's own source, and `get_tags` / `get_links` / `get_backlinks` / `get_timeline` route through the federated source scope. A multi-source client reads tags, links, backlinks, and timeline across exactly its granted sources (union), instead of falling back to a single source.
- **Link reads are scoped on all three endpoints** (source, target, and authoring page) under a federated grant, and untrusted remote single-source callers are scoped the same way — so a cross-source link can't disclose a foreign slug. Trusted local reads keep the full cross-source view.
### Changed
- The engine's `getTags` / `getLinks` / `getBacklinks` / `getTimeline` and `TimelineOpts` accept a `sourceIds[]` federated scope (precedence over the scalar source), mirroring `getPage` from v0.42.37.0. Write-side operations are unchanged — a read grant never widens writes.
### To take advantage of v0.42.46.0
`gbrain upgrade`. No configuration needed — federated clients immediately read tags, links, backlinks, and timeline across their full granted source set, and cross-source link reads stop surfacing foreign slugs. Single-source brains are unaffected.
## [0.42.45.0] - 2026-06-13
**The daily sync cron stops wedging on cost, and the embedding-spend estimate finally matches what a sync actually does (gbrain#2139).** On an active brain the inline-embed cost gate priced the *entire* corpus every time the working tree was dirty — which is always, since agents and crons write to it constantly — so a routine daily sync estimated ~158M tokens / ~$8 when the real delta was a few hundred files / ~$0.04, then blocked the cron with a confirmation it could never answer. Embeds silently stalled until someone noticed. The estimate now mirrors execution: it fetches first and prices only the files this run will pull and import, through the same diff machinery the sync itself uses. A brain whose commits are caught up but whose tree is dirty estimates $0, because an attached-HEAD sync imports only the committed diff.
When the gate does fire in a non-interactive session, it no longer exits with an error — it imports now and defers embedding to capped background jobs (which drain via the jobs worker or `gbrain embed --stale`), so a cron is never wedged again. Operators who have decided cost isn't the constraint get one switch — `gbrain config set spend.posture tokenmax` — that makes every cost gate informational across sync, reindex, enrich, and onboard (spend is still recorded; the switch removes the ceiling, not the accounting). The USD knobs accept `off` / `unlimited`, and every gate message now carries paste-ready commands so the controls are discoverable at the moment they fire.
This release also lifts the rule that blocked `--skip-failed` / `--retry-failed` under parallel sync — failure recovery no longer has to drop to `--serial` (which is what armed the inline gate in the first place).
### Added
- **`spend.posture` config** — `tokenmax` makes every embedding-cost gate informational (print the estimate, proceed, keep the ledger); `gated` (default) enforces as before. Documented end-to-end in `docs/operations/spend-controls.md`.
- **First-class off switches**`sync.cost_gate_min_usd`, `embed.backfill_max_usd_per_source_24h`, `embed.backfill_max_usd`, and `reindex --max-cost` / `enrich --max-usd` accept `off` / `unlimited` / `none`. No more sentinel values like `100000`.
- **Single-source `gbrain sync` cost preview** — plain `gbrain sync` previously embedded inline with no preview; it now carries the same gate as `sync --all` (auto-defers in non-TTY sessions, never blocks).
- **Self-describing gate messages** — every cost-gate / FYI line ends with the exact `gbrain config set` commands to widen, disable, or switch posture, plus a docs pointer.
### Changed
- **`gbrain sync --all` is no longer blocked by the cost gate in cron/agent contexts.** Above the floor in a non-interactive session it auto-defers embeds (exit 0) instead of emitting a `cost_preview_requires_yes` envelope and exiting 2. Cron wrappers that branched on exit 2 now see exit 0 with `status: "auto_deferred"`. A TTY still prompts `[y/N]`; `--yes` still embeds inline.
- **`--skip-failed` / `--retry-failed` now work under parallel sync.** The failure ledger is per-source and lock-serialized, so the previous "not supported under parallel — re-run with --serial" refusal is retired.
- **The six spend-control config keys are now first-class** (`gbrain config set` accepts them without `--force`).
### To take advantage of v0.42.45.0
`gbrain upgrade`. Nothing to configure for the headline fix — the daily sync cron stops wedging and the estimate is accurate out of the box. If you run a high-volume brain where cost genuinely isn't the constraint, `gbrain config set spend.posture tokenmax` makes every gate informational. To widen or disable a specific gate instead, see the table in `docs/operations/spend-controls.md` (e.g. `gbrain config set sync.cost_gate_min_usd off`). Failure-recovery syncs can now stay parallel: `gbrain sync --all --skip-failed` no longer forces `--serial`.
## [0.42.44.0] - 2026-06-13
### Fixed
- **Personal-brain tutorial points at the correct AlphaClaw site.** Step 4 of `docs/tutorials/personal-brain.md` ("Deploy via AlphaClaw on Render") linked to the wrong top-level domain, sending readers to a site that isn't the official AlphaClaw. The link now resolves to the right destination, so the deploy step works as written (gbrain#2165).
## [0.42.43.0] - 2026-06-12
**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it.
Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves.
This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure.
### Added
- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution.
- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C.
- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries.
- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt.
- **Transaction-mode pooler in the local CI gate**`bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after.
### Fixed
- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI.
- **Reflex pointer suppression under multi-turn windows** distinguishes "page already surfaced" from "entity merely mentioned earlier" — without this, window extraction would have suppressed every prior-turn entity by construction.
### To take advantage of v0.42.43.0
`gbrain upgrade` (applies the new feedback-log migration automatically). The wider reflex window is on by default for context-engine hosts — set `retrieval_reflex_window_turns: 1` in `~/.gbrain/config.json` to restore single-turn behavior. Agents without the context engine: call `volunteer_context` per turn (window in, pointers out), or pipe a transcript through `gbrain watch --json`. After a few days, `gbrain volunteer-context --stats` shows which resolution arms are earning their keep; raise or lower `min_confidence` accordingly.
## [0.42.42.0] - 2026-06-12
**`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever.
The teardown contract now lives in one place: every cli.ts disconnect site runs a bounded background-work drain and a bounded disconnect under a backstop whose deadline is computed from the bounds it guards (so it fires only when something violated its own bound), then the process exits explicitly — after fencing stdout/stderr and holding a short aliveness window so piped output is delivered (Bun queues pipe writes in a native buffer that only drains while the process is alive). The most-used command in the CLI now exits in milliseconds-to-a-couple-seconds instead of ten.
Along the way the wave fixed a deeper, silent bug: PGLite's WASM runtime writes its own status into `process.exitCode` at arbitrary points mid-run, which meant **every error exit on PGLite-engine brains has been reporting success (exit 0)** — scripts and agents keying on exit codes never saw failures. The CLI verdict now lives in a gbrain-owned channel that the WASM runtime cannot touch.
### Fixed
- **The flat 10s teardown tax + force-exit banner on transaction-mode poolers (gbrain#2084).** Queries exit promptly; the banner now appears only when a teardown component genuinely violated its own bound.
- **Slow operations are no longer killed mid-run with a false success.** The teardown deadline starts at teardown, never before the operation handler — a 30-second sync or a deep query runs to completion.
- **Error exits on PGLite report exit 1, not 0.** Failed operations (e.g. `gbrain get <missing-page>`) now exit non-zero on every engine; the exit code reports the operation, not the cleanup.
- **Piped output survives the exit.** Output is fenced and given a delivery window before the process exits, on every routed exit path including the backstop (the truncation class from gbrain#1959).
- **`gbrain doctor` no longer leaks its connection pool when DB checks throw**, and `dream`, `doctor`, `ze-switch`, and the search dashboards route their dispatcher teardown through the same bounded path (closing a long-standing drain gap on the overnight-cron path).
- **Daemon safety with space-separated global flags.** `gbrain --timeout 30s serve` is recognized as the daemon it is — the exit gate resolves the command exactly the way dispatch does.
### Added
- **`GBRAIN_TEARDOWN_DEADLINE_MS`** — env override for the teardown backstop deadline (incident escape hatch; the default is computed from the drain and pool bounds).
- **`GBRAIN_FLUSH_GRACE_MS`** — env override for the pre-exit output-delivery window (default 250ms on pipes, 0 on TTYs). Raise it when piping very large payloads into slow consumers; lower it for high-frequency scripted invocations that capture to files.
### To take advantage of v0.42.42.0
`gbrain upgrade`. No configuration needed. If your `gbrain query` has been printing results and then hanging ~10 seconds before a `force-exiting` banner, this release removes both the wait and the banner. If your scripts check gbrain exit codes on a PGLite brain, they will start seeing real failures — previously masked as exit 0 — so a wrapper that suddenly reports errors is the fix working, not a regression.
## [0.42.41.0] - 2026-06-11
**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test.
The headline is data durability. The `extract_facts` cycle phase reconciles a page's facts from its `## Facts` fence by deleting then reinserting — but conversation facts (written by `extract-conversation-facts`) live on pages that have no fence, so a cycle could delete them and reinsert nothing. A failed sync made it worse by escalating the phase to a full-brain walk. Both are fixed: the reconcile now protects non-fence (`cli:`-origin) facts, the destructive phase no longer inherits the failed-sync full-walk, and a reconcile that removes far more than it adds now reports `warn` instead of a silent `ok`.
Three more silent failures, each found in production and now self-healing or loud:
- **Timeline writes that silently stopped.** A migration renumbered during a merge could be recorded as applied without its index change ever running, so every timeline insert failed its `ON CONFLICT`. `gbrain` now repairs the index shape on every migrate pass (dedupe-then-rebuild, even when nothing is pending), `gbrain doctor` reports the drift, and the meetings extractor no longer swallows batch errors.
- **Autopilot crash-loop on transient DB errors.** The health-probe recovery called `connect()` without its config, so every reconnect threw and the process exited on any blip. It now uses `reconnect()`, which restores the captured config; `reconnect()` is a first-class method on both engines.
- **WAL corruption from concurrent PGLite processes.** A live, working process holding the data-dir lock could have it stolen after five minutes. The lock now heartbeats while held and is only reclaimed from a dead or genuinely stalled holder.
### Added
- **`timeline_dedup_index` doctor check + always-run repair (#2038).** Detects and heals a stale `idx_timeline_dedup` shape; `gbrain apply-migrations --force-schema` triggers the repair on demand. Reported by @jbarol.
- **`BrainEngine.reconnect()` on both engines (#2034).** Config-restoring reconnect, replacing the `disconnect()`+bare-`connect()` pattern.
### Fixed
- **Conversation facts survive a fence reconcile (#1928).** The cycle's per-page wipe now excludes non-fence (`cli:`) facts, the destructive phase no longer full-walks on a failed sync, and net-negative reconciles surface as `warn`.
- **`put_page` write-through no longer leaks into an unrelated source's repo (#2018).** It mirrors to the assigned source's own `local_path`; a source without one is skipped rather than written into the global repo path.
- **Autopilot survives transient DB errors instead of crash-looping (#2034).**
- **Concurrent PGLite processes no longer corrupt the WAL (#2058).** Heartbeat + steal-grace replaces the age-only stale-lock check.
- **Timeline migration drift self-heals; the extractor stops swallowing errors (#2038, #2057).** A Date-typed batch date round-trips correctly (verified by test).
- **A cwd `.env` `DATABASE_URL` no longer silently retargets the brain (#2064, closes #427).** Reported by @bomliu.
- **`gbrain sync --strategy code` honors `.gitignore` and skips `vendor`/`dist`/`build`/`venv` (#2052, #2020).** Reported by @aphaiboon and @dMac716.
- **Asymmetric embedding `input_type` reaches the wire across every openai-compatible recipe (#2033, supersedes #1400).** Query vectors are no longer document-typed. By @pabloglzg; original diagnosis by @billy-armstrong.
- **`updateSourceConfig` JSONB merge is atomic (#2074).** Eliminates a concurrent-writer lost-update race. Reported by @pai-scaffolde.
- **`gbrain doctor` correctness pass (#2075):** stale-lock hints, content sanity, graph coverage, exit code, gateway guard. Reported by @pai-scaffolde.
- **`gbrain search` returns results instead of exiting 0 empty on slow poolers; scoped `code-callers`/`code-callees` find their edges (#2073).** Migration v116 backfills NULL edge `source_id` and indexes `from_symbol_qualified`. Reported by @jbarol.
- **OAuth scope handling (#2009, #2072):** an omitted authorize scope now defaults to the client's registered grant (clamped to it, so no widening) instead of an empty grant that never self-heals; legacy token source grants are honored through a single shared scope parser. By @austinrarnett and @maxpetrusenkoagent.
### To take advantage of v0.42.41.0
`gbrain upgrade`. The timeline-index repair and migration v116 run automatically on the next migrate pass — if `gbrain doctor` flagged a timeline or call-graph problem before, re-run it after upgrade. No config changes required; the facts, write-through, autopilot, and lock fixes apply on restart.
## [0.42.40.0] - 2026-06-09
**`gbrain extract --stale` no longer aborts partway through a brain that contains emoji or other non-BMP characters.** On a large brain, link/timeline extraction could die with `invalid input syntax for type json` and commit nothing — and because the staleness bookmark only advances on a clean finish, every retry re-hit the same point and extraction stayed wedged. The cause: the link-context excerpt was sliced by raw UTF-16 index, so a window boundary landing inside an emoji's surrogate pair left an unpaired surrogate half in the text, which Postgres rejects when the batch is serialized to JSONB — taking down the whole batch, not just the one row. (PGLite is more permissive here, so this primarily bit the managed-Postgres engine.)
The fix well-forms free text before it is serialized: any unpaired surrogate half is replaced with the Unicode replacement character before it reaches the database. It is applied at the slicer and, as defense in depth, centrally at the batch-insert boundary for every free-text field — link context, timeline summary/detail/source, take claim/source — across both the batch and single-row write paths and both engines. Identity fields (slugs, source ids, holders) are deliberately left untouched, so a malformed identifier still fails loudly instead of being silently rewritten. The same well-forming helper now also backs the brainstorm prompt path, replacing a hand-rolled version that left back-to-back malformed characters half-cleaned.
### Fixed
- **`extract --stale` runs to completion on brains with emoji / non-BMP text (gbrain#2011).** The link-context slicer no longer leaves an unpaired UTF-16 surrogate that Postgres rejects at the JSONB cast and aborts the whole batch. A 192K-page brain that died at ~1,550 pages now sweeps clean.
- **Defense in depth at the batch-insert boundary.** Every free-text field written to JSONB (link context; timeline summary/detail/source; take claim/source) is now NUL- and surrogate-sanitized centrally, in both the batch and single-row paths and on both engines, so no future slicer can re-trigger this class of crash. Identity/security fields stay un-sanitized and fail closed.
### Changed
- **One shared well-forming primitive.** The brainstorm cross-prompt path now uses the same surrogate-cleaning helper, which also fixes a case where consecutive malformed characters were only half-cleaned.
### To take advantage of v0.42.40.0
`gbrain upgrade`. No configuration needed. If a `gbrain extract` run had been stalling at the same page count every time, this is the release that unsticks it — the next run resumes and completes.
## [0.42.39.0] - 2026-06-09
**Your agent now learns a brain page exists the moment you name someone — instead of talking about a close contact for four messages without ever opening their page.** gbrain was great at *storing* knowledge and at injecting deterministic per-turn context, but it never taught the agent the *policy* of retrieval: when to look something up, and what to pull. That lived in each user's hand-rolled instructions and failed silently. The Retrieval Reflex makes it a property of having a brain.
Two layers, on by default. A **deterministic pointer layer** in the context engine scans each turn's message for salient, resolvable entities (capitalized names, `@handles`) and injects a compact pointer — name → slug → one-line summary → "open the page before relying on details." Zero-LLM, fail-open, capped, and judgment-gated: it points, it never auto-dumps the page body, and it stays silent on trivial mentions or entities already in context. A **policy skill** (installed into your agent's resolver via `gbrain integrations install retrieval-reflex`) encodes the trigger policy and the pointer → full-page → graph-neighbors escalation ladder so the agent knows what to do with the pointer.
It works on every engine without leaking a second database connection. PGLite holds a single connection (your `gbrain serve` owns it), so the context engine resolves *through* the live holder over a local socket rather than opening its own — and on Postgres it uses a cached direct connection. Synopses run through the same privacy boundary `get_page` applies, so private facts never reach the prompt. `gbrain doctor` reports whether the reflex is actually firing.
### Added
- **Retrieval Reflex deterministic pointer layer (gbrain#1981).** The context engine injects compact, privacy-safe entity pointers per turn — on by default, zero-LLM, fail-open, capped. Disable with `GBRAIN_RETRIEVAL_REFLEX=false` or `retrieval_reflex: false` in `~/.gbrain/config.json` (file/env plane).
- **`retrieval-reflex` recipe + policy skill.** Installs the when/what-to-retrieve policy into your agent's resolver: `gbrain integrations install retrieval-reflex --target <host-repo>`.
- **`retrieval_reflex_health` doctor check.** Reports the deterministic layer's real runtime status (observed firing, resolve path, policy-skill install state).
### Fixed
- **Resolver-row install fence is now keyed by recipe id.** Installing a second `copy-into-host-repo` recipe previously wrote a block mislabeled with the first recipe's name; `--refresh`/uninstall now find their own rows.
### To take advantage of v0.42.39.0
`gbrain upgrade`. The deterministic pointer layer is on automatically — no config needed. To give the agent the matching policy skill, run `gbrain integrations install retrieval-reflex --target <your-agent-repo>`, then `gbrain doctor` to confirm `retrieval_reflex_health` is green.
## [0.42.38.0] - 2026-06-09
**Three independent job-layer bugs that left autopilot wedged or swallowed a command's output are fixed, each traced to source.** A triage of the job/lock/teardown layer (gbrain#1972) pulled them into one wave.
A crashed sync (OOM, a recycle, a kill) used to strand its lock row: the source looked "syncing" forever because reclaim only happened when something else came along and contended for the same lock. There was no background sweep, so a low-traffic source could sit falsely locked for a long time. Now every cycle reaps locks whose holder process is provably dead on this host — scoped to the sync/cycle lock namespaces, never to elections or the worker supervisor, and guarded against PID reuse so a recycled PID can never clear a live lock. `gbrain doctor --fix` runs the same reaper for brains that don't run autopilot.
Short one-shot CLI calls also got their full latency and output back. Database teardown could block for the full force-exit deadline against a transaction-mode pooler and then exit hard mid-write, which truncated the command's real output — the reason a relational query could come back empty even though the query itself worked. Teardown is now bounded by gbrain's own deadline instead of the connection driver's, so a short command returns in milliseconds with its output intact.
And the cooperative-abort work started in v0.42.29 (which only covered the embed phase) now covers every long phase a cycle runs — extract, fact extraction, and consolidation all check for cancellation between batches, so a cancelled cycle relinquishes its worker promptly instead of being force-evicted. A cancelled cycle also no longer records itself as a completed full run.
### Fixed
- **Stale dead-holder locks are reaped automatically (gbrain#1972, adjacent to #1470).** A background, host-scoped sweep at cycle start deletes `gbrain-sync:*` / `gbrain-cycle*` locks whose holder PID is dead, with a snapshot-matched delete that's safe against PID reuse and a 60s grace window. Other lock namespaces (elections, supervisor, reindex) keep their existing TTL behavior, untouched. `gbrain doctor --fix` reaps too, for no-autopilot brains.
- **One-shot CLI calls no longer hang on teardown or lose their output (gbrain#1959).** Pool disconnect is bounded by a gbrain-owned deadline (both pools closed concurrently) instead of blocking until the hard force-exit fired and truncated stdout. A short command returns promptly with intact output.
- **Cooperative abort now covers every long cycle phase (gbrain#1737 follow-up).** `extract` (incremental + full-walk), `extract_facts` (including its per-page embed and the phantom-redirect lock-retry), and `consolidate` check the abort signal between batches; `lint` yields periodically so it can be cancelled too. A cycle aborted mid-phase no longer stamps `last_full_cycle_at` as a completed run, and a new per-phase duration warning names any phase that overruns the worker's force-evict deadline.
### To take advantage of v0.42.38.0
`gbrain upgrade`. No configuration needed — the lock reaper, bounded teardown, and abort coverage are all on by default. If a source has looked stuck "syncing" with no live process, the next cycle (or `gbrain doctor --fix`) clears it automatically.
## [0.42.37.0] - 2026-06-08
**Cross-source reads now honor the caller's grant everywhere, a single bad frontmatter value no longer wedges a whole `lint`/`sync` run, and a handful of long-standing papercuts are gone.** A triage of the open issue backlog pulled the highest-impact bugs into one wave.
The headline is a source-isolation hardening pass. Every read that can be scoped to a source now resolves through one shared, fail-closed trust+grant check, so a remote client only ever sees the sources it was granted — whether it asks for one source, all sources, or reads a page by exact slug. Reads route the same way across query, the code-intel traversals, image search, and `get_page`. Legacy bearer tokens now carry the source grant an operator already stored on them, instead of being pinned to `default`.
On ingestion, a non-string frontmatter value (a bare number or date in `title:`, `slug:`, or `type:`) used to throw partway through and abort the entire run — so one malformed file could stop a whole brain from linting or syncing. Now those values are coerced to a usable string (a bare date `2024-06-01` becomes a real slug, not a crash), and `gbrain lint` flags the un-quoted field by name so you can clean it up.
Plus: `gbrain embed --catch-up` runs to completion instead of stopping after the first batch (and tells you when chunks genuinely can't be embedded); the frontmatter pre-commit hook actually matches `.md`/`.mdx` files now instead of silently doing nothing; the skill catalog shows the real description for skills that write it as a YAML block scalar; and `getConfig` retries through a transient connection blip instead of silently falling back to defaults.
### Fixed
- **Source-scoped reads honor the caller's grant across every read op (gbrain#1924, #1371, #1393).** One shared resolver replaces the per-op scope logic: a remote caller's "all sources" request is bounded to its grant, an out-of-grant source is refused, and `get_page`'s exact-slug path is scoped like every other read (both engines).
- **Legacy bearer tokens carry their stored source grant (gbrain#1336).** Tokens with an operator-set source grant read across exactly those sources instead of being limited to `default`.
- **Non-string frontmatter no longer aborts `lint`/`sync` (gbrain#1883, #1658, #1556, #1948).** Title/slug/type are coerced to usable strings instead of throwing mid-run, and `gbrain lint` reports the un-quoted field by name.
- **`embed --catch-up` runs to completion (gbrain#1946).** The mode no longer stops after one batch, and surfaces chunks that can't be embedded instead of looking like a clean finish.
- **Frontmatter pre-commit hook matches `.md`/`.mdx` files (gbrain#1840).** The installed hook was a silent no-op; it now validates staged markdown on commit.
- **Skill catalog shows block-scalar descriptions (gbrain#1711).** Skills written with `description: |` show their real text instead of a stray indicator.
- **`getConfig` retries on a transient connection blip (gbrain#1603)** instead of silently falling through to defaults (which surfaced as the wrong search mode / empty output on remote Postgres).
### To take advantage of v0.42.37.0
`gbrain upgrade`. No configuration needed. If `gbrain lint` now flags a `frontmatter-non-string-field` on a page, quote the value in that page's frontmatter (e.g. `title: "123"`). Reinstall the pre-commit hook with `gbrain frontmatter install-hook` to pick up the fixed matcher.
## [0.42.36.0] - 2026-06-08
**A huge `gbrain sync` that keeps getting killed now converges instead of restarting from zero.** On a high-write source — hundreds of thousands of files, a generator committing faster than each sync can drain — a full sync that ran past its launching session's timeout (SIGTERM) would lose 100% of its progress and re-import the entire backlog on the next run, forever. The bookmark never advanced, the source went quietly stale for hours while the importer burned CPU the whole time, and competing hourly launches stole each other's lock and raced. This release makes a large sync **resumable, durable, and single-flight** so it banks what it imports and picks up where it left off.
Progress is now banked into an append-only checkpoint as files drain, written through a direct session connection so it survives connection-pool exhaustion (the exact condition that used to silently drop every checkpoint write). The write is a delta — one row per drained file — instead of rewriting the whole completed-set each flush, so banking stays cheap even at hundreds of thousands of files. The bookmark still only advances on true completion, so a killed run resumes from the checkpoint rather than re-walking from zero. And the per-source lock now heartbeats through the direct pool and refuses to steal a holder that's alive and actively refreshing — so a long sync that overruns into the next scheduled run is skipped, not break-locked into a thrashing race.
### Fixed
- **Resumable sync survives pool exhaustion (gbrain#1794).** Checkpoint reads/writes route through the direct session pool with bounded retry; `EMAXCONNSESSION` / `too_many_connections` are now classified retryable. A killed run banks its progress and the next run skips already-drained files.
- **Guaranteed final flush on every exit path.** A cooperative timeout, an external SIGTERM (one-shot no-retry flush via the cleanup registry, ordered before lock release), and a clean finish all bank the in-flight delta. The bookmark is never advanced on a partial.
- **Fail-loud instead of burning CPU.** If checkpoint persistence fails repeatedly (pool genuinely dead), the run aborts with a `checkpoint_unavailable` partial rather than importing work it can never bank. Every partial/blocked exit now logs how many files were banked, so a killed run is never misread as total loss.
- **Lock thrash eliminated.** The import loop yields the event loop so the lock-refresh heartbeat fires mid-import; takeover refuses to steal a recently-refreshed (alive-but-starved) holder; a bare `gbrain sync` (no `--source`) now uses the refreshing lock too; and a cron sync that collides with a running one is reported as a skip, not a phase failure.
### Added
- **Append-only checkpoint storage** (`op_checkpoint_paths`, migration v115): one row per drained path; O(delta) writes instead of O(N²) full-set rewrites over a large sync.
### To take advantage of v0.42.36.0
- Nothing to do. `gbrain sync` is resumable by default — a killed sync now banks its progress and the next run converges. Five env knobs tune cadence, fail-loud threshold, event-loop yield, and lock-steal grace if you need them at incident time; see the "Sync resumability + lock tuning" section in CLAUDE.md.
## [0.42.35.0] - 2026-06-07
**A bookmark left pointing at a rewritten-away commit no longer freezes your brain in an endless full re-walk.** When a source's history is rewritten — a force-push, a `master``main` consolidation, a squash — the commit gbrain recorded as "last synced" can fall outside the branch's current history. The old guard treated that the same as a missing commit and fell back to re-importing the entire repository on every run. On a large brain with a cross-region database that full walk never finishes inside the sync timeout, so the bookmark never advanced and the source went quietly stale with no error surfaced.
The fix is a smaller, exact diff. `git diff A..B` compares two trees and does not require A to be an ancestor of B, so when the recorded commit's object is still on disk (the common case right after a rewrite) gbrain now diffs directly against it and imports only the real delta — the changed files, not the whole tree. A clear `[sync] last_commit … history rewritten` line marks the recovery. Only when the commit object is genuinely gone does sync fall back to a full reconcile, and that reconcile now also purges pages whose source files were removed — so a full sync is finally authoritative for deletes, not just imports (manually authored `put_page` pages and metafiles are never swept). Sibling of the v0.42.32.0 silent-staleness fix (gbrain#1939); closes gbrain#1970.
### Fixed
- **Sync recovers from an unreachable `last_commit` instead of full-walking forever (gbrain#1970).** A bookmark orphaned by a history rewrite is now diffed tree-to-tree directly when its object is still present, importing only the changed files; an oversized or failed diff degrades to a full reconcile instead of throwing. Only a truly-absent (gc'd) object forces a full reconcile.
- **A full sync now purges deleted files.** `performFullSync` reconciles deletions — pages whose backing file is gone are removed (gated to file-backed pages via `source_path`; manual `put_page` pages and metafiles are spared). This makes both the object-absent recovery path and every `--full` sync authoritative for deletes, not just imports.
- **Rename to an unsyncable path deletes the stale page.** A syncable file renamed to a non-syncable destination (which git reports as a rename, not a delete) now removes the old page instead of leaving it orphaned.
### To take advantage of v0.42.35.0
- Nothing to do. The next `gbrain sync` after upgrading self-heals a stuck bookmark automatically; watch for the one-line `[sync] last_commit … history rewritten` recovery message. If a source has been stale since a force-push or branch consolidation, this is the release that unsticks it.
## [0.42.34.0] - 2026-06-07
**Relationship questions now get relationship answers.** Ask "who invested in widget-co", "who introduced me to alice-example", or "what connects fund-a and fund-b" and gbrain resolves the named entity and walks its typed-edge graph (`invested_in`, `works_at`, `founded`, `attended`, `advises`, …) to surface the answer — even when no single page mentions both sides. Until now the graph only re-ranked results that keyword/vector search had already found; a relationship that lived purely in the edges (an investor whose page never names the company) was invisible. It now enters retrieval as a first-class candidate.
This is on by default in the `balanced` and `tokenmax` search modes, a pure no-op for non-relational questions and for brains with no typed edges, and off in `conservative`. On a benchmark of relationship queries whose answers are unreachable by content similarity, recall@10 goes from near-zero to over 75%. The traversal is deterministic (same query + brain → same answer), stays within a single source (it never crosses a mounted-brain boundary), excludes noisy body-text "mentions" edges by default, and is depth- and fan-out-bounded so a popular hub entity can't blow up a query.
### Added
- **Typed-edge relational retrieval** (`search.relational_retrieval`, on for balanced/tokenmax). Relational questions resolve their seed entity and traverse the typed-edge graph, injecting edge-derived answers as a fourth fusion arm alongside keyword + vector. Relation vocabulary is schema-pack-extensible: a pack that defines its own link types can declare the query phrases that retrieve them. The `query` operation gains a `relational` flag (omit for the smart default; pass `false` to force lexical/vector-only). Results carry `--explain` attribution ("surfaced via invested_in from widget-co") and, for "what connects A and B", the connecting path.
- **`relationalFanout` engine method** (PGLite + Postgres, in lockstep). Seed-array typed-edge fan-out aggregating to ranked nodes (shortest hop, edge richness, connecting path, canonical chunk), source-scoped and deterministic.
- **`gbrain eval retrieval-quality --ab-relational`**: A/Bs the arm off vs on over a question set and reports the recall@10 lift + latency. The retrieval-quality harness gains recall@k / recall@10 metrics.
### Fixed
- **Cross-source result collapse.** The search fusion/dedup key now carries `source_id`, so two pages that share a slug across mounted brains no longer merge into one result. The semantic query cache is likewise scoped per source-set, so a federated search can't be served a single-source cached result.
### To take advantage of v0.42.34.0
- Just ask relationship questions in natural language — the arm is on by default in balanced/tokenmax. To turn it off: `gbrain config set search.relational_retrieval false`.
- One-time cache note: this release advances the search-cache key version (a relational-on result must not be served to a relational-off lookup), so the first query after upgrade re-runs instead of hitting a stale cache row. No action needed; it self-heals on first use.
## [0.42.33.0] - 2026-06-07
**`gbrain sync` will never delete a repo it didn't create.** If a source was registered with a `remote_url` but its `local_path` pointed at a working tree you manage yourself (not a gbrain-managed clone), a failed or degraded code sync could remove that directory and re-clone over it. Sync now re-clones **only** clones gbrain actually created — identified by an ownership marker, or by gbrain's own clone location for clones made before this release. Anything else, including your live working tree, is treated as read-only: indexed, never deleted. On an unowned path, sync aborts loudly **before touching the filesystem** and tells you how to fix the source registration. Thanks to @zaqwery for the report.
The re-clone path is also crash-safer now: it clones into a sibling temp on the same filesystem and swaps atomically (old aside → new in → drop old), so a cross-device rename can't leave a source deleted-but-not-restored. If the swap ever fails, the error names exactly where your original clone is preserved.
### Fixed
- **`gbrain sync` never deletes an unowned working tree (gbrain#1881).** Re-clone is confined to clones gbrain created (`config.managed_clone` marker, or the default clone location for pre-marker clones). A `remote_url` source whose `local_path` is your own working tree is synced read-only and refused — loudly, before any filesystem op — never removed. `gbrain sources restore` on such a source warns and keeps the tree instead of deleting it. Reported by @zaqwery.
- **Safer re-clone swap.** Re-clone uses a same-filesystem sibling temp plus an atomic swap (no cross-device "deleted but not re-cloned" window); a symlinked clone path is refused; a failed swap reports where the original is preserved.
### To take advantage of v0.42.33.0
Upgrade. Nothing to configure. New `--url` sources are marked gbrain-owned automatically, and existing managed clones at the default location keep auto-recovering. If you registered a source whose `local_path` is a working tree you maintain yourself, `gbrain sync` now syncs it read-only and prints how to re-register it if you want gbrain to manage the clone.
## [0.42.32.0] - 2026-06-07
**A single un-parseable note can no longer silently stop your brain from indexing anything new.** A page whose YAML frontmatter `title:` was a bare date (`title: 2024-06-01`) or number (`title: 1458`) parsed as a Date/number, not text — and the importer threw when it tried to lowercase it. That throw blocked the sync bookmark from advancing, so every later `gbrain sync` re-walked the whole repo, never reached HEAD, and quietly stopped indexing new commits. The page was committed and on GitHub, but `gbrain get` returned `page_not_found` with no surfaced error.
@@ -16251,8 +16878,7 @@ The OAuth provider in `src/core/oauth-provider.ts` got a parallel hardening pass
Smaller hardening: admin cookies set `Secure` when behind HTTPS or a public-URL proxy (F9), magic-link nonces are bounded by an LRU cap (F10), `/mcp` wraps `transport.handleRequest` in try/catch so SDK throws hit a JSON-RPC 500 instead of express's default HTML error page (F14), and OperationError + unexpected exceptions both route through the unified `buildError`/`serializeError` envelope (F15). DCR disable became a constructor option on the provider rather than a serve-http monkey-patch (F12 — cleanup, not security).
To take advantage of v0.26.9
============================
=====================
`gbrain upgrade` is a one-step upgrade. There is no migration; all changes are application-layer.
1. **Upgrade.** `gbrain upgrade`. Confirm `gbrain --version` shows `0.26.9`.
@@ -16386,8 +17012,7 @@ Both run at `--max-concurrency=1` after the parallel pass, same as the existing
Wallclock observed: 74s on a Mac dev box (running `bun run test` with the new quarantines). Already at the v0.26.9 informational target. The full intra-file marker flip (with codemod + per-file `test.concurrent()`) lands in v0.26.9 and aims for the same ≤60s with pinned config.
To take advantage of v0.26.7
============================
=====================
`gbrain upgrade` does nothing functional in this release — it ships test infrastructure, not user-facing code. But if you contribute tests:
1. **Run `bun run verify` before pushing.** The new `check-test-isolation.sh` runs alongside the privacy + jsonb + progress checks. Catches new env-mutation, mock.module, and PGLite-pattern violations before CI does.
+107 -5
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -59,9 +59,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
hand-roll source filtering — a missed thread is a cross-source data leak.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
Guarded by `scripts/check-jsonb-pattern.sh`.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -97,6 +102,8 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
@@ -149,6 +156,7 @@ project resolves through `src/core/search/mode.ts`.
| `intentWeighting` | true | true | true |
| `tokenBudget` | **4000** | **12000** | **off** |
| `expansion` (LLM multi-query) | false | false | **true** |
| `relationalRetrieval` | false | **true** | **true** |
| `searchLimit` default | 10 | 25 | 50 |
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
@@ -207,6 +215,19 @@ written against `embedding` (1536d OpenAI). Existing v=2 rows become
unreachable on first re-query (one-time miss spike on upgrade);
`mode.ts:KNOBS_HASH_VERSION` is the single source of truth.
**v0.42.34.0 knobs_hash v=9 → v=10.** Folds the `relationalRetrieval` knob +
depth into the cache key so a relational-on result set can't be served to a
relational-off lookup (same contamination class as graph_signals). One-time
miss spike on upgrade.
**Relational retrieval (v0.42.34.0).** `relationalRetrieval` (on for
balanced/tokenmax) adds a fourth recall arm: a relational query ("who invested
in X", "what connects A and B") resolves its seed entity and walks the typed-edge
graph (`src/core/search/relational-recall.ts` + `relational-intent.ts`,
`engine.relationalFanout`), injecting edge-derived answers into RRF. Within-source,
deterministic, mentions-excluded by default, pure no-op for non-relational queries.
The `query` op's `relational` flag forces it on/off per call.
**Three CLI surfaces:**
gbrain search modes # what is running, with per-knob attribution
@@ -238,7 +259,7 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
@@ -257,6 +278,17 @@ routing is narrowed to what the skill actually covers.
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
(agent-readable health report).
**Brain-resident skillpacks + advisor (v0.42.47.0, #2180):** A brain repo can carry its
own publishable skillpack (`brain_resident: true` in `skillpack.json` + `schema_pack`);
`gbrain skillpack init-brain-pack` scaffolds one with a 5-section machine-parseable README.
Connecting harnesses discover it on `gbrain sources add` (Topology A advisory, bounded nag
via `nag-state.ts`) and over MCP via the source-scoped `list_brain_skillpack` op +
`get_skill --source_id` (gated by `mcp.publish_skills`). The bundled `gbrain-advisor` skill
+ `gbrain advisor` op compute a ranked, read-only list of high-leverage actions from brain
state (8 collectors in `src/core/advisor/`); `--json`+exit codes for CI/cron, local-only
`--apply <id>` behind confirm, exposed over MCP behind `mcp.publish_advisor` (default off,
read-only on remote). Thin-client binary install stays deferred to PR2 `build_skillpack`.
**Routing-table compression (v0.32.3.0):** `skills/functional-area-resolver/` —
two-layer dispatch pattern for shrinking large AGENTS.md / RESOLVER.md files
(>=12KB) without losing routing accuracy. Replaces one row per skill with one
@@ -356,6 +388,76 @@ For background tasks (`run_in_background: true`), the harness captures the exit
file separately — use it via the bg task's `<id>.exit` file, not the streamed
output.
## Sync resumability + lock tuning (v0.42.x, #1794)
`gbrain sync` is resumable and converges under pool exhaustion + repeated kills.
Progress banks into the append-only `op_checkpoint_paths` table (one row per drained
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
run resumes from the checkpoint and `last_commit` only advances on true completion. The
per-source lock heartbeats through the direct pool and refuses to steal a live,
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
hatches — no config-dashboard surface by design):
| Env var | Default | What it does |
|---|---|---|
| `GBRAIN_SYNC_CHECKPOINT_EVERY` | 1000 | Flush the checkpoint every N drained files. |
| `GBRAIN_SYNC_CHECKPOINT_SECONDS` | 10 | Also flush every N seconds (whichever comes first) — bounds worst-case loss regardless of throughput. Flush also fires after the first file. |
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
## Pace Mode (DB-contention-aware backfill pacing)
A naive `gbrain embed --stale` / large `sync` can saturate a PgBouncer
transaction-mode pooler and starve the minion supervisor's lock renewals
(`lock-renewal-failed` → dead jobs). Pacing is the native, composable fix — it
replaces external SIGSTOP/SIGCONT wrapper scripts. **Opt-in: default mode `off`.**
The composable primitive is `src/core/db-pacer.ts` (`createDbPacer`):
- **Concurrency cap is the real lever** (caps simultaneous in-flight DB writes =
pooler slots held). Embed paths set their worker count to `maxConcurrency`
(single pool, no permit); `sync` uses the shared `acquire()` **permit** because
each parallel worker owns a separate engine (one budget must span pools).
- **In-band signal** (`observe(ms)` EWMA from the work's own queries — never
blind the way an out-of-band probe pool was). **No probe loop, no
`probeLatency` engine method.**
- **Cooperative `pace()` sleep** on `setTimeout` (keeps the lock heartbeat
firing), jittered to avoid a thundering-herd resume. `acquire()`/`pace()` throw
`AbortError` on cancel; everything else is fail-open (a pacer bug never kills a
backfill, never throws an unhandledRejection).
Named bundles resolve through `src/core/pace-mode.ts` (`resolvePaceMode`), mirror
of the search-mode pattern but with **env ABOVE config** (incident escape hatch):
per-call flag → GBRAIN_PACE_* env → config (pace.*) → PACE_BUNDLES[mode] → off
| Knob | off | gentle | balanced | aggressive |
|---|---|---|---|---|
| `maxConcurrency` | (off) | 4 | 8 | 16 |
| `paceAtMs` (EWMA → sleep) | — | 250 | 500 | 1000 |
| `maxSleepMs` (jittered cap) | — | 2000 | 1500 | 1000 |
**Surfaces.** `gbrain embed --stale --pace[=mode]` (bare `--pace` = balanced),
`--pace-max-concurrency=N`. `--background` carries explicit pace OVERRIDES (not
the resolved bundle) into the `embed` job payload; the handler re-resolves
env>config>bundle at execution so `GBRAIN_PACE_*` still wins (CX5). Config-level
`pace.mode` paces EVERY `runEmbedCore` caller (cycle embed, embed-catch-up,
sync-auto-embed) and the prod `embed-backfill` job automatically. `sync` reads
env/config. PGLite / mode `off` → no-op pacer.
**Correctness fixes pacing bundles** (longer paced runs widen these): CLI
`embed --stale` single-flights via the SAME per-source lock key as the
`embed-backfill` handler (`src/core/embed-backfill-lock.ts`; all-source runs lock
every source in sorted order) so a hand-run backfill and a queued job can't race
the NULL→non-NULL upsert (`TODOS:2299`); a **bounded** end-of-run keyset re-entry
(max 3 + forward-progress, paced runs only) catches rows inserted behind the
cursor (`TODOS:2301`); and the embed wall-clock budget timer is re-armed around
`pace()` sleeps so paced time doesn't burn the work budget.
`EmbedResult.pacing` carries the end-of-run telemetry (cap, samples, EWMA, slept
ms, max waiters) for `--json`; a one-line summary prints to stderr.
## Build
`bun build --compile --outfile bin/gbrain src/cli.ts`
+20
View File
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
@@ -289,6 +307,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
## Troubleshooting
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
+12
View File
@@ -87,6 +87,18 @@ and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
without `--enable-dcr`. That footgun is gone.
### DCR consent default (v0.42.55+)
The "disable `client_credentials`, only allow `authorization_code`" guidance
above is now the built-in default for the DCR path, not just advice for custom
wrappers. With `--enable-dcr` on, a self-registered client defaults to the
`authorization_code` (browser-approval) grant, and an explicit
`client_credentials` request is rejected with `invalid_client_metadata`.
Operators who genuinely need the machine-to-machine grant on the registration
endpoint opt in with `--enable-dcr-insecure` (which implies `--enable-dcr`); a
startup WARNING prints whenever DCR is enabled, and a second when the insecure
grant is allowed. Pre-registering clients via the CLI / admin API is unchanged.
### Token Management
```bash
+511 -18
View File
@@ -1,5 +1,453 @@
# TODOS
## community fix-wave follow-ups (filed v0.42.60.0)
- [ ] **P1 — take-writes source scoping fails open when source resolution errors (#2684 residual).**
`resolveTakesSourceId` (src/commands/takes.ts) swallows resolution errors and returns
`undefined`, which falls back to the unscoped slug-only page lookup — so an invalid
`GBRAIN_SOURCE` (or a broken dotfile chain) silently restores the pre-#2698 cross-source
write behavior on multi-source brains. Decide fail-closed semantics: error out when a
source was explicitly requested but doesn't resolve; keep the unscoped fallback only for
brains with no source configuration at all. Add a regression test for the invalid-source
path. Found by cross-model adversarial review during the v0.42.60.0 release ship.
- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent`
before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered.
## v0.42.59.0 follow-ups (five-fix rollup #2735#2739)
Filed as follow-ups from v0.42.59.0 (bootstrap probe for
`timeline_entries.event_page_id`, migrate-engine source catalog + target-aware
resume, entity-resolution quarantine, escape-aware fence cells, think gather
source scope).
- [ ] **P2 — schema-bootstrap-coverage strip block never exercises `timeline_entries.event_page_id`.**
The guard's pre-migration-brain simulation (the strip DDL in
`test/schema-bootstrap-coverage.test.ts`) has no
`ALTER TABLE timeline_entries DROP COLUMN IF EXISTS event_page_id` (or FK drop), so the
coverage entry added for the v121 forward reference is vacuous — the probe never fires
under that harness. The real regression guard lives in `test/bootstrap.test.ts` (which
does drop → re-bootstrap → assert). Add the DROP statements to the strip block so the
coverage test genuinely exercises its own entry.
- [ ] **P2 — extract-facts reconcile still wipes-then-reinserts when the parse emitted MALFORMED warnings.**
`runExtractFacts` (`src/core/cycle/extract-facts.ts`) deletes a page's facts and
reinserts from the parsed fence even when `parseFactsFence` surfaced
`FACTS_TABLE_MALFORMED` warnings — any future parse defect becomes a deletion vector
(rows the parser failed to read get wiped with nothing to reinsert). Consider
skip-wipe-on-warnings: treat a warning-bearing parse as non-authoritative for that page
(skip the wipe, surface a warn), mirroring the empty-fence legacy-row guard's posture.
- [ ] **P3 — bare-name resolution quarantines even on an exact unique match when prefix siblings exist.**
With pages `companies/acme` + `companies/acme-labs`, a bare `"Acme"` yields two
`findPrefixCandidates` rows, so `tryUnambiguousPrefixExpansion` declines — even though
`companies/acme` is an exact `dir/token` slug match (and may be a unique exact title
match). That's an unambiguity signal being wasted. Consider promoting an exact
`dir/token` (or exact-title) hit above the sibling-count check in
`src/core/entities/resolve.ts`.
- [ ] **P2 — `scripts/run-verify-parallel.sh` no-gtimeout fallback reports the watchdog's exit code, not the check's.**
In the fallback branch, `rc=$?` is captured after `wait "$cap_pid"` (the killed
sleep-watchdog, rc=143) rather than after `wait "$pid"` (the actual check) — on a Mac
without coreutils every check false-fails with rc=143. Capture `rc` from `wait "$pid"`
first, then reap the watchdog.
- [ ] **P3 — same-target migrate resume with `--force` still skips checkpointed pages after the wipe.**
`gbrain migrate --to <engine> --force` wipes the target's pages, but the resume
manifest's `completed_slugs` filter still applies, so previously-checkpointed pages are
skipped against the now-empty target (pre-existing behavior; the v0.42.59.0 verification
warns about it). `--force` should clear the manifest when it matches the same target.
Where: `src/commands/migrate-engine.ts`.
- [ ] **P2 — think residual scope gaps.** Two spots in `src/core/think/index.ts` don't yet
inherit the caller's source scope the way the gather stage now does:
`persistCitations` resolves citation slugs with an unscoped
`SELECT id FROM pages WHERE slug = $1 LIMIT 1` (cross-source slug ambiguity can attach
saved evidence to the wrong same-slug page), and the trajectory entity-resolution scalar
is `opts.sourceId ?? 'default'` (a federated caller with `allowedSources` but no scalar
resolves entities against `default` instead of its grant). Mirror the gather-stage
precedence (federated array > scalar > default) at both sites.
## provider-agnostic follow-ups (filed v0.42.58.0)
Deferred from the provider-agnostic plumbing wave (#1249/#1250/#1292/#2271/#2209).
Plan + review trail at `~/.claude/plans/system-instruction-you-are-working-keen-newell.md`.
The eng-review + Codex outside-voice narrowed the wave to these deferrals:
- [ ] **P2 — Capability-aware query expansion on OpenAI-compat providers (#2372).**
Expansion only runs for recipes that declare an `expansion` touchpoint, and only the
native providers (anthropic/openai/google) do. To make expansion work on
litellm/openrouter/groq/together/deepseek you must ADD expansion touchpoints to those
chat-capable recipes AND add a `generateObject``generateText` capability fallback for
backends without strict structured outputs. Feature-shaped; overlaps the general
OpenAI-compat proxy story (`docs/designs/COMMUNITY_IDEAS.md`). Community PR #2373 is a
starting point. Where: `src/core/ai/gateway.ts:expand`, recipe files, `types.ts` (ExpansionTouchpoint).
- [ ] **P2 — LiteLLM as a chat/expansion backend.** `litellm-proxy` declares ONLY an
embedding touchpoint, so `think`/chat on LiteLLM is dead. Add chat (and expansion) so a
LiteLLM proxy is a full LLM backend, not embedding-only. The general OpenAI-compat proxy story.
- [ ] **P3 — Per-model embedding dims metadata on `EmbeddingTouchpoint`.** `default_dims`
is recipe-wide, so a recipe (ollama) can't carry different native dims per model. This
wave added the modern ollama model NAMES + a `trust_custom_dims` passthrough (user supplies
`--embedding-dimensions`); per-model dims would let gbrain pick the right default. Then
ollama could fail-closed at preflight like litellm/llama-server instead of at first embed.
- [ ] **P3 — Google native baseURL normalization (#1250 follow-up).** `resolveNativeBaseUrl`
covers anthropic + openai; Google was deferred because Gemini's native suffix is unproven
(its OpenAI-compat route is `/v1beta/openai`). Verify the correct `@ai-sdk/google` suffix,
then add `google` to the helper. Where: `src/core/ai/gateway.ts:resolveNativeBaseUrl`.
- [ ] **P3 — Fold Voyage/Google/LiteLLM/OpenRouter API keys into `buildGatewayConfig`.**
It folds only OPENAI/ANTHROPIC/ZEROENTROPY file-plane keys today, so `config.json`-set keys
for other providers only work if also in `process.env`. Extend the mapping. Where:
`src/core/ai/build-gateway-config.ts`.
- [ ] **P3 — OpenRouter per-model custom-dim handling.** OpenRouter declares recipe-wide
`dims_options` and mixes fixed-dim + arbitrary models, so it's excluded from `trust_custom_dims`.
A per-model story would let OpenRouter accept custom dims for models that support them.
- [ ] **P1 — Gateway subagent-loop tool-result persistence + Date normalization (#2273/#2256).**
Confirmed crash-block: non-Anthropic subagent jobs dead-letter after any interruption
(tool-result user turns aren't persisted; raw Date values fail the AI SDK's strict JSON
check). Larger self-contained change with 6 competing community PRs
(#2274/#2257/#1934/#2065/#2112/#2336) — pick one canonical impl, preserve authorship.
This is the immediate fast-follow to the provider-agnostic wave. Where:
`src/core/ai/gateway.ts:toolLoop`/`toModelMessages`, `src/core/minions/handlers/subagent.ts`.
## Life Chronicle follow-ups (filed v0.42.56.0, #2390)
Deferred from the Life Chronicle wave (CEO Scope-Expansion + eng review CLEARED,
3 codex rounds absorbed, PR #2533). Every item was an explicit review decision,
not an oversight; each names its decision provenance.
- [ ] **P1 — Eval-gated auto-emit default-flip (D5.5 fast-follow).** Auto-emission
ships OFF (`auto_chronicle=false`) per spend/consent posture. The headline
fast-follow: run `gbrain eval chronicle` + a live-LLM OFF-vs-ON agent arm on a
real brain, and if the lift holds, flip the default ON in the next minor with
an upgrade notice. Where: `src/core/chronicle/config.ts`, upgrade banner in
`src/commands/upgrade.ts`.
- [ ] **P2 — Live-LLM OFF-vs-ON eval arm + LongMemEval temporal slice.** The
shipped `gbrain eval chronicle` is the deterministic CI bar (6 gold tasks).
The full North-Star proof adds (a) a live agent reconstructing a day with the
chronicle ops ON vs OFF, and (b) the LongMemEval `question_type:
temporal-reasoning` slice as secondary corroboration — verify the adapter can
filter by question type first. Where: `src/eval/chronicle/harness.ts`,
`src/commands/eval-longmemeval.ts`.
- [ ] **P2 — Passive diary capture + consent model (D3.5/E5).** Active-only in v1
by explicit decision (highest consent-risk surface). Passive detection of
first-person interiority in transcripts requires a dedicated consent design:
an explicit `chronicle.diary.passive` opt-in, a consent prompt, and
provenance-aware redaction (the facts `visibility` lane is already in place).
- [ ] **P2 — Ontology interval-splitting for backdated conflicts (G4).** A
backdated observation whose validity window overlaps an existing row is
flagged (not rewritten) in v1. Real interval algebra (split the prior window
around the backdated fact) is deliberate follow-up scope; the conflict lane
(`findOntologyConflicts`) is the holding surface. Where: both engines'
`mergeOntologyFact`.
- [ ] **P3 — Cross-brain federated timeline (D3.6/E6).** v1 holds source
isolation (scoped-default, `--all-sources` opt-in within the host brain).
Unifying across mounted team brains is its own epic with an access-policy
surface.
- [ ] **P3 — Place-as-entity (`gbrain where <venue>`).** `event.where` is
captured as free text; resolving venues to entity pages + geo-adjacency
queries is a follow-up.
- [ ] **P3 — Richer meta-ontology dashboard.** `gbrain ontology-dimensions` is
the v1 surface; a full dashboard (per-dimension drill-down, quarantine review
queue for novel dimensions) is deferred until usage shows demand.
- [ ] **P3 — Materialized daily timeline pages / emotional-arc view.** The
query-time aggregator won D5.6; embeddable `life/timeline/YYYY/MM/DD.md`
narrative pages (a single `materialize_timeline` cycle phase) revisit after
the eval shows `reflect`-style recall needs them.
## reliability fix-wave follow-ups (filed v0.42.52.0)
Deferred from the autopilot/supervisor + sync/status/minion reliability wave
(plan-eng-review + codex + adversarial diff review CLEARED). Both surfaced by the
ship-stage pre-landing review; neither blocks the wave.
- [ ] **P2 — Thread a cancellation signal through `importFile` (#1950).** The sync
stall watchdog aborts `opts.signal`, but the per-iteration abort checks observe
it BETWEEN files — a hang inside one `importFile` call (e.g. a stuck embed
network request) isn't interrupted until that call returns. Thread an
`AbortSignal` into `importFromContent`/`importFromFile` and check it at the async
phase boundaries (post-parse, pre-embed, pre-DB-write) so an in-flight wedge is
reaped too. Core hot path (engine-parity + downstream-client surface) — scope it
on its own. Where: `src/core/import-file.ts`, `src/commands/sync.ts`.
- [ ] **P3 — Centralize live-sync liveness onto `liveSyncStatus` (#1950).**
`gbrain sources status` now uses the shared `liveSyncStatus(engine, sourceId)`
helper; retrofit `gbrain doctor` (its own inline lock probe) and `gbrain status`
onto the same helper so there's one source of truth for "is this source
syncing." Where: `src/core/db-lock.ts`, `src/commands/doctor.ts`,
`src/commands/status.ts`.
## Pace Mode follow-ups (filed v0.42.49.0)
Deferred from the paced-backfill wave (CEO + eng review CLEARED). Core shipped:
`db-pacer` + `pace-mode` wired into embed (CLI + shared core + `embed-backfill`
job) and sync. See CLAUDE.md "Pace Mode".
- [ ] **P2 — `doctor` pacing check (E2).** Detect a txn-mode pooler (port 6543)
running unpaced bulk and recommend `--pace`; optionally correlate recent
`minion_jobs` deaths with backfill windows. Where: `src/commands/doctor.ts`.
- [ ] **P2 — `--pace=auto` autotuned thresholds (E3).** Derive `paceAtMs`/cap from
observed baseline latency (rolling median) instead of fixed bundle values,
mirroring `gbrain search tune`. Needs a baseline window + cold-start default +
config persistence — not a small add. Where: `src/core/pace-mode.ts` +
`src/core/db-pacer.ts`.
- [ ] **P3 — First-class pacing in more minion job handlers (E5).** `embed-backfill`
is paced; extend to `extract`/`embed-catch-up`/contextual-reindex handlers with
supervisor-detection downgrade. Today these inherit config/env pacing only when
they call `runEmbedCore`.
- [ ] **P1-companion — Supervisor concurrency 3→2 + job-kind slot fairness (E7).**
The daemon-side root cause the external wrapper's probe was blind to:
`embed-backfill`/`autopilot-cycle` jobs can occupy all supervisor slots
(`:215` below). Pacing makes backfills safe; this fixes the residual death rate.
Where: `src/core/minions/supervisor.ts` + queue slot accounting.
- [ ] **P3 — `gbrain sync --pace` CLI flag.** Sync reads env/config pacing today;
add a per-run `--pace[=mode]` flag for symmetry with `embed`. Where:
`src/commands/sync.ts` arg parsing.
- [ ] **P3 — Real-PG e2e for pacing.** Gated on `DATABASE_URL`: paced
`embed --stale --pace --progress-json` caps concurrency + emits telemetry;
single-flight rejects a 2nd concurrent run; lock heartbeat advances during a
paced sleep (short-TTL). Unit coverage (`db-pacer`/`pace-mode`) already ships.
## brain-repo durability follow-ups (filed v0.42.48.0)
- [ ] **P3 — gbrain write-path calls commit-push synchronously when durability is on.**
v0.42.48.0 ships the synchronous `brain-commit-push.sh` as the guarantee and a local
post-commit hook as a best-effort fallback. The strongest durability (codex outside-voice
D13-C) is to have gbrain's own write-through path call the commit-push helper synchronously
when a source is hardened — that also covers writes that never get committed by an agent.
Deferred because it touches the write path; the hook + mandated helper cover the
agent-driven case today.
- **Where to start:** `src/core/write-through.ts:writePageThrough` + a per-source "hardened"
flag to gate the synchronous push.
- [ ] **P3 — Unify the durability pull cron with autopilot's OS-scheduler.**
v0.42.48.0 ships a minimal launchd/crontab installer inside `brain-repo-durability.ts`
(D12: minimal-now to keep the diff off the load-bearing autopilot feature). Extract a shared
`os-scheduler.ts` (`installPeriodic`/`removePeriodic`) and have both autopilot and brain-pull
call it, so there's one OS-cron path.
- **Where to start:** `src/commands/autopilot.ts` (`installLaunchd`/`installSystemd`/
`installCrontab`/`writeWrapperScript`) + `brain-repo-durability.ts:installDurabilityCron`.
## gbrain#2200 federated-read follow-ups (filed v0.42.46.0)
- [ ] **P1 — Close the federated-read scope on the remaining same-class by-slug read ops.**
v0.42.46.0 (#2200) routed `get_page` tags + `get_tags` / `get_links` / `get_backlinks` /
`get_timeline` through the federated source scope and taught the engine methods to honor
`sourceIds[]`. The adversarial review (Codex + Claude) flagged sibling read ops in the
SAME class that still use scalar-only `ctx.sourceId ? {sourceId} : {}` and never thread
`ctx.auth.allowedSources`: `get_chunks`, `get_raw_data`, `get_versions`, `resolve_slugs`
(the standalone op — `resolve_slugs` passes NO scope at all), plus (per the v0.42.55.0
eng-review codex pass) `takes_search` (`operations.ts:1727` — holder-allowlist only, no
`sourceScopeOpts`) and `code_def` (`operations.ts:4155` — brain-wide raw SQL over
`content_chunks`; confirm whether brain-wide is intentional before scoping). A remote
federated client (grant set, dispatch-default `ctx.sourceId='default'`) reads these against
`default` or unscoped, not its grant.
- **Why:** same cross-source correctness/isolation class #2200 targets; a federated client
can't read chunks/raw-data/versions for an authorized non-default source, `resolve_slugs`
can fuzzy-resolve across all sources, and `takes_search`/`code_def` query without the grant.
The #2399 close-list deliberately did NOT blanket-close #1371/#2200 because of these residual
surfaces — close those issues only after this TODO lands.
- **How to start:** mirror the #2200 pattern — route each handler through `sourceScopeOpts(ctx)`
(or `linkReadScopeOpts` if a far endpoint exists), add `sourceIds?: string[]` to the engine
methods (`getChunks` / `getRawData` / `getVersions` / `resolveSlugs` / the takes-search +
code-def queries) with `source_id = ANY($::text[])` precedence, and add federated/isolation
tests + engine-parity arms.
- **Depends on:** nothing; #2200 established the pattern and the `linkReadScopeOpts` helper.
## Spend-controls wave follow-ups (filed v0.42.45.0, #2139)
Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-lovely-balloon.md`.
- [ ] **P3 — Measured post-import chunk-count gating (#2139 proposal 2b).**
**What:** Gate the inline cost decision on the actual chunk count sync produced
(known after import, before embedding) instead of the pre-sync token estimate.
**Why:** A fully execution-accurate gate with zero estimate error. **Context:**
After v0.42.42.0 the estimator already mirrors execution (fetch-first delta via the
shared `computeSyncDelta`, `--full`=delta+stale, dirty-tree→$0). This is the
belt-and-suspenders fallback if a future case still drifts. **Trigger:** only if the
delta estimator proves insufficient in practice. **Start:** the gate call site in
`src/commands/sync.ts` (`runInlineCostGate`), gate on post-import `chunksCreated`.
- [ ] **P3 — Per-source defer granularity (#2139, D8A road-not-taken).**
**What:** When the aggregate inline gate trips in a non-TTY session, defer embeds
only for sources above a per-source floor; let cheap sources keep embedding inline.
**Why:** Cheap sources would get embeddings minutes sooner instead of waiting for a
backfill-worker drain. **Context:** v0.42.42.0 chose GLOBAL defer (one flag, strictly
dominates the exit-2 it replaced). This is the granularity upgrade. **Trigger:** a
filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates
through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan.
## gbrain#2095 push-based context follow-ups (v0.43+)
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria
are the bar). Plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`.
- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover
pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered
pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
**Blocked by:** a real consumer (revisit when one exists).
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
needed doctor visibility because silent failure was invisible; volunteer is
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
agents not discovering the surface, ship a `push-context` recipe (mirror
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
**Where:** `recipes/`, `src/commands/doctor.ts`.
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
If MCP callers accumulate parsing bugs, add a structured array param beside it.
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver
(`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE
ANY('%/...')`) predates #2095 but now runs per turn on three channels
(reflex window, volunteer_context, watch) federated across sources. Neither
the leading-wildcard suffix arm nor `lower(title)` is index-served. If
per-turn latency telemetry on large brains comes back hot: add
`(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or
gin_trgm) index, or split the OR into three index-friendly queries.
**Where:** `src/core/context/retrieval-reflex.ts`, migration.
- [ ] **P3 — batch the volunteer-events pruner's first run after a long gap.**
`purgeStaleVolunteerEvents` is one unbatched DELETE with a bare
`volunteered_at` predicate (full scan; fine for a TTL-bounded table). Edge:
a brain whose dream cycle was off for months could hit the pooler's ~2min
statement_timeout on the first prune, get swallowed by the catch, and never
make progress. If observed: id-batched chunks (`DELETE ... WHERE id IN
(SELECT ... LIMIT 10000)` looped). **Where:**
`src/core/context/volunteer-events.ts:purgeStaleVolunteerEvents`.
- [ ] **P3 — route `gbrain watch` through the serve resolve-IPC on PGLite.**
`watch` connects directly, so on a PGLite brain it monopolizes the single
connection for its whole (potentially hours-long) session — a concurrent
`gbrain serve` or any write path blocks on the lock until watch exits.
WATCH_HELP documents the monopoly; the fix is an IPC rung in watch's
resolver (reuse `resolveViaIpc` like the ambient reflex's ladder) so a
running serve answers and watch never takes the lock. **Why:** watch +
serve concurrently is the natural agent topology. **Where:**
`src/commands/watch.ts`, `src/core/context/resolve-ipc.ts` (red-team RT2).
- [ ] **P3 — capability/version gate for host-injected reflex resolvers.**
Windowing switched the orchestrator's suppression request to 'slug-only';
a host resolver built against the pre-window contract that still applies
title-whole-word suppression silently self-suppresses every windowed
entity. The contract is documented at `ResolveEntitiesFn` (reflex.ts), but
nothing detects a stale host. Add a capability handshake (e.g. resolver
advertises `supportsSuppressionModes`) and fall back to
`window_turns: 1` semantics when absent. **Where:**
`src/core/context/reflex.ts:ResolveEntitiesFn` + the OpenClaw plugin
contract (red-team RT4).
## gbrain triage wave follow-ups (filed v0.42.41.0)
Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes).
See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-zany-thacker.md`.
- [ ] **P1 — supervisor: retry-with-backoff instead of hard stop on transient DB outages (#1994).**
`max_crashes_exceeded` gives up permanently; a transient pooler blip that trips the
counter wedges the supervisor until manual restart. **Why:** the #2034 reconnect fix
makes the engine recover, but the supervisor still hard-stops. **Where:**
`src/core/minions/supervisor.ts` crash-count loop — add exponential backoff with a
much higher (or no) permanent-give-up threshold for recoverable errors.
- [ ] **P2 — PGLite `reindex-frontmatter` / backfill statement_timeout boost (#1963).**
Community RCA: `SET LOCAL statement_timeout` is gated on `engine.kind === 'postgres'`,
so PGLite inherits the 30s session default and trips on non-trivial batches; the CLI
then swallows the error and exits 0. **Where:** `src/core/backfill-effective-date.ts`
(boost on PGLite too, or per-row updates) + the cli.ts catch that hides it.
- [ ] **P2 — autopilot drain-worker concurrency self-deadlock (#2050).** Drain-worker
runs at concurrency=1, so any cycle phase that spawns a subagent (patterns, synthesize)
deadlocks waiting on a worker slot it can't get. **Where:** autopilot drain-worker
dispatch — raise concurrency or exempt subagent-spawning phases.
- [ ] **P3 — name-keyed migration ledger (#2038 structural follow-up).** The always-run
index drift probe heals the one known case; the general fix is keying applied-migration
tracking by stable name rather than version integer so a renumber can't strand a
migration as recorded-but-not-executed. **Where:** `src/core/migrate.ts` ledger.
## gbrain#1981 Retrieval Reflex follow-ups (v0.43+)
Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extractor
is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`.
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor
(`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin
scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall
on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095
wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and
pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun
coreference for never-named antecedents remains with the LLM-pass idea.)*
- [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver
(`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix)
for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion
arm, gated on an unambiguous single hit, if recall telemetry comes back weak.
## gbrain#1972 job-layer follow-up (v0.43+)
Filed from the #1972 fix (stale-lock reaper + bounded disconnect + complete
cooperative-abort). One item was deliberately gated, not deferred blindly. See plan +
GSTACK REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-curious-pike.md`.
- [ ] **P2 — `findBacklinkGaps` sync→async refactor (gated on telemetry).** The backlinks
phase does its heavy work in a single synchronous call (`findBacklinkGaps`,
`src/commands/backlinks.ts:71` — nested `readdirSync` double-walk, no `await` seam), so it
cannot be cooperatively aborted: a >30s run on a huge brain blocks the event loop and gets
force-evicted. lint was made yield-able this wave (it was already async); backlinks needs
`findBacklinkGaps` converted to async-with-periodic-yields, threaded through
`runBacklinksCore` + `runPhaseBacklinks`. **Why gated:** the trigger is UNCONFIRMED — we
don't know backlinks ever exceeds 30s. This wave added the phase-duration force-evict
attribution log (`FORCE_EVICT_DEADLINE_MS` in `src/core/cycle.ts`), which names any phase
that crosses the deadline. Do this refactor only if a production 24h pull shows backlinks
crossing it; otherwise it's a hot-loop rewrite for a non-occurring case. **Where:**
`src/commands/backlinks.ts`, `src/core/cycle.ts` (runPhaseBacklinks signal threading).
## gbrain#1881 sync reclone ownership follow-ups (v0.43+)
Filed from the #1881 fix (`gbrain sync --strategy code` deleted a user's working
tree; `recloneIfMissing` now only re-clones a clone gbrain OWNS — `config.managed_clone`
marker or exact default-location equality — via `isOwnedClone`). Deliberately scoped
OUT of that PR. Codex outside-voice findings #5/#6. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-golden-valiant.md`.
- [ ] **P2 — `gbrain doctor` misconfigured-source check.** Flag every source row
where `config.remote_url` is set but `isOwnedClone(row)` is false (the shape that
caused #1881: a federated row whose `local_path` is a user working tree). Print a
one-time, actionable hint per row: drop `config.remote_url` to sync it read-only,
or remove + re-add with `--url` so gbrain owns the clone. **Why:** the core guard
now refuses to delete such rows, but they still exist in users' brains (created by
the gstack orchestrator). This is the single surfacing point — it replaces the
per-sync stderr warning that was rejected during eng-review (Codex: it would spam
every healthy sync). **Where:** extend the doctor checks in `src/commands/doctor.ts`;
reuse `isOwnedClone` from `src/core/sources-ops.ts`. No migration.
- [ ] **P3 — Decide the `--clone-dir`-outside-root policy.** `gbrain sources add --url
--clone-dir <path>` lets local callers place a gbrain-owned clone anywhere. The
ownership marker (this PR) makes those safe to reclone, but the dormant
`clone_dir_outside_gbrain` code in `SourceOpErrorCode` (`sources-ops.ts`) is unused —
it hints at a previously-intended confinement rule. Decide: either wire it up (forbid
`--clone-dir` outside `$GBRAIN_HOME/clones/`) or delete the dead code. Don't leave it
half-implemented. Codex finding #5.
- [ ] **P2 — Harden the `managed_clone` ownership marker against forgery.** Ownership
(`isOwnedClone`) authorizes the destructive reclone swap on the strength of a DB JSON
boolean (`config.managed_clone`). Today only `addSource --url` writes it, but it's a
mutable field any future `set-config` / external INSERT / restored dump could set on a
user-tree path. A forged marker on a real (non-symlink) user path would authorize
deletion. (A realpath path-check does NOT close this — it false-positives on ubiquitous
system symlinks like macOS /var, and an owned clone gbrain created is legitimately
deleted through any operator symlink anyway. Path can't prove ownership.) Two follow-ups:
(a) a CI guard asserting NO code path other than `addSource` ever writes the
`managed_clone` key; (b) bind ownership to an unforgeable on-disk stamp (a `.gbrain-clone`
sentinel written into the clone at creation, verified before any destructive op) instead
of / in addition to the DB field — with an equality-fallback for pre-stamp clones. Codex
adversarial (High) + Claude adversarial (Finding 2) from the #1881 ship review.
- [ ] **P3 — Sweep orphaned `.gbrain-reclone-*` temp dirs.** The EXDEV-safe reclone clones
into a sibling temp of `local_path` (`.gbrain-reclone-<leaf>-<rand>`). Every error path
`rmSync`s it, but a hard crash (SIGKILL/power loss) between clone and swap leaves a full
clone orphaned next to the user's `--clone-dir` parent — outside gbrain's swept
`clones/.tmp`. Add a startup/doctor sweep for `.gbrain-reclone-*` / `*.old-*` older than N
minutes. Codex Medium / Claude Finding 4 from the #1881 ship review.
- [ ] **P3 — CLI `gbrain sources remove` leaks the managed clone dir.** `runRemove`
(`src/commands/sources.ts:269`) runs `DELETE FROM sources` directly, bypassing
`removeSource()` and its symlink-safe clone-cleanup guard — so removing a `--url`
source never deletes its on-disk clone (storage leak). Route CLI remove through
`removeSource()` (or replicate its guard) so the clone dir is cleaned with the same
ownership/symlink protections. Orthogonal to the deletion bug; surfaced by Codex
finding #6 during the #1881 review.
## #1737 minion fair-scheduling follow-up (v0.43+)
Filed during the #1737 wave (`/plan-eng-review` decision F7, codex outside-voice
@@ -33,8 +481,9 @@ context). Deliberately scoped OUT of that PR. See plan + GSTACK REVIEW REPORT at
batch error, retry the batch element-by-element so one bad row can't abort a
353K-page `extract --stale` sweep, logging the offending `(from_slug, context)`
instead of dying. The durable JSONB fix removed the known crash class (malformed
array literal) and NUL-stripping removed the other known jsonb-parse failure, so
there is no remaining data-dependent crash for this to catch *today* — it's
array literal), NUL-stripping removed a second jsonb-parse failure, and
v0.42.40.0 lone-surrogate well-forming (#2011) removed a third, so there is no
remaining *known* data-dependent crash for this to catch *today* — it's
belt-and-suspenders against unknown future per-row failures. Wire it in
`addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` (or in `batchRetry` as
a post-classification fallback). Issue #1861 option 2.
@@ -121,17 +570,41 @@ GSTACK REVIEW REPORT at
can't desync per-engine, bounded against CLI-hang by a top-level forced
cleanup. Do this BEFORE introducing any concurrent module-engine connect path.
- [ ] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
last-retrieved queues before the owner disconnect.** The op-dispatch path
(`cli.ts:~282-314`) drains `getFactsQueue().drainPending()` +
`awaitPendingLastRetrievedWrites()` before `engine.disconnect()`; the `dream`
owner-disconnect (`cli.ts:~1164`) and the fall-through owner-disconnect
(`cli.ts:~1785`) do not. If the dream cycle ever enqueues a facts:absorb /
last-retrieved write that's still in flight at disconnect, the owner nulls the
singleton and the write throws "No database connection". Pre-existing (not
introduced by the #1471 ownership fix), surfaced by the Claude adversarial
review (F5). Fix: hoist the same drain-before-disconnect block the op-dispatch
path uses into a shared helper and call it on all three owner-disconnect sites.
- [x] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
last-retrieved queues before the owner disconnect.** DONE in the #2084 fix:
`finishCliTeardown` (`src/core/cli-force-exit.ts`) is exactly the shared
drain-before-disconnect helper this item asked for, and ALL NINE cli.ts
disconnect sites route through it (op-dispatch, fall-through, dream, doctor
×3, ze-switch, search dashboard, read-only timeout path). Structural guard:
no bare `await engine.disconnect()` remains in cli.ts
(`test/fix-wave-structural.test.ts` `#2084` describe).
- [ ] **P2 — command-module `process.exit` sites bypass the #2084 teardown
contract.** Several CLI_ONLY command modules exit directly on their normal
paths (`doctor.ts` ~10 sites incl. its verdict exit, `dream.ts` ~23,
`ze-switch.ts` ~9, plus friction/claw-test/eval verdict exits in cli.ts) —
those exits preempt the call-site `finally`, so the background-work drain,
bounded disconnect, and `flushThenExit` grace are all skipped on those paths
(pre-existing class, NOT introduced by #2084; pre-fix the same exits skipped
the inline drains too). Consequences: `gbrain doctor --json | <slow reader>`
keeps the #1959 truncation exposure; a dream path that exits mid-cycle
discards in-flight facts/search-cache writes. Fix shape: convert in-command
`process.exit(n)` to `setCliExitVerdict(n)` + return (the central seam
exits), or route them through a shared `exitCommand(n)` helper that runs
teardown first. Surfaced by the #2084 cross-model adversarial review (F2).
- [ ] **P3 — opt-in whole-command wallclock cap (`GBRAIN_COMMAND_DEADLINE_MS`),
build ONLY on a real wedged-handler incident.** The #2084 fix deliberately
removed the blanket pre-handler 10s force-exit (it killed slow-legit ops with
exit 0 and truncated output); per-op deadlines (query-embed deadline,
`withTimeout` on read-only commands) own handler wallclock now, and
`connectEngine` hangs — the historically observed zombie class — were never
covered by the old timer anyway. If production ever shows a genuinely wedged
handler (trigger: a non-`serve` command alive >30min with no progress
output), add an opt-in env cap that exits NON-ZERO with a truthful banner.
Attach point: the `GBRAIN_TEARDOWN_DEADLINE_MS` / `computeTeardownDeadlineMs`
plumbing in `src/core/cli-force-exit.ts`. Do not build speculatively —
follow-up from the #2084 eng review (decision D2/D14).
## v0.42.x AI SDK v6 tool-schema fix follow-ups (#1782/#1764)
Surfaced by the codex outside-voice pass during `/plan-eng-review` and
@@ -449,6 +922,19 @@ PR1 shipped the read-only catalog; PR2 is the download-and-install surface,
deferred per the plan's D1 + D8 because it stands up new HTTP/binary/token
infra and reaches into third-party packs that live outside the host skills dir.
> **#2180 update (v0.43+ brain-resident skillpacks + advisor):** brain-resident
> pack DISCOVERY over MCP shipped as a dedicated, source-scoped
> `list_brain_skillpack` op (NOT folded into `list_skills` — the host catalog is
> host-global and ignores `ctx.sourceId`, so per-source packs needed their own
> tenancy-correct surface). `get_skill` gained an optional `source_id` for
> per-source fetch disambiguation. The `tools:` version-skew lint below is now
> implemented (`src/core/skillpack/brain-pack-lint.ts`, run by
> `gbrain skillpack init-brain-pack`). STILL DEFERRED to this PR2: thin-client
> BINARY install (`build_skillpack` download) — a thin client today gets the
> pack's git scaffold spec and `resolveSource`s it on its own machine. The
> `include_skillpacks` host-global merge below is intentionally still open
> (separate concern from per-source brain packs).
- [ ] **v0.41.37+: `build_skillpack` op + `GET /skillpack/download/:token` endpoint.** Build a deterministic `.tgz` on demand (named skillpack, ad-hoc skill subset, or whole repo) and deliver it both base64-inline (universal/stdio) and via an authenticated short-lived download URL when running under `gbrain serve --http`. **What:** new admin-or-write-scoped op + a token-store + cache-dir GC; reuse `packTarball` from `src/core/skillpack/tarball.ts` (already deterministic + symlink-rejecting + size-capped) and the magic-link nonce pattern in `serve-http.ts`. The tarball ships source CODE, so it needs its own trust decision separate from PR1's prose-only catalog. **Why:** lets a thin client install a skillpack into its own setup, not just follow one live. **Depends on:** PR1 (landed in v0.41.36.0). Priority: P2.
- [ ] **v0.41.37+: `include_skillpacks` merge in `list_skills`.** Fold pinned third-party packs (from `~/.gbrain/skillpack-state.json`) into the catalog. Deferred from PR1 (D8) because packs live OUTSIDE the host skills dir and need (a) a per-pack trusted-root realpath confinement and (b) `{name, skillpack_name?}` disambiguation when a pack skill and a host skill share a name. Lands naturally with PR2's pack machinery. Priority: P2.
- [ ] **v0.41.37+: TTL+mtime cache for the skill-catalog walk.** PR1 reads fresh every call (cold path, ~ms). If telemetry shows repeated `list_skills` calls, add a TTL+mtime-keyed cache shared by `list_skills` + `get_skill`. Priority: P3 (do-nothing was the deliberate PR1 call).
@@ -1708,11 +2194,6 @@ Three items deferred:
self-heals via stale-reclaim). The common sync SUCCESS path already drains via
handleCliOnly's finally. Convert for graceful drain on sync error exits.
- [ ] **(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer** so it
wraps `engine.disconnect()` only (it's armed before the handler today, doubling
as a blanket handler watchdog) and fix its misleading "engine.disconnect() did
not return…" message that fires even when the handler (not disconnect) was slow.
- [ ] **(v0.42.20.0 follow-up) Gateway idle-timeout (vs absolute) for streaming
chat.** `withDefaultTimeout` uses an absolute `AbortSignal.timeout`; a streaming
generation actively producing tokens past the chat default (300s) would abort.
@@ -3482,6 +3963,18 @@ keeping both skills' triggers intact for chaining.
## Completed
### ~~(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer~~
**Completed:** v0.42.39.0 (2026-06-10)
The timer now arms at teardown entry (inside the op-dispatch finally, before
drain + disconnect) so it bounds ONLY disconnect — no longer doubling as a
blanket handler watchdog that killed slow-but-healthy ops at 10s with exit 0
and empty stdout. Its "engine.disconnect() did not return…" message is now
accurate by construction (it can only fire during teardown). Read-scope
handlers + context build got their own explicit wallclock bound (180s default,
`--timeout=Ns`, exit 124, hard-exit after teardown) in the same wave. Pinned by
`test/cli-force-exit-teardown-arming.test.ts`.
### ~~Checks 5 + 6 for check-resolvable~~
**Completed:** v0.19.0 (2026-04-22)
+1 -1
View File
@@ -1 +1 @@
0.42.32.0
0.42.64.0
+52 -20
View File
@@ -13,48 +13,52 @@
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"typescript": "^5.8.3",
"vite": "^6.3.3",
"vite": "^6.4.3",
},
},
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10",
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.0", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.28.5", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw=="],
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
"@babel/compat-data": ["@babel/compat-data@7.29.0", "", {}, "sha512-T1NCJqT/j9+cn8fvkt7jtwbLBfLC/1y1c7NtCeXFRgzGTsafi68MRv8yzkYSapBnFA6L3U2VSc02ciDzoAJhJg=="],
"@babel/compat-data": ["@babel/compat-data@7.29.7", "", {}, "sha512-locTkQyKvwIEgBzVrn8693ebc97F2U8ZHjbXwDXJ5Fn2TCpNwTlKcaKLkdHop5c/icOFE7qt7Q9JC5hnKNa6Gg=="],
"@babel/core": ["@babel/core@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-compilation-targets": "^7.28.6", "@babel/helper-module-transforms": "^7.28.6", "@babel/helpers": "^7.28.6", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/traverse": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA=="],
"@babel/core": ["@babel/core@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.7", "@babel/helper-compilation-targets": "^7.29.7", "@babel/helper-module-transforms": "^7.29.7", "@babel/helpers": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/template": "^7.29.7", "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-RgHBCvtjbOK2gXSNBNIkNoEc9qoVEtau3hj8gEqKQuL3HZAibKarWFEI3Lfm6EYKkLalOh8eSrj9b+ch9H/VBA=="],
"@babel/generator": ["@babel/generator@7.29.1", "", { "dependencies": { "@babel/parser": "^7.29.0", "@babel/types": "^7.29.0", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw=="],
"@babel/generator": ["@babel/generator@7.29.7", "", { "dependencies": { "@babel/parser": "^7.29.7", "@babel/types": "^7.29.7", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ=="],
"@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.28.6", "", { "dependencies": { "@babel/compat-data": "^7.28.6", "@babel/helper-validator-option": "^7.27.1", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-JYtls3hqi15fcx5GaSNL7SCTJ2MNmjrkHXg4FSpOA/grxK8KwyZ5bubHsCq8FXCkua6xhuaaBit+3b7+VZRfcA=="],
"@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.29.7", "", { "dependencies": { "@babel/compat-data": "^7.29.7", "@babel/helper-validator-option": "^7.29.7", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-wem6WaBj4NaVYVdNhLPPVacES6ZJ+KBBfSkTMD3YZxbP3rm3Di85tJU5ljaUNhaOynt+Aj0xruhYuzQBt8n71g=="],
"@babel/helper-globals": ["@babel/helper-globals@7.28.0", "", {}, "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw=="],
"@babel/helper-globals": ["@babel/helper-globals@7.29.7", "", {}, "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA=="],
"@babel/helper-module-imports": ["@babel/helper-module-imports@7.28.6", "", { "dependencies": { "@babel/traverse": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw=="],
"@babel/helper-module-imports": ["@babel/helper-module-imports@7.29.7", "", { "dependencies": { "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g=="],
"@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.28.6", "", { "dependencies": { "@babel/helper-module-imports": "^7.28.6", "@babel/helper-validator-identifier": "^7.28.5", "@babel/traverse": "^7.28.6" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA=="],
"@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.29.7", "", { "dependencies": { "@babel/helper-module-imports": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7", "@babel/traverse": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg=="],
"@babel/helper-plugin-utils": ["@babel/helper-plugin-utils@7.28.6", "", {}, "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug=="],
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@babel/helper-string-parser": ["@babel/helper-string-parser@7.29.7", "", {}, "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.29.7", "", {}, "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg=="],
"@babel/helper-validator-option": ["@babel/helper-validator-option@7.27.1", "", {}, "sha512-YvjJow9FxbhFFKDSuFnVCe2WxXk1zWc22fFePVNEaWJEu8IrZVlda6N0uHwzZrUM1il7NC9Mlp4MaJYbYd9JSg=="],
"@babel/helper-validator-option": ["@babel/helper-validator-option@7.29.7", "", {}, "sha512-N9ZErrD+yW5geCDtBqnOoxmR8+tNKiGuxKlDpuJxfsqpa2dFcexaziGAE/qoHLiDDreVNMupxGmSoNlyvsA3gw=="],
"@babel/helpers": ["@babel/helpers@7.29.2", "", { "dependencies": { "@babel/template": "^7.28.6", "@babel/types": "^7.29.0" } }, "sha512-HoGuUs4sCZNezVEKdVcwqmZN8GoHirLUcLaYVNBK2J0DadGtdcqgr3BCbvH8+XUo4NGjNl3VOtSjEKNzqfFgKw=="],
"@babel/helpers": ["@babel/helpers@7.29.7", "", { "dependencies": { "@babel/template": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-1k2lAGRMfHTcwuNYcCNUmaUffmQv8KWMfh2iJUUeRlwlwH4FdNG7mfPI10NPfLHJFThE4Tyr4mv7kTNZOiPuBg=="],
"@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@babel/parser": ["@babel/parser@7.29.7", "", { "dependencies": { "@babel/types": "^7.29.7" }, "bin": "./bin/babel-parser.js" }, "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg=="],
"@babel/plugin-transform-react-jsx-self": ["@babel/plugin-transform-react-jsx-self@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-6UzkCs+ejGdZ5mFFC/OCUrv028ab2fp1znZmCZjAOBKiBK2jXD1O+BPSfX8X2qjJ75fZBMSnQn3Rq2mrBJK2mw=="],
"@babel/plugin-transform-react-jsx-source": ["@babel/plugin-transform-react-jsx-source@7.27.1", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.27.1" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-zbwoTsBruTeKB9hSq73ha66iFeJHuaFkUbwvqElnygoNbj/jHRsSeokowZFN3CZ64IvEqcmmkVe89OPXc7ldAw=="],
"@babel/template": ["@babel/template@7.28.6", "", { "dependencies": { "@babel/code-frame": "^7.28.6", "@babel/parser": "^7.28.6", "@babel/types": "^7.28.6" } }, "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ=="],
"@babel/template": ["@babel/template@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg=="],
"@babel/traverse": ["@babel/traverse@7.29.0", "", { "dependencies": { "@babel/code-frame": "^7.29.0", "@babel/generator": "^7.29.0", "@babel/helper-globals": "^7.28.0", "@babel/parser": "^7.29.0", "@babel/template": "^7.28.6", "@babel/types": "^7.29.0", "debug": "^4.3.1" } }, "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA=="],
"@babel/traverse": ["@babel/traverse@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.7", "@babel/helper-globals": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/template": "^7.29.7", "@babel/types": "^7.29.7", "debug": "^4.3.1" } }, "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw=="],
"@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@babel/types": ["@babel/types@7.29.7", "", { "dependencies": { "@babel/helper-string-parser": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7" } }, "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA=="],
"@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.25.12", "", { "os": "aix", "cpu": "ppc64" }, "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA=="],
@@ -220,7 +224,7 @@
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nanoid": ["nanoid@3.3.11", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="],
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
@@ -228,7 +232,7 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.9", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-7a70Nsot+EMX9fFU3064K/kdHWZqGVY+BADLyXc8Dfv+mTLLVl6JzJpPaCZ2kQL9gIJvKXSLMHhqdRRjwQeFtw=="],
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
@@ -250,8 +254,36 @@
"update-browserslist-db": ["update-browserslist-db@1.2.3", "", { "dependencies": { "escalade": "^3.2.0", "picocolors": "^1.1.1" }, "peerDependencies": { "browserslist": ">= 4.21.0" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w=="],
"vite": ["vite@6.4.2", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ=="],
"vite": ["vite@6.4.3", "", { "dependencies": { "esbuild": "^0.25.0", "fdir": "^6.4.4", "picomatch": "^4.0.2", "postcss": "^8.5.3", "rollup": "^4.34.9", "tinyglobby": "^0.2.13" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["@types/node", "jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss", "terser", "tsx", "yaml"], "bin": { "vite": "bin/vite.js" } }, "sha512-NTKlcQjlAK7MlQoyb6LgaqHc8sso/pVyUJYWMws3jg21uTJw/LddqIFPcPqP6PzpgbIcZyKI85sFE4HBrQDA8A=="],
"yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="],
"@types/babel__core/@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@types/babel__core/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__generator/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__template/@babel/parser": ["@babel/parser@7.29.2", "", { "dependencies": { "@babel/types": "^7.29.0" }, "bin": "./bin/babel-parser.js" }, "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA=="],
"@types/babel__template/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__traverse/@babel/types": ["@babel/types@7.29.0", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A=="],
"@types/babel__core/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__core/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__generator/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__generator/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__template/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__template/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
"@types/babel__traverse/@babel/types/@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="],
"@types/babel__traverse/@babel/types/@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="],
}
}
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -7,7 +7,7 @@
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
<script type="module" crossorigin src="/admin/assets/index-DqP-zmqH.js"></script>
<script type="module" crossorigin src="/admin/assets/index-CoGEje3-.js"></script>
<link rel="stylesheet" crossorigin href="/admin/assets/index-GxkWX7v3.css">
</head>
<body>
+5 -1
View File
@@ -15,7 +15,11 @@
"@types/react": "^19.1.2",
"@types/react-dom": "^19.1.2",
"@vitejs/plugin-react": "^4.4.1",
"vite": "^6.3.3",
"vite": "^6.4.3",
"typescript": "^5.8.3"
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10"
}
}
+1 -1
View File
@@ -21,7 +21,7 @@ export function DashboardPage() {
api.stats().then(setStats).catch(() => {});
api.health().then(setHealth).catch(() => {});
const es = new EventSource('/admin/events');
const es = new EventSource('/admin/events', { withCredentials: true });
eventSourceRef.current = es;
es.onopen = () => setSseStatus('connected');
es.onmessage = (e) => {
+40 -17
View File
@@ -26,8 +26,8 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"js-yaml": "^3.15.0",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
@@ -50,6 +50,17 @@
"trustedDependencies": [
"@electric-sql/pglite",
],
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"js-yaml": "^3.15.0",
"qs": "^6.15.2",
},
"packages": {
"@ai-sdk/anthropic": ["@ai-sdk/anthropic@3.0.74", "", { "dependencies": { "@ai-sdk/provider": "3.0.10", "@ai-sdk/provider-utils": "4.0.26" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-Xew9rfz9WWhDSyF8rNhjT/XWOWelNfJrMlmG0Ahw210hStisRpQZ1s+7VeI9JTJOZ5y5tXqBi5kfPwYnCfyRTA=="],
@@ -151,7 +162,7 @@
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
"@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="],
"@jsquash/avif": ["@jsquash/avif@2.1.1", "", { "dependencies": { "wasm-feature-detect": "^1.2.11" } }, "sha512-LMRxd0fMgfCLtobDh0/sFYJMMiRJTNYSEEWvRDKXlAeZ08t3gI5V+1thIT0XjXJ+SVG7Zug9B0XPyx0Ti5VRNA=="],
@@ -159,6 +170,8 @@
"@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="],
"@nodable/entities": ["@nodable/entities@3.0.0", "", {}, "sha512-8L9xFeTYKhm49xfIypoe2W5wV1m/3Z58kT+7kR9A8OyFxcPduI4VmxaUMQyKYrRjUoLLSXv6EKKID5Tvj9cUVw=="],
"@opentelemetry/api": ["@opentelemetry/api@1.9.0", "", {}, "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg=="],
"@smithy/chunked-blob-reader": ["@smithy/chunked-blob-reader@5.2.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-St+kVicSyayWQca+I1rGitaOEH6uKgE8IUWoYnnEX26SWdWQcL6LvMSD19Lg+vYHKdT9B2Zuu7rd3i6Wnyb/iw=="],
@@ -307,6 +320,8 @@
"ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],
"anynum": ["anynum@1.0.1", "", {}, "sha512-N6//FLET/tXYNM/F6ABca1oH6fWB+KlTt909Le28WMDBk8oaT4vY17DCrwg2MvmuqUKt3Ni4N5dGJ/EoBgcO6A=="],
"argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="],
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
@@ -385,15 +400,15 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
"fast-uri": ["fast-uri@3.1.3", "", {}, "sha512-i70LwGWUduXqzicKXWshooq+sWL1K3WUU5rKZNG/0i3a1OSoX3HqhH5WbWwTmqWfor4urUakGPiRQcleRZTwOg=="],
"fast-xml-builder": ["fast-xml-builder@1.1.4", "", { "dependencies": { "path-expression-matcher": "^1.1.3" } }, "sha512-f2jhpN4Eccy0/Uz9csxh3Nu6q4ErKxf0XIsasomfOihuSUa3/xw6w8dnOtCDgEItQFJG8KyXPzQXzcODDrrbOg=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
"fast-xml-parser": ["fast-xml-parser@5.5.8", "", { "dependencies": { "fast-xml-builder": "^1.1.4", "path-expression-matcher": "^1.2.0", "strnum": "^2.2.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-Z7Fh2nVQSb2d+poDViM063ix2ZGt9jmY1nWhPfHBOK2Hgnb/OW3P4Et3P/81SEej0J7QbWtJqxO05h8QYfK7LQ=="],
"fast-xml-parser": ["fast-xml-parser@5.10.1", "", { "dependencies": { "@nodable/entities": "^3.0.0", "fast-xml-builder": "^1.2.0", "is-unsafe": "^2.0.0", "path-expression-matcher": "^1.6.2", "strnum": "^2.4.1", "xml-naming": "^0.3.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-IEMIf7298kXuZSRFoGfMYrl7is8LpavODgbNz1cwIudv7KwVFnuU+UsMporfq6PD6aXSlawZlARiA3UywCTfMw=="],
"finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "^4.4.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "on-finished": "^2.4.1", "parseurl": "^1.3.3", "statuses": "^2.0.1" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="],
"form-data": ["form-data@4.0.5", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.2", "mime-types": "^2.1.12" } }, "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w=="],
"form-data": ["form-data@4.0.6", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.4", "mime-types": "^2.1.35" } }, "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ=="],
"form-data-encoder": ["form-data-encoder@1.7.2", "", {}, "sha512-qfqtYan3rxrnCk1VYaA4H+Ms9xdpPqvLZa6xmMgFvhO32x7/3J/ExcTd6qpxM0vH2GdMI+poehyBZvqfMTto8A=="],
@@ -417,11 +432,11 @@
"has-tostringtag": ["has-tostringtag@1.0.2", "", { "dependencies": { "has-symbols": "^1.0.3" } }, "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw=="],
"hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"hasown": ["hasown@2.0.4", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A=="],
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.12.10", "", {}, "sha512-mx/p18PLy5og9ufies2GOSUqep98Td9q4i/EF6X7yJgAiIopxqdfIO3jbqsi3jRgTgw88jMDEzVKi+V2EF+27w=="],
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -431,7 +446,7 @@
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="],
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
@@ -439,11 +454,13 @@
"is-promise": ["is-promise@4.0.0", "", {}, "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ=="],
"is-unsafe": ["is-unsafe@2.0.0", "", {}, "sha512-2LdV822R+wmI86unXA93WCFpL6g+av8ynWk0nrHyJqGop5VoocYsSLFgN8jrfalT6iGeLNM4KXuVSsULP53kEA=="],
"isexe": ["isexe@2.0.0", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="],
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
"js-yaml": ["js-yaml@3.14.2", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg=="],
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
@@ -455,7 +472,7 @@
"libheif-js": ["libheif-js@1.19.8", "", {}, "sha512-vQJWusIxO7wavpON1dusciL8Go9jsIQ+EUrckauFYAiSTjcmLAsuJh3SszLpvkwPci3JcL41ek2n+LUZGFpPIQ=="],
"marked": ["marked@18.0.0", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-2e7Qiv/HJSXj8rDEpgTvGKsP8yYtI9xXHKDnrftrmnrJPaFNM7VRb2YCzWaX4BP1iCJ/XPduzDJZMFoqTCcIMA=="],
"marked": ["marked@18.0.6", "", { "bin": { "marked": "bin/marked.js" } }, "sha512-MrV5puXBfuiy6wl6DLaq3BtIJQAJToAd5zt/ZKhRfGRAuFPALE7/4Y7jnxRQoEgK/pBgurGqLyAuRgZ2xOjr6w=="],
"math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],
@@ -487,7 +504,7 @@
"parseurl": ["parseurl@1.3.3", "", {}, "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="],
"path-expression-matcher": ["path-expression-matcher@1.5.0", "", {}, "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ=="],
"path-expression-matcher": ["path-expression-matcher@1.6.2", "", {}, "sha512-enSlaiat05iasnzmgNxRj8reFdj3puY2QpNgP1aPIaVfT6nn9ICuPoFlKHk8EN22HcwewshO+mN2DGbkCEOtqQ=="],
"path-key": ["path-key@3.1.1", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="],
@@ -503,7 +520,7 @@
"pure-rand": ["pure-rand@8.4.0", "", {}, "sha512-IoM8YF/jY0hiugFo/wOWqfmarlE6J0wc6fDK1PhftMk7MGhVZl88sZimmqBBFomLOCSmcCCpsfj7wXASCpvK9A=="],
"qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="],
"qs": ["qs@6.15.3", "", { "dependencies": { "es-define-property": "^1.0.1", "side-channel": "^1.1.1" } }, "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A=="],
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],
@@ -529,9 +546,9 @@
"shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="],
"side-channel": ["side-channel@1.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3", "side-channel-list": "^1.0.0", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw=="],
"side-channel": ["side-channel@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4", "side-channel-list": "^1.0.1", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ=="],
"side-channel-list": ["side-channel-list@1.0.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3" } }, "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA=="],
"side-channel-list": ["side-channel-list@1.0.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4" } }, "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w=="],
"side-channel-map": ["side-channel-map@1.0.1", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3" } }, "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA=="],
@@ -543,7 +560,7 @@
"strip-bom-string": ["strip-bom-string@1.0.0", "", {}, "sha512-uCC2VHvQRYu+lMh4My/sFNmF2klFymLX1wHJeXnbEJERpV/ZsVuonzerjfrGpIGF7LBVa1O7i9kjiWvJiFck8g=="],
"strnum": ["strnum@2.2.3", "", {}, "sha512-oKx6RUCuHfT3oyVjtnrmn19H1SiCqgJSg+54XqURKp5aCMbrXrhLjRN9TjuwMjiYstZ0MzDrHqkGZ5dFTKd+zg=="],
"strnum": ["strnum@2.4.1", "", { "dependencies": { "anynum": "^1.0.1" } }, "sha512-M9eUSMT2dCB2cTNPG7UYj6KuK7RJR2SN2+yCV/fTW3xzTCS6EaGZ5pSMgDIjB7r8zSfTGk+dvvn9rTjpVS9Mwg=="],
"toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="],
@@ -577,6 +594,8 @@
"wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="],
"xml-naming": ["xml-naming@0.3.0", "", {}, "sha512-ghig2TBE/H11aOVgmahA3MhimvkBr6JIYknH/Dhdk10nXwdbIqBJsbfMxpvFPG8bAw77gN29aQWvKpmVoPlvPQ=="],
"zod": ["zod@4.3.6", "", {}, "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg=="],
"zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="],
@@ -595,12 +614,16 @@
"@types/bun/bun-types": ["bun-types@1.3.11", "", { "dependencies": { "@types/node": "*" } }, "sha512-1KGPpoxQWl9f6wcZh57LvrPIInQMn2TQ7jsgxqpRzg+l0QPOFvJVH7HmvHo/AiPgwXy+/Thf6Ov3EdVn1vOabg=="],
"es-set-tostringtag/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"eventsource/eventsource-parser": ["eventsource-parser@3.0.6", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],
"express/cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],
"form-data/mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="],
"get-intrinsic/hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],
"openai/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="],
"@anthropic-ai/sdk/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
+6 -1
View File
@@ -13,4 +13,9 @@ timeout = 60_000
# fixtures still match the schema. v0.37's production default is ZE/1280;
# tests that want the new default call configureGateway() explicitly in
# their own beforeAll.
preload = ["./test/helpers/legacy-embedding-preload.ts"]
#
# #2823: redirect GBRAIN_AUDIT_DIR to a per-run scratch dir BEFORE any test
# runs, so audit-emitting code paths (content-sanity, shell-audit, etc.)
# can't leak fixture events into the operator's real ~/.gbrain/audit/. See
# test/helpers/audit-dir-preload.ts for the full rationale.
preload = ["./test/helpers/legacy-embedding-preload.ts", "./test/helpers/audit-dir-preload.ts"]
+36
View File
@@ -85,6 +85,40 @@ services:
volumes:
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
# v0.43 (#2084 / eng-review TD1): PgBouncer in TRANSACTION pooling mode
# fronting postgres-1 — the production topology (Supabase direct :5432 +
# pooled :6543) behind three consecutive pooler-teardown waves
# (#1972 → #2015 → #2084) that CI could never reproduce.
# test/e2e/pgbouncer-teardown.test.ts uses a DEDICATED database
# (gbrain_pgbouncer) on postgres-1 so it never races shard 1's
# TRUNCATE-based fixtures; pgbouncer's wildcard [databases] section
# forwards any dbname to DB_HOST.
pgbouncer:
image: edoburu/pgbouncer:latest
environment:
DB_HOST: postgres-1
DB_PORT: "5432"
DB_USER: postgres
DB_PASSWORD: postgres
POOL_MODE: transaction
# plain (CI-only): pg16 stores SCRAM verifiers, and pgbouncer can only
# answer the server's SCRAM challenge when its userlist holds the
# PLAINTEXT password — an md5-hashed userlist fails with
# "server login failed: wrong password type".
AUTH_TYPE: plain
MAX_CLIENT_CONN: "200"
DEFAULT_POOL_SIZE: "10"
# gbrain's client sets statement_timeout + idle_in_transaction_session_timeout
# as startup parameters (db.ts buildConnectionParams); the Supabase pooler
# whitelists them, so this pooler must too or every connection is refused
# before the teardown path is even reached.
IGNORE_STARTUP_PARAMETERS: extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path
ports:
- "${GBRAIN_CI_PGBOUNCER_PORT:-6543}:5432"
depends_on:
postgres-1:
condition: service_healthy
runner:
image: oven/bun:1
working_dir: /app
@@ -97,6 +131,8 @@ services:
condition: service_healthy
postgres-4:
condition: service_healthy
pgbouncer:
condition: service_started
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
volumes:
+79 -1
View File
@@ -94,7 +94,7 @@ export interface BrainEngine {
**Slug-based API, not ID-based.** Every method takes slugs, not numeric IDs. The engine resolves slugs to IDs internally. This keeps the interface portable... slugs are strings, IDs are database-specific.
**Embedding is NOT in the engine.** The engine stores embeddings and searches by vector, but it doesn't generate embeddings. `src/core/embedding.ts` handles that. This is intentional: embedding is an external API call (OpenAI), not a storage concern. All engines share the same embedding service.
**Embedding is NOT in the engine.** The engine stores embeddings and searches by vector, but it doesn't generate embeddings. `src/core/embedding.ts` handles that (a thin delegation to the provider-agnostic AI gateway in `src/core/ai/gateway.ts`). This is intentional: embedding is an external API call (OpenAI, Voyage, a local Ollama — whichever provider you configured), not a storage concern. All engines share the same embedding service.
**Chunking is NOT in the engine.** Same logic. `src/core/chunkers/` handles chunking. The engine stores and retrieves chunks. All engines share the same chunkers.
@@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
Defense-in-depth layer for Postgres deployments that want the database itself
to enforce source isolation, in addition to the mandatory app-layer filters
(`sourceScopeOpts` — layer 1, always on).
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
source-scoped read methods wrap their queries in a transaction that first runs
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
bound params). An RLS policy can then filter rows by
`current_setting('app.scopes', true)`.
**Default off.** With the env var unset, reads call through on the shared pool
exactly as before — no per-read transaction, no pool-slot hold (the search
methods keep the transaction they always had for their `SET LOCAL
statement_timeout`). Existing operators see zero behavior change.
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
```sql
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
CREATE POLICY pages_scope_filter ON pages
USING (current_setting('app.scopes', true) = '*'
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
-- Required: connections that don't run through the scoped read helper
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
-- see zero rows once the policy exists:
ALTER ROLE <runtime-role> SET app.scopes = '*';
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
```
Safe to enable in either order: the env var without a policy is a no-op
setting; a policy without the env var is enforced only via the role default.
**Honest caveat:** only read paths routed through the scoped helper carry a
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine (v0.7, ships)
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
@@ -176,6 +221,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
| Form | Verdict |
|---|---|
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
cast then wraps that already-JSON string into a jsonb scalar string instead of
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
why a regression only shows up on Postgres (and why the parity test must run there).
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
`jsonb-guard-ok` comment.
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
and assert `jsonb_typeof` — the assertion PGLite cannot make.
## Adding a new engine
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
+35
View File
@@ -111,3 +111,38 @@ gbrain models doctor # 1-token probe per configured model
```
If anything's yellow, `gbrain doctor` names the fix command in the message. Most issues are missing API keys or stale schema (`gbrain upgrade --force-schema`).
## Troubleshooting
### PGLite crashes on macOS 26.x (Tahoe)
PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL:
```bash
# Install PostgreSQL + pgvector
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
# Build pgvector from source (required for vector search)
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# Point gbrain at your local Postgres
cat > ~/.gbrain/config.json << 'EOF'
{
"engine": "postgres",
"database_url": "postgresql://localhost:5432/gbrain",
"schema_pack": "gbrain-base-v2"
}
EOF
# Run migrations and verify
gbrain apply-migrations --yes
gbrain doctor
```
All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend.
> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again.
+10 -8
View File
@@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite.
Two equivalent paths:
**Path A — local CI gate (recommended, v0.23.1+):**
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
`docker-compose.ci.yml`. Override the host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host),
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer service (unit phase keeps
`DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger
than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins
up + tears down postgres automatically via `docker-compose.ci.yml`. Override
the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
schema/skills/package.json changes. Fast iteration during a focused branch.
(`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/
paths or schema/skills/package.json changes. Fast iteration during a focused
branch.
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
+27 -10
View File
@@ -3,6 +3,8 @@
On-demand reference (see CLAUDE.md Reference map). Current behavior + invariants
only.
`test/e2e/serve-http-oauth.test.ts` additionally pins confidential POST/Basic revocation, public-client SDK fallthrough, malformed/mixed authentication rejection, cross-client isolation, unknown-token opacity, metadata auth methods, no-store responses, strict post-revoke `401`, and retryable backend `503` semantics.
### Test command tiers
Seven test command tiers, each with a clear scope:
@@ -10,16 +12,16 @@ Seven test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set: `check:privacy && check:jsonb && check:progress && check:wasm && bun run typecheck`. The 4 checks `.github/workflows/test.yml` runs on shard 1 + typecheck. Single source of truth — CI literally calls `bun run verify`. | ~12s (wasm-compile dominates) | Before pushing; before `/ship`. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; runs at `--max-concurrency=1`). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` 4-way, which uses FNV-1a hash bucketing and INCLUDES `*.slow.test.ts`. CI EXCLUDES `*.serial.test.ts` from the hash buckets and runs them on shard 1 via `bun run test:serial` at `--max-concurrency=1` — keeping serial files out of the hash buckets is what preserves the `mock.module` quarantine (top-level mocks in serial files would otherwise leak into the parallel files they share a shard process with). CI is the ground truth for "did everything pass."
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
@@ -39,7 +41,7 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
@@ -111,7 +113,7 @@ Rename to `*.serial.test.ts` when:
- The file is genuinely env-coupled (e.g. `gbrain-home-isolation.test.ts`, `claw-test-cli.test.ts`) — module-load env readers + ESM caching defeat dynamic-import-after-env tricks.
- The file's tests intentionally share state across `it()` boundaries.
Quarantine count cap: 10 (informational). Beyond that, push back on the design.
The quarantine has grown to dozens of files — treat it as debt: every addition needs a reason from the list above, and prefer fixing the contention root cause when one exists.
### Unit test inventory
@@ -123,6 +125,15 @@ Unit tests and what they cover:
- `test/chunkers/recursive.test.ts` — chunking.
- `test/parity.test.ts` — operations contract parity.
- `test/cli.test.ts` — CLI structure.
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
- `test/cli-should-force-exit.test.ts``shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
- `test/cli-exit-verdict-pin.test.ts`#2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard.
- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`.
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
- `test/watch-command.test.ts``gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
- `test/watch-sigint.serial.test.ts``gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
- `test/cli-format-volunteer.test.ts``formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
- `test/config.test.ts` — config redaction.
- `test/files.test.ts` — MIME/hash.
- `test/import-file.test.ts` — import pipeline.
@@ -130,7 +141,7 @@ Unit tests and what they cover:
- `test/file-migration.test.ts` — file migration.
- `test/file-resolver.test.ts` — file resolution.
- `test/import-resume.test.ts` — import checkpoints.
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion.
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip).
- `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage.
- `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns.
- `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.
@@ -141,7 +152,7 @@ Unit tests and what they cover:
- `test/yaml-lite.test.ts` — YAML parsing.
- `test/check-update.test.ts` — version check + update CLI.
- `test/pglite-engine.test.ts` — PGLite engine, all BrainEngine methods including `addLinksBatch` / `addTimelineEntriesBatch` (empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100) plus `connect()` error-wrap assertion (original error nested, #223 link in message, lock released).
- `test/links-timeline-jsonb-poison.test.ts` — gbrain#1861 PGLite half (always-on, no `DATABASE_URL`). Locks the `jsonb_to_recordset` batch-insert path for links/timeline/takes against free-text "poison" payloads (commas, quotes, backslashes, braces, em-dashes) and asserts NUL is stripped from free-text body fields but rejected in identity fields. The Postgres lane (`test/e2e/jsonb-batch-poison-postgres.test.ts`) is the one that actually reproduced the original crash.
- `test/links-timeline-jsonb-poison.test.ts` — gbrain#1861 PGLite half (always-on, no `DATABASE_URL`). Locks the `jsonb_to_recordset` batch-insert path for links/timeline/takes against free-text "poison" payloads (commas, quotes, backslashes, braces, em-dashes) and asserts NUL is stripped from free-text body fields but rejected in identity fields. gbrain#2011 adds lone-UTF-16-surrogate cases: every free-text field (link context; timeline summary/detail/source; take claim/source) well-forms to U+FFFD across batch + scalar write paths, while a surrogate in an identity field (slug) still fail-closed rejects the batch. The Postgres lane (`test/e2e/jsonb-batch-poison-postgres.test.ts`) is the one that actually reproduced the original crash.
- `test/engine-factory.test.ts` — engine factory + dynamic imports.
- `test/integrations.test.ts` — recipe parsing, CLI routing, recipe validation.
- `test/publish.test.ts` — content stripping, encryption, password generation, HTML output.
@@ -212,13 +223,16 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's JSONB bind (`jsonb_to_recordset(($1::jsonb)->'rows')`) differs from PGLite's and gets its own coverage.
- `test/e2e/search-quality.test.ts` — search quality against PGLite (no API keys, in-memory).
- `test/e2e/graph-quality.test.ts` — knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory.
- `test/e2e/jsonb-batch-poison-postgres.test.ts` — gbrain#1861 regression, the engine that actually crashed. Seeds free-text "poison" context (Zoom URL with `?pwd=`, commas, quotes, Windows backslash path, braces, em-dash) and asserts the links/timeline/takes batch writers no longer error with "malformed array literal"; also asserts NUL is stripped from free-text bodies (`context`/`summary`/`detail`/`claim`) and still rejected in identity fields. `DATABASE_URL`-gated.
- `test/e2e/jsonb-batch-poison-postgres.test.ts` — gbrain#1861 regression, the engine that actually crashed. Seeds free-text "poison" context (Zoom URL with `?pwd=`, commas, quotes, Windows backslash path, braces, em-dash) and asserts the links/timeline/takes batch writers no longer error with "malformed array literal"; also asserts NUL is stripped from free-text bodies (`context`/`summary`/`detail`/`claim`) and still rejected in identity fields. gbrain#2011 adds the lone-surrogate crash lock: a lone UTF-16 surrogate in free text (the value that aborted `extract --stale` with `22P02` on Supabase) well-forms to U+FFFD across batch + scalar paths (incl. timeline + take `source`), while a surrogate in an identity field still rejects the batch. `DATABASE_URL`-gated.
- `test/e2e/postgres-jsonb.test.ts` — round-trips all 5 JSONB write sites (`pages.frontmatter`, `raw_data.data`, `ingest_log.pages_updated`, `files.metadata`, `page_versions.frontmatter`) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. Guards against the double-encode bug.
- `test/e2e/integrity-batch.test.ts` — parity for `scanIntegrity`'s batch-load fast path vs sequential. Cases (dedup, hits, validate, topPages) seed a fixture and assert both paths return identical results. Dedup case uses raw SQL via `getConn().unsafe()` to seed a `(test-source-2, people/alice)` row alongside the default-source row, since `engine.putPage` doesn't take a `source_id`. Pins multi-source overcounting; the "multi-source duplicate slugs scan once" case expects both batch + sequential paths to report 2.
- `test/e2e/jsonb-roundtrip.test.ts` — companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface drifts from the actual write surface, one of these tests catches it.
- `test/e2e/sync.test.ts``--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972#2015#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
- `test/e2e/volunteer-context-postgres.test.ts``volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
- `test/e2e/openclaw-reference-compat.test.ts``check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts``test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
@@ -227,8 +241,11 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/http-transport.test.ts``gbrain serve --http` end-to-end against real Postgres: bearer auth round-trip, `last_used_at` SQL-level debounce, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the dispatch round-trip with a real operation. Skips without `DATABASE_URL`.
- `test/e2e/serve-http-oauth.test.ts` — real-Postgres E2E against `gbrain serve --http` with full OAuth 2.1. Spawns a subprocess server, registers a client via the CLI, mints `client_credentials` tokens, exercises the `/mcp` JSON-RPC pipeline. Real DCR `/register` HTTP-level response-shape test (asserts `typeof body.client_id_issued_at === 'number'` over the wire, RFC 7591 §3.2.1); real CLI subprocess test for `revoke-client` (registers → mints token → revokes via `execSync` → asserts token rejected at `/mcp` → asserts re-run exits 1); server fixture flips on `--enable-dcr` so `/register` is reachable. **bun execSync env-inheritance contract:** bun's `execSync` does NOT inherit env mutations done via `process.env.X = ...`, only OS-level env from before bun started. helpers.ts loads `.env.testing` and sets `DATABASE_URL` via `process.env` mutation, which is invisible to subprocesses unless `env: { ...process.env }` is passed explicitly — every subprocess call in this file passes `env: { ...process.env }`. Reference fix for the same failure mode in sibling sync/cycle/dream/claw-test E2Es. `afterAll` cleanup is guarded on `clientId` (won't throw if `beforeAll` failed before registration); cleanup errors surface to stderr without throwing so real test failures aren't masked. Also covers the trust-boundary fix: an HTTP MCP `submit_job` for `name: "shell"` MUST reject with a permission error (request handler sets `remote: true` and `submit_job`'s protected-name guard fires), and the same guard rejects subagent submission. Skips without `DATABASE_URL`.
- `test/e2e/sync-parallel.test.ts``DATABASE_URL`-gated. 60-file Postgres sync at concurrency=4 imports all + no connection leak (probes `pg_stat_activity` before/after to confirm worker engines disconnected). 120-file serial-vs-parallel benchmark prints `SYNC_PARALLEL_BENCH N files | serial=Xms | parallel(4)=Yms | speedup=Zx`. Asserts parallel ≤ serial × 1.5 (CI-noise tolerant; not a strict speedup gate).
- `test/e2e/multi-source-bug-class.test.ts` — PGLite in-memory regression suite pinning every multi-source bug site: `listAllPageRefs` ordering by `(source_id, slug)`, `getPage` with sourceId picks the right `(source, slug)` row, `extract-takes` processes both overlapping `people/alice` rows independently, `listPages` filters correctly with `PageFilters.sourceId`, `addLinksBatch` with `from/to_source_id` targets the right rows, `validateSourceId` rejects path traversal, reverse-write disk layout uses `brainDir/.sources/<id>/<slug>.md` for non-default sources. No `DATABASE_URL` needed. Wired into `scripts/e2e-test-map.ts` so changes to extract-takes / patterns / synthesize / embed / extract / migrate-engine auto-trigger it.
- `test/e2e/multi-source-bug-class.test.ts` — PGLite in-memory regression suite pinning every multi-source bug site: `listAllPageRefs` ordering by `(source_id, slug)`, `getPage` with sourceId picks the right `(source, slug)` row, `extract-takes` processes both overlapping `people/alice` rows independently, `listPages` filters correctly with `PageFilters.sourceId`, `addLinksBatch` with `from/to_source_id` targets the right rows, `validateSourceId` rejects path traversal, reverse-write disk layout uses `brainDir/.sources/<id>/<slug>.md` for non-default sources, `copyMigrationSources` lands source metadata before overlapping-slug pages. No `DATABASE_URL` needed. Wired into `scripts/e2e-test-map.ts` so changes to extract-takes / patterns / synthesize / embed / extract / migrate-engine auto-trigger it.
- `test/e2e/migrate-engine-sources-postgres.test.ts``DATABASE_URL`-gated companion for `gbrain migrate --to`: migrates a PGLite brain carrying two non-default sources with overlapping slugs into real Postgres and asserts `copyMigrationSources` created every `sources` FK parent (config JSONB intact, not double-encoded) before any page write. Unit-level manifest identity (crash manifest resumes only against the SAME target; legacy engine-only manifests start fresh) is `test/migrate-engine-resume.test.ts`.
- `test/e2e/facts-fence-reconcile-postgres.test.ts``DATABASE_URL`-gated round-trip for the escape-aware fence parser: renders a `## Facts` fence whose cells carry literal pipes, backslashes (Windows paths), and empty cells via `renderFactsTable`, runs the wipe-and-reinsert reconcile (`runExtractFacts`) on real Postgres, and asserts every cell survives byte-identically with no column shift.
- `test/e2e/source-isolation-pglite.test.ts` — PGLite in-memory regression suite pinning the source-isolation seal at two layers. Engine layer: `searchKeyword` / `searchVector` / `searchKeywordChunks` / `listPages` / `getPage` / `traverseGraph` / `traversePaths` apply `sourceId` (scalar fast path) and `sourceIds` (array path) correctly across both engines. Op-handler layer: routes through `sourceScopeOpts(ctx)` so a `read+write`-scoped OAuth client bound to `--source dept-x` cannot see rows from neighboring sources via `search`, `query`, `list_pages`, `get_page`, or `find_experts`. Covers both `ctx.sourceId` (single-source clients) and `ctx.auth.allowedSources` (federated_read clients) precedence; federated array wins over scalar wins over nothing. No `DATABASE_URL` needed.
- `test/e2e/think-source-isolation-pglite.test.ts` — PGLite in-memory suite pinning the `think` gather stage's source scope: seeds three sources with cross-source links and embedded takes, then asserts `runGather` under a federated `sourceIds` grant (and under a scalar `sourceId`) keeps every stream — hybrid retrieval, takes keyword + vector (`searchTakes`/`searchTakesVector`), and the `traversePaths` graph walk — inside the grant while still reaching authorized neighboring sources. No `DATABASE_URL` needed.
- `test/e2e/skill-brain-first.test.ts` — doctor reports `skill_brain_first` check with structured issues; `--fix --dry-run` previews insertion without writing; `--fix` applies the canonical Convention callout idempotently; `brain_first: exempt` frontmatter resolves the warn; `brain_first_typo` surfaces a paste-ready hint; audit JSONL records `detected` / `resolved` / `fixed` transitions; stable brain emits 0 audit lines/run.
- Tier 2 (`test/e2e/skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI.
- If `.env.testing` doesn't exist in this directory, check sibling worktrees: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
File diff suppressed because one or more lines are too long
+1
View File
@@ -123,6 +123,7 @@ expansion (if enabled)
hybrid search:
├── vector (HNSW on chunk embeddings)
├── keyword (BM25 via tsvector)
├── relational (v0.42.34.0: typed-edge recall arm — relational queries only)
├── source-aware re-rank (CASE in SQL)
└── RRF fusion → top 30
+367
View File
@@ -0,0 +1,367 @@
# Community Ideas Ledger
> A diary of the **valuable ideas** surfaced by the community-PR wave, kept so that
> good thinking survives even when the PR that carried it is closed. gbrain moves
> fast and the maintainer's "cathedral" rewrites supersede most individual PRs —
> but the *idea* behind a closed PR is often still worth something.
>
> **Bar for this file:** an idea only earns a line if it is (a) still live on
> master and (b) genuinely valuable to gbrain users. **Graduating an idea to
> `TODOS.md` is a higher bar still** — it must serve the North Star (next-Postgres-
> for-memory: widest coverage, best-for-the-most-at-the-least) and be worth a
> maintainer-owned implementation. Most lines here will never graduate. That's fine.
>
> Status legend: **OPEN** = PR still open as a real merge candidate · **CLOSED** =
> PR closed, idea captured here · **HELD** = strategic, awaiting maintainer call.
> Provenance is credited to the contributor; scrub real private-network names per
> the repo privacy rule when anything here graduates to a public artifact.
_Generated from a full triage of the open-PR backlog (436 community PRs), 2026-06-07._
---
## 1. Internationalization — non-English brains are second-class
The single biggest coverage gap for "serve a billion people." Several independent
contributors hit the same walls.
- **Configurable FTS language** (#580/#581/#582, @rafaelreis-r) — **OPEN, high.**
Every `to_tsvector`/`tsquery` is hardcoded `'english'` (query side, trigger side,
and no reindex path), so non-English brains run every search through the English
stemmer. A coherent 3-PR set: `GBRAIN_FTS_LANGUAGE` config → migration recreating
triggers with the chosen language → `gbrain reindex-search-vector` to change it
post-install. **Strongest i18n candidate to graduate.**
- **Full-Unicode slugs** (#782, @tamagodo-fu; #514 zh, @JimmyJiang67) — **HELD, high.**
CJK slugs already work (`CJK_SLUG_CHARS`); generalize to all scripts (Cyrillic,
Devanagari, Hangul, …) and widen the remaining ASCII-only validators so non-ASCII
slugs flow end-to-end instead of being generated then rejected. #514 also carries a
corpus-driven `relationships-zh.json` verb dictionary for `inferLinkType` — a
reusable artifact for Chinese relationship typing.
- **CJK entity extraction** (#1637, @alkalide) — **OPEN, high.** Mention extraction is
ASCII-only (`TOKEN_RE`, `MIN_NAME_LENGTH=4`), so 23 char Chinese/Japanese/Korean
names are invisible to the gazetteer (there's an in-code TODO acknowledging it).
CJK detection + lower min-length + single-token pure-CJK titles + substring pass.
## 2. Reliability — the daily-driver failure modes
Recurring, production-observed failures. Many are tiny fixes with outsized impact;
these are the densest source of real bugs in the whole backlog.
- **Embedding egress waste** (#347/#460, @notjbg) — **OPEN, high.** `getChunks` does
`SELECT cc.*`, shipping the ~6KB pgvector embedding that `rowToChunk` immediately
discards — ~1922 GB/day egress on a busy Supabase brain. Enumerate the columns;
add a CI guard. (#460 dup of #347.)
- **Body-keyed embedding reuse** (#1424, @defenestrate2) — **OPEN, high.** Markdown
import re-embeds byte-identical chunks that merely shifted position, turning a
cosmetic edit into ~99K wasted re-embeds. Reuse by chunk-text hash like the code
path already does; add `--force` + a no-hash sentinel.
- **`embed --stale` full re-pull** (#775, @kyledeanjackson) — **CLOSED (partial on
master), high.** Re-pulled all chunks every cycle (~3TB/mo egress); steady-state
brains should do near-zero work. Master added a `countStaleChunks` early-exit;
verify it fully closes this.
- **Config round-trip storm** (#1694, @Omerbahari) — **OPEN, high.** A single query
fires ~85 serial single-key config `SELECT`s — invisible on PGLite, ~85 network
RTTs on a remote pooler. Batch + cache `getConfig` (`getConfigMany`).
- **cgroup-aware worker sizing** (#1244, @tyler3k1) — **OPEN, high.** `defaultWorkers()`
sizes from `os.totalmem()` (host RAM), so containerized installs (Railway/Fly/Render/
Cloud Run/ECS) oversize the pool and get OOM-killed mid-import. Use
`process.constrainedMemory()`.
- **Linux memory-pressure throttle** (#556, @chengzehsu) — **OPEN, high.** `os.freemem()`
is `MemFree` (excludes reclaimable cache), so healthy containers reject every batch
job. Read `MemAvailable` from `/proc/meminfo`.
- **propose_takes never caches empties** (#1218 @AdityaRajeshGadgil / #1760 @notjbg) —
**OPEN, high.** A valid `[]` extractor result writes no cache row, so unchanged pages
re-spend extractor tokens every ~5min cycle (57,885 calls/11 days observed). Sentinel
row keyed on `(source_id, page_slug, content_hash, prompt_version)`.
- **Prompt-cache opt-in on hot paths** (#1761, @notjbg) — **OPEN, high.** Only ~4.9% of
input tokens hit the Anthropic prompt cache because the highest-volume cycle/extraction
call sites don't set `cacheSystem:true` despite gateway support. One-line opt-ins.
- **Autopilot reliability cluster** (#232 @ianderse, #464/#465 @notjbg, #289 @RyanAlberts,
#477 @vinsew, #1935/#1936 @mdcruz88, #1906/#1891 @rayers/@jalagrange) — **OPEN, high.**
A family of distinct live bugs: argless `engine.connect()` wipes saved config and
crash-loops under launchd; `cwd=/` wrappers miss `brain/.env`; mtime-only lock probing
blocks respawn for 10min after OOM; no backoff on the 5-failure suicide cap;
disconnect-before-connect `reconnect()` bricks the engine on a transient blip; config
accessors lack the retry wrapper. **Pick the best fix per layer and land as a wave.**
- **lint `--fix` corrupts mid-doc fences** (#1417 @trinh-macbook, #1597 @chungty) —
**OPEN, high.** Detector/fixer regex disagree, so `lint --fix` strips the closing fence
of mid-document ```` ```markdown ```` blocks and autopilot re-corrupts the page every
cycle. Only unwrap whole-page fences.
- **backlinks worker defaults to `fix`** (#1853 @choomz; #1027 @sliday; #495 @23salus) —
**OPEN, high.** Empty-payload backlinks jobs default to `action='fix'`, silently
rewriting tracked markdown ("Referenced in" bullets) on every sync→embed→backlinks
chain (129 files/day in the wild). Default to `check`; require explicit opt-in. Also
fixes a duplicate-line accumulation bug.
- **`DATABASE_URL` hijack** (#1884, @awilkinson) — **OPEN, high.** A co-located app's
generic `DATABASE_URL` silently overrides the configured brain (wrong DB, or
auto-migrates it). Fix precedence: `GBRAIN_DATABASE_URL` > config.json > `DATABASE_URL`.
- **Engine-switch strips config** (#1088, @samchaudhary) — **OPEN, high.** `migrate --to`
rewrites config to just `{engine,url}`, dropping `embedding_model`/`dimensions`/keys;
migration "succeeds" but new embeds break.
- **Re-init silently corrupts the brain** (#1060, @vincedk-alt) — **OPEN, high.** Flag-less
re-init ignores persisted `embedding_model`/`dimensions` and writes a wrong-shape
OpenAI-1536 brain before the dim-check catches it.
- **IPv6-only direct URL** (#1006, @diazMelgarejo) — **OPEN, high.** `deriveDirectUrl`
turns a Session-Pooler URL into an IPv6-only host, ECONNREFUSED on IPv4-only networks
(the majority). Return null for pooler URLs.
- **HOME-isolation in tests** (#205/#517/#534 @orendi84, #434 @lloydarmbrust) — **OPEN,
high.** The E2E suite spawns `gbrain init/import` against the developer's real
`~/.gbrain/config.json`, clobbering their live DB URL+keys. Isolate HOME to a tmpdir.
*(A footgun that bites contributors of this very repo.)*
- **dim-aware embed write target** (#1263, @DmitryBMsk) — **OPEN, high.** `upsertChunks`
always writes the legacy `embedding vector(1536)` column, so brains on an alternate
column (`embedding_ze halfvec(2560)`) fail with dim-mismatch on every write.
- **Oversized chunks silently unembedded** (#1675, @lubos-buracinsky) — **OPEN, high.**
The code chunker emits giant literals/template strings whole; the embedder rejects
them and they vanish from semantic search. Cap chunk size so they stay embeddable.
- **Token-vs-char truncation** (#557 @chengzehsu, #990 @mgunnin, #1180 @kkroo,
#1281 @mmekkaoui, #1947 @100menotu001) — **OPEN, high.** The embed path truncates by
chars (`MAX_CHARS`) not tokens, so dense pages still exceed the 8192/300K-token ceiling
and loop forever on HTTP 400 with `embedded_at` never cleared; `isTokenLimitError`
misses OpenAI's real error string; llama-server's 32-input limit isn't capped; and
`--catch-up`'s unbounded budget overflows the 32-bit `setTimeout` and aborts after one
batch. A "make embedding backfills never silently wedge" cluster.
## 3. Search & retrieval quality
- **Keyword search ignores page titles** (#1646, @jeades) — **OPEN, high.** `searchKeyword`
ranks only chunk `search_vector`, never `pages.search_vector` (weight-A titles), so an
exact-title `gbrain search` returns nothing while `query` finds it. High-impact, tiny.
- **`code-def` misses most OO symbols** (#1628, @rayers) — **OPEN, high.** `DEF_TYPES`
omits method/constructor/field/struct/protocol, so `code-def` returns 0 for most
object-oriented code. Root-cause fix in `normalizeSymbolType` + `DEF_TYPES`.
(Prefer over #1701's fallback-only approach.)
- **doc-comment column is wired but dead** (#520, @Evode-Manirahari) — **OPEN, high.** FTS
weights `content_chunks.doc_comment` above chunk text but the column is never populated.
Extract JSDoc/docstrings per symbol via AST and thread through import.
- **autocut weak-top collapse** (#1863, @rayers) — **OPEN, high.** The fresh autocut
feature (#1682) normalizes the rerank gap by the top score, so a weak top (0.317→1.0)
looks like a confident cliff and rare cross-source queries collapse to 1 result. Add a
`minTopScore` floor.
- **Graph-hop wikilink rerank** (#717, @gwanghoon91) — **HELD, high.** Zero-token
score-shapers (graph-hop wikilink rerank + query-token disambiguation) claimed
+2.6/+2.8pt P@5/R@5 on BrainBench. Worth re-evaluating against the new retrieval
cathedral's ranker rather than merging the old diff.
- **Effective-date time filters** (#1706, @mvanhorn) — **OPEN, med.** `since`/`until`
filter on `updated_at`, so content dated to the past but edited recently is mis-filtered;
filter on `COALESCE(effective_date, updated_at, created_at)`.
## 4. Extraction & the knowledge graph
- **Obsidian wikilink → typed graph edges** (#87 @franmaranchello; alias/title/basename
fallback #1188 @rwbaker) — **OPEN/HELD, high.** `[[wikilinks]]`/`![[embeds]]` are
invisible to the graph. Materialize them as typed edges with alias (frontmatter
`aliases:`), first-H1-title, and basename fallback resolution (path-equality-only gives
~5.5% edge recall on real vaults). Master shipped global-basename (#1388); the alias/
title fallbacks are the still-novel part.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **OPEN, high.** The
link extractor's `DIR_PATTERN` is a frozen 16-prefix const that ignores pack-declared
`path_prefixes`, so default-pack installs silently lose wikilinks to `person/`,
`writing/`, `wiki/*`. Resolve prefixes from the active pack.
- **DB-source extraction** (#1539, @afshaker) — **OPEN, high.** The cycle's extract phase
only walks the filesystem, so DB-resident pages (imported transcripts, remote-DB brains)
never get links/timeline and `brain_score` is capped. Thread `source:'db'`.
- **source_id threaded through fs-walk extract** (#1719, @seungsu-kr) — **OPEN, high.**
fs-walk extractors omit `source_id`, defaulting to `'default'`, so the `pages` INNER JOIN
drops every row on non-default-source brains — silent 0 inserted.
- **extract `--stale` permanent-lag loop** (#1791, @Nazim22) — **OPEN, high.** Pages last
edited before the link-extractor version bump get stamped below the version threshold and
re-flag every run (~97% pages permanently "stale"). Stamp `GREATEST(updated_at, versionTs)`.
- **Plain-text NER for auto-link** (#1565, @donogeme) — **HELD, med.** Plain mentions of
people (no `[[wikilink]]`) never become edges. The opt-in idea is right; the shipped
implementation (capitalized-bigram regex, Western-names-only) is too crude — needs a
real NER pass to clear the graph-integrity bar.
## 5. Providers & the gateway
The AI-gateway + recipes + `user_provided_models` system already absorbed ~40
per-vendor embedding PRs (Ollama, Gemini, Azure, DashScope, DeepSeek, Zhipu, E5,
bge-m3, Copilot, Composio, Kimi, LM Studio, Mistral, Hunyuan, MiniMax…). The
*residue* worth keeping:
- **litellm proxy unusable for chat** (#1953 @miroslavb, #1938 @BKF-Gitty) — **OPEN, high.**
The `litellm-proxy` recipe declares only an embedding touchpoint (no chat), so
`chat_model=litellm:*` fails validation and `think` degrades to a misleading "set
ANTHROPIC_API_KEY"; and `build-gateway-config` never folds `litellm/openrouter/together`
keys, so configured proxy auth goes out unauthenticated. Plus user-provided custom-dim
embeddings are double-false-rejected in preflight. **The general-OpenAI-compat-proxy
story.**
- **Matryoshka dims threading** (#1072 @mgandal, #1240 @mike7seven) — **OPEN, high.**
Qwen3-Embedding returns its native dim (2560/4096) not the requested one because
`dimensions:N` isn't threaded for the openai-compat path, hard-failing a 1536-dim brain.
- **"Freeze provider at init, clear vectors on dim change"** (#100/#172, @niallobrien/
@nbzy1995) — **CLOSED, med.** A safety insight worth keeping even though the provider
PRs are superseded: persist+freeze the brain's provider/dim at init so a later env change
can't silently corrupt the vector space; clear stale embeddings on an intentional change.
- **China-region provider coverage** (#59 @Magicray1217, #1071 @AzeWZ) — **CLOSED, med.**
Make DashScope/DeepSeek/Zhipu first-class recipes that honor `provider_base_urls` (the
China-region endpoints) and provider batch limits — on-mission for global coverage.
- **Amazon Bedrock native** (#1826, @naterchrdsn) / **Jina asymmetric retrieval**
(#1930, @Whamp) — **HELD, high/med.** The maintainer pattern prefers the universal
litellm-proxy over per-vendor native recipes, but Bedrock (AWS IAM credential chain) and
Jina's asymmetric `input_type=document|query` are distinct enough to warrant a call.
- **Local-first chat parity** (#1854/#1855/#1858 @starm2010, #1423 @pabloglzg,
#1618 @punksterlabs) — **OPEN, high.** `FREE_LOCAL_CHAT_PROVIDERS` doesn't exist (only
embed), brainstorm/cycle/takes hardcode `anthropic:claude-sonnet-4-6`, and the
openai-compat `generateObject` path silently fails on providers that reject
`json_schema`. The "run gbrain fully local" cluster.
- **OpenRouter config key** (#1714 @tmchow), **OAuth bearer for AI providers**
(#1312 @pabloglzg), **API-key files** (#570 @shawnduggan) — **OPEN, med.** Credential
ergonomics: config-file key (not just env), externally-minted bearer tokens, and
`OPENAI_API_KEY_FILE` so OAuth harnesses don't inherit a raw key in `process.env`.
## 6. Auth, federation & access control (security-adjacent)
These cluster into a real theme: **runtime access control for remote/multi-tenant MCP
beyond prompt discipline.** Several are live security gaps (see the security list in the
triage report) and should be treated as a coordinated design, not piecemeal merges.
- **Clamp remote source overrides** (#1372, @jlfetter1) — **OPEN, high, SECURITY.** A
remote MCP caller can pass `source_id` (or `__all__`) to `query`/`get_page` to read
sources outside their OAuth `allowedSources` — the param bypasses `sourceScopeOpts`
(CWE-285). Clamp to token claims, fail-closed. **#1394 (get_page source_id) must land
*with* this clamp, not before it.**
- **Read-side prefix/federation enforcement** (#1860 @choomz, #1790 @colin-atlas,
#470 @AdityaRajeshGadgil, #1508 @tim404x) — **OPEN, high.** `bound_slug_prefixes` is
enforced on write but not read; exact `get_page` uses scalar `ctx.sourceId` while fuzzy
uses the federation ladder; unqualified search can scan isolated `--no-federated` sources.
Unify on one fail-closed visibility predicate across every read surface.
- **Per-OIDC-user access tiers** (#789, @0x471) — **HELD, high, SECURITY.** Map verified
OIDC end-users to `oauth_clients.access_tier` dispatch gates + shape filters — real
runtime access control. Pairs with multi-agent MCP hardening (#1316, @chipoto69, HELD).
- **Federated-read management CLI + admin UI** (#1592/#1601 @bitak1, #1558 @flamerged) —
**OPEN, high.** No CLI/UI to inspect or change a client's `federated_read` scope (raw
SQL only today). Atomic `array_append`/`array_remove` SQL to avoid read-modify-write
races, plus an admin Sources tab.
- **Pre-registration flow flags** (#894, @panda850819) — **OPEN, high, SECURITY.**
`register-client` hardcodes `redirect_uris=[]`, making the SECURITY.md-recommended
pre-registration (DCR-off) flow unusable for Claude.ai/ChatGPT connectors.
- **RFC 9728 `resource_metadata`** (#1410, @rayers) — **OPEN, high.** HTTP MCP 401s omit
the `resource_metadata` param the MCP auth spec + RFC 9728 require, so claude.ai/Cursor
can't discover the auth server and never start OAuth.
- **Server-enforced memory groups** (#1497, @oldmate99) — **HELD, med.** Audience-based
read/write via `memory_groups` + client-to-group assignment — strategic for hosted
multi-tenant, but overlaps the existing source-isolation model; a design call.
## 7. Security hardening (must not be lost)
- **Command injection in transcription** (#245, @aliceagent) — **OPEN, high, SECURITY.**
`transcription.ts` shell-interpolates an agent-controlled `audioPath` into `execSync`
ffprobe/ffmpeg/`rm -rf`. **Confirmed still present on master.** Switch to
`execFileSync` arg arrays + `fs.rmSync`.
- **Dotfile / skills-dir confinement** (#418/#419, @garagon) — **OPEN, high, SECURITY.**
`.gbrain-source` walk-up trusts any ancestor dotfile (source hijack on shared hosts);
`resolveWorkspaceSkillsDir` never canonicalizes (symlink escape). `lstat` ownership/
symlink/world-writable checks + realpath containment.
- **Destructive reclone gate** (#1705, @mvanhorn) — **OPEN, high, SECURITY.**
`recloneIfMissing` does `rm`+rename over `src.local_path` without verifying it's
gbrain-managed, so a re-pointed source can wipe a user's working tree. Gate behind
`isManagedRecloneTarget()` + reject `..`. *(The maintainer's own #1960 is the canonical
landing for this class — cross-check.)*
- **CORS preflight asymmetry** (#983, @yashkot007) — **OPEN, high, SECURITY.** Preflight
returns the full method/header surface unconditionally while the actual-request path
gates on the allowlist — leaks allowed surface to non-allowlisted origins.
- **jsonb double-encode corruption** (#1584 @warkcod, #597 @vinsew) — **OPEN, high,
SECURITY/integrity.** Source-config and subagent writers `JSON.stringify` into a
`::jsonb` cast — the exact postgres.js trap CLAUDE.md forbids; corrupts source config
(freshness/autopilot) and breaks dream synthesize slug-collection on real Postgres.
## 8. Developer experience & platform reach
- **Windows / CRLF portability** (#1294 @xwang4-svg, #1149 @samporter-31, #1554 @Sanjays2402,
#1396 @xuezhaolan) — **OPEN, high.** CRLF breaks frontmatter + skill-trigger parsing
(CI is Ubuntu-only so it never surfaces), `/dev/stdin` doesn't exist, a POSIX postinstall
one-liner hard-fails `bun install`, backslash bundle keys. A coordinated "first-class
Windows" pass. *(A working Windows binary + CI target #180/#181 is the prerequisite for
the full story.)*
- **`.gbrainignore` / per-repo exclusion** (#1483 @eepaul; repo-local code filters
#1011 @AndrewLauder; `--respect-gitignore` #1159 @jetsetterfl) — **OPEN, high.** Sync
indexes every file with no ignore mechanism (`data/`, `*.parquet`, fixtures, vendored
trees), bloating DB + embedding cost. gitignore-parity `.gbrainignore` + per-source
`excludePatterns`. *(See also the maintainer's walker-prune work; #1942 prunes
vendor/dist/build.)*
- **Monorepo sub-path sources** (#774, @jeremyknows) — **HELD, high.** `--src-subpath`
(split repo into git-root + logical-source axes) + `--exclude` so one repo can hold N
sources at subdirs.
- **MCP tool filtering** (#747, @joelwp) — **OPEN, high.** MCP advertises all ~51 ops to
every consumer (~10K tokens of schemas, tool confusion); `GBRAIN_EXPOSED_TOOLS` filters
the advertised surface.
- **Install-method detection for upgrade** (#538, @brucek) — **OPEN, high.** The README's
own recommended git-clone+bun-link install detects as `unknown`, so `gbrain upgrade`
offers three dead ends including a wrong npm package.
- **Runtime subagent defs** (#1282, @dcarolan1) — **OPEN, high.** The plugin loader
validates `SubagentDefinition[]` at startup but the handler never reads
`data.subagent_def`, so the persisted field is dead at runtime — callers must re-embed
the full system body in every job.
- **macOS Tahoe PGLite workaround** (#1671, @roysaurav) — **HELD, med.** PGLite's WASM
engine crashes on macOS 26 (Apple Silicon); document the native Homebrew Postgres+pgvector
fallback. Reader-valuable until the WASM crash is fixed upstream.
## 9. Capabilities & integrations (strategic — maintainer call)
These are net-new surfaces held for a product decision, not auto-closed.
- **Alternative engines** — SQLite/`bun:sqlite`+FTS5 single-file backend (#291, @mvanhorn)
and Neo4j GraphBrain REST backend (#594, @pkyanam). Both conflict with the two-engine
lockstep invariant and the Postgres-for-memory North Star, but the *zero-WASM single-file*
install story (SQLite) is strategically interesting. **HELD.**
- **Page versioning / soft-delete / read audit** (#573, @cropsgg) — **HELD, high.** Snapshots
with provenance, soft-delete tombstones + hard purge, read-path audit treating edits as
derivative works. Ambitious cathedral-scope; maintainer-owned territory.
- **Configurable embedding dimension** (#1051, @vincedk-alt) — **HELD, high.** `schema.sql`
hardcodes `vector(1536)`; read `embedding_dimensions` from config (default 1536). The
canonical fix that dozens of local-provider PRs hack around. *(Pairs with #1263.)*
- **Transcribe skill** (#1449, @RyanAlberts) — **OPEN, high.** Implements the empty
video/audio branch of `media-ingest` (YouTube captions fast path + yt-dlp/whisper
fallback), $0 by default. A genuine capability gap.
- **iPhone backup importer** (#1733, @H4RR1SON) — **HELD, med.** Local-CLI-only importer
for decrypted iPhone backups (contacts→person pages, iMessage→conversation pages); zero
network, thin-client refused.
- **Compounding dream phase** (#509, @durang) — **HELD, high.** An LLM "7th phase" that
*creates* structure (orphan-mention people, knowledge gaps, concept-dup at cosine>0.92,
decay, incomplete pages) vs the deterministic phases. Overlaps `enrich --thin`.
- **Codex-OAuth for dream** (#977, @barronlroth) / **dream gateway + `migrate-embedding-dim`**
(#1013, @cxbitz) — **HELD, high.** OAuth-backed chat for synthesis; a command to resize
the vector schema + clear incompatible embeddings.
- **Voice-extraction skill** (#300, @harjclaw) — **CLOSED, med.** Mine the user's outbound-
email corpus already in the brain to build a queryable writing-voice profile so agents
draft in the user's voice. Overlaps soul-audit.
- **MCP put_page parity + DB→markdown reconciliation** (#438, @rayzhux) — **HELD, high.**
A frontmatter-only safe auto-link mode for remote callers + `GBRAIN_BRAIN_ROOT` to render
remote writes back to markdown so MCP writes reach the git source-of-truth. Touches the
remote trust boundary — a design proposal, not a merge.
- **Recipe discovery convention** (#1279, @ialmeida-jera) — **OPEN, med.** `~/.gbrain/recipes/`
auto-discovery + `--external-dir`, loaded untrusted to keep the command-spawn boundary.
- **Destructive-op audit trail + audit-factory** (#1069/#1070, @vincedk-alt) — **HELD, med.**
Rotating JSONL forensic trail for hard-deletes + a shared `createAuditLogger` factory.
## 10. Doctor & brain-health observability
- **Queue dead-job visibility** (#1185, @ethanbeard) — **OPEN, high.** A collector can
heartbeat green while all its jobs die in the worker (3561 dead in the wild) and doctor
has zero view into the minions queue. Add a cross-cutting `[queue]` dead-jobs check.
- **Orphan-metric alignment** (#1107 @colin477, #915 @xaviroblessarries, #1202 @rwbaker) —
**OPEN, high.** `get_health` counts ingestion-by-design (`daily/`, briefings), soft-deleted,
and hub pages as orphans, distorting `brain_score`; CLI `find_orphans` uses a *different*
predicate than `getHealth`. Unify on one islanded predicate with sensible exclusions.
- **doctor check-name registry drift** (#1839, @mvanhorn) — **OPEN, med.** Several emitted
checks aren't registered in `doctor-categories`, printing `unknown check name` every run;
the drift guard only scanned `doctor.ts`, missing `onboard/checks.ts` emitters.
- **Honest stale-lock hint** (#1553, @Sanjays2402) — **OPEN, med.** doctor always says
`gbrain sync --break-lock`, which silently no-ops on `gbrain-cycle` locks.
---
## Cross-cutting observations for the maintainer
- **The same bug was filed many times.** `extract_facts.entity_hints` missing an `items`
schema came in ≥5 times (#812/#832/#847/#863/…, already fixed); the Postgres-singleton
disconnect class a dozen+ times; sync no-op freshness, slug-casing, and the embedding-
preflight false-reject each 515 times. A short "already fixed / known" note in the
release notes or a CONTRIBUTING "before you file" list would cut the re-file rate.
- **The recipe system is working as a pressure valve** — it correctly absorbed ~40 vendor
PRs into config rather than code. The remaining provider asks are about *capabilities*
the recipe schema doesn't yet express (asymmetric `input_type`, Matryoshka dims, per-item
RPM caps, alternative credential groups), not new vendors.
- **i18n (§1) and local-first chat (§5) are the two biggest "serve a billion" coverage
gaps** the community is repeatedly hitting and the best candidates to graduate to TODOs.
+11 -6
View File
@@ -43,8 +43,8 @@ genuinely has to change.
Switching dimensions requires:
1. Dropping the HNSW vector index (pgvector won't survive an `ALTER COLUMN TYPE`).
2. Altering the column type (Postgres only — PGLite cannot do this).
3. Wiping every existing embedding (the old vectors are unusable in the new space).
2. Wiping every existing embedding (the old vectors are unusable in the new space — and pgvector refuses to cast them across dimensions, so this must happen before the alter).
3. Altering the column type (Postgres only — PGLite cannot do this).
4. Re-embedding the entire corpus (can take hours on a 50K-page brain and costs $1-100 in API calls depending on model).
5. Conditionally recreating the index (HNSW supports up to 2000 dimensions per pgvector; above that you must use exact scans).
@@ -115,12 +115,17 @@ BEGIN;
-- 1. Drop the HNSW index. It can't survive the column type change.
DROP INDEX IF EXISTS idx_chunks_embedding;
-- 2. Alter the column type.
ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(<NEW_DIMS>);
-- 3. Clear stale embeddings so they don't survive into the new space.
-- 2. Clear stale embeddings FIRST. This must happen BEFORE the column
-- alter: pgvector refuses to cast existing vectors across dimensions
-- ("expected <NEW_DIMS> dimensions, not <OLD_DIMS>"), so altering a
-- column that still holds old-width vectors aborts the transaction.
-- NULLs cast fine. (The old vectors are unusable in the new space
-- anyway — this is the wipe step from the rationale above.)
UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;
-- 3. Alter the column type (all rows are NULL now, so the cast succeeds).
ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(<NEW_DIMS>);
-- 4. Recreate the HNSW index ONLY IF dims <= 2000. Above that, leave it
-- indexless and rely on exact scans (gbrain searchVector handles this
-- automatically — search just gets slower, not broken).
+10
View File
@@ -131,6 +131,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
history rewrite still hard-blocks even with `--skip-failed`. Run
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
5. **Import checkpoints name the import target, not the caller's CWD.**
Interrupted `gbrain import <dir>` runs may leave
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
checkpoint `dir` is the absolute, resolved import target captured when
import starts. It is not a cleanup instruction and it must not be
re-derived from the process working directory. Checkpoints written by
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
`kind: "import"` so downstream tools can validate the contract before
deciding whether to resume.
## How to Verify
1. **Edit a file and search for the change.** Edit a brain markdown file,
+97
View File
@@ -0,0 +1,97 @@
# Multi-language full-text search
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
environment variable. Default: `english`.
## How it works
Postgres text-search configurations control stemming and stop-word removal.
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
both sides of the search:
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
(Postgres and PGLite).
- **Write side** — the `update_page_search_vector` and
`update_chunk_search_vector` trigger functions that populate
`pages.search_vector` and `content_chunks.search_vector`.
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
interpolated into SQL (tsvector functions don't accept parameterized config
names). Invalid values fall back to `english` with a warning.
## Built-in languages
Set the env var to any configuration your Postgres instance ships:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
export GBRAIN_FTS_LANGUAGE=spanish
export GBRAIN_FTS_LANGUAGE=german
```
List what's available:
```sql
SELECT cfgname FROM pg_ts_config;
```
PGLite (the embedded default engine) ships the same built-in snowball
configurations as stock Postgres.
## First install vs. changing language later
On first install (or upgrade), the `configurable_fts_language` schema
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
that language. After the migration has run, changing the env var alone does
NOT retokenize existing rows — the migration shows as applied and is skipped.
Use the explicit command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview: language + row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command recreates both trigger functions under the new language and
backfills every existing `pages` and `content_chunks` row in batches,
streaming progress to stderr. It is idempotent: re-running with the same
language produces identical vectors. `--json` prints a machine-readable
result envelope but still requires `--yes` (or an interactive confirm).
## Recipe: accent-insensitive Portuguese (`pt_br`)
Brazilian Portuguese content often mixes accented and unaccented spellings
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
the `unaccent` extension, then stems with the portuguese snowball dictionary:
```sql
CREATE EXTENSION IF NOT EXISTS unaccent;
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
ALTER TEXT SEARCH CONFIGURATION pt_br
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, portuguese_stem;
```
Then point GBrain at it:
```bash
export GBRAIN_FTS_LANGUAGE=pt_br
gbrain reindex-search-vector --yes
```
Note: custom configurations require a real Postgres instance (e.g. the
Supabase engine). The config must exist BEFORE the migration or the reindex
command runs, or Postgres will reject the trigger recreation with
`text search configuration "pt_br" does not exist`.
## Caveats
- One language per brain: the setting is global to the database, not
per-source. Mixed-language brains should pick the dominant language (the
vector-search arm is language-agnostic and covers the rest).
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
the env var tokenizes new rows in `english` until the next reindex.
+97 -1
View File
@@ -114,8 +114,11 @@ Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
Full subcommand reference:
```
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated] [--force]
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
--path must be a git repo (or a subdirectory of one) — see
"The git requirement for --path sources" below. --force
skips that check to register before git-init exists.
gbrain sources list [--json] List all sources with page counts + federation state.
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
Cascade-delete a source (pages, chunks, timeline).
@@ -128,6 +131,47 @@ gbrain sources federate <id>
gbrain sources unfederate <id>
```
## The git requirement for --path sources
Every `--path` source must be a git repository (or live inside one — a
subdirectory of a git repo works too) with at least one committed, tracked
file under that path. `gbrain sources add` validates this at registration
time and refuses a directory that doesn't qualify — no `.git` at all, a
`git init` with no commit yet, or a commit made before `git add` — with an
actionable error instead of silently registering a source that will fail
(or worse, "succeed" while importing nothing) on its first `gbrain sync`.
Fix it with:
```bash
git -C <path> init
git -C <path> add -A
git -C <path> commit -m "initial import"
gbrain sources add <id> --path <path>
```
Two details that are easy to miss:
- **Files must actually be committed, not just present.** The sync walker
reads files through git objects, so `git init` alone — even followed by an
empty commit (`git commit --allow-empty`) — isn't enough. Registration
checks for real tracked content (`git ls-tree HEAD` scoped to the path),
not just a resolvable `HEAD`, so this footgun is caught immediately
instead of surfacing later as a sync that imports nothing.
- **`--force` registers the source anyway**, skipping the check. Use this if
you're registering a path before an automated pipeline gets around to
`git init`-ing it. GBrain never auto-`git init`s a `--path` source for
you — it's your directory, not a gbrain-managed clone (same consent
boundary as sync-time self-heal, which also never mutates a `--path`
source without an explicit ask).
**If sync ever reports a problem with the sync anchor** (`last_commit`) —
after a force-push, a history rewrite, or a from-scratch `git init` on a
directory that was synced before — you do not need to reset anything by
hand. `gbrain sync` detects an unreachable or non-ancestor anchor
automatically and recovers: either a full reimport (anchor object missing)
or a direct tree-to-tree diff against the orphaned bookmark (anchor present
but rewritten), advancing the anchor to the new HEAD when it completes.
## Citation format for agents
When agents receive multi-source results they MUST cite pages in
@@ -155,6 +199,58 @@ Reads span federated sources by default. Writes require a resolved
source (explicit, inferred, or default). The resolver never picks a
source silently when ambiguous — it errors with a clear fix.
## Durability: keep a brain repo in sync (auto-harden)
A long-lived agent that writes to a knowledge-wiki git repo needs three
things to never lose work: pull before it edits, push every write, and not
go stale while it sits idle. `gbrain sources harden` installs all of that,
idempotently. The moment you add a brain repo with a token, it runs
automatically:
```bash
# Clone + register a GitHub repo, then auto-harden it for durability.
# Use a fine-grained PAT scoped to just this repo.
gbrain sources add wiki --url https://github.com/you/brain-wiki.git --pat-file ~/.secrets/wiki-pat
# → clones, then installs: local auto-push hook, scripts/brain-commit-push.sh,
# always-on durability rules in AGENTS.md/RESOLVER.md, a 30-min pull cron,
# and a repo-scoped credential. Verifies push works before declaring done.
# Run the same audit on an existing source any time (idempotent):
gbrain sources harden wiki --pat-file ~/.secrets/wiki-pat
# Pull on demand (the cron calls the --path form, which never opens the DB):
gbrain sources pull wiki
# Remove the durability scaffolding (also runs automatically on `sources remove`):
gbrain sources unharden wiki
```
What hardening guarantees:
- **Pull-first, conflict-safe.** Every pull is a divergence-safe rebase. A
dirty working tree is skipped (your in-progress edits are never touched); a
rebase conflict is aborted cleanly and flagged for attention, never left
half-applied.
- **Push is never deferred.** `scripts/brain-commit-push.sh "<msg>" <path>`
commits and pushes atomically and refuses to report success without a
confirmed push. The post-commit hook is a best-effort background fallback;
the helper is the guarantee.
- **No silent staleness.** A 30-minute background pull keeps an idle session
current. It runs DB-free, so it never contends with a live brain for the
PGLite single-writer lock.
Flags: `--no-cron` skips the scheduled pull, `--no-verify` skips the push
probe, `--dry-run` reports what would change, `--json` emits a machine
report, `--all` hardens every source with a remote (same-account only).
`--no-harden` on `sources add` opts out of auto-harden.
Security: the push automation is installed locally per machine (never
committed into the repo), the token is wired per-repo (an existing
credential helper is reused when present), and it never appears in the repo,
the remote URL, logs, or the JSON report. For a self-hosted git server
reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
(default is HTTPS-only).
## Upgrading an existing brain
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
+79
View File
@@ -0,0 +1,79 @@
# Push-based context (#2095, v0.42.43.0)
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
## How it decides
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
merged with recency / frequency / user-role salience. Assistant-introduced
entities and "what did she invest in?" follow-ups whose antecedent was named
in the window now resolve.
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
+0.05 when mentioned in ≥2 turns or the newest turn.
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
explicit lower gate), suppress pages already surfaced (slug-presence only),
cap at 3 pages (hard cap 5).
## CLI
```bash
# one-shot: pipe recent turns (oldest → newest)
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
| gbrain volunteer-context
# streaming: volunteered pages print as the transcript flows
some-transcript-feed | gbrain watch --json
# the feedback loop: how often were volunteered pages actually opened?
gbrain volunteer-context --stats
```
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
and unrelated reads of the same page cause false positives. Use the per-arm
precision to tune `min_confidence`, not as an exact metric.
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
connection for the whole session — a concurrent `gbrain serve` or any write
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
input exits at EOF) or use the ambient reflex channel instead, which routes
through a running serve's resolve socket rather than taking the lock. Routing
watch through that same socket is a filed follow-up (TODOS.md). Postgres
brains are unaffected.
## Config
| Key | Default | What it does |
|---|---|---|
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
| `retrieval_reflex` | true | the ambient channel's master switch |
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
`session_id` / `turn` attribution params (watch stamps its own per-session id and
turn numbers in the feedback log), and `days` to size the `--stats` window.
## Storage + privacy
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
arm, confidence, channel, optional session/turn — the rationale is a
deterministic template string, never raw conversation text. Event writes are
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
applies to untrusted callers, applied unconditionally here so private fence rows
never reach a prompt regardless of caller trust.
@@ -208,9 +208,10 @@ architectural rounds shipped in the budget-cathedral wave that followed:
- **P3 (judge chunking):** `runJudge` in `src/core/brainstorm/judges.ts`
auto-chunks at 100 ideas/call. Context-window overflow is structurally
prevented.
- **P4 (unicode sanitization):** `sanitizeUnicode` in
`src/core/brainstorm/orchestrator.ts` strips unpaired surrogates before
serialization.
- **P4 (unicode sanitization):** `ensureWellFormed` (in `src/core/text-safe.ts`,
used by `src/core/brainstorm/orchestrator.ts`) replaces unpaired surrogates
with U+FFFD before serialization. (Consolidated from the original hand-rolled
`sanitizeUnicode` in v0.42.40.0 / #2011.)
- **P5 (BudgetTracker at the gateway layer):** new
`src/core/budget/budget-tracker.ts` is the canonical primitive. The
gateway's `withBudgetTracker(tracker, fn)` composes via
+9 -5
View File
@@ -34,7 +34,7 @@ The resolved provider + dimensions get persisted to `~/.gbrain/config.json` atom
| `zhipu` | `ZHIPUAI_API_KEY` | 1024 | varies | no | no |
| `ollama` | (none — runs locally) | 768 | 0 | yes | no |
| `llama-server` | (none — runs locally) | user-set | 0 | yes | no |
| `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | no |
| `litellm` | `LITELLM_API_KEY` (optional) | user-set | varies | yes (proxy) | yes (backend permitting) |
| `together` | `TOGETHER_API_KEY` | 768 | varies | no | no |
| `anthropic` | (no embedding model — chat only) | — | — | — | — |
| `deepseek` | (no embedding model — chat only) | — | — | — | — |
@@ -77,6 +77,8 @@ The doctor distinguishes two repair paths:
Default. Set `OPENAI_API_KEY`. Models: `text-embedding-3-large` (3072 max, 1536 default), `text-embedding-3-small` (1536). Matryoshka via the `dimensions` field — gbrain pins it from `embedding_dimensions` config so existing 1536-dim brains stay aligned across SDK upgrades.
Optional `OPENAI_BASE_URL` — point the native OpenAI provider at an OpenAI-compatible gateway. A bare host is normalized to carry the `/v1` suffix automatically (so `https://gw.example.com` and `https://gw.example.com/v1` both work); when unset, the SDK's default endpoint is untouched. `ANTHROPIC_BASE_URL` gets the same normalization for Anthropic chat/expansion calls.
### Voyage AI
Best-in-class quality on the Voyage 4 family (Jan 2026 release). Set `VOYAGE_API_KEY`. Models: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-4-nano`, `voyage-3.5`, `voyage-code-3` (code-tuned), `voyage-finance-2`, `voyage-law-2`, `voyage-multimodal-3` (text + image).
@@ -141,13 +143,15 @@ Set `ZHIPUAI_API_KEY`. Models: `embedding-3` (current; Matryoshka 256-2048 dims)
No env required — Ollama runs unauthenticated locally. Optional `OLLAMA_BASE_URL` (default `http://localhost:11434/v1`) and `OLLAMA_API_KEY` (for auth-enabled deployments).
Recipe ships with `nomic-embed-text` (768d, recommended), `mxbai-embed-large` (1024d), `all-minilm` (384d). `gbrain providers test --model ollama:nomic-embed-text` smoke-tests the local install.
Recipe ships with `nomic-embed-text` (768d, recommended), `mxbai-embed-large` (1024d), `all-minilm` (384d), plus the larger modern embedders `qwen3-embed-8b` (4096d) and `snowflake-arctic-embed-l-v2` (1024d). `gbrain providers test --model ollama:nomic-embed-text` smoke-tests the local install.
The recipe default is `nomic-embed-text`'s 768 dims. If you run one of the larger models, declare its native dimension with `--embedding-dimensions <N>` at init — gbrain trusts the value you declare for local recipes instead of rejecting a non-768 width.
### llama-server (local, llama.cpp)
`llama.cpp`'s `llama-server --embeddings` endpoint. No env required. Optional `LLAMA_SERVER_BASE_URL` (default `http://localhost:8080/v1`) and `LLAMA_SERVER_API_KEY`.
User-driven models: launch llama-server with `--model <gguf-path> --embeddings`, then run `gbrain init --embedding-model llama-server:<your-id> --embedding-dimensions <N>`. The recipe refuses the implicit shorthand `--model llama-server` because there's no canonical first model.
User-driven models: launch llama-server with `--model <gguf-path> --embeddings`, then run `gbrain init --embedding-model llama-server:<your-id> --embedding-dimensions <N>`. gbrain trusts the dimension you declare (you know the GGUF you launched); the recipe refuses the implicit shorthand `--model llama-server` because there's no canonical first model.
### LiteLLM proxy (universal escape hatch)
@@ -155,6 +159,8 @@ Run [LiteLLM](https://docs.litellm.ai/docs/proxy/quick_start) in front of any pr
This is the catch-all for "my provider isn't in the list above." Set up LiteLLM, then `gbrain init --embedding-model litellm:<your-model-id> --embedding-dimensions <N>`.
**Include the `/v1` suffix in `LITELLM_BASE_URL` if your proxy serves the OpenAI route there** (e.g. `http://localhost:4000/v1`). Many LiteLLM deployments expose the OpenAI-compatible API only under `/v1`; pointing gbrain at the bare host 404s or fails authentication with no hint. gbrain trusts the dimension you declare for the proxy-backed model — the proxy's backend, not gbrain, decides the true width — so `--embedding-dimensions <N>` is required and accepted as-is.
## Choosing dimensions
Three numbers matter:
@@ -183,5 +189,3 @@ The supported paths:
- **Postgres (Supabase / self-hosted):** follow the SQL recipe in `docs/embedding-migrations.md` (drop the HNSW index, ALTER COLUMN TYPE, clear stale embeddings, recreate the index conditionally, then `gbrain init --supabase --embedding-model X --embedding-dimensions N` to update the file plane and re-embed).
`gbrain doctor` 8c "alternative_providers" surfaces unconfigured providers whose env is already set — useful when you've configured OpenAI but also have e.g. `VOYAGE_API_KEY` exported and want to know you can switch without extra setup.
`gbrain doctor` 8c "alternative_providers" surfaces unconfigured providers whose env is already set — useful when you've configured OpenAI but also have e.g. `VOYAGE_API_KEY` exported and want to know you can switch without extra setup.
+2 -1
View File
@@ -7,7 +7,7 @@ brain source's repo that runs `gbrain frontmatter validate` against staged
## What the hook catches
The same seven validation classes the `frontmatter-guard` skill and
The same eight validation classes the `frontmatter-guard` skill and
`gbrain doctor`'s `frontmatter_integrity` subcheck report:
| Code | What it catches |
@@ -18,6 +18,7 @@ The same seven validation classes the `frontmatter-guard` skill and
| `SLUG_MISMATCH` | `slug:` in frontmatter doesn't match path-derived slug |
| `NULL_BYTES` | Binary corruption (`\x00`) anywhere in the content |
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape that breaks YAML |
| `NON_STRING_FIELD` | `title`/`type`/`slug` is an unquoted non-string scalar (`title: 123`) |
| `EMPTY_FRONTMATTER` | `---` ... `---` with nothing meaningful between |
## Install
+8 -1
View File
@@ -74,13 +74,20 @@ to the HTTP server, so no migration is required.
gbrain serve --http --port 3131
```
On first start, the server prints an **admin bootstrap token** to stderr:
On first start in an interactive terminal, the server prints an **admin
bootstrap token** to stderr:
```
Admin bootstrap token: 3a1f9c...
Open http://localhost:3131/admin and paste it to log in.
```
On a non-TTY start (systemd, Docker, any piped or captured logs) the generated
token is hidden so it never lands in log storage. For headless deploys either
set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or
run `gbrain serve --http --print-admin-token` once on a trusted terminal to
force printing.
Save this token. Open `http://localhost:3131/admin` and paste it to access the
dashboard. The dashboard shows live activity, registered clients, request logs,
and per-client config export.
+111
View File
@@ -0,0 +1,111 @@
# Spend controls
GBrain's embedding-spend gates in one place: every gate, its config key, default,
whether it blocks or just informs, how to widen or disable it, and how the
`spend.posture` switch governs all of them.
The orienting idea: **GBrain itself is rounding error; the spend that matters is
downstream embedding.** These gates exist so a routine sync or enrich can't run up
an unexpected embedding bill, while never wedging an unattended cron.
## `spend.posture` — one switch for "cost is not my constraint"
```bash
gbrain config set spend.posture tokenmax # all cost gates become informational
gbrain config set spend.posture gated # default — gates enforce
```
| Value | Effect |
|-------|--------|
| `gated` (default) | Every cost gate enforces its limit as documented below. |
| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. |
`spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs
retrieval payload size, not embedding spend). When a gate fires and
`search.mode=tokenmax` but `spend.posture` is unset, the gate prints a one-line hint
pointing at this switch.
**Precedence:** an explicit per-call cap (`--max-usd N`, `--max-cost N`) always wins
over posture. `tokenmax` only governs the default/absent case — it never overrides a
number you typed on the command line.
## Off switches (`off` / `unlimited` / `none`)
The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to mean
"no limit" — no more setting sentinel values like `100000`.
- `0` is **not** "off". On `sync.cost_gate_min_usd`, `0` means "block on any nonzero
spend" (a real choice). On the backfill caps, `0` falls back to the default.
- Internally "no limit" is the string `unlimited` in any printed/JSON output and "no
cap" inside the budget tracker — never a raw `Infinity` (which would serialize to
`null` in ledger rows).
## The gates
| Gate | Config key | Default | Blocks? | Off switch | tokenmax |
|------|-----------|---------|---------|-----------|----------|
| Sync inline-embed cost gate | `sync.cost_gate_min_usd` | `0.50` | TTY prompt / non-TTY auto-defer | `off` (or `0` = block-on-any) | informational |
| Backfill 24h per-source spend cap | `embed.backfill_max_usd_per_source_24h` | `25` | refuses submission | `off` (`0` → default) | bypassed (still ledgered) |
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
### Sync inline-embed cost gate
Fires only when sync embeds **inline** (federated_v2 off, or `--serial` without
`--no-embed`). Under federated_v2 + parallel, embedding is deferred to capped backfill
jobs and the gate is informational. The estimate prices the **delta** — the files this
sync will actually import (fetched-first, so it sees commits the run is about to pull) —
not the whole tree. A busy brain with a dirty working tree but caught-up commits
estimates `$0`, because an attached-HEAD sync imports only the committed diff.
Behavior above the floor:
- **TTY:** prompts `[y/N]`.
- **Non-interactive (cron/agent):** **auto-defers** embeds to capped backfill jobs and
exits 0 — it never wedges the pipeline. The backlog drains via the jobs worker or
`gbrain embed --stale`. Pass `--yes` to embed inline instead.
Output format splits on the explicit `--json` flag: `--json` emits a structured
envelope; otherwise human text. Every gate message carries paste-ready knobs.
`--full` re-embeds the stale backlog inline (full sync sweeps it), so a `--full`
estimate is `delta + stale backlog`, labeled as such.
### Estimate labels
- `~N tokens (delta: changed files since last sync)` — the precise estimate.
- `<=N tokens (full-tree ceiling for K source(s): <reasons> …)` — a conservative
over-count used only when a precise delta can't be computed: a first sync, a chunker
version drift (forces a full re-chunk), or git being unavailable. Unchanged files
still skip via `content_hash` at execution, so the ceiling over-states real spend.
## Notes & limits
- **Pre-pull window:** the gate fetches before estimating, so it prices what the run
will pull. If a fetch fails (offline), it estimates against local HEAD and labels the
result; the bounded residual is priced on the next run.
- **Single-source `gbrain sync`** carries the same gate as `sync --all` (it previously
embedded inline with no preview).
- **Recovery under parallel:** `--skip-failed` / `--retry-failed` work under parallel
sync (the failure ledger is per-source and lock-serialized) — you no longer have to
drop to `--serial`, which is what used to arm the inline gate.
## Escape hatches at a glance
```bash
# Never gate this brain on cost:
gbrain config set spend.posture tokenmax
# Widen the sync inline floor to $5:
gbrain config set sync.cost_gate_min_usd 5
# Disable the sync inline floor entirely:
gbrain config set sync.cost_gate_min_usd off
# Lift the backfill 24h spend cap:
gbrain config set embed.backfill_max_usd_per_source_24h off
# Run enrich uncapped non-interactively:
gbrain enrich --max-usd off # or: gbrain config set spend.posture tokenmax
```
+1 -1
View File
@@ -158,7 +158,7 @@ gbrain serve --http --port 3131 --bind 0.0.0.0
The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface.
The server prints an admin bootstrap token to stderr on first start. Save it. You'll use it once for the admin dashboard.
The server prints an admin bootstrap token to stderr on first start when run in an interactive terminal. Save it. You'll use it once for the admin dashboard. On a non-TTY start (systemd, Docker, piped logs) the token is hidden from logs — set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` yourself or pass `--print-admin-token` on a trusted terminal instead.
For development, tunnel the local server out via ngrok:
+1 -1
View File
@@ -86,7 +86,7 @@ Save this token. You'll need it for the AlphaClaw setup.
AlphaClaw is the setup harness that manages OpenClaw deployment.
1. Go to [alphaclaw.com](https://alphaclaw.com)
1. Go to [alphaclaw.md](https://alphaclaw.md)
2. Enter your **workspace repo** (not the brain repo): `your-org/myagent`
3. Select "Use existing" if the repo already exists
4. Enter your GitHub PAT from Step 2
@@ -415,6 +415,7 @@ export async function main(argv: string[]): Promise<number> {
chat_model: config?.chat_model ?? modelFull,
chat_fallback_chain: config?.chat_fallback_chain,
base_urls: config?.provider_base_urls,
provider_chat_options: config?.provider_chat_options,
env: { ...process.env } as Record<string, string>,
});
+314 -9
View File
@@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
unset) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
@@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -207,9 +208,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
hand-roll source filtering — a missed thread is a cross-source data leak.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
Guarded by `scripts/check-jsonb-pattern.sh`.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -245,6 +251,8 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
@@ -297,6 +305,7 @@ project resolves through `src/core/search/mode.ts`.
| `intentWeighting` | true | true | true |
| `tokenBudget` | **4000** | **12000** | **off** |
| `expansion` (LLM multi-query) | false | false | **true** |
| `relationalRetrieval` | false | **true** | **true** |
| `searchLimit` default | 10 | 25 | 50 |
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
@@ -355,6 +364,19 @@ written against `embedding` (1536d OpenAI). Existing v=2 rows become
unreachable on first re-query (one-time miss spike on upgrade);
`mode.ts:KNOBS_HASH_VERSION` is the single source of truth.
**v0.42.34.0 knobs_hash v=9 → v=10.** Folds the `relationalRetrieval` knob +
depth into the cache key so a relational-on result set can't be served to a
relational-off lookup (same contamination class as graph_signals). One-time
miss spike on upgrade.
**Relational retrieval (v0.42.34.0).** `relationalRetrieval` (on for
balanced/tokenmax) adds a fourth recall arm: a relational query ("who invested
in X", "what connects A and B") resolves its seed entity and walks the typed-edge
graph (`src/core/search/relational-recall.ts` + `relational-intent.ts`,
`engine.relationalFanout`), injecting edge-derived answers into RRF. Within-source,
deterministic, mentions-excluded by default, pure no-op for non-relational queries.
The `query` op's `relational` flag forces it on/off per call.
**Three CLI surfaces:**
gbrain search modes # what is running, with per-knob attribution
@@ -386,7 +408,7 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
@@ -405,6 +427,17 @@ routing is narrowed to what the skill actually covers.
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
(agent-readable health report).
**Brain-resident skillpacks + advisor (v0.42.47.0, #2180):** A brain repo can carry its
own publishable skillpack (`brain_resident: true` in `skillpack.json` + `schema_pack`);
`gbrain skillpack init-brain-pack` scaffolds one with a 5-section machine-parseable README.
Connecting harnesses discover it on `gbrain sources add` (Topology A advisory, bounded nag
via `nag-state.ts`) and over MCP via the source-scoped `list_brain_skillpack` op +
`get_skill --source_id` (gated by `mcp.publish_skills`). The bundled `gbrain-advisor` skill
+ `gbrain advisor` op compute a ranked, read-only list of high-leverage actions from brain
state (8 collectors in `src/core/advisor/`); `--json`+exit codes for CI/cron, local-only
`--apply <id>` behind confirm, exposed over MCP behind `mcp.publish_advisor` (default off,
read-only on remote). Thin-client binary install stays deferred to PR2 `build_skillpack`.
**Routing-table compression (v0.32.3.0):** `skills/functional-area-resolver/` —
two-layer dispatch pattern for shrinking large AGENTS.md / RESOLVER.md files
(>=12KB) without losing routing accuracy. Replaces one row per skill with one
@@ -504,6 +537,76 @@ For background tasks (`run_in_background: true`), the harness captures the exit
file separately — use it via the bg task's `<id>.exit` file, not the streamed
output.
## Sync resumability + lock tuning (v0.42.x, #1794)
`gbrain sync` is resumable and converges under pool exhaustion + repeated kills.
Progress banks into the append-only `op_checkpoint_paths` table (one row per drained
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
run resumes from the checkpoint and `last_commit` only advances on true completion. The
per-source lock heartbeats through the direct pool and refuses to steal a live,
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
hatches — no config-dashboard surface by design):
| Env var | Default | What it does |
|---|---|---|
| `GBRAIN_SYNC_CHECKPOINT_EVERY` | 1000 | Flush the checkpoint every N drained files. |
| `GBRAIN_SYNC_CHECKPOINT_SECONDS` | 10 | Also flush every N seconds (whichever comes first) — bounds worst-case loss regardless of throughput. Flush also fires after the first file. |
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
## Pace Mode (DB-contention-aware backfill pacing)
A naive `gbrain embed --stale` / large `sync` can saturate a PgBouncer
transaction-mode pooler and starve the minion supervisor's lock renewals
(`lock-renewal-failed` → dead jobs). Pacing is the native, composable fix — it
replaces external SIGSTOP/SIGCONT wrapper scripts. **Opt-in: default mode `off`.**
The composable primitive is `src/core/db-pacer.ts` (`createDbPacer`):
- **Concurrency cap is the real lever** (caps simultaneous in-flight DB writes =
pooler slots held). Embed paths set their worker count to `maxConcurrency`
(single pool, no permit); `sync` uses the shared `acquire()` **permit** because
each parallel worker owns a separate engine (one budget must span pools).
- **In-band signal** (`observe(ms)` EWMA from the work's own queries — never
blind the way an out-of-band probe pool was). **No probe loop, no
`probeLatency` engine method.**
- **Cooperative `pace()` sleep** on `setTimeout` (keeps the lock heartbeat
firing), jittered to avoid a thundering-herd resume. `acquire()`/`pace()` throw
`AbortError` on cancel; everything else is fail-open (a pacer bug never kills a
backfill, never throws an unhandledRejection).
Named bundles resolve through `src/core/pace-mode.ts` (`resolvePaceMode`), mirror
of the search-mode pattern but with **env ABOVE config** (incident escape hatch):
per-call flag → GBRAIN_PACE_* env → config (pace.*) → PACE_BUNDLES[mode] → off
| Knob | off | gentle | balanced | aggressive |
|---|---|---|---|---|
| `maxConcurrency` | (off) | 4 | 8 | 16 |
| `paceAtMs` (EWMA → sleep) | — | 250 | 500 | 1000 |
| `maxSleepMs` (jittered cap) | — | 2000 | 1500 | 1000 |
**Surfaces.** `gbrain embed --stale --pace[=mode]` (bare `--pace` = balanced),
`--pace-max-concurrency=N`. `--background` carries explicit pace OVERRIDES (not
the resolved bundle) into the `embed` job payload; the handler re-resolves
env>config>bundle at execution so `GBRAIN_PACE_*` still wins (CX5). Config-level
`pace.mode` paces EVERY `runEmbedCore` caller (cycle embed, embed-catch-up,
sync-auto-embed) and the prod `embed-backfill` job automatically. `sync` reads
env/config. PGLite / mode `off` → no-op pacer.
**Correctness fixes pacing bundles** (longer paced runs widen these): CLI
`embed --stale` single-flights via the SAME per-source lock key as the
`embed-backfill` handler (`src/core/embed-backfill-lock.ts`; all-source runs lock
every source in sorted order) so a hand-run backfill and a queued job can't race
the NULL→non-NULL upsert (`TODOS:2299`); a **bounded** end-of-run keyset re-entry
(max 3 + forward-progress, paced runs only) catches rows inserted behind the
cursor (`TODOS:2301`); and the embed wall-clock budget timer is re-armed around
`pace()` sleeps so paced time doesn't burn the work budget.
`EmbedResult.pacing` carries the end-of-run telemetry (cap, samples, EWMA, slept
ms, max waiters) for `--json`; a one-line summary prints to stderr.
## Build
`bun build --compile --outfile bin/gbrain src/cli.ts`
@@ -1305,6 +1408,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Morning prep, meeting context, day planning | `skills/daily-task-prep/SKILL.md` |
| Daily briefing, "what's happening today" | `skills/briefing/SKILL.md` |
| Cron scheduling, quiet hours, job staggering | `skills/cron-scheduler/SKILL.md` |
| "get more out of gbrain", "is my brain set up right", "weekly brain checkup", "advise me on my brain", "gbrain advisor" | `skills/gbrain-advisor/SKILL.md` |
| Save or load reports | `skills/reports/SKILL.md` |
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
@@ -1648,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
```
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
```bash
export GBRAIN_FTS_LANGUAGE=portuguese
gbrain reindex-search-vector --dry-run # preview row counts
gbrain reindex-search-vector --yes # recreate triggers + backfill
```
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
@@ -1679,6 +1801,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
## Troubleshooting
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
**Hourly cron sync keeps timing out on a federated brain?** v0.41.13.0 ships
@@ -1935,7 +2059,7 @@ export interface BrainEngine {
**Slug-based API, not ID-based.** Every method takes slugs, not numeric IDs. The engine resolves slugs to IDs internally. This keeps the interface portable... slugs are strings, IDs are database-specific.
**Embedding is NOT in the engine.** The engine stores embeddings and searches by vector, but it doesn't generate embeddings. `src/core/embedding.ts` handles that. This is intentional: embedding is an external API call (OpenAI), not a storage concern. All engines share the same embedding service.
**Embedding is NOT in the engine.** The engine stores embeddings and searches by vector, but it doesn't generate embeddings. `src/core/embedding.ts` handles that (a thin delegation to the provider-agnostic AI gateway in `src/core/ai/gateway.ts`). This is intentional: embedding is an external API call (OpenAI, Voyage, a local Ollama — whichever provider you configured), not a storage concern. All engines share the same embedding service.
**Chunking is NOT in the engine.** Same logic. `src/core/chunkers/` handles chunking. The engine stores and retrieves chunks. All engines share the same chunkers.
@@ -1989,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
Defense-in-depth layer for Postgres deployments that want the database itself
to enforce source isolation, in addition to the mandatory app-layer filters
(`sourceScopeOpts` — layer 1, always on).
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
source-scoped read methods wrap their queries in a transaction that first runs
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
bound params). An RLS policy can then filter rows by
`current_setting('app.scopes', true)`.
**Default off.** With the env var unset, reads call through on the shared pool
exactly as before — no per-read transaction, no pool-slot hold (the search
methods keep the transaction they always had for their `SET LOCAL
statement_timeout`). Existing operators see zero behavior change.
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
```sql
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
CREATE POLICY pages_scope_filter ON pages
USING (current_setting('app.scopes', true) = '*'
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
-- Required: connections that don't run through the scoped read helper
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
-- see zero rows once the policy exists:
ALTER ROLE <runtime-role> SET app.scopes = '*';
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
```
Safe to enable in either order: the env var without a policy is a no-op
setting; a policy without the env var is enforced only via the role default.
**Honest caveat:** only read paths routed through the scoped helper carry a
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
run under the role default and are not backstopped per caller. This is layer 2;
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
live in `test/postgres-engine-rls-scope.test.ts`.
## PGLiteEngine (v0.7, ships)
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
@@ -2017,6 +2186,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
| Form | Verdict |
|---|---|
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
cast then wraps that already-JSON string into a jsonb scalar string instead of
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
why a regression only shows up on Postgres (and why the parity test must run there).
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
`jsonb-guard-ok` comment.
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
and assert `jsonb_typeof` — the assertion PGLite cannot make.
## Adding a new engine
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
@@ -2628,6 +2830,16 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
history rewrite still hard-blocks even with `--skip-failed`. Run
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
5. **Import checkpoints name the import target, not the caller's CWD.**
Interrupted `gbrain import <dir>` runs may leave
`~/.gbrain/import-checkpoint.json` so the next import can resume. The
checkpoint `dir` is the absolute, resolved import target captured when
import starts. It is not a cleanup instruction and it must not be
re-derived from the process working directory. Checkpoints written by
gbrain include `schema_version: 1`, `owner: "gbrain"`, and
`kind: "import"` so downstream tools can validate the contract before
deciding whether to resume.
## How to Verify
1. **Edit a file and search for the change.** Edit a brain markdown file,
@@ -3340,6 +3552,92 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md).
---
## docs/guides/push-context.md
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md
# Push-based context (#2095, v0.42.43.0)
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
## How it decides
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
merged with recency / frequency / user-role salience. Assistant-introduced
entities and "what did she invest in?" follow-ups whose antecedent was named
in the window now resolve.
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
+0.05 when mentioned in ≥2 turns or the newest turn.
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
explicit lower gate), suppress pages already surfaced (slug-presence only),
cap at 3 pages (hard cap 5).
## CLI
```bash
# one-shot: pipe recent turns (oldest → newest)
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
| gbrain volunteer-context
# streaming: volunteered pages print as the transcript flows
some-transcript-feed | gbrain watch --json
# the feedback loop: how often were volunteered pages actually opened?
gbrain volunteer-context --stats
```
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
and unrelated reads of the same page cause false positives. Use the per-arm
precision to tune `min_confidence`, not as an exact metric.
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
connection for the whole session — a concurrent `gbrain serve` or any write
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
input exits at EOF) or use the ambient reflex channel instead, which routes
through a running serve's resolve socket rather than taking the lock. Routing
watch through that same socket is a filed follow-up (TODOS.md). Postgres
brains are unaffected.
## Config
| Key | Default | What it does |
|---|---|---|
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
| `retrieval_reflex` | true | the ambient channel's master switch |
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
`session_id` / `turn` attribution params (watch stamps its own per-session id and
turn numbers in the feedback log), and `days` to size the `--stats` window.
## Storage + privacy
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
arm, confidence, channel, optional session/turn — the rationale is a
deterministic template string, never raw conversation text. Event writes are
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
applies to untrusted callers, applied unconditionally here so private fence rows
never reach a prompt regardless of caller trust.
---
## docs/mcp/DEPLOY.md
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md
@@ -3420,13 +3718,20 @@ to the HTTP server, so no migration is required.
gbrain serve --http --port 3131
```
On first start, the server prints an **admin bootstrap token** to stderr:
On first start in an interactive terminal, the server prints an **admin
bootstrap token** to stderr:
```
Admin bootstrap token: 3a1f9c...
Open http://localhost:3131/admin and paste it to log in.
```
On a non-TTY start (systemd, Docker, any piped or captured logs) the generated
token is hidden so it never lands in log storage. For headless deploys either
set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or
run `gbrain serve --http --print-admin-token` once on a trusted terminal to
force printing.
Save this token. Open `http://localhost:3131/admin` and paste it to access the
dashboard. The dashboard shows live activity, registered clients, request logs,
and per-client config export.
+1
View File
@@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
- [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.
- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
## AI providers
+1
View File
@@ -46,6 +46,7 @@
"skills/data-research",
"skills/enrich",
"skills/functional-area-resolver",
"skills/gbrain-advisor",
"skills/idea-ingest",
"skills/idea-lineage",
"skills/ingest",
+16 -4
View File
@@ -65,6 +65,7 @@
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:search-path": "scripts/check-search-path.sh",
"check:no-double-retry": "scripts/check-no-double-retry.sh",
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
"check:worker-lock-renewal-shape": "scripts/check-worker-lock-renewal-shape.sh",
@@ -83,7 +84,7 @@
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"postinstall": "bun run scripts/postinstall.ts",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
},
@@ -117,8 +118,8 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"js-yaml": "^3.15.0",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
@@ -143,5 +144,16 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.32.0"
"version": "0.42.64.0",
"overrides": {
"@hono/node-server": "^1.19.13",
"fast-uri": "^3.1.2",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"qs": "^6.15.2",
"js-yaml": "^3.15.0"
}
}
+54
View File
@@ -0,0 +1,54 @@
---
id: retrieval-reflex
name: Retrieval Reflex
version: 0.1.0
description: Teaches the host agent WHEN to look something up and WHAT to pull. Ships a policy skill (trigger + retrieval spec) into the host resolver; pairs with the deterministic pointer layer in the context engine.
category: reflex
install_kind: copy-into-host-repo
requires: []
secrets: []
health_checks:
- type: command
argv: [gbrain, doctor, --json]
label: Retrieval reflex wiring (see retrieval_reflex_health)
setup_time: 2 min
cost_estimate: "$0 — zero-LLM deterministic layer + a prose policy skill"
---
# Retrieval Reflex: teach the agent *when* and *what* to retrieve
gbrain is great at **storing** knowledge and at **injecting deterministic
context** every turn. It does not, by itself, teach the host agent the *policy*
of retrieval: **when** to look something up and **what** to pull. Without it,
the agent can discuss a person who has a rich brain page for several messages
without ever opening it — then answer generically about facts the brain already
knew.
This reflex has two halves:
1. **Deterministic pointer layer (automatic, on by default).** The
`gbrain-context` engine scans each turn's user message for salient,
resolvable entities and injects a compact pointer (name → slug → one-line
summary) so the agent *knows the page exists*. Zero-LLM, fail-open. Nothing
to install — it's on unless `retrieval_reflex` is disabled in
`~/.gbrain/config.json` or `GBRAIN_RETRIEVAL_REFLEX=false`.
2. **Policy skill (this recipe installs it).** A SKILL fragment in the host
resolver that encodes the trigger policy and retrieval spec the agent
follows when a pointer appears or an entity becomes the subject.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Run these steps on behalf of the user.
1. Confirm the deterministic layer isn't disabled:
`gbrain doctor --json | jq '.checks[] | select(.name=="retrieval_reflex_health")'`
2. Install the policy skill into the host repo (the OpenClaw/agent repo that
holds `skills/RESOLVER.md` or `AGENTS.md`):
`gbrain integrations install retrieval-reflex --target <host-repo>`
3. Verify: re-run `gbrain doctor` and confirm `retrieval_reflex_health` is `ok`.
The deterministic layer needs no install. On a PGLite brain it resolves through
the running `gbrain serve` (or a host-provided capability); if neither is
available it stays disabled and this policy skill carries the behavior — the
doctor check reports which.
@@ -0,0 +1,16 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"recipe": "retrieval-reflex",
"version": "0.1.0",
"install_kind": "copy-into-host-repo",
"description": "src → target mapping consumed by `gbrain integrations install retrieval-reflex`. Policy-only recipe: ships one SKILL.md into the host resolver and appends a resolver row. The deterministic pointer layer lives in the gbrain context engine and needs no install.",
"target_root_relative_to_host_repo": "skills/retrieval-reflex",
"skills_target_root_relative_to_host_repo": "skills",
"files": [],
"skills": [
{ "src": "skills/retrieval-reflex/SKILL.md", "target": "skills/retrieval-reflex/SKILL.md", "mode": "0644" }
],
"resolver_rows_to_append": [
"retrieval-reflex | a named person/company/project/place becomes the subject; a brain-page pointer appears in context; \"who is\", \"what do we know about\", \"tell me about\"; about to assert a non-trivial detail about a named entity"
]
}
@@ -0,0 +1,59 @@
---
name: retrieval-reflex
version: 0.1.0
description: When/what to retrieve — open the brain page for a salient entity before answering from memory.
triggers:
- "who is"
- "what do we know about"
- "tell me about"
mutating: false
writes_pages: false
writes_to: []
tools: [get_page, query, graph, backlinks]
---
# Retrieval Reflex — retrieve on demand, when an entity is salient
A person doesn't bulk-load their whole address book into working memory. They
retrieve **on demand**, when an entity becomes **salient**, use it, and drop it.
Encode that reflex. The brain probably has the data — if a name is salient and
you haven't opened its page, open it before you answer.
## Trigger policy — WHEN to retrieve
Retrieve when ANY of these holds AND the page isn't already loaded in context:
- An entity (person / company / project / deal / place) is the **subject** of
the message, or a decision/judgment about it is being made, or the exchange is
substantive / relational / emotional about it.
- A **brain-page pointer** appeared in context this turn (the deterministic
layer told you the page exists) — open it before relying on details.
- A name or term appears that you **don't recognize** and that looks notable →
do a quick resolve (the human reflex).
- You're about to **assert a non-trivial detail** about an entity (attribution,
status, history) → verify against the brain first. Say "let me check", not a guess.
**Skip** trivial passing mentions, logistics pings, and anything already loaded.
Judgment first — retrieve when it changes the quality of the reply, not reflexively.
## Retrieval spec — WHAT to pull, and when to stop
Escalate only as far as the task needs:
1. **Pointer / metadata.** If a pointer is already in context (slug + one-line
summary), and the task only needs identity, stop there.
2. **Full page.** When the entity is the subject or details matter, open it:
`get_page <slug>` (MCP) — read the page before relying on specifics.
3. **Linked neighbors.** Only when relationship context is needed, pull
`graph` / `backlinks` for the slug.
**Resolve only the name(s) the current task needs, use them, drop them.** No
bulk-loading the inner circle.
## The failure this prevents
If you've discussed a named person for more than a message without opening their
page, open it now. The write side captures everything; the read side only helps
if you actually look.
See also: `skills/query/SKILL.md` (search the brain), `skills/brain-ops/SKILL.md`.
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env node
/**
* CI guard for the POSITIONAL jsonb double-encode footgun (#2339 / #2324 class).
*
* The legacy scripts/check-jsonb-pattern.sh only catches the template-tag form
* (`${JSON.stringify(x)}::jsonb`). It MISSES the positional-param form:
*
* engine.executeRaw(`... $3::jsonb ...`, [a, b, JSON.stringify(x)])
*
* Under postgres.js `.unsafe(sql, params)` a JS STRING bound to a `$N::jsonb`
* param double-encodes the textjsonb cast wraps the already-JSON string into a
* jsonb *string scalar*. PGLite parses it silently, so the bug is invisible in
* unit tests and only bites on real Postgres (it aborted every sync in #2339).
*
* This scanner flags any executeRaw / executeRawDirect / .unsafe(...) call whose
* balanced argument span contains BOTH a positional `$N::jsonb` cast
* (NOT `$N::text::jsonb`, NOT `$N::text[]`) AND a `JSON.stringify(` the exact
* double-encode shape. It is heuristic by design (whole-span correlation); the
* real backstop is the DATABASE_URL-gated e2e parity test. Keep both.
*
* Allowed forms (NOT flagged):
* - `$N::text::jsonb` + JSON.stringify (the fix: binds as text, cast parses it)
* - `$N::text[]` (the unnest path arrays bind fine)
* - executeRawJsonb(...) (passes raw objects, not strings)
* - sql.json(x) (postgres.js native jsonb serializer)
* - a `jsonb-guard-ok` comment anywhere in the call span (explicit opt-out)
*
* Exit 0 = clean, 1 = violations found. Runs under node or bun.
*/
import { readdirSync, readFileSync, statSync } from 'node:fs';
import { join } from 'node:path';
// Default scan roots; overridable via argv so the guard's own test can point it
// at a fixture dir (e.g. `node check-jsonb-params.mjs /tmp/fixtures`).
const ROOTS = process.argv.slice(2).length > 0 ? process.argv.slice(2) : ['src', 'scripts'];
// executeRawDirect must precede executeRaw in the alternation so the longer name
// wins; executeRawJsonb is deliberately excluded (it passes objects). The
// optional `<...>` handles generic type args, e.g. `executeRaw<{ id: string }>(`.
//
// Only the postgres.js raw path is scanned (executeRaw/executeRawDirect/.unsafe).
// PGLite's native `this.db.query(...)` is intentionally NOT matched: its driver
// parses a text→jsonb cast natively, so the double-encode that bites postgres.js
// `.unsafe()` does not occur there (the `pglite-masks` invariant). The engine
// parity test pins that the resulting jsonb_typeof agrees across both engines.
const CALL_RE = /\b(executeRawDirect|executeRaw|unsafe)\s*(?:<[^>;]*>)?\s*\(/g;
/** Walk from the '(' at openIdx and return [start,end) of the balanced span,
* respecting strings, template literals, and comments. */
function findSpan(src, openIdx) {
let depth = 0;
let mode = 'code'; // code | line | block | sq | dq | tpl
for (let i = openIdx; i < src.length; i++) {
const c = src[i];
const n = src[i + 1];
if (mode === 'line') { if (c === '\n') mode = 'code'; continue; }
if (mode === 'block') { if (c === '*' && n === '/') { mode = 'code'; i++; } continue; }
if (mode === 'sq') { if (c === '\\') { i++; continue; } if (c === "'") mode = 'code'; continue; }
if (mode === 'dq') { if (c === '\\') { i++; continue; } if (c === '"') mode = 'code'; continue; }
if (mode === 'tpl') { if (c === '\\') { i++; continue; } if (c === '`') mode = 'code'; continue; }
// mode === 'code'
if (c === '/' && n === '/') { mode = 'line'; i++; continue; }
if (c === '/' && n === '*') { mode = 'block'; i++; continue; }
if (c === "'") { mode = 'sq'; continue; }
if (c === '"') { mode = 'dq'; continue; }
if (c === '`') { mode = 'tpl'; continue; }
if (c === '(') depth++;
else if (c === ')') { depth--; if (depth === 0) return [openIdx + 1, i]; }
}
return [openIdx + 1, src.length];
}
/** Blank out comments so a commented-out example doesn't trip the JSON.stringify probe. */
function stripComments(s) {
return s.replace(/\/\/[^\n]*/g, '').replace(/\/\*[\s\S]*?\*\//g, '');
}
const violations = [];
function scanFile(file) {
const src = readFileSync(file, 'utf8');
CALL_RE.lastIndex = 0;
let m;
while ((m = CALL_RE.exec(src))) {
const method = m[1];
const openIdx = m.index + m[0].length - 1; // index of the '('
const [s, e] = findSpan(src, openIdx);
const span = src.slice(s, e);
if (/jsonb-guard-ok/.test(span)) continue;
if (!/JSON\.stringify\s*\(/.test(stripComments(span))) continue;
// A positional `$N::jsonb` that is NOT `$N::text::jsonb`.
const jsonbRe = /\$\d+\s*::\s*jsonb\b/g;
let j;
let badText = '';
while ((j = jsonbRe.exec(span))) {
const pre = span.slice(Math.max(0, j.index - 12), j.index);
if (/::\s*text\s*$/.test(pre)) continue; // $N::text::jsonb is the fix — allowed
badText = j[0].replace(/\s+/g, '');
break;
}
if (!badText) continue;
const line = src.slice(0, s).split('\n').length;
violations.push(
`${file}:${line} ${method}(...) binds JSON.stringify into ${badText} — use $N::text::jsonb or pass a raw object (executeRawJsonb / sql.json)`,
);
}
}
function walk(dir) {
let ents;
try { ents = readdirSync(dir); } catch { return; }
for (const ent of ents) {
if (ent === 'node_modules') continue;
const p = join(dir, ent);
const st = statSync(p);
if (st.isDirectory()) walk(p);
else if (p.endsWith('.ts') && !p.endsWith('.test.ts')) scanFile(p);
}
}
for (const root of ROOTS) walk(root);
if (violations.length) {
console.error('JSONB positional double-encode violations (#2339 class):\n');
for (const v of violations) console.error(' ' + v);
console.error(`\n${violations.length} violation(s). Fix: bind through $N::text::jsonb (keeping JSON.stringify), or pass a raw object via executeRawJsonb / sql.json. See docs/ENGINES.md.`);
process.exit(1);
}
console.log('check-jsonb-params: clean (no positional $N::jsonb + JSON.stringify double-encodes)');
+14
View File
@@ -44,3 +44,17 @@ if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/
fi
echo "OK: max_stalled defaults are 5 in all schema sources"
# v0.42.x (#2339 / #2324): positional `$N::jsonb` + JSON.stringify double-encode.
# The template-string grep above only catches `${JSON.stringify(x)}::jsonb`. It
# MISSES the positional-param form — executeRaw(`... $N::jsonb ...`,
# [JSON.stringify(x)]) — which is the exact shape that double-encoded the
# op_checkpoints pin and aborted every sync in #2339. The AST-lite scanner below
# catches it. `set -e` propagates its non-zero exit.
if command -v node >/dev/null 2>&1; then
node scripts/check-jsonb-params.mjs
elif command -v bun >/dev/null 2>&1; then
bun scripts/check-jsonb-params.mjs
else
echo "WARN: neither node nor bun on PATH; skipping check-jsonb-params.mjs" >&2
fi
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard (#1647 / #171): every trigger function in the canonical schema base
# files MUST pin `SET search_path`. Without it, an unqualified reference inside
# the function body resolves through the caller's search_path, so a same-named
# object in a user-controlled schema could shadow it. Migration v120 ALTERs
# existing brains; this guard keeps fresh-install function definitions correct
# so a NEW trigger function can't reintroduce the gap. Mirrors the
# check-jsonb-pattern.sh guard philosophy (a written rule caused the disease;
# a guard cures it).
#
# Scope: schema base files only (src/schema.sql, src/core/pglite-schema.ts).
# Historical migration bodies in migrate.ts are append-only and not rescanned;
# the runtime doctor probe (pg_proc.proconfig) covers the live post-migration
# state on real brains.
#
# Usage: scripts/check-search-path.sh
# Exit: 0 when all trigger functions pin search_path, 1 otherwise.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
FILES="src/schema.sql src/core/pglite-schema.ts src/core/schema-embedded.ts"
# A hardened header reads `... RETURNS trigger SET search_path = ... AS $tag$`.
# An UNHARDENED one reads `... RETURNS trigger AS $tag$` — match that form and
# (belt-and-suspenders) drop any line that already mentions search_path.
BAD="$(grep -nEi 'CREATE OR REPLACE FUNCTION [a-z_]+\(\) RETURNS trigger AS ' $FILES 2>/dev/null | grep -vi 'search_path' || true)"
if [ -n "$BAD" ]; then
echo "ERROR: trigger function(s) missing SET search_path in schema base files:"
echo "$BAD"
echo
echo "Add 'SET search_path = pg_catalog, public' to the function header, e.g.:"
echo " CREATE OR REPLACE FUNCTION foo() RETURNS trigger SET search_path = pg_catalog, public AS \$\$"
echo "See #1647 / #171."
exit 1
fi
echo "OK: all trigger functions in schema base files pin search_path"
+12 -2
View File
@@ -196,7 +196,10 @@ SELECTED=$(bun run scripts/select-e2e.ts)
if [ -z "$SELECTED" ]; then
echo "[runner] selector emitted nothing (doc-only diff); skipping E2E."
else
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
echo "$SELECTED" | xargs bash scripts/run-e2e.sh
fi'
else
RUN_PHASES_CMD='echo "[runner] guards + typecheck"
@@ -208,7 +211,10 @@ bun run typecheck
echo "[runner] unit (unsharded, DATABASE_URL unset)"
env -u DATABASE_URL bash scripts/run-unit-shard.sh
echo "[runner] e2e (unsharded)"
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh'
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
bash scripts/run-e2e.sh'
fi
else
# Tier 1 sharded path. Each shard runs unit+E2E sequentially against its
@@ -257,10 +263,14 @@ printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c '
if [ -s /tmp/e2e-selected.txt ]; then
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1
else
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
bash scripts/run-e2e.sh >> \$log 2>&1
fi
e2e_exit=\$?
+6
View File
@@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [
"Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.",
path: "docs/guides/scaling-skills.md",
},
{
title: "docs/guides/push-context.md",
description:
"Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.",
path: "docs/guides/push-context.md",
},
{
title: "docs/mcp/DEPLOY.md",
description: "MCP server deployment.",
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bun
// scripts/postinstall.ts
//
// Postinstall hook: after `bun install`, apply any pending schema migrations so
// a freshly-installed gbrain is immediately usable. Wired via package.json
// ("postinstall": "bun run scripts/postinstall.ts") as a real Bun script rather
// than an inline `node -e` one-liner.
//
// Why a script file and not an inline command:
// Embedding a program inside the package.json postinstall string lets the
// lifecycle shell mangle it. Bun's Windows script-runner expands `\n` in the
// hint string into a REAL newline before node sees it, producing
// `SyntaxError: Invalid or unexpected token` and aborting the whole install.
// `node` is also not guaranteed present under a Bun install (bun is the
// guaranteed runtime), and `shell: win32` re-opens a quoting surface. A
// checked-in .ts run by `bun run` sidesteps all three.
//
// Uses Bun APIs only — `which()` for Windows-aware PATH resolution (finds
// gbrain.exe / gbrain.cmd) and an argv-array `Bun.spawnSync` (no shell, nothing
// to quote). It NEVER fails the install: every path exits 0.
import { which } from 'bun';
const HINT =
'[gbrain] postinstall skipped. If installed via bun install -g github:...: ' +
'run `gbrain doctor` and `gbrain apply-migrations --yes` manually. ' +
'See https://github.com/garrytan/gbrain/issues/218';
// Windows-aware PATH resolution — finds gbrain, gbrain.exe or gbrain.cmd.
const bin = which('gbrain');
if (!bin) {
// Fresh clone / global install where gbrain isn't on PATH yet: skip cleanly.
console.error(HINT);
process.exit(0);
}
try {
const r = Bun.spawnSync({
cmd: [bin, 'apply-migrations', '--yes', '--non-interactive'],
stdout: 'inherit',
stderr: 'inherit',
});
if (r.exitCode !== 0) console.error(HINT);
} catch {
console.error(HINT);
}
process.exit(0); // never abort the install
+18
View File
@@ -66,6 +66,24 @@ export HOME="$E2E_TMP_HOME"
export GBRAIN_HOME="$E2E_TMP_HOME"
mkdir -p "$E2E_TMP_HOME/.gbrain"
# --- Hermetic env scrub: operator/agent context must not bleed into E2E ---
# A dev shell or a Conductor workspace exports CONDUCTOR_*, MCP_*, OPENCLAW_*,
# and GBRAIN_* config overrides (e.g. a stray GBRAIN_BRAIN_ID, GBRAIN_SOURCE,
# GBRAIN_*_THRESHOLD, GBRAIN_SUPERVISOR_PID_FILE) that would silently change
# test behavior — making "hermetic" E2E non-hermetic and its failures
# unreproducible across machines. Drop them before bun starts. This is a
# DENYLIST of operator-context prefixes (not an allowlist rebuild), so PATH,
# HOME, TMPDIR, CI, DATABASE_URL, and bun internals survive untouched. We keep
# GBRAIN_HOME (just set above for HOME isolation); everything else GBRAIN_* is
# an operator override the suite must not inherit. Adapts GStack's
# buildHermeticEnv() allowlist to gbrain's shell E2E runner.
for _e2e_var in $(env | grep -oE '^(CONDUCTOR_|MCP_|OPENCLAW_|GBRAIN_)[A-Za-z0-9_]*' | sort -u); do
case "$_e2e_var" in
GBRAIN_HOME) ;; # required for HOME isolation (set above) — keep
*) unset "$_e2e_var" || true ;;
esac
done
# --dry-run-list: print the resolved file list (one per line) and exit. Used
# by scripts/ci-local.sh to smoke-test the argv branching at startup.
DRY_RUN_LIST=0
+1
View File
@@ -38,6 +38,7 @@ CHECKS=(
"check:proposal-pii"
"check:test-names"
"check:jsonb"
"check:search-path"
"check:source-id-projection"
"check:source-config-leak"
"check:progress"
+1
View File
@@ -57,6 +57,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Morning prep, meeting context, day planning | `skills/daily-task-prep/SKILL.md` |
| Daily briefing, "what's happening today" | `skills/briefing/SKILL.md` |
| Cron scheduling, quiet hours, job staggering | `skills/cron-scheduler/SKILL.md` |
| "get more out of gbrain", "is my brain set up right", "weekly brain checkup", "advise me on my brain", "gbrain advisor" | `skills/gbrain-advisor/SKILL.md` |
| Save or load reports | `skills/reports/SKILL.md` |
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
+2 -1
View File
@@ -24,7 +24,7 @@ mutating: true
## Contract
This skill guarantees:
- Every brain page is scanned against the seven canonical frontmatter validation classes
- Every brain page is scanned against the eight canonical frontmatter validation classes
- Mechanical errors (nested quotes, missing closing `---`, null bytes, slug mismatch) are auto-repairable on demand with `.bak` backups
- Validation logic is shared with `gbrain doctor`'s `frontmatter_integrity` subcheck — single source of truth
- Reports per source (gbrain is multi-source since v0.18.0); never silently audits the wrong root
@@ -50,6 +50,7 @@ Without a guard, these accumulate silently until `gbrain sync` chokes or search
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes (removes the field) |
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes |
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes |
| `NON_STRING_FIELD` | `title`/`type`/`slug` is an unquoted non-string scalar (e.g. `title: 123`, `slug: 2024-06-01`) | No (quote the value) |
| `EMPTY_FRONTMATTER` | Open + close present but nothing between | No (needs human) |
## Phases
+117
View File
@@ -0,0 +1,117 @@
---
name: gbrain-advisor
version: 1.0.0
description: |
Proactive "make the most of gbrain" coaching. Runs `gbrain advisor` on a
cadence and pings the user with the top high-leverage actions for their brain:
version drift, pending migrations, stalled jobs, low embed coverage, setup
smells, and uninstalled brain skills. Read-only; always asks before fixing.
triggers:
- "what should I do to get more out of gbrain"
- "is my brain set up right"
- "gbrain advisor"
- "advise me on my brain"
- "weekly brain checkup"
tools:
- advisor
mutating: false
---
# gbrain Advisor
> **Convention:** See `skills/conventions/brain-first.md`. This skill is the
> proactive voice of the brain — it tells the owner how to run it better.
## Contract
This skill guarantees:
- **Read-only.** `gbrain advisor` never mutates. It computes a ranked list of
actions from existing brain state.
- **Print, never execute.** You SHOW the user the findings and ASK before running
any fix. The user owns every decision.
- **Bounded nagging.** On a cadence, surface only what changed or what's
critical; don't repeat an ignored low-severity item every run.
## When to run
- On demand when the user asks "how do I get more out of this brain?"
- On a **weekly** cadence via the cron recipe below (even idle brains get a
"here's how to run this better" ping).
## How to run it
```bash
gbrain advisor --json
```
Exit code is the severity gate (E2): `0` clean, `1` warn, `2` critical. The JSON
payload is `{ version, generated_at, worst, findings: [...] }`. Each finding has:
- `severity``critical` | `warn` | `info`
- `title` — one-line why-it-matters
- `fix.command_argv` — the exact command to fix it (a structured argv)
- `fix.dispatch_id` — present when the fix is safe to run via `--apply`
## What to do with the findings
1. Read the findings, highest severity first.
2. Summarize the top 1-3 to the user in their own channel/voice. Lead with any
`critical` item (e.g. pending migrations).
3. For each, show the `fix.command_argv` and **ask** whether to run it.
4. If they say yes and the finding has a `fix.dispatch_id`, you may run it
locally with an explicit confirm:
```bash
gbrain advisor --apply <dispatch_id>
```
`--apply` is local-only, runs the fix as a structured argv (no shell), and
confirms first. Findings without a `dispatch_id` are not auto-runnable — run
their `fix.command_argv` yourself after the user agrees.
5. Never run a fix the user didn't approve.
## Cron recipe (weekly checkup)
Install a weekly job via the `cron-scheduler` skill. Keep the prompt THIN — the
job just reads this skill and runs the advisor:
- **Schedule:** weekly, one quiet-hours-respecting slot (e.g. Monday 09:00 local).
- **Job prompt:** `Read skills/gbrain-advisor/SKILL.md and run gbrain advisor --json. If anything is critical or new since last run, ping me with the top items and the exact fix commands. Ask before fixing.`
- **Idempotent:** the advisor is read-only, so a double-fire is harmless.
The advisor records a local run history, so on each fire you can tell the user
what is **new since last run** rather than re-listing everything.
## Output Format
When you surface advisor findings to the user, lead with severity and keep it
scannable:
```
🧠 gbrain checkup — 2 things worth your attention
CRITICAL Schema migrations are pending.
Fix: gbrain apply-migrations --yes (want me to run it?)
WARN gbrain 0.44 is available (you're on 0.43).
Fix: gbrain upgrade
```
- One block per finding, highest severity first.
- Always show the exact `fix` command and ASK before running it.
- If nothing is pressing, say so in one line ("brain looks healthy") — don't
manufacture work.
## Anti-Patterns
- **Running a fix without asking.** The advisor is read-only by contract. Never
run `--apply` (or any `fix` command) without the user's explicit yes.
- **Dumping the raw JSON at the user.** Translate findings into their voice; lead
with what matters.
- **Re-nagging ignored low-severity items every run.** Use the "new since last
run" delta; respect the user's prior non-action.
- **Treating `info` like `critical`.** Only block/insist on `critical` findings
(pending migrations). `info` is a gentle nudge.
- **Calling the MCP `advisor` op for workspace install state.** Over MCP the
advisor returns brain-state signals only; uninstalled-skill findings are a
local-CLI concern.
+5
View File
@@ -239,6 +239,11 @@
"path": "functional-area-resolver/SKILL.md",
"description": "Compress an agent's routing file (RESOLVER.md or AGENTS.md) by replacing skill-per-row tables with functional-area dispatcher entries. Two-layer dispatch keeps every sub-skill reachable at ~50% of the file size."
},
{
"name": "gbrain-advisor",
"path": "gbrain-advisor/SKILL.md",
"description": "Proactive 'make the most of gbrain' coaching. Runs gbrain advisor on a cadence and pings the user with the top high-leverage actions for their brain. Read-only; always asks before fixing."
},
{
"name": "brain-taxonomist",
"path": "brain-taxonomist/SKILL.md",
+3 -3
View File
@@ -1,13 +1,13 @@
// AUTO-GENERATED — do not edit by hand.
// Run `bun run scripts/build-admin-embedded.ts` to regenerate.
// Source: admin/dist/ at 2026-05-24.
// Source: admin/dist/ at 2026-05-27.
//
// Bun resolves the file: imports to a path that works at runtime even
// inside a compiled binary (`bun build --compile`). The manifest maps
// the request path the express handler sees to (resolved-path, mime).
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
import A_0_assets_index_DqP_zmqH_js from '../admin/dist/assets/index-DqP-zmqH.js' with { type: 'file' };
import A_0_assets_index_CoGEje3__js from '../admin/dist/assets/index-CoGEje3-.js' with { type: 'file' };
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
import A_1_assets_index_GxkWX7v3_css from '../admin/dist/assets/index-GxkWX7v3.css' with { type: 'file' };
// @ts-ignore — type: 'file' is Bun ESM, not in lib.d.ts
@@ -19,7 +19,7 @@ export interface AdminAsset {
}
export const ADMIN_ASSETS: Record<string, AdminAsset> = {
"/admin/assets/index-DqP-zmqH.js": { path: A_0_assets_index_DqP_zmqH_js as unknown as string, mime: "application/javascript; charset=utf-8" },
"/admin/assets/index-CoGEje3-.js": { path: A_0_assets_index_CoGEje3__js as unknown as string, mime: "application/javascript; charset=utf-8" },
"/admin/assets/index-GxkWX7v3.css": { path: A_1_assets_index_GxkWX7v3_css as unknown as string, mime: "text/css; charset=utf-8" },
"/admin/index.html": { path: A_2_index_html as unknown as string, mime: "text/html; charset=utf-8" },
};
+290 -101
View File
@@ -24,9 +24,9 @@ import type { GBrainConfig } from './core/config.ts';
import type { AIGatewayConfig } from './core/ai/types.ts';
import type { BrainEngine } from './core/engine.ts';
import { operations, OperationError } from './core/operations.ts';
import { formatVolunteeredPage } from './core/context/volunteer.ts';
import type { Operation, OperationContext } from './core/operations.ts';
import { drainAllBackgroundWorkForCliExit } from './core/background-work.ts';
import { shouldForceExitAfterMain } from './core/cli-force-exit.ts';
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
import { serializeMarkdown } from './core/markdown.ts';
import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts';
import type { CliOptions } from './core/cli-options.ts';
@@ -43,8 +43,18 @@ for (const op of operations) {
}
}
/**
* JSON replacer: `bigint` string, matching the postgres.js wire shape (int8
* comes back as a string on the routed path). Lets the local-engine output
* normalizer round-trip bigint columns (e.g. a `BIGSERIAL` `id`) instead of
* throwing `TypeError: Do not know how to serialize a BigInt`.
*/
export function bigintToStringReplacer(_key: string, value: unknown): unknown {
return typeof value === 'bigint' ? value.toString() : value;
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -68,6 +78,8 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
// unreachable via help because the dispatcher's generic CLI-only
// short-circuit fired before runSync could print its own usage block.
@@ -233,6 +245,17 @@ async function main() {
command = 'query';
}
// Local patch 2026-06-11 — mark one-shot CLI processes so the facts
// backstop routes absorb work to the durable jobs worker instead of the
// in-process queue that the exit teardown drains-then-aborts after ~1-2s
// (the `pipeline_error: [chat(...)] The operation was aborted.` class in
// ingest_log). Daemons keep the in-process queue: their event loop
// outlives the work. See src/core/facts/cli-process-mode.ts.
if (!['serve', 'jobs', 'autopilot'].includes(command)) {
const { markShortLivedCliProcess } = await import('./core/facts/cli-process-mode.ts');
markShortLivedCliProcess();
}
// T5 — `gbrain search modes|stats|tune` is the read-only config dashboard,
// NOT a free-text search for the literal word "modes". Free-text
// `gbrain search "<query>"` falls through to the cheap-hybrid `search` op
@@ -260,7 +283,11 @@ async function main() {
await withTimeout(runSearch(engine, subArgs), timeoutMs, label);
}
} finally {
await engine.disconnect();
// #2084: `search diagnose` runs real hybrid retrieval (arms search-cache
// writes) — route through the shared bounded teardown like every other
// one-shot path. The connect-timeout process.exit(124) above is reviewed
// and intentionally unchanged: no engine exists at that point.
await finishCliTeardown({ engine });
}
return;
}
@@ -278,6 +305,17 @@ async function main() {
}
}
// DB-free durability pull (v0.42.44 D2): the harden cron calls
// `gbrain sources pull --path <dir>` every ~30 min. It must NOT open PGLite
// (a live long-lived session holds the single-writer lock), so handle it
// BEFORE connectEngine. The `sources pull <id>` form (no --path) still routes
// through handleCliOnly → runSources with an engine.
if (command === 'sources' && subArgs[0] === 'pull' && subArgs.includes('--path')) {
const { runPull } = await import('./commands/sources-harden.ts');
await runPull(null, subArgs.slice(1));
return;
}
// CLI-only commands
if (CLI_ONLY.has(command)) {
await handleCliOnly(command, subArgs);
@@ -350,45 +388,82 @@ async function main() {
// Local engine path (unchanged behavior for local installs).
const engine = await connectEngine();
// v0.41.8.0 (#1247, #1269, #1290): the search / query / get_page
// op handlers fire-and-forget `bumpLastRetrievedAt` after returning
// results. On PGLite that IIFE keeps Bun's event loop alive past
// engine.disconnect(), hanging the CLI at ~95-98% CPU until SIGKILL.
// Drain the fire-and-forget set BEFORE disconnect; force-exit only
// if the drain itself times out (preserves stderr diagnostic signal
// AND guarantees the CLI doesn't re-hang at the disconnect layer).
//
// Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself
// can hang on PGLite (db.close() or releaseLock racing OS-level FS state).
// Install an unref'd setTimeout hard-exit fallback BEFORE entering the
// try/catch/finally so a hung disconnect cannot defeat the force-exit
// contract. Daemons (`serve`) are excluded so they stay alive.
const DISCONNECT_HARD_DEADLINE_MS = 10_000;
let forceExitTimer: ReturnType<typeof setTimeout> | undefined;
if (shouldForceExitAfterMain()) {
forceExitTimer = setTimeout(() => {
console.warn(
`[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`,
);
// v0.42.20.0 (codex): honor an exit code an errored op already set —
// a bare process.exit(0) here would mask a failed op as success if the
// drain/disconnect then hangs.
process.exit(process.exitCode ?? 0);
}, DISCONNECT_HARD_DEADLINE_MS);
// unref so the timer itself doesn't keep the event loop alive — only
// the actual pending work (PGLite WASM handle) does. Without unref,
// we'd block a clean exit by 10s on every successful CLI run.
forceExitTimer.unref?.();
}
// #2084: the teardown contract (bounded drain of every background-work sink,
// bounded disconnect, computed-deadline backstop) lives in finishCliTeardown
// — see src/core/cli-force-exit.ts for the full design. The hard-deadline
// timer arms at TEARDOWN start inside the helper, never before the handler:
// the pre-#2084 placement here measured handler + teardown combined, so a
// slow-but-healthy query burned the teardown budget (the flat-10s-banner
// bug) and any >10s op was force-killed mid-run with exit 0. The explicit
// process exit happens once, in the import.meta.main seam at the bottom of
// this file — NOT here.
// v0.42.41.0 (merged): wallclock bound for READ-scope op handlers. With the
// teardown backstop correctly scoped to teardown, a genuinely WEDGED read
// handler (hung pooler connection mid-query) would otherwise hang the CLI
// forever — the #1633 zombie class the old pre-try timer accidentally
// bounded at 10s. 180s sits far above any healthy slow-pooler run
// (6-10s/connection); --timeout=Ns overrides. Writes/admin stay unbounded:
// a long import/embed must never be killed by a default deadline. On
// timeout the abandoned handler may hold ref'd sockets — harmless here,
// because the import.meta.main seam exits explicitly on every one-shot path.
const READ_OP_TIMEOUT_MS = 180_000;
try {
const ctx = await makeContext(engine, params);
const rawResult = await op.handler(ctx, params);
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
const wallclockMs = getCliOptions().timeoutMs ?? READ_OP_TIMEOUT_MS;
const onWallclockTimeout = (e: InstanceType<typeof OperationTimeoutError>) => {
const hint = getCliOptions().timeoutMs
? ''
: ` (default ${e.ms}ms; pass --timeout=Ns to override)`;
console.error(`${e.label} timed out${hint}.`);
// 124 = timeout convention (matches the read-only dispatch path). Set
// through the verdict channel — a raw process.exitCode write is invisible
// to the exit seam and PGLite's WASM runtime can scribble over it.
setCliExitVerdict(124);
};
// Context build does DB I/O (resolveSourceId) and runs for EVERY op —
// a wedged pooler connection here would otherwise hang reads, writes,
// and admin alike with no bound at all (adversarial review finding).
let ctx: Awaited<ReturnType<typeof makeContext>>;
try {
ctx = await withTimeout(
makeContext(engine, params),
wallclockMs,
`gbrain ${command}: context`,
);
} catch (e: unknown) {
if (e instanceof OperationTimeoutError) {
onWallclockTimeout(e);
return; // the finally drains + disconnects; the import.meta.main seam exits
}
throw e;
}
let rawResult: unknown;
if (op.scope === 'read') {
try {
rawResult = await withTimeout(
op.handler(ctx, params),
wallclockMs,
`gbrain ${command}`,
);
} catch (e: unknown) {
if (e instanceof OperationTimeoutError) {
onWallclockTimeout(e);
return; // the finally drains + disconnects; the import.meta.main seam exits
}
throw e;
}
} else {
rawResult = await op.handler(ctx, params);
}
// ENG-2 (renderer parity by data shape): JSON-round-trip the local-engine
// path's return value so renderers see the same shape they'd see on the
// routed path. Date → ISO string; bigint → string (postgres.js shape);
// Buffer → object. Microsecond-cost; eliminates a whole drift bug class.
const result = JSON.parse(JSON.stringify(rawResult));
const result = JSON.parse(JSON.stringify(rawResult, bigintToStringReplacer));
const output = formatResult(op.name, result);
if (output) process.stdout.write(output);
} catch (e: unknown) {
@@ -396,27 +471,20 @@ async function main() {
// STILL runs (drains every background-work sink + disconnects). A bare
// process.exit(1) here would skip the finally → skip the drain + disconnect
// (leaves facts/cache/eval-capture writes racing teardown). The finally's
// drain bounds teardown; the outer hard-deadline timer bounds a hung one.
// drain bounds teardown; the hard-deadline timer armed at teardown entry
// bounds a hung one.
if (e instanceof OperationError) {
console.error(`Error [${e.code}]: ${e.message}`);
if (e.suggestion) console.error(` Fix: ${e.suggestion}`);
} else {
console.error(e instanceof Error ? e.message : String(e));
}
process.exitCode = 1;
setCliExitVerdict(1);
} finally {
// v0.42.20.0 — drain ALL fire-and-forget sinks (facts, last-retrieved,
// search-cache, eval-capture) via the background-work registry BEFORE
// disconnect, so a PGLite db.close() can't race in-flight work into the
// re-pump busy-loop (#1762). facts drains first (order 0) so its abort-path
// DB logIngest gets the freshest live-engine window. 1s per-sink timeout:
// read paths with no pending work pay the ~0ms fast path; capture/import
// that DO enqueue pay up to 1s (+ facts shutdown grace) while in-flight
// Haiku finishes. The unref'd hard-deadline timer above is the backstop if
// disconnect or a lingering socket keeps Bun's loop alive.
await drainAllBackgroundWorkForCliExit({ timeoutMs: 1000 });
await engine.disconnect();
if (forceExitTimer) clearTimeout(forceExitTimer);
// 1s per-sink drain budget: read paths with no pending work pay the ~0ms
// fast path; capture/import that DO enqueue pay up to 1s (+ facts shutdown
// grace) while in-flight Haiku finishes (#1762 drain-before-disconnect).
await finishCliTeardown({ engine, drainTimeoutMs: 1000 });
}
}
@@ -769,8 +837,27 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
};
}
function formatResult(opName: string, result: unknown): string {
// Exported for tests (same import-safety contract as cliAliases/printOpHelp).
export function formatResult(opName: string, result: unknown): string {
switch (opName) {
case 'volunteer_context': {
const r = result as any;
// Stats mode (the feedback loop).
if (r && r.approximate === true && Array.isArray(r.by_arm)) {
const lines = [
`volunteered-context precision — last ${r.days} day(s) (${r.note})`,
`total: ${r.total_volunteered} volunteered, ${r.total_used} used`,
];
for (const a of r.by_arm) {
lines.push(` ${a.match_arm}/${a.channel}: ${a.used}/${a.volunteered} used (precision ${a.precision})`);
}
if (!r.by_arm.length) lines.push(' (no volunteer events in the window)');
return lines.join('\n') + '\n';
}
const pages = (r?.pages ?? []) as any[];
if (!pages.length) return 'Nothing volunteered (no entity cleared the confidence gate).\n';
return pages.map((p) => formatVolunteeredPage(p)).join('\n') + '\n';
}
case 'get_page': {
const r = result as any;
if (r.error === 'ambiguous_slug') {
@@ -892,6 +979,9 @@ function formatResult(opName: string, result: unknown): string {
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
'repair-jsonb', 'orphans', 'integrity', 'serve',
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
// the volunteer_context MCP op instead.
'watch',
// v0.31.1 (CDX-2 op coverage matrix): more local-only commands
'dream', 'transcripts', 'storage',
// v0.31.1 CDX-2 audit: takes/sources have multiple subcommands; some
@@ -908,6 +998,13 @@ const THIN_CLIENT_REFUSED_COMMANDS = new Set([
// - `code-def`/`code-refs`/`code-callers`/`code-callees` have NO MCP ops
// in operations.ts:2630-2671; cannot be "fixed by routing" yet
'pages', 'files', 'eval', 'code-def', 'code-refs', 'code-callers', 'code-callees',
// scratch-DB audit: `config` get/set operate on the host brain's config
// plane (DB rows / host file-plane). On a thin client they fabricated an
// ephemeral local PGLite (full migration replay per call) and read/wrote
// config nobody would ever see. NOTE: `jobs` is deliberately NOT here —
// it gets a partial dispatch (list/get route over MCP engine-free, the
// rest refuse) in the main dispatch before connectEngine().
'config',
]);
/**
@@ -945,6 +1042,9 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
'code-refs': '`code-refs` has no MCP op yet. Run on the host.',
'code-callers': '`code-callers` has no MCP op yet. Run on the host.',
'code-callees': '`code-callees` has no MCP op yet. Run on the host.',
// scratch-DB audit additions
config: "config reads/writes the host brain's config plane. Edit the host's .gbrain/config.json (file-plane keys) or run on the host with GBRAIN_HOME set.",
jobs: '`jobs list` and `jobs get <id>` are thin-client routable; this subcommand runs against the host queue. Use the submit_job / list_jobs / get_job MCP tools from your agent, or run on the host with GBRAIN_HOME set.',
};
/**
@@ -1123,11 +1223,14 @@ async function handleCliOnly(command: string, args: string[]) {
}
if (command === 'friction') {
const { runFriction } = await import('./commands/friction.ts');
process.exit(runFriction(args));
// #2084 inner-exit sweep: verdict + return so teardown + the flush seam run.
setCliExitVerdict(runFriction(args));
return;
}
if (command === 'claw-test') {
const { runClawTest } = await import('./commands/claw-test.ts');
process.exit(await runClawTest(args));
setCliExitVerdict(await runClawTest(args));
return;
}
if (command === 'report') {
const { runReport } = await import('./commands/report.ts');
@@ -1173,13 +1276,13 @@ async function handleCliOnly(command: string, args: string[]) {
if (args.includes('--remediation-plan')) {
const { runRemediationPlan } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediationPlan(eng, args); } finally { await eng.disconnect(); }
try { await runRemediationPlan(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
return;
}
if (args.includes('--remediate')) {
const { runRemediate } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediate(eng, args); } finally { await eng.disconnect(); }
try { await runRemediate(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
return;
}
@@ -1192,13 +1295,21 @@ async function handleCliOnly(command: string, args: string[]) {
// "user chose --fast while config is present".
await runDoctor(null, args, getDbUrlSource());
} else {
// #2084: both failure kinds (connect throw, runDoctor(eng) throw) still
// fall back to filesystem-only checks — identical to the prior shape.
// The finally closes the gap where a runDoctor(eng) throw used to skip
// the in-try disconnect. NOTE: runDoctor normally calls process.exit
// itself, which preempts this finally — in-command exit sites bypassing
// teardown are a pre-existing class, tracked as a TODOS.md follow-up.
let eng: BrainEngine | null = null;
try {
const eng = await connectEngine();
eng = await connectEngine();
await runDoctor(eng, args);
await eng.disconnect();
} catch {
// DB unavailable — still run filesystem checks
await runDoctor(null, args, getDbUrlSource());
} finally {
if (eng) await finishCliTeardown({ engine: eng });
}
}
return;
@@ -1212,7 +1323,7 @@ async function handleCliOnly(command: string, args: string[]) {
try {
await runZeSwitch(args, eng);
} finally {
await eng.disconnect();
await finishCliTeardown({ engine: eng });
}
return;
}
@@ -1228,7 +1339,7 @@ async function handleCliOnly(command: string, args: string[]) {
execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } });
} catch (e: any) {
// Non-zero exit = some tests failed (exit code = failure count)
process.exit(e.status ?? 1);
setCliExitVerdict(e.status ?? 1);
}
return;
}
@@ -1257,12 +1368,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runDream(eng, args);
} finally {
// #1471 invariant tripwire (the dream-cycle owner): `eng` created the
// module singleton (first module connector) and is disconnected LAST,
// module singleton (first module connector) and is torn down LAST,
// here, after the whole cycle. The ownership fix relies on this owner's
// lifetime strictly dominating every borrower (lint/doctor probe engines
// created mid-cycle). Do NOT disconnect `eng` before runDream returns, or
// created mid-cycle). Do NOT tear down `eng` before runDream returns, or
// a borrower could outlive the owner and lose the shared singleton.
if (eng) await eng.disconnect();
// #2084: routed through the shared bounded teardown — dream runs as an
// overnight cron, where a lingering-socket hang is a silent zombie
// (closes the TODOS.md drain-before-owner-disconnect item).
if (eng) await finishCliTeardown({ engine: eng });
}
return;
}
@@ -1274,7 +1388,8 @@ async function handleCliOnly(command: string, args: string[]) {
// The handler self-configures the AI gateway from loadConfig() + process.env.
if (command === 'eval' && args[0] === 'cross-modal') {
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
process.exit(await runEvalCrossModal(args.slice(1)));
setCliExitVerdict(await runEvalCrossModal(args.slice(1)));
return;
}
// v0.32 EXP-5 (codex review #10): `eval takes-quality replay <receipt>`
@@ -1285,7 +1400,8 @@ async function handleCliOnly(command: string, args: string[]) {
// engine-required path below.
if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') {
const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts');
process.exit(await runReplayNoBrain(args.slice(2)));
setCliExitVerdict(await runReplayNoBrain(args.slice(2)));
return;
}
// v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing
@@ -1311,13 +1427,22 @@ async function handleCliOnly(command: string, args: string[]) {
return;
}
// v0.42.x (#2390): `gbrain eval chronicle` is deterministic — brings its own
// in-memory PGLite, no DB/gateway. CI fixture gate runs anywhere.
if (command === 'eval' && args[0] === 'chronicle') {
const { runEvalChronicle } = await import('./commands/eval-chronicle.ts');
setCliExitVerdict(await runEvalChronicle(args.slice(1)));
return;
}
// v0.41.13.0: `gbrain eval conversation-parser` is pure-function
// (parses fixture JSONL, runs parseConversation, scores results).
// No DB access; bypass connectEngine entirely so the CI fixture
// gate runs on machines with no `~/.gbrain/config.json`.
if (command === 'eval' && args[0] === 'conversation-parser') {
const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts');
process.exit(await runEvalConversationParser(args.slice(1)));
setCliExitVerdict(await runEvalConversationParser(args.slice(1)));
return;
}
// v0.41.13.0: `gbrain conversation-parser list-builtins | validate
@@ -1345,7 +1470,8 @@ async function handleCliOnly(command: string, args: string[]) {
const cfgPre = loadConfig();
if (isThinClient(cfgPre)) {
const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts');
process.exit(await runEvalWhoknows(null, args.slice(1)));
setCliExitVerdict(await runEvalWhoknows(null, args.slice(1)));
return;
}
}
@@ -1359,7 +1485,8 @@ async function handleCliOnly(command: string, args: string[]) {
if (cfgPre && isThinClient(cfgPre)) {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(null, args);
process.exit(result.exitCode);
setCliExitVerdict(result.exitCode);
return;
}
}
@@ -1438,7 +1565,7 @@ async function handleCliOnly(command: string, args: string[]) {
}
throw e;
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
await finishCliTeardown({ engine });
}
return;
}
@@ -1476,6 +1603,27 @@ async function handleCliOnly(command: string, args: string[]) {
}
}
// Thin-client `jobs` dispatch: `list` and `get` route over MCP (v0.32
// routing branches in commands/jobs.ts) and never touch a local engine —
// but falling through to connectEngine() below fabricates an empty
// scratch PGLite in the thin-client GBRAIN_HOME and replays the entire
// migration chain on every invocation before the remote call even runs.
// Dispatch them engine-free here; every other jobs subcommand is
// host-queue-bound, so refuse with a pinpoint hint instead of building
// the scratch store.
if (command === 'jobs') {
const cfgJobs = loadConfig();
if (isThinClient(cfgJobs)) {
const jobsSub = args[0];
if (jobsSub === 'list' || jobsSub === 'get') {
const { runJobs } = await import('./commands/jobs.ts');
await runJobs(null, args);
return;
}
refuseThinClient('jobs', cfgJobs!.remote_mcp!.mcp_url);
}
}
// All remaining CLI-only commands need a DB connection
const engine = await connectEngine();
try {
@@ -1490,7 +1638,7 @@ async function handleCliOnly(command: string, args: string[]) {
// so wrappers (sync, CI scripts, `&& gbrain doctor`) propagate.
const importResult = await runImport(engine, args);
if (importResult.errors > 0) {
process.exitCode = 1;
setCliExitVerdict(1);
}
break;
}
@@ -1672,6 +1820,15 @@ async function handleCliOnly(command: string, args: string[]) {
case 'status': {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(engine, args);
// #2084 inner-exit sweep: a mid-switch exit skips the finally teardown.
setCliExitVerdict(result.exitCode);
break;
}
// v0.43 (#2180) — `gbrain advisor`: ranked, read-only "what to do next".
// CLI surface; the same signals are exposed over MCP via the `advisor` op.
case 'advisor': {
const { runAdvisorCli } = await import('./commands/advisor.ts');
const result = await runAdvisorCli(engine, args);
process.exit(result.exitCode);
// eslint-disable-next-line no-unreachable
break;
@@ -1843,6 +2000,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runQuarantine(engine, args);
break;
}
case 'watch': {
// v0.43 (#2095): push-based context transport. Blocks in the stdin
// iteration (interactive stays alive; piped exits at EOF), then the
// finally below runs finishCliTeardown (volunteer events drain with
// every other sink) and the import.meta.main seam flush-exits.
const { runWatch } = await import('./commands/watch.ts');
await runWatch(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
@@ -1866,6 +2032,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runReindexCodeCli(engine, args);
break;
}
case 'reindex-search-vector': {
// Explicit recreate of FTS trigger functions + batched backfill,
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
// env var on a brain that already ran the configurable_fts_language
// migration.
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
await runReindexSearchVectorCli(engine, args);
break;
}
case 'reindex-frontmatter': {
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
// Mirror of reindex-code shape. Wraps the shared library function in
@@ -1913,31 +2088,16 @@ async function handleCliOnly(command: string, args: string[]) {
}
} finally {
syncWatchdog?.dispose(); // #1633: tear down the hard-deadline watchdog on clean exit
// v0.42.20.0 (#1762) — the CLI_ONLY path (which owns `gbrain capture`)
// lacked the op-dispatch drain-before-disconnect contract. `put_page` fires
// a fire-and-forget facts:absorb job AFTER printing the receipt; on a
// multi-chunk page that job is in flight when this finally tears the engine
// down, and `engine.disconnect()` nulling PGLite's _db mid-job spins
// db.close() into a 100%-CPU busy-loop that pins the single-writer lock.
// Drain every background-work sink first (facts shutdown() abort cancels a
// hung Haiku), THEN disconnect. The drain-before-disconnect is the causal
// fix; the force-exit defense below is secondary (it CANNOT preempt a WASM
// busy-loop on a pinned JS thread — that's exactly why the drain matters).
// #1471: this is also the fall-through OWNER-disconnect — the owner is torn
// down LAST (after the drain), so module-singleton borrowers never outlive it.
// #2084 — the CLI_ONLY fall-through teardown (drain every background-work
// sink, THEN disconnect, under a computed-deadline backstop) lives in
// finishCliTeardown. `gbrain capture`'s fire-and-forget facts:absorb job
// gets its drain window before PGLite's db.close() can race it into the
// re-pump busy-loop (#1762). #1471: this is also the fall-through
// OWNER-disconnect — the owner is torn down LAST (after the drain), so
// module-singleton borrowers never outlive it. `serve` skips teardown
// entirely: the daemon owns its lifecycle.
if (command !== 'serve') {
const forceExit = shouldForceExitAfterMain();
let hardExitTimer: ReturnType<typeof setTimeout> | undefined;
if (forceExit) {
hardExitTimer = setTimeout(() => {
console.warn('[cli] engine.disconnect() did not return within 10000ms — force-exiting');
process.exit(process.exitCode ?? 0);
}, 10_000);
hardExitTimer.unref?.();
}
await drainAllBackgroundWorkForCliExit();
await engine.disconnect();
if (hardExitTimer) clearTimeout(hardExitTimer);
await finishCliTeardown({ engine });
}
}
}
@@ -2129,7 +2289,7 @@ IMPORT/EXPORT
import <dir> [--no-embed] Import markdown directory
sync [--repo <path>] [flags] Git-to-brain incremental sync
sync --watch [--interval N] Continuous sync (loops until stopped)
sync --install-cron Install persistent sync daemon
See also: autopilot --install (continuous daemon).
export [--dir ./out/] Export to markdown
export --restore-only [--repo <p>] Restore missing supabase-only files
[--type T] [--slug-prefix S] With optional filters
@@ -2195,7 +2355,14 @@ BRAIN (capture / ideate / explore — v0.37/v0.38)
SOURCES (multi-repo / multi-brain)
sources list Show registered sources
sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki')
sources remove <id> Remove a source + its pages
sources remove <id> Remove a source + its pages (--confirm-destructive)
sources archive <id> Soft-delete: hide from search, recoverable for 72h
sources restore <id> Un-archive a soft-deleted source
sources archived List soft-deleted sources and their purge expiry
sources purge [<id>] Permanently delete archived sources
sources status Per-source dashboard (sync lag, embed coverage)
sources --help Full subcommand list (rename, default, attach,
current, federate, set-cr-mode, webhook, harden, ...)
sync --all Sync all sources with a local_path
sync --source <id> Sync one specific source
repos ... DEPRECATED alias for 'sources' (v0.19.0)
@@ -2209,6 +2376,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
reconcile-links [--dry-run] Batch-recompute docimpl edges (v0.20.0)
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
reindex-search-vector [--dry-run] [--yes] [--json]
Recreate FTS triggers + backfill under
$GBRAIN_FTS_LANGUAGE (default 'english')
sync --strategy code Sync code files into the brain
JOBS (Minions)
@@ -2234,10 +2404,13 @@ ADMIN
serve MCP server (stdio)
serve --http [--port N] HTTP MCP server with OAuth 2.1
--token-ttl N Access token TTL in seconds (default: 3600)
--enable-dcr Enable Dynamic Client Registration
--enable-dcr Enable Dynamic Client Registration (DCR clients default to authorization_code)
--enable-dcr-insecure Also allow the consent-bypassing client_credentials grant on DCR (implies --enable-dcr)
--public-url URL Public issuer URL (required behind proxy/tunnel)
connect <mcp-url> --token <t> Wire Claude Code to a remote gbrain (bearer token)
[--install] [--json] Print the paste-ready command, or --install to run it
watch [--json] Push-based context: pipe conversation turns in,
volunteered brain pages stream out (#2095)
call <tool> '<json>' Raw tool invocation
version Version info
--tools-json Tool discovery (JSON)
@@ -2249,9 +2422,25 @@ Run gbrain <command> --help for command-specific help.
// Only auto-run when invoked as the entry point (the compiled binary or
// `bun src/cli.ts`). Guarded so tests can import cliAliases / printOpHelp
// without triggering argv parsing + main(). v114 (#1941).
//
// #2084 — the ONE process-exit seam for one-shot commands. Every teardown site
// routes through finishCliTeardown (which returns); the exit itself happens
// here, after main() settles, so the CLI never waits on Bun's event loop to
// drain (stuck PgBouncer sockets kept it alive — endPoolBounded races PAST a
// stuck pool.end() by design). flushThenExit fences stdout/stderr and holds a
// short aliveness grace so piped output is delivered before exit (#1959).
// Daemons (`serve`) are excluded by shouldForceExitAfterMain and keep the
// pre-#2084 behavior: main() resolves and the server's own work keeps the
// process alive. A fatal error still exits 1 for every command, daemons
// included (matches the prior unconditional process.exit(1) on rejection).
if (import.meta.main) {
main().catch(e => {
console.error(e.message || e);
process.exit(1);
});
main().then(
() => {
if (shouldForceExitAfterMain()) flushThenExit(currentExitCode());
},
(e) => {
console.error(e.message || e);
flushThenExit(1);
},
);
}
+140
View File
@@ -0,0 +1,140 @@
/**
* commands/advisor.ts `gbrain advisor` CLI surface.
*
* gbrain advisor # ranked, agent-readable action list (human render)
* gbrain advisor --json # structured findings; exit non-zero on critical (E2)
* gbrain advisor --apply ID # run ONE finding's fix, local-only, after confirm (E5)
*
* The advisor itself never mutates. `--apply` is the only path that runs a fix,
* and it: refuses over MCP (CLI is always local), only acts on allowlisted
* findings (those carrying a dispatch_id), executes the fix as STRUCTURED ARGV
* via a child process (never a shell no injection), and confirms first.
*/
import { spawnSync } from 'child_process';
import { createInterface } from 'readline';
import { resolve as resolvePath } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { VERSION } from '../version.ts';
import { loadConfig } from '../core/config.ts';
import { autoDetectSkillsDir } from '../core/repo-root.ts';
import { runAdvisor } from '../core/advisor/run.ts';
import { renderAdvisorReport } from '../core/advisor/render.ts';
import { appendAdvisorRun, summarizeDeltas } from '../core/advisor/history.ts';
import { resolveApplyTarget } from '../core/advisor/apply.ts';
import type { AdvisorContext, AdvisorReport } from '../core/advisor/types.ts';
export interface AdvisorCliResult {
exitCode: 0 | 1 | 2;
}
function buildContext(engine: BrainEngine): AdvisorContext {
const det = autoDetectSkillsDir();
const skillsDir = det.dir;
const workspace = skillsDir ? resolvePath(skillsDir, '..') : null;
return {
engine,
config: loadConfig() ?? ({} as AdvisorContext['config']),
version: VERSION,
workspace,
skillsDir,
now: new Date(),
remote: false, // CLI is always the trusted local owner
};
}
/** Exit-code contract (E2): 0 clean / 1 warn / 2 critical. */
function exitFor(report: AdvisorReport): 0 | 1 | 2 {
if (report.worst === 'critical') return 2;
if (report.worst === 'warn') return 1;
return 0;
}
export async function runAdvisorCli(engine: BrainEngine, args: string[]): Promise<AdvisorCliResult> {
if (args.includes('--help') || args.includes('-h')) {
console.log(
'gbrain advisor [--json] [--apply <finding-id>]\n\n' +
' (no flags) Ranked, agent-readable list of high-leverage actions for this brain.\n' +
' --json Structured findings. Exit code: 0 clean / 1 warn / 2 critical.\n' +
' --apply <id> Run ONE finding\'s fix (local-only, confirms first). Only findings\n' +
' that report an apply id are runnable.\n\n' +
'Read-only by default; never mutates without --apply + your confirmation.',
);
return { exitCode: 0 };
}
const json = args.includes('--json');
const applyIdx = args.indexOf('--apply');
const applyId = applyIdx >= 0 ? args[applyIdx + 1] : undefined;
const ctx = buildContext(engine);
const report = await runAdvisor(ctx);
if (applyId) {
return applyFinding(report, applyId);
}
// Record run history (local-only) for "since last run" deltas.
let deltaNote = '';
try {
const prior = appendAdvisorRun(report);
deltaNote = summarizeDeltas(prior, report);
} catch {
/* history is best-effort; never block the report */
}
if (json) {
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
} else {
process.stdout.write(renderAdvisorReport(report));
if (deltaNote) process.stdout.write(deltaNote + '\n');
}
return { exitCode: exitFor(report) };
}
/**
* E5: run a single finding's fix. Allowlist = findings carrying a dispatch_id.
* Local-only (refused over MCP by construction this is the CLI path). Executes
* the structured argv via a child process with NO shell.
*/
function applyFinding(report: AdvisorReport, id: string): AdvisorCliResult {
const target = resolveApplyTarget(report, id);
if (!target.ok) {
console.error(
target.error +
(target.runnable.length ? ` Runnable now: ${target.runnable.join(', ')}.` : ' Nothing is runnable right now.'),
);
return { exitCode: 2 };
}
console.error(`About to run: ${target.display}`);
if (!confirmTty('Proceed? [y/N]: ')) {
console.error('Aborted. Nothing was run.');
return { exitCode: 1 };
}
const [cmd, ...rest] = target.argv;
const res = spawnSync(cmd!, rest, { stdio: 'inherit', shell: false });
return { exitCode: (res.status ?? 1) === 0 ? 0 : 2 };
}
/** Synchronous y/N TTY confirm. Non-TTY → false (never auto-run). */
function confirmTty(prompt: string): boolean {
if (!process.stdin.isTTY) return false;
// Bun supports a synchronous prompt via readline only async; use a tiny
// blocking read on the TTY fd instead.
process.stderr.write(prompt);
const buf = Buffer.alloc(8);
try {
const fs = require('fs') as typeof import('fs');
const n = fs.readSync(0, buf, 0, 8, null);
const ans = buf.toString('utf8', 0, n).trim().toLowerCase();
return ans === 'y' || ans === 'yes';
} catch {
return false;
}
}
// readline imported for type-compat with other commands; not used directly.
void createInterface;
+78 -19
View File
@@ -74,8 +74,12 @@ SUBMITTING
--follow Tail status until terminal (default on TTY)
--detach Submit + print job id, exit immediately
Flags after \`run\` up to the first unrecognized token are parsed; the
remainder is the prompt. Use \`--\` to explicitly terminate flag parsing.
Flags before the prompt are parsed normally. The no-value switches
--detach, --follow and --no-follow are ALSO recognized when they trail
the prompt, so \`gbrain agent run "do X" --detach\` detaches. Any other
--word is treated as prompt text (no error). Use \`--\` to end flag
parsing and pass the rest verbatim:
gbrain agent run -- "literally --detach this, with --flags"
VIEWING
gbrain agent logs <job_id>
@@ -102,31 +106,86 @@ interface RunFlags {
detach: boolean;
}
/** No-value switches that may also trail the prompt and get hoisted out (#1738). */
const BOOLEAN_TAIL_FLAGS = new Set(['--follow', '--no-follow', '--detach']);
function applyBooleanFlag(flags: RunFlags, a: string): void {
if (a === '--follow') flags.follow = true;
else if (a === '--no-follow') flags.follow = false;
else { flags.detach = true; flags.follow = false; } // --detach
}
/** Read the value for a value-flag, rejecting a missing or flag-shaped value. */
function requireFlagValue(args: string[], i: number, flag: string): string {
const v = args[i];
if (v === undefined || v.startsWith('--')) {
throw new Error(`gbrain agent run: ${flag} requires a value. Run \`gbrain agent run --help\`.`);
}
return v;
}
function parseIntFlagValue(v: string, flag: string): number {
const n = parseInt(v, 10);
if (Number.isNaN(n)) {
throw new Error(`gbrain agent run: ${flag} expects a number, got "${v}".`);
}
return n;
}
/**
* Parse `agent run` args into flags + prompt (#1738).
*
* args [ leading flag zone ] [ ? ] [ prompt (trailing booleans) ]
*
* Leading zone: known flags (value + boolean) are consumed left-to-right until
* the first positional token, an UNKNOWN --flag, or an explicit `--`. An
* unknown --flag is NOT an error it begins the freeform prompt, so
* `agent run "--note: do X"` works without `--`. Value-flags missing their
* value throw a usage error instead of silently capturing `undefined`/`NaN`.
*
* Prompt zone: a trailing run of the no-value switches (--detach/--follow/
* --no-follow) is hoisted out so `agent run "do X" --detach` detaches. Only
* trailing switches are hoisted; a `--word` elsewhere in the prompt stays
* verbatim. After an explicit `--`, nothing is hoisted.
*/
function parseRunFlags(args: string[]): { flags: RunFlags; rest: string[] } {
const flags: RunFlags = {
follow: process.stdout.isTTY === true,
detach: false,
};
let i = 0;
while (i < args.length) {
const a = args[i];
if (a === '--') { i++; break; }
if (!isKnownFlag(a!)) break;
let escaped = false;
for (; i < args.length; i++) {
const a = args[i]!;
if (a === '--') { i++; escaped = true; break; }
if (!a.startsWith('--')) break;
let known = true;
switch (a) {
case '--subagent-def': flags.subagentDef = args[++i]; i++; break;
case '--model': flags.model = args[++i]; i++; break;
case '--max-turns': flags.maxTurns = parseInt(args[++i] ?? '', 10); i++; break;
case '--tools': flags.tools = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean); i++; break;
case '--timeout-ms': flags.timeoutMs = parseInt(args[++i] ?? '', 10); i++; break;
case '--fanout-manifest': flags.fanoutManifest = args[++i]; i++; break;
case '--follow': flags.follow = true; i++; break;
case '--no-follow': flags.follow = false; i++; break;
case '--detach': flags.detach = true; flags.follow = false; i++; break;
default:
throw new Error(`unknown flag: ${a}. Run \`gbrain agent run --help\` for usage.`);
case '--subagent-def': flags.subagentDef = requireFlagValue(args, ++i, a); break;
case '--model': flags.model = requireFlagValue(args, ++i, a); break;
case '--max-turns': flags.maxTurns = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
case '--tools': flags.tools = requireFlagValue(args, ++i, a).split(',').map(s => s.trim()).filter(Boolean); break;
case '--timeout-ms': flags.timeoutMs = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
case '--fanout-manifest': flags.fanoutManifest = requireFlagValue(args, ++i, a); break;
case '--follow': flags.follow = true; break;
case '--no-follow': flags.follow = false; break;
case '--detach': flags.detach = true; flags.follow = false; break;
default: known = false; break;
}
if (!known) break; // unknown --flag → first token of the (freeform) prompt
}
const rest = args.slice(i);
// An explicit `--` terminates flag parsing wherever it appears — leading
// zone (escaped) OR after a positional (the leading loop breaks before it,
// so `escaped` stays false). Honor both: when the prompt carries a literal
// `--`, hoist nothing, so `agent run note -- --detach` keeps `--detach`
// verbatim instead of silently flipping detach mode.
if (!escaped && !rest.includes('--')) {
while (rest.length > 0 && BOOLEAN_TAIL_FLAGS.has(rest[rest.length - 1]!)) {
applyBooleanFlag(flags, rest.pop()!);
}
}
return { flags, rest: args.slice(i) };
return { flags, rest };
}
export async function runAgentRun(engine: BrainEngine, args: string[]): Promise<void> {
@@ -251,7 +310,7 @@ async function runFanout(engine: BrainEngine, queue: MinionQueue, flags: RunFlag
// do this after submission because each add() returns the committed
// row's id; the aggregator's seed started with an empty array.
await engine.executeRaw(
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::jsonb) WHERE id = $2`,
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::text::jsonb) WHERE id = $2`,
[JSON.stringify(childIds), aggregator.id],
);
+70 -4
View File
@@ -346,6 +346,12 @@ interface RegisterClientArgs {
federatedRead: string[] | undefined;
redirectUris: string[];
tokenEndpointAuthMethod: string | undefined;
boundTools: string[] | undefined;
boundSourceId: string | undefined;
boundBrainId: string | undefined;
boundSlugPrefixes: string[] | undefined;
boundMaxConcurrent: number | undefined;
budgetUsdPerDay: string | undefined;
}
export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
@@ -356,6 +362,12 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
federatedRead: undefined,
redirectUris: [],
tokenEndpointAuthMethod: undefined,
boundTools: undefined,
boundSourceId: undefined,
boundBrainId: undefined,
boundSlugPrefixes: undefined,
boundMaxConcurrent: undefined,
budgetUsdPerDay: undefined,
};
let i = 0;
let grantTypesSet = false;
@@ -389,6 +401,34 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
case '--token-endpoint-auth-method':
out.tokenEndpointAuthMethod = requireValue();
i += 2; break;
case '--bound-tools': {
const v = requireValue();
out.boundTools = v.split(',').map(s => s.trim()).filter(Boolean);
i += 2; break;
}
case '--bound-source': out.boundSourceId = requireValue(); i += 2; break;
case '--bound-brain': out.boundBrainId = requireValue(); i += 2; break;
case '--bound-slug-prefixes': {
const v = requireValue();
out.boundSlugPrefixes = v.split(',').map(s => s.trim()).filter(Boolean);
i += 2; break;
}
case '--bound-max-concurrent': {
const v = Number(requireValue());
if (!Number.isInteger(v) || v < 1) {
throw new Error('--bound-max-concurrent must be a positive integer');
}
out.boundMaxConcurrent = v;
i += 2; break;
}
case '--budget-usd-per-day': {
const v = requireValue();
if (!/^\d+(?:\.\d{1,2})?$/.test(v)) {
throw new Error('--budget-usd-per-day must be a non-negative decimal with at most 2 decimal places');
}
out.budgetUsdPerDay = v;
i += 2; break;
}
default:
throw new Error(`Unknown flag: ${flag}`);
}
@@ -405,7 +445,7 @@ export function parseRegisterClientArgs(args: string[]): RegisterClientArgs {
async function registerClient(name: string, args: string[]) {
if (!name) {
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none]');
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none] [--bound-tools T1,T2] [--bound-source SOURCE] [--bound-brain BRAIN] [--bound-slug-prefixes P1,P2] [--bound-max-concurrent N] [--budget-usd-per-day USD]');
process.exit(1);
}
let parsed: RegisterClientArgs;
@@ -413,17 +453,28 @@ async function registerClient(name: string, args: string[]) {
parsed = parseRegisterClientArgs(args);
} catch (e: any) {
console.error(`Error: ${e.message}`);
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none]');
console.error('Usage: auth register-client <name> [--grant-types G] [--scopes S] [--source SOURCE] [--federated-read SRC1,SRC2,...] [--redirect-uri URI ...] [--token-endpoint-auth-method client_secret_post|client_secret_basic|none] [--bound-tools T1,T2] [--bound-source SOURCE] [--bound-brain BRAIN] [--bound-slug-prefixes P1,P2] [--bound-max-concurrent N] [--budget-usd-per-day USD]');
process.exit(1);
}
const { grantTypes, scopes, sourceId, federatedRead, redirectUris, tokenEndpointAuthMethod } = parsed;
const agentBindings = parsed.boundTools || parsed.boundSourceId || parsed.boundBrainId ||
parsed.boundSlugPrefixes || parsed.boundMaxConcurrent !== undefined || parsed.budgetUsdPerDay !== undefined
? {
boundTools: parsed.boundTools,
boundSourceId: parsed.boundSourceId,
boundBrainId: parsed.boundBrainId,
boundSlugPrefixes: parsed.boundSlugPrefixes,
boundMaxConcurrent: parsed.boundMaxConcurrent,
budgetUsdPerDay: parsed.budgetUsdPerDay,
}
: undefined;
try {
await withConfiguredSql(async (sql) => {
const { GBrainOAuthProvider } = await import('../core/oauth-provider.ts');
const provider = new GBrainOAuthProvider({ sql });
const { clientId, clientSecret } = await provider.registerClientManual(
name, grantTypes, scopes, redirectUris, sourceId, federatedRead, tokenEndpointAuthMethod,
name, grantTypes, scopes, redirectUris, sourceId, federatedRead, tokenEndpointAuthMethod, agentBindings,
);
const effectiveFederated = federatedRead && federatedRead.length > 0 ? federatedRead : [sourceId];
const effectiveAuthMethod = tokenEndpointAuthMethod || 'client_secret_post';
@@ -441,7 +492,16 @@ async function registerClient(name: string, args: string[]) {
console.log(` Redirect URIs: ${redirectUris.join(', ')}`);
}
console.log(` Write source: ${sourceId}`);
console.log(` Federated reads: ${effectiveFederated.join(', ')}\n`);
console.log(` Federated reads: ${effectiveFederated.join(', ')}`);
if (agentBindings) {
console.log(` Bound tools: ${(parsed.boundTools ?? []).join(', ') || '<none>'}`);
console.log(` Bound source: ${parsed.boundSourceId ?? '<none>'}`);
console.log(` Bound brain: ${parsed.boundBrainId ?? '<none>'}`);
console.log(` Bound slug prefixes:${parsed.boundSlugPrefixes ? ' ' + parsed.boundSlugPrefixes.join(', ') : ' <none>'}`);
console.log(` Max concurrency: ${parsed.boundMaxConcurrent ?? 1}`);
console.log(` Daily budget USD: ${parsed.budgetUsdPerDay ?? '<none>'}`);
}
console.log('');
if (clientSecret) {
console.log('Save the client secret — it will not be shown again.');
} else {
@@ -527,6 +587,12 @@ Usage:
--redirect-uri <https://...> (v0.41.3+; repeatable; required for authorization_code)
--token-endpoint-auth-method <method> (v0.41.3+; client_secret_post | client_secret_basic | none;
'none' = public PKCE-only client, no secret minted)
--bound-tools <tool1,tool2> Bind submit_agent to an allow-list of tools
--bound-source <id> Bind submit_agent jobs to a source id
--bound-brain <id> Bind submit_agent jobs to a brain id
--bound-slug-prefixes <prefix1,prefix2> Bind submit_agent writes to slug prefixes
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
`);
+306 -5
View File
@@ -32,9 +32,26 @@
import type { BrainEngine, SourceRow } from '../core/engine.ts';
import type { MinionQueue } from '../core/minions/queue.ts';
import { NON_GLOBAL_PHASES, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY } from '../core/cycle.ts';
const FULL_CYCLE_FLOOR_MIN = 60;
// #2194 fix #2: failure cooldown. A source whose autopilot-cycle keeps
// failing/timing-out re-dispatches every tick today (only SUCCESS gates
// dispatch), so the same handful of sources fail and re-fan-out forever — the
// self-perpetuating dead-job storm. Back a failed source off with bounded
// exponential cooldown so a chronically-slow source can't re-dispatch every
// tick. Disabled with autopilot.failure_cooldown_min=0.
const FAILURE_COOLDOWN_BASE_MIN = 10;
const FAILURE_COOLDOWN_CAP_MIN = 120;
const FAILURE_COOLDOWN_EXP_CAP = 4; // 2^4 = 16× base before the cap clamps
/** Recent-failure record for one source (from minion_jobs dead/failed rows). */
export interface SourceFailure { count: number; lastFailedAt: Date; }
/** Resolved cooldown knobs. baseMin <= 0 means the cooldown is disabled. */
export interface CooldownOpts { baseMin: number; capMin: number; }
export interface FanoutOpts {
repoPath: string;
slot: string;
@@ -58,6 +75,8 @@ export interface FanoutResult {
skipped_fresh: string[];
/** Source ids beyond the fanoutMax cap (will retry next tick). */
skipped_cap: string[];
/** Source ids skipped because they're in failure cooldown (#2194 fix #2). */
skipped_cooldown: string[];
/** True when this tick fell back to the legacy single-job path
* (no sources rows / engine empty). */
legacy_fallback: boolean;
@@ -83,6 +102,62 @@ export async function resolveFanoutMax(engine: BrainEngine): Promise<number> {
return engine.kind === 'pglite' ? 1 : 4;
}
/**
* Read the worker concurrency the supervisor most recently STARTED with, from
* its `started` audit event (the lowest-coupling source no extra lock-row
* column). Filesystem read; returns null when no supervisor has ever started
* (or the event lacks concurrency). Filtered by queue so a `shell`-queue
* supervisor's concurrency doesn't leak into the `default`-queue decision.
*
* ADVISORY use only (doctor warning). Behavior-changing callers (the fanout
* clamp) must additionally gate on a LIVE supervisor see
* resolveEffectiveFanoutMax because a stale `started` row can otherwise
* shrink fan-out for a supervisor that isn't running that config (codex #9/D5).
*/
export async function readSupervisorConcurrency(queue = 'default'): Promise<number | null> {
try {
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const started = events
.filter((e) => e.event === 'started' && (e.queue === undefined || e.queue === queue))
.pop();
const c = started?.concurrency;
return typeof c === 'number' && Number.isFinite(c) ? c : null;
} catch {
return null;
}
}
/**
* Resolve fanoutMax CLAMPED to the worker's effective concurrency (#2194 fix #1).
*
* Fanning out more cycles than the worker can run guarantees waiters that then
* race the stalled-sweeper. Clamp to `max(1, concurrency - 1)` reserving 1
* slot for targeted sync/embed jobs that share the `default` queue.
*
* codex #9 / D5: the clamp is BEHAVIOR-changing, so it trusts only a
* proven-alive supervisor (live DB-lock holder, `ttl_expires_at`-gated). With
* no live holder the concurrency is UNKNOWN and we fall back to the unclamped
* default (4 pg / 1 pglite) the safe direction (never starve on stale data).
* Operators can disable the clamp via `autopilot.fanout_clamp_to_concurrency`.
*/
export async function resolveEffectiveFanoutMax(engine: BrainEngine, queue = 'default'): Promise<number> {
const base = await resolveFanoutMax(engine);
const clampCfg = await engine.getConfig('autopilot.fanout_clamp_to_concurrency');
if (clampCfg === 'false' || clampCfg === '0') return base; // operator opt-out
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId(queue));
if (!snap || !isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) return base; // no live holder → unknown → no clamp
const concurrency = await readSupervisorConcurrency(queue);
if (concurrency === null) return base;
return Math.max(1, Math.min(base, concurrency - 1));
} catch {
return base;
}
}
/**
* Read `last_full_cycle_at` ISO string from a source's config JSONB.
* Returns null when missing or unparseable. Pure function over the row
@@ -111,6 +186,133 @@ export function isSourceStale(src: SourceRow, now = Date.now(), floorMin = FULL_
return ageMin >= floorMin;
}
/**
* Most recent SUCCESSFUL cycle for a source. Prefers `last_source_cycle_at`
* (per-source phases, written by the split cycle) and falls back to the legacy
* `last_full_cycle_at`, so this works before AND after the cycle split.
*/
export function readLastSuccessAt(src: SourceRow): Date | null {
const c = src.config ?? {};
const raw = (typeof c.last_source_cycle_at === 'string' && c.last_source_cycle_at)
|| (typeof c.last_full_cycle_at === 'string' && c.last_full_cycle_at)
|| null;
if (!raw) return null;
const d = new Date(raw);
return Number.isFinite(d.getTime()) ? d : null;
}
/** Bounded exponential cooldown window (minutes) for a given failure count. */
export function cooldownMinForCount(count: number, opts: CooldownOpts): number {
if (count <= 0 || opts.baseMin <= 0) return 0;
const mult = Math.pow(2, Math.min(count - 1, FAILURE_COOLDOWN_EXP_CAP));
return Math.min(opts.baseMin * mult, opts.capMin);
}
/**
* Is a source currently in failure cooldown? Pure drives both the dispatch
* gate and the claim-time guard. A SUCCESS at-or-after the most recent failure
* clears the cooldown (codex #7: operator repair / manual cycle re-eligibility),
* so a recovered source is never suppressed by stale failure history.
*/
export function isInFailureCooldown(
failure: SourceFailure | undefined,
lastSuccessAt: Date | null,
now: number,
opts: CooldownOpts,
): boolean {
if (opts.baseMin <= 0) return false; // disabled
if (!failure || failure.count <= 0) return false;
if (lastSuccessAt && lastSuccessAt.getTime() >= failure.lastFailedAt.getTime()) return false;
const cooldownMs = cooldownMinForCount(failure.count, opts) * 60_000;
return (now - failure.lastFailedAt.getTime()) < cooldownMs;
}
/**
* Resolve cooldown knobs from config. `autopilot.failure_cooldown_min` overrides
* the base (0 = disable entirely exactly today's behavior);
* `autopilot.failure_cooldown_cap_min` overrides the ceiling.
*/
export async function resolveFailureCooldownOpts(engine: BrainEngine): Promise<CooldownOpts> {
let baseMin = FAILURE_COOLDOWN_BASE_MIN;
let capMin = FAILURE_COOLDOWN_CAP_MIN;
const baseCfg = await engine.getConfig('autopilot.failure_cooldown_min');
if (baseCfg !== null && baseCfg !== undefined && baseCfg !== '') {
const n = parseInt(baseCfg, 10);
if (Number.isFinite(n) && n >= 0) baseMin = n;
}
const capCfg = await engine.getConfig('autopilot.failure_cooldown_cap_min');
if (capCfg) {
const n = parseInt(capCfg, 10);
if (Number.isFinite(n) && n >= 1) capMin = n;
}
return { baseMin, capMin };
}
/**
* Read recent dead/failed autopilot-cycle jobs grouped by source. Read-at-
* dispatch (NOT a write hook) because timeouts/RSS-kills/stalls dead-letter via
* SQL in queue.ts and never run handler code a write-only cooldown would miss
* the exact failures that drive the storm. Engine-parity-safe via executeRaw
* (one query, both engines); cutoff is precomputed in JS to avoid INTERVAL
* portability concerns. codex #6: rows with a null source_id are excluded.
*/
export async function readRecentSourceFailures(
engine: BrainEngine,
opts: { sinceMin?: number; sourceId?: string } = {},
): Promise<Map<string, SourceFailure>> {
const sinceMin = opts.sinceMin ?? FAILURE_COOLDOWN_CAP_MIN;
const cutoff = new Date(Date.now() - sinceMin * 60_000).toISOString();
const map = new Map<string, SourceFailure>();
try {
const params: unknown[] = [cutoff];
let sql =
`SELECT data->>'source_id' AS source_id,
count(*)::int AS fail_count,
max(finished_at) AS last_failed_at
FROM minion_jobs
WHERE name = 'autopilot-cycle'
AND status IN ('dead','failed')
AND data->>'source_id' IS NOT NULL
AND finished_at IS NOT NULL
AND finished_at > $1`;
if (opts.sourceId) { params.push(opts.sourceId); sql += ` AND data->>'source_id' = $${params.length}`; }
sql += ` GROUP BY data->>'source_id'`;
const rows = await engine.executeRaw<{ source_id: string | null; fail_count: number; last_failed_at: string | Date }>(sql, params);
for (const r of rows) {
if (!r.source_id) continue; // codex #6 null-source guard (defensive)
const last = r.last_failed_at instanceof Date ? r.last_failed_at : new Date(r.last_failed_at);
if (!Number.isFinite(last.getTime())) continue;
map.set(r.source_id, { count: Number(r.fail_count) || 0, lastFailedAt: last });
}
} catch {
// Pre-migration / transient DB error → no cooldown data (fail open: dispatch).
}
return map;
}
/**
* Claim-time cooldown guard (codex #5 / D4): a job already queued or retrying
* (max_attempts:2) can reach the worker after the dispatch gate decided. The
* handler calls this immediately before runCycle; an in-cooldown claim becomes
* a no-op skip (NOT a failure it must not re-arm the cooldown). Shares the
* exact cooldown math with the dispatch gate (DRY).
*/
export async function isSourceInCooldown(engine: BrainEngine, sourceId: string, now = Date.now()): Promise<boolean> {
const opts = await resolveFailureCooldownOpts(engine);
if (opts.baseMin <= 0) return false;
const failures = await readRecentSourceFailures(engine, { sinceMin: opts.capMin, sourceId });
const failure = failures.get(sourceId);
if (!failure) return false;
let lastSuccessAt: Date | null = null;
try {
const rows = await engine.executeRaw<{ config: Record<string, unknown> | null }>(
`SELECT config FROM sources WHERE id = $1`, [sourceId],
);
if (rows[0]) lastSuccessAt = readLastSuccessAt({ config: rows[0].config ?? {} } as SourceRow);
} catch { /* treat as no success */ }
return isInFailureCooldown(failure, lastSuccessAt, now, opts);
}
/**
* Decide which sources to dispatch this tick. Pure function so tests can
* exercise the freshness gate + cap math without an engine.
@@ -126,11 +328,21 @@ export function selectSourcesForDispatch(
fanoutMax: number,
now = Date.now(),
floorMin = FULL_CYCLE_FLOOR_MIN,
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[] } {
recentFailures: Map<string, SourceFailure> = new Map(),
cooldownOpts: CooldownOpts = { baseMin: FAILURE_COOLDOWN_BASE_MIN, capMin: FAILURE_COOLDOWN_CAP_MIN },
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[]; skippedCooldown: SourceRow[] } {
const stale: SourceRow[] = [];
const fresh: SourceRow[] = [];
const cooldown: SourceRow[] = [];
for (const s of sources) {
(isSourceStale(s, now, floorMin) ? stale : fresh).push(s);
if (!isSourceStale(s, now, floorMin)) { fresh.push(s); continue; }
// #2194 fix #2: a stale source that recently failed is held in cooldown so
// it can't re-dispatch every tick (the storm). Success clears it.
if (isInFailureCooldown(recentFailures.get(s.id), readLastSuccessAt(s), now, cooldownOpts)) {
cooldown.push(s);
continue;
}
stale.push(s);
}
// Oldest-first ordering: NULL last_full_cycle_at sorts before any timestamp.
stale.sort((a, b) => {
@@ -141,7 +353,7 @@ export function selectSourcesForDispatch(
});
const dispatch = stale.slice(0, fanoutMax);
const skippedCap = stale.slice(fanoutMax);
return { dispatch, skippedFresh: fresh, skippedCap };
return { dispatch, skippedFresh: fresh, skippedCap, skippedCooldown: cooldown };
}
/**
@@ -193,10 +405,27 @@ export async function dispatchPerSource(
} else {
log(`[dispatch] job #${job.id} autopilot-cycle (legacy single-source)`);
}
return { dispatched: [], skipped_fresh: [], skipped_cap: [], legacy_fallback: true };
return { dispatched: [], skipped_fresh: [], skipped_cap: [], skipped_cooldown: [], legacy_fallback: true };
}
const { dispatch, skippedFresh, skippedCap } = selectSourcesForDispatch(sources, opts.fanoutMax);
// #2194 fix #2: load recent per-source failures + cooldown knobs so a
// chronically-failing source is backed off instead of re-dispatched every
// tick. Fail-open: cooldown is an optimization, not a correctness gate — if
// config/job-history reads fail (or the engine lacks them), dispatch proceeds
// with no cooldown rather than blocking.
let cooldownOpts: CooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
let recentFailures = new Map<string, SourceFailure>();
try {
cooldownOpts = await resolveFailureCooldownOpts(engine);
if (cooldownOpts.baseMin > 0) {
recentFailures = await readRecentSourceFailures(engine, { sinceMin: cooldownOpts.capMin });
}
} catch {
cooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
}
const { dispatch, skippedFresh, skippedCap, skippedCooldown } =
selectSourcesForDispatch(sources, opts.fanoutMax, Date.now(), FULL_CYCLE_FLOOR_MIN, recentFailures, cooldownOpts);
const dispatched: string[] = [];
for (const src of dispatch) {
@@ -208,6 +437,11 @@ export async function dispatchPerSource(
repoPath: opts.repoPath,
source_id: src.id,
pull: !!remoteUrl,
// #2194 fix #3 (cycle split): per-source cycles run ONLY source-scoped
// (+ mixed) phases. The brain-wide global phases (embed, orphans,
// purge, …) run once in autopilot-global-maintenance, not N times
// concurrently here — the fix for the 4→10GB RSS blowout.
phases: NON_GLOBAL_PHASES,
},
{
queue: 'default',
@@ -261,10 +495,77 @@ export async function dispatchPerSource(
}));
}
if (skippedCooldown.length > 0 && opts.jsonMode) {
emit(JSON.stringify({
event: 'fanout_cooldown_skipped',
sources: skippedCooldown.map(s => s.id),
}));
}
return {
dispatched,
skipped_fresh: skippedFresh.map(s => s.id),
skipped_cap: skippedCap.map(s => s.id),
skipped_cooldown: skippedCooldown.map(s => s.id),
legacy_fallback: false,
};
}
const GLOBAL_FLOOR_MIN = 60;
/** Is the brain-wide maintenance overdue? Null/unparseable → overdue. */
export function isGlobalMaintenanceStale(lastGlobalAtIso: string | null, now = Date.now(), floorMin = GLOBAL_FLOOR_MIN): boolean {
if (!lastGlobalAtIso) return true;
const d = new Date(lastGlobalAtIso);
if (!Number.isFinite(d.getTime())) return true;
return (now - d.getTime()) / 60_000 >= floorMin;
}
/**
* #2194 fix #3 / #2227 bug #3 dispatch the single brain-wide maintenance job
* that runs the `global` cycle phases (embed, orphans, purge, ) ONCE per
* window, instead of N per-source cycles each running them concurrently (the
* RSS blowout). Single-flight is structural: one `idempotency_key` +
* `maxWaiting:1`, so a slow run never stacks. Gated on `autopilot.last_global_at`
* (stamped by the handler on success). Postgres-only fan-out concern; on PGLite
* the file lock already serializes, but the job is still correct there.
*/
export async function dispatchGlobalMaintenance(
engine: BrainEngine,
queue: MinionQueue,
opts: { repoPath: string; slot: string; timeoutMs: number; jsonMode: boolean; emit?: (l: string) => void; log?: (l: string) => void },
): Promise<{ dispatched: boolean; reason: 'stale' | 'fresh' }> {
const emit = opts.emit ?? ((line) => process.stderr.write(line + '\n'));
const log = opts.log ?? ((line) => console.log(line));
let floorMin = GLOBAL_FLOOR_MIN;
const floorCfg = await engine.getConfig('autopilot.global_floor_min');
if (floorCfg) {
const n = parseInt(floorCfg, 10);
if (Number.isFinite(n) && n >= 1) floorMin = n;
}
const lastGlobalAt = await engine.getConfig(LAST_GLOBAL_AT_KEY);
if (!isGlobalMaintenanceStale(lastGlobalAt, Date.now(), floorMin)) {
return { dispatched: false, reason: 'fresh' };
}
const job = await queue.add(
'autopilot-global-maintenance',
{ repoPath: opts.repoPath, phases: GLOBAL_PHASES },
{
queue: 'default',
// Structural single-flight: one global job per slot; maxWaiting:1 coalesces
// any surplus so a slow brain-wide pass never stacks duplicates.
idempotency_key: `autopilot-global:${opts.slot}`,
max_attempts: 2,
timeout_ms: opts.timeoutMs,
maxWaiting: 1,
},
);
if (opts.jsonMode) {
emit(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'global_maintenance', slot: opts.slot }));
} else {
log(`[dispatch] job #${job.id} autopilot-global-maintenance (brain-wide phases)`);
}
return { dispatched: true, reason: 'stale' };
}
+219 -21
View File
@@ -17,7 +17,8 @@
* gbrain autopilot --status [--json]
*/
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync, chmodSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join } from 'path';
import { execSync } from 'child_process';
import type { BrainEngine } from '../core/engine.ts';
@@ -108,7 +109,21 @@ function logError(phase: string, e: unknown) {
*/
export function resolveGbrainCliPath(): string {
try {
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
// #2747: `env: process.env` is required under Bun. Bun's execSync
// snapshots process.env at Bun's OWN startup, not at call time — a
// runtime PATH mutation (dotenv/config loading, shell-profile sourcing
// in a wrapper, etc.) happening between Bun boot and this call is
// invisible to `which` without explicitly forwarding the current env.
// This is why "which gbrain" succeeds when run standalone (fresh Bun
// process, no prior mutation) but can fail from inside autopilot's own
// process at this exact call site. Same fix already applied to
// detectTini() in spawn-helpers.ts (see its comment) — this call site
// was missed.
const which = execSync('which gbrain', {
encoding: 'utf-8',
stdio: ['ignore', 'pipe', 'ignore'],
env: process.env,
}).trim();
if (which) return which;
} catch { /* not on $PATH — fall through */ }
@@ -122,13 +137,145 @@ export function resolveGbrainCliPath(): string {
return arg1;
}
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
// #2747: include what we actually saw so an operator (or a future bug
// report) doesn't have to guess whether PATH/execPath/argv[1] looked
// sane at the moment of failure.
throw new Error(
'Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH ' +
'(e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly. ' +
`Debug: PATH=${JSON.stringify(process.env.PATH ?? '')} execPath=${JSON.stringify(exec)} argv1=${JSON.stringify(arg1)}`,
);
}
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
return !args.includes('--no-worker');
}
/**
* #1525 positional subcommand translation.
*
* Pre-fix, `gbrain autopilot status` silently fell through to "start daemon"
* because `runAutopilot()` only branched on flag forms (`--status`, etc.).
* `status` was treated as a stray positional and ignored.
*
* This translator maps known positional subcommands to their flag form so
* `autopilot status` is equivalent to `autopilot --status`, then rejects
* any unrecognized positional with a fail-loud error before any side
* effect (lockfile, daemon spawn, sync dispatch) runs.
*
* Scope decisions:
* - Known aliases: `status` `--status`, `install` `--install`,
* `uninstall` `--uninstall`, `start` (drop; default daemon launch).
* - `stop` is intentionally NOT aliased here. Stopping a running daemon
* is a new behavior (read PID from lock, SIGTERM, drain) that deserves
* its own design and PR. Users typing `gbrain autopilot stop` today get
* the unknown-positional error with the canonical alternatives.
* - At most one positional allowed; multiple positionals fail loud.
*/
// Every flag that consumes the NEXT argv token. Missing one here makes the
// translator misread the flag's value as a positional subcommand and exit 2
// (e.g. `--install --target linux-cron`). Keep in sync with parseArg call sites.
const AUTOPILOT_VALUE_FLAGS = new Set(['--repo', '--interval', '--target']);
const AUTOPILOT_POSITIONAL_ALIASES: Record<string, string | null> = {
status: '--status',
install: '--install',
uninstall: '--uninstall',
start: null, // drop the positional; default behavior is daemon launch
};
export type PositionalTranslation =
| { ok: true; args: string[] }
| {
ok: false;
reason: 'unknown_subcommand' | 'multiple_subcommands';
message: string;
};
export function translatePositionalSubcommands(args: string[]): PositionalTranslation {
const out: string[] = [];
let positionalSeen = false;
let i = 0;
while (i < args.length) {
const a = args[i];
if (AUTOPILOT_VALUE_FLAGS.has(a)) {
// Pass through the flag and its value untouched. If the value is
// missing at end-of-argv, fall through so the existing parseArg
// path can report the broken usage.
out.push(a);
if (i + 1 < args.length) {
out.push(args[i + 1]);
i += 2;
} else {
i += 1;
}
continue;
}
if (a.startsWith('-')) {
out.push(a);
i += 1;
continue;
}
// Positional subcommand.
if (positionalSeen) {
const known = Object.keys(AUTOPILOT_POSITIONAL_ALIASES).join(', ');
return {
ok: false,
reason: 'multiple_subcommands',
message: `Multiple subcommands given. Use only one of: ${known}.`,
};
}
positionalSeen = true;
if (a in AUTOPILOT_POSITIONAL_ALIASES) {
const alias = AUTOPILOT_POSITIONAL_ALIASES[a];
if (alias) out.push(alias);
i += 1;
continue;
}
const known = Object.keys(AUTOPILOT_POSITIONAL_ALIASES).join(', ');
return {
ok: false,
reason: 'unknown_subcommand',
message:
`Unknown subcommand: \`${a}\`.\n` +
`Allowed subcommands: ${known}.\n` +
`Or use the flag form: --status, --install, --uninstall.\n` +
`Run \`gbrain autopilot --help\` for full usage.`,
};
}
return { ok: true, args: out };
}
export function isPidAlive(pid: number): boolean {
if (!Number.isFinite(pid) || pid <= 0) return false;
try {
process.kill(pid, 0);
return true;
} catch (error: unknown) {
return (error as NodeJS.ErrnoException).code === 'EPERM';
}
}
export function decideLockAcquisition(
lockPath: string,
currentPid: number,
): { action: 'acquire' } | { action: 'exit'; holderPid: number } | { action: 'takeover'; reason: string } {
if (!existsSync(lockPath)) return { action: 'acquire' };
let raw = '';
try {
raw = readFileSync(lockPath, 'utf-8').trim();
} catch {
// An unreadable lock cannot prove another process is alive.
}
const holderPid = Number.parseInt(raw, 10);
const sameProcess = Number.isFinite(holderPid) && holderPid === currentPid;
const alive = !sameProcess && isPidAlive(holderPid);
if (alive) return { action: 'exit', holderPid };
return { action: 'takeover', reason: `dead pid ${raw || '<empty>'}` };
}
// ── Self-upgrade silent channel (v0.42; opt-in, supervisor-relaunch) ─────────
/**
@@ -310,6 +457,11 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
' gbrain autopilot --install [--repo <path>]\n' +
' gbrain autopilot --uninstall\n' +
' gbrain autopilot --status [--json]\n\n' +
'Subcommand aliases:\n' +
' gbrain autopilot status → --status\n' +
' gbrain autopilot install → --install\n' +
' gbrain autopilot uninstall → --uninstall\n' +
' gbrain autopilot start → (default daemon launch)\n\n' +
'Self-maintaining brain daemon. Runs the full maintenance cycle\n' +
'(lint + backlinks + sync + extract + embed + orphans) on an interval.\n\n' +
'For a one-shot cron-triggered cycle, see `gbrain dream`.',
@@ -317,6 +469,16 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
return;
}
// #1525: translate positional subcommands to their flag form BEFORE any
// side effect (lockfile, daemon spawn, sync dispatch). Unknown positionals
// fail loud here rather than silently starting the daemon.
const translated = translatePositionalSubcommands(args);
if (!translated.ok) {
console.error(translated.message);
process.exit(2);
}
args = translated.args;
if (args.includes('--install')) {
await installDaemon(engine, args);
return;
@@ -350,14 +512,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const lockPath = gbrainHomePath('autopilot.lock');
try {
mkdirSync(gbrainHomePath(), { recursive: true });
if (existsSync(lockPath)) {
const stat = require('fs').statSync(lockPath);
const ageMinutes = (Date.now() - stat.mtimeMs) / 60000;
if (ageMinutes < 10) {
console.error('Another autopilot instance is running (lock file is fresh). Exiting.');
process.exit(0);
}
console.log('Stale lock file found (>10 min). Taking over.');
const decision = decideLockAcquisition(lockPath, process.pid);
if (decision.action === 'exit') {
console.error(`Another autopilot instance is running (pid ${decision.holderPid}). Exiting.`);
process.exit(0);
}
if (decision.action === 'takeover') {
console.log(`Stale autopilot lock found (${decision.reason}). Taking over.`);
}
writeFileSync(lockPath, String(process.pid));
} catch { /* best-effort */ }
@@ -529,8 +690,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
autopilotReconnectFails = 0; // reset on success
} catch (probeErr) {
try {
await engine.disconnect();
await (engine as any).connect?.();
// #2034: use reconnect() — it restores the config captured at connect()
// and avoids the null-connection window. The previous
// `disconnect()` + bare `connect()` lost the config (throwing
// `database_url undefined` on every retry → FATAL restart-loop on any
// transient DB blip) AND tore down the pool postgres.js can otherwise
// self-heal.
await engine.reconnect({ error: probeErr });
autopilotReconnectFails = 0;
} catch (e) {
logError('reconnect', e);
@@ -542,7 +708,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
`Exiting so launchd ThrottleInterval can apply backoff.`,
);
stopping = true;
process.exitCode = 1;
setCliExitVerdict(1);
break;
}
if (autopilotReconnectFails >= AUTOPILOT_MAX_RECONNECT_FAILS) {
@@ -551,7 +717,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
`Last error: ${(e as Error).message ?? 'unknown'}. Exiting.`,
);
stopping = true;
process.exitCode = 1;
setCliExitVerdict(1);
break;
}
}
@@ -865,8 +1031,12 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// codex P1-3). Fresh-install brains with no sources rows fall
// back to the legacy single autopilot-cycle so existing
// behavior is preserved.
const { dispatchPerSource, resolveFanoutMax } = await import('./autopilot-fanout.ts');
const fanoutMax = await resolveFanoutMax(engine);
const { dispatchPerSource, dispatchGlobalMaintenance, resolveEffectiveFanoutMax } = await import('./autopilot-fanout.ts');
// #2194 fix #1: clamp fan-out to the worker's effective concurrency
// (reserve ≥1 slot), gated on a LIVE supervisor so a stale audit row
// can't shrink throughput (codex #9/D5). autopilot-cycle jobs run on
// the 'default' queue, so that's the concurrency we compare against.
const fanoutMax = await resolveEffectiveFanoutMax(engine, 'default');
const result = await dispatchPerSource(engine, queue, {
repoPath,
slot,
@@ -874,6 +1044,18 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
fanoutMax,
jsonMode,
});
// #2194 fix #3 / #2227 bug #3: dispatch the single brain-wide
// maintenance job (embed/orphans/purge/…) once per window — the per-
// source cycles above no longer run global phases, so this is where
// the brain-wide work happens (single-flight, no RSS blowout). Only on
// the per-source path (legacy single-source still runs everything).
if (!result.legacy_fallback) {
try {
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs, jsonMode });
} catch (e) {
if (jsonMode) process.stderr.write(JSON.stringify({ event: 'global_maintenance_dispatch_failed', error: e instanceof Error ? e.message : String(e) }) + '\n');
}
}
if (result.dispatched.length > 0 || result.legacy_fallback) {
lastFullCycleAt = Date.now();
}
@@ -883,6 +1065,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
dispatched: result.dispatched,
skipped_fresh: result.skipped_fresh,
skipped_cap: result.skipped_cap,
skipped_cooldown: result.skipped_cooldown,
legacy_fallback: result.legacy_fallback,
fanout_max: fanoutMax,
score,
@@ -890,7 +1073,8 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
} else if (!result.legacy_fallback) {
console.log(
`[dispatch] fanout: ${result.dispatched.length} dispatched, ` +
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped ` +
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped, ` +
`${result.skipped_cooldown.length} cooldown ` +
`(score=${score}, max=${fanoutMax})`,
);
}
@@ -1188,7 +1372,14 @@ function installLaunchd(wrapperPath: string, home: string, repoPath: string) {
try {
const agentsDir = join(home, 'Library', 'LaunchAgents');
mkdirSync(agentsDir, { recursive: true });
writeFileSync(plistPath(), plist);
writeFileSync(plistPath(), plist, { mode: 0o644 });
// launchd rejects group/world-writable agent plists: bootstrap/load fails
// with the opaque "Bootstrap failed: 5: Input/output error" and the login
// scan skips the file silently. writeFileSync's mode only applies on
// create — a reinstall over an existing plist keeps the old bits (a 0666
// plist written under an umask-0 parent stays 0666 forever) — so
// normalize unconditionally.
chmodSync(plistPath(), 0o644);
execSync(`launchctl load "${plistPath()}"`, { stdio: 'pipe' });
console.log('Installed launchd service: com.gbrain.autopilot');
console.log(` Repo: ${repoPath}`);
@@ -1278,7 +1469,11 @@ export function migrateSystemdUnitToRestartAlways(): { rewritten: boolean; reaso
return { rewritten: false, reason: 'hand-edited' };
}
try {
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]));
writeFileSync(unitPath, generateSystemdUnit(execMatch![1]), { mode: 0o644 });
// This path always rewrites an EXISTING unit, so writeFileSync's mode
// never applies — chmod is the only thing that normalizes a unit born
// 0666 under a umask-0 parent (systemd warns on world-writable units).
chmodSync(unitPath, 0o644);
try {
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
} catch {
@@ -1295,7 +1490,10 @@ function installSystemd(wrapperPath: string, repoPath: string) {
try {
const unitPath = systemdUnitPath();
mkdirSync(join(process.env.HOME || '', '.config', 'systemd', 'user'), { recursive: true });
writeFileSync(unitPath, unit);
writeFileSync(unitPath, unit, { mode: 0o644 });
// Same umask-0 hardening as the launchd path (systemd warns on
// world-writable units); mode only applies on create, so normalize.
chmodSync(unitPath, 0o644);
execSync('systemctl --user daemon-reload', { stdio: 'pipe', timeout: 10_000 });
execSync('systemctl --user enable --now gbrain-autopilot.service', { stdio: 'pipe', timeout: 15_000 });
console.log('Installed systemd user service: gbrain-autopilot.service');
+11 -7
View File
@@ -257,7 +257,9 @@ function buildChapterPrompt(
return `You are analyzing one chapter of "${bookTitle}"${authorLine} for the user.
Your output is a markdown two-column table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain.
Your output is a two-column HTML table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain.
CRITICAL: Use an HTML <table> with valign="top" on EVERY cell NOT a markdown pipe table. Markdown pipe tables have no way to set vertical alignment, so every renderer except GitHub middle-aligns the rows, which is unreadable when the two columns have different lengths. The HTML <table valign="top"> form top-aligns everywhere (GitHub, PDF, Obsidian).
This is chapter ${chapter.index} of ${totalChapters}.
@@ -276,11 +278,12 @@ Return ONLY a single markdown section in this exact shape:
### Key Ideas
[2-4 sentence thesis of the chapter what the author is actually arguing.]
| What the Author Says | How This Applies to You |
|---|---|
| [Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`<br><br>\` for paragraph breaks within the cell.] | [Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`<br><br>\` for breaks.] |
| [Next section] | [Next mirror] |
| [4-10 rows depending on chapter density] | |
<table>
<tr><th align="left">What the Author Says</th><th align="left">How This Applies to You</th></tr>
<tr><td valign="top">[Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`<br><br>\` for paragraph breaks within the cell.]</td><td valign="top">[Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`<br><br>\` for breaks.]</td></tr>
<tr><td valign="top">[Next section]</td><td valign="top">[Next mirror]</td></tr>
[4-10 rows depending on chapter density]
</table>
\`\`\`
## RULES
@@ -290,6 +293,7 @@ Return ONLY a single markdown section in this exact shape:
- 4-10 rows per chapter. If a section honestly doesn't apply, write \`*This section is less directly relevant because [specific reason].*\` Don't force connections.
- Never generic ("This might apply if you've ever felt..."). Never sycophantic. Never preach.
- Use \`<br><br>\` for paragraph breaks inside table cells, not literal newlines.
- EVERY <td> MUST carry valign="top". Never emit a markdown pipe table (| ... | ... |) always the HTML <table> form above.
You have ${DEFAULT_MAX_TURNS} turns and read-only tools (get_page, search). You CANNOT call put_page your output is the markdown text in your final message. The CLI assembles all chapters and writes the brain page.
@@ -314,7 +318,7 @@ title: "${opts.title} — Personalized"
type: book-analysis${authorLine}
date: ${today}
context: "${contextSummary.replace(/"/g, '\\"')}"
tags: [book, personalized, two-column]
tags: [book, personalized, two-column-htmltable-valign-top]
---`;
const intro = `# ${opts.title} — Personalized
+2 -1
View File
@@ -13,6 +13,7 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import {
runBrainstorm,
formatBrainstormMarkdown,
@@ -322,7 +323,7 @@ async function runBrainstormCli(
const msg = formatSaveOutcome(outcome, { profileLabel: profile.label, slug });
if (msg.stdout) console.log(msg.stdout);
for (const line of msg.stderr) console.error(line);
if (msg.exitCode) process.exitCode = msg.exitCode;
if (msg.exitCode) setCliExitVerdict(msg.exitCode);
}
}
+16 -3
View File
@@ -25,7 +25,9 @@ import type { GBrainConfig } from '../core/config.ts';
import { GBrainError } from '../core/types.ts';
export interface CalibrationProfileRow {
id: number;
/** BIGSERIAL string (postgres.js int8 wire shape; never Number() int8
* exceeds 2^53). No consumer does arithmetic on it; it's audit/serialize only. */
id: string;
source_id: string;
holder: string;
wave_version: string;
@@ -67,7 +69,12 @@ export async function getLatestProfile(
sql += ` ORDER BY generated_at DESC LIMIT 1`;
const rows = await engine.executeRaw<CalibrationProfileRow>(sql, params);
return rows[0] ?? null;
if (!rows[0]) return null;
// `id` is BIGSERIAL → the pg driver returns it as a JS bigint, which crashes
// JSON.stringify on the --json / MCP output paths once a row exists. Coerce to
// string — matches the postgres.js int8 wire shape (and cli.ts's ENG-2
// "bigint → string" contract); String() has no 2^53 ceiling, unlike Number().
return { ...rows[0], id: String(rows[0].id) };
}
/** Human format the profile for terminal output. */
@@ -125,6 +132,7 @@ export interface RunCalibrationArgs {
regenerate?: boolean;
undoWave?: string;
abReport?: boolean;
source?: string;
}
function parseArgs(args: string[]): { sub?: string; opts: RunCalibrationArgs } {
@@ -144,6 +152,7 @@ function parseArgs(args: string[]): { sub?: string; opts: RunCalibrationArgs } {
else if (a === '--json') opts.json = true;
else if (a === '--regenerate') opts.regenerate = true;
else if (a === '--undo-wave') opts.undoWave = args[++i];
else if (a === '--source') opts.source = args[++i];
}
return { sub, opts };
}
@@ -159,7 +168,11 @@ export async function runCalibration(
): Promise<void> {
const { opts } = parseArgs(args);
const holder = opts.holder ?? 'garry';
const sourceId = 'default';
// Resolve --source / GBRAIN_SOURCE / .gbrain-source so the (now reachable, #2035)
// calibration command targets the right source in a multi-source brain instead
// of always reading `default`. No signal → 'default' (prior behavior).
const { resolveSourceId } = await import('../core/source-resolver.ts');
const sourceId = await resolveSourceId(engine, opts.source ?? null);
if (opts.undoWave) {
// T17 / D18 CDX-3 — reverse the wave's mutations on canonical state.
+5 -1
View File
@@ -1,6 +1,7 @@
import type { BrainEngine } from '../core/engine.ts';
import { handleToolCall } from '../mcp/server.ts';
import { resolveSourceId } from '../core/source-resolver.ts';
import { bigintToStringReplacer } from '../cli.ts';
/**
* `gbrain call <tool> <json>` trusted local op-dispatch surface.
@@ -49,5 +50,8 @@ export async function runCall(engine: BrainEngine, args: string[]) {
// an explicit/env/dotfile id refers to a non-registered source.
const sourceId = await resolveSourceId(engine, explicitSource);
const result = await handleToolCall(engine, tool, params, { sourceId });
console.log(JSON.stringify(result, null, 2));
// `gbrain call` bypasses cli.ts's op-output normalizer entirely, so this
// exit needs its own bigint-safe replacer — any op returning an int8 column
// (BIGSERIAL id) would otherwise crash plain JSON.stringify (#2450).
console.log(JSON.stringify(result, bigintToStringReplacer, 2));
}
+47 -3
View File
@@ -49,6 +49,12 @@ interface RunOpts {
source?: string;
quiet?: boolean;
json?: boolean;
// v0.42.x — Life Chronicle (#2390): manual `--type event` frontmatter sugar.
who?: string; // comma-separated entity slugs
what?: string;
where?: string;
kind?: string;
depth?: string; // the depth page this event backlinks
}
function parseArgs(args: string[]): RunOpts | { help: true; positional: string | undefined } {
@@ -80,6 +86,12 @@ function parseArgs(args: string[]): RunOpts | { help: true; positional: string |
if (v) opts.source = v;
continue;
}
// v0.42.x — Life Chronicle event sugar.
if (a === '--who') { const v = args[++i]; if (v) opts.who = v; continue; }
if (a === '--what') { const v = args[++i]; if (v) opts.what = v; continue; }
if (a === '--where') { const v = args[++i]; if (v) opts.where = v; continue; }
if (a === '--kind') { const v = args[++i]; if (v) opts.kind = v; continue; }
if (a === '--depth') { const v = args[++i]; if (v) opts.depth = v; continue; }
if (a.startsWith('--')) continue; // unknown flag, ignore
positional.push(a);
}
@@ -132,12 +144,21 @@ Examples:
JOB=$(gbrain capture "..." --quiet)
`;
function defaultSlug(content: string, now: Date = new Date()): string {
// v0.42.x — Life Chronicle (#2390): route the default slug prefix by type so
// `gbrain capture --type diary` lands under life/diary/ and `--type event`
// under life/events/ (matching the chronicle path-prefix inference). Everything
// else keeps the inbox/ default.
function slugPrefixForType(type?: string): string {
if (type === 'diary') return 'life/diary';
if (type === 'event') return 'life/events';
return 'inbox';
}
function defaultSlug(content: string, now: Date = new Date(), type?: string): string {
const y = now.getUTCFullYear();
const m = String(now.getUTCMonth() + 1).padStart(2, '0');
const d = String(now.getUTCDate()).padStart(2, '0');
const hashPrefix = computeContentHash(content).slice(0, 8);
return `inbox/${y}-${m}-${d}-${hashPrefix}`;
return `${slugPrefixForType(type)}/${y}-${m}-${d}-${hashPrefix}`;
}
/**
@@ -245,6 +266,22 @@ function deriveTitle(rawBody: string): string {
* stamps a fresh frontmatter block, and if the body doesn't already look
* like markdown (no `#` heading), wraps it under a `# {title}` heading.
*/
// v0.42.x — Life Chronicle (#2390): assemble the `event:` frontmatter block
// from the --who/--what/--where/--kind/--depth flags (only for --type event).
// Returns undefined when no event flags are set so non-event captures are
// untouched.
function buildEventBlock(opts: RunOpts): Record<string, unknown> | undefined {
if (opts.type !== 'event') return undefined;
const who = opts.who ? opts.who.split(',').map((s) => s.trim()).filter(Boolean) : [];
const block: Record<string, unknown> = {};
if (opts.what) block.what = opts.what;
if (who.length) block.who = who;
if (opts.where) block.where = opts.where;
if (opts.kind) block.kind = opts.kind;
if (opts.depth) block.depth = opts.depth;
return Object.keys(block).length ? block : undefined;
}
export function mergeCaptureFrontmatter(rawBody: string, opts: RunOpts): string {
const nowIso = new Date().toISOString();
// Detect frontmatter: leading `---\n` or `---\r\n`, tolerating leading BOM/whitespace.
@@ -263,6 +300,8 @@ export function mergeCaptureFrontmatter(rawBody: string, opts: RunOpts): string
captured_via: opts.source ?? 'capture-cli',
captured_at: nowIso,
};
const ev = buildEventBlock(opts);
if (ev) fm.event = ev;
const looksMarkdown = /^#{1,6}\s/.test(rawBody.trimStart());
const body = looksMarkdown ? rawBody : `# ${title}\n\n${rawBody}`;
return matter.stringify(body, fm);
@@ -290,6 +329,11 @@ export function mergeCaptureFrontmatter(rawBody: string, opts: RunOpts): string
captured_via: userFm.captured_via ?? opts.source ?? 'capture-cli',
captured_at: userFm.captured_at ?? nowIso,
};
// v0.42.x — merge the event block (user-declared keys win per-key).
const ev = buildEventBlock(opts);
if (ev || userFm.event) {
merged.event = { ...(ev ?? {}), ...((userFm.event as Record<string, unknown>) ?? {}) };
}
return matter.stringify(parsed.content, merged);
}
@@ -439,7 +483,7 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
// The daemon's 24h LRU dedup keys on this hash; identical captures must
// produce identical hashes. The DB content_hash (importFromContent at
// src/core/import-file.ts) gets the same treatment in Phase 3d.
const slug = parsed.slug ?? defaultSlug(normalizedBody);
const slug = parsed.slug ?? defaultSlug(normalizedBody, new Date(), parsed.type);
const fullContent = buildContent(rawBody, parsed);
const capturedAt = new Date().toISOString();
const contentHash = computeContentHash(normalizedBody);
+33 -3
View File
@@ -98,9 +98,25 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
const value = args[2];
if (action === 'get' && key) {
const val = await engine.getConfig(key);
if (val !== null) {
console.log(val);
// #2120: `get` used to read only the DB plane, so a runtime-effective key
// in ~/.gbrain/config.json (or env) reported not-found. Resolve the way
// the runtime does — env/file plane wins over DB (loadConfig() already
// overlays env onto the file) — and report which plane answered on
// stderr, keeping stdout a bare value for scripts.
const filePlane = loadConfig() as Record<string, unknown> | null;
const fileVal = filePlane?.[key];
const dbVal = await engine.getConfig(key);
const val = fileVal !== undefined && fileVal !== null ? fileVal : dbVal;
if (val !== null && val !== undefined) {
console.log(typeof val === 'string' ? val : JSON.stringify(val));
if (fileVal !== undefined && fileVal !== null) {
const shadow = dbVal !== null && dbVal !== undefined
? ' — a DB-plane value also exists and is shadowed at runtime'
: '';
console.error(`[config] source: file/env plane (~/.gbrain/config.json or env)${shadow}`);
} else {
console.error(`[config] source: db plane`);
}
} else {
console.error(`Config key not found: ${key}`);
process.exit(1);
@@ -181,6 +197,20 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
const coverageOverride =
args.includes('--coverage-override') || args.includes('--yes');
// v0.42.42.0 (#2139): validate spend.posture at set time so a typo
// ('tokenMax', 'max') doesn't silently fall back to gated.
if (key === 'spend.posture') {
const { isValidSpendPosture } = await import('../core/spend-posture.ts');
if (!isValidSpendPosture(value)) {
console.error(
`[config] spend.posture must be 'gated' or 'tokenmax' (got '${value}').\n` +
`[config] gbrain config set spend.posture tokenmax # cost gates become informational\n` +
`[config] gbrain config set spend.posture gated # default — gates enforce`,
);
process.exit(1);
}
}
if (key === 'embedding_columns') {
try {
const parsed = JSON.parse(value);
+3 -1
View File
@@ -73,7 +73,9 @@ export const LEARN_INSTRUCTION =
'Once connected, call the `get_brain_identity` tool (whose brain this is), then ' +
'`list_skills` (everything it can do; if it errors, the host has not enabled skill ' +
'publishing — these core tools still work: search, query, get_page, put_page, ' +
'think, find_experts). Always search the brain before answering or writing.';
'think, find_experts). Then call `list_brain_skillpack`: if this brain ships a ' +
'skillpack, ask the user whether to install it (gbrain skillpack scaffold <spec>). ' +
'Always search the brain before answering or writing.';
const SECRET_NOTE =
'Note: that bearer token is a long-lived, full-access secret — keep it private and ' +
+2 -2
View File
@@ -16,6 +16,7 @@
import { readFileSync, existsSync } from 'node:fs';
import { BUILTIN_PATTERNS } from '../core/conversation-parser/builtins.ts';
import { readConversationBodyForParsing } from '../core/conversation-parser/body.ts';
import { parseConversation } from '../core/conversation-parser/parse.ts';
import type { BrainEngine } from '../core/engine.ts';
@@ -170,8 +171,7 @@ async function runScan(
process.exit(2);
}
// Concatenate compiled_truth + timeline (matches the real parser's body shape).
const body = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim();
const body = await readConversationBodyForParsing(engine, page);
const result = parseConversation(body, { page, diagnostic: true });
+487 -89
View File
@@ -1,4 +1,5 @@
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import * as db from '../core/db.ts';
import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
@@ -24,7 +25,10 @@ import { categorizeCheck, type CheckCategory } from '../core/doctor-categories.t
import { rankIssues, type RankedIssue } from '../core/doctor-cause-rank.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import type { DbUrlSource } from '../core/config.ts';
import { gbrainPath } from '../core/config.ts';
import { gbrainPath, loadConfig } from '../core/config.ts';
import { reflexEnabled } from '../core/context/reflex.ts';
import { resolveSocketPath } from '../core/context/resolve-ipc.ts';
import { homedir } from 'os';
import { dirname, isAbsolute, join, resolve as resolvePath } from 'path';
import { fileURLToPath } from 'url';
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
@@ -322,6 +326,70 @@ export async function whoknowsHealthCheck(_engine: BrainEngine): Promise<Check>
}
}
/**
* Doctor check: pgvector availability.
*
* Use the active engine instead of the module-level Postgres singleton.
* PGLite exposes pg_extension through its engine connection, but does not
* connect db.ts's Postgres singleton; using db.getConnection() here turns a
* healthy PGLite brain into a false warning.
*/
export async function pgvectorCheck(engine: BrainEngine): Promise<Check> {
try {
const ext = await engine.executeRaw<{ extname: string }>(
`SELECT extname FROM pg_extension WHERE extname = 'vector'`,
);
if (ext.length > 0) {
return { name: 'pgvector', status: 'ok', message: 'Extension installed' };
}
return { name: 'pgvector', status: 'fail', message: 'Extension not found. Run: CREATE EXTENSION vector;' };
} catch {
return { name: 'pgvector', status: 'warn', message: 'Could not check pgvector extension' };
}
}
/**
* Doctor check: JSONB columns are not double-encoded as strings.
*
* This check is valid on both Postgres and PGLite. Route through
* engine.executeRaw() so embedded PGLite brains are checked through their
* actual connection instead of the unrelated Postgres singleton.
*/
export async function jsonbIntegrityCheck(
engine: BrainEngine,
progress?: Pick<ProgressReporter, 'heartbeat'>,
): Promise<Check> {
try {
const targets: Array<{ table: string; col: string; expected: 'object' | 'array' }> = [
{ table: 'pages', col: 'frontmatter', expected: 'object' },
{ table: 'raw_data', col: 'data', expected: 'object' },
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
{ table: 'files', col: 'metadata', expected: 'object' },
{ table: 'page_versions', col: 'frontmatter', expected: 'object' },
];
let totalBad = 0;
const breakdown: string[] = [];
for (const { table, col } of targets) {
progress?.heartbeat(`jsonb_integrity.${table}.${col}`);
const rows = await engine.executeRaw<{ n: number }>(
`SELECT count(*)::int AS n FROM ${table} WHERE jsonb_typeof(${col}) = 'string'`,
);
const n = Number(rows[0]?.n ?? 0);
if (n > 0) { totalBad += n; breakdown.push(`${table}.${col}=${n}`); }
}
if (totalBad === 0) {
return { name: 'jsonb_integrity', status: 'ok', message: 'All JSONB columns store objects/arrays' };
}
return {
name: 'jsonb_integrity',
status: 'warn',
message: `${totalBad} row(s) double-encoded (${breakdown.join(', ')}). Fix: gbrain repair-jsonb`,
};
} catch {
return { name: 'jsonb_integrity', status: 'warn', message: 'Could not check JSONB integrity' };
}
}
export async function takesWeightGridCheck(engine: BrainEngine): Promise<Check> {
try {
const rows = await engine.executeRaw<{ off_grid: string | number; total: string | number }>(
@@ -507,6 +575,60 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
checks.push({ name: 'schema_version', status: 'warn', message: 'Could not check schema version' });
}
// 2b. #2038: idx_timeline_dedup shape. A renumbered-during-merge migration
// (v102) can be recorded-as-applied without its DDL running, leaving the
// 3-column index in place — every timeline write then fails the 4-column
// ON CONFLICT. The version counter can't see this, so check the index SHAPE.
try {
const { checkTimelineDedupIndex } = await import('../core/timeline-dedup-repair.ts');
const idx = await checkTimelineDedupIndex(engine);
if (!idx.tablePresent || !idx.needsRepair) {
checks.push({
name: 'timeline_dedup_index',
status: 'ok',
message: idx.tablePresent ? 'idx_timeline_dedup has the 4-column shape' : 'no timeline_entries table yet',
});
} else {
checks.push({
name: 'timeline_dedup_index',
status: 'fail',
message:
`idx_timeline_dedup is ${idx.indexPresent ? `(${idx.columns.join(', ')})` : 'absent'}, ` +
`expected (page_id, date, summary, source) — timeline writes are failing (#2038). ` +
`Run \`gbrain apply-migrations --force-schema\` to heal it.`,
});
}
} catch {
checks.push({ name: 'timeline_dedup_index', status: 'warn', message: 'Could not check idx_timeline_dedup shape' });
}
// v0.42.x — Life Chronicle (#2390): orphaned event projections. Reads already
// hide projections whose event page is soft-deleted (read-time correctness);
// this always-run probe surfaces the cleanup backlog. Keyed off the real
// schema (event_page_id), NOT a migration verify-hook, per
// migration-verify-hook-never-runs-on-stamped-brains.
try {
const orphans = await engine.executeRaw<{ n: number }>(
`SELECT count(*)::int AS n FROM timeline_entries te
JOIN pages ep ON ep.id = te.event_page_id
WHERE te.event_page_id IS NOT NULL AND ep.deleted_at IS NOT NULL`,
);
const n = Number(orphans[0]?.n ?? 0);
checks.push(
n === 0
? { name: 'chronicle_projection_health', status: 'ok', message: 'No orphaned event projections' }
: {
name: 'chronicle_projection_health',
status: 'warn',
message:
`${n} timeline projection(s) point to soft-deleted event pages ` +
'(hidden at read time; clean up with `gbrain integrity auto`).',
},
);
} catch {
checks.push({ name: 'chronicle_projection_health', status: 'ok', message: 'no event projections yet' });
}
// 3. Brain score
try {
const health = await engine.getHealth();
@@ -664,6 +786,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
// issue #1801 — wedged_queue (cross-surface parity with buildChecks).
checks.push(await computeWedgedQueueCheck(engine));
// #2194 fix #5 — warn when autopilot fan-out exceeds worker concurrency.
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
// v0.41 Bug 2 / Eng D8 — subagent_health surfaces rate-lease pressure to the operator.
checks.push(await checkSubagentHealth(engine));
@@ -1532,6 +1657,49 @@ export async function computeWedgedQueueCheck(engine: BrainEngine): Promise<Chec
}
}
/**
* #2194 fix #5: warn when autopilot's per-tick fan-out exceeds the worker's
* effective concurrency. Fanning out more cycles than there are worker slots
* guarantees waiters that race the stalled-sweeper a silent misconfig today.
* Advisory (started-event concurrency is fine here; the behavior-changing clamp
* in resolveEffectiveFanoutMax is the one that gates on liveness). Surfaces only
* when a supervisor has actually started (no noise on never-supervised brains).
*/
export async function computeAutopilotFanoutConcurrencyCheck(engine: BrainEngine): Promise<Check> {
if (engine.kind !== 'postgres') {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'PGLite — single-writer, fan-out is 1' };
}
try {
const { resolveFanoutMax, readSupervisorConcurrency } = await import('./autopilot-fanout.ts');
const concurrency = await readSupervisorConcurrency('default');
if (concurrency === null) {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'No supervisor observed — skipping fan-out/concurrency check' };
}
const fanoutMax = await resolveFanoutMax(engine);
const effectiveSlots = Math.max(1, concurrency - 1);
if (fanoutMax > effectiveSlots) {
return {
name: 'autopilot_fanout_concurrency',
status: 'warn',
message:
`autopilot fan-out (${fanoutMax}/tick) exceeds worker concurrency (${concurrency}). ` +
`Surplus cycles queue behind the worker and race the stalled-sweeper. ` +
`Lower fan-out: \`gbrain config set autopilot.fanout_max_per_tick ${effectiveSlots}\`, ` +
`or raise the supervisor's \`--concurrency\` to ${fanoutMax + 1}. ` +
`(The clamp in autopilot does this automatically unless disabled.)`,
details: { fanout_max: fanoutMax, concurrency, effective_slots: effectiveSlots },
};
}
return {
name: 'autopilot_fanout_concurrency',
status: 'ok',
message: `fan-out ${fanoutMax}/tick within worker concurrency ${concurrency}`,
};
} catch (e) {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: `Skipped (${e instanceof Error ? e.message : String(e)})` };
}
}
export async function checkBatchRetryHealth(_engine: BrainEngine): Promise<Check> {
try {
// Codex M-10: surface bad env config at doctor time.
@@ -2346,25 +2514,62 @@ export function checkAutopilotLockScope(): Check {
* but the main work is blocked. Requires explicit heartbeat probe;
* speculation until production data shows the case.
*/
export async function checkStaleLocks(engine: BrainEngine): Promise<Check> {
export async function checkStaleLocks(
engine: BrainEngine,
opts: { fix?: boolean; dryRun?: boolean } = {},
): Promise<Check> {
try {
const { listStaleLocks } = await import('../core/db-lock.ts');
const { listStaleLocks, reapDeadHolderLocks } = await import('../core/db-lock.ts');
// #1972: under `gbrain doctor --fix`, reap dead-holder sync/cycle locks
// using the SAME namespace-scoped, host-scoped, snapshot-matched reaper the
// cycle runs at start. This is the self-heal path for no-autopilot brains: a
// brain that never runs `gbrain dream` never hits the cycle-start sweep, so
// doctor --fix is how its crashed-sync locks get cleared. DB-only, so it's
// orthogonal to (and unaffected by) the skills-dir --fix safety gate above.
// Best-effort: a reap failure falls through to the warn path below.
let reapedIds: string[] = [];
if (opts.fix && !opts.dryRun) {
try {
reapedIds = (await reapDeadHolderLocks(engine)).reapedIds;
} catch { /* fall through; listStaleLocks still surfaces remaining locks */ }
}
const reapedNote = reapedIds.length > 0
? `Reaped ${reapedIds.length} dead-holder lock(s): ${reapedIds.join(', ')}.`
: null;
const stale = await listStaleLocks(engine);
if (stale.length === 0) {
return { name: 'stale_locks', status: 'ok', message: 'No stale locks (no rows with ttl_expires_at < NOW())' };
return {
name: 'stale_locks',
status: 'ok',
message: reapedNote
? `${reapedNote} No stale locks remain.`
: 'No stale locks (no rows with ttl_expires_at < NOW())',
};
}
const lines = stale.slice(0, 10).map(s => {
const ageH = Math.floor(s.age_ms / 3600_000);
const source = s.id.startsWith('gbrain-sync:') ? s.id.slice('gbrain-sync:'.length) : null;
const breakHint = source ? `gbrain sync --break-lock --source ${source}` : `gbrain sync --break-lock`;
let breakHint = 'gbrain doctor';
if (s.id.startsWith('gbrain-sync:')) {
breakHint = `gbrain sync --break-lock --source ${s.id.slice('gbrain-sync:'.length)}`;
} else if (s.id.startsWith('gbrain-cycle:')) {
breakHint = `gbrain dream --break-lock --source ${s.id.slice('gbrain-cycle:'.length)}`;
} else if (s.id === 'gbrain-cycle') {
breakHint = 'gbrain dream --break-lock';
}
return ` ${s.id} (pid ${s.holder_pid} on ${s.holder_host}, age ${ageH}h) → ${breakHint}`;
});
const tail = stale.length > 10 ? ` ... and ${stale.length - 10} more.` : null;
const header = opts.fix
? `${stale.length} stale lock(s) remain that could not be auto-reaped (live holder, cross-host, or within the PID-reuse grace):`
: `${stale.length} stale lock(s) detected (ttl_expires_at < NOW()):`;
return {
name: 'stale_locks',
status: 'warn',
message: [
`${stale.length} stale lock(s) detected (ttl_expires_at < NOW()):`,
reapedNote,
header,
...lines,
tail,
].filter(Boolean).join('\n'),
@@ -2580,7 +2785,7 @@ async function checkEmbeddingEnvOverride(engine: BrainEngine): Promise<Check> {
};
}
async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
export async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
try {
const { classifyCapabilities } = await import('../core/ai/capabilities.ts');
const tierSubagent = await engine.getConfig('models.tier.subagent');
@@ -2641,8 +2846,11 @@ async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
const { loadConfig } = await import('../core/config.ts');
const cfg = loadConfig();
const chatModel = cfg?.chat_model;
const gatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null);
const gatewayLoopEnabled = typeof gatewayLoopRaw === 'string'
&& ['true', '1', 'yes', 'on'].includes(gatewayLoopRaw.trim().toLowerCase());
const { isAnthropicProvider } = await import('../core/model-config.ts');
if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY) {
if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) {
return {
name: 'subagent_capability',
status: 'warn',
@@ -2848,7 +3056,7 @@ export async function computeConversationFactsBacklogCheck(
const typesRaw = await engine.getConfig(
'cycle.conversation_facts_backfill.types',
);
let types = ['conversation', 'meeting', 'slack', 'email'];
let types = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'];
if (typesRaw) {
try {
const parsed = JSON.parse(typesRaw);
@@ -3323,6 +3531,40 @@ export async function checkSyncFreshness(
let hasWarnings = false;
let hasFailures = false;
// BUG 4 (v0.42.x): a source with a LIVE, non-expired per-source sync lock is
// actively syncing RIGHT NOW — it must not read as stale or never-synced.
// The live lock is the only honest "in progress" signal. Checkpoint banking
// is NOT usable: a blocked sync banks the good files then writes no anchor
// (test/sync-resumable-import.serial.test.ts), so banking can't tell
// in-progress from wedged. A blocked/failed sync's process has exited (no
// lock row) and a wedged holder stops refreshing (TTL lapses), so either
// correctly falls through to the stale path and is NEVER masked. Same
// dynamic import as the stale_locks check; any throw (stub engine in unit
// tests, pre-lock-table brain) is swallowed to false, so this can only ADD
// an in-progress verdict, never suppress a real stale one.
// Notes for sources caught actively syncing (surfaced in the result
// message so the operator sees "in progress", not just a silent healthy
// bucket). Empty when nothing is syncing — keeps the steady-state messages
// byte-for-byte unchanged.
const inProgress: string[] = [];
let liveSyncSnap: (sourceId: string) => Promise<{ holder_pid: number; holder_host: string } | null> =
async () => null;
try {
const { inspectLock, syncLockId } = await import('../core/db-lock.ts');
liveSyncSnap = async (sourceId: string) => {
try {
const snap = await inspectLock(engine, syncLockId(sourceId));
return snap && !snap.ttl_expired
? { holder_pid: snap.holder_pid, holder_host: snap.holder_host }
: null;
} catch {
return null;
}
};
} catch {
/* db-lock unavailable — skip in-progress detection, staleness stands. */
}
for (const source of sources) {
// Embed source.id in user-visible messages so `gbrain sync --source <id>`
// matches what the user copy-pastes. Show display name in parens when set.
@@ -3330,6 +3572,15 @@ export async function checkSyncFreshness(
? `'${source.id}' (${source.name})`
: `'${source.id}'`;
// BUG 4: actively syncing (live lock) → healthy, count as synced_recently
// and skip the staleness checks. Keeps the 3-bucket invariant intact.
const liveSnap = await liveSyncSnap(source.id);
if (liveSnap) {
inProgress.push(`${display} sync in progress (pid ${liveSnap.holder_pid} on ${liveSnap.holder_host})`);
synced_recently_count++;
continue;
}
if (!source.last_sync_at) {
issues.push(`Source ${display} has never been synced`);
hasFailures = true;
@@ -3415,12 +3666,15 @@ export async function checkSyncFreshness(
// D6 invariant: every source incremented exactly one bucket.
const details = { unchanged_count, synced_recently_count, stale_count };
// BUG 4: append in-progress context when any source is actively syncing.
// Empty otherwise, so steady-state messages are byte-for-byte unchanged.
const inProgressNote = inProgress.length ? `. ${inProgress.join('; ')}` : '';
if (hasFailures) {
return {
name: 'sync_freshness',
status: 'fail',
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` for each stale source`,
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` for each stale source${inProgressNote}`,
details,
};
}
@@ -3428,7 +3682,7 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'warn',
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` to refresh`,
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` to refresh${inProgressNote}`,
details,
};
}
@@ -3439,7 +3693,7 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'ok',
message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)`,
message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)${inProgressNote}`,
details,
};
}
@@ -3447,14 +3701,14 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'ok',
message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync`,
message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync${inProgressNote}`,
details,
};
}
return {
name: 'sync_freshness',
status: 'ok',
message: `All ${sources.length} federated source(s) synced recently`,
message: `All ${sources.length} federated source(s) synced recently${inProgressNote}`,
details,
};
} catch (e) {
@@ -3881,6 +4135,93 @@ export async function computePoolReapHealthCheck(
return null;
}
/**
* Retrieval Reflex health (#1981). Read-only, fail-open. The deterministic
* pointer layer is on by default; this reports the TRUTH, not an aspiration:
* - config/env disabled warn (pointer layer off)
* - heartbeat fired recently ok, "active" (it's demonstrably working,
* whatever path Postgres/IPC/host)
* - enabled, no recent heartbeat ok if a viable path looks present
* (postgres, or pglite serve socket),
* else warn (likely inactive policy
* skill carries). Never claims a host
* capability it can't observe.
* Policy-skill install state is reported in details (it ships into the HOST
* repo, so absence in gbrain's own skills dir is expected, not a failure).
*/
export function buildRetrievalReflexCheck(skillsDir: string | null): Check {
const name = 'retrieval_reflex_health';
try {
const cfg = loadConfig();
const enabled = reflexEnabled(cfg);
const engineKind = cfg?.engine ?? 'unknown';
const skillInstalled = !!skillsDir && existsSync(join(skillsDir, 'retrieval-reflex', 'SKILL.md'));
if (!enabled) {
return {
name,
status: 'ok',
message: 'retrieval reflex intentionally disabled (config/env) — entity pointer layer off',
details: { enabled: false, engine: engineKind, policy_skill_installed: skillInstalled },
};
}
// Heartbeat is the authority for "is it firing".
const hbPath = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl');
let lastFired: string | null = null;
try {
if (existsSync(hbPath)) {
const lines = readFileSync(hbPath, 'utf8').trim().split('\n').filter(Boolean);
const last = lines.length ? JSON.parse(lines[lines.length - 1]) : null;
if (last && typeof last.ts === 'string') lastFired = last.ts;
}
} catch { /* heartbeat unreadable — treat as never fired */ }
const firedRecently =
!!lastFired && Date.now() - new Date(lastFired).getTime() < 7 * 24 * 60 * 60 * 1000;
// Detect a viable resolve path the doctor CAN see (host ctx.brainQuery is invisible).
let pathDesc: string;
let viablePathVisible: boolean;
if (engineKind === 'postgres') {
pathDesc = 'postgres direct';
viablePathVisible = true;
} else if (engineKind === 'pglite' && cfg?.database_path) {
const socket = resolveSocketPath(cfg.database_path);
viablePathVisible = existsSync(socket);
pathDesc = viablePathVisible ? 'pglite via serve IPC' : 'pglite — serve IPC socket not present';
} else {
pathDesc = `engine ${engineKind}`;
viablePathVisible = false;
}
const runtimeMsg = firedRecently
? `active (last fired ${lastFired})`
: viablePathVisible
? 'enabled; not observed firing yet'
: 'enabled but no observed activity and no visible resolve path (host capability may still supply it; policy skill carries otherwise)';
const status: Check['status'] = firedRecently || viablePathVisible ? 'ok' : 'warn';
const skillHint = skillInstalled
? ''
: ' — policy skill not installed; run `gbrain integrations install retrieval-reflex --target <host-repo>`';
return {
name,
status,
message: `${pathDesc}; ${runtimeMsg}${skillHint}`,
details: {
enabled: true,
engine: engineKind,
path: pathDesc,
fired_recently: firedRecently,
last_fired: lastFired,
policy_skill_installed: skillInstalled,
},
};
} catch (e) {
return { name, status: 'warn', message: `could not check: ${(e as Error).message}` };
}
}
export async function buildChecks(
engine: BrainEngine | null,
args: string[],
@@ -3993,9 +4334,18 @@ export async function buildChecks(
checks.push({ name: 'resolver_health', status: 'warn', message: 'Could not find skills directory' });
}
// 1b. Retrieval Reflex health (#1981, SKILL group — gated). Truthful runtime
// status: the deterministic pointer layer is on by default; the heartbeat file
// (written by the context engine when it actually injects) is the authority for
// "is it firing". The doctor cannot see the OpenClaw host capability directly,
// so it never claims "enabled via host"; it reports observed activity instead.
if (scope === 'all') {
checks.push(buildRetrievalReflexCheck(skillsDir));
}
// 2. Skill conformance (SKILL group — gated)
if (scope === 'all' && skillsDir) {
const conformanceResult = checkSkillConformance(skillsDir);
const conformanceResult = skillConformanceCheck(skillsDir);
checks.push(conformanceResult);
}
@@ -4135,7 +4485,22 @@ export async function buildChecks(
const pidStatus = readSupervisorPid(DEFAULT_PID_FILE);
const supervisorPid = pidStatus.pid;
const running = pidStatus.running;
const pidfileRunning = pidStatus.running;
// issue #2227 fix #1/#3: DEFAULT_PID_FILE is HOME-derived, so a supervisor
// started under a different $HOME reads as "not running" even when healthy.
// Consult the queue-scoped DB singleton lock (#1849, HOME-independent) before
// warning. PID-reuse-safe (isLockHolderLive keys on lock freshness).
let detectedViaDbLock = false;
if (!pidfileRunning && engine) {
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId('default'));
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) detectedViaDbLock = true;
} catch { /* pre-migration / transient: pidfile-only */ }
}
const running = pidfileRunning || detectedViaDbLock;
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
@@ -4183,7 +4548,7 @@ export async function buildChecks(
checks.push({
name: 'supervisor',
status: 'ok',
message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
message: `running=true${detectedViaDbLock ? ' (detected via DB lock; pidfile not at the HOME-derived path)' : ` pid=${supervisorPid}`} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
});
}
}
@@ -4560,9 +4925,10 @@ export async function buildChecks(
// triage the misses interactively.
if (engine) {
try {
const { readConversationBodyForParsing } = await import('../core/conversation-parser/body.ts');
const { parseConversation } = await import('../core/conversation-parser/parse.ts');
const allowedTypes = ['conversation', 'meeting', 'slack', 'email'] as const;
// PageFilters supports singular `type` only; iterate the 4 types
const allowedTypes = ['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'] as const;
// PageFilters supports singular `type` only; iterate the allowed types
// and cap at ~50/each to land at ~200 total max.
const sample: import('../core/types.ts').Page[] = [];
for (const t of allowedTypes) {
@@ -4579,7 +4945,7 @@ export async function buildChecks(
const hitsByPattern: Record<string, number> = {};
let unmatched = 0;
for (const page of sample) {
const body = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`.trim();
const body = await readConversationBodyForParsing(engine, page);
const result = parseConversation(body, { page, noPolish: true, noFallback: true });
const id = result.matched_pattern_id ?? '_no_match';
hitsByPattern[id] = (hitsByPattern[id] ?? 0) + 1;
@@ -4772,13 +5138,7 @@ export async function buildChecks(
checks.push({
name: 'multi_source_drift',
status: 'warn',
message:
`${result.count} page slug(s) appear at 'default' but NOT at the intended source ` +
`(e.g., ${sampleStr}). Two possible causes: (1) pre-v0.30.3 putPage misroutes; ` +
`(2) source X never completed initial sync and the default page is unrelated. ` +
`Verify with 'gbrain sources status', then either re-sync with ` +
`'gbrain sync --source <id> --full' or 'gbrain delete <slug>' if the default-source ` +
`row is the misroute. (A 'gbrain sources rehome' cleanup command is tracked for v0.32.0.)`,
message: multiSourceDriftAdvice(result.count, sampleStr),
});
} else {
checks.push({
@@ -4887,17 +5247,7 @@ export async function buildChecks(
// 4. pgvector extension
progress.heartbeat('pgvector');
try {
const sql = db.getConnection();
const ext = await sql`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
if (ext.length > 0) {
checks.push({ name: 'pgvector', status: 'ok', message: 'Extension installed' });
} else {
checks.push({ name: 'pgvector', status: 'fail', message: 'Extension not found. Run: CREATE EXTENSION vector;' });
}
} catch {
checks.push({ name: 'pgvector', status: 'warn', message: 'Could not check pgvector extension' });
}
checks.push(await pgvectorCheck(engine));
// 4b. PgBouncer / prepared-statement compatibility.
// URL-only inspection — no DB roundtrip — so this is cheap and works
@@ -5480,8 +5830,29 @@ export async function buildChecks(
"SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization')",
))[0]?.count ?? 0;
const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0);
const timelinePct = ((health.timeline_coverage ?? 0) * 100).toFixed(0);
// Compute coverage against eligible entities only — exclude test fixtures
// (`tools/gbrain/test/*`) and template stubs (`templates/new-person`) so
// that brains seeded only with code sources don't get spurious warnings
// about missing link/timeline coverage on pages that are test fixtures, not
// real knowledge entities.
const eligibleStats = (await engine.executeRaw<{ entities: number; linked_from: number; timeline: number }>(
`WITH eligible AS (
SELECT id FROM pages
WHERE type IN ('entity','person','company','organization')
AND slug NOT LIKE 'tools/gbrain/test/%'
AND slug <> 'templates/new-person'
)
SELECT
(SELECT count(*)::int FROM eligible) AS entities,
(SELECT count(DISTINCT from_page_id)::int FROM links WHERE from_page_id IN (SELECT id FROM eligible)) AS linked_from,
(SELECT count(DISTINCT page_id)::int FROM timeline_entries WHERE page_id IN (SELECT id FROM eligible)) AS timeline`,
))[0] ?? { entities: entityCount, linked_from: 0, timeline: 0 };
const eligibleEntityCount = Number(eligibleStats.entities ?? entityCount);
const linkCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.linked_from ?? 0) / eligibleEntityCount : 0;
const timelineCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.timeline ?? 0) / eligibleEntityCount : 0;
const linkPct = (linkCoverage * 100).toFixed(0);
const timelinePct = (timelineCoverage * 100).toFixed(0);
if (entityCount === 0) {
// Markdown-only / journal / wiki brain — no entity pages to compute
// coverage against. Coverage formula is structurally inapplicable.
@@ -5490,13 +5861,19 @@ export async function buildChecks(
status: 'ok',
message: 'No entity pages — graph_coverage not applicable (markdown-only brain)',
});
} else if ((health.link_coverage ?? 0) >= 0.5 && (health.timeline_coverage ?? 0) >= 0.5) {
} else if (eligibleEntityCount === 0) {
checks.push({
name: 'graph_coverage',
status: 'ok',
message: `Only code/test fixture entity pages found (${entityCount}); graph_coverage not applicable`,
});
} else if (linkCoverage >= 0.5 && timelineCoverage >= 0.5) {
checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%` });
} else {
checks.push({
name: 'graph_coverage',
status: 'warn',
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${entityCount} entity pages). Run: gbrain extract all`,
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`,
});
}
@@ -5656,37 +6033,7 @@ export async function buildChecks(
// surface matches `repair-jsonb` (the previous 4-target scan missed a
// repair target, per #254/Codex review).
progress.heartbeat('jsonb_integrity');
try {
const sql = db.getConnection();
const targets: Array<{ table: string; col: string; expected: 'object' | 'array' }> = [
{ table: 'pages', col: 'frontmatter', expected: 'object' },
{ table: 'raw_data', col: 'data', expected: 'object' },
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
{ table: 'files', col: 'metadata', expected: 'object' },
{ table: 'page_versions', col: 'frontmatter', expected: 'object' },
];
let totalBad = 0;
const breakdown: string[] = [];
for (const { table, col } of targets) {
progress.heartbeat(`jsonb_integrity.${table}.${col}`);
const rows = await sql.unsafe(
`SELECT count(*)::int AS n FROM ${table} WHERE jsonb_typeof(${col}) = 'string'`,
);
const n = Number((rows as any)[0]?.n ?? 0);
if (n > 0) { totalBad += n; breakdown.push(`${table}.${col}=${n}`); }
}
if (totalBad === 0) {
checks.push({ name: 'jsonb_integrity', status: 'ok', message: 'All JSONB columns store objects/arrays' });
} else {
checks.push({
name: 'jsonb_integrity',
status: 'warn',
message: `${totalBad} row(s) double-encoded (${breakdown.join(', ')}). Fix: gbrain repair-jsonb`,
});
}
} catch {
checks.push({ name: 'jsonb_integrity', status: 'warn', message: 'Could not check JSONB integrity' });
}
checks.push(await jsonbIntegrityCheck(engine, progress));
// 10b. Takes weight grid integrity (v0.32 — EXP-2).
//
@@ -6032,12 +6379,29 @@ export async function buildChecks(
.slice(0, 3)
.map(([s, n]) => `${s}=${n}`)
.join(', ');
// Audit events are evidence, not automatically breakage. A large code
// source can legitimately emit many WARN events (oversize/markup-heavy)
// while remaining searchable and intentionally flagged. Fail on hard
// dispositions (content actually blocked or hidden); warn on soft
// dispositions or volume. This keeps doctor from treating expected
// code-corpus telemetry as an unhealthy brain.
//
// v0.42 renamed the hard path: a rejected page emits `reject` and a
// quarantined (hidden) junk page emits `quarantine`; `hard_block` is now
// only the pre-v0.42 legacy alias. Counting `hard_block` alone let fresh
// junk-ingest evidence (`reject`/`quarantine`) clear as `ok` whenever
// fewer than 10 events landed. `flag` is a warn disposition (still
// searchable, agent warned on retrieval), so it joins `soft_block`.
const hardBlocked =
summary.by_type.hard_block + summary.by_type.reject + summary.by_type.quarantine;
const softBlocked = summary.by_type.soft_block + summary.by_type.flag;
const status: 'ok' | 'warn' | 'fail' =
events.length >= 100 ? 'fail' : events.length >= 10 ? 'warn' : 'ok';
hardBlocked > 0 ? 'fail' :
(softBlocked > 0 || events.length >= 10) ? 'warn' : 'ok';
checks.push({
name: 'content_sanity_audit_recent',
status,
message: `${events.length} events (hard=${summary.by_type.hard_block} soft=${summary.by_type.soft_block} warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`,
message: `${events.length} events (hard=${hardBlocked} [hard_block=${summary.by_type.hard_block} reject=${summary.by_type.reject} quarantine=${summary.by_type.quarantine}] soft=${softBlocked} [soft_block=${summary.by_type.soft_block} flag=${summary.by_type.flag}] warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`,
});
}
} catch (err) {
@@ -6814,9 +7178,17 @@ export async function buildChecks(
let vanished = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
try {
fs.statSync(r.storage_path);
fs.statSync(abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
@@ -6914,6 +7286,9 @@ export async function buildChecks(
// waiting, zero live-lock active, stale completions) as a health error.
progress.heartbeat('wedged_queue');
checks.push(await computeWedgedQueueCheck(engine));
// #2194 fix #5 — autopilot fan-out vs worker concurrency mismatch.
progress.heartbeat('autopilot_fanout_concurrency');
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
// v0.40.4 graph_signals_coverage — global inbound-link density when
// graph_signals is enabled in the active mode bundle.
progress.heartbeat('graph_signals_coverage');
@@ -6951,7 +7326,7 @@ export async function buildChecks(
checks.push(checkAutopilotLockScope());
// v0.41.6.0 D3 — stale_locks (gbrain_cycle_locks rows with ttl_expires_at < NOW())
progress.heartbeat('stale_locks');
checks.push(await checkStaleLocks(engine));
checks.push(await checkStaleLocks(engine, { fix: doFix, dryRun }));
// v0.38 — cycle_phase_scope (informational; no DB cost)
progress.heartbeat('cycle_phase_scope');
checks.push(checkCyclePhaseScope());
@@ -7015,7 +7390,10 @@ export async function runDoctor(
} catch { /* best-effort */ }
}
process.exit(hasFail ? 1 : 0);
// Use process.exitCode instead of process.exit() so cleanup handlers
// (e.g. Bun unload events, open database connections) still run before
// the process terminates. process.exit() is a hard kill that bypasses them.
setCliExitVerdict(hasFail ? 1 : 0);
}
// ---------------------------------------------------------------------------
@@ -7054,15 +7432,13 @@ function printAutoFixReport(report: AutoFixReport, dryRun: boolean, jsonOutput:
/** Quick skill conformance check — frontmatter + required sections */
function checkSkillConformance(skillsDir: string): Check {
const manifestPath = join(skillsDir, 'manifest.json');
if (!existsSync(manifestPath)) {
return { name: 'skill_conformance', status: 'warn', message: 'manifest.json not found' };
}
export function skillConformanceCheck(skillsDir: string): Check {
try {
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8'));
const skills = manifest.skills || [];
// Host workspaces are allowed to omit a gbrain-specific manifest. Keep
// conformance aligned with resolver_health and skill_brain_first by using
// the canonical fallback that derives entries from direct SKILL.md files.
const manifest = loadOrDeriveManifest(skillsDir);
const skills = manifest.skills;
let passing = 0;
const failing: string[] = [];
@@ -7082,7 +7458,8 @@ function checkSkillConformance(skillsDir: string): Check {
}
if (failing.length === 0) {
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass` };
const derivedNote = manifest.derived ? ' (derived from SKILL.md files)' : '';
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass${derivedNote}` };
}
return {
name: 'skill_conformance',
@@ -7090,7 +7467,7 @@ function checkSkillConformance(skillsDir: string): Check {
message: `${passing}/${skills.length} pass. Failing: ${failing.join(', ')}`,
};
} catch {
return { name: 'skill_conformance', status: 'warn', message: 'Could not parse manifest.json' };
return { name: 'skill_conformance', status: 'warn', message: 'Could not load or derive skills manifest' };
}
}
@@ -7698,3 +8075,24 @@ async function checkSchemaPackSourceDrift(engine: BrainEngine): Promise<Check> {
};
}
}
/**
* #1123 multi_source_drift remediation advice. Exported so the regression
* test can pin that it only references CLI surfaces that actually exist
* (the pre-fix text pointed at 'gbrain sources rehome', which was never
* built, and at 'gbrain delete <slug>' without explaining that delete
* targets the ACTIVE source following it literally on a multi-source
* brain deletes the correctly-routed row).
*/
export function multiSourceDriftAdvice(count: number, sampleStr: string): string {
return (
`${count} page slug(s) appear at 'default' but NOT at the intended source ` +
`(e.g., ${sampleStr}). Two possible causes: (1) pre-v0.30.3 putPage misroutes; ` +
`(2) the intended source never completed initial sync and the default page is unrelated. ` +
`Verify with 'gbrain sources status', then re-sync with ` +
`'gbrain sync --source <id> --full' (reconciles drift without deleting data). ` +
`If a misrouted default-source row remains after re-sync, remove it with ` +
`'GBRAIN_SOURCE=default gbrain delete <slug>' — delete targets the active source, ` +
`so pin it to 'default' explicitly.`
);
}
+65
View File
@@ -76,6 +76,18 @@ interface DreamArgs {
drain: boolean;
/** Drain wallclock budget in seconds. Default 300 (5 min). */
windowSeconds: number;
/**
* issue #2860 `--once`. One-shot bypass of the named `--phase`'s own
* `dream.<phase>.enabled` / `cycle.<phase>.enabled` config gate, for this
* invocation only. Never reads or writes config unlike the old
* "toggle enabled true, run, toggle back to false" workaround, a crash
* mid-run can't leave any global state stuck. Requires an explicit
* `--phase <name>`; bare `--once` is a usage error (there'd be no single
* phase to target). Applies only to phases with a config `.enabled` gate
* (patterns, synthesize, conversation_facts_backfill, enrich_thin,
* skillopt) a no-op for phases that always run when named directly.
*/
once: boolean;
}
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
@@ -105,6 +117,14 @@ function collectFlagValues(args: string[], flag: string): string[] | null {
function parseArgs(args: string[]): DreamArgs {
const phaseIdx = args.indexOf('--phase');
// issue #2860 (Codex P3): captured BEFORE --input/--drain get a chance to
// implicitly default `phase` below, so --once's validation can require
// the user actually TYPED --phase, not merely that some phase ended up
// resolved. Without this, `--input <f> --once` and `--drain --once`
// slip past the "explicit --phase required" contract (the derived
// `phase` value is already non-null by the time that check runs) and
// --once becomes silently ineffective for both.
const phaseWasExplicit = phaseIdx !== -1;
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
let phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
? (rawPhase as CyclePhase)
@@ -214,6 +234,35 @@ function parseArgs(args: string[]): DreamArgs {
}
}
// issue #2860: --once requires an EXPLICIT single --phase target (typed
// by the user, not merely implied by --input/--drain — see
// `phaseWasExplicit` above). Bare `--once` (full/default cycle) has no
// single phase to bypass the gate for, and force-enabling EVERY
// currently-disabled phase at once would be exactly the kind of
// surprise-spend risk the flag exists to prevent. An implicit phase
// (from --input or --drain) is rejected too: --drain returns before
// onceForPhase is ever read, and --input already bypasses the
// synthesize gate on its own, so --once would silently do nothing in
// either case — reject loudly instead of pretending it worked (Codex
// review finding).
//
// Codex review finding: `--help` must short-circuit BEFORE this exits(2),
// matching the "IRON RULE" pinned by test/dream.test.ts's
// "--help --source whatever prints help and exits 0" case — `gbrain
// dream --help --once` (no --phase) must show help, not a usage error.
const once = args.includes('--once');
const wantsHelp = args.includes('--help') || args.includes('-h');
if (once && !phaseWasExplicit && !wantsHelp) {
console.error(
'--once requires an explicit --phase <name> (bypasses that one ' +
'phase\'s dream.<phase>.enabled / cycle.<phase>.enabled gate for ' +
'this run only; never touches config). A phase implied by --input ' +
'or --drain does not count — --once would silently do nothing for ' +
'those. Usage: gbrain dream --phase <name> --once',
);
process.exit(2);
}
return {
json: args.includes('--json'),
dryRun: args.includes('--dry-run'),
@@ -229,6 +278,7 @@ function parseArgs(args: string[]): DreamArgs {
source,
drain,
windowSeconds,
once,
};
}
@@ -310,6 +360,17 @@ Options:
"--dry-run" does NOT mean "zero LLM calls."
--json Emit the CycleReport as JSON (agent-readable)
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
--once With --phase <name>: run that phase once even if its
own dream.<phase>.enabled / cycle.<phase>.enabled
config gate is false. Never reads or writes config
unlike toggling the flag on/off around the run, a
crash mid-invocation can't leave it stuck. Applies to
patterns, synthesize, conversation_facts_backfill,
enrich_thin, skillopt; no-op on phases with no such
gate. Requires an EXPLICIT --phase <name> a phase
implied by --input or --drain does not count (bare
--once, or --once with --input/--drain and no
explicit --phase, is a usage error).
--pull git pull the brain repo before syncing (default: no pull)
--dir <path> Brain directory (default: configured brain). On a
postgres/remote brain with no local checkout, the
@@ -353,6 +414,7 @@ Examples:
gbrain dream
gbrain dream --dry-run --json
gbrain dream --phase lint
gbrain dream --phase patterns --once # run once, ignore dream.patterns.enabled=false
gbrain dream --phase synthesize --input ~/transcripts/2026-04-25.txt
gbrain dream --phase synthesize --from 2026-04-01 --to 2026-04-25
0 2 * * * gbrain dream --json # nightly via cron
@@ -594,6 +656,9 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
synthFrom: opts.from ?? undefined,
synthTo: opts.to ?? undefined,
synthBypassDreamGuard: opts.bypassDreamGuard,
// issue #2860: opts.phase is guaranteed non-null here when opts.once is
// set (parseArgs enforces --once requires --phase).
onceForPhase: opts.once ? opts.phase! : undefined,
});
if (opts.json) {
+354 -34
View File
@@ -9,7 +9,16 @@ import { loadConfig } from '../core/config.ts';
import { slog, serr } from '../core/console-prefix.ts';
import { filterOutEmbedSkipped } from '../core/embed-skip.ts';
import { runSlidingPool } from '../core/worker-pool.ts';
import { isAborted, anySignal } from '../core/abort-check.ts';
import { isAborted, anySignal, AbortError } from '../core/abort-check.ts';
import { type DbPacer, createDbPacer, createNoopPacer, observed } from '../core/db-pacer.ts';
import {
resolvePaceMode,
loadPaceModeConfig,
readPaceEnv,
type PaceKeyOverrides,
} from '../core/pace-mode.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
export interface EmbedOpts {
/** Embed ALL pages (every chunk). */
@@ -71,6 +80,33 @@ export interface EmbedOpts {
* with the internal wall-clock budget timer via `anySignal`.
*/
signal?: AbortSignal;
/**
* DB-contention pacing (paced-backfill). Raw inputs resolved in
* runEmbedCore via env > config > bundle (env beats config = incident
* escape hatch). `perCallMode` is from `--pace[=mode]`; `perCall` from
* `--pace-max-concurrency` etc. Absent resolves from env/config (so a
* queued job paced by config alone still throttles). Mode `off` no-op.
*/
pace?: {
perCallMode?: string;
perCall?: PaceKeyOverrides;
};
/**
* When the pace overrides were SERIALIZED from a background-job payload (not
* typed at an interactive CLI), resolve them at the config tier so
* `GBRAIN_PACE_*` on the worker still overrides at execution (Codex P2). Set
* by the `embed` job handler; unset for interactive CLI runs.
*/
paceFromBackground?: boolean;
/**
* E-2 (paced-backfill): single-flight the stale run by taking the SAME
* per-source lock the `embed-backfill` minion handler uses, so a hand-run CLI
* backfill and a queued job can't grind the same source at once (closing the
* NULLnon-NULL upsert race window that paced longer runs widen). Set
* ONLY by the CLI (`runEmbed`); the minion path already locks. All-source
* runs lock every source in sorted order. dryRun skips it.
*/
singleFlight?: boolean;
}
/**
@@ -94,6 +130,24 @@ export interface EmbedResult {
pages_processed: number;
/** True if this run was a dry-run. */
dryRun: boolean;
/**
* E1 (paced-backfill): end-of-run pacing telemetry. Present ONLY when pacing
* was active (enabled bundle). The number the operator could not get from an
* external wrapper ("zero pauses" "queue safe").
*/
pacing?: {
maxConcurrency: number;
/** In-band latency samples folded into the EWMA. */
samples: number;
/** Final EWMA of observed DB-op latency (ms), or null if no samples. */
ewmaMs: number | null;
/** Cumulative cooperative-sleep time (ms). */
totalSleptMs: number;
/** Number of cooperative sleeps. */
sleeps: number;
/** High-water mark of acquirers blocked on the permit (sync path). */
maxWaiters: number;
};
}
/**
@@ -207,11 +261,118 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
return result;
}
if (opts.all || opts.stale) {
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
batchSize: opts.batchSize,
priority: opts.priority,
catchUp: opts.catchUp,
}, opts.signal);
// E-2 (paced-backfill): CLI single-flight. Take the SAME per-source lock as
// the embed-backfill minion handler so a hand-run backfill and a queued job
// are mutually exclusive per source. All-source runs lock every source in
// sorted (deterministic) order to avoid acquire-order deadlock. Released in
// the finally below. Skipped for dryRun and when the caller didn't opt in
// (cycle / catch-up / sync-auto-embed callers never single-flight).
const sfLocks: DbLockHandle[] = [];
if (opts.singleFlight && opts.stale && !opts.dryRun) {
let lockSourceIds: string[];
if (opts.sourceId) {
lockSourceIds = [opts.sourceId];
} else {
try {
const rows = await engine.listAllSources();
lockSourceIds = rows.map((r) => r.id).sort();
} catch {
lockSourceIds = [];
}
}
for (const sid of lockSourceIds) {
let lock: DbLockHandle | null = null;
try {
lock = await tryAcquireDbLock(engine, embedBackfillLockId(sid), 60);
} catch {
// Fail-open: a lock-subsystem error must not crash a backfill. Drop
// single-flight for this run (release what we took) and proceed.
for (const h of sfLocks) {
try { await h.release(); } catch { /* best-effort */ }
}
sfLocks.length = 0;
break;
}
if (!lock) {
// Another backfill (CLI or job) holds this source. Release what we
// took and bail cleanly rather than racing the upsert path.
for (const h of sfLocks) {
try { await h.release(); } catch { /* best-effort */ }
}
serr(` [embed] another backfill is already running for source "${sid}"; skipping (single-flight).`);
return result;
}
sfLocks.push(lock);
}
}
// Resolve DB-contention pacing (env > config > bundle; env is the
// incident escape hatch). dryRun skips it — no writes to pace. A
// disabled bundle yields a no-op pacer (zero overhead on the hot path).
let pacer: DbPacer = createNoopPacer();
let paceMaxConcurrency: number | undefined;
if (!opts.dryRun) {
try {
const cfg = await loadPaceModeConfig(engine);
const { envMode, envOverrides } = readPaceEnv();
// Codex P2: an interactive CLI flag (--pace) is the most immediate
// intent and sits at the per-call tier (beats env). But a flag
// SERIALIZED into a background job payload must sit at the CONFIG tier
// so GBRAIN_PACE_* on the worker can still override it at execution
// (incident escape hatch). paceFromBackground distinguishes the two.
const fromBg = !!opts.paceFromBackground;
const knobs = resolvePaceMode({
mode: fromBg ? (opts.pace?.perCallMode ?? cfg.mode) : cfg.mode,
configOverrides: fromBg
? { ...cfg.configOverrides, ...(opts.pace?.perCall ?? {}) }
: cfg.configOverrides,
envMode,
envOverrides,
perCallMode: fromBg ? undefined : opts.pace?.perCallMode,
perCall: fromBg ? undefined : opts.pace?.perCall,
});
if (knobs.enabled) {
pacer = createDbPacer({ bundle: knobs });
paceMaxConcurrency = knobs.maxConcurrency;
}
} catch {
// Fail-open: pacing must never break a backfill.
pacer = createNoopPacer();
}
}
try {
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, {
batchSize: opts.batchSize,
priority: opts.priority,
catchUp: opts.catchUp,
pacer,
paceMaxConcurrency,
}, opts.signal);
} finally {
// E1: surface pacing telemetry (human + structured) when pacing was on.
const snap = pacer.snapshot();
if (snap.enabled) {
result.pacing = {
maxConcurrency: snap.maxConcurrency,
samples: snap.sampleCount,
ewmaMs: snap.ewmaMs,
totalSleptMs: snap.totalSleptMs,
sleeps: snap.sleepCount,
maxWaiters: snap.maxWaiters,
};
serr(
` [embed] pacing: cap=${snap.maxConcurrency} samples=${snap.sampleCount} ` +
`ewma=${snap.ewmaMs === null ? 'n/a' : Math.round(snap.ewmaMs) + 'ms'} ` +
`slept=${snap.totalSleptMs}ms/${snap.sleepCount}`,
);
}
pacer.dispose();
// E-2: release single-flight locks (reverse order). Best-effort; the
// lock TTL is the backstop if a release fails.
for (const h of sfLocks.reverse()) {
try { await h.release(); } catch { /* best-effort; TTL covers it */ }
}
}
return result;
}
if (opts.slug) {
@@ -221,6 +382,39 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
}
/**
* Parse the `--pace` family from a CLI arg list. Returns ONLY the explicit
* overrides (CX5: never the full resolved bundle) so they can be serialized
* into a background-job payload and re-resolved (env > config > bundle) at
* execution. Returns undefined when no pace flag is present.
*
* Recognized: `--pace` (bare balanced), `--pace=<mode>`,
* `--pace-max-concurrency=<n>` / `--pace-max-concurrency <n>`.
*/
export function parsePaceArgs(
args: string[],
): { perCallMode?: string; perCall?: PaceKeyOverrides } | undefined {
let perCallMode: string | undefined;
let perCall: PaceKeyOverrides | undefined;
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--pace') {
perCallMode = 'balanced';
} else if (a.startsWith('--pace=')) {
perCallMode = a.slice('--pace='.length) || 'balanced';
} else if (a.startsWith('--pace-max-concurrency=')) {
const n = parseInt(a.slice('--pace-max-concurrency='.length), 10);
if (Number.isFinite(n) && n >= 1) (perCall ??= {}).maxConcurrency = n;
} else if (a === '--pace-max-concurrency') {
const n = parseInt(args[i + 1] ?? '', 10);
if (Number.isFinite(n) && n >= 1) (perCall ??= {}).maxConcurrency = n;
i++; // consume the value token so positional parsing can't read it as a slug (Codex P2)
}
}
if (perCallMode === undefined && perCall === undefined) return undefined;
return { ...(perCallMode !== undefined && { perCallMode }), ...(perCall && { perCall }) };
}
export async function runEmbed(engine: BrainEngine, args: string[]): Promise<EmbedResult | undefined> {
// v0.36+ T7: --background submits via Minion queue, returns job_id to
// stdout, exits. Same semantics in TTY and cron (D9).
@@ -239,6 +433,10 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
dryRun: cleanArgs.includes('--dry-run'),
slugs: slugsI >= 0 ? cleanArgs.slice(slugsI + 1).filter(a => !a.startsWith('--')) : undefined,
sourceId: srcI >= 0 ? cleanArgs[srcI + 1] : undefined,
// CX1+CX5: carry explicit pace overrides into the `embed` job payload
// (the job name CLI --background actually submits). The handler
// re-resolves env > config > bundle at execution.
...(parsePaceArgs(cleanArgs) && { pace: parsePaceArgs(cleanArgs) }),
};
},
source: 'cli',
@@ -262,12 +460,14 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
const catchUp = args.includes('--catch-up');
const pace = parsePaceArgs(args);
let opts: EmbedOpts;
if (slugsIdx >= 0) {
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
} else if (all || stale) {
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp };
// E-2: CLI-only single-flight for stale runs (the minion path locks itself).
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) };
} else {
const slug = args.find(a => !a.startsWith('--'));
if (!slug) {
@@ -416,6 +616,10 @@ async function embedAll(
batchSize?: number;
priority?: 'recent';
catchUp?: boolean;
/** DB-contention pacer (paced-backfill); no-op when pacing is off. */
pacer?: DbPacer;
/** Resolved concurrency cap (E-1: the worker count, no separate permit). */
paceMaxConcurrency?: number;
},
signal?: AbortSignal,
) {
@@ -443,6 +647,10 @@ async function embedAll(
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature, signal);
}
// --all path: pacer (no-op when off). E-1: lower the worker count to the
// resolved cap instead of adding a separate permit.
const pacer = staleOpts?.pacer ?? createNoopPacer();
// v0.31.12: when sourceId is set, scope listPages to that source.
// v0.41 (D8 + Codex r2 #11): apply embed-skip filter via the shared
// helper so the `--all` path honors `frontmatter.embed_skip` the same
@@ -466,7 +674,13 @@ async function embedAll(
// (3000+/min for tier 1 = 50+/sec, 20 parallel is safely below) and
// avoids overwhelming postgres connection pools. Users can tune via
// GBRAIN_EMBED_CONCURRENCY env var based on their tier/infra.
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
// Paced runs lower this to the resolved cap (the real lever vs pooler-slot
// starvation); unpaced keeps the env/default 20. Codex P2: only ever LOWER —
// never raise above an operator's existing env cap.
const BASE_CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
const CONCURRENCY = staleOpts?.paceMaxConcurrency
? Math.min(BASE_CONCURRENCY, staleOpts.paceMaxConcurrency)
: BASE_CONCURRENCY;
async function embedOnePage(page: typeof pages[number]) {
// #1737: bail before doing any work for this page if the run was aborted.
@@ -475,7 +689,7 @@ async function embedAll(
// target the correct (source_id, slug) row, not the 'default' source.
const pageSourceId = page.source_id;
const pageOpts = pageSourceId ? { sourceId: pageSourceId } : undefined;
const chunks = await engine.getChunks(page.slug, pageOpts);
const chunks = await observed(pacer, () => engine.getChunks(page.slug, pageOpts));
const toEmbed = chunks; // staleOnly path handled above via embedAllStale
result.total_chunks += chunks.length;
@@ -511,10 +725,12 @@ async function embedAll(
embedding: embeddingMap.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(page.slug, updated, pageOpts);
await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts));
// v0.41.31: stamp embedding provenance so a later model swap is
// detectable as stale.
await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature });
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
result.embedded += toEmbed.length;
} catch (e: unknown) {
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
@@ -523,6 +739,12 @@ async function embedAll(
processed++;
result.pages_processed++;
onProgress?.(processed, pages.length, result.embedded);
// Cooperative DB-contention pace between pages (no-op when unpaced).
try {
await pacer.pace(signal);
} catch (e) {
if (!(e instanceof AbortError)) throw e;
}
}
// v0.41.15.0: sliding worker pool extracted into src/core/worker-pool.ts.
@@ -576,6 +798,10 @@ async function embedAllStale(
batchSize?: number;
priority?: 'recent';
catchUp?: boolean;
/** DB-contention pacer (paced-backfill); no-op when pacing is off. */
pacer?: DbPacer;
/** Resolved concurrency cap (E-1: the worker count, no separate permit). */
paceMaxConcurrency?: number;
},
signature?: string,
externalSignal?: AbortSignal,
@@ -626,18 +852,43 @@ async function embedAllStale(
// (page_id, chunk_index). Each query finishes in <1s.
// v0.41.18.0 (A13): --batch-size N CLI flag overrides hardcoded 2000 default.
const PAGE_SIZE = staleOpts?.batchSize ?? 2000;
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
// Paced runs lower concurrency to the resolved cap (E-1: worker count IS the
// lever on this single pool, no separate permit). Codex P2: pacing only ever
// LOWERS concurrency — never raise above an operator's existing env cap.
const BASE_CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
const CONCURRENCY = staleOpts?.paceMaxConcurrency
? Math.min(BASE_CONCURRENCY, staleOpts.paceMaxConcurrency)
: BASE_CONCURRENCY;
const pacer = staleOpts?.pacer ?? createNoopPacer();
// D3 + D3a + D8: wall-clock budget. 30 min default; env override.
// v0.41.18.0 (A13): --catch-up removes the wall-clock cap entirely so the
// handler runs until countStaleChunks() returns 0. Use Number.MAX_SAFE_INTEGER
// (effectively unbounded) instead of the 30-min default. The AbortController
// still wraps for SIGINT propagation; just the timer never fires.
const BUDGET_MS = staleOpts?.catchUp
? Number.MAX_SAFE_INTEGER
// #1946: --catch-up removes the wall-clock cap. The prior code set BUDGET_MS =
// Number.MAX_SAFE_INTEGER and passed it to setTimeout — but setTimeout's delay
// is a 32-bit signed int, so MAX_SAFE_INTEGER (9e15) overflows and the timer
// fires almost immediately, aborting catch-up after a single batch. The fix is
// to NOT arm the timer in catch-up at all: the keyset pass below terminates on
// its own (the (page_id, chunk_index) cursor advances monotonically), and
// SIGINT / worker-abort still propagate via externalSignal.
const BUDGET_MS: number | null = staleOpts?.catchUp
? null
: parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10);
const budgetController = new AbortController();
const budgetTimer = setTimeout(() => budgetController.abort(), BUDGET_MS);
const budgetStart = Date.now();
let budgetTimer = BUDGET_MS != null
? setTimeout(() => budgetController.abort(), BUDGET_MS)
: undefined;
// E-4 (paced-backfill): the budget measures WORK, not waiting. After each
// batch, re-arm the timer to fire at start + BUDGET + total-paced-sleep, so a
// contended DB that spends time in pace() sleeps converges instead of exiting
// having embedded little. No-op when unpaced (totalSleptMs stays 0) or in
// catch-up (no budget timer).
const rearmBudgetForPacing = (): void => {
if (BUDGET_MS == null) return;
const slept = pacer.snapshot().totalSleptMs;
if (budgetTimer) clearTimeout(budgetTimer);
const fireInMs = budgetStart + BUDGET_MS + slept - Date.now();
budgetTimer = setTimeout(() => budgetController.abort(), Math.max(0, fireInMs));
};
const budgetSignal = budgetController.signal;
// #1737: the effective signal fires when EITHER the internal wall-clock
// budget OR the caller's abort (worker timeout / lock loss / SIGTERM) fires.
@@ -660,6 +911,37 @@ async function embedAllStale(
let afterUpdatedAt: string | null = null;
let totalChunksLoaded = 0;
let budgetExitNotified = false;
// #1946 (OV2a): track chunks that errored out so a catch-up pass that finishes
// with stale chunks still remaining (un-embeddable for a non-transient reason)
// surfaces that loudly instead of looking like a clean run.
let embedFailures = 0;
// E-3 (paced-backfill): bounded end-of-run re-entry. A longer paced run gives
// a live writer (sync / put_page) more time to insert NEW stale rows BEHIND
// the keyset cursor (TODOS:2301). When the cursor exhausts, re-scan from the
// start — capped at MAX_REENTRIES AND requiring forward progress (a pass that
// embeds 0 while count>0 stops) so a writer outrunning embed can't spin
// forever.
const MAX_REENTRIES = 3;
let reentries = 0;
let lastReentryEmbedded = 0;
const maybeReenter = async (): Promise<boolean> => {
// Scoped to PACED runs: pacing lengthens the run, which is what widens the
// behind-cursor window. Unpaced runs keep prior (single-pass) behavior.
if (!pacer.snapshot().enabled) return false;
if (effectiveSignal.aborted) return false;
if (reentries >= MAX_REENTRIES) return false;
const remaining = await engine.countStaleChunks(sourceOpt);
if (remaining === 0) return false;
if (result.embedded === lastReentryEmbedded) return false; // no forward progress
lastReentryEmbedded = result.embedded;
reentries++;
afterPageId = 0;
afterChunkIndex = -1;
afterUpdatedAt = null;
serr(`\n [embed] re-entry ${reentries}/${MAX_REENTRIES}: ${remaining} stale chunk(s) appeared during the run; rescanning from start.`);
return true;
};
try {
// eslint-disable-next-line no-constant-condition
@@ -675,17 +957,22 @@ async function embedAllStale(
break;
}
const batch = await engine.listStaleChunks({
batchSize: PAGE_SIZE,
afterPageId,
afterChunkIndex,
...(orderBy === 'updated_desc' && {
orderBy,
afterUpdatedAt,
const batch = await observed(pacer, () =>
engine.listStaleChunks({
batchSize: PAGE_SIZE,
afterPageId,
afterChunkIndex,
...(orderBy === 'updated_desc' && {
orderBy,
afterUpdatedAt,
}),
...(sourceId && { sourceId }),
}),
...(sourceId && { sourceId }),
});
if (batch.length === 0) break;
);
if (batch.length === 0) {
if (await maybeReenter()) continue;
break;
}
totalChunksLoaded += batch.length;
// Advance cursor to last row in this batch.
@@ -720,7 +1007,7 @@ async function embedAllStale(
try {
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal });
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await engine.getChunks(slug, { sourceId: keySourceId });
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
const staleIdxToEmbedding = new Map<number, Float32Array>();
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
@@ -732,20 +1019,23 @@ async function embedAllStale(
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
// v0.41.31: stamp provenance after the page's chunks are embedded —
// but only when EVERY chunk was stale (fully re-embedded this pass).
// A partially-stale page keeps preserved chunks of unknown/old
// provenance, so don't claim it's current. (After invalidate, a
// signature-drifted page IS fully stale → this stamps it.)
if (signature && stale.length === existing.length) {
await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature });
await observed(pacer, () =>
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
}
result.embedded += stale.length;
} catch (e: unknown) {
// Budget/abort-fired cancellations are expected on the way out; don't
// spam per-page "Error embedding" lines when we're shutting down.
if (effectiveSignal.aborted) return;
embedFailures++;
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
totalProcessedPages++;
@@ -753,6 +1043,17 @@ async function embedAllStale(
// Use staleCount as the estimated total for progress (not exact after
// pagination starts, but directionally correct).
onProgress?.(totalProcessedPages, Math.ceil(staleCount / PAGE_SIZE) * keys.length, result.embedded);
// Cooperative DB-contention pace between keys (no-op when unpaced).
// E-4 (Codex P1): pace() is subject to the EXTERNAL abort only, NOT the
// wall-clock budget — a contended DB's sleep must not be cut by the
// budget timer before its time is credited. Re-arm the budget right
// after each sleep so accrued sleep never eats into work time.
try {
await pacer.pace(externalSignal);
rearmBudgetForPacing();
} catch (e) {
if (!(e instanceof AbortError)) throw e;
}
}
// v0.41.15.0: migrated to shared runSlidingPool. The pool checks
@@ -768,14 +1069,33 @@ async function embedAllStale(
failureLabel: (key) => key,
});
// E-4: extend the work budget by any paced-sleep time accrued this batch.
rearmBudgetForPacing();
// If we got fewer rows than PAGE_SIZE, we've reached the end.
if (batch.length < PAGE_SIZE) break;
if (batch.length < PAGE_SIZE) {
if (await maybeReenter()) continue;
break;
}
}
} finally {
clearTimeout(budgetTimer);
if (budgetTimer) clearTimeout(budgetTimer);
}
slog(`Embedded ${result.embedded} chunks across ${totalProcessedPages} pages`);
// #1946 (OV2a): a catch-up pass that completed without being aborted but left
// chunks unembedded means those chunks are stuck (a non-transient embed
// failure), not that we ran out of time. Surface it loudly so it doesn't read
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
const remaining = await engine.countStaleChunks(
signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined),
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
}
}
}
/**
+40 -9
View File
@@ -33,7 +33,7 @@ import type { BrainEngine } from '../core/engine.ts';
import type { EnrichCandidate, PageType } from '../core/types.ts';
import { operations } from '../core/operations.ts';
import type { OperationContext } from '../core/operations.ts';
import { isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
import { configureGatewayIfUninitialized, isAvailable, chat, getChatModel, withBudgetTracker } from '../core/ai/gateway.ts';
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
import { hybridSearch } from '../core/search/hybrid.ts';
import { serializeMarkdown } from '../core/markdown.ts';
@@ -508,8 +508,13 @@ export async function runEnrichCore(
// One tracker reference for both the run and the post-hoc overage check.
// External tracker (cycle phase): used as-is, no withBudgetTracker wrap (that
// would REPLACE not stack). Internal: capped at maxCostUsd ?? DEFAULT.
// v0.42.42.0 (#2139): Infinity = explicit "uncapped" (off / tokenmax) → pass
// `undefined` so BudgetTracker runs without a ceiling (NOT raw Infinity, which
// would serialize to null in audit rows). undefined-when-unset still → DEFAULT.
const resolvedCap =
opts.maxCostUsd === Infinity ? undefined : (opts.maxCostUsd ?? DEFAULT_MAX_COST_USD);
const tracker = opts.budgetTracker ?? new BudgetTracker({
maxCostUsd: opts.maxCostUsd ?? DEFAULT_MAX_COST_USD,
maxCostUsd: resolvedCap,
label: `enrich:${sourceId}`,
});
try {
@@ -622,8 +627,15 @@ export function parseArgs(args: string[]): ParsedArgs {
continue;
}
if (a === '--max-usd' || a === '--max-cost-usd') {
const n = parseFloat(args[++i] ?? '');
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
const raw = args[++i] ?? '';
// v0.42.42.0 (#2139): off/unlimited/none → run uncapped (Infinity sentinel;
// mapped to "no BudgetTracker ceiling" in runEnrichCore). Spend still ledgered.
if (['off', 'unlimited', 'none'].includes(raw.trim().toLowerCase())) {
out.maxCostUsd = Infinity;
} else {
const n = parseFloat(raw);
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
}
continue;
}
if (a === '--min-context') {
@@ -795,15 +807,32 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
process.exit(1);
}
// Chat gateway required for non-dry-run.
// Chat gateway is required for non-dry-run. Recover a cold singleton before
// reporting an availability error (#2590).
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
if (!parsed.dryRun && !isAvailable('chat')) {
console.error('Chat gateway unavailable. Configure a chat model (e.g. `gbrain config set chat_model anthropic:claude-haiku-4-5`), or pass --dry-run to preview candidates.');
process.exit(1);
}
// v0.42.42.0 (#2139, D15A): enrich runs UNCAPPED when the operator says cost
// isn't the constraint — either explicit `--max-usd off` (parsed to Infinity)
// or `spend.posture=tokenmax` with no per-call cap. Uncapped → the missing-cap
// refusals lift AND runEnrichCore passes no ceiling to the BudgetTracker (spend
// still ledgered; posture removes the ceiling, not the accounting). An explicit
// finite --max-usd always wins (precedence: per-call > posture).
const explicitOff = parsed.maxCostUsd === Infinity;
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const posture = parsed.dryRun ? 'gated' : await resolveSpendPosture(engine);
const uncapped =
!parsed.dryRun && (explicitOff || (parsed.maxCostUsd === undefined && posture === 'tokenmax'));
if (uncapped) {
console.error(`${explicitOff ? '--max-usd off' : 'spend.posture=tokenmax'}: running uncapped, spend ledgered. docs: docs/operations/spend-controls.md`);
}
// Non-TTY execute without --max-usd or --yes is refused (cost guardrail).
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY) {
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> or --yes.');
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY && !uncapped) {
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> (or `off`), --yes, or set spend.posture=tokenmax.');
process.exit(1);
}
@@ -812,7 +841,7 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
: (await listSources(engine)).map((s) => s.id);
// Dry-run cost preview (TTY) before spending.
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined) {
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined && !uncapped) {
const limit = parsed.limit ?? DEFAULT_LIMIT;
const est = (limit * sourceIds.length * COST_ESTIMATE_PER_PAGE_USD).toFixed(2);
console.error(`About to enrich up to ${limit} page(s) per source across ${sourceIds.length} source(s), est. ~$${est}. Re-run with --max-usd or --yes to confirm.`);
@@ -834,7 +863,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
limit: parsed.limit,
workers: parsed.workers,
model: parsed.model,
maxCostUsd: parsed.maxCostUsd,
// uncapped (off / tokenmax) → Infinity sentinel; runEnrichCore maps it
// to "no BudgetTracker ceiling".
maxCostUsd: uncapped ? Infinity : parsed.maxCostUsd,
minContextChars: parsed.minContextChars,
thinThreshold: parsed.thinThreshold,
reenrichAfterMs: parsed.reenrichAfterMs,
+44
View File
@@ -0,0 +1,44 @@
// v0.42.x — Life Chronicle (#2390) `gbrain eval chronicle` (Phase A.9).
// Deterministic, brings its own in-memory PGLite (no DB, no gateway), so the
// CI fixture gate runs anywhere. Exit 0 only on a perfect score.
import { PGLiteEngine } from '../core/pglite-engine.ts';
import { runChronicleEval } from '../eval/chronicle/harness.ts';
const HELP = `Usage: gbrain eval chronicle [--json]
Deterministic Life Chronicle (#2390) feature eval. Builds a synthetic month
corpus with a known gold chronology + a planted ontology supersession + a
planted conflict, then scores the chronicle layer on: day reconstruction
(intra-day order), last-seen exact date, ontology supersession + --asof
time-travel, contradiction surfacing, and source isolation.
Exit code 0 iff every task passes.
`;
export async function runEvalChronicle(args: string[]): Promise<number> {
if (args.includes('--help') || args.includes('-h')) {
process.stdout.write(HELP);
return 0;
}
const json = args.includes('--json');
const engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
try {
const result = await runChronicleEval(engine);
if (json) {
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
} else {
process.stderr.write(
`[eval chronicle] ${result.passed}/${result.total} tasks passed ` +
`(score ${(result.score * 100).toFixed(0)}%)\n`,
);
for (const t of result.tasks) {
process.stderr.write(` ${t.passed ? 'PASS' : 'FAIL'} ${t.id}${t.detail}\n`);
}
}
return result.score === 1 ? 0 : 1;
} finally {
await engine.disconnect();
}
}
+2
View File
@@ -275,6 +275,7 @@ function configureGatewayForCli(): boolean {
chat_model: undefined,
chat_fallback_chain: undefined,
base_urls: undefined,
provider_chat_options: undefined,
env: { ...process.env },
});
return true;
@@ -286,6 +287,7 @@ function configureGatewayForCli(): boolean {
chat_model: config.chat_model,
chat_fallback_chain: config.chat_fallback_chain,
base_urls: config.provider_base_urls,
provider_chat_options: config.provider_chat_options,
env: { ...process.env },
});
return true;
+44 -1
View File
@@ -20,12 +20,13 @@ import {
export async function runEvalRetrievalQuality(engine: BrainEngine, args: string[]): Promise<void> {
const json = args.includes('--json');
const abRelational = args.includes('--ab-relational');
const sourceIdx = args.indexOf('--source');
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
const fixture = args.find(a => !a.startsWith('--') && a !== sourceId);
if (!fixture) {
console.error('Usage: gbrain eval retrieval-quality <fixture.jsonl> [--json] [--source <id>]');
console.error('Usage: gbrain eval retrieval-quality <fixture.jsonl> [--json] [--source <id>] [--ab-relational]');
process.exit(2);
}
@@ -47,6 +48,48 @@ export async function runEvalRetrievalQuality(engine: BrainEngine, args: string[
return results.map(r => r.slug);
};
// v0.43 — A/B the relational recall arm (off vs on) over the same questions
// in a fixed mode (expansion off, skipCache-equivalent via bare hybridSearch)
// so the delta is purely the arm. Headline: graph-relationship recall@10 lift.
if (abRelational) {
const mk = (relationalRetrieval: boolean): SearchFn => async (q) => {
const results = await hybridSearch(engine, q, {
limit: 10, relationalRetrieval, expansion: false,
...(sourceId ? { sourceId } : {}),
});
return results.map(r => r.slug);
};
const t0 = Date.now();
const off = await runRetrievalQuality(questions, mk(false));
const tMid = Date.now();
const on = await runRetrievalQuality(questions, mk(true));
const tEnd = Date.now();
const fam = (r: typeof off, f: string) => r.families.find(x => x.family === f);
const rel = { off: fam(off, 'graph-relationship'), on: fam(on, 'graph-relationship') };
const payload = {
schema_version: 1 as const,
ab: 'relational',
graph_relationship: {
n: rel.on?.n ?? 0,
recall_at_10: { off: rel.off?.recall_at_10 ?? 0, on: rel.on?.recall_at_10 ?? 0,
delta: (rel.on?.recall_at_10 ?? 0) - (rel.off?.recall_at_10 ?? 0) },
hit_at_3: { off: rel.off?.hit_at_3 ?? 0, on: rel.on?.hit_at_3 ?? 0 },
},
latency_ms: { off_total: tMid - t0, on_total: tEnd - tMid },
families: { off: off.families, on: on.families },
};
if (json) {
console.log(JSON.stringify(payload, null, 2));
} else {
console.log(`Relational A/B — graph-relationship n=${payload.graph_relationship.n}`);
const g = payload.graph_relationship;
console.log(` recall@10: off=${(g.recall_at_10.off * 100).toFixed(0)}% on=${(g.recall_at_10.on * 100).toFixed(0)}% Δ=+${(g.recall_at_10.delta * 100).toFixed(0)}pp`);
console.log(` Hit@3: off=${(g.hit_at_3.off * 100).toFixed(0)}% on=${(g.hit_at_3.on * 100).toFixed(0)}%`);
console.log(` latency: off=${payload.latency_ms.off_total}ms on=${payload.latency_ms.on_total}ms (arm adds ${payload.latency_ms.on_total - payload.latency_ms.off_total}ms over ${payload.graph_relationship.n} queries)`);
}
process.exit(0);
}
const report = await runRetrievalQuality(questions, searchFn);
const gate = evaluateGate(report);
+29 -20
View File
@@ -35,9 +35,10 @@
* `listPages({type, sourceId, limit: PAGE_LIST_BATCH})` so worst
* case is BATCH × 25MB per batch (currently 10 × 25MB = 250MB
* bounded). Per-page body cap drops oversize before parsing.
* - Body read covers compiled_truth + timeline. parseMarkdown splits
* conversation imports across both columns; reading only
* compiled_truth silently drops half on iMessage/Slack imports.
* - Body read prefers frontmatter.raw_transcript when present, then
* falls back to compiled_truth + timeline. Meeting pages often
* store the real turn-by-turn transcript in a sidecar file while
* compiled_truth is just the human summary.
* - Page-global row_num accumulator. facts table unique index is
* (source_id, source_markdown_slug, row_num); per-segment row_num
* would collide on segment 2. Per-page counter increments across
@@ -70,7 +71,7 @@ import {
extractFactsFromTurn,
isFactsExtractionEnabled,
} from '../core/facts/extract.ts';
import { isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
import { configureGatewayIfUninitialized, isAvailable, withBudgetTracker } from '../core/ai/gateway.ts';
import { BudgetTracker, BudgetExhausted } from '../core/budget/budget-tracker.ts';
import { listSources } from '../core/sources-ops.ts';
import {
@@ -80,7 +81,6 @@ import {
} from '../core/op-checkpoint.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions, maybeBackground } from '../core/cli-options.ts';
import { loadConfig } from '../core/config.ts';
import { createHash } from 'crypto';
// v0.41.15.0 (T5): worker-pool primitive + per-source-clamp wrapper +
// per-page advisory lock + delete-orphans-first replay safety. See plan
@@ -140,7 +140,14 @@ export const DEFAULT_MAX_COST_USD = 5.0;
* `--types` flag is an explicit per-run override; cycle config is
* the single source of truth.
*/
export const ALLOWED_TYPES = ['conversation', 'meeting', 'slack', 'email'] as const;
export const ALLOWED_TYPES = [
'conversation',
'meeting',
'slack',
'email',
'imessage',
'imessage-daily',
] as const;
export type AllowedType = (typeof ALLOWED_TYPES)[number];
/**
@@ -286,6 +293,7 @@ import {
parseConversation,
type ParseConversationOpts as OrchestratorParseOpts,
} from '../core/conversation-parser/parse.ts';
import { readConversationBodyForParsing } from '../core/conversation-parser/body.ts';
/**
* v0.41.13.0 back-compat shape for direct callers + the existing
@@ -474,16 +482,6 @@ function pageBodyBytes(page: Page): number {
return Buffer.byteLength(compiled, 'utf8') + Buffer.byteLength(timeline, 'utf8');
}
function readPageBody(page: Page): string {
// F1: read BOTH compiled_truth AND timeline; iMessage importers
// place chronological message stream in timeline.
const compiled = page.compiled_truth ?? '';
const timeline = page.timeline ?? '';
if (!compiled) return timeline;
if (!timeline) return compiled;
return `${compiled}\n\n${timeline}`;
}
// ---------------------------------------------------------------------------
// Types config resolver (Eng-v2 A2 — unified single source of truth).
// ---------------------------------------------------------------------------
@@ -682,7 +680,7 @@ async function processPage(
return { newEndIso: null };
}
const body = readPageBody(page);
const body = await readConversationBodyForParsing(state.engine, page);
// v0.41.13.0: thread the full Page through the orchestrator so D8
// date-derivation chain (frontmatter.date > effective_date >
// '1970-01-01') AND timezone_policy warnings apply. The historical
@@ -765,6 +763,12 @@ async function processPage(
source_markdown_slug: page.slug,
source: PER_SEGMENT_SOURCE_PREFIX,
source_session: sessionId,
// Preserve the conversation's valid time instead of defaulting every
// extracted fact to extraction time. Epoch-anchored parses have no
// trustworthy date, so they retain the existing now() fallback.
...(seg.startIso && !seg.startIso.startsWith('1970-')
? { valid_from: new Date(seg.startIso) }
: {}),
context:
fact.context ?? `from ${page.slug} segment ${seg.startIso}..${seg.endIso}`,
}));
@@ -1077,7 +1081,8 @@ export async function runExtractConversationFactsCore(
}
// Fall through to receipt+rollup write so the partial run is
// still observable in extract_health doctor + extracts/ pages.
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
// ...but not under --dry-run: a preview must not persist cache state.
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ true);
// Return partial result — caller (CLI / Minion) decides how to
// surface. NOT a thrown failure.
return result;
@@ -1089,7 +1094,9 @@ export async function runExtractConversationFactsCore(
// (queryable + citable per D-EXTRACT-17/19) AND UPSERTs the per-day
// rollup row (best-effort cache per F-OUT-19). Both are best-effort —
// failures stderr-warn but never fail the parent operation.
await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
// --dry-run must not persist cache/knowledge state: skip the rollup UPSERT +
// receipt-page write so a preview leaves no extract cache row behind.
if (!dryRun) await writeRunReceiptAndRollup(engine, sourceId, result, /* halted */ false);
return result;
}
@@ -1359,7 +1366,9 @@ export async function runExtractConversationFacts(
process.exit(1);
}
// Chat gateway is required for non-dry-run.
// Chat gateway is required for non-dry-run. Recover a cold singleton before
// reporting an availability error (#2590).
if (!parsed.dryRun && !isAvailable('chat')) configureGatewayIfUninitialized();
if (!parsed.dryRun && !isAvailable('chat')) {
console.error('Chat gateway unavailable. Configure an Anthropic or compatible chat model, or pass --dry-run to preview segmentation.');
process.exit(1);
+110 -8
View File
@@ -29,6 +29,7 @@
*/
import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join, relative, dirname } from 'path';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
import type { PageType } from '../core/types.ts';
@@ -60,6 +61,7 @@ import { createHash } from 'crypto';
// v0.41.15.0 (T7, D9): --workers N for the fs-walk inner loops via the
// shared sliding-pool helper + PGLite-clamp wrapper.
import { runSlidingPool } from '../core/worker-pool.ts';
import { isAborted } from '../core/abort-check.ts';
import { parseWorkers, resolveWorkersWithClamp } from '../core/sync-concurrency.ts';
// Batch size for addLinksBatch / addTimelineEntriesBatch.
@@ -186,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
// call isSyncable at all — so it descended into `node_modules/`, emitted
// markdown files from there, AND ignored the canonical exclusion list
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
// (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor
// subtrees before recursion (saving IO), and isSyncable filters the emit
// set against the canonical markdown-strategy rules.
const files: { path: string; relPath: string }[] = [];
@@ -492,6 +494,38 @@ export function extractTimelineFromContent(content: string, slug: string): Extra
entries.push({ slug, date: match[1], source: 'markdown', summary: match[2].trim(), detail: detail || undefined });
}
// Format 3: Inline citation — [Source: <source>, YYYY-MM-DD]
//
// This is the citation convention gbrain's own quality rules require on
// every brain write (skills/conventions/quality.md), so dated evidence is
// pervasive in curated pages — but until now the extractor could not see
// it, and a page whose dates all live in citations scored zero timeline
// coverage. The entry's summary is the sentence the citation annotates
// (the surrounding line with citation markers stripped).
//
// Lines already captured by Format 1 are skipped: a timeline bullet often
// carries its own [Source: ...] citation, and re-extracting it would file
// a duplicate entry under a different (source, summary) shape that the
// DB-level uniqueness cannot collapse.
const citationPattern = /\[Source:\s*([^\]]+?),\s*(\d{4}-\d{2}-\d{2})\s*\]/g;
const bulletLinePattern = /^-\s+\*\*\d{4}-\d{2}-\d{2}\*\*\s*\|/;
for (const line of content.split(/\r?\n/)) {
if (bulletLinePattern.test(line)) continue;
const lineMatches = [...line.matchAll(citationPattern)];
if (lineMatches.length === 0) continue;
// Strip every citation marker from the line to leave the annotated text.
const summary = line
.replace(/\[Source:[^\]]*\]/g, '')
.replace(/^[-*>#\s]+/, '')
.replace(/\s+/g, ' ')
.trim()
.slice(0, 300);
if (!summary) continue; // a bare citation with no surrounding text is not an event
for (const m of lineMatches) {
entries.push({ slug, date: m[2], source: m[1].trim().slice(0, 200), summary });
}
}
return entries;
}
@@ -526,6 +560,28 @@ export interface ExtractOpts {
* own pagination and stay serial in v0.41.15.0.
*/
workers?: number;
/**
* #1972: cooperative-abort signal. Forwarded into the sliding pool (which
* propagates it to every worker) and checked at the top of each onItem, so a
* cancelled cycle's extract (incremental OR full-walk) relinquishes its
* worker slot well under the 30s force-evict. Honored by the cycle-reachable
* paths: extractForSlugs, extractLinksFromDir, extractTimelineFromDir.
*/
signal?: AbortSignal;
/**
* Brain source id to stamp on extracted fs-walk rows (#1747 / #1503).
*
* The fs-walk extractors build LinkBatchInput / TimelineBatchInput rows
* with no source_id, so addLinksBatch / addTimelineEntriesBatch map
* missing literal 'default'. On a brain whose content lives in a
* non-'default' source (e.g. 'wiki'), the batch INSERT's
* `JOIN pages ON (slug, source_id='default')` drops EVERY row 0
* inserted, no error (the "created 0 from N pages" silent no-op).
* Threading the resolved source id here stamps from/to/origin_source_id
* so the JOIN matches. When undefined, rows fall back to 'default' as
* before (single-'default'-source brains unaffected).
*/
sourceId?: string;
}
/**
@@ -564,7 +620,7 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
// Nothing changed — skip entirely.
return result;
}
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers);
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.links_created = r.links_created;
result.timeline_entries_created = r.timeline_created;
result.pages_processed = r.pages;
@@ -573,12 +629,12 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
// Full walk path: CLI `gbrain extract` or first-run.
if (opts.mode === 'links' || opts.mode === 'all') {
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode, workers);
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.links_created = r.created;
result.pages_processed = r.pages;
}
if (opts.mode === 'timeline' || opts.mode === 'all') {
const r = await extractTimelineFromDir(engine, opts.dir, dryRun, jsonMode, workers);
const r = await extractTimelineFromDir(engine, opts.dir, dryRun, jsonMode, workers, opts.signal, opts.sourceId);
result.timeline_entries_created = r.created;
result.pages_processed = Math.max(result.pages_processed, r.pages);
}
@@ -846,6 +902,17 @@ Status (v0.42):
if (!jsonMode) {
console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`);
}
// #2057 (codex): batch failures are no longer swallowed silently — make
// them visible at the command surface (and non-zero exit) instead of
// printing a clean "N entries" success over failed inserts.
if (r.batch_errors > 0) {
console.error(
`[extract timeline] ${r.batch_errors} batch(es) failed to insert` +
(r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') +
` — timeline is incomplete.`,
);
setCliExitVerdict(1);
}
} else if (byMention || ner) {
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
// gazetteer; saves an entire pass on big brains. When only one
@@ -888,11 +955,21 @@ Status (v0.42):
}
}
} else {
// #1747: resolve the brain source id and thread it into the fs-walk
// extractors so batch rows carry from/to_source_id. Without this they
// default to 'default' and addLinksBatch's JOIN drops every row on a
// non-'default' brain → silent "created 0 from N pages". Resolution
// honors --source-id, then GBRAIN_SOURCE / .gbrain-source /
// registered-path / sole-non-default, mirroring the source-aware
// inline hooks (extractLinksForSlugs) that #1204 confirmed correct.
const { resolveSourceId } = await import('../core/source-resolver.ts');
const resolvedSourceId = await resolveSourceId(engine, sourceIdFilter, brainDir);
result = await runExtractCore(engine, {
mode: subcommand as 'links' | 'timeline' | 'all',
dir: brainDir,
dryRun,
jsonMode,
sourceId: resolvedSourceId,
workers,
});
}
@@ -931,6 +1008,9 @@ async function extractForSlugs(
// shared flush primitive; JS single-threaded event loop makes the
// shared counter increments atomic.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id on batch rows (see ExtractOpts.sourceId).
sourceId?: string,
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
// Build the full slug set for link resolution (fast: just readdir, no file reads)
const allFiles = walkMarkdownFiles(brainDir);
@@ -992,8 +1072,12 @@ async function extractForSlugs(
await runSlidingPool({
items: slugs,
workers,
signal,
failureLabel: (slug) => slug,
onItem: async (slug) => {
// #1972: bail before doing any work for this slug on abort. Trailing
// flushLinks/flushTimeline still commit accumulated rows — no torn write.
if (isAborted(signal)) return;
const relPath = slug + '.md';
const fullPath = join(brainDir, relPath);
try {
@@ -1007,7 +1091,9 @@ async function extractForSlugs(
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
linksCreated++;
} else {
linkBatch.push(link);
linkBatch.push(sourceId
? { ...link, from_source_id: sourceId, to_source_id: sourceId, origin_source_id: sourceId }
: link);
if (linkBatch.length >= BATCH_SIZE) await flushLinks();
}
}
@@ -1020,7 +1106,7 @@ async function extractForSlugs(
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date}${entry.summary}`);
timelineCreated++;
} else {
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail, ...(sourceId ? { source_id: sourceId } : {}) });
if (timelineBatch.length >= BATCH_SIZE) await flushTimeline();
}
}
@@ -1048,6 +1134,10 @@ async function extractLinksFromDir(
engine: BrainEngine, brainDir: string, dryRun: boolean, jsonMode: boolean,
// v0.41.15.0 (T7): in-process worker count. Default 1.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id on batch rows so the
// addLinksBatch JOIN matches non-'default' source pages.
sourceId?: string,
): Promise<{ created: number; pages: number }> {
const files = walkMarkdownFiles(brainDir);
const allSlugs = new Set(files.map(f => pathToSlug(f.relPath)));
@@ -1088,8 +1178,11 @@ async function extractLinksFromDir(
await runSlidingPool({
items: files,
workers,
signal,
failureLabel: (f) => f.relPath,
onItem: async (file) => {
// #1972: bail before this file on abort; trailing flush() commits the batch.
if (isAborted(signal)) return;
try {
const content = readFileSync(file.path, 'utf-8');
const links = await extractLinksFromFile(content, file.relPath, allSlugs, { globalBasename });
@@ -1101,7 +1194,9 @@ async function extractLinksFromDir(
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
created++;
} else {
batch.push(link);
batch.push(sourceId
? { ...link, from_source_id: sourceId, to_source_id: sourceId, origin_source_id: sourceId }
: link);
if (batch.length >= BATCH_SIZE) await flush();
}
}
@@ -1123,6 +1218,10 @@ async function extractTimelineFromDir(
engine: BrainEngine, brainDir: string, dryRun: boolean, jsonMode: boolean,
// v0.41.15.0 (T7): in-process worker count. Default 1.
workers: number = 1,
signal?: AbortSignal,
// #1747/#1503: stamp resolved brain source id so addTimelineEntriesBatch
// matches non-'default' source pages.
sourceId?: string,
): Promise<{ created: number; pages: number }> {
const files = walkMarkdownFiles(brainDir);
@@ -1153,8 +1252,11 @@ async function extractTimelineFromDir(
await runSlidingPool({
items: files,
workers,
signal,
failureLabel: (f) => f.relPath,
onItem: async (file) => {
// #1972: bail before this file on abort; trailing flush() commits the batch.
if (isAborted(signal)) return;
try {
const content = readFileSync(file.path, 'utf-8');
const slug = pathToSlug(file.relPath);
@@ -1166,7 +1268,7 @@ async function extractTimelineFromDir(
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date}${entry.summary}`);
created++;
} else {
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail, ...(sourceId ? { source_id: sourceId } : {}) });
if (batch.length >= BATCH_SIZE) await flush();
}
}
+1 -2
View File
@@ -116,8 +116,7 @@ async function listFiles(engine: BrainEngine, slug?: string) {
console.log(`${rows.length} file(s):`);
for (const row of rows) {
const sizeBytes = row.size_bytes as number | null;
const size = sizeBytes ? `${Math.round(sizeBytes / 1024)}KB` : '?';
const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?';
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
}
}
+1 -1
View File
@@ -39,7 +39,7 @@ if ! command -v gbrain >/dev/null 2>&1; then
exit 0
fi
staged=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\\\\.mdx?$' || true)
staged=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\\.mdx?$' || true)
[ -z "$staged" ] && exit 0
failed=0
+14 -5
View File
@@ -16,6 +16,7 @@
*/
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join, relative, resolve } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { loadConfig, toEngineConfig } from '../core/config.ts';
@@ -29,6 +30,7 @@ import {
type AuditReport,
type AuditFix,
} from '../core/brain-writer.ts';
import { collectGitVisibleFiles } from '../core/git-visible-files.ts';
import { isSyncable, pruneDir, slugifyPath } from '../core/sync.ts';
export async function runFrontmatter(args: string[]): Promise<void> {
@@ -63,7 +65,7 @@ export async function runFrontmatter(args: string[]): Promise<void> {
}
console.error(`Unknown frontmatter subcommand: ${sub}\n`);
printHelp();
process.exitCode = 1;
setCliExitVerdict(1);
}
async function connectEngineForAudit(): Promise<BrainEngine> {
@@ -164,14 +166,14 @@ async function runValidate(rest: string[]): Promise<void> {
}
if (!target) {
console.error('error: gbrain frontmatter validate requires a <path> argument');
process.exitCode = 1;
setCliExitVerdict(1);
return;
}
const resolved = resolve(target);
if (!existsSync(resolved)) {
console.error(`error: path not found: ${target}`);
process.exitCode = 1;
setCliExitVerdict(1);
return;
}
@@ -242,7 +244,7 @@ async function runValidate(rest: string[]): Promise<void> {
}
}
process.exitCode = totalErrors > 0 && !flags.fix ? 1 : 0;
setCliExitVerdict(totalErrors > 0 && !flags.fix ? 1 : 0);
}
/**
@@ -271,6 +273,13 @@ export function collectFiles(
if (st.isFile()) {
return [target];
}
const gitFiles = collectGitVisibleFiles(target, (rel) => isSyncable(rel, { strategy: 'markdown' }));
if (gitFiles) {
if (visitDir) visitDir(target);
return gitFiles;
}
const out: string[] = [];
const stack = [target];
if (visitDir) visitDir(target);
@@ -378,7 +387,7 @@ async function runGenerate(args: string[]): Promise<void> {
if (!targetPath) {
console.error('error: gbrain frontmatter generate requires a <path> argument');
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
process.exitCode = 1;
setCliExitVerdict(1);
return;
}
+159 -13
View File
@@ -11,6 +11,9 @@ import {
isCodeFilePath,
isMarkdownFilePath,
isImageFilePath as isImageFilePathFromSync,
matchesAnyGlob,
pruneDir,
SYNC_SKIP_FILES,
type SyncStrategy,
} from '../core/sync.ts';
import { sortNewestFirst } from '../core/sort-newest-first.ts';
@@ -18,6 +21,7 @@ import {
loadCheckpoint,
saveCheckpoint,
clearCheckpoint,
resolveImportTargetDir,
resumeFilter,
} from '../core/import-checkpoint.ts';
@@ -44,7 +48,25 @@ export interface RunImportResult {
export async function runImport(
engine: BrainEngine,
args: string[],
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string; managedBookmark?: boolean } = {},
opts: {
commit?: string;
strategy?: SyncStrategy;
sourceId?: string;
managedBookmark?: boolean;
/**
* #753/#774: glob patterns to exclude from the import (same semantics as
* `isSyncable`'s `exclude` matched against the dir-relative path).
* Threaded by performFullSync for `gbrain sync --exclude`.
*/
exclude?: string[];
/**
* #753/#774 monorepo subdir-source support: when set, slugs and
* `source_path` are computed relative to this root (the git repo root)
* instead of `dir` (the sync scope), so `wiki/page1.md` lands as slug
* `wiki/page1` consistently across full and incremental sync.
*/
slugRoot?: string;
} = {},
): Promise<RunImportResult> {
const noEmbed = args.includes('--no-embed');
const fresh = args.includes('--fresh');
@@ -166,7 +188,19 @@ export async function runImport(
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--json]');
process.exit(1);
}
const dir: string = dirArg; // narrowed; survives closure capture
// #1728: capture the import target ONCE as an absolute real path. Every
// downstream consumer of `dir` (collection, checkpoint load/save, resume
// filtering) sees the same canonical identity — never the caller's `.`/
// relative spelling, which would make the persisted checkpoint `dir`
// resolve against whatever CWD a later process happens to run from.
let dir: string;
try {
dir = resolveImportTargetDir(dirArg);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Import target is not readable: ${dirArg} (${msg})`);
process.exit(1);
}
// v0.31.2: collect under the right strategy. Pre-fix this called
// collectMarkdownFiles unconditionally — code-strategy first sync
@@ -175,13 +209,30 @@ export async function runImport(
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
const _walkT0 = Date.now();
console.error(`[gbrain phase] import.collect_files start dir=${dir} strategy=${strategy}`);
const allFiles = collectSyncableFiles(dir, { strategy });
let allFiles = collectSyncableFiles(dir, { strategy });
console.error(
`[gbrain phase] import.collect_files done ${Date.now() - _walkT0}ms files=${allFiles.length}`,
);
const fileTypeLabel = strategy === 'code' ? 'code'
: strategy === 'auto' ? 'syncable' : 'markdown';
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
// #753/#774: apply --exclude glob patterns (threaded by performFullSync).
if (opts.exclude && opts.exclude.length > 0) {
const beforeExclude = allFiles.length;
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
console.log(
`Found ${allFiles.length} ${fileTypeLabel} files ` +
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
);
// NAV-4: everything excluded is almost always a mistyped pattern — warn.
if (beforeExclude > 0 && allFiles.length === 0) {
console.warn(
`[gbrain sync] No files matched after applying ${opts.exclude.length} --exclude pattern(s). ` +
`Check your --exclude flags. Patterns: ${JSON.stringify(opts.exclude)}`,
);
}
} else {
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
}
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
// See src/core/sort-newest-first.ts for the policy.
@@ -227,6 +278,11 @@ export async function runImport(
async function processFile(eng: BrainEngine, filePath: string) {
const relativePath = relative(dir, filePath);
// #753/#774: slug + source_path base. When performFullSync syncs a
// monorepo subdir, slugRoot is the git root so slugs stay git-root-
// relative (matching the incremental path's git-diff paths). The
// checkpoint (`completed`) stays dir-relative — resumeFilter's contract.
const importRelPath = opts.slugRoot ? relative(opts.slugRoot, filePath) : relativePath;
// v0.31.2 (D5): per-file slow-path log. Fires only when a single
// file takes >5s. The user's hang surfaces as one file taking
// forever — without this, the agent can't see which file.
@@ -237,8 +293,8 @@ export async function runImport(
// up images when GBRAIN_EMBEDDING_MULTIMODAL=true so this branch is
// unreachable when the gate is off; defense-in-depth check anyway.
const result = isImageFilePath(relativePath) && process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'
? await importImageFile(eng, filePath, relativePath, { noEmbed, sourceId })
: await importFile(eng, filePath, relativePath, { noEmbed, sourceId, activePack: importActivePack });
? await importImageFile(eng, filePath, importRelPath, { noEmbed, sourceId })
: await importFile(eng, filePath, importRelPath, { noEmbed, sourceId, activePack: importActivePack });
const _fileMs = Date.now() - _fileT0;
if (_fileMs > 5000) {
console.error(`[gbrain phase] import.process_file slow ${_fileMs}ms ${relativePath}`);
@@ -254,7 +310,9 @@ export async function runImport(
if (result.error && result.error !== 'unchanged') {
console.error(` Skipped ${relativePath}: ${result.error}`);
// Bug 9 — non-"unchanged" skips carry a real error reason.
failures.push({ path: relativePath, error: result.error });
// #774: ledger paths use the slug base so an incremental sync's
// success at the same (git-root-relative) path clears the row.
failures.push({ path: importRelPath, error: result.error });
} else {
// 'unchanged' or no-error skip: content_hash matched a prior
// successful import, so this file IS done for checkpoint purposes.
@@ -272,7 +330,7 @@ export async function runImport(
}
errors++;
skipped++;
failures.push({ path: relativePath, error: msg });
failures.push({ path: importRelPath, error: msg });
}
processed++;
tickProgress();
@@ -286,6 +344,9 @@ export async function runImport(
catch { /* non-fatal */ }
}
saveCheckpoint(checkpointPath, {
schema_version: 1,
owner: 'gbrain',
kind: 'import',
dir,
completedPaths: Array.from(completed),
timestamp: new Date().toISOString(),
@@ -492,12 +553,39 @@ interface CollectOpts {
* The first-sync walker historically admitted them on markdown too when
* `GBRAIN_EMBEDDING_MULTIMODAL=true`. Codex (C5) flagged the contradiction
* preserve the walker semantic explicitly.
*
* Closes #345: exclude `SYNC_SKIP_FILES` metafiles
* (`README.md` / `index.md` / `log.md` / `schema.md` / `RESOLVER.md`).
* Incremental `sync` skips these via `isSyncable`, but the bulk-import
* walker only filtered by extension so a directory import imported every
* directory README as a page, titled by its folder ("People", "Companies",
* ). Those index-titled pages then trigram-corrupt fuzzy entity resolution
* (any `people/X` slug matches the "People" page) and inflate orphan count.
* Funnel both admission paths through the same metafile exclusion so import
* and sync agree on what is a page.
*/
function isCollectibleForWalker(
path: string,
strategy: SyncStrategy,
multimodalOn: boolean,
): boolean {
// #2607: apply the SAME segment-level prune gate as incremental sync's
// `classifySync` (core/sync.ts). The FS walk below prunes at descent time,
// but the git fast path enumerates via `git ls-files` and historically
// filtered only by extension — so `sync --full` imported (and resurrected
// previously-deleted) pages under dot-dirs / vendored trees that incremental
// sync excludes. Full and incremental must agree on the exclusion set.
// (In the FS-walk route `path` is a basename, so this is the same dot-file
// check pruneDir already applied there — no behavior change on that route.)
const segments = path.split('/');
if (segments.some((seg) => !pruneDir(seg))) return false;
// Metafiles are directory scaffolding (READMEs / index / log / schema /
// resolver), not typed brain pages — same exclusion `sync`'s `isSyncable`
// applies. Guards both the FS-walk and the git-fast-path collection routes.
const basename = segments[segments.length - 1] || '';
if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false;
switch (strategy) {
case 'code':
return isCodeFilePath(path);
@@ -512,6 +600,51 @@ function isCollectibleForWalker(
}
}
/**
* Git-aware fast path for `collectSyncableFiles`. Returns the strategy-filtered
* list of syncable files when `dir` is inside a git work tree (paths absolute,
* sorted), or `null` when `dir` is not a git repo / git is unavailable in
* which case the caller falls back to the recursive FS walk.
*
* Honors `.gitignore` (the whole point): `git ls-files --cached --others
* --exclude-standard` lists tracked + untracked-not-ignored files, so vendored
* / build / generated trees never reach the importer. `-z` (NUL-delimited)
* survives paths with spaces/newlines. Each path is lstat-checked to preserve
* the walker's no-symlink policy and to drop submodule gitlinks (which surface
* as a single non-regular entry).
*/
function gitListSyncableFiles(
dir: string,
strategy: SyncStrategy,
multimodalOn: boolean,
): string[] | null {
let stdout: string;
try {
stdout = execFileSync(
'git',
['-C', dir, 'ls-files', '--cached', '--others', '--exclude-standard', '-z'],
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
);
} catch {
return null; // not a git work tree, or git not on PATH → FS-walk fallback
}
const files: string[] = [];
for (const rel of stdout.split('\0')) {
if (!rel) continue;
if (!isCollectibleForWalker(rel, strategy, multimodalOn)) continue;
const full = join(dir, rel);
let st;
try {
st = lstatSync(full);
} catch {
continue; // ls-files raced a deletion, or unreadable
}
if (st.isSymbolicLink() || !st.isFile()) continue;
files.push(full);
}
return files.sort();
}
/**
* v0.31.2 (codex C4 + C5 + C8): unified walker with five hardenings:
*
@@ -532,6 +665,19 @@ function isCollectibleForWalker(
export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): string[] {
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
const multimodalOn = process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true';
// v0.42.x (#1159 --respect-gitignore / #1483 .gbrainignore): when `dir` is a
// git work tree, enumerate via `git ls-files` so the walk honors
// `.gitignore`. Pre-fix the recursive FS walk below descended into every
// git-ignored tree — `vendor/` (PHP Composer), `storage/`, `public/build/`,
// etc. — so a Laravel/PHP repo's `--strategy code` sync tried to import ~50k
// dependency/build files (and bloated DB + embedding cost on any repo with
// vendored data/fixtures). `--cached --others --exclude-standard` = tracked
// PLUS untracked-not-ignored, so uncommitted source is still indexed. Non-git
// dirs (or git unavailable) fall through to the FS walk below.
const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn);
if (gitFiles) return gitFiles;
const maxDepth = resolveMaxWalkDepth();
const visitedInodes = new Map<string, true>();
const files: string[] = [];
@@ -548,11 +694,11 @@ export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): strin
return;
}
for (const entry of entries) {
// Skip hidden dirs (.git, .claude, .raw, etc.) and `node_modules`/`ops`.
// Same set the legacy walkers honored, surfaced once at the top of
// every iteration.
if (entry.startsWith('.')) continue;
if (entry === 'node_modules' || entry === 'ops') continue;
// Descent-time prune through the canonical gate (single source of truth
// in core/sync.ts) instead of a hand-maintained inline list that drifted
// from it. Skips hidden dirs (`.git`, `.raw`, etc.), `node_modules`,
// `vendor`, `dist`, `build`, `venv` (#2020), `ops`, and git submodules.
if (!pruneDir(entry, d)) continue;
const full = join(d, entry);
let stat;
+12 -2
View File
@@ -6,7 +6,7 @@ import { homedir } from 'os';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts';
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
import { createEngine } from '../core/engine-factory.ts';
import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts';
import { runInitEmbedCheck } from '../core/init-embed-check.ts';
@@ -133,7 +133,11 @@ export async function runInit(args: string[]) {
if (manualUrl) {
databaseUrl = manualUrl;
} else if (isNonInteractive) {
const envUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
// effectiveEnvDatabaseUrl applies the #427 guard: a DATABASE_URL that Bun
// auto-loaded from a .env in cwd must not seed a brain config — that is
// the "init inside a web-app checkout writes the app's DB into
// ~/.gbrain/config.json" failure mode.
const envUrl = effectiveEnvDatabaseUrl();
if (envUrl) {
databaseUrl = envUrl;
} else {
@@ -1452,6 +1456,12 @@ export function reportModStatus(): void {
}
console.log('Resolver: skills/RESOLVER.md');
console.log('Soul audit: run `gbrain soul-audit` to customize agent identity');
// Retrieval Reflex (#1981): the deterministic pointer layer is ON by default
// (no action needed). The policy skill is installed into the HOST repo on
// request — we PRINT the command rather than silently mutating the host repo.
console.log('Retrieval reflex: on by default (entity pointers injected per turn)');
console.log(' Install the policy skill into your agent repo:');
console.log(' gbrain integrations install retrieval-reflex --target <host-repo>');
console.log('');
}
+8 -2
View File
@@ -1438,9 +1438,15 @@ export async function installRecipeIntoHostRepo(
} catch { /* not present */ }
}
if (resolverPath) {
const rowsBlock = `\n\n<!-- gbrain:agent-voice:resolver-rows -->\n` +
// Fence the appended rows by the RECIPE id, not a hardcoded recipe name.
// The pre-v0.42 fence was literally `gbrain:agent-voice:resolver-rows`,
// so any second copy-into-host-repo recipe (e.g. retrieval-reflex) wrote
// an agent-voice-labeled block — and `--refresh`/uninstall keyed on the
// recipe id would never find it. Derive the fence from manifest.recipe.
const fenceId = manifest.recipe || 'gbrain';
const rowsBlock = `\n\n<!-- gbrain:${fenceId}:resolver-rows -->\n` +
manifest.resolver_rows_to_append.map((r) => `- ${r}`).join('\n') +
'\n<!-- /gbrain:agent-voice:resolver-rows -->\n';
`\n<!-- /gbrain:${fenceId}:resolver-rows -->\n`;
fsAppendFileSync(resolverPath, rowsBlock);
} else {
console.warn(
+266 -25
View File
@@ -7,7 +7,8 @@ import type { BrainEngine } from '../core/engine.ts';
import { MinionQueue } from '../core/minions/queue.ts';
import { MinionWorker } from '../core/minions/worker.ts';
import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts';
import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts';
import type { MinionHandler, MinionJob, MinionJobStatus } from '../core/minions/types.ts';
import type { PaceKeyOverrides } from '../core/pace-mode.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
import { parseNiceValue, applyNiceness, getEffectiveNiceness, formatNice } from '../core/minions/niceness.ts';
@@ -21,6 +22,49 @@ function hasFlag(args: string[], flag: string): boolean {
return args.includes(flag);
}
/**
* Long-lived workers outlive operator config changes. Re-stamp the AI gateway
* from DB-backed model config immediately before queued jobs enter gateway-backed
* paths, so a stale process-level default cannot route new work to the wrong
* provider.
*/
async function refreshGatewayForJob(engine: BrainEngine): Promise<void> {
const { reconfigureGatewayWithEngine } = await import('../core/ai/gateway.ts');
await reconfigureGatewayWithEngine(engine);
}
const GATEWAY_REFRESH_JOB_NAMES = new Set([
'embed',
'extract-conversation-facts',
'enrich',
'contextual_reindex_per_chunk',
'autopilot-cycle',
'synthesize',
'patterns',
'consolidate',
'extract_facts',
'extract-atoms-drain',
'embed-backfill',
'extract-takes-from-pages',
'embed-catch-up',
]);
function registerBuiltinJob(
worker: MinionWorker,
engine: BrainEngine,
name: string,
handler: MinionHandler,
): void {
if (!GATEWAY_REFRESH_JOB_NAMES.has(name)) {
worker.register(name, handler);
return;
}
worker.register(name, async (job) => {
await refreshGatewayForJob(engine);
return await handler(job);
});
}
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
* Throws on malformed input (caller should surface the error and exit).
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
@@ -131,9 +175,23 @@ function formatJobDetail(job: MinionJob): string {
return lines.join('\n');
}
export async function runJobs(engine: BrainEngine, args: string[]): Promise<void> {
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
const sub = args[0];
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
// with remote MCP routing (`list`, `get`) so no scratch local engine is
// ever built. Any other subcommand arriving with a null engine is a
// routing bug upstream of this function — refuse instead of crashing
// inside MinionQueue.
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
process.exit(1);
}
// Null only ever reaches the MCP-routed `list`/`get` branches, which
// never touch the engine — narrowed once here so the host-only cases
// below typecheck unchanged.
const engine = engineOrNull as BrainEngine;
if (!sub || sub === '--help' || sub === '-h') {
console.log(`gbrain jobs — Minions job queue
@@ -216,6 +274,8 @@ HANDLER TYPES (built in)
return;
}
// The constructor just stores the reference; on the null (thin-client
// list/get) paths no queue method is ever reached.
const queue = new MinionQueue(engine);
switch (sub) {
@@ -1019,10 +1079,38 @@ HANDLER TYPES (built in)
const pidStatus = readSupervisorPid(pidFile);
const supervisorPid = pidStatus.pid;
const running = pidStatus.running;
const pidfileRunning = pidStatus.running;
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
// issue #2227 fix #1/#3: the pidfile is HOME-derived, so a supervisor
// started under a different $HOME (keeper=/root vs ops=/data) reads as
// "not running" here even when it is healthy — the false signal that
// makes an operator spawn a duplicate. Fall back to the queue-scoped DB
// singleton lock (#1849), the HOME-independent authority. PID-reuse-safe:
// isLockHolderLive keys on lock freshness, never process.kill.
const supQueue = parseFlag(args, '--queue') ?? 'default';
let detectedViaDbLock = false;
let dbLockHolder: { holder_pid: number; holder_host: string } | null = null;
if (!pidfileRunning) {
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId(supQueue));
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) {
detectedViaDbLock = true;
dbLockHolder = { holder_pid: snap.holder_pid, holder_host: snap.holder_host };
}
} catch {
// Pre-migration brains / transient DB errors: fall back to pidfile-only.
}
}
const running = pidfileRunning || detectedViaDbLock;
// Surface the supervisor's recorded config from the latest `started`
// event (concurrency + effective --max-rss) so split-$HOME deployments
// see what the live-but-pidfile-invisible supervisor is running.
const startedEvt = events.filter(e => e.event === 'started').pop() ?? null;
// Shared classifier — same code path runs in `gbrain doctor` so the
// two surfaces cannot drift on what counts as a crash. Supersedes
// v0.35.4.0's binary `classifyWorkerExit({code})` on this surface;
@@ -1037,15 +1125,20 @@ HANDLER TYPES (built in)
nice_requested: w.nice_requested,
nice: w.nice_now,
}));
const supervisorNice = running && supervisorPid !== null
const supervisorNice = pidfileRunning && supervisorPid !== null
? getEffectiveNiceness(supervisorPid)
: null;
const status = {
running,
supervisor_pid: supervisorPid,
detected_via: detectedViaDbLock ? 'db_lock' : (pidfileRunning ? 'pidfile' : null),
supervisor_pid: supervisorPid ?? dbLockHolder?.holder_pid ?? null,
db_lock_holder: dbLockHolder,
pid_file: pidFile,
queue: supQueue,
last_start: lastStart,
concurrency: typeof startedEvt?.concurrency === 'number' ? startedEvt.concurrency : null,
max_rss_mb: typeof startedEvt?.max_rss_mb === 'number' ? startedEvt.max_rss_mb : null,
crashes_24h: summary.total,
clean_exits_24h: summary.clean_exits,
crashes_by_cause: summary.by_cause,
@@ -1057,9 +1150,11 @@ HANDLER TYPES (built in)
if (jsonMode) {
console.log(JSON.stringify(status, null, 2));
} else {
console.log(`Supervisor: ${running ? 'running' : 'not running'}`);
if (supervisorPid) console.log(` PID: ${supervisorPid}`);
const via = detectedViaDbLock ? ' (detected via DB lock; pidfile not found at the configured path)' : '';
console.log(`Supervisor: ${running ? 'running' : 'not running'}${via}`);
if (status.supervisor_pid) console.log(` PID: ${status.supervisor_pid}${detectedViaDbLock ? ` @ ${dbLockHolder?.holder_host}` : ''}`);
console.log(` PID file: ${pidFile}`);
if (detectedViaDbLock && status.concurrency !== null) console.log(` Concurrency: ${status.concurrency}${status.max_rss_mb !== null ? ` (max-rss ${status.max_rss_mb}MB)` : ''}`);
if (lastStart) console.log(` Last start: ${lastStart}`);
console.log(` Crashes (24h): ${summary.total} (runtime=${summary.by_cause.runtime_error} oom=${summary.by_cause.oom_or_external_kill} unknown=${summary.by_cause.unknown} legacy=${summary.by_cause.legacy})`);
console.log(` Clean exits (24h): ${summary.clean_exits}`);
@@ -1387,7 +1482,7 @@ export async function registerBuiltinHandlers(
return { ...result, embed_job_id: embedJobId, embed_skip_reason: embedSkipReason };
});
worker.register('embed', async (job) => {
registerBuiltinJob(worker, engine, 'embed', async (job) => {
const { runEmbedCore } = await import('./embed.ts');
// Primary Minion progress channel is job.updateProgress (DB-backed,
// readable via `gbrain jobs get <id>`). Stderr from the worker daemon
@@ -1398,6 +1493,18 @@ export async function registerBuiltinHandlers(
slugs: Array.isArray(job.data.slugs) ? (job.data.slugs as string[]) : undefined,
all: !!job.data.all,
stale: job.data.all ? false : (job.data.stale !== false),
sourceId: typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined,
// CX1+CX5: pace overrides ride in the job payload as explicit overrides
// only; runEmbedCore re-resolves env > config > bundle at execution so
// GBRAIN_PACE_* still wins during an incident.
...(job.data.pace && typeof job.data.pace === 'object'
? {
pace: job.data.pace as { perCallMode?: string; perCall?: PaceKeyOverrides },
// Serialized from the queued payload → config tier so GBRAIN_PACE_*
// on the worker still wins at execution (Codex P2 escape hatch).
paceFromBackground: true,
}
: {}),
onProgress: (done, total, embedded) => {
// Fire-and-forget: progress updates are best-effort and must not
// block the worker loop.
@@ -1422,7 +1529,7 @@ export async function registerBuiltinHandlers(
// BudgetTracker inside its own process. BudgetExhausted is caught at
// the core level and returned as `result.budget_exhausted: true` (NOT
// a job failure) so the user can resume with a higher cap.
worker.register('extract-conversation-facts', async (job) => {
registerBuiltinJob(worker, engine, 'extract-conversation-facts', async (job) => {
const { runExtractConversationFactsCore } = await import('./extract-conversation-facts.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId) {
@@ -1433,7 +1540,7 @@ export async function registerBuiltinHandlers(
}
const types = Array.isArray(job.data.types)
? (job.data.types as string[]).filter((t) =>
['conversation', 'meeting', 'slack', 'email'].includes(t),
['conversation', 'meeting', 'slack', 'email', 'imessage', 'imessage-daily'].includes(t),
)
: undefined;
const result = await runExtractConversationFactsCore(engine, {
@@ -1456,13 +1563,32 @@ export async function registerBuiltinHandlers(
return result;
});
// v0.42.x (#2390) — Life Chronicle event extraction. NOT protected (bounded
// LLM spend per page; no shell). Enqueued by the put_page chronicle backstop
// and by `gbrain chronicle backfill`. Idempotent (content-addressed event
// slugs + projection upsert), so a retry re-runs to the same state.
worker.register('chronicle_extract', async (job) => {
const slug = typeof job.data.slug === 'string' ? job.data.slug : undefined;
if (!slug) throw new Error('chronicle_extract job requires data.slug');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
const { runChronicleExtract } = await import('../core/chronicle/extract-events.ts');
const { chronicleTz } = await import('../core/chronicle/config.ts');
const tz = await chronicleTz(engine);
return await runChronicleExtract(engine, {
slug,
sourceId,
tz,
signal: (job as { signal?: AbortSignal }).signal,
});
});
// v0.41.39 (#1700) — enrich. NOT in PROTECTED_JOB_NAMES: per-call cost is
// bounded by data.maxCostUsd (default DEFAULT_MAX_COST_USD) and the handler
// re-creates the BudgetTracker in its own process. BudgetExhausted is caught
// at the core level and returned as result.budget_exhausted (NOT a failure).
// Strict per-source: the CLI fans out one job per source when --source is
// omitted, so a job ALWAYS carries data.sourceId.
worker.register('enrich', async (job) => {
registerBuiltinJob(worker, engine, 'enrich', async (job) => {
const { runEnrichCore } = await import('./enrich.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
if (!sourceId) {
@@ -1545,6 +1671,43 @@ export async function registerBuiltinHandlers(
return await runBacklinksCore({ action, dir, dryRun: !!job.data.dryRun });
});
// Local patch 2026-06-11: durable facts:absorb. One-shot CLI processes
// (capture/put/sync) can't finish the extraction chat before their exit
// drain aborts it, so backstop.ts submits this job instead and the
// long-lived worker does the LLM work here. Inline mode: errors throw,
// so minion retry/backoff handles transient gateway failures and real
// failures stay visible in `gbrain jobs list --status failed`.
worker.register('facts-absorb', async (job) => {
const slug = typeof job.data.slug === 'string' ? job.data.slug : '';
if (!slug) throw new Error('facts-absorb job requires data.slug');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : 'default';
const page = await engine.getPage(slug, { sourceId });
if (!page) return { skipped: 'page_missing', slug, sourceId };
const { runFactsBackstop } = await import('../core/facts/backstop.ts');
const KNOWN_SOURCES = ['sync:import', 'mcp:put_page', 'mcp:extract_facts', 'file_upload', 'code_import'] as const;
const source = (KNOWN_SOURCES as readonly string[]).includes(job.data.source as string)
? (job.data.source as typeof KNOWN_SOURCES[number])
: 'mcp:put_page';
return await runFactsBackstop(
{
slug: page.slug,
type: page.type,
compiled_truth: page.compiled_truth,
frontmatter: (page.frontmatter ?? {}) as Record<string, unknown>,
},
{
engine,
sourceId,
sessionId: typeof job.data.sessionId === 'string' ? job.data.sessionId : null,
source,
mode: 'inline',
notabilityFilter: job.data.notabilityFilter === 'high-only' ? 'high-only' : 'all',
visibility: job.data.visibility === 'world' ? 'world' : 'private',
...(typeof job.data.model === 'string' && job.data.model ? { model: job.data.model } : {}),
},
);
});
// Autopilot-cycle handler: delegates to runCycle. Shares the exact same
// phase set and ordering as `gbrain dream` and autopilot's inline path —
// one source of truth for what the brain does overnight.
@@ -1565,13 +1728,13 @@ export async function registerBuiltinHandlers(
const { makeContextualReindexHandler } = await import(
'../core/minions/handlers/contextual-reindex-per-chunk.ts'
);
worker.register('contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
registerBuiltinJob(worker, engine, 'contextual_reindex_per_chunk', makeContextualReindexHandler({ engine }));
}
// derivation); the handler returns { partial, status, report } so
// `gbrain jobs get <id>` shows the full structured report. Does NOT
// throw on partial: a flaky phase must not block every future cycle.
worker.register('autopilot-cycle', async (job) => {
registerBuiltinJob(worker, engine, 'autopilot-cycle', async (job) => {
const { runCycle } = await import('../core/cycle.ts');
// v0.41.30 (T2): fall back to null (NOT cwd '.') when no repo is configured.
// The queued cycle is the same primitive `gbrain dream` uses; a checkout-less
@@ -1594,6 +1757,13 @@ export async function registerBuiltinHandlers(
// archived between fan-out and worker claim, skip cleanly.
const rawSourceId = job.data.source_id;
let sourceId: string | undefined;
// issue #2227/#2194 (TODOS:634, codex #8): a per-source cycle must run its
// FILESYSTEM phases (sync/lint/extract) against the SOURCE's own checkout,
// not the global brain's. Pre-fix it inherited `repoPath` (the default
// checkout) while writing DB freshness for `source_id` — mixed scope that
// made cooldown/freshness attribute to the wrong source. We resolve the
// source's `local_path` here and use it as the cycle's brainDir below.
let sourceLocalPath: string | null = null;
if (rawSourceId !== undefined && rawSourceId !== null) {
if (typeof rawSourceId !== 'string') {
throw new Error(`autopilot-cycle: invalid source_id (not a string): ${JSON.stringify(rawSourceId)}`);
@@ -1607,9 +1777,10 @@ export async function registerBuiltinHandlers(
}
// Archive recheck (codex r1 P1-5): cheap pre-cycle lookup. Returns
// immediately if source is gone or archived; runCycle never even
// acquires a lock.
const rows = await engine.executeRaw<{ archived: boolean | null }>(
`SELECT archived FROM sources WHERE id = $1`,
// acquires a lock. Also fetches local_path so FS phases bind to the
// source's own checkout (the #2227/#2194 mixed-scope fix).
const rows = await engine.executeRaw<{ archived: boolean | null; local_path: string | null }>(
`SELECT archived, local_path FROM sources WHERE id = $1`,
[rawSourceId],
);
if (rows.length === 0) {
@@ -1627,8 +1798,17 @@ export async function registerBuiltinHandlers(
};
}
sourceId = rawSourceId;
sourceLocalPath = typeof rows[0].local_path === 'string' && rows[0].local_path.length > 0
? rows[0].local_path
: null;
}
// Effective checkout for FS phases. For a per-source cycle, bind to the
// SOURCE's local_path (or null → skip FS phases for a pure-DB source);
// NEVER fall through to the global repoPath, which would run sync/lint
// against the wrong tree. Legacy (no source_id) keeps the global repoPath.
const effectiveBrainDir: string | null = sourceId ? sourceLocalPath : repoPath;
// Allow callers to select phases via job data (e.g. skip embed for
// fast cycles). Validates against ALL_PHASES to prevent injection.
const { ALL_PHASES } = await import('../core/cycle.ts');
@@ -1640,10 +1820,26 @@ export async function registerBuiltinHandlers(
// Pull default: legacy `true` for back-compat; explicit boolean wins.
const pull = typeof job.data.pull === 'boolean' ? job.data.pull : true;
// #2194 fix #2 / codex #5 (D4): claim-time cooldown guard. A job already
// queued or retrying (max_attempts:2) can reach the worker after the
// dispatch gate decided to back this source off. Skip it here as a NO-OP
// (status 'skipped', NOT a failure — a failure would re-arm the cooldown).
if (sourceId) {
const { isSourceInCooldown } = await import('./autopilot-fanout.ts');
if (await isSourceInCooldown(engine, sourceId)) {
return {
partial: false,
status: 'skipped',
report: { reason: 'source_in_cooldown', source_id: sourceId },
};
}
}
const report = await runCycle(engine, {
brainDir: repoPath,
brainDir: effectiveBrainDir,
pull,
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
...(sourceId ? { sourceId } : {}),
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
yieldBetweenPhases: async () => {
@@ -1659,6 +1855,50 @@ export async function registerBuiltinHandlers(
};
});
// #2194 fix #3 / #2227 bug #3 — brain-wide maintenance. Runs the `global`
// cycle phases (embed, orphans, purge, resolve_symbol_edges, grade_takes,
// calibration_profile, synthesize_concepts, skillopt) ONCE per window instead
// of N times concurrently across per-source cycles (the 4→10GB RSS blowout).
// No source_id → uses the legacy global cycle lock; stamps autopilot.last_global_at
// on success so the dispatch gate backs off.
worker.register('autopilot-global-maintenance', async (job) => {
const { runCycle, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY, ALL_PHASES } = await import('../core/cycle.ts');
const repoPath: string | null = typeof job.data.repoPath === 'string'
? job.data.repoPath
: (await engine.getConfig('sync.repo_path')) ?? null;
const validPhases = new Set(ALL_PHASES);
const requested = Array.isArray(job.data.phases)
? (job.data.phases as string[]).filter((p) => validPhases.has(p as never))
: GLOBAL_PHASES;
const phases = (requested.length > 0 ? requested : GLOBAL_PHASES) as typeof GLOBAL_PHASES;
const report = await runCycle(engine, {
brainDir: repoPath,
pull: false, // brain-wide DB/maintenance work never git-pulls
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
phases,
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
});
// Stamp last_global_at only on a non-failed run so a failed pass stays stale
// and re-dispatches next tick (self-healing retry).
if (report.status === 'ok' || report.status === 'clean' || report.status === 'partial') {
try {
await engine.setConfig(LAST_GLOBAL_AT_KEY, new Date().toISOString());
} catch (e) {
console.warn(`[autopilot-global-maintenance] failed to stamp last_global_at: ${e instanceof Error ? e.message : String(e)}`);
}
}
return {
partial: report.status === 'partial' || report.status === 'failed',
status: report.status,
report,
};
});
// Shell handler is always registered. Runtime env guard lives inside the
// handler so claimed jobs emit a clear rejection log on workers missing
// GBRAIN_ALLOW_SHELL_JOBS=1.
@@ -1783,17 +2023,18 @@ export async function registerBuiltinHandlers(
brainDir: repoPath,
phases: [phase as any],
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
});
return { phase, status: report.status, report };
};
// PROTECTED — internally spawn subagent children
worker.register('synthesize', makePhaseHandler('synthesize'));
worker.register('patterns', makePhaseHandler('patterns'));
worker.register('consolidate', makePhaseHandler('consolidate'));
registerBuiltinJob(worker, engine, 'synthesize', makePhaseHandler('synthesize'));
registerBuiltinJob(worker, engine, 'patterns', makePhaseHandler('patterns'));
registerBuiltinJob(worker, engine, 'consolidate', makePhaseHandler('consolidate'));
// Open — DB writes only, no LLM spend
worker.register('extract_facts', makePhaseHandler('extract_facts'));
registerBuiltinJob(worker, engine, 'extract_facts', makePhaseHandler('extract_facts'));
worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges'));
worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight'));
@@ -1803,7 +2044,7 @@ export async function registerBuiltinHandlers(
// window / defer behavior. On LockUnavailableError (the routine cycle holds
// the per-source lock) the job completes `{ deferred: true }` and retries
// next tick instead of failing — cooperative interleave (CODEX accepted).
worker.register('extract-atoms-drain', async (job) => {
registerBuiltinJob(worker, engine, 'extract-atoms-drain', async (job) => {
const { runExtractAtomsDrainForSource } = await import('../core/cycle/extract-atoms-drain.ts');
const { LockUnavailableError } = await import('../core/db-lock.ts');
const sourceId = typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined;
@@ -1831,7 +2072,7 @@ export async function registerBuiltinHandlers(
// Cost-bounded via D6 ($10/job BudgetTracker) + D19 (source-level cooldown
// + 24h rolling cap, gated at submit time). NOT in PROTECTED_JOB_NAMES —
// embedding-only spend, no API-by-the-minute risk like subagent.
worker.register('embed-backfill', async (job) => {
registerBuiltinJob(worker, engine, 'embed-backfill', async (job) => {
const { makeEmbedBackfillHandler } = await import('../core/minions/handlers/embed-backfill.ts');
return await makeEmbedBackfillHandler(engine)(job);
});
@@ -1852,7 +2093,7 @@ export async function registerBuiltinHandlers(
// (LLM-bearing). Two-gate consent enforced at the handler boundary:
// refuses to run unless takes.bootstrap_enabled config is true, even
// when allowProtectedSubmit was set at queue.add time.
worker.register('extract-takes-from-pages', async (job) => {
registerBuiltinJob(worker, engine, 'extract-takes-from-pages', async (job) => {
const { extractTakesFromPages } = await import('../core/extract-takes-from-pages.ts');
const data = (job.data ?? {}) as { sourceId?: string; maxPages?: number };
const bootstrapCfg = await engine.getConfig('takes.bootstrap_enabled');
@@ -1879,7 +2120,7 @@ export async function registerBuiltinHandlers(
// remediation pipeline. Wraps runEmbedCore with stale + catchUp + the
// priority/batchSize the recommendation supplies. NOT in
// PROTECTED_JOB_NAMES (embedding spend only).
worker.register('embed-catch-up', async (job) => {
registerBuiltinJob(worker, engine, 'embed-catch-up', async (job) => {
const { runEmbedCore } = await import('./embed.ts');
const data = (job.data ?? {}) as {
sourceId?: string;
+19 -1
View File
@@ -18,6 +18,7 @@
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
import { join, relative } from 'path';
import { isAborted } from '../core/abort-check.ts';
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
import {
assessContentSanity,
@@ -45,6 +46,7 @@ const FRONTMATTER_RULE_NAMES: Record<ParseValidationCode, string> = {
SLUG_MISMATCH: 'frontmatter-slug-mismatch',
NULL_BYTES: 'frontmatter-null-bytes',
NESTED_QUOTES: 'frontmatter-nested-quotes',
NON_STRING_FIELD: 'frontmatter-non-string-field',
EMPTY_FRONTMATTER: 'frontmatter-empty',
};
@@ -405,6 +407,13 @@ export interface LintOpts {
* create + disconnect a competing module-style engine that nulls the
* shared db singleton mid-cycle. */
engine?: BrainEngine;
/**
* #1972: cooperative-abort signal. lint's per-page work is synchronous, so
* without a periodic yield the event loop can't deliver an abort and a
* very large lint would block past the worker's 30s force-evict. The loop
* yields + checks this every 200 pages.
*/
signal?: AbortSignal;
}
export interface LintResult {
@@ -443,7 +452,16 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
let totalFixed = 0;
let pagesWithIssues = 0;
for (const page of pages) {
for (let idx = 0; idx < pages.length; idx++) {
const page = pages[idx];
// #1972: every 200 pages, yield to the event loop and honor abort. The
// yield is what lets the abort signal actually fire (the rest of the loop
// is synchronous); the break returns a valid partial LintResult since each
// page is independently read + written.
if (idx > 0 && idx % 200 === 0) {
if (isAborted(opts.signal)) break;
await new Promise<void>((resolve) => setImmediate(resolve));
}
const content = readFileSync(page, 'utf-8');
const issues = lintContent(content, isSingleFile ? page : relative(opts.target, page), lintOpts);
if (issues.length === 0) continue;
+82 -4
View File
@@ -8,10 +8,12 @@
*/
import { createEngine } from '../core/engine-factory.ts';
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
import type { BrainEngine } from '../core/engine.ts';
import type { EngineConfig } from '../core/types.ts';
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
import { createHash } from 'crypto';
import { resolve } from 'path';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -49,12 +51,27 @@ function getManifestPath(): string {
return gbrainPath('migrate-manifest.json');
}
interface MigrateManifest {
export interface MigrateManifest {
completed_slugs: string[];
target_engine: string;
target_id?: string;
schema_version?: number;
started_at: string;
}
export function migrationTargetId(config: EngineConfig): string {
const locator = config.engine === 'postgres'
? config.database_url ?? ''
: resolve(config.database_path ?? gbrainPath('brain.pglite'));
return createHash('sha256')
.update(JSON.stringify([config.engine, locator]))
.digest('hex');
}
export function manifestMatchesTarget(manifest: MigrateManifest, targetId: string): boolean {
return manifest.schema_version === 2 && manifest.target_id === targetId;
}
function loadManifest(): MigrateManifest | null {
const path = getManifestPath();
if (!existsSync(path)) return null;
@@ -74,6 +91,58 @@ function clearManifest(): void {
if (existsSync(path)) unlinkSync(path);
}
interface MigratedSourceRow {
id: string;
name: string;
local_path: string | null;
last_commit: string | null;
last_sync_at: Date | string | null;
config_json: string;
archived: boolean;
archived_at: Date | string | null;
archive_expires_at: Date | string | null;
contextual_retrieval_mode: string | null;
trust_frontmatter_overrides: boolean;
newest_content_at: Date | string | null;
created_at: Date | string;
}
export async function copyMigrationSources(source: BrainEngine, target: BrainEngine): Promise<void> {
const sources = await source.executeRaw<MigratedSourceRow>(`
SELECT id, name, local_path, last_commit, last_sync_at, config::text AS config_json, archived,
archived_at, archive_expires_at, contextual_retrieval_mode,
trust_frontmatter_overrides, newest_content_at, created_at
FROM sources
ORDER BY (id = 'default') DESC, id`);
for (const row of sources) {
await target.executeRaw(`
INSERT INTO sources
(id, name, local_path, last_commit, last_sync_at, config, archived,
archived_at, archive_expires_at, contextual_retrieval_mode,
trust_frontmatter_overrides, newest_content_at, created_at)
VALUES ($1, $2, $3, $4, $5, $6::text::jsonb, $7, $8, $9, $10, $11, $12, $13)
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
local_path = EXCLUDED.local_path,
last_commit = EXCLUDED.last_commit,
last_sync_at = EXCLUDED.last_sync_at,
config = EXCLUDED.config,
archived = EXCLUDED.archived,
archived_at = EXCLUDED.archived_at,
archive_expires_at = EXCLUDED.archive_expires_at,
contextual_retrieval_mode = EXCLUDED.contextual_retrieval_mode,
trust_frontmatter_overrides = EXCLUDED.trust_frontmatter_overrides,
newest_content_at = EXCLUDED.newest_content_at,
created_at = EXCLUDED.created_at`, [
row.id, row.name, row.local_path, row.last_commit, row.last_sync_at,
row.config_json, row.archived, row.archived_at, row.archive_expires_at,
row.contextual_retrieval_mode, row.trust_frontmatter_overrides,
row.newest_content_at, row.created_at,
]);
}
}
export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]): Promise<void> {
const opts = parseArgs(args);
const config = loadConfig();
@@ -91,7 +160,8 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
// Build target config
const targetConfig: EngineConfig = { engine: opts.targetEngine };
if (opts.targetEngine === 'postgres') {
targetConfig.database_url = opts.targetUrl || process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
// #427 guard: don't let a cwd-.env DATABASE_URL become a migration target.
targetConfig.database_url = opts.targetUrl || effectiveEnvDatabaseUrl();
if (!targetConfig.database_url) {
console.error('Target is Supabase but no connection string provided. Use: --url <connection_string>');
process.exit(1);
@@ -99,6 +169,7 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
} else {
targetConfig.database_path = opts.targetPath || gbrainPath('brain.pglite');
}
const targetId = migrationTargetId(targetConfig);
// Connect to target
console.log(`Connecting to target (${opts.targetEngine})...`);
@@ -128,7 +199,7 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
// Load or create manifest for resume
let manifest = loadManifest();
if (manifest && manifest.target_engine !== opts.targetEngine) {
if (manifest && !manifestMatchesTarget(manifest, targetId)) {
console.log('Previous migration was to a different target. Starting fresh.');
manifest = null;
}
@@ -143,10 +214,17 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
manifest = {
completed_slugs: [],
target_engine: opts.targetEngine,
target_id: targetId,
schema_version: 2,
started_at: new Date().toISOString(),
};
}
// Pages.source_id is a foreign key. Copy the complete source catalog first,
// including archived rows and sync/routing metadata, so every page write has
// a valid parent and the target preserves source behavior.
await copyMigrationSources(sourceEngine, targetEngine);
// Get all source pages
const sourceStats = await sourceEngine.getStats();
const allPages = await sourceEngine.listPages({ limit: 100000 });
+23 -6
View File
@@ -45,6 +45,13 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
// No-op without a pack_upgrade_available finding.
const explain = args.includes('--explain');
const targetScore = parseInt10(args, '--target-score') ?? 90;
// v0.42.42.0 (#2139): `--max-usd off`/`unlimited`/`none` → run uncapped. maxUsd
// stays undefined, which runRemediation already treats as no ceiling (skips the
// est-cost refusal + BudgetTracker runs uncapped); `maxUsdOff` lifts the
// missing-cap refusal below. Spend is still ledgered.
const maxUsdIdx = args.indexOf('--max-usd');
const maxUsdVal = maxUsdIdx >= 0 ? (args[maxUsdIdx + 1] ?? '').trim().toLowerCase() : '';
const maxUsdOff = ['off', 'unlimited', 'none'].includes(maxUsdVal);
const maxUsdRaw = parseFloat10(args, '--max-usd');
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
@@ -91,13 +98,23 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
}
// --auto refuses without --max-usd (cron-safety per A12 + A20).
// v0.42.42.0 (#2139, D15A): spend.posture=tokenmax lifts the refusal —
// --auto runs UNCAPPED (maxUsd stays undefined), spend still ledgered by the
// remediation budget tracker. An explicit --max-usd always wins.
if (auto && maxUsd === undefined) {
process.stderr.write(
`gbrain onboard --auto refuses without --max-usd N.\n` +
`Set a cap to avoid surprise spend:\n` +
` gbrain onboard --auto --max-usd 5\n`,
);
process.exit(2);
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const tokenmax = (await resolveSpendPosture(engine)) === 'tokenmax';
if (maxUsdOff || tokenmax) {
process.stderr.write(`${maxUsdOff ? '--max-usd off' : 'spend.posture=tokenmax'}: onboard --auto running uncapped, spend ledgered. docs: docs/operations/spend-controls.md\n`);
} else {
process.stderr.write(
`gbrain onboard --auto refuses without --max-usd N.\n` +
`Set a cap to avoid surprise spend:\n` +
` gbrain onboard --auto --max-usd 5\n` +
`Or pass --max-usd off / set spend.posture=tokenmax to run uncapped. docs: docs/operations/spend-controls.md\n`,
);
process.exit(2);
}
}
// Build the plan: T4 checks supply extra remediations on top of T3's
+9 -1
View File
@@ -53,7 +53,15 @@ const DENY_PREFIXES = [
];
/** First slug segments where no inbound links is expected */
const FIRST_SEGMENT_EXCLUSIONS = new Set(['scratch', 'thoughts', 'catalog', 'entities']);
const FIRST_SEGMENT_EXCLUSIONS = new Set([
'scratch',
'thoughts',
'catalog',
'entities',
'raw',
'atoms',
'skills',
]);
// --- Filter logic ---
+43 -13
View File
@@ -7,6 +7,7 @@
import { listRecipes, getRecipe } from '../core/ai/recipes/index.ts';
import { configureGateway, embedOne, isAvailable as gwIsAvailable, chat as gwChat } from '../core/ai/gateway.ts';
import { buildGatewayConfig } from '../core/ai/build-gateway-config.ts';
import { probeOllama, probeLMStudio } from '../core/ai/probes.ts';
import { loadConfig } from '../core/config.ts';
import { AIConfigError, AITransientError } from '../core/ai/errors.ts';
@@ -33,15 +34,19 @@ interface ProviderOption {
function configureFromEnv(): void {
const config = loadConfig();
configureGateway({
embedding_model: config?.embedding_model,
embedding_dimensions: config?.embedding_dimensions,
expansion_model: config?.expansion_model,
chat_model: config?.chat_model,
chat_fallback_chain: config?.chat_fallback_chain,
base_urls: config?.provider_base_urls,
env: { ...process.env },
});
// Route through buildGatewayConfig — the single ownership seam that folds
// file-plane API keys (openrouter_api_key, zeroentropy_api_key, ...) into
// the gateway env — instead of hand-assembling AIGatewayConfig field by
// field. Hand-building it here let this diagnostic report a provider as
// missing env even when ~/.gbrain/config.json had it and the real gateway
// path resolved it fine (#2728). Pre-init (no file-plane config yet) falls
// back to a bare env passthrough so the command still works before
// `gbrain init`.
if (config) {
configureGateway(buildGatewayConfig(config));
return;
}
configureGateway({ env: { ...process.env } });
}
export function envReady(recipe: Recipe, env: NodeJS.ProcessEnv = process.env): boolean {
@@ -136,7 +141,12 @@ EXAMPLES
}
function runList(_args: string[]): void {
console.log(formatRecipeTable(listRecipes()));
// Same env the gateway actually sees (file-plane keys folded in), not bare
// process.env — keeps this table's STATUS column honest with what
// `providers test` (and the real init/gateway path) would report.
const cfg = loadConfig();
const env = cfg ? buildGatewayConfig(cfg).env : process.env;
console.log(formatRecipeTable(listRecipes(), env));
}
async function runTest(args: string[]): Promise<void> {
@@ -163,8 +173,18 @@ async function runTest(args: string[]): Promise<void> {
// the divergence at the top of the test so the recovery experience
// doesn't repeat the bug-reporter's "providers test ✓ but import still
// broken" trap.
//
// #2863: `cfg` is lifted out of the try block (not just used for the
// warning) so the configureGateway calls below can reuse it. Before this
// fix, the --model override only forwarded embedding_model/chat_model +
// env, dropping config.provider_base_urls entirely — a probe against a
// custom endpoint (e.g. a regional DashScope base URL) would silently
// fall back to the recipe's hardcoded default endpoint and fail with a
// misleading "Incorrect API key" error even though the key was valid for
// the configured endpoint.
let cfg: ReturnType<typeof loadConfig> | null = null;
try {
const cfg = loadConfig();
cfg = loadConfig();
const configuredModel = tpArg === 'embedding' ? cfg?.embedding_model : cfg?.chat_model;
if (!configuredModel) {
console.error(
@@ -180,17 +200,27 @@ async function runTest(args: string[]): Promise<void> {
}
} catch { /* loadConfig throws when no brain configured — first-time install path; the no-config branch above handles it. */ }
// Reuse the SAME resolver the production path uses (buildGatewayConfig —
// also used by cli.ts#connectEngine and init-embed-check.ts) so the probe
// sees the identical base_urls / provider_chat_options / folded API keys
// that a real `gbrain import`/`gbrain query` call would. Only the
// touchpoint's model (+ embedding dims) is overridden on top, so an
// isolated `--model` probe still targets exactly the requested model —
// it just resolves that model's endpoint the way the brain actually
// would. Falls back to bare env when no brain is configured yet (cfg is
// null on first-time install, matching the old behavior for that case).
const baseGatewayConfig = cfg ? buildGatewayConfig(cfg) : { env: { ...process.env } };
if (tpArg === 'embedding') {
const dims = recipe?.touchpoints.embedding?.default_dims ?? 1536;
configureGateway({
...baseGatewayConfig,
embedding_model: modelArg,
embedding_dimensions: dims,
env: { ...process.env },
});
} else {
configureGateway({
...baseGatewayConfig,
chat_model: modelArg,
env: { ...process.env },
});
}
void modelId; // intentionally unused but preserved for readability
+41 -15
View File
@@ -444,14 +444,24 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
}
// F3: --max-cost / --max-cost-usd both accepted for symmetry with brainstorm.
// v0.42.42.0 (#2139): `off`/`unlimited`/`none` → no runtime cap AND an explicit
// "cost isn't the constraint" decision that proceeds past the confirmation gate
// (like --yes). Numeric must be positive; `0`/garbage is rejected.
let maxCostUsd: number | undefined;
let maxCostOff = false;
for (const flag of ['--max-cost', '--max-cost-usd']) {
const idx = args.indexOf(flag);
if (idx >= 0) {
const v = args[idx + 1];
const t = (v ?? '').trim().toLowerCase();
if (['off', 'unlimited', 'none'].includes(t)) {
maxCostUsd = undefined; // no runtime cap (reindex skips the tracker when unset)
maxCostOff = true;
break;
}
const n = v ? parseFloat(v) : NaN;
if (!Number.isFinite(n) || n <= 0) {
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD (got ${v ?? '(missing)'})`);
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD, or off/unlimited (got ${v ?? '(missing)'})`);
process.exit(2);
}
maxCostUsd = n;
@@ -493,20 +503,36 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
}
if (!yes) {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (!isTTY || json) {
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
// on --json now (human refusal on stderr otherwise) — #1784.
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
if (refusal.stdout) console.log(refusal.stdout);
if (refusal.stderr) console.error(refusal.stderr);
process.exit(2);
}
console.log(previewMsg);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return;
// v0.42.42.0 (#2139): spend.posture=tokenmax makes the gate informational
// — print the estimate and proceed (the operator declared cost isn't the
// constraint). The spend is still ledgered by the runtime BudgetTracker.
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const posture = await resolveSpendPosture(engine);
// An explicit `--max-cost off` is the same "cost isn't the constraint"
// signal as spend.posture=tokenmax — proceed past the confirmation gate.
if (posture === 'tokenmax' || maxCostOff) {
const gate = maxCostOff ? 'max_cost_off' : 'posture_tokenmax';
if (json) {
console.log(JSON.stringify({ status: 'proceeding', gate, codePages: preview.totalPages, totalTokens: preview.totalTokens, costUsd, model: getEmbeddingModelName() }));
} else {
console.log(`${previewMsg} ${maxCostOff ? '--max-cost off' : 'spend.posture=tokenmax'}: proceeding (informational). docs: docs/operations/spend-controls.md`);
}
} else {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (!isTTY || json) {
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
// on --json now (human refusal on stderr otherwise) — #1784.
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
if (refusal.stdout) console.log(refusal.stdout);
if (refusal.stderr) console.error(refusal.stderr);
process.exit(2);
}
console.log(previewMsg);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return;
}
}
}
}
+282
View File
@@ -0,0 +1,282 @@
/**
* `gbrain reindex-search-vector` recreate FTS trigger functions and
* backfill existing rows under the language configured via
* GBRAIN_FTS_LANGUAGE.
*
* Why this command exists: schema migration v123 (configurable_fts_language)
* stamps the trigger functions with the configured language at first apply.
* After that, changing the env var has no effect on the write side because
* v123 already shows as "applied" the migrations runner will skip it.
* This command is the documented escape hatch: it re-runs the same
* recreate-and-backfill logic v123 uses, gated on an explicit user
* action so the operation is intentional and visible (writes touch
* every row in pages and content_chunks).
*
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
* the same trigger function bodies and the same tokenized vectors.
*
* Flags:
* --dry-run Show what would happen, exit 0 without touching DB.
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
* --json Machine-readable result envelope. Does NOT imply --yes.
*
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
* so a large brain never holds one giant row lock, and streams progress
* through the shared reporter (stderr; stdout stays clean for --json).
*
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
* expect ~5-15s depending on Postgres CPU and content size.
*/
import type { BrainEngine } from '../core/engine.ts';
import { getFtsLanguage } from '../core/fts-language.ts';
import { createInterface } from 'readline';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
export interface ReindexSearchVectorOpts {
dryRun?: boolean;
yes?: boolean;
json?: boolean;
}
export interface ReindexSearchVectorResult {
status: 'ok' | 'dry_run' | 'cancelled';
language: string;
pagesUpdated: number;
chunksUpdated: number;
triggersRecreated: number;
durationMs: number;
}
interface CountRow {
pages: number;
chunks: number;
}
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
export const BACKFILL_BATCH_SIZE = 5000;
/**
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
* the shared progress reporter after each batch. Terminates when a batch
* returns fewer rows than the batch size (or none).
*/
async function batchedBackfill(
engine: BrainEngine,
table: 'pages' | 'content_chunks',
setClause: string,
tick: (n: number) => void
): Promise<void> {
let cursor = 0;
for (;;) {
const rows = await engine.executeRaw<{ id: number }>(`
UPDATE ${table} SET ${setClause}
WHERE id IN (
SELECT id FROM ${table}
WHERE search_vector IS NOT NULL AND id > ${cursor}
ORDER BY id
LIMIT ${BACKFILL_BATCH_SIZE}
)
RETURNING id
`);
if (rows.length === 0) break;
tick(rows.length);
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
if (rows.length < BACKFILL_BATCH_SIZE) break;
}
}
/**
* Programmatic entrypoint takes a typed opts object. Used by tests and
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
* defined at the bottom of this file.
*/
export async function runReindexSearchVector(
engine: BrainEngine,
opts: ReindexSearchVectorOpts
): Promise<ReindexSearchVectorResult> {
const lang = getFtsLanguage();
const startedAt = Date.now();
// Inventory: how many rows will the backfill touch?
const counts = await engine.executeRaw<CountRow>(
`SELECT
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
);
const pagesCount = counts[0]?.pages ?? 0;
const chunksCount = counts[0]?.chunks ?? 0;
if (opts.dryRun) {
const result: ReindexSearchVectorResult = {
status: 'dry_run',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
}
return result;
}
// Confirm unless --yes. --json does NOT bypass the gate — a machine
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
if (!opts.yes) {
if (!process.stdin.isTTY) {
if (opts.json) {
console.log(JSON.stringify({
error: {
class: 'ConfirmationRequired',
code: 'reindex_requires_yes',
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
hint: 'Pass --yes to proceed, or --dry-run to preview.',
},
language: lang,
pages: pagesCount,
chunks: chunksCount,
}));
} else {
console.error('Refusing to run without --yes in non-TTY environment.');
}
process.exit(2);
}
const rl = createInterface({ input: process.stdin, output: process.stdout });
const answer = await new Promise<string>(resolve => {
rl.question(
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
resolve
);
});
rl.close();
if (!/^y(es)?$/i.test(answer.trim())) {
const result: ReindexSearchVectorResult = {
status: 'cancelled',
language: lang,
pagesUpdated: 0,
chunksUpdated: 0,
triggersRecreated: 0,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log('Cancelled.');
}
return result;
}
}
// Recreate trigger functions. The strings are intentionally identical to
// the v124 migration body — keeping them in lockstep is the contract.
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
// hardening from every brain that runs this command.
//
// #2704: compiled_truth (the unbounded whole-page body) is deliberately
// NOT indexed here — it overflows Postgres's 1MB tsvector cap on large
// pages, and content_chunks.search_vector (populated separately, chunk-
// grain, well under the cap) is what searchKeyword() actually queries.
// See migrate.ts's v124 for the full rationale; keep this copy in sync.
const recreatePagesFn = `
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
DECLARE
timeline_text TEXT;
BEGIN
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
NEW.search_vector :=
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
const recreateChunksFn = `
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
`;
await engine.executeRaw(recreatePagesFn);
await engine.executeRaw(recreateChunksFn);
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
// direct vector compute since the column itself is what we want.
progress.start('reindex_search_vector.pages', pagesCount);
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
progress.finish();
progress.start('reindex_search_vector.chunks', chunksCount);
await batchedBackfill(
engine,
'content_chunks',
`search_vector =
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
n => progress.tick(n)
);
progress.finish();
const result: ReindexSearchVectorResult = {
status: 'ok',
language: lang,
pagesUpdated: pagesCount,
chunksUpdated: chunksCount,
triggersRecreated: 2,
durationMs: Date.now() - startedAt,
};
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
}
return result;
}
/**
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
*
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
*/
export async function runReindexSearchVectorCli(
engine: BrainEngine,
args: string[]
): Promise<void> {
const dryRun = args.includes('--dry-run');
const yes = args.includes('--yes') || args.includes('-y');
const json = args.includes('--json');
await runReindexSearchVector(engine, { dryRun, yes, json });
}
+2 -1
View File
@@ -22,6 +22,7 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
import { importFromContent, importFromFile } from '../core/import-file.ts';
import { serializeMarkdown } from '../core/markdown.ts';
@@ -150,7 +151,7 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
} else {
process.stderr.write('Usage: gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--repo PATH]\n');
}
process.exitCode = 2;
setCliExitVerdict(2);
return { pending: 0, reindexed: 0, skipped: 0, failed: 0, dryRun: !!opts.dryRun, chunkerVersion: MARKDOWN_CHUNKER_VERSION };
}
+18 -12
View File
@@ -105,13 +105,19 @@ function printHelp(): void {
async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>, args: string[]): Promise<void> {
const { json, timeoutMs } = parseFlags(args);
let submitted: { id: number; name: string; state: string };
// submit_job / get_job return the MinionJob row verbatim — the lifecycle
// field is `status` (src/core/minions/types.ts), not `state`. Reading
// `state` here made every poll see `undefined`, so the terminal check
// never matched and ping always exhausted its timeout (exit 1) even when
// the cycle completed. The ping's own JSON *output* keys (`state`,
// `last_state`) are kept as-is for consumers.
let submitted: { id: number; name: string; status: string };
try {
const res = await callRemoteTool(config, 'submit_job', {
name: 'autopilot-cycle',
data: { phases: ['sync', 'extract', 'embed'] },
});
submitted = unpackToolResult<{ id: number; name: string; state: string }>(res);
submitted = unpackToolResult<{ id: number; name: string; status: string }>(res);
} catch (e) {
return failPing(e, json);
}
@@ -122,43 +128,43 @@ async function runRemotePing(config: NonNullable<ReturnType<typeof loadConfig>>,
const startMs = Date.now();
let attempt = 0;
let lastState = submitted.state;
let lastState = submitted.status;
while (Date.now() - startMs < timeoutMs) {
const elapsed = Date.now() - startMs;
const intervalMs = elapsed < 30_000 ? 1_000 : elapsed < 5 * 60_000 + 30_000 ? 5_000 : 10_000;
await sleep(intervalMs);
attempt++;
let job: { id: number; state: string; failed_reason?: string };
let job: { id: number; status: string; failed_reason?: string };
try {
const res = await callRemoteTool(config, 'get_job', { id: submitted.id });
job = unpackToolResult<{ id: number; state: string; failed_reason?: string }>(res);
job = unpackToolResult<{ id: number; status: string; failed_reason?: string }>(res);
} catch (e) {
// Network blip mid-poll: log and keep going. Surface only if persistent.
if (!json) console.error(` poll #${attempt} failed (${e instanceof Error ? e.message : String(e)}); continuing...`);
continue;
}
if (job.state !== lastState) {
lastState = job.state;
if (!json) console.error(` job #${submitted.id}${job.state}`);
if (job.status !== lastState) {
lastState = job.status;
if (!json) console.error(` job #${submitted.id}${job.status}`);
}
const terminal = ['completed', 'failed', 'dead', 'cancelled'];
if (terminal.includes(job.state)) {
const ok = job.state === 'completed';
if (terminal.includes(job.status)) {
const ok = job.status === 'completed';
if (json) {
console.log(JSON.stringify({
status: ok ? 'success' : 'error',
job_id: submitted.id,
state: job.state,
state: job.status,
...(job.failed_reason ? { failed_reason: job.failed_reason } : {}),
elapsed_ms: Date.now() - startMs,
}));
} else {
console.log(ok
? `\nautopilot-cycle complete (${Math.round((Date.now() - startMs) / 1000)}s).`
: `\nautopilot-cycle ended ${job.state}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
: `\nautopilot-cycle ended ${job.status}${job.failed_reason ? `: ${job.failed_reason}` : ''}.`);
}
process.exit(ok ? 0 : 1);
}

Some files were not shown because too many files have changed in this diff Show More