Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 c5952b8714 v0.43.0.0 feat(mcp,search): MEMORY_VERBS v1 five-verb protocol + concept-question query steering (cathedral 1, #2416) (#2172)
* feat(verbs): MEMORY_VERBS v1 core — five frozen verbs over existing machinery

recall extended in place (G1B additive superset: protocol_version + string
fact_id/provenance on every response; query arm via cheap-hybrid with
keyword-only degradation when no embedding provider; op-layer budget packing
facts-first via the new generic packToBudget). remember/entity/synthesize/
forget as first-class Operations in src/core/verbs.ts (runtime leaf — TDZ-safe
both import directions) with MEMORY VERB descriptions, ToolAnnotations,
populated suggestions on every error (verbError + protocol ErrorCodes +
additive detail/protocol_version on OperationError.toJSON).

writeSingleFact (facts/write-single.ts): zero-LLM single-fact seam reusing the
pipeline's dedup; X1 deterministic supersession (same entity+kind, similar,
text differs -> supersedeId / fence expire+link); validUntil threaded through
FenceInputFact (was silently dropped). buildEntityCard (verbs/entity-card.ts):
reflex resolution arms + parallel depth-1 reads, per-arm degradation on
pre-page_aliases brains, near-miss suggestions with create_safety. runThink
gains additive usage aggregation; synthesize converts the no-LLM graceful
result to the unavailable protocol error and prices the cost block via
canonical model-pricing. RESPONSE_SCHEMAS registry + ERROR_SCHEMA for the
conformance kit.

Cathedral 1 stage 1/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(mcp): MEMORY_VERBS v1 surface mode + fail-closed dispatch + usage sidecar

gbrain serve --surface verbs|full (config mcp_surface registered in the merge
allowlist; default full — existing installs unchanged). Enforcement is
two-layer and fail-closed: filtered ListTools AND dispatchToolCall's new
allowedOps set, applied on stdio + BOTH HTTP paths (serve-http.ts OAuth and
the second http-transport.ts). ToolAnnotations passthrough in buildToolDefs +
the HTTP inline mapper, emitted only when an op defines them (byte-stability
for existing tools). Verb validation/internal errors speak the protocol
envelope (suggestion + protocol_version) at the dispatch layer.

E4 observability: per-verb JSONL sidecar (usage-log.ts — local only, never
uploaded; O_APPEND line-atomic, lock-free 10MB rotation) written from dispatch
so param-validation failures count; brain_id via gbrainPath; doctor gains the
memory_verbs_usage check. CLI: human-readable rendering for remember/entity/
synthesize with trailing --json escape; gbrain recall gains --query +
--budget-tokens routed through the recall OP (same arm MCP exercises) with the
budget footer; the forget verb carries no cliHint (CLI_ONLY forget dispatches
first and would shadow it).

Cathedral 1 stage 2/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(protocol): gbrain protocol — schemas, conformance kit, stats + TTHW

gbrain protocol [--json] emits the live verb input schemas + the hand-authored
RESPONSE_SCHEMAS + error contract + versioning policy. gbrain protocol
conformance certifies ANY MCP endpoint (default: self-spawned stdio; --target
http://...[--token] or a stdio command) against the embedded fixture set —
shape/enum/behavior/round-trip only, deterministic; entity-page cases seed via
put_page when available and skip honestly on verbs-only targets; synthesize is
cost-gated behind --synthesize (asserts the clean unavailable error on key-less
CI). Fixtures double as BrainBench seeds (test/fixtures/memory-verbs/cases.json
mirror). Self-certification passes 23/23 live.

gbrain protocol stats [--days N]: per-verb calls/error-rate/latency, budget
drops, entity hit rate, and measured TTHW (init stamps protocol_installed_at;
delta to the sidecar's first verb call) — local JSONL only, never uploaded.
init epilogues print the 3-command quickstart (mcp add --surface verbs +
key-free resurrection demo) with the visibility consent line. entity-card fix:
raw namespaced-slug input now matches exactly (slugify flattens slashes).

Cathedral 1 stage 3/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test+docs(protocol): conformance suite, p99 gate, MEMORY_VERBS_v1 doc, funnel

test/memory-verbs-conformance.test.ts: 26 tests through dispatchToolCall —
G1B superset regression (legacy fields byte-equal, additions allowed), budget
math incl. budget<first-item, keyword degradation (never an error), remember
provenance/ttl-trap/null-echo, remote round-trip via the world default +
private facts hidden, entity card schema-validated with the chat gateway
rigged to throw (zero-LLM guard) + all arms + fence test, synthesize
[EXPENSIVE + annotations + clean unavailable, forget idempotency/not_found,
writeSingleFact X1 supersession via the embed-transport seam + degraded
dedup, the F3 negative self-test (the runner FAILS lying servers: dropped
fields, bad enums, re-typed ids), fixture-mirror drift guard.
test/mcp-surface.test.ts: verbs=exactly 5, full=identity, dispatch-layer
fail-closed, strict flag parsing. test/entity-card-perf.slow.test.ts: 20K
pages/100K links/30K aliases/40K facts — measured p99 22.9ms vs the 100ms
contract + a recalibrated 50x ratio guard; wired into the CI perf job along
with stdio self-certification (23/23 CONFORMANT live). tool-defs byte-
stability test now pins per-op: ops WITHOUT annotations keep the exact
pre-v1 shape.

docs/protocol/MEMORY_VERBS_v1.md: the frozen contract — envelopes, error
codes, additive-forever policy, ttl forms (P30D rejected), status-not-
status_text, surface modes + why default full, per-harness installs, consent
line, conformance pass criteria + copyable --target examples, local-only
stats privacy, 200K manual bench recipe. README quickstart + docs/mcp pages
moved to --surface verbs with the claude-not-found recovery; llms-config
registered + bundles rebuilt; KEY_FILES entries for the 7 new modules.

Cathedral 1 stage 4/4 (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): verify-gate compliance — test seam, allow-comment placement, import hygiene

usage-log gains __setUsageLogPathForTests so the verb tests redirect the
sidecar without mutating process.env.GBRAIN_HOME (test-isolation lint);
the gbrain-allow-direct-insert rationale moves onto the insertFact call line
(system-of-record guard greps per-line); protocol.ts drops its unused
`operations` import (operations-filter-bypass guard — operationsByName only).
The synthesize unit test now drives the chat-transport seam (deterministic
success envelope + priced cost block, zero real API spend); the keyless
`unavailable` conversion stays covered end-to-end by the CI conformance step,
which runs credential-free.

Cathedral 1 verify pass (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): close 3 ship-review P1s — forget cross-source, entity backlink leak, conformance exit code

Pre-landing review (codex, on the committed diff) caught three P1s the
plan-stage reviews flagged only in the abstract:

P1.1 trust boundary — forget expired by GLOBAL fact id with no source/visibility
scope: a remote caller in source A could expire facts in source B (incl.
private) by guessing ids. forgetFactInFence gains sourceId + worldOnly opts
(scope-check before any state inspection; out-of-scope → not_found, no
existence leak); the forget verb passes ctx.sourceId + worldOnly=remote.

P1.2 source isolation — entity card incoming edges + backlink_count leaked
across sources: getBacklinks(slug,{sourceId}) scopes only the TARGET page and
getBacklinkCounts has no source param, so a foreign-source page's slug surfaced
in a same-named entity's card. Replaced both with a both-sides-scoped query
(f.source_id = t.source_id = entity source, mentions excluded) inside the card
module — no engine-interface change.

P1.3 CLI exit code — protocol conformance/unknown-subcommand set
process.exitCode directly, which the PGLite/WASM force-exit seam ignores
(it reads the gbrain-owned setCliExitVerdict). A NOT CONFORMANT target printed
the banner but exited 0 — CI would green-light a non-conformant server. Now
routes through setCliExitVerdict; proven: unknown subcommand exits 1.

Regression tests: cross-source forget → not_found + foreign fact stays active;
remote can't forget a private fact; entity card omits foreign-source backlinks
from both edges and the count.

Cathedral 1 ship-review fixes (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.43.0.0 feat(mcp): MEMORY_VERBS v1 — frozen five-verb memory protocol + conformance kit (cathedral 1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: surface --surface verbs in local-stdio quickstart paths for v0.43.0.0

The MEMORY_VERBS v1 quickstart flag (gbrain serve --surface verbs) was added to
the README quickstart, CLAUDE_CODE.md, and CODEX.md during the build but three
other docs that show the local-stdio wire-up still presented bare `gbrain serve`:
the connect-coding-agent Path B walkthrough (the doc the README links to),
INSTALL.md, and DEPLOY.md's Local stdio path. Brought them in line so the
five-verb surface is consistent across every local-quickstart entry point.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(doctor): assign memory_verbs_usage check to the SKILL category set

The new memory_verbs_usage doctor check (Cathedral 1, E4) was added without a
category, tripping the doctor-categories drift guard (every check name must
belong to exactly one category set). It's an agent-capability adoption signal,
a sibling of retrieval_reflex_health — assigned to SKILL_CHECK_NAMES.

Cathedral 1 ship-review fix (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(verbs): propagate MEMORY_VERBS v1 to existing harnesses on upgrade

The verbs shipped reachable + self-describing, but an EXISTING install that
upgrades to 0.43 and keeps its skills had no active push toward them: no
post-upgrade notice (default surface stays 'full'), and no skill/guidance
routing — agents kept calling search/query/get_page/extract_facts.

Two additive propagation fixes:

- Post-upgrade feature pitch: src/commands/migrations/v0_43_0.ts (pitch-only,
  no-op idempotent orchestrator — the verbs ride existing tables, no schema/
  data migration). Registered in the migrations index so
  and the self-upgrade NOTIFY channel announce the five verbs +  +  on any upgrade past 0.43.
  Verified live: the pitch fires for an install upgrading from 0.42.43.0.
- Skill routing: skills/query (memory reads) and skills/brain-ops (read/write
  cycle) now teach the verbs — recall/entity/synthesize vs search/get_page/
  query, remember vs extract_facts, forget — with a fall-back-to-classic-ops
  note for older brains / full surface. The in-MCP tool descriptions already
  steered tool-list readers; this adds the narrative layer the audit found
  missing across all 29 skills.

apply-migrations future-list fixtures updated for the new 0.43.0.0 registry
entry. No DB migration; no contract change; verbs unaffected.

Cathedral 1 propagation (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(verbs): green the 3 failing CI shards — 3-seg migration version, hermetic synthesize test, fresh llms

Three real CI-shard failures from the propagation commit + master merges:

- migrations-registry (shard 10): v0_43_0 used a 4-segment version 0.43.0.0,
  but the registry enforces 3-segment semver (/^\d+\.\d+\.\d+$/). Changed
  the migration version to 0.43.0 (the RELEASE stays 0.43.0.0; migrations have
  always used 3-seg, e.g. 0.32.2). apply-migrations future-list fixtures
  updated to match.
- synthesize conformance test (shard 5): asserted a successful answer via the
  chat seam, but runThink builds its client through a real-key check (not the
  seam), so CI (credential-free) took the NO_ANTHROPIC_API_KEY path → the
  verb's unavailable conversion → isError. Now wraps the call in withEnv with a
  fake key AND the chat seam, so it's deterministic regardless of ambient
  credentials.
- build-llms (shard 8): the v0.42.44.0 master merge changed an inlined doc;
  regenerated llms.txt/llms-full.txt.

Cathedral 1 CI fixes (#cathedral-1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(search): steer concept/landscape questions to query — descriptions, docs, lookup chain, CLI nudge (#2416)

The search/query routing guidance shipped a stale and backwards model:
SEARCH_DESCRIPTION still led with "Keyword search using full-text search"
(search has been cheap-hybrid — vector + keyword + RRF, expansion off —
since D4/D15), and the mandatory lookup chain in brain-first.md ordered
search-first with query only "if search is thin". For concept/landscape
questions ("all the X that do Y"), keyword+vector recall always returns
something, so the fallback never fires and expansion-recoverable
synonym-phrased matches drop silently.

- operations-descriptions.ts: fix the stale SEARCH lead (cheap-hybrid, no
  LLM expansion); both constants now route concept/synonym/landscape
  questions to query, cap the claim honestly (both are top-K — exhaustive
  enumeration belongs to list_pages), and keep every pinned redirect.
- brain-first.md: lookup chain is intent-driven (exact token → search;
  concept/landscape → query FIRST), cost claims corrected, and the
  anti-pattern is explicit: a nonzero search count is not a completeness
  signal. Same sweep in brain-ops/SKILL.md and query/SKILL.md, which
  duplicated the backwards order.
- search-modes.md: decision tree + quick-ref updated to the cheap-hybrid
  reality, new Tricky Spot #6 (count ≠ completeness), example names moved
  to placeholder people per the privacy convention. RETRIEVAL.md gets the
  matching one-liner.
- query-intent.ts: looksConceptShaped + conceptNudge — deterministic
  concept-shape classifier in the existing classifier home, tuned to favor
  false-negatives; cues owned by other routers (find_experts, salience
  ops) deliberately excluded. cli.ts prints the hint-only stderr nudge on
  BOTH result paths (local engine + thin-client routed), --quiet-gated;
  search stays the cheap hot path (never auto-reroutes).
- salience-llm-routing e2e: new concept-phrasings block asserts the
  descriptions actually route concept questions to query (verified live:
  20/20 — all 12 personal-query regressions still route to salience ops,
  all 8 concept phrasings route to query).
- KEY_FILES.md: query-intent entry updated to current state (the old
  entry pointed at a renamed file); llms bundle regenerated (no diff —
  entry is link-only).
- TODOS.md: MCP-envelope hint follow-up filed (remote agents are the
  higher-value half; needs a return-contract change kept out of this
  commit).

Verification: typecheck clean; 70 targeted unit tests + 20 live routing
evals green; all 24 check:all guards green; failing full-suite shards
triaged — think-pipeline.serial.test.ts:434 fails identically on the
clean tree (pre-existing on this branch, filed separately), the rest are
PGLite parallel-load contention that pass in isolation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): pin think's stub-path usage at null per the E2 accounting contract

The MEMORY_VERBS [E2] usage-accounting change deliberately moved "no LLM
ran" from `undefined` to `null` (the frozen cost-block spelling), but the
older #1698 stub-path test still pinned `undefined` and failed on every
run of this branch. The protected semantic is unchanged: no fake usage or
cost on the stub path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): cover the #2416 query-steering wave under v0.43.0.0

The 0.43.0.0 entry described only MEMORY_VERBS; the #2416 retrieval-routing
wave rides the same release. Adds its user-facing summary and refreshes the
release date to the actual ship date.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(protocol): reword so the #3502 docs-lint doesn't read a phantom command

Master's docs-lint (landed after this branch forked) scans for
`gbrain <verb>` references; the MEMORY_VERBS diagram's "gbrain reference
implementation" phrasing parsed as a nonexistent `gbrain reference`
command. Post-merge-only failure; both sides were green alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.43.0.0

Post-ship /document-release drift pass over the MEMORY_VERBS v1 +
#2416 query-steering waves:

- docs/tutorials/connect-coding-agent.md: the pasteable brain-first
  protocol still taught the pre-#2416 "Search first" rule and named
  only full-surface tools; step 1 now routes by question shape
  (exact tokens -> search, concept/landscape -> query first) and both
  steps map the five-verb surface names (recall/synthesize/remember).
  find_experts and query callouts marked with their surface.
- README.md + tutorial: "90-tool wall" count refreshed to 110 (the
  actual operation catalog).
- CLAUDE.md: contract-first op count ~90 -> ~110 with the v0.43.0.0
  MEMORY_VERBS clause; reference-map row for the memory-verbs
  protocol / --surface / conformance docs.
- TODOS.md: blank line before the v0.42.74.0 section header.
- llms-full.txt regenerated (build:llms; freshness test green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: apply cross-model doc-review fixes for v0.43.0.0

Verified findings from the /document-release Codex pass:

- MEMORY_VERBS_v1.md: disclose that conformance WRITES (marker-suffixed
  seed page + remember/forget facts, write-scope creds, artifacts not
  auto-deleted); quickstart header says 4 commands because it has 4.
- CHANGELOG v0.43.0.0: cost claim scoped per verb (budget/latency/cost
  block); remember "updates in place" -> supersedes-with-history (what
  write-single.ts actually does).
- DEPLOY.md: legacy bearer fallback works on BOTH engines now (both
  schemas carry access_tokens per http-transport.ts) — was Postgres-only.
- CLAUDE_CODE.md: list_skills/core-tools verify note marked full-surface
  (on --surface verbs those tools aren't on the surface).
- skills/query/SKILL.md: --surface full INCLUDES the verbs; fallback is
  for pre-0.43 servers only.
- skills/brain-ops/SKILL.md: search/query tool labels updated to the
  #2416 reality (cheap hybrid vs hybrid + LLM expansion).
- search-modes.md: stale "keyword search returns chunks" label.
- RETRIEVAL.md: note the query op defaults expand:true per call.
- Counts: 43 skills -> 52 (README x2, INSTALL), 30 skills -> 52
  (CLAUDE.md), "30+ tools over MCP" -> 110 (README).
- TODOS.md: filed 2 code-vs-spec follow-ups (recall filter composition +
  limit cap; synthesize unavailable-mapping breadth) instead of silently
  editing the frozen contract.
- llms-full.txt + skills.lock.json regenerated; freshness guards green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): regenerate the flag registry for the cathedral-1 command set

Master's #2185 strict flag validation ships a committed, generated
CLI_FLAG_REGISTRY with drift + freshness guards keyed off CLI_ONLY. The
merge added this branch's `protocol` command to CLI_ONLY, but the
committed registry was generated on master without it, failing CI shard
7's drift guard. Regenerated via `bun run build:flag-registry` on the
merged tree (95 commands; picks up `protocol` and the `--surface`
help-text mentions from the verbs work).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 17:25:00 -07:00
Garry TanandClaude Fable 5 130d321d23 v0.42.76.0 fix(upgrade,security,cli): strict flag validation, bootstrap-wedge class kill, federated chunk scope (#3902)
* fix(schema): bootstrap timeline_entries.event_page_id forward reference — un-wedge pre-v121 upgrades (#2626 #2594 #2579 #2537 #2536)

v0.42.56.0 (Chronicle, migration v121) added timeline_entries.event_page_id
and two partial indexes in the embedded schema blobs without extending
applyForwardReferenceBootstrap — any brain whose timeline_entries predates
v121 wedged initSchema at blob replay ("column event_page_id does not
exist") before runMigrations could apply v121, with no in-band recovery.

- Add the timeline_entries.event_page_id probe + column-only ALTER to
  applyForwardReferenceBootstrap in BOTH engines; FK + partial indexes land
  via the idempotent v121 / blob replay afterwards. Stays in the always-run
  bootstrap (never a migration hook — those skip oddly-stamped brains).
- REQUIRED_BOOTSTRAP_COVERAGE entry + strip blocks in both runtime tests.
- e2e: pre-v121 rewind → full initSchema converges (indexes re-created);
  wedged-brain recovery — a brain that already FAILED the upgrade attempt
  converges on retry with full final shape (column + FK + both partial
  indexes) and no ledger residue.

Absorbs PR #2548 (@chetan-guevara) and the e2e test from PR #2623
(@colinagent) — thank you both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(schema): coverage guard cross-references migration-added columns — close the scan hole that shipped the v121 wedge

The A2 static check treated a column as covered when the current CREATE
TABLE body declared it. But a column that is BOTH in the blob's CREATE
TABLE AND added by a migration is a forward reference by definition — on
pre-existing tables CREATE TABLE IF NOT EXISTS no-ops and the blob's
CREATE INDEX crashes initSchema before runMigrations can help. That mask
is exactly how timeline_entries.event_page_id passed the guard while
wedging every pre-v121 brain.

- buildIndexRefCoveragePredicate: migration-added columns (from
  extractAddedColumnsFromMigrations over the MIGRATIONS array) require a
  bootstrap ALTER; CREATE TABLE presence no longer counts for them.
- Unit test pins the v121 regression shape red/green with synthetic inputs;
  the A2 test pins the incident triple directly (migration-added +
  blob-indexed + bootstrap-covered).
- The strengthened predicate immediately surfaced two more latent wedges of
  the same class: minion_jobs.timeout_at + minion_jobs.idempotency_key
  (migration v7, blob-indexed, unprobed). Added probes in both engines +
  coverage entries + runtime strip blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): serve --http hides the generated admin token from non-TTY output by default (#2624)

Generated admin bootstrap tokens printed into container/log-aggregator
stdout on every headless start. shouldSuppressBootstrapPrint now defaults
to hidden unless stderr is an interactive TTY; env-sourced tokens are
never printed; --print-admin-token is the explicit escape hatch for
capturing the value on a trusted non-TTY start; --suppress-bootstrap-token
still overrides everything. Unit-tested across all five postures.

Absorbs PR #2625 (@irresi) — thank you.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): legacy bearer tokens honor permissions.takes_holders over serve --http (#2529)

GBrainOAuthProvider.verifyAccessToken never returned takesHoldersAllowList,
so the serve --http dispatch site always fell back to ['world'] — remote
MCP callers with an operator-configured takes_holders grant saw only
public takes. The legacy branch now extracts permissions.takes_holders
exactly like src/mcp/http-transport.ts (fail-safe ['world'] default,
non-string entries dropped, malformed permissions JSON fails closed
without throwing), and AuthInfo carries the field as a typed contract.

OAuth-registered clients have no takes_holders storage on oauth_clients;
that lane is design work tracked in TODOS (column migration + DCR/
registration surface), not part of this hotfix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): get_chunks honors the federated source grant + stops shipping embedding vectors (#2555, getChunks half of #2544)

The get_chunks op still used the pre-#2200 scalar pattern
(ctx.sourceId ? {sourceId} : {}) and engine.getChunks had no sourceIds[]
support — a federated client that could read a page via get_page got []
from get_chunks. The op now routes through sourceScopeOpts (canonical
ladder: federated array > scalar floor > nothing) and both engines gain
the getPage-style sourceIds[] precedence branch; the unset-opts 'default'
floor is preserved for local callers (importCodeFile contract).

While in the function: SELECT cc.* pulled every embedding vector over the
wire per chunk only for rowToChunk to discard them — replaced with the
explicit non-vector column list in both engines (the getChunks half of
#2544; the per-put_page getAllSlugs half is tracked separately).

getChunksWithEmbeddings stays scalar-only by design (engine-internal,
zero remote-reachable callers — documented at the interface).

Tests: op-level federated repro + isolation + default-floor bleed guard
(PGLite), engine precedence + Chunk-shape pin, and a DATABASE_URL-gated
engine-parity test covering all three scope shapes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): /admin/api/register-client accepts source + federatedRead bindings (#2143 enabler)

The HTTP register endpoint hardcoded source_id='default' and
federated_read=undefined — only the CLI could mint a client bound to a
non-default source, so HTTP-registered MCP clients wrote into 'default'
regardless of intent. The endpoint now accepts optional source /
federatedRead body fields, validated via assertValidSourceId with a
structured 400 on bad input; omitting both preserves the historical
default. The admin-UI form layer is a tracked follow-up.

Absorbs PR #2016 — thank you.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scope,calibration): think reads over MCP; source-scoped takes reads; calibration CLI reachability + model resolution (#2078-class #2451)

- think op → scope:'read' for OAuth/MCP clients: the handler already forces
  save/take off for remote callers before persistence, so a read-scoped
  token can think without a write grant; local CLI persistence unchanged.
  Scope-annotation test carries an explicit remote-gated allowlist.
- takes_list / takes_search / takes_scorecard / takes_calibration route
  through sourceScopeOpts (the #2200 class on the takes read lane) with
  engine support in BOTH engines + tests.
- 'calibration' added to CLI_ONLY (the command was registered but
  unreachable — dispatch-gap class) and calibration_profile/voice-gate
  resolve models through the canonical gateway tier resolver instead of
  bare ids that parseModelId rejects.
- BigInt-safe local-op output normalization (bigintToStringReplacer,
  postgres.js wire parity) — first half of the #2450 fix; the
  formatResult default case lands with the cli-output commit.

Absorbs PR #2598 (@colinagent) and PR #2452 (@spinsirr) — thank you both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): output correctness — BigInt-safe rendering, search --json, files bigint, and 5 unreachable commands (#2450 #2527 #2042 #2035-class)

- normalizeLocalResult wraps the local-op output round-trip with the
  bigint→string replacer (postgres.js wire parity; a bare stringify THROWS
  on BIGSERIAL keys); formatResult's default renderer gets the same
  replacer so nothing upstream can crash it.
- search/query --json: CLI-local formatter flag threaded through the shared
  formatter — stdout is a parseable result array, never human text on the
  --json path (the #2042 residual).
- file_list normalizes size_bytes (Postgres BIGINT → Number) so MCP
  serialization and the CLI KB math survive; null preserved.
- NEW dispatch-gap guard: every handleCliOnly top-level case label must be
  reachable via CLI_ONLY. It immediately caught FIVE live unreachable
  commands: pages, backfill, reconcile-links, notability-eval (added to
  CLI_ONLY), and the documented 'gbrain search modes|stats|tune' dashboards
  (pre-fix, 'search modes' silently keyword-searched the word "modes") —
  now routed via a pre-dispatch subcommand gate. 'whoknows' stays on its
  op-alias route (collision guard); tracked with PR #2509.

Absorbs PR #2494 and PR #2531 (@javieraldape) and adapts PR #472
(@vinsew) — thank you.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(db): updateSourceConfig survives mixed-array config rows + repair/doctor cover the subagent jsonb columns (#2251)

The array-coercion branch called jsonb_each(elem) bare — a mixed array
(e.g. '["x", {"last_full_cycle_at": ...}]') threw 'cannot call
jsonb_each on a non-object' DURING row production, permanently failing
every subsequent updateSourceConfig (last_full_cycle_at could never be
written again). Non-object elements are now neutralized inline via a
CASE-guarded jsonb_each; the row self-heals to a flat object on the next
write, object elements' keys recovered. Pinned ungated on PGLite (real
Postgres semantics) and via a DATABASE_URL-gated e2e on the real engine.

repair-jsonb + doctor's jsonb_integrity check extend from 5 to 8 columns
(subagent_messages.content_blocks, subagent_tool_executions.input/output
— historical damage rows from the pre-v0.42.53.0 positional double-encode;
the write paths themselves were fixed in #2375) with a to_regclass skip
for brains predating those tables.

Adapts the repair/doctor extension from PR #597 (@vinsew) — thank you.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): dry-run honesty — strict unknown-flag rejection CLI-wide + unify-types worker defaults to dry-run (#2185 #1575)

#2185: 'gbrain init --migrate-only --dry-run' applied REAL migrations —
flags are read ad hoc (args.includes) so anything a handler doesn't look
for was silently ignored, including intent-bearing safety flags. The CLI
now validates every flag pre-dispatch and pre-engine:

- Op commands validate against the operation contract (op.params +
  CLI-local --json/--explain), mirroring parseOpArgs so flag values that
  begin with '--' are never misread. parseOpArgs also gains the
  --key=value inline form (previously parsed as a junk key that consumed
  the NEXT token).
- CLI_ONLY commands validate against a GENERATED per-command registry
  (scripts/generate-flag-registry.ts scans each command's case block +
  imported modules + one level of relative imports; deliberately
  over-inclusive so a missed flag can't break a working invocation).
  Committed as src/core/cli-flag-registry.generated.ts;
  'bun run build:flag-registry' regenerates.
- Passthrough by construction: everything after '--', plus call / config /
  'jobs submit' payloads (handler-defined params are their contract).
- Guards: sweep test (every command × nonsense flag → error), acceptance
  tests (real flags, --no- negation, = form, -- passthrough), drift guard
  (every CLI_ONLY member has an entry), freshness guard (committed
  registry == fresh generator run), subprocess smokes incl. the literal
  #2185 repro failing loud with zero engine work.

BREAKING: scripts passing stray flags now fail loud with
"Unknown flag --x for 'gbrain <cmd>'" — that is the point.

#1575: the unify-types worker registration passed apply ?? true while the
handler documents 'Default false (dry-run)' — the canonical operator
invocation destructively retyped 25K+ pages by default. Now ?? false with
a structural test; explicit --params '{"apply":true}' is the only way to
mutate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(todos): file fix-wave 1 follow-ups (OAuth takes_holders design, parseFlags end-state, whoknows routing, #2544 half, #1558 UI, #2536 diagnostics)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): memory-safe unit runner — adaptive concurrency + serial OOM rescue pass

A default run (4 shards × 4 intra-shard files) holds up to 16 concurrent
PGLite WASM instances (~1.5GB each). With sibling Conductor workspaces
running their own suites, PGLite connect failed with 'Out of memory'
across every shard at once — 369 phantom test failures on a healthy
branch, indistinguishable from real breakage at a glance.

Two default-on layers in scripts/run-unit-parallel.sh:
1. Memory-aware sizing: total concurrency is capped to available memory
   (vm_stat on macOS, MemAvailable on Linux) at GBRAIN_TEST_MEM_PER_FILE_MB
   (default 1536) per concurrent file, shedding shards before intra-shard
   width. Quiet machines are unaffected (banner: mem-ok); pressured ones
   degrade instead of OOMing (banner: mem-adapted AxB→CxD).
2. Serial OOM rescue: failures whose shard log carries the WASM
   out-of-memory signature are re-run at --max-concurrency 1 after the
   fan-out drains. Phantoms pass serially → run goes green with an
   oom_rescued note and the failure blocks marked superseded; real
   failures fail again and stay red. Plain assertion failures never match
   the signature and never enter the rescue lane (existing exit-code and
   failure-log contract tests unchanged).

Escape hatches: GBRAIN_TEST_NO_MEM_ADAPT=1, GBRAIN_TEST_NO_OOM_FALLBACK=1.
Tests: OOM-once fixture rescued to exit 0; kill-switch stays red; banner
advertises the sizing verdict. Documented in docs/TESTING.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test,cli): merge-seam repairs — shard timeout for the tripled suite, init flag-error contract, think scope exemption

Three post-merge repairs surfaced by the first full-suite run:
- Shard timeout 1500s -> 3000s: the suite roughly tripled since the cap was
  sized (~3900 -> 11k+ tests; PGLite inits replay 120 migrations, was 92).
  Two shards were killed mid-progress at 1500s.
- The #2185 pre-dispatch validator now emits the same error contract as
  init.ts's in-handler check it preempts: lowercase 'unknown flag' on
  stderr + structured {status:'error', reason:'invalid_flag'} on stdout
  for --json callers (pinned by test/init-migrate-only.test.ts).
- test/operations-trust-boundary.test.ts gets the same documented
  remote-gated allowlist for think's read scope (#2598) that
  test/oauth.test.ts already carries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): rescue lane also covers externally-killed shards (sibling-workspace pkill / memory jetsam)

Second phantom class observed on a multi-workspace Conductor machine:
3 shards SIGTERM'd + 1 SIGKILL'd at ~700s under a 3000s cap, all
mid-progress — an external killer, not a wedge. The dead shards then
poisoned the serial pass (lock/state residue → 18 more phantoms), and
every one of the 18 passed standalone.

The runner now stamps per-shard start/end epochs; a shard dying on
143/137 before 80% of SHARD_TIMEOUT is classified externally-killed and
its file list joins the serial rescue queue (real wedges die AT the cap
and stay red). Serial-pass failures that occur while any shard was
externally killed are treated as suspect residue and rescued too.
Structural tests pin the detector, threshold, and routing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): keep tokenEndpointAuthMethod terminal in the register-client destructure (PKCE structural contract)

The #2016 absorb appended source/federatedRead after
tokenEndpointAuthMethod; test/fix-wave-structural.test.ts pins
tokenEndpointAuthMethod as the final destructured field (v0.36.1.x #1077
PKCE regression contract). The added fields move into the regex's
optional-middle slot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): re-apply the #2598 remote-gated scope exemption to master's oauth scope-annotation test

Taking master's v0.42.74.0 oauth.test.ts (its #2529 implementation) dropped
the think read-scope allowlist that PR #2598 carries; re-applied to match
test/operations-trust-boundary.test.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): pre-landing review batch — CLI-local flag acceptance, json= coherence, re-landed getChunks trim, payload-aware jsonb repair, rescue-lane tightening

Ship Step 9 findings (checklist pass + 4 specialists), all verified before fixing:
- CRITICAL: the #2185 validator rejected --source/--dry-run on op commands
  (makeContext CLI-locals consumed outside the op contract) — 'gbrain search
  "x" --source y' exited 1. Exempted with parser-mirroring value consumption
  + unit/subprocess tests incl. global-flag acceptance.
- CRITICAL: --json=<v> diverged between validator (accepted) and parseOpArgs
  (junk-key path consumed the NEXT token, corrupting positionals). Parser now
  handles --json=true|false; =-forms of bare-only CLI-locals reject loud.
  parseOpArgs inline-= suite added (regression rule).
- CRITICAL: the master merge silently restored SELECT cc.* in both engines'
  getChunks while docs claimed the #2544 trim. Re-landed the explicit
  non-vector column list + a source-level structural pin so a merge can't
  silently undo it again.
- repair-jsonb/doctor: the subagent columns legitimately hold jsonb string
  scalars (persistToolExec binds pre-serialized strings) — unconditional
  unwrap would abort the repair run or corrupt legit values. jsonPayloadOnly
  predicate (JSON-container content only) on those 3 targets, mirrored in
  doctor + parameterized to_regclass + behavioral test (damage flagged,
  legit string ignored, absent table skipped).
- runner rescue-lane tightening: serial failures rescue-eligible only with
  their own OOM signature or after an external shard kill (residue), never
  because a sibling shard OOM'd — flaky serial tests stay red. Rescue passes
  no longer double-count into TOTAL_PASS; shard timeout scales when
  mem-adaptation sheds shards; negative-path tests (mixed run stays red,
  deterministic OOM-signature failure stays red).
- fail-closed remote spelling at 2 forward sites (ctx.remote !== false per
  the CLAUDE.md invariant); registry generator drops template-literal flag
  prefixes; real-PG e2es for the v121 + minion_jobs wedge classes; stale
  comments corrected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: regenerate flag registry after master merge (#3864 added extract help flags)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): red-team batch — dispatch-order validation, real --dry-run boolean, rescue-lane timeout/isolation/cap, parseable-payload repair gate

Red-team pass (post-specialist) findings, all verified before fixing:
- CRITICAL: validateCommandFlags checked the op lane before CLI_ONLY while
  dispatch runs CLI_ONLY first — dual-lane commands (think/salience/
  anomalies) were validated against the WRONG contract, rejecting documented
  invocations ('salience --kind entity'). Lane order now mirrors dispatch.
- CRITICAL: --dry-run was blessed as legal on op commands but parseOpArgs
  never SET it (trailing → nothing → ctx.dryRun false → the REAL destructive
  action ran; leading → consumed the next token). Now a CLI-local boolean
  exactly like --json, with --dry-run=false support and regression tests.
- CRITICAL: the rescue lane ran bun test WITHOUT --timeout=60000 (bun default
  5s) — PGLite phantoms re-failed on timeout and were mislabeled 'confirmed
  real'. Both rescue invocations now mirror the shard flags; serial files
  re-run one process per file (run-serial-tests.sh isolation contract);
  rescue wallclock capped at 2x the shard timeout.
- CRITICAL: the jsonPayloadOnly probe matched container-LOOKING invalid JSON
  ('[INFO] fetch complete') whose repair cast would throw and abort the run
  mid-loop. Predicate now gates on pg_input_is_valid (PG16+, same floor as
  the existing IS JSON usage) + per-target catch records and continues;
  doctor mirrors; behavioral test covers the lookalike row.
- Registry generator bounded at handleCliOnly's closing brace (the LAST case
  block absorbed ~100 junk flags from the rest of cli.ts, neutering strict
  validation for it); uppercase flag typos reject loudly in both lanes
  (handlers are lowercase-sensitive); --json=true spelling gets the
  structured invalid_flag envelope; get_chunks __all__ narrowing filed as a
  Wave 3 TODO.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engines): getChunks trimmed SELECT must carry modality (codex P1-B)

The #2544 egress trim replaced SELECT cc.* with an explicit column list but
omitted cc.modality — every rowToChunk field except the vector must survive
the trim, or the embed round-trip (getChunks -> upsertChunks) rewrites image
chunks as text. Both engines; the structural pin now iterates the full
rowToChunk field list instead of spot-checking.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): safety flags require consumption evidence in the flag registry (codex P1-A)

upgrade.ts prints a help hint naming another command's --dry-run; that
literal is depth-0 text for post-upgrade, so the generator allowlisted
--dry-run there — recreating the exact #2185 repro this wave kills
(post-upgrade --dry-run accepted, ignored, migrations run for real).
Safety flags now need a tight-quoted standalone literal (an args read
like has('--dry-run')) before the registry grants them; prose bleed
embeds the flag inside a longer string and never qualifies. Regenerated
registry drops --dry-run from post-upgrade, keeps genuine consumers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.42.76.0 fix(upgrade,security,cli): strict flag validation, bootstrap-wedge class kill, federated chunk scope

Version bump + CHANGELOG for the fix wave: CLI-wide unknown-flag
rejection with a generated per-command registry (#2185, #1575 class),
minion_jobs bootstrap probes + migration-aware coverage guard (v121
wedge class), get_chunks federated scope + egress trim (#2555, half of
#2544), think read-scope over MCP (#2598), register-client source
bindings (#2016, #2143 enabler), repair-jsonb/doctor subagent columns
with a parse-validated damage predicate, memory-safe unit-test runner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.42.76.0

CONTRIBUTING.md: test-runner claims match the memory-safe 4-shard default
(was 8-shard) and the CLI-only command recipe now includes the
build:flag-registry regen step. KEY_FILES.md: repair-jsonb entry updated
to the 8-column parse-gated current state; new entries for the strict
flag-validation subsystem and src/core/source-id.ts; operations/engine/
serve-http entries updated for get_chunks scope ladder + trimmed SELECT
and the register-client HTTP source bindings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: apply doc-review fixes for v0.42.76.0

Cross-model doc review caught stragglers: docs/TESTING.md still said
8-shard in the file taxonomy, carried a two-generations-stale shard
timeout default (600s -> 3000s), and didn't name the new
run-unit-parallel regression test or the remaining runner knobs;
CONTRIBUTING.md's fast-loop file count predated the tripled suite
(92+ -> 1000+); test-count claims unified at 3700+; the
CLIENT_FENCED_WRITE_OPS comment in operations.ts still described
think as scope write; KEY_FILES names the exported findUnknownFlag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): rescue lane survives CI — strip ::group:: prefixes, satisfy the bun-test-timeout guard

Two CI-only breaks from the master merge: (1) under GITHUB_ACTIONS the
shard wraps file sections as ::group::path.test.ts, so the rescue pass
extracted literal ::group:: non-paths that matched zero test files —
failing_files_in_log now strips the prefix; (2) master's new
check-bun-test-timeout guard greps for bare 'bun test' and tripped on
run_rescue's comment text (the invocations themselves carry
--timeout=60000) — comment reworded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): shard-mechanics tests disable mem-adaptation — CI's 7GB runner collapsed explicit 2 shards to 1

The runner deliberately adapts even explicit --shards to available memory
(GBRAIN_TEST_NO_MEM_ADAPT=1 is the escape hatch); on GitHub's ~7GB runners
that collapsed the tests' 2-shard sandbox runs to 1 shard, breaking every
'shard 1/2:' expectation while passing locally. The tests pin shard
MECHANICS with tiny synthetic files, so they now set the escape hatch;
the one test that checks the mem banner overrides it back on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 09:49:54 -07:00
MasaandClaude Fable 5 485c4773ca fix(think): extend 16K output headroom to OpenAI reasoning models (#3916)
maxOutputTokensFor() gave the 16000-token output budget only to
thinking-by-default Anthropic Claude 5 models; every other model kept the
4000 default. OpenAI reasoning models (the gpt-5 family and the numbered
o-series) bill internal reasoning tokens as output tokens counting against
max_tokens, so the 4000 cap starved think's actual answer the same way.

Add OPENAI_REASONING_MODEL_RE alongside the existing Claude-5 regex,
deliberately scoped to the gpt-5 family + numbered o-series with the
non-reasoning ChatGPT -chat snapshots (gpt-5-chat-latest,
gpt-5.2-chat-latest) excluded. Defaults (4000/16000) are unchanged and no
config knob is added; gpt-4o, other providers, and provider-prefixed
spellings outside openai:/openai/ keep 4000.

Tests pin the new 16000 cases (colon + slash forms), the chat-snapshot and
codex-mini-latest exclusions, and name/version boundaries (gpt-50, o3foo,
bare model names, openrouter-nested ids).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 19:14:00 +07:00
MasaandClaude Fable 5 f8878be54e fix(cycle): price synthesize_concepts estimates from the canonical table (#3915)
The T1/T2 narrative cost estimate hardcoded Sonnet rates ($3/M in,
$15/M out) regardless of which model actually answered, violating the
one-canonical-chat-pricing-table invariant. On non-Sonnet models
(e.g. openai:gpt-5.2) the ~2.4x over-estimate tripped the $1.50 budget
gate early — degrading remaining T1/T2 groups to the deterministic
template with budget still left — and persisted the inflated cost into
receipts and rollups.

The estimate now resolves through canonicalLookup(result.model), the
model that actually answered. A canonical miss falls back to Sonnet-tier
pricing (derived from the canonical table, not hand-copied), mirroring
skillopt/preflight's lookupPrice policy: conservative, non-throwing, and
it keeps the budget gate effective.

The $1.50 gate, concept-generation logic, and model resolution are
untouched (model resolution is in flight separately on the same file).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 19:00:13 +07:00
MasaandClaude Fable 5 0df56f1efa docs(brainstorm): correct estimateCost's cost figures to match its own formula (#3905)
The doc comment claimed brainstorm ~$0.05-0.15 and lsd ~$0.20-0.40 while
stating it anchors on Sonnet pricing. Plugging the function's own token
budgets (3000 in/cross, 250 out/idea, 350 judge-in/idea, 200 judge-out/idea)
and the profile constants (brainstorm 4x6x3, lsd 2x12x4) into Sonnet's
canonical $3/M-in $15/M-out yields ~$0.78 and ~$0.96 — roughly 5x and
3x the stated ranges. No single model in the canonical pricing table makes
both stated ranges true, and the comment, formula, and profile constants
were all introduced in the same commit (613da940), so this was an arithmetic
slip rather than drift.

Comment-only; no behavior change. The runtime preview already prints the
real number via canonicalLookup.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 18:46:05 +07:00
Ziyang Guo 01d9639204 fix(facts): salvage valid extractor candidates (#3894) 2026-08-09 18:31:44 +07:00
Garry TanandClaude Fable 5 f15480b9d0 v0.42.75.0 fix(pglite): in-place WAL auto-repair for the macOS Aborted() startup crash (#2575, #223, #1670) (#3901)
* fix(pglite): in-place WAL auto-repair for the Aborted() startup crash (#223, #1670, #2575)

The 'macOS 26.x WASM bug' was a misdiagnosis: an unclean shutdown (typically
the OS-upgrade reboot) tears the data dir's WAL, and every subsequent open
fails WAL replay inside WASM with an opaque RuntimeError: Aborted(). This
ports the pg_resetwal recovery upstream rejected (electric-sql/pglite#994,
by @yestheboxer) and wires it into connect() as bounded auto-repair:

- src/core/pglite-resetwal.ts: pg_resetwal for PG17 NodeFS dirs, fail-closed
  layout validation, atomic+durable writes (tmp+fsync+rename), idempotent.
- src/core/pglite-repair.ts: whole-pg_wal-dir rename backup (zero transient
  disk), overwrite-order restore with mtime guard, cooldown sidecar +
  episode-scoped backup retention (newest 3 episodes), and a never-throws
  engine seam. Kill-switch: GBRAIN_PGLITE_WAL_REPAIR=off.
- pglite-engine.ts: verdict rename macos-26-3 -> wasm-abort, classifier now
  matches the real production message (it previously fell to 'unknown'),
  corrupt-beats-wasm precedence preserved, honest per-outcome error copy
  incl. the failed-not-restored arm, and repair only under a cleanly-acquired
  lock (new LockHandle.reaped provenance; never after reaping a holder).
- gbrain pglite-repair: manual dry-run/repair command (validate-before-lock,
  serve/reaped refusals, no --force by design).
- doctor: pglite_data_dir fs-check with recurrence escalation and backup
  inventory when a PGLite brain fails to connect.
- reinit-pglite: embedding flags default from file-only config so the
  recovery ladder's rebuild rung works bare mid-outage.
- stringifyPgliteInitError: message-less Emscripten ErrnoError objects no
  longer surface as [object Object].

Regression-tested against real brains: corrupt every WAL segment (truncate
and garbage variants), reopen, auto-repair fires, original rows readable,
process.exitCode stays contained (#2084).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): replace the macOS-26.x misdiagnosis with the corrupt-WAL recovery ladder

README + INSTALL.md shipped (via #1671) the claim that PGLite is incompatible
with macOS 26.x and that a Bun/WASM fix would restore it. The real cause is
torn WAL state from the upgrade reboot, now auto-repaired in place. Rewrites
those sections around the recovery ladder (auto-repair -> gbrain pglite-repair
-> reinit-pglite -> engine switch; native-Postgres recipe kept, credit
@roysaurav), adds the ENGINES.md troubleshooting section, updates the
KEY_FILES.md entries to current truth, files the two follow-up TODOs
(SIGTERM engine-close extension; pglite upgrade blocker), and regenerates
the llms bundles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): harden WAL auto-repair (pre-landing + adversarial review)

Review-army (security/testing/maintainability/perf) + Claude & Codex
adversarial passes on the WAL-repair wave. Correctness + safety hardening,
no behavior change to the happy path:

- Live-writer safety: repair refuses any reaped lock acquisition, a corrupt
  (unknowable-liveness) reap writes a cross-process quarantine marker that
  gates auto-repair AND the manual command for 10 min, isProcessAlive treats
  only ESRCH as dead (EPERM/malformed-pid read as alive), and a live
  postmaster.pid (native Postgres) is refused. Lock heartbeat + initial write
  are atomic (tmp+rename) so a torn read can't misclassify a healthy holder;
  an in-flight acquisition is no longer mistaken for corrupt.
- resetWal verifies the stored pg_control CRC before trusting/re-signing it —
  a damaged control file routes to rebuild instead of laundering corrupt
  checkpoint counters under a fresh CRC. Atomic 'wx' writes (no symlink
  follow), whole-pg_wal-dir rename backup, 64MB seg-size cap.
- Honest failure reporting: repairPgliteWal threads the real restore result
  out via WalRepairError so the 'failed-restored' vs 'failed-not-restored'
  message never lies; the not-restored copy names the correct restore paths.
- Episode lifecycle: episodes close on the next healthy connect (not just on
  a verified repair), a gutted (restored) backup loses its pin, stale (>24h)
  episode backups aren't reused, and the cooldown also caps repaired-only
  crash loops. Empty backup dirs are pruned on refusal.
- Command: rejects unknown flags and valueless --path (a destructive command
  must not silently mis-parse), confirm prompt goes to stderr (stdout stays
  clean for --json), embedding-flag defaults come from the config file only.
- Symlink confinement extended to global/; sidecar reuse path validated
  (prefix + no '..' + must still hold pg_wal); sidecar writes atomic.
- doctor recurrence escalation counts all attempts; data dir absolutized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): current-state KEY_FILES + WAL-repair follow-up TODOs

KEY_FILES.md pglite entries updated to the hardened truth (reap marker +
quarantine, atomic writes, CRC gate, global-symlink refusal, WalRepairError,
episode lifecycle). TODOS.md files the deferred judgment-call follow-ups
(unclean-shutdown gate on auto-repair; non-gbrain pglite consumer boundary;
mixed-version torn-lock double-read).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.75.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:01:20 -07:00
Ziyang Guo 0b47afbf40 fix(doctor): resolve MSYS executable paths (#3865) 2026-08-08 19:37:38 +07:00
Ziyang Guo 9f81e1d8b2 fix(cli): expose extract flags in help (#3864) 2026-08-08 19:21:05 +07:00
Ziyang Guo 86555e39bb fix(autopilot): honor source pull policy (#3861)
Fixes #3835

Use a canonical positive pull flag for sync and autopilot-cycle jobs while preserving legacy noPull payloads. Parse source config consistently so local-only sources skip pulls and PGLite remote sources still pull.

Test: bun test test/job-pull-policy.test.ts test/autopilot-fanout.test.ts test/autopilot-fanout-wiring.test.ts test/sources-load.test.ts
2026-08-08 19:03:36 +07:00
zay 3257758492 fix(doctor): align maintenance scope with canonical files (#3854) 2026-08-08 18:45:53 +07:00
Garry TanandClaude Fable 5 a948dfd6e2 v0.42.74.0 fix(security): honor takes_holders over serve --http + agent-voice default-deny CORS (#2529 #2477) (#3868)
* fix(auth): honor permissions.takes_holders for legacy bearer tokens over serve --http (#2529)

The OAuth provider's legacy access_tokens branch parsed permissions.source_id
but never read permissions.takes_holders, so the /mcp dispatch site's
fail-closed default pinned every remote caller to world-only takes visibility
— set-takes-holders was a silent no-op over serve --http, in both directions
(grants above world never applied; restrictions below world didn't either).

- src/core/legacy-token-scope.ts: new parseTakesHoldersAllowList shared by
  BOTH transports (the drift between the legacy HTTP transport's correct
  inline parse and the OAuth provider is how this bug shipped). [] preserved
  as explicit deny-all; non-array → undefined → consumer defaults ['world'].
- src/core/operations.ts: AuthInfo.takesHoldersAllowList typed field
  (same ride-along as sourceId/allowedSources).
- src/core/oauth-provider.ts: legacy branch threads the stored grant.
  OAuth-client tokens unchanged (no per-client storage — TODO filed).
- src/mcp/http-transport.ts: converged on the shared helper (behavior no-op).
- src/commands/serve-http.ts: sidecar cast replaced by the typed field.
- src/core/facts/meta-hook.ts: hashAllowList gives [] its own cache key
  (cache identity only — payload filtering stays visibility-based).

Tests: 7 verifyAccessToken cases (grant/absent/garbage/deny-all/mixed/
oauth-client/column-default), pure-helper describe, meta-hook cache-key pin,
and a Postgres e2e (test/e2e/serve-http-takes-holders.test.ts) pinning the
issue repro end-to-end over POST /mcp — the seam that had no coverage.

Reported by @Derek95king.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(agent-voice): default-deny CORS + origin gate + loopback bind in reference server (#2477)

The recipe reference server reflected any Origin into
Access-Control-Allow-Origin on every route, including the unauthenticated
side-effectful POSTs. Recipe is install_kind: copy-into-host-repo, so the
reference is the source of truth for every future install.

- Default-deny CORS: headers emitted only for exact matches against the new
  AGENT_VOICE_CORS_ORIGIN allowlist (comma-separated), with Vary: Origin;
  Allow-Credentials never set. Same-origin /call needs no configuration.
- Origin gate on /session and /tool: CORS headers gate response reads, not
  request sends — a no-preflight "simple" cross-origin POST still executes.
  Disallowed Origins now 403 before any body read / upstream fetch / tool
  dispatch. No-Origin callers (curl, Twilio, native) and same-origin pages
  (Origin host == Host, tunnels included) pass. DNS rebinding stays a
  documented production-checklist item (TODO filed).
- Loopback-default bind: HOST env, default 127.0.0.1 (mirrors gbrain
  serve --http --bind default); HOST=0.0.0.0 for containers/LAN.
- Startup log prints the bind + CORS posture; recipe md + install manifest
  bumped to 0.1.1 with the production checklist rewritten to match.

Tests: test/agent-voice-cors.serial.test.ts spawns the real server twice and
pins default-deny, allowlist echo + trimming, preflight behavior, and the
gate's ordering (evil-origin 403 vs no-origin reaching the handler).

Reported by @sebastiondev.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: agent-voice origin gate fails closed on malformed Origin (#2477)

Coverage-audit follow-up: a cross-origin POST with an unparseable Origin
header must 403 (new URL() throws → originAllowed returns false), never
fall through to the handler. A bypass here would defeat the gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: pin loopback-bind default + wire-level fail-closed takes default (#2529 #2477)

Pre-landing review (testing specialist) flagged two revert-catching gaps:

- #2477 HOST default (127.0.0.1) had no assertion — a regression to
  all-interfaces would pass every test. Capture the server's startup log
  and assert the loopback bind; add a HOST=0.0.0.0 override case.
- #2529 serve-http `?? ['world']` default branch was only unit-covered.
  Add a 4th e2e case: a legacy token with no takes_holders key sees
  world-held takes but NOT brain-held ones over POST /mcp, pinning the
  fail-closed default end-to-end. Also assert the deny-all case returns a
  successful (non-error) tool result so the negatives can't pass vacuously.

Verified: agent-voice 4/4, serve-http-takes-holders e2e 4/4 (real Postgres).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): shared permissions decode + collision-free hot-memory cache key (#2529)

Adversarial-review hardening on the #2529 fix:

- The two transports shared parseTakesHoldersAllowList but still decoded the
  permissions column differently: the OAuth provider JSON.parse'd a
  string-typed value, the legacy HTTP transport didn't. On a double-encoded
  jsonb string scalar (#2339 class) a deny-all token would fail open to
  ['world'] on the HTTP transport while the provider honored it. Extract
  coerceLegacyPermissions into the shared module and route both through it, so
  "the two transports cannot drift" is literally true (shared decode + shared
  parse). Arrays/scalars/malformed strings → undefined (no grant).
- hashAllowList used bare sentinels ('_' for undefined, '(empty)' for []),
  which collided with real holder values ['_'] and ['(empty)']. Encode
  collision-free (undefined → 'none', else JSON.stringify(sorted)) so the
  []-vs-undefined cache separation the #2529 change relies on holds for every
  holder value.

Tests: coerceLegacyPermissions unit cases (object/JSON-string/malformed/
array/scalar), all existing takes-holders + meta-hook + e2e suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* v0.42.74.0 fix(security): honor takes_holders over serve --http + agent-voice default-deny CORS (#2529 #2477)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: document agent-voice HOST + AGENT_VOICE_CORS_ORIGIN env vars in install hint (v0.42.74.0)

The post-install hint's env-var quick-start predated the #2477 hardening and
listed neither the loopback-default HOST bind nor the default-deny
AGENT_VOICE_CORS_ORIGIN allowlist. Add both as optional entries (safe by
default) and refresh the stale startup-log line to match the server's actual
loopback-bind output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: correct CHANGELOG command guidance + TODOS filing version (#2529 #2477)

Ship-stage document-release caught two wrong CLI invocations in the v0.42.74.0
CHANGELOG "To take advantage" block: `gbrain auth permissions <token>` has no
read-only view form (that shape errors + exits 1 — set the scope directly with
`set-takes-holders <values>`), and `integrations install agent-voice --refresh`
requires `--target <host-repo>`. Also correct the follow-ups TODO header from
the plan's stale v0.42.56.0 guess to the actual ship version v0.42.74.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(deps): bump js-yaml to 3.15.1 — osv GHSA-5p4m-2wfm-xmqj (#2529 #2477)

osv-scan flagged js-yaml@3.15.0 (High, CVSS 7.5), fixed in 3.15.1. The
transitive copy (gray-matter → js-yaml) was pinned to ^3.15.0 by the
package.json `overrides` block; bump both the direct dependency and the
override to ^3.15.1 so the vulnerable version is gone from bun.lock entirely
(gray-matter/js-yaml now resolves to 3.15.1). Patch bump, in-range, frontmatter
parsing verified (markdown + frontmatter + import + oauth suites green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(deps): bump admin nanoid to 3.3.18 — osv GHSA-2v37-7h3g-55p8 (#2529 #2477)

osv-scan flagged nanoid@3.3.16 (High, CVSS 8.2) in admin/bun.lock, fixed in
3.3.17. nanoid is transitive (postcss → nanoid), so pin it in the admin
overrides block; refresh resolves to 3.3.18 (latest patched 3.x). Admin SPA
build verified green; both root and admin lockfiles now scan clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-07 15:31:53 -07:00
15b9863d13 v0.42.73.2 fix(security): fence dedup-resolved writes to the caller's own write scope (#3809)
* fix(security): fence the dedup-resolved slug under the caller's own confinement

put_page's resolved-slug re-check tested `ctx.auth.boundSlugPrefixes` only.
The delegated submit_agent -> subagent context carries `viaSubagent` +
`allowedSlugPrefixes` but no `auth`, so a slug-bound client holding `agent`
scope could delegate a write and have importFromContent's dedup pre-check
redirect it onto a page outside its grant — where the disk write-through
then re-rendered the victim's file with the caller's provenance.

The re-check now applies whichever confinement the caller is actually
under (OAuth binding and/or subagent allow-list / legacy namespace) via
`slugOutsideCallerFence`, which composes the existing match rules rather
than re-deriving them. Dedup returns status 'skipped' before any DB write,
so the throw still rolls nothing back. The denial does not name the
resolved slug (slug-enumeration oracle) and reads "your write scope",
since either confinement can trigger it.

Reported privately by Aleksei Razsadin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: coverage for the OAuth in-fence redirect and the missing-subagentId guard

* v0.42.73.2 fix(security): fence dedup-resolved writes to the caller's own write scope

VERSION + package.json + CHANGELOG for 0.42.73.2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: state that the write fence follows a delegated write

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 07:59:50 +07:00
aecb33e795 v0.42.73.1 revert(ci): remove the PR gate and withdraw the v0.42.72.1 contribution requirements (#3805)
* v0.42.73.1 revert(ci): remove the PR gate — it cannot function on this repository

The gate needs an ANTHROPIC_API_KEY Actions secret for its verdict and
read-write workflow permissions to post a comment or set a label. This
repository grants neither, and both are admin-only, so it can only ever skip.

It also caused a real incident before that was understood: a read-only token
turned every API call into a 403, the code treated that as a crash, and the
check went red on an outside contributor's PR four times with no comment
explaining why. v0.42.73.0 fixed the crash, but a check that runs on every PR
and can never reach a verdict does not earn a place in the repo.

Removes the workflow, the script, its type surface, and its test file. The code
is preserved in git history at v0.42.73.0. If it is ever restored, the
mechanical half (intent/screenshot policy, title rule, red flags) should render
to the Actions job summary rather than a comment — that needs no token
permission and no API key.

CONTRIBUTING.md's intent-paragraph and screenshot requirement is unchanged and
stands as written; it is enforced by maintainers reading PRs, as before.

typecheck clean, verify 34/34, llms bundles regenerated.

* v0.42.73.1 revert(docs): withdraw the human-intent-paragraph + screenshot contribution requirement

This reverts commit 6d1232d5a6 (v0.42.72.1).

CONTRIBUTING.md, both issue templates, and the pull-request template return
to their pre-2026-08-02 state. VERSION/package.json/CHANGELOG keep moving
forward (0.42.73.1); the v0.42.72.1 CHANGELOG entry stays as historical
record, and the 0.42.73.1 entry now describes the withdrawal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 04:20:30 +07:00
Garry Tan e5dee4fb78 fix(deps): bump four override pins past newly-published CVEs
osv-scan passed on the last two release PRs and fails on this one, so these are
newly published, not tolerated debt: 2 High + 4 Medium across fast-uri (3.1.4 →
3.1.5), hono (4.12.25 → 4.12.34), ip-address (10.1.1 → 10.3.1), and admin's
postcss (8.5.10 → 8.5.23). All four are transitive and pinned through the
overrides block, so the fix is the pin, not a dependency addition.

Resolved: fast-uri@3.1.5, hono@4.13.0, ip-address@10.4.0, postcss@8.5.25.
2026-08-04 22:25:25 +08:00
Garry Tan 3c26f2eaf2 fix(ci): a permission failure must never red-X a contributor's PR (live incident)
The gate's first live runs put a red X on an outside contributor's PR five
times over, with no comment explaining why. This repo's GITHUB_TOKEN is
read-only, so every comment and label call returned 403; the throw reached the
top-level handler as exit 2.

The gate is advisory. A repository permission problem is an operator condition,
never a statement about the PR under review. 401/403/404 from GitHub now emit a
loud operator-facing warning naming both blockers (workflow permissions and the
missing ANTHROPIC_API_KEY secret) and exit 0. A genuine outage or a bug in here
still fails visibly at exit 2. Pinned by four tests including the entry-handler
routing; mutation-tested.
2026-08-04 22:25:25 +08:00
Garry Tan 615c33e5b5 v0.42.73.0 feat(ci): strict PR usefulness gate + five contributed correctness fixes
Release-only commit: VERSION, package.json, CHANGELOG. All code already on
master. Covers the 8 gate commits (#3794, closes #3698), plus #3764 import
stdout, #3759 dry-run chmod, #3726 cycle model telemetry, #3751 integrity
counter, #3739 doc comment, and #2655 slug_filter.
2026-08-04 22:25:25 +08:00
40d1d4cabc fix(import): make imported pages readable by gbrain's own conversation parser (#3788)
* fix(envelope importer): honor the envelope's own counts, and stop a second import destroying the first

Two ways this importer lost conversation history at exit 0 with zero stderr
bytes. Both reproduced against output from the real memvelope converter, not
hand-authored envelopes — the blind spot that let them through review.

F1. envelope-v0 makes meta.conversation_count and meta.message_count mandatory:
the envelope states its own totals. The script read neither. A file declaring
353 conversations and 9412 messages that actually held one of each imported
one, exited 0, and printed a receipt internally consistent with the reduced
output. The receipt also counted only pages, never messages, so every
message-level loss was invisible by construction — a conversation arriving with
one turn instead of forty still writes exactly one page.

Each declared count is now checked on its own, before the first write. A count
that disagrees with the contents, or that is present but is not a non-negative
integer, refuses the import (exit 2). A count that is simply absent cannot be
compared, so the envelope still imports — but stderr names the field whose half
of the check was skipped, because an unchecked import must never look like a
checked one. The receipt reports messages as well as pages, and says so out
loud when messages read exceed messages written.

F2. filesWritten was per-process and the default outDir is a fixed literal, so
a second import into the same directory clobbered the first with no read-back
and no warning. The trigger is the positional fallback conv-N, which fires
whenever c.id is not a non-empty string — and the spec is explicit that id is
string | null and that a converter MUST NOT synthesize one, so null is the
CONFORMING shape, not malformed input. The reference converter emits it for any
ChatGPT export lacking conversation_id and id. Filenames then key off array
position rather than identity, and two unrelated exports both put their first
conversation at conv-1.

Target files are now examined before anything is written. Byte-identical
content is a re-import; a page this importer wrote from the SAME conversation
id is a refreshed export legitimately updating its own page. Anything else — a
foreign file, or one of ours whose conversation id cannot be matched — refuses
the import (exit 2) and writes nothing. Refusing rather than disambiguating
keeps the importer from inventing an identity the envelope declined to state,
and the remedy is a different output directory.

Both checks run before the first write, so a refused import leaves no partial
output to be mistaken for a whole one, and does not even create the directory.

Verified against all 12 golden fixtures from the memvelope reference converter
and 7 envelopes freshly produced by running that converter over synthetic
ChatGPT and Claude exports: every one imports at exit 0 with zero stderr bytes,
and all 19 produce byte-identical pages to the previous script. The guards do
not false-positive on legitimate producer output.

Known limit, documented in the header: the target-file check is check-then-
write, not atomic. Two simultaneous imports into one directory can both pass it
— measured 19 refused / 21 raced over 40 trials, against 0 / 40 before.

17 new tests; the 13 existing tests are unchanged and still pass. No new
dependencies: package.json and bun.lock are byte-identical.

* fix(envelope importer): close the three holes the adversarial pass found in the guards

An eight-agent refutation round attacked every claim made for the previous
commit. Four claims survived untouched; three did not, and all three were in
code that commit introduced. Each is reproduced, then closed, with tests.

Identity was compared on the TRIMMED conversation id, because that is what the
filename slugs and what the frontmatter recorded. So two ids differing only by
surrounding whitespace — both copied verbatim by the reference converter, both
schema-valid — looked like one conversation to the guard, and the second import
destroyed the first at exit 0 with zero stderr bytes. The surviving page then
recorded an id that appeared nowhere in the envelope that wrote it. The
frontmatter now records the id verbatim, as the spec requires, and identity is
matched raw. The filename still slugs the trimmed form, so no page's bytes
change for any id without surrounding whitespace.

The identity scan required a page to start with exactly `---\n`. A page THIS
IMPORTER WROTE that later picked up CRLF line endings or a UTF-8 BOM — a git
checkout with core.autocrlf, a cross-platform sync, an editor save — was
therefore reclassified as foreign, and one such page refused the ENTIRE
envelope at exit 2. The previous script treated the same mutation as a harmless
overwrite, so the guard had turned a cosmetic byte change into an unrecoverable
block. Both are normalized away before the scan. The message for a file that
genuinely cannot be recognized no longer asserts "was not written by this
importer" — a claim this code cannot make, and one that was false for exactly
the pages it was being printed about.

The new message-delta warning announced "N message(s) in the envelope are not on
disk" whenever pages collided. Duplicate ids are conforming input — the spec has
merging never deduplicate — so converting an old export together with a newer
one, which is what the memvelope CLI tells users to do, fired it routinely while
every unique turn was on disk. It now states what it knows: the overwritten
pages carried N messages that are not on disk, and if those were earlier copies
the surviving page may already hold them. The raw tally stays, since hiding it
is what made message-level loss invisible to begin with.

Two header claims were false as written and are corrected: message text is
byte-verbatim for 18 of the 19 producer envelopes, not all 19 — the
lone-surrogate fixture writes U+FFFD, unchanged from before and out of scope —
and sequential coverage is no longer described as complete. Three further limits
are now stated rather than left implicit: an id-less conversation cannot be
refreshed in place (refused, not applied — the deliberate trade, since the same
ambiguity resolved the other way is the defect being fixed); a conversation
whose created_at moves orphans its earlier page rather than updating it; and a
hand-written lookalike is indistinguishable from a page this importer wrote.

Rejected after reproducing it: a "silent loss" via a duplicate top-level
conversations key in the JSON text. JSON.parse keeps the last such key, which is
language semantics, is identical on the previous script, and cannot be produced
by JSON.stringify.

36 tests pass, 0 fail. The 13 pre-existing tests remain byte-identical as the
first 306 lines of the file. All 19 real-producer envelopes still produce
byte-identical pages. package.json and bun.lock unchanged.

* fix(envelope importer): write pages gbrain's own conversation parser can read

Every page this importer wrote declared `type: conversation` — which opens the
gate to conversation-facts extraction, chronicle eligibility and the
conversation_format_coverage check — and then presented a turn header

    **Assistant** (2025-11-02T14:22:51.000Z · m2):

matching none of the 17 built-in patterns in the conversation parser. The
extractor parsed zero messages, incremented `pages_skipped`, and said nothing.
Pages were stored and searchable; no fact was ever extracted from any of them.

Measured on two throwaway PGLite brains fed the same 13 conversations, one
written each way: `conversation-parser scan` goes from 13/13 `no_match` with 0
messages to 13/13 `imessage-slack` with 33; `extract-conversation-facts
--dry-run` goes from "Skipped 13 page(s)" to every page segmenting and reaching
the extractor; `doctor` conversation_format_coverage goes from warn "13/13 ...
match NO built-in pattern" to ok "13 pages: imessage-slack=13".

The turn header is now `**Me** (2025-11-02 14:22):`, the one shape that parser
reads. 24-hour, not 12-hour-with-AM/PM: both match and both were measured to
reconstruct all 24 hours exactly, so the tie is broken on the fact that 24-hour
is a substring of the envelope's own `ts` (no hour arithmetic, so the 12/0
boundary cannot be got wrong) and sorts chronologically within a day.

That header can carry a wall clock and nothing else, so per-message identity
moves to frontmatter as an array of maps:

    messages:
      - id: "m1"
        ts: "2025-11-02T14:22:51.000Z"

An array, not a map keyed by id — a map discards order and collapses the
duplicate ids the spec permits. Consumers index it by position.

Every scalar is JSON-encoded, so every timestamp is quoted. Unquoted, js-yaml
reads an RFC 3339 scalar as a JS Date: microseconds truncate, a +05:30 offset
normalises away, and gbrain's own coerceFrontmatterString slices a Date to 10
characters — the time of day gone. It is sticky, too. The suite fails if a
timestamp is ever emitted unquoted, and carries a sentinel proving that guard
fires.

Also drops the `---` rule between turns: it is a non-blank line matching no
pattern, so the parser appended it to the preceding message and every extracted
text ended `...\n---`.

Verified over all 12 golden fixtures from the memvelope reference converter
(exit 0, zero stderr, 33/33 messages, every id and ts byte-identical after the
round trip) and over fresh envelopes built by running that converter over
synthetic vendor exports. 52 tests green, up from 36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(envelope importer): build the offset-shifted header clock without Date.UTC

Self-review of the branch, before the refutation round. `headerClock()` is the
only place in this script that does arithmetic on a timestamp, and its offset
branch had two ways to emit a header that is off — or that does not parse at
all.

`Date.UTC` applies MakeFullYear: a year of 0..99 is read as 1900+y. The shape
regex accepts any four digits, so `0050-01-01T00:30:00+05:30` was rendered as
1949-12-31 — the page moved by nineteen hundred years, silently, at exit 0.
Built with the UTC setters instead, which do not remap.

And `getUTCFullYear()` returns `49`, not `0049`. The pattern's regex requires
`\d{4}`, so an unpadded year emits a header that matches nothing and the turn is
appended to its neighbour as a continuation — one message where there were two.
The year is now padded like every other field.

Neither is reachable from the reference converter, which emits `Z`. Both are
reachable from a conforming envelope: the spec types `ts` as any date-time
string, and RFC 3339 offsets are legal.

18 new tests pin the clock on its own: every Z / offset / designator-less form,
both day-boundary crossings, `+0530` without a colon, the sub-100 year in both
branches, and the six unusable-`ts` shapes that must fall back to the
conversation's date at midnight rather than fabricate a clock or drop a turn.
Three more hostile ids added — U+2028, U+2029 and U+0085 are line breaks in
YAML 1.1 and JSON.stringify emits all three raw, so they are the sharpest
version of the injection the quoting exists to stop. js-yaml 3.14 does not
honor them as breaks; measured, not assumed.

72 tests pass, 0 fail. Adjacent suites (conversation-parser, markdown,
extract-conversation-facts, doctor backlog): 380 pass, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(envelope importer): name the built-in pattern count the header claims

'None of the built-in patterns' is a claim about a number the reader cannot
see. There are 17, and `gbrain conversation-parser list-builtins` is how to
count them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(envelope importer): stop the conversation title manufacturing a turn

An adversarial round against the new format. Three claims in the file header
were wrong or overstated, and one of them was a defect.

The H1 heading is the only place a third-party string reaches the BODY — every
frontmatter value is JSON-escaped, the heading was interpolated raw. That was
merely untidy while nothing parsed the body. Now that the turn headers are
legible, a title carrying a newline manufactures a TURN, and one that lands
AHEAD of every real one, so `messages[0]` in frontmatter names content the user
never sent and every id after it is off by one. Measured: a 2-message
conversation titled "Real Title\n\n**Me** (2020-01-01 09:00): INJECTED" parses
to 3 turns, the first attributed to 2020. The heading is flattened to one line;
the verbatim title, newlines and all, is still recorded in frontmatter.

The empty-title fallback also disagreed with itself — "Untitled conversation"
in frontmatter, "Conversation" in the heading, two names for the same missing
thing, and parseMarkdown prefers the body's H1 when frontmatter has no title.
One fallback now.

Corrected in the header, because a false claim there is worse than a limit:

"the frontmatter array still holds exactly the real turns, so the two can be
reconciled by count" was FALSE. parse.ts picks one pattern per page, scored on
the first 10 body lines, and only re-scores full-body under 0.3 — so a pasted
Slack or Telegram snippet inside one message only has to win that window, and
the length of the real conversation is irrelevant. With four `**[09:0N] Name:**`
lines quoted inside message 1: 40 real turns are replaced by 4 fabricated
speakers at fabricated times, at exit 0, with phase regex_match, so
pages_skipped stays 0 and doctor's conversation_format_coverage reports OK. And
at 4 real turns against 4 pasted lines the COUNTS AGREE while every speaker and
timestamp is fabricated — which is exactly the check the old wording offered as
the remedy. The table is in the header now, with what count-comparison does and
does not catch. Closing it needs a parse.ts change, not an importer change.

"every timestamp is QUOTED" was true only for conforming input. JSON quotes
strings; an envelope whose `ts` is a number emits `ts: 1762093371000` unquoted.
That is a YAML integer, not a Date, so it is lossless and carries no truncation
hazard — but the claim as written was wrong, and the behavior is now pinned by
test rather than described.

Two more limits stated: minute resolution is the parser's ceiling, not this
format's — every branch of buildIso hardcodes :00 and no built-in captures
seconds, so two turns in the same minute collide (claude-basic m1 15:02:00 and
m2 15:02:31 both parse to 15:02:00Z) and frontmatter `ts` is the only
full-resolution value on the page. And the array survives a gbrain rewrite
semantically, not textually: serializeMarkdown re-emits `id: "m1"` as `id: m1`.

81 tests pass, 0 fail. Adjacent suites: 389 pass, 0 fail. All 12 golden
fixtures still import at exit 0 with zero stderr.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(envelope importer): range-check the per-message clock, and arm two guards

Second adversarial round. The shape regex counts digits; it does not know a
calendar. `2025-99-99T99:99:00Z` was copied straight into a turn header — and
`imessage-slack` MATCHES that header, so gbrain stored an instant no calendar
contains. `2025-02-30` is the sharper one: it yields a VALID JS Date, silently
shifted to March 2. The script already validates `created_at` before it reaches
a header, with a comment explaining why; the per-message clock is the same
untrusted surface and is used far more often. It is validated now, by round
trip — a date that does not survive its own UTC round trip was never a date.

Also removes a provably unreachable `Number.isFinite` check: over the whole
space the shape regex admits, the constructed instant is always finite.

Two guards had no test that fires. Both are now pinned:

- The `created_at` validation. Dropping it let a hostile value break the turn
  header it was supposed to anchor, and the whole suite stayed green.
- The determinism guard — the reason TS_SHAPE exists rather than
  `new Date(string)`, which parses a designator-less date-time as LOCAL time.
  The one test that touched it only caught the mutation because this box is
  America/New_York; on a UTC runner it passed. TZ is now pinned explicitly and
  the case runs under four zones including UTC and Pacific/Kiritimati.

94 tests pass, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(envelope importer): correct five claims the adversarial round falsified

None of these change behavior. All five were statements in the file header that
an independent pass could not reproduce, and a false claim in a header is worse
than a limitation, because a reader has no way to tell.

"the field is read nowhere outside builtins.ts", of `time_format`, was flatly
false: src/commands/conversation-parser.ts reads it twice. Both are display —
`list-builtins` prints it — and the operative half stands, since parse.ts never
reads it and converts off the captured AM/PM group instead. Said that way now.

"turns averaging more than ~19 non-blank lines fall below the floor" was
optimistic by one line: measured line by line, 18 parses (0.0526) and 19 does
not (0.0499), because the H1 is in the denominator too.

"a fallback when `ts` is null" understated the condition. It fires for any
string this script will not read a clock out of — a date with no time, a basic-
format 20251102T142251Z, an impossible 2025-02-30 — silently preferring the
conversation's date over a value that looked like a time.

"Every emitted scalar is JSON-encoded" was false for the four keys the script
writes itself (`type:`, `origin:`, `date: null`, `messages: []`), which are
literals under its own control rather than envelope data.

"collapses the duplicate ids the spec permits" cited the wrong half of the
spec: duplicates are tolerated for CONVERSATION ids; message ids are positional
and unique within their conversation. The real argument for an array is that
order IS the join. And since a conforming envelope's message id is derivable
from its index, `ts` is the only genuinely new value the array carries — worth
saying rather than implying more is being rescued than is.

Also: the STATUS block quoted `gbrain conversation-parser scan` as producing a
13-page aggregate; it takes one slug and has no aggregate form. The numbers
were reproduced per-slug, so the receipt now shows the command that produced
them.

And two tests changed by this branch still used `page.split('---')[1]` to reach
the frontmatter — the exact idiom this branch's own helper documents as unsound,
since it cuts on the substring anywhere including inside a quoted value. Both
now go through that helper.

94 tests pass, 0 fail. Adjacent suites: 402 pass, 0 fail. All 12 golden
fixtures still exit 0 with zero stderr and 33/33 messages parsed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(envelope importer): break a duplicate id on updated_at, not array position

A merged re-export kept the STALE copy, deterministically.

The memvelope CLI's own USAGE tells users to pass every downloaded export
at once; folder expansion sorts by filename (cli/convert.mjs expandInputs)
and SPEC.md rule 8 forbids the converter from re-sorting conversations
afterwards. Every automatic duplicate-namer a browser or OS gives a second
download of `conversations.json` inserts a character that sorts below `.`
— ` (1)`, `(1)`, `-1`, ` 2` — so the RE-EXPORT lands first in
`conversations[]` and the ORIGINAL lands last. Resolving the collision by
array index therefore threw away the newer copy every time, not half the
time.

Reproduced with an envelope built by the reference converter from two
ChatGPT downloads named the way a browser names them: 4-turn re-export at
index 0, 2-turn original at index 1, and the 2-turn page reached disk at
exit 0.

`updated_at` is a required conversation key in envelope-v0, is populated
by both vendor paths of the reference converter, and was read by nothing
here. It now decides, compared as an INSTANT (a `+05:30` value sorts above
a `Z` value it precedes) at full sub-second resolution (the converter
emits milliseconds always).

The tiebreak applies only when BOTH copies carry an orderable value.
Equal, absent, non-string or unparseable on either side falls back to
array order — a missing timestamp is not evidence of being older, and
preferring the copy that has one is a guess dressed as a rule. `created_at`
is not a secondary key: it is identical in both copies of a re-export.

The collision warning is not quieter — it is louder. It now names which
copy survived, on what basis, and with both timestamps, because
"overwriting the earlier page" is false whenever the tiebreak fires. The
run summary says "discarded" rather than "overwritten" for the same
reason: the losing copy is now sometimes never written at all.

`updated_at` is still written to no page. That is a separate, tracked gap.

Receipts: 107 pass / 0 fail (94 pre-existing, unweakened; 13 new, red-first
against cd88d1de at 96 pass / 11 fail). Over all 12 golden fixtures from
the reference converter plus the repo's sample, output pages, stdout,
stderr and exit code are byte-identical to cd88d1de — this fires only on a
filename collision.

* docs(envelope importer): name the extraction path, the UTC day, and drop a contradicted key claim

R2. The header promised `type: conversation` "keeps pages eligible for
conversation-facts extraction ... after sync" and stopped there, which
reads as "sync extracts facts from these." It does not. The accepting
path is `gbrain extract-conversation-facts`, run by hand, and its
autopilot wrapper — the `conversation_facts_backfill` cycle phase — is
opt-in and OFF by default. Both names verified against upstream/master
before writing them: src/cli.ts:1964 dispatches the command,
src/core/cycle/conversation-facts-backfill.ts:4-6 and :34 document the
phase as "Default OFF" with `enabled (false)`, and doctor.ts:3373 reads
an absent key as disabled. What the type actually buys is admission to
`ALLOWED_TYPES`, and that is now what the header says.

R3. The page `date` and every turn header are the UTC calendar day, not
the user's, and the limits block did not say so. Reproduced on this box:
a conversation held at 19:30 on Sunday 2 November 2025 in California is
`created_at: "2025-11-03T03:30:00.000Z"`, and the page comes out
`date: "2025-11-03"` — a Monday — with a Monday turn header. The output
is byte-identical under TZ=UTC, America/Los_Angeles, Asia/Kolkata,
Pacific/Kiritimati, Pacific/Midway and Europe/London (one sha across all
six), so this is the format's day, not the importing machine's.

Named as unfixable, with the reason rather than an apology: envelope-v0
renders every timestamp as `...Z` and SPEC.md rule 3 says why — "The
source's true offset is unknowable, and UTC is the only
machine-independent choice." A source offset is normalised away before
this script opens the file, and the vendor fields the reference
converter reads carry none to preserve: over the 22 conversation-level
timestamps in its own input fixtures, 16 are bare unix-epoch numbers, 4
end in `Z`, 2 have no designator, and 0 carry a numeric offset. No fix
is invented; the one available guess (the importing machine's zone)
would break determinism.

R4. Deleted the claim that "the id is the natural key," which the header
disclaimed 127 lines later in its own identity limit. The key is the
PAIR — date and id name the file — and the surviving limit already says
so correctly. The same claim appeared a second time as an in-code
comment at the filename site ("the date only leads as a ... prefix; the
id carries uniqueness"); deleting one statement of a false claim and
leaving its twin is a half-correction, so both went. Flagged as a
deviation: R4 as written names the header only.

The bullet R4 emptied also still described the pre-R1 duplicate-id
behaviour, so it is rewritten to the behaviour that now ships.

Doc-only: 107 pass / 0 fail, unchanged.

* fix(envelope importer): correct what the adversarial gate falsified in R1-R4

Six independent refuters plus a completeness critic ran against 81deb83d.
Two of my own corrections were themselves false, one broke a repo-wide CI
gate, and one real defect nobody had measured turned up. Everything here
was re-derived on this box before it was written.

BLOCKER — test/fixtures/memvelope/merged-re-export.mve.json shipped with
no trailing newline, taking scripts/check-trailing-newline.sh from green
to red. It is wired into `bun run check:all`, `bun run check:newlines` and
three call sites in scripts/ci-local.sh, so the branch could not pass the
project's own local CI. Now `trailing-newline check: ok (1547 files)`.

BLOCKER — the R2 paragraph claimed `ALLOWED_TYPES` "requires"
`type: conversation`, "so a page typed anything else is not merely un-run,
it is ineligible." False. ALLOWED_TYPES (extract-conversation-facts.ts:142)
admits six types — conversation, meeting, slack, email, imessage,
imessage-daily — and :569 defaults to the whole list. A new false claim
about a named upstream constant, in a branch whose purpose is deleting
false claims. Rewritten to say what the type actually buys: admission, not
a trigger, and one admission among six.

Also in R2: "extracts no facts from any of them until one of those two is
invoked" is unsafe as an absolute. facts/eligibility.ts ORs the type test
with RESCUE_SLUG_PREFIXES = ['meetings/','personal/','daily/'], so a page
written into an outDir syncing under one of those IS picked up by a plain
sync — subject to the 80-char MIN_BODY_CHARS floor, which nobody in the
gate noticed either. Both named; the default outDir is unaffected and now
says so.

MAJOR — R4's replacement sentence, "Two conversations carrying the SAME id
land on one filename", is false and reinstated the exact contradiction R4
removed. Verified: one id at two created_at dates writes TWO files and
zero collisions. The filename is the PAIR.

MAJOR — the same bullet stated the fallback condition backwards ("when
neither carries an orderable one"). The code (`a === null || b === null ||
a === b`) falls back when EITHER side is unorderable, or when the two are
equal, and the function's own doc said so correctly 460 lines down.

MAJOR — the R3 census was wrong by four in both figures: 22/16 should be
26/20. chatgpt-split.json is an array of PARTS, each an array of
conversations, and my counter walked the parts as if they were
conversations and scored zero. Recounted with the level flattened: 13
conversations, 26 conversation-level timestamps, 20 bare unix-epoch, 4 Z,
2 no-designator, 0 numeric offset. The conclusion is unharmed and slightly
stronger. Also narrowed "not fixable, here or upstream" to "not fixable
HERE" — a converter CAN hold a source offset and this format drops it by
design, which is a decision, not an impossibility.

MINOR (found independently by four of six refuters) — the message-delta
warning still said "the overwritten page(s)", the exact word this branch
deleted from the summary line one commit earlier for being false whenever
the tiebreak keeps the earlier copy. Now "discarded" in both places.

R1's own limits, all newly documented rather than newly introduced:
  - when only ONE copy carries an orderable `updated_at` the rule changes
    nothing and the stale copy still wins. `updated_at: null` is
    producer-reachable (both normalizers end in `|| null`) though it is 0
    of 13 in the reference corpus. Refusing to promote a copy for merely
    HAVING a timestamp is the deliberate choice; the header now states
    what it costs instead of implying coverage the rule does not have.
    The argument is also repaired: both copies come from ONE converter
    run, so the asymmetry is the vendor's and its direction is
    unrecoverable — not the weaker "an older converter" reasoning.
  - the reduction is PAIRWISE. With every copy orderable it is a true
    maximum in all six permutations; with an unorderable copy in the
    middle transitivity is lost and [later, absent, earlier] keeps
    `earlier`. Now documented and pinned by a 7-case test.
  - the tiebreak is WITHIN one envelope. Across runs, check 2 still lets
    an older export refresh a newer page at exit 0 with no warning at all.

Test-file note, stated plainly: exactly ONE pre-existing test was touched
— 'the message-delta warning ties itself to the ... pages' — whose
assertion moved from 'overwritten page(s) carried' to 'discarded page(s)
carried' to track the deliberate message change. Its claim is unchanged
and nothing was loosened. Run against the untouched cd88d1de script in a
detached worktree, the shipped test file is 100 pass / 14 fail: that one
renamed pre-existing test plus 13 of the 20 new ones.

Receipts: 114 pass / 0 fail. Adjacent suites (conversation-parser x6,
markdown x3, extract-conversation-facts, doctor backlog) 301 pass / 0
fail. Over all 12 golden fixtures from the reference converter plus the
repo's sample, pages, stdout, stderr and exit code remain byte-identical
to cd88d1de.

NOT fixed here, escalated instead: a message body containing any of
gbrain's four timeline sentinels truncates the conversation at
markdown.ts before parse.ts runs. Reproduced against a pristine
`git archive upstream/master`: five sentinel forms each take a 4-turn
conversation to 2 parsed turns with the tail reclassified into the page's
timeline, at exit 0 with stdout reporting "4 message(s)"; two controls
stay clean. The re-grade graded this class NOT-A-DEFECT on a different
measurement and the packet puts its drop table out of scope, so this is
the RE's call, not a change smuggled in here.

* test(envelope importer): make the offset guard discriminate, cover the leap-second ceiling

The test named "an offset-bearing updated_at is compared as an instant, not
lexically" asserted only `kept === 'second'`. Array order keeps the second copy
too, so the assertion could not tell "compared as an instant" from "not compared
at all". Measured on the parent commit: dropping the offset alternation from
UPDATED_AT_SHAPE — so an offset-bearing value stops parsing and the pair falls
back to array order — leaves the file at 114 pass / 0 fail. The guard was
decorative against exactly the failure it is named for.

It now runs both offset signs and asserts the VERDICT STRING, which names the
branch that decided. The negative-offset case additionally makes the
offset-bearing copy win from array position 0, an outcome the fallback cannot
produce. Both cases fail under that mutant, and under the sign-arithmetic
mutant that the old test did already catch.

The `s > 60` leap-second ceiling had no test at all. Three cases now pin it.
Two of them pin the ROLL ACROSS THE MINUTE BOUNDARY rather than inside it:
23:59:60Z IS 2027-01-01T00:00:00.000Z, so a copy carrying each of those two
strings is a TIE and falls to array order, and 23:59:60Z outranks
23:59:59.999Z by one millisecond. Asserting only that 23:59:60Z beats
23:59:59Z would have pinned nothing but f(60) > f(59) — measured: making the
seconds field contribute milliseconds instead of seconds (`s * 1000` -> `s`)
destroys the date roll and still passes such a test at 117/0. The third case
is 23:59:61Z, the only value in the file rejected solely by the ceiling.

NOT CLOSED, so nobody reads this as a mutation-clean function: two sibling
guards in `updatedAtInstant` remain decorative on this commit. Deleting either
`if (h > 23 || mi > 59) return null` or the offset-range `if (oh > 23 || om >
59) return null` leaves the file green. They are the same class; they are not
this commit's scope.

Header prose only, no behavior change:
  - The Output-layout paragraph said the filename is the (date, id) PAIR. Both
    halves are slugged, so two DISTINCT ids reach one filename and the tiebreak
    then discards one of two UNRELATED conversations while stderr calls the id
    "not unique". Stated on the SLUG, not the raw id, because raw ids predict
    nothing either way: `AbC-123`/`abc-123` differ at character 1 and collide,
    while sixty `-` plus `a` and sixty `-` plus `b` agree on all of their first
    60 characters and do not collide — `slug` strips leading/trailing `-`
    before it truncates. Check 2 below already named this class; it REFUSES
    what the tiebreak resolves.
  - The header explains at length why check 2 refuses to resolve an id-less
    `conv-1` across runs, then the code resolves that same name WITHIN one
    envelope. Both sides are now stated together, with the asymmetry named:
    the within-run path is loud and has `updated_at` on both copies, which is
    exactly the evidence check 2 lacks. Also widened — any id that SLUGS to
    `conv-N` collides with the positional name, and any id that slugs to EMPTY
    enters that namespace while still recording a non-null id in frontmatter.
  - "two different values" -> "two different INSTANTS": two different strings
    can name one instant (`09:00:00Z` and `14:30:00+05:30`) and fall back.
  - The collision block's own comment still glossed "mapping to the same
    filename" as "carrying duplicate ids" — the equation the header now
    corrects. Fixed in place.

Suite: 114 pass / 0 fail before, 118 pass / 0 fail after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(envelope importer): remove a duplicate helper tsc rejects

Two functions were both named writeEnvelope. Bun hoists the later declaration,
so the earlier raw-envelope form was already dead at runtime - every call was
executing the fields form, which rebuilds the envelope from .conversations and
.meta and therefore handles both call shapes. All 118 tests passing under bun
is the proof; tsc's TS2393 was the only honest complaint.

Deleting the dead function keeps runtime behavior byte-identical. Removing it
also unmasked the two type mismatches TS2393 had hidden: envelopeWith declared
a return type of unknown for a value with a concrete shape, and writeEnvelope's
parameter did not admit the memvelope key that full-envelope call sites pass.
Both signatures now state what the values actually are.

bun run verify: 2/34 failing, identical to upstream/master's own baseline
(check:skill-brain-first and check:conversation-parser fail on the untouched
base as well).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(envelope importer): recognize our own pages after gbrain re-serializes them

The pre-file sweep proved existingPageIdentity() accepted only this importer's
own JSON.stringify output. gbrain rewrites pages it holds - export --dir, the
DB-only restore path, and put_page write-through all re-emit frontmatter
through gray-matter, which writes a UUID as a plain unquoted scalar and an
all-digit or boolean-looking id single-quoted. 16 of 17 id shapes, and 13 of
13 golden-fixture ids, came back in a form JSON.parse rejects - so the page
turned foreign, a later refresh refused the whole envelope at exit 2, and the
error's remedy advised deleting gbrain's own copy, provenance and body edits
included.

idScalar() now reads the three shapes a YAML round trip produces: our own
JSON, YAML single-quoted (doubled-quote escape), and plain scalars. YAML
null shapes and the empty string stay foreign, exactly as before - refusal
remains the answer for anything not confidently recognized.

Red-first: both new tests (plain-scalar UUID, single-quoted digits) fail on
the parent and pass at this commit. 120 pass / 0 fail; typecheck clean;
bun run verify at the repo baseline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Sean Gearin <sean@virgilknows.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 19:20:15 +08:00
Garry TanandClaude Fable 5 d610a845a8 fix(ci): gate — CommonMark info-string rule, hash the real model payload, close the YAML scanner gap (blind review round 5)
Three defects a second blind reviewer found that survived round 5. All three
reproduced against cf7a1616 before the fix and are pinned after it.

1. The fence scanner still falsely closed compliant descriptions. CommonMark 4.5
   forbids a backtick in a BACKTICK fence's info string, so ```foo`bar is an
   ordinary paragraph. The scanner opened a fence on it, never found a closer,
   and stripped the body to EOF:

     body = '```foo`bar' + three sentences of real prose + an image embed
     before: intentWordCount 0, misses [missing_intent, missing_screenshot]
     after:  intentWordCount 52, misses []

   Same class as round 5's closing-length bug and the same cost — opening a
   block CommonMark would not open deletes the author's prose exactly the way
   closing one late did. Tilde fences keep the permissive rule (4.5 restricts
   backtick fences only).

2. The spend guard hashed less than the model consumes. hashInputs covered
   title, modelBody, head.sha, exemption and policy ids — but the payload also
   carries the changed-file list and the diff, and the workflow degrades the
   diff to a marker line when the API 406s on a huge one. So a run that
   classified with no diff cached a diff-blind verdict, and the next run — real
   diff in hand, same title/body/sha — matched the hash and was served that
   verdict permanently (measured: anthropicCalls 1 across both runs). The
   assembled payload is now folded in as a fixed-width digest, inside the JSON
   tuple where quoting still makes a forged boundary impossible. Round 5's
   property is re-pinned: a policy fix past the model's body cap invalidates
   even when the payload is byte-identical.

3. The workflow's ${{ }}-in-run scanner missed a legal commented block header.
   `run: | # shell block` is valid YAML — js-yaml puts the following
   interpolation in the script — but the header fell through to the
   single-line branch, which captured `| # shell block` as the whole command
   and never looked at the block body. The rule protecting against shell
   injection reported clean over an interpolating workflow. The scanner now
   handles a trailing comment on the header, and scans the comment text too
   rather than leaving itself a hiding place. Guarded against js-yaml's actual
   parse, not against the scanner's own opinion.

Also corrected one overstated claim in the file's security block: a BARE url in
a sanitized string still autolinks under GFM. That is a self-labelled link and
the deliberate stopping point — the escaping targets the masking characters so
it cannot forge `![APPROVED](…)` or `[click to approve](…)` — but "no live
links" was too strong for what the code does.

Self-audit: all 13 interpolations into the sticky comment are either literals in
this file or pass sanitizeModelText/sanitizeList; the only unsanitized one
(titleCheck.reason) is a hardcoded literal and renders inside a code span. Every
rendered statement matches behavior the code performs. The four realistic-human
fixtures and the three new fence cases all PASS; empty, "fixes bug", the
unfilled template, ten words of lorem and a fenced-away screenshot all still
FAIL.

test/pr-gate-workflow.test.ts 158 -> 163 pass, 0 fail. Six mutations (the
CommonMark rule, the tilde exemption, the payload digest, the runGate wiring,
the scanner's comment group, the header-comment scan) each fail at least one
test. typecheck, actionlint, verify (34/34) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 f87488ff36 fix(ci): gate — stop red-Xing genuine contributors; neutralize markdown embeds (blind review round 4)
A blind reviewer rejected this branch on false positives and the repro held up.
The gate's intent check deleted every list and blockquote LINE before counting
words, then measured what was left against an arbitrary 40. Real people write
their story as bullets. Measured on the four realistic-human descriptions from
the repro:

  description                              before  after  floor 40  floor 20
  own prose written as four bullets             0     55  CLOSE      PASS
  short non-native-English paragraph           38     38  CLOSE      PASS
  specific first-person bug report             34     34  CLOSE      PASS
  mostly a stack trace + a real explanation    28     27  CLOSE      PASS

The file's own exemption rationale already says a check that is red on every
release is a check somebody disables inside a week. Red on every terse-but-
genuine contribution dies the same way, and it costs real contributors on the
way there — worse than the forgery risks rounds 1-3 chased.

- intentWordCount strips the list/quote MARKER and keeps the author's words.
  Still stripped: fenced code, indented code (new — the other spelling of a
  fence, guarded so it never eats a list continuation), headings, HTML
  comments, raw HTML, bare URLs, inline code, link/image syntax, and the
  template's own bold prompts. An untouched pull_request_template.md still
  scores 0, pinned against the real file on disk.
- INTENT_MIN_WORDS 40 -> 20, documented as a floor against empty/boilerplate-
  only descriptions, not a quality bar. CONTRIBUTING.md documents no word count
  at all. Empty, "fixes bug", the unfilled template, 10 words of lorem and an
  indented log paste all still fail.
- The fix-it comment no longer tells contributors to reopen a PR that was never
  closed — the gate has no close call in it. It now says what actually happens:
  editing the description re-runs the check (the workflow triggers on `edited`),
  the PR stays open, a maintainer decides.
- sanitizeModelText backslash-escapes `[` and `]`. Escaping HTML killed the
  <details> forgery, but Markdown needs no angle brackets: an image embed
  renders a green "APPROVED" picture and a link renders a live phishing target
  inside a close-lane comment. The file asserted nothing attacker-controlled
  reaches Markdown unescaped; that is now true.
- A label-API failure during a NEUTRAL run no longer throws to exit 2 with no
  comment. NEUTRAL has no verdict to record, so it logs, still posts, exits 0 —
  and the comment stops claiming labels were cleared when they were not. The
  verdict path keeps failing loudly on purpose (ordering note).

Four verbatim human descriptions are pinned as PASSING forever. Eight mutations
(floor, list markers, quote markers, indented code, markdown escaping, the
reopen copy, the NEUTRAL try/catch, the label claim) each fail at least one test.

test/pr-gate-workflow.test.ts 145 -> 158 pass, 0 fail. typecheck, actionlint,
verify (34/34) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry Tan ddb39df23a fix(ci): gate — escape comment HTML, tighten screenshot floor, fix fence false-positive, repair label ordering (blind review round 3)
Four findings from a second independent blind review, all reproduced against
HEAD before the fix and all pinned by a mutation-tested case.

- sanitizeModelText escapes &, < and > after the existing comment/mention/
  block-marker stripping, so raw HTML from any dynamic string (model output,
  mechanical flag details built from PR filenames, neutralReason, the
  policy-exempt note) renders as literal text. A <details open><summary>MERGE
  LANE — approved</summary> string is a forged verdict inside a close-lane
  comment; it now renders as &lt;details.
- hasScreenshot requires a markdown image whose URL looks like a URL or path,
  an <img> carrying a non-empty src=, or a bare paste URL, and strips HTML
  comments before scanning. `![proof](x)`, `<img alt=proof>` and an image
  hidden inside `<!-- -->` no longer clear it. Documented in the code as a
  FLOOR against zero-effort submissions, not proof.
- stripCodeFences is a line scanner following the CommonMark rule that a
  closing fence must be the same character and at least as long as the
  opening one. The old backreference read ```` as a NEW opening fence and
  stripped to EOF, so a compliant body documenting fence syntax lost its
  intent paragraph and was closed. The scanner is also linear, retiring the
  superlinear-backtracking hazard the 16KB cap was sized against.
- Labels are reconciled BEFORE the sticky comment carrying the cached state.
  Written the other way, one transient label-API 500 left stale/missing/
  duplicate labels forever: the rerun short-circuited on the persisted state
  and returned success without repairing them.

Also:
- hashInputs covers what the run consumes — the truncated model body plus the
  mechanical policy outcome — so an edit past the 6KB model cap no longer
  mints a new hash and buys an identical paid call, while a policy fix landing
  past that cap still invalidates the cached verdict.
- The test's YAML block-scalar scanner accepts indentation indicators in both
  legal orders (`>2-` as well as `|-2`), with a guard-the-guard case proving
  it sees a `run: >2-` block interpolating attacker-controlled text.
- A block at the top of the script states plainly that this gate is a triage
  signal and a reviewer checklist, not an authorization boundary; one line of
  the sticky comment footer says the same to the contributor.

145 pass / 0 fail (was 126), typecheck clean, actionlint clean, verify 34/34.
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 88731d8cf3 fix(ci): gate — sanitize mechanical flag details, anchor path regexes, exempt maintainer/bot/draft from the policy check (blind review round 2)
Blind review round 2 rejected the branch. Five findings, all fixed.

BLOCKING 1 — the verdict was forgeable through a PR filename.
renderComment wrote mechanical red-flag details raw, and two of them
interpolate filenames (adds_recipe, deletes_tests). git allows a newline
inside a filename and JS `[^/]` matches one, so RECIPE_RE's anchors were
decorative: a PR adding `src/core/ai/recipes/x\n## PR Gate — ...\ncc
@octocat\n<!-- ...state... -->\nz.ts` put that text verbatim into the bot's
comment — forged heading, live third-party mention, and on the NEUTRAL
render (which writes no state block of its own) parseState() returned the
ATTACKER's block, so the next run hit the spend guard and silently skipped
the verdict with no label and exit 0. Three layers:
  (a) flag details go through sanitizeList, exactly like the model's
      strings. Audited every other interpolation into the comment; the rest
      are literals in the file, Number()-coerced, or already sanitized.
  (b) every path regex spells its segment class [^/\n], not [^/] —
      RECIPE_RE, SOURCE_EXT_RE and the test-path check.
  (c) parseState reads the state block only off line 2 of a marker-leading
      comment (where renderComment writes it) and STATE_RE is whole-line
      anchored. A block anywhere else is somebody else's text.

BLOCKING 2 — no exemption, so every release PR was close-lane.
Measured: 40 of the last 40 merged PRs would be close-lane on
missing_screenshot, including every /ship release PR. A check that is red
on every release gets switched off within a week, and then it filters
nothing. The #3745 policy exists to filter INCOMING OUTSIDE CONTRIBUTIONS;
release automation cannot take a screenshot of itself. It is now waived for
OWNER/MEMBER/COLLABORATOR, bot authors and drafts — the usefulness verdict,
the title rule and every mechanical red flag still run, and the sticky
comment says the check was skipped. author_association / draft / user.type
are read from the pr.json the workflow already fetches: no new API call, one
source of truth. `draft` is the one author-settable input, so
ready_for_review joins the trigger list and the exemption is folded into the
spend-guard hash — the draft-era verdict cannot be reused after the flip.
Stated as a deliberate decision in both the workflow header and the script.

3 — DOWNGRADE_FLAG_IDS omitted deletes_tests, adds_symlink and
adds_node_modules, so a PR deleting test/e2e/engine-parity.test.ts kept
merge-lane and a green check on the strength of its prose. All three added.
The old test used deletes_tests as its example of a NON-downgrading flag;
rewritten to pin the stronger invariant instead — every id detectRedFlags
can emit is a downgrade trigger (derived from the detector, so a new flag
fails until it is classified on purpose), and the set is still an allowlist
(an unrecognized id changes nothing).

4 — FENCE_RE backtracks superlinearly on a hostile body: 65KB of backticks
(GitHub's max body length) measured 8.2s across the two policy scans on a
pull_request_target runner. stripCodeFences now caps the scan at 16KB —
same input, 0.40s. Tradeoff documented at the constant: the intent
paragraph and the screenshot both sit near the top in practice (the PR
template puts them in the first two sections, and the model payload already
caps the same body at 6KB), so a real contributor is not judged on a
truncated tail.

Verified: test/pr-gate-workflow.test.ts 126 pass / 0 fail (was 95),
typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout / check-key-files-current-state all exit 0. Each new
pin was mutation-tested against the pre-fix behavior: all six mutants fail
the suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 4a00c31b12 fix(ci): gate policy check survives an API outage; merge master; drop NUL separators
Three fixes on the strict PR usefulness gate (#3698), plus the merge that
brings in the policy it enforces.

1. The mechanical policy check now outlives the model. The
   ANTHROPIC_API_KEY guard used to sit above detectPolicyMisses, so a PR
   with no intent paragraph and no screenshot got a green NEUTRAL skip
   whenever the key was absent or Anthropic was down — "wait for a 500"
   was a documented way past the one hard requirement. The #3745 branch
   now sits above the key guard and the spend guard: a policy miss is
   close-lane + the friendly fix-it comment + exit 1 with no API
   dependency at all. A compliant PR that hits a missing key or a dead
   API keeps the round-1 NEUTRAL behavior unchanged (loud comment,
   ::warning::, exit 0, stale gate:* labels cleared) — and the NEUTRAL
   comment now says plainly that the *usefulness verdict* did not run,
   while still reporting the title check and mechanical red flags it was
   able to compute without a model.

2. Merged origin/master, which carries #3745's CONTRIBUTING.md section
   and .github/pull_request_template.md. No conflicts: this branch never
   touched VERSION / package.json / CHANGELOG.md, so master's 0.42.72.1
   carried through untouched — the feature branch adds no version bump.
   The test's inlined pull_request_template fallback (only needed while
   the branch predated the merge) is gone; it now reads the real file, so
   growing the template's own prose past the 40-word bar fails here
   instead of silently letting an untouched template through. The
   CONTRIBUTING_URL deep link is pinned against a GitHub-style slug of
   every heading in the merged CONTRIBUTING.md, with the slugger itself
   pinned so it cannot "pass" against an anchor GitHub never generates.

3. hashInputs joined its three fields with literal NUL bytes, which made
   grep treat the whole of scripts/pr-gate.mjs as binary — any future
   grep-based CI guard over that file would have matched nothing and
   passed silently. Replaced with JSON.stringify of the tuple: still
   unforgeable (each field is quoted and escaped), still stable by
   construction, and printable. `grep -c hashInputs scripts/pr-gate.mjs`
   now returns 2 instead of nothing. Existing sticky-comment state hashes
   are invalidated once, costing one re-verdict per open PR.

Tests: 95 pass / 0 fail in test/pr-gate-workflow.test.ts. The no-API-key
policy-miss case was verified to fail against the pre-fix ordering.
Verified live against the Anthropic API: HTTP 200, strict JSON, all seven
required keys, merge-lane on a compliant fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 94662eb1e1 feat(ci): gate enforces the #3745 intent-paragraph + screenshot requirement
CONTRIBUTING.md (#3745) requires every PR to carry a paragraph the author
wrote themselves and a screenshot of gbrain actually in use. The gate now
checks both, mechanically, before it spends anything on review.

What it checks (no LLM, both exported for testing):
- hasScreenshot: markdown `![alt](url)`, a bare user-images.githubusercontent
  or github.com/user-attachments/assets URL, or an <img> tag. Anything inside
  a fenced code block does not count — pasting the syntax is not attaching
  the picture.
- hasIntentParagraph: >= 40 words of prose left after stripping fenced code,
  blockquotes, list items, headings, HTML comments, links and the PR
  template's own boilerplate. Per-character scripts are tokenized per
  character, so a paragraph written in Chinese counts as one.

The two lane consequences:
- missing_screenshot OR missing_intent forces close-lane from any recommended
  lane (exit 1) and skips the model call entirely — closed without review is
  the documented consequence, so there is nothing to spend a review on. The
  sticky comment leads with what is missing, how to fix it, and the reopen
  path; both misses are also recorded in the existing "Mechanical downgrades
  applied" section.
- The model's new advisory intent_authenticity verdict forces
  needs-maintainer (exit 0) when it reads "ai_generated", and never
  close-lane on that signal alone. The comment says only that a maintainer
  will read the paragraph personally; the model's reasoning is consumed and
  never published at the contributor.

Rubric + strict-JSON schema gain intent_authenticity and
intent_authenticity_reason, with explicit instructions that rough grammar,
terseness and non-native English are evidence of a HUMAN and that "unclear"
is the answer whenever the evidence is not clear-cut.

test/pr-gate-workflow.test.ts: 63 -> 88 tests, all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 2f65ed8da6 fix(ci): harden the PR gate — comment-ownership, output sanitization, deterministic lane downgrades (blind review round 1)
Two independent blind reviews of #3698 (one APPROVE, one REJECT). Every
confirmed finding from the reject side, plus the cheap hardening both
reviews flagged.

BLOCKING

1. Sticky-comment hijack. upsertStickyComment adopted ANY comment containing
   the marker, so a contributor could pre-post `<!-- gbrain-pr-gate -->`, have
   the gate PATCH it, then edit it into a fake green verdict. isOwnComment()
   now requires user.type === 'Bot' AND login === 'github-actions[bot]' AND
   the body to START WITH the marker; anything else gets a fresh comment.

2. LLM output injected as raw Markdown. Model-produced reasons[] and
   reviewer_checklist[] reached the comment unescaped — PR-body-driven
   injection could forge headings, a second marker, and live @mentions.
   sanitizeModelText()/sanitizeList() are now the single choke point in
   renderComment(): HTML comments stripped, mentions zero-width-broken,
   leading block markers removed, newlines collapsed, 300 chars per string,
   8 entries per list, both caps self-marking.

3. Lane was purely model-decided. A well-written feature pitch could talk
   itself into merge-lane. The model now RECOMMENDS; applyMechanicalDowngrades
   forces merge-lane -> needs-maintainer on any of: workflow edits, a new
   package.json dependency, a new src/core/ai/recipes/ provider file, new
   KNOWN_CONFIG_KEYS entries, >40 changed files, >400 net source lines outside
   test/, or a src/ change with no test file touched (#3665). The sticky
   comment reports them under "Mechanical downgrades applied".

4. The test file did not pin what it claimed. Added: no gh pr checkout /
   git fetch / refs/pull / pull/*/head in any spelling; exact permissions
   key->value map plus a single-permissions-block assertion so no job-level
   grant re-widens contents; the ${{ }}-in-run scanner now covers folded
   (`run: >`) and chomped blocks, with a guard-the-guard test; and mocked
   end-to-end runGate() runs for close-lane exit 1, marker hijack, sanitizer,
   truncation, refusal routing, NEUTRAL label clearing, label swap, and the
   spend guard.

5. Version-first title regex rejected the documented suffix form.
   `v0.31.1.1-fixwave fix: ...` now passes. VERSION_AT_END_RE no longer
   false-positives on `chore: bump zod (3.25.76)`: it fires only on a
   v-prefixed or 4-segment trailing version, i.e. this project's own shape.

6. Refusal fail-open. stop_reason=refusal exhausted retries into a green
   NEUTRAL — a deterministic way to dodge the red X. Refusal and
   schema-invalid output now route to needs-maintainer with an explicit
   note; only transport failure stays NEUTRAL. Refusal also short-circuits
   the retry loop, since retrying a deterministic refusal only burns spend.

ALSO

7. persist-credentials: false on the checkout step.
8. Dropped pull-requests:write. Everything the script calls is the issues
   API (comments, label create, label add/remove), so issues:write is the
   only grant that is actually needed.
9. Spend guard for the edited/synchronize amplification: the LLM call is
   skipped when sha256(title+body+head_sha) matches the hash recorded in the
   previous sticky comment's state block, and the stored lane's exit code is
   reused.
10. NEUTRAL runs now clear every gate:* label instead of leaving a stale
    verdict behind.

Verified: bun test test/pr-gate-workflow.test.ts 63 pass / 0 fail,
bun run typecheck clean, actionlint clean, check-privacy /
check-no-tracked-symlinks / check-progress-to-stdout /
check-bun-test-timeout clean, plus a live Anthropic smoke of the exact
request shape (HTTP 200, valid strict JSON, injection attempt rejected).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Garry TanandClaude Fable 5 ca1eed3e95 feat(ci): strict PR usefulness gate — merge-lane/close-lane verdict before any review effort (#3698)
Runs on pull_request_target against master. PR code is never checked out
or executed: metadata + a 120KB-capped diff come from the GitHub API only;
the base-repo checkout provides the rubric script. All interpolations are
env-bound; actions SHA-pinned; permissions limited to contents:read +
pull-requests:write + issues:write; per-PR concurrency with cancel.

scripts/pr-gate.mjs (no deps, fetch only) classifies via the strict
usefulness rubric (claude-sonnet-5, strict JSON schema, thinking disabled;
sampling params omitted — rejected on this model), posts one sticky
comment (<!-- gbrain-pr-gate -->), applies exactly one gate:* label, and
exits 1 only on close-lane. Missing ANTHROPIC_API_KEY or API failure after
2 retries NEUTRAL-skips loudly (comment + warning, exit 0). Mechanical
(no-LLM) checks: version-first title rule and diff red flags (>40 files,
node_modules, symlinks, workflow edits force needs-maintainer, new
package.json deps, deleted tests).

test/pr-gate-workflow.test.ts pins the security invariants and unit-tests
the exported title rule + red-flag detector (import side-effect guarded).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-04 10:46:09 +08:00
Sean Gearin 82fe0216ff fix(import): keep stdout JSON-only under --json (#3637) (#3764)
runImport printed its informational lines ('Found N markdown files' and four
siblings) through console.log unconditionally, so `gbrain import <dir> --json`
emitted them ahead of the payload and stdout did not parse as JSON. A consumer
parsing stdout reads that as zero imports while its own bookkeeping records the
files as ingested, so the next run skips them permanently.

Route those five lines through an `info` helper that switches to stderr when
--json is set. The lines are relocated, not removed: human mode is byte-for-byte
unchanged, and the JSON payload itself is untouched. This is the contract
CLAUDE.md already states ('Stdout stays clean for data output (--json
payloads)') and that import.ts's own progress comment repeats.

scripts/check-progress-to-stdout.sh only greps for process.stdout.write('\\r…),
so this class was never guarded; the new spawn-level test pins it.
2026-08-03 19:55:04 +08:00
Masa fa313f2969 fix(durability): gate installHelper's exec-bit chmod on dryRun (#3736) (#3759)
installHelper's "already current" fast path ran chmodSync(helperPath,
0o755) unconditionally, before the dryRun check further down — so
`gbrain sources harden --dry-run` mutated the helper's permissions even
though dry-run is documented as a pure preview. This is the 5th
instance of the #3594 class (see #3692 for the sibling fix to the
step-1 pull, same function).

Gate the chmod with `if (!dryRun)`, mirroring the guard pattern already
used in installLocalHook's own "already current" branch. Non-dry-run
behavior is unchanged.

Adds a regression pair to test/brain-repo-durability.serial.test.ts:
one asserting dry-run leaves a drifted-permission helper untouched,
and a control asserting the real run still restores the exec bit.
2026-08-03 19:37:18 +08:00
2e554500e2 fix(cycle): grade_takes and calibration_profile record the model they actually run — follow the gateway chat model (#3726)
Two cycle phases still carried the label-vs-actual defect that
propose_takes shed in v0.42.62 (and that the #2805 review noted was worth
tracking so it isn't lost):

- grade_takes hardcoded 'claude-sonnet-4-6' into judge_model_id, the
  evidence signature, and budget metering, while the default judge call
  passed NO model hint and rode the gateway's chat_model. On any brain
  with a non-default chat_model, the verdict cache and telemetry recorded
  a model that never ran.
- calibration_profile persisted TIER_DEFAULTS.reasoning to model_id but
  never passed a model to the patterns generator at all, so the recorded
  model and the executed one were unrelated.

Fix, matching the propose_takes convention: each phase resolves ONE
string via explicit override > getChatModel(), and that string drives the
chat call, the cache key, and the stored id.

- grade_takes: the judge hint gets the FULL provider-prefixed string; the
  stored judge_model_id and evidence signature keep the historical bare
  tail. Stock installs are unchanged: getChatModel() defaults to
  'anthropic:claude-sonnet-4-6', whose tail equals the old hardcoded
  value, so no verdict cache invalidates. A genuinely different
  chat_model invalidates, which is correct — the judge really changed.
- calibration_profile: getChatModel() is provider-prefixed, preserving
  the #2451 contract, and its default IS the old TIER_DEFAULTS.reasoning
  value — stock behavior unchanged. The generator now receives the same
  resolved string that is persisted to model_id.

Tests: regression per phase pinning configured-chat-model routing (full
string to the call, bare tail to the cache key for grade_takes, full
string persisted for calibration); all existing suites pass unchanged,
pinning the no-change-on-stock property. 112 tests across the four
affected suites; typecheck clean.

Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-02 20:26:51 +08:00
Masa 630786dc65 docs(integrity): correct the --dry-run line in the subcommand comment (#3739)
The comment listed `gbrain integrity --dry-run` as a subcommand, but
runIntegrity dispatches on `check`/`auto` and parses --dry-run inside
the auto path, so that form exits 1 with "Unknown subcommand". The
runtime help already shows --dry-run indented under `auto [options]`;
only the source comment disagreed.

Addresses #3738.
2026-08-02 20:10:31 +08:00
Masa d92f0dc86f fix(integrity): stop counting dead-link findings as review-queue entries (#3750) (#3751)
The dead-link branch in cmdAuto called logSkip() (writing to
integrity.log.jsonl) but incremented bucketReview, the same counter fed by
the bare-tweet path's appendReview() calls (which write to
integrity-review.md). The printed "Review queue" summary line therefore
included findings that were never written to the review file, so
`gbrain integrity review` silently disagreed with `integrity auto`'s own
summary.

Give dead-link findings their own counter (bucketDeadLink) and print it as
a separate line. No change to bare-tweet bucketing or CLI surface.
2026-08-02 19:54:49 +08:00
a68379050e feat(schema-pack): add slug_filter to retype mapping rules (#2655)
retype mapping_rules' existing path_filter matches pages.source_path,
which is only populated for pages synced from a git repo. Pages ingested
via the put_page MCP tool (or any write path that doesn't go through
sync) have source_path = NULL, so path_filter can never disambiguate
them — a same-from_type retype rule targeting a slug prefix has no way
to address this class of page at all.

slug_filter adds an independent, orthogonal LIKE filter on pages.slug
(the field that's always populated), combinable with path_filter via AND
when both are given. Wired through the full call path: schema validation
(manifest-v1.ts), the RetypeRule interface + probeRule/applyRetypeRule
(retype.ts), and the pack-manifest → RetypeRule conversion in the
unify-types job handler (unify-types-handler.ts) — the last one matters
because a field only present in the zod schema but not carried through
that conversion would validate fine yet silently no-op at execution time.

- manifest-v1.ts: add optional slug_filter to RetypeMappingRuleSchema
- retype.ts: slug_filter on RetypeRule; probeRule + applyRetypeRule both
  apply `AND slug LIKE $N` when present, independent of path_filter
- unify-types-handler.ts: carry rule.slug_filter through the
  pack-mapping-rule → RetypeRule conversion
- tests: skips-outside-slug_filter, matches-despite-NULL-source_path
  (the motivating case), path_filter+slug_filter combined via AND

320/320 existing schema-pack tests pass; 3 new tests added; tsc --noEmit
clean.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-02 08:12:46 +08:00
6d1232d5a6 v0.42.72.1 docs(contributing): require a human-written intent paragraph + live screenshot on every issue and PR (#3745)
Effective 2026-08-02. Every issue and PR must carry a paragraph the author
wrote themselves explaining why they are opening it, and a screenshot showing
gbrain actually in use in that situation. AI-generated or AI-polished intent
text is not accepted; AI assistance for the code is still fine. Missing either
one means closed without review, reopenable once added.

Stated in CONTRIBUTING.md and pre-filled in both issue templates plus a new
pull request template so the fields are in front of the author.

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 07:47:23 +08:00
Sina Matian aff34a428a v0.42.72.0 feat(auth): server-enforced slug-prefix write fence for OAuth clients + qm-harness integration guide (#3712)
Registering an OAuth client with --bound-slug-prefixes now makes the write boundary real: writes outside the bound prefixes are refused on every op that can name a page, and ops that write by something other than a slug are refused outright rather than left unfenced. Deny-by-default at dispatch, so a write op added later is refused to bound clients until it is explicitly fenced.

Adds docs/integrations/qm-harness.md (gbrain as the company brain for a qm deployment) with a roster-driven provisioning script and deployment templates, plus a Known limitations section stating plainly that this is a write boundary and not a privacy boundary.

Five review rounds, including three clean-room passes by codex gpt-5.6-sol and Claude Fable 5 against an instruction-stripped tree.
2026-08-02 07:05:13 +08:00
Garry Tan f07e9dff11 fix(ci): release build gates on the artifact, not a serial re-run of the whole suite
The first Release run (30698650484) failed before building anything: the
build job re-ran the entire unit suite serially on the release runner —
a different environment from the sharded Test workflow that had already
gated the exact SHA at merge — and died on ambient-env tests. The review
of #3573 predicted this ('a flaky test now blocks the release'). The
build job now compiles and smoke-tests the binary (--version must match
the VERSION file), which validates the actual artifact — something the
test suite never did. The Test workflow remains the code gate.
2026-08-01 20:43:58 +08:00
Garry Tan 72ec53bb49 v0.42.71.0 feat(ci): publish GitHub releases on every version bump — organized notes + binaries, self-update unbroken (#3521 #3716) 2026-08-01 19:55:28 +08:00
Garry Tan 0244104b8d fix(ci): env-bind the release-notes version interpolation
Review hardening from the #3573 gatekeeper pass: a ${{ }} inside run: is
shell injection by construction; bind through env instead. Not
attacker-reachable today (VERSION comes from master), correct anyway.
2026-08-01 19:55:28 +08:00
Garry TanandTime Attakc 48618bd4bf feat(ci): publish GitHub releases so binary self-update can work (#3521) (#3573)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 19:55:28 +08:00
Garry Tan dcaea7fdbb fix(wave): revert extractor watermark to merge-day midnight — a future watermark masks concurrent edits (D4)
CI caught what the composite review's watermark 'fix' actually did: the
stamp path clamps links_extracted_at up to the watermark
(GREATEST(updated_at, versionTs), extract.ts:1835), so a future watermark
stamps every page at 2026-08-02 and masks any concurrent edit until then —
the exact race D4 guards — and makes the doctor lag check warn on fresh
brains. The limitation the future date tried to cover (stamps written by
pre-wave code after the watermark) is inherent: no fixed watermark covers
code that keeps running past it. Documented instead.
2026-08-01 10:06:17 +08:00
Garry Tan c6bfab582e v0.42.70.0 fix: community fix wave two — 18 contributed fixes for dead flags, dry-run safety, link resolution, and Windows support 2026-08-01 10:06:17 +08:00
Garry Tan c5ac3efe9f fix(wave): composite-review findings — mask wikilink interiors from the bare-path scan; future-safe extractor watermark
Two cross-PR findings from the wave-i hostile review (Codex xhigh):
1. The #3560-ungated bare-path pass scanned inside [[...]] spans, so a
   dir-qualified wikilink's lowercase prefix (its parent page) became a
   spurious 'markdown' edge whenever the parent existed. Wikilink spans
   are now masked with equal-length blanks before pass 2; discriminating
   test added (fails without the mask).
2. LINK_EXTRACTOR_VERSION_TS was midnight today with a strict-< staleness
   predicate, so same-day stamps from pre-wave code read as fresh and
   never re-extracted. Bumped to 2026-08-02T00:00:00Z.
Plus two comment corrections from both reviewers: upgrade.ts's X1 hook
rationale (stale after #3085) and PGLiteEngine.transaction's tx-engine
db-proxy hazard for #3613's searchVector wrapper.
2026-08-01 10:06:17 +08:00
Garry Tan 0a6b697070 fixup(#3161): pin #1964 assertions to the wikilink-resolved lane
In-wave interaction with #3560: extractPageLinks now also emits raw-literal and bare-path candidates that downstream existence checks drop; the tests' whole-list toEqual predates that. Filter to linkSource='wikilink-resolved' — the PR's resolution + no-cross-dir-leak claims are still fully pinned.
2026-08-01 10:06:17 +08:00
Garry Tan 873587f831 fixup(#3453): regenerate skills.lock.json against current master skills/
The PR's committed manifest predates master's latest SKILL.md edits; regenerated with the PR's own generator per the check's instructions.
2026-08-01 10:06:17 +08:00
Garry Tan 78391e8b64 fix(extract): resolve cross-directory wikilinks via sync-consistent slugification (#3161)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc f08a51d9de fix(extract): stop silently dropping links from non-whitelisted directories (#2576) (#3560)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc 7cbb99ffef feat(doctor): silent-failure check batch — content-hash duplicates, undeclared DB-only pages, heartbeat staleness, db_only collector collision (#2250 #2784 #2787 #2788) (#3457)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc aa5b9e6e2d feat(doctor): tamper-evident skills/ manifest — generator, doctor drift check, CI freshness guard (#159) (#3453)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 8e3699156f fix(sources): honor unfederate for unqualified reads (#2928) (#3533)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry Tan cc1783c0e4 fix(cli): make --brain actually route to the named brain (#3576)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego e77a15f0d8 fix(windows): use native path boundaries and assertions (#3578)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandYMYD 522cfb032a fix(chunker): measured hard-split budgets + token-aware capByChars — the follow-up invited in #3477's merge review (#3564)
Co-Authored-By: YMYD <paul@ymyd.co.kr>
2026-08-01 10:06:17 +08:00
Garry Tananddialthewolff 273bd0e2be fix: scope residual federated reads (#3550)
Co-Authored-By: dialthewolff <jordan@jcwolff.com>
2026-08-01 10:06:17 +08:00
Garry Tan 7522b90db9 fix(search): implement documented relative since/until durations (#3442) (#3539)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandTime Attakc 42f810c406 fix(search): raise hnsw.ef_search to match the vector candidate request (#3613)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 1ca1a14f24 fix(apply-migrations): stop reporting 'All migrations up to date' while schema is behind (#3085)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego 64c191b1d7 fix(build): force LF for Markdown too (frontmatter parsers anchor on LF) (#3517)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego d49ea83db4 fix(minions): settle the spawn loop when a child never launches (#3536)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry Tan d71d50503d fix(serve,import): stop the SIGINT orphan; quote imported frontmatter (#3618)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandDiego a228776b8f docs(security): tighten responsible disclosure language (#3619)
Co-Authored-By: Diego <diegodearagao@gmail.com>
2026-08-01 10:06:17 +08:00
Garry TanandMammad M. c0b28f104d fix(sync): prevent dry-run pull and out-of-strategy page deletion (#3624)
Co-Authored-By: Mammad M. <mammad@digit.az>
2026-08-01 10:06:17 +08:00
Garry Tanandcvillarroel2 61ef727710 fix(contextual): route synopsis through configured models (#3678)
Co-Authored-By: cvillarroel2 <20239888+cvillarroel2@users.noreply.github.com>
2026-08-01 10:06:17 +08:00
Garry Tan 532e591655 fix(test): recipe-ollama-dims pins a truncated width — native-width requests now omit the param (#3699)
Master's #1072 test asserted bare qwen3-embedding@1024 threads dimensions:1024;
wave PR #3699 suppresses the param when the request equals the native width
(1024 for the bare id), so the same input now correctly returns undefined —
pinned by dims-qwen3-native.test.ts. Moved this test to 512 to keep its actual
intent (bare id recognized as Matryoshka-capable) without contradicting the
suppression. Third composition defect of the wave; caught by CI shard 8.
2026-08-01 07:40:48 +08:00
Garry Tan 3acd511b80 v0.42.69.0 fix: community fix wave — 22 contributed fixes for silent-failure paths, local-model support, and multi-source routing 2026-08-01 07:40:48 +08:00
Garry Tan 25e4c0c3b1 fix(skills,docs): unify-types playbooks must pass apply:true after the #3574 default flip
#3574 flipped the unify-types worker default to dry-run (jobs.ts:2221,
apply: data.apply ?? false) and updated the architecture docs, but three
agent-facing surfaces still presented the bare submit as the Apply step:
skills/schema-unify/SKILL.md 'Phase 3: Apply', skills/conventions/
schema-evolution.md, and README.md. Because #3545 also edited SKILL.md in
this wave, each PR looked self-consistent in isolation — only the composed
branch shipped a playbook whose apply step silently retypes nothing and
never flips the active pack. Skills distribute downstream via the skillpack,
so this would have propagated. Found by an independent cross-PR review pass.
2026-08-01 07:40:48 +08:00
Garry Tan ba27a186ec fix(budget): reconcile #3691 and #3541 — free local providers are priceable at $0
#3691's regression test used llama-server as an 'unknown provider' example.
#3541 (same wave) prices ollama/llama-server at $0 via FREE_LOCAL_CHAT_PROVIDERS,
so that example is now priceable and the assertion inverted. Swapped in groq —
the paid-but-unpriced case #3691's own description cites — and added the
positive assertion that free local providers keep their cap enforced.
2026-08-01 07:40:48 +08:00
Garry TanandChristoph 4bb313cd80 feat(by-mention): Unicode-aware word tokenizer — Vietnamese/diacritic name extraction (#3563)
Co-Authored-By: Christoph <astaran@herr-der-ringe-film.de>
2026-08-01 07:40:48 +08:00
Garry Tanand老赵 f39b059ad8 test(cli): cover import side-effect guard (#3581)
Co-Authored-By: 老赵 <273731059@qq.com>
2026-08-01 07:40:48 +08:00
Garry TanandKyzcreig 42c2d56df3 fix(import): checkpoint on a time interval and before preserve, not only every 100 files (#3585)
Co-Authored-By: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandVyacheslav Zakharov e3806cf46f fix(test): kill timeout-resistant unit shards (#3631)
Co-Authored-By: Vyacheslav Zakharov <vyacheslav.zakharov@avers.kz>
2026-08-01 07:40:48 +08:00
Garry TanandAlexey a82a83dbc3 fix: health metric that cannot converge, and facts lost to a row_num collision (#3634)
Co-Authored-By: Alexey (CTO) <cto@phrase.local>
2026-08-01 07:40:48 +08:00
Garry TanandBen Young 03de3246f3 fix(budget): price free local chat providers at $0, matching embed and rerank (#3541)
Co-Authored-By: Ben Young <Grimnoth@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandAustin Wilhite 37ad1d2104 fix(cycle): stop extract_atoms silently skipping all work on unpriced models (#3691)
Co-Authored-By: Austin Wilhite <austinw80@gmail.com>
2026-08-01 07:40:48 +08:00
Garry Tanandzenspam aeb75dc839 fix(dims): skip dimensions param when it equals qwen3-embedding native width (#3699)
Co-Authored-By: zenspam <zenspam@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 298b6b01b8 fix(docs): remove references to the nonexistent gbrain install command (#3502) (#3545)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry Tan d6f929bfbd fix(embed): stop one bad chunk darkening a page, and exit non-zero on failures (#3037) (#3532)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandRyan Ayers 9c1a4b8fce fix(cycle): surface stamp-write failure on CycleReport and degrade status (#3504) (#3589)
Co-Authored-By: Ryan Ayers <rayers@dividia.net>
2026-08-01 07:40:48 +08:00
Garry Tan 3523d8fd7e fix(jobs): unify-types worker defaults to dry-run per its handler contract (#1575) (#3574)
Co-Authored-By: Garry Tan <garrytan@gmail.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc a3000d4630 fix(skillpack): resolve gbrain root from module path when cwd walk fails (#3144)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandBrett 64ad743d98 fix(models): hint the /v1 base-URL suffix when a doctor chat probe 401s on an openai-compatible proxy (#3553)
Co-Authored-By: Brett <brettdavies@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 9ada48e600 fix(doctor): bound embedding provider health probe (#3364)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc aa05255887 fix(doctor): stop reporting Windows-drive image paths as missing under WSL (#1835) (#3523)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 241603aab8 fix(dream): honor a configured 0 in synthesize + auto_think config resolution (stop coercing to the default) (#3552)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 69be8bb707 fix(minions): reconnect the stall timer, stop failJob masking job errors, unify retry matchers (#1720) (#3555)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 1116a95926 fix(entities): resolveEntitySlug fallback keeps the path separator (#3447) (#3567)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc c8ea38421a fix(cycle): log swallowed lock.release() failures (#1470) (#3572)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc 163a83baa3 fix(operations): sync_brain threads ctx.sourceId to performSync (#2830) (#3568)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
Garry TanandTime Attakc f7295e3308 fix(cli): route init --help to its own usage text (#3652)
Co-Authored-By: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-08-01 07:40:48 +08:00
321 changed files with 27454 additions and 1544 deletions
+13
View File
@@ -14,3 +14,16 @@
# through bash. `eol=lf` pins the checkout regardless of the user's
# core.autocrlf setting.
*.sh text eol=lf
# Markdown gets the same pin, for a different failure mode: the frontmatter
# parsers anchor on LF. Under a CRLF checkout the opening fence becomes
# "---\r\n", which an LF-only /^---\n/ (or a startsWith("---\n")) does not
# match, so a well-formed document silently parses as having no frontmatter.
# There is no error -- the field just comes back empty. That has surfaced as
# blank skill descriptions, a fixer inserting its banner above the
# frontmatter instead of below it, resolver trigger extraction dropping
# entries, and a generated-doc freshness check reporting every line as
# drifted. The parsers stay CR-tolerant on their own merits (gbrain reads
# Markdown it does not own), but pinning this repo's own .md checkout to LF
# removes the whole class for anyone working here.
*.md text eol=lf
+88 -8
View File
@@ -1,14 +1,67 @@
name: Release
# Publishes a GitHub release for every VERSION bump that lands on master:
# tag + title `v<VERSION>`, notes from that version's CHANGELOG.md entry,
# compiled binaries attached (#3521).
#
# Why every bump: `gbrain check-update` resolves the latest version from the
# VERSION file on master, but binary self-update
# (src/core/binary-self-update.ts) downloads assets from `releases/latest`.
# If releases lag VERSION, binary installs are told an upgrade exists that
# self-update cannot apply. Keeping releases/latest == VERSION closes that gap.
#
# Idempotent: the `version` job skips build+release when a release for
# v<VERSION> already exists WITH all expected assets. A half-published release
# (tag exists / assets incomplete) is repaired on the next run — softprops
# updates the existing release in place. Historical 3-segment tags are never
# touched; a new 4-segment VERSION always mints a new tag.
#
# The asset names are a contract with expectedAssetName() in
# src/core/binary-self-update.ts, pinned by test/release-workflow.test.ts.
on:
push:
tags: ['v*']
branches: [master]
paths: [VERSION]
workflow_dispatch: {} # manual first run / backfill of the current VERSION
permissions:
contents: write
contents: read
concurrency:
group: release
cancel-in-progress: false
jobs:
version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- id: v
name: Read VERSION and check for an existing complete release
env:
GH_TOKEN: ${{ github.token }}
run: |
version="$(tr -d '[:space:]' < VERSION)"
echo "version=$version" >> "$GITHUB_OUTPUT"
# Complete = release exists AND carries every asset the self-updater
# can request. A partial release must NOT short-circuit, so a re-run
# can repair it.
assets="$(gh release view "v$version" --repo "$GITHUB_REPOSITORY" \
--json assets --jq '[.assets[].name] | sort | join(",")' 2>/dev/null || true)"
if [ "$assets" = "gbrain-darwin-arm64,gbrain-linux-x64" ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
echo "Release v$version already published with all assets — nothing to do."
else
echo "exists=false" >> "$GITHUB_OUTPUT"
fi
build:
needs: version
if: needs.version.outputs.exists == 'false'
strategy:
matrix:
include:
@@ -29,11 +82,19 @@ jobs:
with:
bun-version: 1.3.13
- run: bun install
# --timeout matches every scripts/ runner and covers hook budgets too
# (bunfig.toml's timeout key is ignored by bun; hooks default to 5s).
- run: bun test --timeout=60000
- run: bun run verify
# No test re-run here: the Test workflow already gated this exact SHA at
# merge (10 shards + E2E). Re-running the whole suite serially on the
# release runner is a flakier duplicate gate — it blocked the first
# release on ambient-env tests (run 30698650484). The build job's gate
# is the artifact itself: compile, then smoke-test the binary.
- run: bun build --compile --target=${{ matrix.target }} --outfile bin/${{ matrix.artifact }} src/cli.ts
- name: Smoke-test the compiled binary
run: |
chmod +x bin/${{ matrix.artifact }}
out="$(./bin/${{ matrix.artifact }} --version)"
echo "binary reports: $out"
v="$(tr -d '[:space:]' < VERSION)"
case "$out" in *"$v"*) echo "version matches VERSION file" ;; *) echo "binary version '$out' does not contain '$v'" >&2; exit 1 ;; esac
- name: Attest build provenance
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
@@ -44,16 +105,35 @@ jobs:
path: bin/${{ matrix.artifact }}
release:
needs: build
needs: [version, build]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
permissions:
contents: write # create the tag + release (scoped to this job only)
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
path: artifacts
- name: Extract CHANGELOG entry for release notes
# env-bound, not inlined into the script: VERSION comes from master so
# it isn't attacker-reachable today, but a `${{ }}` inside `run:` is
# shell injection by construction if that ever changes.
env:
RELEASE_VERSION: ${{ needs.version.outputs.version }}
run: |
v="$RELEASE_VERSION"
if ! bash scripts/changelog-entry.sh "$v" > /tmp/release-notes.md || ! [ -s /tmp/release-notes.md ]; then
echo "See [CHANGELOG.md](https://github.com/${GITHUB_REPOSITORY}/blob/master/CHANGELOG.md) for v$v." > /tmp/release-notes.md
fi
- name: Create release
uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2
with:
tag_name: v${{ needs.version.outputs.version }}
name: v${{ needs.version.outputs.version }}
target_commitish: ${{ github.sha }}
body_path: /tmp/release-notes.md
fail_on_unmatched_files: true
files: |
artifacts/gbrain-darwin-arm64/gbrain-darwin-arm64
artifacts/gbrain-linux-x64/gbrain-linux-x64
generate_release_notes: true
+14 -1
View File
@@ -87,7 +87,7 @@ jobs:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@dcedce43c6f43de0b836d1fe38946645c9c638dc # v2
- uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -187,6 +187,19 @@ jobs:
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
# (20K-page corpus + ratio guard) shares this runner — same perf-job
# shape, runs in parallel with the matrix.
- run: bun test test/entity-card-perf.slow.test.ts --timeout=300000
# Protocol self-certification: init a scratch brain and run the
# conformance kit against gbrain's own stdio server. --synthesize is
# safe here: no LLM key in CI, so it asserts the clean `unavailable`
# protocol error instead of spending tokens.
- name: MEMORY_VERBS conformance (self-certify, stdio)
run: |
export GBRAIN_HOME="$RUNNER_TEMP/gbrain-conformance"
bun run src/cli.ts init --pglite --no-embedding --non-interactive
bun run src/cli.ts protocol conformance --synthesize
test:
# Pure matrix shard — no verify, no serial. Each shard runs its slice
+368
View File
@@ -2,6 +2,374 @@
All notable changes to GBrain will be documented in this file.
## [0.43.0.0] - 2026-08-08
**Your agent now has five memory verbs it can actually reach.** Cathedral 1 freezes
a stable, versioned memory protocol — `recall`, `remember`, `entity`, `synthesize`,
`forget` — over the brain's operation catalog, the way Postgres speaks one wire
protocol to every client. Point any MCP harness at it (`claude mcp add gbrain --
gbrain serve --surface verbs`, or the Codex/OpenClaw equivalents) and the agent sees
exactly five self-describing tools instead of a wall of internal ops. Every response
carries what it is, why it matched, where it came from, and what it spent (the token
budget on `recall`, latency on `entity`, the full cost block on `synthesize`) — and the
contract never breaks: v1 field names and meanings are frozen, changes are
additive-forever.
What you can do now that you couldn't before:
- **Remember a fact once, recall it in a fresh session — in any harness.** `remember`
takes mandatory provenance (where the fact came from) and an optional expiry, dedupes
against what's already known, and supersedes the old fact when it changes
("X joined acme-example" → "X left acme-example" — the outdated fact expires, the
history stays). `recall` retrieves saved facts and,
with a query, budget-packed page snippets — the server enforces the token budget and
tells you what it dropped instead of trusting the client to trim.
- **Look up one person/company/project as a compact card in well under 100ms, zero LLM
calls.** `entity` resolves a name to a privacy-safe card (who/what, aliases,
last-touched, open threads, top typed edges) — or, on a miss, near-miss suggestions
instead of a dead end.
- **Reason across pages when you actually need it.** `synthesize` is the explicitly
expensive verb (it says so in its own description), with a best-effort cost block so
agents choose it deliberately. No API key configured? It says `unavailable` with a
fix, never a fake answer.
- **Certify any memory server against the contract.** `gbrain protocol --json` publishes
the machine-readable spec; `gbrain protocol conformance [--target <endpoint>]` runs the
frozen-contract test suite against gbrain's own server or any MCP endpoint — and
provably fails servers that don't comply. `gbrain protocol stats` shows per-verb
adoption and your real time-to-first-use, all from a local file that never leaves
your machine.
`gbrain serve` keeps every operation by default (`--surface full`) — existing installs
are unchanged. The new `--surface verbs` is the quickstart surface for agents. No schema
migration; the verbs ride the existing facts, pages, and typed-graph tables. The full
contract, per-harness install blocks, and the additive-forever versioning policy live in
[docs/protocol/MEMORY_VERBS_v1.md](docs/protocol/MEMORY_VERBS_v1.md).
**This release also fixes which search verb your agent reaches for (#2416).** The
`search` and `query` tool descriptions, the mandatory lookup chain, and the search
guides had drifted from reality: they described `search` as keyword-only (it has been
cheap hybrid — vector + keyword, no LLM expansion — for many releases) and told agents
to try `search` first for everything, falling back to `query` only when results looked
"thin". For concept questions ("all the X that do Y", "the ecosystem around Z") that
fallback never fired, and synonym-phrased matches dropped silently.
- **Concept questions now route to `query` by default.** The tool descriptions and the
lookup-chain convention are intent-driven: exact known tokens → `search` (cheaper, no
expansion call); concept / synonym / exhaustive-set questions → `query` first. Verified
with a live LLM routing eval: all concept phrasings route to `query`, and personal
questions still route to the salience ops.
- **"Got results" is no longer treated as "got everything."** The descriptions, docs, and
conventions now say it plainly: a populated `search` result set is not proof of
coverage, and `query` is still top-K — literal exhaustive enumeration belongs to
`list_pages` pagination.
- **The CLI nudges you when it can help.** A concept-shaped `gbrain search "..."` prints a
one-line hint on stderr suggesting the equivalent `gbrain query` call (results stay
clean on stdout; `--quiet` silences it; never auto-reroutes).
- **`think` cost accounting reads consistently.** When no LLM call ran (stubbed or no
API key), `usage` is now uniformly `null` in `--json` output rather than sometimes
missing.
To take advantage of v0.43.0.0: re-run `gbrain serve` with `--surface verbs` to give
your coding agent the five-verb memory protocol (or keep `--surface full` for the
complete operation catalog — both speak the verbs). Run `gbrain protocol conformance`
to self-certify, and `gbrain protocol stats` to watch adoption. Memories your agent
saves are readable by every agent connected to the brain by default; pass
`visibility: "private"` for local-only facts. If your agent instructions or skill
files copy the old "search first, query if thin" rule, refresh them from
`skills/conventions/brain-first.md` — the shipped skillpack carries the corrected
routing.
## [0.42.76.0] - 2026-08-08
**Mistyped or unsupported flags now fail loudly instead of being silently ignored — including the ones that were supposed to make a command safe.**
**Strict flag validation, CLI-wide.** Every gbrain command now rejects a flag it does not understand, with a clear error naming the flag and the command, before any work runs. Before, commands read their flags ad hoc and ignored the rest — so `gbrain post-upgrade --dry-run` accepted the flag, ignored it, and applied migrations for real. That class is gone: the legal flags for every command are derived from each command's own source into a generated registry, checked before dispatch, and a command may only advertise a safety flag like `--dry-run` if its code actually reads it. On commands routed through the operations contract, a trailing `--dry-run` is now a real rehearsal switch rather than a no-op. `--json` invocations get the same error as a structured payload, so scripts fail cleanly too.
**A word of warning (intentional breaking change):** cron jobs or scripts that pass stray, misspelled, or long-removed flags have been running on luck — the flag did nothing. Those invocations now exit with an error naming the flag. That is the point: fix the invocation once and it means what it says forever. Everything after `--` is passthrough and remains untouched.
**Upgrades can't wedge on forward-referenced columns anymore — as a class.** The v0.42.56.0-era startup wedge (a schema blob referencing a column that pre-existing brains didn't have yet) had two more latent instances waiting in the jobs table. Both are now probed and healed at startup, and the schema coverage guard was rewritten to cross-reference every column referenced by the embedded schema against the set of columns any migration has ever added — so a new forward reference cannot ship without its startup probe. A recovery test walks the exact journey an affected brain takes: failed upgrade, retry on the fixed binary, converge with no leftover state blocking the way.
**Remote agents get more, within the same fences.** The `think` operation is now available to remote MCP callers as a read-only synthesis — the local CLI can still persist results, while remote callers are forced read-only. Chunk reads now resolve through the same source-scope rules as page reads, so a federated grant that can open a page can also read that page's chunks, and a caller without the grant cannot reach chunks outside its own floor. Chunk payloads also stop carrying raw embedding vectors over the wire — noticeably smaller responses with no behavior change, since no consumer ever read them. Two internal call sites that forward caller identity now treat anything ambiguous as untrusted, matching the fail-closed rule the rest of the codebase already follows.
**Source-bound clients can be minted over HTTP.** The `/admin/api/register-client` endpoint now accepts `source` and `federatedRead` bindings, mirroring the CLI's `--source` / `--federated-read` flags — so an admin UI or provisioning proxy can create a client confined to a specific brain source without shelling out to the CLI. Omitting both preserves the historical default, and invalid source ids get a structured 400.
**`gbrain doctor` and `repair-jsonb` see further and misfire less.** The double-encoded-JSON scan now covers the subagent execution columns, and the damage test requires the stored text to actually parse as JSON before flagging it — a legitimate string value that merely starts with `[` or `{` (a log line, a code snippet) is no longer misclassified, and a repair pass can no longer corrupt it. One damaged table no longer aborts the scan of the rest.
### To take advantage of v0.42.76.0
## [0.42.75.0] - 2026-08-08
**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.**
The dreaded `RuntimeError: Aborted()` at startup — the one that made zero-config brains unusable after a macOS upgrade and pushed people onto Homebrew Postgres — was never a macOS or WASM bug. An unclean shutdown (typically the upgrade reboot) tears the write-ahead log inside the data dir, and every open after that dies replaying it. gbrain now detects that failure on any command, backs up the WAL state to a sibling directory, resets it in place (the pg_resetwal recovery Postgres has shipped for decades, ported to run against PGLite data dirs), and reopens your brain — pages, embeddings, and history intact. Transactions that never reached a checkpoint may be lost; that is the standard trade for a database that would otherwise not open at all.
### Added
- **Automatic WAL repair on startup.** A torn-WAL abort self-heals on the next gbrain command: backup → in-place reset → retry, with a loud notice naming the backup and recommending `gbrain doctor`. Disable with `GBRAIN_PGLITE_WAL_REPAIR=off`.
- **`gbrain pglite-repair`** — the deliberate version: `--dry-run` gives a read-only diagnosis of the data dir; `--yes` runs the same in-place repair manually. Refuses to operate while any live process holds the brain, and never force-removes another process's lock.
- **`gbrain doctor` diagnoses unopenable PGLite brains.** A new `pglite_data_dir` check reads the data dir from disk when connect fails, names the right recovery rung (repair vs rebuild), inventories repair backups, and escalates when repairs keep recurring — the signal that something is still killing gbrain mid-write.
- **Recovery guardrails throughout:** repair runs only under a cleanly-acquired lock (never after taking over another process's lock, with a quarantine window when a lock's holder couldn't be verified); a live database — including a native Postgres one — is refused by a `postmaster.pid` liveness check; repeated attempts inside one corruption episode reuse one backup instead of stacking copies (newest three episodes retained); a cooldown stops repair loops from silently eating data on machines where crashes keep recurring; and every restore path reports honestly whether your original files are back in place or waiting in the backup.
### Changed
- **`gbrain reinit-pglite` works bare.** The embedding model and dimensions now default from your config file, so the rebuild rung of the recovery ladder is one command mid-outage (explicit flags still win; environment overrides are deliberately ignored so a stale shell export can't change the rebuild target).
- **Honest error messages.** The startup-abort hint now names the real cause (torn WAL after an unclean shutdown), states exactly what auto-repair did or why it stood down, and lays out the full ladder: repair → rebuild → engine switch. The docs that claimed PGLite is "incompatible with macOS 26.x" have been rewritten (README, INSTALL, ENGINES) — thanks @roysaurav for the original native-Postgres walkthrough, which remains the engine-switch rung.
- Message-less WASM error objects no longer surface as `[object Object]`.
### Fixed
- The classifier that routes startup failures now matches the abort message PGLite actually produces (it previously fell through to a generic hint), while catalog corruption keeps routing to rebuild — WAL repair is never suggested for damage it cannot fix.
- Lock-file reads can no longer misclassify a healthy live holder as corrupt (writes are atomic now), a holder owned by another user is treated as alive, and an in-flight acquisition is no longer mistaken for a corrupt lock.
Credit where due: @yang1996202-cpu (#2575), @AndreLYL (#223), and @roysaurav (#1670) for reports and diagnosis, the #223 thread contributors whose recoveries proved the root cause, and @yestheboxer, whose rejected upstream recovery PR (electric-sql/pglite#994) this port builds on.
### To take advantage of v0.42.75.0
```bash
gbrain upgrade
```
Nothing to configure. If a cron job or script starts failing with `unknown flag`, that invocation was passing a flag that did nothing — remove or fix the flag and it will not regress silently again.
### For contributors
Community fixes absorbed with credit: @colinagent (#2598 think read-scope; the upgrade-rewind e2e pattern from #2623), @guim4dev (#2016 register-client source bindings), @vinsew (#597 repair-jsonb coverage extension), @javieraldape (#2494/#2531 output-correctness class — BigInt-safe local rendering and the search `--json` regression pin land here; parts of both PRs shipped earlier from master). Thank you — superseded PRs are being closed with notes.
The unit-test runner is now memory-safe on machines running multiple workspaces: shard concurrency adapts to actually-available memory, and a serial rescue lane re-runs files that died to OOM or external kills before calling them failures — a red suite now means real failures, not memory pressure. The flag registry regenerates via `bun run build:flag-registry` and is pinned by freshness, drift, and consumption-evidence guards.
If your brain currently won't open, that's it — the next command repairs it. If you'd rather look first: `gbrain pglite-repair --dry-run`.
## [0.42.74.0] - 2026-08-07
**Two fixes for agents that reach a brain over the network: takes-holder visibility now works the way you set it, and the voice recipe is safe by default.**
Legacy bearer tokens served over `gbrain serve --http` now honor the takes-holder allow-list you set with `gbrain auth permissions <token> set-takes-holders`. Before, that setting was read on one serving path but silently ignored on the other, so a remote agent saw only world-held takes no matter what you granted — a token you widened to see brain-held takes saw none of them, and a token you narrowed still saw the public ones. Both directions now behave as configured, an empty grant means "no takes" (not "the default set"), and the two serving paths decode and apply the grant through one shared piece of code so they cannot drift apart again. Tokens with no grant continue to fall back to public-only, so nothing widens on upgrade.
The bundled voice-agent recipe (`recipes/agent-voice`) ships secure by default. Its reference server now refuses cross-origin browser requests unless you name the origins in `AGENT_VOICE_CORS_ORIGIN`, gates the endpoints that spend your OpenAI key or read your brain so a stray web page can't trigger them, and listens on loopback only until you set `HOST` to expose it. The voice page you run locally is unaffected. Because this recipe is copied into your own repo at install time, `gbrain integrations install agent-voice --refresh` picks up the hardened version.
### To take advantage of v0.42.74.0
```bash
gbrain upgrade
```
Then, if you serve a brain to remote agents, set each token's takes-holder scope with `gbrain auth permissions <token> set-takes-holders world,brain` (or your desired holders). Voice-recipe operators run `gbrain integrations install agent-voice --refresh --target <your-host-repo>`, then set `AGENT_VOICE_CORS_ORIGIN` if a browser on another origin needs access and `HOST=0.0.0.0` only if the server must listen beyond loopback.
### For contributors
Both issues were reported by external security researchers who supplied fixes. Ship-stage adversarial review hardened two more spots: the two serving paths now share one permissions-decode helper (not just the allow-list parser) so a malformed double-encoded row can't make them disagree, and the hot-memory cache key encodes the allow-list collision-free so the empty-vs-absent distinction holds for every holder value. Credit @Derek95king (takes-holder threading) and @sebastiondev (voice-recipe CORS).
## [0.42.73.2] - 2026-08-05
**A write that deduplication redirects onto an existing page is now checked against the write scope of whoever asked for it.** When the same content arrives under a new slug, gbrain recognises it and points the write at the page that already holds it. That redirected target is now tested against the caller's own scope — under whichever mechanism confines that caller. One of the two mechanisms was consulted at that point; both are now.
Nothing changes for local CLI use, or for clients that hold unrestricted write access — neither was ever scope-confined. A confined caller whose write dedups onto a page **inside** its own scope keeps working exactly as before; that redirect is a feature and it is preserved, with a regression test to keep it that way. A confined caller whose write dedups onto a page **outside** its scope now gets `permission_denied`, with the remedy in the message: drop the `id:` frontmatter field, or change the content, to write a new page under your own prefix. The denial does not name the page the write resolved to.
Recommended for any brain served over HTTP to scope-restricted clients.
### To take advantage of v0.42.73.2
```bash
gbrain upgrade
```
Nothing to configure. Existing clients keep their scopes unchanged, and no re-registration is needed.
### For contributors
Reported privately by an external security researcher, who supplied a fix and a regression test with it. The version that shipped composes the two existing scope-matching rules into a single predicate rather than restating either one, so the check at the door and the check after a redirect cannot drift apart; the audit the report prompted closed the same gap on one further caller path.
## [0.42.73.1] - 2026-08-05
**Removes the PR gate that v0.42.73.0 added, and reverts the v0.42.72.1 contribution-policy change it enforced.** The gate cannot function on this repository, and it caused a real incident before that was understood.
The gate needed two things this repository does not grant it: an `ANTHROPIC_API_KEY` Actions secret for its verdict, and read-write workflow permissions to post a comment or set a label. Without them it can only skip. Worse, on its first live runs a read-only token turned every API call into a 403, the code treated that as a crash, and the check went red on an outside contributor's pull request four times with no comment explaining why. That was fixed in v0.42.73.0, but a check that runs on every pull request and can never reach a verdict does not earn its place in the repository.
The v0.42.72.1 contribution policy is also withdrawn: the human-written intent paragraph and gbrain-in-use screenshot are no longer required on issues and pull requests. `CONTRIBUTING.md`, both issue templates, and the pull-request template return to their pre-2026-08-02 state, and issues and PRs are reviewed on their content by maintainers, as before.
The code is preserved in git history at v0.42.73.0 and can be restored if the repository ever grants those permissions. If it is restored, the mechanical half — the intent and screenshot check, the version-first title rule, the red flags — should render to the Actions job summary instead of a comment, because that needs no token permission and no API key.
### To take advantage of v0.42.73.1
```bash
gbrain upgrade
```
Nothing to change. Everything else v0.42.73.0 shipped — the five contributed correctness fixes, `slug_filter`, and the four dependency pins that cleared six CVEs — is unaffected and stays.
## [0.42.73.0] - 2026-08-04
**Every incoming pull request now gets a verdict before anyone reads it — and five contributed fixes for silent wrong answers.**
**The PR gate.** Open a pull request against gbrain and an automated check now posts a single verdict comment within a minute: **merge-lane**, **close-lane**, or **needs-maintainer**, with its reasons and a checklist of what a human reviewer should verify for that specific diff. It also checks mechanically that the description carries the human-written intent paragraph and the screenshot of gbrain in use that `CONTRIBUTING.md` requires, and that the title leads with its version.
It is deliberately **advisory** — a triage signal and a reviewer checklist, not an authorization boundary. A green verdict is not permission to merge; a maintainer still decides. Pull-request code is never checked out or executed: the verdict comes from the description and the diff read through the API. Maintainer, bot, and draft pull requests are exempt from the intent-and-screenshot floor only (release automation cannot screenshot itself); they still receive the full verdict. Where the rubric can be argued with, the decision is taken away from it: a merge-lane recommendation is downgraded automatically when a diff adds a dependency, a new provider recipe, or new config keys, edits workflows, deletes a test, exceeds 40 files or 400 net source lines, or changes `src/` without touching a single test.
**Your import output parses again.** `gbrain import <dir> --json` printed five informational lines to stdout ahead of the JSON payload, so anything parsing that output read zero imports while its own bookkeeping recorded the files as ingested — and the next run skipped them permanently. Those lines now go to stderr under `--json`; human output is byte-for-byte unchanged.
**`sources harden --dry-run` no longer changes anything.** It reset the helper's executable bit before reaching the dry-run check, so a documented preview quietly mutated permissions.
**Telemetry records the model that actually ran.** Two nightly-cycle phases wrote a hardcoded or unrelated model name into their verdict cache, evidence signature, and spend metering while the gateway ran whatever chat model you configured. On any brain with a non-default model, the recorded history was fiction.
**`gbrain integrity` stops contradicting itself.** Dead-link findings were counted in the "Review queue" total but written to a different file, so `integrity review` disagreed with `integrity auto`'s own summary. They now get their own line.
**Retype rules can address API-ingested pages.** Mapping rules could only filter on a file path, which is empty for every page written through `put_page` — so no rule could target that whole class. A new `slug_filter` filters on the slug instead, and combines with the path filter when both are given.
Also: the `integrity` source comment no longer documents a `--dry-run` subcommand form that exits with an error.
### To take advantage of v0.42.73.0
```bash
gbrain upgrade
gbrain import <dir> --json | jq . # now parses
gbrain integrity auto # dead links reported separately
```
Nothing to configure for the gate — it runs on pull requests to this repository. If you maintain a fork and want it, the workflow needs an `ANTHROPIC_API_KEY` secret; without one it skips loudly rather than blocking anyone.
### For contributors
The gate went through six rounds against two independent blind reviewers, each judging cold. The findings that changed the design most were not exploits but false positives: a code fence that swallowed the rest of a description, an explanation written as bullet points scoring zero words, a word floor stricter than the published policy, and a comment telling contributors to reopen a pull request that was never closed. Those four descriptions are now permanent regression fixtures — a gate that insults a first-time contributor is worse than no gate. Two properties are deliberate and documented rather than fixed: the mechanical floor is a floor (a determined author clears it in seconds), and a bare URL in a cited reason still autolinks.
Contributed by @YiconZiwei (#2655), @time-attack (#3764, #3759, #3726, #3751, #3739, and the gate groundwork in #3573/#3698).
## [0.42.72.1] - 2026-08-02
**Every issue and pull request now needs a human-written paragraph and a screenshot of gbrain actually being used.**
Effective immediately, opening an issue or a PR requires two things from you personally: a paragraph you wrote yourself saying why you're opening it — what you were doing, what went wrong or what you needed, why it matters — and a screenshot of your terminal, agent session, or logs showing the real situation. Rough grammar is fine and preferred over polish. AI-generated or AI-polished intent text is not accepted; the paragraph is the human part. AI assistance for the *code* is still welcome.
Issues and PRs missing either are closed without review, and can be reopened once both are added. Scrub private names, companies, keys, and brain contents from screenshots before attaching — a redacted screenshot is fine, a missing one is not.
The requirement is stated in `CONTRIBUTING.md` and pre-filled in the bug-report and feature-request issue templates plus a new pull-request template, so the fields are in front of you when you open one.
## [0.42.72.0] - 2026-08-01
**Per-person write isolation inside a shared source, and a guide for putting gbrain behind a multi-user agent harness.**
Until now, `--source` was the only write boundary: a client could write anywhere inside the source it was scoped to, and keeping each person in their own folder was a convention the agent had to honor by itself. Registering a client with `--bound-slug-prefixes` now makes that boundary real. Writes outside the bound prefixes are refused by the server, on every op that can name a page.
**Adding a binding to an existing client narrows it on purpose.** Ops that write by something other than a page slug can't be confined to a prefix, so a bound client is refused them outright rather than left with an unfenced path: `extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`, and `POST /ingest`. `put_page`'s automatic fact extraction is skipped for the same reason — it writes to entity pages the caller never named. Reads are unaffected, and unbound clients behave exactly as before. The gate keys on "anything that is not a plain read", so an op added in a future release is refused to bound clients until it is explicitly fenced.
Both prefix spellings work: the `wiki/agents/alice/*` glob that `submit_agent` bindings already use, and the plainer `emp-alice/` form. Change a binding in place with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>` — existing tokens pick it up on their next request, so no secret rotation is needed when someone joins or leaves a team.
**New guide: [gbrain as the company brain for a qm deployment](docs/integrations/qm-harness.md).** qm is a multiplayer agent harness where each employee and each channel gets an isolated agent scope. The guide covers the whole path — one central `gbrain serve --http`, the thin-client binary baked into the sandbox image, one OAuth client per scope, and a roster-driven provisioning script that converges the brain to a list of people and channels. It also states plainly what the model does *not* give you: within a shared source, reads stay source-granular, so prefix isolation is a write boundary, not a privacy boundary.
gbrain upgrade # or: bun install -g gbrain@0.42.72.0
gbrain apply-migrations --yes # required: the fence refuses writes it cannot evaluate
```
To fence an existing client to a folder:
```bash
gbrain auth rescope-client <client_id> --bound-slug-prefixes partners/alice-example/
gbrain auth rescope-client <client_id> --bound-slug-prefixes none # undo
```
Verify it took, from a client holding that credential — the first write should succeed and the second should be refused:
```bash
gbrain put partners/alice-example/notes/test --content "mine"
gbrain put partners/bob-example/notes/test --content "not mine"
```
## [0.42.71.0] - 2026-08-01
**GBrain now publishes real releases. Every version bump from here on lands on the [Releases page](https://github.com/garrytan/gbrain/releases) with organized notes and downloadable binaries — and binary self-update finally works.**
Until now the repo had no releases at all: `gbrain check-update` could tell you a new version existed, but `gbrain self-upgrade` downloaded from an empty releases API and failed every time, and anyone trying to follow what shipped had to read raw commit history. That's what people have been (rightly) complaining about.
From this release forward, every version bump automatically:
- **Tags the commit** (`v0.42.71.0`) so versions are addressable in git.
- **Publishes a GitHub Release** whose notes are that version's CHANGELOG entry — the same organized, user-facing writeup, not a commit dump.
- **Attaches compiled binaries** for macOS (arm64) and Linux (x64), so `gbrain self-upgrade` and fresh binary installs work without a toolchain.
The pipeline is idempotent: a partial release (tag exists, assets incomplete) is repaired on the next run instead of wedging. It runs post-merge, so a flaky release build can never turn master red. Releases for today's two fix waves (v0.42.69.0 and v0.42.70.0) have been backfilled with their CHANGELOG notes so the Releases page tells the whole story of the day; binaries attach from v0.42.71.0 onward.
### To take advantage of v0.42.71.0
```bash
gbrain check-update # now resolves against real releases
gbrain self-upgrade # now actually downloads a binary
```
Or browse https://github.com/garrytan/gbrain/releases for organized per-version notes.
### For contributors
`docs/RELEASING.md` gains the release-publication section; `scripts/changelog-entry.sh` extracts a version's CHANGELOG section (used for release notes — keep entries under the standard `## [X.Y.Z.W]` headers and they publish verbatim). The workflow keeps all actions SHA-pinned, tightens top-level permissions to `contents: read` with write scoped to the release job only, and env-binds all interpolations.
Contributed by @time-attack (#3573, closing #3521).
## [0.42.70.0] - 2026-08-01
**Community fix wave two: 18 contributed fixes. The headline: several things you asked gbrain to do were being quietly ignored — and now they aren't.**
**`--brain` now actually routes.** The documented `gbrain query "X" --brain media-team` parsed the flag and then ran against your host brain anyway. It now routes to the named brain, and an unknown brain name fails loudly instead of silently answering from the wrong database.
**`sync --dry-run` no longer touches anything.** A dry run could pull from the remote and — if your sync strategy had changed — delete indexed pages before the "dry run" early-return was reached. Previews are now read-only, full stop.
**`apply-migrations --yes` applies.** It previously warned that your schema was behind and then printed "All migrations up to date" with exit 0. If you have wedged brains that upgrade never healed, this was why.
**Links between your pages resolve the way you write them.** Dir-qualified wikilinks with raw Obsidian names (`[[wiki/entities/AI 3.0]]`) now resolve to the sync-slugified page; references in non-whitelisted directories are no longer silently dropped; and a scan bug that could add an edge to a *parent* page you never referenced was caught in the wave's composite review and fixed before shipping.
**Windows and self-hosters.** Markdown files keep LF endings so frontmatter parsers stop mis-reading on Windows checkouts; the archive-crawler path gate no longer denies every real Windows path (and no longer fail-opens on NTFS case-insensitivity); a chat-synopsis tier that was hardcoded to one provider now follows your configured models; vector search asks the index for as many candidates as it was told to consider.
**Quieter, more honest infrastructure.** `serve --http` no longer leaves an orphan holding the database lock after Ctrl-C; a minion child that fails to launch settles immediately instead of hanging its slot; doctor gains checks for content-hash duplicates, undeclared database-only pages, stale heartbeats, and a tamper-evident manifest for the skills directory; federated reads respect per-source isolation settings in two more paths; and the security docs were rewritten to describe fixes without cataloguing attack surface.
gbrain upgrade
gbrain extract --stale # re-extracts links under the fixed resolver
gbrain doctor # includes the new silent-failure checks
```
If your brain uses `link_resolution.global_basename` and was populated before this release, a small number of superseded `wikilink_basename` edges can linger beside their newer typed replacements after re-extraction (edge writes are append-only by design). `gbrain reconcile-links` cleans them up; they are harmless to queries that dedup on target.
### For contributors
The composite review of this wave (two independent max-effort review passes over the combined branch) caught two interaction defects that per-PR review could not: the ungated bare-path scanner reading inside wikilink spans, and an extraction watermark set to a date that same-day stamps would already outrun. Both were fixed in the wave with discriminating tests. One reviewed-and-approved PR was deliberately held out: it conflicts semantically with its author's own sibling PR in this wave, and choosing between their two path-resolution mechanisms is the author's call.
Contributed by @time-attack (#3618, #3085, #3539, #3576, #3533, #3453, #3457, #3560, #3161), @daragao3 (#3619, #3536, #3517, #3578), @paul-0320 (#3613, #3564), @cvillarroel2 (#3678), @mamedov (#3624), @dialthewolff (#3550).
## [0.42.69.0] - 2026-08-01
**A community fix wave: 22 contributed fixes, most of them for work your brain was quietly not doing.**
The theme of this release is silent failure. A nightly cycle that reported `ok` while extracting nothing. An `embed` run that left a whole page unsearchable because one chunk in it failed, then exited 0. A health metric that recommended the same step forever because it counted one thing and the fix measured another. None of these looked broken from the outside, which is exactly why they lasted.
**If you run a local or non-Anthropic model, atom extraction was doing nothing.** With a cost cap set, any model absent from the pricing tables made the first work item hard-fail, which latched a budget flag and skipped every remaining item — while the phase still reported success. Local models (`ollama`, `llama-server`) now price at $0, because local inference costs electricity rather than tokens, so their caps stay enforceable. Genuinely unpriced paid providers still skip, but loudly now instead of silently.
**`gbrain embed` no longer lets one bad chunk darken an entire page,** and it exits non-zero when embeddings actually fail. If you have a cron wrapping `gbrain embed`, a brain holding permanently un-embeddable content will now turn that cron red. That is the intended change — it was previously green while silently incomplete.
**Non-Latin and diacritic names now survive mention extraction.** The by-mention tokenizer matched ASCII letters and digits only, so `Đà Nẵng` shredded into one- and two-character fragments and never matched anything. Names in Vietnamese, and any script outside ASCII, are now tokenized properly.
**Self-hosted embedding backends work.** Fixed-dimension OpenAI-compatible servers that reject an explicit `dimensions` parameter no longer get sent one when the requested width already matches the model's native width. A vector search on the embedded database also now asks the index for as many candidates as it was told to consider, instead of silently truncating the pool to the driver default.
**Multi-source brains route correctly in two more places.** A programmatic `sync_brain` call now syncs the source it was handed rather than the global default, and entity slug resolution keeps its path separators instead of flattening `people/alice-example` into an id no page can hold.
**Safer default on a destructive migration.** Submitting the type-unification job without an explicit `apply` now previews instead of applying. If you have that command in a runbook, add `"apply":true` — the playbooks and README were updated to show it.
Also: interrupted imports keep their tail instead of losing progress below the next 100-file boundary; `gbrain init --help` prints its own help instead of a stub; `doctor` stops reporting Windows drive paths as missing files under WSL and bounds its embedding health probe instead of retrying a permanent auth failure three times; cycle lock-release and stamp-write failures are visible instead of swallowed; and references to a `gbrain install` command that never existed are gone from the docs.
### To take advantage of v0.42.69.0
```bash
gbrain upgrade # or: bun install -g gbrain@0.42.69.0
gbrain doctor # confirms the health metric now converges
gbrain embed --stale # exits non-zero if anything is genuinely un-embeddable
```
If you use a local chat model for the nightly cycle, re-run it once and check that atoms actually land:
```bash
gbrain dream --json | jq '.phases[] | select(.name=="extract_atoms")'
```
If you have `gbrain jobs submit unify-types` in a runbook or script, add `"apply":true` to its `--params` or it will now preview only.
### For contributors
Two defects existed only in the *combination* of otherwise-sound fixes, and were caught by reviewing the composed branch rather than the individual changes:
- `isModelPriceable` was introduced with a test asserting `llama-server` is unpriced, while a second fix in the same wave priced `llama-server` at $0. Together the assertion inverted. Reconciled by using a genuinely unpriced provider in the test and pinning the positive case: free local providers are priceable at $0, so their caps stay enforced.
- The type-unification default flipped to dry-run, but three agent-facing playbooks still presented a bare submit as the apply step. Because a second fix in the same wave also edited one of those files, each change looked self-consistent alone. Skills ship downstream via the skillpack, so this would have propagated a playbook whose apply step silently did nothing.
One reviewed fix was deliberately held back: extending the inline subagent drain to Postgres composes badly with this wave's minion connection-recovery work, since the drain calls the same queue operations without the new recovery path and can strand a child job in a per-run queue no worker will claim.
Contributed by @alexey-metaengage (#3652), @time-attack (#3568, #3572, #3567, #3555, #3523, #3144, #3574, #3532, #3545), @brettdavies (#3552, #3553), @mattchronicle (#3364), @rayers (#3589), @zenspam (#3699), @awilhite (#3691), @Grimnoth (#3541), @georgell-ceo (#3634), @Vyacheslav-Zakharov (#3631), @Kyzcreig (#3585), @HammerTech-Z (#3581), @cfeddersen (#3563).
## [0.42.68.1] - 2026-07-30
**If you run `gbrain reindex-frontmatter` or `gbrain backfill` on the default embedded database, they now work. Until this release both failed every time, after waiting 30 seconds.**
+3 -2
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~110 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`; v0.43.0.0 adds the five frozen MEMORY_VERBS — `recall`, `remember`, `entity`, `synthesize`, `forget` — servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -119,6 +119,7 @@ detail on demand.)
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| memory verbs / MCP tool surface (`--surface`) / conformance | `docs/protocol/MEMORY_VERBS_v1.md` + the `verbs*`/`surface.ts`/`protocol.ts` entries in `KEY_FILES.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
| running or writing tests | `docs/TESTING.md` |
| bulk-command progress wiring | `docs/progress-events.md` |
@@ -272,7 +273,7 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 52 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
+26 -8
View File
@@ -19,14 +19,20 @@ The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` that Git for Windows installs by default. A fresh clone is
correct with no extra steps.
If you cloned before that pin existed, your working copy still has the old
Windows line endings and bash will fail with `$'\r': command not found`. Refresh
it once, from the repository root:
`.gitattributes` pins `*.md text eol=lf` for the same reason. The frontmatter
readers anchor on a `---` fence followed by a Unix line ending, so a CRLF
checkout makes a well-formed document parse as having no frontmatter. That
failure is silent: no error, the field just comes back empty.
If you cloned before either pin existed, your working copy still has the old
Windows line endings. Bash will fail with `$'\r': command not found`, and
frontmatter will read as absent. Refresh it once, from the repository root:
```bash
git rm --cached -r . -q
git reset --hard
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
bash -n scripts/run-unit-parallel.sh # silence means bash can read the scripts
git ls-files --eol -- '*.md' | grep -c w/crlf # 0 means Markdown is clean
```
Every `check:*` entry in `package.json` invokes its script as `bash scripts/<name>.sh`
@@ -75,7 +81,7 @@ docs/ Architecture docs
```bash
# Inner edit loop (~85s on a Mac dev box, 3700+ unit tests)
bun run test # parallel 8-shard fan-out + serial post-pass
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
bun test test/markdown.test.ts # specific unit test
# Pre-push gate (matches what CI runs on shard 1 + typecheck)
@@ -112,9 +118,12 @@ trailing-newline and exports-count checks.
### Writing tests that survive the parallel loop
`bun run test` shards 92+ unit-test files across 8 worker processes. Files in the
same shard share a process, so process-global state leaks between them. Four
lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
`bun run test` shards 1000+ unit-test files across up to 4 worker processes,
capping total concurrency (shards × intra-shard files) to available memory and
re-running OOM-killed or externally-killed files serially before calling them
failures (see `docs/TESTING.md` for the rescue-pass details and knobs). Files
in the same shard share a process, so process-global state leaks between them.
Four lint rules (`scripts/check-test-isolation.sh`, R1-R4) enforce isolation:
| Rule | What it bans | Fix |
|---|---|---|
@@ -211,6 +220,15 @@ automatically appears in the CLI, MCP server, and tools-json:
For CLI-only commands (init, upgrade, import, export, files, embed, doctor, sync):
1. Create `src/commands/mycommand.ts`
2. Add the case to `src/cli.ts`
3. Regenerate the flag registry: `bun run build:flag-registry`. The CLI rejects
unknown flags before dispatch; each CLI-only command's legal flag set is
derived from its source into `src/core/cli-flag-registry.generated.ts`.
`test/cli-flag-validation.test.ts` pins registry freshness, drift, and
consumption evidence (a safety flag like `--dry-run` may only be advertised
if the command's code actually reads it), so a stale registry fails the
build. At runtime a missing registry entry fails open — a forgotten regen
never bricks a command. Rerun the regen whenever you add or remove a flag
on an existing command, too.
Parity tests (`test/parity.test.ts`) verify CLI/MCP/tools-json stay in sync.
+10 -8
View File
@@ -91,7 +91,7 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
The agent installs GBrain, creates the brain, asks for your API keys, loads 52 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
@@ -99,13 +99,15 @@ The agent installs GBrain, creates the brain, asks for your API keys, loads 43 s
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel:
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full 110-tool wall; drop the flag for every operation:
```bash
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve # or: codex mcp add gbrain -- gbrain serve
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve --surface verbs # or: codex mcp add gbrain -- gbrain serve --surface verbs
```
If `claude` is not found, install Claude Code first — or use the per-harness blocks in the [protocol doc](docs/protocol/MEMORY_VERBS_v1.md). Heads-up: memories agents save default to brain-wide visibility (every connected agent can recall them); pass `visibility: "private"` for local-only facts.
**Already have a brain on a remote host** (OpenClaw, Hermes, or any `gbrain serve --http`)? Point your laptop agents at it with one command each — `--install` wires it up and smoke-tests the token before handoff:
```bash
@@ -117,7 +119,7 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install
### Install the full autonomous setup into your existing agent
Want the whole thing — local brain, 43 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
Want the whole thing — local brain, 52 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
```
Retrieve and follow the instructions at:
@@ -140,7 +142,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
### Connect GBrain to your AI client (MCP)
GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:
GBrain exposes 110 tools over MCP (stdio and HTTP) — or exactly the five memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
@@ -216,7 +218,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain``gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -317,7 +319,7 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
## Troubleshooting
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
+18 -23
View File
@@ -34,15 +34,18 @@ enforced structurally by actionlint on every workflow change.
## Remote MCP Security
### ⚠️ Do NOT use open OAuth client registration for remote MCP
### Keep dynamic client registration disabled unless explicitly needed
If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1
support, **never allow unauthenticated client registration**. An attacker
who discovers your server URL can:
GBrain disables Dynamic Client Registration (DCR) by default. Keep that
default for internet-reachable deployments and pre-register trusted clients
with operator-approved scopes and source access. Enabling DCR lets network
callers create OAuth client records, so use it only when the deployment's
trust model requires self-service registration and browser approval remains
part of the authorization flow.
1. Register a new OAuth client via `POST /register`
2. Use `client_credentials` grant to obtain a bearer token
3. Access all brain data via the MCP tools
Do not enable `--enable-dcr-insecure` on an untrusted network. That option is
reserved for deployments that intentionally allow self-registered
machine-to-machine clients without browser approval.
### Recommended: `gbrain serve --http`
@@ -104,12 +107,10 @@ Auth methods (`--token-endpoint-auth-method`):
- `none` — public PKCE-only client (no secret minted; ChatGPT custom
connector, Claude Code, Cursor)
The validator rejects unknown methods at the registration boundary, and
the same gate applies to the admin endpoint `POST /admin/api/register-client`
and the DCR `POST /register` path. Pre-v0.41.3 the CLI hard-coded
`redirect_uris = []` and `token_endpoint_auth_method = NULL`, forcing
operators to UPDATE `oauth_clients` rows by hand to make claude.ai work
without `--enable-dcr`. That footgun is gone.
The same validator applies to CLI, admin, and DCR registration paths, so
unknown authentication methods are rejected consistently. Browser-based
clients can be configured entirely through the supported CLI flags; operators
do not need to edit OAuth database rows by hand.
### DCR consent default (v0.42.55+)
@@ -186,16 +187,10 @@ When the request `Origin` matches the allowlist, the server echoes it
back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no
CORS header is sent and the browser blocks the request.
**v0.41.3:** the same allowlist now gates every OAuth endpoint (`/mcp`,
`/token`, `/authorize`, `/register`, `/revoke`). Pre-v0.41.3 these used
default-wide-open `cors()` middleware, leaking
`Access-Control-Allow-Origin: *` on every response — any web origin could
complete a token exchange from a logged-in operator's browser. The CORS
preflight handler in the legacy bearer transport was also asymmetric
(actual-request path correctly default-deny, but OPTIONS preflight leaked
`Access-Control-Allow-Methods` + `Access-Control-Allow-Headers` to every
Origin); both are now consolidated through a single allowlist-gated path.
A startup stderr WARN fires when `--bind 0.0.0.0` is set without
The same allowlist gates the complete MCP and OAuth HTTP surface. Actual
requests and browser preflight requests use one allowlist-gated policy, so
unlisted origins receive no cross-origin authorization. A startup stderr
warning fires when `--bind 0.0.0.0` is set without
`GBRAIN_HTTP_CORS_ORIGIN`, surfacing the default-deny posture before the
first request.
+191 -1
View File
@@ -1,5 +1,195 @@
# TODOS
## #2416 follow-ups (query-steering wave)
- [ ] **P2 — MCP-envelope `hint` field for concept-shaped `search` calls.**
**What:** surface the concept→query nudge to remote/MCP agent callers, not
just the CLI. **Why:** MCP agents are the primary misrouting class the
#2416 issue describes; the shipped CLI stderr nudge covers the caller class
*least* at risk. **Context:** the `search` op returns a bare
`SearchResult[]` (`src/core/operations.ts` — both return sites), so a hint
needs an envelope change that ripples into `formatResult`, MCP
serialization, and array-shape tests — deliberately kept out of the atomic
#2416 commit. The pure classifier already exists
(`looksConceptShaped`/`conceptNudge` in `src/core/search/query-intent.ts`);
only the transport is missing. Consider a sibling metadata channel (like
`_meta.metric_glossary`) rather than changing the array shape.
**Depends on:** agreeing an envelope pattern that doesn't break existing
MCP consumers.
## MEMORY_VERBS v1 follow-ups (filed v0.43.0.0 — Cathedral 1)
Deferred from the Cathedral 1 ship (CEO review, EXPANSION mode). Both are
additive to the frozen v1 contract — neither breaks it. See plan + GSTACK
REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-agile-iverson.md`
and the scope record at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-06-12-memory-verbs-protocol.md`.
- [ ] **P3 — external-implementation certification PROGRAM.** The conformance
TOOLING shipped (`gbrain protocol conformance --target <endpoint>`); the
PROGRAM around it (badges, a registry of conformant implementations, listed
third-party servers) waits for a second implementation to exist. **Why:** the
protocol-not-product thesis only pays off once someone else implements
MEMORY_VERBS; until then a certification program certifies an empty set.
**Where:** new — would build on `src/commands/protocol.ts` conformance output.
- [ ] **P3 — persistent open-threads model for the entity card.** v1 derives
`entity.open_threads` from active commitment-kind facts + recent timeline
entries (best-effort, possibly empty). A richer model (a real threads table:
conversation id, opened/closed state, last activity) would make open-threads
authoritative. **Why:** the card's open-threads field is the weakest signal
in v1; a first-class threads store would make it load-bearing. **Where:**
`src/core/verbs/entity-card.ts` open-threads assembly + a new schema table
(additive — the card field already exists, so this is a quality upgrade, not
a contract change).
- [ ] **P2 — `recall` filter composition vs the spec (found by the v0.43.0.0
cross-model doc review).** The handler dispatch is first-match
(`supersessions` > `entity` > `session_id` > `since`), so `since` is
silently ignored when `entity`/`session_id` is supplied, and `limit` has no
server-side cap. Either compose the filters (additive — the spec's "filters
the FACTS arm" wording already reads that way) or spell the precedence out
in `docs/protocol/MEMORY_VERBS_v1.md`. **Where:** the `recall` handler in
`src/core/operations.ts`.
- [ ] **P3 — widen `synthesize`'s `unavailable` mapping.** Only the
missing-key gateway warning maps to the `unavailable` error today; other
no-usable-model failures can surface as `internal` (contract-legal but less
actionable) or, worst case, a stubbed success. Audit the gateway failure
modes and map every model-unusable path to `unavailable` with a fix
suggestion. **Where:** the `synthesize` handler in `src/core/verbs.ts`.
## Fix-wave 1 follow-ups (upgrade-wedge + trust-seam wave, 2026-08)
Deferred from the un-wedge-v121 hotfix wave (eng review + codex outside voice
CLEARED; every item an explicit review decision). Waves 26 of the sequence are
planned separately (provider-compat rescue is next; its original 2026-07-24
DeepSeek-deprecation deadline has now PASSED — re-verify each cluster against
master before starting, several fixes landed independently).
- [ ] **P2 — Shared strict `parseFlags` helper as the #2185 end-state (eng
review 2B).** This wave ships the generated known-flags registry +
pre-dispatch validator (parser and registry can drift only until the
freshness guard fires). The structural end-state migrates commands onto one
shared strict parser so parser == registry by construction; mechanical but
touches 60+ command files — its own PR. Where: `src/commands/*.ts`,
`src/cli.ts`, `scripts/generate-flag-registry.ts` (retires).
- [ ] **P2 — `whoknows` CLI routing (surfaced by the #2035-class sweep).**
`handleCliOnly`'s `whoknows` case (the dedicated CLI renderer with
thin-client routing) is dead code — the command resolves via the
`find_experts` op alias, and adding it to CLI_ONLY trips the alias-collision
guard. Decide the intended surface alongside PR #2509 (whoknows --explain
per-result factor breakdown) and delete whichever lane loses. Where:
`src/cli.ts`, `src/commands/whoknows.ts`, PR #2509.
- [ ] **P3 — #2544 second half: per-put_page `getAllSlugs` full scan.** The
getChunks egress half shipped in this wave (explicit non-vector column
list). The remaining Postgres-egress cost is put_page's per-call
`getAllSlugs` table scan — needs a targeted existence probe or cached slug
set. Where: `src/core/operations.ts` put_page path, both engines.
- [ ] **P3 — #1558 admin-UI register form.** The `/admin/api/register-client`
API now accepts `source` + `federatedRead` (this wave, PR #2016 absorbed);
the admin SPA form fields + `/admin/api/sources` picker are the UI layer.
Where: `src/commands/serve-http.ts` admin SPA blob.
- [ ] **P3 — jsonb-integrity surfaces: batch + share (ship-review follow-up).**
doctor's jsonbIntegrityCheck runs 2 queries per target (16 round-trips) and
duplicates the TARGETS table with repair-jsonb (already drifted once on the
jsonPayloadOnly predicate before being mirrored by hand). Batch the counts
into one UNION ALL query and extract a shared targets constant
(src/core/jsonb-integrity-targets.ts) consumed by both. Where:
`src/commands/doctor.ts` jsonbIntegrityCheck, `src/commands/repair-jsonb.ts`.
- [ ] **P3 — register-client HTTP-level e2e (ship-review follow-up).** The
source/federatedRead lane is covered by unit normalizers + a structural
route pin; a DATABASE_URL-gated serve-http e2e (register with bindings →
assert stored client via /admin/api/agents; invalid source → 400
invalid_source) closes the wire-level gap. Where:
`test/e2e/serve-http-oauth.test.ts`.
- [ ] **P3 — get_chunks `__all__` sentinel narrows to 'default' (red-team,
Wave 3 territory).** `sourceScopeOpts` returns `{}` for a trusted local
`--source __all__` caller (documented "spans the brain"), but both engines'
getChunks map empty scope to the 'default' floor — the one read op where
`{}` is reinterpreted. Fold into the Wave 3 source-federation cluster's
`__all__` work (an explicit unscoped signal in the engine signature, or
handler-side expansion for trusted callers). Where: `src/core/operations.ts`
get_chunks, both engines' getChunks.
- [ ] **P3 — #2536 wedged-migration diagnostics.** The v121 wedge aborted
initSchema BEFORE runMigrations, so the wedged-migration diagnostics row was
never written — operators got a bare SQL error with no remediation hint.
Write the diagnostics row (or a stderr remediation block) from the blob-replay
catch path too. Where: `src/core/migrate.ts`, `src/commands/apply-migrations.ts`.
## WAL-repair wave follow-ups (#223/#1670/#2575)
- [ ] **P2 — gate auto-repair on an unclean-shutdown marker (adversarial F7).** The classifier
deliberately over-matches (`RuntimeError`/`unreachable``wasm-abort`). If an unclean
shutdown leaves a REPLAYABLE WAL tail (normal crash recovery would restore those committed
txns) and the reopen then fails on a transient WASM error (OOM), auto-repair fires, layout
validation can't tell torn from replayable, and resetWal discards the tail while the notice
says "data preserved." Bounded today (backup always taken + restore + honest failure + repeated
attempts capped), but a false-positive-with-successful-retry silently drops committed data.
Fix direction (probe-verified): PGLite removes `postmaster.pid` on clean close, so gate AUTO
repair (not the manual command) on `postmaster.pid` presence — a clean dir that aborts is not
torn-WAL. Requires making the serial regression test stamp a `postmaster.pid` before corrupting
(it currently clean-disconnects then corrupts, which the red-team flagged as unfaithful anyway).
Needs a recall/precision call before landing.
- [ ] **P3 — live non-gbrain PGLite consumer not caught by the postmaster.pid liveness guard
(adversarial F8).** PGLite writes a sentinel `postmaster.pid` of `-42`; the liveness refusal in
`validateWalRepairTarget` requires `pid > 0`, so it protects native Postgres dirs but not a
non-gbrain pglite app that has the dir open (such an app writes no `.gbrain-lock`). Deliberate
misuse of `pglite-repair --path <foreign pglite dir>` required. Option: refuse when
postmaster.pid holds pid ≤ 0 with a very recent mtime, or document the boundary.
- [ ] **P3 — mixed-version torn-lock read (adversarial F10 residual).** The heartbeat + initial
lock writes are atomic (tmp+rename) now, but an OLD gbrain binary writing heartbeats IN PLACE
while a NEW binary poll-reads can still catch a torn read → corrupt-lock verdict → a live
holder's lock reaped → two writers (the #2348 class, version-skew-triggered). The reap marker
quarantines repair, not the concurrent open. Cheap hardening: double-read the lock file (~50ms
apart) before declaring it corrupt.
- [ ] **P2 — graceful PGLite close on SIGTERM for the remaining long-running paths.**
The torn-WAL genesis this wave repairs is an unclean shutdown: `src/core/process-cleanup.ts`
releases locks on SIGTERM but never closes the PGlite handle, so `serve` / `jobs work` /
`sync` killed mid-write (macOS-upgrade reboot, `systemctl stop`) leave the WAL torn.
Autopilot already ships the pattern (d2fd1f29, #3178/#1872: `registerCleanup('autopilot-engine-close', ...)`
— abort in-flight work → ≤2s bounded wait inside the 3s cleanup deadline →
`engine.disconnect()`, double-call safe; rationale comment at autopilot.ts:438-452).
Extend that exact pattern to the remaining long-running PGLite paths (register in
connect()/command scope; dedupe so autopilot doesn't double-close), pinned by a serial
lifecycle test. Interacts with #2084 exitCode containment + #1337 close ordering — read
those comments in pglite-engine.ts first. Auto-repair makes recurrence self-healing
meanwhile, so this is prevention, not recovery.
- [ ] **P3 — pglite upgrade blocker tracker.** Two couplings make a "routine" pglite bump a
breaking change: (a) pglite ≥0.5 removes the `@electric-sql/pglite/vector` export that
`pglite-engine.ts` imports (verified against npm); (b) the pg_resetwal port
(`src/core/pglite-resetwal.ts`) is coupled to the PG17 pg_control layout
(`PG_CONTROL_VERSION` 1700 — guarded at runtime by `WalResetUnsupportedError`, so a
mismatched bump makes the repair tool refuse every dir rather than corrupt, but it still
means the repair feature silently dies). Any future pglite upgrade wave must revisit BOTH
together and re-derive the ControlFileData offset table for the new PG major.
## serve --http takes-holders + agent-voice hardening follow-ups (filed v0.42.74.0)
Deferred from the #2529/#2477 security-fix wave (plan-eng-review + codex outside
voice CLEARED). None block the wave.
- [ ] **P2 — Per-OAuth-client `takes_holders` storage (#2529 follow-up).** Legacy
bearer tokens honor `access_tokens.permissions.takes_holders` through
`verifyAccessToken`; OAuth clients have no equivalent column on `oauth_clients`,
so OAuth-minted tokens fail closed to `['world']`. Needs a schema migration
(`oauth_clients.takes_holders` JSONB or TEXT[]) + a `register-client` flag +
the `verifyAccessToken` JOIN projection. Include surfacing the EFFECTIVE
takes-holder scope in `whoami` output as part of this follow-up, so operators
can self-diagnose the legacy-vs-OAuth semantic split instead of reading docs.
Where: `src/schema.sql`, `src/core/migrate.ts`, `src/core/oauth-provider.ts`,
`src/commands/auth.ts`, `src/core/operations.ts` (whoami).
- [ ] **P3 — agent-voice Host-header allowlist (DNS-rebinding hardening).** The
#2477 fix ships default-deny CORS + an Origin gate on `/session`/`/tool`, but
the gate derives self-origin from the `Host` header, so a DNS-rebound page
(attacker origin whose host resolves to the operator's loopback) still passes.
Validate `Host` against `localhost`/`127.0.0.1`/operator-configured hosts and
403 otherwise; slots beside `originAllowed()` in the router. Issue #2477
explicitly deferred this. Where: `recipes/agent-voice/code/server.mjs`.
- [ ] **P3 — Debounce `last_used_at` in the oauth-provider legacy path.** The
legacy branch of `verifyAccessToken` fires an unconditional
`UPDATE access_tokens SET last_used_at = now()` on EVERY request, while the
legacy HTTP transport debounces the same write to once per 60s via a WHERE
clause (`src/mcp/http-transport.ts` validateToken). Apply the same pattern —
one fewer write per request on the `serve --http` hot path.
Where: `src/core/oauth-provider.ts`.
## v0.42.67.0 follow-ups (Windows build tooling)
Filed as follow-ups from v0.42.67.0 (`.gitattributes` LF pin for `*.sh` +
@@ -2580,7 +2770,7 @@ contributor traps.
- [ ] **v0.37.x: Adopt `resolveDefaultHeaders` for Together / Groq / other attribution-bearing recipes.** v0.37.6.0's `default_headers` / `resolveDefaultHeaders` seam is generic — any recipe whose provider benefits from app-attribution headers can opt in. Together and Groq both have rankings/analytics tied to per-app headers. Add their respective attribution headers to each recipe, similar to OR's `HTTP-Referer` + `X-OpenRouter-Title`. No type-system or gateway changes needed; just `default_headers` blocks on the existing recipes plus `<PROVIDER>_REFERER` / `<PROVIDER>_TITLE` env vars in their `auth_env.optional`. Filed during v0.37.6.0 eng review as a D4 generalization opportunity.
- [ ] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
- [x] **v0.37.x: Guard cli.ts `main()` so importing `buildGatewayConfig` doesn't print help.** v0.37.6.0 exported `buildGatewayConfig` from `src/cli.ts` for test access. Importing it triggers the file's top-level `main()` which prints help to stdout during tests — functionally harmless (tests pass) but noisy. Fix: wrap `main()` in `if (import.meta.main)` so it only runs when cli.ts is the entry point, not when imported. Touches one line; trivial. Filed during v0.37.6.0 implementation.
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
+1 -1
View File
@@ -1 +1 @@
0.42.68.1
0.43.0.0
+4 -3
View File
@@ -19,7 +19,8 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10",
"nanoid": "^3.3.17",
"postcss": "^8.5.23",
},
"packages": {
"@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="],
@@ -224,7 +225,7 @@
"ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],
"nanoid": ["nanoid@3.3.16", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q=="],
"nanoid": ["nanoid@3.3.18", "", { "bin": { "nanoid": "bin/nanoid.cjs" } }, "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w=="],
"node-releases": ["node-releases@2.0.37", "", {}, "sha512-1h5gKZCF+pO/o3Iqt5Jp7wc9rH3eJJ0+nh/CIoiRwjRxde/hAHyLPXYN4V3CqKAbiZPSeJFSWHmJsbkicta0Eg=="],
@@ -232,7 +233,7 @@
"picomatch": ["picomatch@4.0.4", "", {}, "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A=="],
"postcss": ["postcss@8.5.19", "", { "dependencies": { "nanoid": "^3.3.12", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-Mz8SaolMd8nB+G13WkORcxQKHZ/NE4xXevtkJHVuG+guo9/wYKlIMTKAqGdEmYOXR2ijPjTYNHssizdaVSUNdQ=="],
"postcss": ["postcss@8.5.25", "", { "dependencies": { "nanoid": "^3.3.16", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw=="],
"react": ["react@19.2.5", "", {}, "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA=="],
+2 -1
View File
@@ -20,6 +20,7 @@
},
"overrides": {
"@babel/core": "^7.29.6",
"postcss": "^8.5.10"
"postcss": "^8.5.23",
"nanoid": "^3.3.17"
}
}
+9 -9
View File
@@ -26,7 +26,7 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.15.0",
"js-yaml": "^3.15.1",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
@@ -53,13 +53,13 @@
"overrides": {
"@hono/node-server": "^2.0.5",
"body-parser": "^2.3.0",
"fast-uri": "^3.1.4",
"fast-uri": "^3.1.5",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"js-yaml": "^3.15.0",
"hono": "^4.12.34",
"ip-address": "^10.3.1",
"js-yaml": "^3.15.1",
"qs": "^6.15.2",
},
"packages": {
@@ -401,7 +401,7 @@
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.4", "", {}, "sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw=="],
"fast-uri": ["fast-uri@3.1.5", "", {}, "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw=="],
"fast-xml-builder": ["fast-xml-builder@1.3.0", "", { "dependencies": { "path-expression-matcher": "^1.6.2", "xml-naming": "^0.3.0" } }, "sha512-F74cZEdCvuw9P41GAC3rod4X04jjWGM1JPEv/GWSqFTWLsdyMSBMBMlm9Hk3GLBgLBbdBNY8yee0pQh2RBVESQ=="],
@@ -437,7 +437,7 @@
"heic-decode": ["heic-decode@2.1.0", "", { "dependencies": { "libheif-js": "^1.19.8" } }, "sha512-0fB3O3WMk38+PScbHLVp66jcNhsZ/ErtQ6u2lMYu/YxXgbBtl+oKOhGQHa4RpvE68k8IzbWkABzHnyAIjR758A=="],
"hono": ["hono@4.12.30", "", {}, "sha512-emn+JoJjrN9YTpRDS5it/UI2SO9BAE37T6I3d963RxcZ81G9A4pr2SZTEiiaiKbzx+NKRg5BZ89fCL7gCJCUog=="],
"hono": ["hono@4.13.0", "", {}, "sha512-jhunvfHWxd7J5EFfSgH4xsYJzSe/lfqbUCxiyyeaQasUsXeEHXtzVid+7EOGByc5JnFa23SSFL3Y2RV/z1T+eQ=="],
"http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],
@@ -447,7 +447,7 @@
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"ip-address": ["ip-address@10.2.0", "", {}, "sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA=="],
"ip-address": ["ip-address@10.4.0", "", {}, "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ=="],
"ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
@@ -461,7 +461,7 @@
"jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="],
"js-yaml": ["js-yaml@3.15.0", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog=="],
"js-yaml": ["js-yaml@3.15.1", "", { "dependencies": { "argparse": "^1.0.7", "esprima": "^4.0.0" }, "bin": { "js-yaml": "bin/js-yaml.js" } }, "sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag=="],
"json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="],
+60
View File
@@ -221,6 +221,66 @@ live in `test/postgres-engine-rls-scope.test.ts`.
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
**Symptom:** every PGLite-touching command dies at startup with
`PGLite failed to initialize its WASM runtime … Aborted(). Build with
-sASSERTIONS for more info.` — commonly first seen right after a macOS
upgrade.
**Real root cause:** corrupt WAL/checkpoint state in the data dir after an
unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the
write-ahead log; every subsequent open fails WAL replay inside WASM and
Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM
incompatibility — the same signature reproduces across macOS versions and on
Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun
version bump changes it.
**Recovery ladder** (top rung first):
1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort,
backs up `pg_wal/` + `pg_control` into a sibling
`<dataDir>.wal-repair-backup-<ts>/` dir, resets the WAL in place
(pg_resetwal semantics — data files preserved; transactions not
checkpointed before the corruption may be lost), and retries once. On
success it prints a loud stderr notice naming the backup and recommending
`gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired
data-dir lock (never after reaping another process's lock), skips for a
cooldown window after a failed attempt
(`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one
backup per corruption episode (newest 3 episodes retained), and restores
the original files if the retry still fails. Kill-switch:
`GBRAIN_PGLITE_WAL_REPAIR=off`.
2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir
(read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset
deliberately. Refuses when another gbrain process holds the brain (a live
`gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`.
3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default
from your config) wipes and re-creates the brain from your brain repo, or
manually: back up `~/.gbrain`, move `brain.pglite` aside,
`gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`.
Required for *catalog* corruption (58P01 / pgvector load failure) — WAL
repair cannot fix that class.
4. **Switch engines.** `gbrain init --supabase`, or native Postgres +
pgvector (recipe below, contributed by @roysaurav):
```bash
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# ~/.gbrain/config.json: { "engine": "postgres",
# "database_url": "postgresql://localhost:5432/gbrain" }
gbrain apply-migrations --yes && gbrain doctor
```
`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails
to connect: it diagnoses the dir from disk, names the repair command, reports
retained repair backups, and escalates when repairs keep recurring (that
means the unclean-shutdown genesis is still active — see the ladder's rung 4).
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
+20 -8
View File
@@ -9,7 +9,7 @@ Already running [OpenClaw](https://github.com/garrytan/openclaw) or [Hermes](htt
```bash
bun install -g github:garrytan/gbrain
gbrain init --pglite # 2 seconds; no server
gbrain skillpack scaffold --all # 43 skills scaffolded into your agent workspace
gbrain skillpack scaffold --all # 52 skills scaffolded into your agent workspace
gbrain doctor # green checks all the way down
```
@@ -58,16 +58,17 @@ gbrain autopilot --install # background daemon for nightly enrichment
**Wire this same local brain into your coding agent** — zero server, zero token:
```bash
claude mcp add gbrain -- gbrain serve # Claude Code
codex mcp add gbrain -- gbrain serve # Codex
claude mcp add gbrain -- gbrain serve --surface verbs # Claude Code
codex mcp add gbrain -- gbrain serve --surface verbs # Codex
```
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
The agent spawns `gbrain serve` as a stdio subprocess against your local brain. `--surface verbs` gives the agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](protocol/MEMORY_VERBS_v1.md)) instead of the full tool catalog; drop the flag (default `full`) for every operation. Full walkthrough (both this local path and connecting to a remote brain), plus the brain-first protocol to paste into `CLAUDE.md` / `AGENTS.md`: **[Give your coding agent a memory](tutorials/connect-coding-agent.md)**.
## 3. MCP server (any MCP client)
```bash
gbrain serve # stdio MCP (Claude Desktop / Code / Cursor)
gbrain serve --surface verbs # stdio MCP, just the 5 memory verbs (quickstart)
gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard
```
@@ -117,7 +118,20 @@ If anything's yellow, `gbrain doctor` names the fix command in the message. Most
### PGLite crashes on macOS 26.x (Tahoe)
PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL:
This crash (`RuntimeError: Aborted()` at engine startup, typically first seen
after a macOS upgrade) is **not** a macOS/WASM incompatibility. The upgrade
reboot kills gbrain mid-write and tears the data dir's write-ahead log; every
subsequent open then fails WAL replay. Recovery ladder:
1. **Auto-repair (default):** just run any gbrain command — gbrain detects the
abort, resets the WAL in place (data preserved; a backup of the pre-repair
state is kept next to the data dir), and continues. Then run `gbrain doctor`.
2. **Manual repair:** `gbrain pglite-repair --dry-run` to diagnose,
`gbrain pglite-repair --yes` to repair in place.
3. **Rebuild:** `gbrain reinit-pglite` (wipes and re-creates the brain from
your brain repo; embedding settings default from your config).
4. **Switch engines** — if you prefer a server database anyway, native
Homebrew PostgreSQL works great and supports multiple concurrent agents:
```bash
# Install PostgreSQL + pgvector
@@ -144,6 +158,4 @@ gbrain apply-migrations --yes
gbrain doctor
```
All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend.
> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again.
Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend (plus multi-connection support: several agents can share one Postgres brain, which PGLite's single-process lock doesn't allow).
+33
View File
@@ -362,6 +362,39 @@ done
If any SHA differs from what's in the workflow files, update the pin and version comment.
## GitHub releases (binary assets + self-update) — #3521
`.github/workflows/release.yml` publishes a GitHub release automatically for
**every VERSION bump that lands on master** (trigger: push to master touching
`VERSION`, plus `workflow_dispatch` for a manual first run or repair). No
manual tag push is part of the ship flow — the workflow reads `VERSION` (the
single source of truth), mints tag `v<VERSION>` at the pushed commit, titles
the release the same, uses that version's `CHANGELOG.md` entry as the notes
(`scripts/changelog-entry.sh`; falls back to a CHANGELOG link if the entry is
missing), and attaches the compiled binaries.
Why every bump, not selective: `gbrain check-update` resolves the latest
version from `VERSION` on master, while binary self-update
(`src/core/binary-self-update.ts`) downloads assets from `releases/latest`.
Any release that lags `VERSION` tells binary installs an upgrade exists that
self-update cannot apply. `releases/latest` must track `VERSION`.
Invariants:
- **Asset names are a contract.** The build matrix's `artifact:` names must
equal what `expectedAssetName()` in `src/core/binary-self-update.ts`
returns (`gbrain-darwin-arm64`, `gbrain-linux-x64` today). Adding a
platform means updating BOTH plus the version job's completeness check;
`test/release-workflow.test.ts` pins all of it.
- **Idempotent + self-repairing.** The version job skips when a release for
`v<VERSION>` already exists with all expected assets; a partial release
(tag but no release, or missing assets) is completed on re-run. Racing
master pushes queue via the `release` concurrency group — a skipped
intermediate version is fine, latest is what matters.
- **Historical tags are never rewritten.** Old 3-segment versions keep their
history; every new 4-segment `VERSION` mints a fresh tag.
- **Permissions stay scoped.** `contents: write` lives on the release job
only; everything else runs read-only.
## PR descriptions cover the whole branch
+19 -7
View File
@@ -11,7 +11,7 @@ Seven test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
| `bun run test` | Parallel unit-test fast loop. Up-to-4-shard fan-out via `scripts/run-unit-parallel.sh` (min(CPUs, 4)), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | ~85s on a Mac dev box (3700+ tests) | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
@@ -32,9 +32,12 @@ CI, macOS) rejects CRLF and dies on the script's first meaningful line; the Cygw
bash that ships with Git for Windows tolerates it, so a green local run is not by
itself evidence that a script is CRLF-clean.
The root `.gitattributes` pins `*.sh text eol=lf`, which overrides the
`core.autocrlf=true` default that Git for Windows installs. Working copies cloned
before that pin need a one-time `git rm --cached -r . -q && git reset --hard` to
pick it up; see the Windows section of `CONTRIBUTING.md`.
`core.autocrlf=true` default that Git for Windows installs. It pins `*.md` the
same way, because the frontmatter readers anchor on a `---` fence followed by a
Unix line ending and a CRLF checkout makes a document parse as having no
frontmatter, silently. Working copies cloned
before those pins need a one-time `git rm --cached -r . -q && git reset --hard` to
pick them up; see the Windows section of `CONTRIBUTING.md`.
Wallclock figures in the table above are from a Mac dev box. Windows is
substantially slower because each check pays full process-creation cost, and three
@@ -47,7 +50,7 @@ there even though they pass on Linux and macOS.
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
### Failure-first logging
@@ -58,17 +61,26 @@ When `bun run test` finds any failure, the wrapper:
3. Writes a one-line-per-shard summary to `.context/test-summary.txt` (`shard N/M: pass=X fail=Y skip=Z rc=W`).
4. Exits non-zero. Empty failure log + non-zero exit = infrastructure problem (wedged shard, killed child); the banner says so.
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 3000s; `GBRAIN_TEST_SHARD_KILL_AFTER` grace after TERM before KILL, default 30s), the wrapper writes `--- shard N: WEDGED after ${SHARD_TIMEOUT}s ---` to the failure log, includes the last 50 lines of the shard log, and proceeds with other shards' results.
### File taxonomy
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
### Skills-manifest freshness guard
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
### Test-isolation lint and helpers
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
File diff suppressed because one or more lines are too long
+6 -2
View File
@@ -97,7 +97,11 @@ Promote or reject them via `gbrain extraction-pending` / `gbrain
extraction-review`.
The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool +
title + alias, expansion off); `query` is the full-control variant. NamedThingBench
title + alias, expansion off); `query` is the full-control variant. Route
concept / landscape / "all-of-X" questions to `query` — expansion recovers
synonym-phrased matches `search` can miss, and a populated `search` result set
is not proof of coverage (both are top-K; exhaustive enumeration belongs to
`list_pages`). NamedThingBench
(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a
specific miss with `gbrain search diagnose "<q>" --target <slug>`.
@@ -116,7 +120,7 @@ The classifier is deterministic (no LLM call). Wrong classification degrades gra
For `detail: 'high'` searches, `src/core/search/expansion.ts` runs a Haiku-class LLM call to produce 2-3 query variants. Each variant runs through the full hybrid stack; results merge via RRF. Catches synonym misses without recall loss.
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale.
Expansion is opt-in per mode bundle (`tokenmax` on by default; `balanced` + `conservative` off). Default off in the cheap tiers because the LLM call adds ~$0.001/query and ~200ms — real money at scale. The `query` op is the exception: it defaults `expand: true` per call (pass `expand: false` to opt out) — expansion-by-default is what makes it the concept/landscape verb.
## Putting it together
+3 -2
View File
@@ -56,7 +56,8 @@ that tuple lights up the `pack_upgrade_available` onboard check.
│ gbrain onboard --check --explain shows per-cluster narrative │
│ User reviews; if OK, runs: │
│ gbrain jobs submit unify-types --allow-protected \ │
│ --params '{"target_pack":"gbrain-base-v2"}'
│ --params '{"target_pack":"gbrain-base-v2","apply":true}'
│ (omit "apply":true for a dry-run; that is the default) │
│ (Autopilot never auto-fires this; manual_only) │
└──────────────────────────┬─────────────────────────────────────┘
@@ -229,7 +230,7 @@ add `GBRAIN_AUDIT_FULL=1` (v0.43+ TODO; not yet wired).
- Per-source pack-upgrade (the handler accepts `sourceId` but
`findPackSuccessors` doesn't yet pass it through)
- Cross-brain federated mounts that disagree on canonical packs
- Automatic rollback (today: manual SQL or `gbrain pages restore`)
- Automatic rollback (today: manual SQL or `gbrain restore`)
- LLM-assisted mapping_rules codegen from production data (`gbrain
schema detect-mappings`; deferred to v0.43+)
+1 -1
View File
@@ -214,7 +214,7 @@ gbrain schema downgrade
1. `git revert <merge-commit>` — restores the code.
2. `gbrain schema downgrade --to gbrain-base` — restores config.
3. (Optional) `gbrain pages purge-deleted --older-than 0h` — drops
3. (Optional) `gbrain purge-deleted --older-than 0h` — drops
v0.39-typed pages that no longer have a matching type in the active
pack.
+10 -10
View File
@@ -19,11 +19,13 @@ entire DB from scratch.
This means:
- **Disaster recovery is one command.** If your DB volume corrupts, if
Postgres eats itself, if PGLite's WASM lock wedges — you don't need
a backup. You wipe the DB, re-import from your brain repo, and the
derived state regenerates. v0.32.3 ships `gbrain rebuild
--confirm-destructive` as the documented one-liner.
- **Disaster recovery is a short, boring sequence.** If your DB volume
corrupts, if Postgres eats itself, if PGLite's WASM lock wedges — you
don't need a backup. You wipe the derived tables (on PGLite,
`gbrain reinit-pglite` wipes the whole embedded DB), re-import from
your brain repo with `gbrain sync`, and `gbrain extract all`
regenerates the derived state. See "Disaster recovery" below for the
exact commands.
- **Multi-machine sync is git.** Your brain is a repo. Push from one
machine, pull from another, and the second machine's DB rebuilds on
its next sync. No "back up the database" step.
@@ -146,11 +148,9 @@ The promise the rule makes:
# Snapshot what's there
gbrain stats > /tmp/before.txt
# Wipe and rebuild
gbrain rebuild --confirm-destructive # v0.32.3 — deletes derived tables
# (pages + content_chunks survive
# the CASCADE-safe design)
# OR manually for v0.32.2:
# Wipe and rebuild — delete the derived tables (pages + content_chunks
# survive the CASCADE-safe design), then re-derive from the repo.
# On PGLite, `gbrain reinit-pglite` wipes the whole embedded DB instead.
psql -c 'DELETE FROM facts; DELETE FROM takes; DELETE FROM links; DELETE FROM timeline_entries;'
gbrain sync
gbrain extract all
+4 -4
View File
@@ -64,7 +64,7 @@ Key files:
thin-client routing branches. These commands bypass the operation-layer
dispatch in cli.ts (call `engine.foo()` directly), so each gets its own
`if (isThinClient(cfg)) { callRemoteTool(...) }` branch that maps CLI flags
to op params. `think` is a special case: the server's `think` op
intentionally disables `--save`/`--take` for remote callers
(operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns
loudly when those flags are set.
to op params. `think` is a special case: the server's `think` op is
read-scoped for OAuth/MCP and intentionally disables `--save`/`--take` for
remote callers in its trust-boundary gate; thin-client `think` warns loudly
when those flags are set.
+6 -4
View File
@@ -76,7 +76,8 @@ gbrain onboard --check --explain # per-cluster narrative dry-run
gbrain jobs submit unify-types \ # PROTECTED + manual_only
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
# omit "apply":true → dry-run (default)
Handler runs 4 phases:
┌─────────────────────────────────────┐
@@ -108,8 +109,8 @@ Every primitive ships with a documented rollback:
| Operation | Rollback |
|-----------|----------|
| Retype | `frontmatter.legacy_type = <original>` preserved on every page (D8). One SQL UPDATE restores types: `UPDATE pages SET type = frontmatter->>'legacy_type' WHERE frontmatter ? 'legacy_type'`. |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain pages restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Page-to-link | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Link row stays harmless if source restored. |
| Page-to-alias | Source page soft-deleted with 72h TTL. `gbrain restore <slug>` within 72h. Alias row stays harmless (or `DELETE FROM slug_aliases WHERE alias_slug = <slug>` to clean up). |
| Active-pack flip | `gbrain schema use gbrain-base` reverses the flip. |
## What if my brain doesn't fit?
@@ -127,7 +128,8 @@ For brains with substantial custom types that deserve their own canonical
2. Edit your fork to add page_types + mapping_rules covering your
custom domain.
3. Target your fork: `gbrain jobs submit unify-types --allow-protected
--params '{"target_pack":"my-pack"}'`
--params '{"target_pack":"my-pack","apply":true}'` (omit `"apply":true`
for a dry-run preview — that is the default)
Your fork can also declare `migration_from: {pack: gbrain-base-v2,
version: "1.x"}` to register itself as a successor — future agents
+5 -4
View File
@@ -154,10 +154,11 @@ these are the densest source of real bugs in the whole backlog.
`aliases:`), first-H1-title, and basename fallback resolution (path-equality-only gives
~5.5% edge recall on real vaults). Master shipped global-basename (#1388); the alias/
title fallbacks are the still-novel part.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **OPEN, high.** The
link extractor's `DIR_PATTERN` is a frozen 16-prefix const that ignores pack-declared
`path_prefixes`, so default-pack installs silently lose wikilinks to `person/`,
`writing/`, `wiki/*`. Resolve prefixes from the active pack.
- **Schema-pack-aware link extraction** (#1547, @billy-armstrong) — **RESOLVED via #2576.**
The extractor no longer gates on the frozen `DIR_PATTERN` whitelist: any dir-shaped
path produces a candidate and the persist paths' page-existence checks decide, so
pack-declared directories (`person/`, `writing/`, `wiki/*`, `ops/`) link without a
prefix registry.
- **DB-source extraction** (#1539, @afshaker) — **OPEN, high.** The cycle's extract phase
only walks the filesystem, so DB-resident pages (imported transcripts, remote-DB brains)
never get links/timeline and `brain_score` is capped. Thread `source:'db'`.
+1 -1
View File
@@ -183,6 +183,6 @@ This also means the best AI agent setups will be open source by default. Closed,
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain install voice-agent`
`gbrain skillpack scaffold voice-agent`
That's it.
+1 -1
View File
@@ -69,7 +69,7 @@ update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain add_timeline_entry {slug} {
gbrain timeline-add {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
+9 -9
View File
@@ -46,10 +46,10 @@ on user_shares_media(url_or_file):
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
gbrain link <slug> <person_slug>
gbrain link <person_slug> <slug>
gbrain timeline-add <person_slug> {date} \
"Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
@@ -80,8 +80,8 @@ on user_shares_media(url_or_file):
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
@@ -109,8 +109,8 @@ on user_shares_media(url_or_file):
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
gbrain link <slug> <entity_slug>
gbrain link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
@@ -127,7 +127,7 @@ on user_shares_media(url_or_file):
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
2. Run `gbrain call get_links '{"slug": "media/youtube/{slug}"}'`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
+9 -9
View File
@@ -49,23 +49,23 @@ on enrich(entity, trigger):
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
gbrain call put_raw_data \
'{"slug": "<entity_slug>", "data": {"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
gbrain timeline-add <entity_slug> {date} "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
gbrain timeline-add <entity_slug> {date} "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
gbrain link <person_slug> <company_slug> # person -> company
gbrain link <company_slug> <person_slug> # company -> person
gbrain link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
@@ -94,8 +94,8 @@ on enrich(entity, trigger):
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
2. Run `gbrain call get_raw_data '{"slug": "<slug>"}'`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain call get_links '{"slug": "<slug>"}'`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
+5 -5
View File
@@ -53,7 +53,7 @@ on upcoming_meeting(meeting):
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain get_links <attendee_slug>,
"relevant_deals": gbrain call get_links '{"slug": "<attendee_slug>"}',
}
else:
briefing[attendee] = "No brain page -- consider enriching"
@@ -67,14 +67,14 @@ on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
gbrain timeline-add <sender_slug> {date} \
"Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
+7 -7
View File
@@ -32,15 +32,15 @@ on new_meeting_transcript(meeting):
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
gbrain timeline-add <person_slug> {date} \
"Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
gbrain timeline-add <company_slug> {date} \
"Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
@@ -49,8 +49,8 @@ on new_meeting_transcript(meeting):
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
gbrain link <slug> <entity_slug> # meeting -> entity
gbrain link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
@@ -73,7 +73,7 @@ on new_meeting_transcript(meeting):
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
4. Run `gbrain call get_links '{"slug": "meetings/{date}-{slug}"}'`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
---
+3 -3
View File
@@ -91,7 +91,7 @@ first):
6. The seeded `default` source.
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
`~/.gstack` via `.gbrain-source`, `gbrain put` implicitly writes to
the `gstack` source. Outside any registered directory with no env/dotfile
set, it writes to the default.
@@ -188,10 +188,10 @@ citations keep working.
```bash
# Pass --source explicitly
gbrain put-page topics/ai ... --source wiki
gbrain put topics/ai ... --source wiki
# Or rely on the dotfile / env / CWD match
cd ~/.gstack && gbrain put-page plans/multi-repo ...
cd ~/.gstack && gbrain put plans/multi-repo ...
# → source auto-resolves to gstack
```
+8 -8
View File
@@ -20,8 +20,8 @@ on every_inbound_message(message):
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
gbrain timeline-add <entity_slug> {date} \
"{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
@@ -64,13 +64,13 @@ on nightly_schedule("02:00"):
# The brain COMPOUNDS overnight.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list_pages
pages = gbrain list
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain get_links <page.slug>
existing_links = gbrain call get_links '{"slug": "<page.slug>"}'
for mention in mentions:
if mention not in existing_links:
gbrain add_link <page.slug> <mention_slug> # fix broken graph
gbrain link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
@@ -80,7 +80,7 @@ on nightly_schedule("02:00"):
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain get_timeline <page.slug>
timeline = gbrain timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
@@ -110,11 +110,11 @@ on nightly_schedule("02:00"):
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain call get_links '{"slug": "<slug>"}'`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*
+3 -3
View File
@@ -47,8 +47,8 @@ on user_message(message):
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
gbrain link originals/{slug} <entity_slug>
gbrain link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
@@ -79,7 +79,7 @@ on user_message(message):
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
3. Run `gbrain call get_links '{"slug": "originals/ambition-debt"}'`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
+1 -1
View File
@@ -87,7 +87,7 @@ expect it.
| `version` | string | yes | Your plugin's semver. Informational. |
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
| `description` | string | no | Shown in future `gbrain plugin list`. |
| `description` | string | no | Shown in a future plugin-listing command. |
## Subagent definition files
+28 -19
View File
@@ -15,40 +15,48 @@ on user_asks_about(topic):
if know_exact_slug(topic):
# MODE 3: Direct get -- instant, no search overhead
result = gbrain get <slug>
# e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi
# e.g., "Tell me about Alice" -> gbrain get alice-example
# Returns the FULL page -- compiled truth + timeline
elif topic.is_exact_name or topic.is_keyword:
# MODE 1: Keyword search -- fast, no embeddings needed, day-one ready
# MODE 1: Cheap-hybrid search -- vector + keyword + RRF, NO LLM
# expansion. Embeds the query when embeddings are configured; the
# keyword arm still works day-one without them (keyword-only is
# also available via the search.mcp_keyword_only opt-out).
results = gbrain search "{name_or_keyword}"
# e.g., "Find anything about Series A" -> gbrain search "Series A"
# Returns CHUNKS, not full pages
# IMPORTANT: keyword search returns chunks
# IMPORTANT: search returns chunks
# If the chunk confirms relevance, THEN load the full page:
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
elif topic.is_semantic_question:
# MODE 2: Hybrid search -- semantic + keyword, needs embeddings
elif topic.is_semantic_question or topic.is_concept_or_landscape:
# MODE 2: Full hybrid -- adds multi-query LLM expansion on top of
# vector + keyword + RRF. Owns concept / landscape / "all-of-X"
# questions: expansion recovers synonym- and outcome-phrased
# matches a single embedding misses. Costs one LLM expansion call
# per query -- worth it for these question shapes.
results = gbrain query "{natural language question}"
# e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts"
# Returns ranked chunks via vector + keyword + RRF
# e.g., "all the companies doing offshore wind" -> gbrain query "..."
# Returns ranked chunks via vector + keyword + expansion + RRF
# Same rule: chunks first, then get full page if needed
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
# Quick reference:
# | Mode | Command | Needs Embeddings | Speed | Best For |
# |---------|----------------------|------------------|---------|---------------------------------|
# | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches |
# | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match |
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
# | Mode | Command | Needs Embeddings | Speed | Best For |
# |-------------|----------------------|------------------|---------|-------------------------------------------|
# | Cheap-hybrid| gbrain search "term" | Uses if present | Fastest | Known names, exact tokens |
# | Full hybrid | gbrain query "..." | Yes | Fast | Concept / landscape / "all-of-X", synonyms |
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
# Progression over time:
# Day 1: keyword search (works without embeddings)
# After first embed: hybrid search unlocked
# Day 1: search (keyword arm works without embeddings)
# After first embed: vector arm + full hybrid (query) unlocked
# Once you know slugs: direct get for speed
# Precedence for conflicting information within a page:
@@ -61,16 +69,17 @@ on user_asks_about(topic):
## Tricky Spots
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available.
3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug.
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page.
5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
2. **Search works without embeddings.** On day one before any embedding run, `gbrain search` still works (the keyword arm carries it; the vector arm joins once embeddings exist). Don't tell the user "search isn't available yet" -- search is always available.
3. **Don't use full hybrid for known names.** `gbrain query "Alice Example"` wastes an LLM expansion call. Use `gbrain search "Alice Example"` or better yet `gbrain get alice-example` if you know the slug.
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Alice" -- get the full page.
5. **Full hybrid needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
6. **A populated `gbrain search` result set is not proof you found everything.** Search runs without query expansion, so synonym- and outcome-phrased matches can be missed even when it returns plenty of hits. For "find every / all / the landscape of" questions, use `gbrain query`; for literal exhaustive enumeration ("list every page of type X"), use `list_pages` pagination. A nonzero count is not a completeness signal.
## How to Verify
1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references.
1. Run `gbrain search "Alice"` -- confirm it returns chunks with matching text and slug references.
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline.
3. Run `gbrain get alice-example` -- confirm it returns the full page with compiled truth and timeline.
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
+9 -1
View File
@@ -44,6 +44,7 @@ These require manual setup (no self-installing recipe yet):
|-------|-------------|
| [Credential Gateway](credential-gateway.md) | Set up ClawVisor or Hermes for Gmail, Calendar, Contacts access |
| [Meeting & Call Webhooks](meeting-webhooks.md) | Circleback meeting transcripts + Quo/OpenPhone SMS/calls |
| [qm Harness](qm-harness.md) | gbrain as the company brain for a qm (multi-user agent harness) deployment — central HTTP MCP, per-scope clients, roster provisioning, write fencing |
## How to Read a Recipe
@@ -69,6 +70,12 @@ health_checks: # typed DSL to verify the integration is working
auth_user: "$TWILIO_ACCOUNT_SID"
auth_token: "$TWILIO_AUTH_TOKEN"
label: "Twilio account"
- type: heartbeat_max_age # staleness gate: FAILS `integrations doctor`
max_age: 48h # when the newest heartbeat event is older.
label: "Data freshness" # The other types are point-in-time and stay
# green even when a sense stops producing data.
output_paths: # repo-relative dirs the collector writes files to;
- daily/voice/ # lets doctor/sync warn if one lands in db_only
setup_time: 30 min # estimated time to complete setup
---
@@ -86,7 +93,8 @@ a source install, or the global install copy) are trusted. Recipes discovered at
runtime from `$GBRAIN_RECIPES_DIR` or a cwd-local `./recipes/` are marked untrusted:
they cannot run `command` health checks, cannot run `http` health checks (SSRF
defense), and cannot use the deprecated string health_check form. Untrusted recipes
can still use `env_exists` and `any_of` compositions. To ship a recipe that runs
can still use `env_exists`, `heartbeat_max_age` (reads only the local heartbeat
file — no exec, no network), and `any_of` compositions. To ship a recipe that runs
live checks, contribute it upstream so it becomes package-bundled.
## The Deterministic Collector Pattern
@@ -0,0 +1,79 @@
---
name: gbrain
description: Search and write the company knowledge brain. Use for any question about the org, people, projects, decisions, or history, and to persist durable knowledge beyond this scope's notebook.
---
# gbrain — the company brain
This sandbox has the `gbrain` CLI connected (thin-client) to the org's central
brain. It is the deep, indexed, cross-source memory: org docs, shared channel
knowledge, and every agent's durable notes. Your scope's own notebook stays the
fast per-turn memory; the brain is where knowledge outlives a scope and becomes
searchable by everyone entitled to it.
## First-run setup (once per sandbox — skip if `gbrain remote doctor` passes)
Your scope's brain credentials arrive via the deployment's secret handoff
(keychain entry or one-time secret drop named `gbrain`). Then:
```bash
gbrain init --mcp-only \
--issuer-url "https://brain.<org>.com" \
--mcp-url "https://brain.<org>.com/mcp" \
--oauth-client-id "<client id from the handoff>" \
--oauth-client-secret "<client secret from the handoff>"
gbrain whoami # must succeed before using any other command
```
Pass the secret with `--oauth-client-secret`, not via `GBRAIN_REMOTE_CLIENT_SECRET`:
an env-sourced secret is deliberately NOT written to `~/.gbrain/config.json`, so
every later command would fail with "No client_secret available" once the
variable is out of scope. The flag persists it to the config file on this
sandbox's durable disk, which is what the tool's credential capture expects.
Do not run `gbrain remote doctor` — it needs `admin` scope, which your client
does not have (by design). `gbrain whoami` is the read-scope health check.
## Reading (do this liberally)
```bash
gbrain search "who decided X and why" # hybrid semantic + keyword search
gbrain get <slug> # read one page
gbrain query "question" --json # search tuned for agent consumption
```
You can read: the shared agent-memory source, org read-only sources (wiki,
handbook), and everything under them. Reads are isolation-enforced server-side;
you only ever see sources your client is entitled to.
## Writing (durable knowledge only, under YOUR prefixes)
Your client is write-fenced to slug prefixes — your own namespace plus the
channels you belong to. Writes outside them are rejected server-side.
```bash
# personal durable memory (your namespace):
gbrain put emp-<your-slug>/people/jane-example --content "..."
# shared channel knowledge (channels you are in):
gbrain put chan-eng/decisions/2026-08-database-choice --content "..."
```
Conventions:
- Write conclusions and durable facts, not chat transcripts. One page per
entity/decision/topic; update the page rather than appending near-duplicates.
- Markdown with YAML frontmatter; the brain chunks, embeds, and links it.
- Cross-reference liberally: `gbrain link <from> <to>` (from must be in your
namespace; linking TO any readable page is fine).
- When you learn something channel-relevant in personal work, mirror the
conclusion into the channel prefix with a `(said in <where>)` provenance
note.
## When to reach for the brain
- Any question about the org, a person, a project, a decision, or history →
`gbrain search` FIRST, then answer.
- You produced knowledge with value beyond this conversation → `gbrain put`.
- Something looks wrong (auth errors, empty results you don't expect) →
`gbrain whoami` to confirm which client and scopes you're using, and report
its output.
+258
View File
@@ -0,0 +1,258 @@
#!/usr/bin/env bash
# provision-scopes.sh — roster-driven gbrain provisioning for a qm deployment
# (or any multi-user agent harness with per-person + per-channel scopes).
#
# Reads a roster of channels + employees and converges the brain to it:
# - ensures the shared agent-memory source exists (path-less: agents write
# pages into it over MCP; `gbrain sync` skips it; if the brain host has
# sync.repo_path configured, pages also write through to .sources/<id>/
# on disk for git-backed durability)
# - registers one OAuth client per employee, write-fenced via
# bound_slug_prefixes to emp-<slug>/ plus chan-<c>/ for each channel
# they are in, with federated reads over the memory source + any
# read-only sources you pass
# - re-running after roster edits rescopes existing clients IN PLACE
# (client ids are remembered in the state file; secrets never rotate
# unless you revoke + delete the state row)
#
# Usage:
# provision-scopes.sh roster.tsv \
# [--memory-source agents] [--read-sources org-wiki,handbook] \
# [--budget-usd-per-day 5] [--state-file roster.state.tsv] \
# [--secrets-out new-credentials.tsv] [--gbrain gbrain] [--dry-run]
#
# Roster format (one entry per line; '#' comments and blank lines ignored):
# channel <slug>
# employee <slug> [comma-separated channel slugs]
#
# SECURITY: --secrets-out receives client secrets for NEW registrations,
# written exactly once (gbrain never re-shows them). Deliver each row to its
# scope's sandbox (e.g. via the harness keychain or a one-time secret drop),
# then delete the file.
#
# ponytail: sequential CLI loop, one gbrain invocation per roster row — fine
# to hundreds of employees; batch via the admin API if that ever hurts.
# -f (noglob) is load-bearing, not stylistic: roster lines are word-split
# unquoted below, so without it a line like `employee * eng` would expand
# against the working directory and silently provision a filename as a
# person — i.e. the wrong write fence. Nothing here needs globbing.
set -euf -o pipefail
# Client secrets and the id state file are written by this script; 077 makes
# them 0600 instead of the default 0644. Set before the first file is created.
umask 077
die() { echo "ERROR: $*" >&2; exit 1; }
# Slugs become source ids, client names, AND slug-prefix write fences. The
# fence list is comma-separated, so an unvalidated slug containing a comma
# would inject an EXTRA prefix and hand the client write access to someone
# else's namespace. Fail closed on anything that isn't plain kebab-case.
valid_slug() {
case "$1" in
'') return 1 ;;
-*|*-) return 1 ;;
*[!a-z0-9-]*) return 1 ;;
*) return 0 ;;
esac
}
require_slug() {
valid_slug "$2" || die "roster: invalid $1 slug '$2' (allowed: lowercase a-z, 0-9, interior hyphens)"
}
ROSTER="${1:-}"
[ -n "$ROSTER" ] && [ -f "$ROSTER" ] || die "usage: provision-scopes.sh <roster-file> [flags] (roster not found: '$ROSTER')"
shift
GBRAIN="${GBRAIN:-gbrain}"
MEMORY_SOURCE="agents"
READ_SOURCES=""
BUDGET="5"
STATE_FILE=""
SECRETS_OUT=""
DRY_RUN=0
while [ $# -gt 0 ]; do
case "$1" in
--memory-source) MEMORY_SOURCE="$2"; shift 2 ;;
--read-sources) READ_SOURCES="$2"; shift 2 ;;
--budget-usd-per-day) BUDGET="$2"; shift 2 ;;
--state-file) STATE_FILE="$2"; shift 2 ;;
--secrets-out) SECRETS_OUT="$2"; shift 2 ;;
--gbrain) GBRAIN="$2"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
*) die "unknown flag: $1" ;;
esac
done
STATE_FILE="${STATE_FILE:-${ROSTER}.state.tsv}"
SECRETS_OUT="${SECRETS_OUT:-${ROSTER}.new-credentials.tsv}"
# The roster usually lives in the deployment repo, so the default secrets and
# state paths land there too — one `git add -A` from committing live
# credentials. The STATE file matters as much as the secrets file: it maps
# employee -> client_id, and this script feeds that id straight to
# `rescope-client`, so whoever can write it decides which client receives a
# given employee's write authority. Treat both as privileged infrastructure,
# at the same trust level as the roster itself.
for f in "$SECRETS_OUT" "$STATE_FILE"; do
if git -C "$(dirname "$f")" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
echo "WARN: $f is inside a git work tree. Never commit it;" >&2
echo " gitignore it, or pass --secrets-out/--state-file outside the repo." >&2
fi
done
# A group/world-writable parent directory defeats the symlink and ownership
# checks below: anyone with write access there can swap the file between our
# check and our append. Refuse rather than pretend the checks hold.
for d in "$(dirname "$SECRETS_OUT")" "$(dirname "$STATE_FILE")"; do
perms=$(ls -ld "$d" | awk '{print $1}')
case "$perms" in
?????w*|????????w*) die "refusing to write credentials into a group/world-writable directory: $d ($perms)" ;;
esac
done
# Secure the credential sinks BEFORE anything is appended. umask only governs
# files this script creates; a pre-existing world-readable file would receive
# secrets first and be chmod'ed only afterwards, and a symlink planted at
# either path would redirect them entirely.
for f in "$SECRETS_OUT" "$STATE_FILE"; do
[ -L "$f" ] && die "refusing to write credentials through a symlink: $f"
if [ -e "$f" ]; then
[ -f "$f" ] || die "refusing to write credentials to a non-regular file: $f"
[ -O "$f" ] || die "refusing to write credentials to a file owned by another user: $f"
else
: > "$f"
fi
chmod 600 "$f"
done
run() {
if [ "$DRY_RUN" = 1 ]; then echo "DRY-RUN: $GBRAIN $*" >&2; return 0; fi
# shellcheck disable=SC2086 — $GBRAIN may carry args ("bun run src/cli.ts")
$GBRAIN "$@"
}
state_lookup() { # state_lookup <employee-slug> -> client_id or empty
[ -f "$STATE_FILE" ] || return 0
awk -F'\t' -v s="$1" '$1 == s { print $2; exit }' "$STATE_FILE"
}
# ── Pass 1: parse roster, collect declared channels ─────────────────────────
CHANNELS=""
EMPLOYEES=""
lineno=0
while IFS= read -r line || [ -n "$line" ]; do
lineno=$((lineno + 1))
line="${line%%#*}"
line="${line%$'\r'}" # a CRLF roster would otherwise yield 'emp-alice\r/' prefixes that fence everything out
[ -z "${line//[[:space:]]/}" ] && continue
# shellcheck disable=SC2086 — deliberate word split; globbing is off (set -f above)
set -- $line
[ "$#" -le 3 ] || die "roster line $lineno: too many fields ('$line'). Channels are ONE comma-separated field with no spaces: 'employee alice eng,product'"
case "$1" in
channel)
require_slug channel "${2:-}"
CHANNELS="$CHANNELS $2"
;;
employee)
require_slug employee "${2:-}"
case " $EMPLOYEES " in *" $2:"*) die "roster line $lineno: employee '$2' listed twice" ;; esac
if [ -n "${3:-}" ]; then
for c in ${3//,/ }; do require_slug "channel-reference" "$c"; done
fi
EMPLOYEES="$EMPLOYEES $2:${3:-}"
;;
*) die "roster line $lineno: unknown entry type '$1' (expected 'channel' or 'employee')" ;;
esac
done < "$ROSTER"
# ── Pass 2: ensure the shared memory source exists (path-less) ──────────────
if out=$(run sources add "$MEMORY_SOURCE" --name "agent memory ($MEMORY_SOURCE)" 2>&1); then
echo "source '$MEMORY_SOURCE': created"
else
echo "$out" | grep -q "already registered" || die "sources add failed: $out"
echo "source '$MEMORY_SOURCE': already exists"
fi
# ── Pass 3: converge one client per employee ────────────────────────────────
FED_READ="$MEMORY_SOURCE${READ_SOURCES:+,$READ_SOURCES}"
new_secrets=0
for entry in $EMPLOYEES; do
slug="${entry%%:*}"
chans="${entry#*:}"
prefixes="emp-$slug/"
if [ -n "$chans" ]; then
for c in ${chans//,/ }; do
echo " $CHANNELS " | grep -q " $c " || echo "WARN: employee '$slug' references undeclared channel '$c'" >&2
prefixes="$prefixes,chan-$c/"
done
fi
client_id="$(state_lookup "$slug")"
if [ -n "$client_id" ]; then
# The state file usually sits in the deployment repo, so anyone who can
# edit it could otherwise retarget this privileged rescope at an arbitrary
# client id (e.g. point alice's row at an admin client). Shape-check it.
case "$client_id" in
gbrain_cl_) die "state file: empty client id for '$slug'" ;;
gbrain_cl_*[!a-zA-Z0-9_]*) die "state file: malformed client id for '$slug': $client_id" ;;
gbrain_cl_*) ;;
*) die "state file: client id for '$slug' does not look like a gbrain client: $client_id" ;;
esac
# --source too, so a re-run actually CONVERGES the client to the roster:
# without it, changing --memory-source (or inheriting a state row written
# against an older one) silently leaves the old write source in place
# while the script reports success.
run auth rescope-client "$client_id" --source "$MEMORY_SOURCE" \
--federated-read "$FED_READ" --bound-slug-prefixes "$prefixes" >/dev/null
echo "employee '$slug': rescoped $client_id [write: $prefixes]"
elif [ "$DRY_RUN" = 1 ]; then
echo "employee '$slug': WOULD register qm-emp-$slug [write: $prefixes] [read: $FED_READ]"
continue
else
out=$(run auth register-client "qm-emp-$slug" \
--grant-types client_credentials --scopes "read write" \
--source "$MEMORY_SOURCE" --federated-read "$FED_READ" \
--bound-slug-prefixes "$prefixes" --budget-usd-per-day "$BUDGET" 2>&1) \
|| die "register-client failed for '$slug' (output withheld: it can contain a secret). Re-run the command by hand to see why."
client_id=$(echo "$out" | sed -n 's/.*Client ID:[[:space:]]*\(gbrain_cl_[^[:space:]]*\).*/\1/p' | head -1)
secret=$(echo "$out" | sed -n 's/.*Client Secret:[[:space:]]*\(gbrain_cs_[^[:space:]]*\).*/\1/p' | head -1)
if [ -z "$client_id" ] || [ -z "$secret" ]; then
# The client may well have been created — dying silently would strand a
# live credential nobody can find. Say so WITHOUT echoing the captured
# output: it contains the freshly minted secret, and this path ends up
# in CI logs.
die "could not parse client id/secret for '$slug' from register-client output (output withheld: it contains a secret). A client MAY have been created; check \`gbrain auth list\` and revoke any stray 'qm-emp-$slug'."
fi
printf '%s\t%s\n' "$slug" "$client_id" >> "$STATE_FILE"
printf '%s\t%s\t%s\n' "$slug" "$client_id" "$secret" >> "$SECRETS_OUT"
chmod 600 "$STATE_FILE" "$SECRETS_OUT" 2>/dev/null || true # umask covers new files; this covers pre-existing ones
new_secrets=$((new_secrets + 1))
echo "employee '$slug': registered $client_id [write: $prefixes]"
fi
done
# ── Pass 4: flag offboarded employees ───────────────────────────────────────
# Removing someone from the roster is the highest-stakes edit there is, and
# this script cannot safely revoke on its own (a typo'd roster would nuke live
# credentials). Report instead, with the exact command.
if [ -f "$STATE_FILE" ]; then
while IFS=$'\t' read -r st_slug st_client _rest; do
[ -n "${st_slug:-}" ] || continue
case " $EMPLOYEES " in
*" $st_slug:"*) ;;
*) echo "STALE: '$st_slug' ($st_client) is no longer in the roster but its credentials still work." >&2
echo " Revoke with: $GBRAIN auth revoke-client $st_client" >&2 ;;
esac
done < "$STATE_FILE"
fi
echo
echo "Done. State: $STATE_FILE"
if [ "$new_secrets" -gt 0 ]; then
echo "$new_secrets NEW client secret(s) written to $SECRETS_OUT — deliver to each scope's sandbox, then DELETE the file."
fi
@@ -0,0 +1,10 @@
# Roster for provision-scopes.sh — one line per channel / employee.
# channel <slug>
# employee <slug> [comma-separated channels they belong to]
channel eng
channel product
employee alice-example eng,product
employee bob-example eng
employee carol-example
1 # Roster for provision-scopes.sh — one line per channel / employee.
2 # channel <slug>
3 # employee <slug> [comma-separated channels they belong to]
4 channel eng
5 channel product
6 employee alice-example eng,product
7 employee bob-example eng
8 employee carol-example
@@ -0,0 +1,19 @@
{
"id": "gbrain",
"label": "gbrain company brain",
"advertise": "gbrain",
"hints": [
"Company knowledge brain: searchable, cross-source, persistent.",
"Search it BEFORE answering questions about the org, people, projects, decisions, or history: `gbrain search \"<question>\"`.",
"Write durable knowledge with `gbrain put <slug> --content ...`, only under your own slug prefixes.",
"See the gbrain skill for slug conventions and first-run setup."
],
"auth": {
"check": "gbrain whoami",
"reauth": "gbrain init --mcp-only --force --issuer-url \"$GBRAIN_ISSUER_URL\" --mcp-url \"$GBRAIN_MCP_URL\" --oauth-client-id \"$GBRAIN_CLIENT_ID\" --oauth-client-secret \"$GBRAIN_CLIENT_SECRET\"",
"credentialPaths": [
{ "path": ".gbrain/config.json", "kind": "file" }
]
},
"install": { "binary": "gbrain" }
}
+232
View File
@@ -0,0 +1,232 @@
# qm (multi-user agent harness) — gbrain as the company brain
Connect gbrain to [qm](https://github.com/yc-software/qm) — the multiplayer
agent harness where each employee and each channel gets an isolated agent
scope — so every scope's agent can search and write one shared, indexed,
isolation-enforced company brain. The same recipe fits any harness with
per-person sandboxes that can run a CLI.
**Shape:** one central `gbrain serve --http` (OAuth 2.1) next to qm's core;
the `gbrain` binary baked into qm's sandbox image as a thin client; one OAuth
client per employee, read-fenced by source federation and write-fenced by
`bound_slug_prefixes`. Zero qm code changes — everything lives in the qm
*deployment directory*.
qm's native memory (per-scope notebook) stays as-is for fast per-turn recall.
gbrain adds what qm doesn't have: semantic + hybrid search, cross-scope
knowledge, entity graphs, and durable memory that outlives a scope.
## Topology
| gbrain concept | qm concept |
|---|---|
| one brain (one Postgres/Supabase DB) | the org |
| source `agents` (path-less, shared) | all agent-written memory |
| slug prefix `emp-<slug>/` in `agents` | an employee's personal scope |
| slug prefix `chan-<slug>/` in `agents` | a channel/room scope |
| source `org-wiki` (git-backed, read-only) | company docs |
| OAuth client `qm-emp-<slug>` | one employee's agent identity |
Isolation model:
- **Reads** are source-granular, SQL-enforced (`federated_read`): every
employee client reads `agents` + the read-only sources you grant.
- **Writes** are slug-prefix-granular, server-enforced (`bound_slug_prefixes`,
v0.42.72.0+): a client can only mutate pages under its own `emp-<slug>/`
and its channels' `chan-<x>/` prefixes — on `put_page`, `delete_page`,
`restore_page`, `add_tag`, `remove_tag`, `add_link`/`remove_link`,
`add_timeline_entry`, `revert_version` and `put_raw_data`, plus the
`POST /ingest` webhook route. Not by convention.
- **Every op that is not a plain read is denied unless allow-listed.** Ops
that write by a key other than a slug — `extract_entities` and
`extract_facts` (which mutate `people/*` and `companies/*`), `forget_fact`
(targets a fact by numeric id, across sources), `ontology_propose`, and the
`sources_admin` pair `sources_add`/`sources_remove` — cannot be fenced by
slug, so a bound client gets `permission_denied` at dispatch. The gate keys
on "not a pure read", not on a list of scope strings, so a write op added
later (or one carrying a bespoke scope) is denied until it is explicitly
fenced and added to `CLIENT_FENCED_WRITE_OPS` (`src/core/operations.ts`).
`think` is allow-listed because remote callers cannot persist from it;
`submit_agent` because it enforces this same column itself.
- **Indirect write paths are gated too, not just the ops.** `put_page`'s
facts backstop would otherwise extract entities from the page body and
write fact rows (and a `## Facts` fence on git-backed sources) onto
`people/*` pages the caller never named — the same capability
`extract_facts` is denied for, reached through an in-prefix write. It is
skipped for bound clients. `POST /ingest` is refused outright: its handler
bypasses the op layer *and* discards the source grant for untrusted
payloads, so it would write into the `default` source.
### Known limitations — read these before you rely on the fence
The write fence is a **write** boundary within a source. It is not a privacy
boundary, and it does not make every side effect prefix-clean. As of
v0.42.73.2:
- **The fence follows a delegated write.** When a client with `agent` scope
hands work to a subagent via `submit_agent`, that subagent runs under its own
slug confinement rather than the parent's OAuth binding. Both confinements are
enforced, including on the path where deduplication redirects a write onto an
existing page: the redirected target is checked against whichever confinement
the calling context actually carries, so delegation does not widen what a
client can write.
- **`add_link`/`remove_link` fence the `from` endpoint only.** A bound client
can create an edge pointing AT a page it cannot write; the edge's `context`
text surfaces in that page's backlinks and contributes to its search
ranking. Fencing `to` would break legitimate cross-referencing into
`org-wiki`, so this is deliberate — treat inbound-edge context as untrusted
content, the same way you treat page bodies.
- **Reads are source-granular, never prefix-granular.** Everyone entitled to
a source can read every prefix in it. If a scope needs genuine read
privacy, give it its own source.
- **`put_page` can create one reverse graph edge outside the fence.** If a
page body cites a code location (`src/x.ts:42`) and a code page for it
exists *in the same source*, doc↔impl reconciliation adds an edge
originating from that code page. It affects graph/backlink ranking, not
page content. Unreachable in the layout above (the `agents` source is
path-less and holds no code pages); it applies only if you point employee
writes at a code-synced source.
- **A few read ops are still brain-wide** and ignore the federated grant:
`get_recent_salience`, `find_anomalies`, `find_contradictions`, and
`sources_list`/`sources_status` (which expose source ids, paths and URLs).
A read-scoped client can learn facts derived from sources it was not
granted. Pre-existing, not introduced by the fence; if that matters for
your deployment, withhold those tools at the harness layer for now.
- **Reads touch `last_retrieved_at`** on the pages they return, including
pages in read-only sources. Freshness/usage signals are therefore
writable-by-reading; nothing else about the page is.
- **`POST /ingest` writes land in the `default` source** regardless of the
calling client's `source_id`, because the handler discards the source for
untrusted payloads. Bound clients are refused the route outright for this
reason; if you point a webhook integration at it, scope that brain's
`default` source deliberately.
- **Tradeoff to state out loud:** read isolation is per-source, so within the
shared `agents` source every employee can *read* every prefix (including
other employees' `emp-*/`). That matches qm's transparent-by-default,
everything-audited posture. If you need hard read privacy for personal
memory, give those employees their own write source instead of a prefix
(one `sources add emp-<slug>` + `--source emp-<slug>` per client) and keep
channel prefixes in `agents` via a second, channels-only client — at the
cost of two credentials in that sandbox.
## Host setup (the machine running qm's core, or any box its sandboxes can reach)
```bash
# 1. Engine: Postgres/Supabase. PGLite is single-process and cannot serve
# many concurrent sandboxes.
gbrain init --supabase --embedding-model voyage:voyage-4-large
# 2. Modes + gates (publish_* default OFF and fail as silent 403s):
gbrain config set search.mode balanced
gbrain config set mcp.publish_skills true
gbrain config set mcp.publish_advisor true
# 3. Read-only org sources + first sync:
gbrain sources add org-wiki --path ~/brains/org-wiki
gbrain sync --all # cron this
# 4. Serve over HTTP MCP (OAuth 2.1):
gbrain serve --http --bind 0.0.0.0 --port 3131 \
--public-url https://brain.acme-example.com
```
Never hand sandboxes `DATABASE_URL` — direct DB access bypasses OAuth, source
federation, and the write fence entirely.
## Provision scopes from a roster
[`qm-harness-snippets/provision-scopes.sh`](qm-harness-snippets/provision-scopes.sh)
converges the brain to a roster file
([`roster.example.tsv`](qm-harness-snippets/roster.example.tsv)):
```bash
bash provision-scopes.sh roster.tsv --read-sources org-wiki
```
- Creates the path-less `agents` source (agent-written memory needs no git
clone; if the host has `sync.repo_path` configured, pages also write
through to `.sources/agents/` for git-backed durability).
- Registers `qm-emp-<slug>` clients: `--scopes "read write"`,
`--source agents`, `--federated-read agents,org-wiki`,
`--bound-slug-prefixes emp-<slug>/,chan-<a>/,...`, per-day budget.
- **Idempotent:** re-run after every roster edit; existing clients are
`rescope-client`ed in place (channel joins/leaves update the write fence
without rotating secrets).
- New client secrets land once in `<roster>.new-credentials.tsv` — deliver
each row to its scope (qm keychain / one-time secret drop), then delete
the file.
## qm deployment directory
In the org's qm deployment repo (the directory `qm init` produced):
1. **Tool:** copy [`qm-harness-snippets/tool.json`](qm-harness-snippets/tool.json)
to `sandbox/tools/gbrain/tool.json` and drop the compiled `gbrain` binary
beside it (`bun build --compile --outfile gbrain src/cli.ts`, built for
the sandbox image's OS/arch). `auth.credentialPaths` marks
`~/.gbrain/config.json` as the scope's resident credential file;
`auth.check` wires `gbrain whoami` into qm's connector status (read-scope;
see the note below on why `remote doctor` cannot be used here).
2. **Skill:** copy [`qm-harness-snippets/SKILL.md`](qm-harness-snippets/SKILL.md)
to `sandbox/skills/gbrain/SKILL.md` (edit slug conventions to taste).
3. Ship it: `qm sandbox build && qm sandbox publish && qm up`.
Per scope, one-time (agent- or operator-run, credentials from the handoff):
```bash
gbrain init --mcp-only \
--issuer-url https://brain.acme-example.com \
--mcp-url https://brain.acme-example.com/mcp \
--oauth-client-id gbrain_cl_... --oauth-client-secret gbrain_cs_...
gbrain whoami # must succeed
```
Use `--oauth-client-secret`, not `GBRAIN_REMOTE_CLIENT_SECRET`: an env-sourced
secret is deliberately not written to `~/.gbrain/config.json`
(`src/commands/init.ts`), so with the env var alone every later command fails
once it leaves scope — and qm's `sandbox.secretEnv` is org-wide, so there is no
per-scope env to keep it in. With the flag, the credential lands in the config
file on the scope's durable disk and this runs once per scope, ever.
`gbrain remote doctor` is **not** the health check here: `run_doctor` is an
`admin`-scope op and these clients are `read write` on purpose. `gbrain whoami`
is read-scope and reports the client's identity, source, and grants.
## Verify isolation before rollout
From two differently-scoped sandboxes (or two thin-client configs):
```bash
# alice-example (bound to emp-alice-example/, chan-eng/):
gbrain put emp-alice-example/notes/test --content "mine" # OK
gbrain put chan-eng/notes/test --content "shared" # OK
gbrain put emp-bob-example/notes/test --content "not mine" # permission_denied
gbrain put chan-product/notes/test --content "not my channel" # permission_denied
gbrain search "test" # sees agents + org-wiki only
```
## Cost + operations
- `search.mode balanced` (12K token budget, relational retrieval on) is the
right default for a startup fleet; see `docs/guides/search-modes.md` for
the cost matrix before changing it.
- Budgets: `--budget-usd-per-day` is recorded on the client but only enforced
on the `submit_agent` path (`src/core/minions/budget-meter.ts`), which these
`read write` clients cannot reach — so it does **not** cap spend from
ordinary `search`/`put_page` traffic. Treat runaway-agent containment as an
open item: watch the admin SPA (`/admin`) and `gbrain search stats`, and cap
at the model/harness layer.
- Backfills on a live brain: `gbrain embed --stale --pace` (see Pace Mode in
CLAUDE.md / `docs/operations/spend-controls.md`).
## Deliberately deferred
- **qm `MemoryService` decorator** (mirror notebook captures into gbrain,
fan `recall` out and merge, `volunteer_context` push): needs a qm code
change; today's integration is agent-initiated via the CLI + skill.
- **MCP-native attach:** qm pins `strictMcpConfig` with only its in-process
server, so gbrain's MCP-discovered brain-resident skillpacks don't reach
qm agents; the sandbox skill above covers it.
- **Read-side prefix fencing** (hard privacy for `emp-*/` inside a shared
source) — tracked upstream; the roster layout is forward-compatible with
it.
+11 -3
View File
@@ -8,12 +8,18 @@
## Option 1: Local (recommended, zero server needed)
```bash
claude mcp add gbrain -- gbrain serve
claude mcp add gbrain -- gbrain serve --surface verbs
```
That's it. Claude Code spawns `gbrain serve` as a stdio subprocess. No server, no
tunnel, no token needed. Works with both PGLite and Supabase engines.
`--surface verbs` exposes the five-verb memory protocol (`recall`, `remember`,
`entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)),
the surface built for agents and quickstarts. Drop the flag for the full
operation catalog (`get_page`, `put_page`, `search`, graph ops, …) — `full` is
the default and what existing installs already run.
## Option 2: Remote, one command (fastest from a bearer token)
If GBrain is running somewhere as an HTTP server (`gbrain serve --http`, see the
@@ -79,8 +85,10 @@ You should see results from your GBrain knowledge base.
> **`list_skills` returns nothing?** Skill discovery is gated by `mcp.publish_skills`
> on the host. New brains from `gbrain init` default it ON; brains upgraded from an
> older release stay OFF until you opt in. Enable it on the host with
> `gbrain config set mcp.publish_skills true`. The core tools (search, query,
> get_page, put_page, think, find_experts) work regardless. Note: `capture` is a
> `gbrain config set mcp.publish_skills true`. Skill discovery and the core tools
> named here (search, query, get_page, put_page, think, find_experts) are
> full-surface — on `--surface verbs` the agent sees only the five memory verbs,
> and `list_skills` isn't on the surface at all. Note: `capture` is a
> CLI-only command, not an MCP tool — the agent writes over MCP with `put_page`.
## Remove
+3 -1
View File
@@ -68,4 +68,6 @@ codex mcp remove gbrain
- The token is a long-lived, full-access secret. Keep `GBRAIN_REMOTE_TOKEN` out of
version control and prefer a scoped token if your host supports one.
- Local stdio also works if you run the brain on the same machine:
`codex mcp add gbrain -- gbrain serve`.
`codex mcp add gbrain -- gbrain serve --surface verbs` — the five-verb memory
protocol ([MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)); drop the flag
for the full operation catalog.
+9 -4
View File
@@ -5,8 +5,8 @@
> dashboard at `/admin`, scoped operations, and a live SSE activity feed.
> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back
> to the `access_tokens` table and grandfathers tokens to `read+write+admin`.
> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only);
> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md)
> Both the legacy fallback and the OAuth tables work on PGLite and Postgres
> (both engine schemas carry `access_tokens`). See [SECURITY.md](../../SECURITY.md)
> for env vars and tunable defaults.
Access your brain from any device, any AI client. GBrain ships two transports:
@@ -18,11 +18,16 @@ for remote clients over OAuth 2.1.
### Local stdio (zero setup)
```bash
gbrain serve
gbrain serve # full operation catalog (default)
gbrain serve --surface verbs # just the 5 memory verbs (quickstart surface)
```
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
`--surface verbs` exposes exactly the five-verb memory protocol (`recall`,
`remember`, `entity`, `synthesize`, `forget`
[MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)) instead of the full catalog;
omit the flag (default `full`) for every operation.
### Remote over OAuth 2.1 (recommended, v0.26.0+)
@@ -250,7 +255,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
+260
View File
@@ -0,0 +1,260 @@
# MEMORY_VERBS v1 — the memory wire protocol
GBrain's frozen five-verb memory interface over MCP: `recall`, `remember`,
`entity`, `synthesize`, `forget`. The contract every harness can rely on the
way every Postgres client relies on the wire protocol — and the contract any
OTHER memory server can implement and certify against
(`gbrain protocol conformance --target <endpoint>`).
```
agent (any MCP harness)
│ remember("picked Stripe over Adyen", provenance: "chat 2026-06-11")
five verbs ── recall ── remember ── entity ── synthesize ── forget
│ self-describing envelopes: protocol_version, evidence, provenance,
│ budget meta, cost block, enumerated error codes + a populated fix
your brain (reference implementation: gbrain; any conformant server)
```
**Machine-readable spec:** `gbrain protocol --json` emits the input schemas
from the live operation definitions plus the response-shape registry — doc and
code structurally cannot drift; conformance validates live responses against
the same registry.
## Versioning policy (the point of the freeze)
- Every field NAME and its SEMANTICS in v1 are frozen forever — never removed,
renamed, or re-typed; meanings never change.
- New OPTIONAL params and new OPTIONAL response fields may be added at any
time (additive-forever). A conformant CLIENT must ignore unknown fields; a
conformant SERVER must never reject unknown-to-v1 additions it itself ships.
- `protocol_version` (integer, starts at `1`) rides every verb response and
every verb error. It increments ONLY on a breaking change, which by policy
requires a new `MEMORY_VERBS_v2` document — expected never.
- Conformance pins a minimum version; certification asserts shape, enum
validity, contract behavior, and round-trips — never ranking quality (that
is BrainBench's job).
- Enum values are part of the contract. Where an enum's DERIVATION is
implementation-defined (noted per field), implementations may improve the
derivation without a version bump; the values and their meanings stay fixed.
## Install (the 4-command quickstart)
```bash
gbrain init --pglite # 2-second local brain
claude mcp add gbrain -- gbrain serve --surface verbs # the five-verb surface
gbrain remember "I prefer dark mode in every editor" --provenance demo --entity people/me
gbrain recall --entity people/me # …now ask your agent in a NEW session
```
> Memories agents save are readable by every agent connected to this brain;
> pass `visibility: "private"` for local-CLI-only facts.
If `claude` is not found: install Claude Code first, or use a block below.
**Codex**
```bash
codex mcp add gbrain -- gbrain serve --surface verbs
```
**OpenClaw / any stdio MCP host** — register the server command
`gbrain serve --surface verbs`. Remote brains: `gbrain serve --http` on the
host, then `gbrain connect https://host/mcp --token gbrain_xxx --install` on
each client.
**Surface modes:** `--surface verbs` exposes EXACTLY the five verbs —
advertised list AND dispatch are filtered fail-closed (a hidden op returns
`unknown_tool` even when called by name). `--surface full` (the default)
exposes every operation, verbs included. Why default full: verbs is for
agents and quickstarts; full preserves existing advanced tooling. Persist a
default with `gbrain config set mcp_surface verbs`.
## The verbs
### recall(query?, entity?, budget_tokens?, since?, session_id?, limit?, …) — read
Retrieve saved facts and (with `query`) budget-packed page snippets.
- `entity` scopes the FACTS arm; `query` runs the hybrid-search arm over
pages; both present ⇒ both arms run.
- `since`: ISO 8601 date/datetime — filters the FACTS arm only in v1. (The
reference implementation also accepts relative phrases like `"8 hours ago"`
as a convenience; only ISO 8601 is part of the frozen contract.)
- `limit` is a PER-ARM cap (facts and search results each).
- `budget_tokens`: SERVER-side packing — facts pack first (limit-capped
one-liners, so search-arm starvation is bounded), search results take the
remainder. The estimator is char/4 (±1015%); `budget_used` reports packed
tokens, `dropped_count` what didn't fit. Never advisory, never client-side.
- No embedding provider configured? The search arm degrades to keyword-only
and the response notes `search_degraded` — never an error.
Response — an additive SUPERSET of the pre-v1 facts envelope on EVERY call
(all legacy fields unchanged; JSON consumers ignore additions):
| field | type | semantics |
|---|---|---|
| `protocol_version` | int | always present (every verb, every call) |
| `facts[]` | array | legacy fact fields unchanged, PLUS per fact: `fact_id` (opaque STRING — the value `forget` accepts; the legacy numeric `id` stays for pre-v1 consumers) and `provenance` (the stored source attribution) |
| `total` | int | count of facts returned |
| `results[]` | array | search arm only: `slug`, `title`, `chunk`, `evidence`, `create_safety`, `provenance` (origin page slug) |
| `search_degraded` | string? | present when keyword-only fallback fired |
| `budget_tokens` / `budget_used` / `dropped_count` | int? | present when `budget_tokens` was passed |
**evidence** (enum, zero-LLM heuristic): `alias_hit` \| `exact_title_match` \|
`high_vector_match` \| `keyword_exact` \| `weak_semantic` — why each result
matched. **create_safety** (enum): `exists` (a page for this already exists)
\| `probable` (likely exists; check before creating) \| `unknown` (no
signal). The derivation of both is implementation-defined and may improve;
the values are frozen.
### remember(fact, provenance, ttl?, entity?, kind?, visibility?) — write
Save ONE fact with mandatory attribution.
- `provenance` (REQUIRED, free text ≤500 chars, stored verbatim): e.g.
`"conversation 2026-06-12"`, `"user said in chat"`, `"import: notes.md"`.
Empty ⇒ `provenance_required` error with a fix.
- `entity`: set whenever the fact is about a specific person/company/project —
entity-scoped recall will not find unattributed facts.
- `ttl`: duration shorthand (`"30d"`, `"12h"`, `"45m"`) or an absolute ISO 8601
timestamp. ISO-8601 DURATIONS (`P30D`) are rejected with a self-correcting
suggestion. Omitted ⇒ never expires.
- `kind`: `event` \| `preference` \| `commitment` \| `belief` \| `fact`
(default).
- `visibility`: `world` (DEFAULT — readable by every agent connected to this
brain; required for the remote remember→recall round-trip) \| `private`
(local CLI reads only). The init quickstart carries the consent line.
Response: `{ id, status, status_text, entity_slug, valid_until,
protocol_version }` (+ `degraded_dedup: true` when no embedding provider —
near-duplicates may insert; dedup and supersession ride embedding similarity).
- `id` — opaque STRING (gbrain serializes integers; another implementation may
use UUIDs). On `status: "duplicate"` it is the EXISTING fact's id.
- `status``inserted` \| `duplicate` \| `superseded`. **Branch on `status`,
never on `status_text`** (the human rendering). Supersession is
implementation-defined; the reference rule: same entity + same kind +
similarity above the dedup threshold + different text = the new fact
supersedes the old ("X at acme-example" → "X left acme-example").
- Omitted optional inputs echo as `null`, never absent.
### entity(name) — read, zero LLM, p99 < 100ms
One known person/company/project card. NEVER errors on a miss.
Resolution (frozen precedence): alias > exact title > slug/slug-suffix; ties
break on most-recently-touched. Multi-hit ⇒ best match's card + runners-up in
`suggestions`. Miss ⇒ `found: false` + keyword near-misses with
`create_safety` hints.
Response: `{ protocol_version, found, latency_ms, card?, suggestions? }`.
Card: `{ entity{slug,title,type}, aka[], summary, last_touched{updated_at,
last_retrieved_at, last_timeline_date}, open_threads[], edges[],
backlink_count, active_fact_count }`.
- `summary` passes the same privacy fences as `get_page` (takes + private
facts stripped); remote callers never see private facts in the card.
- `open_threads` (best-effort in v1): active commitment-kind facts + timeline
entries from the last 90 days, capped at 3.
- `edges`: top ~10 typed edges, mentions excluded, out-edges first.
- The p99 < 100ms promise is op-layer latency (transport excluded), CI-gated
on a 20K-page corpus. 200K validation recipe below.
### synthesize(question, since?, until?) — read, EXPENSIVE
`[EXPENSIVE / SLOW — makes LLM calls, seconds-to-minutes latency, costs
money]` — the deliberately-priced slow verb. Prefer `recall`/`entity` for
lookups; use synthesize only when the answer requires combining evidence
across pages.
Response: `{ answer, sources[], gaps[], cost{model, input_tokens,
output_tokens, usd_estimate}, protocol_version }`.
- The `cost` block is a BEST-EFFORT AGGREGATE (retries/multi-call flows sum;
cache hits may undercount; token fields are `null` when a provider returns
no accounting). Honest signal, not an invoice.
- No LLM configured ⇒ the protocol error `unavailable` with a fix — never a
fake answer.
### forget(id, reason?) — write
Expire a fact by its opaque string id (from `remember` or
`recall.facts[].fact_id` — never a page slug). Idempotent: re-forgetting an
already-expired fact returns `expired: false` (success); unknown id ⇒
`not_found`. Facts are expired with an audit trail, never deleted.
Response: `{ id, expired, reason, protocol_version }`.
## Error contract (uniform across all five verbs)
```json
{ "error": "<code>", "message": "...", "suggestion": "problem + cause + fix",
"detail": "freeform specifics", "protocol_version": 1 }
```
Codes (coarse on purpose — codes are for branching; `detail` carries the
story): `invalid_params`, `provenance_required`, `not_found`, `scope_denied`,
`unavailable` (a required dependency cannot serve: no API key, gateway down,
model refusal — configure/retry, not a server bug), `budget_unsatisfiable`
(RESERVED — schema-listed, never returned in v1), `internal`.
Every verb error carries a POPULATED `suggestion`. Specific cases: `recall` on
an empty brain returns empty arrays (success, not an error); auth/scope
failures fail closed via the standard dispatch.
## Trust boundary
Verbs are ordinary operations: they inherit fail-closed `remote` semantics,
OAuth scope enforcement (`remember`/`forget` are write-scope), and per-source
isolation on every read. Remote callers see `visibility = world` facts only.
## Conformance + certification
```bash
gbrain protocol conformance # self-certify (stdio)
gbrain protocol conformance --target http://localhost:3131/mcp --token gbrain_xxx
gbrain protocol conformance --target "bun run src/cli.ts serve"
gbrain protocol conformance --synthesize # also live-call synthesize
```
Pass criteria: response SHAPE (required fields, enum validity), CONTRACT
BEHAVIOR (provenance rejected when empty; budget arithmetic consistent;
entity miss ⇒ `found:false`, not an error; private facts absent from remote
cards; idempotent forget), and ROUND-TRIP (remember → recall by entity — a
plain indexed read, deterministic). It does NOT judge ranking quality.
Entity-card cases need a seedable page (`put_page`); against verbs-only
targets they skip honestly. `--synthesize` is cost-gated: with no LLM key it
asserts the clean `unavailable` error (what CI does); with a key it spends
real tokens.
Conformance is a LIVE test that WRITES: it seeds a marker-suffixed synthetic
entity page (`people/conformance-<marker>`, when the target exposes
`put_page`) and writes/expires facts through `remember`/`forget`. Point it at
write-capable credentials and a brain you're comfortable leaving those
synthetic artifacts in — they're marker-named for easy cleanup, not
auto-deleted. The fixture set ships as data
(`test/fixtures/memory-verbs/cases.json`) and seeds BrainBench's
protocol-compliance arm. gbrain's CI certifies its own stdio + HTTP
transports; external certification is best-effort tooling until a second
implementation exists.
## Observability (local only)
Every verb call appends one line to
`~/.gbrain/integrations/memory-verbs/usage.jsonl` — **local JSONL only, never
uploaded**, stats-only (lock-free rotation may drop lines; POSIX O_APPEND
line-atomic, best-effort on Windows). `gbrain protocol stats [--days N]`
aggregates per-verb calls, error rate, latency, budget drops, entity hit rate,
and the measured TTHW (install → first verb call, from the
`protocol_installed_at` stamp). `gbrain doctor` carries a
`memory_verbs_usage` health line.
## 200K-page latency validation (manual recipe)
CI gates entity() p99 < 100ms on a 20K-page corpus
(`test/entity-card-perf.slow.test.ts`). To validate at 200K, edit the
constants at the top of that file (`PAGES = 200_000`, `LINKS = 1_000_000`,
`ALIASES = 300_000`, `FACTS = 400_000`) and run
`bun test test/entity-card-perf.slow.test.ts --timeout=1800000` — seeding
dominates (~minutes); the measured calls report p50/p99 + the ratio guard.
+12
View File
@@ -51,6 +51,18 @@ When storage configuration is present, `gbrain sync` automatically manages `.git
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
- Warns when a configured collector's declared output dir (recipe `output_paths`
frontmatter) sits inside a `db_only` path: gitignored files never appear in the
git-walking sync diff, and `gbrain import` honors `.gitignore` too — the
collector would run green while nothing reaches the DB. The
`db_only_collector_collision` doctor check surfaces the same trap.
Related doctor coverage: `undeclared_db_only_pages` warns about DB pages with no
backing file that sit outside every declared `db_only` path. The engine's own
derive-phase output prefixes (`life/events/`, `atoms/`, `extracts/`,
`dream-cycle-summaries/`) count as implicitly declared for that check, so healthy
brains stay quiet without adding them to `gbrain.yml`. They are NOT auto-added to
`.gitignore` — only explicitly declared `db_only` dirs are.
Example `.gitignore` addition:
+1 -1
View File
@@ -13,7 +13,7 @@ Step-by-step walkthroughs that take you from zero to a working outcome. Concrete
These are the next tutorials on the roadmap. Open an issue if one of them is the one you need most; that's how we'll prioritize.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find_trajectory`, and `gbrain founder scorecard` on real workflows.
- **Set up GBrain for VC dealflow** — the operator's recipe. People pages for founders, companies with typed Facts fence carrying ARR / team-size / runway across dates, meetings auto-ingested, deal pages linking everything. Shows `gbrain whoknows`, `gbrain find-trajectory`, and `gbrain founder scorecard` on real workflows.
- **Migrate your existing vault into GBrain** — for Notion / Obsidian / Roam users with a vault that doesn't match GBrain's default layout. Walks through `gbrain schema detect``suggest``review-candidates` so the brain learns your shape instead of forcing you to learn its.
+3 -3
View File
@@ -93,7 +93,7 @@ There are two ways to scope teammates' access. They suit different deployment sh
**Model A: separate sources with OAuth scoping (recommended for true multi-user with different AI clients).** What this tutorial walks you through. Each teammate gets their own OAuth client, which carries `--source` + `--federated-read` flags. The brain refuses cross-source reads at the SQL layer; isolation is database-enforced. Each teammate can run their own MCP-aware client (Claude Code, Cursor, their own OpenClaw, etc.) and the scoping holds.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. There's no OAuth-enforced isolation; the agent itself enforces "Alice's writes go to her partners/ subdir." This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth, but the scoping is convention-only.
**Model B: one source, directory-based per-person scoping (simpler for one-agent-serves-everyone setups).** The shape I actually run in production: a single source called `default`, with a `partners/<slug>/` convention inside it (e.g. `partners/alice-example/`, `partners/bob-example/`). Each partner gets their own subdirectory holding their personal pages: `partners/alice-example/USER.md`, `partners/alice-example/concepts/`, `partners/alice-example/sources/`, etc. This is the right model when ONE agent (yours) serves everyone over Telegram or a single shared interface. It's simpler ops, no per-user OAuth. **Write scoping within the shared source can be server-enforced:** register each per-person client with `--bound-slug-prefixes partners/alice-example/` and every slug-mutating write outside that prefix is rejected with `permission_denied` (v0.42.72.0+). Without the binding, the scoping is convention-only (the agent polices itself). Read scoping stays source-granular in both models — within a shared source, everyone entitled to the source can read every folder.
For most company-brain installs (10+ teammates each with their own AI client), Model A is the right starting point. If you're running the fat-agent-serves-everyone pattern from the personal-brain tutorial, Model B is genuinely simpler. You can also mix: separate sources for the obviously-different ones (customer notes vs internal-only) AND a `partners/<slug>/` convention inside the shared source for per-person workspace.
@@ -210,7 +210,7 @@ Each `register-client` command prints a `client_id` and a `client_secret`. Save
A note on the flags:
- `--scopes read,write` lets the client query the brain and write new pages. You can omit `write` for read-only clients (executive summaries, dashboards). The `admin` scope is needed for operational commands like `gbrain remote doctor` and is usually reserved for your own admin client.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder.
- `--source` controls write authority. A client can only write to one source. Within that source, your folder convention from Part 3 keeps each person's writes in their own subfolder — and you can make that server-enforced with `--bound-slug-prefixes alice-example/` (v0.42.72.0+): every slug-mutating write op (put_page, delete_page, tags, links, timeline, revert, raw data) outside the bound prefixes is rejected with `permission_denied`. Update the binding later with `gbrain auth rescope-client <id> --bound-slug-prefixes <p1,p2|none>`. **Adding a binding to an existing client narrows it in ways you should expect:** ops that write by something other than a slug (`extract_entities`, `extract_facts`, `forget_fact`, `ontology_propose`, `sources_add`/`sources_remove`) and `POST /ingest` become unavailable to that client, and `put_page`'s automatic fact extraction is skipped — all because none of them can be confined to a prefix. Reads are unaffected. See [the qm-harness guide](../integrations/qm-harness.md) for the full model.
- `--federated-read` controls read scope. A client can read from one or more sources.
### Verify the scoping actually scopes
@@ -554,7 +554,7 @@ What to do next:
- **Wire ingestion** from external systems (Granola, Linear, Slack) using the [ingestion source contract](../skillpack-anatomy.md). Most companies want their meetings auto-ingested so the brain stays current without anyone typing notes.
- **Set up team-specific dashboards** through the admin UI. Each team lead can have their own view of brain health and activity.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find_trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
- **Explore the rest of the brain layer.** `gbrain whoknows` (find the expert on a topic), `gbrain find-trajectory` (how a metric changed over time), `gbrain founder scorecard` (especially useful for VC and ops teams), the contradiction-detection cycle that surfaces conflicts between different people's notes.
If you're building in this space (which YC has flagged as the [company-brain category in its Request for Startups](https://www.ycombinator.com/rfs#company-brain)), you might as well build on this. Everything described above is open source, MIT licensed, and what I run in production behind my own AI agents.
+28 -14
View File
@@ -142,15 +142,23 @@ generate while working, and is genuinely useful by day two.
```bash
# Claude Code
claude mcp add gbrain -- gbrain serve
claude mcp add gbrain -- gbrain serve --surface verbs
# Codex
codex mcp add gbrain -- gbrain serve
codex mcp add gbrain -- gbrain serve --surface verbs
```
That's the whole wire-up. No token, no URL, no tunnel. The agent spawns
`gbrain serve` as a stdio subprocess and talks to your local brain directly.
`--surface verbs` exposes exactly the five-verb memory protocol
(`recall`, `remember`, `entity`, `synthesize`, `forget`
[MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md), frozen + additive-forever)
instead of the full operation catalog, so the agent sees a tight, stable surface
instead of a 110-tool wall. Drop the flag (or pass `--surface full`) for every
operation. The default when the flag is omitted is `full`, so existing wire-ups
are unchanged.
### B4. Verify
In the agent: *"search my brain for PGLite"* (or whatever you just captured). You
@@ -175,14 +183,19 @@ on the patterns.
You have a knowledge brain connected over MCP. Before answering any question
about people, companies, decisions, projects, or past context:
1. **Search first.** Call `search` (or `query` for a synthesized answer) against
the brain BEFORE answering from memory or asking me. If the brain has the
answer, use it. Never ask "who is X?" or "what did we decide about Y?" before
searching — the brain probably already knows.
1. **Brain first — route by the shape of the question.** Exact names or known
tokens → `search` (cheap hybrid, no expansion). Concept, landscape, or
"all the X that do Y" questions → `query` FIRST — it recovers synonym
phrasings `search` misses, and a populated `search` result set is not proof
of coverage. On the five-verb surface the same split is `recall` (retrieve)
vs `synthesize` (reasoned answer). Check the brain BEFORE answering from
memory or asking me. Never ask "who is X?" or "what did we decide about Y?"
before checking — the brain probably already knows.
2. **Write back.** When I make a decision, mention a new person/company, or land
on an idea worth keeping, write it to the brain with `put_page` (entity pages
under people/, companies/; decisions under decisions/ or notes/). One insight,
one page, linked.
on an idea worth keeping, write it to the brain: `remember` on the five-verb
surface (one fact, with provenance), or `put_page` on the full surface
(entity pages under people/, companies/; decisions under decisions/ or
notes/). One insight, one page, linked.
3. **Cite.** When you answer from the brain, name the page you used.
```
@@ -204,13 +217,14 @@ hundreds of linked pages and patterns you didn't know were there.
**3. Briefing from your brain (not from the internet).** *"What do I need to know
before my 2pm with the Acme team?"* pulls your meeting history, the people,
what's still open, what the brain doesn't know yet. The agent does your prep
because it read your context. (`query` gives you the synthesized answer with
citations; this is the example on the [README](../../README.md).)
because it read your context. (`query` `synthesize` on the five-verb surface —
gives you the synthesized answer with citations; this is the example on the
[README](../../README.md).)
**4. whoknows (expertise routing).** *"Who do I know who's shipped a rate
limiter in Postgres?"* The `find_experts` tool ranks people in your brain by
relevance + recency. Useful the moment your brain has more than a handful of
people in it.
limiter in Postgres?"* The `find_experts` tool (full surface) ranks people in
your brain by relevance + recency. Useful the moment your brain has more than a
handful of people in it.
That's the spine of it. Two commands to connect, one protocol to paste, four
habits to build. Your agent stops being amnesiac.
+9 -9
View File
@@ -115,21 +115,21 @@ You can use the same keys across multiple agents.
## Step 6: Install GBrain
Once OpenClaw is running:
Once OpenClaw is running, installation is two commands — one in the brain repo, one in the agent workspace:
```bash
gbrain install
# In the BRAIN repo (the git repo that holds your markdown pages):
gbrain init --supabase
# In the AGENT WORKSPACE repo (where OpenClaw runs):
gbrain skillpack scaffold --all
```
This installs:
`gbrain init --supabase` walks a short wizard that asks for your Supabase connection string and creates the schema. You'll get that connection string in Step 7 — read 7a and 7b first so you paste the right one (the transaction pooler, not the direct connection). If you'd rather try things locally before paying for a database, `gbrain init --pglite` gives you a zero-config embedded engine instead; you can migrate to Supabase later with `gbrain migrate --to supabase`.
- About 60 skills
- About 9 skill packs
- Default brain structure
- MCP server configuration
- Supabase connection (for embeddings and search)
`gbrain skillpack scaffold --all` copies the ~43 bundled skills into your agent workspace as first-class files you can edit freely. (The old managed-install model was retired in v0.36.0.0; see `docs/INSTALL.md` if you're upgrading from an older release.)
GBrain populates the brain repo with its default directory structure, skill files, and configuration. From this point, the agent has working memory and access to every skill.
From this point, the agent has working memory and access to every skill.
---
+3 -3
View File
@@ -32,7 +32,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -62,7 +62,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -143,7 +143,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
+352 -17
View File
@@ -187,7 +187,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~110 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`; v0.43.0.0 adds the five frozen MEMORY_VERBS — `recall`, `remember`, `entity`, `synthesize`, `forget` — servable alone via `gbrain serve --surface verbs`, see `docs/protocol/MEMORY_VERBS_v1.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -268,6 +268,7 @@ detail on demand.)
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| memory verbs / MCP tool surface (`--surface`) / conformance | `docs/protocol/MEMORY_VERBS_v1.md` + the `verbs*`/`surface.ts`/`protocol.ts` entries in `KEY_FILES.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
| running or writing tests | `docs/TESTING.md` |
| bulk-command progress wiring | `docs/progress-events.md` |
@@ -421,7 +422,7 @@ audit trail lives in the source repo's git history.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 30 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 52 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
@@ -1605,7 +1606,7 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
The agent installs GBrain, creates the brain, asks for your API keys, loads 43 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
The agent installs GBrain, creates the brain, asks for your API keys, loads 52 skills, configures the dream cycle, and verifies the install end-to-end. ~30 minutes. You answer questions, it does the work.
> **Never set up an AI agent platform before?** The [personal-brain tutorial](docs/tutorials/personal-brain.md) walks the whole path end-to-end — picking OpenClaw vs Hermes, deploying it, pointing it at INSTALL_FOR_AGENTS.md, getting the API keys, and verifying the first query. Start there if any of the above is new.
@@ -1613,13 +1614,15 @@ The agent installs GBrain, creates the brain, asks for your API keys, loads 43 s
Already running Claude Code or Codex? There are two ways to wire GBrain in, depending on what you want.
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel:
**Just want a memory for your coding agent (recommended starting point).** Spin up a local brain and connect it in two commands — zero server, zero token, zero tunnel. `--surface verbs` gives your agent the five-verb memory protocol (`recall`, `remember`, `entity`, `synthesize`, `forget` — [MEMORY_VERBS v1](docs/protocol/MEMORY_VERBS_v1.md), frozen + additive-forever) instead of the full 110-tool wall; drop the flag for every operation:
```bash
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve # or: codex mcp add gbrain -- gbrain serve
gbrain init --pglite # 2-second local brain (no Docker)
claude mcp add gbrain -- gbrain serve --surface verbs # or: codex mcp add gbrain -- gbrain serve --surface verbs
```
If `claude` is not found, install Claude Code first — or use the per-harness blocks in the [protocol doc](docs/protocol/MEMORY_VERBS_v1.md). Heads-up: memories agents save default to brain-wide visibility (every connected agent can recall them); pass `visibility: "private"` for local-only facts.
**Already have a brain on a remote host** (OpenClaw, Hermes, or any `gbrain serve --http`)? Point your laptop agents at it with one command each — `--install` wires it up and smoke-tests the token before handoff:
```bash
@@ -1631,7 +1634,7 @@ gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex --install
### Install the full autonomous setup into your existing agent
Want the whole thing — local brain, 43 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
Want the whole thing — local brain, 52 skills, the overnight dream cycle that enriches while you sleep? Paste this into Codex, Claude Code, Cursor, or another coding agent:
```
Retrieve and follow the instructions at:
@@ -1654,7 +1657,7 @@ Postgres-at-scale, Supabase, and thin-client setup paths live in [`docs/INSTALL.
### Connect GBrain to your AI client (MCP)
GBrain exposes 30+ tools over MCP (stdio and HTTP). The specific snippet depends on which client you use:
GBrain exposes 110 tools over MCP (stdio and HTTP) — or exactly the five memory verbs with `--surface verbs`. The specific snippet depends on which client you use:
- **[Claude Code](docs/mcp/CLAUDE_CODE.md)** — local: one command, `claude mcp add gbrain -- gbrain serve` (zero server, zero tunnel). Remote with just a bearer token: `gbrain connect https://your-host/mcp --token gbrain_xxx` prints a paste-ready block (or `--install` wires it up and smoke-tests the token).
- **[Codex](docs/mcp/CODEX.md)** — `gbrain connect https://your-host/mcp --token gbrain_xxx --agent codex` (or `--install`). Codex reads the bearer from `$GBRAIN_REMOTE_TOKEN` at runtime, so the token never lands in Codex config.
@@ -1730,7 +1733,7 @@ Most personal-knowledge tools force one fixed layout: their idea of "notes" + "p
**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit:
- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`.
- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2","apply":true}'` (omit `"apply":true` for a dry-run preview — that is the default).
- **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`.
- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md).
@@ -1831,7 +1834,7 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h
## Troubleshooting
**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe).
**`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model <provider>:<model>` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing.
@@ -2216,6 +2219,66 @@ live in `test/postgres-engine-rls-scope.test.ts`.
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
### Troubleshooting: startup abort (`RuntimeError: Aborted()`)
**Symptom:** every PGLite-touching command dies at startup with
`PGLite failed to initialize its WASM runtime … Aborted(). Build with
-sASSERTIONS for more info.` — commonly first seen right after a macOS
upgrade.
**Real root cause:** corrupt WAL/checkpoint state in the data dir after an
unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the
write-ahead log; every subsequent open fails WAL replay inside WASM and
Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM
incompatibility — the same signature reproduces across macOS versions and on
Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun
version bump changes it.
**Recovery ladder** (top rung first):
1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort,
backs up `pg_wal/` + `pg_control` into a sibling
`<dataDir>.wal-repair-backup-<ts>/` dir, resets the WAL in place
(pg_resetwal semantics — data files preserved; transactions not
checkpointed before the corruption may be lost), and retries once. On
success it prints a loud stderr notice naming the backup and recommending
`gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired
data-dir lock (never after reaping another process's lock), skips for a
cooldown window after a failed attempt
(`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one
backup per corruption episode (newest 3 episodes retained), and restores
the original files if the retry still fails. Kill-switch:
`GBRAIN_PGLITE_WAL_REPAIR=off`.
2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir
(read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset
deliberately. Refuses when another gbrain process holds the brain (a live
`gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`.
3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default
from your config) wipes and re-creates the brain from your brain repo, or
manually: back up `~/.gbrain`, move `brain.pglite` aside,
`gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`.
Required for *catalog* corruption (58P01 / pgvector load failure) — WAL
repair cannot fix that class.
4. **Switch engines.** `gbrain init --supabase`, or native Postgres +
pgvector (recipe below, contributed by @roysaurav):
```bash
brew install postgresql@17
brew services start postgresql@17
createdb gbrain
cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector && make && make install
psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;"
# ~/.gbrain/config.json: { "engine": "postgres",
# "database_url": "postgresql://localhost:5432/gbrain" }
gbrain apply-migrations --yes && gbrain doctor
```
`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails
to connect: it diagnoses the dir from disk, names the repair command, reports
retained repair backups, and escalates when repairs keep recurring (that
means the unclean-shutdown genesis is still active — see the ladder's rung 4).
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
@@ -2346,7 +2409,7 @@ gbrain schema sync --apply
The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now:
- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text.
- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The `extract_facts` cycle runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`.
- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files.
One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did.
@@ -2376,7 +2439,7 @@ gbrain schema add-link-type led-by --page-type deal --target-type inves
gbrain schema sync --apply
```
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." The `extract_facts` cycle starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`.
The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them.
@@ -2457,7 +2520,7 @@ Re-run the same `whoknows` query. Top-3 should shift, because the new type is no
Three things gbrain does that generic note systems can't:
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. The `extract_facts` cycle only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached.
**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion.
@@ -3682,8 +3745,8 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY
> dashboard at `/admin`, scoped operations, and a live SSE activity feed.
> Pre-v0.26 legacy bearer tokens still work — `verifyAccessToken` falls back
> to the `access_tokens` table and grandfathers tokens to `read+write+admin`.
> Postgres-only for the legacy fallback (the `access_tokens` table is Postgres-only);
> OAuth tables work on both PGLite and Postgres. See [SECURITY.md](../../SECURITY.md)
> Both the legacy fallback and the OAuth tables work on PGLite and Postgres
> (both engine schemas carry `access_tokens`). See [SECURITY.md](../../SECURITY.md)
> for env vars and tunable defaults.
Access your brain from any device, any AI client. GBrain ships two transports:
@@ -3695,11 +3758,16 @@ for remote clients over OAuth 2.1.
### Local stdio (zero setup)
```bash
gbrain serve
gbrain serve # full operation catalog (default)
gbrain serve --surface verbs # just the 5 memory verbs (quickstart surface)
```
Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio.
No server, no tunnel, no token needed. Works on both PGLite and Postgres engines.
`--surface verbs` exposes exactly the five-verb memory protocol (`recall`,
`remember`, `entity`, `synthesize`, `forget` —
[MEMORY_VERBS v1](../protocol/MEMORY_VERBS_v1.md)) instead of the full catalog;
omit the flag (default `full`) for every operation.
### Remote over OAuth 2.1 (recommended, v0.26.0+)
@@ -3927,7 +3995,7 @@ All 30 GBrain operations are available remotely, including `sync_brain` and
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
CLI callers (`gbrain files upload ...`) keep unrestricted filesystem access since
the user owns the machine.
## Deployment Options
@@ -4006,6 +4074,273 @@ built-in server is the recommended path.
---
## docs/protocol/MEMORY_VERBS_v1.md
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/protocol/MEMORY_VERBS_v1.md
# MEMORY_VERBS v1 — the memory wire protocol
GBrain's frozen five-verb memory interface over MCP: `recall`, `remember`,
`entity`, `synthesize`, `forget`. The contract every harness can rely on the
way every Postgres client relies on the wire protocol — and the contract any
OTHER memory server can implement and certify against
(`gbrain protocol conformance --target <endpoint>`).
```
agent (any MCP harness)
│ remember("picked Stripe over Adyen", provenance: "chat 2026-06-11")
five verbs ── recall ── remember ── entity ── synthesize ── forget
│ self-describing envelopes: protocol_version, evidence, provenance,
│ budget meta, cost block, enumerated error codes + a populated fix
your brain (reference implementation: gbrain; any conformant server)
```
**Machine-readable spec:** `gbrain protocol --json` emits the input schemas
from the live operation definitions plus the response-shape registry — doc and
code structurally cannot drift; conformance validates live responses against
the same registry.
## Versioning policy (the point of the freeze)
- Every field NAME and its SEMANTICS in v1 are frozen forever — never removed,
renamed, or re-typed; meanings never change.
- New OPTIONAL params and new OPTIONAL response fields may be added at any
time (additive-forever). A conformant CLIENT must ignore unknown fields; a
conformant SERVER must never reject unknown-to-v1 additions it itself ships.
- `protocol_version` (integer, starts at `1`) rides every verb response and
every verb error. It increments ONLY on a breaking change, which by policy
requires a new `MEMORY_VERBS_v2` document — expected never.
- Conformance pins a minimum version; certification asserts shape, enum
validity, contract behavior, and round-trips — never ranking quality (that
is BrainBench's job).
- Enum values are part of the contract. Where an enum's DERIVATION is
implementation-defined (noted per field), implementations may improve the
derivation without a version bump; the values and their meanings stay fixed.
## Install (the 4-command quickstart)
```bash
gbrain init --pglite # 2-second local brain
claude mcp add gbrain -- gbrain serve --surface verbs # the five-verb surface
gbrain remember "I prefer dark mode in every editor" --provenance demo --entity people/me
gbrain recall --entity people/me # …now ask your agent in a NEW session
```
> Memories agents save are readable by every agent connected to this brain;
> pass `visibility: "private"` for local-CLI-only facts.
If `claude` is not found: install Claude Code first, or use a block below.
**Codex**
```bash
codex mcp add gbrain -- gbrain serve --surface verbs
```
**OpenClaw / any stdio MCP host** — register the server command
`gbrain serve --surface verbs`. Remote brains: `gbrain serve --http` on the
host, then `gbrain connect https://host/mcp --token gbrain_xxx --install` on
each client.
**Surface modes:** `--surface verbs` exposes EXACTLY the five verbs —
advertised list AND dispatch are filtered fail-closed (a hidden op returns
`unknown_tool` even when called by name). `--surface full` (the default)
exposes every operation, verbs included. Why default full: verbs is for
agents and quickstarts; full preserves existing advanced tooling. Persist a
default with `gbrain config set mcp_surface verbs`.
## The verbs
### recall(query?, entity?, budget_tokens?, since?, session_id?, limit?, …) — read
Retrieve saved facts and (with `query`) budget-packed page snippets.
- `entity` scopes the FACTS arm; `query` runs the hybrid-search arm over
pages; both present ⇒ both arms run.
- `since`: ISO 8601 date/datetime — filters the FACTS arm only in v1. (The
reference implementation also accepts relative phrases like `"8 hours ago"`
as a convenience; only ISO 8601 is part of the frozen contract.)
- `limit` is a PER-ARM cap (facts and search results each).
- `budget_tokens`: SERVER-side packing — facts pack first (limit-capped
one-liners, so search-arm starvation is bounded), search results take the
remainder. The estimator is char/4 (±1015%); `budget_used` reports packed
tokens, `dropped_count` what didn't fit. Never advisory, never client-side.
- No embedding provider configured? The search arm degrades to keyword-only
and the response notes `search_degraded` — never an error.
Response — an additive SUPERSET of the pre-v1 facts envelope on EVERY call
(all legacy fields unchanged; JSON consumers ignore additions):
| field | type | semantics |
|---|---|---|
| `protocol_version` | int | always present (every verb, every call) |
| `facts[]` | array | legacy fact fields unchanged, PLUS per fact: `fact_id` (opaque STRING — the value `forget` accepts; the legacy numeric `id` stays for pre-v1 consumers) and `provenance` (the stored source attribution) |
| `total` | int | count of facts returned |
| `results[]` | array | search arm only: `slug`, `title`, `chunk`, `evidence`, `create_safety`, `provenance` (origin page slug) |
| `search_degraded` | string? | present when keyword-only fallback fired |
| `budget_tokens` / `budget_used` / `dropped_count` | int? | present when `budget_tokens` was passed |
**evidence** (enum, zero-LLM heuristic): `alias_hit` \| `exact_title_match` \|
`high_vector_match` \| `keyword_exact` \| `weak_semantic` — why each result
matched. **create_safety** (enum): `exists` (a page for this already exists)
\| `probable` (likely exists; check before creating) \| `unknown` (no
signal). The derivation of both is implementation-defined and may improve;
the values are frozen.
### remember(fact, provenance, ttl?, entity?, kind?, visibility?) — write
Save ONE fact with mandatory attribution.
- `provenance` (REQUIRED, free text ≤500 chars, stored verbatim): e.g.
`"conversation 2026-06-12"`, `"user said in chat"`, `"import: notes.md"`.
Empty ⇒ `provenance_required` error with a fix.
- `entity`: set whenever the fact is about a specific person/company/project —
entity-scoped recall will not find unattributed facts.
- `ttl`: duration shorthand (`"30d"`, `"12h"`, `"45m"`) or an absolute ISO 8601
timestamp. ISO-8601 DURATIONS (`P30D`) are rejected with a self-correcting
suggestion. Omitted ⇒ never expires.
- `kind`: `event` \| `preference` \| `commitment` \| `belief` \| `fact`
(default).
- `visibility`: `world` (DEFAULT — readable by every agent connected to this
brain; required for the remote remember→recall round-trip) \| `private`
(local CLI reads only). The init quickstart carries the consent line.
Response: `{ id, status, status_text, entity_slug, valid_until,
protocol_version }` (+ `degraded_dedup: true` when no embedding provider —
near-duplicates may insert; dedup and supersession ride embedding similarity).
- `id` — opaque STRING (gbrain serializes integers; another implementation may
use UUIDs). On `status: "duplicate"` it is the EXISTING fact's id.
- `status` — `inserted` \| `duplicate` \| `superseded`. **Branch on `status`,
never on `status_text`** (the human rendering). Supersession is
implementation-defined; the reference rule: same entity + same kind +
similarity above the dedup threshold + different text = the new fact
supersedes the old ("X at acme-example" → "X left acme-example").
- Omitted optional inputs echo as `null`, never absent.
### entity(name) — read, zero LLM, p99 < 100ms
One known person/company/project card. NEVER errors on a miss.
Resolution (frozen precedence): alias > exact title > slug/slug-suffix; ties
break on most-recently-touched. Multi-hit ⇒ best match's card + runners-up in
`suggestions`. Miss ⇒ `found: false` + keyword near-misses with
`create_safety` hints.
Response: `{ protocol_version, found, latency_ms, card?, suggestions? }`.
Card: `{ entity{slug,title,type}, aka[], summary, last_touched{updated_at,
last_retrieved_at, last_timeline_date}, open_threads[], edges[],
backlink_count, active_fact_count }`.
- `summary` passes the same privacy fences as `get_page` (takes + private
facts stripped); remote callers never see private facts in the card.
- `open_threads` (best-effort in v1): active commitment-kind facts + timeline
entries from the last 90 days, capped at 3.
- `edges`: top ~10 typed edges, mentions excluded, out-edges first.
- The p99 < 100ms promise is op-layer latency (transport excluded), CI-gated
on a 20K-page corpus. 200K validation recipe below.
### synthesize(question, since?, until?) — read, EXPENSIVE
`[EXPENSIVE / SLOW — makes LLM calls, seconds-to-minutes latency, costs
money]` — the deliberately-priced slow verb. Prefer `recall`/`entity` for
lookups; use synthesize only when the answer requires combining evidence
across pages.
Response: `{ answer, sources[], gaps[], cost{model, input_tokens,
output_tokens, usd_estimate}, protocol_version }`.
- The `cost` block is a BEST-EFFORT AGGREGATE (retries/multi-call flows sum;
cache hits may undercount; token fields are `null` when a provider returns
no accounting). Honest signal, not an invoice.
- No LLM configured ⇒ the protocol error `unavailable` with a fix — never a
fake answer.
### forget(id, reason?) — write
Expire a fact by its opaque string id (from `remember` or
`recall.facts[].fact_id` — never a page slug). Idempotent: re-forgetting an
already-expired fact returns `expired: false` (success); unknown id ⇒
`not_found`. Facts are expired with an audit trail, never deleted.
Response: `{ id, expired, reason, protocol_version }`.
## Error contract (uniform across all five verbs)
```json
{ "error": "<code>", "message": "...", "suggestion": "problem + cause + fix",
"detail": "freeform specifics", "protocol_version": 1 }
```
Codes (coarse on purpose — codes are for branching; `detail` carries the
story): `invalid_params`, `provenance_required`, `not_found`, `scope_denied`,
`unavailable` (a required dependency cannot serve: no API key, gateway down,
model refusal — configure/retry, not a server bug), `budget_unsatisfiable`
(RESERVED — schema-listed, never returned in v1), `internal`.
Every verb error carries a POPULATED `suggestion`. Specific cases: `recall` on
an empty brain returns empty arrays (success, not an error); auth/scope
failures fail closed via the standard dispatch.
## Trust boundary
Verbs are ordinary operations: they inherit fail-closed `remote` semantics,
OAuth scope enforcement (`remember`/`forget` are write-scope), and per-source
isolation on every read. Remote callers see `visibility = world` facts only.
## Conformance + certification
```bash
gbrain protocol conformance # self-certify (stdio)
gbrain protocol conformance --target http://localhost:3131/mcp --token gbrain_xxx
gbrain protocol conformance --target "bun run src/cli.ts serve"
gbrain protocol conformance --synthesize # also live-call synthesize
```
Pass criteria: response SHAPE (required fields, enum validity), CONTRACT
BEHAVIOR (provenance rejected when empty; budget arithmetic consistent;
entity miss ⇒ `found:false`, not an error; private facts absent from remote
cards; idempotent forget), and ROUND-TRIP (remember → recall by entity — a
plain indexed read, deterministic). It does NOT judge ranking quality.
Entity-card cases need a seedable page (`put_page`); against verbs-only
targets they skip honestly. `--synthesize` is cost-gated: with no LLM key it
asserts the clean `unavailable` error (what CI does); with a key it spends
real tokens.
Conformance is a LIVE test that WRITES: it seeds a marker-suffixed synthetic
entity page (`people/conformance-<marker>`, when the target exposes
`put_page`) and writes/expires facts through `remember`/`forget`. Point it at
write-capable credentials and a brain you're comfortable leaving those
synthetic artifacts in — they're marker-named for easy cleanup, not
auto-deleted. The fixture set ships as data
(`test/fixtures/memory-verbs/cases.json`) and seeds BrainBench's
protocol-compliance arm. gbrain's CI certifies its own stdio + HTTP
transports; external certification is best-effort tooling until a second
implementation exists.
## Observability (local only)
Every verb call appends one line to
`~/.gbrain/integrations/memory-verbs/usage.jsonl` — **local JSONL only, never
uploaded**, stats-only (lock-free rotation may drop lines; POSIX O_APPEND
line-atomic, best-effort on Windows). `gbrain protocol stats [--days N]`
aggregates per-verb calls, error rate, latency, budget drops, entity hit rate,
and the measured TTHW (install → first verb call, from the
`protocol_installed_at` stamp). `gbrain doctor` carries a
`memory_verbs_usage` health line.
## 200K-page latency validation (manual recipe)
CI gates entity() p99 < 100ms on a 20K-page corpus
(`test/entity-card-perf.slow.test.ts`). To validate at 200K, edit the
constants at the top of that file (`PAGES = 200_000`, `LINKS = 1_000_000`,
`ALIASES = 300_000`, `FACTS = 400_000`) and run
`bun test test/entity-card-perf.slow.test.ts --timeout=1800000` — seeding
dominates (~minutes); the measured calls report p50/p99 + the ratio guard.
---
# AI providers
# Debugging
+1
View File
@@ -27,6 +27,7 @@ Repo: https://github.com/garrytan/gbrain
- [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.
- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
- [docs/protocol/MEMORY_VERBS_v1.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/protocol/MEMORY_VERBS_v1.md): The frozen five-verb memory protocol (recall/remember/entity/synthesize/forget): response envelopes, error contract, additive-forever versioning, surface modes, conformance certification, per-harness installs.
## AI providers
+8 -6
View File
@@ -37,6 +37,7 @@
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"build:flag-registry": "bun run scripts/generate-flag-registry.ts",
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
@@ -75,6 +76,7 @@
"check:privacy": "bash scripts/check-privacy.sh",
"check:proposal-pii": "bash scripts/check-proposal-pii.sh",
"check:eval-glossary": "bash scripts/check-eval-glossary-fresh.sh",
"check:skills-manifest": "bash scripts/check-skills-manifest-fresh.sh",
"check:test-names": "bash scripts/check-test-real-names.sh",
"check:progress": "bash scripts/check-progress-to-stdout.sh",
"check:no-tracked-symlinks": "bash scripts/check-no-tracked-symlinks.sh",
@@ -121,7 +123,7 @@
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.15.0",
"js-yaml": "^3.15.1",
"marked": "^18.0.2",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
@@ -147,17 +149,17 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.68.1",
"version": "0.43.0.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.4",
"fast-uri": "^3.1.5",
"body-parser": "^2.3.0",
"fast-xml-builder": "^1.1.7",
"fast-xml-parser": "^5.7.0",
"form-data": "^4.0.6",
"hono": "^4.12.25",
"ip-address": "^10.1.1",
"hono": "^4.12.34",
"ip-address": "^10.3.1",
"qs": "^6.15.2",
"js-yaml": "^3.15.0"
"js-yaml": "^3.15.1"
}
}
+4 -2
View File
@@ -1,7 +1,7 @@
---
id: agent-voice
name: Voice Personas (Mars + Venus)
version: 0.1.0
version: 0.1.1
description: WebRTC-first voice agent reference (Mars + Venus personas, optional Twilio adapter). Skillpack-as-reference paradigm — the install-time agent COPIES code into your host agent repo where it becomes user-owned and mutable, NOT a runtime gbrain dependency.
category: voice
install_kind: copy-into-host-repo
@@ -132,7 +132,9 @@ Reference code ships intentionally minimal. Before public deployment:
- **Twilio signature validation** on `/voice` — currently absent; add `X-Twilio-Signature` header validation.
- **Rate limiting** on `/session` and `/tool` — currently absent.
- **CORS allowlist**currently `*`; restrict to your deployed origins.
- **CORS allowlist**default-deny out of the box: no `Access-Control-Allow-Origin` header is emitted unless the request's Origin exactly matches `AGENT_VOICE_CORS_ORIGIN` (comma-separated origins, e.g. `AGENT_VOICE_CORS_ORIGIN=https://your.app,https://staging.your.app`). The served `/call` page is same-origin and needs no configuration. `/session` and `/tool` additionally reject cross-origin browser requests (403) unless allowlisted — CORS headers alone can't stop a no-preflight "simple" POST from executing.
- **Bind address** — the server listens on `127.0.0.1` by default; set `HOST=0.0.0.0` for containers or direct LAN exposure (and prefer a tunnel for anything public).
- **Host-header allowlist** — not shipped; the origin gate derives self-origin from the `Host` header, so DNS rebinding is not covered. Add a `Host` allowlist before exposing beyond loopback.
- **Auth on /tool** — voice-side tool calls currently trust the in-process connection; if you expose `/tool` publicly, gate it behind a session token.
- **HTTPS** — required for browser mic access in production. Use ngrok / Caddy / Cloudflare Tunnel.
- **Twilio fallback URL**`/fallback` is a TwiML stub; wire to your operator's cell for crash recovery.
+86 -20
View File
@@ -19,16 +19,26 @@
* against `lib/twilio-bridge.mjs` (port-ready stubs included).
*
* Configuration via env:
* PORT default 8765
* OPENAI_API_KEY required for /session
* OPENAI_REALTIME_MODEL default 'gpt-4o-realtime-preview'
* DEFAULT_PERSONA default 'venus' (one of 'mars' | 'venus')
* BRAIN_ROOT passed through to context-builder
* TIMEZONE passed through to context-builder
* PORT default 8765
* HOST default '127.0.0.1' (loopback-only; set 0.0.0.0
* for containers / direct LAN exposure)
* OPENAI_API_KEY required for /session
* OPENAI_REALTIME_MODEL default 'gpt-4o-realtime-preview'
* DEFAULT_PERSONA default 'venus' (one of 'mars' | 'venus')
* AGENT_VOICE_CORS_ORIGIN comma-separated exact origins allowed via CORS
* (default unset = default-deny; the served /call
* page is same-origin and needs nothing)
* BRAIN_ROOT passed through to context-builder
* TIMEZONE passed through to context-builder
*
* Security posture: this is reference code. It does NOT ship hardening for
* production deployment (no rate limiting, no Twilio signature validation,
* no CORS allowlist). Operators add those at install time per the recipe's
* Security posture: CORS is default-deny (exact-origin allowlist via
* AGENT_VOICE_CORS_ORIGIN), the side-effectful POSTs (/session, /tool) are
* gated on the Origin header (CORS headers gate response READS, not request
* SENDS a cross-origin "simple" POST skips preflight, so without the gate
* an attacker page could blind-fire /session and burn the OpenAI key), and
* the listener binds loopback by default. Still reference code: rate
* limiting, Twilio signature validation, and a Host-header allowlist
* (DNS-rebinding hardening) are operator-added per the recipe's
* "production checklist."
*/
@@ -43,10 +53,60 @@ const __dirname = dirname(fileURLToPath(import.meta.url));
const PUBLIC_DIR = join(__dirname, 'public');
const PORT = parseInt(process.env.PORT || '8765', 10);
// Loopback by default (mirrors `gbrain serve --http --bind 127.0.0.1`).
// Tunnels (ngrok / Caddy / Cloudflare) target localhost, so the documented
// flows keep working; container/LAN deployments set HOST=0.0.0.0.
const HOST = process.env.HOST || '127.0.0.1';
const DEFAULT_PERSONA = (process.env.DEFAULT_PERSONA || 'venus').toLowerCase();
const OPENAI_REALTIME_MODEL = process.env.OPENAI_REALTIME_MODEL || 'gpt-4o-realtime-preview';
const OPENAI_REALTIME_URL = 'https://api.openai.com/v1/realtime/calls';
// ── CORS + origin gate (default-deny) ─────────────────────────────────
// Mirrors the GBRAIN_HTTP_CORS_ORIGIN pattern in gbrain's own HTTP
// transport: exact-origin allowlist, no header emitted otherwise, and
// Access-Control-Allow-Credentials is never set.
function parseCorsAllowlist() {
const v = process.env.AGENT_VOICE_CORS_ORIGIN;
if (!v) return null;
const entries = v.split(',').map((s) => s.trim()).filter(Boolean);
return entries.length > 0 ? new Set(entries) : null;
}
const CORS_ALLOWLIST = parseCorsAllowlist();
function applyCors(req, res) {
const origin = req.headers.origin;
if (!(CORS_ALLOWLIST && origin && CORS_ALLOWLIST.has(origin))) return;
res.setHeader('Access-Control-Allow-Origin', origin);
res.setHeader('Vary', 'Origin');
if (req.method === 'OPTIONS') {
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS');
res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');
}
}
// CORS headers gate response READS, not request SENDS: a cross-origin
// "simple" POST (e.g. text/plain) skips preflight entirely, executes
// server-side, and only the response is withheld from the attacker's JS.
// For endpoints with side effects (/session spends OPENAI_API_KEY, /tool
// dispatches brain reads) that isn't enough — reject disallowed Origins
// BEFORE doing any work. Requests without an Origin header (curl, Twilio
// webhooks, native apps) pass; browser requests pass only when same-origin
// (Origin host matches the Host header — covers the served /call page,
// including through a tunnel) or explicitly allowlisted. Known limit: a
// DNS-rebound page's Origin host matches the rebound Host header, so this
// does not defend against DNS rebinding (Host-header allowlist is the
// production-checklist follow-up).
function originAllowed(req) {
const origin = req.headers.origin;
if (!origin) return true;
if (CORS_ALLOWLIST && CORS_ALLOWLIST.has(origin)) return true;
try {
return new URL(origin).host === req.headers.host;
} catch {
return false;
}
}
const MIME = {
'.html': 'text/html; charset=utf-8',
'.js': 'application/javascript; charset=utf-8',
@@ -223,10 +283,9 @@ function handleVoiceTwiml(req, res) {
const server = createServer(async (req, res) => {
const url = new URL(req.url, `http://${req.headers.host}`);
// CORS: allow same-origin only by default. Operators relax in production.
res.setHeader('Access-Control-Allow-Origin', req.headers.origin || '*');
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS');
res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');
// CORS: default-deny. Headers are emitted only for allowlisted origins
// (AGENT_VOICE_CORS_ORIGIN); a 204 without CORS headers is browser-blocked.
applyCors(req, res);
if (req.method === 'OPTIONS') return send(res, 204, '');
try {
@@ -239,11 +298,13 @@ const server = createServer(async (req, res) => {
if (url.pathname === '/directory') {
return serveStatic(res, 'directory.html');
}
if (url.pathname === '/session') {
return handleSession(req, res);
}
if (url.pathname === '/tool') {
return handleTool(req, res);
if (url.pathname === '/session' || url.pathname === '/tool') {
// Origin gate BEFORE any body read / upstream fetch / tool dispatch —
// blocks blind cross-origin "simple" POSTs that CORS headers can't.
if (!originAllowed(req)) {
return sendJson(res, 403, { error: 'origin not allowed' });
}
return url.pathname === '/session' ? handleSession(req, res) : handleTool(req, res);
}
if (url.pathname === '/voice') {
return handleVoiceTwiml(req, res);
@@ -266,11 +327,16 @@ const server = createServer(async (req, res) => {
}
});
server.listen(PORT, () => {
server.listen(PORT, HOST, () => {
// eslint-disable-next-line no-console
console.log(`[agent-voice] listening on http://localhost:${PORT}`);
console.log(`[agent-voice] listening on http://${HOST}:${PORT} (bind: ${HOST}${HOST === '127.0.0.1' ? ' — set HOST=0.0.0.0 to expose beyond loopback' : ''})`);
console.log(`[agent-voice] default persona: ${DEFAULT_PERSONA}`);
console.log(`[agent-voice] read-only tools: ${getEffectiveAllowlist().join(', ')}`);
if (!CORS_ALLOWLIST) {
console.log('[agent-voice] CORS: default-deny. Set AGENT_VOICE_CORS_ORIGIN=https://your.app to allow cross-origin browser clients.');
} else {
console.log(`[agent-voice] CORS allowlist: ${[...CORS_ALLOWLIST].join(', ')}`);
}
});
process.on('SIGTERM', () => server.close(() => process.exit(0)));
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"recipe": "agent-voice",
"version": "0.1.0",
"version": "0.1.1",
"install_kind": "copy-into-host-repo",
"description": "src → target mapping consumed by `gbrain integrations install agent-voice`. The install command reads this manifest, computes SHA-256 of each source file at copy time, writes the per-file hash to <host>/services/voice-agent/.gbrain-source.json so --refresh can do three-way classification (unchanged-identical / unchanged-stale / locally-modified).",
"target_root_relative_to_host_repo": "services/voice-agent",
@@ -15,8 +15,14 @@ OPENAI_API_KEY=sk-... # required (OpenAI Realtime API)
DEFAULT_PERSONA=venus # optional (one of: venus, mars)
BRAIN_ROOT=/path/to/your/brain # optional (enables live context)
TIMEZONE=US/Pacific # optional
HOST=0.0.0.0 # optional — binds 127.0.0.1 (loopback) by default; set only to expose beyond localhost
AGENT_VOICE_CORS_ORIGIN=https://your.app # optional — CORS is default-deny; list exact origins (comma-separated) only if a browser on another origin needs access
```
The two security env vars ship safe by default: the server listens on loopback
only and refuses cross-origin browser requests. The local `/call` flow below
needs neither. See the recipe's production checklist before exposing publicly.
Optional for inbound Twilio:
```bash
TWILIO_ACCOUNT_SID=AC...
@@ -56,7 +62,7 @@ If any prompt-shape test fails, the privacy guard has caught a name you'd want t
```bash
cd <target-repo>/services/voice-agent
bun run start # or `npm start`
# → listening on http://localhost:8765
# → listening on http://127.0.0.1:8765 (bind: 127.0.0.1 — set HOST=0.0.0.0 to expose beyond loopback)
```
Open `http://localhost:8765/call` in a browser, click Connect, grant mic permission. You should be talking to Venus (or Mars if you set `DEFAULT_PERSONA=mars`).
+6 -1
View File
@@ -1,7 +1,7 @@
---
id: calendar-to-brain
name: Calendar-to-Brain
version: 0.7.0
version: 0.8.0
description: Google Calendar events become searchable brain pages. Daily files with attendees, locations, and meeting prep context.
category: sense
requires: [credential-gateway]
@@ -28,6 +28,11 @@ health_checks:
- type: env_exists
name: GOOGLE_CLIENT_ID
label: "Google OAuth"
- type: heartbeat_max_age
max_age: 48h
label: "Calendar data freshness"
output_paths:
- daily/calendar/
setup_time: 20 min
cost_estimate: "$0 (both options are free)"
---
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Print the CHANGELOG.md body for one version (Keep-a-Changelog format),
# without its `## [X.Y.Z.W] - date` header line. Used by
# .github/workflows/release.yml as the GitHub release notes; tested by
# test/release-workflow.test.ts.
#
# Usage: changelog-entry.sh <version> [changelog-file]
# Exits 1 when the version has no entry (caller falls back to a link stub).
set -euo pipefail
ver="${1:?usage: changelog-entry.sh <version> [changelog-file]}"
file="${2:-CHANGELOG.md}"
# Exact-string prefix match on "## [<ver>]" — no regex, so dots in the
# version can't glob and a 3-segment version can't match a 4-segment header.
awk -v ver="$ver" '
index($0, "## [" ver "]") == 1 { found = 1; next }
found && /^## \[/ { exit }
found { print }
END { exit found ? 0 : 1 }
' "$file"
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# CI guard for skills/skills.lock.json freshness (#159).
#
# Mirrors scripts/check-eval-glossary-fresh.sh: regenerate the manifest into
# a tmp file, diff against the committed version, fail the build if they
# drift. Tamper-evidence, not a signature system — the point is that any
# change under skills/ ships with an explicit manifest diff.
#
# Run: bash scripts/check-skills-manifest-fresh.sh
# Wired through `bun run verify` (scripts/run-verify-parallel.sh) so PRs that
# edit skills/ without regenerating the manifest are caught before review.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
COMMITTED="$REPO_ROOT/skills/skills.lock.json"
TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT
if [ ! -f "$COMMITTED" ]; then
echo "ERROR: $COMMITTED not found." >&2
echo "Run: bun run scripts/generate-skills-manifest.ts" >&2
exit 1
fi
cd "$REPO_ROOT"
# Render directly via bun + a one-liner that exposes the module function.
bun -e "import { renderSkillsManifest } from './src/core/skills-integrity.ts'; process.stdout.write(renderSkillsManifest('skills'));" > "$TMP"
if ! diff -q "$COMMITTED" "$TMP" >/dev/null 2>&1; then
echo "ERROR: skills/skills.lock.json is stale." >&2
echo "" >&2
echo "Diff between committed and freshly-generated:" >&2
echo "" >&2
diff -u "$COMMITTED" "$TMP" >&2 || true
echo "" >&2
echo "To regenerate: bun run scripts/generate-skills-manifest.ts" >&2
exit 1
fi
echo "✓ skills/skills.lock.json is fresh"
File diff suppressed because it is too large Load Diff
+177
View File
@@ -0,0 +1,177 @@
/**
* #2185 known-flags registry generator for CLI_ONLY commands.
*
* gbrain's CLI_ONLY commands read flags ad hoc (`args.includes('--force')`,
* per-command parseFlags helpers), so there is no parser to make strict. The
* pre-dispatch validator in src/cli.ts needs to know each command's legal
* flags; this script derives them from the source instead of a hand-typed
* list that would rot.
*
* How: parse handleCliOnly's top-level `case 'X': {` blocks out of src/cli.ts,
* collect every `import('./commands/Y.ts')` inside each block, then scan the
* case-block text plus each imported module (plus one level of that module's
* ./relative same-directory imports) for `--flag` string literals including
* help text, which deliberately over-includes: accepting a flag the handler
* ignores is the pre-#2185 status quo for that flag, while missing a real
* flag would break working invocations on upgrade.
*
* Output: src/core/cli-flag-registry.generated.ts (committed; freshness is
* pinned by test/cli-flag-validation.test.ts the same way build:llms pins the
* llms bundles). Regenerate: bun run build:flag-registry
*
* Hand-tuning lane: EXTRA_FLAGS below, for flags that live deeper than the
* one-level scan (add with a comment naming the deep module).
*/
import { readFileSync, writeFileSync, existsSync } from 'fs';
import { dirname, resolve as resolvePath, join } from 'path';
import { fileURLToPath } from 'url';
const ROOT = resolvePath(dirname(fileURLToPath(import.meta.url)), '..');
/** Flags that live deeper than the one-level module scan. Keep commented. */
const EXTRA_FLAGS: Record<string, string[]> = {
// embed's pace knobs resolve inside src/core/pace-mode.ts (two levels deep).
embed: ['--pace', '--pace-max-concurrency'],
// sync shares the same pace surface via env/config plus CLI passthrough.
sync: ['--pace', '--pace-max-concurrency'],
};
/** Universal helper flags every command may see (parsed or short-circuited upstream). */
const UNIVERSAL_FLAGS = ['--help', '--json', '--brain', '--source'];
const FLAG_RE = /--[a-z0-9][a-z0-9-]*/g;
function flagsInText(text: string): Set<string> {
const out = new Set<string>();
for (const m of text.matchAll(FLAG_RE)) {
// Template-literal prefixes (`--bound-${key}` scans as `--bound-`) are
// not real flags — a trailing hyphen would make the validator accept
// every typo sharing the prefix.
if (!m[0].endsWith('-')) out.add(m[0]);
}
return out;
}
/** One level of ./relative imports (static or dynamic) from a module's source. */
function relativeImports(src: string, fromDir: string): string[] {
const paths = new Set<string>();
for (const m of src.matchAll(/from\s+'(\.\.?\/[^']+\.ts)'/g)) paths.add(m[1]);
for (const m of src.matchAll(/import\('(\.\.?\/[^']+\.ts)'\)/g)) paths.add(m[1]);
return [...paths]
.map(p => resolvePath(fromDir, p))
.filter(p => existsSync(p));
}
export function buildFlagRegistry(): Record<string, string[]> {
const cliSource = readFileSync(join(ROOT, 'src/cli.ts'), 'utf-8');
// CLI_ONLY membership (the single source of truth in src/cli.ts). Strip
// line comments first — the set literal carries commentary whose quoted
// words ('Unknown command', 'pages') must not parse as members.
const onlyMatch = cliSource.match(/const CLI_ONLY = new Set(?:<string>)?\(\[([\s\S]*?)\]\)/);
if (!onlyMatch) throw new Error('CLI_ONLY set not found in src/cli.ts');
const onlyBody = onlyMatch[1].replace(/\/\/[^\n]*/g, '');
const commands = [...onlyBody.matchAll(/'([^']+)'/g)].map(m => m[1]);
// handleCliOnly body — bounded at the function's closing brace (column 0).
// Unbounded, the LAST case block absorbed every --flag literal in the rest
// of cli.ts (printHelp's full flag surface included), handing whichever
// command sits last in the switch a ~100-flag junk allowlist that made
// strict validation a no-op for it.
const fnStart = cliSource.indexOf('async function handleCliOnly');
if (fnStart < 0) throw new Error('handleCliOnly not found in src/cli.ts');
const fnTail = cliSource.slice(fnStart);
const fnEndRel = fnTail.search(/\n\}\n/);
const fnSrc = fnEndRel > 0 ? fnTail.slice(0, fnEndRel) : fnTail;
// handleCliOnly dispatches through TWO styles: an `if (command === 'X')`
// chain (DB-free commands like init/auth/schema) and a switch with
// `case 'X':` labels. Segment on BOTH marker kinds; the text between a
// marker and the next marker belongs to that label.
const markRe = /(?:^\s*if \(command === '([a-z0-9-]+)'\)|^ case '([a-z0-9-]+)':)/gm;
const marks: Array<{ label: string; start: number }> = [];
let m: RegExpExecArray | null;
while ((m = markRe.exec(fnSrc)) !== null) {
marks.push({ label: (m[1] ?? m[2])!, start: m.index });
}
const blocks = new Map<string, string>();
for (let i = 0; i < marks.length; i++) {
const end = i + 1 < marks.length ? marks[i + 1].start : fnSrc.length;
const body = fnSrc.slice(marks[i].start, end);
// Fall-through labels share the following block.
blocks.set(marks[i].label, (blocks.get(marks[i].label) ?? '') + body);
}
// Safety flags carry destructive-bypass semantics: allowlisting one that
// the handler never reads recreates the #2185 repro (`post-upgrade
// --dry-run` accepted, ignored, migrations run for real). Presence isn't
// enough — upgrade.ts prints a HINT naming another command's --dry-run,
// which is depth-0 text for post-upgrade. These flags are only legal with
// CONSUMPTION evidence in the command's own code: the flag as a TIGHT-QUOTED
// standalone literal (`includes('--dry-run')`, `has('--dry-run')`,
// `=== '--dry-run'`). Prose bleed embeds the flag inside a longer string, so
// it never has quotes on both sides of the bare flag.
const SAFETY_FLAGS = new Set(['--dry-run']);
const consumes = (text: string, flag: string): boolean =>
new RegExp(`['"\`]${flag}['"\`]`).test(text);
const registry: Record<string, string[]> = {};
for (const command of commands) {
const block = blocks.get(command) ?? '';
const flags = new Set<string>(UNIVERSAL_FLAGS);
const depthZero = new Set<string>();
let depthZeroText = block;
for (const f of flagsInText(block)) { flags.add(f); depthZero.add(f); }
// Modules imported inside the case block, plus one level of each module's
// own ./relative imports.
const commandModules = [...block.matchAll(/import\('(\.\/[^']+\.ts)'\)/g)]
.map(mm => resolvePath(join(ROOT, 'src'), mm[1]))
.filter(p => existsSync(p));
for (const modPath of commandModules) {
const modSrc = readFileSync(modPath, 'utf-8');
depthZeroText += modSrc;
for (const f of flagsInText(modSrc)) { flags.add(f); depthZero.add(f); }
for (const dep of relativeImports(modSrc, dirname(modPath))) {
for (const f of flagsInText(readFileSync(dep, 'utf-8'))) flags.add(f);
}
}
for (const f of EXTRA_FLAGS[command] ?? []) { flags.add(f); depthZero.add(f); }
for (const f of SAFETY_FLAGS) {
if (flags.has(f) && !consumes(depthZeroText, f)) flags.delete(f);
}
registry[command] = [...flags].sort();
}
return registry;
}
export function renderRegistryModule(registry: Record<string, string[]>): string {
const entries = Object.keys(registry)
.sort()
.map(cmd => ` '${cmd}': [${registry[cmd].map(f => `'${f}'`).join(', ')}],`)
.join('\n');
return `// AUTO-GENERATED by scripts/generate-flag-registry.ts — do not edit by hand.
// Regenerate: bun run build:flag-registry
// Freshness + drift pinned by test/cli-flag-validation.test.ts (#2185).
//
// Per-command legal flags for CLI_ONLY commands, derived from each command's
// source (case block + imported modules + one level of relative imports +
// scripts/generate-flag-registry.ts EXTRA_FLAGS). Deliberately over-inclusive
// (help-text mentions count): accepting an ignored flag is the pre-#2185
// status quo; missing a real one breaks working invocations.
export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
${entries}
};
`;
}
if (import.meta.main) {
const registry = buildFlagRegistry();
const outPath = join(ROOT, 'src/core/cli-flag-registry.generated.ts');
writeFileSync(outPath, renderRegistryModule(registry));
const n = Object.keys(registry).length;
const total = Object.values(registry).reduce((a, v) => a + v.length, 0);
console.log(`wrote ${outPath} (${n} commands, ${total} flag entries)`);
}
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bun
/**
* Regenerates skills/skills.lock.json the tamper-evidence manifest mapping
* every bundled file under skills/ to its sha256 (#159). Not a signature
* system: it turns silent skill edits into explicit diffs. `gbrain doctor`
* warns (never fails) on drift; scripts/check-skills-manifest-fresh.sh keeps
* the committed manifest in sync in CI.
*
* Run after any change under skills/:
* bun run scripts/generate-skills-manifest.ts
*/
import { writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import {
SKILLS_MANIFEST_FILENAME,
renderSkillsManifest,
} from '../src/core/skills-integrity.ts';
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
const skillsDir = join(repoRoot, 'skills');
const outPath = join(skillsDir, SKILLS_MANIFEST_FILENAME);
writeFileSync(outPath, renderSkillsManifest(skillsDir));
console.log(`Wrote ${outPath}`);
+6
View File
@@ -162,6 +162,12 @@ export const SECTIONS: DocSection[] = [
description: "MCP server deployment.",
path: "docs/mcp/DEPLOY.md",
},
{
title: "docs/protocol/MEMORY_VERBS_v1.md",
description:
"The frozen five-verb memory protocol (recall/remember/entity/synthesize/forget): response envelopes, error contract, additive-forever versioning, surface modes, conformance certification, per-harness installs.",
path: "docs/protocol/MEMORY_VERBS_v1.md",
},
],
},
{
+302 -10
View File
@@ -13,8 +13,29 @@
#
# Env overrides:
# SHARDS=N same as --shards
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 600)
# GBRAIN_TEST_SHARD_TIMEOUT per-shard wallclock cap, seconds (default 3000)
# GBRAIN_TEST_SHARD_KILL_AFTER grace after TERM before KILL (default 30)
# GBRAIN_TEST_MAX_CONCURRENCY passed through to bun test (default 4)
# GBRAIN_TEST_MEM_PER_FILE_MB memory budget per concurrent test file used by
# the adaptive sizing below (default 1536 — a
# PGLite WASM instance reserves ~1-1.5GB)
# GBRAIN_TEST_NO_MEM_ADAPT=1 disable memory-aware concurrency reduction
# GBRAIN_TEST_NO_OOM_FALLBACK=1 disable the serial OOM-rescue pass
#
# Memory safety (two layers; both default-on):
# 1. ADAPTIVE SIZING — before spawning, total concurrency (shards ×
# intra-shard --max-concurrency) is capped to what available memory can
# hold at GBRAIN_TEST_MEM_PER_FILE_MB per concurrent file. Concurrent
# Conductor workspaces running their own suites shrink the budget
# automatically instead of OOMing each other.
# 2. SERIAL PHANTOM RESCUE — two phantom classes are re-run serially
# (--max-concurrency 1) after the parallel pass: (a) failures whose
# shard log carries the PGLite WASM out-of-memory signature, and
# (b) shards killed EXTERNALLY (SIGTERM/SIGKILL well before the shard
# timeout — sibling Conductor workspaces' process cleanup, macOS memory
# jetsam). Phantoms pass serially and the run goes green with an
# oom_rescued note; real failures fail again and stay red. Plain
# assertion failures never match either signature.
#
# Output files (workspace-local; falls back to /tmp if .context/ unwritable):
# .context/test-failures.log failure blocks (cleared at start)
@@ -37,6 +58,35 @@ detect_cpus() {
echo 4
}
# ──────────────────────────────────────────────────────────────────────────
# Available-memory detection (MB). macOS: vm_stat free + inactive +
# speculative + purgeable pages (inactive/purgeable are reclaimable on
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
# Unknown platform → 0, and the caller skips adaptation entirely.
# ──────────────────────────────────────────────────────────────────────────
detect_available_mem_mb() {
if command -v vm_stat >/dev/null 2>&1; then
vm_stat 2>/dev/null | awk '
/page size of/ { psize = $8 }
/Pages free/ { free = $NF }
/Pages inactive/ { inactive = $NF }
/Pages speculative/ { spec = $NF }
/Pages purgeable/ { purge = $NF }
END {
gsub(/\./, "", free); gsub(/\./, "", inactive)
gsub(/\./, "", spec); gsub(/\./, "", purge)
if (psize == 0) psize = 16384
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
}'
return
fi
if [ -r /proc/meminfo ]; then
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
return
fi
echo 0
}
# ──────────────────────────────────────────────────────────────────────────
# Argument parsing. --shards N override wins over $SHARDS; both are clamped.
# ──────────────────────────────────────────────────────────────────────────
@@ -78,7 +128,59 @@ INTRA_CONC="${MAX_CONCURRENCY_OVERRIDE:-${GBRAIN_TEST_MAX_CONCURRENCY:-4}}"
# had completed in 968s. 1500s cap gives ~55% headroom over observed
# 4-shard wallclock; real hangs still hit it. Override via
# GBRAIN_TEST_SHARD_TIMEOUT=N.
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-1500}"
# v0.42.74 sizing: 1500 -> 3000. The suite roughly tripled since the 1500s
# cap was set (June: ~3900 tests, 92-migration PGLite replay; now: 11k+
# tests, 120-migration replay per PGLite init). At 4 shards, two shards were
# killed at 1500s while making steady per-test progress. 3000s keeps the
# same ~55%-headroom doctrine over observed wallclock; real hangs still die.
SHARD_TIMEOUT="${GBRAIN_TEST_SHARD_TIMEOUT:-3000}"
SHARD_KILL_AFTER="${GBRAIN_TEST_SHARD_KILL_AFTER:-30}"
if ! printf '%s' "$SHARD_KILL_AFTER" | grep -qE '^[0-9]+$' || [ "$SHARD_KILL_AFTER" -lt 1 ]; then
echo "ERROR: invalid shard kill-after: $SHARD_KILL_AFTER" >&2; exit 2
fi
# ──────────────────────────────────────────────────────────────────────────
# Memory-aware concurrency (layer 1). Total concurrent test files =
# N shards × INTRA_CONC; each concurrent file can hold a PGLite WASM
# instance (~1-1.5GB reserved). 4×4 = 16 concurrent instances OOM'd on a
# 128GB machine when other Conductor workspaces ran their suites at the
# same time — every PGLite connect across every shard failed at once
# ("Out of memory" at PGlite.create). Cap total concurrency to what's
# actually available, keeping a 4GB reserve for the OS + bun itself.
# Applies to explicit --shards overrides too (an operator who wants an
# over-committed run sets GBRAIN_TEST_NO_MEM_ADAPT=1).
# ──────────────────────────────────────────────────────────────────────────
MEM_PER_FILE_MB="${GBRAIN_TEST_MEM_PER_FILE_MB:-1536}"
MEM_NOTE=""
if [ "${GBRAIN_TEST_NO_MEM_ADAPT:-0}" != "1" ]; then
AVAIL_MB=$(detect_available_mem_mb)
if [ "${AVAIL_MB:-0}" -gt 0 ] 2>/dev/null; then
BUDGET_MB=$((AVAIL_MB - 4096))
[ "$BUDGET_MB" -lt "$MEM_PER_FILE_MB" ] && BUDGET_MB="$MEM_PER_FILE_MB"
MAX_TOTAL=$((BUDGET_MB / MEM_PER_FILE_MB))
[ "$MAX_TOTAL" -lt 1 ] && MAX_TOTAL=1
ORIG_N="$N"; ORIG_INTRA="$INTRA_CONC"
# Shed shards before intra-shard concurrency: fewer bun processes frees
# more than narrower ones (each process carries its own heap + WASM).
while [ $((N * INTRA_CONC)) -gt "$MAX_TOTAL" ]; do
if [ "$N" -gt 1 ]; then N=$((N - 1))
elif [ "$INTRA_CONC" -gt 1 ]; then INTRA_CONC=$((INTRA_CONC - 1))
else break
fi
done
if [ "$N" != "$ORIG_N" ] || [ "$INTRA_CONC" != "$ORIG_INTRA" ]; then
# Fewer shards → more files per shard → each shard legitimately runs
# longer. Scale the per-shard cap by the shed ratio so adaptation
# doesn't convert memory safety into false WEDGED verdicts.
if [ "$N" -lt "$ORIG_N" ]; then
SHARD_TIMEOUT=$((SHARD_TIMEOUT * ORIG_N / N))
fi
MEM_NOTE=" | mem-adapted ${ORIG_N}x${ORIG_INTRA}${N}x${INTRA_CONC} (avail=${AVAIL_MB}MB, ${MEM_PER_FILE_MB}MB/file, timeout→${SHARD_TIMEOUT}s)"
else
MEM_NOTE=" | mem-ok (avail=${AVAIL_MB}MB)"
fi
fi
fi
# ──────────────────────────────────────────────────────────────────────────
# Output directories. Prefer workspace-local .context/, fall back to /tmp.
@@ -95,7 +197,7 @@ else
mkdir -p "$LOG_DIR" || { echo "ERROR: cannot create log dir" >&2; exit 2; }
fi
# Clear from prior run.
rm -f "$LOG_DIR"/shard-*.log "$LOG_DIR"/shard-*.exit "$LOG_DIR"/shard-*.wedged 2>/dev/null
rm -f "$LOG_DIR"/shard-*.log "$LOG_DIR"/shard-*.exit "$LOG_DIR"/shard-*.wedged "$LOG_DIR"/shard-*.start "$LOG_DIR"/shard-*.end 2>/dev/null
: > "$FAILURES_LOG"
: > "$SUMMARY_FILE"
@@ -109,7 +211,7 @@ elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
fi
START_TS=$(date +%s)
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | logs=$LOG_DIR" >&2
echo "[unit-parallel] N=$N shards | --max-concurrency=$INTRA_CONC | timeout=${SHARD_TIMEOUT}s | kill-after=${SHARD_KILL_AFTER}s | logs=$LOG_DIR${MEM_NOTE}" >&2
if [ "$DRY_RUN" = "1" ]; then
echo "[unit-parallel] dry-run: would spawn $N shards with the above settings."
@@ -128,8 +230,9 @@ SHARD_PIDS=()
for i in $(seq 1 "$N"); do
(
SHARD_LOG="$LOG_DIR/shard-$i.log"
date +%s > "$LOG_DIR/shard-$i.start"
if [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" "${SHARD_TIMEOUT}s" \
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${SHARD_TIMEOUT}s" \
env SHARD="$i/$N" \
bash scripts/run-unit-shard.sh --max-concurrency="$INTRA_CONC" \
> "$SHARD_LOG" 2>&1
@@ -140,7 +243,7 @@ for i in $(seq 1 "$N"); do
> "$SHARD_LOG" 2>&1 &
pid=$!
( sleep "$SHARD_TIMEOUT" && kill -TERM "$pid" 2>/dev/null && \
sleep 5 && kill -KILL "$pid" 2>/dev/null ) &
sleep "$SHARD_KILL_AFTER" && kill -KILL "$pid" 2>/dev/null ) &
cap_pid=$!
wait "$pid" 2>/dev/null
# Capture the shard's exit code from ITS `wait`, before any watchdog
@@ -157,8 +260,9 @@ for i in $(seq 1 "$N"); do
kill "$cap_pid" 2>/dev/null
wait "$cap_pid" 2>/dev/null
fi
date +%s > "$LOG_DIR/shard-$i.end"
echo "$rc" > "$LOG_DIR/shard-$i.exit"
[ "$rc" = "124" ] && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
{ [ "$rc" = "124" ] || [ "$rc" = "137" ]; } && echo "WEDGED" > "$LOG_DIR/shard-$i.wedged"
) &
SHARD_PIDS+=($!)
done
@@ -311,6 +415,40 @@ TOTAL_FAILURES=0
TOTAL_PASS=0
TOTAL_SKIP=0
TOTAL_RC=0
# Layer 2 state (serial OOM rescue). A shard whose log carries the WASM
# out-of-memory signature gets its failing files queued for a serial re-run;
# NON_OOM_FAIL records that at least one failure exists that the rescue lane
# must NOT absolve (plain assertion failures, wedges without the signature).
OOM_RE='Out of memory|WebAssembly\.Memory|RuntimeError: [Aa]borted|Aborted\(\)'
OOM_RESCUE_LIST="$LOG_DIR/oom-rescue-files.txt"
: > "$OOM_RESCUE_LIST"
NON_OOM_FAIL=0
# Set when any shard was killed externally — killed-midrun shards leave lock/
# state residue that can poison the LATER serial pass, so serial failures are
# only rescue-eligible under this flag (or their own OOM signature). A flaky
# serial test in an otherwise-clean run must stay red.
EXTERNAL_KILL_ANY=0
# failing_files_in_log: attribute each `(fail)` block to the test file whose
# `path.test.ts:` header most recently preceded it in bun's output. Under
# GITHUB_ACTIONS the shard wraps each file section as `::group::path.test.ts:`
# — strip that prefix or the rescue pass feeds bun literal `::group::...`
# non-paths that match zero test files (CI-only; local runs have no groups).
failing_files_in_log() {
local file="$1"
[ -f "$file" ] || return 0
awk '
/^(::group::)?[^ ].*\.test\.ts:$/ {
current = $0
sub(/^::group::/, "", current)
current = substr(current, 1, length(current) - 1)
next
}
/^\(fail\) / && current != "" { print current }
' "$file" | sort -u
}
for i in $(seq 1 "$N"); do
SHARD_LOG="$LOG_DIR/shard-$i.log"
EXIT_FILE="$LOG_DIR/shard-$i.exit"
@@ -325,17 +463,74 @@ for i in $(seq 1 "$N"); do
TOTAL_FAILURES=$((TOTAL_FAILURES + fail_count))
TOTAL_SKIP=$((TOTAL_SKIP + skip_count))
shard_oom=0
if [ "$rc" != "0" ] && [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
&& [ -f "$SHARD_LOG" ] && grep -qE "$OOM_RE" "$SHARD_LOG"; then
shard_oom=1
fi
# External-kill detection: rc 143 (SIGTERM) / 137 (SIGKILL) with the shard
# dying before 80% of the shard timeout means something OUTSIDE the runner
# killed it — sibling Conductor workspaces' process cleanup and macOS
# memory jetsam both present exactly this way (observed: 3 shards TERM'd +
# 1 KILL'd at ~700s under a 3000s cap, all mid-progress). A REAL wedge is
# killed BY the runner at ~SHARD_TIMEOUT and stays red. Externally-killed
# shards are phantoms: queue for the serial rescue lane like OOM.
shard_external_kill=0
if [ "$shard_oom" = "0" ] && [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
&& { [ "$rc" = "143" ] || [ "$rc" = "137" ]; }; then
s_start=$(cat "$LOG_DIR/shard-$i.start" 2>/dev/null) || s_start=""
s_end=$(cat "$LOG_DIR/shard-$i.end" 2>/dev/null) || s_end=""
if [ -n "$s_start" ] && [ -n "$s_end" ]; then
s_elapsed=$((s_end - s_start))
if [ "$s_elapsed" -lt $((SHARD_TIMEOUT * 80 / 100)) ]; then
shard_external_kill=1
EXTERNAL_KILL_ANY=1
fi
fi
fi
if [ -f "$WEDGED_FILE" ]; then
TOTAL_RC=1
if [ "$shard_external_kill" = "1" ]; then
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
echo "shard $i/$N: KILLED externally after ${s_elapsed}s (rc=$rc, well before ${SHARD_TIMEOUT}s cap — queued for serial rescue)" >> "$SUMMARY_FILE"
elif [ "$shard_oom" = "1" ]; then
# Wedged UNDER memory pressure: we can't attribute failures, so queue
# the shard's entire file list for the serial rescue pass.
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc, OOM signature — queued for serial rescue)" >> "$SUMMARY_FILE"
else
NON_OOM_FAIL=1
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc)" >> "$SUMMARY_FILE"
fi
{
echo "--- shard $i: WEDGED after ${SHARD_TIMEOUT}s ---"
[ -f "$SHARD_LOG" ] && tail -50 "$SHARD_LOG"
echo ""
} >> "$FAILURES_LOG"
echo "shard $i/$N: WEDGED after ${SHARD_TIMEOUT}s (rc=$rc)" >> "$SUMMARY_FILE"
continue
fi
if [ "$rc" != "0" ]; then
if [ "$shard_oom" = "1" ]; then
# One scan, reused for both the queue append and the emptiness check.
shard_failing_files=$(failing_files_in_log "$SHARD_LOG")
if [ -n "$shard_failing_files" ]; then
printf '%s\n' "$shard_failing_files" >> "$OOM_RESCUE_LIST"
else
# OOM signature but no attributable files (e.g. bun died before any
# file header) → rescue the whole shard.
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
fi
elif [ "$shard_external_kill" = "1" ]; then
SHARD="$i/$N" bash scripts/run-unit-shard.sh --dry-run-list 2>/dev/null >> "$OOM_RESCUE_LIST"
echo "shard $i/$N: KILLED externally after ${s_elapsed}s (rc=$rc — queued for serial rescue)" >> "$SUMMARY_FILE"
else
NON_OOM_FAIL=1
fi
fi
echo "shard $i/$N: pass=$pass_count fail=$fail_count skip=$skip_count rc=$rc" >> "$SUMMARY_FILE"
if [ "$rc" != "0" ]; then
@@ -390,6 +585,17 @@ if [ "$SERIAL_FILES_COUNT" -gt 0 ]; then
cat "$LOG_DIR/serial.log"
if [ "$SERIAL_RC" != "0" ]; then
TOTAL_RC=1
if [ "${GBRAIN_TEST_NO_OOM_FALLBACK:-0}" != "1" ] \
&& { grep -qE "$OOM_RE" "$LOG_DIR/serial.log" || [ "$EXTERNAL_KILL_ANY" = "1" ]; }; then
# Serial failures are rescue-eligible ONLY with their own OOM signature
# or when an externally-killed shard ran earlier in this invocation
# (killed-midrun shards leave lock/state residue that poisons the serial
# pass). A merely-OOM'd sibling shard is NOT grounds — a flaky serial
# test must stay red rather than get silently absolved.
failing_files_in_log "$LOG_DIR/serial.log" >> "$OOM_RESCUE_LIST"
else
NON_OOM_FAIL=1
fi
s_fail=$(bun_summary_count "fail" "$LOG_DIR/serial.log")
TOTAL_FAILURES=$((TOTAL_FAILURES + s_fail))
if [ "$s_fail" -gt 0 ]; then
@@ -415,6 +621,92 @@ if [ "$SERIAL_FILES_COUNT" -gt 0 ]; then
fi
fi
# ──────────────────────────────────────────────────────────────────────────
# Layer 2: serial OOM rescue. Re-run every file that failed inside an
# OOM-signature shard, one at a time (1 shard, --max-concurrency 1), after
# the parallel fan-out has fully drained. Phantom failures (the WASM ran out
# of memory because 16 instances were up at once) pass here and the run goes
# green with an oom_rescued note; real failures fail again and stay red.
# ──────────────────────────────────────────────────────────────────────────
OOM_RESCUED=0
OOM_RESCUE_NOTE=""
sort -u "$OOM_RESCUE_LIST" -o "$OOM_RESCUE_LIST" 2>/dev/null
# grep -c exits 1 on zero matches — assign in two steps so an empty rescue
# list yields a single "0" (the grep_count double-output bug, same class).
RESCUE_COUNT=$(grep -c . "$OOM_RESCUE_LIST" 2>/dev/null) || RESCUE_COUNT=0
if [ "$TOTAL_RC" != "0" ] && [ "${RESCUE_COUNT:-0}" -gt 0 ]; then
echo "════════════ OOM rescue pass ($RESCUE_COUNT files, serial) ════════════"
echo "[unit-parallel] OOM signature detected — re-running $RESCUE_COUNT failing file(s) at --max-concurrency 1" >&2
RESCUE_LOG="$LOG_DIR/oom-rescue.log"
# 60s-per-file floor with the shard cap as a minimum, and 2x the shard cap
# as a CEILING: a wedged shard queueing its whole file list must not turn
# `bun run test` into an unbounded multi-hour serial re-run — hitting the
# ceiling reads as a red rescue, not silence.
RESCUE_TIMEOUT=$((RESCUE_COUNT * 60))
[ "$RESCUE_TIMEOUT" -lt "$SHARD_TIMEOUT" ] && RESCUE_TIMEOUT="$SHARD_TIMEOUT"
[ "$RESCUE_TIMEOUT" -gt $((SHARD_TIMEOUT * 2)) ] && RESCUE_TIMEOUT=$((SHARD_TIMEOUT * 2))
# Split the queue: *.serial.test.ts files require one bun PROCESS per file
# (run-serial-tests.sh's isolation contract — top-level mock.module leaks
# across files in a shared registry); the remainder batches in one process.
# Both lanes mirror the shard invocation's --timeout=60000 — bun's default
# 5s per-test timeout would re-fail PGLite phantoms (120-migration replay)
# and mislabel them 'confirmed real'.
grep -v '\.serial\.test\.ts$' "$OOM_RESCUE_LIST" > "$LOG_DIR/oom-rescue-batch.txt" || true
grep '\.serial\.test\.ts$' "$OOM_RESCUE_LIST" > "$LOG_DIR/oom-rescue-serial.txt" || true
RESCUE_RC=0
: > "$RESCUE_LOG"
run_rescue() { # $1 = per-invocation timeout seconds; rest = test-file args
local t="$1"; shift
if [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" --signal=TERM --kill-after="${SHARD_KILL_AFTER}s" "${t}s" \
bun test --max-concurrency 1 --timeout=60000 "$@" >> "$RESCUE_LOG" 2>&1
else
bun test --max-concurrency 1 --timeout=60000 "$@" >> "$RESCUE_LOG" 2>&1
fi
}
if [ -s "$LOG_DIR/oom-rescue-batch.txt" ]; then
# shellcheck disable=SC2046
run_rescue "$RESCUE_TIMEOUT" $(cat "$LOG_DIR/oom-rescue-batch.txt") || RESCUE_RC=1
fi
if [ -s "$LOG_DIR/oom-rescue-serial.txt" ]; then
while IFS= read -r serial_file; do
[ -n "$serial_file" ] || continue
run_rescue 300 "$serial_file" || RESCUE_RC=1
done < "$LOG_DIR/oom-rescue-serial.txt"
fi
cat "$RESCUE_LOG"
r_pass=$(bun_summary_count "pass" "$RESCUE_LOG")
r_fail=$(bun_summary_count "fail" "$RESCUE_LOG")
if [ "$RESCUE_RC" = "0" ] && [ "$NON_OOM_FAIL" = "0" ]; then
# Every failure in the run was OOM-phantom and every rescued file passed
# serially: the run is green. Adjust the headline numbers so they reflect
# the rescue verdict, and mark the earlier failure blocks superseded.
TOTAL_RC=0
OOM_RESCUED=1
# Do NOT fold r_pass into TOTAL_PASS — the failing shard's own summary
# already counted the rescued files' passing tests, so folding would
# double-count. Rescue results ride in the note instead.
TOTAL_FAILURES=0
OOM_RESCUE_NOTE=" | oom_rescued=${RESCUE_COUNT}files(${r_pass}p serial)"
{
echo "--- OOM rescue: all $RESCUE_COUNT file(s) passed serially (${r_pass} tests) ---"
echo "--- failure blocks above were WASM out-of-memory phantoms, superseded ---"
} >> "$FAILURES_LOG"
echo "oom-rescue: $RESCUE_COUNT files pass=$r_pass rc=0 (phantom OOM failures superseded)" >> "$SUMMARY_FILE"
else
# Real failures confirmed serially (or a non-OOM failure exists anyway).
OOM_RESCUE_NOTE=" | oom_rescue_failed=${r_fail}real"
awk '
/^\(fail\) / { in_block=1; print "--- oom-rescue (serial, confirmed real): " $0; next }
in_block {
if (/^\(pass\)/ || /^\(skip\)/ || /^[[:space:]]*$/ || /__bun_test_summary__/) { in_block=0; print ""; next }
print $0
}
' "$RESCUE_LOG" >> "$FAILURES_LOG"
echo "oom-rescue: $RESCUE_COUNT files pass=$r_pass fail=$r_fail rc=$RESCUE_RC (real failures confirmed)" >> "$SUMMARY_FILE"
fi
fi
END_TS=$(date +%s)
ELAPSED=$((END_TS - START_TS))
@@ -431,10 +723,10 @@ if [ "$TOTAL_RC" != "0" ]; then
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
tail -30 "$FAILURES_LOG"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP"
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP${OOM_RESCUE_NOTE}"
} >&2
exit 1
fi
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP" >&2
echo "[unit-parallel] elapsed=${ELAPSED}s | pass=$TOTAL_PASS fail=$TOTAL_FAILURES skip=$TOTAL_SKIP${OOM_RESCUE_NOTE}" >&2
exit 0
+1
View File
@@ -50,6 +50,7 @@ CHECKS=(
"check:cli-exec"
"check:system-of-record"
"check:eval-glossary"
"check:skills-manifest"
"check:no-pii-agent-voice"
"check:synthetic-corpus-privacy"
"check:skill-brain-first"
+1 -1
View File
@@ -248,7 +248,7 @@ before submission.
After the brain page is written, render to PDF using `skills/brain-pdf`:
```bash
gbrain put_page # already done by the CLI; nothing to add here
gbrain put # already done by the CLI; nothing to add here
# Then invoke brain-pdf:
# (see skills/brain-pdf/SKILL.md for the make-pdf invocation)
```
+15 -4
View File
@@ -34,6 +34,17 @@ flows through in both directions.
> **Convention:** See `skills/conventions/brain-first.md` for the 5-step lookup protocol.
> **Convention:** See `skills/conventions/quality.md` for citation and back-link rules.
> **Memory verbs (MEMORY_VERBS v1, gbrain ≥ 0.43).** Over MCP, prefer the five
> frozen memory verbs for the read/write cycle: **`remember(fact, provenance,
> ttl?)`** to save a single durable fact (mandatory provenance; dedupes +
> supersedes), **`recall(query | entity, budget_tokens)`** to read it back
> budget-packed, **`entity(name)`** for a zero-LLM card, **`synthesize(question)`**
> for the expensive cross-page answer, **`forget(id)`** to expire a fact. Use
> `remember` instead of `extract_facts` when you already have ONE formed fact;
> `put_page` / `add_link` / `add_timeline_entry` stay the page/graph write path.
> Fall back to the classic ops when the verbs aren't on the surface. Contract:
> `docs/protocol/MEMORY_VERBS_v1.md`.
## Contract
This skill guarantees:
@@ -56,8 +67,8 @@ broken brain. See `skills/conventions/quality.md` for format.
Before using ANY external API to research a person, company, or topic:
1. `gbrain search "name"`keyword search for existing pages
2. `gbrain query "natural question about name"`hybrid search for context
1. `gbrain search "name"`exact-token lookup for existing pages (cheap hybrid, no expansion)
2. `gbrain query "natural question about name"`concept/landscape questions go here FIRST (expansion recovers synonym phrasings; a nonzero `search` count is not proof of completeness)
3. `gbrain get <slug>` — if you know the slug, read the full page
4. Check backlinks: who references this entity?
5. Check timeline: recent events involving this entity
@@ -153,8 +164,8 @@ the citation is `[gstack:plans/foo]`. That's the whole rule.
## Tools Used
- `search`keyword search
- `query` — hybrid vector+keyword search
- `search`cheap hybrid search (vector + keyword, no expansion)
- `query` — hybrid search + LLM multi-query expansion (concept/landscape questions)
- `get_page` — read a brain page
- `put_page` — create/update brain pages
- `add_link` — cross-reference entities
+18 -6
View File
@@ -11,8 +11,8 @@ Your tool inventory includes these (prefixed `gbrain__` in OpenClaw):
| Tool | Use for |
|------|---------|
| `gbrain__search` / `search` | Keyword search — fast, always works |
| `gbrain__query` / `query` | Hybrid search (keyword + semantic) — best quality |
| `gbrain__search` / `search` | Exact tokens / known names — cheap hybrid, no expansion |
| `gbrain__query` / `query` | Concept / landscape questions — hybrid + LLM expansion |
| `gbrain__get_page` / `get_page` | Direct page read when you know the slug |
| `gbrain__get_links` / `get_links` | Outgoing links from a page |
| `gbrain__get_backlinks` / `get_backlinks` | Who references this entity |
@@ -28,10 +28,22 @@ Tool names vary by transport (MCP uses short names, OpenClaw plugin uses
## The Lookup Chain (MANDATORY ORDER)
1. **`search`** first — keyword search, fast, zero API cost
2. **`query`** if search is thin — hybrid semantic search, uses embedding API
3. **`get_page`** if you found a slug — read the full compiled truth
4. **External APIs only after steps 1-2 return nothing useful**
Route by the SHAPE of the question, then escalate:
1. **Exact known token / name / structured field****`search`** — cheap
hybrid (vector + keyword, no expansion; embedding-only cost).
2. **Concept / landscape / synonym-phrased question** ("all the X that do Y",
"the landscape of Z") → **`query`** FIRST — multi-query expansion recovers
phrasings `search` misses. Costs one extra LLM expansion call; worth it
for these.
3. **`get_page`** if you found a slug — read the full compiled truth.
4. **External APIs only after steps 1-2 return nothing useful.**
**A nonzero `search` count is NOT a completeness signal.** For "did I capture
everything about X?" run `query` even if `search` already returned hits —
synonym- and outcome-phrased matches drop silently otherwise. And `query` is
still top-K: for literal "list every page that…" enumeration, use `list_pages`
with pagination.
Never skip to external APIs without completing steps 1-2. The brain has
thousands of pages. The answer is almost always there.
+6 -6
View File
@@ -73,13 +73,13 @@ stock worker auto-loads on startup) registers handlers before `start()`.
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
using `agentTurn`. Respect that. No auto-rewrite.
## Forward note (v0.12.0)
## Forward note
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
`gbrain jobs work` that owns cron expressions natively — no more
handing off to host schedulers. Until v0.12.0 lands, the host
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
layer (what the cron trigger *does*), not the scheduling layer.
A native scheduler loop inside `gbrain jobs work` (owning cron
expressions directly, with no host-scheduler hand-off) has been on the
roadmap since v0.11.1 but has not shipped. The host scheduler keeps
firing on schedule; this convention only replaces the execution layer
(what the cron trigger *does*), not the scheduling layer.
## Related
+2 -1
View File
@@ -125,7 +125,8 @@ v0.41.22 ships **gbrain-base-v2** as the declared successor to
gbrain-base@1.x — collapses 94 noisy types to 15 canonical via
declarative mapping_rules. Run via `gbrain onboard --check --explain`
(preview) → `gbrain jobs submit unify-types --allow-protected --params
'{"target_pack":"gbrain-base-v2"}'` (apply). See
'{"target_pack":"gbrain-base-v2","apply":true}'` (apply — `apply`
defaults to false, so a bare submit is a dry run). See
`skills/schema-unify/SKILL.md` for the full playbook.
Authoring a successor pack: declare
+2 -2
View File
@@ -54,8 +54,8 @@ Ask the user what they want to track. Either:
- Define a custom recipe with: source queries, classification rules, extraction schema,
tracker page path, tracker format
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Use `gbrain research init`
to scaffold a new one.
Recipes are YAML files at `~/.gbrain/recipes/{name}.yaml`. Scaffold a new one by
copying a built-in recipe file and editing its fields.
### Phase 2: Search Sources
+1 -1
View File
@@ -201,7 +201,7 @@ Use the brain page template. MUST include:
### 4b. Entity pages (people, companies)
For each entity mentioned:
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get_page people/<slug>`).
- Check if a brain page exists (`gbrain search "<name>"` or `gbrain get people/<slug>`).
- If exists: update State, append Timeline entry citing this research.
- If not: create with enrichment.
+1 -1
View File
@@ -112,7 +112,7 @@ gbrain query "<topic keywords>"
# -d '{"model": "sonar-pro", "messages": [{"role":"user","content":"..."}]}'
# 4. Write the structured research page via put_page:
gbrain put_page research/<slug> # via the put_page operation
gbrain put research/<slug> # via the put_page operation
# 5. Cross-link entities mentioned (people, companies) per Iron Law.
```
+17 -2
View File
@@ -33,6 +33,21 @@ mutating: false
Answer questions using the brain's knowledge with 3-layer search and synthesis.
> **Memory verbs (MEMORY_VERBS v1, gbrain ≥ 0.43).** When connected to a brain
> over MCP, prefer the five frozen memory verbs for memory work — they carry
> provenance, evidence, and a server-enforced token budget:
> - **`recall(query | entity, budget_tokens)`** — the budget-packed memory read.
> Use it instead of bare `search` for "what do we know that we SAVED about X".
> - **`entity(name)`** — a zero-LLM person/company/project card (aliases,
> last-touched, open threads, top edges). Use it instead of `get_page` +
> `get_backlinks` when you just need the card.
> - **`synthesize(question)`** — the explicitly-expensive cross-page answer; the
> heavy version of `query`. Reach for it only when the answer must combine
> evidence across pages.
> Fall back to `search`/`query`/`get_page` when the verbs aren't on the surface
> (pre-0.43 servers; `--surface full` includes the verbs alongside every other
> op). See `docs/protocol/MEMORY_VERBS_v1.md`.
## Contract
This skill guarantees:
@@ -49,8 +64,8 @@ This skill guarantees:
- Semantic query for conceptual questions
- Structured queries (list by type, backlinks) for relational questions
2. **Execute searches:**
- Keyword search gbrain for FTS matches (search)
- Hybrid search gbrain for semantic+keyword with expansion (query)
- Cheap-hybrid search gbrain for exact tokens / known names (search)
- Full-hybrid search gbrain with multi-query expansion for concept questions (query)
- List pages in gbrain by type or check backlinks for structural queries
3. **Read top results.** Read the top 3-5 pages from gbrain to get full context.
4. **Synthesize answer** with citations. Every claim traces back to a specific page slug.
+9 -5
View File
@@ -11,7 +11,7 @@ tools:
- gbrain schema active
- gbrain schema use
- gbrain schema stats
- gbrain pages restore
- gbrain restore
- mcp:run_onboard
triggers:
- "unify my types"
@@ -90,9 +90,13 @@ The handler is PROTECTED (manual_only per D17) — autopilot will never auto-fir
```bash
gbrain jobs submit unify-types \
--allow-protected \
--params '{"target_pack":"gbrain-base-v2"}'
--params '{"target_pack":"gbrain-base-v2","apply":true}'
```
`apply` defaults to **false** (dry-run) per the handler contract, so
`"apply":true` is required here or the job reports success having retyped
nothing and left the active pack unflipped. Omit it to preview.
Watch progress per phase:
```bash
@@ -143,7 +147,7 @@ WHERE source_id = 'default' AND frontmatter->>'legacy_type' IS NOT NULL;
Page-to-alias and page-to-link source pages soft-delete with 72h TTL. Restore within that window:
```bash
gbrain pages restore <slug>
gbrain restore <slug>
```
Revert the active pack flip:
@@ -197,7 +201,7 @@ Outputs:
- Active pack flipped to `gbrain-base-v2` atomically at end of successful run.
Side effects:
- Source pages soft-deleted with 72h restore TTL (`gbrain pages restore <slug>`).
- Source pages soft-deleted with 72h restore TTL (`gbrain restore <slug>`).
- One-time cache invalidation on KNOBS_HASH_VERSION bump (5→6); self-healing in `cache.ttl_seconds`.
- Query-time `--type X` alias-expands via `expandTypeFilter` (D14 back-compat).
@@ -212,7 +216,7 @@ DON'T:
- Submit `unify-types` directly via the MCP `submit_job` op without `--allow-protected`. PROTECTED handlers require trusted local callers; remote MCP rejection is the intentional trust boundary.
- Edit `mapping_rules` in `gbrain-base-v2.yaml` to skip clusters you don't trust. Fork the pack instead (`gbrain schema fork`) so the source-of-truth migration stays consistent across brains.
- Run `unify-types` from inside an autopilot tick. The check is `manual_only` per D17 — autopilot deliberately never auto-fires it because pack upgrades are one-time consenting taxonomy decisions.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain pages restore <slug>` first if rollback is needed.
- Hard-delete soft-deleted source pages before the 72h restore window. Use `gbrain restore <slug>` first if rollback is needed.
- Assume `frontmatter.legacy_type` survives every roundtrip. The marker is canonical for the immediate post-migration window; downstream re-imports may overwrite it.
## Output Format
+130
View File
@@ -0,0 +1,130 @@
{
"RESOLVER.md": "ad9f85d0f953ad0ec80c70091cb31c7227a89e4a08c924454be1d61d4df5b65c",
"_AGENT_README.md": "3dd82df125ceb87bdbc1ad16be4306fb3ca6e3a491c70122f6c7724a862f0b21",
"_brain-filing-rules.json": "cf850df6a7425464c6d63b3ace71991cc93497fa0cc8cd21acd31883e17939c6",
"_brain-filing-rules.md": "2d2d75b7c76081c56f41b2c0a5a978c355ce957300f9b0a5575dc4079ef1f877",
"_friction-protocol.md": "1b6e7cfa58725a6a5dc2dc787242141bc33f5fde524540d85b14ec22266140f7",
"_output-rules.md": "239022bd9003b4d45870fea2ddb177a132f0684d898ed5075dc5dcb90cc56ab7",
"academic-verify/SKILL.md": "1c19e27e75249d869da428ce8d060075feef8fbbfe146af58b305d11a260ebbc",
"academic-verify/routing-eval.jsonl": "90d894a9829d9936e6ac7a6507e4de67ad26e46a1fe13b7a34e7dec1c0d887dd",
"archive-crawler/SKILL.md": "10c529797e1cba75da5bdb13bdd3609ae5f962f7dcc429c3b85d51c0032d9a26",
"archive-crawler/routing-eval.jsonl": "a90c607d69737adf58771d1b986c5a7d9d11f5303df7de4c7def74aa2c86a2f5",
"article-enrichment/SKILL.md": "fcdbce0f250aa2c38b86299dae510b8dc7f1e1cd854961cb8520c61642309549",
"article-enrichment/routing-eval.jsonl": "408fa8cc80caf1a2208afeb049e5ada69beb378f22d6b7114828201a4657e6e3",
"ask-user/SKILL.md": "a40f484721e548a3a14d4b33a4636111d92f99619ecc4e4ea54c3da3a15f8331",
"book-mirror/SKILL.md": "e8b8cc7a6eba4ecd302a840446b48c0e1aecc245e8f240e233c738daa8dff78a",
"book-mirror/routing-eval.jsonl": "79fd23642cfa37b1255a799907e71bb2904585cf79dd6a596e3a2f019787e54c",
"brain-ops/SKILL.md": "5f221e3de45845b050b90fac935ac70c55ed5148649fb47c4a30989c1d42c40a",
"brain-pdf/SKILL.md": "13c3e3162763a4503685db0a10663475d3687c4874b5f04d539af83a990f643e",
"brain-pdf/routing-eval.jsonl": "119e4fa113ea45783cee4499e63a729fdeecb4d9a45d47497754b4f5b21d0734",
"brain-taxonomist/SKILL.md": "dea4557b540868ec2c56bf43ee7f63c5d03a22d4047cd0dfbeaf19adef334f60",
"brain-taxonomist/routing-eval.jsonl": "8b485b3d735aace60be703854f0f2e9d97c52d52564efdaf7334a0c39e8d20ae",
"briefing/SKILL.md": "a661804c3eb2ce5bd4eb6f3e7106283046913945d9dba0b967bdc8483a58dd66",
"capture/SKILL.md": "98568ac96331f57397ea072749641d9748b1ce31e8b09d512b8db25c8fcda65f",
"citation-fixer/SKILL.md": "abdadbf0740a529b9c4f86f05bba416417624503fdcbc6054402d5546afd08b4",
"citation-fixer/routing-eval.jsonl": "52b23b71e66fdc18aee67d0576099b0c83997d648cf4ecf8fe7753b91b6c9c53",
"cold-start/SKILL.md": "a2c42dd7c4eceb7d3ce6449a414b417d55195aa445e723e6c798d90906cbf4e6",
"concept-synthesis/SKILL.md": "2bc060ae6d706c4e8e7d784cbe3e577b85e211a21c68cba094a1754d3f34436b",
"concept-synthesis/routing-eval.jsonl": "51d1da894158503ce18b892a34edd203f40732e79ac1c0e85141fd37e0b9922f",
"conventions/brain-first.md": "14370d89209c7d7e2673c6a4d4e7545fd3330f0744ab598170865a41ae20210b",
"conventions/brain-routing.md": "a8035f7dbadff0ea68b8babb8314b3d044cafbed8242dce5b931fa08b028fc45",
"conventions/calibration.md": "eda7ca76f80c8a17ae546110484389f805c5b21fc0a57f951bbe8b6abba26e03",
"conventions/cron-via-minions.md": "badb1cd6cd825d6f1ac0b6b28cc47e5d80facc783a3e59a14146ae901ee0f933",
"conventions/cross-modal.yaml": "c012c3d72614a87b1ee698173dce2a0fb0d057a54df7aab87993c4b07fff6280",
"conventions/model-routing.md": "fb7ae8746a578500d6789b68ff40049037aa4d337b65b42f7c1745ae7080c2db",
"conventions/quality.md": "8aa681001114689d34268ccadaf0e2ff07b8f68aa5987c093a8c4a7a744f12a6",
"conventions/salience-and-recency.md": "62b0b303bf48bef10adcf08d3b51f23d3b1f1187b3850476a4b5532c2ee88926",
"conventions/schema-evolution.md": "b5cdd5a17e43b4f2d0546ebf8c421cc0378cbcf74cb2fe97199e115ccb05f9c2",
"conventions/search-modes.md": "2a920225d1c95ea978fb1c77c5170f6a598377ab86fc0024b70962d5a84d54d0",
"conventions/subagent-routing.md": "59afd362ff0cbaf3a63586e97c53f68feb837a258a2bd43f9177af4e57d2b20e",
"conventions/test-before-bulk.md": "5073e5b93d570445f72f3c10ed3e6ec10c1fee7574ad73c2e2695993c850eca5",
"cron-scheduler/SKILL.md": "e3f9745c4f8e2dacba5b055f408b1ffe541b450aee6bbfe4773f7b424d90e04c",
"cross-modal-review/SKILL.md": "685233b1afd477e96697562c502233eea22eda5db8df116b81dd2fa78f01f80c",
"daily-task-manager/SKILL.md": "e616f74a6befffc7bb64c0e29b2d2caa37c51bc470bca96df70101c772429d83",
"daily-task-prep/SKILL.md": "9fe89f85fae139adac25c3bdc6f23bbf64239f3738a9e679c447e686175516f0",
"data-research/SKILL.md": "990ccec01a23d3e46c7b5abbd2650ac5507480b50f7ec4de8b466863b3b61cb6",
"eiirp/SKILL.md": "9d42d6a5f61bba30cb47400db58927c501cd89e5cc3241b5218d7962d1a68804",
"eiirp/routing-eval.jsonl": "416459ff68da2e5f5eb216a0c24368c8eebdf4edbbc540a28fe730ceeb4c9700",
"enrich/SKILL.md": "9988168348f6c3391d3aeec6621f9c99c8d75d1e1bfdd6c44d48e4c1067ab775",
"frontmatter-guard/SKILL.md": "5142ab53f5428ebc084ded78f1fb7eb4bfa386d276ee034bf4d57273256570c6",
"frontmatter-guard/routing-eval.jsonl": "243c28b04b557bac5360318c0f47bb6f1dd56d2f720aad3763b93fc0b2ea081e",
"functional-area-resolver/SKILL.md": "52df04bc4f8e678f931c3b2078b2126524e6d2d72676ad46b6b710d13271b46c",
"functional-area-resolver/routing-eval.jsonl": "f80674d915acdfe229046737a5b171da834be15ac6524b5a3fd18048e9b37028",
"gbrain-advisor/SKILL.md": "c15c7a88bee2c96733d718a168dd9afcb123b7b6b2c0014c5260d37c72e8736a",
"gbrain-upgrade/SKILL.md": "7cd05f43027fa20d56ed651f696288f9e4814b7bdf7b396f132fde7c96dac620",
"idea-ingest/SKILL.md": "e118bc32d5044a4a6fba4fed1911b8be50210ade14957e6cfe8e7ed26ff95298",
"idea-lineage/SKILL.md": "bbf37781d93b71ddc7909ecc5ab635872c874fb8591995dbf88b45ffeac6b1de",
"idea-lineage/routing-eval.jsonl": "ee2e00704b9accb7dd58bb8f126a3bc04a2c40be499180fa505dbf6d5061cd41",
"ingest/SKILL.md": "dc40ecc0072806fb8c7bb6ab9cf1f103842e05653eb55d67632d7e3ffc4dd7d2",
"install/SKILL.md": "881bd0a422f34c6df4642aae66c51e2a4cc18ad5ca6d0b52d44b4de93512a3c4",
"maintain/SKILL.md": "e80cf5bb170c979b773a67a18e0d880c086e0b983db9a576a207074b0112c4f8",
"manifest.json": "52b970cfc3ed340ee4f25323b6fadeb5d125741b02d1d2dc2f85c39f0c701258",
"media-ingest/SKILL.md": "33db12830ed31a4ff4a6a58c4f126bf2596ee28d54cee0027680c83fee648a20",
"meeting-ingestion/SKILL.md": "7767334c63ff3bd8e60cd4d7cd1d1b44f0d6b0a7e0ac411529a1d59cc0a7781b",
"migrate/SKILL.md": "442c117cfe50026a142ff4f934e489c56ee226d454b639d20bba3b111a86d8fd",
"migrations/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"migrations/v0.10.3.md": "1e12db6fdb88d21df479659e236bad111a041a205d47aa0894afde2f04903124",
"migrations/v0.11.0.md": "ac45c6144eeee5033f2b24308a5063ceced2be7831b04e48154aa0b6f8ff55d8",
"migrations/v0.12.0.md": "f30b452caed5ce2679cbd60dc2d0b9e60449bab20c3a3e0c895e6cd167644f44",
"migrations/v0.12.1.md": "a461d8fcd6b19dabe3a0ce849a54e3c577667ae2af41870c6f456c6195e76064",
"migrations/v0.13.0.md": "3bf61eb1da933e6c3c35d67d0ea6d9dada5d5f878bbd14020616d4e1fe48feb2",
"migrations/v0.14.0.md": "1025395f033b91480451aad6a98dc10f2c18bfacd1b1a4e2a9fb4dc99f8fb571",
"migrations/v0.15.2.md": "468a58e8a9f0f5c5867e0ed8eb9dd1bd60dbc8d66111647997ebd8d9d2119915",
"migrations/v0.17.0.md": "761dda3b6090bfe20c9b4354aff5248d5a9b6a79a24702ecc2b29873d4f5b413",
"migrations/v0.18.0.md": "94efc1d571070600a78aca13c2735866a987e6a8f188210b7062f81b54839986",
"migrations/v0.19.0.md": "c47584e8306680f21ddc7f21a4a89f2184da522acb177f0dd73a09f1cd77ae57",
"migrations/v0.21.0.md": "83569ce6fb2843f81964833affe5eae72c94a2983eba265c836aa05be7876957",
"migrations/v0.22.14.md": "8898c109716fbc48c058a44c606f50fb7c02be599541d367380aa2e63bffea9e",
"migrations/v0.22.4.md": "5f9b2ed86cb64d92416448e744ca6c4984b89728895cd4e363f9c76a860074d9",
"migrations/v0.23.0.md": "f0363274e5071fa35d0c60e3981c4caee0494c43adb254d65acf6b743d4762d5",
"migrations/v0.25.1.md": "981ef49abadf19a891cb68b655b9f35ee67d9f4ac232ceb29d83dfc3b1c9982e",
"migrations/v0.27.1.md": "719dd4ebcc7ac81413ee79af20f80de5141577e3860e5049a49fc8f6d09ddf89",
"migrations/v0.28.0.md": "21929a392af2e1f454272e80aade6c58c5f6ccd7253d24344d08c6b53bcdf4f9",
"migrations/v0.29.1.md": "7b85373a62a8500ee8c4238ffc9878318b44ee62543adb8b2462049c6c0cd1f8",
"migrations/v0.32.2.md": "101623fa48f946242d7edc8f126967822563c3fb30a6ad50d029894f5fe3499d",
"migrations/v0.32.6.md": "e9438a910afe8a1f7861c5222fd7db04a0f8d8b8283778476b07b7c3d539d3f0",
"migrations/v0.33.0.md": "11710cb11d6eb7dc3ea54b764e3c4a25f8679cf76590acd330f97bfa1c684945",
"migrations/v0.33.3.0.md": "188a03ca86a97a9aa697cbbc83cc8ca37843fab24db2bd82f1383c400173d5bd",
"migrations/v0.34.0.0.md": "d421c5ecff0765ac1de3592d3175734db7df52e8658ec101567779c7c56c2db2",
"migrations/v0.35.0.0.md": "0fc21dc0b098f87fff1ac79a669b00a3d69ab1510ebfc5eac4a66f5c6d783809",
"migrations/v0.35.7.0.md": "c6d4454bd39e2aa243b3b3d9bc72fe5a4fd25d097be7be2bb14b25604b5c2cc5",
"migrations/v0.36.2.0.md": "1b59328240ae19c5e7e8d3eafda245809cca1fea27146607334dbee53cbeb270",
"migrations/v0.36.5.0.md": "a01a722202dfc3c799693596750c8bee611fe4dafe3cb662f6b4cd0b635cb429",
"migrations/v0.40.3.0.md": "5f500f8c543c2b6f41778b0bd3beedada68f7284f7933ad8b769322b433a8fe9",
"migrations/v0.40.5.md": "b9837d52a030517698dfb31c439f562cde60a1015ae488dab09be2c16ff182e5",
"migrations/v0.41.11.0.md": "5c6873ab969d14def4a450d792f070f1259d08b3aca43bc7825d0a9114b2b36b",
"migrations/v0.5.0.md": "5e0dabc451595295c4d971e19bcb33c258a127223d25859d8321cb7e1ce60711",
"migrations/v0.7.0.md": "97c2740445a10b1c5c7123c17dbd625fa27a94095b85d27c2b278da756c4c59a",
"migrations/v0.8.0.md": "1919ff8b8f3680612ff888e7cfcc0d86ece5d5304ae19af4497bdf40b050561a",
"migrations/v0.8.1.md": "fad7341cfb5e02545fb8a23221d12ab395fc3d8db15d1d8ee8a18844aea6563a",
"migrations/v0.9.0.md": "773fab0a8d7f330576265a3f510c1f318f47789b6136c46d43e08121acbc20eb",
"migrations/v0.9.1.md": "75761bad6c0ad37b69ec8197c6a678bb6a1484f9a76e4b70f2d1e86dc80102b3",
"minion-orchestrator/SKILL.md": "669e23f485561cf6fef16d445dfb8a6d05f76127d0ebf9feb17534af8843df5f",
"perplexity-research/SKILL.md": "c25f5c471cbe3c6e0f975d8397e8382b00a85f8aa75302231d53c52855369e97",
"perplexity-research/routing-eval.jsonl": "f1a40d87e710d5d2acd602a372d83f46c95da022b6e635228fffeaacb3bb2b27",
"publish/SKILL.md": "e06b609db780a3cc93a1755a87b30ff08ffdc0fdbc834c1422b2ad2489b57497",
"query/SKILL.md": "8672fb9c9315f01274b1a7d3ad35f903df9705b2e806e2fa4c6add027ecef96f",
"query/routing-eval.jsonl": "74f5a91e52fabc54e0e9403fa17db87ee26bb7ebb8ae8005148c51142abc62fe",
"repo-architecture/SKILL.md": "4ec2b8f45d168aaa55f17ecd1ed404ab04217a75c2317f0710c71705846f5394",
"reports/SKILL.md": "5dc190a0c3a2ee518254e8b596418dbe19ff389ea5ec8c8d30fcb0dfef4d0ed5",
"schema-author/SKILL.md": "4da9a472c966f8e4fb43d97a3a3608c67ec3d43da6b8c626ee0c30a4e70da26f",
"schema-unify/SKILL.md": "e1d50a54a6ff29434d38a841a572e815d236a8167d459609ae697e548770f500",
"setup/SKILL.md": "4a47a5f6ee99ac8a2649a304abb261fd34810cf6185fd7d6a53e1b6c4cbc0764",
"signal-detector/SKILL.md": "64e4547f5a8624c53d875001b423d240ec73ee9fd026a96c7b799d287c5fb6e4",
"skill-creator/SKILL.md": "4a11f8935d4214b21b4664a5c0c03149733731020ec0d5d09dc9fd8c40bd92f6",
"skill-optimizer/SKILL.md": "ba3028c7351dec3e644a7114e08e59cae7c60dc367dc4fd10560265a162baa90",
"skill-optimizer/routing-eval.jsonl": "48f7fc04414b194ee8674577c3e58e03ebd7f74d836766bd16cb5abfd4effb76",
"skill-optimizer/skillopt-benchmark.jsonl": "5552457d6eaa32486b79796d12fcbe2c078b0c53d7fbdaf582f0fd1a17e9a381",
"skillify/SKILL.md": "a154e43409458136e1e1cea084aa973e4cf46cd8450c6be8e9891a4a3ddf6146",
"skillpack-check/SKILL.md": "3f347ec8b498530a662be212d05f4cd06b205bce5795c2231b6a4cecef149ea0",
"skillpack-harvest/SKILL.md": "3c4c591b33f03a5ccf11ca0ddde56b54fba541efef0d590b6d435687733182d7",
"skillpack-harvest/routing-eval.jsonl": "cb4783288e95af3132b32ecb40a54cffc095f57b36240f96a25c2d2adf5e68c6",
"smoke-test/SKILL.md": "f2f2172d41e63e288095451132a0c56848ccc34101d265b330b6cf00e5769f5d",
"soul-audit/SKILL.md": "7f162dddcc511e97a24db3a46136295fcbda76023019ad8994546744d7b8eb0a",
"strategic-reading/SKILL.md": "5be656c39c830153ec7c2f328dc8bdeac05c1412b01b3415de6b5b008926e7a2",
"strategic-reading/routing-eval.jsonl": "eb0fc239c93aac53cf7d856190967bb65eaf970b4fe8bd0aad1c87495fbb8ccd",
"testing/SKILL.md": "f1846ba7c35076d910744a6b867c9ee850c1895f75a104a319c9b7d18b1d5e90",
"voice-note-ingest/SKILL.md": "145c02e636430abba5648e026aef77606d6ba2594285fbfee4f4dd4253cf833a",
"voice-note-ingest/routing-eval.jsonl": "374aaec16fbde336d1e376edce89e51adc4ecaa93c13fb4a69b967ba299b8742",
"webhook-transforms/SKILL.md": "b774293297af4d513c7efa92a79cf65b8438a714adafc16802b492613e679d17"
}
+6 -4
View File
@@ -43,8 +43,9 @@ The Analysis section can interpret; the transcript section is sacred.
The user sends an audio or voice message via any channel (Telegram, voice
memo upload, openclaw audio attachment). The host agent typically provides
the transcript text. If not, transcribe via `gbrain transcription` (Groq
Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
the transcript text. If not, transcribe it with your host's transcription
tool (Groq Whisper is fast and cheap; OpenAI Whisper works too — segment
audio > 25MB via ffmpeg first).
## The pipeline
@@ -52,8 +53,9 @@ Whisper by default; OpenAI fallback for audio > 25MB segmented via ffmpeg).
1. STORE → Upload original audio to gbrain storage backend
(S3 / Supabase Storage / local — pluggable per
src/core/storage.ts).
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR call
gbrain transcription if no transcript was supplied.
2. TRANSCRIBE → Use the agent-provided transcript verbatim, OR
transcribe the audio yourself (see "When to invoke")
if no transcript was supplied.
3. ROUTE → Apply the decision tree (below) to find the right
destination directory.
4. WRITE → Create / update the destination brain page; preserve the
+398 -11
View File
@@ -30,9 +30,11 @@ import type { Operation, OperationContext } from './core/operations.ts';
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
import { serializeMarkdown } from './core/markdown.ts';
import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts';
import { conceptNudge } from './core/search/query-intent.ts';
import type { CliOptions } from './core/cli-options.ts';
import { callRemoteTool, RemoteMcpError, unpackToolResult } from './core/mcp-client.ts';
import { maybePromptForUpgrade } from './core/thin-client-upgrade-prompt.ts';
import { CLI_FLAG_REGISTRY } from './core/cli-flag-registry.generated.ts';
import { VERSION } from './version.ts';
// Build CLI name -> operation lookup
@@ -54,13 +56,29 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
return typeof value === 'bigint' ? value.toString() : value;
}
// ENG-2 renderer parity: round-trip a local-engine op's return value so
// renderers see the same shape the routed path produces. Bigint-safe via
// bigintToStringReplacer. Exported for tests (same import-safety contract as
// cliAliases/formatResult). (#2450)
export function normalizeLocalResult(rawResult: unknown): unknown {
return JSON.parse(JSON.stringify(rawResult, bigintToStringReplacer));
}
// CLI-only commands that bypass the operation layer
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'backfill']);
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'pglite-repair', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'protocol', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill',
// v0.42.58 (#2035 class, caught by the handleCliOnly reachability sweep):
// full handler at `case 'notability-eval'` but never dispatchable.
'notability-eval']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
const CLI_ONLY_SELF_HELP = new Set([
'upgrade', 'post-upgrade', 'check-update',
// #3502 sweep: pages + bench print their own usage (pages.ts printHelp,
// bench-publish.ts printHelp). Both were documented but undispatchable —
// `pages` had a live handleCliOnly case but was missing from CLI_ONLY
// (the #2035 calibration bug class); `bench` was never wired at all.
'pages', 'bench',
'embed', 'config',
'skillpack', 'skillpack-check',
'integrations', 'friction',
@@ -88,11 +106,17 @@ const CLI_ONLY_SELF_HELP = new Set([
// short-circuit fired before runSync could print its own usage block.
// Adding `sync` here routes `gbrain sync --help` into runSync.
'sync',
// #3834: extract ships detailed help for its mode-specific flags. Keep the
// generic CLI-only stub from hiding that contract.
'extract',
// v0.37 fix wave (deferred TODO, shipped): reinit-pglite has its
// own --help in runReinitPglite. Routing through SELF_HELP avoids
// the generic short-circuit so the destructive-action warning text
// reaches the user.
'reinit-pglite',
// WAL-repair wave: pglite-repair ships its own --help with the
// dry-run/repair semantics + the un-checkpointed-tail caveat.
'pglite-repair',
// v0.40.6.0 Schema Cathedral v3 — `gbrain schema --help` should hit
// schema.ts printHelp() with the full 22+ verb taxonomy, not the
// generic short-circuit's one-line stub.
@@ -107,6 +131,14 @@ const CLI_ONLY_SELF_HELP = new Set([
// `gbrain connect --help` prints its own usage (flags + examples) from
// runConnect; route around the generic one-line short-circuit.
'connect',
// MEMORY_VERBS v1 (Cathedral 1): protocol ships its own detailed HELP
// (subcommands, conformance targets, the cost-gated --synthesize flag).
'protocol',
// `gbrain init --help` prints its own usage from runInit; route around the
// generic one-line short-circuit (matches `connect`). Without this, `init`
// is in CLI_ONLY but not CLI_ONLY_SELF_HELP, so the dispatcher's generic
// short-circuit fires and the printInitHelp() guard in init.ts is dead code.
'init',
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
// (engine transfer) keeps its own dispatch too.
@@ -312,6 +344,30 @@ async function main() {
}
}
// #2185: strict unknown-flag validation — pre-dispatch, pre-engine. A flag
// no handler consults (the repro: `init --migrate-only --dry-run` applying
// REAL migrations while the user asked for a rehearsal) fails loud here
// instead of silently doing the destructive thing. Runs after the --help
// short-circuit so `gbrain x --help` never errors; runs before any dispatch
// or engine connect so the error is instant and side-effect-free.
{
const unknown = validateCommandFlags(command, subArgs);
if (unknown) {
// Message contract shared with init.ts's in-handler check (which this
// pre-dispatch validator now reaches first): lowercase 'unknown flag'
// on stderr; --json callers get the structured error on stdout with
// reason 'invalid_flag' (pinned by test/init-migrate-only.test.ts).
const message = `unknown flag ${unknown} for 'gbrain ${command}'`;
// Both --json spellings get the structured envelope (--json=false opts out).
if (subArgs.some(a => a === '--json' || (a.startsWith('--json=') && a !== '--json=false'))) {
process.stdout.write(JSON.stringify({ status: 'error', reason: 'invalid_flag', message }) + '\n');
}
console.error(`gbrain ${command}: ${message}`);
console.error(`Run: gbrain ${command} --help`);
process.exit(1);
}
}
// DB-free durability pull (v0.42.44 D2): the harden cron calls
// `gbrain sources pull --path <dir>` every ~30 min. It must NOT open PGLite
// (a live long-lived session holds the single-writer lock), so handle it
@@ -394,6 +450,18 @@ async function main() {
if (op.localOnly) {
refuseThinClient(command, cfgPre!.remote_mcp!.mcp_url);
}
// A thin client has no local mounts — an explicit --brain cannot be
// honored and must not be silently dropped (same loud-beats-silent rule
// as applyThinClientSourceScope's --source refusal). Ambient tiers
// (GBRAIN_BRAIN_ID / .gbrain-mount) are ignored here, matching the
// source axis's ambient-with-nowhere-to-send behavior.
if (cliOpts.brain) {
console.error(
'--brain is not supported on a thin-client install: the remote server is a single brain. ' +
'Remove the flag, or run from a machine with local mounts (gbrain mounts list).',
);
process.exit(1);
}
// #2098: the local path resolves --source / GBRAIN_SOURCE / .gbrain-source
// inside makeContext (ctx.sourceId), which this route never reaches — so
// scope must be mapped onto the op's source_id wire param before the call.
@@ -484,9 +552,10 @@ async function main() {
// path's return value so renderers see the same shape they'd see on the
// routed path. Date → ISO string; bigint → string (postgres.js shape);
// Buffer → object. Microsecond-cost; eliminates a whole drift bug class.
const result = JSON.parse(JSON.stringify(rawResult, bigintToStringReplacer));
const output = formatResult(op.name, result);
const result = normalizeLocalResult(rawResult);
const output = formatResult(op.name, result, params);
if (output) process.stdout.write(output);
maybePrintConceptNudge(op.name, params);
} catch (e: unknown) {
// v0.42.20.0 (codex D4): on error, set exitCode + return so the `finally`
// STILL runs (drains every background-work sink + disconnects). A bare
@@ -568,8 +637,9 @@ async function runThinClientRouted(
signal: sigintController.signal,
});
const result = unpackToolResult(raw);
const output = formatResult(op.name, result);
const output = formatResult(op.name, result, params);
if (output) process.stdout.write(output);
maybePrintConceptNudge(op.name, params);
} catch (e: unknown) {
if (e instanceof RemoteMcpError) {
const url = cfg.remote_mcp!.mcp_url;
@@ -786,6 +856,29 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg.startsWith('--')) {
// #2185: `--key=value` inline form. Pre-fix this parsed as junk key
// 'key=value' and consumed the NEXT token as its value, corrupting
// positional parsing. Recognized here so the strict-flag validator and
// the parser agree on the idiom.
const eq = arg.indexOf('=');
if (eq > 2) {
const key = arg.slice(2, eq).replace(/-/g, '_');
// CLI-local booleans: `--json=<v>` / `--dry-run=<v>` must parse as
// booleans, not fall through to the junk-key path (which would
// consume the NEXT token as a value and corrupt positional parsing).
if (key === 'json' || key === 'dry_run') {
params[key] = arg.slice(eq + 1) !== 'false';
continue;
}
const def = op.params[key];
if (def) {
const raw = arg.slice(eq + 1);
params[key] = def.type === 'boolean' ? raw !== 'false'
: def.type === 'number' ? Number(raw)
: raw;
continue;
}
}
if (arg.startsWith('--no-')) {
const positiveKey = arg.slice(5).replace(/-/g, '_');
const positiveDef = op.params[positiveKey];
@@ -798,6 +891,14 @@ export function parseOpArgs(op: Operation, args: string[]): Record<string, unkno
const paramDef = op.params[key];
if (paramDef?.type === 'boolean') {
params[key] = true;
} else if (key === 'json' || key === 'dry_run') {
// CLI-local booleans, intentionally NOT on the operation contract
// exposed over MCP/tools: json is the formatter flag; dry_run feeds
// makeContext's ctx.dryRun. Both must never consume a value token —
// pre-fix, `gbrain delete x --dry-run` (trailing) set NOTHING, so
// ctx.dryRun stayed false and the REAL delete ran despite the
// rehearsal request (the resurrected #2185 class the red team caught).
params[key] = true;
} else if (i + 1 < args.length) {
params[key] = args[++i];
if (paramDef?.type === 'number') params[key] = Number(params[key]);
@@ -959,6 +1060,112 @@ export function applyThinClientSourceScope(
}
// Exported for tests (same import-safety contract as applyThinClientSourceScope).
// ─────────────────────────────────────────────────────────────────
// #2185 — strict unknown-flag validation (pre-dispatch, pre-engine).
// A flag no handler consults must fail loud instead of silently doing the
// destructive thing (`init --migrate-only --dry-run` applied REAL migrations
// while the user asked for a rehearsal). Two lanes:
// - op commands: legal flags derive from the operation contract
// (op.params) + the CLI-local formatter flags, mirroring parseOpArgs's
// traversal so values that begin with '--' are never misread.
// - CLI_ONLY commands: legal flags come from the generated
// CLI_FLAG_REGISTRY (scripts/generate-flag-registry.ts scans each
// command's source; freshness + coverage pinned by
// test/cli-flag-validation.test.ts).
// Everything after a literal `--` is passthrough and never validated.
// ─────────────────────────────────────────────────────────────────
// Exempt by contract, not oversight:
// - call: the generic op invoker — arbitrary --param names are its interface.
// - config: `config set <key> <value>` values are arbitrary strings.
// - jobs submit: job payloads carry handler-defined params (shell lane incl.).
function flagValidationExempt(command: string, subArgs: string[]): boolean {
return command === 'call' || command === 'config'
|| (command === 'jobs' && subArgs[0] === 'submit');
}
/** Returns the first unknown flag (e.g. '--dry-run') or null when clean. */
export function validateCommandFlags(command: string, subArgs: string[]): string | null {
if (flagValidationExempt(command, subArgs)) return null;
// Lane order MUST mirror dispatch order (CLI_ONLY first): commands that are
// BOTH an op and a CLI_ONLY member (think, salience, anomalies) dispatch to
// handleCliOnly, whose handlers parse flags the op contract doesn't declare
// (`salience --kind`, `think --with-calibration`) — validating those
// against op.params rejected documented invocations.
if (CLI_ONLY.has(command)) {
const legal = CLI_FLAG_REGISTRY[command];
// Registry drift fails OPEN at runtime (never brick a command); the
// drift-guard test fails the build instead.
if (!legal) return null;
return findUnknownFlag(subArgs, new Set(legal));
}
const op = cliOps.get(command) ?? cliAliases.get(command);
if (op) return findUnknownOpFlag(op, subArgs);
return null; // unknown command — the dispatcher's own error handles it
}
/** CLI_ONLY lane: token scan against the generated legal set. */
export function findUnknownFlag(args: string[], legal: ReadonlySet<string>): string | null {
for (const a of args) {
if (a === '--') break;
const m = /^--([a-z0-9][a-z0-9-]*)(?:=.*)?$/i.exec(a);
if (!m) continue;
// Casing typo = unknown flag: every handler in the repo is
// case-sensitive-lowercase, so `--MIGRATE-ONLY` passing validation would
// just be silently ignored downstream — the exact class this validator
// exists to kill.
if (/[A-Z]/.test(m[1])) return `--${m[1]}`;
const name = `--${m[1]}`;
if (legal.has(name)) continue;
// --no-<flag> negation of a known flag is legal.
if (name.startsWith('--no-') && legal.has(`--${name.slice(5)}`)) continue;
return name;
}
return null;
}
/** Op lane: mirrors parseOpArgs so flag VALUES starting with '--' are skipped. */
export function findUnknownOpFlag(op: Operation, args: string[]): string | null {
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--') break;
const m = /^--([a-z0-9][a-z0-9-]*)(?:=(.*))?$/i.exec(a);
if (!m) continue;
// Casing typo = unknown flag (see findUnknownFlag).
if (/[A-Z]/.test(m[1])) return `--${m[1]}`;
const rawKey = m[1];
// CLI-local flags consumed OUTSIDE the op contract (never wire params):
// json/explain — formatter flags; help — short-circuits pre-dispatch;
// source — makeContext's 6-tier source resolution (deleted before wire);
// dry-run — makeContext's ctx.dryRun projection.
// Pre-fix, rejecting these broke documented invocations
// (`gbrain search "x" --source y`, `gbrain put x --dry-run`).
if (rawKey === 'json') continue;
if ((rawKey === 'explain' || rawKey === 'help') && m[2] === undefined) continue;
if (rawKey === 'source' || rawKey === 'dry-run') {
// Non-boolean-style CLI-locals consume the next token as their value
// in parseOpArgs (source does; dry-run is boolean-read) — mirror the
// parser: source consumes a value when not inline-`=`.
if (rawKey === 'source' && m[2] === undefined) i++;
continue;
}
if (rawKey.startsWith('no-')) {
const positive = rawKey.slice(3).replace(/-/g, '_');
if (op.params[positive]?.type === 'boolean') continue;
}
const key = rawKey.replace(/-/g, '_');
const paramDef = op.params[key];
if (paramDef) {
// Non-boolean flags consume the next token as their value unless
// provided inline via `=` — exactly like parseOpArgs.
if (paramDef.type !== 'boolean' && m[2] === undefined) i++;
continue;
}
return `--${rawKey}`;
}
return null;
}
export async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
@@ -1004,12 +1211,34 @@ export async function makeContext(engine: BrainEngine, params: Record<string, un
// table). Matches dispatch.ts's auto-fill so the contract holds across
// every transport.
sourceId: sourceId ?? 'default',
// Brain axis: the id connectEngine resolved for this process. Module
// state, NEVER params — caller-supplied params.brain must not select a
// brain (that would be an untrusted-caller cross-brain hole over MCP).
brainId: activeBrainId,
...(localFederated ? { localFederatedSourceIds: localFederated } : {}),
};
}
// Exported for tests (same import-safety contract as cliAliases/printOpHelp).
export function formatResult(opName: string, result: unknown): string {
/**
* #2416: hint-only steering a concept-shaped `search` gets a one-line
* stderr nudge toward `query`. Never fires for other ops, never reroutes
* (search stays the cheap hot path), and honors --quiet the same silence
* discipline as the identity banner. Called from BOTH result paths (local
* engine + thin-client routed); formatResult can't host this because it
* never sees the query text.
*/
export function maybePrintConceptNudge(opName: string, params: Record<string, unknown>): void {
if (opName !== 'search' || getCliOptions().quiet) return;
const nudge = conceptNudge(String(params.query ?? ''));
if (nudge) process.stderr.write(nudge + '\n');
}
export function formatResult(
opName: string,
result: unknown,
params: Record<string, unknown> = {},
): string {
switch (opName) {
case 'volunteer_context': {
const r = result as any;
@@ -1048,6 +1277,7 @@ export function formatResult(opName: string, result: unknown): string {
case 'search':
case 'query': {
const results = result as any[];
if (params.json === true) return JSON.stringify(results, null, 2) + '\n';
if (results.length === 0) return 'No results.\n';
// v0.40.4 — --explain switches to per-stage attribution formatter.
// Reads CliOptions.explain via the module-level singleton.
@@ -1133,9 +1363,70 @@ export function formatResult(opName: string, result: unknown): string {
`#${v.id} ${v.snapshot_at?.toString().slice(0, 19) || '?'} ${v.compiled_truth?.slice(0, 60) || ''}...`,
).join('\n') + '\n';
}
// MEMORY_VERBS v1 [F-E]: human-readable by default; trailing `--json`
// escapes to the raw envelope (parseOpArgs ignores an unmatched trailing
// flag, so the argv probe is safe).
case 'remember': {
if (process.argv.includes('--json')) break;
const r = result as any;
if (r.dry_run) return `[dry-run] would remember: ${r.fact}\n`;
const lines = [r.status_text || `${r.status} (fact #${r.id})`];
if (r.entity_slug) lines.push(` entity: ${r.entity_slug}`);
if (r.valid_until) lines.push(` expires: ${r.valid_until}`);
if (r.degraded_dedup) lines.push(' note: no embedding provider — duplicate detection degraded');
return lines.join('\n') + '\n';
}
case 'entity': {
if (process.argv.includes('--json')) break;
const r = result as any;
if (!r.found) {
const lines = [`No entity found. (${r.latency_ms}ms)`];
if (Array.isArray(r.suggestions) && r.suggestions.length) {
lines.push('Did you mean:');
for (const s of r.suggestions) lines.push(` ${s.slug}${s.title} [${s.create_safety}]`);
}
return lines.join('\n') + '\n';
}
const c = r.card;
const lines = [`${c.entity.title} (${c.entity.slug})${c.entity.type ? ` [${c.entity.type}]` : ''} (${r.latency_ms}ms)`];
if (c.summary) lines.push(` ${c.summary}`);
if (c.aka?.length) lines.push(` aka: ${c.aka.join(', ')}`);
const lt = c.last_touched || {};
const touched = lt.updated_at || lt.last_retrieved_at || lt.last_timeline_date;
if (touched) lines.push(` last touched: ${String(touched).slice(0, 10)}`);
if (c.open_threads?.length) {
lines.push(' open threads:');
for (const t of c.open_threads) lines.push(` [${t.kind}] ${t.text}${t.date ? ` (${String(t.date).slice(0, 10)})` : ''}`);
}
if (c.edges?.length) {
lines.push(' edges:');
for (const e of c.edges) lines.push(` ${e.direction === 'out' ? '→' : '←'} ${e.type} ${e.slug}`);
}
lines.push(` backlinks: ${c.backlink_count} | active facts: ${c.active_fact_count}`);
if (Array.isArray(r.suggestions) && r.suggestions.length) {
lines.push(' other matches:');
for (const s of r.suggestions) lines.push(` ${s.slug}${s.title}`);
}
return lines.join('\n') + '\n';
}
case 'synthesize': {
if (process.argv.includes('--json')) break;
const r = result as any;
const lines = [r.answer || '(no answer)'];
if (Array.isArray(r.sources) && r.sources.length) lines.push('', `sources: ${r.sources.join(', ')}`);
if (Array.isArray(r.gaps) && r.gaps.length) lines.push(`gaps: ${r.gaps.join('; ')}`);
const cost = r.cost || {};
const tok = cost.input_tokens != null ? `${cost.input_tokens} in / ${cost.output_tokens} out` : 'tokens n/a';
const usd = cost.usd_estimate != null ? ` (~$${Number(cost.usd_estimate).toFixed(4)})` : '';
lines.push(`cost: ${cost.model}${tok}${usd}`);
return lines.join('\n') + '\n';
}
default:
return JSON.stringify(result, null, 2) + '\n';
// bigintToStringReplacer keeps this fallback renderer crash-proof even
// if a future caller hands it a not-yet-normalized result. (#2450)
return JSON.stringify(result, bigintToStringReplacer, 2) + '\n';
}
return JSON.stringify(result, null, 2) + '\n';
}
/**
@@ -1260,11 +1551,33 @@ async function handleCliOnly(command: string, args: string[]) {
await runSchema(args);
return;
}
// MEMORY_VERBS v1 (Cathedral 1): protocol introspection + conformance +
// local usage stats. No pre-bound engine — conformance spawns its own
// server; stats reads the local JSONL sidecar.
if (command === 'protocol') {
const { runProtocol } = await import('./commands/protocol.ts');
await runProtocol(args);
return;
}
if (command === 'init') {
const { runInit } = await import('./commands/init.ts');
await runInit(args);
return;
}
if (command === 'bench') {
// #3502 sweep: `gbrain bench publish` was documented (docs/eval-bench.md,
// KEY_FILES.md, and eval-gate's own --help text) but never dispatched —
// the promised-but-unwired class retrieval-upgrade (#3390) fixed before.
// Pure file-in/file-out (NDJSON → baseline); no DB, no engine.
if (args[0] === 'publish') {
const { runBenchPublish } = await import('./commands/bench-publish.ts');
await runBenchPublish(args.slice(1));
return;
}
console.error('Usage: gbrain bench publish --from <captured.ndjson> --to <X.baseline.ndjson> [flags]');
console.error('Run `gbrain bench publish --help` for the full flag list.');
process.exit(args[0] === '--help' || args[0] === '-h' ? 0 : 2);
}
// v0.37 fix wave (deferred TODO, shipped): one-command wipe-and-reinit.
// Spawns its own engine internally so no pre-bound engine needed.
if (command === 'reinit-pglite') {
@@ -1272,6 +1585,13 @@ async function handleCliOnly(command: string, args: string[]) {
await runReinitPglite(args);
return;
}
// WAL-repair wave (#223/#1670/#2575): in-place torn-WAL recovery. Never
// connects an engine — the whole point is that the DB won't open.
if (command === 'pglite-repair') {
const { runPgliteRepair } = await import('./commands/pglite-repair.ts');
setCliExitVerdict(await runPgliteRepair(args));
return;
}
if (command === 'auth') {
const { runAuth } = await import('./commands/auth.ts');
await runAuth(args);
@@ -1677,6 +1997,14 @@ async function handleCliOnly(command: string, args: string[]) {
return;
}
// #3834: extract help is engine-independent and must work on a fresh
// install before a brain has been configured.
if (command === 'extract' && (args.includes('--help') || args.includes('-h'))) {
const { runExtract } = await import('./commands/extract.ts');
await runExtract(null as never, args);
return;
}
// v0.39.3.0 WARN-5: same pattern for `capture --help`. CLI_ONLY_SELF_HELP
// now includes 'capture' so the generic short-circuit at :101 stays out
// of the way, but the dispatch case at :1229 still needs an engine. The
@@ -1827,7 +2155,14 @@ async function handleCliOnly(command: string, args: string[]) {
}
case 'embed': {
const { runEmbed } = await import('./commands/embed.ts');
await runEmbed(engine, args);
// #3037: mirror the `import` case above — the CLI was discarding the
// result, so a run where every chunk failed to embed still exited 0
// and cron/CI/health gates read total silence as success. Surface
// non-zero on failures > 0. (undefined = backgrounded via --background.)
const embedResult = await runEmbed(engine, args);
if (embedResult && embedResult.failures > 0) {
setCliExitVerdict(1);
}
break;
}
case 'serve': {
@@ -2368,7 +2703,52 @@ async function dispatchReadOnlyCommand(engine: BrainEngine, command: string, arg
import { buildGatewayConfig } from './core/ai/build-gateway-config.ts';
export { buildGatewayConfig };
/**
* Which brain this process's engine targets. Set by connectEngine after brain
* resolution; read by makeContext so ctx.brainId carries the audit id. Never
* derived from op params an untrusted caller must not be able to name a
* brain (same fail-closed shape as the #3524 remote source sentinel).
*/
let activeBrainId: string = 'host';
/**
* Connect to a mounted brain (brain axis, non-host). Routes through
* BrainRegistry so:
* - an unknown/disabled mount id throws UnknownBrainError. Fail-closed:
* the pre-fix CLI silently fell back to the host brain, returning
* confident wrong answers (mirror of #3524's explicit --source decision);
* - postgres mounts get a per-instance pool, never the db.ts singleton;
* - NO migrations run against the mount schema is the publisher's job
* (same decision as BrainRegistry.initMountBrain). Write access control
* is the mount's own DB credential grants: a read-only role rejects
* writes at the database; gbrain does not re-implement that client-side.
* The AI gateway still configures from the HOST config (the caller's API
* keys + model tiers) embedding/expansion spend stays the caller's, and
* a mount's DB-plane model config is never merged into the caller's gateway.
*/
async function connectMountEngine(brainId: string): Promise<BrainEngine> {
const config = loadConfig();
if (config) {
const { configureGateway } = await import('./core/ai/gateway.ts');
configureGateway(buildGatewayConfig(config));
}
const { loadRegistry } = await import('./core/brain-registry.ts');
const handle = await loadRegistry().getBrain(brainId);
activeBrainId = brainId;
return handle.engine;
}
async function connectEngine(opts?: { probeOnly?: boolean }): Promise<BrainEngine> {
// Brain axis: resolve WHICH DATABASE this invocation targets before touching
// the host engine. --brain (global flag) / GBRAIN_BRAIN_ID / .gbrain-mount /
// mount-path-prefix resolve via the canonical 6-tier chain — the mirror of
// the source axis in makeContext. connectEngine is the single choke point
// every local CLI command routes through (shared ops, CLI-only commands,
// and the search-dashboard path), so routing lands here once.
const { resolveBrainId } = await import('./core/brain-resolver.ts');
const brainId = resolveBrainId(getCliOptions().brain);
if (brainId !== 'host') return connectMountEngine(brainId);
const config = loadConfig();
if (!config) {
console.error('No brain configured. Run: gbrain init');
@@ -2561,10 +2941,13 @@ TIMELINE
TOOLS
extract <links|timeline|all> Extract links/timeline (idempotent)
[--source fs|db] fs (default) walks .md files; db iterates engine pages
[--dir <brain>] brain dir for fs source
[--type T] [--since DATE] filters (db source)
[--dry-run] [--json]
[--source fs|db] [--source-id ID] [--dir <brain>]
[--type T] [--since DATE] [--include-frontmatter]
[--workers N|--concurrency N] [--dry-run] [--json]
extract links --by-mention [--ner] --source db
extract timeline --from-meetings [--infer-dates] --source db
extract --stale [--source-id ID] [--catch-up] [--dry-run] [--json]
extract --explain <kind> [--json] Full details: gbrain extract --help
publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256)
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
@@ -2634,9 +3017,13 @@ ADMIN
features [--json] [--auto-fix] Scan usage + recommend unused features
autopilot [--repo] [--interval N] Self-maintaining brain daemon
config [show|get|set] <key> [val] Brain config
protocol [conformance|stats] MEMORY_VERBS v1: schemas, conformance
certification, local usage stats + TTHW
storage status [--repo <path>] Storage tier status and health
[--json] (git-tracked vs supabase-only)
serve MCP server (stdio)
--surface verbs|full Tool surface: the 5 memory verbs only, or
every op (default full; verbs = quickstart)
serve --http [--port N] HTTP MCP server with OAuth 2.1
--token-ttl N Access token TTL in seconds (default: 3600)
--enable-dcr Enable Dynamic Client Registration (DCR clients default to authorization_code)
+60 -12
View File
@@ -108,7 +108,7 @@ Flags:
Exit codes:
0 Success (including "nothing to do").
1 An orchestrator failed.
1 An orchestrator failed, or schema migrations are pending (re-run with --yes).
2 Invalid arguments.
`);
}
@@ -260,6 +260,41 @@ function printDryRun(plan: Plan, installed: string): void {
}
}
/**
* #1530: schema-drift pre-flight resolution. When the schema version is
* behind, `--yes`/`--non-interactive` runs the schema migrations right there
* (the engine is already connected); interactive runs warn and return true so
* the caller exits non-zero instead of claiming "All migrations up to date".
* All output goes to stderr (migrations never print to stdout).
*
* Returns true when the schema is STILL behind after this call.
*/
async function resolveSchemaBehind(opts: {
schemaVer: number;
latest: number;
autoApply: boolean;
run: () => Promise<{ applied: number; current: number }>;
}): Promise<boolean> {
const { schemaVer, latest, autoApply, run } = opts;
if (schemaVer >= latest) return false;
if (autoApply) {
console.error(`Schema version ${schemaVer} is behind latest ${latest}; running schema migrations...`);
try {
const result = await run();
console.error(`Applied ${result.applied} schema migration(s); now at v${result.current}.`);
return false;
} catch (err) {
console.error(`Schema migration failed: ${err instanceof Error ? err.message : String(err)}`);
return true;
}
}
console.warn(
`\n⚠️ Schema version ${schemaVer} is behind latest ${latest}.\n` +
` Run \`gbrain apply-migrations --yes\` to apply now, or \`gbrain init --migrate-only\`.\n`,
);
return true;
}
function orchestratorOptsFrom(cli: ApplyMigrationsArgs): OrchestratorOpts {
return {
yes: cli.yes || cli.nonInteractive,
@@ -355,10 +390,13 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
if (cli.forceAll) return; // both surfaces flushed
}
// Pre-flight: warn if schema migrations (migrate.ts) are behind.
// apply-migrations runs orchestrator migrations only; schema migrations
// run via connectEngine() / initSchema(). Users often expect this CLI
// to handle everything (Issue 1 from v0.18.0 field report).
// Pre-flight: detect schema migrations (migrate.ts) being behind.
// apply-migrations historically ran orchestrator migrations only; schema
// migrations run via connectEngine() / initSchema(). Users expect this CLI
// to handle everything (Issue 1 from v0.18.0 field report; #1530). With
// --yes/--non-interactive we apply them here; otherwise we warn and make
// sure the run does NOT report "All migrations up to date" with exit 0.
let schemaBehind = false;
try {
const { LATEST_VERSION } = await import('../core/migrate.ts');
const { loadConfig: lc, toEngineConfig } = await import('../core/config.ts');
@@ -378,14 +416,16 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
await eng.connect(toEngineConfig(cfg));
const verStr = await eng.getConfig('version');
const schemaVer = parseInt(verStr || '1', 10);
const { runMigrations } = await import('../core/migrate.ts');
schemaBehind = await resolveSchemaBehind({
schemaVer,
latest: LATEST_VERSION,
// --list and --dry-run are read-only surfaces: never mutate schema
// even when combined with --yes/--non-interactive.
autoApply: (cli.yes || cli.nonInteractive) && !cli.dryRun && !cli.list,
run: () => runMigrations(eng),
});
await eng.disconnect();
if (schemaVer < LATEST_VERSION) {
console.warn(
`\n⚠️ Schema version ${schemaVer} is behind latest ${LATEST_VERSION}.\n` +
` Schema migrations run automatically on next connectEngine() / initSchema().\n` +
` To run them now: gbrain init --migrate-only\n`,
);
}
}
}
} catch {
@@ -420,6 +460,13 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
const toRun: Migration[] = [...plan.partial, ...plan.pending];
if (toRun.length === 0) {
if (schemaBehind) {
console.error(
'Orchestrator migrations are up to date, but schema migrations are behind. ' +
'Run `gbrain apply-migrations --yes` (or `--force-schema`) to apply them.',
);
process.exit(1);
}
console.log('All migrations up to date.');
process.exit(0);
}
@@ -511,4 +558,5 @@ export const __testing = {
buildPlan,
indexCompleted,
statusForVersion,
resolveSchemaBehind,
};
+25 -6
View File
@@ -524,13 +524,17 @@ async function registerClient(name: string, args: string[]) {
* /admin/api/rescope-client endpoint.
*/
async function rescopeClient(clientId: string, args: string[]) {
const usage = 'Usage: auth rescope-client <client_id> [--source SOURCE] [--federated-read SRC1,SRC2,...]';
const usage = 'Usage: auth rescope-client <client_id> [--source SOURCE] [--federated-read SRC1,SRC2,...] [--bound-slug-prefixes P1,P2|none]';
if (!clientId) {
console.error(usage);
process.exit(1);
}
let sourceId: string | undefined;
let federatedRead: string[] | undefined;
// v0.42.72.0: tri-state — undefined = untouched, null = clear ('none'),
// array = replace. Lets roster churn (channel joins/leaves) update the
// write fence in place instead of register+rotate.
let boundSlugPrefixes: string[] | null | undefined;
for (let i = 0; i < args.length; i += 2) {
const flag = args[i];
const value = args[i + 1];
@@ -542,14 +546,18 @@ async function rescopeClient(clientId: string, args: string[]) {
if (flag === '--source') sourceId = value;
else if (flag === '--federated-read') {
federatedRead = value.split(',').map(s => s.trim()).filter(Boolean);
} else if (flag === '--bound-slug-prefixes') {
boundSlugPrefixes = value === 'none'
? null
: value.split(',').map(s => s.trim()).filter(Boolean);
} else {
console.error(`Error: Unknown flag: ${flag}`);
console.error(usage);
process.exit(1);
}
}
if (sourceId === undefined && federatedRead === undefined) {
console.error('Error: pass --source and/or --federated-read');
if (sourceId === undefined && federatedRead === undefined && boundSlugPrefixes === undefined) {
console.error('Error: pass --source, --federated-read, and/or --bound-slug-prefixes');
console.error(usage);
process.exit(1);
}
@@ -557,10 +565,13 @@ async function rescopeClient(clientId: string, args: string[]) {
await withConfiguredSql(async (sql) => {
const { GBrainOAuthProvider } = await import('../core/oauth-provider.ts');
const provider = new GBrainOAuthProvider({ sql });
const result = await provider.rescopeClient(clientId, { sourceId, federatedRead });
const result = await provider.rescopeClient(clientId, { sourceId, federatedRead, boundSlugPrefixes });
console.log(`OAuth client rescoped: "${result.clientName}" (${result.clientId})\n`);
console.log(` Write source: ${result.sourceId}`);
console.log(` Federated reads: ${result.federatedRead.join(', ') || '<none>'}`);
if (result.boundSlugPrefixes !== undefined) {
console.log(` Bound slug prefixes: ${result.boundSlugPrefixes?.join(', ') ?? '<none — full-source write authority>'}`);
}
console.log('\nTakes effect on the client\'s next request (existing tokens included).');
});
} catch (e: any) {
@@ -645,14 +656,22 @@ Usage:
--bound-tools <tool1,tool2> Bind submit_agent to an allow-list of tools
--bound-source <id> Bind submit_agent jobs to a source id
--bound-brain <id> Bind submit_agent jobs to a brain id
--bound-slug-prefixes <prefix1,prefix2> Bind submit_agent writes to slug prefixes
--bound-slug-prefixes <prefix1,prefix2> Fence ALL direct slug writes (put_page, delete_page,
tags, links, timeline, revert, raw data) AND
submit_agent to these prefixes. Each MUST end with
'/' or '/*' a boundary-less 'emp-alice' would also
name 'emp-alice-2/...'. Ops that write by something
other than a slug (extract_*, forget_fact,
ontology_propose, sources_*) and POST /ingest become
unavailable to a bound client. Omit = full-source writes.
--bound-max-concurrent <n> Bound submit_agent concurrency (default: 1)
--budget-usd-per-day <usd> Bound submit_agent daily spend cap
gbrain auth rescope-client <client_id> [options] Change an existing client's source scope (e.g. a DCR
client stuck on the 'default' source). Only the flags
you pass change; the other axis is left as-is.
you pass change; the other axes are left as-is.
--source <id> New write source
--federated-read <id1,id2,...> New read-scope source list
--bound-slug-prefixes <p1,p2|none> Replace the slug-prefix write fence ('none' clears it)
gbrain auth revoke-client <client_id> Hard-delete an OAuth 2.1 client (cascades to tokens + codes)
gbrain auth test <url> --token <token> Smoke-test a remote MCP server
`);
+5 -4
View File
@@ -33,6 +33,7 @@
import type { BrainEngine, SourceRow } from '../core/engine.ts';
import type { MinionQueue } from '../core/minions/queue.ts';
import { NON_GLOBAL_PHASES, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY } from '../core/cycle.ts';
import { sourceConfigHasRemoteUrl } from '../core/sources-load.ts';
const FULL_CYCLE_FLOOR_MIN = 60;
@@ -430,13 +431,13 @@ export async function dispatchPerSource(
const dispatched: string[] = [];
for (const src of dispatch) {
try {
const remoteUrl = typeof src.config?.remote_url === 'string' ? src.config.remote_url : null;
const shouldPull = sourceConfigHasRemoteUrl(src.config);
const job = await queue.add(
'autopilot-cycle',
{
repoPath: opts.repoPath,
source_id: src.id,
pull: !!remoteUrl,
pull: shouldPull,
// #2194 fix #3 (cycle split): per-source cycles run ONLY source-scoped
// (+ mixed) phases. The brain-wide global phases (embed, orphans,
// purge, …) run once in autopilot-global-maintenance, not N times
@@ -465,11 +466,11 @@ export async function dispatchPerSource(
job_id: job.id,
mode: 'per_source',
source_id: src.id,
pull: !!remoteUrl,
pull: shouldPull,
slot: opts.slot,
}));
} else {
log(`[dispatch] job #${job.id} autopilot-cycle source=${src.id}${remoteUrl ? ' pull=yes' : ''}`);
log(`[dispatch] job #${job.id} autopilot-cycle source=${src.id}${shouldPull ? ' pull=yes' : ''}`);
}
} catch (e) {
// Per-source submit failure does NOT abort the tick (codex E1 F1
+2 -1
View File
@@ -739,7 +739,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
try {
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
if (await isFederatedV2Enabled(engine)) {
const { loadAllSources } = await import('../core/sources-load.ts');
const { loadAllSources, sourceConfigHasRemoteUrl } = await import('../core/sources-load.ts');
const sources = await loadAllSources(engine);
const intervalMs = baseInterval * 1000;
const now = Date.now();
@@ -754,6 +754,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
{
sourceId: src.id,
repoPath: src.local_path,
pull: sourceConfigHasRemoteUrl(src.config),
auto_embed_backfill: true,
embed_reason: 'autopilot_freshness',
},
+9 -6
View File
@@ -45,12 +45,15 @@ function upgradeCommandForMethod(method: string): string {
}
}
/** Where the latest version is resolved from. gbrain publishes NO GitHub
* releases (the `releases/latest` API is a permanent 404), so the release
* train's source of truth is the `VERSION` file on master same trusted host
* `fetchChangelog` already uses. An npm fallback was rejected: the `gbrain`
* package on npm is an unrelated GPU library (#505), so it would produce false
* upgrade prompts pointing at a stranger's package. */
/** Where the latest version is resolved from. The release train's source of
* truth is the `VERSION` file on master same trusted host `fetchChangelog`
* already uses. GitHub releases are published from it per VERSION bump
* (`.github/workflows/release.yml`, #3521) and carry the binary assets, but
* this check deliberately does NOT read `releases/latest`: it was a permanent
* 404 before releases existed (#3520) and can still lag master. An npm
* fallback was rejected: the `gbrain` package on npm is an unrelated GPU
* library (#505), so it would produce false upgrade prompts pointing at a
* stranger's package. */
const VERSION_SOURCE_URL = 'https://raw.githubusercontent.com/garrytan/gbrain/master/VERSION';
const RELEASE_NOTES_URL = 'https://github.com/garrytan/gbrain/blob/master/CHANGELOG.md';
+108
View File
@@ -0,0 +1,108 @@
/**
* #1835 storage_path resolution for the doctor `image_assets` check.
*
* `files.storage_path` rows written by a Windows gbrain install carry Windows
* drive paths (`D:/foo/img.jpg`, `D:\foo\img.jpg`). On POSIX,
* `path.isAbsolute()` is false for those, so the old code joined them onto the
* repo root and produced a path that can never exist a false-positive
* "missing from disk, restore from git" WARN under WSL and macOS.
*
* Policy:
* - win32: drive paths are absolute; stat them as-is.
* - WSL (linux + "microsoft" in /proc/version): translate `D:/x` to
* `<automount root>/d/x` (automount root read from /etc/wsl.conf
* `[automount] root`, default `/mnt`) and stat that.
* - any other POSIX host (macOS, plain Linux): the path is unresolvable on
* this platform report it as foreign so the caller SKIPS the stat
* instead of inventing a path that will never exist.
*
* Kept in its own module (not doctor.ts) so the pure tests don't pull the
* 7k-line doctor dep graph, and so open PRs rewriting the image_assets block
* (e.g. a `resolveImageAssetPath` helper) can adopt it with a one-line call.
*/
import { readFileSync } from 'node:fs';
import { join, posix, win32 } from 'node:path';
const WINDOWS_DRIVE_RE = /^([A-Za-z]):[\\/](.*)$/;
export interface AssetPathResolution {
/** Absolute path to stat, or null when the path is unresolvable here. */
abs: string | null;
/** True when storage_path is a Windows drive path this host cannot stat. */
foreign: boolean;
}
/**
* Resolve a files.storage_path to a stat-able absolute path.
* `opts.platform` / `opts.wslMountRoot` exist for tests; production callers
* pass neither (process.platform + detected WSL automount root).
* `wslMountRoot: null` means "not under WSL".
*/
export function resolveAssetPath(
storagePath: string,
repoRoot: string,
opts: { platform?: NodeJS.Platform; wslMountRoot?: string | null } = {},
): AssetPathResolution {
const platform = opts.platform ?? process.platform;
if (platform !== 'win32') {
const m = WINDOWS_DRIVE_RE.exec(storagePath);
if (m) {
const root = opts.wslMountRoot !== undefined ? opts.wslMountRoot : detectWslMountRoot();
if (root === null) return { abs: null, foreign: true };
const abs = `${root.replace(/\/+$/, '')}/${m[1].toLowerCase()}/${m[2].replace(/\\/g, '/')}`;
return { abs, foreign: false };
}
}
// Platform-appropriate absoluteness (not the host's) so injected-platform
// tests behave identically everywhere; in production platform === host.
const isAbs = platform === 'win32' ? win32.isAbsolute(storagePath) : posix.isAbsolute(storagePath);
return {
abs: isAbs ? storagePath : join(repoRoot, storagePath),
foreign: false,
};
}
/**
* Extract the `[automount] root` value from /etc/wsl.conf content.
* Defaults to `/mnt` (WSL's own default) when absent/unparseable.
*/
export function parseWslAutomountRoot(conf: string): string {
let inAutomount = false;
for (const raw of conf.split(/\r?\n/)) {
const line = raw.replace(/[#;].*$/, '').trim();
if (line.startsWith('[')) {
inAutomount = /^\[automount\]$/i.test(line);
continue;
}
if (!inAutomount) continue;
const m = /^root\s*=\s*"?([^"]+?)"?\s*$/.exec(line);
if (m) return m[1];
}
return '/mnt';
}
let cachedWslMountRoot: string | null | undefined;
/**
* Detect the WSL Windows-drive automount root. Returns null when not running
* under WSL (including macOS and plain Linux). Memoized per process.
*/
export function detectWslMountRoot(): string | null {
if (cachedWslMountRoot === undefined) cachedWslMountRoot = computeWslMountRoot();
return cachedWslMountRoot;
}
function computeWslMountRoot(): string | null {
if (process.platform !== 'linux') return null;
try {
// The standard WSL tell: kernel version string names Microsoft.
if (!/microsoft/i.test(readFileSync('/proc/version', 'utf8'))) return null;
} catch {
return null;
}
try {
return parseWslAutomountRoot(readFileSync('/etc/wsl.conf', 'utf8'));
} catch {
return '/mnt'; // WSL default when wsl.conf is absent.
}
}
+475 -16
View File
@@ -6,6 +6,11 @@ import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
import { autoDetectSkillsDirReadOnly } from '../core/repo-root.ts';
import {
SKILLS_MANIFEST_FILENAME,
verifySkillsManifest,
type SkillsManifest,
} from '../core/skills-integrity.ts';
import { loadOrDeriveManifest } from '../core/skill-manifest.ts';
import { parseSkillFrontmatter } from '../core/skill-frontmatter.ts';
import {
@@ -47,6 +52,13 @@ import { lagFromContentMs } from '../core/source-health.ts';
import { CHUNKER_VERSION } from '../core/chunkers/code.ts';
import { LINK_EXTRACTOR_VERSION_TS } from '../core/link-extraction.ts';
import { isUndefinedColumnError } from '../core/utils.ts';
import {
loadStorageConfig,
effectiveDbOnlyDirs,
DERIVE_PHASE_DB_ONLY_DEFAULTS,
findDbOnlyCollisions,
} from '../core/storage-config.ts';
import { slugifyPath } from '../core/sync.ts';
// issue #1777: hidden_by_search_policy — count chunked pages withheld from
// default search by the hard-exclude prefix policy. Reuses the canonical
// exclude resolver + LIKE escaper + visibility clause so the doctor count can't
@@ -364,19 +376,38 @@ export async function jsonbIntegrityCheck(
progress?: Pick<ProgressReporter, 'heartbeat'>,
): Promise<Check> {
try {
const targets: Array<{ table: string; col: string; expected: 'object' | 'array' }> = [
{ table: 'pages', col: 'frontmatter', expected: 'object' },
{ table: 'raw_data', col: 'data', expected: 'object' },
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
{ table: 'files', col: 'metadata', expected: 'object' },
{ table: 'page_versions', col: 'frontmatter', expected: 'object' },
const targets: Array<{ table: string; col: string; expected: 'object' | 'array'; jsonPayloadOnly?: boolean }> = [
{ table: 'pages', col: 'frontmatter', expected: 'object' },
{ table: 'raw_data', col: 'data', expected: 'object' },
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
{ table: 'files', col: 'metadata', expected: 'object' },
{ table: 'page_versions', col: 'frontmatter', expected: 'object' },
// Subagent persistence — second double-encode site (historical damage
// rows from the pre-v0.42.53.0 positional bind; write paths fixed in
// #2375). Mirrors repair-jsonb's targets incl. jsonPayloadOnly: these
// columns can legitimately hold jsonb STRING scalars (persistToolExec
// binds pre-serialized string payloads as-is), so only JSON-container
// content counts as damage.
{ table: 'subagent_messages', col: 'content_blocks', expected: 'array', jsonPayloadOnly: true },
{ table: 'subagent_tool_executions', col: 'input', expected: 'object', jsonPayloadOnly: true },
{ table: 'subagent_tool_executions', col: 'output', expected: 'object', jsonPayloadOnly: true },
];
let totalBad = 0;
const breakdown: string[] = [];
for (const { table, col } of targets) {
for (const { table, col, jsonPayloadOnly } of targets) {
progress?.heartbeat(`jsonb_integrity.${table}.${col}`);
// Skip targets whose table doesn't exist on this brain (subagent_*
// tables are v0.15+; pre-v0.15 brains naturally lack them).
const existsRows = await engine.executeRaw<{ exists: boolean }>(
`SELECT to_regclass($1) IS NOT NULL AS exists`,
[table],
);
if (!existsRows[0]?.exists) continue;
const damage = jsonPayloadOnly
? `jsonb_typeof(${col}) = 'string' AND (${col} #>> '{}') ~ '^[[:space:]]*[\\[{]' AND pg_input_is_valid(${col} #>> '{}', 'jsonb')`
: `jsonb_typeof(${col}) = 'string'`;
const rows = await engine.executeRaw<{ n: number }>(
`SELECT count(*)::int AS n FROM ${table} WHERE jsonb_typeof(${col}) = 'string'`,
`SELECT count(*)::int AS n FROM ${table} WHERE ${damage}`,
);
const n = Number(rows[0]?.n ?? 0);
if (n > 0) { totalBad += n; breakdown.push(`${table}.${col}=${n}`); }
@@ -3667,6 +3698,200 @@ export async function checkUnverifiedExtractions(
}
}
/**
* issue #2250 (reported by @615Works) content_hash_duplicates.
*
* `gbrain import` run from the wrong root (one level too deep) drops the
* path prefix from every slug, leaving `people/x` and `x` coexisting with
* identical content. `dream --phase purge` never removes them (they aren't
* file-backed orphans) and nothing surfaced the condition. One GROUP BY
* never an N² hash comparison flags hash groups that contain BOTH a bare
* slug (no '/') and a path-prefixed slug.
*/
export async function checkContentHashDuplicates(engine: BrainEngine): Promise<Check> {
const name = 'content_hash_duplicates';
const fix = 'Fix: gbrain pages delete <bare-slug> for each pair, then gbrain pages purge-deleted --older-than 0';
try {
const rows = await engine.executeRaw<{ source_id: string; content_hash: string; slugs: string }>(
`SELECT source_id, content_hash,
string_agg(slug, '|' ORDER BY length(slug), slug) AS slugs
FROM pages
WHERE deleted_at IS NULL AND content_hash IS NOT NULL AND content_hash <> ''
GROUP BY source_id, content_hash
HAVING count(*) > 1
AND count(*) FILTER (WHERE strpos(slug, '/') = 0) > 0
AND count(*) FILTER (WHERE strpos(slug, '/') > 0) > 0
LIMIT 50`,
);
if (rows.length === 0) {
return { name, status: 'ok', message: 'No content-hash duplicate pairs (bare vs path-prefixed slugs)' };
}
let pairCount = 0;
const samples: string[] = [];
for (const r of rows) {
const slugs = String(r.slugs).split('|');
const prefixed = slugs.filter(s => s.includes('/'));
for (const bare of slugs.filter(s => !s.includes('/'))) {
const twin = prefixed.find(p => p.endsWith('/' + bare)) ?? prefixed[0];
pairCount++;
if (samples.length < 5) samples.push(`${bare} <-> ${twin}`);
}
}
return {
name,
status: 'warn',
message: `${pairCount} content-hash duplicate pair(s) detected (same content, differing slug forms — usually an import run from the wrong root, which drops the path prefix). Sample: ${samples.join('; ')}. ${fix}`,
details: { pair_count: pairCount, hash_groups: rows.length, sample_pairs: samples },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check content-hash duplicates: ${(e as Error).message}` };
}
}
/** Walk a repo for markdown files and return their slugified (lowercased) slugs. */
function collectMarkdownSlugs(root: string): Set<string> {
const out = new Set<string>();
const stack = [''];
while (stack.length > 0) {
const rel = stack.pop()!;
let entries;
try {
entries = readdirSync(rel ? join(root, rel) : root, { withFileTypes: true });
} catch {
continue;
}
for (const e of entries) {
// Hidden directories can contain canonical, tracked knowledge (for
// example `.archive/`). Only implementation metadata is never a page.
if (e.name === '.git' || e.name === 'node_modules') continue;
const childRel = rel ? `${rel}/${e.name}` : e.name;
if (e.isDirectory()) stack.push(childRel);
else if (/\.mdx?$/i.test(e.name)) out.add(slugifyPath(childRel).toLowerCase());
}
}
return out;
}
/**
* issue #2784 (reported by @alexputici) undeclared_db_only_pages.
*
* A markdown page with no backing file that sits outside every declared
* db_only path is invisible to any file-lane backup/recovery reasoning: an
* operator auditing "what would survive a DB loss" gets a silently wrong
* answer. The engine's own derive-phase output prefixes
* (DERIVE_PHASE_DB_ONLY_DEFAULTS) count as implicitly declared so the check
* stays quiet on healthy brains. Deliberately allowed to stat the source
* repo (the one thing the SQL-only check registry could never see).
*/
export async function checkUndeclaredDbOnlyPages(engine: BrainEngine): Promise<Check> {
const name = 'undeclared_db_only_pages';
try {
const sources = await engine.executeRaw<{ id: string; local_path: string | null }>(
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL`,
);
const checkable = sources.filter(s => s.local_path && existsSync(s.local_path));
if (checkable.length === 0) {
return { name, status: 'ok', message: 'Not applicable (no sources with a local repo path on this host)' };
}
let total = 0;
const samples: string[] = [];
const perSource: Record<string, number> = {};
for (const src of checkable) {
let declared: string[] = [];
try {
declared = loadStorageConfig(src.local_path)?.db_only ?? [];
} catch {
// invalid gbrain.yml — treated as no declarations; the sync path
// already surfaces the config error itself.
}
const dbOnlyDirs = effectiveDbOnlyDirs(declared);
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT slug FROM pages WHERE deleted_at IS NULL AND source_id = $1 AND page_kind = 'markdown'`,
[src.id],
);
if (rows.length === 0) continue;
const backed = collectMarkdownSlugs(src.local_path!);
for (const { slug } of rows) {
if (dbOnlyDirs.some(dir => slug.startsWith(dir))) continue;
if (backed.has(slug)) continue;
total++;
perSource[src.id] = (perSource[src.id] ?? 0) + 1;
if (samples.length < 5) samples.push(`${slug} (src=${src.id})`);
}
}
if (total === 0) {
return {
name,
status: 'ok',
message: `Every DB page is file-backed or under a declared/default db_only path (derive-phase defaults: ${DERIVE_PHASE_DB_ONLY_DEFAULTS.join(' ')})`,
};
}
return {
name,
status: 'warn',
message: `${total} DB page(s) have no backing file and sit outside every declared/default db_only path — invisible to file-lane backup/recovery. Sample: ${samples.join('; ')}. Fix: restore or export the files, or declare their prefixes under storage.db_only in gbrain.yml (derive-phase defaults already cover: ${DERIVE_PHASE_DB_ONLY_DEFAULTS.join(' ')})`,
details: { total, per_source: perSource, sample_slugs: samples },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check undeclared db-only pages: ${(e as Error).message}` };
}
}
/**
* issue #2788 (reported by @alexputici) db_only_collector_collision.
*
* Declaring a collector's output dir in storage.db_only silently kills its
* ingestion: manageGitignore auto-gitignores the dir, the git-walking sync
* never sees the files, and import honors .gitignore too everything stays
* green while nothing reaches the DB (a 7-week outage in the field). The
* recipe's `output_paths` frontmatter is the ground truth; the same warning
* also fires at .gitignore-write time inside sync's manageGitignore.
*/
export async function checkDbOnlyCollectorCollision(
engine: BrainEngine,
opts?: { collectors?: Array<{ id: string; output_path: string }> },
): Promise<Check> {
const name = 'db_only_collector_collision';
try {
let collectors = opts?.collectors;
if (!collectors) {
const { getConfiguredCollectorOutputs } = await import('./integrations.ts');
collectors = getConfiguredCollectorOutputs();
}
if (collectors.length === 0) {
return { name, status: 'ok', message: 'No configured collectors declare output paths' };
}
const sources = await engine.executeRaw<{ id: string; local_path: string | null }>(
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL`,
);
const hits: string[] = [];
for (const src of sources) {
if (!src.local_path || !existsSync(src.local_path)) continue;
let dbOnly: string[] = [];
try {
dbOnly = loadStorageConfig(src.local_path)?.db_only ?? [];
} catch {
continue;
}
if (dbOnly.length === 0) continue;
for (const hit of findDbOnlyCollisions(collectors, dbOnly)) {
hits.push(`collector '${hit.id}' writes to '${hit.output_path}' which is inside db_only path '${hit.db_only_dir}' (source ${src.id})`);
}
}
if (hits.length === 0) {
return { name, status: 'ok', message: 'No collector output dir falls inside a db_only path' };
}
return {
name,
status: 'warn',
message: `${hits.length} collector/db_only collision(s): ${hits.join('; ')}. db_only dirs are auto-gitignored, so sync AND import silently skip files there — the collector runs green while nothing reaches the DB. Fix: remove the prefix from storage.db_only in gbrain.yml, or move the collector output.`,
details: { collisions: hits },
};
} catch (e) {
return { name, status: 'warn', message: `Could not check collector/db_only collisions: ${(e as Error).message}` };
}
}
/**
* issue #1678 extract_atoms_backlog doctor check.
*
@@ -4445,6 +4670,96 @@ export async function checkCycleFreshness(
* - `progress` reporter writes to stderr (heartbeats per check)
* - `engine.executeRaw` / handler-leaf calls (the actual probe work)
*/
// ≥2 failed repair attempts inside 7 days = the corruption keeps regenerating.
const REPAIR_RECURRENCE_WINDOW_MS = 7 * 24 * 3600 * 1000;
const REPAIR_RECURRENCE_THRESHOLD = 2;
/**
* WAL-repair wave (#223/#1670/#2575): when the DB failed to connect on a
* PGLite brain, diagnose the data dir from the FILESYSTEM (the connect error
* itself was swallowed by doctor's fs-only fallback this check re-derives
* the state from disk). Pure: interprets an `inspectPgliteDataDir` diagnosis
* into a Check; exported so `test/doctor-pglite-datadir.test.ts` drives it
* directly (same convention as computeWorkerOomLoopCheck). Returns a Check
* always the call site only runs it when connect already failed, so even a
* healthy-looking dir warrants a pointer at the repair tooling.
*
* Recurrence escalation (eng-review 2A): repeated failed repair attempts on
* record mean the corruption keeps regenerating (unclean-shutdown genesis)
* escalate to the engine-switch ladder instead of letting the brain silently
* lose a WAL tail per cycle. Backup-dir inventory rides along (same
* disk-visibility class as orphan_clones).
*/
export function computePgliteDataDirCheck(
dataDir: string,
diagnosis: import('../core/pglite-repair.ts').PgliteDirDiagnosis,
): Check {
const backupNote = diagnosis.backupDirs.length > 0
? ` ${diagnosis.backupDirs.length} repair backup dir(s) on disk (newest: ${diagnosis.backupDirs[0]}) — delete old ones to reclaim space once the brain is healthy.`
: '';
// Count BOTH outcomes (adversarial review F12): a >1h-period crash loop where
// each repair "succeeds" discards a WAL tail per cycle with zero FAILED
// attempts on record — escalation must still fire.
const recentAttempts = diagnosis.recentAttempts.filter(
(a) => Date.now() - a.ts < REPAIR_RECURRENCE_WINDOW_MS,
).length;
const recurrence = recentAttempts >= REPAIR_RECURRENCE_THRESHOLD
? ` Auto-repair has run ${recentAttempts}x this week — the corruption keeps regenerating (likely an unclean-shutdown loop). Consider switching engines (docs/ENGINES.md: \`gbrain init --supabase\` or native Postgres).`
: '';
switch (diagnosis.verdict) {
case 'locked':
return {
name: 'pglite_data_dir',
status: 'warn',
message:
`Could not connect, and the PGLite data-dir lock is held by live PID ${diagnosis.lockHolderPid}` +
`another gbrain process (often \`gbrain serve\`) has the brain open. Stop it and re-run.${backupNote}`,
remediation_status: 'human_only',
};
case 'missing':
return {
name: 'pglite_data_dir',
status: 'warn',
message: `No PGLite data dir at ${dataDir}. Run \`gbrain init --pglite\` to create one.`,
remediation_status: 'human_only',
};
case 'unsupported-layout':
return {
name: 'pglite_data_dir',
status: 'fail',
message:
`PGLite data dir at ${dataDir} is not repairable in place (${diagnosis.detail}). ` +
`Rebuild from your brain repo: \`gbrain reinit-pglite\` (or back up ~/.gbrain, move the dir aside, ` +
`\`gbrain init --pglite\`, re-add sources + sync + embed).${backupNote}${recurrence}`,
remediation_status: 'human_only',
};
case 'wal-corruption-likely':
return {
name: 'pglite_data_dir',
status: 'fail',
message:
`PGLite failed to open and the data dir shows unclean-shutdown state (${diagnosis.detail}). ` +
`This is the torn-WAL class behind issue #223 — repairable in place, data preserved: ` +
`\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair.${backupNote}${recurrence}`,
remediation_status: 'human_only',
};
case 'looks-healthy':
default:
return {
name: 'pglite_data_dir',
status: 'fail',
message:
`PGLite failed to open but the data dir layout validates (${diagnosis.detail}). ` +
`IF the connect error mentions \`Aborted()\` this is likely torn WAL state — ` +
`\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair in place ` +
`(repair discards the un-checkpointed WAL tail — don't run it for lock-contention or ` +
`catalog-corruption errors; 58P01/pgvector load failures need \`gbrain reinit-pglite\` instead).${backupNote}${recurrence}`,
remediation_status: 'human_only',
};
}
}
/**
* issue #1685 (GAP A) the single authoritative "worker is OOM-looping" signal.
*
@@ -4629,6 +4944,46 @@ export async function computePoolReapHealthCheck(
* Policy-skill install state is reported in details (it ships into the HOST
* repo, so absence in gbrain's own skills dir is expected, not a failure).
*/
/**
* MEMORY_VERBS v1 (Cathedral 1, E4) usage-sidecar health. Read-only,
* fail-open. Stats only (local JSONL, never uploaded; never source of truth):
* - no sidecar file ok, "no verb calls recorded yet" (fresh install)
* - recent events parse ok, names the last verb + timestamp
* - file exists, unreadable warn (observability degraded, verbs unaffected)
*/
export async function buildMemoryVerbsCheck(): Promise<Check> {
const name = 'memory_verbs_usage';
try {
const { readVerbUsage, usageLogPath } = await import('../core/verbs/usage-log.ts');
if (!existsSync(usageLogPath())) {
return {
name,
status: 'ok',
message: 'no verb calls recorded yet (sidecar appears on first remember/recall/entity/synthesize/forget)',
};
}
const events = await readVerbUsage({ days: 30 });
if (events.length === 0) {
return { name, status: 'ok', message: 'sidecar present; no verb calls in the last 30 days' };
}
const last = events[events.length - 1];
const byVerb = new Map<string, number>();
for (const e of events) byVerb.set(e.verb, (byVerb.get(e.verb) ?? 0) + 1);
const mix = [...byVerb.entries()].map(([v, n]) => `${v}:${n}`).join(' ');
return {
name,
status: 'ok',
message: `${events.length} verb calls in 30d (${mix}); last ${last.verb} at ${last.ts} — local JSONL only, never uploaded`,
};
} catch (e) {
return {
name,
status: 'warn',
message: `verb usage sidecar unreadable (${e instanceof Error ? e.message : String(e)}) — observability degraded; verbs unaffected`,
};
}
}
export function buildRetrievalReflexCheck(skillsDir: string | null): Check {
const name = 'retrieval_reflex_health';
try {
@@ -4822,6 +5177,13 @@ export async function buildChecks(
checks.push(buildRetrievalReflexCheck(skillsDir));
}
// 1c. MEMORY_VERBS v1 usage sidecar health (Cathedral 1, E4). Read-only,
// fail-open: reports whether the local JSONL sidecar is present + parseable
// and when a verb last fired. Local file only — never uploaded.
if (scope === 'all') {
checks.push(await buildMemoryVerbsCheck());
}
// 2. Skill conformance (SKILL group — gated)
if (scope === 'all' && skillsDir) {
const conformanceResult = skillConformanceCheck(skillsDir);
@@ -4845,6 +5207,15 @@ export async function buildChecks(
checks.push(skillBrainFirstCheck(skillsDir));
}
// 2c. Skills manifest integrity (#159): tamper-evidence, not signatures.
// Compares the skills tree against its committed skills.lock.json and
// WARNS on drift — never fails, never blocks. No manifest (e.g. a user
// workspace skills dir, or a compiled binary far from the repo) → ok/skip.
// SKILL group — gated.
if (scope === 'all' && skillsDir) {
checks.push(skillsManifestIntegrityCheck(skillsDir));
}
// 3. Half-migrated Minions detection (filesystem-only).
// If completed.jsonl has any status:"partial" entry with no later
// status:"complete" for the same version, the install is mid-migration.
@@ -5732,6 +6103,27 @@ export async function buildChecks(
// Filesystem read failure is non-fatal.
}
// 3d. PGLite data-dir diagnosis (WAL-repair wave). Only meaningful when the
// connect already FAILED on a PGLite brain (engine === null): the connect
// error was swallowed by the fs-only fallback, so this check re-derives the
// dir state from disk and names the repair ladder. Skipped under --fast
// (connect wasn't attempted, so "engine === null" proves nothing there).
if (!fastMode && !engine) {
try {
const cfg = loadConfig();
if (cfg?.engine === 'pglite') {
const { inspectPgliteDataDir } = await import('../core/pglite-repair.ts');
const { resolve } = await import('node:path');
// Absolutize: a RELATIVE database_path would make the sidecar/backup
// lookups resolve against doctor's cwd instead of the engine's.
const pgliteDataDir = resolve(cfg.database_path || gbrainPath('brain.pglite'));
checks.push(computePgliteDataDirCheck(pgliteDataDir, inspectPgliteDataDir(pgliteDataDir)));
}
} catch {
// Best-effort: an unreadable config or fs failure must not stop doctor.
}
}
// --- DB checks (skip if --fast or no engine) ---
if (fastMode || !engine) {
@@ -6102,7 +6494,10 @@ export async function buildChecks(
} else {
// Live embed test
const start = Date.now();
const vec = await embedOne('gbrain doctor embedding smoke test');
// Doctor is itself the provider-health circuit breaker. A permanent
// billing/auth failure must be sampled once, not multiplied by the AI
// SDK's default retries (which can add ~90s to every health check).
const vec = await embedOne('gbrain doctor embedding smoke test', { maxRetries: 0 });
const ms = Date.now() - start;
const actualDims = vec.length;
@@ -7582,33 +7977,44 @@ export async function buildChecks(
`SELECT storage_path FROM files WHERE mime_type LIKE 'image/%' LIMIT 1000`
);
let vanished = 0;
let foreign = 0;
const vanishedPaths: string[] = [];
const fs = await import('node:fs');
const nodePath = await import('node:path');
const { resolveAssetPath } = await import('./doctor-asset-paths.ts');
// storage_path is repo-relative for sync-ingested assets. Resolving
// against cwd made this check a false-positive WARN whenever doctor
// ran outside the brain repo.
const repoRoot = (await engine.getConfig('sync.repo_path')) ?? process.cwd();
for (const r of rows) {
const abs = nodePath.isAbsolute(r.storage_path)
? r.storage_path
: nodePath.join(repoRoot, r.storage_path);
// #1835: Windows drive paths (D:/…) translate to the WSL automount
// (/mnt/d/…) under WSL, and are SKIPPED (not "missing") on hosts
// where they cannot exist (macOS / plain Linux) — never joined onto
// repoRoot, which produced a false "restore from git" WARN.
const resolved = resolveAssetPath(r.storage_path, repoRoot);
if (resolved.abs === null) {
foreign++;
continue;
}
try {
fs.statSync(abs);
fs.statSync(resolved.abs);
} catch {
vanished++;
if (vanishedPaths.length < 5) vanishedPaths.push(r.storage_path);
}
}
const checked = rows.length - foreign;
const foreignNote = foreign > 0
? ` (${foreign} Windows-drive path(s) skipped — not resolvable on this platform)`
: '';
if (rows.length === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: 'No image assets indexed yet' });
} else if (vanished === 0) {
checks.push({ name: 'image_assets', status: 'ok', message: `${rows.length} image(s) all present on disk` });
checks.push({ name: 'image_assets', status: 'ok', message: `${checked} image(s) all present on disk${foreignNote}` });
} else {
checks.push({
name: 'image_assets',
status: 'warn',
message: `${vanished} of ${rows.length} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')}). ` +
message: `${vanished} of ${checked} image(s) missing from disk (e.g. ${vanishedPaths.join(', ')})${foreignNote}. ` +
`Fix: restore from git, or \`gbrain sync --skip-failed\` to acknowledge.`,
});
}
@@ -7668,6 +8074,14 @@ export async function buildChecks(
// per-source dispatch gate sees.
progress.heartbeat('cycle_freshness');
checks.push(await checkCycleFreshness(engine));
// Silent-failure batch (#2250 / #2784 / #2788): wrong-root import
// duplicates, undeclared DB-only pages, collector-output-in-db_only.
progress.heartbeat('content_hash_duplicates');
checks.push(await checkContentHashDuplicates(engine));
progress.heartbeat('undeclared_db_only_pages');
checks.push(await checkUndeclaredDbOnlyPages(engine));
progress.heartbeat('db_only_collector_collision');
checks.push(await checkDbOnlyCollectorCollision(engine));
}
// v0.32.3 search-lite — mode + eval_drift surfaces. Status stays 'ok' per
@@ -7902,6 +8316,51 @@ export function skillConformanceCheck(skillsDir: string): Check {
* Test seam: pure function, no `process.exit`. Direct call from tests
* with a synthetic skills dir under tempdir.
*/
/**
* Skills-manifest integrity check (#159). Verifies the skills tree against
* the committed skills.lock.json tamper-evidence manifest. Advisory only:
* drift is a WARN (local edits are legitimate), and a missing/unreadable
* manifest is an ok/skip a user's workspace skills dir or a compiled
* binary far from the repo has no manifest, and that is not a problem.
*/
export function skillsManifestIntegrityCheck(skillsDir: string): Check {
const name = 'skills_manifest_integrity';
const manifestPath = join(skillsDir, SKILLS_MANIFEST_FILENAME);
if (!existsSync(manifestPath)) {
return { name, status: 'ok', message: `No ${SKILLS_MANIFEST_FILENAME} in ${skillsDir} — integrity check not applicable` };
}
let drift: ReturnType<typeof verifySkillsManifest>;
let tracked: number;
try {
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8')) as SkillsManifest;
tracked = Object.keys(manifest).length;
drift = verifySkillsManifest(skillsDir, manifest);
} catch (err) {
// Fail-safe: an unreadable/unparseable manifest or a filesystem error
// skips the check rather than warning — this check must never block.
const msg = err instanceof Error ? err.message : String(err);
return { name, status: 'ok', message: `Could not verify ${SKILLS_MANIFEST_FILENAME} (${msg}) — integrity check skipped` };
}
const total = drift.modified.length + drift.missing.length + drift.extra.length;
if (total === 0) {
return { name, status: 'ok', message: `${tracked} bundled skill files match ${SKILLS_MANIFEST_FILENAME}` };
}
const sample = (files: string[]): string =>
files.slice(0, 5).join(', ') + (files.length > 5 ? `, … +${files.length - 5} more` : '');
const parts: string[] = [];
if (drift.modified.length > 0) parts.push(`${drift.modified.length} modified (${sample(drift.modified)})`);
if (drift.missing.length > 0) parts.push(`${drift.missing.length} missing (${sample(drift.missing)})`);
if (drift.extra.length > 0) parts.push(`${drift.extra.length} extra (${sample(drift.extra)})`);
return {
name,
status: 'warn',
message:
`skills/ drifted from ${SKILLS_MANIFEST_FILENAME} (advisory — local edits are fine): ${parts.join('; ')}. ` +
`If intentional, regenerate: bun run scripts/generate-skills-manifest.ts`,
details: { modified: drift.modified, missing: drift.missing, extra: drift.extra },
};
}
export function skillBrainFirstCheck(skillsDir: string): Check {
let manifest: ReturnType<typeof loadOrDeriveManifest>;
try {
+217 -36
View File
@@ -19,10 +19,26 @@ import {
} from '../core/pace-mode.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts';
import { embedBackfillLockId } from '../core/embed-backfill-lock.ts';
import { AITransientError } from '../core/ai/errors.ts';
import { wrapChunkTextsForStoredMode } from '../core/embedding-context.ts';
import { titleTierCorpusGeneration } from '../core/contextual-retrieval-service.ts';
import type { Page } from '../core/types.ts';
/** #3037: cap failure samples so a corpus-wide outage doesn't bloat --json. */
const FAILURE_SAMPLE_CAP = 10;
/**
* #3037: record embed failures on the run result. `chunkCount` is the number
* of chunks left un-embedded by this failure (1 for page-level errors where
* the chunk count isn't known at the catch site).
*/
function recordFailure(result: EmbedResult, chunkCount: number, slug: string, e: unknown): void {
result.failures += chunkCount;
if (result.failure_samples.length < FAILURE_SAMPLE_CAP) {
result.failure_samples.push(`${slug}: ${e instanceof Error ? e.message : String(e)}`);
}
}
/**
* #3507 after a plain re-embed fully re-embedded a `per_chunk_synopsis`
* page at the title-only tier (see wrapChunkTextsForStoredMode), restamp the
@@ -166,6 +182,20 @@ export interface EmbedResult {
total_chunks: number;
/** Number of pages processed (whether or not they had stale chunks). */
pages_processed: number;
/**
* #3037: chunks that FAILED to embed this run (batch failures + per-chunk
* isolation failures). Callers must not read total silence as success:
* `src/cli.ts` turns `failures > 0` into a non-zero exit verdict (mirrors
* the `import` errors>0 guard), and structured consumers (--json, minion
* handlers) can surface it. 0 on a clean run. Additive field.
*/
failures: number;
/**
* #3037: up to 10 `slug: error-message` samples of what failed, so the
* operator gets a diagnosis without scrolling stderr. Capped so a
* corpus-wide outage doesn't bloat structured output. Additive field.
*/
failure_samples: string[];
/** True if this run was a dry-run. */
dryRun: boolean;
/**
@@ -284,6 +314,8 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
would_embed: 0,
total_chunks: 0,
pages_processed: 0,
failures: 0,
failure_samples: [],
dryRun: !!opts.dryRun,
};
@@ -293,6 +325,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal, opts.quiet);
} catch (e: unknown) {
if (isAborted(opts.signal)) break; // shutdown, not a failure
// #3037: a page-level error (not found, DB write) must not exit 0.
// Chunk-level embed failures are counted inside embedPage; this
// counts the page itself (chunk count unknown at this site).
recordFailure(result, 1, s, e);
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
}
@@ -535,6 +572,12 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
try {
const result = await runEmbedCore(engine, opts);
if (progressStarted) progress.finish();
// #3037: loud end-of-run summary so failures are visible even when the
// per-page stderr lines scrolled away. cli.ts turns failures>0 into a
// non-zero exit verdict.
if (result.failures > 0) {
serr(`[embed] ${result.failures} chunk(s) failed to embed. First error: ${result.failure_samples[0] ?? 'unknown'}`);
}
return result;
} catch (e) {
if (progressStarted) progress.finish();
@@ -623,10 +666,32 @@ async function embedPage(
// contextual prefix when the page was embedded wrapped), not raw
// chunk_text — otherwise a re-embed silently strips the contextual
// prefixes the sync path applied. fenced_code chunks stay unwrapped.
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed), { abortSignal: signal });
// #3037: per-chunk failure isolation — one bad chunk must not leave the
// page's sibling chunks NULL. The wrapped texts (computed once) feed the
// fan-out too, so an isolation retry never strips the prefixes. Total
// embed failure is recorded here (where the chunk count is known) and
// swallowed: the page stays NULL exactly as before, but the run now
// reports it (result.failures → non-zero exit) instead of pretending
// success. Abort (shutdown) still propagates.
let embeddings: (Float32Array | null)[];
let failed = 0;
let firstError: unknown;
try {
({ embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
));
} catch (e: unknown) {
if (isAborted(signal)) throw e;
recordFailure(result, toEmbed.length, slug, e);
result.pages_processed++;
serr(` Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
return;
}
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
const updated: ChunkInput[] = chunks.map(c => preserveCodeMetadata(c, {
chunk_index: c.chunk_index,
@@ -643,16 +708,21 @@ async function embedPage(
// Guard: only stamp when EVERY chunk was (re)embedded this pass. If some
// chunks were preserved from a prior embed (unknown/old provenance), the
// page is mixed — don't claim it's current. `embed --all` fully re-embeds
// such a page and then stamps it.
if (toEmbed.length === chunks.length) {
// such a page and then stamps it. #3037: a partial failure leaves failed
// chunks NULL, so don't stamp then either.
if (failed === 0 && toEmbed.length === chunks.length) {
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
// #3507: a fully re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest.
await restampIfDemotedToTitleTier(engine, page, slug, page.source_id);
}
result.embedded += toEmbed.length;
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(` ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
result.pages_processed++;
if (!quiet) slog(`${slug}: embedded ${toEmbed.length} chunks`);
if (!quiet) slog(`${slug}: embedded ${toEmbed.length - failed} chunks`);
}
/**
@@ -791,11 +861,18 @@ async function embedAll(
try {
// #3507: reproduce the page's stored wrapping convention (see embedPage).
const embeddings = await embedBatch(wrapChunkTextsForStoredMode(page, toEmbed));
// #3037: per-chunk failure isolation — one bad chunk costs one chunk,
// not the whole page's siblings. The wrapped texts feed the fan-out
// too, so an isolation retry never strips the contextual prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(page, toEmbed),
signal ? { abortSignal: signal } : {},
);
// Build a map of new embeddings by chunk_index
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) embeddingMap.set(toEmbed[j].chunk_index, emb);
}
// Preserve ALL chunks, only update embeddings for stale ones.
// preserveCodeMetadata threads code-chunk metadata (#769) so re-embed
@@ -809,17 +886,30 @@ async function embedAll(
}));
await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts));
// v0.41.31: stamp embedding provenance so a later model swap is
// detectable as stale.
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest.
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
result.embedded += toEmbed.length;
// detectable as stale. #3037: not on partial failure — failed chunks
// stay NULL under unknown provenance.
if (failed === 0) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }),
);
// #3507: --all fully re-embeds; a per_chunk_synopsis page landed at
// the title tier — keep the stamped mode honest. #3037: gated on
// failed === 0 — a partially-failed page was NOT fully re-embedded,
// so restamping would make contextual_retrieval_mode lie again
// (the exact #3461 bug).
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, page, page.slug, pageSourceId),
);
}
result.embedded += toEmbed.length - failed;
if (failed > 0) {
recordFailure(result, failed, page.slug, firstError);
serr(`\n ${page.slug}: ${failed} chunk(s) failed to embed; embedded the other ${toEmbed.length - failed}`);
}
} catch (e: unknown) {
// #3037: count the darkened page so the run can't exit 0 (abort is a
// shutdown, not a failure).
if (!isAborted(signal)) recordFailure(result, toEmbed.length, page.slug, e);
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
}
@@ -1037,10 +1127,8 @@ async function embedAllStale(
let afterUpdatedAt: string | null = null;
let totalChunksLoaded = 0;
let budgetExitNotified = false;
// #1946 (OV2a): track chunks that errored out so a catch-up pass that finishes
// with stale chunks still remaining (un-embeddable for a non-transient reason)
// surfaces that loudly instead of looking like a clean run.
let embedFailures = 0;
// #1946 (OV2a) + #3037: embed failures are tracked on result.failures so
// the catch-up warning below AND the CLI exit verdict both see them.
// E-3 (paced-backfill): bounded end-of-run re-entry. A longer paced run gives
// a live writer (sync / put_page) more time to insert NEW stale rows BEHIND
@@ -1137,12 +1225,19 @@ async function embedAllStale(
// NORMAL post-model-migration path, so raw-text embedding here
// quietly converted whole corpora to the unwrapped convention.
const pageRow = await observed(pacer, () => engine.getPage(slug, { sourceId: keySourceId }));
const embeddings = await embedBatchWithBackoff(wrapChunkTextsForStoredMode(pageRow, stale), { abortSignal: effectiveSignal });
// #3037: per-chunk failure isolation — one bad chunk costs one
// chunk, not the whole page's siblings. The wrapped texts feed the
// fan-out too, so an isolation retry never strips the prefixes.
const { embeddings, failed, firstError } = await embedPageTexts(
wrapChunkTextsForStoredMode(pageRow, stale),
{ abortSignal: effectiveSignal },
);
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId }));
const staleIdxToEmbedding = new Map<number, Float32Array>();
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
const emb = embeddings[j];
if (emb) staleIdxToEmbedding.set(stale[j].chunk_index, emb);
}
// preserveCodeMetadata threads code-chunk metadata (#769) so the
// autopilot --stale path doesn't clobber language/symbol_name/etc
@@ -1160,7 +1255,8 @@ async function embedAllStale(
// A partially-stale page keeps preserved chunks of unknown/old
// provenance, so don't claim it's current. (After invalidate, a
// signature-drifted page IS fully stale → this stamps it.)
if (signature && stale.length === existing.length) {
// #3037: not on partial failure — failed chunks stay NULL.
if (signature && failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }),
);
@@ -1168,17 +1264,24 @@ async function embedAllStale(
// #3507: a FULLY re-embedded per_chunk_synopsis page landed at the
// title tier — keep the stamped mode honest. Partially-stale pages
// stay stamped as-is (mixed provenance; reindex sweeps fix them).
if (stale.length === existing.length) {
// #3037: `failed === 0` is part of "fully re-embedded" — if the
// per-chunk isolation left some chunks NULL, restamping would make
// contextual_retrieval_mode lie again (the exact #3461 bug).
if (failed === 0 && stale.length === existing.length) {
await observed(pacer, () =>
restampIfDemotedToTitleTier(engine, pageRow, slug, keySourceId),
);
}
result.embedded += stale.length;
result.embedded += stale.length - failed;
if (failed > 0) {
recordFailure(result, failed, slug, firstError);
serr(`\n ${slug}: ${failed} chunk(s) failed to embed; embedded the other ${stale.length - failed}`);
}
} catch (e: unknown) {
// Budget/abort-fired cancellations are expected on the way out; don't
// spam per-page "Error embedding" lines when we're shutting down.
if (effectiveSignal.aborted) return;
embedFailures++;
recordFailure(result, stale.length, slug, e);
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
totalProcessedPages++;
@@ -1231,14 +1334,14 @@ async function embedAllStale(
// chunks unembedded means those chunks are stuck (a non-transient embed
// failure), not that we ran out of time. Surface it loudly so it doesn't read
// as a clean run — re-running won't help until the underlying failure is fixed.
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
if (staleOpts?.catchUp && !effectiveSignal.aborted && result.failures > 0) {
const remaining = await engine.countStaleChunks(
signature
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
: (sourceId ? { sourceId } : undefined),
);
if (remaining > 0) {
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${result.failures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
}
}
}
@@ -1358,12 +1461,7 @@ export async function embedBatchWithBackoff(
// If the budget fired we may have been aborted mid-fetch; bubble out.
if (signal?.aborted) throw e;
const msg = e instanceof Error ? e.message : String(e);
// D4: structured detection first (handles gateway-wrapped errors via
// cause chain); message-match as fallback for providers whose wrappers
// strip `cause.status`.
const isRateLimit = detect429FromCause(e)
|| /rate.?limit|429/i.test(msg);
if (!isRateLimit || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
if (!isRateLimitError(e) || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
const delayMs = parseRetryDelayMs(msg);
serr(` [rate-limit] attempt ${attempt + 1}/${MAX_RATE_LIMIT_RETRIES}, waiting ${delayMs}ms...`);
@@ -1373,3 +1471,86 @@ export async function embedBatchWithBackoff(
// Unreachable, but TypeScript needs it.
return embedBatch(texts);
}
/**
* 429 judgment shared by embedBatchWithBackoff (retry decision) and
* embedPageTexts (fan-out decision). D4: structured detection first
* (gateway-wrapped errors via cause chain); message-match as fallback for
* providers whose wrappers strip `cause.status`.
*/
function isRateLimitError(e: unknown): boolean {
const msg = e instanceof Error ? e.message : String(e);
return detect429FromCause(e) || /rate.?limit|429/i.test(msg);
}
/** Walk the cause chain (like detect429FromCause) for the first HTTP status. */
function statusFromCause(e: unknown): number | undefined {
let cur: unknown = e;
for (let depth = 0; depth < 5 && cur !== undefined && cur !== null; depth++) {
const obj = cur as { status?: unknown; statusCode?: unknown; cause?: unknown };
if (typeof obj.status === 'number') return obj.status;
if (typeof obj.statusCode === 'number') return obj.statusCode;
cur = obj.cause;
}
return undefined;
}
/**
* #3037: embed one page's chunk texts with per-chunk failure isolation.
*
* All three embed paths used to send a page's chunks in ONE
* embedBatch call, so one bad chunk (e.g. an oversized chunk the provider
* 400s) left EVERY sibling chunk NULL an ~8.6x blast radius. This wrapper
* tries the batch first (the cheap, common path), and only on a
* PERMANENT-looking batch failure retries once per chunk so one bad chunk
* costs one chunk.
*
* Cost bounding when we do NOT fan out (rethrow instead):
* - 429 / rate limit: embedBatchWithBackoff already retried with backoff;
* fanning out N single-chunk calls would hammer the same limiter N-fold.
* - AITransientError (5xx / network / unknown, per normalizeAIError): the
* batch CONTENT isn't the problem, so isolation can't help during an
* outage it would just multiply failing calls per page.
* - 401/403 (auth): nothing chunk-specific; every call would fail.
* When we DO fan out (permanent request-shaped 4xx like 400/413/422), the
* per-chunk pass happens at most ONCE per page per run and re-spends roughly
* the same tokens the failed batch would have bounded, no recursion. A
* fresh 429 arising DURING the fan-out still gets the normal backoff (each
* single-chunk call goes through embedBatchWithBackoff).
*
* Throws when nothing could be embedded (total failure same contract as
* the pre-#3037 single batch call). Returns `null` at the index of each
* failed chunk otherwise.
*/
async function embedPageTexts(
texts: string[],
opts: EmbedBatchWithBackoffOpts = {},
): Promise<{ embeddings: (Float32Array | null)[]; failed: number; firstError?: unknown }> {
try {
return { embeddings: await embedBatchWithBackoff(texts, opts), failed: 0 };
} catch (e: unknown) {
if (opts.abortSignal?.aborted) throw e; // shutdown, not a chunk problem
if (texts.length <= 1) throw e; // nothing to isolate
if (isRateLimitError(e) || e instanceof AITransientError) throw e;
const status = statusFromCause(e);
if (status === 401 || status === 403) throw e;
const embeddings: (Float32Array | null)[] = [];
let failed = 0;
let firstError: unknown;
for (const t of texts) {
try {
const single = await embedBatchWithBackoff([t], opts);
embeddings.push(single[0] ?? null);
if (single[0] === undefined) { failed++; firstError ??= e; }
} catch (chunkErr: unknown) {
if (opts.abortSignal?.aborted) throw chunkErr;
embeddings.push(null);
failed++;
firstError ??= chunkErr;
}
}
if (failed === texts.length) throw firstError ?? e; // total failure: pre-#3037 contract
return { embeddings, failed, firstError };
}
}
+69 -35
View File
@@ -43,7 +43,7 @@ import {
} from '../core/link-extraction.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { pathToSlug, pruneDir, isSyncable } from '../core/sync.ts';
import { pathToSlug, slugifyPath, pruneDir, isSyncable } from '../core/sync.ts';
// v0.41.18.0: withRetry + isRetryableConnError + WithRetryOpts moved to
// src/core/retry.ts as the canonical primitive. Engine methods
// (addLinksBatch/addTimelineEntriesBatch/upsertChunks) now self-retry via
@@ -269,14 +269,24 @@ export function extractMarkdownLinks(content: string): { name: string; relTarget
export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<string>): string | null {
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
const s1 = join(fileDir, targetNoExt);
if (allSlugs.has(s1)) return s1;
// Issue #1964: wikilinks carry raw Obsidian paths (`[[llm-wiki/entities/AI 3.0]]`)
// but allSlugs holds sync-slugified slugs (`llm-wiki/entities/ai-3.0`). Try the
// raw candidate first (back-compat), then the sync-consistent slugified form.
const hit = (candidate: string): string | null => {
if (allSlugs.has(candidate)) return candidate;
const slugified = slugifyPath(candidate);
if (slugified !== candidate && allSlugs.has(slugified)) return slugified;
return null;
};
const s1 = hit(join(fileDir, targetNoExt));
if (s1) return s1;
const parts = fileDir.split('/').filter(Boolean);
for (let strip = 1; strip <= parts.length; strip++) {
const ancestor = parts.slice(0, parts.length - strip).join('/');
const candidate = ancestor ? join(ancestor, targetNoExt) : targetNoExt;
if (allSlugs.has(candidate)) return candidate;
const candidate = hit(ancestor ? join(ancestor, targetNoExt) : targetNoExt);
if (candidate) return candidate;
}
return null;
@@ -662,7 +672,43 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
return result;
}
const EXTRACT_HELP = `Usage: gbrain extract <subcommand> [flags]
Extraction:
gbrain extract links [--source fs|db] [--source-id <id>] [--dir <brain-dir>]
[--type T] [--since DATE] [--include-frontmatter]
[--workers N|--concurrency N] [--dry-run] [--json]
gbrain extract timeline [--source fs|db] [--source-id <id>] [--dir <brain-dir>]
[--type T] [--since DATE] [--include-frontmatter]
[--infer-dates] [--workers N|--concurrency N]
[--dry-run] [--json]
gbrain extract all [--source fs|db] [--source-id <id>] [--dir <brain-dir>]
[--type T] [--since DATE] [--include-frontmatter]
[--infer-dates] [--workers N|--concurrency N]
[--dry-run] [--json]
gbrain extract <links|timeline> --by-mention --source db
gbrain extract <links|timeline|all> --ner --source db
gbrain extract <timeline|all> --from-meetings --source db
Incremental sweep:
gbrain extract --stale [--source-id <id>] [--include-frontmatter]
[--catch-up] [--dry-run] [--json]
Re-extract links + timeline only for stale pages. DB-source; safe to
cron. --catch-up loops past the 30-minute budget until none remain.
Inspection:
gbrain extract --explain <kind> [--json]
gbrain extract benchmark --pack <name> --kind <type> [--json]
Status:
gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]`;
export async function runExtract(engine: BrainEngine, args: string[]) {
if (args.includes('--help') || args.includes('-h')) {
console.log(EXTRACT_HELP);
return;
}
const subcommand = args[0];
// v0.42 Wave C+D dispatch — new operator surfaces. These intercept
@@ -790,32 +836,7 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
}
if (!subcommand || !['links', 'timeline', 'all'].includes(subcommand)) {
console.error(`Usage: gbrain extract <subcommand> [flags]
Extraction (existing):
gbrain extract links [--source fs|db] [--source-id <id>] [--dir <brain-dir>] [--dry-run] [--json] [--type T] [--since DATE] [--workers N]
gbrain extract timeline [--source fs|db] [--source-id <id>] [--dir <brain-dir>] [--dry-run] [--json] [--type T] [--since DATE] [--workers N]
gbrain extract all [--source fs|db] [--source-id <id>] [--dir <brain-dir>] [--dry-run] [--json] [--type T] [--since DATE] [--workers N]
gbrain extract <links|timeline> --by-mention --source db
gbrain extract <links|timeline|all> --ner --source db
gbrain extract <timeline|all> --from-meetings
Incremental sweep (v0.42.7):
gbrain extract --stale [--source-id <id>] [--catch-up] [--dry-run] [--json]
Re-extract links + timeline ONLY for pages whose extraction is stale
(never extracted, edited since, or extractor bumped). DB-source; safe to
cron. --catch-up loops past the 30-min wall-clock budget until 0 remain.
Inspection (v0.42):
gbrain extract --explain <kind> [--json]
Print resolution chain for one pack-declared extractable kind.
gbrain extract benchmark --pack <name> --kind <type> [--json]
Run a pack's fixture corpus through the extractor (v0.42 reports
fixture shape; LLM dispatch comes in v0.43+).
Status (v0.42):
gbrain extract status [--source-id ID] [--kind X] [--verbose] [--json]
Per-kind 7-day rollup: cost, halt rate, eval pass/fail counts.`);
console.error(EXTRACT_HELP);
process.exit(1);
}
@@ -1458,6 +1479,8 @@ async function extractLinksFromDB(
slugToSources.set(ref.slug, list);
}
let processed = 0, created = 0;
// #2576: skipped-candidate counter — see extractStaleFromDB's twin.
let skippedMissingTarget = 0;
// v0.42.7 (#1696): pages whose links we extracted this run — stamped after
// the loop so a manual `gbrain extract links|all --source db` clears the
// links_extraction_lag doctor signal. Non-dry-run only.
@@ -1514,7 +1537,7 @@ async function extractLinksFromDB(
// endpoint-validation + from/to source-id picking (null = skip: missing
// endpoint OR target only in a non-origin/non-default source).
const resolved = resolveCandidateSources(c, slug, source_id, allSlugs, slugToSources);
if (!resolved) continue;
if (!resolved) { skippedMissingTarget++; continue; }
const { fromSlug, fromSourceId, toSourceId } = resolved;
if (dryRunSeen) {
@@ -1571,6 +1594,9 @@ async function extractLinksFromDB(
if (!jsonMode) {
const label = dryRun ? '(dry run) would create' : 'created';
console.log(`Links: ${label} ${created} from ${processed} pages (db source)`);
if (skippedMissingTarget > 0) {
console.log(`Skipped ${skippedMissingTarget} candidate(s) whose target page doesn't exist (references to non-pages are never persisted).`);
}
if (includeFrontmatter && unresolved.length > 0) {
// Top-20 preview of unresolvable frontmatter names so the user can
// see where the graph has holes (codex tension 6.4).
@@ -1716,7 +1742,7 @@ export async function extractStaleFromDB(
sourceIdFilter?: string;
catchUp: boolean;
},
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number }> {
): Promise<{ linksCreated: number; timelineCreated: number; pagesProcessed: number; staleRemaining: number; skippedMissingTarget?: number }> {
const { dryRun, jsonMode, includeFrontmatter, sourceIdFilter, catchUp } = opts;
const versionTs = LINK_EXTRACTOR_VERSION_TS;
@@ -1768,6 +1794,10 @@ export async function extractStaleFromDB(
let afterPageId = 0;
let linksCreated = 0, timelineCreated = 0, pagesProcessed = 0;
let budgetHit = false;
// #2576: candidates whose endpoint pages don't exist are skipped, not
// persisted. Counted so a dropped reference is observable in the summary
// instead of vanishing silently (the failure mode that hid bug 2).
let skippedMissingTarget = 0;
for (;;) {
const rows = await engine.listStalePagesForExtraction({
@@ -1787,7 +1817,7 @@ export async function extractStaleFromDB(
);
for (const c of extracted.candidates) {
const r = resolveCandidateSources(c, page.slug, page.source_id, allSlugs, slugToSources);
if (!r) continue;
if (!r) { skippedMissingTarget++; continue; }
linkRows.push({
from_slug: r.fromSlug, to_slug: c.targetSlug, link_type: c.linkType,
context: c.context, link_source: c.linkSource, origin_slug: c.originSlug,
@@ -1848,6 +1878,9 @@ export async function extractStaleFromDB(
if (!jsonMode) {
console.log(`Extract --stale: ${linksCreated} link(s) + ${timelineCreated} timeline entr(ies) from ${pagesProcessed} page(s).`);
if (skippedMissingTarget > 0) {
console.log(`Skipped ${skippedMissingTarget} candidate(s) whose target page doesn't exist (references to non-pages are never persisted).`);
}
if (budgetHit && staleRemaining > 0) {
console.log(`Time budget reached — ${staleRemaining} page(s) still stale. Re-run 'gbrain extract --stale' (or pass --catch-up) to continue.`);
}
@@ -1855,9 +1888,10 @@ export async function extractStaleFromDB(
process.stdout.write(JSON.stringify({
action: 'extract_stale_done', links_created: linksCreated, timeline_created: timelineCreated,
pages_processed: pagesProcessed, stale_remaining: staleRemaining, budget_hit: budgetHit,
skipped_missing_target: skippedMissingTarget,
}) + '\n');
}
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining };
return { linksCreated, timelineCreated, pagesProcessed, staleRemaining, skippedMissingTarget };
}
/**
+57 -6
View File
@@ -78,6 +78,17 @@ export async function runImport(
const jsonOutput = args.includes('--json');
const includeGitignored = args.includes('--include-gitignored') || opts.includeGitignored === true;
// #3637: under --json, stdout belongs to the JSON document alone. The
// informational lines below are useful — they just belong on the other
// channel, the same rule progress already follows (CLAUDE.md: "Progress
// always writes to stderr. Stdout stays clean for data output (--json
// payloads)"). Pre-fix, `import --json` prefixed the payload with
// "Found N markdown files", so JSON.parse of stdout failed outright.
const info = (msg: string): void => {
if (jsonOutput) console.error(msg);
else console.log(msg);
};
// T7 (D9): refuse cleanly when init persisted the deferred-setup sentinel,
// unless the user is explicitly skipping embedding via `--no-embed` (in
// which case the chunks land without vectors and the user can backfill
@@ -225,7 +236,7 @@ export async function runImport(
if (opts.exclude && opts.exclude.length > 0) {
const beforeExclude = allFiles.length;
allFiles = allFiles.filter(abs => !matchesAnyGlob(relative(dir, abs), opts.exclude));
console.log(
info(
`Found ${allFiles.length} ${fileTypeLabel} files ` +
`(${beforeExclude - allFiles.length} excluded by --exclude patterns)`,
);
@@ -237,7 +248,7 @@ export async function runImport(
);
}
} else {
console.log(`Found ${allFiles.length} ${fileTypeLabel} files`);
info(`Found ${allFiles.length} ${fileTypeLabel} files`);
}
// Sort newest-first so date-prefixed brain paths get embedded before older ones.
@@ -253,7 +264,7 @@ export async function runImport(
const cp = loadCheckpoint(checkpointPath, dir);
if (cp) {
for (const p of cp.completedPaths) completed.add(p);
console.log(`Resuming from checkpoint: skipping ${completed.size} already-processed files`);
info(`Resuming from checkpoint: skipping ${completed.size} already-processed files`);
}
}
const files = resumeFilter(allFiles, dir, completed);
@@ -261,13 +272,19 @@ export async function runImport(
// Determine actual worker count
const actualWorkers = workerCount > 1 ? workerCount : 1;
if (actualWorkers > 1) {
console.log(`Using ${actualWorkers} parallel workers`);
info(`Using ${actualWorkers} parallel workers`);
}
let imported = 0;
let skipped = 0;
let errors = 0;
let processed = 0;
// Time-based checkpoint floor (see the save site below). Chunking cost scales
// with paragraph count, not bytes, so a single reference-style file can take
// many minutes; a count-only trigger leaves that work undurable.
const CHECKPOINT_MAX_INTERVAL_MS = 120_000;
let lastCheckpointMs = Date.now();
let lastCheckpointSize = completed.size;
let chunksCreated = 0;
const importedSlugs: string[] = [];
const errorCounts: Record<string, number> = {};
@@ -343,7 +360,17 @@ export async function runImport(
// Save checkpoint every 100 SUCCESSFUL adds (not every 100 processed).
// Failed files never enter `completed`, so a flaky file can't push the
// checkpoint past it — the next run will retry it.
if (completed.size > 0 && completed.size % 100 === 0) {
// ...and ALSO save on a time interval. On a corpus with an expensive tail
// `completed` can advance ~1 file per several minutes, so the next
// 100-boundary may be hours away; any kill before it discards every file
// since the last boundary and the run can never converge.
const nowMs = Date.now();
const dueByCount = completed.size > 0 && completed.size % 100 === 0;
const dueByTime = completed.size > lastCheckpointSize
&& nowMs - lastCheckpointMs >= CHECKPOINT_MAX_INTERVAL_MS;
if (dueByCount || dueByTime) {
lastCheckpointMs = nowMs;
lastCheckpointSize = completed.size;
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
try { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
@@ -429,13 +456,37 @@ export async function runImport(
}
}
// Final checkpoint save BEFORE the clear/preserve decision below. The
// periodic triggers above are gated on a 100-file boundary or an interval,
// so a run that ends between them would otherwise leave its tail unsaved.
// This must run before clearCheckpoint() so a clean run still ends with no
// checkpoint file — it only makes the ERROR path's preserved checkpoint
// complete.
if (errors > 0 && completed.size > lastCheckpointSize) {
try {
const cpDir = gbrainPath();
if (!existsSync(cpDir)) {
const { mkdirSync } = await import('fs');
mkdirSync(cpDir, { recursive: true });
}
saveCheckpoint(checkpointPath, {
schema_version: 1,
owner: 'gbrain',
kind: 'import',
dir,
completedPaths: Array.from(completed),
timestamp: new Date().toISOString(),
});
} catch { /* non-fatal: the next run simply redoes the tail */ }
}
// Clear checkpoint on clean completion. On error, the path-based checkpoint
// preserves only the successfully-completed paths, so the next run retries
// failed files automatically (they never entered `completed`).
if (errors === 0) {
clearCheckpoint(checkpointPath);
} else if (existsSync(checkpointPath)) {
console.log(` Checkpoint preserved (${errors} errors). Run again to retry failed files.`);
info(` Checkpoint preserved (${errors} errors). Run again to retry failed files.`);
}
const totalTime = ((Date.now() - startTime) / 1000).toFixed(1);
+27
View File
@@ -1049,6 +1049,9 @@ async function initPGLite(opts: {
// gbrain invocation). mode_prompted=true so the upgrade-time banner doesn't
// also fire on a fresh install. Hands-off: gbrain config set self_upgrade.mode auto
config.self_upgrade = { mode: 'notify', mode_prompted: true, ...(config.self_upgrade ?? {}) };
// MEMORY_VERBS v1 [D6C]: TTHW stamp — `gbrain protocol stats` derives
// install→first-verb-call from this. Idempotent on re-init.
config.protocol_installed_at = config.protocol_installed_at ?? new Date().toISOString();
saveConfig(config);
if (opts.schemaPack) {
process.stderr.write(
@@ -1086,6 +1089,7 @@ async function initPGLite(opts: {
} else {
console.log('Next: gbrain import <dir>');
}
printMemoryVerbsQuickstart();
console.log('');
console.log('When you outgrow local: gbrain migrate --to supabase');
reportModStatus();
@@ -1103,6 +1107,26 @@ async function initPGLite(opts: {
}
}
/**
* MEMORY_VERBS v1 quickstart funnel (E3 + D4B + T1 consent). Printed at the
* end of both init epilogues. The copy-next block is EXACTLY three commands
* (codex DX 9): wire the harness, write a memory, prove the resurrection.
* The demo uses the facts arm only, so it works with NO embedding key [F-B].
*/
function printMemoryVerbsQuickstart(): void {
console.log('');
console.log('Give your agent memory (copy these three commands):');
console.log(' claude mcp add gbrain -- gbrain serve --surface verbs');
console.log(' gbrain remember "I prefer dark mode in every editor" --provenance demo --entity people/me');
console.log(' gbrain recall --entity people/me');
console.log('Now ask your agent in a NEW session — it remembers.');
console.log('');
console.log('Note: memories agents save are readable by every agent connected to');
console.log('this brain; use visibility:"private" for local-only facts.');
console.log('Other harnesses (Codex, OpenClaw): docs/protocol/MEMORY_VERBS_v1.md');
console.log('If `claude` is not found: install Claude Code first, or use the per-harness blocks in that doc.');
}
async function initPostgres(opts: {
databaseUrl: string;
jsonOutput: boolean;
@@ -1297,6 +1321,8 @@ async function initPostgres(opts: {
// gbrain invocation). mode_prompted=true so the upgrade-time banner doesn't
// also fire on a fresh install. Hands-off: gbrain config set self_upgrade.mode auto
config.self_upgrade = { mode: 'notify', mode_prompted: true, ...(config.self_upgrade ?? {}) };
// MEMORY_VERBS v1 [D6C]: TTHW stamp (see the PGLite path).
config.protocol_installed_at = config.protocol_installed_at ?? new Date().toISOString();
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
if (opts.schemaPack) {
@@ -1331,6 +1357,7 @@ async function initPostgres(opts: {
} else {
console.log('Next: gbrain import <dir>');
}
printMemoryVerbsQuickstart();
reportModStatus();
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
const { VERSION } = await import('../version.ts');
+84 -1
View File
@@ -55,6 +55,13 @@ interface RecipeFrontmatter {
health_checks: HealthCheck[];
setup_time: string;
cost_estimate?: string;
/**
* Repo-relative dirs (slug prefixes, trailing '/') this recipe's collector
* writes files to. Ground truth for the `db_only_collector_collision`
* doctor check (issue #2788): output inside a db_only path is silently
* skipped by sync and import (auto-gitignored).
*/
output_paths: string[];
}
interface ParsedRecipe {
@@ -106,7 +113,20 @@ interface AnyOfCheck {
checks: HealthCheck[];
}
type HealthCheck = string | HttpCheck | EnvExistsCheck | CommandCheck | AnyOfCheck;
/**
* Staleness-aware check type (issue #2787, reported by @alexputici). All
* other types are point-in-time a sense whose gateway is up and env vars
* are set passes forever even when zero data flows. This one reads the
* integration's heartbeat file and FAILS when the newest event is older
* than the declared cadence (`max_age`, e.g. "48h", "2d", "90m").
*/
interface HeartbeatMaxAgeCheck {
type: 'heartbeat_max_age';
max_age: string;
label?: string;
}
type HealthCheck = string | HttpCheck | EnvExistsCheck | CommandCheck | AnyOfCheck | HeartbeatMaxAgeCheck;
interface CheckResult {
integration: string;
@@ -141,6 +161,26 @@ export function secretEnv(): Record<string, string | undefined> {
return process.env;
}
/**
* Parse a heartbeat_max_age duration string ("30s", "90m", "48h", "2d")
* into milliseconds. Returns null on anything unparseable.
*/
export function parseMaxAge(s: string): number | null {
const m = /^(\d+(?:\.\d+)?)\s*(s|m|h|d)$/i.exec(String(s).trim());
if (!m) return null;
const n = Number(m[1]);
if (!Number.isFinite(n) || n <= 0) return null;
const unit = { s: 1_000, m: 60_000, h: 3_600_000, d: 86_400_000 }[m[2].toLowerCase() as 's' | 'm' | 'h' | 'd'];
return n * unit;
}
/** Human-readable age for heartbeat_max_age output ("3d", "17h", "42m"). */
function formatAge(ms: number): string {
if (ms >= 86_400_000) return `${Math.floor(ms / 86_400_000)}d`;
if (ms >= 3_600_000) return `${Math.floor(ms / 3_600_000)}h`;
return `${Math.max(0, Math.floor(ms / 60_000))}m`;
}
/** Expand $VAR references with gateway-env (config-folded) values */
export function expandVars(s: string): string {
const env = secretEnv();
@@ -299,6 +339,29 @@ export async function executeHealthCheck(
}
}
case 'heartbeat_max_age': {
// No embedded gate: reads only the local heartbeat file — no exec, no
// network. Safe for user-provided recipes.
const maxMs = parseMaxAge(check.max_age);
if (maxMs === null) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat_max_age'}: invalid max_age '${check.max_age}' (use e.g. 90m, 48h, 2d)` };
}
const entries = readHeartbeat(integrationId);
if (entries.length === 0) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat'}: no heartbeat events in the last 30 days (expected activity within ${check.max_age}) — the sense has stopped producing data` };
}
let newest = 0;
for (const e of entries) {
const t = new Date(e.ts).getTime();
if (Number.isFinite(t) && t > newest) newest = t;
}
const ageMs = Date.now() - newest;
if (ageMs > maxMs) {
return { ...base, status: 'fail', output: `${check.label || 'heartbeat'}: last event ${formatAge(ageMs)} ago exceeds max_age ${check.max_age} — the sense has stopped producing data` };
}
return { ...base, status: 'ok', output: `${check.label || 'heartbeat'}: last event ${formatAge(ageMs)} ago (within ${check.max_age})` };
}
case 'any_of': {
for (const sub of check.checks) {
const result = await executeHealthCheck(sub, integrationId, isEmbedded);
@@ -340,6 +403,7 @@ export function parseRecipe(content: string, filename: string): ParsedRecipe | n
health_checks: (data.health_checks || []) as HealthCheck[],
setup_time: data.setup_time || 'unknown',
cost_estimate: data.cost_estimate,
output_paths: Array.isArray(data.output_paths) ? data.output_paths.map(String) : [],
},
body: body.trim(),
filename,
@@ -403,6 +467,25 @@ function loadAllRecipes(): ParsedRecipe[] {
return recipes;
}
/**
* Output paths of every CONFIGURED recipe (secrets present the collector
* can actually be running). Ground truth for the
* `db_only_collector_collision` doctor check and the sync-time warning
* (issue #2788). Unconfigured recipes are skipped: a collector that can't
* run can't silently die.
*/
export function getConfiguredCollectorOutputs(): Array<{ id: string; output_path: string }> {
const out: Array<{ id: string; output_path: string }> = [];
for (const r of loadAllRecipes()) {
if (r.frontmatter.output_paths.length === 0) continue;
if (getStatus(r) === 'available') continue;
for (const p of r.frontmatter.output_paths) {
out.push({ id: r.frontmatter.id, output_path: p });
}
}
return out;
}
function findRecipe(id: string): ParsedRecipe | null {
const recipes = loadAllRecipes();
const exact = recipes.find(r => r.frontmatter.id === id);
+10 -2
View File
@@ -12,7 +12,7 @@
* Subcommands:
* gbrain integrity check Read-only report to stdout
* gbrain integrity auto Three-bucket repair with confidence
* gbrain integrity --dry-run Same as auto, no writes
* gbrain integrity auto --dry-run Same as auto, no writes
*
* Three-bucket confidence (contract with x_handle_to_tweet resolver):
* >= 0.8 auto-repair through BrainWriter transaction
@@ -449,6 +449,7 @@ async function cmdAuto(args: string[]): Promise<void> {
let bucketReview = 0;
let bucketSkip = 0;
let bucketErr = 0;
let bucketDeadLink = 0;
let pagesProcessed = 0;
const { createProgress } = await import('../core/progress.ts');
@@ -549,7 +550,13 @@ async function cmdAuto(args: string[]): Promise<void> {
hit: { slug, line: hit.line, rawLine: hit.url, phrase: 'dead-link' },
reason: `dead link: ${result.value.reason ?? 'unknown'}`,
});
bucketReview++;
// Dead links have no confidence score and are never written to
// the review file (appendReview() is only called from the
// bare-tweet path above) — they land in the skip log via
// logSkip() a few lines up. Count them separately so the
// printed "Review queue" total only ever reflects what's
// actually in ~/.gbrain/integrity-review.md.
bucketDeadLink++;
}
} catch {
/* transient; don't fail the run */
@@ -568,6 +575,7 @@ async function cmdAuto(args: string[]): Promise<void> {
console.log(`Review queue (≥${reviewLower} <${confidenceThreshold}): ${bucketReview}`);
console.log(`Skipped (<${reviewLower}): ${bucketSkip}`);
if (bucketErr > 0) console.log(`Resolver errors: ${bucketErr}`);
if (bucketDeadLink > 0) console.log(`Dead links surfaced (see skipped log): ${bucketDeadLink}`);
console.log(`\nReview queue: ${getReviewFile()}`);
console.log(`Skipped log: ${getLogFile()}`);
console.log(`Progress: ${getProgressFile()}`);
+20 -6
View File
@@ -22,6 +22,16 @@ function hasFlag(args: string[], flag: string): boolean {
return args.includes(flag);
}
/**
* Resolve the canonical positive-polarity pull flag while preserving queued
* jobs that still carry the legacy inverse `noPull` key.
*/
export function resolveJobPull(data: Record<string, unknown>): boolean {
if (typeof data.pull === 'boolean') return data.pull;
if (typeof data.noPull === 'boolean') return !data.noPull;
return true;
}
/**
* Long-lived workers outlive operator config changes. Re-stamp the AI gateway
* from DB-backed model config immediately before queued jobs enter gateway-backed
@@ -1414,7 +1424,7 @@ export async function registerBuiltinHandlers(
worker.register('sync', async (job) => {
const { performSync } = await import('./sync.ts');
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
const noPull = !!job.data.noPull;
const noPull = !resolveJobPull(job.data);
// noEmbed defaults to true (embed is a separate job — submit `embed --stale`
// after sync, OR run via the autopilot cycle which has its own embed phase).
// Caller can opt in by passing { noEmbed: false } in job params.
@@ -1855,8 +1865,7 @@ export async function registerBuiltinHandlers(
? (job.data.phases as string[]).filter(p => validPhases.has(p as any))
: undefined;
// Pull default: legacy `true` for back-compat; explicit boolean wins.
const pull = typeof job.data.pull === 'boolean' ? job.data.pull : true;
const pull = resolveJobPull(job.data);
// #2194 fix #2 / codex #5 (D4): claim-time cooldown guard. A job already
// queued or retrying (max_attempts:2) can reach the worker after the
@@ -2194,8 +2203,9 @@ export async function registerBuiltinHandlers(
// migration that retypes 25K+ pages, creates alias rows, converts edge-
// shaped pages to link rows, AND flips the active pack at end of run.
// manual_only via src/core/onboard/render.ts:MANUAL_ONLY_PROTECTED_JOBS.
// Operator path: `gbrain jobs submit unify-types --allow-protected --params
// '{"target_pack":"gbrain-base-v2"}'`.
// Dry-run preview: `gbrain jobs submit unify-types --allow-protected
// --params '{"target_pack":"gbrain-base-v2"}'`; apply with
// '{"target_pack":"gbrain-base-v2","apply":true}'.
worker.register('unify-types', async (job) => {
const { runUnifyTypes } = await import('../core/schema-pack/unify-types-handler.ts');
const data = (job.data ?? {}) as {
@@ -2213,7 +2223,11 @@ export async function registerBuiltinHandlers(
} as unknown as import('../core/operations.ts').OperationContext;
return await runUnifyTypes(ctx, {
target_pack: data.target_pack,
apply: data.apply ?? true,
// #1575: default matches the handler interface's "Default false
// (dry-run)" — a destructive one-shot migration must be opted into
// with apply:true (the onboard remediation + the printed migration
// command both carry it explicitly).
apply: data.apply ?? false,
sourceId: data.sourceId,
onProgress: (msg: string) => {
job.updateProgress({ phase: 'unify-types', message: msg }).catch(() => {});
+2
View File
@@ -26,6 +26,7 @@ import { v0_28_0 } from './v0_28_0.ts';
import { v0_29_1 } from './v0_29_1.ts';
import { v0_31_0 } from './v0_31_0.ts';
import { v0_32_2 } from './v0_32_2.ts';
import { v0_43_0 } from './v0_43_0.ts';
export const migrations: Migration[] = [
v0_11_0,
@@ -43,6 +44,7 @@ export const migrations: Migration[] = [
v0_29_1,
v0_31_0,
v0_32_2,
v0_43_0,
];
/** Look up a migration by exact version string. */
+33
View File
@@ -0,0 +1,33 @@
/**
* v0.43.0.0 migration MEMORY_VERBS v1 (Cathedral 1).
*
* PITCH-ONLY. There is NO schema or data migration: the five frozen memory
* verbs (recall/remember/entity/synthesize/forget) ride the existing facts,
* pages, and typed-graph tables. This entry exists solely so `gbrain
* post-upgrade` / the self-upgrade NOTIFY channel actively tells an existing
* install that the verbs landed and how to switch a harness onto them the
* propagation path the verbs otherwise lacked (the default surface stays
* `full`, so an upgrade alone does NOT steer agents to the verbs).
*
* The orchestrator is a no-op that reports `complete` immediately
* idempotent by construction (it does nothing), so apply-migrations records
* the ledger row and moves on.
*/
import type { Migration, OrchestratorOpts, OrchestratorResult } from './types.ts';
async function orchestrator(_opts: OrchestratorOpts): Promise<OrchestratorResult> {
// No schema/data work — MEMORY_VERBS v1 is a façade over existing tables.
return { version: '0.43.0', status: 'complete', phases: [] };
}
export const v0_43_0: Migration = {
version: '0.43.0',
featurePitch: {
headline:
'Five memory verbs — recall, remember, entity, synthesize, forget — are now the agent-facing memory protocol (MEMORY_VERBS v1).',
description:
'Point any MCP harness at `gbrain serve --surface verbs` to expose exactly these five self-describing tools instead of the full op wall: remember(fact, provenance) writes durable facts; recall(query|entity, budget_tokens) returns budget-packed memory; entity(name) is a zero-LLM card; synthesize(question) is the explicitly-expensive cross-page answer; forget(id) expires a fact. Existing `gbrain serve` (full surface) keeps working and now also lists the verbs, but `--surface verbs` is the clean agent surface. Set a default with `gbrain config set mcp_surface verbs`. Verify any endpoint with `gbrain protocol conformance`; see usage with `gbrain protocol stats`. Full contract: docs/protocol/MEMORY_VERBS_v1.md.',
},
orchestrator,
};
+107 -10
View File
@@ -34,10 +34,19 @@ import {
resolveModel,
type ModelTier,
} from '../core/model-config.ts';
import { resolveRecipe } from '../core/ai/model-resolver.ts';
const TIERS: ModelTier[] = ['utility', 'reasoning', 'deep', 'subagent'];
const PER_TASK_KEYS: Array<{ key: string; tier: ModelTier; description: string }> = [
interface PerTaskModelRoute {
key: string;
tier: ModelTier;
description: string;
deprecatedConfigKey?: string;
envVar?: string;
}
const PER_TASK_KEYS: PerTaskModelRoute[] = [
{ key: 'models.dream.synthesize', tier: 'reasoning', description: 'Dream synthesis (conversation → brain pages)' },
{ key: 'models.dream.synthesize_verdict', tier: 'utility', description: 'Dream synthesis verdict (Haiku judge)' },
{ key: 'models.dream.patterns', tier: 'reasoning', description: 'Pattern discovery (cross-take themes)' },
@@ -49,6 +58,13 @@ const PER_TASK_KEYS: Array<{ key: string; tier: ModelTier; description: string }
{ key: 'models.eval.longmemeval', tier: 'reasoning', description: 'LongMemEval benchmark answer-gen' },
{ key: 'models.eval.contradictions_judge', tier: 'utility', description: 'Contradiction probe judge (v0.34 temporal-aware)' },
{ key: 'models.expansion', tier: 'utility', description: 'Query expansion for hybrid search' },
{
key: 'models.contextual_synopsis',
tier: 'utility',
description: 'Per-chunk contextual synopsis generation',
deprecatedConfigKey: 'contextual_retrieval.haiku_model',
envVar: 'GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL',
},
{ key: 'models.chat', tier: 'reasoning', description: 'Default `gateway.chat()` model' },
];
@@ -66,12 +82,26 @@ interface ModelsReport {
aliases: { defaults: Record<string, string>; user: Record<string, string> };
}
async function probeSource(engine: BrainEngine, configKey: string, envVar: string): Promise<string | null> {
async function probeSource(
engine: BrainEngine,
route: Pick<PerTaskModelRoute, 'key' | 'tier' | 'deprecatedConfigKey' | 'envVar'>,
): Promise<string | null> {
// For per-task probes, return the source the resolver USED (config / env /
// tier default / hardcoded). The resolver itself is the source of truth;
// we re-walk a subset of its precedence here to attribute the value.
const configVal = await engine.getConfig(configKey);
if (configVal && configVal.trim()) return `config: ${configKey}`;
// tier default / hardcoded). Keep this walk in the same order as
// resolveModel so dedicated task env vars and compatibility keys are
// attributed truthfully in `gbrain models` output.
const configVal = await engine.getConfig(route.key);
if (configVal && configVal.trim()) return `config: ${route.key}`;
if (route.deprecatedConfigKey) {
const deprecated = await engine.getConfig(route.deprecatedConfigKey);
if (deprecated && deprecated.trim()) return `config: ${route.deprecatedConfigKey}`;
}
const globalDefault = await engine.getConfig('models.default');
if (globalDefault && globalDefault.trim()) return 'config: models.default';
const tierKey = `models.tier.${route.tier}`;
const tierValue = await engine.getConfig(tierKey);
if (tierValue && tierValue.trim()) return `config: ${tierKey}`;
const envVar = route.envVar ?? 'GBRAIN_MODEL';
if (process.env[envVar] && process.env[envVar]!.trim()) return `env: ${envVar}`;
return null;
}
@@ -96,9 +126,16 @@ async function buildReport(engine: BrainEngine): Promise<ModelsReport> {
}
const per_task: ModelsReport['per_task'] = [];
for (const { key, tier, description } of PER_TASK_KEYS) {
const resolved = await resolveModel(engine, { configKey: key, tier, fallback: TIER_DEFAULTS[tier] });
const explicit = await probeSource(engine, key, 'GBRAIN_MODEL');
for (const route of PER_TASK_KEYS) {
const { key, tier, description, deprecatedConfigKey, envVar } = route;
const resolved = await resolveModel(engine, {
configKey: key,
deprecatedConfigKey,
envVar,
tier,
fallback: TIER_DEFAULTS[tier],
});
const explicit = await probeSource(engine, route);
const source = explicit ?? `tier.${tier}`;
per_task.push({ key, tier, resolved, source, description });
}
@@ -186,6 +223,44 @@ function classifyError(err: unknown): { status: ProbeStatus; message: string } {
return { status: 'unknown', message: msg };
}
const OPENAI_COMPAT_V1_HINT =
'If the API key is correct, the base URL may be missing the /v1 suffix. ' +
'OpenAI-shaped proxies (codex-proxy, Azure-OpenAI mirrors, LiteLLM fronting an OpenAI route) ' +
'serve /v1/chat/completions and 401 on the bare path. ' +
'Confirm with: `curl <base>/models` returns 200 with the same bearer, then append /v1 to the base URL.';
/**
* Fix-hint for the openai-compatible-proxy `/v1`-suffix trap.
*
* An OpenAI-shaped proxy whose base URL omits `/v1` (codex-proxy, some
* Azure-OpenAI mirrors, a LiteLLM proxy fronting an OpenAI-route backend)
* serves `/v1/chat/completions` and returns 401 on the bare `/chat/completions`
* the AI SDK appends to the base. `classifyError` reads that 401 as `auth` and
* points the operator at the bearer token, when the real fix is the URL shape.
*
* Returns the corrective hint only when the model routes through an
* openai-compatible recipe (proxy tier, not native anthropic/openai/google),
* `baseURL` is set, and `baseURL` does not already end in `/v1` (optionally with
* a trailing slash). Pure: recipe resolution is synchronous and does no
* network/engine work; any resolution failure returns undefined.
*
* @internal exported for tests.
*/
export function openAiCompatV1Hint(
modelStr: string,
baseURL: string | undefined | null,
): string | undefined {
if (!baseURL || !baseURL.trim()) return undefined;
if (/\/v1\/?$/.test(baseURL.trim())) return undefined;
try {
const { recipe } = resolveRecipe(modelStr);
if (recipe.tier !== 'openai-compat') return undefined;
return OPENAI_COMPAT_V1_HINT;
} catch {
return undefined;
}
}
/**
* Validate the configured embedding model + dims combo without spending tokens.
* Catches the bug class where a brain configured for Voyage with a missing or
@@ -523,7 +598,29 @@ async function probeModel(modelStr: string, touchpoint: 'chat' | 'expansion'): P
}
} catch (err) {
const { status, message } = classifyError(err);
return { model: modelStr, touchpoint, status, message, elapsed_ms: Date.now() - start };
const result: ProbeResult = { model: modelStr, touchpoint, status, message, elapsed_ms: Date.now() - start };
// An openai-compatible proxy whose base URL omits `/v1` returns 401 (not
// 404) on the bare `/chat/completions` path, which classifyError reads as
// `auth`. Attach the URL-shape hint so the operator doesn't chase the
// bearer token. Fail open: any error resolving the base URL yields no hint
// and never breaks the probe.
if (status === 'auth') {
try {
const { loadConfig } = await import('../core/config.ts');
const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts');
const fileCfg = loadConfig();
if (fileCfg) {
const cfg = buildGatewayConfig(fileCfg);
const { recipe } = resolveRecipe(modelStr);
const baseURL = cfg.base_urls?.[recipe.id] ?? recipe.base_url_default;
const hint = openAiCompatV1Hint(modelStr, baseURL);
if (hint) result.fix = hint;
}
} catch {
// fail open — no hint
}
}
return result;
}
}
+1 -1
View File
@@ -251,7 +251,7 @@ async function renderPackUpgradeExplain(
` Page-to-link: ${result.per_phase.page_to_link.would_convert} edges across ${result.per_phase.page_to_link.rules} rules\n` +
` Page-to-alias: ${result.per_phase.page_to_alias.would_alias} aliases across ${result.per_phase.page_to_alias.rules} rules\n` +
`\nRun the migration with:\n` +
` gbrain jobs submit unify-types --allow-protected --params '${JSON.stringify({ target_pack: targetPack })}'\n`,
` gbrain jobs submit unify-types --allow-protected --params '${JSON.stringify({ target_pack: targetPack, apply: true })}'\n`,
);
if (result.warnings.length > 0) {
process.stdout.write(`\nWarnings:\n`);
+340
View File
@@ -0,0 +1,340 @@
/**
* `gbrain pglite-repair` diagnose and repair a torn-WAL PGLite data dir
* in place (#223 / #1670 / #2575 recovery, the manual surface for the
* auto-repair that `PGLiteEngine.connect()` runs on `wasm-abort` failures).
*
* Never connects an engine (the whole point is that the DB won't open), so it
* works when auto-repair is disabled (GBRAIN_PGLITE_WAL_REPAIR=off) or was
* skipped. `--dry-run` is strictly read-only. The real run:
*
* validate (BEFORE locking `acquireLock` mkdirs the data dir, and a
* typo'd --path must not create directories) TTY confirm unless --yes
* acquireLock (refuses a reaped acquisition: a corrupt-lock reap cannot
* prove the holder is dead, and WAL surgery under a possibly-live writer is
* never correct; a live `gbrain serve` holder fast-fails via
* LiveServeLockError) re-validate under the lock repair receipt.
*
* There is deliberately NO --force: force-removing `.gbrain-lock` while the
* holder is alive would reopen exactly the concurrent-writer corruption hole
* #2348 closed. For catalog corruption (the `corrupt` classifier verdict) WAL
* repair does not help `gbrain reinit-pglite` is the rebuild path.
*/
import { createInterface } from 'readline';
import { loadConfig, gbrainPath } from '../core/config.ts';
import { acquireLock, releaseLock, LiveServeLockError, msSinceLastReap } from '../core/pglite-lock.ts';
import {
inspectPgliteDataDir,
listRepairBackups,
readRepairSidecar,
recordRepairAttempt,
repairPgliteWal,
validateWalRepairTarget,
WalRepairError,
} from '../core/pglite-repair.ts';
interface RepairCmdOpts {
dryRun: boolean;
yes: boolean;
jsonOutput: boolean;
customPath: string | null;
help: boolean;
}
class UnknownFlagError extends Error {}
function parseArgs(args: string[]): RepairCmdOpts {
const opts: RepairCmdOpts = { dryRun: false, yes: false, jsonOutput: false, customPath: null, help: false };
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--dry-run') opts.dryRun = true;
else if (a === '--yes' || a === '-y') opts.yes = true;
else if (a === '--json') opts.jsonOutput = true;
else if (a === '--path') {
const val = args[++i];
// F1: a valueless --path (typo / shell mangling) must NOT fall through
// to the configured default brain and run surgery on the wrong dir.
if (val === undefined || val.startsWith('-')) throw new UnknownFlagError('--path requires a directory argument');
opts.customPath = val;
}
else if (a === '--help' || a === '-h') opts.help = true;
// Reject unknown args on a DESTRUCTIVE command (codex): silently ignoring
// a typo like `--dry-rnu` would run a real WAL reset instead of a dry run.
else throw new UnknownFlagError(`unknown argument: ${a}`);
}
return opts;
}
function printHelp(): void {
console.log(`gbrain pglite-repair — repair a torn-WAL PGLite data dir in place
The default gbrain engine (PGLite) can fail to open after an unclean shutdown
(commonly a macOS-upgrade reboot) with "RuntimeError: Aborted()". The cause is
torn WAL/checkpoint state on disk, not a macOS WASM bug. This command resets
the WAL in place (pg_resetwal semantics): data files are preserved;
transactions not checkpointed before the corruption may be lost. The
pre-repair pg_wal + pg_control are kept in a sibling backup directory.
Usage:
gbrain pglite-repair --dry-run [--json] [--path <dir>] diagnose only
gbrain pglite-repair [--yes] [--json] [--path <dir>] repair (confirm on TTY)
Flags:
--dry-run Read-only diagnosis of the data dir. Mutates nothing.
--yes, -y Skip the confirmation prompt (required in non-TTY runs).
--json Machine-readable output on stdout.
--path <dir> Repair a specific data dir (default: the configured brain).
Notes:
Auto-repair runs on ordinary commands by default; disable it with
GBRAIN_PGLITE_WAL_REPAIR=off and use this command deliberately.
Catalog corruption (58P01 / pgvector load failure) is NOT repairable in
place use \`gbrain reinit-pglite\` for that class.`);
}
function emitError(jsonOutput: boolean, code: string, message: string): void {
if (jsonOutput) {
console.log(JSON.stringify({ status: 'error', code, message }));
} else {
console.error(`Error (${code}): ${message}`);
}
}
async function promptYesNo(question: string): Promise<boolean> {
// Prompt on stderr: stdout stays clean for --json payloads.
const rl = createInterface({ input: process.stdin, output: process.stderr });
return new Promise((resolve) => {
rl.question(`${question} [y/N] `, (answer) => {
rl.close();
resolve(/^y(es)?$/i.test(answer.trim()));
});
});
}
export async function runPgliteRepair(args: string[]): Promise<number> {
let opts: RepairCmdOpts;
try {
opts = parseArgs(args);
} catch (err) {
if (err instanceof UnknownFlagError) {
const jsonOut = args.includes('--json');
emitError(jsonOut, 'unknown_flag', `${err.message}. Run \`gbrain pglite-repair --help\`.`);
return 2;
}
throw err;
}
if (opts.help) {
printHelp();
return 0;
}
// Resolve dir: --path > config > default brain path. With an explicit
// --path we skip the engine check (repairing an arbitrary dir is the point).
let dataDir = opts.customPath;
if (!dataDir) {
const cfg = loadConfig();
if (cfg?.engine !== 'pglite') {
emitError(
opts.jsonOutput,
'not_pglite',
`gbrain pglite-repair is for PGLite brains (current engine: ${cfg?.engine || 'none'}). ` +
'Pass --path <dir> to repair a specific data dir.',
);
return 1;
}
dataDir = cfg.database_path || gbrainPath('brain.pglite');
}
// Read-only diagnosis first — BEFORE any lock (acquireLock mkdirs the data
// dir; a typo'd --path must produce a clean refusal with zero side effects).
const validation = validateWalRepairTarget(dataDir);
const diagnosis = inspectPgliteDataDir(dataDir);
if (opts.dryRun) {
if (opts.jsonOutput) {
console.log(JSON.stringify({
status: 'ok',
action: 'dry-run',
data_dir: dataDir,
validation,
diagnosis,
}));
} else {
console.log(`PGLite data dir: ${dataDir}`);
console.log(` Verdict: ${diagnosis.verdict}${diagnosis.detail}`);
console.log(` PG_VERSION: ${diagnosis.pgVersion ?? '(unreadable)'} pg_control: ${diagnosis.pgControlOk ? 'ok (8192 bytes)' : 'BAD'}`);
console.log(` WAL segments: ${diagnosis.walSegments.length} stale postmaster.pid: ${diagnosis.postmasterPid ? 'YES' : 'no'}`);
console.log(` Lock: ${diagnosis.lockHeld ? `HELD by live PID ${diagnosis.lockHolderPid}` : 'free'}`);
if (diagnosis.recentAttempts.length > 0) {
console.log(` Repair attempts on record: ${diagnosis.recentAttempts.map((a) => `${a.outcome}@${new Date(a.ts).toISOString()}`).join(', ')}`);
}
if (diagnosis.backupDirs.length > 0) {
console.log(` Repair backups on disk: ${diagnosis.backupDirs.join(', ')}`);
}
if (!validation.ok) {
console.log(` Repairable: NO — ${validation.detail}`);
} else if (diagnosis.verdict === 'looks-healthy') {
console.log(' Repairable: yes — but no unclean-shutdown markers found; repair is likely');
console.log(' unnecessary. Run `gbrain pglite-repair --yes` ONLY if PGLite fails to open');
console.log(' with `RuntimeError: Aborted()` (repair discards the un-checkpointed WAL tail).');
} else {
console.log(' Repairable: yes — run `gbrain pglite-repair --yes` to reset the WAL in place.');
}
}
return 0;
}
if (!validation.ok) {
emitError(opts.jsonOutput, `refused_${validation.reason}`, validation.detail);
return 1;
}
if (diagnosis.lockHeld) {
emitError(
opts.jsonOutput,
'refused_locked',
`another gbrain process (PID ${diagnosis.lockHolderPid}) is using this brain — stop it, then re-run.`,
);
return 1;
}
const sinceReap = msSinceLastReap(dataDir);
const REAP_QUARANTINE_MS = 10 * 60 * 1000;
if (sinceReap !== null && sinceReap >= 0 && sinceReap < REAP_QUARANTINE_MS) {
emitError(
opts.jsonOutput,
'refused_reap_quarantine',
`a lock on this brain was reaped ${Math.round(sinceReap / 1000)}s ago from a holder whose ` +
'liveness could not be verified — that process may still be writing. Confirm no gbrain ' +
`process is running (\`pgrep -af gbrain\`), wait ${Math.ceil((REAP_QUARANTINE_MS - sinceReap) / 60000)} more minute(s), then re-run.`,
);
return 1;
}
if (!opts.yes) {
if (!process.stdin.isTTY) {
emitError(opts.jsonOutput, 'no_tty_no_yes', 'Non-TTY environment requires --yes to confirm the WAL reset.');
return 1;
}
console.error(`About to reset the WAL of ${dataDir} in place.`);
console.error('Data files are preserved; un-checkpointed transactions may be lost.');
console.error('The current pg_wal + pg_control are kept in a sibling backup directory.');
const confirmed = await promptYesNo('Repair now?');
if (!confirmed) {
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'aborted', reason: 'user_declined' }));
} else {
console.log('Aborted. Data dir untouched.');
}
return 0;
}
}
// Short timeout: the diagnosis said the lock is free; if we still can't get
// it quickly, someone raced us — refuse rather than queue behind them.
let lock;
try {
lock = await acquireLock(dataDir, { timeoutMs: 5_000 });
} catch (err) {
if (err instanceof LiveServeLockError) {
emitError(
opts.jsonOutput,
'refused_live_serve',
`a live \`gbrain serve\` (MCP) process holds this brain — stop \`gbrain serve\` first, then re-run. ${String((err as Error).message)}`,
);
return 1;
}
emitError(opts.jsonOutput, 'refused_lock_timeout', String((err as Error)?.message ?? err));
return 1;
}
try {
if (!lock.acquired) {
emitError(opts.jsonOutput, 'refused_locked', 'could not acquire the PGLite data-dir lock — another gbrain process is using this brain.');
return 1;
}
if (lock.reaped) {
// A reaped acquisition (dead-PID or corrupt-lock-file reap) cannot prove
// the prior holder is gone. WAL surgery under a possibly-live writer is
// never correct — no --force by design.
emitError(
opts.jsonOutput,
'refused_reaped_lock',
'the data-dir lock was acquired by reaping a prior holders lock — ' +
'another gbrain process may still be using this brain. Confirm no gbrain ' +
'process is running, then re-run (a cleanly-acquired lock enables repair).',
);
return 1;
}
// Cheap re-validate under the lock (TOCTOU window between diagnosis and
// lock acquisition).
const revalidation = validateWalRepairTarget(dataDir);
if (!revalidation.ok) {
emitError(opts.jsonOutput, `refused_${revalidation.reason}`, revalidation.detail);
return 1;
}
process.stderr.write(`Repairing WAL of ${dataDir} in place…\n`);
const sidecar = readRepairSidecar(dataDir);
// F4: only reuse a FRESH (<24h) episode backup — a stale pin may predate
// real data (same bound as the auto seam's episodeFresh).
const episodeFresh =
sidecar.episodeStartedAt !== null &&
Date.now() - sidecar.episodeStartedAt >= 0 &&
Date.now() - sidecar.episodeStartedAt < 24 * 3600 * 1000;
let receipt;
try {
receipt = await repairPgliteWal(dataDir, {
reuseBackupPath: episodeFresh ? sidecar.episodeBackupPath ?? undefined : undefined,
});
} catch (err) {
if (err instanceof WalRepairError) {
// Reset failed after the backup was taken — report the restore's REAL
// outcome so the user knows whether the dir is back or in reset state.
recordRepairAttempt(dataDir, 'failed', err.receipt.backupPath);
emitError(
opts.jsonOutput,
'repair_failed',
err.message + (err.restore.restored
? ` (data dir restored to its pre-repair state; backup kept at ${err.receipt.backupPath})`
: ` (RESTORE ALSO FAILED — the dir is in a reset state; your pre-repair files are intact at ${err.receipt.backupPath}: ${err.restore.detail ?? ''})`),
);
return 1;
}
recordRepairAttempt(dataDir, 'failed', sidecar.episodeBackupPath);
emitError(opts.jsonOutput, 'repair_failed', String((err as Error)?.message ?? err));
return 1;
}
// "repaired" here means the reset completed; the next connect PROVES it.
// Record a FAILED attempt (not repaired-with-closeEpisode:false, which
// leaves the episode null when none was open — codex): this opens/keeps an
// episode pinned to this backup so a later healthy connect closes it and
// prunes, and repeated manual runs during one incident reuse the pinned
// backup instead of deleting the pre-damage forensic copy.
recordRepairAttempt(dataDir, 'failed', receipt.backupPath);
if (opts.jsonOutput) {
console.log(JSON.stringify({
status: 'ok',
action: 'repaired',
data_dir: receipt.dataDir,
backup_path: receipt.backupPath,
backed_up: receipt.backedUpFiles,
reused_episode_backup: receipt.reusedEpisodeBackup,
reset_segment: receipt.resetSegment,
timeline_id: receipt.timelineId,
wal_seg_size: receipt.walSegSize,
repaired_at: receipt.repairedAt,
backups_on_disk: listRepairBackups(dataDir),
}));
} else {
console.log('WAL reset complete.');
console.log(` Data dir: ${receipt.dataDir}`);
console.log(` Backup: ${receipt.backupPath}${receipt.reusedEpisodeBackup ? ' (reused this episodes existing backup)' : ''}`);
console.log(` Reset segment: ${receipt.resetSegment} (timeline ${receipt.timelineId}, ${receipt.walSegSize / (1024 * 1024)}MB segments)`);
console.log(' Data files were preserved; un-checkpointed transactions may be lost.');
console.log(' Next: run any gbrain command to reopen the brain, then `gbrain doctor`.');
}
return 0;
} finally {
await releaseLock(lock);
}
}

Some files were not shown because too many files have changed in this diff Show More