mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 00:48:18 +00:00
master
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry TanandClaude Fable 5
|
d35c9c9e44 |
v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code (#3975)
* docs(designs): agent-bootstrap plan + design docs (normative, review-absorbed) The scrubbed, in-repo sources of truth for the gbrain bootstrap wave: AGENT_BOOTSTRAP_DESIGN.md (product scope/sequencing) and AGENT_BOOTSTRAP_PLAN.md (implementation; all review-finding IDs inlined). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): format spec, question bank, identity templates, bundled assets agent.json manifest (format_version 1, initialized sentinel) + machine-local install receipt [CX2-1, CX2-12]; 12-question/6-required interview bank with consent keys and a persist:false sink for the optional provider key [CX2-13]; ten {{TOKEN}} identity templates (generic, adapted to gbrain ops — gates call recall/query/put_page, write-through-ops rule, keyless agent-authored facts, silence contract); assets embedded compiled-binary-safe via file-type imports [ENG-6]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): runbook, README paste block, bootstrap guide, TODOS entries BOOTSTRAP_FOR_AGENTS.md (agent-driven install runbook: CLI phase list is the source of truth, never-invent rules, Codex approvals preflight, keyless posture, failure-modes table, version stamp for the skew check); README gains the full-agent paste block pinned to latest-stable inside the Claude Code/Codex quick start (memory-only tier stays); docs/guides/bootstrap.md carries the full install/security/consent/degradation/uninstall contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(designs): spike instrument for the bootstrap wave (build order 0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): interview + render engines Interview gate with read-back confirm-hash (any later answer change clears the confirmation — the hostile single-batch case is structurally impossible), per-answer provenance, caps + escaping at set time, config-sink routing for the provider key; renderer with hard-fail token sweep, subordinate fencing of principal input, never-clobber + backups, deterministic minimal mode for the template repo, scaled byte floors. 58 unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): private-repo lifecycle — repo create, attach, uninstall, run lock gh-gated private repo creation with API-verified privacy (rate-limit distinct from public), refuse-foreign-origin with attach as the sanctioned path, atomic bootstrap mutex (pid liveness + age + token), receipt-keyed uninstall that never wholesale-deletes the gbrain home and only offers --delete-brain for a brain it created; read-only PGLite lock probe (never opens the engine). 54 unit tests, injectable exec seam throughout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(release): latest-stable ref, template-repo publish job, bootstrap CI guards release.yml advances the latest-stable tag only after assets publish (the paste block's permanent ref — copies in the wild never rot) and gains a PAT-gated publish-template job verified against the vendored tree; two skip-graceful guards (sanctioned-ref + runbook stamp; template/token bijection + placeholder assertion + generator byte-diff) wired into verify; README + runbook re-admitted to the CI cache hash; vendored deterministic template tree generated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(context): IPC v2 turn_context + 8KB assembly + visibility resolver + session identity Discriminated-union IPC with handler map, protocol echo (stale-serve detection), shared-secret gate, server-side source binding, per-kind budgets; turn-context assembly (reflex pointers + volunteered pages + world-only hot facts) under a data-not-instructions envelope trimmed to the harness's 10KB hook-output cap; facts.default_visibility resolved through one helper at all four sites (explicit caller wins, typos fail closed); typed sessionId threads _meta.session_id into the hot-memory cache key. 50 new tests; 180 adjacent tests confirmed green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(persistence): secret-scan, gbrain sources push, durability unification Pattern secret scanner (own runtime allowlist, redacted previews, corpus-write redaction mode); sources push runs the whole scan→stage→commit→pull→push sequence under one cross-platform lock (mkdir-atomic, pid+age+token) with a deny-glob backstop, commit-first divergence-safe pull, refuse-unverifiable visibility, and push-status telemetry; gbrain-home choke point unifies GBRAIN_HOME semantics with config (0700); durability is parent-repo-aware and rotates its push log at 0600. 35 new tests; 200 existing green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sources): harden/pull gates accept sources inside a parent git repo The bootstrap workspace registers brain/ (a subdirectory) as the source; the durability core already resolves the repo root, so the command gates now check inside-a-repo rather than .git-right-here [CX2-3]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(serve): resident maintenance sweep + keyless capability probe The lock-owning serve process now closes the persistence loop: startup (3s post-connect, best-effort, unref'd) and idle (10-min quiet intervals through the injectable timer seam) sweeps run facts-fence reconciliation, deterministic link/timeline extraction over recent workspace pages, and spend-gated corpus ingest (skipped keyless — agent-authored fences cover it). gbrain sweep --once is the trusted CLI seam bootstrap verify uses. Capability probe renders the honest keyless/keyed report. Full reuse of the cycle extractor + extract cores; 26 new tests, neighbors green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(hooks): engine-free gbrain hook command, settings writers, transcript parser Four hook events (session-start digest + crashed-session recovery push, user-prompt turn-context injection under an 800ms deadline and the 10KB cap, stop buffers, session-end corpus write with redaction/retention/dedup + best-effort push); structural JSON settings merger keyed by a _gbrain marker (foreign hooks and permissions survive); dated host-spec registry; Claude Code .jsonl parser as a spec-target with a scrubbed 7-shape fixture. Heartbeat is counters-only by construction. 59 tests; zero engine modules in the import graph. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(bootstrap): cross-link the full-agent path from the connection docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): dispatcher, verify, status — the command assembled gbrain bootstrap {status,interview,render,repo,hooks,verify,uninstall,attach}: engine-free except verify (owns its engine, in-process sweep — no live-serve conflict); phase list is the TS source of truth with install.jsonl telemetry and the support blob; verify's fail-soft check suite covers the real write path (put_page → write-through file → sweep → graph floor → recall), passes keyless, persists snapshots, and ends with the first-run tour. cli.ts wired per the three-touchpoint rule; doctor gains the bootstrap check group (silent on machines with no bootstrap state). 28 new tests; 353 adjacent green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: KEY_FILES bootstrap cluster + CLAUDE.md dispatcher row (+ build:llms) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bootstrap): e2e pins — hook-under-live-serve, attach, degraded modes, compiled binary, Docker harness The permanent pins: a real serve holds the PGLite lock while the engine-free hook completes (and a direct engine open provably throws LiveServeLockError); stale-socket fail-open; machine-2 attach with marker-keyed hook repair; decline-everything installs verify green with every degradation named; the compiled binary renders bundled templates in an empty cwd. Offline Docker harness (networkless, read-only) gated into heavy-tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bootstrap): register doctor check categories + system-of-record allow comments The six bootstrap doctor checks join OPS_CHECK_NAMES; the sweep's batch link/ timeline inserts carry the explicit extract-path allow comments (the sweep IS the extraction path for workspace pages). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): shard wedge cap tracks suite growth (1500s -> 1800s) At ~9000 tests a healthy shard finished at 1466s and two progressing shards were false-killed at the old cap; 1800s restores ~25% headroom over the slowest observed healthy shard. Real hangs still hit it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): cache-hash policy — README + runbook edits must invalidate [C2] The old deny-list assertion predates the paste block; README.md and BOOTSTRAP_FOR_AGENTS.md are policy-doc re-admissions now, so their edits must change the hash (a paste-block edit shipping under a cached green was the C2 hole). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): classify post-suite exit-hangs as warn-pass; file the leak forensics A shard killed by the wedge watchdog with every assigned file started and zero fail markers did all its work and leaked a handle at exit — pre-existing and master-reproducible (P1 TODO carries the full bisect forensics). Bun's per-test timeout turns a hung test into a (fail), so the classifier cannot mask one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): shard cap 2400s — the count-balanced heavy shard needs it under contention Observed: the heavy shard still progressing 22s before an 1800s kill while siblings finish at 1150-1550s (split balances file count, not weight). Filed the load-sensitive WAL-repair flake (pre-existing, master's v0.42.75.0 wave). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bootstrap): quarantine env-mutating suites to the serial lane check-test-isolation R1: six new files mutate GBRAIN_HOME/env at module scope — the serial lane (one process per file) is the guard's prescribed home for them. All 114 tests pass post-rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): per-harness install sections — Codex, Claude Code, then OpenClaw/Hermes Each harness gets its own complete paste-block section (desktop app first, terminal noted — Claude Code CLI is the identical harness; Codex CLI works pull-based today); the OpenClaw/Hermes platform path keeps equal weight with its one-click deploys and INSTALL_FOR_AGENTS block intact; memory-only and remote-connect tiers consolidated under 'Lighter ways in'. Supersedes the review's D5 ordering by user direction; stale heading references updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): Codex as the recommended first step; OpenClaw/Hermes framed as-intended, high-cost The install section now routes newcomers explicitly: Codex first (subscription-priced, nothing to deploy), OpenClaw/Hermes as GBrain used the way it was designed — always on, at real server + API cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: doc-audit code chasers — broken recovery hints, stale op description, auto_link key Four small code fixes surfaced by the markdown accuracy audit: - doctor's auto-RLS recovery hint pointed at `apply-migrations --force-retry 35`, which cannot work (--force-retry targets the vX.Y.Z orchestrator registry, not the numeric schema MIGRATIONS array). Hint now points at the recreate SQL in docs/guides/rls-and-you.md; test pins against regression. - v0_11_0 migration printed the same broken-mechanism class of hint (`config set minion_mode` writes DB config nothing reads); now names `apply-migrations --mode` + preferences.json, the real setter. - submit_job's op description hardcoded a stale handler list; now points at registerBuiltinHandlers as the source plus the --follow discovery trick. - `auto_link` added to KNOWN_CONFIG_KEYS: read by link-extraction, reconcile-links, and sweep, and documented as the off-switch in brain-ops/maintain, but the allowlist rejected `config set auto_link false`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: repo-wide accuracy + MECE reform from the 9-bucket markdown audit A code-grounded audit of every markdown file (root, architecture, guides, mcp, tutorials, docs-root, operations/eval/designs, skills, recipes) followed by a fix wave with per-bucket ownership. Four classes of change: Accuracy — every documented command/flag verified against src/ before writing: dead commands replaced with working ones (pages purge-deleted, jobs watch --follow, gbrain restore, import-based Obsidian flow, space-separated --scopes, real thin-client recipes, working isolation verification, real supervisor restart procedure, curl-based ngrok health check, real minion_mode setter); count drift fixed with rot-proof phrasing (100+ ops, 50+ bundled skills via skills/manifest.json, 140+ engine methods, KNOBS_HASH_VERSION pointer instead of hardcoded versions); stale claims corrected (search-mode defaults, RETRIEVAL pipeline order incl. autocut, sentinel rules, refusal-list mechanism, engine snapshot, shard cap 2400s + EXIT-HANG classifier in TESTING.md, latest-stable + publish-template documented in RELEASING.md as release.yml promises). MECE — one home per concept, pointers elsewhere: test isolation → TESTING.md; OAuth registration + --bind/--public-url lore → DEPLOY.md; mode bundles → guides/search-modes.md (the home the CLAUDE.md dispatcher always promised); merge contract → schema-packs.md; WAL ladder → ENGINES.md; quiet-hours → quiet-hours.md; capture taxonomy → entity-detection.md; person-page taxonomy → compiled-truth.md; brain-first protocol → brain-first-lookup.md; refresh semantics → refresh-algorithm.md; KEY_FILES.md deduplicated (58 extension entries merged, one entry per file); infra-layer.md rewritten as a pointer page. Privacy — placeholder sweep across guides, docs, skills, and recipes per the iron rule; per-release narration stripped from reference docs (current-state prose only). Bootstrap coverage — AGENTS.md pointer, RESOLVER routing row, INSTALL.md path, tutorial cross-links, keyless-mode sections in spend-controls/headless-install. skills.lock.json regenerated; llms.txt/llms-full.txt rebuilt. Gates: verify 36/36, typecheck clean, doctor 96/96, skills-integrity + resolver + build-llms + config-set + migrations all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): refresh production-brain stats to current brain-repo counts 155,795 pages / 24,589 people / 5,340 companies, counted from the brain repo's current HEAD; the "100K-page brain" framing moves to 150K to match. Cron-fleet count unchanged (its store lives on the deployment host, not in the repos available for verification). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scan,push): modern OpenAI/Voyage key patterns, scan staged blobs not disk - secret-scan matches sk-proj-/sk-svcacct-/sk-None- and pa- Voyage keys (the bare sk- pattern missed every current OpenAI key format). - workspacePush stages first, then scans the staged index blobs via git cat-file, closing the scan-then-stage TOCTOU where a file changed between snapshot and commit shipped unscanned. - shared binary-sniff helper, memoized glob regexes, atomic push-status write, and tests for pull_conflict + gitignored deny-match paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): regenerate flag registry for new commands, harden shard classifier + release token - cli-flag-registry.generated.ts regenerated: bootstrap/hook/sweep and sources push --message/--allow-unverified-remote were missing, so the strict #2185 validator rejected real invocations and skipped the new commands entirely. - EXIT-HANG shard classifier now requires every assigned file to have started before warn-passing a watchdog kill (was fail-open). - release.yml passes TEMPLATE_REPO_PAT via http.extraheader, off the argv. - compiled-binary e2e fails loud in CI instead of a silent permanent skip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(hook,sweep,ipc): non-blocking hook pushes, bounded sweep + cache, source-bound resolve - session-start/session-end no longer run synchronous git + inline push inside their self-deadline; a detached child does the push and the hook returns immediately (blocked Claude Code startup for minutes on a dirty tree before). - serve sweep drops the unbounded listAllPageRefs, resolves only candidate targets, claims corpus files atomically (no double-LLM-spend race), and caps the fence LIKE scan; heartbeat writes are O_APPEND with rare compaction. - hot-memory cache evicts expired entries and bounds entry count (the key is caller-controlled via _meta.session_id). - v1 resolve IPC honors boundSourceId like turn_context; turn-context runs its arms concurrently. doctor reads push/heartbeat thresholds from hook.ts. - new tests: doctor bootstrap checks, hook push-gate + deadline, concurrent sweep claims, cache eviction, bound-source resolve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bootstrap): origin-ownership gate, world visibility, collision-safe source id, consent + templates - repo adoption requires an exact receipt repo_url match or authed-owner check (undefined repo_url was a wildcard); create verifies privacy BEFORE the first push. - verify sets facts.default_visibility=world if unset, so agent-authored facts surface in per-turn context (they defaulted private before). - source_id derives a path-hash suffix when 'workspace' is taken by another checkout; every consumer reads manifest.source_id. - skipped HOOKS_CONSENT now declines (was falling through to default yes); --minimal refuses on an initialized manifest; tilde fences escaped. - MCP registration pins --surface full; status hard-fails a public origin (template door); receipt writers guard against newer/corrupt receipts; uninstall only claims brain-deleted after a real rm. - templates ship jobs disabled + provider-consent + support-relay lines; soul-audit re-runs over the shared interview bank. TODOS: 11 follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): close adversarial-review findings — scan fails closed, whole-PEM redaction, bound repo push Cross-model adversarial pass (Claude + Codex) on the bootstrap wave: - secret scan fails CLOSED: an unreadable, oversized, or binary staged blob now blocks the push (blocked_unscannable, exit 5) instead of committing unscanned; only a confirmed staged deletion is skipped. This was the headline "block secrets before they leave the machine" property failing open. - private-key redaction spans the whole PEM block (header+body+footer), not just the header line — the base64 body no longer survives into the corpus the sweep sends to an extraction provider. - bootstrap repo commits the workspace (secret-scan-gated) before the first push and verifies the remote actually received it, so a push-fail retry can't adopt an empty remote as success. - privacy verify is re-bound to origin immediately before push (a concurrent origin rewrite between verify and push is refused). - session-end corpus write is atomic and clears the stale ingested/in-progress sidecars so a resumed session's appended transcript is re-ingested. - public-origin refusal enforced at render (not only status); MCP "already registered" is verified to target this workspace, not blessed blindly; verify probe cleanup scopes deletes to its own slugs, not a token substring; allowlist fingerprint floor raised 8→16 hex. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v0.45.0.0 feat(bootstrap): paste-in personal-agent install for Codex + Claude Code Turns a Codex or Claude Code session into a persistent personal agent: interview-rendered identity files, a local PGLite brain, per-turn context via serve IPC (Claude Code hooks / Codex pull protocol), session-triggered persistence, and a private GitHub repo as the agent's portable body. Keyless- first (the harness model is the LLM; one optional key adds embeddings + extraction). New `gbrain bootstrap` command family + `gbrain hook` + `gbrain sweep`; doctor bootstrap health checks; latest-stable distribution ref + template-repo publish job. Opt-in, additive — existing installs untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: regenerate flag registry for security-fix flags; drop fabricated gbrain capabilities doc ref CI caught two real failures under the merged state: - the flag registry lagged the blocked_unscannable/exit-5 flags the security round added, tripping the #2185 freshness guard. - headless-install.md described the keyless capability report as a `gbrain capabilities` command, which the #3502 doc-command resolver rejects — reworded to prose (the real surface is bootstrap verify's report). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: sync KEY_FILES + bootstrap plan to security-fix behavior Cross-referenced the security-fix round against the reference docs and corrected the drift those commits introduced: - workspace-push.ts entry: stage-FIRST-then-scan order (the TOCTOU fix), fail-closed blocked_unscannable, and the sources-push status -> exit-code map. - hooks.ts entry: MCP registration pins `serve --surface full`. - hook.ts entry: session-start/session-end pushes run in a detached child (non-blocking); atomic corpus write clears stale sidecars. - bootstrap.ts entry: render hard-refuses a public origin (template door). - verify.ts entry: source_id collision resolution (workspace-<path-hash>). - AGENT_BOOTSTRAP_PLAN as-shipped delta note for the scan/stage reorder. llms bundle unchanged (KEY_FILES is link-only); build:llms and test/build-llms.test.ts green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): silence SC2016 on the intentional askpass literal in release.yml The one-shot GIT_ASKPASS script must contain literal $1 and $TEMPLATE_REPO_PAT so they expand when /bin/sh runs it at git's credential prompt, not when the outer shell writes the file — single quotes are correct. Add a scoped shellcheck disable so actionlint passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(skills): declare 'bootstrap my data' trigger in cold-start frontmatter The doc reform added 'bootstrap my data' to cold-start's RESOLVER.md row (to disambiguate data-bootstrap from agent-bootstrap) but not to the skill's own frontmatter triggers, tripping the RESOLVER↔frontmatter round-trip contract (resolver.test.ts). Declare it; regenerate skills.lock.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(bootstrap): default per-turn hooks + search mode ON without a prompt Installing gbrain for your coding agent IS the consent for the behaviors that make it work, so stop re-litigating them with install-time questions whose "no" defeats the product: - Per-turn hooks (Claude Code) install ON by default — no prompt. Off-ramps: `--no-hooks` at install, `GBRAIN_HOOKS=0` at runtime, `bootstrap uninstall`. The "hooks installed" line now surfaces the kill switch so default-on is never silent. A persisted HOOKS_CONSENT=no (interview --skip) still declines. - Search mode defaults to `balanced` silently (nobody knows the modes at install; `gbrain search modes` changes it any time). - MCP scope stays the ONE deliberate prompt — project vs user is a real cross-repo privacy choice, not friction. Marks the two consents `silent: true` in the question bank (new QuestionSpec field), rewrites the runbook phases so the agent no longer asks them, adds the `--no-hooks` flag (+ registry regen), and adds default-on / opt-out tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bootstrap): real end-to-end coverage — cross-session recall, per-turn content, Codex door, realistic corpus Closes the seven e2e gaps a coverage audit surfaced: the plumbing was well-unit-tested but the product claims ("Codex works, context shows up every turn with real content, it remembers across restarts, machine two recovers, Postgres works") were unproven end to end. Test-only wave — zero src changes. - Hermetic synthetic corpus (test/fixtures/bootstrap-corpus/ + a loader helper): 12 interlinked pages (52 edges, timelines), 12 world/private beliefs, 8 gold queries — curated from the gbrain-evals synthetic corpora, 100% placeholder names, so recall is asserted on a real multi-entity brain instead of a 2-node self-planted probe. - GAP1 magic moment: author a fact via the real write path, disconnect the engine, reopen against the same DB, recall it — a real session boundary, not verify.ts's same-connection SQL read-back. Plus a source-isolation assertion. - GAP2 per-turn content: hook-under-serve Pin 1 now seeds a known fact and asserts its text lands in the injected block AND private beliefs never do (was: empty brain, empty_block accepted as a pass). - GAP3 Codex door: assert the rendered AGENTS.md carries the Gate-3 brain-first pull protocol; make the fake codex shim implement `mcp get` so the [FIX7] target-verification can actually fail; the Docker cold-machine harness now exercises the hooks/MCP registration step instead of skipping it. - GAP4 corpus recall: turn-context + verify graph-floor/qrels run on the real multi-entity brain with real edges. - GAP5 attach: machine-two now re-ingests the cloned brain/ into a fresh DB and recalls a fact authored only on machine one — the multi-device payoff. - GAP6 keyed + Postgres (env-gated): real embeddings prove semantic recall a paraphrase query can reach but keyless BM25 cannot; bootstrap verify drives a real Postgres engine (skipIf DATABASE_URL/keys absent). - GAP7 persistence: session-end runs the REAL push (not the mocked seam) to a local bare remote and the remote receives the content; a planted secret is blocked at the gate; the 15-min cron installs and fires a scan-gated push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bootstrap): real-agent e2e — drive the actual claude + codex binaries end to end Closes the audit's biggest gap ("no real harness ever drives a turn"). Adapts gstack's PTY/headless agent harness to prove the bootstrap install + smoke work against the REAL binaries, not PATH shims. Test/CI/docs only — zero src changes. - test/helpers/agent-harness.ts: hermetic clean-room child env (ported from gstack; drops CONDUCTOR_/CLAUDE_/GSTACK_/MCP_/GBRAIN_, promotes GSTACK_ANTHROPIC_API_KEY→ANTHROPIC_API_KEY), real-binary resolvers + auth probes, headless `claude -p --output-format stream-json` and `codex exec --json` turn runners, a gbrain stdio MCP-config writer, and a keyless brain seeder. + a fixture-parse unit test (no binary needed). - test/e2e/bootstrap-real-claude.serial.test.ts: real `gbrain bootstrap` install → REAL `claude mcp add` (verified via `claude mcp get`) → verify exit 0 → a real `claude -p --mcp-config --strict-mcp-config` turn that invokes mcp__gbrain__search and answers from the brain (proven: toolCalls include mcp__gbrain__search, final text carries the seeded fact). - test/e2e/bootstrap-real-codex.serial.test.ts: same install with REAL `codex mcp add` into a real ~/.codex/config.toml + Gate-3 pull-protocol assertion, then a real `codex exec --json` turn surfacing the fact (MCP or the pull- protocol shell path). Bounded retry absorbs codex's occasional MCP-call cancellation without softening the fact-requiring assertion. - Everything hermetic (temp HOME/CLAUDE_CONFIG_DIR/CODEX_HOME/GBRAIN_HOME; real ~/.codex auth copied read-only) and skipIf-gated so it self-skips cleanly where the binaries/auth are absent. - heavy-tests.yml: gated `real-agent-e2e` job (nightly/label, never the PR shard; no-op on a runner without authed binaries). - TODOS: compiled `gbrain` binary can't serve a PGLite brain (bun compile omits the WASM/extension payloads); harness falls back to `bun run` serve. Verified against live claude 4.6 + codex 0.147.0: 15 pass / 0 fail; verify 36/36; typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): real-agent-e2e job — bash array + --timeout (actionlint SC2086 + bun-test-timeout guard) The real-agent-e2e job's file loop used an unquoted $FILES (SC2086) and ran `bun test` without --timeout (check-bun-test-timeout guard). Switch to a bash array and add --timeout=600000 (real-agent turns are slow; the tests self-skip without authed binaries so it's a no-op elsewhere). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite): embed WASM + extension assets so the compiled binary can serve A `bun build --compile` gbrain binary could not `serve` a PGLite brain: the compile bundles JS but not PGLite's runtime payload (pglite.wasm, initdb.wasm, pglite.data, vector/pg_trgm tarballs), so `serve` on PGLite died with a bunfs/ENOENT. Now the assets ride inside the binary. - src/core/pglite-embedded-assets.ts: embeds the five assets via `import … with { type: 'file' }` (the ENG-6 idiom) and exposes getEmbeddedPgliteOptions() → { pgliteWasmModule, initdbWasmModule, fsBundle, extensions:{vector,pg_trgm} }. WASM/fsBundle are consumed as bytes; the two extension tarballs are materialized to a content-addressed temp file (atomic, size-verified reuse) because PGLite reads them via fs.createReadStream, which cannot read a /$bunfs path. Unconditional (works in bun-run and compiled), so no fragile mode branch. - src/core/pglite-engine.ts: static-import getEmbeddedPgliteOptions (engine path stays static per the engine-dynamic-import invariant); spread into both PGlite.create sites (initial + WAL-repair retry). The bunfs classifier stays as a backstop but no longer fires for a correct binary. - scripts/check-pglite-embedded.sh (+ smoketest): compiles a focused binary and asserts it boots PGLite, CREATE EXTENSION vector/pg_trgm, and round-trips a page — wired into `bun run verify` (now 37 checks), check:all, and check:pglite-embedded. Fail-soft only when compile is unavailable. - agent-harness.ts probeCompiledPglite now passes → the real-agent e2e uses the fast compiled MCP server. TODOS: the P2 "can't serve PGLite" item is closed. Verified: fresh compiled binary ran `search`/`query` against a PGLite brain and returned the seeded row (no bunfs/ENOENT); verify 37/37; pglite-engine 120/0 source-mode; typecheck clean; engine-dynamic-import + parity guards pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Garry TanandClaude Opus 4.7
|
0e9f8814a5 |
feat: v0.16.0 — durable agent runtime (gbrain agent + subagent handler + plugin loader) (#258)
* refactor(mcp): extract buildToolDefs helper for subagent tool registry reuse
The inline operations.map(...) block in src/mcp/server.ts became the only
source of truth for agent-facing tool definitions. Extract into a reusable
exported helper so the v0.15 subagent tool registry can call it with a
filtered OPERATIONS subset instead of duplicating the shape.
Byte-for-byte equivalence regression pinned in test/mcp-tool-defs.test.ts —
legacy inline mapping kept verbatim inside the test so any future drift
between the new helper and the pre-extraction MCP schema fails loudly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(operations): subagent-aware OperationContext + put_page namespace
Adds three optional fields to OperationContext:
- jobId?: number — the currently running Minion job id
- subagentId?: number — the owning subagent job id for tool-dispatched calls
- viaSubagent?: boolean — FAIL-CLOSED flag for agent-path gating
put_page now enforces a namespace rule when invoked on the subagent tool
dispatch path (viaSubagent=true): writes MUST target
`wiki/agents/<subagentId>/...`. Anchored, slash-boundary enforced so a
collision like `wiki/agents/12evil/...` can't impersonate subagent 12.
The check runs BEFORE the dry-run short-circuit so preview calls surface
the same rejection. Fail-closed: a missing subagentId with viaSubagent=true
rejects every slug rather than letting a dispatcher bug open a hole.
Existing callers unaffected — all three fields are optional and the legacy
put_page behavior is unchanged when viaSubagent is undefined/false.
12 regression + namespace tests pin:
- local CLI writes (viaSubagent unset) accept arbitrary slugs
- MCP writes (remote=true, viaSubagent unset) accept arbitrary slugs
- subagent-path: anchored prefix accepted, wrong id rejected, prefix-
collision defeated, leading-slash rejected, bare-prefix rejected,
fail-closed on missing/NaN subagentId, permission_denied code emitted
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(schema): v0.15.0 subagent runtime tables + migration orchestrator
Adds three new tables for the durable LLM agent runtime:
subagent_messages — Anthropic message-block persistence.
Parallel tool_use blocks in one assistant
message live in content_blocks JSONB, not
across rows (fixes the (job_id, turn_idx, role)
misdesign codex caught in v0.13 drafting).
subagent_tool_executions — Two-phase tool ledger. INSERT pending before
execute, UPDATE complete/failed after. Replay
re-runs pending rows only if the tool is
idempotent (v1 ships only idempotent tools so
this is preventive).
subagent_rate_leases — Lease-based concurrency cap for outbound
providers (e.g. anthropic:messages). Stale
leases auto-prune on next acquire so crashed
workers can't strand capacity.
All DDL uses CREATE TABLE/INDEX IF NOT EXISTS — order-independent vs
PR #244's initSchema() reorder, and idempotent across fresh-install +
upgrade paths. Shipped in both src/schema.sql (Postgres) and
src/core/pglite-schema.ts (PGLite); schema-embedded.ts regenerated.
Migration orchestrator v0_15_0.ts (phases: schema → verify → record).
v0_14_0.ts is a no-op stub so the registry's version sequence stays
gapless (v0.14.0 shipped shell-jobs — code change, no DB migration).
10 unit tests for registry wiring, ordering, dry-run phase behavior, and
schema-embedded table presence. test/apply-migrations.test.ts updated for
the two new registry entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): emit child_done on every terminal + max_stalled per-job + terminal set fix
Three correctness fixes the v0.15 subagent aggregator spine depends on:
1. child_done emission on ALL terminal transitions, not just success.
- completeJob already emitted on success — now also tags outcome='complete'.
- failJob newly emits on terminal 'failed' or 'dead' (outcome='failed'|'dead',
error=<text>), BEFORE the parent-terminal UPDATE so the EXISTS guard on
the inbox INSERT doesn't skip it on fail_parent paths (codex catch).
- cancelJob now emits outcome='cancelled' per descendant with a parent.
- handleTimeouts now emits outcome='timeout' per timed-out child.
ChildDoneMessage gains optional { outcome, error } — backwards compatible
(legacy writers omitted them; consumers treat absent outcome as 'complete').
2. Parent-resolution terminal set now includes 'failed'.
Pre-v0.15 the `NOT EXISTS (... status NOT IN ('completed','dead','cancelled'))`
guard treated a failed child as still-pending, stranding aggregator parents
that chose on_child_fail='continue' or 'ignore' in waiting-children forever.
Expanded to {completed, failed, dead, cancelled} everywhere parent resolution
reads child status (completeJob inline, failJob remove_dep + continue,
cancelJob sweep, handleTimeouts sweep, and the resolveParent method itself).
3. MinionJobInput.max_stalled threads through MinionQueue.add() on INSERT.
Column exists with default 1 — that is "first stall → dead", which defeats
crash recovery for long-running handlers. Subagent children will set
max_stalled: 3 to survive mid-run worker kills. Second-submitter under an
idempotency-key hit does NOT mutate the existing row (codex-flagged
footgun — first-submit options are load-bearing state).
13 unit tests pin: emission on each of completeJob/failJob/cancelJob/
handleTimeouts, insertion order on fail_parent, terminal-set expansion with
continue policy, max_stalled default + override + idempotency behavior.
E2E tier 1 (Postgres) passes 141 tests unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): rate-leases + waitForCompletion infra for v0.15 subagent
Two infrastructure modules the subagent handler spine depends on:
rate-leases.ts — lease-based concurrency cap for outbound providers
(anthropic:messages, openai:*, etc.). Counter-based limiters leak capacity
on worker crash; leases are owner-tagged rows with expires_at that
auto-prune on the next acquire. Two-phase: txn-scoped pg_advisory_xact_lock
guards the check-then-insert so concurrent acquires can't both win the
"last slot". renewLeaseWithBackoff retries 3x (250/500/1000ms) for mid-
call DB blips — on persistent failure the LLM-loop caller aborts with a
renewable error so the worker re-claims and the rate invariant is
preserved. Owner FK cascades clean up leases on job deletion.
wait-for-completion.ts — poll-until-terminal helper for CLI callers.
Minions' NOTIFY is worker-side only; `gbrain agent run --follow` polls
getJob() until status is {completed, failed, dead, cancelled}. TimeoutError
carries jobId + elapsedMs and does NOT cancel the job — the user can
inspect via `gbrain jobs get <id>` later. Supports AbortSignal for Ctrl-C
without throwing. Default pollMs is 1000 on Postgres, 250 on PGLite (inline
CLI has no network RTT).
21 unit tests cover: single/multi acquire under cap, rejection past cap,
release frees slot, different keys are independent, stale prune, cascade
on owner delete, renew bumps expires_at, renew on missing is false,
backoff path success + pruned short-circuit. waitForCompletion: fast-path
terminal, transitions mid-wait (completed/failed/cancelled), TimeoutError
shape, abort-signal early exit, non-existent job error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent ToolDef types + brain-tool registry (v0.15)
Types first so the handler has a stable contract:
- SubagentHandlerData / AggregatorHandlerData — the two job.data shapes
- ToolCtx (engine, jobId, remote, signal) + ToolDef (name, description,
input_schema, idempotent, execute) — Anthropic-envelope, distinct from
the MCP McpToolDef extraction landed earlier
- ContentBlock discriminated union for subagent_messages.content_blocks
- SubagentStopReason + SubagentResult emitted on terminal completion
brain-allowlist.ts derives one ToolDef per allow-listed OPERATION. Reuses
the ParamDef → JSONSchema shape from the MCP extraction in a local helper
(Anthropic's input_schema field diverges from MCP's inputSchema by a
character). The 11-name allow-list is read-safe + put_page — every
destructive / filesystem / identity-mutating op stays off by default.
put_page gets a namespace-wrapped tool schema: `slug` pattern = anchored
`^wiki/agents/<subagentId>/.+`. The server-side check in put_page op
(shipped in prior commit) is still the authoritative gate — the schema
just helps the model write correct slugs first-try. `subagentId` is
plumbed into the ToolCtx so the viaSubagent=true fail-closed path lights
up on every tool-dispatched put_page.
filterAllowedTools narrows a registry by subagent_def's allowed_tools
frontmatter field. Rejects unknown names at load time (no silent drop —
typos in a skills/subagents/*.md would otherwise ship to prod with a
tool silently missing).
18 tests pin: every allowlist name exists in OPERATIONS (catches upstream
rename), Anthropic name regex, put_page namespace pattern per-subagent,
execute() routes through the op handler with viaSubagent=true, out-of-
namespace put_page throws permission_denied, filter passes prefixed +
unprefixed names, rejects unknowns, deduplicates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent-audit JSONL + transcript renderer
Two small plumbing pieces the v0.15 subagent handler + `gbrain agent logs`
depend on:
subagent-audit.ts — JSONL-rotated audit log mirroring the shell-audit
pattern. Two event flavors: submission (one line per job submit) and
heartbeat (one line per turn boundary — llm_call_started / completed /
tool_called / tool_result / tool_failed). Heartbeats fix the "--follow on
a long Anthropic call shows nothing for 30 seconds" problem codex flagged.
Never logs prompts or tool inputs (PII risk — subagent input_vars may
carry user-supplied free text); DOES log tokens, ms_elapsed, tool_name,
first 200 chars of error text. Rotates weekly via ISO week. `readSubagent
AuditForJob` is the readback path for `gbrain agent logs` — scans the
current + prior week file so job boundaries across weeks still resolve.
`GBRAIN_AUDIT_DIR` overrides the default ~/.gbrain/audit/ for container
deploys.
transcript.ts — renders subagent_messages + subagent_tool_executions to
markdown. Message order is authoritative; tool rows splice under their
owning assistant tool_use by tool_use_id. Handles text, tool_use (with
pending / complete / failed execution rows), tool_result (skipped if
we already rendered the owning tool_use — avoids double-printing), and
unknown block types (fenced JSON dump for diagnostics). Output is
UTF-8-safe truncated at maxOutputBytes.
21 unit tests: ISO week filename rotation (incl. 2027-01-01 → W53-2026
boundary), submission + heartbeat write shapes, 200-char error cap, best-
effort write failure doesn't throw, readback filters by job_id and
sinceIso. Transcript: empty input, ordering, token line, tool_use +
complete/failed/pending execution rendering, truncation, unknown-block
diagnostic dump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent LLM-loop handler with crash-resumable replay
The main event: runs one Anthropic Messages API conversation with tool
use, persists every turn + tool execution, and resumes cleanly after a
worker kill anywhere in the loop.
Design points that carry the v0.15 guarantees:
1. Two-phase tool persistence. INSERT status='pending' before dispatch,
UPDATE to 'complete' or 'failed' after. subagent_messages rows are
the canonical conversation; subagent_tool_executions rows are the
canonical "did this tool run + what did it return". Either DB commit
is atomic, so replay has a single source of truth.
2. Replay reconciliation. If the last persisted message is an assistant
with tool_use blocks AND no following synthesized user message, we
crashed mid-dispatch. On resume, finish those tools first (respecting
idempotent flag for 'pending' rows), synthesize the user turn, and
THEN call the LLM again. Non-idempotent pending rows abort the job
with a clear error — v0.15 ships only idempotent tools so this is
preventive.
3. Rate lease around every LLM call. acquireLease before, releaseLease
after (both success and error paths). acquired=false throws
RateLeaseUnavailableError — the worker treats it as a renewable
error and re-claims later, so a temporary capacity cap doesn't fail
the job terminally.
4. Anthropic prompt caching. system block gets cache_control=ephemeral;
the LAST tool def gets it too (Anthropic caches everything up to and
including the marked block). ~10x cost reduction on multi-turn
agents per the plan.
5. Dual-signal abort. AbortSignal.any merges ctx.signal (timeout / lock
loss / cancel) with ctx.shutdownSignal (worker SIGTERM). Both feed
the Anthropic call's AbortSignal; mid-turn abort bails before the
next LLM call with whatever turns are already persisted. Node ≥ 20
has AbortSignal.any; older runtimes get a manual-merge polyfill.
6. Injectable Anthropic client. The real SDK implements MessagesClient
structurally; tests inject a FakeMessagesClient that scripts
responses.
12 unit tests pin: no-tool happy path, single tool_use complete, tool
throws → failed row + loop continues, unknown tool name rejection,
max_turns cap, crash-then-resume with partial state, replay skips already-
complete tool execs without re-invoking execute, non-idempotent pending
rejects on resume, lease acquire + release roundtrip, RateLeaseUnavailable
under cap-full, missing prompt validation, allowed_tools unknown-name.
NOT in v0.15: refusal detection (stop_reason + content shape), stop_reason
=max_tokens partial recovery, mid-call lease renewal with backoff loop.
All three are documented as P2 items in the plan file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent_aggregator handler with mixed-outcome rendering
Claims AFTER all subagent children resolve — by then Lane 1B's queue
changes have posted one child_done message per terminal transition into
this job's inbox (complete / failed / dead / cancelled / timeout). The
aggregator reads those, builds a deterministic markdown summary, and
returns it as the handler result.
Not an LLM call in v0.15 — output is reproducible concatenation so
fan-out runs stay comparable. v0.16+ can add an LLM synthesis pass
behind an opt-in flag.
Contract:
- empty children_ids → `(no children)` marker
- missing child_done (shouldn't happen under v0.15 invariants but
possible if a terminal-state path slipped past Lane 1B) → counted as
failed with "no child_done message observed" error
- non-complete outcomes: result is null in the output so no payload
leaks alongside a failure label
- children appear in the order children_ids was supplied
- custom aggregate_prompt_template replaces the markdown header
13 unit tests cover: empty input, all-success, mixed outcomes, result
suppression on failure, missing child_done handling, order preservation,
custom template, progress + log emission, stringified JSONB payload
parsing, non-child_done inbox filtering, legacy-writer outcome fallback,
and internal helper edges.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): GBRAIN_PLUGIN_PATH loader + plugin-authors guide (v0.15)
Plumbing that makes Wintermute (and future downstream agents) day-1
usable on v0.15. Host repos drop a `gbrain.plugin.json` + `subagents/`
directory somewhere, set GBRAIN_PLUGIN_PATH (colon-separated like \$PATH),
and their custom subagent defs load at worker startup.
Path policy is strict: absolute paths only. Relative, ~-prefixed, and
URL-style (https://, file://) all rejected with warnings — the user
controls where plugins live. Non-existent paths and files (not dirs) are
warned and skipped so a typo doesn't crash worker startup.
Collision policy: left-wins. If two plugins ship a subagent with the same
name, the first one in GBRAIN_PLUGIN_PATH keeps it and the other gets a
warning naming both sources. Deterministic + debuggable.
Trust policy: plugins ship subagent defs ONLY. Cannot declare new tools,
cannot extend the brain allow-list, cannot override safety flags. The
subagent def's `allowed_tools:` frontmatter MUST subset the derived
registry — validation happens at load time (worker startup), not at
dispatch time, so a typo in a skill gives a loud startup error instead
of silently "tool never fires at 3am."
Manifest `plugin_version: "gbrain-plugin-v1"` locks the contract. Unknown
versions rejected. `subagents` field escape attempts (`../../../etc` etc)
rejected. gray-matter handles the markdown frontmatter parse — subagent
defs don't conform to the page schema, so we don't use parseMarkdown.
docs/guides/plugin-authors.md is the Wintermute-facing walkthrough.
Covers the minimum viable plugin shape, the three policies, the
frontmatter fields, known caveats (audit JSONL is local-only, tool calls
always run remote=true, put_page is namespace-scoped).
22 unit tests pin path rejection, missing/invalid manifest, unsupported
version, escape-attempt, basename fallback for missing frontmatter.name,
allowed_tools round-trip, unknown-tool rejection with validAgentToolNames,
empty env, multi-path, collision warning with left-wins, trimmed paths,
manifest-rejection as warning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): gbrain agent run + logs + worker registration (v0.15 Lane 4H)
Three integration seams wired:
src/commands/agent.ts — \`gbrain agent run\`. Submits subagent jobs (or a
fan-out of N + aggregator) under the trusted-submit flag so the
PROTECTED_JOB_NAMES guard doesn't reject. Fan-out path creates the
aggregator first (so children can reference its id as parent), submits
each child with on_child_fail='continue' (required by Lane 1B's terminal-
set + child_done machinery), then jsonb_set's the aggregator's
children_ids. Short-circuits a 1-entry manifest to a single subagent
with no aggregator. Follow mode runs agent-logs streaming + waitFor
Completion in parallel and exits on terminal status; detach prints the
job id and exits. Ctrl-C is handled as detach, not cancel — the job
keeps running, consistent with durability invariants.
src/commands/agent-logs.ts — \`gbrain agent logs\`. Merges ~/.gbrain/audit/
subagent-jobs-*.jsonl (heartbeats + submissions) with subagent_messages
(persisted conversation) in one chronological stream. --follow polls at
1s and exits when the job hits terminal. --since accepts ISO-8601 OR
relative shorthand (5m / 1h / 2d). Writes transcript tail (full message
+ tool tree) only for terminal jobs, so mid-run --follow doesn't spam a
half-rendered transcript.
src/commands/jobs.ts registerBuiltinHandlers — matches the shell-handler
opt-in shape. GBRAIN_ALLOW_LLM_JOBS=1 registers the subagent +
subagent_aggregator handlers, then loads plugins from GBRAIN_PLUGIN_PATH
with validAgentToolNames pulled from BRAIN_TOOL_ALLOWLIST. Every plugin
warning + loaded-plugin line prints to stderr, mirroring the openclaw-
seam startup convention.
src/core/minions/protected-names.ts — subagent + subagent_aggregator
join the protected set. MCP submit_job returns permission_denied; only
trusted-CLI callers (with allowProtectedSubmit) can insert these rows.
src/cli.ts — adds 'agent' to CLI_ONLY + dispatches it like 'jobs'.
Test fallout: subagent-handler.test.ts + subagent-transcript.test.ts
helpers now submit under allowProtectedSubmit (they insert rows named
'subagent' directly against the queue). 23 new tests in agent-cli.test.ts
cover: flag parsing (including --detach implies !follow, --tools comma
split, -- terminator, unknown flag throw), --since parse (ISO, relative
5m/2h/1d, unparseable error), protected-name guard for all three names,
trusted-submit gate, and a fan-out integration check that verifies the
aggregator + children shape after --fanout-manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): rename max_children test's spawned jobs off the protected 'subagent' name
The spawn-storm test submitted 50 literal-string 'subagent' children to
exercise the max_children row-lock serialization. In v0.15 'subagent' is
a PROTECTED_JOB_NAME (CLI-only; trusted submit required), so the old
literal submission now throws before reaching the row-lock check.
The test is about max_children semantics, not the v0.15 subagent runtime
specifically — rename the child name to 'child_worker' so the test
exercises the exact same queue.add path without tripping the new guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ship): v0.15.0 — VERSION, CHANGELOG, README, upgrading-agents, CLAUDE.md
Bumps VERSION → 0.15.0 and package.json → 0.15.0 (resolves the pre-existing
drift — on master, VERSION=0.14.0 but package.json=0.13.1; src/version.ts
reads package.json, so this is what the binary prints now).
CHANGELOG lands the release-summary entry in the GStack voice + the full
itemized change list (11 new modules, 3 new tables, queue correctness
fixes, trust-model additions, 159 new unit tests). Voice rules respected
— no em dashes, no AI vocabulary, real file names + real numbers.
README gets a "Durable agents: `gbrain agent` (v0.15)" section next to
the Minions block, with the three canonical CLI shapes (single run,
fanout-manifest, logs --follow) and a pointer to plugin-authors.md.
docs/UPGRADING_DOWNSTREAM_AGENTS.md gets a full v0.15.0 section covering
the four adoption steps downstream agents (Wintermute and similar) need:
(1) worker opt-in via GBRAIN_ALLOW_LLM_JOBS, (2) moving custom subagent
defs to a plugin repo, (3) replacing ephemeral subagent runs with durable
`gbrain agent run`, (4) the put_page namespace rule for agent-driven writes.
CLAUDE.md updated with concise per-file descriptions for every new module:
the handler, aggregator, audit, rate-leases, wait-for-completion,
transcript, plugin-loader, brain-allowlist, tool-defs extraction, agent
CLI + logs CLI, and the registerBuiltinHandlers wiring for subagent
handlers + plugin-loader.
Verified: binary builds (940 modules, 89ms compile), prints `gbrain 0.15.0`,
`gbrain agent --help` shows the new subcommand shape. 170 new tests pass
(full v0.15 surface). Full unit suite passes bar one parallel-load
flake on a pre-existing E2E (graph-quality, passes in isolation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): drop GBRAIN_ALLOW_LLM_JOBS flag — subagent handlers always-on
The env flag was ceremony. Shell jobs need the flag because they execute
arbitrary CLI commands (RCE surface). Subagent jobs don't — they call the
Anthropic API with whatever ANTHROPIC_API_KEY is in env, so the key is
already the cost gate (no key → SDK fails on the first turn). And
who-can-submit is already protected by PROTECTED_JOB_NAMES +
TrustedSubmitOpts: MCP callers get permission_denied; only `gbrain agent
run` with allowProtectedSubmit can insert subagent / subagent_aggregator
rows. The flag added nothing the existing guards didn't already give us.
registerBuiltinHandlers now always registers subagent + subagent_aggregator
and loads GBRAIN_PLUGIN_PATH plugins. Worker startup prints:
[minion worker] subagent handlers enabled
instead of the conditional enabled/disabled pair. Plugin discovery runs
unconditionally — empty PATH is a no-op.
README, CHANGELOG, docs/UPGRADING_DOWNSTREAM_AGENTS, CLAUDE.md, agent CLI
help text, and subagent handler docstring all updated to drop the flag
reference. Shell handler's GBRAIN_ALLOW_SHELL_JOBS gate is untouched —
separate concern (RCE, not billing).
Full suite: 1859 pass, 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: scrub private agent-fork name from all public artifacts
Enforces the rule added to CLAUDE.md (privacy section): never say
`Wintermute` in any CHANGELOG, README, doc, PR, or commit message.
Reader-facing copy says `your OpenClaw` (the term covers every
downstream OpenClaw deployment — Wintermute, Hermes, AlphaClaw — in
one umbrella the reader already recognizes). First-person /
origin-story copy says `Garry's OpenClaw` (honest that this is the
production deployment driving the feature, without exposing the
private agent's name).
Swept across:
CHANGELOG.md (v0.15 entry + 4 historical mentions)
README.md
TODOS.md
docs/UPGRADING_DOWNSTREAM_AGENTS.md
docs/guides/plugin-authors.md (including example plugin names)
docs/guides/plugin-handlers.md
docs/guides/minions-fix.md
docs/designs/KNOWLEDGE_RUNTIME.md (27 refs, mostly analytical)
docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md
skills/migrations/v0.11.0.md
skills/skillpack-check/SKILL.md
scripts/skillify-check.ts
src/commands/doctor.ts
src/commands/migrations/v0_15_0.ts
src/commands/skillpack-check.ts
src/core/enrichment/completeness.ts
src/core/minions/plugin-loader.ts
src/core/operations.ts
src/core/output/scaffold.ts
Intentionally kept (these mentions define/test the rule itself):
CLAUDE.md — the privacy rule section necessarily uses the literal
name to define the restriction and examples
test/plugin-loader.test.ts — fixture name in a plugin-loading test;
renaming risks breaking assertion logic
test/integrations.test.ts — the word appears in a privacy-regex
test that explicitly enforces name redaction
test/doctor-minions-check.test.ts — a comment referencing the rule
CEO plan artifact at ~/.gstack/projects/… — private, not distributed
Binary builds (941 modules), 198/198 relevant tests pass, `gbrain --version`
prints `0.15.0`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: gitignore bun --compile artifacts with a glob, not specific hashes
Each `bun build --compile` emits a fresh hash-named `.*-*.bun-build` file
in cwd. The prior entries listed two specific hashes that were already
stale, so every build after those created a new untracked file requiring
manual cleanup.
Replace the two stale entries with `*.bun-build` so any current or future
compile artifact is ignored automatically.
Verified: ran `bun build --compile`, got two new `.*-*.bun-build` files,
`git status` stays clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ship): rename v0.15.0 → v0.16.0
gbrain master is at 0.14.2. Other 0.15.x PRs may land before/after
this one — we bump the minor (new capability) and lock to 0.16.0 so
ordering with concurrent work doesn't matter.
Touches:
- VERSION: 0.15.0 → 0.16.0
- package.json: 0.15.0 → 0.16.0
- Rename src/commands/migrations/v0_15_0.ts → v0_16_0.ts (+ all
version strings inside + import in index.ts registry)
- Rename test/migrations-v0_15_0.test.ts → migrations-v0_16_0.test.ts
- test/apply-migrations.test.ts: skippedFuture lists now reference
'0.16.0'
- test/put-page-namespace.test.ts + test/mcp-tool-defs.test.ts: Lane
comment refs updated
- src/schema.sql + src/core/pglite-schema.ts: "v0.15.0" section
comment updated; src/core/schema-embedded.ts regenerated
- CHANGELOG.md: top entry renamed to [0.16.0]; inline v0_15_0 /
v0.15.0 refs swept
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: section heading v0.15.0 → v0.16.0
Verified: `gbrain --version` prints 0.16.0, migration registry /
buildPlan / put_page / mcp-tool-defs / handlers tests all green
(49/49).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: reframe v0.16 durability headline around OpenClaw crashes
"Laptop closed mid-run" framing implied a consumer workflow. Real pain is
OpenClaw subagents dying daily on worker kill, memory blip, or timeout.
Headline + README copy match the body now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: regenerate llms-full.txt after README copy change
Regen drift guard caught the README edit from
|
||
|
|
d8613366a5 |
Minions v7 + v0.11.1 canonical migration + skillify (#130)
* feat: add minion_jobs schema, migration v5, and executeRaw to BrainEngine Foundation for the Minions job queue system. Adds: - minion_jobs table (20 columns) with CHECK constraints, partial indexes, and RLS. Inspired by BullMQ's job model, adapted for Postgres. - Migration v5 creates the table for existing databases. - executeRaw<T>() method on BrainEngine interface for raw SQL access, needed by the Minions module for claim queries (FOR UPDATE SKIP LOCKED), token-fenced writes, and atomic stall detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions job queue — queue, worker, backoff, types BullMQ-inspired Postgres-native job queue built into GBrain. No Redis. No external dependencies. Postgres transactions replace Lua scripts. - MinionQueue: submit, claim (FOR UPDATE SKIP LOCKED), complete/fail (token-fenced), atomic stall detection (CTE), delayed promotion, parent-child resolution, prune, stats - MinionWorker: handler registry, lock renewal, graceful SIGTERM, exponential backoff with jitter, UnrecoverableError bypass - MinionJobContext: updateProgress(), log(), isActive() for handlers - 8-state machine: waiting/active/completed/failed/delayed/dead/ cancelled/waiting-children Patterns stolen from: BullMQ (lock tokens, stall detection, flows), Sidekiq (dead set, backoff formula), Inngest (checkpoint/resume). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 43 tests for Minions job queue Full coverage of the Minions module against PGLite in-memory: - Queue CRUD (9): submit, get, list, remove, cancel, retry, duplicate - State machine (6): waiting→active→completed/failed, retry→delayed→waiting - Backoff (4): exponential, fixed, jitter range, attempts_made=0 edge - Stall detection (3): detect stalled, counter increment, max→dead - Dependencies (5): parent waits, fail_parent, continue, remove_dep, orphan - Worker lifecycle (5): register, start-without-handlers, claim+execute, non-Error throws, UnrecoverableError bypass - Lock management (3): renewal, token mismatch, claim sets lock fields - Claim mechanics (4): empty queue, priority ordering, name filtering, delayed promotion timing - Cancel & retry (2): cancel active, retry dead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions CLI commands and MCP operations Wire Minions into the GBrain CLI and MCP layer: CLI (gbrain jobs): submit <name> [--params JSON] [--follow] [--dry-run] list [--status S] [--queue Q] [--limit N] get <id> — detailed view with attempt history cancel/retry/delete <id> prune [--older-than 30d] stats — job health dashboard work [--queue Q] [--concurrency N] — Postgres-only worker daemon 6 MCP operations (contract-first, auto-exposed via MCP server): submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress Built-in handlers: sync, embed, lint, import. --follow runs inline. Worker daemon blocked on PGLite (exclusive file lock). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for Minions job queue CLAUDE.md: added Minions files to key files, updated operation count (36), BrainEngine method count (38), test file count (45), added jobs CLI commands. CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall detection, worker daemon). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions v2 — agent orchestration primitives (pause/resume, inbox, tokens, replay) Adds the foundation for Minions as universal agent orchestration infrastructure. GBrain's Postgres-native job queue now supports durable, observable, steerable background agents. The OpenClaw plugin (separate repo) will consume these via library import, not MCP, for zero-latency local integration. ## New capabilities - **Concurrent worker** — Promise pool replaces sequential loop. Per-job AbortController for cooperative cancellation. Graceful shutdown waits for all in-flight jobs via Promise.allSettled. - **Pause/resume** — pauseJob clears the lock and fires AbortSignal on active jobs. Handlers check ctx.signal.aborted and exit cleanly. resumeJob returns paused jobs to waiting. Catch block skips failJob when signal.aborted. - **Inbox (separate table)** — minion_inbox table for sidechannel messages. sendMessage with sender validation (parent job or admin). readInbox is token-fenced and marks read_at atomically. Separate table avoids row bloat from rewriting JSONB on every send. - **Token accounting** — tokens_input/tokens_output/tokens_cache_read columns. updateTokens accumulates; completeJob rolls child tokens up to parent. USD cost computed at read time (no cost_usd column — pricing too volatile). - **Job replay** — replayJob clones a terminal job with optional data overrides. New job, fresh attempts, no parent link. ## Handler contract additions MinionJobContext now provides: - `signal: AbortSignal` — cooperative cancellation - `updateTokens(tokens)` — accumulate token usage - `readInbox()` — check for sidechannel messages - `log()` — now accepts string or TranscriptEntry ## MCP operations added pause_job, resume_job, replay_job, send_job_message — all auto-generate CLI commands and MCP server endpoints. ## Library exports package.json exports map adds ./minions and ./engine-factory paths so plugins can `import { MinionQueue } from 'gbrain/minions'` for direct library use. ## Instruction layer (the teaching) - skills/minion-orchestrator/SKILL.md — when/how to use Minions, decision matrix, lifecycle management, anti-patterns - skills/conventions/subagent-routing.md — cross-cutting rule: all background work goes through Minions - RESOLVER.md — trigger entries for agent orchestration - manifest.json — registered ## Schema migration v6 Additive: 3 token columns, paused status, minion_inbox table with unread index. Full Postgres + PGLite support. No backfill needed. ## Tests 65 tests (was 43): pause/resume (5), inbox (6), tokens (4), replay (4), concurrent worker context (3), plus all existing coverage. ## What's NOT in this commit Deferred to follow-up PRs: - LISTEN/NOTIFY subscribe (needs real Postgres E2E) - Resource governor (depends on concurrent worker stress testing) - Routing eval harness (needs API keys + benchmark data) - OpenClaw plugin (separate @gbrain/openclaw-minions-plugin repo) See docs/designs/MINIONS_AGENT_ORCHESTRATION.md for full CEO-approved design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): migration v7 — agent_parity_layer schema Adds columns on minion_jobs (depth, max_children, timeout_ms, timeout_at, remove_on_complete, remove_on_fail, idempotency_key) plus the new minion_attachments table. Three partial indexes for bounded scans: idx_minion_jobs_timeout, idx_minion_jobs_parent_status, and uniq_minion_jobs_idempotency. Check constraints enforce non-negative depth and positive child cap / timeout. Additive migration — existing installs pick it up via ensureSchema on next use. No user action required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): extend types for v7 parity layer Extends MinionJob with depth/max_children/timeout_ms/timeout_at/ remove_on_complete/remove_on_fail/idempotency_key. Extends MinionJobInput with the same options plus max_spawn_depth override. Adds MinionQueueOpts (maxSpawnDepth default 5, maxAttachmentBytes default 5 MiB). Adds AttachmentInput/Attachment shapes and ChildDoneMessage in the InboxMessage union. rowToMinionJob updated to pick up the new columns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): attachments validator New module validateAttachment() gates every attachment write. Rejects empty filenames, path traversal (.., /, \), null bytes, oversized content (5 MiB default, per-queue override), invalid base64, and implausible content_type headers. Returns normalized { filename, content_type, content (Buffer), sha256, size } on success. The DB also enforces UNIQUE (job_id, filename) as defense-in-depth for concurrent addAttachment races — JS-only checks are not sufficient. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): queue v7 — depth, child cap, timeouts, cascade, idempotency, child_done Wraps completeJob and failJob in engine.transaction() so parent hook invocations (resolveParent, failParent, removeChildDependency) fold into the same transaction as the child update. A process crash between child and parent can't strand the parent in waiting-children anymore. Adds v7 behaviors: - Depth tracking. add() computes depth = parent.depth + 1 and rejects past maxSpawnDepth (default 5). - Per-parent child cap. add() takes SELECT ... FOR UPDATE on the parent, counts non-terminal children, rejects when count >= max_children. NULL max_children = no cap. - Per-job wall-clock timeout. claim() populates timeout_at when timeout_ms is set. New handleTimeouts() dead-letters expired rows with error_text='timeout exceeded'. Terminal — no retry. - Cascade cancel. cancelJob() walks descendants via recursive CTE with depth-100 runaway cap. Returns the root row. Re-parented descendants (parent_job_id NULL) are naturally excluded. - Idempotency. add() uses INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING; falls back to SELECT when RETURNING is empty. Same key always yields the same job id. - child_done inbox. completeJob inserts {type:'child_done', child_id, job_name, result} into the parent's inbox in the same transaction as the token rollup, guarded by EXISTS so terminal/deleted parents skip without FK violation. New readChildCompletions(parent_id, lock_token, since?) helper; token-fenced like readInbox. - removeOnComplete / removeOnFail. Deletes the row after the parent hook fires, so parent policy sees consistent state. - Attachment methods. addAttachment validates via validateAttachment then INSERTs; UNIQUE (job_id, filename) backs the JS dup check. listAttachments, getAttachment, deleteAttachment round out the API. Fixes pre-existing inverted status bug: add() now puts children in waiting/delayed (not waiting-children) and atomically flips the parent to waiting-children in the same transaction. Tests no longer need manual UPDATE workarounds. Two correctness fixes: - Sibling completion race. Under READ COMMITTED, two grandchildren completing concurrently each saw the other as still-active in the pre-commit snapshot and neither flipped the parent. Fixed by taking SELECT ... FOR UPDATE on the parent row at the start of completeJob and failJob transactions, serializing siblings on the parent lock. - JSONB double-encode. postgres.js conn.unsafe(sql, params) auto- JSON-encodes parameters. Calling JSON.stringify(obj) first stored a JSON string literal (jsonb_typeof=string) and broke payload->>'key' queries silently. Removed JSON.stringify from three call sites (child_done inbox post, updateProgress, sendMessage). PGLite tolerated both forms so unit tests missed it — real-PG E2E caught it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): worker — timeout safety net + handleTimeouts tick Worker tick now calls handleStalled() first, then handleTimeouts() — stall requeue wins over timeout dead-letter when both could fire in the same cycle. handleTimeouts() guards on lock_until > now() so stalled jobs take the retryable path. launchJob schedules a per-job setTimeout(timeout_ms) that fires ctx.signal as a best-effort handler interrupt. The timer is always cleared in .finally so process exit isn't delayed by a dangling timer. Handlers that respect AbortSignal stop cleanly; handlers that ignore it still get dead-lettered by the DB-side handleTimeouts. Removed post-completeJob and post-failJob parent-hook calls from the worker — those are now inside the queue method transactions. Worker becomes simpler and crash-safer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(minions): 33 new unit tests for v7 parity layer Covers depth cap, per-parent child cap, timeout dead-letter, cascade cancel (including the re-parent edge case), removeOnComplete / removeOnFail, idempotency (single + concurrent), child_done inbox (posted in txn + survives child removeOnComplete + since cursor), attachment validation (oversize, path traversal, null byte, duplicates, base64), AbortSignal firing on pause mid-handler, catch-block skipping failJob when aborted, worker in-flight bookkeeping, token-rollup guard when parent already terminal, and setTimeout safety-net cleanup. Existing tests updated to remove the inverted-status manual UPDATE workarounds that the add() fix made obsolete. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(e2e): Minions v7 concurrency + OpenClaw resilience coverage minions-concurrency.test.ts spins two MinionWorker instances against the test Postgres, submits 20 jobs, and asserts zero double-claims (every job runs exactly once). This is the only test that actually proves FOR UPDATE SKIP LOCKED under real concurrency — PGLite runs on a single connection and can't exercise the race. minions-resilience.test.ts covers the six OpenClaw daily pains: 1. Spawn storm caps enforce under concurrent submit. 2. Agent stall → handleStalled() requeues; handleTimeouts() skips (lock_until guard). 3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade cancel stops grandchildren mid-flight. 5. Deep tree fan-in (parent → 3 children → 2 grandchildren each) completes with the full inbox chain. 6. Parent crash/recovery resumes from persisted state. helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and minion_jobs (FK dependents first) so E2E teardown doesn't leak rows between runs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: release v0.11.0 — Minions v7 agent orchestration primitives Bumps VERSION / package.json to 0.11.0. Adds CHANGELOG entry covering depth tracking, max_children, per-job timeouts, cascade cancel, idempotency keys, child_done inbox, removeOnComplete/Fail, attachments, migration v7, plus the two correctness fixes (sibling completion race and JSONB double-encode). TODOS.md captures the four v7 follow-ups: per-queue rate limiting, repeat/cron scheduler, worker event emitter, and waitForChildren convenience helpers. 1066 unit + 105 E2E = 1171 tests passing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): unify JSONB inserts, tighten nullish coalescing Three non-blocker cleanups from post-ship review of v0.11.0: - queue.ts add() and completeJob(): pre-stringifying with JSON.stringify while other sites pass raw objects with $n::jsonb casts. postgres.js double-encodes if you stringify first — works on PGLite (text→JSONB auto-cast), fails silently on real PG. Unify on raw object + explicit $n::jsonb cast. - queue.ts readChildCompletions: since clause used sent_at > $2 relying on PG's implicit text→TIMESTAMPTZ coercion. Explicit $2::timestamptz is safer and clearer. - types.ts rowToMinionJob: parent_job_id used || which coerces 0 to null. Harmless today (SERIAL IDs start at 1) but ?? is semantically correct. All 110 unit tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): updateProgress missed $1::jsonb cast in unification Residual from |