mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 17:02:19 +00:00
Compare commits
28
Commits
@@ -1,39 +0,0 @@
|
||||
<!--
|
||||
Tier 5.5 Externally-Authored Query Submission template
|
||||
See eval/CONTRIBUTING.md for the full workflow.
|
||||
-->
|
||||
|
||||
## Summary
|
||||
|
||||
Submitting **N** Tier 5.5 queries for BrainBench.
|
||||
|
||||
- Author handle: `@your-handle`
|
||||
- File location: `eval/external-authors/your-handle/queries.json`
|
||||
- Queries authored fresh (not copy-pasted from a model output)
|
||||
- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`)
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes
|
||||
- [ ] At least 20 queries
|
||||
- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true`
|
||||
- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601)
|
||||
- [ ] Phrasing is varied (not all the same template)
|
||||
- [ ] `author` field matches my handle
|
||||
|
||||
## Phrasing variety (optional self-audit)
|
||||
|
||||
Tick the styles represented in your batch:
|
||||
|
||||
- [ ] Full sentence questions
|
||||
- [ ] Fragment-style ("crypto founder Goldman Sachs background")
|
||||
- [ ] Comparison ("X vs Y")
|
||||
- [ ] Follow-up ("And who else...")
|
||||
- [ ] Imperative ("Pull up Alice Davis")
|
||||
- [ ] Trait-based ("the demanding engineering leader")
|
||||
- [ ] Abstention bait (answer is "not in corpus")
|
||||
|
||||
## Notes to reviewer
|
||||
|
||||
Anything worth flagging — ambiguous cases, corpus gaps you found, specific
|
||||
phrasings you were uncertain about.
|
||||
@@ -28,4 +28,4 @@ jobs:
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: bun install
|
||||
- run: bun run test
|
||||
- run: bun test
|
||||
|
||||
+2
-8
@@ -5,15 +5,9 @@ bin/
|
||||
.env
|
||||
.env.*
|
||||
!.env.*.example
|
||||
# Bun --compile temp artifacts. Each build emits a new hash-named .bun-build
|
||||
# file in cwd; glob catches all of them.
|
||||
*.bun-build
|
||||
.18a49dfd730ff378-00000000.bun-build
|
||||
.18a49f9dfb996f70-00000000.bun-build
|
||||
.gstack/
|
||||
supabase/.temp/
|
||||
.claude/skills/
|
||||
.idea
|
||||
eval/reports/
|
||||
eval/data/world-v1/world.html
|
||||
|
||||
# BrainBench amara-life-v1 Opus cache (regenerate via eval:generate-amara-life)
|
||||
eval/data/amara-life-v1/_cache/
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
# Agents working on GBrain
|
||||
|
||||
This is your install + operating protocol. Claude Code reads `./CLAUDE.md` automatically.
|
||||
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
|
||||
start here.
|
||||
|
||||
## Install (5 min)
|
||||
|
||||
1. Clone: `git clone https://github.com/garrytan/gbrain ~/gbrain && cd ~/gbrain`
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
1. `./AGENTS.md` (this file) — install + operating protocol.
|
||||
2. [`./CLAUDE.md`](./CLAUDE.md) — architecture reference, key files, trust boundaries,
|
||||
test layout.
|
||||
3. [`./skills/RESOLVER.md`](./skills/RESOLVER.md) — skill dispatcher. Read before any task.
|
||||
|
||||
## Trust boundary (critical)
|
||||
|
||||
GBrain distinguishes **trusted local CLI callers** (`OperationContext.remote = false`,
|
||||
set by `src/cli.ts`) from **untrusted agent-facing callers** (`remote = true`, set by
|
||||
`src/mcp/server.ts`). Security-sensitive operations like `file_upload` tighten filesystem
|
||||
confinement when `remote = true` and default to strict behavior when unset. If you are
|
||||
writing or reviewing an operation, consult `src/core/operations.ts` for the contract.
|
||||
|
||||
## Common tasks
|
||||
|
||||
- **Configure:** [`docs/ENGINES.md`](./docs/ENGINES.md),
|
||||
[`docs/guides/live-sync.md`](./docs/guides/live-sync.md),
|
||||
[`docs/mcp/DEPLOY.md`](./docs/mcp/DEPLOY.md).
|
||||
- **Debug:** [`docs/GBRAIN_VERIFY.md`](./docs/GBRAIN_VERIFY.md),
|
||||
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
|
||||
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
## Before shipping
|
||||
|
||||
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
|
||||
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
|
||||
not by hand.
|
||||
|
||||
## Privacy
|
||||
|
||||
Never commit real names of people, companies, or funds into public artifacts. See the
|
||||
Privacy rule in `./CLAUDE.md`. GBrain pages reference real contacts; public docs must
|
||||
use generic placeholders (`alice-example`, `acme-example`, `fund-a`).
|
||||
|
||||
## Forks
|
||||
|
||||
If you are a fork, regenerate `llms.txt` + `llms-full.txt` with your own URL base before
|
||||
publishing: `LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms`.
|
||||
+1
-2891
File diff suppressed because it is too large
Load Diff
@@ -9,111 +9,45 @@ cron scheduling, reports, identity, and access control.
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~41 shared operations (adds `find_orphans` in v0.12.3). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~30 shared operations. CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
|
||||
**Trust boundary:** `OperationContext.remote` distinguishes trusted local CLI callers
|
||||
(`remote: false` set by `src/cli.ts`) from untrusted agent-facing callers
|
||||
(`remote: true` set by `src/mcp/server.ts`). Security-sensitive operations like
|
||||
`file_upload` tighten filesystem confinement when `remote=true` and default to
|
||||
strict behavior when unset.
|
||||
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation)
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine)
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 37 BrainEngine methods
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency).
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted)
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
- `src/core/errors.ts` (v0.19.0) — `StructuredAgentError` + `buildError` + `serializeError`. Every new v0.19.0 agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches v0.17.0 `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` (v0.19.0) — 36 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` (v0.19.0) — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. As of v0.22.0, `searchKeyword` / `searchKeywordChunks` / `searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `wintermute/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided)
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level)
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic.
|
||||
- `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass. **v0.19:** AGENTS.md workspaces now resolve natively (see `src/core/resolver-filenames.ts`) — gbrain inspects the 107-skill OpenClaw deployment whether the routing file is `RESOLVER.md` or `AGENTS.md`. `DEFERRED[]` is empty — Checks 5 + 6 shipped as real code, not issue URLs.
|
||||
- `src/core/resolver-filenames.ts` (v0.19) — central list of accepted routing filenames (`RESOLVER.md`, `AGENTS.md`). Shared by `findRepoRoot`, `check-resolvable`, and skillpack install so every code path walks the same fallback chain.
|
||||
- `src/commands/skillify.ts` + `src/core/skillify/{generator,templates}.ts` (v0.19) — `gbrain skillify scaffold <name>` creates all stubs for a new skill in one command: SKILL.md, script, tests, routing-eval.jsonl, resolver entry, filing-rules pointer. `gbrain skillify check <script>` runs the 10-step checklist (LLM evals, routing evals, check-resolvable gate, filing audit) against a candidate skill before it lands.
|
||||
- `src/commands/skillify-check.ts` (v0.19) — `gbrain skillpack-check` agent-readable health report. Exit 0/1/2 for CI pipeline gating; JSON for debugging. Wraps `check-resolvable --json`, `doctor --json`, and migration ledger into one payload so agents can decide whether a human action is required.
|
||||
- `src/commands/skillpack.ts` + `src/core/skillpack/{bundle,installer}.ts` (v0.19) — `gbrain skillpack install` drops gbrain's curated 25-skill bundle into a host workspace, managed-block style. Never clobbers local edits; tracks a skill manifest so subsequent `install --update` diffs cleanly. Bundle builder (`skillpack/bundle.ts`) packages the set from `skills/` into a versioned payload.
|
||||
- `src/core/skill-manifest.ts` (v0.19) — parser for `skill-manifest.json` records. Used by skillpack installer to detect drift between the shipped bundle and the user's local edits, so updates merge instead of overwriting.
|
||||
- `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` (v0.19) — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills/<name>/routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost); `--llm` opts into a Haiku tie-break layer for CI. False positives surface before users hit them.
|
||||
- `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` (v0.19) — Check 6 of `check-resolvable`. Parses new `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON. Warning-only in v0.19, upgrades to error in v0.20.
|
||||
- `src/core/dry-fix.ts` — `gbrain doctor --fix` engine. `autoFixDryViolations(fixes, {dryRun})` rewrites inlined rules to `> **Convention:** see [path](path).` callouts via three shape-aware expanders (bullet / blockquote / paragraph). Five guards: working-tree-dirty (`getWorkingTreeStatus()` returns 3-state `'clean' | 'dirty' | 'not_a_repo'`), no-git-backup, inside-code-fence, already-delegated (40-line proximity, consistent with detector), ambiguous-multi-match, block-is-callout. `execFileSync` array args (no shell — no injection surface). EOF newline preserved.
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects
|
||||
- `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier
|
||||
- `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. v0.22.1 (#409, contributed by @atrevino47): `--stale` path now starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire). On a fully-embedded brain that's a 1-line short-circuit — no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload). Caller groups by slug, embeds via OpenAI, re-upserts via `upsertChunks`. Replaces the prior page-walk that pulled every chunk's embedding column over the wire and discarded most.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`.
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't. v0.22.1 (#403): per-job timeout fires `abort.abort(new Error('timeout'))` then a 30-second grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead in DB if the handler ignores the abort signal — frees the slot even when a handler wedges (the 98-waiting-0-active prod incident driver).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. v0.22.1 (#406): `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets the counter. Worker exit classifier emits `likely_cause` field on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
|
||||
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
|
||||
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
|
||||
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
|
||||
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (v0.15.1 #219 regression guard — must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
|
||||
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output is not analogous to `schema-embedded.ts` (no runtime consumer); we commit for GitHub browsing and fork-safe fetching.
|
||||
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
|
||||
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
|
||||
- `src/commands/upgrade.ts` — Self-update CLI with post-upgrade feature discovery
|
||||
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
|
||||
- `src/commands/integrations.ts` — Standalone integration recipe management (no DB needed). Exports `getRecipeDirs()` (trust-tagged recipe sources), SSRF helpers (`isInternalUrl`, `parseOctet`, `hostnameToOctets`, `isPrivateIpv4`). Only package-bundled recipes are `embedded=true`; `$GBRAIN_RECIPES_DIR` and cwd `./recipes/` are untrusted and cannot run `command`/`http`/string health checks.
|
||||
- `src/core/search/expansion.ts` — Multi-query expansion via Haiku. Exports `sanitizeQueryForPrompt` + `sanitizeExpansionOutput` (prompt-injection defense-in-depth). Sanitized query is only used for the LLM channel; original query still drives search.
|
||||
- `src/commands/integrations.ts` — Standalone integration recipe management (no DB needed)
|
||||
- `recipes/` — Integration recipe files (YAML frontmatter + markdown setup instructions)
|
||||
- `docs/guides/` — Individual SKILLPACK guides (broken out from monolith)
|
||||
- `docs/integrations/` — "Getting Data In" guides and integration docs
|
||||
@@ -128,9 +62,9 @@ strict behavior when unset.
|
||||
- `docs/guides/diligence-ingestion.md` — Data room to brain pages pipeline
|
||||
- `docs/designs/HOMEBREW_FOR_PERSONAL_AI.md` — 10-star vision for integration system
|
||||
- `docs/mcp/` — Per-client setup guides (Claude Desktop, Code, Cowork, Perplexity)
|
||||
- BrainBench (benchmark suite + corpus): lives in the separate [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo. Not installed alongside gbrain.
|
||||
- `docs/benchmarks/` — Search quality benchmark results (reproducible, fictional data)
|
||||
- `skills/_brain-filing-rules.md` — Cross-cutting brain filing rules (referenced by all brain-writing skills)
|
||||
- `skills/RESOLVER.md` — Skill routing table (based on the agent-fork AGENTS.md pattern)
|
||||
- `skills/RESOLVER.md` — Skill routing table (modeled on Wintermute's AGENTS.md)
|
||||
- `skills/conventions/` — Cross-cutting rules (quality, brain-first, model-routing, test-before-bulk, cross-modal)
|
||||
- `skills/_output-rules.md` — Output quality standards (deterministic links, no slop, exact phrasing)
|
||||
- `skills/signal-detector/SKILL.md` — Always-on idea+entity capture on every message
|
||||
@@ -150,7 +84,6 @@ strict behavior when unset.
|
||||
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
|
||||
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
|
||||
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
|
||||
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
|
||||
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
|
||||
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
|
||||
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
|
||||
@@ -159,21 +92,6 @@ strict behavior when unset.
|
||||
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
|
||||
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
|
||||
|
||||
### BrainBench — in a sibling repo (v0.20+)
|
||||
|
||||
BrainBench — the public benchmark for personal-knowledge agent stacks — lives in
|
||||
[github.com/garrytan/gbrain-evals](https://github.com/garrytan/gbrain-evals). It
|
||||
depends on gbrain as a consumer; gbrain never pulls in the ~5MB eval corpus or
|
||||
the pdf-parse dev dep at install time.
|
||||
|
||||
gbrain's public API surface (the exports map in `package.json`) is what
|
||||
gbrain-evals consumes: `gbrain/engine`, `gbrain/types`, `gbrain/operations`,
|
||||
`gbrain/pglite-engine`, `gbrain/link-extraction`, `gbrain/import-file`,
|
||||
`gbrain/transcription`, `gbrain/embedding`, `gbrain/config`, `gbrain/markdown`,
|
||||
`gbrain/backoff`, `gbrain/search/hybrid`, `gbrain/search/expansion`,
|
||||
`gbrain/extract`. Removing any of these is a breaking change for the
|
||||
gbrain-evals consumer.
|
||||
|
||||
## Commands
|
||||
|
||||
Run `gbrain --help` or `gbrain --tools-json` for full command reference.
|
||||
@@ -182,113 +100,45 @@ Key commands added in v0.7:
|
||||
- `gbrain init` — defaults to PGLite (no Supabase needed), scans repo size, suggests Supabase for 1000+ files
|
||||
- `gbrain migrate --to supabase` / `gbrain migrate --to pglite` — bidirectional engine migration
|
||||
|
||||
Key commands added for Minions (job queue):
|
||||
- `gbrain jobs submit <name> [--params JSON] [--follow] [--dry-run]` — submit a background job. v0.13.1 adds first-class flags for every `MinionJobInput` tuning knob: `--max-stalled N`, `--backoff-type fixed|exponential`, `--backoff-delay Nms`, `--backoff-jitter 0..1`, `--timeout-ms N`, `--idempotency-key K`.
|
||||
- `gbrain jobs list [--status S] [--queue Q]` — list jobs with filters
|
||||
- `gbrain jobs get <id>` — job details with attempt history
|
||||
- `gbrain jobs cancel/retry/delete <id>` — manage job lifecycle
|
||||
- `gbrain jobs prune [--older-than 30d]` — clean old completed/dead jobs
|
||||
- `gbrain jobs stats` — job health dashboard
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.12.2:
|
||||
- `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`.
|
||||
|
||||
Key commands added in v0.12.3:
|
||||
- `gbrain orphans [--json] [--count] [--include-pseudo]` — surface pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. The natural consumer of the v0.12.0 knowledge graph layer: once edges are captured, find the gaps.
|
||||
- `gbrain doctor` gains two new reliability detection checks: `jsonb_integrity` (v0.12.0 Postgres double-encode damage) and `markdown_body_completeness` (pages truncated by the old splitBody bug). Detection only; fix hints point at `gbrain repair-jsonb` and `gbrain sync --force`.
|
||||
|
||||
Key commands added in v0.14.2:
|
||||
- `gbrain sync --skip-failed` — acknowledge the current set of failed-parse files recorded in `~/.gbrain/sync-failures.jsonl` so the sync bookmark advances past them. Doctor's `sync_failures` check shows previously-skipped as "all acknowledged" instead of warning.
|
||||
- `gbrain sync --retry-failed` — re-walk the unacknowledged failures and re-attempt parsing. If the files now succeed, they clear from the set and the bookmark advances naturally.
|
||||
- `gbrain apply-migrations --force-retry <version>` — reset a wedged migration (3 consecutive partials with no completion) by appending a `'retry'` marker. Next `apply-migrations --yes` treats the version as fresh. `complete` status never regresses to `partial` either before or after a retry marker.
|
||||
- `GBRAIN_POOL_SIZE` env var — honored by both the singleton pool (`src/core/db.ts`) and the parallel-import worker pool (`src/commands/import.ts`). Default is 10; lower to 2 for Supabase transaction pooler to avoid MaxClients crashes during `gbrain upgrade` subprocess spawns. Read at call time via `resolvePoolSize()`.
|
||||
- `gbrain doctor` gains two new checks: `sync_failures` (surfaces unacknowledged parse failures with exact paths + fix hints) and `brain_score` (renders the 5-component breakdown when score < 100: embed coverage / 35, link density / 25, timeline coverage / 15, orphans / 15, dead links / 10 — sum equals total).
|
||||
|
||||
Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain doctor --index-audit` — opt-in Postgres-only check reporting zero-scan indexes from `pg_stat_user_indexes`. Informational only; never auto-drops.
|
||||
- `gbrain doctor` schema_version check fails loudly when `version=0` — catches `bun install -g github:...` postinstall failures (#218) and routes users to `gbrain apply-migrations --yes`.
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
`bun test` runs all tests (34 unit test files + 5 E2E test files). Unit tests run
|
||||
without a database. E2E tests skip gracefully when `DATABASE_URL` is not set.
|
||||
|
||||
Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts`
|
||||
(chunking), `test/parity.test.ts` (operations contract
|
||||
(chunking), `test/sync.test.ts` (sync logic), `test/parity.test.ts` (operations contract
|
||||
parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redaction),
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations),
|
||||
`test/upgrade.test.ts` (schema migrations), `test/doctor.test.ts` (doctor command),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100 + v0.13.1 `connect()` error-wrap assertion (original error nested, #223 link in message, lock released)),
|
||||
`test/engine-factory.test.ts` (engine factory + dynamic imports),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 37 BrainEngine methods),
|
||||
`test/utils.test.ts` (shared SQL utilities), `test/engine-factory.test.ts` (engine factory + dynamic imports),
|
||||
`test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation),
|
||||
`test/publish.test.ts` (content stripping, encryption, password generation, HTML output),
|
||||
`test/backlinks.test.ts` (entity extraction, back-link detection, timeline entry generation),
|
||||
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
|
||||
`test/report.test.ts` (report format, directory structure),
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/sql-ranking.test.ts` (v0.22.0 source-boost helpers: 39 cases covering longest-prefix-match in SQL CASE, detail=high temporal-bypass, three-meta-char LIKE escape (%, _, \\), single-quote SQL-literal doubling, env override parsing for GBRAIN_SOURCE_BOOST + GBRAIN_SEARCH_EXCLUDE, resolveBoostMap / resolveHardExcludes merge semantics),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels),
|
||||
`test/check-resolvable.test.ts` (resolver reachability, MECE overlap, gap detection, DRY checks + v0.14.1 proximity-based DRY detection + `extractDelegationTargets` coverage — 13 DRY cases),
|
||||
`test/dry-fix.test.ts` (v0.14.1 auto-fix: three shape-aware expander pure-function tests, five guards — working-tree-dirty, no-git-backup, inside-code-fence, already-delegated within 40 lines, ambiguous-multi-match, block-is-callout — 28 cases),
|
||||
`test/doctor-fix.test.ts` (v0.14.1 `gbrain doctor --fix` CLI integration: dry-run preview, apply path, JSON output shape — 3 cases),
|
||||
`test/check-resolvable.test.ts` (resolver reachability, MECE overlap, gap detection, DRY checks),
|
||||
`test/backoff.test.ts` (load-aware throttling, concurrency limits, active hours),
|
||||
`test/fail-improve.test.ts` (deterministic/LLM cascade, JSONL logging, test generation, rotation),
|
||||
`test/transcription.test.ts` (provider detection, format validation, API key errors),
|
||||
`test/enrichment-service.test.ts` (entity slugification, extraction, tier escalation),
|
||||
`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping),
|
||||
`test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail + v0.13.1 `max_stalled` clamp/default/plumbing coverage),
|
||||
`test/extract.test.ts` (link extraction, timeline extraction, frontmatter parsing, directory type inference),
|
||||
`test/extract-db.test.ts` (gbrain extract --source db: typed link inference, idempotency, --type filter, --dry-run JSON output),
|
||||
`test/extract-fs.test.ts` (gbrain extract --source fs: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard — the v0.12.1 N+1 dedup bug),
|
||||
`test/link-extraction.test.ts` (canonical extractEntityRefs both formats, extractPageLinks dedup, inferLinkType heuristics, parseTimelineEntries date variants, isAutoLinkEnabled config),
|
||||
`test/graph-query.test.ts` (direction in/out/both, type filter, indented tree output),
|
||||
`test/features.test.ts` (feature scanning, brain_score calculation, CLI routing, persistence),
|
||||
`test/file-upload-security.test.ts` (symlink traversal, cwd confinement, slug + filename allowlists, remote vs local trust),
|
||||
`test/query-sanitization.test.ts` (prompt-injection stripping, output sanitization, structural boundary),
|
||||
`test/search-limit.test.ts` (clampSearchLimit default/cap behavior across list_pages and get_ingest_log),
|
||||
`test/repair-jsonb.test.ts` (v0.12.2 JSONB repair: TARGETS list, idempotency, engine-awareness),
|
||||
`test/migrations-v0_12_2.test.ts` (v0.12.2 orchestrator phases: schema → repair → verify → record),
|
||||
`test/markdown.test.ts` (splitBody sentinel precedence, horizontal-rule preservation, inferType wiki subtypes),
|
||||
`test/orphans.test.ts` (v0.12.3 orphans command: detection, pseudo filtering, text/json/count outputs, MCP op),
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases),
|
||||
`test/check-resolvable-cli.test.ts` (v0.19 CLI wrapper: exit codes, JSON envelope shape, AGENTS.md fallback chain),
|
||||
`test/regression-v0_16_4.test.ts` (findRepoRoot regression guard — hermetic startDir parameterization),
|
||||
`test/filing-audit.test.ts` (v0.19 Check 6: `writes_pages` / `writes_to` frontmatter, filing-rules JSON validation),
|
||||
`test/routing-eval.test.ts` (v0.19 Check 5: fixture parsing, structural routing, ambiguous_with, Haiku tie-break layer),
|
||||
`test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers),
|
||||
`test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures),
|
||||
`test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics),
|
||||
`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source).
|
||||
`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys)
|
||||
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
|
||||
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered.
|
||||
- `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset.
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
`find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found.
|
||||
@@ -340,122 +190,26 @@ stop and remove it before starting a new one.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 25 skills
|
||||
organized by `skills/RESOLVER.md`:
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
briefing, migrate, setup, publish.
|
||||
|
||||
**Brain skills (ported from an upstream agent fork):** signal-detector, brain-ops, idea-ingest, media-ingest,
|
||||
**Brain skills (from Wintermute):** signal-detector, brain-ops, idea-ingest, media-ingest,
|
||||
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
|
||||
|
||||
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
|
||||
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
|
||||
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
|
||||
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
|
||||
routing is narrowed to what the skill actually covers.
|
||||
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
testing, soul-audit, webhook-transforms.
|
||||
|
||||
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
|
||||
## Bulk-action progress reporting
|
||||
|
||||
All bulk commands (doctor, embed, import, export, sync, extract, migrate,
|
||||
repair-jsonb, orphans, check-backlinks, lint, integrity auto, eval, files
|
||||
sync, and apply-migrations) stream progress through the shared reporter
|
||||
at `src/core/progress.ts`. Agents get heartbeats within 1 second of every
|
||||
iteration regardless of how slow the underlying work is.
|
||||
|
||||
Rules:
|
||||
- Progress always writes to **stderr**. Stdout stays clean for data output
|
||||
(`--json` payloads, final summaries, JSON action events from `extract`).
|
||||
- Non-TTY default: plain one-line-per-event human text. JSON requires the
|
||||
explicit `--progress-json` flag.
|
||||
- Global flags (`--quiet`, `--progress-json`, `--progress-interval=<ms>`)
|
||||
are parsed by `src/core/cli-options.ts` BEFORE command dispatch.
|
||||
- Phase names are machine-stable `snake_case.dot.path` (e.g.
|
||||
`doctor.db_checks`, `sync.imports`). Documented in
|
||||
`docs/progress-events.md`; additive changes only.
|
||||
- `scripts/check-progress-to-stdout.sh` is a CI guard that fails the build
|
||||
if any new code writes `\r` progress to stdout. Wired into `bun run test`.
|
||||
- Minion handlers pass `job.updateProgress` as the `onProgress` callback
|
||||
to core functions (DB-backed primary progress channel); stderr from
|
||||
`jobs work` stays coarse for daemon liveness only.
|
||||
|
||||
When wiring a new bulk command: `import { createProgress } from '../core/progress.ts'`
|
||||
and `import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'`.
|
||||
Create a reporter with `createProgress(cliOptsToProgressOptions(getCliOptions()))`,
|
||||
`start(phase, total?)` before the loop, `tick()` inside it, `finish()` after.
|
||||
For single long-running queries, use `startHeartbeat(reporter, note)` with a
|
||||
try/finally to guarantee cleanup. Never call `process.stdout.write('\r...')`
|
||||
in bulk paths, the CI guard will fail the build.
|
||||
|
||||
## Build
|
||||
|
||||
`bun build --compile --outfile bin/gbrain src/cli.ts`
|
||||
|
||||
## Version locations (single source of truth: `VERSION` file)
|
||||
|
||||
Every release advances the version in **five files at once**. Keep these in
|
||||
sync. `/ship` enforces this via Step 12's idempotency check (VERSION vs
|
||||
package.json drift), but the canonical list lives here so future runs and
|
||||
the auto-update agent know where to look.
|
||||
|
||||
**Required (every release must update all five):**
|
||||
|
||||
| File | What lives there | Format |
|
||||
|---|---|---|
|
||||
| `VERSION` | The single source of truth. Read first by `/ship`, the binary, and CI version-gate. | Bare 4-digit string `MAJOR.MINOR.PATCH.MICRO` (e.g. `0.22.1`), no leading `v`, no trailing newline-sensitivity issues. |
|
||||
| `package.json` | Bun/npm package version. `gbrain --version` reads it via the compiled binary's bundled package metadata. CI version-gate cross-checks this against `VERSION` and fails if they drift. | `"version": "0.22.1"` |
|
||||
| `CHANGELOG.md` | Top entry header `## [0.22.1] - YYYY-MM-DD` plus the "To take advantage of v0.22.1" block. | Standard Keep-a-Changelog header. |
|
||||
| `TODOS.md` | Any TODO entries that mention "follow-up from vX.Y.Z" use the version of the release that filed them. Update only when filing NEW follow-up TODOs. | Inline `vX.Y.Z` references in TODO bodies. |
|
||||
| `CLAUDE.md` | The Key Files section's per-file annotations carry `vX.Y.Z (#NNN)` tags noting which release introduced a behavior. Update whenever a wave's annotations get folded in. | Inline `vX.Y.Z (#NNN, contributed by @user)` references. |
|
||||
|
||||
**Auto-derived (no manual edit; refreshed by their own commands):**
|
||||
|
||||
- `bun.lock` — root-package version is auto-pinned from `package.json`. After
|
||||
bumping `package.json`, run `bun install` to refresh the lockfile.
|
||||
- `llms-full.txt` / `llms.txt` — auto-generated documentation bundles. After
|
||||
any release ship that touches the Key Files annotations in `CLAUDE.md`,
|
||||
run `bun run build:llms` to regenerate. The bundles do not contain a
|
||||
version pin per se; they reflect the current state of the docs they index.
|
||||
|
||||
**Historical (DO NOT bump on release):**
|
||||
|
||||
- `skills/migrations/v0.21.0.md` — migration files use the version they
|
||||
shipped FROM as their filename. v0.21.0's migration always says v0.21.0.
|
||||
- `src/commands/migrations/v0_21_0.ts` — same: migration code references
|
||||
the schema version it migrates to.
|
||||
- `test/migrations-v0_21_0.test.ts`, `test/migration-orchestrator-v0_21_0.test.ts`,
|
||||
`test/migrate.test.ts` — migration tests reference historical migration
|
||||
versions; these are correct as-is and should not move.
|
||||
- `src/core/db.ts`, `src/core/migrate.ts`, `src/core/import-file.ts`,
|
||||
`src/commands/reindex-code.ts` — code comments cite the release that
|
||||
introduced a feature. Once written, these are historical record.
|
||||
- `README.md` — references the latest published feature names by version
|
||||
(e.g. "v0.21.0 Code Cathedral"); update only when the README's marketing
|
||||
copy is intentionally being refreshed, NOT on every micro/patch bump.
|
||||
|
||||
**The /ship workflow's version idempotency check:** Step 12 reads
|
||||
`VERSION` and `package.json`, classifies as FRESH / ALREADY_BUMPED /
|
||||
DRIFT_STALE_PKG / DRIFT_UNEXPECTED, and refuses to proceed on
|
||||
DRIFT_UNEXPECTED. This is why the two must move together.
|
||||
|
||||
**The CI version-gate** rejects pushes where `VERSION` and
|
||||
`package.json` disagree, OR where `VERSION` is not strictly greater
|
||||
than master's VERSION. If a queue collision claims your version on
|
||||
master before yours lands, /ship's queue-aware allocator (Step 12)
|
||||
will detect drift and re-bump on the next run.
|
||||
|
||||
## Pre-ship requirements
|
||||
|
||||
Before shipping (/ship) or reviewing (/review), always run the full test suite:
|
||||
@@ -484,152 +238,11 @@ Files that MUST be checked on every ship:
|
||||
|
||||
A ship without updated docs is an incomplete ship. Period.
|
||||
|
||||
## CHANGELOG + VERSION are branch-scoped
|
||||
## CHANGELOG voice
|
||||
|
||||
**VERSION and CHANGELOG describe what THIS branch adds vs master, not how we got
|
||||
here.** Every feature branch that ships gets its own version bump and CHANGELOG
|
||||
entry. The entry is product release notes for users; it is not a log of internal
|
||||
decisions, review rounds, or codex findings.
|
||||
|
||||
**Write the CHANGELOG entry at /ship time, not during development.** Mid-branch
|
||||
iterations, review rounds (CEO/Eng/Codex/DX), and implementation detours belong
|
||||
in the plan file at `~/.claude/plans/`, not in the CHANGELOG. One unified entry
|
||||
per branch, covering what the branch added vs the base branch.
|
||||
|
||||
**Never edit a CHANGELOG entry that already landed on master.** If master has
|
||||
v0.18.2 and your branch adds features, bump to the next version (v0.19.0, not
|
||||
editing master's v0.18.2). When merging master into your branch, master may
|
||||
bring new CHANGELOG entries above yours — push your entry above master's
|
||||
latest and verify:
|
||||
|
||||
- Does CHANGELOG have your branch's own entry separate from master's entries?
|
||||
- Is VERSION higher than master's VERSION?
|
||||
- Is your entry the topmost `## [X.Y.Z]` entry?
|
||||
- `grep "^## \[" CHANGELOG.md` shows a contiguous version sequence?
|
||||
|
||||
If any answer is no, fix it before continuing.
|
||||
|
||||
**CHANGELOG is for users, not contributors.** Write like product release notes:
|
||||
|
||||
- Lead with what the user can now **do** that they couldn't before. Sell the capability.
|
||||
- Plain language, not implementation details. "You can now..." not "Refactored the..."
|
||||
- **Never mention internal artifacts**: plan file IDs, decision tags (D-CX-#, F-ENG-#),
|
||||
review rounds, codex findings, subcontractor credits. These are invisible to users.
|
||||
- Put contributor-facing changes in a separate `### For contributors` section at the bottom.
|
||||
- Every entry should make someone think "oh nice, I want to try that."
|
||||
|
||||
**What to omit:**
|
||||
- "Codex caught X that the CEO review missed" — private process detail.
|
||||
- "D-CX-3 split errors/warnings" — tag is meaningless to users; name the feature instead.
|
||||
- "Fix-wave PR #N supersedes #M" — supersede chains belong in PR bodies, not release notes.
|
||||
- "215 new cases, 3 decisions applied, 7 reviews cleared" — these are planning-mode metrics.
|
||||
|
||||
**What to keep:**
|
||||
- The user-facing change: what commands exist now, what flag was added, what behavior fixed.
|
||||
- Numbers that mean something to the user: TTHW, commands that timed out before, detection counts.
|
||||
- Upgrade instructions: `gbrain upgrade` + any manual step if needed.
|
||||
- Credit to external contributors when a community PR was incorporated.
|
||||
|
||||
## CHANGELOG voice + release-summary format
|
||||
|
||||
Every version entry in `CHANGELOG.md` MUST start with a release-summary section in
|
||||
the GStack/Garry voice — one viewport's worth of prose + tables that lands like a
|
||||
verdict, not marketing. The itemized changelog (subsections, bullets, files) goes
|
||||
BELOW that summary, separated by a `### Itemized changes` header.
|
||||
|
||||
The release-summary section gets read by humans, by the auto-update agent, and by
|
||||
anyone deciding whether to upgrade. The itemized list is for agents that need to
|
||||
know exactly what changed.
|
||||
|
||||
### Release-summary template
|
||||
|
||||
Use this structure for the top of every `## [X.Y.Z]` entry:
|
||||
|
||||
1. **Two-line bold headline** (10-14 words total) ... should land like a verdict, not
|
||||
marketing. Sound like someone who shipped today and cares whether it works.
|
||||
2. **Lead paragraph** (3-5 sentences) ... what shipped, what changed for the user.
|
||||
Specific, concrete, no AI vocabulary, no em dashes, no hype.
|
||||
3. **A "The X numbers that matter" section** with:
|
||||
- One short setup paragraph naming the source of the numbers (real production
|
||||
deployment OR a reproducible benchmark ... name the file/command to run).
|
||||
- A table of 3-6 key metrics with BEFORE / AFTER / Δ columns.
|
||||
- A second optional table for per-category breakdown if relevant.
|
||||
- 1-2 sentences interpreting the most striking number in concrete user terms.
|
||||
4. **A "What this means for [audience]" closing paragraph** (2-4 sentences) tying
|
||||
the metrics to a real workflow shift. End with what to do.
|
||||
|
||||
Voice rules:
|
||||
- No em dashes (use commas, periods, "...").
|
||||
- No AI vocabulary (delve, robust, comprehensive, nuanced, fundamental, etc.) or
|
||||
banned phrases ("here's the kicker", "the bottom line", etc.).
|
||||
- Real numbers, real file names, real commands. Not "fast" but "~30s on 30K pages."
|
||||
- Short paragraphs, mix one-sentence punches with 2-3 sentence runs.
|
||||
- Connect to user outcomes: "the agent does ~3x less reading" beats "improved
|
||||
precision."
|
||||
- Be direct about quality. "Well-designed" or "this is a mess." No dancing.
|
||||
|
||||
Source material to pull from:
|
||||
- CHANGELOG.md previous entry for prior context
|
||||
- Latest `gbrain-evals/docs/benchmarks/[latest].md` for headline numbers (sibling repo)
|
||||
- Recent commits (`git log <prev-version>..HEAD --oneline`) for what shipped
|
||||
- Don't make up numbers. If a metric isn't in a benchmark or production data, don't
|
||||
include it. Say "no measurement yet" if asked.
|
||||
|
||||
Target length: ~250-350 words for the summary. Should render as one viewport.
|
||||
|
||||
### "To take advantage of v[version]" block (required, v0.13+)
|
||||
|
||||
After the release-summary and BEFORE `### Itemized changes`, every `## [X.Y.Z]`
|
||||
entry MUST include a human-readable self-repair block under the heading
|
||||
`## To take advantage of v[version]`.
|
||||
|
||||
Why: `gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`.
|
||||
This chain has a known weak link — `upgrade.ts` catches post-upgrade failures as
|
||||
best-effort (so the binary still works). When that chain silently fails, users end
|
||||
up with half-upgraded brains. The self-repair block gives them a paste-ready
|
||||
recovery path; the v0.13+ `~/.gbrain/upgrade-errors.jsonl` trail + `gbrain doctor`
|
||||
integration close the loop.
|
||||
|
||||
Template (adapt the verify commands per release):
|
||||
|
||||
```markdown
|
||||
## To take advantage of v[version]
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
|
||||
warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Your agent reads `skills/migrations/v[version].md` the next time you interact with it.**
|
||||
[One sentence on whether headless agents need manual action, or whether the
|
||||
orchestrator already handled the mechanical side.]
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
[release-specific verify commands, e.g. `gbrain graph ... --depth 2`]
|
||||
gbrain stats
|
||||
```
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
```
|
||||
|
||||
**Skip this block** for patches that are pure bug fixes with zero user-facing action
|
||||
(rare). If the release has a schema migration, data backfill, or new feature the
|
||||
user needs to verify, the block is required.
|
||||
|
||||
The v0.13.0 entry in CHANGELOG.md is the canonical example.
|
||||
|
||||
### Itemized changes (the existing rules)
|
||||
|
||||
Below the release summary, write `### Itemized changes` and continue with the
|
||||
detailed subsections (Knowledge Graph Layer, Schema migrations, Security hardening,
|
||||
Tests, etc.). Same rules as before:
|
||||
CHANGELOG.md is read by agents during auto-update (Section 17). The agent summarizes
|
||||
the changelog to convince the user to upgrade. Write changelog entries that sell the
|
||||
upgrade, not document the implementation.
|
||||
|
||||
- Lead with what the user can now DO that they couldn't before
|
||||
- Frame as benefits and capabilities, not files changed or code written
|
||||
@@ -643,13 +256,6 @@ Tests, etc.). Same rules as before:
|
||||
a community PR, name the contributor with `Contributed by @username`. Contributors
|
||||
did real work. Thank them publicly every time, no exceptions.
|
||||
|
||||
### Reference: v0.12.0 entry as canonical example
|
||||
|
||||
The v0.12.0 entry in CHANGELOG.md is the canonical example of the format. Match its
|
||||
structure for every future version: bold headline, lead paragraph, "numbers that
|
||||
matter" with BrainBench-style before/after table, "what this means" closer, then
|
||||
`### Itemized changes` with the detailed sections below.
|
||||
|
||||
## Version migrations
|
||||
|
||||
Create a migration file at `skills/migrations/v[version].md` when a release
|
||||
@@ -679,130 +285,6 @@ Write migration files as agent instructions, not technical notes. Tell the agent
|
||||
what to do, step by step, with exact commands. See `skills/migrations/v0.5.0.md`
|
||||
for the pattern.
|
||||
|
||||
## Migration is canonical, not advisory
|
||||
|
||||
GBrain's job is to deliver a canonical, working setup to every user on upgrade.
|
||||
Anything that looks like a "host-repo change" — AGENTS.md, cron manifests,
|
||||
launchctl units, config files outside `~/.gbrain/` — is a GBrain migration
|
||||
step, not a nudge we leave for the host-repo maintainer. Migrations edit host
|
||||
files (with backups) to make the canonical setup real. Exceptions: changes
|
||||
that require human judgment (content edits, renames that break semantics,
|
||||
host-specific handler registration where shell-exec would be an RCE surface).
|
||||
Everything mechanical ships in the migration.
|
||||
|
||||
**Test:** if shipping a feature requires a sentence that starts with "in
|
||||
your AGENTS.md, add…" or "in your cron/jobs.json, rewrite…", the migration
|
||||
orchestrator should be doing that edit, not the user.
|
||||
|
||||
**The exception is host-specific code.** For custom Minion handlers
|
||||
(host-specific integrations like inbox sweeps or third-party API scanners), shipping them as a
|
||||
data file the worker would exec is an RCE surface. Those get registered in
|
||||
the host's own repo via the plugin contract (`docs/guides/plugin-handlers.md`);
|
||||
the migration orchestrator emits a structured TODO to
|
||||
`~/.gbrain/migrations/pending-host-work.jsonl` + the host agent walks the
|
||||
TODOs using `skills/migrations/v0.11.0.md` — stays host-agnostic, still
|
||||
canonical.
|
||||
|
||||
## Privacy rule: scrub real names from public docs
|
||||
|
||||
**Never reference real people, companies, funds, or private agent names in any
|
||||
public-facing artifact.** Public artifacts include: `CHANGELOG.md`, `README.md`,
|
||||
`docs/`, `skills/`, PR titles + bodies, commit messages, and comments in checked-in
|
||||
code. Query examples, benchmark stories, and migration guides MUST use generic
|
||||
placeholders.
|
||||
|
||||
Why: gbrain runs a personal knowledge brain containing notes on real people and
|
||||
real companies (YC founders, portfolio companies, funds, investors, meeting
|
||||
attendees). When a doc copies a query like `gbrain graph diana-hu --depth 2` or
|
||||
names a specific agent fork like `Wintermute`, that real name gets indexed by
|
||||
search engines, surfaced in cross-references, and distributed with every release.
|
||||
|
||||
**Name mapping** to use in examples:
|
||||
- Agent forks → `your agent fork`, `a downstream agent`, or `agent-fork`
|
||||
- Example person → `alice-example`, `charlie-example`, or `a-founder`
|
||||
- Example company → `acme-example`, `widget-co`, or `a-company`
|
||||
- Example fund → `fund-a`, `fund-b`, `fund-c`
|
||||
- Example deal → `acme-seed`, `widget-series-a`
|
||||
- Example meeting → `meetings/2026-04-03` (generic date is fine)
|
||||
- Example user → `you` or `the user`, never a proper name
|
||||
|
||||
**Specific rule: never say `Wintermute` in any CHANGELOG, README, doc, PR, or
|
||||
commit message.** When the temptation is to illustrate with the real fork name:
|
||||
- Reader-facing copy → `your OpenClaw` (covers Wintermute, Hermes, AlphaClaw,
|
||||
and any other downstream OpenClaw deployment in one term the reader already
|
||||
recognizes).
|
||||
- First-person / origin-story copy → `Garry's OpenClaw` (honest that this is
|
||||
the production deployment driving the feature, without exposing the private
|
||||
agent's name).
|
||||
|
||||
`Wintermute` may appear in private artifacts (scratch plans under
|
||||
`~/.gstack/projects/…`, memory files, conversation transcripts, CEO-review
|
||||
plans) — those aren't distributed. Anything checked into this repo or shipped
|
||||
in a release must use the OpenClaw phrasing above. Sweeping a stale reference
|
||||
is a small clean-up PR, not a debate.
|
||||
|
||||
**When in doubt, ask yourself:** "Would this query reveal private information
|
||||
about the user's contacts, investments, or portfolio if it were read by a
|
||||
stranger?" If yes, replace with generic placeholders.
|
||||
|
||||
**Illustrative API examples with household-brand companies** (Stripe, Brex, OpenAI,
|
||||
GitHub, etc.) are fine — they're public entities, not contacts in anyone's brain.
|
||||
Do not confuse illustrative API examples with queries that reveal real
|
||||
relationships.
|
||||
|
||||
## Responsible-disclosure rule: don't broadcast attack surface in release notes
|
||||
|
||||
**When a release fixes a security gap or a user-impacting bug, describe the fix
|
||||
functionally. Do not enumerate the attack surface, quantify the exposure window,
|
||||
or highlight the most sensitive records by name in public-facing artifacts.**
|
||||
|
||||
Public-facing artifacts include: `CHANGELOG.md`, `README.md`, `docs/`, PR titles
|
||||
and bodies, commit messages, GitHub issue titles and comments, release pages,
|
||||
tweets, blog posts.
|
||||
|
||||
**Don't write:**
|
||||
- "10 tables were publicly readable by the anon key for months, including X, Y, Z"
|
||||
- "X and Y are the most sensitive ones"
|
||||
- "N tables exposed. Fix: enable RLS on these specific tables: ..."
|
||||
|
||||
**Do write:**
|
||||
- "Security hardening pass. Fresh installs secure by default. Existing brains
|
||||
brought to the same bar automatically on upgrade."
|
||||
- "If `gbrain doctor` still flags anything after upgrade, the message names each
|
||||
table and gives the exact fix."
|
||||
|
||||
Why: anyone reading the release page before they've upgraded now has a directed
|
||||
probe list for unpatched installs. The source code ships the specifics anyway
|
||||
(`src/schema.sql`, `src/core/migrate.ts`, test fixtures) — reverse engineers can
|
||||
get them. But the release page is a broadcast channel. Don't hand attackers a
|
||||
curated list with a banner.
|
||||
|
||||
**The test:** if a reader with no prior context could read the release note and
|
||||
walk away knowing "gbrain at version X has table Y readable by anon key until
|
||||
they patch," the note is too specific. Rewrite until that's no longer possible.
|
||||
|
||||
**What IS fine in public artifacts:**
|
||||
- The mechanism of the fix ("the check now scans every public table instead of
|
||||
a hardcoded allowlist").
|
||||
- User-facing operator ergonomics (the escape-hatch SQL template, the upgrade
|
||||
commands, the breaking-change flag).
|
||||
- Credit to contributors.
|
||||
- Generic framing of severity ("security posture tightening pass") without
|
||||
quantification.
|
||||
|
||||
**What stays in private artifacts (plan files, private memories, internal docs):**
|
||||
- Specific table names, record counts, exposure duration.
|
||||
- Which records stand out as highest-risk.
|
||||
- Detailed before/after tables in the "numbers that matter" format.
|
||||
|
||||
If the CEO/Eng review of a plan produces a detailed exposure table, keep it in
|
||||
the plan file under `~/.claude/plans/` or `~/.gstack/projects/`. Don't copy it
|
||||
into the CHANGELOG or PR body.
|
||||
|
||||
Applies retroactively: if you see a prior CHANGELOG entry naming attack-surface
|
||||
specifics, scrub it as a small cleanup commit, the same way a stale Wintermute
|
||||
reference gets swept.
|
||||
|
||||
## Schema state tracking
|
||||
|
||||
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
|
||||
@@ -825,22 +307,6 @@ done
|
||||
|
||||
If any SHA differs from what's in the workflow files, update the pin and version comment.
|
||||
|
||||
## PR descriptions cover the whole branch
|
||||
|
||||
Pull request titles and bodies must describe **everything in the PR diff against the
|
||||
base branch**, not just the most recent commit you made. When you open or update a
|
||||
PR, walk the full commit range with `git log --oneline <base>..<head>` and write the
|
||||
body to cover all of it. Group by feature area (schema, code, tests, docs) — not
|
||||
chronologically by commit.
|
||||
|
||||
This matters because reviewers read the PR body to understand what's shipping. If
|
||||
the body only covers your last commit, they miss everything else and can't review
|
||||
properly. A 7-commit PR with a body that describes commit 7 is worse than no body
|
||||
at all — it actively misleads.
|
||||
|
||||
When in doubt, run `gh pr view <N> --json commits --jq '[.commits[].messageHeadline]'`
|
||||
to see what's actually in the PR before writing the body.
|
||||
|
||||
## Community PR wave process
|
||||
|
||||
Never merge external PRs directly into master. Instead, use the "fix wave" workflow:
|
||||
|
||||
+3
-58
@@ -3,17 +3,6 @@
|
||||
Read this entire file, then follow the steps. Ask the user for API keys when needed.
|
||||
Target: ~30 minutes to a fully working brain.
|
||||
|
||||
## Step 0: If you are not Claude Code
|
||||
|
||||
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
|
||||
protocol (install, read order, trust boundary, common tasks). Claude Code reads
|
||||
`CLAUDE.md` automatically and can skip ahead.
|
||||
|
||||
If you fetched this file by URL without cloning yet, the companion files live at:
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` — start here
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms.txt` — full doc map
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms-full.txt` — same map, inlined
|
||||
|
||||
## Step 1: Install GBrain
|
||||
|
||||
```bash
|
||||
@@ -26,11 +15,6 @@ bun install && bun link
|
||||
Verify: `gbrain --version` should print a version number. If `gbrain` is not found,
|
||||
restart the shell or add the PATH export to the shell profile.
|
||||
|
||||
> **Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
> postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
> aborts with `Aborted()` when it opens PGLite. Use the `git clone + bun link` path
|
||||
> above. Tracking issue: [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
## Step 2: API Keys
|
||||
|
||||
Ask the user for these:
|
||||
@@ -69,30 +53,6 @@ gbrain embed --stale # generate vector embeddings
|
||||
gbrain query "key themes across these documents?"
|
||||
```
|
||||
|
||||
## Step 4.5: Wire the Knowledge Graph
|
||||
|
||||
If the user already had a brain repo (Step 3 imported existing markdown), backfill
|
||||
the typed-link graph and structured timeline. This populates the `links` and
|
||||
`timeline_entries` tables that future writes will maintain automatically.
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db --dry-run | head -20 # preview
|
||||
gbrain extract links --source db # commit
|
||||
gbrain extract timeline --source db # dated events
|
||||
gbrain stats # verify links > 0
|
||||
```
|
||||
|
||||
For brand-new empty brains, skip this step — auto-link populates the graph as the
|
||||
agent writes pages going forward. There is nothing to backfill yet.
|
||||
|
||||
After this step:
|
||||
- `gbrain graph-query <slug> --depth 2` works (relationship traversal)
|
||||
- Search ranks well-connected entities higher (backlink boost)
|
||||
- Every future `put_page` auto-creates typed links and reconciles stale ones
|
||||
|
||||
If a user has a very large brain (>10K pages), `extract --source db` is idempotent
|
||||
and supports `--since YYYY-MM-DD` for incremental runs.
|
||||
|
||||
## Step 5: Load Skills
|
||||
|
||||
Read `~/gbrain/skills/RESOLVER.md`. This is the skill dispatcher. It tells you which
|
||||
@@ -143,28 +103,13 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
|
||||
|
||||
## Step 9: Verify
|
||||
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 6 verification checks. Check #4 (live sync
|
||||
actually works) is the most important.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
cd ~/gbrain && git pull origin master && bun install
|
||||
gbrain init # apply schema migrations (idempotent)
|
||||
gbrain post-upgrade # show migration notes for the version range
|
||||
cd ~/gbrain && git pull origin main && bun install
|
||||
```
|
||||
|
||||
Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
|
||||
For v0.12.2+ specifically: if your brain is Postgres- or Supabase-backed and
|
||||
predates v0.12.2, the `v0_12_2` migration runs `gbrain repair-jsonb`
|
||||
automatically during `gbrain post-upgrade` to fix the double-encoded JSONB
|
||||
columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
Then run `gbrain init` to apply any schema migrations (idempotent, safe to re-run).
|
||||
|
||||
@@ -4,14 +4,10 @@ Your AI agent is smart but forgetful. GBrain gives it a brain.
|
||||
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain powering his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up and the brain is smarter than when you went to bed.
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
GBrain is those patterns, generalized. 25 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
|
||||
|
||||
## Install
|
||||
|
||||
### On an agent platform (recommended)
|
||||
@@ -28,12 +24,7 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
agent operating protocol (install, read order, trust boundary, common tasks). For
|
||||
the full doc map, use `llms.txt` at the same URL root.
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 25 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
### Standalone CLI (no agent)
|
||||
|
||||
@@ -44,11 +35,6 @@ gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
```
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
&& bun link` as shown above. See [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
```
|
||||
3 results (hybrid search, 0.12s):
|
||||
|
||||
@@ -87,25 +73,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
### Using gbrain with GStack
|
||||
## The 25 Skills
|
||||
|
||||
If your engineering agent runs on [GStack](https://github.com/garrytan/gstack), point it at gbrain for code lookup instead of grep+read. Cathedral II (v0.21.0) ships call-graph edges and two-pass retrieval — `/investigate`, `/review`, `/plan-eng-review`, and `/office-hours` all benefit when the agent walks the symbol graph instead of scanning files line by line.
|
||||
|
||||
The five magical-moment commands:
|
||||
|
||||
```bash
|
||||
gbrain code-callers searchKeyword # who calls this symbol?
|
||||
gbrain code-callees searchKeyword # what does this symbol call?
|
||||
gbrain code-def BrainEngine # where is X defined?
|
||||
gbrain code-refs BrainEngine # all reference sites
|
||||
gbrain query "how does N+1 handling work" --near-symbol BrainEngine.searchKeyword --walk-depth 2
|
||||
```
|
||||
|
||||
All five auto-emit JSON on non-TTY (gh-CLI convention) so a GStack subagent shelling out via bash gets a clean parseable response. Run `gbrain sources add <repo> --strategy code` to index a repo, then your agent's brain-first lookup covers code, not just markdown. ([Cathedral II release notes](CHANGELOG.md#0210---2026-04-25))
|
||||
|
||||
## The 29 Skills
|
||||
|
||||
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
GBrain ships 25 skills organized by `skills/RESOLVER.md`. The resolver tells your agent which skill to read for any task.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
@@ -149,10 +119,6 @@ GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
| **webhook-transforms** | External events (SMS, meetings, social mentions) converted into brain pages with entity extraction. |
|
||||
| **testing** | Validates every skill has SKILL.md with frontmatter, manifest coverage, resolver coverage. |
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
@@ -180,7 +146,6 @@ Signal arrives (meeting, email, tweet, link)
|
||||
-> Brain-ops: check the brain first (gbrain search, gbrain get)
|
||||
-> Respond with full context
|
||||
-> Write: update brain pages with new information + citations
|
||||
-> Auto-link: typed relationships extracted on every write (zero LLM calls)
|
||||
-> Sync: gbrain indexes changes for next query
|
||||
```
|
||||
|
||||
@@ -194,171 +159,6 @@ The system gets smarter on its own. Entity enrichment auto-escalates: a person m
|
||||
> "What have I said about the relationship between shame and founder performance?"
|
||||
> ... searches YOUR thinking, not the internet
|
||||
|
||||
## Minions: your sub-agents won't drop work anymore
|
||||
|
||||
A durable, Postgres-native job queue built into the brain. Every long-running agent task is now a job that survives gateway restarts, streams progress, gets paused / resumed / steered mid-flight, and shows up in `gbrain jobs list`. Zero infra beyond your existing brain.
|
||||
|
||||
### The production numbers that matter
|
||||
|
||||
Here's my personal OpenClaw deployment: one Render container. Supabase Postgres holding a 45,000-page brain. 19 cron jobs firing on schedule. Real gateway load from real daily work. The task: pull a month of my social posts from an external API and ingest them end-to-end into the brain as a structured page.
|
||||
|
||||
| | Minions | `sessions_spawn` |
|
||||
|--- |--- |--- |
|
||||
| Wall time | **753ms** | **>10,000ms** (gateway timeout) |
|
||||
| Token cost | **$0.00** | ~$0.03 per run |
|
||||
| Success rate | **100%** | **0%** (couldn't even spawn) |
|
||||
| Memory/job | ~2 MB | ~80 MB |
|
||||
|
||||
Under that 19-cron load, sub-agent spawn couldn't clear the 10-second gateway wall. Minions landed it in under a second for zero tokens. **Scaling:** 19,240 posts across 36 months, single bash loop, ~15 min total, $0.00. Sub-agents: ~9 min best case, ~$1.08 in tokens, ~40% spawn failure. **Lab:** durability ∞ (SIGKILL mid-flight, 10/10 rescued), throughput ~10× faster, fan-out ~21× with no failure wall, memory ~400× less.
|
||||
|
||||
Full benchmarks live in [gbrain-evals](https://github.com/garrytan/gbrain-evals/tree/main/docs/benchmarks).
|
||||
|
||||
### The routing rule
|
||||
|
||||
> **Deterministic** (same input → same steps → same output) → **Minions**
|
||||
> **Judgment** (input requires assessment or decision) → **Sub-agents**
|
||||
|
||||
Pull posts, parse JSON, write a brain page, run a sync — deterministic. $0 tokens, survives restart, millisecond runtime. Triage the inbox, assess meeting priority, decide if a cold email deserves a reply — judgment. What sub-agents are actually good at. `minion_mode: pain_triggered` (the default) automates the routing.
|
||||
|
||||
### What's fixed
|
||||
|
||||
The six daily pains — spawn storms, agents that stop responding, forgotten dispatches, gateway crashes mid-run, runaway grandchildren, debugging soup — all belonged to the "deterministic work through a reasoning model" mistake. Minions fixes them by not making that mistake: `max_children` cap, `timeout_ms` + AbortSignal, `child_done` inbox, full `parent_job_id`/`depth`/transcript per job, Postgres durability with stall detection, cascade cancel via recursive CTE. Plus idempotency keys, attachment validation, `removeOnComplete`, and `gbrain jobs smoke` that proves the install in half a second.
|
||||
|
||||
```bash
|
||||
gbrain jobs smoke # verify install
|
||||
gbrain jobs submit sync --params '{}' # fire a background job
|
||||
gbrain jobs stats # health dashboard
|
||||
gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
|
||||
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
|
||||
```
|
||||
|
||||
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
|
||||
|
||||
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
|
||||
|
||||
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.
|
||||
|
||||
### Health check and self-heal
|
||||
|
||||
Minions is canonical as of v0.11.1 — every `gbrain upgrade` runs the migration automatically (schema → smoke → prefs → host rewrites → env-aware autopilot install). If you ever want to verify manually or wire a cron into your morning briefing:
|
||||
|
||||
```bash
|
||||
gbrain doctor # half-migrated state? prints loud banner + exits non-zero
|
||||
gbrain skillpack-check --quiet # exit 0/1/2 for pipeline gating
|
||||
gbrain skillpack-check | jq # full JSON: {healthy, summary, actions[], doctor, migrations}
|
||||
```
|
||||
|
||||
If anything's off, `actions[]` tells you the exact command to run. For deeper troubleshooting: [`docs/guides/minions-fix.md`](docs/guides/minions-fix.md).
|
||||
|
||||
Moving gateway crons to Minions (deterministic scripts, zero LLM tokens per fire): [`docs/guides/minions-shell-jobs.md`](docs/guides/minions-shell-jobs.md).
|
||||
|
||||
## Durable agents: `gbrain agent` (v0.15)
|
||||
|
||||
Your subagent runs survive crashes now. OpenClaw died mid-run? The worker re-claims on restart and replays from the last committed turn. Fan-out across 50 shards, one shard crashes — the aggregator still claims after every child reaches a terminal state and writes a mixed-outcome summary. Tool calls persist as a two-phase ledger (`pending` → `complete | failed`) so replay is safe by construction, not by hope.
|
||||
|
||||
```bash
|
||||
# Submit a single-subagent run
|
||||
gbrain agent run "summarize my last 10 journal pages"
|
||||
|
||||
# Fan out N prompts across N subagent children + 1 aggregator
|
||||
gbrain agent run "analyze every page" \
|
||||
--fanout-manifest manifests/pages.json \
|
||||
--subagent-def analyzer
|
||||
|
||||
# Tail a running job (heartbeat per turn + full transcript on completion)
|
||||
gbrain agent logs 1247 --follow --since 5m
|
||||
```
|
||||
|
||||
Durability is the point: every Anthropic turn commits to `subagent_messages`, every tool call to `subagent_tool_executions`. Worker kills, OpenClaw crashes, timeouts — all resumable. Host repos (your OpenClaw, etc.) ship their own subagent definitions via `GBRAIN_PLUGIN_PATH` + a `gbrain.plugin.json` manifest: see [`docs/guides/plugin-authors.md`](docs/guides/plugin-authors.md). Requires `ANTHROPIC_API_KEY` on the worker.
|
||||
|
||||
## Skillify: say "skillify it!" and the bug becomes structurally impossible to repeat
|
||||
|
||||
Your OpenClaw hit a new failure. You fix it once in conversation. You say "skillify it!"
|
||||
And now the fix is permanent: a SKILL.md with triggers, a deterministic script with tests, a
|
||||
routing fixture the agent re-evaluates daily, a filing audit that keeps the output from
|
||||
drifting. Ten items. Every one required. The bug can't recur.
|
||||
|
||||
Hermes and similar agent frameworks auto-create skills as a background behavior. Fine until
|
||||
you don't know what the agent shipped. Checklists decay. Tests drift. Resolver entries get
|
||||
stale. Six months later it's an opaque pile nobody has read, nobody has tested, and nobody
|
||||
is sure still works. GBrain ships the same capability except the human stays in the loop
|
||||
and every step is a command you can run.
|
||||
|
||||
### The four verbs you need (v0.19)
|
||||
|
||||
```bash
|
||||
# 1. Scaffold all 5 stub files for a new skill in one shot.
|
||||
gbrain skillify scaffold webhook-verify \
|
||||
--description "verify ngrok webhooks" \
|
||||
--triggers "verify the webhook,check tunnel" \
|
||||
--writes-pages --writes-to people/,companies/
|
||||
|
||||
# 2. Replace the SKILLIFY_STUB sentinels with real logic + real tests.
|
||||
$EDITOR skills/webhook-verify/scripts/webhook-verify.mjs
|
||||
$EDITOR test/webhook-verify.test.ts
|
||||
|
||||
# 3. Run the 10-item audit: SKILL.md exists, script exists, unit + E2E tests,
|
||||
# LLM evals, resolver entry, trigger eval, check-resolvable gate, brain filing.
|
||||
gbrain skillify check skills/webhook-verify/scripts/webhook-verify.mjs
|
||||
|
||||
# 4. Verify the whole tree: reachability, MECE overlap, DRY, routing gaps,
|
||||
# filing audit, SKILLIFY_STUB sentinels (fails if any skill still has one).
|
||||
gbrain check-resolvable # warnings advisory, errors block
|
||||
gbrain check-resolvable --strict # warnings block too (CI opt-in)
|
||||
```
|
||||
|
||||
Idempotent re-runs. `--force` regenerates stub files but NEVER duplicates a resolver row.
|
||||
Scaffold completes in under 2 seconds. The real work (your rule, your script, your tests)
|
||||
is what you spend time on. Everything else is boilerplate the CLI writes for you.
|
||||
|
||||
### `gbrain routing-eval` — catch the routing gaps your users actually hit
|
||||
|
||||
Drop a `routing-eval.jsonl` fixture next to any skill. Each line is `{intent, expected_skill,
|
||||
ambiguous_with?}`. `gbrain check-resolvable` runs the structural layer by default; `gbrain
|
||||
routing-eval --llm` runs an LLM tie-break layer for CI. False positives (wrong skill matched),
|
||||
missed routes (no skill matched), and tautological fixtures (intent copies trigger verbatim)
|
||||
all surface as specific advisories with the exact file:line to fix.
|
||||
|
||||
### Works on your OpenClaw, not just gbrain's repo
|
||||
|
||||
v0.19 teaches `gbrain check-resolvable` to accept `AGENTS.md` as a resolver file alongside
|
||||
`RESOLVER.md`, at either the skills directory OR one level up (OpenClaw-native workspace-root
|
||||
layout). The skill manifest auto-derives from walking `skills/*/SKILL.md` when `manifest.json`
|
||||
is missing. Set `OPENCLAW_WORKSPACE=~/your-openclaw/workspace` and everything just works:
|
||||
|
||||
```bash
|
||||
export OPENCLAW_WORKSPACE=~/your-openclaw/workspace
|
||||
gbrain check-resolvable --verbose
|
||||
# Auto-detects: AGENTS.md at workspace root, 107 skills derived from SKILL.md walk,
|
||||
# 15 unreachable errors surfaced, 108 advisory warnings for overlaps and gaps.
|
||||
```
|
||||
|
||||
First run on a real OpenClaw deployment found 15 unreachable skills out of 102 — about 15%
|
||||
of the tree was dark. The essay's "skills the agent can never reach" footgun, now visible.
|
||||
|
||||
### `gbrain skillpack install` — drop 25 curated skills into your OpenClaw
|
||||
|
||||
The skills gbrain ships are a curated bundle. Install them into your workspace with
|
||||
dependency closure (shared conventions come along), per-file diff protection (your local
|
||||
edits are never clobbered without `--overwrite-local`), a file lock that serializes
|
||||
concurrent installers, and an atomic managed-block update to your AGENTS.md so you can
|
||||
see exactly what gbrain wrote.
|
||||
|
||||
```bash
|
||||
gbrain skillpack list # 25 curated skills
|
||||
gbrain skillpack install brain-ops # one skill + its shared conventions
|
||||
gbrain skillpack install --all # the full bundle
|
||||
gbrain skillpack install brain-ops --dry-run # preview; no writes
|
||||
gbrain skillpack diff brain-ops # compare bundle vs your local copy
|
||||
```
|
||||
|
||||
Re-running is safe. The managed-block markers in your AGENTS.md let `skillpack install`
|
||||
accumulate rows across separate single-skill installs instead of overwriting each other.
|
||||
|
||||
**Skillify is the piece that makes the skills tree survive six months of compounding work.**
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
@@ -394,7 +194,7 @@ Run `gbrain integrations` to see status.
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 25 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
@@ -430,36 +230,6 @@ want, which you can't learn any other way.
|
||||
|
||||
Above the `---`: **compiled truth**. Your current best understanding. Gets rewritten when new evidence changes the picture. Below: **timeline**. Append-only evidence trail. Never edited, only added to.
|
||||
|
||||
## Knowledge Graph
|
||||
|
||||
Pages aren't just text. Every mention of a person, company, or concept becomes a typed link in a structured graph. The brain wires itself.
|
||||
|
||||
```
|
||||
Write a meeting page mentioning Alice and Acme AI
|
||||
-> Auto-link extracts entity refs from content (zero LLM calls)
|
||||
-> Infers types: meeting page + person ref => `attended`
|
||||
"CEO of X" pattern => `works_at`
|
||||
"invested in" => `invested_in`
|
||||
"advises", "advisor" => `advises`
|
||||
"founded", "co-founded" => `founded`
|
||||
-> Reconciles stale links: edits remove links no longer in content
|
||||
-> Backlinks rank well-connected entities higher in search
|
||||
```
|
||||
|
||||
```bash
|
||||
gbrain graph-query people/alice --type attended --depth 2
|
||||
# returns who Alice met with, transitively
|
||||
```
|
||||
|
||||
The graph powers questions vector search can't: "who works at Acme AI?", "what has Bob invested in?", "find the connection between Alice and Carol". Backfill an existing brain in one command:
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db # wire up the existing 29K pages
|
||||
gbrain extract timeline --source db # extract dated events from markdown timelines
|
||||
```
|
||||
|
||||
Then ask graph questions or watch the search ranking improve. Benchmarked side-by-side against ripgrep-BM25, vector-only RAG (same embedder), and gbrain-with-graph-disabled: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating hybrid-nograph by **+31.4 points P@5**. Isolate the contribution: v0.11→v0.12 moved the same gbrain codebase from P@5 22.1% → 49.1% on identical inputs, so typed-link extract quality is load-bearing. Full scorecards + reproducible corpus: [gbrain-evals](https://github.com/garrytan/gbrain-evals).
|
||||
|
||||
## Search
|
||||
|
||||
Hybrid search: vector + keyword + RRF fusion + multi-query expansion + 4-layer dedup.
|
||||
@@ -477,76 +247,6 @@ Query
|
||||
|
||||
Keyword alone misses conceptual matches. Vector alone misses exact phrases. RRF gets both. Search quality is benchmarked and reproducible: `gbrain eval --qrels queries.json` measures P@k, Recall@k, MRR, and nDCG@k. A/B test config changes before deploying them.
|
||||
|
||||
## Why it works: many strategies in concert
|
||||
|
||||
The brain isn't one trick. Every retrieval question goes through ~20 deterministic
|
||||
techniques layered together. No single one is magic; the win comes from stacking
|
||||
them so each layer covers what the others miss.
|
||||
|
||||
```
|
||||
Question
|
||||
│
|
||||
├─ INGESTION (every put_page)
|
||||
│ ├─ Recursive markdown chunking (or semantic / LLM-guided)
|
||||
│ ├─ Embedding cache invalidation on edit
|
||||
│ └─ Idempotent imports (content-hash dedup)
|
||||
│
|
||||
├─ GRAPH EXTRACTION (auto-link post-hook, zero LLM)
|
||||
│ ├─ Entity-ref regex (markdown links + bare slugs)
|
||||
│ ├─ Code-fence stripping (no false-positive slugs in code blocks)
|
||||
│ ├─ Typed inference cascade (FOUNDED → INVESTED → ADVISES → WORKS_AT)
|
||||
│ ├─ Page-role priors (partner-bio language → invested_in)
|
||||
│ ├─ Within-page dedup (same target collapses to one link)
|
||||
│ ├─ Stale-link reconciliation (edits remove dropped refs)
|
||||
│ └─ Multi-type link constraint (same person can works_at AND advises)
|
||||
│
|
||||
├─ SEARCH PIPELINE (every query)
|
||||
│ ├─ Intent classifier (entity / temporal / event / general — auto-routes)
|
||||
│ ├─ Multi-query expansion (Haiku rephrases the question 3 ways)
|
||||
│ ├─ Vector search (HNSW cosine over OpenAI embeddings)
|
||||
│ ├─ Keyword search (Postgres tsvector + websearch_to_tsquery)
|
||||
│ ├─ Source-aware ranking (curated dirs outrank chat/daily swamp at SQL layer)
|
||||
│ ├─ Hard-exclude (test/ archive/ attachments/ .raw/ filtered before retrieval)
|
||||
│ ├─ Reciprocal Rank Fusion (score = sum 1/(60+rank) across both)
|
||||
│ ├─ Cosine re-scoring (re-rank chunks against actual query embedding)
|
||||
│ ├─ Compiled-truth boost (assessments outrank timeline noise)
|
||||
│ ├─ Backlink boost (well-connected entities rank higher)
|
||||
│ └─ Source-aware dedup (one CT chunk per page guaranteed)
|
||||
│
|
||||
├─ GRAPH TRAVERSAL (relational queries)
|
||||
│ ├─ Recursive CTE with cycle prevention (visited-array check)
|
||||
│ ├─ Type-filtered edges (--type works_at, attended, etc.)
|
||||
│ ├─ Direction control (in / out / both)
|
||||
│ └─ Depth-capped (≤10 for remote MCP; DoS prevention)
|
||||
│
|
||||
└─ AGENT WORKFLOW (graph-confident hybrid)
|
||||
├─ Graph-query first (high-precision typed answers)
|
||||
├─ Grep fallback when graph returns nothing
|
||||
└─ Graph hits ranked first in top-K (better P@K and R@K)
|
||||
```
|
||||
|
||||
End-to-end on the BrainBench v1 corpus (240 rich-prose pages, before/after PR #188):
|
||||
|
||||
| Metric | BEFORE PR #188 | AFTER PR #188 | Δ |
|
||||
|-------------------------|----------------|---------------|-------------|
|
||||
| **Precision@5** | 39.2% | **44.7%** | **+5.4 pts**|
|
||||
| **Recall@5** | 83.1% | **94.6%** | **+11.5 pts**|
|
||||
| Correct in top-5 | 217 | 247 | **+30** |
|
||||
| Graph-only F1 (ablation)| 57.8% (grep) | **86.6%** | **+28.8 pts**|
|
||||
|
||||
Plus 5 orthogonal capability checks (identity resolution, temporal queries,
|
||||
performance at 10K-page scale, robustness to malformed input, MCP operation
|
||||
contract). All pass. Full report: [gbrain-evals](https://github.com/garrytan/gbrain-evals).
|
||||
|
||||
The point: each technique handles a class of inputs the others miss. Vector
|
||||
search misses exact slug refs; keyword catches them. Keyword misses conceptual
|
||||
matches; vector catches them. RRF picks the best of both. Compiled-truth boost
|
||||
keeps assessments above timeline noise. Auto-link extraction wires the graph
|
||||
that lets backlink boost rank well-connected entities higher. Graph traversal
|
||||
answers questions search alone can't reach. The agent picks graph-first for
|
||||
precision and falls back to keyword for recall. **All deterministic, all in
|
||||
concert, all measured.**
|
||||
|
||||
## Voice
|
||||
|
||||
Call a phone number. Your AI answers. It knows who's calling, pulls their full context from the brain, and responds like someone who actually knows your world. When the call ends, a brain page appears with the transcript, entity detection, and cross-references.
|
||||
@@ -625,45 +325,16 @@ EMBEDDINGS
|
||||
gbrain embed [<slug>|--all|--stale] Generate/refresh embeddings
|
||||
|
||||
LINKS + GRAPH
|
||||
gbrain link|unlink|backlinks Cross-reference management
|
||||
gbrain extract links|timeline|all Batch backfill from existing pages
|
||||
(--source db|fs, --type, --since, --dry-run)
|
||||
gbrain graph-query <slug> Typed traversal (--type T --depth N
|
||||
--direction in|out|both)
|
||||
|
||||
JOBS (Minions)
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] Submit a background job
|
||||
gbrain jobs list [--status S] [--queue Q] List jobs with filters
|
||||
gbrain jobs get|cancel|retry|delete <id> Manage job lifecycle
|
||||
gbrain jobs prune [--older-than 30d] Clean completed/dead jobs
|
||||
gbrain jobs stats Job health dashboard
|
||||
gbrain jobs smoke One-command health check
|
||||
gbrain jobs work [--queue Q] [--concurrency N] Start worker daemon
|
||||
|
||||
SKILLS (v0.19)
|
||||
gbrain skillify scaffold <name> Create 5 stub files + idempotent resolver row
|
||||
gbrain skillify check [path] 10-item audit of a skill
|
||||
gbrain skillpack list Print the 25 curated skills in the bundle
|
||||
gbrain skillpack install <name> Copy one skill + its shared conventions into target
|
||||
gbrain skillpack install --all Install the full curated bundle
|
||||
gbrain skillpack diff <name> Per-file diff: bundle vs target workspace
|
||||
gbrain check-resolvable [--strict] Resolver audit (reachability, MECE, DRY, routing, filing,
|
||||
SKILLIFY_STUB). Accepts RESOLVER.md OR AGENTS.md.
|
||||
gbrain routing-eval [--llm] [--json] Intent→skill routing accuracy on fixtures
|
||||
gbrain link|unlink|backlinks|graph Cross-reference management
|
||||
|
||||
ADMIN
|
||||
gbrain doctor [--json] [--fast] Health checks (resolver, skills, DB, embeddings)
|
||||
gbrain doctor --fix [--dry-run] Auto-fix DRY violations (delegate inlined rules to conventions)
|
||||
gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only)
|
||||
gbrain doctor --fix Auto-fix resolver issues
|
||||
gbrain stats Brain statistics
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain sources list|add|remove|... Multi-source brain management (v0.18)
|
||||
gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly)
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
gbrain lint [--fix] LLM artifact detection
|
||||
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
|
||||
gbrain orphans [--json] [--count] Find pages with zero inbound wikilinks
|
||||
gbrain transcribe <audio> Transcribe audio (Groq Whisper)
|
||||
gbrain research init <name> Scaffold a data-research recipe
|
||||
gbrain research list Show available recipes
|
||||
@@ -683,7 +354,7 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
|
||||
**For agents:**
|
||||
- **[skills/RESOLVER.md](skills/RESOLVER.md)** ... Start here. The skill dispatcher.
|
||||
- [Individual skill files](skills/) ... 28 standalone instruction sets (25 ship in the curated `gbrain skillpack install` bundle)
|
||||
- [Individual skill files](skills/) ... 25 standalone instruction sets
|
||||
- [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) ... Legacy reference architecture
|
||||
- [Getting Data In](docs/integrations/README.md) ... Integration recipes and data flow
|
||||
- [GBRAIN_VERIFY.md](docs/GBRAIN_VERIFY.md) ... Installation verification
|
||||
@@ -697,9 +368,6 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
- [GBRAIN_V0.md](docs/GBRAIN_V0.md) ... Full product spec
|
||||
- [CHANGELOG.md](CHANGELOG.md) ... Version history
|
||||
|
||||
**Benchmarks:**
|
||||
- [gbrain-evals](https://github.com/garrytan/gbrain-evals) ... BrainBench, the sibling repo that holds the eval harness, corpus, scorecards, and 4-adapter comparisons. Depends on gbrain; not installed alongside gbrain.
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. E2E tests: spin up Postgres with pgvector, run `bun run test:e2e`, tear down.
|
||||
|
||||
@@ -1,231 +1,7 @@
|
||||
# TODOS
|
||||
|
||||
## resolver / check-resolvable (v0.22.4 follow-ups)
|
||||
|
||||
### D10 — Extend `check-resolvable` to parse RESOLVER.md disambiguation rules
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Extend `src/core/check-resolvable.ts:357-390` to parse a structured
|
||||
disambiguation block in `RESOLVER.md` (e.g. a `## Disambiguation rules`
|
||||
numbered list with parseable `<trigger>` → `<winning-skill>` shape) and treat
|
||||
resolved overlaps as non-issues. Then the action message at
|
||||
`src/core/check-resolvable.ts:388` ("Add disambiguation rule in RESOLVER.md OR
|
||||
narrow triggers") stops lying about the OR — currently only the second branch
|
||||
silences the warning.
|
||||
|
||||
**Why:** The current MECE-overlap fix path forces authors to delete user-facing
|
||||
triggers from skill frontmatter. That's wrong for cases where two skills
|
||||
legitimately respond to the same phrase under different contexts (e.g.
|
||||
"citation audit" → focused fix vs broader brain health). A real
|
||||
disambiguation parser would let `RESOLVER.md` carry the resolution while
|
||||
keeping both skills' triggers intact for chaining.
|
||||
|
||||
**Pros:**
|
||||
- The action message stops misleading users.
|
||||
- v0.22.4 D2 used the "narrow triggers" path because the disambiguation
|
||||
parser doesn't exist yet; landing this would let v0.23+ keep dual triggers
|
||||
for genuinely-overlapping skills.
|
||||
- Aligns RESOLVER.md's stated role (the dispatcher) with what the checker
|
||||
actually reads.
|
||||
|
||||
**Cons:**
|
||||
- Introduces a new `RESOLVER.md` syntactic contract that other tooling now
|
||||
has to respect (parser, lint, downstream forks reading the same file).
|
||||
- Risk of false-positive resolution if the parser is loose.
|
||||
- ~80 lines of parser + tests; not blocking anything in v0.22.4.
|
||||
|
||||
**Context:**
|
||||
- The "OR" in the action message is misleading today. Confirmed at
|
||||
`src/core/check-resolvable.ts:388`.
|
||||
- The MECE detector loop is at `src/core/check-resolvable.ts:357-390`.
|
||||
- The disambiguation rules already exist as prose in
|
||||
`skills/RESOLVER.md` (the citation-audit row added in v0.22.4 is the
|
||||
pattern). They're agent-facing routing hints today, not parsed structure.
|
||||
|
||||
**Effort:** S (human: ~4-6 hours / CC: ~30 min for parser + 12-16 test cases).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
## code-indexing (v0.21.0 Cathedral II follow-ups)
|
||||
|
||||
### B2 — Magika auto-detect for extension-less files (Layer 9 deferred)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Embed Google's Magika ML classifier (~1MB ONNX) as a bundled asset. Wire into `detectCodeLanguage` as the fallback for files with no recognized extension (Dockerfile, Makefile, `.envrc`, shell scripts with shebangs but no `.sh`). The chunker already has `setLanguageFallback(fn)` as a module-level hook.
|
||||
|
||||
**Why:** v0.20.0 widens the file classifier from 9 to 35 extensions (Layer 2), covering most real-world cases. Extension-less files still slip through to recursive chunks. Magika would close the last common case.
|
||||
|
||||
**Pros:** Completes the file-classification story. Unblocks chunker on real-world configs + build scripts.
|
||||
|
||||
**Cons:** ~1MB asset bundled with `bun --compile`. Integration risk: Magika's ONNX runtime needs WASM compat with bun. The plan explicitly allowed deferring B2 because bundling surprises late in implementation are costly.
|
||||
|
||||
**Context:**
|
||||
- `src/core/chunkers/code.ts` exports `setLanguageFallback(fn: LanguageFallback | null)` — call at process start with a Magika-powered classifier.
|
||||
- `detectCodeLanguage(filePath, content?)` already accepts optional content for fallback paths.
|
||||
- The NPM `magika` package is the first thing to try; needs bun-compile compatibility verification.
|
||||
|
||||
**Effort:** M (human: ~2-3 days / CC: ~2 hours for the integration + CI guard).
|
||||
|
||||
**Depends on / blocked by:** Nothing. Hook is in place as of v0.20.0.
|
||||
|
||||
### A4 — full doc_comment extraction at chunk time
|
||||
**Priority:** P2
|
||||
|
||||
**What:** When the chunker emits a method/class/function, look at the comment node(s) immediately preceding the declaration and persist them as `content_chunks.doc_comment`. The FTS trigger from Layer 1b already weights `doc_comment` 'A' above `chunk_text` 'B' — the ranking is ready, the column is populated NULL today.
|
||||
|
||||
**Why:** "how does X handle N+1" should rank the docstring that explains N+1 above the function body or any prose paragraph. Layer 1b paved the ranking half; extraction is the remaining half.
|
||||
|
||||
**Pros:** Material MRR lift on natural-language queries. Zero schema work (column + trigger already in place).
|
||||
|
||||
**Cons:** Per-language convention detection — JSDoc blocks, Python docstrings (first string expression in a function body), C-style doc comments, etc. Not hard but each language has edge cases.
|
||||
|
||||
**Context:**
|
||||
- `src/core/chunkers/code.ts` emits chunks in `chunkCodeTextFull`. Walk each declaration's preceding sibling(s) for comment nodes.
|
||||
- ChunkInput already has `doc_comment?: string`. Populate at chunk time and it flows through `upsertChunks` (Layer 6 wired those columns).
|
||||
- Per-language config: leading-comment type names per language (`comment`, `line_comment`, `block_comment`, `documentation_comment`).
|
||||
- Test hook: `test/cathedral-ii-brainbench.test.ts` has a `doc_comment_matching` placeholder — flesh it out end-to-end.
|
||||
|
||||
**Effort:** M (human: ~2 days / CC: ~90 min for the 8 Layer-5 langs).
|
||||
|
||||
**Depends on / blocked by:** Nothing. Layer 1b + Layer 6 both in place.
|
||||
|
||||
### C6 — gbrain code-signature "(A, B) => C"
|
||||
**Priority:** P3 (stretch)
|
||||
|
||||
**What:** Type-signature retrieval via tree-sitter type captures per language. "Find every function whose signature returns a Promise<User>" or "(string, number) => boolean".
|
||||
|
||||
**Why:** Each language's type system is its own mini-cathedral. Ship per-language rather than as one item.
|
||||
|
||||
**Effort:** L per language (typescript-first).
|
||||
|
||||
**Depends on / blocked by:** Nothing — additive on the Layer 5 edge schema.
|
||||
|
||||
### Cross-file edge resolution (Layer 5 precision upgrade)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Today every call edge lands unresolved in `code_edges_symbol` with to_symbol_qualified = bare callee name. Second-pass resolution: after all code files import, walk every `code_edges_symbol` row and try to resolve `to_symbol_qualified` via `symbol_name_qualified` join; if found within the same source, write a resolved row to `code_edges_chunk`.
|
||||
|
||||
**Why:** `getCallersOf("searchKeyword")` currently returns the Layer 6 ambiguity — every `searchKeyword` call site in any class. Receiver-type analysis lifts this.
|
||||
|
||||
**Effort:** L. Needs receiver-type inference; can ship per-language.
|
||||
|
||||
**Depends on / blocked by:** Nothing — UNION-on-read path keeps unresolved edges surfaced even without this.
|
||||
|
||||
## Completed
|
||||
|
||||
### ~~Checks 5 + 6 for check-resolvable~~
|
||||
**Completed:** v0.19.0 (2026-04-22)
|
||||
|
||||
Both checks shipped as real implementations, not just filed issues:
|
||||
- **Check 5 (trigger routing eval):** `src/core/routing-eval.ts` + `gbrain routing-eval` CLI. Structural layer runs in `check-resolvable` by default; `--llm` opts into LLM tie-break. Fixtures live at `skills/<name>/routing-eval.jsonl`.
|
||||
- **Check 6 (brain filing):** `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json`. New `writes_pages:` + `writes_to:` frontmatter. Warning-only in v0.19, error in v0.20.
|
||||
|
||||
`DEFERRED[]` in `src/commands/check-resolvable.ts` is now empty — v0.19 shipped both deferred checks as working code paths, not as issue URLs. The export stays in place for future deferred checks.
|
||||
|
||||
### ~~BrainBench Cats 5/6/8/9/11 — shipped to sibling repo~~
|
||||
**Completed:** v0.20.0 (2026-04-23)
|
||||
|
||||
All five previously-deferred BrainBench categories shipped as working runners
|
||||
in the sibling repo [github.com/garrytan/gbrain-evals](https://github.com/garrytan/gbrain-evals):
|
||||
|
||||
- **Cat 5 Provenance** — `eval/runner/cat5-provenance.ts` with dedicated `classify_claim` tool (3-way label: `supported | unsupported | over-generalized`)
|
||||
- **Cat 6 Prose-scale auto-link precision** — `eval/runner/cat6-prose-scale.ts` (baseline-only) + `eval/runner/adversarial-injections.ts` (6 injection kinds)
|
||||
- **Cat 8 Skill Compliance** — `eval/runner/cat8-skill-compliance.ts` (brain-first / back-link / citation-format / tier-escalation, deterministic from tool-bridge trace)
|
||||
- **Cat 9 End-to-End Workflows** — `eval/runner/cat9-workflows.ts` (rubric-graded)
|
||||
- **Cat 11 Multi-modal Ingestion** — `eval/runner/cat11-multimodal.ts` (PDF/audio/HTML)
|
||||
|
||||
Plus supporting infrastructure: agent adapter (Sonnet + 12 read + 3 dry_run tools),
|
||||
structured-evidence Haiku judge contract, PublicPage/PublicQuery sealed qrels,
|
||||
6-artifact flight-recorder, 6 portable JSON schemas for v1→v2 driver swap.
|
||||
|
||||
Scope pivot: originally planned for in-tree v1.1 delta; mid-PR pivoted to extract
|
||||
the entire eval harness so gbrain users don't download the ~5MB corpus at install
|
||||
time. BrainBench is now a public sibling benchmark; gbrain ships clean.
|
||||
|
||||
### ~~v0.10.5: inferLinkType residuals (works_at, advises)~~
|
||||
**Completed:** v0.20.0 (2026-04-23)
|
||||
|
||||
`src/core/link-extraction.ts` — WORKS_AT_RE and ADVISES_RE expanded with
|
||||
rank-prefixed engineer patterns ("senior/staff/principal/lead engineer at"),
|
||||
discipline-prefixed ("backend/frontend/ML/security engineer at"), broader role
|
||||
verbs ("manages engineering at", "running product at", "heads up X at"),
|
||||
possessive time ("his/her/their time at"), role-noun forms ("tenure as",
|
||||
"stint as", "role at"), advisory capacity phrasings, "as an advisor" forms,
|
||||
and qualifier-specific advisors. New EMPLOYEE_ROLE_RE prior fires for
|
||||
self-identified employees at the page level, biasing outbound company refs
|
||||
toward works_at when per-edge verbs are absent. Precedence: investor > advisor
|
||||
> employee. Existing tests in `test/link-extraction.test.ts` cover the new
|
||||
patterns.
|
||||
|
||||
## P1 (BrainBench v1.1 — remaining categories)
|
||||
|
||||
Cats 5/6/8/9/11 shipped to the sibling repo in v0.20.0 — see the Completed
|
||||
section above. One remaining scope item:
|
||||
|
||||
### BrainBench Cat 1+2 at full scale
|
||||
**What:** Existing benchmark-search-quality.ts (29 pages, 20 queries) and benchmark-graph-quality.ts (80 pages, 5 queries) currently pass at small scale. v1.1 extends both to 2-3K rich-prose pages generated via Opus to surface scale-dependent failures (tied keyword clusters, hub-node fan-out, prose-noise extraction precision).
|
||||
|
||||
**Why deferred from PR #188:** Needs ~$200-300 of Opus tokens for the rich corpus. The 80-page version already proves algorithmic correctness; scale-up proves it survives real-world load.
|
||||
|
||||
**Threshold:** maintain v1 metrics at 30x scale.
|
||||
|
||||
### ~~v0.10.4: inferLinkType prose precision fix~~
|
||||
**Shipped in PR #188.** BrainBench Cat 2 rich-corpus type accuracy went from
|
||||
70.7% → 88.5%. Fix: widened verb regexes (added "led the seed/Series A",
|
||||
"early investor", "invests in", "portfolio company", etc.), tightened
|
||||
ADVISES_RE to require explicit advisor rooting (generic "board member"
|
||||
matches investors too), widened context window 80→240 chars, added
|
||||
person-page role prior (partner-bio language → invested_in for outbound
|
||||
company refs only). Per-type after fix: invested_in 91.7% (was 0%),
|
||||
mentions 100%, attended 100%. works_at 58% and advises 41% are next
|
||||
iteration's residuals.
|
||||
|
||||
### v0.10.4: gbrain alias resolution feature (driven by Cat 3)
|
||||
**What:** Add an alias table to gbrain so "Sarah Chen" / "S. Chen" / "@schen" / "sarah.chen@example.com" resolve to one canonical entity. Schema: `aliases (id, slug, alias_text)` with a unique index. Search blends alias matches into hybrid scoring.
|
||||
|
||||
**Why:** BrainBench Cat 3 measured 31% recall on undocumented aliases — that's the v0.10.x baseline. With alias table, should jump to 80%+.
|
||||
|
||||
**Depends on:** Cat 3 baseline (shipped in PR #188).
|
||||
|
||||
## P1
|
||||
|
||||
### Minions shell jobs — Phase 2 scheduling (deferred from v0.13.0)
|
||||
|
||||
**What:** `minion_schedules` table + autopilot-cycle scanner that submits due shell jobs.
|
||||
|
||||
**Why:** v0.13.0 moves shell scripts to Minions but still leaves scheduling in the host crontab. Your OpenClaw's `scripts/service-manager.sh` + crontab is the only piece left on the host side. A DB-driven scheduler would mean a single `gbrain autopilot --install` replaces the host crontab entirely, scheduling is visible via `gbrain jobs list --scheduled`, and downtime-on-one-machine tolerance improves (schedule is shared DB state, not per-host crontab).
|
||||
|
||||
**Pros:** Canonical host-agnostic deployment. No more host-specific crontab.
|
||||
|
||||
**Cons:** Cross-engine migration complexity (new table on both PGLite + Postgres). Autopilot-cycle scanner needs to handle missed-schedule semantics (fire-once-on-startup or skip-if-past-now), and this is where every other cron-like system has historically accrued bugs.
|
||||
|
||||
**Depends on:** v0.13.0 shell jobs shipped. ✅
|
||||
|
||||
### `gbrain crontab-to-minions <file>` migration helper (deferred from v0.13.0)
|
||||
|
||||
**What:** Parse an existing crontab file, emit a proposed rewrite using `gbrain jobs submit shell ...` for each deterministic entry, keep LLM-requiring entries as-is.
|
||||
|
||||
**Why:** Hand-rewriting ~14 OpenClaw cron entries is error-prone and one-shot. A helper would make the migration reversible and auditable (diff the before/after crontab, dry-run the first N, commit).
|
||||
|
||||
**Pros:** Removes the "rewrite 14 lines by hand" tax every agent operator pays on adoption.
|
||||
|
||||
**Cons:** Crontab parsing is historically fiddly (5-field vs 6-field, `@hourly` aliases, Vixie extensions, env vars in crontab). Could misrewrite entries with shell substitution.
|
||||
|
||||
**Depends on:** v0.13.0 shell jobs shipped. ✅
|
||||
|
||||
### Batch the DB-source extract read path (deferred from v0.12.1)
|
||||
**What:** `extractLinksFromDB` and `extractTimelineFromDB` at `src/commands/extract.ts:447, 504` issue one `engine.getPage(slug)` per slug after `engine.getAllSlugs()`. On a 47K-page brain that's still 47K serial reads over the Supabase pooler.
|
||||
|
||||
**Why:** v0.12.1 fixed the write-side N+1 with batched INSERTs (~100x fewer round-trips). The read side still does serial `getPage()` calls — each fetches `compiled_truth + timeline + frontmatter` (tens of KB per page). On a 47K-page Supabase brain that's ~10-20 minutes of read latency before any work happens. The v0.12.0 orchestrator's backfill uses `--source db`, so this stays slow until fixed.
|
||||
|
||||
**Pros:** Mirrors the write-side fix on the read path. Combined with batched writes, full re-extract on a 47K-page brain should drop from "minutes" to "seconds" end-to-end. Eliminates the implicit `listPages-pagination-mutation` learning risk by giving you a snapshot read.
|
||||
|
||||
**Cons:** New engine method (`getPagesBatch(slugs: string[]) → Promise<Page[]>` or a streaming cursor) needs to land on both PGLite and Postgres. Memory budget — a 47K-page brain with ~30KB/page is ~1.4GB if loaded all at once; needs chunked iteration (e.g., 500 slugs/query, stream-process).
|
||||
|
||||
**Context:** Codex's plan-time review and the testing/performance specialists at ship time both flagged this. Filed during v0.12.1 to ship the bug fix without scope creep. Approach: add `getPagesBatch(slugs)` returning chunked results, then update the 4 DB-source extract paths to consume it.
|
||||
|
||||
**Depends on:** v0.12.1 ships first.
|
||||
|
||||
### Batch embedding queue across files
|
||||
**What:** Shared embedding queue that collects chunks from all parallel import workers and flushes to OpenAI in batches of 100, instead of each worker batching independently.
|
||||
|
||||
@@ -278,7 +54,7 @@ iteration's residuals.
|
||||
|
||||
**Cons:** Requires adding `sender_id` or `access_tier` to `OperationContext`. Each mutating operation needs a permission check. Medium implementation effort.
|
||||
|
||||
**Context:** From CEO review + Codex outside voice (2026-04-13). Prompt-layer access control works in practice (same model as Garry's OpenClaw) but is not sufficient for remote MCP where direct tool calls bypass the agent's prompt.
|
||||
**Context:** From CEO review + Codex outside voice (2026-04-13). Prompt-layer access control works in practice (same model as Wintermute) but is not sufficient for remote MCP where direct tool calls bypass the agent's prompt.
|
||||
|
||||
**Depends on:** v0.10.0 GStackBrain skill layer (shipped).
|
||||
|
||||
@@ -287,142 +63,8 @@ iteration's residuals.
|
||||
### ~~Constrained health_check DSL for third-party recipes~~
|
||||
**Completed:** v0.9.3 (2026-04-12). Typed DSL with 4 check types (`http`, `env_exists`, `command`, `any_of`). All 7 first-party recipes migrated. String health checks accepted with deprecation warning + metachar validation for non-embedded recipes.
|
||||
|
||||
## P1 (new from v0.11.0 — Minions)
|
||||
|
||||
### Per-queue rate limiting for Minions
|
||||
**What:** Token-bucket rate limiting per queue via a new `minion_rate_limits` table (queue, capacity, refill_rate, tokens, updated_at), with acquire/release in `claim()`.
|
||||
|
||||
**Why:** The #1 daily OpenClaw pain is spawn storms hitting OpenAI/Anthropic rate limits. `max_children` caps fan-out per parent, but a queue with 50 ready jobs will still slam the API. Every Minions consumer currently reinvents token-bucket in user code.
|
||||
|
||||
**Pros:** First-class rate limiting means no consumer has to roll their own. Composes with `max_children` (which is per-parent) to give two orthogonal throttles.
|
||||
|
||||
**Cons:** Adds a write hotspot on the rate-limit row. Mitigate by keeping it a simple `UPDATE ... WHERE tokens > 0 RETURNING` that fails fast and puts the claim back in the pool.
|
||||
|
||||
**Effort:** ~2 hours. Deferred from v0.11.0 to keep the parity PR at a reviewable size.
|
||||
|
||||
**Depends on:** Minions (shipped in v0.11.0).
|
||||
|
||||
### Minions repeat/cron scheduler
|
||||
**What:** BullMQ-style repeatable jobs. `queue.add(name, data, { repeat: { cron: '0 * * * *' } })`.
|
||||
|
||||
**Why:** Idempotency keys (shipped in v0.11.0) are the foundation. Consumers currently use launchd/cron to fire `gbrain jobs submit`, but a native scheduler inside the worker would be cleaner and portable across deployments.
|
||||
|
||||
**Pros:** One mental model for both immediate and scheduled work. Idempotency prevents double-fire.
|
||||
|
||||
**Cons:** Every cron library has edge cases (DST, missed intervals on worker restart). Use a battle-tested parser.
|
||||
|
||||
**Effort:** ~1 day.
|
||||
|
||||
**Depends on:** Idempotency keys (shipped in v0.11.0).
|
||||
|
||||
### Minions worker event emitter
|
||||
**What:** `worker.on('job:completed', handler)` / `worker.on('job:failed', ...)` instead of polling.
|
||||
|
||||
**Why:** Consumers currently poll `getJob(id)` to watch state changes. An event API is the ergonomic BullMQ has and Minions doesn't.
|
||||
|
||||
**Effort:** ~4 hours.
|
||||
|
||||
### `waitForChildren(parent_id, n)` / `collectResults(parent_id)` helpers
|
||||
**What:** Convenience wrappers over `readChildCompletions` for common fan-in patterns.
|
||||
|
||||
**Why:** The `child_done` inbox primitive shipped in v0.11.0. Now add the ergonomic API on top so orchestrators don't have to write the polling loop.
|
||||
|
||||
**Effort:** ~2 hours.
|
||||
|
||||
**Depends on:** `child_done` inbox primitive (shipped in v0.11.0).
|
||||
|
||||
## P2
|
||||
|
||||
### Orchestrator + runner double-write to migrations ledger (deferred from v0.18.2 codex review)
|
||||
|
||||
**What:** `src/commands/migrations/v0_18_0.ts:200-208` appends an entry to `~/.gbrain/migrations/completed.jsonl` while `src/commands/apply-migrations.ts:374-386` also appends one for the same orchestrator run. The dedupe guard in `src/core/preferences.ts:120-131` only suppresses duplicate `complete` entries, not `partial` entries. Result: distorted wedge counting (3-consecutive-partials-triggers-wedge logic sees 6 partials when it should see 3).
|
||||
|
||||
**Why:** Codex plan-review caught this during PR #356 while verifying the two-migration-systems resume boundary. Not blocking v0.18.2 shipping because it only affects the wedge detection threshold, not correctness of the migration itself.
|
||||
|
||||
**Fix:** Pick one writer (prefer `apply-migrations.ts` runner as the single source of truth, remove the orchestrator-side append). Fold into `feat/agent-migration-devex` follow-up PR, which already touches both files for the migrate-command consolidation work.
|
||||
|
||||
**Depends on:** v0.18.2 shipped. ✅
|
||||
|
||||
### 22K-page resync is 30+ minutes on large brains (deferred from v0.18.2 codex review)
|
||||
|
||||
**What:** When a schema migration requires data backfill (e.g., computing `page_id` from `page_slug` across all `files` rows), `src/commands/sync.ts:248-251, 311-337` iterates per-file. None of v0.18.2's hardening work shrinks this path. On a 22K-page brain the resync takes 30+ minutes; at 500K pages it would be several hours.
|
||||
|
||||
**Why:** Codex explicitly called out that none of PR #356 or the two follow-up PRs addresses the resync execution model. This is a separate performance-design problem.
|
||||
|
||||
**Options to explore:**
|
||||
- (a) Parallel page import via worker pool (Minions-based).
|
||||
- (b) Bulk COPY-based import replacing the per-file INSERT.
|
||||
- (c) Incremental resync that only rewrites changed rows (needs content hash or updated_at gating).
|
||||
|
||||
**Priority:** P2 now, upgrade to P1 if another heavy migration ships that needs backfill at this scale.
|
||||
|
||||
**Depends on:** v0.18.2 shipped. ✅
|
||||
|
||||
### Minions: `gbrain jobs stats --orphaned` (deferred from v0.13.0)
|
||||
|
||||
**What:** New CLI flag / output column surfacing jobs that are waiting with no registered handler on any live worker.
|
||||
|
||||
**Why:** v0.13.0 adds shell jobs that require `GBRAIN_ALLOW_SHELL_JOBS=1` on the worker. If an operator submits a shell job but no worker with the flag is running, the row sits in `waiting` silently. The CLI's starvation warning + docs help at submit time; this TODO surfaces the problem at operational-check time.
|
||||
|
||||
**Pros:** Closes the "did my cron actually run" ambiguity for multi-machine deployments.
|
||||
|
||||
**Cons:** Knowing "no worker has this handler registered" requires worker heartbeat tracking, which Minions doesn't have yet (it's stateless at DB level beyond `lock_token`). Could be approximated by "no jobs of this name have completed in last N minutes AND count of waiting is > 0."
|
||||
|
||||
**Depends on:** v0.13.0 shell jobs shipped. ✅
|
||||
|
||||
### Minions: AbortReason plumbing on MinionJobContext (deferred from v0.13.0)
|
||||
|
||||
**What:** Handlers today can't distinguish whether `ctx.signal.aborted` fired due to timeout, cancel, or lock-loss. v0.13.0 derives this at worker-catch-time from `abort.signal.reason`, but the handler can't see it directly. Expose `ctx.abortReason?: 'timeout' | 'cancel' | 'lock-lost' | 'shutdown'` on the context.
|
||||
|
||||
**Why:** Shell handler's kill-sequence today can't decide "retry this" (lock-lost) vs "don't retry, user cancelled" (cancel) — they look the same. A typed AbortReason lets handlers make that decision for themselves.
|
||||
|
||||
**Pros:** Handlers get richer signals.
|
||||
|
||||
**Cons:** Small surface-area addition to the handler API. Not strictly required since the worker already makes the retry/dead decision for them.
|
||||
|
||||
**Depends on:** v0.13.0 shell jobs shipped. ✅
|
||||
|
||||
### Minions: blocking-mode audit log for true forensic integrity (deferred from v0.13.0)
|
||||
|
||||
**What:** Opt-in mode for `shell-audit` where `appendFileSync` failures DO block submission instead of logging-and-continuing.
|
||||
|
||||
**Why:** v0.13.0 ships the audit log in best-effort mode, which means a disk-full attacker can silently disable the forensic trail. Acceptable for v0.13.0 because the primary use is operational ("what did this cron do last Tuesday"), not security forensics. Operators who want fail-closed semantics should have a flag.
|
||||
|
||||
**Pros:** Enables true forensic integrity for deployments that need it.
|
||||
|
||||
**Cons:** Fail-closed means a transient disk issue blocks shell submissions, which can be worse than a missing log line for most operators. Opt-in is the right shape but adds surface area.
|
||||
|
||||
**Depends on:** v0.13.0 shell jobs shipped. ✅
|
||||
|
||||
### Minions: configurable per-job output buffer sizes (deferred from v0.13.0)
|
||||
|
||||
**What:** Add `max_stdout_bytes` / `max_stderr_bytes` to ShellJobParams; override the 64KB/16KB defaults.
|
||||
|
||||
**Why:** 64KB/16KB covers typical OpenClaw scripts today but a verbose benchmark or a debug-dump script could need more.
|
||||
|
||||
**Depends on:** First shell-job author who actually needs it. Don't pre-build the flag.
|
||||
|
||||
### Security hardening follow-ups (deferred from security-wave-3)
|
||||
**What:** Close remaining security gaps identified during the v0.9.4 Codex outside-voice review that didn't make the wave's in-scope cut.
|
||||
|
||||
**Why:** Wave 3 closed 5 blockers + 4 mediums. These are the known residuals. Each is an independent hardening item that becomes trivial as Runtime MCP access control (P0 above) lands.
|
||||
|
||||
**Items (each a separate small task):**
|
||||
- **DNS rebinding protection for HTTP health_checks.** Current `isInternalUrl` validates the hostname string; DNS resolution happens later inside `fetch`. A malicious DNS server can return a public IP on first lookup and an internal IP on the actual request. Fix: resolve hostname via `dns.lookup` before fetch, pin the IP with a custom `http.Agent` `lookup` override, re-validate post-resolution. Alternative: use `ssrf-req-filter` library.
|
||||
- **Extended IPv6 private-range coverage.** Block `fc00::/7` (Unique Local Addresses), `fe80::/10` (link-local), `2002::/16` (6to4), `2001::/32` (Teredo), `::/128`. Current code covers `::1`, `::`, and IPv4-mapped (`::ffff:*`) via hex hextet parsing.
|
||||
- **IPv4 shorthand parsing.** `127.1` (legacy 2-octet form = 127.0.0.1), `127.0.1` (3-octet), mixed-radix with trailing dots. Current code handles hex/octal/decimal integer-form IPs but not these shorthand variants.
|
||||
- **Broader operation-layer limit caps.** `traverse_graph` `depth` param, plus `get_chunks`, `get_links`, `get_backlinks`, `get_timeline`, `get_versions`, `get_raw_data`, `resolve_slugs` — all currently accept unbounded `limit`/`depth`. Wave 3 only clamped `list_pages` and `get_ingest_log`.
|
||||
- **`sync_brain` repo path validation.** The `repo` parameter accepts an arbitrary filesystem path. Same threat model as `file_upload` before wave 3. Add `validateUploadPath` (strict) for remote callers.
|
||||
- **`file_upload` size limit.** `readFileSync` loads the entire file into memory. Trivial memory-DoS from MCP. Add ~100MB cap (matches CLI's TUS routing threshold) and stream for larger files.
|
||||
- **`file_upload` regular-file check.** Reject directories, devices, FIFOs, Unix sockets via `stat.isFile()` before `readFileSync`.
|
||||
- **Explicit confinement root (H2).** `file_upload` strict mode currently uses `process.cwd()`. Move to `ctx.config.upload_root` (or derive from where the brain's schema lives) so MCP server cwd can't be the wrong anchor.
|
||||
|
||||
**Effort:** M total (human: ~1 day / CC: ~1-2 hrs).
|
||||
|
||||
**Priority:** P2 — deferred consciously. Wave 3 closed the easily-exploitable paths. These are the defense-in-depth follow-ups.
|
||||
|
||||
**Depends on:** Security wave 3 shipped. None are blockers for Runtime MCP access control, but all three security workstreams (this, that P0, and the health-check DSL) converge on the same zero-trust MCP goal.
|
||||
|
||||
### Community recipe submission (`gbrain integrations submit`)
|
||||
**What:** Package a user's custom integration recipe as a PR to the GBrain repo. Validates frontmatter, checks constrained DSL health_checks, creates PR with template.
|
||||
|
||||
@@ -462,105 +104,7 @@ iteration's residuals.
|
||||
|
||||
**Depends on:** v0.8.0 (Edge Function removal shipped).
|
||||
|
||||
## P2 (knowledge graph follow-ups)
|
||||
|
||||
### Auto-link skipped writes generate redundant SQL
|
||||
**What:** When `gbrain put` is called with identical content (status=skipped), runAutoLink still does a full getLinks + per-candidate addLink loop. On N identical writes of a 50-entity page that's 50N round trips.
|
||||
|
||||
**Why:** Defensive reconciliation catches drift between page text and links table, but on truly idempotent writes it's wasted work.
|
||||
|
||||
**Pros:** Lower DB load on cron-style re-syncs. Keeps put_page latency tight under bulk MCP usage.
|
||||
|
||||
**Cons:** Need to track whether links could have drifted independent of content (e.g., a target page was deleted). Conservative approach: only skip auto-link reconciliation if status=skipped AND existing links match desired set (which still requires the getLinks call).
|
||||
|
||||
**Context:** Caught in /ship adversarial review (2026-04-18). Acceptable for v0.10.3 because auto-link runs in a transaction with row locks, so amplification cost is bounded.
|
||||
|
||||
**Effort estimate:** S (CC: ~10min)
|
||||
**Priority:** P2
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Audit `extract --source db` against auto_link config flag
|
||||
**What:** `gbrain extract links --source db` writes to the same `links` table that `auto_link=false` is supposed to opt out of. The two are conceptually distinct (extract is intentional batch op, auto_link is implicit on write), but a user who turned off auto_link expecting "no automatic link writes" might be surprised.
|
||||
|
||||
**Why:** Either the behavior should match (extract checks auto_link too) or the docs should explicitly state extract is a superset.
|
||||
|
||||
**Pros:** Less surprise for users who treat auto_link as a master switch.
|
||||
|
||||
**Cons:** Some users want extract to work even when auto_link is off (e.g. one-time backfill).
|
||||
|
||||
**Context:** Caught in /ship adversarial review (2026-04-18). Documenting for now.
|
||||
|
||||
**Effort estimate:** S (CC: ~10min for docs OR ~20min for code change).
|
||||
**Priority:** P2
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Doctor --fix polish from v0.14.1 adversarial review
|
||||
**What:** Six deferred findings from v0.14.1 ship-time adversarial review on `src/core/dry-fix.ts`:
|
||||
1. **TOCTOU between read and write.** `attemptFix` reads once, writes later. Concurrent editor saves silently overwritten. Fix: re-read immediately before write and compare snapshot, or `O_EXCL` tempfile + rename.
|
||||
2. **Fence detection misses 4-backtick and `~~~` fences.** `isInsideCodeFence` only catches `^```$`. CommonMark-legal alternates slip through.
|
||||
3. **`expandBullet` walk-up is dead code.** Loop breaks immediately because `baseIndent` matches the current line. Remove or make it actually walk up.
|
||||
4. **Multi-match guard too strict.** Skills with the pattern in a table-of-contents AND body get `ambiguous_multiple_matches` forever. Consider: fix first, re-scan, repeat until fixed-point.
|
||||
5. **Subprocess spam.** `getWorkingTreeStatus` spawns `git status` N×M times per `doctor --fix`. Cache per-skill per-invocation.
|
||||
6. **`doctor --fix --json` swallows the auto-fix report.** `printAutoFixReport` returns early on `jsonOutput`; agents don't see fix outcomes. Emit `auto_fix` as a top-level key.
|
||||
|
||||
**Why:** None are ship-blockers; all surfaced during v0.14.1 Codex adversarial review. Bundle into one follow-up PR.
|
||||
|
||||
**Pros:** Closes the adversarial findings loop. Better correctness under concurrent edits and JSON-consumer agents.
|
||||
|
||||
**Cons:** Concurrent-edit test is finicky.
|
||||
|
||||
**Context:** v0.14.1 shipped with the 4 critical fixes (shell-injection via execFileSync, no-git-backup detection, EOF newline preservation, proximity-window consistency). These six are the deferred remainder.
|
||||
|
||||
**Effort estimate:** M (CC: ~45min for all six + tests).
|
||||
**Priority:** P2
|
||||
**Depends on:** Nothing.
|
||||
|
||||
## Completed
|
||||
|
||||
### Implement AWS Signature V4 for S3 storage backend
|
||||
**Completed:** v0.6.0 (2026-04-10) — replaced with @aws-sdk/client-s3 for proper SigV4 signing.
|
||||
|
||||
### Caller-opt-in retry for `executeRaw` (D3 follow-up from v0.22.1)
|
||||
**What:** Add `PostgresEngine.executeRawIdempotent(sql, params)` (or a `{retry: true}` parameter flag on `executeRaw`) so callers explicitly opt into auto-retry for statements they know are idempotent. Audit existing call sites and migrate the read-only ones (search, page fetches, etc.) to the new method.
|
||||
|
||||
**Why:** Closes the gap left by D3's drop-the-wrapper decision in v0.22.1. The original #406 wrapped `executeRaw` in a regex-gated retry that was unsound for writable CTEs and side-effecting SELECTs. Recovery moved up to the supervisor watchdog, but per-call recovery for reads (the bulk of `executeRaw` traffic from MCP, search, page fetches) is gone. A caller-opt-in flag puts the idempotency decision where it belongs (at the call site, with full statement context).
|
||||
|
||||
**Pros:** Restores per-call auto-recovery for reads without the phantom-write risk on mutations. Explicit > clever: each call site declares its own idempotency posture. Future caller-added mutations get safe-by-default behavior.
|
||||
|
||||
**Cons:** Touches every existing `executeRaw` call site (~25). Requires careful audit — accidentally tagging a mutation as idempotent re-introduces the phantom-write bug.
|
||||
|
||||
**Context:** Codex F3 demonstrated that `READ_ONLY_PREFIX = /^(\s|--.*\n)*(SELECT|WITH)\b/i` is unsound — `WITH x AS (UPDATE … RETURNING …) SELECT …` matches the prefix but updates a row; `SELECT pg_advisory_xact_lock(...)` is a SELECT with side effects. The plan-eng-review wrap-up in `~/.claude/plans/system-instruction-you-are-working-tender-horizon.md` has the full discussion.
|
||||
|
||||
**Effort estimate:** M (human: ~1 day / CC: ~30 min including call-site audit).
|
||||
**Priority:** P2 — current behavior (no retry, supervisor recovers within ~3 min) is acceptable but per-call recovery is a real ergonomic win.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Replace `walkMarkdownFiles` with `engine.getAllSlugs()` in `extractForSlugs` (F1 follow-up from v0.22.1)
|
||||
**What:** The cycle path's `extractForSlugs()` at `src/commands/extract.ts:455` still does a `walkMarkdownFiles(brainDir)` to build the `allSlugs` set for link resolution. On a 54K-page brain that's a single `readdir` traversal (~hundreds of ms — acceptable, dominated by the file-content-read elimination from #417). But `engine.getAllSlugs()` exists at `extract.ts:728` and produces the same set via a single SQL query (~tens of ms).
|
||||
|
||||
**Why:** Eliminates the residual directory walk on every cycle. Codex F1 noted that the v0.22.1 plan's "cycle never re-walks the whole tree again" claim was overstated — it stops READING file contents but still walks the directory. This TODO closes that gap honestly.
|
||||
|
||||
**Pros:** Cycle becomes O(slugs sync touched), not O(total brain size). No more readdir on a growing brain. ~5 LOC change.
|
||||
|
||||
**Cons:** Crosses an FS-vs-DB consistency boundary in the FS-source extract path. Edge case: a file deleted from disk but still in DB. Currently `extractForSlugs` skips with `if (!existsSync(fullPath)) continue` — unchanged. But if a markdown file references a slug whose page exists in DB but file was deleted, the link would resolve via DB but the original extractor caught it. Needs a careful test for this case.
|
||||
|
||||
**Context:** Codex plan-review during v0.22.1 wrap, verified at `extract.ts:455-456`. The plan-eng-review session captured the rationale.
|
||||
|
||||
**Effort estimate:** S (human: ~2 hr / CC: ~10 min including the consistency-edge-case test).
|
||||
**Priority:** P3 — pure perf, no correctness gap.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### `err.code`-based connection-error matching in `postgres-engine.ts` (B1 follow-up from v0.22.1)
|
||||
**What:** The CONNECTION_ERROR_PATTERNS array (~12 strings: `ECONNREFUSED`, `connection terminated`, `password authentication failed`, etc.) matched against `err.message` and `err.code`. Replace with structured matching against `err.code` only, using postgres.js's typed error classes (`PostgresError` with structured codes).
|
||||
|
||||
**Why:** String matching against error messages breaks on library upgrades (postgres.js could change its error message phrasing without bumping major). Code matching is durable. The Layer 1 cleanup follows: gbrain itself doesn't define connection-error codes; it should defer to postgres.js's classification.
|
||||
|
||||
**Pros:** More durable across library updates. Less code (drop the 12-string array). Follows the typed-errors pattern v0.21.0 introduced (`src/core/errors.ts`).
|
||||
|
||||
**Cons:** Requires verifying which `err.code` values postgres.js actually exposes for each connection-failure mode. May need fallback to message-substring matching for codes that postgres.js doesn't surface.
|
||||
|
||||
**Context:** Section 2/B1 from the v0.22.1 plan-eng-review. After D3 dropped the per-call retry, `isConnectionError` is no longer in the hot path — only the supervisor watchdog cares about classifying connection errors, and it currently catches *anything*. This TODO is a cleanup pass when someone next touches that surface.
|
||||
|
||||
**Effort estimate:** S (human: ~2 hr / CC: ~10 min).
|
||||
**Priority:** P3.
|
||||
**Depends on:** The above caller-opt-in retry (#1) is the natural co-lander since both touch the same error-classification surface.
|
||||
|
||||
@@ -7,26 +7,19 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@electric-sql/pglite": "^0.4.4",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
},
|
||||
"trustedDependencies": [
|
||||
"@electric-sql/pglite",
|
||||
],
|
||||
"packages": {
|
||||
"@anthropic-ai/sdk": ["@anthropic-ai/sdk@0.30.1", "", { "dependencies": { "@types/node": "^18.11.18", "@types/node-fetch": "^2.6.4", "abort-controller": "^3.0.0", "agentkeepalive": "^4.2.1", "form-data-encoder": "1.7.2", "formdata-node": "^4.3.2", "node-fetch": "^2.6.7" } }, "sha512-nuKvp7wOIz6BFei8WrTdhmSsx5mwnArYyJgh4+vYu3V4J0Ltb8Xm3odPm51n1aSI0XxNCrDl7O88cxCtUdAkaw=="],
|
||||
|
||||
@@ -110,9 +103,7 @@
|
||||
|
||||
"@aws/lambda-invoke-store": ["@aws/lambda-invoke-store@0.2.4", "", {}, "sha512-iY8yvjE0y651BixKNPgmv1WrQc+GZ142sb0z4gYnChDDY2YqI4P/jsSopBWrKfAt7LOJAkOXt7rC/hms+WclQQ=="],
|
||||
|
||||
"@dqbd/tiktoken": ["@dqbd/tiktoken@1.0.22", "", {}, "sha512-RYhO8xeHkMNX5Ixqf4M1Ve3siCYJY/dI0yLnlX4M4oIEDOvjMIQ+E+3OUpAaZcWTaMtQJzGcDAghYfllpx3i/w=="],
|
||||
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.4", "", {}, "sha512-g/6CWAJ4XOkObWCWAQ2IReZD8VvsDy3poRHSKvpRR2F96F8WJ3HVbjpso3gN7l0q6QPPgvxSSpl/qo5k8a7mkQ=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
|
||||
|
||||
@@ -458,14 +449,10 @@
|
||||
|
||||
"tr46": ["tr46@0.0.3", "", {}, "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw=="],
|
||||
|
||||
"tree-sitter-wasms": ["tree-sitter-wasms@0.1.13", "", { "dependencies": { "tree-sitter-wasms": "^0.1.11" } }, "sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ=="],
|
||||
|
||||
"tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
|
||||
|
||||
"type-is": ["type-is@2.0.1", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="],
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
|
||||
@@ -474,8 +461,6 @@
|
||||
|
||||
"web-streams-polyfill": ["web-streams-polyfill@4.0.0-beta.3", "", {}, "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug=="],
|
||||
|
||||
"web-tree-sitter": ["web-tree-sitter@0.22.6", "", {}, "sha512-hS87TH71Zd6mGAmYCvlgxeGDjqd9GTeqXNqTT+u0Gs51uIozNIaaq/kUAbV/Zf56jb2ZOyG8BxZs2GG9wbLi6Q=="],
|
||||
|
||||
"webidl-conversions": ["webidl-conversions@3.0.1", "", {}, "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ=="],
|
||||
|
||||
"whatwg-url": ["whatwg-url@5.0.0", "", { "dependencies": { "tr46": "~0.0.3", "webidl-conversions": "^3.0.0" } }, "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw=="],
|
||||
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
[test]
|
||||
# PGLite initialization can be slow under parallel test execution.
|
||||
# Default 5s is too short when many test files boot PGLite instances at once.
|
||||
# 60s is the empirical ceiling we observed before the first file's beforeAll
|
||||
# completed on a loaded machine.
|
||||
#
|
||||
# NOTE: this bunfig.toml `timeout` key is read by `bun test` but empirically
|
||||
# does NOT apply to beforeEach/afterEach hook timeouts under `bun run test`
|
||||
# chained behind `bun run typecheck`. The test script in package.json passes
|
||||
# `--timeout=60000` explicitly to cover both per-test and per-hook timeouts.
|
||||
# Leaving both in place as belt-and-suspenders.
|
||||
timeout = 60_000
|
||||
@@ -49,25 +49,11 @@ Running a production brain.
|
||||
| Guide | What It Covers |
|
||||
|-------|---------------|
|
||||
| [Reference Cron Schedule](guides/cron-schedule.md) | 20+ recurring jobs, quiet hours, dream cycle |
|
||||
| [Cron via Minions](../skills/conventions/cron-via-minions.md) | Why scheduled work runs as Minion jobs, not `agentTurn`. Auto-applied by v0.11.0 migration for built-in handlers; host-specific handlers use the plugin contract below. |
|
||||
| [Plugin Handlers](guides/plugin-handlers.md) | Registering host-specific Minion handlers via code (no data-file exec surface). |
|
||||
| [Minions fix](guides/minions-fix.md) | Repairing a half-migrated v0.11.0 install. |
|
||||
| [Shell jobs (v0.14.0+)](guides/minions-shell-jobs.md) | Move deterministic crons (API fetch, token refresh, scrape+write) off the LLM gateway. Zero tokens per fire, ~60% gateway headroom. Follow `skills/migrations/v0.14.0.md` for the adoption playbook. |
|
||||
| [Quiet Hours & Timezone](guides/quiet-hours.md) | Hold notifications during sleep, timezone-aware delivery |
|
||||
| [Executive Assistant Pattern](guides/executive-assistant.md) | Email triage, meeting prep, scheduling |
|
||||
| [Operational Disciplines](guides/operational-disciplines.md) | Signal detection, brain-first, sync-after-write, heartbeat, dream cycle |
|
||||
| [Skill Development Cycle](guides/skill-development.md) | 5-step cycle: concept, prototype, evaluate, codify, cron |
|
||||
|
||||
**Subagent routing (v0.11.0+):** agents that dispatch background work should route through
|
||||
`skills/conventions/subagent-routing.md` — it reads `~/.gbrain/preferences.json#minion_mode`
|
||||
and branches between native subagents and Minion jobs. The v0.11.0 migration auto-injects
|
||||
a marker into AGENTS.md pointing at this convention.
|
||||
|
||||
**Cron routing (v0.11.0+):** scheduled work goes through Minions, not OpenClaw's `agentTurn`.
|
||||
See `skills/conventions/cron-via-minions.md` for the rewrite pattern. The v0.11.0 migration
|
||||
auto-rewrites entries whose handler is a gbrain builtin; host-specific handlers (e.g.
|
||||
`ea-inbox-sweep`) need a code-level registration per `docs/guides/plugin-handlers.md`.
|
||||
|
||||
## Architecture
|
||||
|
||||
How to structure your system.
|
||||
|
||||
+1
-85
@@ -183,84 +183,6 @@ system context. See `skills/setup/SKILL.md` Phase D.
|
||||
|
||||
---
|
||||
|
||||
## 7. Knowledge Graph Wired
|
||||
|
||||
The v0.12.0 graph layer needs to be populated for existing brains. New writes are
|
||||
auto-linked, but historical pages need a one-time backfill.
|
||||
|
||||
**Command:**
|
||||
|
||||
```bash
|
||||
gbrain stats | grep -E 'links|timeline'
|
||||
```
|
||||
|
||||
**Expected:** Both `links` and `timeline_entries` are non-zero (assuming the brain
|
||||
has content with entity references and dated markdown).
|
||||
|
||||
**If it's zero on a brain with imported content:** Run the backfill.
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db --dry-run | head -5 # preview
|
||||
gbrain extract links --source db # commit
|
||||
gbrain extract timeline --source db
|
||||
gbrain stats # confirm > 0
|
||||
```
|
||||
|
||||
**Bonus check** — graph traversal works:
|
||||
|
||||
```bash
|
||||
# Pick any well-connected slug from your brain
|
||||
gbrain graph-query people/<some-person-slug> --depth 2
|
||||
```
|
||||
|
||||
**Expected:** Indented tree of typed edges (`--attended-->`, `--works_at-->`, etc.).
|
||||
If the slug has no inbound or outbound links, try a different one or run extract
|
||||
again.
|
||||
|
||||
**If extract finds nothing:** Your pages may not use entity-reference syntax. The
|
||||
extractor matches `[Name](people/slug)`, `[Name](../people/slug.md)`, and bare
|
||||
`people/slug` references. If your brain uses a different format, the auto-link
|
||||
heuristics won't find them — file an issue with a sample page.
|
||||
|
||||
---
|
||||
|
||||
## 8. JSONB Frontmatter Integrity (v0.12.2)
|
||||
|
||||
Postgres-backed brains created before v0.12.2 had double-encoded JSONB columns
|
||||
(`frontmatter->>'key'` returned NULL, GIN indexes were inert). `gbrain upgrade`
|
||||
runs `gbrain repair-jsonb` automatically via the `v0_12_2` orchestrator.
|
||||
Verify the repair succeeded.
|
||||
|
||||
**Command:**
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb --dry-run --json
|
||||
```
|
||||
|
||||
**Expected:** `totalRepaired: 0` across all 5 columns (`pages.frontmatter`,
|
||||
`raw_data.data`, `ingest_log.pages_updated`, `files.metadata`,
|
||||
`page_versions.frontmatter`). A zero count means every row is properly-typed
|
||||
JSON objects, not string-encoded JSON.
|
||||
|
||||
**If the count is > 0:** The repair didn't run or was interrupted. Re-run
|
||||
without `--dry-run`:
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb
|
||||
```
|
||||
|
||||
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
|
||||
|
||||
**Bonus check** — frontmatter-keyed queries actually resolve:
|
||||
|
||||
```bash
|
||||
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
|
||||
```
|
||||
|
||||
If this returns rows on a brain with person pages, the JSONB path is healthy.
|
||||
|
||||
---
|
||||
|
||||
## Quick Verification (all checks in one pass)
|
||||
|
||||
```bash
|
||||
@@ -281,13 +203,7 @@ gbrain embed --stale
|
||||
|
||||
# 6. Auto-update
|
||||
gbrain check-update --json
|
||||
|
||||
# 7. Knowledge graph populated (links + timeline > 0)
|
||||
gbrain stats | grep -E 'links|timeline'
|
||||
|
||||
# 8. JSONB integrity (v0.12.2 — Postgres only, PGLite always 0)
|
||||
gbrain repair-jsonb --dry-run --json
|
||||
```
|
||||
|
||||
If all eight return successfully, the installation is healthy. For the full
|
||||
If all six return successfully, the installation is healthy. For the full
|
||||
end-to-end sync test (4c), push a real change and verify it appears in search.
|
||||
|
||||
@@ -1,540 +0,0 @@
|
||||
# Upgrading Downstream Agents
|
||||
|
||||
GBrain ships skills in `skills/`. Downstream agents (custom OpenClaw deployments,
|
||||
agent forks of any kind) often **copy** these skill files into their own workspace and
|
||||
diverge over time — adding agent-specific phases, removing irrelevant ones, tightening
|
||||
language. Once that happens, gbrain can't push updates to those forks. The agent has
|
||||
to apply the diffs by hand.
|
||||
|
||||
This doc lists the exact diffs each downstream agent needs to apply when upgrading.
|
||||
Cross-reference against your fork's local skill files.
|
||||
|
||||
## Why this exists
|
||||
|
||||
`gbrain upgrade` ships the new binary. `gbrain post-upgrade [--execute --yes]` runs
|
||||
the schema migrations and backfills the data. But the **skill files themselves**
|
||||
that tell the agent how to behave — those are user-owned. If your `~/git/<your-agent>/workspace/skills/brain-ops/SKILL.md`
|
||||
says `# Based on gbrain v0.10.0` at the top, it doesn't know about v0.12.0 features.
|
||||
|
||||
The agent will keep manually calling `gbrain link` after every `put_page` (now redundant —
|
||||
auto-link does it), miss out on `gbrain graph-query` for relationship questions, and
|
||||
not know to backfill the structured timeline.
|
||||
|
||||
## How to apply
|
||||
|
||||
1. Identify your forked skill files. Typically at `~/git/<your-agent>/workspace/skills/` or wherever your agent's skill directory lives.
|
||||
2. For each skill listed below, find the matching phase/section in your fork.
|
||||
3. Apply the diff (paste the new block in the indicated location).
|
||||
4. Update the version banner at the top of your fork (`# Based on gbrain v0.12.0`).
|
||||
5. Verify: ask the agent to write a test page and confirm the response includes
|
||||
`auto_links: { created, removed, errors }`.
|
||||
|
||||
Total time: ~10 minutes for all four skills.
|
||||
|
||||
---
|
||||
|
||||
## 1. brain-ops/SKILL.md
|
||||
|
||||
**Where:** Insert a new `### Phase 2.5` section immediately after `### Phase 2: On Every Inbound Signal`.
|
||||
|
||||
**Why:** Phase 2.5 declares that auto-link runs automatically. Without this, the
|
||||
agent's mental model says it must call `gbrain link` after every `put_page`, which
|
||||
is now redundant and can cause double-add warnings.
|
||||
|
||||
```markdown
|
||||
### Phase 2.5: Structured Graph Updates (automatic)
|
||||
|
||||
Every `put_page` call automatically extracts entity references and writes them
|
||||
to the graph (`links` table) with inferred relationship types. Stale links
|
||||
(refs no longer in the page text) are removed in the same call. This is
|
||||
"auto-link" reconciliation.
|
||||
|
||||
- No manual `add_link` calls needed for ordinary page writes.
|
||||
- Inferred link types: `attended` (meeting -> person), `works_at`, `invested_in`,
|
||||
`founded`, `advises`, `source` (frontmatter), `mentions` (default).
|
||||
- The `put_page` MCP response includes `auto_links: { created, removed, errors }`
|
||||
so the agent can verify outcomes.
|
||||
- To disable: `gbrain config set auto_link false`. Default is on.
|
||||
- Timeline entries with specific dates still need explicit `gbrain timeline-add`
|
||||
(or batch via `gbrain extract timeline --source db`).
|
||||
```
|
||||
|
||||
**Also update the Iron Law section.** If your fork still says "Back-links maintained
|
||||
on every brain write (Iron Law)" without qualification, append:
|
||||
|
||||
```markdown
|
||||
**v0.12.0 update:** Auto-link satisfies the Iron Law for entity-reference links
|
||||
on every `put_page`. The agent's Iron Law obligation is now: include the
|
||||
entity reference in the page content (e.g., `[Alice](people/alice)`); auto-link
|
||||
handles the structured row. Manual `add_link` calls are reserved for
|
||||
relationships you can't express in markdown content.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. meeting-ingestion/SKILL.md
|
||||
|
||||
**Where:** Append to the end of `### Phase 3: Attendee enrichment`.
|
||||
|
||||
**Why:** Eliminates redundant `gbrain link` calls per attendee (auto-link handles them
|
||||
when the meeting page references attendees as `[Name](people/slug)`).
|
||||
|
||||
```markdown
|
||||
**Note (v0.12.0):** Once the meeting page is written via `gbrain put`, the
|
||||
auto-link post-hook automatically creates `attended` links from the meeting
|
||||
to each attendee whose page is referenced as `[Name](people/slug)`. You don't
|
||||
need to call `gbrain link` for attendees. You DO still need `gbrain timeline-add`
|
||||
for dated events (auto-link only handles links, not timeline entries).
|
||||
```
|
||||
|
||||
**Where:** In `### Phase 4: Entity propagation`, the line "Back-link from entity page
|
||||
to meeting page" can be replaced with:
|
||||
|
||||
```markdown
|
||||
4. Entity references in the meeting page body auto-create the link via auto-link.
|
||||
For incoming references on the entity page (entity page → meeting page), edit
|
||||
the entity page to mention the meeting and `put_page` it — auto-link handles
|
||||
the rest.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. signal-detector/SKILL.md
|
||||
|
||||
**Where:** Append to the end of `### Phase 2: Entity Detection`.
|
||||
|
||||
**Why:** Same logic as brain-ops — eliminates manual `gbrain link` after writing
|
||||
originals/ideas pages that reference people or companies.
|
||||
|
||||
```markdown
|
||||
**Auto-link (v0.12.0):** When you write/update an originals or ideas page that
|
||||
references a person or company, the auto-link post-hook on `put_page`
|
||||
automatically creates the link from the new page to that entity. You don't
|
||||
need to call `gbrain link` manually. Timeline entries still need explicit calls.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. enrich/SKILL.md
|
||||
|
||||
**Where:** Replace `### Step 7: Cross-reference` with the v0.12.0 version.
|
||||
|
||||
**Why:** Step 7 used to be primarily about creating links between related entity
|
||||
pages. With auto-link, that's automatic. Step 7 is now about content updates,
|
||||
not link creation.
|
||||
|
||||
Old (delete):
|
||||
```markdown
|
||||
### Step 7: Cross-reference
|
||||
|
||||
- Update company pages from person enrichment (and vice versa)
|
||||
- Update related project/deal pages if relevant context surfaced
|
||||
- Check index files if the brain uses them
|
||||
- Add back-links manually via `gbrain link` for any new entity references
|
||||
```
|
||||
|
||||
New (paste):
|
||||
```markdown
|
||||
### Step 7: Cross-reference
|
||||
|
||||
- Update company pages from person enrichment (and vice versa)
|
||||
- Update related project/deal pages if relevant context surfaced
|
||||
- Check index files if the brain uses them
|
||||
|
||||
**Note (v0.12.0):** Links between brain pages are auto-created on every
|
||||
`put_page` call (auto-link post-hook). Step 7 focuses on content
|
||||
cross-references (updating related pages' compiled truth with new signal
|
||||
from this enrichment), not on creating links. Verify via the `auto_links`
|
||||
field in the put_page response (`{ created, removed, errors }`).
|
||||
Timeline entries still need explicit `gbrain timeline-add` calls.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## After all four diffs are applied
|
||||
|
||||
1. **Bump the version banner** at the top of each forked file:
|
||||
```
|
||||
# Based on gbrain v0.12.0 skills/<skill-name>, extended with <your-agent>-specific config
|
||||
```
|
||||
|
||||
2. **Run the v0.12.0 backfill** (this populates the graph for your existing brain):
|
||||
```bash
|
||||
gbrain post-upgrade
|
||||
```
|
||||
The v0.12.0 release wires post-upgrade to call `apply-migrations --yes`
|
||||
automatically, which runs the v0_12_0 orchestrator (schema → config check →
|
||||
`extract links --source db` → `extract timeline --source db` → verify).
|
||||
Idempotent; cheap when nothing is pending.
|
||||
|
||||
3. **Verify auto-link works:** ask the agent to write a test page that references
|
||||
`[Some Person](people/some-person)`. Confirm the put_page response includes
|
||||
`auto_links: { created: 1, removed: 0, errors: 0 }`.
|
||||
|
||||
4. **Verify graph traversal works:**
|
||||
```bash
|
||||
gbrain graph-query people/some-well-connected-person --depth 2
|
||||
```
|
||||
Should return an indented tree of typed edges.
|
||||
|
||||
---
|
||||
|
||||
## v0.12.2 hotfix (data-correctness, no skill edits)
|
||||
|
||||
v0.12.2 is a Postgres data-correctness hotfix. No forked skill files need to
|
||||
change — the skill contracts are unchanged. But you DO need to run the migration,
|
||||
and you should know about one behavior change in markdown parsing.
|
||||
|
||||
### 1. Run the migration (Postgres-backed brains)
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
The `v0_12_2` orchestrator runs `gbrain repair-jsonb` automatically. It rewrites
|
||||
rows where `jsonb_typeof = 'string'` across `pages.frontmatter`, `raw_data.data`,
|
||||
`ingest_log.pages_updated`, `files.metadata`, and `page_versions.frontmatter`.
|
||||
Idempotent, safe to re-run. PGLite brains no-op cleanly.
|
||||
|
||||
Verify after upgrade:
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb --dry-run --json # expect totalRepaired: 0
|
||||
```
|
||||
|
||||
### 2. Recover any truncated wiki articles
|
||||
|
||||
If your brain imported wiki-style markdown before v0.12.2, some pages were
|
||||
silently truncated (any standalone `---` in body content was treated as a
|
||||
timeline separator). Re-import from source:
|
||||
|
||||
```bash
|
||||
gbrain sync --full
|
||||
```
|
||||
|
||||
The new `splitBody` rebuilds `compiled_truth` correctly.
|
||||
|
||||
### 3. Know the splitBody contract going forward
|
||||
|
||||
`splitBody` now requires an explicit timeline sentinel. Recognized markers
|
||||
(priority order):
|
||||
|
||||
1. `<!-- timeline -->` (preferred — what `serializeMarkdown` emits)
|
||||
2. `--- timeline ---` (decorated separator)
|
||||
3. `---` directly before `## Timeline` or `## History` heading (backward-compat)
|
||||
|
||||
A bare `---` in body text is now a markdown horizontal rule, not a timeline
|
||||
separator. If your agent writes pages with a bare `---` delimiter, migrate to
|
||||
`<!-- timeline -->` — the `serializeMarkdown` helper already does this.
|
||||
|
||||
### 4. Wiki subtypes now auto-typed
|
||||
|
||||
`inferType` now auto-detects five additional directory patterns as their own
|
||||
page types (previously they all defaulted to `concept`):
|
||||
|
||||
| Path pattern | New type |
|
||||
|------------------------|----------------|
|
||||
| `/wiki/analysis/` | `analysis` |
|
||||
| `/wiki/guides/` | `guide` |
|
||||
| `/wiki/hardware/` | `hardware` |
|
||||
| `/wiki/architecture/` | `architecture` |
|
||||
| `/writing/` | `writing` |
|
||||
|
||||
If your skills or queries filter by `type=concept` and expect wiki content in
|
||||
that bucket, update them to include the new types.
|
||||
|
||||
---
|
||||
|
||||
## v0.13.0 — Frontmatter Relationship Indexing
|
||||
|
||||
**Verdict: no action required for most skills.** v0.13 projects YAML frontmatter fields into the graph as typed edges. The ingestion API is unchanged — keep calling `put_page` with frontmatter the way you do today; the graph auto-populates behind the scenes.
|
||||
|
||||
Three skills get an optional new phase if you want to consume the new `auto_links.unresolved` response field. Without this, unresolvable frontmatter names silently skip (same as v0.12 behavior).
|
||||
|
||||
### 1. meeting-ingestion/SKILL.md (optional)
|
||||
|
||||
**Where:** Add a new section after "Phase 3: Write Meeting Page".
|
||||
|
||||
```markdown
|
||||
### Phase 3.5: Check for unresolved attendees (v0.13+)
|
||||
|
||||
After `put_page`, inspect `response.auto_links.unresolved` — an array of frontmatter
|
||||
references that did not resolve to existing pages. For meetings, this usually means
|
||||
attendees you haven't created a person page for yet.
|
||||
|
||||
If `unresolved.length > 0`:
|
||||
- Option 1 (create pages now): trigger an enrichment pass to build the missing people pages.
|
||||
- Option 2 (defer): log the unresolved names to the enrichment queue for later.
|
||||
- Option 3 (accept the gap): the attendee edge will not be created until a page exists.
|
||||
Re-running `gbrain extract links --source db --include-frontmatter` after creating
|
||||
the page fills in the missing edges.
|
||||
```
|
||||
|
||||
### 2. enrich/SKILL.md (optional)
|
||||
|
||||
**Where:** Add to the enrichment trigger list.
|
||||
|
||||
```markdown
|
||||
### Drain unresolved frontmatter names (v0.13+)
|
||||
|
||||
If any `put_page` response includes `auto_links.unresolved` entries, the enrichment
|
||||
tier should pick up those (field, name) pairs and try to create the missing entity
|
||||
pages. Example flow:
|
||||
|
||||
1. signal-detector captures a meeting with `attendees: [Alice Known, Unknown Person]`
|
||||
2. put_page returns `auto_links.unresolved = [{field: 'attendees', name: 'Unknown Person'}]`
|
||||
3. enrichment tier consumes `Unknown Person` → web search → creates `people/unknown-person.md`
|
||||
4. The next put_page (or a backfill run) wires up the `attended` edge automatically
|
||||
```
|
||||
|
||||
### 3. idea-ingest/SKILL.md (optional)
|
||||
|
||||
**Where:** Same pattern as meeting-ingestion — check `auto_links.unresolved` after `put_page`, route names to enrichment.
|
||||
|
||||
### Unchanged skills (no diffs needed)
|
||||
|
||||
- **brain-ops/SKILL.md** — auto-link mechanics are internal; the write path stays the same.
|
||||
- **signal-detector/SKILL.md** — signal capture path unchanged.
|
||||
- **query/SKILL.md** — `traverse_graph` now returns richer results automatically.
|
||||
- **daily-task-manager/SKILL.md**, **briefing/SKILL.md**, **citation-fixer/SKILL.md**, **media-ingest/SKILL.md** — unchanged.
|
||||
|
||||
### New edge types you can filter in graph queries
|
||||
|
||||
v0.13 edges carry new `link_type` values. If your fork has graph-query skills that filter by type, these are now available:
|
||||
|
||||
- `works_at` (person → company) — from `company:`, `companies:`, or `key_people:`
|
||||
- `founded` (person → company) — from `founded:`
|
||||
- `invested_in` (investor → deal/company) — from `investors:` or `lead:`
|
||||
- `led_round` (lead → deal) — from `lead:`
|
||||
- `yc_partner` (partner → company) — from `partner:`
|
||||
- `attended` (person → meeting) — from `attendees:`
|
||||
- `discussed_in` (source → page) — from `sources:`
|
||||
- `source` (page → source) — from `source:`
|
||||
- `related_to` (page → target) — from `related:` or `see_also:`
|
||||
|
||||
### Migration timing
|
||||
|
||||
`gbrain upgrade` takes 2-5 min on a 46K-page brain (one-time). Runs out-of-process via `gbrain post-upgrade`. If your agent holds a DB connection during the upgrade, reconnect after; otherwise keep serving.
|
||||
|
||||
### Type normalization NOT in v0.13
|
||||
|
||||
Legacy rows with `link_type='attendee'` or `link_type='mention'` coexist with new `'attended'` / `'mentions'` rows. Your queries filtering on old type names keep working. A separate opt-in `gbrain normalize-types` command in v0.14 handles the rename.
|
||||
## v0.14.0 shell jobs (optional adoption, no skill edits)
|
||||
|
||||
Adds a `shell` job type to Minions so deterministic cron scripts (API fetch, token
|
||||
refresh, scrape + write) move off the LLM gateway. Zero tokens per fire. ~60%
|
||||
gateway CPU headroom at typical scale. Feature is **off by default**, existing
|
||||
installs keep running exactly as they did before. Nothing breaks.
|
||||
|
||||
To adopt, follow `skills/migrations/v0.14.0.md`. The short version:
|
||||
|
||||
1. Set `GBRAIN_ALLOW_SHELL_JOBS=1` on the worker process, then `gbrain jobs work`
|
||||
(Postgres). On PGLite, every crontab invocation uses `--follow` for inline
|
||||
execution; no persistent worker.
|
||||
2. Classify each of your host's cron entries: LLM-requiring (keep on gateway) vs
|
||||
deterministic (candidate for shell). Typical splits:
|
||||
- **Deterministic → shell:** `ycli-token-refresh`, `x-oauth2-refresh`,
|
||||
`x-garrytan-unified`, `calendar-sync-to-brain`, `github-pulse`,
|
||||
`frameio-scan`, `flight-tracker`, `x-raw-json-backfill`.
|
||||
- **LLM-requiring → stay:** `social-radar`, `content-ideas`, `adversary-vacuum`,
|
||||
`ea-inbox-sweep`, `morning-briefing`, `brain-maintenance`.
|
||||
3. For each deterministic cron, rewrite as:
|
||||
```cron
|
||||
3 13,16,19,22,1,4,7,10 * * * \
|
||||
gbrain jobs submit shell \
|
||||
--params '{"cmd":"node scripts/your-script.mjs","cwd":"/data/.openclaw/workspace"}' \
|
||||
--max-attempts 3 --timeout-ms 300000
|
||||
```
|
||||
4. Watch `gbrain jobs get <id>` for exit_code / stdout_tail / stderr_tail on each
|
||||
fire. Compare against pre-migration behavior before approving the next batch.
|
||||
|
||||
**No skill edits required.** The handler runs worker-side; skill files don't
|
||||
change. If your host exposed custom handlers via the plugin contract (v0.11.0),
|
||||
they still work the same way.
|
||||
|
||||
Iron rule: **never auto-rewrite the operator's crontab.** Every rewrite is
|
||||
per-cron, human-approved, with a diff. If you want automation later, the
|
||||
upcoming `gbrain crontab-to-minions <file>` helper is P1 in TODOS.
|
||||
|
||||
---
|
||||
|
||||
## v0.16.0: durable agent runtime
|
||||
|
||||
v0.15 ships `gbrain agent run` / `gbrain agent logs`, a new `subagent` handler
|
||||
type in Minions, and a plugin contract for host-repo subagent defs. None of the
|
||||
existing skills need surgery. The question for downstream agents is *how* to
|
||||
adopt the new runtime, not how to patch around a breaking change.
|
||||
|
||||
### 1. Run a worker with an Anthropic key
|
||||
|
||||
The subagent handlers (`subagent` and `subagent_aggregator`) are always
|
||||
registered on the worker. No separate opt-in flag — `ANTHROPIC_API_KEY` is
|
||||
the natural cost gate (no key, the SDK call fails on the first turn), and
|
||||
who-can-submit is already protected (`PROTECTED_JOB_NAMES` + trusted-submit:
|
||||
MCP callers get `permission_denied`; only `gbrain agent run` can insert
|
||||
these rows).
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY=sk-ant-... gbrain jobs work
|
||||
```
|
||||
|
||||
Worker startup prints:
|
||||
|
||||
```
|
||||
[minion worker] subagent handlers enabled
|
||||
```
|
||||
|
||||
### 2. Ship your subagents as a plugin (OpenClaw + similar)
|
||||
|
||||
Move your custom subagent definitions out of your gbrain fork and into your own
|
||||
repo as a plugin. Concretely:
|
||||
|
||||
```
|
||||
~/<your-agent>/gbrain-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
├── meeting-ingestion.md
|
||||
├── signal-detector.md
|
||||
└── daily-task-prep.md
|
||||
```
|
||||
|
||||
`gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "your-openclaw",
|
||||
"version": "2026.4.20",
|
||||
"plugin_version": "gbrain-plugin-v1"
|
||||
}
|
||||
```
|
||||
|
||||
Each `subagents/*.md` is a plain-text agent definition — YAML frontmatter +
|
||||
body-as-system-prompt. Recognized frontmatter fields: `name`, `model`,
|
||||
`max_turns`, `allowed_tools` (must subset the derived brain-tool registry).
|
||||
|
||||
Turn it on:
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="$HOME/<your-agent>/gbrain-plugin"
|
||||
```
|
||||
|
||||
Worker startup prints `[plugin-loader] loaded '<name>' v<ver> (N subagents)`
|
||||
per plugin; any rejection (bad manifest, unknown tool in `allowed_tools`,
|
||||
version mismatch) shows up as a loud warning at startup, not a silent dispatch-
|
||||
time failure. See `docs/guides/plugin-authors.md` for the full contract.
|
||||
|
||||
### 3. Replace ephemeral subagent runs with durable ones
|
||||
|
||||
If your agent currently spawns ephemeral subagents (OpenClaw `Agent()`, ad-hoc
|
||||
Anthropic API calls, etc.) for work that should survive crashes, sleeps, or
|
||||
worker restarts, migrate those to `gbrain agent run`. The durability is free:
|
||||
|
||||
```bash
|
||||
gbrain agent run "analyze my last 50 journal pages for recurring themes" \
|
||||
--subagent-def analyzer --fanout-manifest manifests/journal-pages.json
|
||||
```
|
||||
|
||||
Every turn persists to `subagent_messages`, every tool call is a two-phase
|
||||
ledger, and `gbrain agent logs <job>` shows where it died + what the last
|
||||
successful call returned. No more "re-run from scratch because the session
|
||||
context evaporated."
|
||||
|
||||
### 4. `put_page` from subagents writes under an agent namespace
|
||||
|
||||
If you adopted the v0.15 subagent runtime, note that `put_page` calls
|
||||
originating from a subagent's tool dispatch MUST target
|
||||
`wiki/agents/<subagent_id>/...`. The schema shown to the model enforces this
|
||||
on first try; a server-side fail-closed check rejects anything else. This
|
||||
does NOT affect your skill files, CLI put_page calls, or MCP put_page —
|
||||
only tool-dispatched writes from inside an LLM loop.
|
||||
|
||||
Aggregation output (the final "here's what all N children found" brain page)
|
||||
goes via a separate trusted CLI path, not through a subagent tool call, so
|
||||
it can write anywhere you want.
|
||||
|
||||
Iron rule: **never grant an agent write access beyond its namespace**. The
|
||||
server-side check exists because dispatcher bugs happen; treat it as defense
|
||||
in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## v0.22.4 — frontmatter-guard adoption
|
||||
|
||||
### 1. Stop hand-rolling frontmatter validators
|
||||
|
||||
If your fork has scripts that call `js-yaml` directly to validate brain page
|
||||
frontmatter, replace them with `gbrain frontmatter validate` calls. The CLI
|
||||
covers the seven canonical error classes and ships a `--json` envelope that's
|
||||
stable across releases.
|
||||
|
||||
```diff
|
||||
- # Custom validator script
|
||||
- node scripts/validate-frontmatter.mjs <path>
|
||||
+ gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
For consumers that need the validator inside another script, import from
|
||||
gbrain's `markdown` export instead of duplicating logic:
|
||||
|
||||
```ts
|
||||
import { parseMarkdown } from 'gbrain/markdown';
|
||||
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true, expectedSlug });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// err.code: MISSING_OPEN | MISSING_CLOSE | YAML_PARSE | SLUG_MISMATCH |
|
||||
// NULL_BYTES | NESTED_QUOTES | EMPTY_FRONTMATTER
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Drop any references to `lib/brain-writer.mjs`
|
||||
|
||||
If your fork's skills or scripts referenced an aspirational
|
||||
`lib/brain-writer.mjs` (it never shipped — the spec was in PR #392 and never
|
||||
landed), replace those references with the gbrain CLI. The `frontmatter-guard`
|
||||
skill lives at `skills/frontmatter-guard/SKILL.md` and points at
|
||||
`gbrain frontmatter validate` / `audit` / `install-hook`.
|
||||
|
||||
### 3. Wire the doctor subcheck into your health pipeline
|
||||
|
||||
`gbrain doctor` now reports `frontmatter_integrity` automatically. If your
|
||||
fork has a custom health pipeline (e.g. a daily Slack post about brain
|
||||
health), pull from `gbrain doctor --json` and surface the
|
||||
`frontmatter_integrity` row counts.
|
||||
|
||||
### 4. (Optional) Install the pre-commit hook on brain repos
|
||||
|
||||
For sources backed by git, the v0.22.4 install-hook helper drops a
|
||||
pre-commit script that blocks commits with malformed frontmatter:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
Skip this if your brain isn't a git repo or if your downstream agent already
|
||||
enforces validation at write time. See `docs/integrations/pre-commit.md` for
|
||||
the full recipe.
|
||||
|
||||
### 5. Migration ergonomics — read pending-host-work.jsonl
|
||||
|
||||
After `gbrain apply-migrations --yes` runs the v0.22.4 audit, your agent
|
||||
should read `~/.gbrain/migrations/pending-host-work.jsonl` (filter to
|
||||
`migration === "0.22.4"`) and walk each entry's `command` field. Each entry
|
||||
points to a per-source `gbrain frontmatter validate <source_path> --fix`
|
||||
command — surface counts to the user, get explicit consent, then run.
|
||||
|
||||
The migration is **audit-only**. It never mutates brain content during
|
||||
`apply-migrations`. Your agent runs the fix command with user consent.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
version. Each new version appends a section; old sections stay so you can catch up
|
||||
multiple versions at once.
|
||||
|
||||
To check what your fork is missing:
|
||||
```bash
|
||||
diff <(grep -A3 "Based on gbrain" ~/<your-fork>/skills/brain-ops/SKILL.md) \
|
||||
<(grep "v[0-9]" ~/gbrain/skills/migrations/ | tail -3)
|
||||
```
|
||||
@@ -0,0 +1,167 @@
|
||||
# Search Quality Benchmark — PR #64
|
||||
|
||||
**Date:** 2026-04-14
|
||||
**Branch:** garrytan/search-quality-boost
|
||||
**Inspired by:** Ramp Labs' "Latent Briefing" paper (April 2026)
|
||||
|
||||
## What this PR does
|
||||
|
||||
GBrain stores knowledge in brain pages. Each page has two sections: **compiled truth**
|
||||
(your distilled assessment of a person, company, or concept) and **timeline** (dated
|
||||
entries like meeting notes, announcements, funding rounds).
|
||||
|
||||
Before this PR, search treated both sections equally. Ask "who is Alice Chen?" and you
|
||||
might get a meeting note from March instead of the actual assessment. Ask "when did we
|
||||
last meet Alice?" and you might get the assessment instead of the date.
|
||||
|
||||
This PR teaches search to understand the difference. It picks the right section based
|
||||
on what you're asking.
|
||||
|
||||
## How we test it
|
||||
|
||||
We built a synthetic brain with **29 fictional pages** and **58 chunks** (2 per page:
|
||||
one compiled truth, one timeline). The pages span 10 people, 10 companies, and 9
|
||||
concept pages across topics like AI, fintech, climate, crypto, robotics, education,
|
||||
biotech, and design.
|
||||
|
||||
The embeddings share dimensions to simulate real-world overlap. "AI" shows up in
|
||||
health pages, education pages, design pages, and robotics pages. A query about "AI
|
||||
companies" has to sort through 5+ relevant pages, not just find one obvious match.
|
||||
|
||||
We run **20 queries** with hand-labeled ground truth:
|
||||
- 11 entity queries ("who is X?", "what does Y do?", "tell me about Z")
|
||||
- 7 temporal queries ("when did we last meet?", "recent updates", "what launched?")
|
||||
- 1 negative control (irrelevant topic, no matches expected)
|
||||
- 1 ambiguous query (could go either way)
|
||||
|
||||
Each query has **graded relevance**: the primary answer gets grade 3, related pages get
|
||||
2 or 1. A query about climate investing has 4 relevant pages ranked by importance.
|
||||
|
||||
We compare three configurations:
|
||||
- **A. Baseline** — how search worked before this PR
|
||||
- **B. Boost only** — compiled truth chunks get a 2x score multiplier (the naive approach)
|
||||
- **C. Boost + Intent** — the full PR: boost + intent classifier that auto-detects query type
|
||||
|
||||
## Results: finding the right page
|
||||
|
||||
These are standard information retrieval metrics. They answer: "did search find the
|
||||
right page?"
|
||||
|
||||
| Metric | What it measures | A. Before | C. After | Change |
|
||||
|--------|-----------------|-----------|----------|--------|
|
||||
| **P@1** | Is the #1 result relevant? | 94.7% | 94.7% | same |
|
||||
| **MRR** | How far down is the first relevant result? | 0.974 | 0.974 | same |
|
||||
| **nDCG@5** | Are the top 5 results in the right order? | 1.191 | 1.069 | -10% |
|
||||
|
||||
Page-level retrieval is roughly the same. The right page was already being found. This
|
||||
is not where the improvement lives.
|
||||
|
||||
## Results: finding the right chunk (the actual improvement)
|
||||
|
||||
These metrics answer: "did search find the right SECTION of the right page?" This is
|
||||
what matters when an agent reads search results to answer a question.
|
||||
|
||||
| Metric | What it measures | A. Before | C. After | Change |
|
||||
|--------|-----------------|-----------|----------|--------|
|
||||
| **Source accuracy** | Is the top chunk the right type for this query? (assessment for "who is X?", timeline for "when did we meet?") | 89.5% | 89.5% | same |
|
||||
| **CT-first rate** | For entity lookups, does the assessment show up before timeline noise? | 100% | 100% | same |
|
||||
| **Timeline accessible** | For temporal queries, can you actually find the dates? | 100% | 100% | same |
|
||||
| **Unique pages** | How many different pages appear in top 10? (more = broader context) | 7.2 | **8.7** | **+21%** |
|
||||
| **Compiled truth ratio** | What % of returned chunks are assessments vs timeline noise? | 51.6% | **66.8%** | **+29%** |
|
||||
|
||||
Two big improvements:
|
||||
|
||||
1. **21% more page coverage.** The agent sees 8.7 unique pages per query instead of 7.2.
|
||||
When you ask "AI companies building real products", you get results from MindBridge,
|
||||
EduStack, PixelCraft, GenomeAI, AND the AI-first thesis page. Before, some of those
|
||||
were crowded out.
|
||||
|
||||
2. **29% more signal in results.** Two thirds of returned chunks are now compiled truth
|
||||
(assessments) instead of roughly half. The agent reads more distilled knowledge and
|
||||
less timeline noise.
|
||||
|
||||
## Why the boost alone isn't enough
|
||||
|
||||
We also tested configuration B: the 2x compiled truth boost without the intent classifier.
|
||||
This is the naive version that just says "rank assessments higher, always."
|
||||
|
||||
| What broke | Before | Boost only | With intent |
|
||||
|-----------|--------|------------|-------------|
|
||||
| Source accuracy | 89.5% | **63.2%** | 89.5% |
|
||||
| Timeline accessible | 100% | **71.4%** | 100% |
|
||||
| P@1 | 94.7% | **89.5%** | 94.7% |
|
||||
|
||||
The boost forces compiled truth to the top even when timeline IS the right answer. Ask
|
||||
"what launched this year?" and the boost pushes assessment chunks above the actual launch
|
||||
dates. The source accuracy drops from 89.5% to 63.2%.
|
||||
|
||||
The **intent classifier** fixes this. It reads the query text (zero latency, no LLM call)
|
||||
and detects whether you're asking an entity question or a temporal question:
|
||||
|
||||
- "Who is Alice Chen?" → entity → boost compiled truth
|
||||
- "When did we last meet Alice?" → temporal → skip boost, show timeline
|
||||
- "Recent funding rounds" → temporal → skip boost, show dates
|
||||
- "AI companies building real products" → general → moderate boost
|
||||
|
||||
This recovers all the regressions while keeping the improvements.
|
||||
|
||||
## Per-query results
|
||||
|
||||
Every query, every configuration. "Src" column shows which chunk type ranked first.
|
||||
|
||||
| Query | Expected | Before src | After src | Before pages | After pages |
|
||||
|-------|----------|-----------|-----------|-------------|-------------|
|
||||
| Who is Alice Chen? | assessment | assessment | assessment | 7 | 10 |
|
||||
| What does MindBridge do? | assessment | assessment | assessment | 6 | 10 |
|
||||
| Tell me about climate investing | assessment | assessment | assessment | 5 | 10 |
|
||||
| When did we last meet Alice? | timeline | timeline | timeline | 9 | 9 |
|
||||
| Recent updates on GenomeAI | timeline | timeline | timeline | 8 | 8 |
|
||||
| CloudScale acquisition | timeline | timeline | timeline | 8 | 8 |
|
||||
| Alice Chen NovaPay payments | assessment | assessment | assessment | 7 | 8 |
|
||||
| Carol Nakamura MindBridge AI | assessment | assessment | assessment | 6 | 8 |
|
||||
| AI companies building products | assessment | assessment | assessment | 9 | 10 |
|
||||
| Who raised funding recently? | timeline | timeline | timeline | 10 | 10 |
|
||||
| Bob and James climate investments | assessment | assessment | assessment | 5 | 9 |
|
||||
| AI replacing designers | assessment | assessment | assessment | 7 | 8 |
|
||||
| Everything on RoboLogic | timeline | assessment | assessment | 6 | 6 |
|
||||
| Deep dive on crypto custody | timeline | assessment | assessment | 6 | 6 |
|
||||
| Education technology Africa | assessment | assessment | assessment | 7 | 10 |
|
||||
| What launched this year? | timeline | timeline | timeline | 10 | 10 |
|
||||
| MPC multi-party computation | assessment | assessment | assessment | 7 | 9 |
|
||||
| Protein folding drug discovery | assessment | assessment | assessment | 7 | 9 |
|
||||
| EduStack Nigeria | assessment | assessment | assessment | 7 | 8 |
|
||||
|
||||
The "pages" column tells the clearest story. Entity lookups with `detail=low` (the
|
||||
intent classifier's choice) go from 5-7 pages to 8-10 pages. The agent gets significantly
|
||||
broader context for the same query.
|
||||
|
||||
## What shipped in PR #64
|
||||
|
||||
1. **Compiled truth boost** — 2.0x score multiplier after RRF normalization
|
||||
2. **Intent classifier** — zero-latency regex that auto-selects detail level per query
|
||||
3. **Detail parameter** — `--detail low/medium/high` for explicit agent control
|
||||
4. **Source-aware dedup** — guarantees compiled truth chunk per page in results
|
||||
5. **Cosine re-scoring** — re-ranks chunks against the actual query embedding
|
||||
6. **RRF normalization** — scores normalized to 0-1 before boosting
|
||||
7. **CJK word count fix** — Chinese/Japanese/Korean queries now expand correctly
|
||||
8. **Eval harness** — `gbrain eval --qrels` with P@k, R@k, MRR, nDCG@k + A/B comparison
|
||||
9. **This benchmark** — 29 pages, 20 queries, reproducible, no private data
|
||||
|
||||
## How to reproduce
|
||||
|
||||
```bash
|
||||
bun run test/benchmark-search-quality.ts
|
||||
```
|
||||
|
||||
Runs in ~2 seconds against in-memory PGLite. No API keys, no database, no network.
|
||||
|
||||
## Methodology notes
|
||||
|
||||
- All data is fictional. No private information from any real brain.
|
||||
- Embeddings use 25 topic dimensions with shared axes (not orthogonal basis vectors).
|
||||
"AI" and "health" share signal so that an AI health query naturally ranks both the
|
||||
AI-health concept page and the MindBridge company page.
|
||||
- Each page has exactly 2 chunks (1 compiled truth, 1 timeline) for clean measurement.
|
||||
Real brains have more chunks per page, which would amplify the boost's effect.
|
||||
- The baseline uses the old text-prefix dedup key. The new configurations use chunk_id.
|
||||
- Graded relevance: 3 = primary answer, 2 = strongly related, 1 = tangentially related.
|
||||
@@ -1,162 +0,0 @@
|
||||
# Code Cathedral II — v0.20.0 Design
|
||||
|
||||
**Status:** Accepted. CEO + Eng + 2 codex passes CLEARED (2026-04-24). 16 cross-model findings absorbed total: 7 codex pass 1 (structural prereqs) + 6 codex pass 2 (absorption errors including the CHUNKER_VERSION silent-no-op gate and inbound-edge invalidation) + 3 eng-review architectural decisions. DX review recommended post-Layer 8 (new CLI surfaces) before ship.
|
||||
**Supersedes:** Cathedral I (planned v0.18.0–v0.19.0 code indexing, shipped v0.19.0).
|
||||
**Mode:** SCOPE EXPANSION (user explicit: "I want the best code search in the world").
|
||||
**Scale:** 14 bisectable layers, ~20–25 CC hours, 3–5 human-weeks. One schema migration with split edge tables (`code_edges_chunk` + `code_edges_symbol`). Backfill via `CHUNKER_VERSION` bump (automatic on next sync) + explicit `gbrain reindex-code` command.
|
||||
|
||||
## Why v0.20.0
|
||||
|
||||
v0.19.0 shipped code indexing: tree-sitter chunker, 29 active languages, symbol columns, forward doc↔impl linking, incremental embed cache, BrainBench code category. Four cathedral-I items got deferred during shipping: `query --lang` filter, `sync --all` cost preview, markdown fence extraction, reverse-scan doc↔impl backfill.
|
||||
|
||||
Cathedral II is a promise-keeping release for those four, bundled with the leap that makes gbrain *the* code search: structural edges (call graph + references + imports + inheritance), parent-scope capture, doc-comment FTS binding, and two-pass retrieval. No more grep-class retrieval on code.
|
||||
|
||||
## The 10x leap
|
||||
|
||||
Today: agent asks "how does hybrid search handle N+1?" → gets 3 prose chunks of `hybrid.ts`.
|
||||
|
||||
Cathedral II: same query returns the anchor function + its 3 callers + its 2 callees + its JSDoc + the guide in `/docs` that cites it + the test file exercising it + parent scope chain. One walk. Code-aware brain.
|
||||
|
||||
## Scope (5 tiers + Layer 0 prerequisites, 14 bisectable layer commits)
|
||||
|
||||
### Tier 0 — Prerequisites (surfaced by codex outside voice)
|
||||
|
||||
**0a. File-classification widening.** `sync.ts:35` currently classifies only 9 extensions as code (TS, JS, Python, Go, Rust, Ruby, Java, C, C++). Cathedral II's B1 ships 165 lazy-loadable grammars, so the classifier needs to accept any extension the chunker can handle. Also reorders `detectCodeLanguage` so Magika (B2) runs as a fallback for extension-less files, not after a null-return gate.
|
||||
|
||||
**0b. Chunk-grain FTS.** Current keyword search lives on `pages.search_vector`. Adding doc-comments or two-pass anchoring at the chunk level has zero ranking effect against a page-grain primitive. Layer 0b adds `content_chunks.search_vector` with a trigger building from qualified symbol name + doc-comment (weight A) and chunk_text (weight B), plus rewrites `searchKeyword` to rank chunks directly. Page-level search_vector stays for title-heavy searches.
|
||||
|
||||
Both Layer 0 items are prerequisites for the 10x leap to actually move retrieval metrics.
|
||||
|
||||
### Tier A — Structural edges (the 10x leap)
|
||||
|
||||
**A1. Call-graph + reference extraction with qualified symbol identity.** Per-language tree-sitter queries at `importCodeFile` time capture:
|
||||
|
||||
- `calls` — function call-sites
|
||||
- `imports` — module deps
|
||||
- `extends` / `implements` — type hierarchies
|
||||
- `mixes_in` — Ruby `include`/`extend`/`prepend`
|
||||
- `type_refs` — parameter + return type usage
|
||||
- `declares` — chunk owns a symbol definition
|
||||
|
||||
**Qualified symbol identity across all 8 langs.** `parent_symbol_path` (A3) is the source of truth for scope; edges use qualified names built from it. Examples: `Admin::UsersController#render` (Ruby instance), `Admin::UsersController.find_all` (Ruby singleton), `admin.users_controller.UsersController.render` (Python), `(*UsersController).Render` (Go), `users::UsersController::render` (Rust), `com.acme.admin.UsersController.render` (Java). Per-lang delimiter + method/class-method distinction. Ruby ships fully in ranker (CLI + A2 two-pass) — no deferral.
|
||||
|
||||
**Split schema (two tables, not one polymorphic):**
|
||||
```sql
|
||||
CREATE TABLE code_edges_chunk (
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
UNIQUE (from_chunk_id, to_chunk_id, edge_type)
|
||||
);
|
||||
CREATE TABLE code_edges_symbol (
|
||||
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
|
||||
from_symbol_qualified TEXT NOT NULL,
|
||||
to_symbol_qualified TEXT NOT NULL,
|
||||
edge_type TEXT NOT NULL,
|
||||
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
|
||||
UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
|
||||
);
|
||||
```
|
||||
`code_edges_chunk` = resolved (both endpoints known). `code_edges_symbol` = unresolved (target symbol exists by qualified name, definition chunk not yet seen). Promotion from symbol→chunk table happens on later import. `source_id` is TEXT matching actual `sources.id` type.
|
||||
|
||||
**Shipped languages:** TypeScript, TSX, JavaScript, Ruby, Python, Go, Rust, Java (8 langs, ~85% of real brain code). Other languages chunk normally (via B1 lazy-load) but don't emit edges in v0.20.0 — extension is one query file + delimiter config per language, shippable as small follow-up PRs.
|
||||
|
||||
**A2. Two-pass retrieval.** Current: keyword + vector → RRF → dedup. New: keyword + vector → anchor set → expand 1–2 hops on `code_edges_chunk` with structural-distance decay → blend into RRF.
|
||||
|
||||
**Default OFF in all cases.** Opt-in only via `--walk-depth N` or `--near-symbol <name>`. Exact-symbol-match auto-on was unsafe (symbol names collide across files). Neighbor cap 50 per hop, depth cap 2. Dedup's per-page cap (currently 2) lifts to `min(10, walkDepth × 5)` when walking so structural neighbors from one file aren't clipped. Distance decay: `1/(1 + hop)` on expanded-neighbor RRF contributions.
|
||||
|
||||
**A3. Parent-scope capture + nested-chunk emission.** Two parts:
|
||||
|
||||
*Part 1:* Nested symbols get `parent_symbol_path text[]` on `content_chunks`. Embedded into chunk header: `[TypeScript] src/foo.ts:42-58 function formatResult (in BrainEngine.searchKeyword)`. Scope flows into embedding. Dual-use: drives A1's qualified symbol identity.
|
||||
|
||||
*Part 2:* Extend `splitLargeNode` to emit nested functions/methods/inner-classes as their own chunks. The current chunker is top-level-node oriented — a `class Foo { method1() {} method2() {} }` emits one chunk. Parent_symbol_path on top-level nodes is empty (no parent above top level), so A3 contributes nothing without sub-top-level chunks. Part 2 makes the scope annotation load-bearing.
|
||||
|
||||
**A4. Doc-comment → symbol binding.** Leading AST comment extracted to `doc_comment text`. Lands on **chunk-grain** search_vector (Layer 0b prerequisite) with FTS weight `'A'`. Natural-language queries rank docstring matches above body text and below title. `'A' > 'B' > 'C' > 'D'` per Postgres FTS weight convention.
|
||||
|
||||
### Tier B — Coverage (honest Chonkie parity)
|
||||
|
||||
**B1.** Lazy-load tree-sitter-language-pack (~165 languages). Replace 36 committed WASMs with a manifest + per-process parser cache. Cathedral I promised this and didn't deliver — Cathedral II does.
|
||||
|
||||
**B2.** Magika auto-detect for extension-less files (Dockerfile, Makefile, `.envrc`). ~1MB bundled asset. Falls back to null → recursive chunker if classifier fails to load.
|
||||
|
||||
### Tier C — Agent CLI surfaces
|
||||
|
||||
- `query --lang <lang>` — filter by `content_chunks.language`
|
||||
- `query --symbol-kind function|class|method|type|interface|enum` — filter by `symbol_type`
|
||||
- `query --near-symbol <name> --depth 1..2` — two-pass retrieval anchored at a known symbol
|
||||
- `code-callers <symbol>` — uses A1 `calls` edges, reversed
|
||||
- `code-callees <symbol>` — uses A1 `calls` edges, forward
|
||||
|
||||
All auto-JSON on non-TTY. `StructuredAgentError` envelopes on failure. `code-signature` deferred to v0.20.1 (needs per-language type captures).
|
||||
|
||||
### Tier D — Bridge items (cathedral I promises)
|
||||
|
||||
**D1.** `sync --all` cost preview. `estimateTokens` extracted from `chunkers/code.ts` to new `tokens.ts` module. Before per-source loop: walk sync-diff set, sum tokens, compute $ estimate. TTY + !json + !yes → interactive `[y/N]`. Non-TTY or `--json` or piped → emit `ConfirmationRequired` envelope, exit 2. `--yes` skips. `--dry-run` previews + exit 0. Preview on `--all` only, not single-source (DX review pain is first-time large-sync surprise bills).
|
||||
|
||||
**D2.** Markdown fence extraction in `importFromContent`. After `parseMarkdown`, iterate marked lexer tokens for `{type:'code', lang, text}`. Map fence tag → language. Chunk each fence through `chunkCodeText`. Persist as `chunk_source='fenced_code'`. Cap 100 fences per markdown page (DOS defense). Per-fence try/catch — one bad fence doesn't break the page import.
|
||||
|
||||
**D3.** `reconcile-links` batch command. Walks markdown pages, calls existing v0.19.0 `extractCodeRefs` per page, emits `addLink(md, code, ..., 'documents')` + reverse. `ON CONFLICT DO NOTHING` handles idempotency. Statement-timeout scoped via `sql.begin` + `SET LOCAL`. Progress reporter + final summary (edges added / existed / missing-target). Respects `auto_link` config.
|
||||
|
||||
### Tier E — Eval, backfill, honesty
|
||||
|
||||
**E1.** BrainBench code sub-categories: `call_graph_recall` (callers of X → expected set), `parent_scope_coverage` (nested-symbol queries return correct scope), `doc_comment_matching` (NL queries rank doc-comments above prose). Regression gates against A1/A3/A4 drift.
|
||||
|
||||
**E2.** Backfill: schema migrates automatically (zero cost). **`CHUNKER_VERSION` bumps 3 → 4** — that constant is folded into each code page's `content_hash`, so every code page's hash changes on upgrade. Next `gbrain sync` won't short-circuit on "git HEAD unchanged"; it re-chunks every code file. New `gbrain reindex-code [--source <id>] [--dry-run] [--yes] [--force]` provides explicit full backfill with cost preview (reuses D1 infra) and `--force` bypasses content_hash skip entirely. Users control when to pay; silent no-op path closed.
|
||||
|
||||
**E3.** Honest CHANGELOG. Retire "Chonkie superset" framing. Run BrainBench before/after for real numbers: 150+ languages loaded (after B1), MRR on NL→code queries, P@1 call-graph precision, P@k on symbol_name queries, sync cost preview on 5K-file repo. Back every claim with a runnable command.
|
||||
|
||||
## Implementation ordering (14 layers, post-codex)
|
||||
|
||||
1. **0a** — File-classification widening (sync.ts:35) + Magika reordered as fallback
|
||||
2. **0b** — Chunk-grain FTS (content_chunks.search_vector + trigger + searchKeyword chunk-level rewrite)
|
||||
3. **Foundation** — schema migration (split edge tables, qualified name columns on content_chunks) + engine method stubs + types
|
||||
4. **B1** — lazy-load grammar manifest + bun --compile guard
|
||||
5. **A1** — edge-extractor + 8 per-lang query files + qualified symbol identity + tests
|
||||
6. **A3** — parent-scope column + doc-comment column + splitLargeNode nested-chunk emission
|
||||
7. **A4** — doc-comment FTS weight A on chunk-grain search_vector
|
||||
8. **A2** — two-pass retrieval, default OFF, opt-in only; dedup cap lifts when walking
|
||||
9. **D tier bundled** — cost preview + fence extraction + reconcile-links
|
||||
10. **B2** — Magika auto-detect
|
||||
11. **C tier** — 5 CLI surfaces
|
||||
12. **E1** — BrainBench sub-categories + CHUNKER_VERSION 3→4 bump
|
||||
13. **E2** — `reindex-code` with `--force` + migration orchestrator with backfill-prompt phase
|
||||
14. **E3 + release** — honest CHANGELOG + docs + migration skill + `/ship`
|
||||
|
||||
## Size and cost
|
||||
|
||||
- Diff: ~5500–6500 lines (~2.5x v0.19.0 post-codex expansion)
|
||||
- Tests: ~2000 lines (8 langs × qualified-name + edge-extraction fixtures + Layer 0b FTS migration tests)
|
||||
- Files: ~36 new, ~25 modified
|
||||
- CC time: ~20–25 hours focused (was 14–18 pre-codex; +6h for Layer 0a/0b + qualified identity across 8 langs + nested-chunk emission + CHUNKER_VERSION bump layer)
|
||||
- Human-equivalent: 3–5 weeks
|
||||
- First-sync cost bump for upgraded v0.19.0 users: every code page re-chunks on first sync after upgrade (CHUNKER_VERSION bump forces invalidation). Users run `gbrain reindex-code --dry-run` for cost preview, then `--yes` or accept gradual backfill over time as files change.
|
||||
- Daily autopilot cost post-backfill: unchanged (edges extracted at chunk time, no per-query LLM)
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
1. **Schema migration on live Postgres.** Test against production-shape DB before ship. v0.12.0 JSONB incident is the canary.
|
||||
2. **Per-language tree-sitter queries are fiddly.** Hand-verified edge-set fixtures per language. Ruby gets extra coverage for dynamic-dispatch false negatives.
|
||||
3. **Two-pass retrieval regression.** Default off for prose. BrainBench Cat 1 MUST show no regression before shipping.
|
||||
4. **Backfill shape (G1 resolved).** Three composable layers: schema-auto migrates columns empty (zero cost). Lazy on-touch catches 80% over time (zero cost). Explicit `reindex-code` with cost preview for users wanting immediate full benefit. No surprise bills.
|
||||
5. **Magika bundle (G2 resolved).** +1MB asset, `bun --compile` guard extension. If bundling surfaces bugs late in implementation, B2 is the only tier that can fall back to v0.20.1 without blocking the cathedral — it's self-contained at Layer 8.
|
||||
6. **High-fan-out symbols.** `console.log`-style symbols have 100K callers. Neighbor cap 50, depth cap 2. Chaos test fixture required.
|
||||
|
||||
## Review gates
|
||||
|
||||
- CEO review (cathedral II) — CLEARED 2026-04-24
|
||||
- Outside voice (codex) — run during cathedral II CEO review
|
||||
- `/plan-devex-review` — up next (per user request, 5 new CLI surfaces + reindex-code need DX polish review before eng)
|
||||
- `/plan-eng-review` — required before implementation begins
|
||||
- `/review` + `/codex review` — required before `/ship`
|
||||
|
||||
## What's deferred to later cathedrals
|
||||
|
||||
- **C6** `code-signature "(A, B) => C"` — per-language type captures. v0.20.1.
|
||||
- **Call-graph langs beyond 8 shipped** — PHP, Swift, Kotlin, Scala, C#, C++, Elixir, etc. One small PR per language.
|
||||
- **LSP integration** for live precision. v0.22+ cathedral.
|
||||
- **Code-tour generator** (cathedral I T1).
|
||||
- **Private-code redaction pre-embed** (cathedral I T3).
|
||||
- **`gbrain doctor --chunker-debug`** AST dump.
|
||||
@@ -1,717 +0,0 @@
|
||||
# GBrain Knowledge Runtime — Design Doc
|
||||
|
||||
**Status:** DRAFT for CEO review.
|
||||
**Date:** 2026-04-18.
|
||||
**Supersedes:** The earlier "Feynman Ideas Assessment + Phase A/B" plan.
|
||||
|
||||
---
|
||||
|
||||
## 0. Context
|
||||
|
||||
During a CEO review of a narrow two-feature plan (bare-tweet citation repair + completeness score, borrowed from Feynman), the scope was reframed. The narrow plan duplicated work Garry's OpenClaw already does and missed the real leverage point: **the bespoke abstractions hiding inside OpenClaw — resolvers, enrichment orchestration, scheduling, deterministic output — should live in GBrain as first-class primitives.**
|
||||
|
||||
North star: *"When Garry's OpenClaw's Claw upgrades to this version of GBrain, it should immediately recognize brilliance and completeness and say 'It's time to switch to these abstractions.'"*
|
||||
|
||||
That is the test this document is designed against. Everything else is downstream.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Four Layers
|
||||
|
||||
The design is four layered abstractions. Each is independently useful; together they are the Knowledge Runtime.
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────────┐
|
||||
│ KNOWLEDGE RUNTIME (new) │
|
||||
├───────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 4: Deterministic Output Builder │
|
||||
│ BrainWriter · Scaffolds · Back-link enforcer · Slug registry │
|
||||
│ Rule: LLM picks WHAT to write. Code guarantees WHERE and HOW. │
|
||||
├───────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Scheduler │
|
||||
│ ScheduledResolver · TZ-aware quiet hours (enforced) · │
|
||||
│ Auto-stagger · Durable state · Retry/circuit-break │
|
||||
├───────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 2: Enrichment Orchestrator │
|
||||
│ Trigger convergence · Tier routing · Budget · Cascade · │
|
||||
│ Evidence-weighted completeness · Fail-safe transactions │
|
||||
├───────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 1: Resolver SDK │
|
||||
│ Resolver<I,O> interface · Registry · Factory · Plugin recipes │
|
||||
│ Ported reference impls: X-API, Perplexity, Mistral, brain │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
REUSES (polished primitives already in GBrain) REPLACES (ad-hoc code)
|
||||
FailImproveLoop · backoff · storage factory · enrichment-service ·
|
||||
check-resolvable · operations validators · embedding · transcription ·
|
||||
engine interface · publish · backlinks 2 recipe formats
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Why This Order (L1 → L4)
|
||||
|
||||
Every higher layer depends on the lower one. **L1 must land first or the rest leaks abstractions.**
|
||||
|
||||
- **L1 (Resolvers)** is the substrate. Without a uniform lookup interface, every orchestrator + writer has bespoke callers.
|
||||
- **L2 (Orchestrator)** uses L1 to fetch; without L1 it's still ad-hoc.
|
||||
- **L3 (Scheduler)** runs L2 periodically; without L2 it's scheduling nothing structured.
|
||||
- **L4 (Output Builder)** is what every layer ultimately writes through; without it we have 14 call sites doing `fs.writeFile` with hand-rolled citation discipline.
|
||||
|
||||
An earlier implementation could ship L1 + L4 first (the two "purest" layers) and have the most immediate integrity impact, then add L2 + L3. But the end-state must include all four.
|
||||
|
||||
---
|
||||
|
||||
## 3. Layer 1 — Resolver SDK
|
||||
|
||||
### 3.1 What's broken today
|
||||
|
||||
Garry's OpenClaw has **69 distinct external-lookup patterns** across X API (14 shapes), Perplexity, Mistral OCR, Gmail, Calendar, Slack, GitHub, YouTube, Diarize.io, YC tools, OSINT collectors, and brain-local lookups. Each one is a bespoke script under `scripts/` with its own error handling, retry logic, and output shape. GBrain has 3 ad-hoc wrappers (`embedding.ts`, `transcription.ts`, `enrichment-service.ts`) that don't share an interface.
|
||||
|
||||
Common consequences:
|
||||
- No uniform retry/backoff strategy (some scripts retry, most don't)
|
||||
- No cost tracking (Perplexity bills eaten silently when calls return no-substance results)
|
||||
- No confidence/provenance propagation (callers can't tell if an answer is verified or inferred)
|
||||
- Users can't add a resolver without forking GBrain
|
||||
|
||||
### 3.2 Interface
|
||||
|
||||
```typescript
|
||||
// src/core/resolvers/interface.ts
|
||||
|
||||
export type ResolverCost = 'free' | 'rate-limited' | 'paid';
|
||||
|
||||
export interface ResolverRequest<I> {
|
||||
input: I;
|
||||
context: ResolverContext;
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ResolverResult<O> {
|
||||
value: O;
|
||||
confidence: number; // 0.0–1.0; 1.0 = deterministic from ground-truth API
|
||||
source: string; // e.g. "x-api-v2", "perplexity-sonar", "brain-local"
|
||||
fetchedAt: Date;
|
||||
costEstimate?: number; // dollars; 0 if free
|
||||
raw?: unknown; // for sidecar preservation via put_raw_data
|
||||
}
|
||||
|
||||
export interface Resolver<I, O> {
|
||||
readonly id: string; // stable, slug-like: "x_handle_to_tweet"
|
||||
readonly cost: ResolverCost;
|
||||
readonly backend: string; // "x-api-v2", "perplexity", "brain-local"
|
||||
readonly inputSchema: JSONSchema;
|
||||
readonly outputSchema: JSONSchema;
|
||||
|
||||
available(ctx: ResolverContext): Promise<boolean>;
|
||||
resolve(req: ResolverRequest<I>): Promise<ResolverResult<O>>;
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 Context
|
||||
|
||||
```typescript
|
||||
export interface ResolverContext {
|
||||
engine: BrainEngine;
|
||||
storage: StorageBackend;
|
||||
config: GBrainConfig;
|
||||
logger: Logger;
|
||||
metrics: MetricsRecorder;
|
||||
budget: BudgetLedger; // hard spend caps, queried pre-resolve
|
||||
requestId: string;
|
||||
remote: boolean; // trust boundary — untrusted callers get stricter validation
|
||||
deadline?: Date;
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 Registry + Factory (mirrors `src/core/storage.ts`)
|
||||
|
||||
```typescript
|
||||
// src/core/resolvers/registry.ts
|
||||
export class ResolverRegistry {
|
||||
register<I, O>(r: Resolver<I, O>): void;
|
||||
get(id: string): Resolver<unknown, unknown>;
|
||||
list(filter?: { cost?: ResolverCost; backend?: string }): Resolver[];
|
||||
async resolve<I, O>(id: string, input: I, ctx: ResolverContext): Promise<ResolverResult<O>>;
|
||||
}
|
||||
|
||||
// src/core/resolvers/factory.ts (dynamic import like engine-factory)
|
||||
export async function createResolver(
|
||||
type: 'x-api' | 'perplexity' | 'mistral-ocr' | 'brain-local' | 'plugin',
|
||||
config: ResolverConfig,
|
||||
): Promise<Resolver>;
|
||||
```
|
||||
|
||||
### 3.5 Plugin format (unifies `recipes/` + `data-research` formats)
|
||||
|
||||
A plugin is YAML + JS module, discovered via filesystem scan of `~/.gbrain/resolvers/` and `recipes/`.
|
||||
|
||||
```yaml
|
||||
# Example: resolvers/x-api/handle-to-tweet.yaml
|
||||
id: x_handle_to_tweet
|
||||
version: 1
|
||||
category: lookup
|
||||
cost: rate-limited
|
||||
backend: x-api-v2
|
||||
module: ./handle-to-tweet.ts
|
||||
input_schema:
|
||||
type: object
|
||||
properties:
|
||||
handle: { type: string, pattern: "^[A-Za-z0-9_]{1,15}$" }
|
||||
keywords: { type: string }
|
||||
required: [handle]
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
url: { type: string, format: uri }
|
||||
tweet_id: { type: string }
|
||||
text: { type: string }
|
||||
created_at: { type: string, format: date-time }
|
||||
requires:
|
||||
env: [X_API_BEARER_TOKEN]
|
||||
health_check:
|
||||
kind: http
|
||||
url: https://api.twitter.com/2/tweets/1
|
||||
expect: { status: [200, 401] } # 401 = auth failure but endpoint reachable
|
||||
tests:
|
||||
- input: { handle: "garrytan" }
|
||||
expect: { url: { pattern: "^https://x\\.com/garrytan/status/\\d+$" } }
|
||||
```
|
||||
|
||||
Trust flagging follows the existing `src/commands/integrations.ts` pattern: only package-bundled resolvers are `embedded=true` and may run arbitrary commands; user-provided resolvers are restricted to `http` and validated schemas.
|
||||
|
||||
### 3.6 Wraps every resolver with `FailImproveLoop`
|
||||
|
||||
Existing `src/core/fail-improve.ts` is the deterministic-first/LLM-fallback pattern. Every resolver automatically gets wrapped: if the deterministic path (e.g. X API) returns a valid result, use it; if it fails, optionally fall back to an LLM-based resolver; log both paths for future pattern analysis and auto-test generation.
|
||||
|
||||
### 3.7 Reference implementations to ship
|
||||
|
||||
The OpenClaw survey inventoried 69 resolver shapes. Shipping all of them is wrong (over-scoped); shipping zero is under-scoped. The dogfood set:
|
||||
|
||||
| # | Resolver | Purpose | Used by |
|
||||
|---|---|---|---|
|
||||
| 1 | `x_handle_to_tweet` | Bare-tweet citation repair (original Phase A) | `gbrain integrity` |
|
||||
| 2 | `url_reachable` | Dead-link detection | `gbrain integrity` |
|
||||
| 3 | `brain_slug_lookup` | Name/email → slug (wraps existing `resolveSlugs`) | Output Builder |
|
||||
| 4 | `openai_embedding` | Refactor of `src/core/embedding.ts` into Resolver | Import pipeline |
|
||||
| 5 | `perplexity_query` | Query → synthesis + citations | Enrichment Orchestrator |
|
||||
| 6 | `text_to_entities` | LLM entity extraction (structured JSON) | Enrichment Orchestrator |
|
||||
|
||||
The remaining 63 OpenClaw patterns port incrementally, driven by user need. Each port is a new YAML + module under `recipes/` or `~/.gbrain/resolvers/` with no framework changes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Layer 2 — Enrichment Orchestrator
|
||||
|
||||
### 4.1 What's broken today
|
||||
|
||||
Garry's OpenClaw's enrichment is **polished at the data layer, hacky at the control layer**:
|
||||
|
||||
- **Completeness = "length > 500 chars + no `needs-enrichment` tag"** (`lib/enrich.mjs:351-355`). Naïve. A rich page of repetitive Perplexity summaries (see `brain/people/0interestrates.md` — 38 repeating blocks) passes this check.
|
||||
- **30-day auto-re-enrichment** runs forever. No "done" state. A person met once in 2023 still gets re-researched monthly.
|
||||
- **Cascade is convention-only.** Person→company stubs are created automatically; company→investors, company→employees traversals are documented but never implemented.
|
||||
- **No hard budget cap.** Cost is estimated per batch, never enforced across batches or per day.
|
||||
- **Failure is silent.** A bad Perplexity response logs and continues; partial writes can leave a page with a timeline entry but no raw-data sidecar.
|
||||
|
||||
### 4.2 The orchestrator
|
||||
|
||||
```typescript
|
||||
// src/core/enrichment/orchestrator.ts
|
||||
|
||||
export interface EnrichmentRequest {
|
||||
entitySlug: string;
|
||||
trigger: 'mention' | 'stub-creation' | 'cron-sweep' | 'manual' | 'cascade';
|
||||
tier?: 1 | 2 | 3; // optional override; auto-computed if absent
|
||||
cascadeDepth?: number; // 0 = no cascade; default 1
|
||||
}
|
||||
|
||||
export interface EnrichmentResult {
|
||||
entitySlug: string;
|
||||
completenessBefore: number;
|
||||
completenessAfter: number;
|
||||
resolversUsed: string[]; // e.g. ["perplexity_query", "x_handle_to_tweet"]
|
||||
costSpent: number;
|
||||
writtenTo: string[]; // page paths touched, for transaction audit
|
||||
cascadedTo: string[]; // related entities enriched
|
||||
status: 'enriched' | 'skipped' | 'failed' | 'budget-exhausted';
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export class EnrichmentOrchestrator {
|
||||
constructor(
|
||||
private registry: ResolverRegistry,
|
||||
private writer: BrainWriter,
|
||||
private budget: BudgetLedger,
|
||||
private scorer: CompletenessScorer,
|
||||
private graph: EntityGraph,
|
||||
) {}
|
||||
|
||||
async enrich(req: EnrichmentRequest): Promise<EnrichmentResult>;
|
||||
async enrichBatch(reqs: EnrichmentRequest[]): Promise<EnrichmentResult[]>;
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 Evidence-weighted completeness (replaces length heuristic)
|
||||
|
||||
Completeness is a per-entity-type rubric, stored in frontmatter on write and recomputed on demand.
|
||||
|
||||
```typescript
|
||||
// src/core/enrichment/completeness.ts
|
||||
export interface CompletenessRubric<Page> {
|
||||
entityType: PageType;
|
||||
dimensions: {
|
||||
name: string;
|
||||
weight: number; // sum must = 1.0
|
||||
check: (page: Page) => number; // 0.0–1.0
|
||||
}[];
|
||||
}
|
||||
|
||||
// Example rubric for persons:
|
||||
// - has_role_and_company 0.20
|
||||
// - has_source_urls 0.20 (≥1 URL with resolver-verified reachability)
|
||||
// - has_timeline_entries 0.15 (≥1)
|
||||
// - has_citations 0.15 (every claim has [Source: ...])
|
||||
// - has_backlinks 0.10 (every linked page links back)
|
||||
// - recency_score 0.10 (last_verified within 90 days)
|
||||
// - non_redundancy 0.10 (no repeated blocks; distinct-lines/total-lines > 0.8)
|
||||
```
|
||||
|
||||
**Key property:** `non_redundancy` + `recency_score` explicitly kill the two brain pathologies observed in the audit (Wilco-style repeating blocks; stale pages without `last_verified`).
|
||||
|
||||
The `completeness` field goes in frontmatter as `0.0–1.0`. It becomes queryable via `list_pages(where: completeness < 0.5)`.
|
||||
|
||||
### 4.4 Tier routing with hard budget
|
||||
|
||||
Two-dimensional routing: **importance** (tier 1/2/3 from person-score) × **budget state**.
|
||||
|
||||
```typescript
|
||||
// src/core/enrichment/tiers.ts
|
||||
export const TIER_CONFIG = {
|
||||
1: { models: ['opus', 'sonar-deep'], maxCostUsd: 0.10, cascadeDepth: 2 },
|
||||
2: { models: ['sonar'], maxCostUsd: 0.02, cascadeDepth: 1 },
|
||||
3: { models: ['sonar'], maxCostUsd: 0.005, cascadeDepth: 0 },
|
||||
};
|
||||
|
||||
// src/core/enrichment/budget.ts
|
||||
export class BudgetLedger {
|
||||
// Hard caps. Queryable pre-resolve.
|
||||
dailyCapUsd: number;
|
||||
perEntityCapUsd: number;
|
||||
perResolverCapUsd: Map<string, number>;
|
||||
|
||||
async reserve(resolverId: string, estimateUsd: number): Promise<Reservation | 'exhausted'>;
|
||||
async commit(reservation: Reservation, actualUsd: number): Promise<void>;
|
||||
async rollback(reservation: Reservation): Promise<void>;
|
||||
async state(): Promise<{ spent: number; remaining: number; perResolver: Record<string, number> }>;
|
||||
}
|
||||
```
|
||||
|
||||
**Property:** if the daily cap is reached, `orchestrator.enrich()` returns `status: 'budget-exhausted'` immediately. No silent overages. Circuit-breaker resets at midnight in the user's configured TZ.
|
||||
|
||||
### 4.5 Cascade (entity graph traversal)
|
||||
|
||||
```typescript
|
||||
// src/core/enrichment/cascade.ts
|
||||
export class EntityGraph {
|
||||
// Deterministic, no LLM. Uses engine.getLinks() + engine.getBacklinks().
|
||||
async neighbors(slug: string, depth: number): Promise<string[]>;
|
||||
async cascadeFrom(trigger: string, depth: number): Promise<EnrichmentRequest[]>;
|
||||
}
|
||||
```
|
||||
|
||||
If person X is enriched and gains a new `company: Acme` field, cascade checks: does `companies/acme` exist? If not, create stub + enqueue at tier 2. Does `companies/acme` link back to X? If not, write the back-link. **Iron Law is machine-enforced, not skill-enforced.**
|
||||
|
||||
### 4.6 Fail-safe transactions
|
||||
|
||||
Every enrichment is wrapped in a BrainWriter transaction (Layer 4). Partial writes are rolled back. No asymmetric state like timeline-entry-without-raw-sidecar.
|
||||
|
||||
```typescript
|
||||
await writer.transaction(async (tx) => {
|
||||
const research = await registry.resolve('perplexity_query', {...}, ctx);
|
||||
await tx.appendTimeline(slug, {...});
|
||||
await tx.putRawData(slug, 'perplexity', research.raw);
|
||||
await tx.setFrontmatterField(slug, 'completeness', score);
|
||||
// All-or-nothing commit on exit.
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Layer 3 — Scheduler
|
||||
|
||||
### 5.1 What's broken today
|
||||
|
||||
Garry's OpenClaw's cron is **externally-driven JSON** (`cron/jobs.json`) with ~30 jobs manually stagger-offset at different minutes. GBrain has **zero native scheduling** — `src/commands/autopilot.ts` is a single daemon loop, and `docs/guides/cron-schedule.md` is architectural guidance, not code.
|
||||
|
||||
Failures observed in Garry's OpenClaw's actual state:
|
||||
- `X OAuth2 Token Refresh`: 11 consecutive timeouts (critical-path silent failure)
|
||||
- `flight-tracker daily scan`: 5 consecutive timeouts
|
||||
- `morning-briefing`: 4 consecutive timeouts
|
||||
- Quiet hours are checked at runtime in skills, so a skill that forgets to check will DM at 3 a.m.
|
||||
- Staggering is manual convention; no protection against two jobs colliding after a config edit.
|
||||
|
||||
### 5.2 ScheduledResolver interface
|
||||
|
||||
```typescript
|
||||
// src/core/scheduling/scheduler.ts
|
||||
export interface Schedule {
|
||||
kind: 'cron' | 'interval';
|
||||
expr?: string; // cron string
|
||||
intervalMs?: number;
|
||||
tz: string; // IANA: "America/Los_Angeles"
|
||||
quietHours?: {
|
||||
startHour: number; // 22 = 10 PM local
|
||||
endHour: number; // 7 = 7 AM local
|
||||
policy: 'skip' | 'defer' | 'silent-run';
|
||||
};
|
||||
staggerKey?: string; // jobs with same key auto-offset
|
||||
maxConcurrent?: number; // global concurrency cap
|
||||
maxDurationMs?: number; // timeout
|
||||
}
|
||||
|
||||
export interface ScheduledResolver extends Resolver<void, ScheduledResult> {
|
||||
schedule: Schedule;
|
||||
retryPolicy: { maxRetries: number; backoffMs: number };
|
||||
circuitBreaker: { failureThreshold: number; cooldownMs: number };
|
||||
state: DurableState; // watermark, content-hash, idempotency key
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Enforcement vs convention (the key delta from Garry's OpenClaw)
|
||||
|
||||
| Concern | Garry's OpenClaw today | Knowledge Runtime |
|
||||
|---|---|---|
|
||||
| Quiet hours | Checked inside each skill (trust-based) | Enforced at scheduler, skill cannot override |
|
||||
| Staggering | Manual minute-offset in `jobs.json` | Scheduler assigns slots via hashed staggerKey |
|
||||
| Concurrency | `MAX_BATCH_PROCESSES=2` in backoff, ignored by cron | Global semaphore in scheduler |
|
||||
| Timeout | Per-job string in JSON, not always respected | Enforced via `AbortController`, timeout raises `TimeoutError` caught by orchestrator |
|
||||
| Retry | None at cron level | `retryPolicy` with exponential backoff |
|
||||
| Silent failure | "11 consecutive timeouts" unnoticed | Circuit breaker opens at threshold → escalation to user |
|
||||
| Idempotency | State files per job, no framework | `DurableState` primitive: watermark/ID/content-hash |
|
||||
|
||||
### 5.4 Native engine + OS cron adapter
|
||||
|
||||
The scheduler runs as either:
|
||||
1. **Embedded** (default for `gbrain autopilot`): native event loop inside the daemon process. One process, many ScheduledResolvers.
|
||||
2. **OS-driven** (for Railway/launchd/systemd): `gbrain schedule run <id>` invoked by OS cron, scheduler state is durable so cross-invocation dedup still works.
|
||||
|
||||
Both modes share the same `Schedule` config + state.
|
||||
|
||||
### 5.5 Observability
|
||||
|
||||
Every scheduled run emits structured events: `started`, `skipped-quiet-hours`, `deferred-to-active-hours`, `failed-retrying`, `circuit-opened`, `completed`. Events go to:
|
||||
- `~/.gbrain/scheduler/events.jsonl` (local, always)
|
||||
- `engine.logIngest` (audit trail in brain DB)
|
||||
- Optional webhook (Slack/Telegram for the user)
|
||||
|
||||
`gbrain doctor` reads the event log and reports: current circuit-breaker state, any resolver with > 3 consecutive failures, any resolver that hasn't fired within 3× its interval (freshness SLA like Garry's OpenClaw's `freshness-check.mjs` but built-in).
|
||||
|
||||
---
|
||||
|
||||
## 6. Layer 4 — Deterministic Output Builder
|
||||
|
||||
### 6.1 The anti-hallucination invariant
|
||||
|
||||
**Iron Law: LLM picks WHAT. Code guarantees WHERE and HOW.**
|
||||
|
||||
Garry's OpenClaw's existing `lib/enrich.mjs:buildTweetEntry` is close to this — tweet URLs are built from `tweet.id` returned by the X API, never from LLM memory. But:
|
||||
|
||||
- A past incident: *"Sub-agent test #2 FAILED — hallucinated 'Philip Leung' entity links across all daily files. LLM rewriting of daily files is too error-prone."* (Garry's OpenClaw memory log, 2026-04-13.)
|
||||
- Back-links depend on `appendTimeline` being called everywhere; skips are silent.
|
||||
- Slug collisions are unchecked (no conflict detection on `slugify`).
|
||||
- Citation format is post-hoc linted weekly, not pre-write enforced.
|
||||
|
||||
### 6.2 BrainWriter
|
||||
|
||||
```typescript
|
||||
// src/core/output/writer.ts
|
||||
export class BrainWriter {
|
||||
constructor(
|
||||
private engine: BrainEngine,
|
||||
private slugRegistry: SlugRegistry,
|
||||
private scaffolder: Scaffolder,
|
||||
) {}
|
||||
|
||||
async transaction<T>(fn: (tx: WriteTx) => Promise<T>): Promise<T>;
|
||||
}
|
||||
|
||||
export interface WriteTx {
|
||||
// High-level typed operations; never raw string writes.
|
||||
createEntity(input: EntityInput): Promise<string>; // returns slug, conflict-checked
|
||||
appendTimeline(slug: string, entry: TimelineInput): Promise<void>;
|
||||
setCompiledTruth(slug: string, body: CompiledTruthInput): Promise<void>;
|
||||
setFrontmatterField(slug: string, key: string, value: unknown): Promise<void>;
|
||||
putRawData(slug: string, source: string, data: object): Promise<void>;
|
||||
addLink(from: string, to: string, context: string): Promise<void>; // auto-creates reverse back-link
|
||||
|
||||
// Validators (called implicitly on commit)
|
||||
validate(): Promise<ValidationReport>;
|
||||
}
|
||||
```
|
||||
|
||||
### 6.3 Scaffolder — deterministic link + citation construction
|
||||
|
||||
Every user-visible URL/link/citation is built by code from resolver outputs, not from LLM text.
|
||||
|
||||
```typescript
|
||||
// src/core/output/scaffold.ts
|
||||
export class Scaffolder {
|
||||
tweetCitation(handle: string, tweetId: string, dateISO: string): string {
|
||||
// "[Source: [X/garrytan, 2026-04-18](https://x.com/garrytan/status/123456)]"
|
||||
}
|
||||
emailCitation(account: string, messageId: string, subject: string): string {
|
||||
// deterministic Gmail URL per OpenClaw pattern
|
||||
}
|
||||
sourceCitation(resolverResult: ResolverResult<unknown>): string {
|
||||
// pulls .source, .fetchedAt, .raw from the result
|
||||
}
|
||||
entityLink(slug: string): string {
|
||||
// slugRegistry checks existence; returns resolvable wikilink
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.4 SlugRegistry — conflict detection
|
||||
|
||||
```typescript
|
||||
// src/core/output/slug-registry.ts
|
||||
export class SlugRegistry {
|
||||
async create(desiredSlug: string, displayName: string, type: PageType): Promise<CreatedSlug>;
|
||||
// Throws SlugCollision if another entity already occupies desiredSlug and isn't
|
||||
// confirmed as the same person (via email / x_handle / disambiguator).
|
||||
// Auto-resolves near-collisions by appending disambiguator.
|
||||
|
||||
async confirmSame(slugA: string, slugB: string, confidence: number): Promise<void>;
|
||||
async merge(canonical: string, duplicate: string): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
### 6.5 Pre-write validators (fail-closed for integrity)
|
||||
|
||||
On `WriteTx.validate()` before commit:
|
||||
|
||||
1. **Citation validator.** Every factual sentence in `compiled_truth` must have an inline `[Source: ...]` within N lines. Non-compliant paragraphs are flagged. Configurable: strict-mode rejects the transaction, lint-mode warns.
|
||||
2. **Link validator.** Every `[text](path)` must point to a page that exists OR to a URL the Scaffolder built (so it's guaranteed-valid). No raw LLM-composed URLs.
|
||||
3. **Back-link validator.** Every outbound link must have a reverse link written in the same transaction.
|
||||
4. **Triple-HR validator.** Compiled truth / timeline split enforced at the schema level.
|
||||
|
||||
**Fails closed**: the default is strict-mode. Loosening requires explicit `writer.transaction({ strictMode: false }, ...)` and logs a warning to the ingest log.
|
||||
|
||||
### 6.6 LLM output sanitization
|
||||
|
||||
Any LLM output destined for a brain page passes through a JSON-Schema-validated parser first. No free-form markdown goes to disk.
|
||||
|
||||
- Entity extraction: JSON array of `{ name, type, context }` per existing `extractEntities` pattern — strict validation.
|
||||
- Compiled-truth synthesis: LLM emits structured `{ sections: [{heading, paragraphs: [{text, sources: [...]}]}]}`, scaffolder renders to markdown.
|
||||
- Timeline entries: LLM emits `{ date, summary, detail, sources }`, scaffolder renders.
|
||||
|
||||
LLM never sees file paths, never writes files, never emits finished markdown.
|
||||
|
||||
---
|
||||
|
||||
## 7. Integration with existing GBrain
|
||||
|
||||
### 7.1 Reuse (already polished)
|
||||
|
||||
| Existing | Used by | Change |
|
||||
|---|---|---|
|
||||
| `src/core/fail-improve.ts` (9/10) | Wraps every Resolver in L1 | None; becomes default wrapper |
|
||||
| `src/core/backoff.ts` (9/10) | ResolverContext.backoff | None |
|
||||
| `src/core/storage.ts` (9/10) | Template for Resolver factory pattern | None; serves as pattern reference |
|
||||
| `src/core/check-resolvable.ts` (9/10) | Extend to validate Resolver plugins | Add `checkResolvers()` mode |
|
||||
| `src/commands/publish.ts` (9/10) | Uses BrainWriter under the hood | Minor: route through L4 |
|
||||
| `src/commands/backlinks.ts` (8/10) | Folded into L4 validator | Keep as CLI-facing lint entry point |
|
||||
| `src/core/operations.ts` validators | Reused in ResolverContext trust enforcement | None |
|
||||
| `src/core/engine.ts` BrainEngine (35 methods) | ResolverContext.engine | Extend with `getResolverRegistry()` |
|
||||
|
||||
### 7.2 Replace (ad-hoc today)
|
||||
|
||||
| Existing | Replace with |
|
||||
|---|---|
|
||||
| `src/core/enrichment-service.ts` (5/10) | `src/core/enrichment/orchestrator.ts` (L2) |
|
||||
| `src/core/embedding.ts` (monolithic) | `src/core/resolvers/builtin/embedding/openai.ts` |
|
||||
| `src/core/transcription.ts` (monolithic) | `src/core/resolvers/builtin/transcription/{groq,openai}.ts` |
|
||||
| `src/commands/integrations.ts` recipe format | Unified Resolver plugin format (§3.5) |
|
||||
| `src/core/data-research.ts` recipe format | Same unified format |
|
||||
| `src/commands/autopilot.ts` hard-coded daemon loop | Wraps a set of ScheduledResolvers |
|
||||
|
||||
### 7.3 Extend
|
||||
|
||||
- `src/core/engine.ts`: add `getResolverRegistry()`, `getWriter()`, `getScheduler()`. Engine becomes the runtime's root container.
|
||||
- `src/core/operations.ts`: `OperationContext` inherits from `ResolverContext` (or vice-versa). Trust flags unified.
|
||||
- `src/core/types.ts`: add `completeness: number` to `Page`, `sourcedBy: string[]` for provenance.
|
||||
|
||||
---
|
||||
|
||||
## 8. Migration Path (phased, shippable)
|
||||
|
||||
Each phase ships independently, passes full E2E, is feature-flagged, and is reversible. No big-bang.
|
||||
|
||||
### Phase 0 — Foundation (human: ~1 wk / CC: ~4 h)
|
||||
- Define `Resolver<I,O>`, `ResolverContext`, `ResolverRegistry`, `ResolverResult` (§3.2–3.4).
|
||||
- Add `src/core/resolvers/index.ts` wiring + tests for registry (register/get/list).
|
||||
- No behavioral change; ship as `v0.11.0-alpha` with feature flag.
|
||||
|
||||
### Phase 1 — Three reference resolvers (human: ~1 wk / CC: ~4 h)
|
||||
- Port `src/core/embedding.ts` → `resolvers/builtin/embedding/openai.ts`.
|
||||
- Implement `resolvers/builtin/brain-local/slug-lookup.ts` (wraps `engine.resolveSlugs`).
|
||||
- Implement `resolvers/builtin/url-reachable.ts` (HEAD-check).
|
||||
- Prove the interface: old callers swap to `registry.resolve('openai_embedding', ...)`.
|
||||
|
||||
### Phase 2 — BrainWriter + Slug Registry (human: ~1.5 wk / CC: ~6 h)
|
||||
- L4 core: `BrainWriter.transaction`, `Scaffolder`, `SlugRegistry` with conflict detection.
|
||||
- Pre-write validators: citation, link, back-link, triple-HR.
|
||||
- Migrate `src/commands/publish.ts` + `src/commands/backlinks.ts` to route through BrainWriter.
|
||||
- **Now** Garry's OpenClaw's "Philip Leung" hallucination is structurally impossible — LLM output passes through JSON-Schema validator before reaching Scaffolder.
|
||||
|
||||
### Phase 3 — `gbrain integrity` command (human: ~0.5 wk / CC: ~2 h)
|
||||
- Ship the originally-scoped user-facing feature on top of the new foundation.
|
||||
- Uses Resolver SDK: `x_handle_to_tweet` + `url_reachable`.
|
||||
- Uses BrainWriter: all auto-repairs go through validated writes.
|
||||
- `--auto --confidence 0.8` mode as user approved in cherry-pick #1.
|
||||
- **User-visible value ships in Phase 3, not Phase 7.**
|
||||
|
||||
### Phase 4 — Enrichment Orchestrator (human: ~2 wk / CC: ~8 h)
|
||||
- L2 core: `EnrichmentOrchestrator`, `BudgetLedger`, `CompletenessScorer`, `EntityGraph.cascadeFrom`.
|
||||
- Migrate `src/core/enrichment-service.ts` callers (deprecate the old file after).
|
||||
- Completeness score in frontmatter on every write (dogfooding cascades).
|
||||
|
||||
### Phase 5 — Scheduler (human: ~2 wk / CC: ~8 h)
|
||||
- L3 core: `Scheduler`, `ScheduledResolver`, `DurableState`, circuit breaker, quiet-hours enforcer.
|
||||
- Migrate `src/commands/autopilot.ts` to a ScheduledResolver set.
|
||||
- Ship `gbrain schedule list|run|pause|tail` CLI for observability.
|
||||
|
||||
### Phase 6 — Port 5–8 OpenClaw resolvers (human: ~1.5 wk / CC: ~6 h)
|
||||
- `perplexity_query`, `text_to_entities`, `mistral_ocr_pdf`, `x_search_all`, `x_user_to_tweets`, `gmail_query_to_threads`, `calendar_date_to_events`.
|
||||
- Each ships as YAML + TS module under `resolvers/builtin/` — **proof of the plugin format.**
|
||||
|
||||
### Phase 7 — OpenClaw Adoption Integration (human: ~1 wk / CC: ~4 h)
|
||||
- Write `docs/openclaw/ADOPTION.md` showing your OpenClaw how to replace its 69 bespoke scripts with calls to `gbrain registry.resolve(...)`.
|
||||
- Ship a `gbrain claw-bridge` subcommand that proxies Garry's OpenClaw's current script invocations to the resolver registry — zero-edit adoption path.
|
||||
- **This is the test of the north star.** If your OpenClaw can stand up a 1-line shim and drop `scripts/x-api-client.mjs`, the abstraction succeeded.
|
||||
|
||||
Total: human: ~10 weeks / CC: ~42 hours / calendar with single implementer: ~3–4 weeks.
|
||||
|
||||
---
|
||||
|
||||
## 9. Critical Files
|
||||
|
||||
### New directories / files
|
||||
|
||||
```
|
||||
src/core/
|
||||
runtime/
|
||||
index.ts # RuntimeContext (engine, storage, config, logger, metrics, budget)
|
||||
registry.ts # ResolverRegistry
|
||||
factory.ts # createResolver()
|
||||
resolvers/
|
||||
interface.ts # Resolver<I, O>
|
||||
fail-improve-wrapper.ts # auto-wraps every resolver in FailImproveLoop
|
||||
builtin/
|
||||
x-api/
|
||||
handle-to-tweet.ts
|
||||
handle-to-tweet.yaml
|
||||
perplexity/
|
||||
query.ts
|
||||
query.yaml
|
||||
brain-local/
|
||||
slug-lookup.ts
|
||||
url-reachable.ts
|
||||
embedding/
|
||||
openai.ts # refactored from src/core/embedding.ts
|
||||
transcription/
|
||||
groq.ts
|
||||
openai.ts
|
||||
enrichment/
|
||||
orchestrator.ts # EnrichmentOrchestrator
|
||||
tiers.ts # TIER_CONFIG
|
||||
budget.ts # BudgetLedger
|
||||
completeness.ts # CompletenessScorer + per-type rubrics
|
||||
cascade.ts # EntityGraph
|
||||
scheduling/
|
||||
scheduler.ts # Scheduler + ScheduledResolver
|
||||
schedule.ts # Schedule type, cron expr parser
|
||||
state.ts # DurableState primitives
|
||||
quiet-hours.ts # TZ-aware enforcement
|
||||
stagger.ts # deterministic slot assignment
|
||||
output/
|
||||
writer.ts # BrainWriter
|
||||
scaffold.ts # Scaffolder (typed URL builders)
|
||||
slug-registry.ts # SlugRegistry (conflict detection)
|
||||
validators/
|
||||
citation.ts
|
||||
link.ts
|
||||
back-link.ts
|
||||
triple-hr.ts
|
||||
|
||||
src/commands/
|
||||
integrity.ts # ships in Phase 3, replaces Feynman Phase A/B
|
||||
schedule.ts # gbrain schedule list|run|pause|tail (Phase 5)
|
||||
|
||||
docs/openclaw/
|
||||
ADOPTION.md # written in Phase 7
|
||||
```
|
||||
|
||||
### Replaced / removed
|
||||
- `src/core/enrichment-service.ts` — folded into `enrichment/orchestrator.ts`
|
||||
- `src/core/embedding.ts` — moved into `resolvers/builtin/embedding/openai.ts`
|
||||
- `src/core/transcription.ts` — moved into `resolvers/builtin/transcription/`
|
||||
|
||||
### Extended
|
||||
- `src/core/engine.ts` — add `getResolverRegistry()`, `getWriter()`, `getScheduler()`
|
||||
- `src/core/operations.ts` — unify with ResolverContext; every operation validator reusable by resolvers
|
||||
- `src/core/types.ts` — add `completeness: number`, `sourcedBy: string[]`, `lastVerified: Date`
|
||||
|
||||
---
|
||||
|
||||
## 10. Testing Strategy
|
||||
|
||||
### Contract tests
|
||||
Every Resolver implementation tested against the interface spec. Table-driven: run the same suite against `openai_embedding`, `x_handle_to_tweet`, etc. Ensures plugin authors can't ship broken resolvers.
|
||||
|
||||
### Property tests
|
||||
- **Idempotency:** running a ScheduledResolver twice with the same state produces the same output and doesn't double-write.
|
||||
- **Atomicity:** a BrainWriter transaction that throws mid-flight leaves the brain bit-for-bit identical to pre-transaction.
|
||||
- **Deterministic scaffolds:** given the same resolver outputs, the Scaffolder produces byte-identical citations/links.
|
||||
|
||||
### Integration tests
|
||||
- `EnrichmentOrchestrator` end-to-end against PGLite (in-memory, no API keys) with mocked resolver registry.
|
||||
- `Scheduler` with fake clock + quiet-hours scenarios.
|
||||
- BrainWriter transaction rollback on validator failure.
|
||||
|
||||
### Chaos tests
|
||||
- Kill the process mid-enrichment; next run must resume cleanly.
|
||||
- Simulate API timeout mid-transaction; transaction must roll back completely.
|
||||
- Corrupted state file; scheduler must escalate, not silently skip.
|
||||
|
||||
### Regression tests vs. Garry's OpenClaw behavior
|
||||
For each OpenClaw pattern we port (e.g. X-handle → tweet URL), a regression test proves the new resolver produces the same answer on real-world inputs from the brain audit. This is the "your OpenClaw would adopt" proof.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open Questions (flagged for CEO re-review)
|
||||
|
||||
1. **Scope shape.** Is this the right four-layer decomposition, or are some layers better left to OpenClaw (e.g. Scheduling lives above GBrain, not in it)?
|
||||
2. **Phase 3 user-value break.** Does Phase 3 (user-visible `gbrain integrity`) ship early enough, or do we need an even smaller MVP?
|
||||
3. **LLM-as-resolver.** Should `text_to_entities` be a Resolver, or does that blur the "code vs LLM" line the invariant relies on?
|
||||
4. **Plugin format.** YAML + TS module (§3.5) vs. pure TS module with decorator-style metadata. Latter is more type-safe; former is more discoverable.
|
||||
5. **Cross-resolver transactions.** Do we support "atomic fetch-from-Perplexity + write-to-brain" at the L2 layer? Current design says yes; implementation is tricky (Perplexity call isn't rollbackable).
|
||||
6. **OpenClaw bridge scope.** Phase 7 `gbrain claw-bridge` — is that worth a phase of its own, or should adoption be documentation-only?
|
||||
7. **Completeness rubric coverage.** Do we define rubrics for all 9 PageTypes upfront, or ship people/company/meeting first and extend incrementally?
|
||||
8. **Budget config UX.** Hard daily cap is strict; should we also expose a soft-cap warning mode, and how is the cap set (env var? config file? prompt on first use?)
|
||||
9. **Backwards compat.** `src/commands/publish.ts` and `src/commands/backlinks.ts` have been running cleanly for weeks. Refactoring through BrainWriter carries migration risk. Acceptable?
|
||||
10. **Existing TODOS alignment.** `TODOS.md` has P0 "Runtime MCP access control" and P2 security hardening. The new RuntimeContext.remote flag interacts with both — do we fold MCP access control into Phase 0 or keep separate?
|
||||
|
||||
---
|
||||
|
||||
## 12. Verification (the "your OpenClaw would adopt" test)
|
||||
|
||||
The design succeeds iff:
|
||||
|
||||
- [ ] A user can add a new resolver by dropping a YAML + TS module in `~/.gbrain/resolvers/` without editing GBrain source.
|
||||
- [ ] Your OpenClaw can delete `scripts/x-api-client.mjs` and replace all callers with 1-line `await registry.resolve('x_handle_to_tweet', ...)`.
|
||||
- [ ] No brain page can be written with a bare tweet reference, a missing back-link, or an unverified URL (validators catch it pre-commit).
|
||||
- [ ] Running `gbrain integrity --auto --confidence 0.8` over a real brain fixes ≥1,000 of the 1,424 known bare-tweet citations without human review.
|
||||
- [ ] Full E2E test suite passes on both PGLite + Postgres engines.
|
||||
- [ ] The Knowledge Runtime ships across 7 phases with each phase individually shippable and reversible.
|
||||
@@ -1,448 +0,0 @@
|
||||
---
|
||||
status: ACTIVE
|
||||
---
|
||||
# CEO Plan: Minions as Universal Agent Orchestration Protocol
|
||||
Generated by /plan-ceo-review on 2026-04-15
|
||||
Branch: garrytan/minions-jobs | Mode: SCOPE EXPANSION
|
||||
Repo: garrytan/gbrain
|
||||
|
||||
## Vision
|
||||
|
||||
### 10x Check
|
||||
Instead of "GBrain has a queue, OpenClaw uses it," make Minions a universal agent
|
||||
orchestration protocol. Any platform (OpenClaw, Hermes, Claude Code, Codex, custom
|
||||
scripts) submits, monitors, steers, and composes agents through the same Postgres-native
|
||||
protocol. GBrain IS the agent control plane.
|
||||
|
||||
### Platonic Ideal (aspirational North Star, NOT in v1 scope)
|
||||
Open a terminal, type `gbrain jobs dashboard`. See every agent across every platform.
|
||||
Their progress, tool calls, token spend. Click any agent for full execution trace.
|
||||
Type a message to redirect a running agent mid-flight. See the governor's decisions
|
||||
visualized. Run A/B tests between agent configurations. The feeling: complete
|
||||
situational awareness of your AI workforce.
|
||||
|
||||
**Note:** The dashboard, A/B testing, and visual governor are future phases. This plan
|
||||
builds the primitives they would sit on top of: real-time events, structured progress,
|
||||
token accounting, inbox with ack, and session transcripts.
|
||||
|
||||
## Scope Decisions
|
||||
|
||||
| # | Proposal | Effort | Decision | Reasoning |
|
||||
|---|----------|--------|----------|-----------|
|
||||
| 1 | pg LISTEN/NOTIFY real-time events | S | ACCEPTED | Sub-second event delivery vs 5s polling. Every platform benefits. |
|
||||
| 2 | Structured progress protocol | S | ACCEPTED | Standard progress makes unified dashboard possible. |
|
||||
| 3 | Job cost tracking (token accounting) | M | ACCEPTED | Token cost is #1 thing users want to know about agent work. |
|
||||
| 4 | Job replay | S | ACCEPTED | Small surface area, high utility for debugging failures. |
|
||||
| 5 | Job groups / waves | M | DEFERRED | Parent-child already provides grouping. Overlap concern. |
|
||||
| 6 | Inbox acknowledgment (read receipts) | S | ACCEPTED | Without it, inbox is fire-and-forget — same problem we're fixing. |
|
||||
| 7 | Universal agent protocol | S | ACCEPTED | Design framing, not extra code. Platform-agnostic naming/docs. |
|
||||
| 8 | Session transcript capture | M | ACCEPTED | Full audit trail of every agent run. |
|
||||
|
||||
## Accepted Scope — Implementation Detail
|
||||
|
||||
### 0a. Pause/resume (from base plan)
|
||||
|
||||
**Schema:** Add `'paused'` to `MinionJobStatus` (already in migration v6 constraint).
|
||||
|
||||
**New methods:**
|
||||
- `MinionQueue.pauseJob(id): MinionJob | null`
|
||||
Transitions `waiting` or `active` → `paused`. For `active` jobs, clears `lock_token`
|
||||
and `lock_until` (worker will detect lock loss and stop). Returns null if job not
|
||||
in pausable state.
|
||||
- `MinionQueue.resumeJob(id): MinionJob | null`
|
||||
Transitions `paused` → `waiting`. Resets for claiming. Returns null if not paused.
|
||||
|
||||
**Worker integration:** Worker's lock renewal loop checks `isActive()`. When a job
|
||||
is paused, the lock is cleared, so `renewLock()` returns false and the worker stops
|
||||
execution gracefully (same path as stall detection). The job's progress and state
|
||||
are preserved in the DB for when it resumes.
|
||||
|
||||
**MCP operations:** `pause_job`, `resume_job` (added in Step 3 of implementation plan).
|
||||
|
||||
**PGLite compatibility:** Full.
|
||||
|
||||
### 0b. Resource governor (from base plan)
|
||||
|
||||
**New file:** `src/core/minions/governor.ts`
|
||||
|
||||
```typescript
|
||||
interface GovernorConfig {
|
||||
maxConcurrency: number; // ceiling
|
||||
minConcurrency: number; // floor (default 1)
|
||||
checkIntervalMs: number; // default 10000
|
||||
cpuThreshold: number; // default 0.80 (80%)
|
||||
memoryThreshold: number; // default 0.85 (85%)
|
||||
circuitBreakerMemory: number; // default 0.90 (90%)
|
||||
}
|
||||
|
||||
class ResourceGovernor {
|
||||
getEffectiveConcurrency(): number; // current allowed concurrency
|
||||
start(): void; // begin polling system metrics
|
||||
stop(): void; // stop polling
|
||||
onCircuitBreak(cb: (jobId) => void): void; // kill callback
|
||||
}
|
||||
```
|
||||
|
||||
**System metrics:** Reuse `getSystemLoad()` from `src/core/backoff.ts` (already
|
||||
implements CPU and memory checks). Add event loop lag measurement via
|
||||
`perf_hooks.monitorEventLoopDelay()`.
|
||||
|
||||
**Worker integration:** `MinionWorker.start()` consults `governor.getEffectiveConcurrency()`
|
||||
before claiming new jobs. If current in-flight count >= effective concurrency, skip claim.
|
||||
|
||||
**Circuit breaker:** If memory > 90%, governor calls `onCircuitBreak` with the
|
||||
lowest-priority active job ID. Worker cancels that job via `failJob()` with
|
||||
`UnrecoverableError("circuit breaker: memory pressure")`.
|
||||
|
||||
**Prerequisite:** Concurrent job processing must be implemented first (see
|
||||
Concurrency Note below).
|
||||
|
||||
**PGLite compatibility:** Full (governor is app-level, not DB-level).
|
||||
|
||||
### 1. pg LISTEN/NOTIFY (real-time events)
|
||||
|
||||
**Schema:** No new columns. Add NOTIFY triggers to state transitions.
|
||||
|
||||
**SQL trigger:**
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
PERFORM pg_notify('minion_jobs', json_build_object(
|
||||
'id', NEW.id, 'status', NEW.status, 'name', NEW.name,
|
||||
'queue', NEW.queue, 'prev_status', COALESCE(OLD.status, 'new')
|
||||
)::text);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER minion_job_notify AFTER INSERT OR UPDATE OF status ON minion_jobs
|
||||
FOR EACH ROW EXECUTE FUNCTION notify_minion_job_change();
|
||||
```
|
||||
|
||||
**New method:** `MinionQueue.subscribe(callback: (event) => void): () => void`
|
||||
Returns unsubscribe function. Requires direct Postgres connection (NOT pooled).
|
||||
|
||||
**PGLite compatibility:** PGLite does NOT support LISTEN/NOTIFY. Fallback: polling
|
||||
via `getJob()` at configurable interval (default 2s). The `subscribe()` method
|
||||
detects engine type and uses polling fallback automatically.
|
||||
|
||||
**Supabase constraint:** Requires direct connection (port 5432), not pgBouncer
|
||||
pooler (port 6543). Document in skill file and setup guide.
|
||||
|
||||
### 2. Structured progress protocol
|
||||
|
||||
**TypeScript interface (convention, not enforced at DB level):**
|
||||
```typescript
|
||||
interface AgentProgress {
|
||||
step: number; // current step (1-based)
|
||||
total: number; // total expected steps (0 = unknown)
|
||||
message: string; // human-readable status
|
||||
tokens_in: number; // cumulative input tokens
|
||||
tokens_out: number; // cumulative output tokens
|
||||
last_tool: string; // name of last tool called
|
||||
started_at: string; // ISO 8601 when this step started
|
||||
}
|
||||
```
|
||||
|
||||
**Storage:** Existing `progress JSONB` column. No schema change needed.
|
||||
Handlers use `ctx.updateProgress(agentProgress)`. Non-agent jobs can use
|
||||
any JSONB shape (backward compatible).
|
||||
|
||||
**Validation:** `updateProgress()` accepts any JSONB. The `AgentProgress`
|
||||
interface is a convention enforced by the agent handler, not by the queue.
|
||||
|
||||
### 3. Job cost tracking (token accounting)
|
||||
|
||||
**Schema changes (migration v6):**
|
||||
```sql
|
||||
ALTER TABLE minion_jobs ADD COLUMN tokens_input INTEGER DEFAULT 0;
|
||||
ALTER TABLE minion_jobs ADD COLUMN tokens_output INTEGER DEFAULT 0;
|
||||
ALTER TABLE minion_jobs ADD COLUMN tokens_cache_read INTEGER DEFAULT 0;
|
||||
ALTER TABLE minion_jobs ADD COLUMN cost_usd NUMERIC(10,6) DEFAULT 0;
|
||||
```
|
||||
|
||||
**New method:** `MinionQueue.updateTokens(id, lockToken, { input, output, cache_read, cost_usd })`
|
||||
Accumulates (adds to existing values, does not replace).
|
||||
|
||||
**Parent rollup:** When `completeJob()` is called, if `parent_job_id` is set,
|
||||
add this job's token counts to the parent's via:
|
||||
```sql
|
||||
UPDATE minion_jobs SET
|
||||
tokens_input = tokens_input + $child_input,
|
||||
tokens_output = tokens_output + $child_output,
|
||||
tokens_cache_read = tokens_cache_read + $child_cache,
|
||||
cost_usd = cost_usd + $child_cost
|
||||
WHERE id = $parent_id;
|
||||
```
|
||||
|
||||
**PGLite compatibility:** Full support (standard columns).
|
||||
|
||||
### 4. Job replay
|
||||
|
||||
**New method:** `MinionQueue.replayJob(id, dataOverrides?: Record<string, unknown>): MinionJob`
|
||||
|
||||
Implementation: Read the completed/failed/dead job. Create a NEW job with:
|
||||
- Same `name`, `queue`, `priority`, `max_attempts`, `backoff_type`, `backoff_delay`
|
||||
- `data` = deep merge of original data + overrides
|
||||
- Fresh `attempts_made: 0`, `status: 'waiting'`
|
||||
- `parent_job_id` = null (replay is a new top-level job, not a child)
|
||||
- Does NOT clone children (replay is a single job, not a DAG)
|
||||
|
||||
**Constraint:** Only works on terminal statuses (completed/failed/dead).
|
||||
Returns the new job record.
|
||||
|
||||
**Idempotency:** Each replay creates a distinct new job. No deduplication.
|
||||
If the original had side effects, the replay may repeat them. Document this
|
||||
in the skill file as a user responsibility.
|
||||
|
||||
### 5. Inbox (sidechannel messaging)
|
||||
|
||||
**Schema changes (migration v6):**
|
||||
```sql
|
||||
ALTER TABLE minion_jobs ADD COLUMN inbox JSONB DEFAULT '[]';
|
||||
```
|
||||
|
||||
**Inbox message format:**
|
||||
```typescript
|
||||
interface InboxMessage {
|
||||
id: string; // UUIDv4
|
||||
sent_at: string; // ISO 8601
|
||||
read_at: string | null; // null until worker reads it
|
||||
sender: string; // 'parent' | 'user' | job ID
|
||||
payload: unknown; // arbitrary directive
|
||||
}
|
||||
```
|
||||
|
||||
**New methods:**
|
||||
- `MinionQueue.sendMessage(jobId, payload, sender?): InboxMessage`
|
||||
Appends message to inbox array via atomic JSONB append
|
||||
(`inbox = inbox || $1::jsonb`), not read-modify-write. Returns the message with id + sent_at.
|
||||
- `MinionQueue.readInbox(jobId, lockToken): InboxMessage[]`
|
||||
Returns unread messages (read_at = null). Marks them as read (sets read_at).
|
||||
Token-fenced: only the worker holding the lock can read.
|
||||
|
||||
**Worker integration:** Agent handler calls `readInbox()` on each iteration.
|
||||
If messages exist, injects them into the agent's context as system messages.
|
||||
|
||||
**PGLite compatibility:** Full support (standard JSONB column).
|
||||
|
||||
### 6. Inbox acknowledgment (read receipts)
|
||||
|
||||
Built into the inbox design above. The `read_at` field on each `InboxMessage`
|
||||
provides the receipt. `sendMessage()` returns the message ID; the sender can
|
||||
later check `getJob(id)` and inspect `inbox` to see which messages have been
|
||||
read.
|
||||
|
||||
No additional schema or methods needed beyond what's in #5.
|
||||
|
||||
### 7. Universal agent protocol (platform-agnostic framing)
|
||||
|
||||
**This is a design decision, not code.** It means:
|
||||
|
||||
1. The skill file (`skills/minion-orchestrator/SKILL.md`) is written for ANY
|
||||
agent platform, not just OpenClaw. Examples show MCP tool calls, not
|
||||
OpenClaw-specific commands.
|
||||
|
||||
2. The agent handler (`agent-handler.ts`) accepts a generic interface:
|
||||
```typescript
|
||||
interface AgentJobData {
|
||||
prompt: string;
|
||||
tools?: string[]; // MCP tool names
|
||||
model?: string; // e.g., 'claude-opus-4-6', 'gpt-4o'
|
||||
context?: string; // additional context
|
||||
platform?: string; // 'openclaw' | 'hermes' | 'claude-code' | 'custom'
|
||||
max_iterations?: number; // agent loop budget
|
||||
}
|
||||
```
|
||||
|
||||
3. The OpenClaw plugin is ONE consumer. Hermes, Claude Code extensions,
|
||||
or custom scripts can submit `agent` jobs through the same MCP operations.
|
||||
|
||||
4. **NOT in v1 scope:** Multi-tenant auth, cross-network connectivity,
|
||||
protocol versioning, API key isolation. These are Phase 2 concerns when
|
||||
actual multi-platform usage materializes. v1 is single-user, single-brain.
|
||||
|
||||
### Agent Handler Architecture (critical design decision)
|
||||
|
||||
The agent handler does NOT live in GBrain. GBrain provides the queue infrastructure
|
||||
and a clean handler contract. The actual agent execution lives in the platform plugin.
|
||||
|
||||
```
|
||||
GBrain (this repo):
|
||||
MinionQueue — queue/claim/complete/inbox/tokens/NOTIFY
|
||||
MinionWorker — poll/lock/stall/governor framework
|
||||
Handler contract — AgentJobData interface + MinionJobContext
|
||||
|
||||
OpenClaw plugin (separate repo):
|
||||
Registers "agent" handler with MinionWorker
|
||||
Handler calls OpenClaw's PI agent core (the actual LLM loop)
|
||||
Each iteration: readInbox → inject as system message, updateProgress, updateTokens
|
||||
Completion: store result + session transcript in job.result + job.stacktrace
|
||||
|
||||
GBrain ships a test/echo handler for unit testing only.
|
||||
```
|
||||
|
||||
**Handler contract (GBrain side):**
|
||||
```typescript
|
||||
// The handler receives this context (already exists in worker.ts)
|
||||
interface MinionJobContext {
|
||||
id: number;
|
||||
name: string;
|
||||
data: Record<string, unknown>; // AgentJobData when name="agent"
|
||||
attempts_made: number;
|
||||
updateProgress(progress: unknown): Promise<void>;
|
||||
updateTokens(tokens: TokenUpdate): Promise<void>; // NEW
|
||||
log(message: string | TranscriptEntry): Promise<void>;
|
||||
isActive(): Promise<boolean>;
|
||||
readInbox(): Promise<InboxMessage[]>; // NEW
|
||||
}
|
||||
```
|
||||
|
||||
**Why this is right:** GBrain is orchestration, not execution. OpenClaw has the
|
||||
PI agent core. Hermes has AIAgent. Claude Code has its own loop. Each platform
|
||||
brings its own engine and registers a handler. GBrain manages lifecycle, progress,
|
||||
steering, cost tracking, and persistence around it.
|
||||
|
||||
### 8. Session transcript capture
|
||||
|
||||
**Extends existing stacktrace mechanism.** The `stacktrace` field (JSONB array
|
||||
of strings) already captures log messages. Session transcripts use the same
|
||||
field with structured entries:
|
||||
|
||||
```typescript
|
||||
type TranscriptEntry =
|
||||
| { type: 'log'; message: string; ts: string }
|
||||
| { type: 'tool_call'; tool: string; args_size: number; result_size: number; ts: string }
|
||||
| { type: 'llm_turn'; model: string; tokens_in: number; tokens_out: number; ts: string }
|
||||
| { type: 'error'; message: string; stack?: string; ts: string };
|
||||
```
|
||||
|
||||
**Storage:** Existing `stacktrace JSONB` column. No schema change.
|
||||
The agent handler appends `TranscriptEntry` objects instead of plain strings.
|
||||
Backward compatible: non-agent jobs continue appending strings.
|
||||
|
||||
**Size concern:** Long agent runs could generate large transcripts. Add a
|
||||
`max_transcript_entries` option (default 1000) that rotates oldest entries
|
||||
when exceeded (FIFO). The full transcript for forensic analysis can be
|
||||
stored as a brain file via `gbrain files upload-raw`.
|
||||
|
||||
## Schema Migration v6
|
||||
|
||||
All schema changes are additive (ALTER TABLE ADD COLUMN). No backfill needed.
|
||||
Existing jobs continue to work with default values.
|
||||
|
||||
```sql
|
||||
-- Migration v6: Agent orchestration primitives
|
||||
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS tokens_input INTEGER DEFAULT 0;
|
||||
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS tokens_output INTEGER DEFAULT 0;
|
||||
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS tokens_cache_read INTEGER DEFAULT 0;
|
||||
|
||||
-- Separate inbox table (not JSONB on job row)
|
||||
CREATE TABLE IF NOT EXISTS minion_inbox (
|
||||
id SERIAL PRIMARY KEY,
|
||||
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
||||
sender TEXT NOT NULL,
|
||||
payload JSONB NOT NULL,
|
||||
sent_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
read_at TIMESTAMPTZ
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_minion_inbox_unread
|
||||
ON minion_inbox (job_id) WHERE read_at IS NULL;
|
||||
|
||||
-- Status constraint update: add 'paused'
|
||||
ALTER TABLE minion_jobs DROP CONSTRAINT IF EXISTS minion_jobs_status_check;
|
||||
ALTER TABLE minion_jobs ADD CONSTRAINT minion_jobs_status_check
|
||||
CHECK (status IN ('waiting','active','completed','failed','delayed','dead','cancelled','waiting-children','paused'));
|
||||
|
||||
-- NOTIFY trigger for real-time events (Postgres only, not PGLite)
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
PERFORM pg_notify('minion_jobs', json_build_object(
|
||||
'id', NEW.id, 'status', NEW.status, 'name', NEW.name,
|
||||
'queue', NEW.queue, 'prev_status', COALESCE(OLD.status, 'new')
|
||||
)::text);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER minion_job_notify AFTER INSERT OR UPDATE OF status ON minion_jobs
|
||||
FOR EACH ROW EXECUTE FUNCTION notify_minion_job_change();
|
||||
```
|
||||
|
||||
## PGLite Compatibility Matrix
|
||||
|
||||
| Feature | Postgres | PGLite | Fallback |
|
||||
|---|---|---|---|
|
||||
| Pause/resume | Full | Full | — |
|
||||
| Inbox + ack | Full | Full | — |
|
||||
| Token accounting | Full | Full | — |
|
||||
| Job replay | Full | Full | — |
|
||||
| LISTEN/NOTIFY | Full | NO | Polling (2s interval) |
|
||||
| NOTIFY trigger | Full | NO | Skipped in PGLite schema |
|
||||
| Structured progress | Full | Full | — |
|
||||
| Session transcripts | Full | Full | — |
|
||||
| Resource governor | Full | Full | — |
|
||||
| Worker daemon | Full | NO (existing limitation) | — |
|
||||
|
||||
## Concurrency Note
|
||||
|
||||
The current `MinionWorker.start()` processes jobs sequentially (one at a time)
|
||||
despite `concurrency` being declared in `MinionWorkerOpts`. Implementing actual
|
||||
concurrent job processing (Promise pool) is a prerequisite for the resource
|
||||
governor to be meaningful. The governor adjusts effective concurrency, which
|
||||
requires actual concurrent processing to exist.
|
||||
|
||||
**Action:** Implement concurrent job processing in `worker.ts` before or as
|
||||
part of the governor step. Use a semaphore pattern: maintain up to N in-flight
|
||||
promises, claim new jobs as slots free up.
|
||||
|
||||
## Outside Voice Decisions (from adversarial review)
|
||||
|
||||
1. **AbortController for pause/resume** — Handler contract gets `signal: AbortSignal`.
|
||||
Pause clears lock AND signals abort. Handler must check `signal.aborted` on each
|
||||
iteration. Without this, pausing active jobs creates duplicate execution.
|
||||
|
||||
2. **Drop cost_usd column** — Token counts (input/output/cache_read) are stable facts.
|
||||
USD pricing is volatile. Compute cost at display/read time from a pricing table,
|
||||
not at write time. Removes `cost_usd NUMERIC(10,6)` from migration v6.
|
||||
|
||||
3. **Separate minion_inbox table** — Instead of JSONB array on job row, use a dedicated
|
||||
table for inbox messages. Avoids row bloat from rewriting entire inbox on every send.
|
||||
Properly concurrent-safe with standard INSERT (no JSONB append concerns).
|
||||
```sql
|
||||
CREATE TABLE minion_inbox (
|
||||
id SERIAL PRIMARY KEY,
|
||||
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
||||
sender TEXT NOT NULL,
|
||||
payload JSONB NOT NULL,
|
||||
sent_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
read_at TIMESTAMPTZ
|
||||
);
|
||||
CREATE INDEX idx_minion_inbox_unread ON minion_inbox (job_id) WHERE read_at IS NULL;
|
||||
```
|
||||
|
||||
4. **One release, not two** — Ship all features in one migration (v6). User prefers
|
||||
cohesive release over incremental delivery for this feature set.
|
||||
|
||||
5. **Selective column projection** — Fix SELECT * queries in getJobs(), claim(),
|
||||
handleStalled() to exclude stacktrace column. Include stacktrace only in getJob()
|
||||
detail view. Prevents transcript bloat from affecting query performance.
|
||||
|
||||
## Future Phases (accepted trajectory)
|
||||
|
||||
- **Phase 2: Dashboard CLI** — `gbrain jobs dashboard` live TUI showing all agents.
|
||||
Enabled by: LISTEN/NOTIFY, structured progress, token accounting.
|
||||
- **Phase 3: Multi-tenant auth** — Runtime MCP access control, per-platform API keys.
|
||||
Enabled by: platform-agnostic framing, sender validation on inbox.
|
||||
- **Phase 4: Agent composition patterns** — Map-reduce, pipeline, approval gates as
|
||||
first-class primitives. Enabled by: parent-child DAGs, inbox sidechannel.
|
||||
|
||||
## Deferred to TODOS.md
|
||||
- Job groups / waves (parent-child covers this; revisit if real grouping need emerges)
|
||||
- cost_usd column (compute from pricing table at read time when pricing API exists)
|
||||
|
||||
## Key Premises Confirmed
|
||||
1. GBrain is intentionally evolving from knowledge brain to agent infrastructure (user confirmed)
|
||||
2. Coupling between OpenClaw and GBrain's Postgres is acceptable (OpenClaw already depends on GBrain)
|
||||
3. Full Infrastructure approach (all 8+ steps) selected over Minimal Viable or Sidecar Tracking
|
||||
4. Prior learning [agent-dx-instruction-layer] validates that the teaching layer (skill + evals) is mandatory
|
||||
@@ -1,13 +0,0 @@
|
||||
# Procfile — Render / Railway / Heroku.
|
||||
#
|
||||
# Fly.io users: see fly.toml.partial instead.
|
||||
#
|
||||
# Set secrets via the platform's env UI or CLI (e.g. `heroku config:set`,
|
||||
# `render env:set`, `railway variables set`). At minimum:
|
||||
# DATABASE_URL=postgresql://...
|
||||
# GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
|
||||
|
||||
# Two-layer supervision: the platform restarts the container on host
|
||||
# events (OOM, deploy); `gbrain jobs supervisor` restarts the worker
|
||||
# on in-process crashes with exponential backoff.
|
||||
worker: gbrain jobs supervisor --concurrency 2
|
||||
@@ -1,24 +0,0 @@
|
||||
# fly.toml — partial. Merge into your existing fly.toml.
|
||||
#
|
||||
# Set secrets once (never commit them):
|
||||
# fly secrets set DATABASE_URL='postgresql://user:pass@host:6543/db?prepare=false'
|
||||
# fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
|
||||
# fly secrets set ANTHROPIC_API_KEY=... # optional
|
||||
#
|
||||
# Two-layer supervision: Fly restarts the VM on host events; the
|
||||
# `gbrain jobs supervisor` process restarts the worker on in-process
|
||||
# crashes with exponential backoff and a structured audit trail.
|
||||
|
||||
[processes]
|
||||
worker = "gbrain jobs supervisor --concurrency 2"
|
||||
|
||||
# Scale the worker process to 1 machine (job queue serializes work; more
|
||||
# machines means higher concurrency but also more Postgres connections).
|
||||
# fly scale count worker=1
|
||||
|
||||
# If you want the worker in its own VM size:
|
||||
# [[vm]]
|
||||
# processes = ["worker"]
|
||||
# memory = "512mb"
|
||||
# cpu_kind = "shared"
|
||||
# cpus = 1
|
||||
@@ -1,35 +0,0 @@
|
||||
# /etc/gbrain.env — secrets + env for the gbrain worker.
|
||||
#
|
||||
# Install:
|
||||
# sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
|
||||
# gbrain.env.example /etc/gbrain.env
|
||||
# sudoedit /etc/gbrain.env # fill in real values
|
||||
#
|
||||
# Referenced from crontab via BASH_ENV=/etc/gbrain.env, or from systemd
|
||||
# via EnvironmentFile=/etc/gbrain.env. Never commit real secrets.
|
||||
|
||||
# --- Required ---------------------------------------------------------------
|
||||
|
||||
# Postgres connection string. For Supabase transaction pooler, include
|
||||
# prepare=false (see CLAUDE.md #284/#286).
|
||||
DATABASE_URL=postgresql://user:pass@host:6543/db?prepare=false
|
||||
|
||||
# --- Required if you submit `shell` jobs ------------------------------------
|
||||
# Only the worker process needs this. Submitters do not.
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1
|
||||
|
||||
# --- Optional ---------------------------------------------------------------
|
||||
|
||||
# LLM provider keys (needed for `subagent` handler, transcription, enrichment).
|
||||
# ANTHROPIC_API_KEY=
|
||||
# OPENAI_API_KEY=
|
||||
|
||||
# Custom handler plugins (see docs/guides/plugin-handlers.md).
|
||||
# GBRAIN_PLUGIN_PATH=/etc/gbrain/plugins
|
||||
|
||||
# Pool size tuning for Supabase transaction pooler (default 10; drop to 2
|
||||
# if you hit MaxClients during upgrade subprocess spawns).
|
||||
# GBRAIN_POOL_SIZE=2
|
||||
|
||||
# Connection-level concurrency cap for Anthropic Messages API.
|
||||
# GBRAIN_ANTHROPIC_MAX_INFLIGHT=4
|
||||
@@ -1,50 +0,0 @@
|
||||
[Unit]
|
||||
Description=gbrain minion worker
|
||||
Documentation=https://github.com/garrytan/gbrain/blob/master/docs/guides/minions-deployment.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Runs as an unprivileged user that owns the brain repo and any shell-job cwds.
|
||||
# Create with: sudo useradd --system --home /srv/gbrain --shell /usr/sbin/nologin gbrain
|
||||
User=gbrain
|
||||
Group=gbrain
|
||||
WorkingDirectory=/srv/gbrain
|
||||
|
||||
# Env file is mode 600, owned by User=. Do not put secrets in this unit.
|
||||
EnvironmentFile=/etc/gbrain.env
|
||||
|
||||
# Two-layer supervision: systemd restarts `gbrain jobs supervisor` on host
|
||||
# events (reboot, unit crash); the supervisor restarts `gbrain jobs work`
|
||||
# on in-process crashes with exponential backoff + structured audit.
|
||||
ExecStart=/usr/local/bin/gbrain jobs supervisor --concurrency 2
|
||||
|
||||
# systemd restarts the supervisor on any non-zero exit. The supervisor
|
||||
# itself handles worker-level crash recovery.
|
||||
Restart=always
|
||||
RestartSec=10s
|
||||
|
||||
# Graceful shutdown: SIGTERM → wait → SIGKILL. 30s matches worker grace
|
||||
# for in-flight jobs and the shell handler's 5s child SIGTERM window.
|
||||
KillSignal=SIGTERM
|
||||
TimeoutStopSec=30s
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=gbrain-worker
|
||||
|
||||
# Default 1024 is tight for Bun + Postgres pool + concurrent subagent LLM calls.
|
||||
LimitNOFILE=65535
|
||||
|
||||
# Hardening (optional — remove if they break your deployment).
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
# ReadWritePaths must include the brain workspace AND ~/.gbrain (PID file +
|
||||
# audit log written by the supervisor).
|
||||
ReadWritePaths=/srv/gbrain /home/gbrain/.gbrain
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,332 +0,0 @@
|
||||
# Minions Worker Deployment Guide
|
||||
|
||||
Keep `gbrain jobs work` running across crashes, reboots, and Postgres
|
||||
connection blips. Written for agents to execute line-by-line.
|
||||
|
||||
## The problem
|
||||
|
||||
The persistent worker can die silently from:
|
||||
|
||||
- Database connection drops (Supabase/Postgres maintenance or network blips).
|
||||
- Lock-renewal failures → the stall detector eventually dead-letters jobs.
|
||||
- Bun process crashes with no automatic restart.
|
||||
- Internal event-loop death (PID alive, worker loop stopped).
|
||||
|
||||
When the worker dies, submitted jobs sit in `waiting` forever. The
|
||||
canonical answer is `gbrain jobs supervisor` — a first-class CLI that
|
||||
spawns `gbrain jobs work` as a child and auto-restarts it on crash.
|
||||
|
||||
## Worker supervision
|
||||
|
||||
### The canonical pattern
|
||||
|
||||
`gbrain jobs supervisor` is an auto-restarting wrapper around
|
||||
`gbrain jobs work`. It writes a PID file, restarts the worker on crash
|
||||
with exponential backoff (1s → 60s cap), emits lifecycle events to an
|
||||
audit file, and drains gracefully on SIGTERM (35s worker-drain window
|
||||
before SIGKILL). Exit codes are documented so agents can branch on them.
|
||||
|
||||
**Typical commands:**
|
||||
|
||||
```bash
|
||||
# Start in the foreground (blocks; Ctrl-C to stop).
|
||||
gbrain jobs supervisor --concurrency 4
|
||||
|
||||
# Start detached — returns {"event":"started","supervisor_pid":…} on stdout.
|
||||
gbrain jobs supervisor start --detach --json
|
||||
|
||||
# Check liveness without reading log files.
|
||||
gbrain jobs supervisor status --json
|
||||
|
||||
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
|
||||
gbrain jobs supervisor stop
|
||||
```
|
||||
|
||||
**Exit codes:**
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) |
|
||||
| 1 | Max crashes exceeded (worker kept dying) |
|
||||
| 2 | Another supervisor holds the PID lock |
|
||||
| 3 | PID file unwritable (permission / path error) |
|
||||
|
||||
An agent seeing exit=2 can safely treat it as "one is already running";
|
||||
exit=1 should page a human.
|
||||
|
||||
### Which supervisor when?
|
||||
|
||||
The supervisor solves in-process crash recovery. Platform-level
|
||||
supervision (systemd, Fly, Render) handles host-level failures. You
|
||||
usually want both.
|
||||
|
||||
| Environment | Recommendation |
|
||||
|---|---|
|
||||
| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). |
|
||||
| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). |
|
||||
| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. |
|
||||
|
||||
### Variables used in this guide
|
||||
|
||||
Substitute these once before copy-pasting any snippet.
|
||||
|
||||
| Variable | Meaning | Typical value |
|
||||
|---|---|---|
|
||||
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
|
||||
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
|
||||
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
|
||||
| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) |
|
||||
|
||||
### Preconditions
|
||||
|
||||
Run these before any deployment step.
|
||||
|
||||
```bash
|
||||
# 1. gbrain is on PATH and resolves to an absolute location.
|
||||
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
|
||||
|
||||
# 2. DATABASE_URL points at reachable Postgres.
|
||||
# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the
|
||||
# separate worker process. If `config.engine === 'pglite'` the CLI rejects
|
||||
# with a clear error.)
|
||||
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
|
||||
|
||||
# 3. Schema is up to date. If version=0 or status=="fail":
|
||||
# gbrain apply-migrations --yes
|
||||
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
|
||||
|
||||
# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the
|
||||
# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting).
|
||||
# Without the flag, the shell handler is disabled at worker startup.
|
||||
```
|
||||
|
||||
## Agent usage (OpenClaw / Hermes / Cursor / Codex)
|
||||
|
||||
Three-command pattern an agent can drive without shell archaeology:
|
||||
|
||||
```bash
|
||||
# Start (returns PIDs + pid_file on stdout as JSON, then detaches)
|
||||
gbrain jobs supervisor start --detach --json
|
||||
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
|
||||
|
||||
# Check health (machine-parseable JSON, no log scraping)
|
||||
gbrain jobs supervisor status --json
|
||||
# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...}
|
||||
|
||||
# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback)
|
||||
gbrain jobs supervisor stop
|
||||
```
|
||||
|
||||
Every lifecycle event (spawn, crash, backoff, health warning, max-crashes,
|
||||
shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
|
||||
for historical inspection. `gbrain doctor` reads that file and surfaces
|
||||
a `supervisor` check in its health report.
|
||||
|
||||
## Deployment: systemd
|
||||
|
||||
For long-running Linux VMs with shell access.
|
||||
|
||||
```bash
|
||||
# Create the worker user if it doesn't exist.
|
||||
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
|
||||
2>/dev/null || true
|
||||
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
|
||||
|
||||
# Install the env file (secrets stay out of the unit file).
|
||||
sudo install -m 600 -o gbrain -g gbrain \
|
||||
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
|
||||
sudoedit /etc/gbrain.env
|
||||
# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1.
|
||||
|
||||
# Install the unit file, substituting /srv/gbrain → your workspace path.
|
||||
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
|
||||
/etc/systemd/system/gbrain-worker.service
|
||||
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
|
||||
/etc/systemd/system/gbrain-worker.service
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now gbrain-worker
|
||||
sudo systemctl status gbrain-worker
|
||||
journalctl -u gbrain-worker -n 50
|
||||
```
|
||||
|
||||
The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work`
|
||||
directly) so you get two-layer supervision: systemd restarts the supervisor
|
||||
on host reboot, supervisor restarts the worker on in-process crash.
|
||||
|
||||
`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery.
|
||||
The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`,
|
||||
and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and
|
||||
audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent
|
||||
LLM subagent calls without hitting the default 1024 cap.
|
||||
|
||||
## Deployment: Fly.io
|
||||
|
||||
```bash
|
||||
# Merge the [processes] block from fly.toml.partial into your fly.toml.
|
||||
cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml
|
||||
# Review + edit as needed.
|
||||
|
||||
# Set secrets (Fly handles restart on crash).
|
||||
fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1
|
||||
```
|
||||
|
||||
The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly
|
||||
restarts the container on host failure; the supervisor restarts the
|
||||
worker on in-process crash.
|
||||
|
||||
## Deployment: Render / Railway / Heroku
|
||||
|
||||
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo
|
||||
root. The shipped Procfile calls `gbrain jobs supervisor`. Set
|
||||
`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's
|
||||
env UI or CLI.
|
||||
|
||||
## Deployment: inline `--follow` (no persistent worker)
|
||||
|
||||
For short deterministic scripts on a fixed schedule where you don't need
|
||||
a persistent worker between runs. Each cron run brings its own temporary
|
||||
worker. `--follow` starts one on the queue and blocks until the
|
||||
just-submitted job reaches a terminal state (`completed` / `failed` /
|
||||
`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job
|
||||
duration for scheduled work.
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
|
||||
--queue nightly-enrich \
|
||||
--params "{\"cmd\":\"$GBRAIN_BIN embed --stale\",\"cwd\":\"$GBRAIN_WORKSPACE\"}" \
|
||||
--follow \
|
||||
--timeout-ms 600000
|
||||
```
|
||||
|
||||
Replace `gbrain embed --stale` with whichever gbrain subcommand you're
|
||||
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
|
||||
`lint`, `autopilot`). For strict single-job semantics on shared queues,
|
||||
use a dedicated queue name like `nightly-enrich` above.
|
||||
|
||||
## Upgrading from an older deployment
|
||||
|
||||
### From `minion-watchdog.sh` (pre-v0.20)
|
||||
|
||||
Earlier versions of this guide shipped a 68-line bash watchdog
|
||||
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
|
||||
which handles everything the script did, plus atomic PID locking,
|
||||
structured audit events, queue-scoped health checks, and graceful
|
||||
drain on SIGTERM.
|
||||
|
||||
**Migration:**
|
||||
|
||||
```bash
|
||||
# 1. Stop and remove the old watchdog.
|
||||
sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null
|
||||
sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \
|
||||
/tmp/gbrain-worker.log
|
||||
crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line
|
||||
|
||||
# 2. Start the supervisor (systemd users: reinstall the unit from
|
||||
# docs/guides/minions-deployment-snippets/systemd.service, which
|
||||
# now calls `gbrain jobs supervisor`).
|
||||
gbrain jobs supervisor start --detach --json
|
||||
# Or: sudo systemctl restart gbrain-worker
|
||||
|
||||
# 3. Verify.
|
||||
gbrain jobs supervisor status --json
|
||||
gbrain doctor # 'supervisor' check should report running=true
|
||||
```
|
||||
|
||||
### Schema / migration hygiene
|
||||
|
||||
Regardless of which deployment path you're upgrading from:
|
||||
|
||||
1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop`
|
||||
(or `sudo systemctl stop gbrain-worker`). Skipping this risks an
|
||||
in-flight job landing partial schema.
|
||||
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
|
||||
`gbrain doctor` reports any migration as `partial` or `pending`.
|
||||
3. **If you run shell jobs:** from v0.14 onward, pass
|
||||
`--allow-shell-jobs` to the supervisor (or keep
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
|
||||
need the flag; only the worker does.
|
||||
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
|
||||
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
|
||||
should show no unexplained growth in `dead` between pre- and
|
||||
post-upgrade.
|
||||
|
||||
## Known issues
|
||||
|
||||
### Supabase connection drops
|
||||
|
||||
The worker uses a single Postgres connection. If Supabase drops it
|
||||
(maintenance, connection limits, network blip), lock renewal fails
|
||||
silently. The stall detector then dead-letters the job after
|
||||
`max_stalled` misses.
|
||||
|
||||
**Current defaults that make this worse:**
|
||||
|
||||
- `lockDuration: 30000` (30 s) — too short for long jobs during
|
||||
connection blips.
|
||||
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
|
||||
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
|
||||
- `stalledInterval: 30000` (30 s) — checks too aggressively.
|
||||
|
||||
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
|
||||
(since v0.13.1). These write onto the job row at submit time — which is
|
||||
what `handleStalled()` reads — so per-job tuning is the real knob today.
|
||||
|
||||
### DO NOT pass `maxStalledCount` to `MinionWorker`
|
||||
|
||||
It's a no-op. The stall detector reads the row's `max_stalled` column
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
|
||||
Use `gbrain jobs submit --max-stalled N` per-job instead.
|
||||
|
||||
### Zombie shell children
|
||||
|
||||
When the Bun worker crashes hard, child processes from shell jobs can
|
||||
become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window
|
||||
covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For
|
||||
long-running shell jobs, prefer timeouts via `--timeout-ms` on submit
|
||||
over relying on hard kills.
|
||||
|
||||
## Smoke test
|
||||
|
||||
```bash
|
||||
# Supervisor alive?
|
||||
gbrain jobs supervisor status --json | jq .running
|
||||
|
||||
# Aggregate queue health.
|
||||
gbrain jobs stats
|
||||
|
||||
# Jobs currently stalled (still `active` with expired lock_until, pre-requeue).
|
||||
gbrain jobs list --status active --limit 10
|
||||
|
||||
# Dead-lettered jobs.
|
||||
gbrain jobs list --status dead --limit 10
|
||||
|
||||
# Shell handler registered? (check supervisor audit log or worker stderr.)
|
||||
gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs'
|
||||
```
|
||||
|
||||
## Uninstall
|
||||
|
||||
**`gbrain jobs supervisor`** (foreground or `--detach`):
|
||||
|
||||
```bash
|
||||
gbrain jobs supervisor stop
|
||||
```
|
||||
|
||||
**systemd:**
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now gbrain-worker
|
||||
sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
**Fly / Render / Railway:** delete the `worker` process from `fly.toml`
|
||||
/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until
|
||||
`fly secrets unset`.
|
||||
|
||||
**Inline `--follow`:** remove the cron entry. Nothing else to clean up
|
||||
— temporary workers exit with their jobs.
|
||||
@@ -1,159 +0,0 @@
|
||||
# Minions fix — repairing a half-migrated install
|
||||
|
||||
**tl;dr:** on v0.11.1+ everything should self-heal. If Minions is partially
|
||||
set up (no `~/.gbrain/preferences.json`, autopilot still inline, cron jobs
|
||||
still on `agentTurn`), run:
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
It's idempotent. On v0.11.1 installs that already migrated it's a cheap
|
||||
no-op.
|
||||
|
||||
## Context
|
||||
|
||||
v0.11.0 shipped the Minions schema, queue, worker, and migration skill —
|
||||
but the migration skill itself never fired on upgrade. `runPostUpgrade`
|
||||
printed the feature pitch and stopped. v0.11.0 was never released
|
||||
publicly; v0.11.1 is the first public Minions ship and fixes the
|
||||
mega-bug (migration fires automatically on `gbrain upgrade` and via
|
||||
the `postinstall` hook).
|
||||
|
||||
If you're on a pre-v0.11.1 branch build (e.g. running the
|
||||
`minions-jobs` branch before v0.11.1 tagged), Minions may be installed
|
||||
but not wired: schema is v7, but no `~/.gbrain/preferences.json`,
|
||||
autopilot still runs inline, cron jobs still call `agentTurn`.
|
||||
|
||||
This guide covers both paths: the canonical v0.11.1+ fix, and the
|
||||
stopgap for pre-v0.11.1 binaries that don't have `apply-migrations`.
|
||||
|
||||
## Detecting the half-migrated state
|
||||
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
If the install is half-migrated, you'll see:
|
||||
|
||||
```
|
||||
[FAIL] minions_migration: MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
[FAIL] minions_config: MINIONS HALF-INSTALLED (schema v7+ but no ~/.gbrain/preferences.json). Run: gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
For a machine-readable report (cron-friendly):
|
||||
|
||||
```bash
|
||||
gbrain skillpack-check --quiet && echo healthy || echo needs_action
|
||||
gbrain skillpack-check | jq -r '.actions[]' # prints the exact commands to run
|
||||
```
|
||||
|
||||
## The fix (v0.11.1 or later)
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
Reads `~/.gbrain/migrations/completed.jsonl`, diffs against the TS
|
||||
migration registry, runs whatever's pending. Seven phases:
|
||||
|
||||
```
|
||||
A. Schema gbrain init --migrate-only
|
||||
B. Smoke gbrain jobs smoke
|
||||
C. Mode prompt (or --yes default pain_triggered)
|
||||
D. Prefs write ~/.gbrain/preferences.json
|
||||
E. Host AGENTS.md marker injection + cron rewrites for gbrain
|
||||
builtins; JSONL TODOs for host-specific handlers
|
||||
F. Install gbrain autopilot --install (env-aware)
|
||||
G. Record append completed.jsonl status:"complete"
|
||||
```
|
||||
|
||||
If Phase E emits TODOs for host-specific handlers (e.g. your OpenClaw's
|
||||
~29 non-gbrain crons), the migration finishes with `status: "partial"`.
|
||||
Your host agent walks the TODOs using `skills/migrations/v0.11.0.md` +
|
||||
`docs/guides/plugin-handlers.md`, ships handler registrations in the
|
||||
host repo, then re-runs `gbrain apply-migrations --yes`. Newly
|
||||
registerable cron entries get rewritten and the JSONL rows mark
|
||||
`status: "complete"`.
|
||||
|
||||
## The stopgap (pre-v0.11.1 binary, no apply-migrations yet)
|
||||
|
||||
If you're stuck on a branch build that doesn't have `apply-migrations`:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/garrytan/gbrain/v0.11.1/scripts/fix-v0.11.0.sh | bash
|
||||
```
|
||||
|
||||
This bash script does what apply-migrations does from a shell environment:
|
||||
|
||||
1. `gbrain init --migrate-only` — schema v7.
|
||||
2. `gbrain jobs smoke` — verify Minions health.
|
||||
3. Prompt for `minion_mode` (defaults `pain_triggered` on non-TTY).
|
||||
4. Write `~/.gbrain/preferences.json` atomically.
|
||||
5. Append `~/.gbrain/migrations/completed.jsonl` with `status: "partial"`
|
||||
and `apply_migrations_pending: true`. That partial record is the
|
||||
signal to v0.11.1's `apply-migrations` to pick up remaining phases
|
||||
after the user upgrades.
|
||||
6. Detect host agent repos and PRINT rewrite instructions (never
|
||||
auto-edits from a curl-piped script).
|
||||
7. Print the next step: `Run: gbrain autopilot --install`.
|
||||
|
||||
Once v0.11.1 is installed, re-run `gbrain apply-migrations --yes` to
|
||||
finish the remaining phases (host rewrites + autopilot install). The
|
||||
stopgap's `status: "partial"` record is designed to resume cleanly
|
||||
(it doesn't poison the permanent migration path).
|
||||
|
||||
## Verify the fix landed
|
||||
|
||||
```bash
|
||||
# 1. Preferences exist and are readable
|
||||
cat ~/.gbrain/preferences.json
|
||||
|
||||
# 2. Migration recorded
|
||||
cat ~/.gbrain/migrations/completed.jsonl
|
||||
|
||||
# 3. Autopilot is supervising a Minions worker child
|
||||
gbrain autopilot --status
|
||||
ps aux | grep 'jobs work'
|
||||
|
||||
# 4. Jobs show up in the queue
|
||||
gbrain jobs list
|
||||
|
||||
# 5. Any host-specific TODOs still pending
|
||||
cat ~/.gbrain/migrations/pending-host-work.jsonl 2>/dev/null || echo "(none — all host work is done)"
|
||||
|
||||
# 6. Doctor + skillpack-check should both be clean
|
||||
gbrain doctor
|
||||
gbrain skillpack-check --quiet && echo ok
|
||||
```
|
||||
|
||||
## If the fix fails
|
||||
|
||||
Each phase is idempotent. Re-running is safe. Common failure modes:
|
||||
|
||||
- **Phase B smoke fails:** the schema didn't apply. Check
|
||||
`~/.gbrain/config.json` has a valid `database_url` (or `database_path`
|
||||
for PGLite). Run `gbrain init --migrate-only` directly and look at
|
||||
the error.
|
||||
- **Phase F install fails:** your host environment doesn't match any
|
||||
detected target. Pass `--target <macos|linux-systemd|ephemeral-container|linux-cron>`
|
||||
explicitly.
|
||||
- **Pending host work never clears:** your host agent hasn't shipped
|
||||
handler registrations yet. Read
|
||||
`~/.gbrain/migrations/pending-host-work.jsonl`, open
|
||||
`skills/migrations/v0.11.0.md`, and follow the host-agent instruction
|
||||
manual.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/migrations/v0.11.0.md` — full migration skill for host agents.
|
||||
- `skills/skillpack-check/SKILL.md` — when and how to run the health check.
|
||||
- `docs/guides/plugin-handlers.md` — plugin contract for host-specific
|
||||
handlers.
|
||||
- `skills/conventions/cron-via-minions.md` — the canonical cron rewrite
|
||||
pattern.
|
||||
@@ -1,167 +0,0 @@
|
||||
# Minions shell jobs — move deterministic crons off the gateway
|
||||
|
||||
## 30 seconds
|
||||
|
||||
```bash
|
||||
# Run your first shell job:
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
|
||||
--params '{"cmd":"echo hello","cwd":"/tmp"}' --follow
|
||||
# → exit_code: 0, stdout_tail: "hello\n", duration_ms: 43
|
||||
```
|
||||
|
||||
That's it. Your cron scripts now have a home with retry, backoff, DLQ, and
|
||||
`gbrain jobs list` visibility, without each one booting a full LLM session.
|
||||
|
||||
**PGLite users:** `gbrain jobs work` does not run on PGLite (exclusive file
|
||||
lock). Every crontab invocation must use `--follow` for inline execution.
|
||||
Postgres users can run a persistent worker; see recipes below.
|
||||
|
||||
---
|
||||
|
||||
## Why it exists
|
||||
|
||||
If your agent runs deterministic scripts from cron (token refresh, API fetch,
|
||||
scrape + write), each one pays the cost of a full LLM session on the gateway.
|
||||
Fourteen simultaneous fires on a Series A deployment pin CPU at 100% and block
|
||||
live messages. None of those scripts need reasoning. They need a shell.
|
||||
|
||||
Shell jobs move them to the Minions worker: one deterministic-script execution
|
||||
per cron, zero LLM tokens, unified visibility and retry.
|
||||
|
||||
---
|
||||
|
||||
## Security model (read this)
|
||||
|
||||
Shell exec is a large blast radius. We ship two independent gates, both must
|
||||
pass:
|
||||
|
||||
1. **MCP boundary.** `submit_job` with `name: 'shell'` is rejected when
|
||||
`ctx.remote === true` (MCP callers). Independent of the env flag. Remote
|
||||
agents can never submit shell jobs. `MinionQueue.add('shell', ...)` has its
|
||||
own guard too, so an in-process handler can't programmatically bypass this.
|
||||
2. **Env flag.** The worker only registers the shell handler when
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` is set on the worker process. Default: off. Your
|
||||
agent opts in per-host.
|
||||
|
||||
**What the env allowlist does AND does not do.** Shell jobs run with a minimal
|
||||
env: `PATH, HOME, USER, LANG, TZ, NODE_ENV`. Your secrets like `OPENAI_API_KEY`
|
||||
and `DATABASE_URL` are NOT passed to the child. You opt-in additional keys per
|
||||
job via `env: { ... }`. This stops accidental `$OPENAI_API_KEY` interpolation in
|
||||
a user-authored script. It does **not** sandbox filesystem reads: a shell
|
||||
script can `cat ~/.env` or any file the worker process can read. The operator
|
||||
picks a safe `cwd`. That is the trust boundary.
|
||||
|
||||
**Audit trail, not forensic insurance.** Every submission writes a JSONL line
|
||||
to `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override
|
||||
with `GBRAIN_AUDIT_DIR`). Failures log to stderr and don't block submission, so
|
||||
a disk-full adversary could silently disable the trail. Good for "what did
|
||||
this cron submit last Tuesday", not for security-critical forensics.
|
||||
|
||||
**The command text is logged as-is.** If you embed a secret in `cmd`
|
||||
(`curl -H 'Authorization: Bearer ...'`), it shows up in the audit file. Put
|
||||
secrets in `env:` instead.
|
||||
|
||||
---
|
||||
|
||||
## Migrate a cron
|
||||
|
||||
### Postgres worker (recommended)
|
||||
|
||||
On one terminal, start a persistent worker:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work
|
||||
```
|
||||
|
||||
Rewrite crontab to submit shell jobs (no `--follow`):
|
||||
|
||||
```cron
|
||||
# Before (LLM gateway):
|
||||
# OpenClaw cron: x-garrytan-unified
|
||||
# After (Minions worker):
|
||||
3 13,16,19,22,1,4,7,10 * * * \
|
||||
gbrain jobs submit shell \
|
||||
--params '{"cmd":"node scripts/x-garrytan-daily.mjs","cwd":"/data/.openclaw/workspace"}' \
|
||||
--max-attempts 3 --timeout-ms 300000
|
||||
```
|
||||
|
||||
Worker claims the job on next poll, runs it, records `exit_code` +
|
||||
`stdout_tail` + `stderr_tail` in the result. Failures retry per
|
||||
`--max-attempts` with exponential backoff.
|
||||
|
||||
### PGLite (inline execution)
|
||||
|
||||
PGLite doesn't support the persistent worker daemon. Every crontab invocation
|
||||
uses `--follow` to run inline:
|
||||
|
||||
```cron
|
||||
# Each cron tick spawns a short-lived worker that runs the job inline.
|
||||
3 13,16,19,22,1,4,7,10 * * * \
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
|
||||
--params '{"cmd":"node scripts/x-garrytan-daily.mjs","cwd":"/data/.openclaw/workspace"}' \
|
||||
--follow --timeout-ms 300000
|
||||
```
|
||||
|
||||
Note: `--follow` blocks the crontab slot until the job finishes. If 14 shell
|
||||
crons land at the same minute and each takes 30s, they serialize through
|
||||
crontab's spawning limits. Postgres + persistent worker scales better.
|
||||
|
||||
### Submitting with `argv` (no shell interpolation)
|
||||
|
||||
For programmatic callers assembling commands from JSON, use `argv` instead of
|
||||
`cmd`. No shell, no injection surface:
|
||||
|
||||
```bash
|
||||
gbrain jobs submit shell \
|
||||
--params '{"argv":["node","scripts/fetch.mjs","--date","2026-04-19"],"cwd":"/data"}' \
|
||||
--follow
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debug a failed job
|
||||
|
||||
```bash
|
||||
# List dead shell jobs
|
||||
gbrain jobs list --status dead
|
||||
|
||||
# Inspect one
|
||||
gbrain jobs get 42
|
||||
# → error_text, stacktrace, result.stdout_tail, result.stderr_tail
|
||||
|
||||
# Submission audit log (operator trail, not forensic)
|
||||
cat ~/.gbrain/audit/shell-jobs-*.jsonl | jq '.'
|
||||
|
||||
# First-time failure mode: submitted without env flag on the worker
|
||||
gbrain jobs list --status waiting --name shell
|
||||
# If rows pile up here, no worker with GBRAIN_ALLOW_SHELL_JOBS=1 is running.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Filesystem reads are not sandboxed.** See "Security model" above. Don't
|
||||
point `cwd` at a directory full of secrets.
|
||||
- **Audit log is advisory.** Disk-full or EACCES silently disables it.
|
||||
- **Cancel latency is lock-renewal-bounded** (~7-15 s by default). A cancelled
|
||||
child keeps running until the next lock-renewal tick fails.
|
||||
- **`--follow` claim order** is by priority/created_at. If another job is
|
||||
waiting in the same queue at the time of `--follow`, that one runs first.
|
||||
- **`cwd` symlink TOCTOU.** The absolute-path check doesn't guard against
|
||||
symlinks pointing elsewhere at execution time. Operator-scope concern.
|
||||
|
||||
---
|
||||
|
||||
## Errors {#errors}
|
||||
|
||||
| Error | What it means | Fix |
|
||||
|---|---|---|
|
||||
| `shell: specify exactly one of cmd or argv` | `cmd` and `argv` are mutually exclusive. Both absent is also invalid. | Choose one. `cmd` for shell-interpolated strings; `argv` for structured args. |
|
||||
| `shell: cwd is required and must be an absolute path` | `cwd` must be a string starting with `/`. | Set `cwd` in `--params` to an absolute path. |
|
||||
| `shell: argv must be an array of strings` | `argv` has a non-string entry or isn't an array. | Pass `argv: ["bin","arg1","arg2"]`. |
|
||||
| `shell: env values must all be strings` | `env` has a number/bool/object value. | Stringify: `"env":{"COUNT":"3"}` not `"env":{"COUNT":3}`. |
|
||||
| `permission_denied: shell jobs cannot be submitted over MCP` | An MCP client tried to submit a shell job. By design CLI-only. | Submit from CLI or via a trusted operation handler (`ctx.remote === false`). |
|
||||
| `protected job name 'shell' requires CLI or operation-local submitter` | A caller invoked `MinionQueue.add('shell', ...)` without the `trusted` opt-in. | Pass `{ allowProtectedSubmit: true }` as the 4th arg. CLI and `submit_job` do this automatically. |
|
||||
| `aborted: timeout` / `aborted: cancel` / `aborted: shutdown` / `aborted: lock-lost` | The worker's abort signal fired mid-execution. Child got SIGTERM, 5s grace, then SIGKILL. | Expected: timeout / user cancel / deploy restart / stall. Inspect `gbrain jobs get` to see which. |
|
||||
| `exit N: <stderr_tail_500>` | Script exited non-zero. | Read `stderr_tail` in `gbrain jobs get`. |
|
||||
@@ -1,182 +0,0 @@
|
||||
# Multi-source brains
|
||||
|
||||
**A single gbrain database can hold multiple knowledge repos.** Each one
|
||||
is a `source`: a logical brain-within-the-brain with its own slug
|
||||
namespace, its own sync state, and its own federation policy. The rest
|
||||
of this guide walks the three canonical scenarios.
|
||||
|
||||
## The three scenarios
|
||||
|
||||
### 1. Unified knowledge recall (wiki + gstack)
|
||||
|
||||
You have a personal wiki and a `gstack` checkout. Both belong to you,
|
||||
both are knowledge you want your agent to recall across. When you ask
|
||||
"what did I learn about X?" you want the best hit whether it lives in
|
||||
the wiki or in a gstack plan.
|
||||
|
||||
```bash
|
||||
# Register the gstack source, federate so it joins cross-source search
|
||||
gbrain sources add gstack --path ~/.gstack --federated
|
||||
|
||||
# Pin the directory so `gbrain sync` knows which source it's walking
|
||||
cd ~/.gstack && gbrain sources attach gstack
|
||||
|
||||
# Initial sync
|
||||
gbrain sync --source gstack
|
||||
|
||||
# Now `gbrain search "retry budgets"` returns hits from BOTH wiki and
|
||||
# gstack. Each result includes source_id so the agent can cite properly.
|
||||
```
|
||||
|
||||
Result: wiki pages and gstack plans are separate (different source_ids,
|
||||
different slug namespaces) but share the search surface.
|
||||
|
||||
### 2. Purpose-separated brains (yc-media + garrys-list)
|
||||
|
||||
You run two completely different content pipelines on the same backend.
|
||||
YC Media covers portfolio news and founder profiles. Garry's List is
|
||||
personal writing. You explicitly DON'T want them mixed in search — YC
|
||||
portfolio content leaking into essay searches is a bug, not a feature.
|
||||
|
||||
```bash
|
||||
# Two sources, both isolated (federated=false)
|
||||
gbrain sources add yc-media --path ~/yc-media --no-federated
|
||||
gbrain sources add garrys-list --path ~/writing --no-federated
|
||||
|
||||
# Pin each checkout directory
|
||||
(cd ~/yc-media && gbrain sources attach yc-media)
|
||||
(cd ~/writing && gbrain sources attach garrys-list)
|
||||
|
||||
# Sync each independently
|
||||
gbrain sync --source yc-media
|
||||
gbrain sync --source garrys-list
|
||||
```
|
||||
|
||||
Result: searching from neither directory returns the `default` source
|
||||
(your main brain). Searching from inside `~/yc-media` returns only yc-
|
||||
media hits. Searching from inside `~/writing` returns only garrys-list.
|
||||
Federation is opt-in, not leaked.
|
||||
|
||||
To search across them explicitly on demand:
|
||||
|
||||
```bash
|
||||
gbrain search "tech layoffs" --source yc-media,garrys-list
|
||||
```
|
||||
|
||||
### 3. Mixed (wiki federated + sessions isolated)
|
||||
|
||||
Your main wiki is federated with a few trusted sources. Your session
|
||||
transcripts (coming in v0.18) land in a separate isolated source so
|
||||
they don't dominate every search result.
|
||||
|
||||
```bash
|
||||
# Federated sources
|
||||
gbrain sources add gstack --path ~/.gstack --federated
|
||||
|
||||
# Isolated source (future v0.18 — sessions use this shape today for ingest)
|
||||
gbrain sources add sessions --path ~/.claude/sessions --no-federated
|
||||
```
|
||||
|
||||
## Resolution priority
|
||||
|
||||
When any command needs to pick a source, gbrain walks this list (highest
|
||||
first):
|
||||
|
||||
1. Explicit `--source <id>` flag.
|
||||
2. `GBRAIN_SOURCE` environment variable.
|
||||
3. `.gbrain-source` dotfile in CWD or any ancestor directory.
|
||||
4. A registered source whose `local_path` contains the CWD (longest
|
||||
prefix wins for nested checkouts).
|
||||
5. The brain-level default set via `gbrain sources default <id>`.
|
||||
6. The seeded `default` source.
|
||||
|
||||
So inside `~/.gstack/plans/` on a brain that pinned `gstack` to
|
||||
`~/.gstack` via `.gbrain-source`, `gbrain put-page` implicitly writes to
|
||||
the `gstack` source. Outside any registered directory with no env/dotfile
|
||||
set, it writes to the default.
|
||||
|
||||
## Federation flag
|
||||
|
||||
Every source row stores `config.federated: boolean` in its JSONB config.
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `true` | Source participates in unqualified `gbrain search "X"` results. |
|
||||
| `false` (default for new sources) | Source only searched when explicitly named via `--source <id>` or qualified citation. |
|
||||
|
||||
The seeded `default` source is `federated=true` so pre-v0.17 brains
|
||||
behave exactly as before — every page appears in search.
|
||||
|
||||
Flip later with `gbrain sources federate <id>` / `unfederate <id>`.
|
||||
|
||||
## Commands
|
||||
|
||||
Full subcommand reference:
|
||||
|
||||
```
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
||||
Register a source. id: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
|
||||
gbrain sources list [--json] List all sources with page counts + federation state.
|
||||
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
Cascade-delete a source (pages, chunks, timeline).
|
||||
gbrain sources rename <id> <new-name>
|
||||
Change display name only; id is immutable.
|
||||
gbrain sources default <id> Set the brain-level default.
|
||||
gbrain sources attach <id> Write .gbrain-source in CWD (like kubectl context).
|
||||
gbrain sources detach Remove .gbrain-source from CWD.
|
||||
gbrain sources federate <id>
|
||||
gbrain sources unfederate <id>
|
||||
```
|
||||
|
||||
## Citation format for agents
|
||||
|
||||
When agents receive multi-source results they MUST cite pages in
|
||||
`[source-id:slug]` form. Example:
|
||||
|
||||
> You told me about the distillation protocol — see [wiki:topics/ai]
|
||||
> and [gstack:plans/multi-repo] for where this came from.
|
||||
|
||||
The citation key is `sources.id` (immutable). Renaming a source via
|
||||
`gbrain sources rename` changes the display name only; existing
|
||||
citations keep working.
|
||||
|
||||
## Writing to a specific source
|
||||
|
||||
```bash
|
||||
# Pass --source explicitly
|
||||
gbrain put-page topics/ai ... --source wiki
|
||||
|
||||
# Or rely on the dotfile / env / CWD match
|
||||
cd ~/.gstack && gbrain put-page plans/multi-repo ...
|
||||
# → source auto-resolves to gstack
|
||||
```
|
||||
|
||||
Reads span federated sources by default. Writes require a resolved
|
||||
source (explicit, inferred, or default). The resolver never picks a
|
||||
source silently when ambiguous — it errors with a clear fix.
|
||||
|
||||
## Upgrading an existing brain
|
||||
|
||||
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your
|
||||
existing pages all move under `source_id='default'`. Behavior is
|
||||
unchanged until you add a second source.
|
||||
|
||||
To add one:
|
||||
|
||||
```bash
|
||||
gbrain sources add gstack --path ~/.gstack --federated
|
||||
cd ~/.gstack && gbrain sources attach gstack && gbrain sync
|
||||
```
|
||||
|
||||
Two commands. The existing default source is untouched.
|
||||
|
||||
## Not in v0.18.0
|
||||
|
||||
- Session transcript ingest (`.jsonl`, raised size cap, session
|
||||
PageType) — v0.18.
|
||||
- Per-source retention/TTL (`gbrain sources prune`) — v0.18.
|
||||
- ACL enforcement via caller-identity — v0.17.1.
|
||||
- `gbrain sources import-from-github <url>` one-shot bootstrap — patch
|
||||
release after the core plumbing stabilizes.
|
||||
|
||||
All of these build on the `sources` primitive shipped here.
|
||||
@@ -1,163 +0,0 @@
|
||||
# Plugin authors guide (v0.15)
|
||||
|
||||
`gbrain` discovers subagent definitions from outside this repo via
|
||||
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
|
||||
deployment, a workflow host, a private tool) and want to ship custom
|
||||
subagents alongside it, drop a plugin directory on that env path.
|
||||
|
||||
This guide is for plugin authors. The CLI user doesn't need to read it.
|
||||
|
||||
## Minimum viable plugin
|
||||
|
||||
```
|
||||
/path/to/my-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
└── my-summarizer.md
|
||||
```
|
||||
|
||||
`gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"version": "1.0.0",
|
||||
"plugin_version": "gbrain-plugin-v1"
|
||||
}
|
||||
```
|
||||
|
||||
`subagents/my-summarizer.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: my-summarizer
|
||||
model: claude-sonnet-4-6
|
||||
allowed_tools:
|
||||
- brain_search
|
||||
- brain_get_page
|
||||
---
|
||||
|
||||
You are a brain page summarizer. Given a slug, fetch the page and produce
|
||||
a 3-sentence summary.
|
||||
```
|
||||
|
||||
## Turning it on
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="/path/to/my-plugin"
|
||||
gbrain jobs work # worker startup prints the plugin load line
|
||||
gbrain agent run "summarize meetings/2026-04-20" --subagent-def my-summarizer
|
||||
```
|
||||
|
||||
Multiple plugins: colon-separated, just like `$PATH`.
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="/path/to/plugin-a:/path/to/plugin-b"
|
||||
```
|
||||
|
||||
## Rules (strict by design)
|
||||
|
||||
**Path policy.** Absolute paths only. Relative paths, `~`-prefixed paths,
|
||||
and URL-style paths (`https://`, `file://`) are rejected with a warning.
|
||||
You control where your plugin lives on disk; `gbrain` doesn't guess.
|
||||
|
||||
**Collision policy.** If two plugins ship a subagent with the same `name`,
|
||||
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
|
||||
with a warning naming both sources.
|
||||
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
|
||||
|
||||
- You **cannot** declare new tools.
|
||||
- You **cannot** extend the brain tool allow-list.
|
||||
- You **cannot** override any `agentSafe` or similar flag.
|
||||
- Your `allowed_tools:` frontmatter field MUST subset the derived brain
|
||||
tool registry. Names not in the registry are rejected at plugin load
|
||||
time (worker startup), NOT at subagent dispatch time — so a typo in
|
||||
your plugin gives you a loud startup error, not a silent "tool never
|
||||
fires" at 3am.
|
||||
|
||||
v0.16+ may open up plugin-declared tools with a separate contract. Don't
|
||||
expect it.
|
||||
|
||||
## `gbrain.plugin.json`
|
||||
|
||||
| field | type | required | notes |
|
||||
|------------------|--------|----------|--------------------------------------------------------------------|
|
||||
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in future `gbrain plugin list`. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
Plain markdown with YAML frontmatter. The body is the system prompt. The
|
||||
frontmatter controls runtime behavior.
|
||||
|
||||
Recognized frontmatter fields:
|
||||
|
||||
| field | type | required | notes |
|
||||
|-----------------|----------|----------|-----------------------------------------------------------------------------------------|
|
||||
| `name` | string | no | Subagent identifier used as `--subagent-def`. Defaults to the file basename. |
|
||||
| `model` | string | no | Anthropic model id. Defaults to the handler default (sonnet). |
|
||||
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
|
||||
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
|
||||
|
||||
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
|
||||
may consume more of them.
|
||||
|
||||
## Caveats that will bite you
|
||||
|
||||
1. **Plugin definitions can't change during a run.** The loader reads the
|
||||
disk once at worker startup. Editing a subagent def doesn't re-take
|
||||
effect until you restart the worker. This is deliberate — live
|
||||
reloads would break crash-resumable replay.
|
||||
|
||||
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
|
||||
worker runs on a different host than the `gbrain agent logs` caller,
|
||||
the CLI won't see heartbeats from that worker. v0.16 will unify this;
|
||||
for now assume worker + CLI share a filesystem.
|
||||
|
||||
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
|
||||
invocation. Tools that gate on `remote=true` (file_upload's strict
|
||||
confinement, put_page's namespace check) will apply. Good default; a
|
||||
subagent definition that wants local-filesystem reach beyond the brain
|
||||
can't have it.
|
||||
|
||||
4. **`put_page` writes are namespace-scoped.** A subagent with id 42 can
|
||||
only write under `wiki/agents/42/...`. This is enforced both in the
|
||||
tool schema (the slug pattern shown to the model) AND server-side in
|
||||
the `put_page` operation (fail-closed if `viaSubagent=true`). Don't
|
||||
try to route around it; you'll get `permission_denied`.
|
||||
|
||||
## Example: a downstream-OpenClaw plugin
|
||||
|
||||
```
|
||||
~/your-openclaw/
|
||||
└── gbrain-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
├── meeting-ingestion.md
|
||||
├── signal-detector.md
|
||||
└── daily-task-prep.md
|
||||
```
|
||||
|
||||
`~/your-openclaw/gbrain-plugin/gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "your-openclaw",
|
||||
"version": "2026.4.20",
|
||||
"plugin_version": "gbrain-plugin-v1",
|
||||
"description": "Your OpenClaw's personal-brain subagents"
|
||||
}
|
||||
```
|
||||
|
||||
Environment:
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="$HOME/your-openclaw/gbrain-plugin"
|
||||
```
|
||||
|
||||
Then your OpenClaw calls `gbrain agent run --subagent-def meeting-ingestion
|
||||
--fanout-by transcript ...` and its definitions load automatically.
|
||||
@@ -1,137 +0,0 @@
|
||||
# Plugin handlers — registering host-specific Minion handlers
|
||||
|
||||
GBrain's Minion worker ships with seven built-in handlers: `sync`,
|
||||
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
|
||||
These cover every background operation the gbrain CLI itself performs.
|
||||
|
||||
Host platforms (OpenClaw deployments, future hosts) register their own
|
||||
handlers via a plugin bootstrap that imports
|
||||
`gbrain/minions`. No `handlers.json`-style data file — handlers are
|
||||
code, loaded by the worker, with the same trust model as any other
|
||||
code in the host's repo.
|
||||
|
||||
## Why code, not data
|
||||
|
||||
An earlier design draft shipped `~/.claude/gbrain-handlers.json` where
|
||||
each entry was a shell command the worker would exec on job claim.
|
||||
Codex flagged this as a durable RCE surface: an agent-writable data
|
||||
file that spawns arbitrary shell. We dropped the data-file approach;
|
||||
handlers are code that the host imports explicitly and ships through
|
||||
code review.
|
||||
|
||||
## The plugin contract
|
||||
|
||||
A host worker bootstrap looks like this (TypeScript):
|
||||
|
||||
```ts
|
||||
import { MinionQueue, MinionWorker } from 'gbrain/minions';
|
||||
import type { BrainEngine } from 'gbrain/engine';
|
||||
|
||||
async function main() {
|
||||
const engine: BrainEngine = /* your engine setup */;
|
||||
await engine.connect({});
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: 'default' });
|
||||
|
||||
// Register every host-specific handler the host's cron manifest references.
|
||||
// Each handler returns a plain object (serialized as the job result).
|
||||
// Throw on failure — the worker catches and retries per max_attempts.
|
||||
|
||||
worker.register('ea-inbox-sweep', async (ctx) => {
|
||||
const slot = ctx.data.slot ?? new Date().toISOString();
|
||||
// Host-specific agent turn: call your LLM, scan the inbox, write
|
||||
// brain pages, return a summary. ctx.signal.aborted indicates the
|
||||
// worker wants you to cooperate with shutdown — honor it.
|
||||
return { swept: true, slot };
|
||||
});
|
||||
|
||||
worker.register('morning-briefing', async (ctx) => {
|
||||
/* host logic */
|
||||
return { briefed: true };
|
||||
});
|
||||
|
||||
// Call start() AFTER every handler is registered. The worker's
|
||||
// stall-detector ignores jobs whose name is not in the registered set.
|
||||
await worker.start();
|
||||
}
|
||||
|
||||
main().catch(err => { console.error(err); process.exit(1); });
|
||||
```
|
||||
|
||||
Ship this as a separate binary in the host repo (e.g. `your-openclaw-worker`)
|
||||
or as a side-effect module that the stock `gbrain jobs work` command
|
||||
auto-loads on startup (configurable via a host-provided entry point).
|
||||
|
||||
## Handler contract
|
||||
|
||||
Every handler receives a `MinionJobContext`:
|
||||
|
||||
```ts
|
||||
interface MinionJobContext {
|
||||
data: Record<string, unknown>; // job params (whatever the cron submit passed)
|
||||
job: MinionJob; // full job row (id, queue, attempts, etc.)
|
||||
signal: AbortSignal; // set to aborted when the worker is shutting down
|
||||
inbox: MinionInbox; // read messages sent to this job while it runs
|
||||
}
|
||||
```
|
||||
|
||||
Return a serializable object on success. Throw on failure (the worker
|
||||
will log + retry per `max_attempts`).
|
||||
|
||||
**Abort cooperation.** When `ctx.signal.aborted` becomes true, finish
|
||||
gracefully. The worker will wait 30s for you to return before SIGKILL.
|
||||
Long-running LLM calls should pass the signal through to whatever
|
||||
network library they use.
|
||||
|
||||
**Idempotency.** The queue enforces unique `idempotency_key` at the DB
|
||||
layer, so you don't need to worry about double-submits from a cron that
|
||||
fires while the previous invocation is still running.
|
||||
|
||||
## Gbrain's migration flow
|
||||
|
||||
The v0.11.0 migration orchestrator (run by `gbrain apply-migrations`)
|
||||
detects cron entries whose handler name is NOT in GBrain's builtin set
|
||||
and emits a structured TODO to `~/.gbrain/migrations/pending-host-work.jsonl`.
|
||||
Each TODO has shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "cron-handler-needs-host-registration",
|
||||
"handler": "ea-inbox-sweep",
|
||||
"cron_schedule": "0 */30 * * *",
|
||||
"manifest_path": "/path/to/cron/jobs.json",
|
||||
"current_cmd": "agentTurn ea-inbox-sweep",
|
||||
"recommendation": "Add a handler registration for `ea-inbox-sweep` in your host worker bootstrap per docs/guides/plugin-handlers.md. Once registered, re-run `gbrain apply-migrations` to auto-rewrite this entry.",
|
||||
"status": "pending"
|
||||
}
|
||||
```
|
||||
|
||||
The host agent walks these entries using `skills/migrations/v0.11.0.md`:
|
||||
|
||||
1. Read `~/.gbrain/migrations/pending-host-work.jsonl`.
|
||||
2. For each `cron-handler-needs-host-registration` row, ship a handler
|
||||
registration in the host's worker bootstrap following the pattern
|
||||
above.
|
||||
3. Deploy the updated worker.
|
||||
4. Re-run `gbrain apply-migrations --yes`. The orchestrator now
|
||||
recognizes the newly-registerable handler (worker writes the
|
||||
registered names to a discovery file on startup) and rewrites the
|
||||
cron entry to use `gbrain jobs submit`. The JSONL row is marked
|
||||
`status: "complete"`.
|
||||
|
||||
## Trust boundary
|
||||
|
||||
Handler code runs inside the worker process with the same privileges
|
||||
as the rest of the host binary. There is no elevation. But there is
|
||||
also no runtime sandbox — handlers can read + write anywhere the
|
||||
worker user can. Review handler PRs the same way you review any other
|
||||
code that touches production data.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/conventions/cron-via-minions.md` — the rewrite convention
|
||||
for cron manifests.
|
||||
- `skills/migrations/v0.11.0.md` — how the migration orchestrator
|
||||
drives the host agent through this work.
|
||||
- `skills/minion-orchestrator/SKILL.md` — patterns for submitting,
|
||||
monitoring, steering, and replaying jobs once the handler is live.
|
||||
@@ -1,76 +0,0 @@
|
||||
# Queue operations runbook
|
||||
|
||||
"My queue looks wedged — what do I run?" The commands below are in the order
|
||||
you probably want them. Shipped with v0.19.1 after a production incident
|
||||
where the queue held for 90+ minutes before the operator noticed.
|
||||
|
||||
## First signal: jobs aren't running
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
|
||||
```
|
||||
|
||||
`queue_health` flags two patterns:
|
||||
|
||||
- **stalled-forever**: active job whose `started_at` is older than 1h.
|
||||
- **waiting-depth**: any per-name queue deeper than 10 (override via
|
||||
`GBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.
|
||||
|
||||
## Triage commands
|
||||
|
||||
```bash
|
||||
# Who's active right now?
|
||||
gbrain jobs list --status active
|
||||
|
||||
# Who's waiting, biggest pile first?
|
||||
gbrain jobs list --status waiting --limit 50
|
||||
|
||||
# What's wrong with a specific job?
|
||||
gbrain jobs get <id>
|
||||
```
|
||||
|
||||
## Rescue actions (in order of escalation)
|
||||
|
||||
```bash
|
||||
# Force-kill a single stuck job:
|
||||
gbrain jobs cancel <id>
|
||||
|
||||
# Clear a specific job entirely (last resort):
|
||||
gbrain jobs delete <id>
|
||||
|
||||
# Health smoke on the mechanism itself:
|
||||
gbrain jobs smoke --wedge-rescue
|
||||
```
|
||||
|
||||
## What each subcheck means
|
||||
|
||||
- **stalled-forever** — A worker claimed a job, started executing, and has
|
||||
held the row for over an hour. The wall-clock sweep evicts jobs past
|
||||
2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
|
||||
or the sweep is newly deployed and this job predates it. Cancel it.
|
||||
- **waiting-depth** — Submitters are piling up jobs faster than workers
|
||||
drain them. Set `--max-waiting N` on the submission or on the programmatic
|
||||
`queue.add()` call. If you want a taller pile, raise the threshold via
|
||||
`GBRAIN_QUEUE_WAITING_THRESHOLD=50 gbrain doctor`.
|
||||
|
||||
## Self-check: is a worker even running?
|
||||
|
||||
```bash
|
||||
# If you're running autopilot with --no-worker, check that your external
|
||||
# worker (systemd / Docker / OpenClaw service-manager) is alive:
|
||||
gbrain jobs list --status active | head -5
|
||||
```
|
||||
|
||||
If the list is empty AND your submissions keep piling up, no worker is
|
||||
claiming. Start one:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
## Follow-ups tracked for v0.20+
|
||||
|
||||
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
|
||||
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
|
||||
subcheck both need this).
|
||||
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
|
||||
@@ -1,154 +0,0 @@
|
||||
# RLS and you
|
||||
|
||||
Short version: every table in your gbrain's `public` schema needs Row Level
|
||||
Security enabled. If one doesn't, `gbrain doctor` now fails, not warns, and the
|
||||
process exits 1.
|
||||
|
||||
This guide explains why, what to do when you hit the check, and the escape hatch
|
||||
for the cases where you really do want a table to stay readable by the anon key.
|
||||
|
||||
## Why RLS matters
|
||||
|
||||
Supabase exposes everything in the `public` schema via PostgREST. Whatever's
|
||||
there is reachable by the anon key, which is a client-side secret by design.
|
||||
If RLS is off on a public table, the anon key can read it. On anything sensitive
|
||||
(auth tokens, chat history, financial data) that's an exfiltration vector, not
|
||||
a footgun.
|
||||
|
||||
gbrain's service-role connection holds `BYPASSRLS`, so enabling RLS without
|
||||
policies does NOT break gbrain itself. It just blocks the anon key's default
|
||||
read. That's the security posture: deny-by-default to anon, full access for
|
||||
the service role.
|
||||
|
||||
## What to do when doctor fails
|
||||
|
||||
Doctor's message names every table missing RLS and gives you a `ALTER TABLE`
|
||||
line per table:
|
||||
|
||||
```
|
||||
1 table(s) WITHOUT Row Level Security: expenses_ramp.
|
||||
Fix: ALTER TABLE "public"."expenses_ramp" ENABLE ROW LEVEL SECURITY;
|
||||
If a table should stay readable by the anon key on purpose, see
|
||||
docs/guides/rls-and-you.md for the GBRAIN:RLS_EXEMPT comment escape hatch.
|
||||
```
|
||||
|
||||
99% of the time, you want the fix. Run the SQL. Re-run `gbrain doctor`. Done.
|
||||
|
||||
## The 1% case: deliberate exemption
|
||||
|
||||
Sometimes a public table is supposed to be readable by the anon key. An
|
||||
analytics view backing a public dashboard. A read-only reference table. A
|
||||
plugin that ships its own frontend and intentionally uses the anon key for
|
||||
reads.
|
||||
|
||||
gbrain has an escape hatch for these. It is deliberately painful to set up.
|
||||
That is the feature.
|
||||
|
||||
### The format
|
||||
|
||||
```sql
|
||||
-- In psql, connected as a BYPASSRLS role (e.g. postgres):
|
||||
COMMENT ON TABLE public.your_table IS
|
||||
'GBRAIN:RLS_EXEMPT reason=<why this is anon-readable on purpose>';
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- The comment value MUST start with `GBRAIN:RLS_EXEMPT` (case-sensitive).
|
||||
- It MUST include `reason=` followed by at least 4 characters of justification.
|
||||
- No other prefix, no checkbox in a config file, no environment variable. Only
|
||||
a Postgres table comment counts.
|
||||
- If RLS is also off on the table (which it must be for the anon key to
|
||||
actually read), you also need `ALTER TABLE ... DISABLE ROW LEVEL SECURITY;`
|
||||
explicitly. Disabling alone is not enough; the comment is what tells doctor
|
||||
this is intentional.
|
||||
|
||||
### Example
|
||||
|
||||
```sql
|
||||
ALTER TABLE public.expenses_ramp DISABLE ROW LEVEL SECURITY;
|
||||
COMMENT ON TABLE public.expenses_ramp IS
|
||||
'GBRAIN:RLS_EXEMPT reason=analytics-only, anon-readable ok, owner=garry, 2026-04-22';
|
||||
```
|
||||
|
||||
After that, `gbrain doctor` reports:
|
||||
|
||||
```
|
||||
rls: ok — RLS enabled on 20/21 public tables (1 explicitly exempt: expenses_ramp)
|
||||
```
|
||||
|
||||
Note that every subsequent run re-enumerates your exemptions by name. That's
|
||||
intentional. The escape hatch is not a one-time sign-off, it's a recurring
|
||||
reminder. If you ever want to know which tables are open, run `gbrain doctor`.
|
||||
|
||||
## Why SQL and not a CLI subcommand
|
||||
|
||||
gbrain does NOT ship a `gbrain rls-exempt add <table>` command. A CLI command
|
||||
would make it easy for an agent to silently open a table to anon reads. The
|
||||
comment-in-psql requirement forces the operator to type the justification
|
||||
in SQL, which is:
|
||||
|
||||
- Visible in shell history.
|
||||
- Visible in a git-tracked schema dump.
|
||||
- Visible in `pg_dump` output the next time you restore.
|
||||
- Visible in `gbrain doctor` output on every run.
|
||||
|
||||
An agent CAN still run the SQL, but it can't do it without the user seeing the
|
||||
action. That's the "write it in blood" design.
|
||||
|
||||
## Auditing exemptions later
|
||||
|
||||
To see every exemption in the current DB:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
c.relname AS table_name,
|
||||
obj_description(c.oid, 'pg_class') AS comment
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'public'
|
||||
AND c.relkind = 'r'
|
||||
AND obj_description(c.oid, 'pg_class') LIKE 'GBRAIN:RLS_EXEMPT%';
|
||||
```
|
||||
|
||||
If that list is longer than you remember signing off on, that's the signal.
|
||||
|
||||
## Removing an exemption
|
||||
|
||||
Just drop the comment and re-enable RLS:
|
||||
|
||||
```sql
|
||||
ALTER TABLE public.expenses_ramp ENABLE ROW LEVEL SECURITY;
|
||||
COMMENT ON TABLE public.expenses_ramp IS NULL;
|
||||
```
|
||||
|
||||
`gbrain doctor` stops listing the table as exempt and goes back to checking
|
||||
it like any other.
|
||||
|
||||
## PGLite
|
||||
|
||||
If you're on PGLite (the zero-config default), doctor skips this check
|
||||
entirely: PGLite is embedded, single-user, and has no PostgREST in front of
|
||||
it. The public-schema-exposure risk doesn't exist. You'll see:
|
||||
|
||||
```
|
||||
rls: ok — Skipped (PGLite — no PostgREST exposure, RLS not applicable)
|
||||
```
|
||||
|
||||
If you migrate to Supabase or self-hosted Postgres later, the check starts
|
||||
running and will flag any table that came over without RLS.
|
||||
|
||||
## Self-hosted Postgres
|
||||
|
||||
If you're running Postgres without PostgREST in front, the anon-key exposure
|
||||
doesn't apply. But gbrain still fails the check on missing RLS, because:
|
||||
|
||||
- The framing is "RLS on all public tables" is a gbrain security invariant,
|
||||
not a Supabase-specific workaround.
|
||||
- The `ALTER TABLE ... ENABLE RLS` fix is harmless on any Postgres: it only
|
||||
constrains non-bypass roles, which gbrain doesn't use.
|
||||
- If you ever put PostgREST or a similar tool in front later, the guard is
|
||||
already in place.
|
||||
|
||||
If this framing doesn't fit your deployment, file an issue with the specifics
|
||||
so we can decide whether a self-hosted-exempt mode is justified.
|
||||
@@ -62,13 +62,8 @@ secrets: # API keys and credentials needed
|
||||
- name: TWILIO_ACCOUNT_SID
|
||||
description: Twilio account SID
|
||||
where: https://console.twilio.com # exact URL to get this key
|
||||
health_checks: # typed DSL to verify the integration is working
|
||||
- type: http
|
||||
url: "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json"
|
||||
auth: basic
|
||||
auth_user: "$TWILIO_ACCOUNT_SID"
|
||||
auth_token: "$TWILIO_AUTH_TOKEN"
|
||||
label: "Twilio account"
|
||||
health_checks: # commands to verify the integration is working
|
||||
- "curl -sf https://api.twilio.com/..."
|
||||
setup_time: 30 min # estimated time to complete setup
|
||||
---
|
||||
|
||||
@@ -79,16 +74,6 @@ setup_time: 30 min # estimated time to complete setup
|
||||
the markdown body and executes the setup steps. It asks you for API keys, validates
|
||||
each one, configures the integration, and runs a smoke test.
|
||||
|
||||
### Recipe trust boundary
|
||||
|
||||
Only recipes shipped inside the gbrain package itself (the `recipes/` directory in
|
||||
a source install, or the global install copy) are trusted. Recipes discovered at
|
||||
runtime from `$GBRAIN_RECIPES_DIR` or a cwd-local `./recipes/` are marked untrusted:
|
||||
they cannot run `command` health checks, cannot run `http` health checks (SSRF
|
||||
defense), and cannot use the deprecated string health_check form. Untrusted recipes
|
||||
can still use `env_exists` and `any_of` compositions. To ship a recipe that runs
|
||||
live checks, contribute it upstream so it becomes package-bundled.
|
||||
|
||||
## The Deterministic Collector Pattern
|
||||
|
||||
When an LLM keeps failing at a mechanical task despite repeated prompt fixes,
|
||||
|
||||
@@ -1,105 +0,0 @@
|
||||
# Pre-commit hook for brain repos (v0.22.4+)
|
||||
|
||||
`gbrain frontmatter install-hook` installs a git pre-commit hook in your
|
||||
brain source's repo that runs `gbrain frontmatter validate` against staged
|
||||
`.md` and `.mdx` files. Malformed frontmatter blocks the commit. Bypass with
|
||||
`git commit --no-verify`.
|
||||
|
||||
## What the hook catches
|
||||
|
||||
The same seven validation classes the `frontmatter-guard` skill and
|
||||
`gbrain doctor`'s `frontmatter_integrity` subcheck report:
|
||||
|
||||
| Code | What it catches |
|
||||
|-------------------|---------------------------------------------------------------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading |
|
||||
| `YAML_PARSE` | YAML failed to parse (syntax or structure) |
|
||||
| `SLUG_MISMATCH` | `slug:` in frontmatter doesn't match path-derived slug |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) anywhere in the content |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape that breaks YAML |
|
||||
| `EMPTY_FRONTMATTER` | `---` ... `---` with nothing meaningful between |
|
||||
|
||||
## Install
|
||||
|
||||
For all registered sources that are git repos:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
For one source:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --source <id>
|
||||
```
|
||||
|
||||
For force-overwrite of an existing pre-commit hook (writes a `.bak`):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --force
|
||||
```
|
||||
|
||||
The hook lands at `<source>/.githooks/pre-commit`. If `core.hooksPath` is
|
||||
unset, the install also runs `git config core.hooksPath .githooks` so the
|
||||
hook is picked up without manual git config.
|
||||
|
||||
## Bypass
|
||||
|
||||
Standard git escape hatch:
|
||||
|
||||
```bash
|
||||
git commit --no-verify
|
||||
```
|
||||
|
||||
This skips ALL pre-commit hooks. Use sparingly — the next time the user
|
||||
runs `gbrain doctor`, the issues will surface.
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --uninstall
|
||||
```
|
||||
|
||||
If a `.bak` was saved during install, it's restored as the active hook.
|
||||
Otherwise the hook is removed cleanly.
|
||||
|
||||
## Behavior on machines without gbrain installed
|
||||
|
||||
The hook script checks for `gbrain` on `$PATH`. When missing, it prints a
|
||||
one-line warning to stderr and exits 0 — commits aren't blocked just because
|
||||
a developer hasn't installed gbrain locally. Once gbrain is installed, the
|
||||
hook resumes blocking malformed pages.
|
||||
|
||||
## For downstream agent forks
|
||||
|
||||
If your fork (Wintermute, Hermes, OpenClaw) wraps gbrain in a host repo
|
||||
that's not the brain repo itself, you may want a separate hook strategy:
|
||||
|
||||
- **Brain repo IS the host repo** (gbrain skills + brain pages in one repo):
|
||||
install via `gbrain frontmatter install-hook` as above.
|
||||
- **Brain repo is a separate registered source** (e.g. `~/brain` registered
|
||||
as a source, host repo is `~/agent-fork`): install in the brain repo only;
|
||||
agent-fork code doesn't need this hook.
|
||||
- **Brain repo is auto-generated** (e.g. by a sync daemon writing to a
|
||||
bucket): skip the hook entirely; gate at the writer instead via
|
||||
`import { writeBrainPage } from 'gbrain/brain-writer'` (planned in a
|
||||
later release; currently the CLI is the surface).
|
||||
|
||||
## How it fits into the broader frontmatter pipeline
|
||||
|
||||
```
|
||||
agent writes a page git commit doctor scan
|
||||
↓ ↓ ↓
|
||||
[source content] → [pre-commit hook validates] → [frontmatter_integrity check]
|
||||
↓ ↓ ↓
|
||||
raw file on disk blocks malformed commits surfaces existing issues
|
||||
↓
|
||||
`gbrain frontmatter validate
|
||||
<source-path> --fix`
|
||||
(writes .bak backups)
|
||||
```
|
||||
|
||||
The hook is the write-time gate; doctor is the audit gate; the CLI is the
|
||||
fix tool. They share `parseMarkdown(..., {validate:true})` as the single
|
||||
source of truth for what counts as malformed.
|
||||
@@ -1,66 +0,0 @@
|
||||
# Reliability repair (v0.12.2)
|
||||
|
||||
If you ran v0.12.0 on real Postgres or Supabase, two bugs may have corrupted
|
||||
data already in your brain. v0.12.1 fixed the code going forward.
|
||||
v0.12.2 adds detection in `gbrain doctor` and a standalone `gbrain repair-jsonb`
|
||||
command for the mechanically fixable class. PGLite users are not affected.
|
||||
|
||||
## What got corrupted
|
||||
|
||||
**JSONB double-encode.** Four write sites used
|
||||
`${JSON.stringify(x)}::jsonb` with postgres.js, which stored a JSONB
|
||||
*string literal* instead of an object. `frontmatter ->> 'key'` returns NULL;
|
||||
GIN indexes are ineffective. Affected: `pages.frontmatter`,
|
||||
`raw_data.data`, `ingest_log.pages_updated`, `files.metadata`.
|
||||
|
||||
**Markdown body truncation.** `splitBody()` treated `---` horizontal rules
|
||||
as a body/timeline delimiter, dropping everything after the first rule.
|
||||
Wiki-style pages with multiple `##`/`###` sections lost the bulk of their
|
||||
content at import time.
|
||||
|
||||
## Detect
|
||||
|
||||
```
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
Reports two new checks:
|
||||
|
||||
- `jsonb_integrity` — counts double-encoded rows per table and points you
|
||||
at `gbrain repair-jsonb`.
|
||||
- `markdown_body_completeness` — heuristic for pages whose `compiled_truth`
|
||||
is suspiciously short compared to `raw_data.data ->> 'content'`.
|
||||
|
||||
## Repair
|
||||
|
||||
For JSONB (mechanically fixable):
|
||||
|
||||
```
|
||||
gbrain repair-jsonb
|
||||
```
|
||||
|
||||
Runs `UPDATE <table> SET <col> = (<col>#>>'{}')::jsonb WHERE jsonb_typeof(<col>) = 'string'`
|
||||
across every affected column. Idempotent. Second run reports 0 rows. Use
|
||||
`--dry-run` to preview, `--json` for structured output. The `v0_12_2`
|
||||
migration runs this automatically on `gbrain upgrade`.
|
||||
|
||||
For truncated markdown bodies (source-dependent):
|
||||
|
||||
```
|
||||
gbrain sync --force
|
||||
# or per-page
|
||||
gbrain import <slug> --force
|
||||
```
|
||||
|
||||
v0.12.2 cannot recover content that was already lost if you no longer have
|
||||
the source markdown file. `gbrain doctor` tells you which pages look short;
|
||||
you decide whether to re-import from source or accept the truncation.
|
||||
|
||||
## Verify
|
||||
|
||||
```
|
||||
gbrain doctor
|
||||
```
|
||||
|
||||
All four `jsonb_integrity` rows should read zero. `markdown_body_completeness`
|
||||
should match your expectations for the corpus.
|
||||
@@ -78,13 +78,6 @@ bun run src/commands/auth.ts test \
|
||||
All 30 GBrain operations are available remotely, including `sync_brain` and
|
||||
`file_upload` (no timeout limits with self-hosted server).
|
||||
|
||||
**Security note on `file_upload`:** remote MCP callers are confined to the working
|
||||
directory where `gbrain serve` was launched. Symlinks, `..` traversal, and absolute
|
||||
paths outside cwd are rejected. Page slugs and filenames are allowlist-validated
|
||||
(alphanumeric + hyphens; no control chars, RTL overrides, or backslashes). Local
|
||||
CLI callers (`gbrain file upload ...`) keep unrestricted filesystem access since
|
||||
the user owns the machine.
|
||||
|
||||
## Deployment Options
|
||||
|
||||
See [ALTERNATIVES.md](ALTERNATIVES.md) for a comparison of ngrok, Tailscale
|
||||
|
||||
@@ -1,191 +0,0 @@
|
||||
# Progress events
|
||||
|
||||
Canonical reference for the JSONL progress stream that `gbrain` writes to
|
||||
`stderr` when a bulk command runs with `--progress-json`. Stable from
|
||||
v0.15.2. Additive changes only; no renames or removals without a major
|
||||
version bump.
|
||||
|
||||
Most humans won't read this page. Agents parsing progress will.
|
||||
|
||||
## When do I get these events?
|
||||
|
||||
Any of these commands stream events when `--progress-json` is set:
|
||||
|
||||
- `gbrain doctor` (DB checks, JSONB integrity, markdown body completeness,
|
||||
integrity sample)
|
||||
- `gbrain orphans`
|
||||
- `gbrain embed`
|
||||
- `gbrain files sync`
|
||||
- `gbrain export`
|
||||
- `gbrain extract [links|timeline|all]` (fs or db source)
|
||||
- `gbrain import`
|
||||
- `gbrain sync`
|
||||
- `gbrain migrate --to …`
|
||||
- `gbrain repair-jsonb`
|
||||
- `gbrain check-backlinks`
|
||||
- `gbrain lint`
|
||||
- `gbrain integrity auto`
|
||||
- `gbrain eval`
|
||||
- `gbrain apply-migrations` (the orchestrator + every child command)
|
||||
|
||||
Non-bulk commands (`stats`, `graph-query`, `get`, `put`, etc.) don't emit
|
||||
events — they return in under a second.
|
||||
|
||||
## Channel
|
||||
|
||||
- Progress events: **`stderr`**, one JSON object per line, `\n`-terminated.
|
||||
- Data results (`--json` payloads from each command): **`stdout`**.
|
||||
- Final human summaries: **`stdout`**.
|
||||
|
||||
Agents can safely capture stdout for their result parsing and read stderr
|
||||
separately for progress.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Behavior |
|
||||
|---|---|
|
||||
| *(none)* | Auto. TTY: `\r`-rewriting single line. Non-TTY: plain line-per-event on stderr. |
|
||||
| `--progress-json` | Force JSON-lines mode on stderr (this doc). |
|
||||
| `--quiet` | Suppress progress entirely. Warnings and final output still print. |
|
||||
| `--progress-interval=<ms>` | Override the minimum interval between tick emits (default 1000). |
|
||||
|
||||
Global flags: parsed by `src/core/cli-options.ts` before command dispatch,
|
||||
so `gbrain --progress-json doctor` works the same as
|
||||
`gbrain doctor --progress-json` (the latter also works — per-command
|
||||
parsers see the flag via the shared `CliOptions` singleton).
|
||||
|
||||
## Event types
|
||||
|
||||
Every event is a single-line JSON object with these common fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `event` | string | One of: `start`, `tick`, `heartbeat`, `finish`, `abort`. |
|
||||
| `phase` | string | Machine-stable snake_case, dot-separated. See "Phase names" below. |
|
||||
| `ts` | ISO 8601 UTC string | Event emission time. |
|
||||
| `elapsed_ms` | number | Ms since the phase started. Present on `tick`/`heartbeat`/`finish`/`abort`. |
|
||||
|
||||
### `start`
|
||||
|
||||
Emitted when a phase begins.
|
||||
|
||||
```json
|
||||
{"event":"start","phase":"doctor.db_checks","ts":"2026-04-20T12:34:56.789Z"}
|
||||
{"event":"start","phase":"import.files","total":52000,"ts":"2026-04-20T12:34:56.789Z"}
|
||||
```
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `total` — the total item count if known at start.
|
||||
|
||||
### `tick`
|
||||
|
||||
Emitted periodically during iteration. Time- and item-gated: the reporter
|
||||
won't emit more often than `minIntervalMs` (default 1000) and
|
||||
`minItems` (default `max(10, ceil(total/100))`).
|
||||
|
||||
```json
|
||||
{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"eta_ms":10300,"ts":"..."}
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `done` — items completed in this phase.
|
||||
- `total` — total items, if known. Omitted when the scan doesn't have a
|
||||
total up front (e.g. a streaming iterator).
|
||||
- `pct` — `done/total * 100`, one decimal. Omitted when `total` is unknown.
|
||||
- `eta_ms` — projected ms until `done === total`, from the observed rate.
|
||||
Omitted when `total` is unknown.
|
||||
- `note` — optional string with the current item (e.g. a slug or filename).
|
||||
|
||||
### `heartbeat`
|
||||
|
||||
Emitted for long-running single operations that don't iterate
|
||||
(e.g. `SELECT` against a 50K-row table). No `done`, no `total` — just a
|
||||
signal that work is still happening.
|
||||
|
||||
```json
|
||||
{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation…","elapsed_ms":1000,"ts":"..."}
|
||||
```
|
||||
|
||||
### `finish`
|
||||
|
||||
Emitted when a phase completes normally.
|
||||
|
||||
```json
|
||||
{"event":"finish","phase":"import.files","done":52000,"total":52000,"elapsed_ms":187000,"ts":"..."}
|
||||
```
|
||||
|
||||
### `abort`
|
||||
|
||||
Emitted by a single process-level SIGINT/SIGTERM handler that tracks every
|
||||
live phase. After `abort`, no further events emit for that phase.
|
||||
|
||||
```json
|
||||
{"event":"abort","phase":"doctor.markdown_body_completeness","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}
|
||||
```
|
||||
|
||||
## Phase names
|
||||
|
||||
Phases use `snake_case.dot.path` naming. A fresh reporter starts at the
|
||||
root; `child()` composition appends to the parent's current phase, so a
|
||||
sync that calls import emits `sync.import.<file>`, not `import.<file>`.
|
||||
|
||||
Stable phase names shipped in v0.15.2:
|
||||
|
||||
- `doctor.db_checks` (umbrella for all DB-side doctor checks)
|
||||
- `orphans.scan`
|
||||
- `embed.pages`
|
||||
- `extract.links_fs`, `extract.timeline_fs`, `extract.links_db`, `extract.timeline_db`
|
||||
- `import.files`
|
||||
- `sync.deletes`, `sync.renames`, `sync.imports`
|
||||
- `migrate.copy_pages`, `migrate.copy_links`
|
||||
- `repair_jsonb.run`, `repair_jsonb.<table>.<column>`
|
||||
- `backlinks.scan`
|
||||
- `lint.pages`
|
||||
- `integrity.auto`
|
||||
- `eval.single`, `eval.ab`
|
||||
- `export.pages`
|
||||
- `files.sync`
|
||||
|
||||
Sub-phases exposed via `child()`:
|
||||
|
||||
- `sync.import.files` — nested inside a sync
|
||||
- `apply_migrations.v0_12_2.jsonb_repair` — nested inside the orchestrator
|
||||
|
||||
## Subprocess inheritance
|
||||
|
||||
When a parent CLI spawns `gbrain …` child processes (mostly in
|
||||
`src/commands/migrations/*`), global flags (`--quiet`, `--progress-json`,
|
||||
`--progress-interval`) are propagated to the child's argv via the
|
||||
`childGlobalFlags()` helper in `src/core/cli-options.ts`. Child stderr
|
||||
passes straight through `stdio: 'inherit'` so the event stream is one
|
||||
merged JSONL feed on the parent's stderr.
|
||||
|
||||
One exception: the orchestrator phase in `migrations/v0_12_2.ts` that
|
||||
captures child stdout (`repair-jsonb --dry-run --json` for verification)
|
||||
does not pass `--progress-json` to avoid any risk of stdout pollution
|
||||
breaking the orchestrator's `JSON.parse`. Its stdio is explicit:
|
||||
`['ignore', 'pipe', 'inherit']` so stderr still flows through.
|
||||
|
||||
## Minion jobs
|
||||
|
||||
`gbrain jobs work` (the Minion worker daemon) keeps progress in the DB,
|
||||
not on stderr. Each Minion handler that runs a bulk core (embed, sync,
|
||||
extract, import, backlinks) calls `job.updateProgress({done, total,
|
||||
…})` per iteration. Agents read per-job progress via the
|
||||
`get_job_progress` MCP operation or `gbrain jobs get <id>`.
|
||||
|
||||
The `jobs work` daemon itself emits coarse one-line-per-job stderr output
|
||||
for liveness only. Per-page detail lives in the DB.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Added**: only. A new event type, a new field, a new phase name — all
|
||||
safe. Agents must ignore unknown fields and unknown event types.
|
||||
- **Removed/renamed**: never without a major version bump.
|
||||
- **Schema changes**: announced in `CHANGELOG.md` and in
|
||||
`skills/migrations/v<next>.md`.
|
||||
|
||||
If your agent depends on this schema and something surprises you, open
|
||||
an issue with the event you received and what you expected.
|
||||
-5319
File diff suppressed because it is too large
Load Diff
@@ -1,52 +0,0 @@
|
||||
# GBrain
|
||||
|
||||
> GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.
|
||||
|
||||
Repo: https://github.com/garrytan/gbrain
|
||||
|
||||
## Core entry points
|
||||
|
||||
- [AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md): Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.
|
||||
- [CLAUDE.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CLAUDE.md): Architecture reference. Key files, trust boundaries, engine factory, test layout.
|
||||
- [INSTALL_FOR_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md): 9-step agent installation.
|
||||
- [skills/RESOLVER.md](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/RESOLVER.md): Skill dispatcher. Read first for any task.
|
||||
- [README.md](https://raw.githubusercontent.com/garrytan/gbrain/master/README.md): Project overview, benchmarks, 30-minute setup.
|
||||
|
||||
## Configuration
|
||||
|
||||
- [docs/ENGINES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ENGINES.md): PGLite vs Postgres trade-off and when to migrate.
|
||||
- [docs/GBRAIN_RECOMMENDED_SCHEMA.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md): MECE directory structure (people/, companies/, concepts/).
|
||||
- [docs/guides/live-sync.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md): Incremental markdown sync setup.
|
||||
- [docs/guides/cron-schedule.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/cron-schedule.md): Recurring job scheduling.
|
||||
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
|
||||
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
|
||||
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
|
||||
|
||||
## Debugging
|
||||
|
||||
- [docs/GBRAIN_VERIFY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_VERIFY.md): 7-check post-setup verification. Start here when something feels off.
|
||||
- [docs/guides/minions-fix.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-fix.md): Troubleshooting the Minions job queue.
|
||||
- [docs/integrations/reliability-repair.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/integrations/reliability-repair.md): Data integrity recovery.
|
||||
|
||||
## Migrations
|
||||
|
||||
- [docs/UPGRADING_DOWNSTREAM_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/UPGRADING_DOWNSTREAM_AGENTS.md): Patches for downstream agent skill forks. One section per release.
|
||||
- [skills/migrations/](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/migrations/): Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.
|
||||
- [CHANGELOG.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CHANGELOG.md): Release-summary voice + itemized changes + self-repair block per version.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- [docs/ethos/THIN_HARNESS_FAT_SKILLS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/THIN_HARNESS_FAT_SKILLS.md): Why skills live in markdown.
|
||||
- [docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md): Homebrew for Personal AI.
|
||||
|
||||
## Optional
|
||||
|
||||
- [docs/designs/](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/designs/): Forward-looking designs.
|
||||
- [docs/architecture/infra-layer.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/architecture/infra-layer.md): Shared infra patterns.
|
||||
|
||||
## Operational tips
|
||||
|
||||
- `gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.
|
||||
- `gbrain orphans [--json]` - pages with zero inbound wikilinks.
|
||||
- `gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.
|
||||
- `gbrain upgrade` runs post-upgrade + apply-migrations.
|
||||
+5
-34
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.19.0",
|
||||
"version": "0.4.1",
|
||||
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
|
||||
"family": "bundle-plugin",
|
||||
"configSchema": {
|
||||
@@ -24,42 +24,13 @@
|
||||
}
|
||||
},
|
||||
"skills": [
|
||||
"skills/brain-ops",
|
||||
"skills/briefing",
|
||||
"skills/citation-fixer",
|
||||
"skills/cross-modal-review",
|
||||
"skills/cron-scheduler",
|
||||
"skills/daily-task-manager",
|
||||
"skills/daily-task-prep",
|
||||
"skills/data-research",
|
||||
"skills/enrich",
|
||||
"skills/idea-ingest",
|
||||
"skills/ingest",
|
||||
"skills/maintain",
|
||||
"skills/media-ingest",
|
||||
"skills/meeting-ingestion",
|
||||
"skills/minion-orchestrator",
|
||||
"skills/query",
|
||||
"skills/reports",
|
||||
"skills/repo-architecture",
|
||||
"skills/signal-detector",
|
||||
"skills/skill-creator",
|
||||
"skills/skillify",
|
||||
"skills/skillpack-check",
|
||||
"skills/soul-audit",
|
||||
"skills/testing",
|
||||
"skills/webhook-transforms"
|
||||
],
|
||||
"shared_deps": [
|
||||
"skills/conventions",
|
||||
"skills/_brain-filing-rules.md",
|
||||
"skills/_brain-filing-rules.json",
|
||||
"skills/_output-rules.md"
|
||||
],
|
||||
"excluded_from_install": [
|
||||
"skills/setup",
|
||||
"skills/maintain",
|
||||
"skills/enrich",
|
||||
"skills/briefing",
|
||||
"skills/migrate",
|
||||
"skills/publish"
|
||||
"skills/setup"
|
||||
],
|
||||
"openclaw": {
|
||||
"compat": {
|
||||
|
||||
+7
-33
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.6",
|
||||
"version": "0.10.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -11,34 +11,15 @@
|
||||
".": "./src/core/index.ts",
|
||||
"./engine": "./src/core/engine.ts",
|
||||
"./types": "./src/core/types.ts",
|
||||
"./operations": "./src/core/operations.ts",
|
||||
"./minions": "./src/core/minions/index.ts",
|
||||
"./engine-factory": "./src/core/engine-factory.ts",
|
||||
"./pglite-engine": "./src/core/pglite-engine.ts",
|
||||
"./link-extraction": "./src/core/link-extraction.ts",
|
||||
"./import-file": "./src/core/import-file.ts",
|
||||
"./transcription": "./src/core/transcription.ts",
|
||||
"./embedding": "./src/core/embedding.ts",
|
||||
"./config": "./src/core/config.ts",
|
||||
"./markdown": "./src/core/markdown.ts",
|
||||
"./backoff": "./src/core/backoff.ts",
|
||||
"./search/hybrid": "./src/core/search/hybrid.ts",
|
||||
"./search/expansion": "./src/core/search/expansion.ts",
|
||||
"./extract": "./src/commands/extract.ts"
|
||||
"./operations": "./src/core/operations.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "bun run src/cli.ts",
|
||||
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
||||
"test": "bun test",
|
||||
"test:e2e": "bun test test/e2e/",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
},
|
||||
@@ -50,23 +31,16 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@dqbd/tiktoken": "^1.0.22",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@electric-sql/pglite": "^0.4.4",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6"
|
||||
"postgres": "^3.4.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.6.0"
|
||||
"@types/bun": "latest"
|
||||
},
|
||||
"trustedDependencies": [
|
||||
"@electric-sql/pglite"
|
||||
],
|
||||
"license": "MIT"
|
||||
}
|
||||
|
||||
@@ -1,193 +0,0 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* build-llms — generate llms.txt + llms-full.txt from scripts/llms-config.ts.
|
||||
*
|
||||
* Run: `bun run build:llms` (or `bun run scripts/build-llms.ts`).
|
||||
*
|
||||
* Outputs:
|
||||
* - llms.txt — llmstxt.org-spec index (H1 / blockquote / H2 sections).
|
||||
* - llms-full.txt — concatenated full content of non-optional entries.
|
||||
*
|
||||
* Deterministic: no timestamps, sorted within categories by config order.
|
||||
* Warns (does not fail) if llms-full.txt exceeds FULL_SIZE_BUDGET. CI catches
|
||||
* drift via test/build-llms.test.ts.
|
||||
*
|
||||
* Fork override: set LLMS_REPO_BASE to regenerate with a different URL base.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
import { join, dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import {
|
||||
FULL_SIZE_BUDGET,
|
||||
INLINE_TIPS,
|
||||
PROJECT,
|
||||
SECTIONS,
|
||||
type DocEntry,
|
||||
type DocSection,
|
||||
} from "./llms-config";
|
||||
|
||||
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
function urlFor(entry: DocEntry): string {
|
||||
return `${PROJECT.rawBaseUrl}/${entry.path}`;
|
||||
}
|
||||
|
||||
function isDirectoryPath(path: string): boolean {
|
||||
return path.endsWith("/");
|
||||
}
|
||||
|
||||
function renderLlmsTxt(): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${PROJECT.name}`);
|
||||
lines.push("");
|
||||
lines.push(`> ${PROJECT.summary}`);
|
||||
lines.push("");
|
||||
lines.push(`Repo: ${PROJECT.repoUrl}`);
|
||||
lines.push("");
|
||||
|
||||
for (const section of SECTIONS) {
|
||||
lines.push(`## ${section.heading}`);
|
||||
lines.push("");
|
||||
for (const entry of section.entries) {
|
||||
lines.push(
|
||||
`- [${entry.title}](${urlFor(entry)}): ${entry.description}`,
|
||||
);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
lines.push("## Operational tips");
|
||||
lines.push("");
|
||||
for (const tip of INLINE_TIPS) {
|
||||
lines.push(`- ${tip}`);
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function renderLlmsFullTxt(): { content: string; sizes: Array<{ path: string; bytes: number }> } {
|
||||
const lines: string[] = [];
|
||||
const sizes: Array<{ path: string; bytes: number }> = [];
|
||||
|
||||
lines.push(`# ${PROJECT.name} — Full Context`);
|
||||
lines.push("");
|
||||
lines.push(`> ${PROJECT.summary}`);
|
||||
lines.push("");
|
||||
lines.push(
|
||||
`This file concatenates core GBrain documentation for single-fetch ingestion.`,
|
||||
);
|
||||
lines.push(
|
||||
`For the link-only index, see \`llms.txt\`. Source of truth: ${PROJECT.repoUrl}.`,
|
||||
);
|
||||
lines.push("");
|
||||
|
||||
for (const section of SECTIONS) {
|
||||
if (section.optional) continue;
|
||||
lines.push(`# ${section.heading}`);
|
||||
lines.push("");
|
||||
for (const entry of section.entries) {
|
||||
if (entry.includeInFull === false) continue;
|
||||
if (isDirectoryPath(entry.path)) continue;
|
||||
|
||||
const absPath = join(repoRoot, entry.path);
|
||||
if (!existsSync(absPath)) {
|
||||
// build-llms won't silently skip — surface the problem. Test case 1
|
||||
// catches this too, but fail fast for manual runs.
|
||||
throw new Error(
|
||||
`llms-config references missing file: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
|
||||
const body = readFileSync(absPath, "utf8");
|
||||
const bytes = Buffer.byteLength(body, "utf8");
|
||||
sizes.push({ path: entry.path, bytes });
|
||||
|
||||
lines.push(`## ${entry.path}`);
|
||||
lines.push("");
|
||||
lines.push(`Source: ${urlFor(entry)}`);
|
||||
lines.push("");
|
||||
lines.push(body.trimEnd());
|
||||
lines.push("");
|
||||
lines.push("---");
|
||||
lines.push("");
|
||||
}
|
||||
}
|
||||
|
||||
return { content: lines.join("\n"), sizes };
|
||||
}
|
||||
|
||||
function validateConfig(): void {
|
||||
for (const section of SECTIONS) {
|
||||
for (const entry of section.entries) {
|
||||
const absPath = join(repoRoot, entry.path);
|
||||
if (!existsSync(absPath)) {
|
||||
throw new Error(
|
||||
`llms-config references missing path: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
const st = statSync(absPath);
|
||||
if (isDirectoryPath(entry.path) && !st.isDirectory()) {
|
||||
throw new Error(
|
||||
`llms-config path ends with '/' but is a file: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
if (!isDirectoryPath(entry.path) && !st.isFile()) {
|
||||
throw new Error(
|
||||
`llms-config path is a directory but missing trailing '/': ${entry.path}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function buildLlmsFiles(): {
|
||||
llmsTxt: string;
|
||||
llmsFullTxt: string;
|
||||
sizes: Array<{ path: string; bytes: number }>;
|
||||
} {
|
||||
validateConfig();
|
||||
const llmsTxt = renderLlmsTxt();
|
||||
const { content: llmsFullTxt, sizes } = renderLlmsFullTxt();
|
||||
return { llmsTxt, llmsFullTxt, sizes };
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const { llmsTxt, llmsFullTxt, sizes } = buildLlmsFiles();
|
||||
|
||||
const llmsPath = join(repoRoot, "llms.txt");
|
||||
const llmsFullPath = join(repoRoot, "llms-full.txt");
|
||||
|
||||
writeFileSync(llmsPath, llmsTxt);
|
||||
writeFileSync(llmsFullPath, llmsFullTxt);
|
||||
|
||||
const fullBytes = Buffer.byteLength(llmsFullTxt, "utf8");
|
||||
console.log(`wrote ${llmsPath} (${Buffer.byteLength(llmsTxt, "utf8")} bytes)`);
|
||||
console.log(`wrote ${llmsFullPath} (${fullBytes} bytes)`);
|
||||
|
||||
if (fullBytes > FULL_SIZE_BUDGET) {
|
||||
console.warn("");
|
||||
console.warn(
|
||||
`WARN: llms-full.txt (${fullBytes} bytes) exceeds FULL_SIZE_BUDGET (${FULL_SIZE_BUDGET} bytes).`,
|
||||
);
|
||||
console.warn(
|
||||
"Add `includeInFull: false` to the biggest entries in scripts/llms-config.ts:",
|
||||
);
|
||||
const sorted = [...sizes].sort((a, b) => b.bytes - a.bytes);
|
||||
for (const entry of sorted.slice(0, 5)) {
|
||||
console.warn(` ${entry.bytes} bytes ${entry.path}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isMainModule = fileURLToPath(import.meta.url) === process.argv[1];
|
||||
if (isMainModule) {
|
||||
try {
|
||||
main();
|
||||
} catch (err) {
|
||||
console.error(err instanceof Error ? err.message : err);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any source file uses the buggy `${JSON.stringify(x)}::jsonb`
|
||||
# template-string pattern instead of postgres.js's `sql.json(x)`.
|
||||
#
|
||||
# This is best-effort static analysis. It catches the common copy-paste form
|
||||
# that caused the v0.12.0 silent-data-loss bug (JSONB columns stored as
|
||||
# string literals on Postgres while PGLite hid the bug). Multi-line and
|
||||
# helper-wrapped variants are NOT caught here — those are covered by
|
||||
# test/e2e/postgres-jsonb.test.ts which round-trips actual writes through
|
||||
# real Postgres and asserts `frontmatter->>'k'` returns objects, not strings.
|
||||
#
|
||||
# Usage: scripts/check-jsonb-pattern.sh
|
||||
# Exit: 0 when no matches, 1 when matches found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Match the interpolated form: ${JSON.stringify(...)}::jsonb
|
||||
# Using grep -P for Perl-compatible regex (lookahead-free pattern is enough here).
|
||||
PATTERN='\$\{JSON\.stringify\([^)]*\)\}::jsonb'
|
||||
|
||||
if grep -rEn "$PATTERN" src/ 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Found JSON.stringify(...)::jsonb pattern in src/."
|
||||
echo " postgres.js v3 stringifies again, producing JSONB string literals."
|
||||
echo " Use sql.json(x) instead. See feedback_postgres_jsonb_double_encode.md."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no JSON.stringify(x)::jsonb interpolation pattern in src/"
|
||||
|
||||
# v0.13.1 #219: guard against max_stalled DEFAULT 1 regressing in any schema
|
||||
# source file. DEFAULT 1 dead-lettered any SIGKILL'd job on first stall, making
|
||||
# the "10/10 rescued" claim false for out-of-the-box users. Default is 5 now.
|
||||
MAX_STALLED_PATTERN='max_stalled\s+INTEGER\s+NOT\s+NULL\s+DEFAULT\s+1\b'
|
||||
|
||||
if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/pglite-schema.ts src/core/schema-embedded.ts 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: max_stalled DEFAULT 1 reintroduced in schema."
|
||||
echo " Must be DEFAULT 5 to preserve SIGKILL-rescue guarantee. See #219."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: max_stalled defaults are 5 in all schema sources"
|
||||
@@ -1,133 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# check-privacy.sh — CLAUDE.md:550 enforcement.
|
||||
#
|
||||
# CLAUDE.md forbids the private OpenClaw fork name in public artifacts:
|
||||
# CHANGELOG.md, README.md, docs/, skills/, PR titles + bodies, commit
|
||||
# messages, and comments in checked-in code. This script greps for the
|
||||
# banned name in either the staged index (for pre-commit hooks) or the
|
||||
# working tree (for CI) and fails loudly if found.
|
||||
#
|
||||
# The allow-list below whitelists files where the name is legitimate
|
||||
# — specifically, this script itself (where we reference the name to
|
||||
# describe the rule) and upgrade guides that historically referenced
|
||||
# the pre-rename fork name.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/check-privacy.sh # scan working tree
|
||||
# scripts/check-privacy.sh --staged # scan git staged index
|
||||
# scripts/check-privacy.sh --help
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 clean
|
||||
# 1 banned name found (or --help printed)
|
||||
# 2 git / grep not available
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
BANNED_NAME='wintermute'
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
scripts/check-privacy.sh — scan for the banned OpenClaw fork name.
|
||||
|
||||
USAGE:
|
||||
scripts/check-privacy.sh Scan all tracked files in the working tree.
|
||||
scripts/check-privacy.sh --staged Scan only files staged for commit.
|
||||
scripts/check-privacy.sh --help Show this message.
|
||||
|
||||
The script greps for '${BANNED_NAME}' (case-insensitive) in:
|
||||
- CHANGELOG.md, README.md
|
||||
- docs/**
|
||||
- skills/**
|
||||
- src/**
|
||||
- test/**
|
||||
- scripts/**
|
||||
|
||||
Allow-list (references to the name are permitted):
|
||||
- scripts/check-privacy.sh itself
|
||||
- docs/UPGRADING_DOWNSTREAM_AGENTS.md (historical context for pre-rename upgrades)
|
||||
- .git/** (branch names, commit history — not checked in artifacts)
|
||||
|
||||
Exit codes: 0 clean, 1 banned name found, 2 setup error.
|
||||
EOF
|
||||
}
|
||||
|
||||
MODE=working
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--staged) MODE=staged ;;
|
||||
--help|-h) usage; exit 1 ;;
|
||||
*)
|
||||
echo "Unknown argument: $arg" >&2
|
||||
usage >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if ! command -v git >/dev/null 2>&1; then
|
||||
echo "check-privacy: git not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Build the file list by scanning-mode.
|
||||
if [ "$MODE" = staged ]; then
|
||||
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null || true)
|
||||
else
|
||||
FILES=$(git ls-files 2>/dev/null || true)
|
||||
fi
|
||||
|
||||
if [ -z "$FILES" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Allow-list: files in which the banned name is legitimate.
|
||||
# Meta-rule docs (define the rule itself), auto-generated LLM indexes,
|
||||
# historical upgrade guides, and the test that enforces the rule
|
||||
# against recipes/ all reference the banned name by necessity.
|
||||
ALLOW_LIST=(
|
||||
'scripts/check-privacy.sh'
|
||||
'CLAUDE.md'
|
||||
'llms-full.txt'
|
||||
'docs/UPGRADING_DOWNSTREAM_AGENTS.md'
|
||||
'test/integrations.test.ts'
|
||||
)
|
||||
|
||||
is_allowed() {
|
||||
local f="$1"
|
||||
for a in "${ALLOW_LIST[@]}"; do
|
||||
if [ "$f" = "$a" ]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
FOUND=0
|
||||
while IFS= read -r file; do
|
||||
[ -z "$file" ] && continue
|
||||
[ ! -f "$file" ] && continue
|
||||
if is_allowed "$file"; then
|
||||
continue
|
||||
fi
|
||||
# Case-insensitive grep; only specific extensions + known docs.
|
||||
case "$file" in
|
||||
*.md|*.ts|*.mjs|*.js|*.py|*.sh|*.json|*.yaml|*.yml|*.txt|README*|CHANGELOG*|CLAUDE*|AGENTS*)
|
||||
if grep -in "$BANNED_NAME" "$file" >/dev/null 2>&1; then
|
||||
echo "[check-privacy] BANNED NAME in $file:" >&2
|
||||
grep -in "$BANNED_NAME" "$file" | sed 's|^| |' >&2
|
||||
FOUND=1
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done <<< "$FILES"
|
||||
|
||||
if [ "$FOUND" -eq 1 ]; then
|
||||
echo "" >&2
|
||||
echo "The private OpenClaw fork name is banned in public artifacts." >&2
|
||||
echo "CLAUDE.md:550. Replace with 'your OpenClaw', 'OpenClaw reference deployment', or 'openclaw-reference'." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any new code emits \r-progress to stdout.
|
||||
#
|
||||
# Since v0.14.2, bulk-action progress lives on stderr via the shared
|
||||
# src/core/progress.ts reporter. \r-rewriting on stdout breaks every
|
||||
# piped-output scenario: agents that capture stdout for structured
|
||||
# results see progress garbage mixed with the data, and CI logs show
|
||||
# a single line per command because everything after the last \r
|
||||
# is truncated by the terminal emulator when played back.
|
||||
#
|
||||
# This script greps for the anti-pattern. Legitimate uses of \r inside
|
||||
# string literals (e.g. Windows line-ending normalization, regex
|
||||
# patterns) are expected to contain \r without being preceded by
|
||||
# `process.stdout.write`. We match the full write-call form only.
|
||||
#
|
||||
# Usage: scripts/check-progress-to-stdout.sh
|
||||
# Exit: 0 when clean, 1 when a banned pattern is found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# The banned pattern: process.stdout.write('\r... or process.stdout.write("\r...
|
||||
# Greedy quote character class so both quote styles match.
|
||||
PATTERN="process\.stdout\.write\([\`'\"]\\\\r"
|
||||
|
||||
# Files allowed to use this pattern historically. Empty allowlist — the point
|
||||
# of v0.14.2 was to remove every one of them. Add entries only if you really
|
||||
# need a \r on stdout (if so, add the rationale as a comment at the call site
|
||||
# and list the file here).
|
||||
ALLOWLIST=()
|
||||
|
||||
matches=""
|
||||
if command -v rg >/dev/null 2>&1; then
|
||||
matches="$(rg -n --no-heading "$PATTERN" src/ 2>/dev/null || true)"
|
||||
else
|
||||
matches="$(grep -rEn "$PATTERN" src/ 2>/dev/null || true)"
|
||||
fi
|
||||
|
||||
if [ -n "$matches" ]; then
|
||||
# Filter out allowlisted files.
|
||||
filtered="$matches"
|
||||
for f in "${ALLOWLIST[@]:-}"; do
|
||||
[ -z "$f" ] && continue
|
||||
filtered="$(echo "$filtered" | grep -v "^${f}:" || true)"
|
||||
done
|
||||
|
||||
if [ -n "$filtered" ]; then
|
||||
echo "ERROR: found process.stdout.write('\\r…') pattern(s) in src/:"
|
||||
echo
|
||||
echo "$filtered"
|
||||
echo
|
||||
echo "Bulk-action progress must go through src/core/progress.ts"
|
||||
echo "(writes to stderr, handles TTY vs non-TTY, honors --quiet /"
|
||||
echo " --progress-json / --progress-interval). If you genuinely"
|
||||
echo "need a \\r on stdout, add the file to the ALLOWLIST at the"
|
||||
echo "top of this script and explain why at the call site."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "check-progress-to-stdout: OK (no banned stdout \\r patterns)"
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: verify that bun --compile binaries ship with embedded tree-sitter
|
||||
# WASMs and produce real semantic chunks (not recursive-fallback chunks).
|
||||
#
|
||||
# This is the #1 silent-failure mode for v0.19.0 code indexing. If the WASM
|
||||
# import attributes regress or the asset path drifts, the compiled binary
|
||||
# silently falls through to the recursive text chunker. Users see no error,
|
||||
# just degraded chunking quality. This script catches that regression.
|
||||
#
|
||||
# Fails the build when:
|
||||
# - bun build --compile fails
|
||||
# - The resulting binary can't parse TypeScript
|
||||
# - Chunks come back without real symbol names (fallback signature)
|
||||
#
|
||||
# Runs as part of `bun test` via the package.json pre-test pipeline.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
OUT_BIN="$(mktemp /tmp/gbrain-wasm-check.XXXXXX)"
|
||||
trap 'rm -f "$OUT_BIN"' EXIT
|
||||
|
||||
# Build a minimal smoketest binary that imports the chunker. We compile this
|
||||
# instead of the full gbrain CLI so the failure mode is laser-focused on
|
||||
# chunker + WASM path resolution, not unrelated CLI wiring.
|
||||
bun build --compile --outfile "$OUT_BIN" scripts/chunker-smoketest.ts >/dev/null 2>&1
|
||||
|
||||
# Run it and capture JSON output.
|
||||
OUTPUT="$("$OUT_BIN" 2>&1)"
|
||||
|
||||
# Sanity: JSON parses and has expected shape.
|
||||
# - has_symbol_names: at least one chunk carries a concrete symbol name
|
||||
# (proves tree-sitter AST extraction, not recursive-fallback chunks).
|
||||
# - has_typescript_header: the structured header is emitted with the
|
||||
# correct language tag (proves the language map reached displayLang).
|
||||
# - calculateScore by name: specific function that MUST appear as a
|
||||
# top-level semantic node. If it's missing, the chunker either fell
|
||||
# through to recursive or the TypeScript grammar didn't load.
|
||||
if ! echo "$OUTPUT" | grep -q '"has_symbol_names": true'; then
|
||||
echo "[check-wasm-embedded] FAIL: compiled binary returned no symbol names (fallback chunks)." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! echo "$OUTPUT" | grep -q '"has_typescript_header": true'; then
|
||||
echo "[check-wasm-embedded] FAIL: chunk header missing TypeScript language tag." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! echo "$OUTPUT" | grep -q '"calculateScore"'; then
|
||||
echo "[check-wasm-embedded] FAIL: tree-sitter did not extract the calculateScore function symbol." >&2
|
||||
echo "[check-wasm-embedded] Output was:" >&2
|
||||
echo "$OUTPUT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[check-wasm-embedded] OK — compiled binary produced real semantic chunks."
|
||||
@@ -1,51 +0,0 @@
|
||||
import { chunkCodeText } from '../src/core/chunkers/code.ts';
|
||||
|
||||
// Large function body so it doesn't merge with siblings — the CI guard
|
||||
// needs at least one chunk with a concrete symbol name to prove the
|
||||
// tree-sitter WASM is actually resolving (not just recursive fallback).
|
||||
const src = `export function calculateScore(
|
||||
items: Array<{ value: number; weight: number }>,
|
||||
opts: { normalize?: boolean; cap?: number } = {}
|
||||
): number {
|
||||
if (items.length === 0) return 0;
|
||||
const sum = items.reduce((acc, it) => acc + it.value * it.weight, 0);
|
||||
const totalWeight = items.reduce((acc, it) => acc + it.weight, 0);
|
||||
if (totalWeight === 0) return 0;
|
||||
const raw = sum / totalWeight;
|
||||
if (opts.normalize) {
|
||||
const clamped = Math.max(0, Math.min(1, raw));
|
||||
return opts.cap !== undefined ? Math.min(opts.cap, clamped) : clamped;
|
||||
}
|
||||
return opts.cap !== undefined ? Math.min(opts.cap, raw) : raw;
|
||||
}
|
||||
|
||||
export class UserRegistry {
|
||||
private users: Map<string, { name: string; score: number }> = new Map();
|
||||
|
||||
register(id: string, name: string, score: number): void {
|
||||
this.users.set(id, { name, score });
|
||||
}
|
||||
|
||||
lookup(id: string): { name: string; score: number } | null {
|
||||
return this.users.get(id) ?? null;
|
||||
}
|
||||
|
||||
topK(k: number): Array<{ id: string; name: string; score: number }> {
|
||||
const entries = Array.from(this.users.entries());
|
||||
entries.sort((a, b) => b[1].score - a[1].score);
|
||||
return entries.slice(0, k).map(([id, v]) => ({ id, ...v }));
|
||||
}
|
||||
}
|
||||
|
||||
export type UserId = string;
|
||||
`;
|
||||
const result = await chunkCodeText(src, 'smoketest.ts');
|
||||
const hasSymbolNames = result.some(c => c.metadata.symbolName !== null);
|
||||
const hasTypeScriptHeader = result.some(c => c.text.startsWith('[TypeScript]'));
|
||||
console.log(JSON.stringify({
|
||||
count: result.length,
|
||||
has_symbol_names: hasSymbolNames,
|
||||
has_typescript_header: hasTypeScriptHeader,
|
||||
first_header: result[0]?.text.split('\n')[0],
|
||||
symbol_names: result.map(c => c.metadata.symbolName),
|
||||
}, null, 2));
|
||||
@@ -1,171 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# fix-v0.11.0.sh — stopgap for broken v0.11.0 installs where the Minions
|
||||
# migration never fired on upgrade.
|
||||
#
|
||||
# Usage:
|
||||
# curl -fsSL https://raw.githubusercontent.com/garrytan/gbrain/v0.11.1/scripts/fix-v0.11.0.sh | bash
|
||||
#
|
||||
# What it does:
|
||||
# 1. gbrain init --migrate-only — applies schema v7 without touching config.
|
||||
# 2. gbrain jobs smoke — fails loudly if Minions isn't healthy.
|
||||
# 3. Prompts for minion_mode (or defaults to pain_triggered on non-TTY).
|
||||
# 4. Atomically writes ~/.gbrain/preferences.json (0o600).
|
||||
# 5. Appends ~/.gbrain/migrations/completed.jsonl with status:"partial" and
|
||||
# apply_migrations_pending: true — the v0.11.1 `apply-migrations` runner
|
||||
# will pick up where we left off (host rewrites, autopilot install).
|
||||
# 6. Detects host AGENTS.md / cron/jobs.json and PRINTS the rewrite guidance
|
||||
# as text. Never auto-edits host files from a curl-piped script — too
|
||||
# high blast-radius (user trust model is "I pasted this").
|
||||
# 7. Final line: tells the user to run `gbrain autopilot --install` as the
|
||||
# one-stop finisher (autopilot forks the Minions worker as a child).
|
||||
#
|
||||
# Retires when v0.11.1 is out: the canonical fix becomes
|
||||
# gbrain upgrade && gbrain apply-migrations
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RED=$'\033[1;31m'
|
||||
GREEN=$'\033[1;32m'
|
||||
YELLOW=$'\033[1;33m'
|
||||
NC=$'\033[0m'
|
||||
|
||||
say() { printf "%s%s%s\n" "$1" "$2" "$NC"; }
|
||||
info() { say "" "$1"; }
|
||||
ok() { say "$GREEN" "$1"; }
|
||||
warn() { say "$YELLOW" "$1"; }
|
||||
die() { say "$RED" "$1"; exit 1; }
|
||||
|
||||
command -v gbrain >/dev/null 2>&1 || die "gbrain not found on \$PATH. Install it first (\`bun add -g gbrain\` or download a binary)."
|
||||
|
||||
GBRAIN_DIR="${HOME}/.gbrain"
|
||||
PREFS_PATH="${GBRAIN_DIR}/preferences.json"
|
||||
COMPLETED_PATH="${GBRAIN_DIR}/migrations/completed.jsonl"
|
||||
|
||||
mkdir -p "${GBRAIN_DIR}/migrations"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 1: schema
|
||||
# ------------------------------------------------------------
|
||||
info "[1/8] Applying schema (gbrain init --migrate-only)..."
|
||||
if ! gbrain init --migrate-only; then
|
||||
die "Schema migration failed. Check ~/.gbrain/config.json has a valid database_url (or database_path for PGLite), then re-run."
|
||||
fi
|
||||
ok " schema ok"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 2: smoke
|
||||
# ------------------------------------------------------------
|
||||
info "[2/8] Running Minions smoke test (gbrain jobs smoke)..."
|
||||
if ! gbrain jobs smoke; then
|
||||
die "Smoke test failed. See the error above. Fix before continuing."
|
||||
fi
|
||||
ok " smoke ok"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 3: mode prompt
|
||||
# ------------------------------------------------------------
|
||||
info "[3/8] Choose minion_mode..."
|
||||
MODE="pain_triggered"
|
||||
if [ -t 0 ] && [ -t 1 ]; then
|
||||
echo ""
|
||||
echo " [1] always — route every background task through Minions (most durable)"
|
||||
echo " [2] pain_triggered — default to native subagents, switch to Minions on pain signals (recommended)"
|
||||
echo " [3] off — disable Minions; keep native subagents"
|
||||
echo ""
|
||||
read -r -p " Choice [2]: " CHOICE
|
||||
case "${CHOICE:-2}" in
|
||||
1) MODE="always" ;;
|
||||
3) MODE="off" ;;
|
||||
*) MODE="pain_triggered" ;;
|
||||
esac
|
||||
else
|
||||
warn " non-interactive shell → defaulting to pain_triggered (change later: \`gbrain config set minion_mode <mode>\`)"
|
||||
fi
|
||||
ok " mode=${MODE}"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 4: atomic write preferences.json (0o600)
|
||||
# ------------------------------------------------------------
|
||||
info "[4/8] Writing ~/.gbrain/preferences.json..."
|
||||
NOW_ISO=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
TMP_PREFS=$(mktemp)
|
||||
cat > "${TMP_PREFS}" <<EOF
|
||||
{
|
||||
"minion_mode": "${MODE}",
|
||||
"set_at": "${NOW_ISO}",
|
||||
"set_in_version": "0.11.0"
|
||||
}
|
||||
EOF
|
||||
chmod 600 "${TMP_PREFS}"
|
||||
mv "${TMP_PREFS}" "${PREFS_PATH}"
|
||||
chmod 600 "${PREFS_PATH}"
|
||||
ok " wrote ${PREFS_PATH}"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 5: append completed.jsonl as status:"partial"
|
||||
# ------------------------------------------------------------
|
||||
info "[5/8] Recording migration as partial..."
|
||||
# We write "partial" + apply_migrations_pending:true. v0.11.1 apply-migrations
|
||||
# detects this and resumes the remaining phases (host rewrites + autopilot
|
||||
# install). If we wrote "complete" here, apply-migrations would SKIP the
|
||||
# remaining phases and the broken install would stay broken (Codex H2).
|
||||
echo "{\"version\":\"0.11.0\",\"status\":\"partial\",\"apply_migrations_pending\":true,\"mode\":\"${MODE}\",\"ts\":\"${NOW_ISO}\",\"source\":\"fix-v0.11.0.sh\"}" >> "${COMPLETED_PATH}"
|
||||
ok " appended ${COMPLETED_PATH}"
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 6: detect AGENTS.md — PRINT guidance, do not auto-edit
|
||||
# ------------------------------------------------------------
|
||||
info "[6/8] Scanning for AGENTS.md..."
|
||||
AGENTS_FOUND=()
|
||||
for CANDIDATE in "${HOME}/.claude/AGENTS.md" "${HOME}/.openclaw/AGENTS.md" "${PWD}/AGENTS.md"; do
|
||||
[ -f "${CANDIDATE}" ] && AGENTS_FOUND+=("${CANDIDATE}")
|
||||
done
|
||||
if [ ${#AGENTS_FOUND[@]} -eq 0 ]; then
|
||||
ok " no AGENTS.md found — nothing to suggest"
|
||||
else
|
||||
for F in "${AGENTS_FOUND[@]}"; do
|
||||
warn " AGENTS.md detected: ${F}"
|
||||
echo " - Next steps (this script does NOT auto-edit):"
|
||||
echo " 1. Add a pointer to skills/conventions/subagent-routing.md"
|
||||
echo " 2. The v0.11.1 binary's \`gbrain apply-migrations --yes\` will inject"
|
||||
echo " this automatically once v0.11.1 is installed."
|
||||
done
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 7: detect cron/jobs.json and scan for agentTurn
|
||||
# ------------------------------------------------------------
|
||||
info "[7/8] Scanning for cron manifests..."
|
||||
CRON_FOUND=()
|
||||
for CANDIDATE in "${HOME}/.claude/cron/jobs.json" "${HOME}/.openclaw/cron/jobs.json" "${PWD}/cron/jobs.json"; do
|
||||
[ -f "${CANDIDATE}" ] && CRON_FOUND+=("${CANDIDATE}")
|
||||
done
|
||||
if [ ${#CRON_FOUND[@]} -eq 0 ]; then
|
||||
ok " no cron/jobs.json found — nothing to suggest"
|
||||
else
|
||||
for F in "${CRON_FOUND[@]}"; do
|
||||
warn " cron manifest detected: ${F}"
|
||||
COUNT=$(grep -c 'agentTurn' "${F}" 2>/dev/null || echo 0)
|
||||
echo " - ${COUNT} agentTurn entries"
|
||||
echo " - v0.11.1 apply-migrations will:"
|
||||
echo " * auto-rewrite builtin handlers (sync/embed/lint/import/"
|
||||
echo " extract/backlinks/autopilot-cycle) to gbrain jobs submit"
|
||||
echo " * emit a pending-host-work.jsonl TODO for every non-builtin"
|
||||
echo " handler; host agent walks those per skills/migrations/v0.11.0.md"
|
||||
done
|
||||
fi
|
||||
|
||||
# ------------------------------------------------------------
|
||||
# Step 8: final line
|
||||
# ------------------------------------------------------------
|
||||
info "[8/8] Done. Next step:"
|
||||
echo ""
|
||||
echo " ${GREEN}gbrain autopilot --install${NC}"
|
||||
echo ""
|
||||
echo " That ONE command does the rest: supervises autopilot, forks the"
|
||||
echo " Minions worker, and installs the right entry for your host (launchd"
|
||||
echo " on macOS, systemd on Linux, bootstrap hook on ephemeral containers)."
|
||||
echo ""
|
||||
echo " Once v0.11.1 is out:"
|
||||
echo " ${GREEN}gbrain upgrade && gbrain apply-migrations${NC}"
|
||||
echo " becomes the canonical fix. This script retires then."
|
||||
@@ -1,205 +0,0 @@
|
||||
/**
|
||||
* llms-config — single source of truth for llms.txt + llms-full.txt.
|
||||
*
|
||||
* Consumed by scripts/build-llms.ts (emits llms.txt, llms-full.txt) and
|
||||
* test/build-llms.test.ts (asserts paths resolve, content contract holds).
|
||||
*
|
||||
* Adding a doc? Add it here and run `bun run build:llms`. The drift-detection
|
||||
* test fails CI if you forget.
|
||||
*
|
||||
* Fork-friendliness: `rawBaseUrl` reads from `LLMS_REPO_BASE` so forks can
|
||||
* regenerate without manual URL rewrites:
|
||||
* LLMS_REPO_BASE=https://raw.githubusercontent.com/fork-org/gbrain/main bun run build:llms
|
||||
*/
|
||||
|
||||
export type DocEntry = {
|
||||
title: string;
|
||||
description: string;
|
||||
path: string;
|
||||
includeInFull?: boolean;
|
||||
};
|
||||
|
||||
export type DocSection = {
|
||||
heading: string;
|
||||
optional?: boolean;
|
||||
entries: DocEntry[];
|
||||
};
|
||||
|
||||
export const PROJECT = {
|
||||
name: "GBrain",
|
||||
summary:
|
||||
"GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.",
|
||||
repoUrl: "https://github.com/garrytan/gbrain",
|
||||
rawBaseUrl:
|
||||
process.env.LLMS_REPO_BASE ??
|
||||
"https://raw.githubusercontent.com/garrytan/gbrain/master",
|
||||
};
|
||||
|
||||
export const SECTIONS: DocSection[] = [
|
||||
{
|
||||
heading: "Core entry points",
|
||||
entries: [
|
||||
{
|
||||
title: "AGENTS.md",
|
||||
description:
|
||||
"Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.",
|
||||
path: "AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "CLAUDE.md",
|
||||
description:
|
||||
"Architecture reference. Key files, trust boundaries, engine factory, test layout.",
|
||||
path: "CLAUDE.md",
|
||||
},
|
||||
{
|
||||
title: "INSTALL_FOR_AGENTS.md",
|
||||
description: "9-step agent installation.",
|
||||
path: "INSTALL_FOR_AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "skills/RESOLVER.md",
|
||||
description: "Skill dispatcher. Read first for any task.",
|
||||
path: "skills/RESOLVER.md",
|
||||
},
|
||||
{
|
||||
title: "README.md",
|
||||
description: "Project overview, benchmarks, 30-minute setup.",
|
||||
path: "README.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Configuration",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/ENGINES.md",
|
||||
description: "PGLite vs Postgres trade-off and when to migrate.",
|
||||
path: "docs/ENGINES.md",
|
||||
},
|
||||
{
|
||||
title: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
|
||||
description:
|
||||
"MECE directory structure (people/, companies/, concepts/).",
|
||||
path: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/live-sync.md",
|
||||
description: "Incremental markdown sync setup.",
|
||||
path: "docs/guides/live-sync.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/cron-schedule.md",
|
||||
description: "Recurring job scheduling.",
|
||||
path: "docs/guides/cron-schedule.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/minions-deployment.md",
|
||||
description:
|
||||
"Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.",
|
||||
path: "docs/guides/minions-deployment.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/quiet-hours.md",
|
||||
description: "Notification hold + timezone-aware delivery.",
|
||||
path: "docs/guides/quiet-hours.md",
|
||||
},
|
||||
{
|
||||
title: "docs/mcp/DEPLOY.md",
|
||||
description: "MCP server deployment.",
|
||||
path: "docs/mcp/DEPLOY.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Debugging",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/GBRAIN_VERIFY.md",
|
||||
description:
|
||||
"7-check post-setup verification. Start here when something feels off.",
|
||||
path: "docs/GBRAIN_VERIFY.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/minions-fix.md",
|
||||
description: "Troubleshooting the Minions job queue.",
|
||||
path: "docs/guides/minions-fix.md",
|
||||
},
|
||||
{
|
||||
title: "docs/integrations/reliability-repair.md",
|
||||
description: "Data integrity recovery.",
|
||||
path: "docs/integrations/reliability-repair.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Migrations",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
|
||||
description:
|
||||
"Patches for downstream agent skill forks. One section per release.",
|
||||
path: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "skills/migrations/",
|
||||
description:
|
||||
"Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.",
|
||||
path: "skills/migrations/",
|
||||
},
|
||||
{
|
||||
title: "CHANGELOG.md",
|
||||
description:
|
||||
"Release-summary voice + itemized changes + self-repair block per version.",
|
||||
path: "CHANGELOG.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Philosophy",
|
||||
optional: true,
|
||||
entries: [
|
||||
{
|
||||
title: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
|
||||
description: "Why skills live in markdown.",
|
||||
path: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
|
||||
description: "Homebrew for Personal AI.",
|
||||
path: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Optional",
|
||||
optional: true,
|
||||
entries: [
|
||||
{
|
||||
title: "docs/designs/",
|
||||
description: "Forward-looking designs.",
|
||||
path: "docs/designs/",
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/architecture/infra-layer.md",
|
||||
description: "Shared infra patterns.",
|
||||
path: "docs/architecture/infra-layer.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
export const INLINE_TIPS = [
|
||||
"`gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.",
|
||||
"`gbrain orphans [--json]` - pages with zero inbound wikilinks.",
|
||||
"`gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.",
|
||||
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
|
||||
];
|
||||
|
||||
// Target ~600KB so llms-full.txt fits in ~150k-token contexts with room to spare.
|
||||
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
|
||||
export const FULL_SIZE_BUDGET = 600_000;
|
||||
@@ -1,71 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run E2E tests ONE FILE AT A TIME.
|
||||
#
|
||||
# Bun's default is to run test files in parallel (each in its own worker).
|
||||
# Our E2E suite shares one Postgres database across all 13 files, and
|
||||
# `setupDB()` does TRUNCATE CASCADE + fixture import. When files run in
|
||||
# parallel, file A's TRUNCATE can race with file B's fixture import,
|
||||
# producing observed fails like "expected 16 pages, got 8", missing
|
||||
# links, orphaned timeline entries, etc. The flakiness was visible on
|
||||
# ~3 of every 5 runs pre-fix.
|
||||
#
|
||||
# Running files sequentially eliminates the race entirely. It also costs
|
||||
# some startup overhead (each file spins up a fresh bun process) but for
|
||||
# a suite this size that is measured in ~1-2s per file, amortized under
|
||||
# the natural per-file test time of 5-10s.
|
||||
#
|
||||
# Exits non-zero on the first failing file so CI fails fast.
|
||||
#
|
||||
# `--timeout=60000` matches the unit test suite. Bun's default is 5s,
|
||||
# which is too tight for setupDB's TRUNCATE CASCADE on ~30 tables on
|
||||
# CI runners under load (one CI flake observed on PR #475 hitting
|
||||
# exactly 5000.09ms in the Tags beforeAll).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
pass_files=0
|
||||
fail_files=0
|
||||
fail_list=()
|
||||
total_pass=0
|
||||
total_fail=0
|
||||
|
||||
for f in test/e2e/*.test.ts; do
|
||||
name=$(basename "$f")
|
||||
echo ""
|
||||
echo "=== $name ==="
|
||||
if output=$(bun test --timeout=60000 "$f" 2>&1); then
|
||||
pass_files=$((pass_files + 1))
|
||||
# Extract pass/fail counts from bun's summary (e.g., "123 pass")
|
||||
p=$(echo "$output" | grep -oE '[0-9]+ pass' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
total_pass=$((total_pass + p))
|
||||
echo "$output" | tail -8
|
||||
else
|
||||
fail_files=$((fail_files + 1))
|
||||
fail_list+=("$name")
|
||||
p=$(echo "$output" | grep -oE '[0-9]+ pass' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
fl=$(echo "$output" | grep -oE '[0-9]+ fail' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
total_pass=$((total_pass + p))
|
||||
total_fail=$((total_fail + fl))
|
||||
echo "$output"
|
||||
echo ""
|
||||
echo "FAILED: $name"
|
||||
# Continue so we see all failures; exit nonzero at the end.
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo "E2E SUMMARY (sequential execution)"
|
||||
echo "========================================"
|
||||
echo "Files: $((pass_files + fail_files)) total, $pass_files passed, $fail_files failed"
|
||||
echo "Tests: $total_pass passed, $total_fail failed"
|
||||
if [ ${#fail_list[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "Failing files:"
|
||||
for f in "${fail_list[@]}"; do
|
||||
echo " - $f"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,13 +0,0 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* scripts/skillify-check.ts — thin shim (v0.17+).
|
||||
*
|
||||
* The 10-item audit logic lives in `src/commands/skillify-check.ts`
|
||||
* and is exposed as `gbrain skillify check` (D-CX-2). This file stays
|
||||
* as a shim so existing callers (tests, docs, cron entries) continue
|
||||
* to work. New code should prefer `gbrain skillify check ...`.
|
||||
*/
|
||||
|
||||
import { runSkillifyCheckInline } from '../src/commands/skillify-check.ts';
|
||||
|
||||
await runSkillifyCheckInline(process.argv.slice(2));
|
||||
@@ -1,204 +0,0 @@
|
||||
#!/bin/bash
|
||||
# smoke-test.sh — GBrain post-restart smoke tests + auto-fix
|
||||
#
|
||||
# Ships with gbrain. Tests gbrain core services + OpenClaw plugin health.
|
||||
# Users extend via ~/.gbrain/smoke-tests.d/*.sh (user-defined tests).
|
||||
#
|
||||
# Usage:
|
||||
# gbrain smoke-test # run all tests
|
||||
# bash scripts/smoke-test.sh # direct invocation
|
||||
#
|
||||
# Each test: check → if broken, attempt fix → re-check → report.
|
||||
# Exit code = number of remaining failures (0 = all pass).
|
||||
|
||||
set -a
|
||||
[ -f /data/.env ] && . /data/.env 2>/dev/null || true
|
||||
set +a
|
||||
|
||||
LOG="${GBRAIN_SMOKE_LOG:-/tmp/gbrain-smoke-test.log}"
|
||||
FAILURES=0
|
||||
FIXES=0
|
||||
TOTAL=0
|
||||
SKIPPED=0
|
||||
|
||||
timestamp() { date -u '+%Y-%m-%d %H:%M:%S'; }
|
||||
pass() { TOTAL=$((TOTAL + 1)); echo "✅ $1"; echo "$(timestamp) PASS: $1" >> "$LOG"; }
|
||||
fail() { TOTAL=$((TOTAL + 1)); FAILURES=$((FAILURES + 1)); echo "❌ $1"; echo "$(timestamp) FAIL: $1" >> "$LOG"; }
|
||||
fixed() { FIXES=$((FIXES + 1)); echo "🔧 Fixed: $1"; echo "$(timestamp) FIXED: $1" >> "$LOG"; }
|
||||
skip() { SKIPPED=$((SKIPPED + 1)); echo "⏭️ $1"; echo "$(timestamp) SKIP: $1" >> "$LOG"; }
|
||||
|
||||
echo "$(timestamp) === GBrain Smoke Tests ===" >> "$LOG"
|
||||
echo "🧪 Running gbrain smoke tests..."
|
||||
echo ""
|
||||
|
||||
# ── Resolve paths ───────────────────────────────────────────
|
||||
# Find gbrain — could be global install, workspace dep, or /data/gbrain
|
||||
GBRAIN_DIR=""
|
||||
for candidate in \
|
||||
"${GBRAIN_DIR_OVERRIDE:-}" \
|
||||
"/data/gbrain" \
|
||||
"$(dirname "$0")/.." \
|
||||
"${OPENCLAW_WORKSPACE:-/data/.openclaw/workspace}/node_modules/gbrain" \
|
||||
"./node_modules/gbrain"; do
|
||||
[ -n "$candidate" ] && [ -f "$candidate/src/cli.ts" ] && GBRAIN_DIR="$candidate" && break
|
||||
done
|
||||
|
||||
# Find bun
|
||||
BUN_PATH=""
|
||||
for bp in "/root/.bun/bin/bun" "/data/.bun/bin/bun" "$(which bun 2>/dev/null)"; do
|
||||
[ -n "$bp" ] && [ -x "$bp" ] && BUN_PATH="$bp" && break
|
||||
done
|
||||
|
||||
# Resolve database URL
|
||||
DB_URL="${GBRAIN_DATABASE_URL:-${DATABASE_URL:-}}"
|
||||
# Fallback: grep from .env (handles files with parse-breaking lines)
|
||||
[ -z "$DB_URL" ] && DB_URL=$(grep '^GBRAIN_DATABASE_URL=' /data/.env 2>/dev/null | head -1 | cut -d= -f2-)
|
||||
[ -z "$DB_URL" ] && DB_URL=$(grep '^DATABASE_URL=' /data/.env 2>/dev/null | head -1 | cut -d= -f2-)
|
||||
|
||||
# ── 1. Bun runtime ─────────────────────────────────────────
|
||||
if [ -n "$BUN_PATH" ]; then
|
||||
export PATH="$(dirname "$BUN_PATH"):$PATH"
|
||||
pass "Bun runtime ($BUN_PATH)"
|
||||
else
|
||||
# Auto-fix: install bun
|
||||
curl -fsSL https://bun.sh/install | bash 2>/dev/null
|
||||
if [ -x "/root/.bun/bin/bun" ]; then
|
||||
BUN_PATH="/root/.bun/bin/bun"
|
||||
export PATH="/root/.bun/bin:$PATH"
|
||||
fixed "Bun runtime installed"
|
||||
pass "Bun runtime"
|
||||
else
|
||||
fail "Bun runtime — install failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 2. GBrain CLI loads ────────────────────────────────────
|
||||
if [ -n "$GBRAIN_DIR" ] && [ -n "$BUN_PATH" ]; then
|
||||
if timeout 15 "$BUN_PATH" run "$GBRAIN_DIR/src/cli.ts" --help >/dev/null 2>&1; then
|
||||
pass "GBrain CLI ($GBRAIN_DIR)"
|
||||
else
|
||||
# Auto-fix: reinstall deps
|
||||
cd "$GBRAIN_DIR" && "$BUN_PATH" install --frozen-lockfile 2>/dev/null
|
||||
if timeout 15 "$BUN_PATH" run "$GBRAIN_DIR/src/cli.ts" --help >/dev/null 2>&1; then
|
||||
fixed "GBrain deps reinstalled"
|
||||
pass "GBrain CLI (after dep fix)"
|
||||
else
|
||||
fail "GBrain CLI — won't start"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
[ -z "$GBRAIN_DIR" ] && fail "GBrain CLI — not found"
|
||||
[ -z "$BUN_PATH" ] && skip "GBrain CLI — bun not available"
|
||||
fi
|
||||
|
||||
# ── 3. GBrain database ────────────────────────────────────
|
||||
if [ -n "$DB_URL" ] && [ -n "$GBRAIN_DIR" ] && [ -n "$BUN_PATH" ]; then
|
||||
DOCTOR_OUT=$(DATABASE_URL="$DB_URL" GBRAIN_DATABASE_URL="$DB_URL" timeout 20 "$BUN_PATH" run "$GBRAIN_DIR/src/cli.ts" doctor 2>&1)
|
||||
if echo "$DOCTOR_OUT" | grep -q "Health score\|brain_score\|Health Check"; then
|
||||
SCORE=$(echo "$DOCTOR_OUT" | grep -oP 'Health score: \K[0-9]+' || echo '?')
|
||||
pass "GBrain database (health score: $SCORE/100)"
|
||||
else
|
||||
fail "GBrain database — doctor returned no health data"
|
||||
fi
|
||||
else
|
||||
[ -z "$DB_URL" ] && fail "GBrain database — no DATABASE_URL or GBRAIN_DATABASE_URL"
|
||||
[ -z "$GBRAIN_DIR" ] && skip "GBrain database — gbrain not found"
|
||||
fi
|
||||
|
||||
# ── 4. GBrain worker process ──────────────────────────────
|
||||
if [ -n "$GBRAIN_DIR" ] && [ -n "$BUN_PATH" ] && [ -n "$DB_URL" ]; then
|
||||
if [ -f /tmp/gbrain-worker.pid ] && kill -0 "$(cat /tmp/gbrain-worker.pid)" 2>/dev/null; then
|
||||
pass "GBrain worker (PID: $(cat /tmp/gbrain-worker.pid))"
|
||||
else
|
||||
# Auto-fix: start worker
|
||||
DATABASE_URL="$DB_URL" GBRAIN_DATABASE_URL="$DB_URL" GBRAIN_ALLOW_SHELL_JOBS=1 \
|
||||
nohup "$BUN_PATH" run "$GBRAIN_DIR/src/cli.ts" jobs work --concurrency 2 > /tmp/gbrain-worker.log 2>&1 &
|
||||
echo $! > /tmp/gbrain-worker.pid
|
||||
sleep 2
|
||||
if kill -0 "$(cat /tmp/gbrain-worker.pid)" 2>/dev/null; then
|
||||
fixed "GBrain worker started"
|
||||
pass "GBrain worker (PID: $(cat /tmp/gbrain-worker.pid))"
|
||||
else
|
||||
fail "GBrain worker — failed to start (check /tmp/gbrain-worker.log)"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
skip "GBrain worker — prerequisites missing"
|
||||
fi
|
||||
|
||||
# ── 5. OpenClaw plugin health (if OpenClaw is installed) ──
|
||||
OPENCLAW_CODEX_ZOD="/app/node_modules/openclaw/dist/extensions/codex/node_modules/zod"
|
||||
if [ -d "$OPENCLAW_CODEX_ZOD" ]; then
|
||||
CORE_CJS="$OPENCLAW_CODEX_ZOD/v4/core/core.cjs"
|
||||
if [ -f "$CORE_CJS" ] && node -e "require('$OPENCLAW_CODEX_ZOD/v4/core/index.cjs')" 2>/dev/null; then
|
||||
pass "OpenClaw Codex plugin (Zod CJS)"
|
||||
else
|
||||
# Auto-fix: reinstall zod
|
||||
cd "$OPENCLAW_CODEX_ZOD" && npm install zod@4 --force --silent 2>/dev/null
|
||||
if [ -f "$CORE_CJS" ] && node -e "require('$OPENCLAW_CODEX_ZOD/v4/core/index.cjs')" 2>/dev/null; then
|
||||
fixed "Codex Zod core.cjs reinstalled"
|
||||
pass "OpenClaw Codex plugin (Zod CJS after fix)"
|
||||
else
|
||||
fail "OpenClaw Codex plugin — Zod fix failed"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
skip "OpenClaw Codex plugin — not installed"
|
||||
fi
|
||||
|
||||
# ── 6. OpenClaw gateway (if running) ─────────────────────
|
||||
OPENCLAW_PORT="${OPENCLAW_GATEWAY_PORT:-18789}"
|
||||
if curl -sf "http://127.0.0.1:$OPENCLAW_PORT/" >/dev/null 2>&1; then
|
||||
pass "OpenClaw gateway (port $OPENCLAW_PORT)"
|
||||
else
|
||||
skip "OpenClaw gateway — not responding (may not be running yet)"
|
||||
fi
|
||||
|
||||
# ── 7. Embedding API key ─────────────────────────────────
|
||||
EMBED_KEY="${OPENAI_API_KEY:-${VOYAGE_API_KEY:-}}"
|
||||
if [ -n "$EMBED_KEY" ]; then
|
||||
pass "Embedding API key set"
|
||||
else
|
||||
fail "Embedding API key — neither OPENAI_API_KEY nor VOYAGE_API_KEY is set"
|
||||
fi
|
||||
|
||||
# ── 8. Brain repo (if configured) ────────────────────────
|
||||
BRAIN_PATH="${GBRAIN_BRAIN_PATH:-/data/brain}"
|
||||
if [ -d "$BRAIN_PATH/.git" ]; then
|
||||
PAGE_COUNT=$(find "$BRAIN_PATH" -name "*.md" -not -path "*/.git/*" 2>/dev/null | wc -l)
|
||||
pass "Brain repo ($PAGE_COUNT pages at $BRAIN_PATH)"
|
||||
elif [ -d "$BRAIN_PATH" ]; then
|
||||
pass "Brain directory exists ($BRAIN_PATH, not a git repo)"
|
||||
else
|
||||
skip "Brain repo — $BRAIN_PATH not found"
|
||||
fi
|
||||
|
||||
# ── User-defined tests (~/.gbrain/smoke-tests.d/*.sh) ────
|
||||
USER_TESTS_DIR="${HOME}/.gbrain/smoke-tests.d"
|
||||
if [ -d "$USER_TESTS_DIR" ]; then
|
||||
for test_script in "$USER_TESTS_DIR"/*.sh; do
|
||||
[ -f "$test_script" ] || continue
|
||||
TEST_NAME=$(basename "$test_script" .sh)
|
||||
echo " Running user test: $TEST_NAME"
|
||||
if bash "$test_script" 2>/dev/null; then
|
||||
pass "User: $TEST_NAME"
|
||||
else
|
||||
fail "User: $TEST_NAME"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Summary ─────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
PASSED=$((TOTAL - FAILURES))
|
||||
echo "Results: $PASSED/$TOTAL passed, $FIXES auto-fixed, $SKIPPED skipped"
|
||||
if [ $FAILURES -gt 0 ]; then
|
||||
echo "⚠️ $FAILURES failure(s) remain — manual intervention needed"
|
||||
else
|
||||
echo "✅ All smoke tests passed"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo "$(timestamp) Summary: $PASSED/$TOTAL passed, $FIXES fixed, $FAILURES failed, $SKIPPED skipped" >> "$LOG"
|
||||
|
||||
exit $FAILURES
|
||||
+1
-14
@@ -13,15 +13,12 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "What do we know about", "tell me about", "search for", "who is", "background on", "notes on" | `skills/query/SKILL.md` |
|
||||
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
|
||||
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
|
||||
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
|
||||
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
|
||||
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
|
||||
| "citation audit", "check citations", "fix citations" | `skills/citation-fixer/SKILL.md` (focused fix). For broader brain health, chain into `skills/maintain/SKILL.md` |
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
@@ -54,13 +51,9 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Cron scheduling, quiet hours, job staggering | `skills/cron-scheduler/SKILL.md` |
|
||||
| Save or load reports | `skills/reports/SKILL.md` |
|
||||
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
|
||||
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
|
||||
| "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` |
|
||||
| Post-restart health + auto-fix, "did the container restart break anything", smoke test | `skills/smoke-test/SKILL.md` |
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
@@ -69,12 +62,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Set up GBrain", first boot | `skills/setup/SKILL.md` |
|
||||
| "Migrate from Obsidian/Notion/Logseq" | `skills/migrate/SKILL.md` |
|
||||
| Brain health check, maintenance run | `skills/maintain/SKILL.md` |
|
||||
| "Extract links", "build link graph", "populate timeline" | `skills/maintain/SKILL.md` (extraction sections) |
|
||||
| "Brain health", "what features am I missing", "brain score" | Run `gbrain features --json` |
|
||||
| "Set up autopilot", "run brain maintenance", "keep brain updated" | Run `gbrain autopilot --install --repo ~/brain` |
|
||||
| Agent identity, "who am I", customize agent | `skills/soul-audit/SKILL.md` |
|
||||
| "Populate links", "extract links", "backfill graph" | `skills/maintain/SKILL.md` (graph population phase) |
|
||||
| "Populate timeline", "extract timeline entries" | `skills/maintain/SKILL.md` (graph population phase) |
|
||||
|
||||
## Identity & access (always-on)
|
||||
|
||||
@@ -99,6 +87,5 @@ When multiple skills could match:
|
||||
These apply to ALL brain-writing skills:
|
||||
- `skills/conventions/quality.md` — citations, back-links, notability gate
|
||||
- `skills/conventions/brain-first.md` — check brain before external APIs
|
||||
- `skills/conventions/subagent-routing.md` — when to use Minions vs inline work
|
||||
- `skills/_brain-filing-rules.md` — where files go
|
||||
- `skills/_output-rules.md` — output quality standards
|
||||
|
||||
@@ -1,101 +0,0 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"companion": "_brain-filing-rules.md",
|
||||
"description": "Canonical (machine-readable) brain filing rules. The .md companion is the human explainer; this JSON is what `gbrain check-resolvable` audits against. Keep both in sync.",
|
||||
"rules": [
|
||||
{
|
||||
"kind": "person",
|
||||
"directory": "people/",
|
||||
"examples": ["founders", "investors", "attendees", "contacts"],
|
||||
"description": "A page whose primary subject is one person."
|
||||
},
|
||||
{
|
||||
"kind": "company",
|
||||
"directory": "companies/",
|
||||
"examples": ["portfolio companies", "acquirers", "vendors"],
|
||||
"description": "A page whose primary subject is one company or organization."
|
||||
},
|
||||
{
|
||||
"kind": "deal",
|
||||
"directory": "deals/",
|
||||
"examples": ["seed rounds", "acquisitions"],
|
||||
"description": "A page whose primary subject is a financing or M&A transaction."
|
||||
},
|
||||
{
|
||||
"kind": "meeting",
|
||||
"directory": "meetings/",
|
||||
"examples": ["1:1s", "pitches", "pods"],
|
||||
"description": "A meeting transcript or minutes. Propagate entities to companies/ and people/ pages."
|
||||
},
|
||||
{
|
||||
"kind": "concept",
|
||||
"directory": "concepts/",
|
||||
"examples": ["mental models", "theses", "frameworks"],
|
||||
"description": "A reusable idea, framework, or mental model not tied to a specific person/company."
|
||||
},
|
||||
{
|
||||
"kind": "project",
|
||||
"directory": "projects/",
|
||||
"examples": ["internal initiatives", "multi-session work"],
|
||||
"description": "A multi-session piece of work with its own arc."
|
||||
},
|
||||
{
|
||||
"kind": "analysis",
|
||||
"directory": "analysis/",
|
||||
"examples": ["deep dives", "comparative studies"],
|
||||
"description": "A long-form analysis of a specific topic."
|
||||
},
|
||||
{
|
||||
"kind": "civic",
|
||||
"directory": "civic/",
|
||||
"examples": ["policy analysis", "government topics"],
|
||||
"description": "Public-sector, policy, or civic-issue content."
|
||||
},
|
||||
{
|
||||
"kind": "writing",
|
||||
"directory": "writing/",
|
||||
"examples": ["essays", "drafts", "published pieces"],
|
||||
"description": "A piece of prose authored by the user."
|
||||
},
|
||||
{
|
||||
"kind": "guide",
|
||||
"directory": "guides/",
|
||||
"examples": ["runbooks", "how-to docs"],
|
||||
"description": "A guide or runbook authored for future reference."
|
||||
},
|
||||
{
|
||||
"kind": "tech",
|
||||
"directory": "tech/",
|
||||
"examples": ["APIs", "libraries", "language notes"],
|
||||
"description": "Technical references and tooling notes not tied to a specific company."
|
||||
},
|
||||
{
|
||||
"kind": "finance",
|
||||
"directory": "finance/",
|
||||
"examples": ["market data", "metrics"],
|
||||
"description": "Financial reference data not tied to a single deal."
|
||||
},
|
||||
{
|
||||
"kind": "personal",
|
||||
"directory": "personal/",
|
||||
"examples": ["logistics", "family"],
|
||||
"description": "Personal-life content — kept separate from work."
|
||||
},
|
||||
{
|
||||
"kind": "openclaw",
|
||||
"directory": "openclaw/",
|
||||
"examples": ["agent-state notes"],
|
||||
"description": "Notes about the host OpenClaw agent itself, not the underlying entities."
|
||||
}
|
||||
],
|
||||
"sources_dir": {
|
||||
"directory": "sources/",
|
||||
"purpose": "ONLY for raw data: bulk imports, API dumps, periodic captures. A page with a clear primary subject (person, company, concept) does NOT belong here.",
|
||||
"not_for": ["articles about a person", "analyses of a company", "reusable frameworks"]
|
||||
},
|
||||
"notes": [
|
||||
"The PRIMARY SUBJECT of the content determines the directory, not the format or source skill.",
|
||||
"When in doubt: what would you search for to find this page again?",
|
||||
"Cross-link from related directories via back-links — do not duplicate content."
|
||||
]
|
||||
}
|
||||
@@ -17,13 +17,6 @@ tools:
|
||||
- get_backlinks
|
||||
- sync_brain
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- meetings/
|
||||
---
|
||||
|
||||
# Brain Operations — The Ambient Context Layer
|
||||
@@ -77,22 +70,6 @@ Every message, meeting, email, or conversation that references a person or compa
|
||||
**User's direct statements are the highest-value data source.** Write them to brain
|
||||
pages immediately with attribution `[Source: User, YYYY-MM-DD]`.
|
||||
|
||||
### Phase 2.5: Structured Graph Updates (automatic)
|
||||
|
||||
Every `put_page` call automatically extracts entity references and writes them
|
||||
to the graph (`links` table) with inferred relationship types. Stale links
|
||||
(refs no longer in the page text) are removed in the same call. This is
|
||||
"auto-link" reconciliation.
|
||||
|
||||
- No manual `add_link` calls needed for ordinary page writes.
|
||||
- Inferred link types: `attended` (meeting -> person), `works_at`, `invested_in`,
|
||||
`founded`, `advises`, `source` (frontmatter), `mentions` (default).
|
||||
- The `put_page` MCP response includes `auto_links: { created, removed, errors }`
|
||||
so the agent can verify outcomes.
|
||||
- To disable: `gbrain config set auto_link false`. Default is on.
|
||||
- Timeline entries with specific dates still need explicit `gbrain timeline-add`
|
||||
(or batch via `gbrain extract timeline --source db`).
|
||||
|
||||
### Phase 3: On Every Outbound Response (READ → PULL → RESPOND)
|
||||
|
||||
Before answering any question about a person, company, or topic:
|
||||
@@ -123,25 +100,6 @@ ingest event.
|
||||
No separate output. Brain-ops is an always-on behavior layer, not a report generator.
|
||||
The output is updated brain pages and enriched responses.
|
||||
|
||||
## Cross-source citation format (v0.18.0+)
|
||||
|
||||
When a brain has multiple sources (wiki, gstack, yc-media, etc.), every
|
||||
citation MUST include the source id: `[source-id:slug]`. Example:
|
||||
|
||||
> You told me about the retry budget approach — see
|
||||
> [wiki:topics/resilience] and [gstack:plans/retry-policy] for where
|
||||
> this came from.
|
||||
|
||||
Rules:
|
||||
- The key is `sources.id` (immutable), never `sources.name` (mutable display).
|
||||
- Single-source brains still write `[default:slug]` OR may omit the prefix
|
||||
for backward compat.
|
||||
- Every page payload returned by `search`, `query`, `get_page`, `list_pages`
|
||||
carries `source_id` — always use it when citing, never guess.
|
||||
|
||||
If a search result has `source_id: "gstack"` and `slug: "plans/foo"`,
|
||||
the citation is `[gstack:plans/foo]`. That's the whole rule.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- Answering questions about people/companies without checking the brain first
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
// Routing eval fixtures for skills/citation-fixer. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please fix citations in the latest batch of brain pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "I need to fix citations across these pages", "expected_skill": "citation-fixer"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "What does this book say about mentorship", "expected_skill": null, "ambiguous_with": []}
|
||||
@@ -1,93 +0,0 @@
|
||||
# Cron via Minions Convention
|
||||
|
||||
How cron-scheduled agent work is dispatched in a GBrain-backed install.
|
||||
|
||||
## Rule: scheduled work runs as Minion jobs, not `agentTurn`
|
||||
|
||||
When a cron fires, it should submit a Minion job. Not call OpenClaw's
|
||||
native `agentTurn` (300s timeout, no durability, no transcript). Not
|
||||
start an isolated session that races the gateway for resources.
|
||||
|
||||
```
|
||||
# Bad: agentTurn with a fixed timeout, no durability.
|
||||
{ "schedule": "*/30 * * * *", "kind": "agentTurn", "skill": "ea-inbox-sweep" }
|
||||
|
||||
# Good (Postgres): fire-and-forget submit with an idempotency key per
|
||||
# cycle slot. The queue dedupes long-running overlaps at the DB layer.
|
||||
{
|
||||
"schedule": "*/30 * * * *",
|
||||
"kind": "shell",
|
||||
"cmd": "gbrain jobs submit ea-inbox-sweep --params '{\"slot\":\"$(date -u +%Y-%m-%dT%H:%M)\"}' --idempotency-key ea-inbox-sweep:$(date -u +%Y-%m-%dT%H:%M)"
|
||||
}
|
||||
|
||||
# Good (PGLite): inline execution with --follow. PGLite's exclusive file
|
||||
# lock blocks a separate worker daemon, so the cron runs the job directly.
|
||||
{
|
||||
"schedule": "*/30 * * * *",
|
||||
"kind": "shell",
|
||||
"cmd": "gbrain jobs submit ea-inbox-sweep --params '{}' --follow"
|
||||
}
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
- **Durability.** Gateway restart mid-task? Worker picks the job up on
|
||||
boot. No lost state.
|
||||
- **Observability.** `gbrain jobs list` + `gbrain jobs get <id>` show
|
||||
every run, its duration, its transcript, its token accounting.
|
||||
- **Steering.** Running jobs accept inbox messages. "Skip the
|
||||
newsletter thread, focus on the urgent DMs" lands as context on the
|
||||
next iteration.
|
||||
- **Concurrency safety.** Idempotency-key on the cycle slot means a cron
|
||||
that fires during a still-running previous invocation produces a noop
|
||||
at the queue layer. Without this, a 5-min cron running 8-min jobs
|
||||
stacks 4 overlapping copies at steady state.
|
||||
|
||||
## Who registers the handler?
|
||||
|
||||
**GBrain only rewrites cron entries whose handler name matches a
|
||||
gbrain builtin** (`sync`, `embed`, `lint`, `import`, `extract`,
|
||||
`backlinks`, `autopilot-cycle`). For host-specific handlers
|
||||
(`ea-inbox-sweep`, `morning-briefing`, whatever your deployment runs
|
||||
on cron), the host platform ships the handler as code.
|
||||
|
||||
See `docs/guides/plugin-handlers.md` for the plugin contract. In short:
|
||||
|
||||
```ts
|
||||
import { MinionQueue, MinionWorker } from 'gbrain/minions';
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: 'default' });
|
||||
worker.register('ea-inbox-sweep', async (ctx) => {
|
||||
// Host-specific agent turn. Call whatever LLM + tools the host has.
|
||||
// ctx.data contains the cron slot payload; return a result object.
|
||||
});
|
||||
await worker.start();
|
||||
```
|
||||
|
||||
Ship the bootstrap in the host repo. Autopilot spawns the worker as a
|
||||
child; the host's custom worker binary (or a side-effect module the
|
||||
stock worker auto-loads on startup) registers handlers before `start()`.
|
||||
|
||||
## Off mode
|
||||
|
||||
Users who set `minion_mode: off` in `~/.gbrain/preferences.json` keep
|
||||
using `agentTurn`. Respect that. No auto-rewrite.
|
||||
|
||||
## Forward note (v0.12.0)
|
||||
|
||||
GBrain v0.12.0 ships `gbrain cron`: a scheduler loop inside
|
||||
`gbrain jobs work` that owns cron expressions natively — no more
|
||||
handing off to host schedulers. Until v0.12.0 lands, the host
|
||||
scheduler keeps firing on schedule; v0.11.1 only replaces the execution
|
||||
layer (what the cron trigger *does*), not the scheduling layer.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/conventions/subagent-routing.md` — native subagents vs
|
||||
Minions for ad-hoc (not cron-scheduled) work.
|
||||
- `skills/minion-orchestrator/SKILL.md` — patterns for managing jobs
|
||||
once they're in the queue.
|
||||
- `skills/cron-scheduler/SKILL.md` — scheduling guidance (quiet hours,
|
||||
staggering, idempotency). Now references this convention.
|
||||
- `skills/migrations/v0.11.0.md` — how GBrain migrates an existing host
|
||||
cron manifest to this convention.
|
||||
@@ -1,97 +0,0 @@
|
||||
# Subagent Routing Convention
|
||||
|
||||
How your agent decides between native subagents and GBrain Minions for
|
||||
background work.
|
||||
|
||||
## Rule: read the user's mode first
|
||||
|
||||
Every time you're about to spawn a subagent or submit a Minion, check the
|
||||
user's preference:
|
||||
|
||||
```bash
|
||||
cat ~/.gbrain/preferences.json 2>/dev/null
|
||||
```
|
||||
|
||||
The file has `minion_mode` set to one of three values. If the file doesn't
|
||||
exist, default to `pain_triggered`.
|
||||
|
||||
Then branch on mode:
|
||||
|
||||
### Mode A: `always`
|
||||
|
||||
All background agent work goes through Minions. `submit_job` is the default.
|
||||
Never fall back to `sessions_spawn` with `runtime: "subagent"`.
|
||||
|
||||
Reference: `skills/minion-orchestrator/SKILL.md`
|
||||
|
||||
### Mode B: `pain_triggered` (default)
|
||||
|
||||
Default to native subagents. They're zero-setup, low-latency, and fine for
|
||||
most tasks. Switch to Minions when one or more of these pain signals fires:
|
||||
|
||||
- **Gateway restart mid-task.** A subagent was running, the gateway crashed
|
||||
or restarted, and you can't resume it.
|
||||
- **State drop.** The user asks "what happened to the agent" or "why did
|
||||
you forget what you were doing" and the underlying cause is the subagent
|
||||
runtime losing context.
|
||||
- **Parallel > 3.** The task needs more than 3 concurrent agents. Native
|
||||
subagents start to drop at this width.
|
||||
- **Long runtime.** The task is expected to take > 5 minutes. The user will
|
||||
want to check progress, and polling a subagent doesn't give structured
|
||||
progress the way `get_job_progress` does.
|
||||
- **User explicit frustration.** The user says any of: "this keeps breaking",
|
||||
"the agent keeps losing track", "why is this so flaky", "can we do this
|
||||
in the background properly."
|
||||
|
||||
When ≥1 signal fires, pause and offer the switch:
|
||||
|
||||
> "Subagents keep dropping state on you. GBrain Minions is a durable queue
|
||||
> for the same work — it survives gateway restarts, every job has progress
|
||||
> tracking and a transcript, and you can pause / resume / steer running
|
||||
> agents. Want me to flip this task to Minions? (~10s, no extra setup.)"
|
||||
|
||||
If the user says yes, submit the task as a Minion job with the same prompt.
|
||||
Optionally propose flipping the default: `gbrain config set minion_mode always`.
|
||||
|
||||
### Mode C: `off`
|
||||
|
||||
Never suggest Minions. Use native subagents for everything. The user has
|
||||
opted out. Respect it.
|
||||
|
||||
## Inline vs background (applies in all modes)
|
||||
|
||||
Even when Minions is the default (mode A), some work should run inline:
|
||||
|
||||
| Condition | Action |
|
||||
|---|---|
|
||||
| Single tool call, < 30s | Inline, always |
|
||||
| Read-only query | Inline |
|
||||
| User is waiting in real-time for the answer | Inline |
|
||||
| Multi-step, user can walk away | Minion |
|
||||
| Parallel 2+ streams | Minion (parent + children) |
|
||||
| Needs to survive restart | Minion |
|
||||
| User wants progress updates | Minion |
|
||||
| Research / bulk operation | Minion |
|
||||
|
||||
**Rule of thumb:** if the user might ask "is it done yet?", use a Minion.
|
||||
|
||||
## Concurrency budget
|
||||
|
||||
Before submitting batch jobs:
|
||||
|
||||
- Check active queue depth via `list_jobs --status active` (MCP-callable) or `gbrain jobs stats` (CLI)
|
||||
- If active > 5, stagger new jobs with `delay` so you don't swarm
|
||||
- The resource governor auto-throttles but don't dump 20 jobs at once
|
||||
|
||||
## Flipping modes
|
||||
|
||||
The user can change their mind at any time:
|
||||
|
||||
```bash
|
||||
gbrain config set minion_mode always # switch to always-on
|
||||
gbrain config set minion_mode pain_triggered # back to default
|
||||
gbrain config set minion_mode off # disable suggestions
|
||||
```
|
||||
|
||||
Or edit `~/.gbrain/preferences.json` directly. The convention reads the file
|
||||
on every decision, so changes take effect next tool call.
|
||||
@@ -39,7 +39,7 @@ This skill guarantees:
|
||||
- Override: user-awake flag (if user is active, quiet hours suspended)
|
||||
- During quiet hours: save output to held queue
|
||||
- Morning contact releases the backlog
|
||||
4. **Register with host scheduler.** OpenClaw cron, Railway cron, crontab, or process manager. **Each registered entry should execute via Minions, not `agentTurn`.** See `skills/conventions/cron-via-minions.md` for the rewrite pattern (PGLite uses `--follow`, Postgres uses fire-and-forget + `--idempotency-key` on the cycle slot). GBrain's v0.11.0 migration auto-rewrites entries for built-in handlers; host-specific handlers need a code-level registration per `docs/guides/plugin-handlers.md`.
|
||||
4. **Register with host scheduler.** OpenClaw cron, Railway cron, crontab, or process manager.
|
||||
5. **Write thin prompt.** Job prompt is one line: "Read skills/{name}/SKILL.md and run it."
|
||||
|
||||
## Idempotency Requirement
|
||||
|
||||
+14
-13
@@ -20,10 +20,6 @@ tools:
|
||||
- add_timeline_entry
|
||||
- get_backlinks
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- companies/
|
||||
---
|
||||
|
||||
# Enrich Skill
|
||||
@@ -41,7 +37,7 @@ This skill guarantees:
|
||||
|
||||
> **Filing rule:** Read `skills/_brain-filing-rules.md` before creating any new page.
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link
|
||||
FROM that entity's page TO the page mentioning them. An unlinked mention is a
|
||||
@@ -55,7 +51,18 @@ they building, what makes them tick, where are they headed.
|
||||
|
||||
## Citation Requirements (MANDATORY)
|
||||
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation formats and source precedence.
|
||||
Every fact must carry an inline `[Source: ...]` citation.
|
||||
|
||||
Three formats:
|
||||
- **Direct attribution:** `[Source: User, {context}, YYYY-MM-DD]`
|
||||
- **API/external:** `[Source: {provider} enrichment, YYYY-MM-DD]`
|
||||
- **Synthesis:** `[Source: compiled from {list of sources}]`
|
||||
|
||||
Source precedence (highest to lowest):
|
||||
1. User's direct statements
|
||||
2. Compiled truth (pre-existing brain synthesis)
|
||||
3. Timeline entries (raw evidence)
|
||||
4. External sources (API enrichment, web search)
|
||||
|
||||
When sources conflict, note the contradiction with both citations.
|
||||
|
||||
@@ -272,15 +279,9 @@ Active items, pending decisions, things to track.
|
||||
|
||||
- Update company pages from person enrichment (and vice versa)
|
||||
- Update related project/deal pages if relevant context surfaced
|
||||
- Add back-links from every entity mentioned (MANDATORY)
|
||||
- Check index files if the brain uses them
|
||||
|
||||
**Note (v0.10.1):** Links between brain pages are auto-created on every
|
||||
`put_page` call (auto-link post-hook). Step 7 focuses on content
|
||||
cross-references (updating related pages' compiled truth with new signal
|
||||
from this enrichment), not on creating links. Verify via the `auto_links`
|
||||
field in the put_page response (`{ created, removed, errors }`).
|
||||
Timeline entries still need explicit `gbrain timeline-add` calls.
|
||||
|
||||
## Bulk Enrichment Rules
|
||||
|
||||
- **Test on 3-5 entities first.** Read actual output. Check quality.
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
---
|
||||
name: frontmatter-guard
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Validate and auto-repair YAML frontmatter on brain pages. Catches malformed
|
||||
pages before they enter the brain (missing closing ---, nested quotes, slug
|
||||
mismatches, null bytes, empty frontmatter, YAML parse failures). Wraps the
|
||||
`gbrain frontmatter` CLI for agent-driven workflows.
|
||||
triggers:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
tools:
|
||||
- exec
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Frontmatter Guard Skill
|
||||
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation rules; this skill is structural validation, not citation auditing.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
- Every brain page is scanned against the seven canonical frontmatter validation classes
|
||||
- Mechanical errors (nested quotes, missing closing `---`, null bytes, slug mismatch) are auto-repairable on demand with `.bak` backups
|
||||
- Validation logic is shared with `gbrain doctor`'s `frontmatter_integrity` subcheck — single source of truth
|
||||
- Reports per source (gbrain is multi-source since v0.18.0); never silently audits the wrong root
|
||||
|
||||
## Why This Exists
|
||||
|
||||
Brain pages pile up over months. Agents write them with malformed frontmatter:
|
||||
- Missing closing `---` (entity detector bugs)
|
||||
- Unstructured YAML in meeting pages (ingestion bugs)
|
||||
- Slug mismatches (path renames not propagated)
|
||||
- Null bytes (binary corruption from copy-paste accidents)
|
||||
- Nested double quotes in titles (`title: "Phil "Nick" Last"`)
|
||||
|
||||
Without a guard, these accumulate silently until `gbrain sync` chokes or search returns garbage. The guard makes the failure visible at audit time and trivially fixable.
|
||||
|
||||
## Validation classes
|
||||
|
||||
| Code | Meaning | Auto-fixable? |
|
||||
|------|---------|---------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` | No (needs human) |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading | Yes |
|
||||
| `YAML_PARSE` | YAML failed to parse | Sometimes (depends on cause) |
|
||||
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes (removes the field) |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes |
|
||||
| `EMPTY_FRONTMATTER` | Open + close present but nothing between | No (needs human) |
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Audit
|
||||
|
||||
Run a read-only scan across all registered sources (or one with `--source <id>`).
|
||||
|
||||
```bash
|
||||
gbrain frontmatter audit --json
|
||||
```
|
||||
|
||||
Reports:
|
||||
- Per-source counts grouped by error code
|
||||
- Sample of up to 20 affected pages per source
|
||||
- Total count
|
||||
- Scan timestamp
|
||||
|
||||
Output is JSON; agents parse `errors_by_code` and `per_source` to decide next steps.
|
||||
|
||||
### Phase 2: Validate one path
|
||||
|
||||
Validate a single file or directory (does not require source registration):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
Exit code 0 = clean; 1 = errors found. Use this in CI pipelines or pre-commit hooks.
|
||||
|
||||
### Phase 3: Fix
|
||||
|
||||
When issues are found:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes `<file>.bak` for every modified file before mutating. The backup is the safety contract — works whether the brain is a git repo or a plain directory.
|
||||
|
||||
`--dry-run` previews without writing. Use this before applying fixes in batch.
|
||||
|
||||
### Phase 4: Pre-commit hook (optional)
|
||||
|
||||
For brain repos that ARE git repos, install the pre-commit hook to block malformed pages from being committed in the first place:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
The hook runs `gbrain frontmatter validate` against staged `.md`/`.mdx` files. Bypass with `git commit --no-verify`.
|
||||
|
||||
## Trigger words
|
||||
|
||||
When the user says any of these, route here:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
|
||||
## Output rules
|
||||
|
||||
- Always run `gbrain frontmatter audit --json` first; never assume a brain is clean.
|
||||
- Surface counts to the user in plain language; do not dump raw JSON.
|
||||
- For `--fix` operations: state how many files will be modified BEFORE running, then confirm.
|
||||
- `SLUG_MISMATCH` fixes remove the frontmatter `slug:` field — gbrain derives slug from path. Mention this when the user's title is intentionally renamed.
|
||||
- Never auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without explicit user input — these usually mean a human author started a page and didn't finish.
|
||||
|
||||
## Chains with
|
||||
|
||||
- `gbrain doctor` — the `frontmatter_integrity` subcheck reports the same counts as `audit`.
|
||||
- `skills/maintain/SKILL.md` — broader brain health audit; chain after this skill if other classes of issue are suspected.
|
||||
- `skills/lint/SKILL.md` (via `gbrain lint`) — overlapping rules for skill-file lint; the `frontmatter-*` rule names in lint output come from this skill's validation surface.
|
||||
|
||||
## Output Format
|
||||
|
||||
Audit summary (terse, agent-friendly):
|
||||
|
||||
```
|
||||
Frontmatter audit — 17 issue(s) across 1 source(s)
|
||||
|
||||
[default] /Users/me/brain
|
||||
17 issue(s)
|
||||
MISSING_CLOSE: 8
|
||||
NESTED_QUOTES: 5
|
||||
NULL_BYTES: 4
|
||||
sample:
|
||||
people/jane.md — MISSING_CLOSE
|
||||
companies/acme.md — NESTED_QUOTES
|
||||
(+ 12 more)
|
||||
|
||||
Fix with: gbrain frontmatter validate /Users/me/brain --fix
|
||||
```
|
||||
|
||||
JSON envelope (when `--json` is passed):
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
`gbrain frontmatter validate <path> --json` returns a similar envelope keyed on per-file results instead of per-source.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
**Don't auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without user input.** These usually mean a human author started a page and didn't finish — silently inserting `---` markers around an unfinished draft is wrong.
|
||||
|
||||
**Don't use `--fix` to "make doctor green" without reading the audit first.** SLUG_MISMATCH cases are surfaced for manual review specifically because gbrain derives the slug from path. A mismatch usually means the user renamed a file intentionally; auto-removing the slug field is the right outcome only when you've confirmed the rename was deliberate.
|
||||
|
||||
**Don't skip the `.bak` backups.** The `.bak` is the safety contract for non-git brain repos. If `.bak` files accumulate after a fix run, that's a feature, not a bug — the user can review the diffs and delete the backups when satisfied.
|
||||
|
||||
**Don't run `audit` on a brain where sources aren't registered.** The CLI returns "no registered sources to audit" gracefully, but the migration emits a `skipped: no_sources` phase result. Don't paper over this with a manual path-walk; the right fix is to register the source via `gbrain sources add`.
|
||||
|
||||
**Don't install the pre-commit hook on non-git brain dirs.** The install-hook command skips them automatically with a one-line note. If you see "skipped — not a git repo" and want validation at write time anyway, use the `audit` command on a cron schedule.
|
||||
@@ -1,8 +0,0 @@
|
||||
// Routing eval fixtures for skills/frontmatter-guard. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please validate frontmatter on the latest batch of brain pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "fix frontmatter on these pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "I want to run a frontmatter audit across the brain", "expected_skill": "frontmatter-guard"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "what's for breakfast", "expected_skill": null, "ambiguous_with": []}
|
||||
@@ -20,11 +20,6 @@ tools:
|
||||
- add_timeline_entry
|
||||
- file_upload
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- concepts/
|
||||
- sources/
|
||||
---
|
||||
|
||||
# Idea Ingest Skill
|
||||
@@ -41,7 +36,7 @@ This skill guarantees:
|
||||
- Every fact has an inline `[Source: ...]` citation
|
||||
- Filing follows primary subject rules (not format-based)
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link.
|
||||
Format: `- **YYYY-MM-DD** | Referenced in [page title](path) — brief context`
|
||||
|
||||
@@ -13,13 +13,6 @@ tools:
|
||||
- add_timeline_entry
|
||||
- sync_brain
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- companies/
|
||||
- concepts/
|
||||
- meetings/
|
||||
- sources/
|
||||
---
|
||||
|
||||
# Ingest Skill
|
||||
@@ -36,7 +29,7 @@ Ingest meetings, articles, media, documents, and conversations into the brain.
|
||||
- State sections are rewritten with current best understanding, never appended to.
|
||||
- Entity detection fires on every inbound message; notable entities get pages or updates.
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link
|
||||
FROM that entity's page TO the page mentioning them. An unlinked mention is a
|
||||
|
||||
@@ -8,15 +8,10 @@ description: |
|
||||
triggers:
|
||||
- "brain health"
|
||||
- "check backlinks"
|
||||
- "citation audit"
|
||||
- "maintenance"
|
||||
- "orphan pages"
|
||||
- "stale pages"
|
||||
- "extract links"
|
||||
- "build link graph"
|
||||
- "populate timeline"
|
||||
- "populate links"
|
||||
- "backfill graph"
|
||||
- "extract timeline entries"
|
||||
tools:
|
||||
- get_health
|
||||
- get_page
|
||||
@@ -64,56 +59,6 @@ Links pointing to pages that don't exist.
|
||||
Pages that mention entity names but don't have formal links.
|
||||
- Read compiled_truth from gbrain, extract entity mentions, create links in gbrain
|
||||
|
||||
### Link graph extraction
|
||||
If link_count is 0 or low relative to page_count, run batch extraction:
|
||||
```bash
|
||||
gbrain extract links --dir ~/brain
|
||||
```
|
||||
This scans all markdown files for entity references, See Also sections, and
|
||||
frontmatter fields, then creates typed links in the database.
|
||||
|
||||
### Timeline extraction
|
||||
If timeline_entry_count is 0, extract structured timeline from markdown:
|
||||
```bash
|
||||
gbrain extract timeline --dir ~/brain
|
||||
```
|
||||
Parses `- **YYYY-MM-DD** | Source — Summary` and `### YYYY-MM-DD — Title` formats.
|
||||
Note: extracted entries improve structured queries (`gbrain timeline`), not vector search.
|
||||
|
||||
### Autopilot check
|
||||
Verify autopilot is running:
|
||||
```bash
|
||||
gbrain autopilot --status
|
||||
```
|
||||
If not running, install it:
|
||||
```bash
|
||||
gbrain autopilot --install --repo ~/brain
|
||||
```
|
||||
Autopilot runs sync, extract, and embed in a continuous loop with adaptive scheduling.
|
||||
In v0.11.1+, autopilot dispatches each cycle as a single `autopilot-cycle`
|
||||
Minion job and supervises the worker child — one install step gives you
|
||||
sync + extract + embed + backlinks + durable job processing.
|
||||
|
||||
### Fix a half-migrated install
|
||||
A v0.11.0 install where the migration skill never fired leaves Minions
|
||||
partially set up: schema is applied, but `~/.gbrain/preferences.json`
|
||||
doesn't exist, autopilot runs inline, host manifests still reference
|
||||
`agentTurn`. Repair:
|
||||
|
||||
```bash
|
||||
# Check migration status
|
||||
gbrain apply-migrations --list
|
||||
|
||||
# Apply pending migrations (idempotent; safe on healthy installs)
|
||||
gbrain apply-migrations --yes
|
||||
|
||||
# If host-specific handlers are flagged in ~/.gbrain/migrations/pending-host-work.jsonl:
|
||||
# walk them per skills/migrations/v0.11.0.md + docs/guides/plugin-handlers.md,
|
||||
# ship handler registrations in the host repo, then re-run apply-migrations.
|
||||
```
|
||||
|
||||
Full troubleshooting guide: `docs/guides/minions-fix.md`.
|
||||
|
||||
### Back-link enforcement
|
||||
Check that the back-linking iron law is being followed:
|
||||
- For each recently updated page, check if entities mentioned in it have
|
||||
@@ -141,34 +86,6 @@ Spot-check pages for missing `[Source: ...]` citations:
|
||||
Inconsistent tagging (e.g., "vc" vs "venture-capital", "ai" vs "artificial-intelligence").
|
||||
- Standardize to the most common variant using gbrain tag operations
|
||||
|
||||
### Graph population (v0.10.3+)
|
||||
|
||||
The `links` and `timeline_entries` tables are the structured graph layer.
|
||||
Populate them periodically or after major imports:
|
||||
|
||||
- `gbrain extract links --source db` — backfill structured links by walking pages
|
||||
from the engine. Reads `[Name](people/slug)` / `[Name](companies/slug)` references
|
||||
and infers relationship types (`attended`, `works_at`, `invested_in`, `founded`,
|
||||
`advises`, `mentions`, `source`). Idempotent. Use `--source fs --dir <brain>`
|
||||
if you have a markdown checkout to walk instead.
|
||||
- `gbrain extract timeline --source db` — backfill structured timeline entries.
|
||||
Parses `- **YYYY-MM-DD** | summary` lines from page content. Idempotent (DB
|
||||
UNIQUE constraint).
|
||||
- `gbrain extract all --source db` — both in one run.
|
||||
- `gbrain graph-query <slug> --depth 2` — verify connectivity (use any well-known
|
||||
entity slug as a probe).
|
||||
- `gbrain stats` — verify `link_count > 0` and `timeline_entry_count > 0` after extraction.
|
||||
- `gbrain health` — review `link_coverage` and `timeline_coverage` percentages
|
||||
on entity pages (person/company). Below 50% means more extraction is needed.
|
||||
|
||||
Available link types (use with `gbrain graph-query --type`):
|
||||
`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, `source`.
|
||||
|
||||
Going forward, every `gbrain put` call auto-creates and reconciles links via the
|
||||
auto-link post-hook (default on; disable: `gbrain config set auto_link false`).
|
||||
So link-extract is mostly a one-time backfill. timeline-extract should be re-run
|
||||
after bulk imports or content edits that add new dated entries.
|
||||
|
||||
### Embedding freshness
|
||||
Chunks without embeddings, or chunks embedded with an old model.
|
||||
- For large embedding refreshes (>1000 chunks), use nohup:
|
||||
|
||||
+1
-26
@@ -2,7 +2,7 @@
|
||||
"name": "gbrain",
|
||||
"version": "0.10.0",
|
||||
"conformance_version": "1.0.0",
|
||||
"description": "Personal knowledge brain with hybrid RAG search \u2014 GStack mod for agent platforms",
|
||||
"description": "Personal knowledge brain with hybrid RAG search — GStack mod for agent platforms",
|
||||
"skills": [
|
||||
{
|
||||
"name": "ingest",
|
||||
@@ -44,11 +44,6 @@
|
||||
"path": "publish/SKILL.md",
|
||||
"description": "Share brain pages as beautiful password-protected HTML (code + skill pair, zero LLM calls)"
|
||||
},
|
||||
{
|
||||
"name": "frontmatter-guard",
|
||||
"path": "frontmatter-guard/SKILL.md",
|
||||
"description": "Validate and auto-repair YAML frontmatter on brain pages; gates against malformed YAML, missing closing ---, nested quotes, slug mismatches, null bytes"
|
||||
},
|
||||
{
|
||||
"name": "signal-detector",
|
||||
"path": "signal-detector/SKILL.md",
|
||||
@@ -133,26 +128,6 @@
|
||||
"name": "data-research",
|
||||
"path": "data-research/SKILL.md",
|
||||
"description": "Structured data research: search, extract, archive, deduplicate, track. Parameterized YAML recipes for investor updates, donations, company metrics."
|
||||
},
|
||||
{
|
||||
"name": "minion-orchestrator",
|
||||
"path": "minion-orchestrator/SKILL.md",
|
||||
"description": "Unified Minions skill for deterministic shell jobs and LLM subagent orchestration. Submit, monitor, steer, pause/resume, replay. Replaces the older gbrain-jobs routing intent and sessions_spawn for durable observable background work."
|
||||
},
|
||||
{
|
||||
"name": "skillify",
|
||||
"path": "skillify/SKILL.md",
|
||||
"description": "Meta skill. Turn any raw feature into a properly-skilled, tested, resolvable, evaled unit. Paired with gbrain check-resolvable gives user-controllable auto-skill-creation."
|
||||
},
|
||||
{
|
||||
"name": "skillpack-check",
|
||||
"path": "skillpack-check/SKILL.md",
|
||||
"description": "Agent-readable gbrain health report. Wraps doctor + apply-migrations --list into one JSON blob with exit codes. Cron-friendly for morning-briefing pipelines."
|
||||
},
|
||||
{
|
||||
"name": "smoke-test",
|
||||
"path": "smoke-test/SKILL.md",
|
||||
"description": "Post-restart smoke tests + auto-fix for gbrain and OpenClaw environments"
|
||||
}
|
||||
],
|
||||
"dependencies": {
|
||||
|
||||
@@ -22,12 +22,6 @@ tools:
|
||||
- add_timeline_entry
|
||||
- file_upload
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- concepts/
|
||||
- people/
|
||||
- companies/
|
||||
- sources/
|
||||
---
|
||||
|
||||
# Media Ingest Skill
|
||||
@@ -45,7 +39,7 @@ This skill guarantees:
|
||||
- Raw source files preserved via `gbrain files upload-raw`
|
||||
- Filing by primary subject, not by media format
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link.
|
||||
|
||||
|
||||
@@ -18,11 +18,6 @@ tools:
|
||||
- add_link
|
||||
- add_timeline_entry
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- meetings/
|
||||
- people/
|
||||
- companies/
|
||||
---
|
||||
|
||||
# Meeting Ingestion Skill
|
||||
@@ -39,7 +34,7 @@ This skill guarantees:
|
||||
- Meeting is NOT fully ingested until enrich runs for every entity
|
||||
- Back-links created bidirectionally
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every attendee and company mentioned MUST get a back-link from their page to
|
||||
the meeting page. An unlinked mention is a broken brain.
|
||||
@@ -84,14 +79,7 @@ For EACH attendee:
|
||||
1. `gbrain search "{name}"` — does a people page exist?
|
||||
2. If NO → create via enrich skill (this is mandatory, not optional)
|
||||
3. If YES → update compiled truth with meeting context
|
||||
4. Add timeline entry on the person's page:
|
||||
`gbrain timeline-add <person-slug> <date> "Attended <meeting-title>"`
|
||||
|
||||
**Note (v0.10.1):** Once the meeting page is written via `gbrain put`, the
|
||||
auto-link post-hook automatically creates `attended` links from the meeting
|
||||
to each attendee whose page is referenced as `[Name](people/slug)`. You don't
|
||||
need to call `gbrain link` for attendees. You DO still need `gbrain timeline-add`
|
||||
for dated events (auto-link only handles links, not timeline entries).
|
||||
4. Add timeline entry: `- **{date}** | Attended [{meeting title}](path) — {context}`
|
||||
|
||||
### Phase 4: Entity propagation (MANDATORY)
|
||||
|
||||
|
||||
+4
-11
@@ -50,17 +50,10 @@ Universal migration from any wiki, note tool, or brain system into GBrain.
|
||||
## Obsidian Migration
|
||||
|
||||
1. Import the vault directory into gbrain (Obsidian vaults are markdown directories)
|
||||
2. Wire the graph with native wikilink support (v0.12.1+):
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db --dry-run | head -20 # preview
|
||||
gbrain extract links --source db # commit
|
||||
```
|
||||
|
||||
`extract links` natively parses `[[relative/path]]` and `[[relative/path|Display Text]]`
|
||||
alongside standard `[text](page.md)` markdown syntax. Ancestor-search resolution handles
|
||||
wiki KBs where authors omit one or more leading `../` prefixes. The `.md` suffix is
|
||||
inferred automatically for wikilinks.
|
||||
2. Convert `[[wikilinks]]` to gbrain links:
|
||||
- Read each page from gbrain
|
||||
- For each `[[Name]]` found, resolve to a slug and create a link in gbrain
|
||||
- `[[Name|alias]]` uses the alias for context
|
||||
|
||||
Obsidian-specific:
|
||||
- Tags (`#tag`) become gbrain tags
|
||||
|
||||
@@ -1,142 +0,0 @@
|
||||
---
|
||||
version: 0.10.3
|
||||
feature_pitch:
|
||||
headline: "Knowledge graph layer — your brain now wires itself"
|
||||
description: |
|
||||
Auto-link on every page write creates and reconciles links automatically.
|
||||
Typed relationships (works_at, attended, invested_in, founded, advises).
|
||||
Graph-powered search boost. New `extract --source db` mode for live brains.
|
||||
Graph traversal queries via `gbrain graph-query`.
|
||||
recipe: null
|
||||
tiers: null
|
||||
auto_execute:
|
||||
- cmd: gbrain init
|
||||
description: Apply schema migrations v5/v6/v7 (idempotent, safe to re-run)
|
||||
- cmd: gbrain extract links --source db
|
||||
description: Backfill typed links from existing pages (~30s for 30K pages, idempotent)
|
||||
- cmd: gbrain extract timeline --source db
|
||||
description: Backfill structured timeline entries (idempotent via UNIQUE index)
|
||||
- cmd: gbrain stats
|
||||
description: Verify links and timeline_entry_count are non-zero
|
||||
---
|
||||
|
||||
# v0.10.3 Migration: Knowledge Graph Layer
|
||||
|
||||
This release turns the structured `links` and `timeline_entries` tables into a
|
||||
real knowledge graph. Brains that have been accumulating page content but
|
||||
showing 0 links and 0 timeline entries (because no command populated them) can
|
||||
now backfill in seconds and keep the graph in sync going forward.
|
||||
|
||||
## What Changed
|
||||
|
||||
### Auto-link on every page write
|
||||
Every `gbrain put` (and MCP `put_page`) now extracts entity references from the
|
||||
page content and writes them to the `links` table with inferred relationship
|
||||
types. Stale links (refs no longer in the page) are removed in the same call.
|
||||
|
||||
The MCP `put_page` response now includes an `auto_links` field:
|
||||
```
|
||||
{ status: "created_or_updated", chunks: 5, auto_links: { created: 3, removed: 1, errors: 0 } }
|
||||
```
|
||||
|
||||
To disable: `gbrain config set auto_link false`. Default is on.
|
||||
|
||||
### Extended `gbrain extract` + new `gbrain graph-query`
|
||||
|
||||
- `gbrain extract links --source db` — backfill structured links by walking pages from
|
||||
the engine (works on live brains with no local checkout). FS-source still works:
|
||||
`gbrain extract links --source fs --dir <path>` walks markdown files (v0.10.1 behavior preserved).
|
||||
Includes typed link inference, within-page dedup, content-hash-based `--since` filter.
|
||||
- `gbrain extract timeline --source db` — backfill structured timeline entries from page
|
||||
content via the engine. FS-source path unchanged.
|
||||
- `gbrain extract all --source db` — both in one run.
|
||||
- `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]` —
|
||||
relationship traversal returning typed edges (GraphPath[]).
|
||||
|
||||
### Schema migrations (auto-applied on `gbrain init`)
|
||||
- v5: `links` UNIQUE constraint widened to `(from, to, link_type)`. Same person
|
||||
can now both `works_at` AND `advises` the same company (separate rows).
|
||||
- v6: `timeline_entries` gets a UNIQUE index on `(page_id, date, summary)` plus
|
||||
ON CONFLICT DO NOTHING in `addTimelineEntry`. Idempotent inserts.
|
||||
- v7: Drops the trigger that updated `pages.updated_at` on every timeline insert.
|
||||
Structured timeline entries are now graph data only, not search text. The
|
||||
markdown timeline section in `pages.timeline` still feeds search via the
|
||||
pages trigger.
|
||||
|
||||
### Graph health metrics in `gbrain health`
|
||||
- `link_coverage`: % of entity pages (person/company) with >= 1 inbound link.
|
||||
- `timeline_coverage`: % of entity pages with >= 1 structured timeline entry.
|
||||
- `most_connected`: top 5 entities by total link count.
|
||||
- `dead_links` removed (always 0 under ON DELETE CASCADE — was a phantom metric).
|
||||
|
||||
### Search ranking improvement
|
||||
Hybrid search now applies a small backlink boost after cosine re-scoring:
|
||||
`score *= (1 + 0.05 * log(1 + backlink_count))`. Well-connected entities surface
|
||||
higher in results. Works in both keyword-only and full hybrid paths.
|
||||
|
||||
### Graph-query MCP exposure
|
||||
The existing `traverse_graph` MCP operation now accepts optional `link_type` and
|
||||
`direction` params. With either set, returns `GraphPath[]` (list of typed edges)
|
||||
instead of the legacy `GraphNode[]` (list of nodes with outgoing-only links).
|
||||
Backward compat: zero-param calls still return GraphNode[].
|
||||
|
||||
## Agent Instructions
|
||||
|
||||
### Step 1: Apply schema migrations
|
||||
```bash
|
||||
gbrain init
|
||||
```
|
||||
Applies migrations v5/v6/v7 to existing brain. Idempotent. Existing data is
|
||||
preserved (duplicates deduplicated before constraints are added).
|
||||
|
||||
### Step 2: Backfill links from existing pages
|
||||
```bash
|
||||
gbrain extract links --source db
|
||||
```
|
||||
Iterates every page from the engine, extracts entity references, writes typed
|
||||
links. Should take ~30 seconds for a 30K-page brain. Run with `--dry-run --json`
|
||||
first to preview (JSON-lines output, agent-parseable).
|
||||
|
||||
If you have a local markdown checkout you'd rather walk: `gbrain extract links
|
||||
--source fs --dir <brain-dir>` (v0.10.1 behavior preserved).
|
||||
|
||||
### Step 3: Backfill timeline entries from page content
|
||||
```bash
|
||||
gbrain extract timeline --source db
|
||||
```
|
||||
Parses `- **YYYY-MM-DD** | summary` lines from page content and writes them
|
||||
as structured rows. Same scale and timing as links extraction.
|
||||
|
||||
### Step 4: Verify the graph populated
|
||||
```bash
|
||||
gbrain stats
|
||||
```
|
||||
Both `link_count` and `timeline_entry_count` should now be non-zero.
|
||||
|
||||
### Step 5: Try a graph traversal
|
||||
```bash
|
||||
gbrain graph-query <any-person-slug> --depth 2
|
||||
```
|
||||
Should show connected entities with their typed relationships. Useful slugs to
|
||||
probe: a known YC partner, a well-connected founder, a frequently-attended meeting.
|
||||
|
||||
### Step 6: Available link types
|
||||
- `attended` — meeting page references a person attendee
|
||||
- `works_at` — context like "CEO of", "works at", "VP at"
|
||||
- `invested_in` — context like "invested in", "backed by", "funding from"
|
||||
- `founded` — context like "founded", "co-founded"
|
||||
- `advises` — context like "advises", "advisor to", "board member"
|
||||
- `source` — frontmatter `source:` field
|
||||
- `mentions` — default for any reference that doesn't match the above
|
||||
|
||||
Use `gbrain graph-query <slug> --type <link_type>` to filter.
|
||||
|
||||
### Step 7: Auto-link is on by default
|
||||
No action needed. Every future `gbrain put` automatically updates the graph.
|
||||
To disable: `gbrain config set auto_link false`.
|
||||
|
||||
### Step 8: Record completion
|
||||
```bash
|
||||
mkdir -p ~/.gbrain/migrations
|
||||
echo '{"version":"0.10.3","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","status":"complete"}' >> ~/.gbrain/migrations/completed.jsonl
|
||||
```
|
||||
@@ -1,198 +0,0 @@
|
||||
---
|
||||
version: 0.11.0
|
||||
feature_pitch:
|
||||
headline: "GBrain Minions — durable background agents"
|
||||
description: "Turn any long-running agent task into a durable job that survives gateway restarts, streams progress, and can be paused, resumed, or steered mid-flight. Postgres-native, zero infra beyond your existing brain. Replaces flaky subagent spawns for multi-step work, parallel fan-out, and anything the user might ask about later."
|
||||
recipe: null
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.11.0 Migration: Minions — host-agent instruction manual
|
||||
|
||||
**Audience: host agents (OpenClaw deployments, future
|
||||
hosts) reading this AFTER `gbrain apply-migrations` has run its
|
||||
mechanical phases.** The orchestrator in
|
||||
`src/commands/migrations/v0_11_0.ts` is the runtime source of truth for
|
||||
the mechanical work (schema, smoke, prefs, built-in cron rewrites,
|
||||
autopilot install). This file tells the host agent how to finish the
|
||||
host-specific work that can't be done host-agnostically.
|
||||
|
||||
## Check what gbrain left you
|
||||
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/pending-host-work.jsonl
|
||||
```
|
||||
|
||||
Empty? You're done. The migration is complete; nothing for you to do.
|
||||
|
||||
Non-empty? Each line is a TODO. Each `type` routes to a section below.
|
||||
|
||||
## `type: cron-handler-needs-host-registration`
|
||||
|
||||
Gbrain rewrites cron entries whose handler name matches a gbrain
|
||||
builtin (`sync`, `embed`, `lint`, `import`, `extract`, `backlinks`,
|
||||
`autopilot-cycle`). For host-specific handlers (e.g. `ea-inbox-sweep`,
|
||||
`frameio-scan`, `x-dm-triage`, `calendar-sync` on your OpenClaw), gbrain
|
||||
leaves the manifest alone and emits a TODO with shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "cron-handler-needs-host-registration",
|
||||
"handler": "ea-inbox-sweep",
|
||||
"cron_schedule": "0 */30 * * *",
|
||||
"manifest_path": "/path/to/cron/jobs.json",
|
||||
"current_cmd": "agentTurn ea-inbox-sweep",
|
||||
"recommendation": "Add a handler registration for ...",
|
||||
"status": "pending"
|
||||
}
|
||||
```
|
||||
|
||||
**Each TODO is a full skillify, not a drive-by handler add.** A properly
|
||||
migrated cron meets every one of these gates before its JSONL row gets
|
||||
marked `complete`:
|
||||
|
||||
### (a) Plugin contract
|
||||
|
||||
Read `docs/guides/plugin-handlers.md`. Add a `worker.register()` call
|
||||
in your host's worker bootstrap:
|
||||
|
||||
```ts
|
||||
import { MinionQueue, MinionWorker } from 'gbrain/minions';
|
||||
const worker = new MinionWorker(engine, { queue: 'default' });
|
||||
worker.register('ea-inbox-sweep', async (ctx) => {
|
||||
// Host-specific agent turn.
|
||||
});
|
||||
// ...register every TODO handler...
|
||||
await worker.start();
|
||||
```
|
||||
|
||||
### (b) Ship the bootstrap in your host repo
|
||||
|
||||
Autopilot already spawns `gbrain jobs work` as a child. Configure it to
|
||||
spawn your custom worker binary (e.g. `your-openclaw-worker`) instead, or
|
||||
register handlers as a side-effect module that the stock worker loads on
|
||||
startup. Either path is documented in `plugin-handlers.md`.
|
||||
|
||||
### (c) Unit tests for the handler body
|
||||
|
||||
Mock the LLM. Assert the happy path, the empty-input path, the
|
||||
network-timeout path, the LLM-failure path. Every branch of the handler's
|
||||
deterministic logic has a unit test.
|
||||
|
||||
### (d) Integration tests
|
||||
|
||||
Submit a real Minion job against your test environment. Assert:
|
||||
- The job completes (not dead-lettered).
|
||||
- The side effects happen (brain pages written, external API called).
|
||||
- The result shape matches what your cron consumers expect.
|
||||
|
||||
### (e) LLM evals
|
||||
|
||||
Whatever prompt the handler runs, add a case in your eval suite. At
|
||||
minimum: one happy case, one edge case, one adversarial case. If the
|
||||
handler's job is "sweep the EA inbox," the eval asserts urgent items
|
||||
surface and routine items don't.
|
||||
|
||||
### (f) Resolver trigger update
|
||||
|
||||
The host's AGENTS.md dispatcher used to reference the skill name
|
||||
(`ea-inbox-sweep`) as an `agentTurn` route. Rewrite that route to
|
||||
`submit_job` with the same trigger phrases — the user-facing language
|
||||
doesn't change; the execution path does.
|
||||
|
||||
### (g) Resolver trigger eval
|
||||
|
||||
Add a test: feed each trigger phrase to the resolver, assert it routes
|
||||
to the new `submit_job` path, not the old `agentTurn`. Without this
|
||||
test, your router can silently keep old behavior and nobody notices
|
||||
until the cron quietly breaks.
|
||||
|
||||
### (h) E2E smoke
|
||||
|
||||
One test that exercises the full pipeline: scheduler → `gbrain jobs
|
||||
submit` → worker claim → handler body → side effect. Gated by your
|
||||
host's E2E DB fixture.
|
||||
|
||||
### (i) Brain filing
|
||||
|
||||
If the handler writes brain pages, `brain/RESOLVER.md` needs an entry
|
||||
for the directory. Orphaned brain pages are worse than no brain pages —
|
||||
nobody can find them on read.
|
||||
|
||||
### (j) check-resolvable
|
||||
|
||||
Run `gbrain check-resolvable`. It validates reachability (is the skill
|
||||
mentioned from RESOLVER.md?), MECE overlap, DRY, gap detection. If it
|
||||
fails, fix the skill or extend an existing one instead of creating a
|
||||
duplicate.
|
||||
|
||||
### Finalize
|
||||
|
||||
When all ten gates are green for a handler:
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
The orchestrator detects the newly-registerable handler, rewrites the
|
||||
cron entry from `agentTurn ea-inbox-sweep` to `gbrain jobs submit ...`
|
||||
with the right engine-aware form (PGLite uses `--follow`; Postgres uses
|
||||
fire-and-forget + `--idempotency-key`), and marks the JSONL row
|
||||
`status: "complete"`.
|
||||
|
||||
**Iron rule:** `scripts/skillify-check.ts <handler-code-path>` must
|
||||
return 10/10 before you mark a handler migrated. No partial-credit
|
||||
"we'll add tests later." See `skills/skillify/SKILL.md` for the meta.
|
||||
|
||||
## `type: agents-md-dispatcher-needs-host-review`
|
||||
|
||||
Gbrain injects a marker into each AGENTS.md pointing to
|
||||
`skills/conventions/subagent-routing.md`. But if your dispatcher has
|
||||
inline `sessions_spawn` routing logic (custom branching on tool names
|
||||
or intent patterns), that's not a mechanical rewrite — it involves host
|
||||
judgment about which routes should flip to `submit_job` and which should
|
||||
stay on `sessions_spawn` for real-time tasks.
|
||||
|
||||
Read `skills/conventions/subagent-routing.md`. Walk your dispatcher's
|
||||
`sessions_spawn` blocks. For each one, decide:
|
||||
|
||||
- **Brain-write, multi-step, user will ask later** → route through
|
||||
`submit_job` (Minions).
|
||||
- **Real-time, user waiting, < 30s** → keep on `sessions_spawn`.
|
||||
- **Somewhere in between** → `minion_mode: pain_triggered` is the
|
||||
right default; let the convention's pain signals decide.
|
||||
|
||||
Mark the JSONL row `status: "complete"` by appending a new line with
|
||||
the same shape + updated status. Or: edit the dispatcher, rerun
|
||||
`gbrain apply-migrations`, and let gbrain re-detect + update
|
||||
automatically (it dedupes on file path, so no duplicate rows).
|
||||
|
||||
## When every TODO clears
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --list
|
||||
```
|
||||
|
||||
Should show v0.11.0 as `applied`. Your host is fully migrated to
|
||||
Minions. Autopilot is dispatching through the queue, cron jobs are
|
||||
durable, and every handler you registered appears in
|
||||
`worker.registeredNames` on next worker startup.
|
||||
|
||||
## Related
|
||||
|
||||
- `skills/migrations/index.ts` (code, not file) — the TS registry the
|
||||
runtime consults; `apply-migrations --list` reads from here, not
|
||||
from this markdown file.
|
||||
- `skills/conventions/cron-via-minions.md` — the rewrite pattern
|
||||
gbrain applies for builtins + the one your host ships for custom
|
||||
handlers.
|
||||
- `skills/conventions/subagent-routing.md` — runtime dispatcher rules
|
||||
(native subagent vs Minion) that AGENTS.md should reference.
|
||||
- `docs/guides/plugin-handlers.md` — the plugin contract for host
|
||||
handler registration.
|
||||
- `skills/skillify/SKILL.md` — the 10-item checklist every handler
|
||||
must pass before its JSONL row clears.
|
||||
- `scripts/skillify-check.ts` — machine-readable version of the
|
||||
checklist; use with `--json` in CI.
|
||||
- `docs/guides/minions-fix.md` — user-facing troubleshooting for
|
||||
broken-v0.11.0 installs that never ran the migration at all.
|
||||
@@ -1,132 +0,0 @@
|
||||
---
|
||||
version: 0.12.0
|
||||
feature_pitch:
|
||||
headline: "Knowledge Graph wires itself — every page write extracts typed links automatically"
|
||||
description: |
|
||||
Every gbrain put_page now extracts entity references and creates typed links
|
||||
(attended, works_at, invested_in, founded, advises) with zero LLM calls.
|
||||
Hybrid search. Self-wiring graph. Backlink-boosted ranking. Ask "who works
|
||||
at Acme?" or "what did Bob invest in?" — answers vector search alone can't reach.
|
||||
Benchmarked end-to-end on a 240-page rich-prose corpus: Recall@5 83% → 95%,
|
||||
Precision@5 39% → 45%, +30 more correct answers in the agent's top-5.
|
||||
Graph-only F1: 86.6% vs grep's 57.8% (+28.8 pts).
|
||||
recipe: null
|
||||
---
|
||||
|
||||
# v0.12.0 Migration: Knowledge Graph Auto-Wire
|
||||
|
||||
This release ships the v0.12.0 graph layer (originally tracked as PR #188 v0.10.3,
|
||||
merged on top of v0.11.1 Minions). The migration is **automatic** — `gbrain
|
||||
post-upgrade` calls `gbrain apply-migrations --yes` which invokes the v0.12.0
|
||||
orchestrator at `src/commands/migrations/v0_12_0.ts`. You normally don't need to
|
||||
do anything; this doc is the reference for what happens under the hood.
|
||||
|
||||
## What ships
|
||||
|
||||
- **Auto-link on every page write.** `put_page` extracts entity references from
|
||||
content and creates typed links (`attended`, `works_at`, `invested_in`,
|
||||
`founded`, `advises`, `mentions`) with deterministic regex inference. Zero LLM
|
||||
calls. Stale links reconciled on edits.
|
||||
- **Schema migrations v8/v9/v10**: multi-type link constraint, timeline dedup
|
||||
index, drop legacy timeline search trigger.
|
||||
- **`gbrain extract --source db`** for batch backfill on existing brains.
|
||||
- **`gbrain graph-query <slug>`** for typed-edge relationship traversal with
|
||||
cycle prevention.
|
||||
- **Backlink-boosted hybrid search**: well-connected entities rank higher
|
||||
(`score *= 1 + 0.05 * log(1 + n)`).
|
||||
- **Graph health metrics in `gbrain health`**: `link_coverage`,
|
||||
`timeline_coverage`, `most_connected`.
|
||||
|
||||
## What the orchestrator does (automatic, idempotent)
|
||||
|
||||
The v0_12_0 migration runs these phases in order. All are idempotent — safe to
|
||||
re-run. Failure in any one phase records `partial` status; re-running picks up
|
||||
where it left off.
|
||||
|
||||
### Phase A — Schema
|
||||
```bash
|
||||
gbrain init --migrate-only
|
||||
```
|
||||
Applies migrations v8/v9/v10 if not already applied. Idempotent.
|
||||
|
||||
### Phase B — Config check
|
||||
Reads `auto_link` config. If user set it to `false`, skips the backfill phases
|
||||
(don't override user intent). Default is enabled; unset = enabled.
|
||||
|
||||
### Phase C — Backfill links
|
||||
```bash
|
||||
gbrain extract links --source db
|
||||
```
|
||||
Walks every page from the engine (mutation-immune snapshot iteration), extracts
|
||||
entity refs from content, creates typed links. Idempotent via the
|
||||
`(from_page_id, to_page_id, link_type)` UNIQUE constraint.
|
||||
|
||||
### Phase D — Backfill timeline
|
||||
```bash
|
||||
gbrain extract timeline --source db
|
||||
```
|
||||
Parses dated bullet entries from page content. Idempotent via the
|
||||
`(page_id, date, summary)` UNIQUE index.
|
||||
|
||||
### Phase E — Verify
|
||||
```bash
|
||||
gbrain stats
|
||||
```
|
||||
Confirms `link_count` and `timeline_entry_count`. Expected outcomes:
|
||||
- **Empty brain (0 pages)**: success, message "auto-link will wire entities as you write pages"
|
||||
- **Pages but 0 links**: success, message "no entity refs in content" (the brain
|
||||
works fine; just no extractable references)
|
||||
- **Pages and links**: success, message "Graph layer wired up"
|
||||
- **`auto_link` disabled**: success, message "auto_link_disabled_by_user"
|
||||
|
||||
### Phase F — Record
|
||||
Appends to `~/.gbrain/migrations/completed.jsonl` so future `gbrain
|
||||
apply-migrations` runs know this version is done.
|
||||
|
||||
## Manual recovery (if you ever need it)
|
||||
|
||||
If the orchestrator fails or you want to re-wire a brain manually:
|
||||
|
||||
```bash
|
||||
gbrain init --migrate-only # Phase A
|
||||
gbrain extract links --source db # Phase C
|
||||
gbrain extract timeline --source db # Phase D
|
||||
gbrain stats # Phase E (look for link_count > 0)
|
||||
gbrain graph-query <some-person-slug> --depth 2 # smoke test
|
||||
```
|
||||
|
||||
## Available link types
|
||||
|
||||
After backfill, you can query the graph with:
|
||||
|
||||
```bash
|
||||
gbrain graph-query people/<slug> --type attended --depth 2
|
||||
gbrain graph-query companies/<slug> --type works_at --direction in
|
||||
```
|
||||
|
||||
Types in this version: `attended`, `works_at`, `invested_in`, `founded`,
|
||||
`advises`, `source`, `mentions`.
|
||||
|
||||
## Disable auto-link
|
||||
|
||||
If you don't want auto-link populating the graph on every write:
|
||||
|
||||
```bash
|
||||
gbrain config set auto_link false
|
||||
```
|
||||
|
||||
Re-enable with `gbrain config set auto_link true`.
|
||||
|
||||
## Branch-install recovery (very rare)
|
||||
|
||||
If you ran the `garrytan/link-timeline-extract` branch BEFORE this merge (so
|
||||
your local PGLite db has migration v7 = `drop_timeline_search_trigger` from
|
||||
the pre-renumber world), you're missing master's v5/v6/v7 (`minion_jobs_table`
|
||||
etc.). Recovery: drop your PGLite db and re-init.
|
||||
|
||||
```bash
|
||||
rm ~/.gbrain/brain.pglite
|
||||
gbrain init --pglite
|
||||
```
|
||||
|
||||
This applies all v2-v10 cleanly.
|
||||
@@ -1,107 +0,0 @@
|
||||
# v0.12.1 Migration: Extract Performance + Migration Timeout Fix
|
||||
|
||||
This release is a pure performance bug fix. **No manual steps are needed for most
|
||||
users** — re-run `gbrain init --migrate-only` and re-run `gbrain extract all` if
|
||||
desired. Both should now complete successfully on any brain size.
|
||||
|
||||
## What changed
|
||||
|
||||
Two production-blocking bugs are fixed:
|
||||
|
||||
1. **`gbrain extract` no longer hangs on large brains.** The N+1 dedup pre-load
|
||||
that ran 47K serial `getLinks()` calls before any work started is gone. Both
|
||||
engines already enforced uniqueness at the SQL layer; the in-memory dedup was
|
||||
redundant. Combined with new batched 100-row INSERTs, a full re-extract on a
|
||||
47K-page brain drops from "10+ min hang then ~minutes more" to "immediate
|
||||
work, ~30-60s total."
|
||||
|
||||
2. **v0.12.0 schema migration no longer times out on duplicate-heavy brains.**
|
||||
Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create
|
||||
a btree helper index before the `DELETE ... USING` self-join, then drop it
|
||||
after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the
|
||||
migration completes in under a second instead of timing out at 60 seconds.
|
||||
|
||||
## What you need to do
|
||||
|
||||
### If your v0.12.0 upgrade succeeded — you're already done
|
||||
|
||||
The migration is idempotent. The fix only matters if your migration FAILED on
|
||||
the v0.12.0 upgrade attempt. Run `gbrain init --migrate-only` once to confirm
|
||||
your schema is at version 10, then run `gbrain extract all --dir <brain>` if
|
||||
you want to reuse it now that it's fast.
|
||||
|
||||
### If your v0.12.0 upgrade FAILED on `idx_timeline_dedup` creation
|
||||
|
||||
You may have the brain in a partial-migration state with duplicate rows in
|
||||
`timeline_entries` (or `links`). Run:
|
||||
|
||||
```bash
|
||||
gbrain init --migrate-only
|
||||
```
|
||||
|
||||
Migration v9 will pre-create the helper index, dedup any existing duplicates
|
||||
(now sub-second instead of timing out), drop the helper, and create the unique
|
||||
index. Idempotent — safe to re-run.
|
||||
|
||||
If you previously ran the manual `CREATE TABLE _clean AS SELECT DISTINCT ON
|
||||
... + table swap` workaround Garry posted, your schema should already be at
|
||||
version 10. Confirm with:
|
||||
|
||||
```bash
|
||||
gbrain config get version
|
||||
```
|
||||
|
||||
If it shows `10`, you're done.
|
||||
|
||||
### If you previously did manual SQL surgery on duplicate timeline rows
|
||||
|
||||
The unique index `idx_timeline_dedup` should now be present after the workaround.
|
||||
Re-running `gbrain init --migrate-only` is a no-op for v9 (uses
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS`). The new migration code adds the helper
|
||||
btree on the dedup columns first — but the helper is dropped at the end of the
|
||||
migration, so even if v9 re-ran (it won't, version is already at 10), it would
|
||||
leave your schema unchanged.
|
||||
|
||||
### Re-running `gbrain extract` on a previously-stuck brain
|
||||
|
||||
This is the most common case. Run:
|
||||
|
||||
```bash
|
||||
gbrain extract all --dir <brain-dir>
|
||||
# or for live brains with no local checkout:
|
||||
gbrain extract all --source db
|
||||
```
|
||||
|
||||
Expect immediate output (`Links: created N from M pages` lines streaming as files
|
||||
process), not a 10-minute hang. On a re-run of a fully-extracted brain you should
|
||||
see `Done: 0 links, 0 timeline entries from N pages` — that's the truthful counter
|
||||
at work, confirming nothing changed.
|
||||
|
||||
## New engine API (informational, optional)
|
||||
|
||||
For plugin authors building integrations on the `BrainEngine` interface, two
|
||||
new methods are available:
|
||||
|
||||
- `addLinksBatch(LinkBatchInput[]) → Promise<number>`
|
||||
- `addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>`
|
||||
|
||||
Both return the count of rows actually inserted (excluding ON CONFLICT no-ops
|
||||
and JOIN-dropped rows whose slugs don't exist). Existing per-row `addLink` /
|
||||
`addTimelineEntry` are unchanged — no migration required for plugin code.
|
||||
|
||||
## Verification
|
||||
|
||||
After the migration completes:
|
||||
|
||||
```bash
|
||||
# Confirm schema version
|
||||
gbrain config get version
|
||||
# Expect: 10
|
||||
|
||||
# Confirm the unique indexes exist (Postgres / Supabase only)
|
||||
gbrain stats
|
||||
# Expect: link_count and timeline_entry_count both populated, no duplicates
|
||||
```
|
||||
|
||||
If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the
|
||||
output of `gbrain init --migrate-only` and `gbrain config get version`.
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
name: v0.13.0
|
||||
version: 0.13.0
|
||||
headline: YAML frontmatter now creates typed graph edges automatically
|
||||
---
|
||||
|
||||
# v0.13.0 Migration: Frontmatter Relationship Indexing
|
||||
|
||||
**TL;DR:** this release teaches the knowledge graph to read your YAML frontmatter. Every `company:`, `investors:`, `attendees:`, `key_people:`, `partner:`, `lead:`, and `related:` field you already wrote now surfaces as a typed graph edge. `gbrain graph <hub-entity> --depth 2` goes from returning ~7 nodes to 50+ on a real brain without you changing a word of content.
|
||||
|
||||
For most users: run `gbrain upgrade` and you're done. The orchestrator handles schema + backfill in 2-5 minutes on a 46K-page brain. You immediately see richer results from `gbrain graph` queries.
|
||||
|
||||
## What changed
|
||||
|
||||
**Before v0.13:** graph edges came only from `[Name](path)` markdown refs. Your frontmatter was indexed for search but did not create graph relationships.
|
||||
|
||||
**After v0.13:**
|
||||
- YAML frontmatter fields project into the `links` table with inferred types.
|
||||
- Direction respects the subject-of-verb: `people/alice --attended--> meetings/2026-04-03` reads naturally because the person is the subject.
|
||||
- `link_source` column distinguishes `markdown` from `frontmatter` from `manual` edges. Reconciliation on `put_page` only touches edges this page's frontmatter created — never other pages' edges.
|
||||
- `origin_page_id` provenance tracks WHICH page's frontmatter authored each edge, so multi-page overlap stays safe.
|
||||
|
||||
## The field → type map
|
||||
|
||||
| Frontmatter field | On page type | Edge type | Direction |
|
||||
|-------------------|--------------|-----------|-----------|
|
||||
| `company`, `companies` | person | `works_at` | person → company |
|
||||
| `founded` | person | `founded` | person → company |
|
||||
| `key_people` | company | `works_at` | person → company (incoming) |
|
||||
| `partner` | company | `yc_partner` | person → company (incoming) |
|
||||
| `investors` | deal, company | `invested_in` | investor → target (incoming) |
|
||||
| `lead` | deal | `led_round` | lead → deal (incoming) |
|
||||
| `attendees` | meeting | `attended` | person → meeting (incoming) |
|
||||
| `sources` | any | `discussed_in` | source → page (incoming) |
|
||||
| `source` | any | `source` | page → source (outgoing) |
|
||||
| `related`, `see_also` | any | `related_to` | page → target (outgoing) |
|
||||
|
||||
Fields on pages not matching the `On page type` column are ignored for that mapping. E.g. a person page with `key_people:` is ignored (makes no sense); only company pages produce `works_at` incoming from `key_people`.
|
||||
|
||||
## How to upgrade
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
That runs the v0.13.0 orchestrator:
|
||||
|
||||
1. **Schema phase** — ALTER TABLE adds `link_source`, `origin_page_id`, `origin_field`. Swaps unique constraint to include them. ~10s.
|
||||
2. **Backfill phase** — walks every page, extracts frontmatter edges via the batch-mode resolver (pg_trgm fuzzy match, zero LLM calls, zero API costs). Progress prints every 500 pages. 2-5 min on a 46K-page brain.
|
||||
3. **Verify phase** — asserts the backfill produced rows + records completion.
|
||||
|
||||
The migration is resumable. If it dies mid-backfill (OOM, network blip), re-run `gbrain upgrade` and it picks up where it left off via `ON CONFLICT DO NOTHING` on the new unique constraint.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# Link count should reflect the ~15-20K new frontmatter edges on a typical brain.
|
||||
gbrain stats
|
||||
|
||||
# Sample a hub entity from your brain — depth-2 should return many more nodes than before.
|
||||
gbrain graph <hub-entity-slug> --depth 2
|
||||
|
||||
# Filter to specific edge types (new in v0.13).
|
||||
gbrain graph <hub-entity-slug> --depth 2 --type yc_partner,invested_in
|
||||
|
||||
# Count edges by provenance.
|
||||
gbrain call get_stats --json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Migration failed mid-backfill.** Re-run `gbrain upgrade`. Resumable via ON CONFLICT DO NOTHING + origin_page_id scoping.
|
||||
|
||||
**PGLite without pg_trgm GIN index.** The migration logs an INFO line and falls back to ILIKE matching. Fuzzy-match quality reduced, but migration completes successfully. No user action required.
|
||||
|
||||
**Unresolvable names in the extract summary.** The backfill prints a top-20 preview of frontmatter names that didn't resolve to any page. These are usually people/companies you've mentioned in frontmatter but never created pages for. Options:
|
||||
- Create the missing pages (then run `gbrain extract links --source db --include-frontmatter` to backfill).
|
||||
- Ignore — unresolved names stay unresolved until you create a page for them.
|
||||
|
||||
**Agents using `put_page` see a new `unresolved` field in `auto_links`.** This is additive. Existing agents that ignore unknown response fields keep working. Agents that want to escalate unresolved names: read `response.auto_links.unresolved`.
|
||||
|
||||
**`attendee` vs `attended` type normalization.** Legacy rows with `link_type='attendee'` or `link_type='mention'` keep working. Normalization to the v0.13 canonical names (`attended`, `mentions`) is deliberately NOT in this migration — it's a separate semantic concern. The `gbrain normalize-types` command (v0.14) handles it opt-in.
|
||||
|
||||
## For downstream agent skill forks
|
||||
|
||||
If you maintain a fork of GBrain skills (a custom OpenClaw deployment or agent-fork), check `docs/UPGRADING_DOWNSTREAM_AGENTS.md` for the v0.13 section. Verdict: **no action required for most skills.** Three skills (`meeting-ingestion`, `enrich`, `idea-ingest`) get a new optional phase if you want to consume the `auto_links.unresolved` field.
|
||||
|
||||
## If something goes wrong
|
||||
|
||||
1. `gbrain doctor` — surfaces any partial migrations and any post-upgrade failures recorded in `~/.gbrain/upgrade-errors.jsonl`.
|
||||
2. Paste the recovery hint doctor prints.
|
||||
3. If that fails too, file an issue: https://github.com/garrytan/gbrain/issues with doctor output + upgrade-errors.jsonl contents. This is how the gbrain maintainers find fragile upgrade paths.
|
||||
@@ -1,185 +0,0 @@
|
||||
---
|
||||
version: 0.14.0
|
||||
feature_pitch:
|
||||
headline: "Move deterministic crons off the LLM gateway. Zero tokens per fire."
|
||||
description: |
|
||||
OpenClaw operators: your gateway pins at 100% CPU because most cron jobs
|
||||
boot a full Opus session to do deterministic work (API fetch, token
|
||||
refresh, scrape + markdown write) that doesn't need reasoning. v0.14.0
|
||||
adds a `shell` job type to Minions so those jobs run as child processes
|
||||
under the existing Minions worker. ~60% gateway CPU reduction at typical
|
||||
scale. Retry, backoff, DLQ, unified `gbrain jobs list` visibility, all
|
||||
free. The LLM-reasoning crons stay on the gateway where they belong.
|
||||
recipe: docs/guides/minions-shell-jobs.md
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.14.0 Migration: Adopt the `shell` job type
|
||||
|
||||
**Audience: host agents (OpenClaw deployments, Hermes operators, future
|
||||
hosts) reading this AFTER `gbrain apply-migrations` has run. There is no
|
||||
mechanical orchestrator for this release — every change described here
|
||||
is host-specific and requires operator judgment per cron.** Shell jobs
|
||||
are off by default on upgrade (nothing breaks). This file walks the
|
||||
host agent through enabling and adopting them.
|
||||
|
||||
## Iron rules
|
||||
|
||||
1. **Never auto-rewrite the operator's crontab.** Crontab entries are
|
||||
host-specific code per the CLAUDE.md "host-specific code" exception.
|
||||
Every rewrite is an explicit human approval, per cron, with a diff.
|
||||
2. **LLM-requiring crons stay on the gateway.** If the cron's work
|
||||
includes reasoning (sentiment classification, triage, synthesis,
|
||||
prose generation), it belongs on the gateway. Moving it to a shell
|
||||
job throws away the LLM session the cron needs.
|
||||
3. **Deterministic crons are the only candidates.** API fetch, token
|
||||
refresh, database read + markdown write, HTTP refresh call, scrape.
|
||||
If the script is `node scripts/x.mjs` or `curl ... | jq` or similar
|
||||
and does zero LLM work, it's a candidate.
|
||||
|
||||
## Step 1: Enable the worker
|
||||
|
||||
Pick the engine the operator is on:
|
||||
|
||||
**Postgres** (most OpenClaw/Hermes deployments):
|
||||
|
||||
```bash
|
||||
# In the worker bootstrap, export the env flag and run the daemon:
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work
|
||||
```
|
||||
|
||||
The worker claims shell jobs from the queue and executes them. Retries,
|
||||
backoff, and dead-letter all work the same as sync/embed jobs.
|
||||
|
||||
**PGLite**: no persistent worker, per-tick inline execution only:
|
||||
|
||||
```bash
|
||||
# Every crontab invocation must use --follow; PGLite's worker daemon
|
||||
# exits immediately due to exclusive file lock.
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
|
||||
--params '{"cmd":"...","cwd":"..."}' --follow
|
||||
```
|
||||
|
||||
## Step 2: Audit the operator's cron manifest
|
||||
|
||||
Read the operator's cron manifest. Typical locations:
|
||||
|
||||
- `~/.claude/cron/jobs.json` (OpenClaw)
|
||||
- `scripts/service-manager.sh` in the host repo
|
||||
- System crontab (`crontab -l`)
|
||||
|
||||
For each entry, classify:
|
||||
|
||||
| Pattern | Class | Action |
|
||||
|---------|-------|--------|
|
||||
| `agentTurn <skill>` or any OpenClaw-dispatched LLM skill | LLM-requiring | **Leave as-is.** Needs gateway. |
|
||||
| `node scripts/*.mjs` that hits an API and writes markdown | Deterministic | Propose shell-job rewrite. |
|
||||
| Token refresh (`ycli token-refresh`, `x-oauth2-refresh`) | Deterministic | Propose shell-job rewrite. |
|
||||
| Scrape + write (`frameio-scan`, `flight-tracker`) | Deterministic | Propose shell-job rewrite. |
|
||||
| Audio transcription or any LLM-dependent extract | LLM-requiring | Leave as-is. |
|
||||
| `bash` wrapper scripts that may call LLM tools internally | Ambiguous | Ask the operator. Don't assume. |
|
||||
|
||||
## Step 3: Propose rewrites per cron
|
||||
|
||||
For each deterministic cron, propose the exact rewrite with a diff. Show
|
||||
the operator both sides. Let them approve per-cron, not in bulk.
|
||||
|
||||
**Before** (LLM gateway):
|
||||
```
|
||||
OpenClaw cron: x-garrytan-unified, 3 13,16,19,22,1,4,7,10 * * *
|
||||
→ runs agentTurn x-garrytan-unified
|
||||
→ boots Opus context, invokes script, returns
|
||||
```
|
||||
|
||||
**After** (Minions worker):
|
||||
```cron
|
||||
3 13,16,19,22,1,4,7,10 * * * \
|
||||
gbrain jobs submit shell \
|
||||
--params '{"cmd":"node /data/.openclaw/workspace/scripts/x-garrytan-daily.mjs","cwd":"/data/.openclaw/workspace"}' \
|
||||
--max-attempts 3 --timeout-ms 300000
|
||||
```
|
||||
|
||||
Rewrite rules:
|
||||
- `cwd` is required and must be an absolute path. Operator picks it. It
|
||||
should be the directory the script expects to run in (the host repo
|
||||
root, typically).
|
||||
- `--max-attempts 3` matches the default Minions retry policy. Override
|
||||
if the script is non-idempotent and should only run once per fire.
|
||||
- `--timeout-ms N` caps the child's wall-clock runtime. Set to the 95th
|
||||
percentile of the script's observed runtime, plus slack. Examples:
|
||||
token refresh → 30s; API fetch → 300s; scrape → 600s.
|
||||
- **PGLite operators:** add `--follow` to every line. Skip Step 1.
|
||||
|
||||
## Step 4: Secrets that the script needs
|
||||
|
||||
Shell jobs receive a minimal env allowlist by default: `PATH, HOME,
|
||||
USER, LANG, TZ, NODE_ENV`. They do NOT inherit `OPENAI_API_KEY`,
|
||||
`ANTHROPIC_API_KEY`, `DATABASE_URL`, or any other worker env vars.
|
||||
|
||||
If a cron's script needs an API key, name it explicitly:
|
||||
|
||||
```bash
|
||||
gbrain jobs submit shell \
|
||||
--params '{"cmd":"node scripts/yc-sync.mjs","cwd":"/data/.openclaw/workspace","env":{"YC_API_TOKEN":"'"$YC_API_TOKEN"'"}}'
|
||||
```
|
||||
|
||||
The shell expands `$YC_API_TOKEN` at submit time. The worker receives
|
||||
the JSON with the literal token value. Audit log does not log env
|
||||
values (keys don't carry sensitive data; values never appear).
|
||||
|
||||
## Step 5: Verify the first migrated cron
|
||||
|
||||
After rewriting ONE cron with the operator's approval:
|
||||
|
||||
1. Wait for the next scheduled fire (or trigger manually: `gbrain jobs
|
||||
submit shell --params '...' --follow`).
|
||||
2. Check `gbrain jobs list --status completed --name shell --limit 5`
|
||||
for the result.
|
||||
3. `gbrain jobs get <id>` shows `exit_code`, `stdout_tail`, `stderr_tail`,
|
||||
`duration_ms`.
|
||||
4. Compare against the pre-migration behavior: did it do the same work?
|
||||
Same output files changed? Same side effects?
|
||||
|
||||
Only after one cron is verified working end-to-end should the operator
|
||||
approve the next batch.
|
||||
|
||||
## Step 6: Starvation sanity check
|
||||
|
||||
If the operator submits shell jobs but forgot to set
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` on the worker, jobs sit in `waiting`
|
||||
indefinitely. The CLI warns on submission, but for daemon-style
|
||||
deployments the warning scrolls past. Add this to the operator's
|
||||
ops-check runbook:
|
||||
|
||||
```bash
|
||||
gbrain jobs list --status waiting --name shell
|
||||
```
|
||||
|
||||
If rows pile up here, either (a) no worker has the env flag set, or
|
||||
(b) the worker crashed. Fix by restarting with the flag.
|
||||
|
||||
## Non-goals (explicitly deferred to later releases)
|
||||
|
||||
- **Automatic crontab rewrites.** Deferred to a future `gbrain
|
||||
crontab-to-minions <file>` helper. P1 in TODOS.md.
|
||||
- **DB-backed scheduler.** `minion_schedules` table replaces host
|
||||
crontab entirely. P1 in TODOS.md.
|
||||
- **Orphaned-shell-job stats.** `gbrain jobs stats --orphaned` would
|
||||
surface the "no worker with env flag" case. P2 in TODOS.md.
|
||||
- **Configurable buffer sizes.** Output tails are fixed at 64KB stdout
|
||||
/ 16KB stderr. P2 in TODOS.md.
|
||||
|
||||
## When to stop
|
||||
|
||||
The migration is done when:
|
||||
|
||||
1. The worker runs with `GBRAIN_ALLOW_SHELL_JOBS=1` (Postgres) or every
|
||||
cron uses `--follow` (PGLite).
|
||||
2. Every deterministic cron the operator approved has been rewritten.
|
||||
3. The operator has verified at least one full cron fire cycle
|
||||
end-to-end and confirmed the output matches pre-migration.
|
||||
4. `gbrain jobs stats` shows shell jobs completing at expected rates
|
||||
with few or zero retries.
|
||||
|
||||
Gateway CPU should visibly drop after the first few rewrites. That's
|
||||
the signal the adoption is working.
|
||||
@@ -1,164 +0,0 @@
|
||||
---
|
||||
version: 0.15.2
|
||||
feature_pitch:
|
||||
headline: "Silent binaries are dead. Every bulk action now heartbeats."
|
||||
description: |
|
||||
`gbrain doctor` on a 52K-page brain used to sit silent for 10+
|
||||
minutes before an agent timeout killed it. Same pattern on embed,
|
||||
sync, import, extract, migrate, and every orchestrator. v0.15.2
|
||||
routes 14 bulk commands through one shared reporter that writes
|
||||
to stderr. Non-TTY default is plain human lines; agents that
|
||||
want structured events add `--progress-json` and get one JSON
|
||||
object per line. Stdout stays clean for data output. Event
|
||||
schema is locked in docs/progress-events.md.
|
||||
recipe: docs/progress-events.md
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.15.2 Migration: Bulk-action progress streaming
|
||||
|
||||
**Audience: host agents reading this after `gbrain apply-migrations`
|
||||
has run. v0.15.2 is purely additive to the CLI surface, there is no
|
||||
schema change, no data rewrite, and no orchestrator for this release.**
|
||||
Your binaries just got observable. This file tells you how to use it.
|
||||
|
||||
## Mechanical migration: nothing
|
||||
|
||||
There is no mechanical step. If `gbrain upgrade` completed, progress
|
||||
events are already flowing the next time you invoke a bulk command.
|
||||
Read on to know what's there and how to consume it.
|
||||
|
||||
## What's new at the CLI
|
||||
|
||||
### Three new global flags
|
||||
|
||||
These work on any `gbrain` subcommand:
|
||||
|
||||
- `--progress-json` — emit one JSON event per line on stderr.
|
||||
- `--quiet` — suppress progress output entirely.
|
||||
- `--progress-interval=<ms>` — minimum ms between progress emits
|
||||
(default 1000).
|
||||
|
||||
Parsed before command dispatch, so both work:
|
||||
|
||||
```
|
||||
gbrain --progress-json doctor --json
|
||||
gbrain doctor --json --progress-json
|
||||
```
|
||||
|
||||
### Per-TTY behavior
|
||||
|
||||
Without `--progress-json`:
|
||||
|
||||
- **TTY:** `\r`-rewriting single-line progress on stderr (fancy).
|
||||
- **Non-TTY (pipe, CI, agent):** one plain-text line per event on
|
||||
stderr. No JSON, no noise. Human-readable.
|
||||
|
||||
The default was deliberately NOT JSON-on-non-TTY. Shell pipelines
|
||||
that just pipe `gbrain ... | less` should get readable logs, not a
|
||||
JSON blob. Agents opt in to JSON explicitly.
|
||||
|
||||
## What's new per command
|
||||
|
||||
Fourteen commands now stream progress through the shared reporter:
|
||||
|
||||
| Command | What you'll see |
|
||||
|---------|-----------------|
|
||||
| `doctor` | `doctor.db_checks` phase + per-check heartbeats, including a 1s heartbeat while `markdown_body_completeness` scans |
|
||||
| `orphans` | `orphans.scan` heartbeat while the anti-join runs |
|
||||
| `embed` | `embed.pages` with per-page ticks |
|
||||
| `files sync` | `files.sync` with per-file ticks |
|
||||
| `export` | `export.pages` with per-page ticks |
|
||||
| `import` | `import.files` with per-file ticks (replaces per-100 stdout logs) |
|
||||
| `extract [links|timeline|all]` (fs + db) | `extract.links_fs` / `extract.timeline_db` etc. |
|
||||
| `sync` | `sync.deletes`, `sync.renames`, `sync.imports` phases |
|
||||
| `migrate --to ...` | `migrate.copy_pages`, `migrate.copy_links` |
|
||||
| `repair-jsonb` | `repair_jsonb.run` + per-column heartbeats |
|
||||
| `check-backlinks` | `backlinks.scan` heartbeat |
|
||||
| `lint` | `lint.pages` per-page ticks |
|
||||
| `integrity auto` | `integrity.auto` per-page ticks |
|
||||
| `eval` | `eval.single` / `eval.ab` per-query ticks |
|
||||
| `apply-migrations` (v0_11/v0_12_0/v0_12_2) | Child processes inherit the parent's progress mode |
|
||||
|
||||
## JSON event schema
|
||||
|
||||
Documented in `docs/progress-events.md` (canonical reference). Stable
|
||||
from v0.15.2, additive changes only.
|
||||
|
||||
Quick agent cheat sheet:
|
||||
|
||||
```json
|
||||
{"event":"start","phase":"doctor.db_checks","ts":"..."}
|
||||
{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"eta_ms":10300,"ts":"..."}
|
||||
{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation...","elapsed_ms":1000,"ts":"..."}
|
||||
{"event":"finish","phase":"doctor.db_checks","elapsed_ms":187000,"ts":"..."}
|
||||
{"event":"abort","phase":"orphans.scan","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}
|
||||
```
|
||||
|
||||
Parser rules:
|
||||
|
||||
1. One JSON object per line on stderr.
|
||||
2. Ignore unknown event types and unknown fields. Schema is additive.
|
||||
3. Group by `phase` prefix to track one run: all `doctor.*` events
|
||||
belong to the same `doctor` invocation.
|
||||
4. `total` / `pct` / `eta_ms` are absent when the scan doesn't have a
|
||||
total up front (e.g. heartbeat-only paths). Don't assume they exist.
|
||||
|
||||
## Minion jobs
|
||||
|
||||
`gbrain jobs work` (the Minion worker daemon) writes progress to the
|
||||
DB via `job.updateProgress`, not to stderr. Read per-job progress via
|
||||
the `get_job_progress` MCP op or:
|
||||
|
||||
```bash
|
||||
gbrain jobs submit embed
|
||||
# while it runs:
|
||||
gbrain jobs get <id> # .progress updates live as the handler ticks
|
||||
```
|
||||
|
||||
The `embed` Minion handler is wired as of v0.15.2. Other bulk cores
|
||||
(`sync`, `extract`, `backlinks`, `import`, `autopilot-cycle`) have the
|
||||
callback plumbing ready and will follow.
|
||||
|
||||
## Backward-compatibility warnings
|
||||
|
||||
Five commands moved per-page progress from stdout to stderr:
|
||||
|
||||
- `embed` (was `\r`-on-stdout)
|
||||
- `files sync` (was `\r`-on-stdout)
|
||||
- `export` (was `\r`-on-stdout, newly in scope)
|
||||
- `migrate-engine` (was per-50 `console.log` to stdout)
|
||||
- `import` (was per-100 `console.log` to stdout)
|
||||
|
||||
If you have scripts that grep `stdout` for progress strings like
|
||||
`Progress: 1234/52000` or `\r 1234/52000 pages...` — those strings
|
||||
now live on stderr. The final data summaries (`Embedded N chunks
|
||||
across M pages`, `Import complete`, etc.) remain on stdout so the
|
||||
"did it finish" signal is unchanged.
|
||||
|
||||
`integrity auto` still writes `~/.gbrain/integrity-progress.jsonl`,
|
||||
but its role is now "resume marker only" — live progress goes through
|
||||
the reporter. If you depended on tailing that file for real-time
|
||||
progress, switch to the stderr stream.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# Your agent sees structured events; stdout stays JSON-parseable:
|
||||
gbrain --progress-json doctor --json > doctor.json 2> doctor.progress.log
|
||||
wc -l doctor.progress.log # should be non-zero
|
||||
jq . doctor.json # should parse cleanly
|
||||
|
||||
# For a very large brain, watch the heartbeat:
|
||||
gbrain --progress-json doctor 2>&1 >/dev/null | grep '"event"'
|
||||
```
|
||||
|
||||
If you see silence for more than a second or two on a non-trivial
|
||||
command, file an issue with the exact command and the first 100 lines
|
||||
of stderr.
|
||||
|
||||
## That's the whole migration
|
||||
|
||||
No mechanical step. No config change. Agents that parse `stdout` keep
|
||||
working; agents that want progress now have it on a clean stderr
|
||||
channel with a documented schema.
|
||||
@@ -1,167 +0,0 @@
|
||||
---
|
||||
version: 0.17.0
|
||||
feature_pitch:
|
||||
headline: "One brain maintenance cycle, two CLIs. `gbrain dream` delivers the README promise."
|
||||
description: |
|
||||
The README has said "the agent runs while I sleep, the dream cycle
|
||||
scans every conversation, enriches missing entities, fixes broken
|
||||
citations, consolidates memory" for a year. v0.17 makes that real
|
||||
as a first-class command (`gbrain dream`) backed by one shared
|
||||
primitive (`runCycle`). Autopilot users get lint + orphan sweep
|
||||
added to their nightly cycle automatically — no config change.
|
||||
Cron users get a single legible verb: `0 2 * * * gbrain dream`.
|
||||
Both converge on the same phase order (lint → backlinks → sync →
|
||||
extract → embed → orphans) so file fixes land in the DB the same
|
||||
night, not the next.
|
||||
recipe: null
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.17.0 Migration: `gbrain dream` + unified maintenance cycle
|
||||
|
||||
**Audience: agents + humans upgrading from v0.16.x. There is no
|
||||
mechanical migration step required — the schema migration (v16
|
||||
cycle-lock table) and behavior changes all apply automatically on
|
||||
upgrade. This file documents what changed, how to verify it, and
|
||||
the one opt-out users may care about.**
|
||||
|
||||
## What changed
|
||||
|
||||
### New command: `gbrain dream`
|
||||
|
||||
The brand-promise one-liner. Runs one brain maintenance cycle and
|
||||
exits. Designed for cron.
|
||||
|
||||
```
|
||||
gbrain dream # full 6-phase cycle
|
||||
gbrain dream --dry-run --json # preview, agent-readable
|
||||
gbrain dream --phase lint # single-phase (fast, targeted)
|
||||
gbrain dream --pull # git pull before syncing
|
||||
0 2 * * * gbrain dream --json # nightly cron
|
||||
```
|
||||
|
||||
See `gbrain dream --help` for the full flag reference.
|
||||
|
||||
### Autopilot now runs lint + orphan sweep
|
||||
|
||||
`gbrain autopilot --install` users: on upgrade, your daemon's cycle
|
||||
gains two phases it didn't run before:
|
||||
|
||||
- **lint --fix** — auto-fixes LLM artifacts, placeholder dates, bad
|
||||
citations across the brain. Modifies files on disk.
|
||||
- **orphan sweep** — reports (read-only) pages with no inbound
|
||||
wikilinks. Visible in `gbrain jobs list` output for each
|
||||
`autopilot-cycle` job.
|
||||
|
||||
No action required. The new phases run on the daemon's existing
|
||||
interval.
|
||||
|
||||
### Shared primitive: `src/core/cycle.ts`
|
||||
|
||||
Three callers (dream CLI, autopilot inline path, autopilot-cycle
|
||||
Minions handler) now all delegate to `runCycle(engine, opts)`. One
|
||||
source of truth for what happens overnight.
|
||||
|
||||
### Cycle coordination via a DB lock table
|
||||
|
||||
`gbrain_cycle_locks` (new table, migration v16) replaces
|
||||
session-scoped `pg_try_advisory_lock` which the v0.15.4
|
||||
PgBouncer-transaction-pooler fix silently broke. The table has a
|
||||
TTL (30 min), refreshed between phases, so crashed holders
|
||||
auto-release.
|
||||
|
||||
## Verify after upgrade
|
||||
|
||||
```bash
|
||||
# 1. Dream command exists:
|
||||
gbrain dream --help
|
||||
|
||||
# 2. Run a dry cycle (safe, no writes):
|
||||
gbrain dream --dry-run --json
|
||||
|
||||
# 3. If you run autopilot --install:
|
||||
gbrain jobs list --status complete | head -5
|
||||
# Each `autopilot-cycle` entry now has 6 phases in its report,
|
||||
# not 4. Check a recent one with `gbrain jobs get <id>`.
|
||||
|
||||
# 4. Schema migration landed:
|
||||
gbrain doctor # should show no pending migrations
|
||||
```
|
||||
|
||||
Expected `gbrain dream --dry-run` output on a healthy brain:
|
||||
|
||||
```
|
||||
Brain is healthy. 6 phase(s) checked in 1.3s.
|
||||
```
|
||||
|
||||
Or with `--json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "1",
|
||||
"status": "clean",
|
||||
"phases": [...],
|
||||
"totals": { "lint_fixes": 0, "backlinks_added": 0, ... }
|
||||
}
|
||||
```
|
||||
|
||||
## Opt-outs for autopilot-installed users
|
||||
|
||||
If you explicitly do NOT want autopilot's daemon modifying files
|
||||
(lint + backlinks phases write to disk):
|
||||
|
||||
**Option 1: disable those phases in cron-dream but keep autopilot
|
||||
running.** Since dream is separate, you can run just the phases
|
||||
you want from cron without touching autopilot:
|
||||
|
||||
```bash
|
||||
# e.g. only re-embed and orphan-sweep nightly, skip file mutations:
|
||||
0 2 * * * gbrain dream --phase orphans
|
||||
```
|
||||
|
||||
**Option 2: uninstall autopilot and use cron-dream only.**
|
||||
|
||||
```bash
|
||||
gbrain autopilot --uninstall
|
||||
# Then add to your crontab:
|
||||
0 2 * * * gbrain dream --pull
|
||||
```
|
||||
|
||||
**Option 3: accept the default.** The new phases are conservative:
|
||||
lint only fixes known-safe artifacts (em dashes, placeholder dates),
|
||||
never destructive. Back-link fills are additive. If something does
|
||||
go wrong, `gbrain dream --dry-run` always tells you what WOULD
|
||||
change before you run it for real.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"cycle_already_running" in dream output:**
|
||||
Another cycle (probably autopilot's daemon) is holding the lock.
|
||||
Expected behavior — dream skipped to avoid racing the daemon. The
|
||||
daemon's next interval will pick up the work.
|
||||
|
||||
**`gbrain dream --dry-run` reports changes when you expected none:**
|
||||
Check `gbrain doctor` for drift: lint issues, stale embeddings,
|
||||
missing back-links. Dream's dry-run is the honest preview of what
|
||||
autopilot's daemon will do on its next cycle.
|
||||
|
||||
**Minion `autopilot-cycle` jobs failing after upgrade:**
|
||||
Open a GitHub issue with the output of `gbrain jobs get <id>` for a
|
||||
failing job. The new runCycle-backed handler preserves the
|
||||
partial-failure semantic (one phase failing doesn't block future
|
||||
cycles), but specific phases may surface new error classes.
|
||||
|
||||
## What did NOT change
|
||||
|
||||
- `gbrain autopilot --install` machinery (launchd / systemd /
|
||||
crontab generators). Existing installs keep working.
|
||||
- `~/.gbrain/autopilot.lock` daemon-singleton lockfile. Separate
|
||||
concern from the new per-cycle lock.
|
||||
- `gbrain jobs` interface. `gbrain jobs get <id>` now shows a
|
||||
richer report structure (schema_version:"1"), but the surface
|
||||
API is stable.
|
||||
|
||||
---
|
||||
|
||||
*This migration file is informational only. No mechanical step is
|
||||
required — all changes apply automatically on `gbrain upgrade`.*
|
||||
@@ -1,161 +0,0 @@
|
||||
---
|
||||
version: 0.18.0
|
||||
feature_pitch:
|
||||
headline: "Multi-source brains: one DB, many repos. Federated and isolated sources coexist."
|
||||
description: |
|
||||
v0.17.0 introduces sources as a first-class primitive. A single
|
||||
gbrain backend can now hold multiple knowledge repos (wiki, gstack,
|
||||
yc-media, garrys-list, etc.) with clean scoping. Every page, file,
|
||||
and ingest_log row is scoped to a `sources(id)` row. Slugs are
|
||||
unique PER source, not globally — so two sources can both have
|
||||
`topics/ai` and they're different pages.
|
||||
|
||||
Per-source federation controls whether a source participates in
|
||||
unqualified default search. `federated=true` (the default source
|
||||
post-upgrade) joins the cross-source recall pool. `federated=false`
|
||||
is isolation — only searched when explicitly named via `--source`.
|
||||
This supports both "unified knowledge brain" (wiki + gstack, both
|
||||
federated) and "purpose-separated brains" (yc-media + garrys-list,
|
||||
both isolated) at the same time.
|
||||
|
||||
Per-directory default via `.gbrain-source` dotfile walk-up +
|
||||
`GBRAIN_SOURCE` env var. Matches how kubectl / terraform / git
|
||||
scope context. `cd ~/yc-media && gbrain query "X"` just works.
|
||||
recipe: docs/guides/multi-source-brains.md
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.17.0 Migration: Multi-source brains
|
||||
|
||||
**Audience: host agents reading this after `gbrain apply-migrations`
|
||||
has run. v0.17.0 installs a schema primitive for multi-source and
|
||||
exposes a `sources` CLI subcommand. Existing single-source brains
|
||||
keep working unchanged — they live under a seeded `default` source
|
||||
that preserves all prior behavior.**
|
||||
|
||||
## Mechanical migration: automatic, no action required
|
||||
|
||||
`gbrain upgrade` chains to `gbrain apply-migrations --yes`, which
|
||||
runs:
|
||||
|
||||
- **migration v16** — creates the `sources` table, seeds `default`
|
||||
with `{"federated": true}` config, inherits your pre-v0.17
|
||||
`sync.repo_path` and `sync.last_commit` into the default row.
|
||||
- **migration v17** — adds `pages.source_id TEXT NOT NULL DEFAULT
|
||||
'default' REFERENCES sources(id)`. Swaps the global `UNIQUE(slug)`
|
||||
constraint for composite `UNIQUE(source_id, slug)`. Engine
|
||||
upserts simultaneously re-target `ON CONFLICT (source_id, slug)`
|
||||
so the constraint swap and the write path land atomically.
|
||||
|
||||
Both migrations are idempotent. Safe to re-run.
|
||||
|
||||
Later point releases (v0.17.1 and v0.18.0) will layer:
|
||||
- v0.17.1: ACL enforcement via a caller-identity primitive (the
|
||||
JSONB slot for `access_policy` ships now; enforcement waits for
|
||||
identity to be designed).
|
||||
- v0.18.0: Session ingest (`.jsonl` transcripts, raised size cap,
|
||||
session PageType) AND per-source retention/TTL at the same time.
|
||||
|
||||
## What's new for agents
|
||||
|
||||
### `sources` CLI subcommand
|
||||
|
||||
```
|
||||
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
|
||||
gbrain sources list [--json]
|
||||
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
|
||||
gbrain sources rename <id> <new-display-name>
|
||||
gbrain sources default <id>
|
||||
gbrain sources attach <id> # write .gbrain-source in CWD
|
||||
gbrain sources detach # remove .gbrain-source
|
||||
gbrain sources federate <id>
|
||||
gbrain sources unfederate <id>
|
||||
```
|
||||
|
||||
Source id rules: `[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?` — start + end
|
||||
with alnum, optional interior hyphens, max 32 chars. Immutable after
|
||||
creation (rename only changes the display name). Used as the stable
|
||||
citation key in `[source:slug]` references.
|
||||
|
||||
### Per-directory default
|
||||
|
||||
Running `gbrain sources attach gstack` inside `~/.gstack/` writes a
|
||||
`.gbrain-source` file containing the single word `gstack`. Any
|
||||
gbrain command run from that directory (or any subdirectory) auto-
|
||||
selects `gstack` as the default source. `gbrain sources detach`
|
||||
removes the dotfile.
|
||||
|
||||
Resolution priority for the source a command targets:
|
||||
|
||||
1. Explicit `--source <id>` flag.
|
||||
2. `GBRAIN_SOURCE` env var.
|
||||
3. `.gbrain-source` dotfile in CWD or any ancestor.
|
||||
4. Registered source whose `local_path` contains CWD (longest
|
||||
prefix wins — nested `~/gstack` + `~/gstack/plans` resolves to
|
||||
`plans` when deeper).
|
||||
5. Brain-level default set via `gbrain sources default <id>`.
|
||||
6. Literal `default` (backward-compat fallback).
|
||||
|
||||
### Federation semantics
|
||||
|
||||
- `federated=true` (only the `default` source has this out of the
|
||||
box, by migration): appears in unqualified `gbrain search "X"`
|
||||
results.
|
||||
- `federated=false` (new sources default to this): only appears
|
||||
when `--source <id>` is passed.
|
||||
|
||||
Interactive `gbrain sources add` prompts for federation; non-
|
||||
interactive uses `--federated` / `--no-federated`. Flip later with
|
||||
`gbrain sources federate <id>` / `unfederate <id>`.
|
||||
|
||||
### Citation contract (for agents)
|
||||
|
||||
When agents get multi-source search results they MUST cite pages
|
||||
in `[source-id:slug]` form. Example:
|
||||
|
||||
> You told me about the distillation protocol — see
|
||||
> [wiki:topics/ai] and [gstack:plans/multi-repo] for where this
|
||||
> came from.
|
||||
|
||||
Citations are keyed on `sources.id` (immutable), never
|
||||
`sources.name` (mutable display). If a user renames a source via
|
||||
`gbrain sources rename`, existing citations stay valid.
|
||||
|
||||
## What's NOT in v0.17.0 yet
|
||||
|
||||
The following land in later Steps of this release cycle (already
|
||||
on the branch but gated until the matching code ships):
|
||||
|
||||
- `ingest_log.source_id` — lands with Step 5 sync rewrite.
|
||||
- `links.resolution_type` + qualified `[[source:slug]]` wikilink
|
||||
parsing — lands with Step 4 link-extraction rewrite.
|
||||
- `files.page_slug → page_id` FK rewrite + `file_migration_ledger`
|
||||
+ storage object prefixing — lands with Step 7 storage backfill.
|
||||
- Source-aware search dedup — lands with Step 3.
|
||||
- `gbrain sources import-from-github <url>` — deferred to a patch
|
||||
release after the plumbing stabilizes.
|
||||
|
||||
Existing callers continue to work against the `default` source. No
|
||||
agent behavioral change is required; the new capabilities are
|
||||
opt-in via the new `sources` CLI surface.
|
||||
|
||||
## Host-repo actions
|
||||
|
||||
None required. If your host agent manages the brain via the
|
||||
standard `gbrain sync` flow, it continues to target the default
|
||||
source and sees no behavioral change. To start using multi-source:
|
||||
|
||||
```
|
||||
# Register a new source
|
||||
gbrain sources add gstack --path ~/.gstack --no-federated
|
||||
|
||||
# Pin that directory to it so no --source flag is needed
|
||||
cd ~/.gstack
|
||||
gbrain sources attach gstack
|
||||
|
||||
# Ingest
|
||||
gbrain sync --source gstack
|
||||
```
|
||||
|
||||
Or see `docs/guides/multi-source-brains.md` for the full three
|
||||
canonical scenarios (unified, purpose-separated, mixed).
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
version: 0.19.0
|
||||
feature_pitch:
|
||||
headline: "Your code is now first-class in the brain."
|
||||
one_liner: "gbrain code-refs BrainEngine --json returns every usage site in <100ms."
|
||||
user_action_required: true
|
||||
---
|
||||
|
||||
# v0.19.0 — Code Indexing
|
||||
|
||||
This release makes code a first-class citizen in the brain. Tree-sitter parses 29 languages into semantic chunks. `gbrain code-def` and `gbrain code-refs` let agents find symbol definitions and references without grep. Incremental chunking drops daily autopilot embedding cost by ~95%. The chunker is a strict superset of Chonkie's CodeChunker plus a structured header Chonkie lacks.
|
||||
|
||||
## Schema migrations applied automatically
|
||||
|
||||
- **v25 — `pages.page_kind`** — distinguishes markdown vs code pages at the DB level. Existing rows backfill to `'markdown'`. Postgres uses `ADD CONSTRAINT ... NOT VALID` + `VALIDATE CONSTRAINT` so large tables don't block.
|
||||
- **v26 — `content_chunks` code metadata** — adds `language`, `symbol_name`, `symbol_type`, `start_line`, `end_line`. All nullable. Partial indexes on `symbol_name` and `language` for fast symbol lookup.
|
||||
|
||||
These run as part of `gbrain upgrade` → `gbrain apply-migrations`. No manual DDL needed.
|
||||
|
||||
## What the agent should do after upgrading
|
||||
|
||||
1. **Confirm migrations landed:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for `schema_version: 26`. If lower, run `gbrain apply-migrations --yes`.
|
||||
|
||||
2. **Register a code source** if the user wants their code indexed:
|
||||
```bash
|
||||
gbrain sources add <id> --path <path-to-repo>
|
||||
```
|
||||
Pick a short `<id>` (e.g. `wiki`, `gbrain`, `yc-media`). Shorter is better — it's used in citation keys.
|
||||
|
||||
3. **Sync the code source:**
|
||||
```bash
|
||||
gbrain sync --source <id>
|
||||
```
|
||||
First sync may run tens of minutes depending on repo size. Each TypeScript function becomes a chunk with a structured header like `[TypeScript] src/core/sync.ts:380-415 function performFullSync`.
|
||||
|
||||
4. **Verify code-def and code-refs work:**
|
||||
```bash
|
||||
gbrain code-def BrainEngine # prints the file + line of the definition
|
||||
gbrain code-refs BrainEngine --json # JSON array of every usage site
|
||||
```
|
||||
If both return non-empty arrays, code indexing is working end-to-end.
|
||||
|
||||
5. **Observe incremental chunking.** Edit one function in a 20-function file, re-run `sync --source <id>`. Embedding cost should be ~5% of the first sync because unchanged chunks reuse their existing embeddings.
|
||||
|
||||
## Migration from Wintermute's `repos` (if you used it)
|
||||
|
||||
v0.19.0 deletes `~/.gbrain/config.json`'s `repos` array in favor of the `sources` table. The CLI surface is preserved as a deprecated alias: `gbrain repos add` still works, but routes into `runSources` with a one-line deprecation notice on stderr. Existing scripts keep working; prefer `gbrain sources` going forward.
|
||||
|
||||
If you had repos configured in `~/.gbrain/config.json`, re-register them:
|
||||
```bash
|
||||
gbrain sources add <name> --path <path>
|
||||
```
|
||||
Per-repo sync bookmarks live in the `sources` table now (not config.json).
|
||||
|
||||
## Flag in `pending-host-work.jsonl`
|
||||
|
||||
Per the v0.11.0 convention, the migration orchestrator writes an entry to `~/.gbrain/migrations/pending-host-work.jsonl` flagging the new CLI surfaces so headless agents can walk the TODOs:
|
||||
|
||||
```json
|
||||
{"version": "0.19.0", "action": "register_code_source", "status": "pending"}
|
||||
```
|
||||
|
||||
Agents that handle pending-host-work should offer the user a `gbrain sources add ...` prompt.
|
||||
|
||||
## When NOT to run the migration
|
||||
|
||||
Never. v0.19.0 is fully backward-compatible. Existing markdown-only brains see zero behavior change until the user adds a code source.
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
version: 0.21.0
|
||||
feature_pitch:
|
||||
headline: "Code Cathedral II — chunk-grain FTS, qualified symbols, structural edges."
|
||||
one_liner: "Natural-language queries now rank docstring matches first. CHUNKER_VERSION 3→4 rolls the new chunker over existing code pages automatically on next sync."
|
||||
user_action_required: true
|
||||
---
|
||||
|
||||
# v0.21.0 — Code Cathedral II
|
||||
|
||||
This release is the biggest code-search upgrade in gbrain history. Chunk-grain FTS with doc_comment Weight A ranks natural-language queries against docstrings above prose. CHUNKER_VERSION 3 → 4 folds into content_hash so every existing code page re-chunks. File classifier widened from 9 to 35 extensions. Markdown fence extraction, `sync --all` cost preview, and `reconcile-links` batch command ship alongside the chunker upgrade.
|
||||
|
||||
## Schema migrations applied automatically
|
||||
|
||||
- **v27 — cathedral_ii_foundation** — adds `code_edges_chunk`, `code_edges_symbol`, `sources.chunker_version`, and new `content_chunks` columns (`parent_symbol_path`, `doc_comment`, `symbol_name_qualified`, `search_vector`). Includes the chunk-grain FTS trigger that builds from `setweight(to_tsvector('english', doc_comment), 'A') || setweight(to_tsvector('english', chunk_text), 'B') || setweight(to_tsvector('english', symbol_name_qualified), 'A')`.
|
||||
- **v28 — cathedral_ii_chunk_fts_backfill** — populates `search_vector` on every existing chunk so day-1 queries already rank correctly.
|
||||
|
||||
These run as part of `gbrain upgrade` → `gbrain apply-migrations`. No manual DDL needed.
|
||||
|
||||
## What the agent should do after upgrading
|
||||
|
||||
1. **Confirm migrations landed:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for `schema_version: 28`. If lower, run `gbrain apply-migrations --yes`.
|
||||
|
||||
2. **Pick a backfill path.** The `CHUNKER_VERSION` bump + `sources.chunker_version` gate means existing code pages must re-chunk for the new shape to take effect. Two paths:
|
||||
|
||||
**Automatic (recommended):** next `gbrain sync --source <id>` detects the version mismatch and forces a full re-walk regardless of git HEAD equality. No cost preview, no user interaction. The Layer 12 SP-1 fix from codex's second-pass review.
|
||||
|
||||
**Immediate:** preview cost, then reindex every code page now.
|
||||
```bash
|
||||
gbrain reindex-code --dry-run # preview token count + $USD cost
|
||||
gbrain reindex-code --yes # reindex all code pages
|
||||
gbrain reindex-code --source <id> --yes # scope to one source
|
||||
```
|
||||
On non-TTY (automation / cron) `reindex-code` without `--yes` emits a `ConfirmationRequired` envelope and exits 2 — same shape as `sync --all`. The envelope matches v0.19.0's `StructuredAgentError`.
|
||||
|
||||
3. **Verify chunk-grain FTS works.** A query that mentions a concept in a docstring (not just the function body) should rank higher than a page that mentions it only in prose:
|
||||
```bash
|
||||
gbrain query "whatever your docstring says"
|
||||
```
|
||||
Expected: top hit is the chunk whose `doc_comment` matches.
|
||||
|
||||
4. **(Optional) Reconcile doc↔impl links.** v0.19.0 Layer 6 forward-extracted code refs from markdown pages when they imported, but edges dropped if the code page hadn't imported yet. v0.21.0's `reconcile-links` batch-scans every markdown page and idempotently reinserts missing edges:
|
||||
```bash
|
||||
gbrain reconcile-links # full run
|
||||
gbrain reconcile-links --dry-run # preview only
|
||||
gbrain reconcile-links --json # machine output
|
||||
```
|
||||
Respects `auto_link=false` config (prints warn + exits 0 when disabled).
|
||||
|
||||
## Widened file classifier — more languages sync now
|
||||
|
||||
Previous classifier recognized 9 extensions as code. v0.21.0 widens to 35 (Rust, Ruby, Java, C#, C/C++, Swift, Kotlin, Scala, PHP, Elixir, Elm, OCaml, Dart, Zig, Solidity, Lua, shell, etc.). If your repo contains source files in languages beyond TS/JS/Py/Go, they'll flow through the code chunker on next sync. No action needed — `detectCodeLanguage` handles dispatch.
|
||||
|
||||
## Flag in `pending-host-work.jsonl`
|
||||
|
||||
The migration orchestrator emits a backfill-prompt phase that prints the two backfill choices directly. No `pending-host-work.jsonl` entry is written — the choice is user-driven and ephemeral (either reindex now or wait for next sync).
|
||||
|
||||
## When NOT to run the migration
|
||||
|
||||
Never skip it. v0.21.0 is fully backward-compatible at the API level (page-grain FTS shape preserved externally; chunk-grain is internal only). Skipping the migration means agents miss doc_comment Weight A ranking and the chunker_version gate never fires — existing code pages silently stay on CHUNKER_VERSION 3 forever until you sync with `--full`.
|
||||
@@ -1,113 +0,0 @@
|
||||
# v0.22.4 — Frontmatter Guard
|
||||
|
||||
## What ships
|
||||
|
||||
- `gbrain frontmatter` CLI (validate / audit / install-hook)
|
||||
- `frontmatter_integrity` subcheck under `gbrain doctor`
|
||||
- New `frontmatter-guard` skill (`skills/frontmatter-guard/SKILL.md`)
|
||||
- Pre-commit hook helper for git-backed brain repos
|
||||
- Audit-only migration that scans every registered source, writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`, and queues per-source TODO
|
||||
entries to `~/.gbrain/migrations/pending-host-work.jsonl`
|
||||
- 0 warnings on `gbrain check-resolvable` (down from 7 on master)
|
||||
|
||||
## What the agent should do post-upgrade
|
||||
|
||||
The orchestrator handles the mechanical side. Your job is to surface the audit
|
||||
to the user and apply fixes with their consent.
|
||||
|
||||
### 1. Run the orchestrator
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
This runs three idempotent phases:
|
||||
|
||||
- `schema` (no-op in v0.22.4)
|
||||
- `audit` — `scanBrainSources()` over every registered source; writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- `emit-todo` — appends one entry per source-with-issues to
|
||||
`~/.gbrain/migrations/pending-host-work.jsonl`. Each entry contains the
|
||||
exact `gbrain frontmatter validate <source-path> --fix` command.
|
||||
|
||||
The migration **never mutates brain pages**. Auto-fixes only run when the user
|
||||
explicitly invokes the validate-with-fix command below.
|
||||
|
||||
### 2. Read the audit report
|
||||
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/v0.22.4-audit.json
|
||||
```
|
||||
|
||||
The shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [
|
||||
{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }
|
||||
]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Surface the report to the user
|
||||
|
||||
State the per-source counts in plain language. Example:
|
||||
|
||||
> "v0.22.4 ships frontmatter-guard. I ran an audit and found 17 issues across
|
||||
> 1 source (default: 8 MISSING_CLOSE, 5 NESTED_QUOTES, 4 NULL_BYTES). The
|
||||
> mechanical errors are auto-fixable; SLUG_MISMATCH cases (if any) need your
|
||||
> review. Want me to fix the auto-fixable ones now?"
|
||||
|
||||
### 4. Run the fix (with consent)
|
||||
|
||||
Per source with issues, the queued command is:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <source_path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes a `.bak` backup for every modified file. SLUG_MISMATCH errors
|
||||
are surfaced for manual review (not auto-fixed) — gbrain derives slugs from
|
||||
path, so a mismatched slug usually means the user renamed the file
|
||||
intentionally or the slug field is stale.
|
||||
|
||||
### 5. (Optional) Install the pre-commit hook
|
||||
|
||||
For git-backed sources only:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
This blocks future malformed-frontmatter commits at the git layer. Bypass with
|
||||
`git commit --no-verify`. Skip this step for non-git brains.
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "frontmatter_integrity")'
|
||||
gbrain frontmatter audit --json | jq '.total'
|
||||
```
|
||||
|
||||
Both should report 0 issues after fixes are applied.
|
||||
|
||||
### 7. If anything fails
|
||||
|
||||
Open an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
@@ -1,302 +0,0 @@
|
||||
---
|
||||
name: minion-orchestrator
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Unified Minions skill for both deterministic shell jobs and LLM subagent
|
||||
orchestration. Replaces the older `gbrain-jobs` routing intent. Use when:
|
||||
submitting gbrain jobs, shell/background tasks, spawning subagents,
|
||||
checking progress, steering running work, pausing/resuming, parallel
|
||||
fan-out. One durable, observable, steerable queue interface.
|
||||
triggers:
|
||||
- "gbrain jobs submit"
|
||||
- "submit a gbrain job"
|
||||
- "submit a shell job"
|
||||
- "shell job"
|
||||
- "run shell command in background"
|
||||
- "deterministic background task"
|
||||
- "spawn agent"
|
||||
- "background task"
|
||||
- "run in background"
|
||||
- "check on agent"
|
||||
- "agent progress"
|
||||
- "what's running"
|
||||
- "steer agent"
|
||||
- "change direction"
|
||||
- "tell the agent"
|
||||
- "pause agent"
|
||||
- "stop agent"
|
||||
- "resume agent"
|
||||
- "parallel tasks"
|
||||
- "fan out"
|
||||
- "do these in parallel"
|
||||
tools:
|
||||
- submit_job
|
||||
- get_job
|
||||
- list_jobs
|
||||
- cancel_job
|
||||
- pause_job
|
||||
- resume_job
|
||||
- replay_job
|
||||
- send_job_message
|
||||
- get_job_progress
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Minion Orchestrator
|
||||
|
||||
## Contract
|
||||
|
||||
Minions is a Postgres-native job queue for durable, observable background work.
|
||||
This single skill handles two lanes:
|
||||
- Deterministic shell jobs (`gbrain jobs submit shell ...`)
|
||||
- LLM subagent jobs (`gbrain agent run ...`)
|
||||
|
||||
When to route to Minions: durable, observable work that must survive restarts,
|
||||
fan out across many parallel tasks, or persist across sessions. Routing policy
|
||||
is defined in `skills/conventions/subagent-routing.md` — the project default is
|
||||
`pain_triggered` (native subagents first, Minions after specific pain signals
|
||||
fire); Mode A (all-through-Minions) is opt-in.
|
||||
|
||||
Guarantees:
|
||||
- Jobs survive gateway restart (Postgres-backed)
|
||||
- Every job has structured progress, token accounting, and session transcripts
|
||||
- Running agents can be steered mid-flight via inbox messages
|
||||
- Jobs can be paused, resumed, or cancelled at any time
|
||||
- Parent-child DAGs with configurable failure policies
|
||||
|
||||
## Route the Request: Shell Job vs Subagent
|
||||
|
||||
| Condition | Action |
|
||||
|---|---|
|
||||
| User asks for deterministic command/script run | Shell job (CLI: `gbrain jobs submit shell ...`) |
|
||||
| User asks to "run in minions" + explicit command/argv | Shell job (CLI, `--params` with `cmd` or `argv`) |
|
||||
| User asks for research/reasoning/iterative agent | Subagent job (CLI: `gbrain agent run`) |
|
||||
| User asks to steer/pause/resume an agent | Subagent job lifecycle tools (MCP-callable) |
|
||||
| Single simple operation under ~30s | Consider inline execution first |
|
||||
| Needs restart durability/observability | Submit as Minion job |
|
||||
| Parallel work (2+ streams) | `gbrain agent run --fanout-manifest` or parent + child subagents |
|
||||
|
||||
If intent is ambiguous, ask one clarification:
|
||||
"Do you want a deterministic shell command job, or an LLM agent job?"
|
||||
|
||||
## Shell Jobs (Deterministic Scripts)
|
||||
|
||||
Use for reproducible command execution, ETL steps, cron work, and scriptable
|
||||
tasks where no LLM reasoning loop is needed.
|
||||
|
||||
### Preconditions (read before submitting your first shell job)
|
||||
|
||||
- **`GBRAIN_ALLOW_SHELL_JOBS=1` must be set on the worker environment.**
|
||||
Without it, the shell handler refuses to register and submissions sit in
|
||||
`waiting` silently. Gate lives in `src/core/minions/handlers/shell.ts`.
|
||||
- **Security:** flipping `GBRAIN_ALLOW_SHELL_JOBS=1` authorizes arbitrary
|
||||
command execution on the worker. On a shared queue, this is a remote code
|
||||
execution surface. Treat as privileged infrastructure authorization.
|
||||
- **Execution mode — pick one:**
|
||||
- **Postgres + daemon:** `gbrain jobs work` runs a persistent worker that
|
||||
claims and executes jobs from the queue.
|
||||
- **PGLite + --follow:** `gbrain jobs submit ... --follow` runs inline.
|
||||
The daemon mode is not available on PGLite (exclusive file lock). See
|
||||
`docs/guides/minions-shell-jobs.md`.
|
||||
- **MCP boundary:** shell-job submission is CLI-only. `submit_job name="shell"`
|
||||
over MCP throws an `OperationError` with code `permission_denied` ("'shell'
|
||||
jobs cannot be submitted over MCP") because `shell` is in `PROTECTED_JOB_NAMES`.
|
||||
Agents CAN observe shell jobs via `get_job` / `list_jobs` / `get_job_progress`
|
||||
(not protected), but cannot submit them. Operator or autopilot submits;
|
||||
agent observes.
|
||||
- **Verify setup:** after configuration, run `gbrain jobs stats` (CLI) to
|
||||
confirm the worker is registered and consuming the queue.
|
||||
|
||||
### Submit (CLI, operator or autopilot)
|
||||
|
||||
Shell jobs take their command via `--params` as a JSON object with `cmd` (string)
|
||||
or `argv` (array), plus `cwd` and optional `env`.
|
||||
|
||||
Command string form:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Argv form (no shell expansion):
|
||||
```
|
||||
gbrain jobs submit shell --params '{"argv":["bash","-lc","echo hello"],"cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Inline execution on PGLite or any one-shot deployment:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/tmp"}' --follow
|
||||
```
|
||||
|
||||
Queue/lifecycle flags exposed by `gbrain jobs submit --help`: `--queue`,
|
||||
`--priority`, `--delay`, `--max-attempts`, `--max-stalled`, `--backoff-type`,
|
||||
`--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`,
|
||||
`--dry-run`.
|
||||
|
||||
### Monitor (agents or operator)
|
||||
|
||||
These operations are MCP-callable and safe for agent use:
|
||||
|
||||
```
|
||||
list_jobs --name shell --status active
|
||||
get_job ID
|
||||
get_job_progress ID
|
||||
```
|
||||
|
||||
Check structured result fields (exit code, stdout/stderr tails, attempts,
|
||||
timings) from `get_job`. Use `gbrain jobs stats` (CLI) for worker/queue
|
||||
health dashboard.
|
||||
|
||||
### Control (MCP-callable)
|
||||
|
||||
```
|
||||
cancel_job id=ID
|
||||
replay_job id=ID
|
||||
```
|
||||
|
||||
`replay_job` is not protected — only shell *submission* is. Agents can
|
||||
cancel or replay a shell job without CLI access.
|
||||
|
||||
Use idempotency keys for recurring shell workloads to avoid duplicate runs.
|
||||
|
||||
## Subagent Jobs (LLM Orchestration)
|
||||
|
||||
Use for open-ended reasoning, tool-using research, and fan-out synthesis.
|
||||
|
||||
**User-facing entrypoint:** `gbrain agent run <prompt>` is the canonical way
|
||||
to submit subagent work. It handles the elevated-trust plumbing — `subagent`
|
||||
and `subagent_aggregator` are both in `PROTECTED_JOB_NAMES`, so direct MCP
|
||||
submission requires `{allowProtectedSubmit: true}`, which `gbrain agent run`
|
||||
supplies.
|
||||
|
||||
## Phase 1: Submit
|
||||
|
||||
```
|
||||
gbrain agent run "Research Acme Corp revenue" --tools "search,query"
|
||||
```
|
||||
|
||||
`--tools` accepts a comma-separated subset of `BRAIN_TOOL_ALLOWLIST` (see
|
||||
`src/core/minions/tools/brain-allowlist.ts`): `query`, `search`, `get_page`,
|
||||
`list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`,
|
||||
`resolve_slugs`, `get_ingest_log`, `put_page`. Anything outside the allow-list
|
||||
is rejected at submit time with `allowed_tools references unknown tool`.
|
||||
|
||||
For parallel work with a fan-out manifest:
|
||||
```
|
||||
gbrain agent run --fanout-manifest companies.json
|
||||
```
|
||||
|
||||
The manifest describes N children + 1 aggregator. Each child runs
|
||||
`name="subagent"` under the hood; the aggregator runs `name="subagent_aggregator"`
|
||||
and claims AFTER every child terminates. See
|
||||
`src/core/minions/handlers/subagent.ts` and
|
||||
`src/core/minions/handlers/subagent-aggregator.ts`.
|
||||
|
||||
Flags (from `src/commands/agent.ts`):
|
||||
- `--subagent-def <name>` — named subagent definition
|
||||
- `--model <id>` — override model
|
||||
- `--max-turns <N>` — cap the LLM loop
|
||||
- `--tools <csv>` — allow-listed brain tools (see above)
|
||||
- `--timeout-ms <N>` — hard timeout per job
|
||||
- `--fanout-manifest <file>` — N children + 1 aggregator
|
||||
- `--follow` / `--no-follow` — stream logs + wait (default on TTY)
|
||||
- `--detach` — submit and return immediately
|
||||
|
||||
Queue/priority/retry tuning is not exposed by `gbrain agent run`; submit the
|
||||
raw `subagent` handler via `gbrain jobs submit` (requires CLI trust) if you
|
||||
need those knobs.
|
||||
|
||||
## Phase 2: Monitor
|
||||
|
||||
```
|
||||
list_jobs --status active # MCP — what's running?
|
||||
get_job ID # MCP — full details + logs + tokens
|
||||
get_job_progress ID # MCP — structured progress snapshot
|
||||
gbrain jobs stats # CLI — queue health dashboard
|
||||
gbrain agent logs ID --follow # CLI — streaming transcript + heartbeat
|
||||
```
|
||||
|
||||
Progress includes: step count, total steps, message, token usage, last tool called.
|
||||
|
||||
## Phase 3: Steer
|
||||
|
||||
Send a message to redirect a running agent:
|
||||
```
|
||||
send_job_message id=ID payload={"directive":"focus on revenue, skip headcount"}
|
||||
```
|
||||
|
||||
The agent handler reads inbox messages on each iteration and injects them as
|
||||
context. Messages are acknowledged (read receipts tracked).
|
||||
|
||||
Only the parent job or admin can send messages (sender validation).
|
||||
|
||||
## Phase 4: Lifecycle
|
||||
|
||||
```
|
||||
pause_job id=ID # freeze without losing state
|
||||
resume_job id=ID # pick up where it left off
|
||||
cancel_job id=ID # hard stop
|
||||
replay_job id=ID # re-run with same or modified params
|
||||
replay_job id=ID data_overrides={"depth":"deep"} # replay with changes
|
||||
```
|
||||
|
||||
All lifecycle ops are MCP-callable.
|
||||
|
||||
## Phase 5: Review Results
|
||||
|
||||
```
|
||||
get_job ID # result, token counts, transcript
|
||||
```
|
||||
|
||||
Token accounting: every job tracks `tokens_input`, `tokens_output`, `tokens_cache_read`.
|
||||
Child tokens roll up to parent automatically on completion.
|
||||
|
||||
## Output Format
|
||||
|
||||
When reporting job status to the user:
|
||||
|
||||
```
|
||||
Job #ID (name) — status
|
||||
Progress: step/total — last action
|
||||
Tokens: input_count in / output_count out (+ cache_read cached)
|
||||
Runtime: Xs
|
||||
Children: N pending, M completed
|
||||
```
|
||||
|
||||
When reporting completion:
|
||||
|
||||
```
|
||||
Job #ID completed in Xs
|
||||
Tokens used: input / output / cache_read
|
||||
Result: <summary>
|
||||
```
|
||||
|
||||
When reporting batch status (parent with children):
|
||||
|
||||
```
|
||||
Parent #ID — waiting-children
|
||||
#A subagent(Acme) — active, 3/5 steps, 2.5k tokens
|
||||
#B subagent(Beta) — completed, 1.8k tokens
|
||||
#C subagent(Gamma) — paused
|
||||
Total tokens so far: 4.3k
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- Don't spawn a Minion for a single search query (use search tool directly)
|
||||
- Don't fire-and-forget without checking results
|
||||
- Don't spawn > 5 concurrent agents without checking `gbrain jobs stats` first
|
||||
- For subagent work, don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available (use `gbrain agent run` instead)
|
||||
- Don't poll `get_job` in a tight loop (use `get_job_progress` for lightweight checks)
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Submit a background job — `submit_job` (MCP, non-protected names only; shell jobs are CLI-only, subagent jobs via `gbrain agent run`)
|
||||
- Get job details — `get_job` (MCP)
|
||||
- List jobs with filters — `list_jobs` (MCP)
|
||||
- Cancel a job — `cancel_job` (MCP)
|
||||
- Pause a job — `pause_job` (MCP)
|
||||
- Resume a paused job — `resume_job` (MCP)
|
||||
- Replay a completed/failed job — `replay_job` (MCP)
|
||||
- Send sidechannel message — `send_job_message` (MCP)
|
||||
- Get structured progress — `get_job_progress` (MCP)
|
||||
- Queue stats — `gbrain jobs stats` (CLI; no MCP equivalent)
|
||||
@@ -12,12 +12,6 @@ triggers:
|
||||
- "what happened"
|
||||
- "search for"
|
||||
- "look up"
|
||||
- "background on"
|
||||
- "notes on"
|
||||
- "who knows who"
|
||||
- "relationship between"
|
||||
- "connections"
|
||||
- "graph query"
|
||||
tools:
|
||||
- search
|
||||
- query
|
||||
@@ -113,28 +107,6 @@ When referencing brain pages in your answer, propagate inline citations:
|
||||
the user can trace facts to their origin
|
||||
- When you synthesize across multiple pages, cite all sources
|
||||
|
||||
## Graph Traversal (v0.10.1+)
|
||||
|
||||
For relationship questions ("who knows who at X?", "connections between A and B",
|
||||
"who works at Acme?", "who attended the standup?"), use the graph layer instead
|
||||
of full-text search:
|
||||
|
||||
- `gbrain graph-query <slug> --type <link_type> --depth N --direction in|out|both`
|
||||
- Available link types: `attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, `source`
|
||||
- `--direction in` answers "who points to X?" (e.g., who works at company X)
|
||||
- `--direction out` answers "what does X point to?" (default)
|
||||
- `--depth N` controls multi-hop traversal (default 5)
|
||||
|
||||
Examples:
|
||||
- "Who works at Acme?" → `gbrain graph-query companies/acme --type works_at --direction in`
|
||||
- "Who attended Demo Day W26?" → `gbrain graph-query meetings/demo-day-w26 --type attended --direction out`
|
||||
- "What companies has Emily advised?" → `gbrain graph-query people/emily --type advises --direction out`
|
||||
- "Who has Alice met (via meetings)?" → `gbrain graph-query people/alice --type attended --depth 2`
|
||||
|
||||
Combine with `gbrain query` for queries that need BOTH semantic similarity AND
|
||||
graph structure. Search results are ranked with a small backlink boost so well-
|
||||
connected entities surface higher.
|
||||
|
||||
## Search Quality Awareness
|
||||
|
||||
If search results seem off (wrong results, missing known pages, irrelevant hits):
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
// Routing eval fixtures for skills/query. Check 5 (W2, v0.17).
|
||||
// Each line: {intent, expected_skill, ambiguous_with?}.
|
||||
// D-CX-6 rule: intent must paraphrase the trigger, not copy it.
|
||||
{"intent": "Who is Paul Graham and what has he written about startups", "expected_skill": "query"}
|
||||
{"intent": "I need background on the Series B round Acme raised last year", "expected_skill": "query", "ambiguous_with": ["brain-ops"]}
|
||||
{"intent": "Do we already have notes on this person", "expected_skill": "query", "ambiguous_with": ["brain-ops"]}
|
||||
+2
-53
@@ -142,24 +142,7 @@ echo "=== Discovery Complete ==="
|
||||
4. **Start embeddings.** Refresh stale embeddings (runs in background). Keyword
|
||||
search works NOW, semantic search improves as embeddings complete.
|
||||
|
||||
5. **Backfill the knowledge graph.** Populate typed links and structured timeline
|
||||
from the imported pages. Auto-link maintains both going forward, but historical
|
||||
pages need a one-time backfill.
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db --dry-run | head -20 # preview
|
||||
gbrain extract links --source db # commit
|
||||
gbrain extract timeline --source db # dated events
|
||||
gbrain stats # verify links > 0
|
||||
```
|
||||
|
||||
After this, `gbrain graph-query <slug> --depth 2` works and search ranks
|
||||
well-connected entities higher. Idempotent — safe to re-run anytime.
|
||||
Supports `--since YYYY-MM-DD` for incremental runs on huge brains.
|
||||
|
||||
Skip if Phase C imported zero pages (auto-link handles new writes).
|
||||
|
||||
6. **Offer file migration.** If the repo has binary files (.raw/ directories with
|
||||
5. **Offer file migration.** If the repo has binary files (.raw/ directories with
|
||||
images, PDFs, audio):
|
||||
> "You have N binary files (X GB) in your brain repo. Want to move them to cloud
|
||||
> storage? Your git repo will drop from X GB to Y MB. All links keep working."
|
||||
@@ -185,40 +168,6 @@ echo "=== Discovery Complete ==="
|
||||
If no markdown repos are found, create a starter brain with a few template pages
|
||||
(a person page, a company page, a concept page) from docs/GBRAIN_RECOMMENDED_SCHEMA.md.
|
||||
|
||||
## Phase C.5: One-step autopilot + Minions install (v0.11.1+)
|
||||
|
||||
Run the migration runner once, then install autopilot. Two commands, done:
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes # applies any pending migrations; idempotent on healthy installs
|
||||
gbrain autopilot --install # supervises itself + forks the Minions worker; env-aware
|
||||
```
|
||||
|
||||
What `gbrain autopilot --install` does:
|
||||
|
||||
- On **macOS**: writes a launchd plist at `~/Library/LaunchAgents/com.gbrain.autopilot.plist`.
|
||||
- On **Linux with systemd**: writes `~/.config/systemd/user/gbrain-autopilot.service`
|
||||
with `Restart=on-failure`.
|
||||
- On **ephemeral containers** (Render / Railway / Fly / Docker): writes
|
||||
`~/.gbrain/start-autopilot.sh` and prints the one-line your agent's
|
||||
bootstrap should source to launch autopilot on every container start.
|
||||
Auto-injects into OpenClaw's `hooks/bootstrap/ensure-services.sh` if
|
||||
detected (use `--no-inject` to opt out).
|
||||
- On **Linux without systemd**: installs a crontab entry (every 5 min).
|
||||
|
||||
Autopilot then supervises the Minions worker as a child process. Users get
|
||||
sync + extract + embed + backlinks + durable Postgres-backed job processing
|
||||
from ONE install step. No separate `gbrain jobs work` daemon to manage.
|
||||
|
||||
On PGLite, autopilot runs inline (PGLite's exclusive file lock blocks a
|
||||
separate worker process). Everything else still works.
|
||||
|
||||
If `apply-migrations` prints "N host-specific items need your agent's
|
||||
attention," read `~/.gbrain/migrations/pending-host-work.jsonl` + walk
|
||||
`skills/migrations/v0.11.0.md` + `docs/guides/plugin-handlers.md` to
|
||||
register host-specific handlers. Re-run `apply-migrations` after each
|
||||
batch.
|
||||
|
||||
## Phase D: Brain-First Lookup Protocol
|
||||
|
||||
Inject the brain-first lookup protocol into the project's AGENTS.md (or equivalent).
|
||||
@@ -275,7 +224,7 @@ Inject the key patterns into the agent's system context or AGENTS.md:
|
||||
1. **Brain-agent loop** (Section 2): read before responding, write after learning
|
||||
2. **Entity detection** (Section 3): spawn on every message, capture people/companies/ideas
|
||||
3. **Source attribution** (Section 7): every fact needs `[Source: ...]`
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
4. **Iron law back-linking** (Section 15.4): every mention links back to the entity page
|
||||
|
||||
Tell the user: "The production agent guide is at docs/GBRAIN_SKILLPACK.md. It covers
|
||||
the brain-agent loop, entity detection, enrichment, meeting ingestion, and cron
|
||||
|
||||
@@ -15,11 +15,6 @@ tools:
|
||||
- add_link
|
||||
- add_timeline_entry
|
||||
mutating: true
|
||||
writes_pages: true
|
||||
writes_to:
|
||||
- people/
|
||||
- companies/
|
||||
- concepts/
|
||||
---
|
||||
|
||||
# Signal Detector — Ambient Brain Capture
|
||||
@@ -44,7 +39,7 @@ This skill guarantees:
|
||||
- Back-links all entity mentions (Iron Law)
|
||||
- Citations on every fact written
|
||||
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
|
||||
Every time this skill creates or updates a brain page that mentions a person or company:
|
||||
1. Check if that person/company has a brain page
|
||||
@@ -74,12 +69,7 @@ meetings, and concepts. An original without cross-links is a dead original.
|
||||
- If NO page → check notability. If notable, create page with enrichment.
|
||||
- If page exists but THIN → trigger enrich
|
||||
- If page exists and RICH → no action
|
||||
3. For new FACTS with specific dates → call `gbrain timeline-add <slug> <date> "<summary>"`
|
||||
|
||||
**Auto-link (v0.10.1):** When you write/update an originals or ideas page that
|
||||
references a person or company, the auto-link post-hook on `put_page`
|
||||
automatically creates the link from the new page to that entity. You don't
|
||||
need to call `gbrain link` manually. Timeline entries still need explicit calls.
|
||||
3. For new FACTS about existing entities → add timeline entry
|
||||
|
||||
### Phase 3: Signal Logging
|
||||
|
||||
|
||||
@@ -1,185 +0,0 @@
|
||||
---
|
||||
name: skillify
|
||||
version: 1.0.0
|
||||
description: |
|
||||
The meta skill. Turn any raw feature or script into a properly-skilled,
|
||||
tested, resolvable, evaled unit of agent-visible capability. Use when
|
||||
the user says "skillify this", "is this a skill?", "make this proper",
|
||||
or after a new feature is built without the full skill infrastructure.
|
||||
|
||||
Paired with `gbrain check-resolvable`, skillify gives a user-controllable
|
||||
equivalent of Hermes' auto-skill-creation: you build, skillify checks the
|
||||
checklist, check-resolvable verifies nothing is orphaned. The human keeps
|
||||
judgment; the tooling keeps the checklist honest.
|
||||
triggers:
|
||||
- "skillify this"
|
||||
- "skillify"
|
||||
- "is this a skill?"
|
||||
- "make this proper"
|
||||
- "add tests and evals for this"
|
||||
- "check skill completeness"
|
||||
tools:
|
||||
- search
|
||||
- list_pages
|
||||
mutating: false
|
||||
---
|
||||
|
||||
# Skillify — The Meta Skill
|
||||
|
||||
## Contract
|
||||
|
||||
A feature is "properly skilled" when all ten checklist items are present:
|
||||
|
||||
1. `SKILL.md` — skill file with YAML frontmatter, triggers, contract, phases.
|
||||
2. Code — deterministic script if applicable.
|
||||
3. Unit tests — cover every branch of deterministic logic.
|
||||
4. Integration tests — exercise live endpoints, not just in-memory shape.
|
||||
5. LLM evals — quality/correctness cases if the feature includes any LLM call.
|
||||
6. Resolver trigger — `skills/RESOLVER.md` entry with the trigger patterns
|
||||
the user actually types.
|
||||
7. Resolver trigger eval — test that feeds trigger phrases to the resolver
|
||||
and asserts they route to this skill, not the old pre-skillify path.
|
||||
8. Check-resolvable — `gbrain check-resolvable` passes (skill is reachable,
|
||||
MECE against its siblings, no DRY violations).
|
||||
9. E2E test — exercises the full pipeline from user turn to side effect.
|
||||
10. Brain filing — if the feature writes brain pages, `brain/RESOLVER.md`
|
||||
has an entry for the directory so the pages aren't orphaned.
|
||||
|
||||
## Trigger
|
||||
|
||||
- "skillify this" / "skillify" / "is this a skill?" / "make this proper"
|
||||
- "add tests and evals for this"
|
||||
- After building any new feature that touches user-facing behavior
|
||||
- When you grep the repo and notice a script with no SKILL.md next to it
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Audit what exists
|
||||
|
||||
For the feature being skillified, answer:
|
||||
|
||||
- **Feature name**: what does it do in one line?
|
||||
- **Code path**: where does the implementation live (file path)?
|
||||
- **Checklist status**: run `gbrain skillify check <path>` (preferred)
|
||||
or the legacy `scripts/skillify-check.ts <path>` shim. Both produce
|
||||
the same 10-item scorecard. Note which items are missing.
|
||||
|
||||
### Phase 2: Create missing pieces in order
|
||||
|
||||
**Fast path — brand-new skill:** run `gbrain skillify scaffold <name>
|
||||
--description "..." [--triggers "p1,p2,p3"] [--writes-pages --writes-to
|
||||
"people/,companies/"]`. This creates all 5 stub files atomically and
|
||||
appends an idempotent resolver row. Every scaffolded file carries the
|
||||
`SKILLIFY_STUB` sentinel; `gbrain check-resolvable --strict` will fail
|
||||
CI until you replace the stubs with real content.
|
||||
|
||||
**Manual path — extending an existing skill:** work the list top-down.
|
||||
Each earlier item constrains what later items look like (the SKILL.md
|
||||
contract determines what tests assert; tests determine what evals gate;
|
||||
the resolver entry determines what trigger-eval checks).
|
||||
|
||||
1. Write `SKILL.md` first. Frontmatter must include `name`, `version`,
|
||||
`description`, `triggers[]`, `tools[]`, `mutating`. Body has at minimum
|
||||
Contract, Phases, and Output Format sections.
|
||||
2. Extract deterministic code into a script if applicable (scripts/*.ts
|
||||
for gbrain; host projects may use .mjs / .py / whatever their runtime
|
||||
uses).
|
||||
3. Write unit tests for every branch of the script. Mock external calls
|
||||
(LLM, DB, network) so tests run fast and deterministic.
|
||||
4. Add integration tests that hit real endpoints. These catch bugs the
|
||||
unit tests' mocks hide (see the `files-test-reimplements-production`
|
||||
learning: reimplementation in tests lets production vulnerabilities
|
||||
slip through).
|
||||
5. Add LLM evals if the feature includes any LLM call. Even a three-case
|
||||
eval (happy / edge / adversarial) is cheap insurance against prompt
|
||||
regressions.
|
||||
6. Add the resolver trigger to `skills/RESOLVER.md`. Use the trigger
|
||||
patterns the user ACTUALLY types, not what you think they should type.
|
||||
7. Add a resolver trigger eval that feeds those patterns in and asserts
|
||||
they route to the new skill.
|
||||
8. Run `gbrain check-resolvable` (auto-detects skill trees) or
|
||||
`gbrain check-resolvable --skills-dir <path>` for custom locations.
|
||||
OpenClaw workspaces are auto-detected from
|
||||
`~/.openclaw/workspace/skills/`. The check validates reachability (is
|
||||
the skill mentioned from RESOLVER.md?), MECE overlap (does it duplicate
|
||||
an existing skill's trigger?), gap detection (are there user intents
|
||||
that fall through the resolver with no match?), and DRY. If it fails,
|
||||
fix the skill (or extend an existing one instead of creating a
|
||||
duplicate).
|
||||
9. Add an E2E smoke test. For gbrain: submit a Minion job or run a CLI
|
||||
invocation end-to-end against a fixture brain; assert side effects.
|
||||
10. Update `brain/RESOLVER.md` if the skill writes brain pages. Orphaned
|
||||
brain pages are worse than no brain pages.
|
||||
|
||||
### Phase 3: Verify
|
||||
|
||||
Run each of these and confirm green:
|
||||
|
||||
```bash
|
||||
# Unit tests
|
||||
bun test test/<skill-name>.test.ts
|
||||
|
||||
# Integration tests (when applicable)
|
||||
bun run test:e2e
|
||||
|
||||
# Resolver reachability + MECE + DRY
|
||||
gbrain check-resolvable
|
||||
|
||||
# Conformance tests (skill YAML + required sections)
|
||||
bun test test/skills-conformance.test.ts
|
||||
```
|
||||
|
||||
## Quality gates
|
||||
|
||||
A feature is NOT properly skilled until:
|
||||
|
||||
- All tests pass (unit + integration + evals).
|
||||
- It appears in `skills/RESOLVER.md` with accurate trigger patterns.
|
||||
- The resolver trigger eval confirms patterns route to the new skill.
|
||||
- `gbrain check-resolvable` shows no orphaned skills, no MECE overlaps,
|
||||
no DRY violations.
|
||||
- If it writes brain pages, `brain/RESOLVER.md` has the directory.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- ❌ Code with no SKILL.md — invisible to the resolver; the agent will
|
||||
never run it.
|
||||
- ❌ SKILL.md with no tests — untested contract; one prompt change
|
||||
regresses silently.
|
||||
- ❌ Tests that reimplement production code — the reimplementation's
|
||||
bugs don't catch production's bugs (the `files-test-reimplements-
|
||||
production` lesson).
|
||||
- ❌ Resolver entry that uses internal jargon the user never types —
|
||||
trigger patterns must mirror real user language.
|
||||
- ❌ Feature that writes to brain without a `brain/RESOLVER.md` entry —
|
||||
orphaned pages the agent will never find.
|
||||
- ❌ Deterministic logic in LLM space — should be a script.
|
||||
- ❌ LLM judgment in deterministic space — should be an eval.
|
||||
|
||||
## Why skillify + check-resolvable is the right pair
|
||||
|
||||
Hermes and similar agent frameworks auto-create skills as a background
|
||||
behavior. That's fine until you don't know what the agent shipped —
|
||||
checklists decay, tests drift, resolver entries get stale.
|
||||
|
||||
Gbrain ships the same capability as two user-controlled tools:
|
||||
|
||||
- `/skillify` builds the checklist and helps you fill in the gaps.
|
||||
- `gbrain check-resolvable` validates the whole skill tree: reachability,
|
||||
MECE, DRY, gap detection, orphaned skills.
|
||||
|
||||
You decide when and what. The human keeps judgment. The tooling keeps the
|
||||
checklist honest. In practice this combo produces zero orphaned skills,
|
||||
every feature with tests + evals + resolver triggers + evals of the
|
||||
triggers.
|
||||
|
||||
## Output Format
|
||||
|
||||
A skillify run produces, in order:
|
||||
|
||||
1. An audit printout listing which of the 10 items exist and which are
|
||||
missing for the target feature.
|
||||
2. The files created to close each gap (SKILL.md, test files, resolver
|
||||
entries).
|
||||
3. The final `gbrain check-resolvable` output confirming reachability.
|
||||
4. A one-line summary of the resulting skill completeness score (N/10).
|
||||
@@ -1,132 +0,0 @@
|
||||
---
|
||||
name: skillpack-check
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Run `gbrain skillpack-check` to produce an agent-readable JSON health report
|
||||
for the gbrain install. Wraps `gbrain doctor` + `gbrain apply-migrations
|
||||
--list` so a host agent (your OpenClaw's morning-briefing, any OpenClaw cron)
|
||||
can see at a glance whether the skillpack needs attention.
|
||||
|
||||
Use when the user asks "is gbrain healthy?", when a cron fires a morning
|
||||
check, or proactively when something seems off (jobs not running, brain
|
||||
not updating, autopilot silent).
|
||||
triggers:
|
||||
- "skillpack check"
|
||||
- "is gbrain healthy"
|
||||
- "gbrain health"
|
||||
- "check the brain"
|
||||
- "is the brain working"
|
||||
tools:
|
||||
- shell
|
||||
mutating: false
|
||||
---
|
||||
|
||||
# Skillpack Check
|
||||
|
||||
## Contract
|
||||
|
||||
Running `gbrain skillpack-check` returns a JSON report with:
|
||||
|
||||
- **`healthy`** (bool): true if no action needed.
|
||||
- **`summary`** (string): one-line summary safe to quote in a briefing.
|
||||
- **`actions`** (string[]): every remediation command. If non-empty, run them.
|
||||
- **`doctor`**: full `gbrain doctor --fast --json` output (filesystem checks).
|
||||
- **`migrations`**: applied/pending/partial counts from `apply-migrations --list`.
|
||||
|
||||
Exit code:
|
||||
- `0` — healthy, nothing to do.
|
||||
- `1` — action needed. Read `actions[]` and execute.
|
||||
- `2` — could not determine (binary crash or missing subcommand). Investigate.
|
||||
|
||||
## When to run
|
||||
|
||||
- **Daily cron** (e.g. your OpenClaw's `morning-briefing`): `gbrain skillpack-check --quiet`.
|
||||
Exit code alone tells you if anything is wrong; surface a one-liner in the
|
||||
briefing only when exit != 0. No JSON noise in happy-path briefings.
|
||||
- **On demand**: `gbrain skillpack-check` for the full JSON when debugging.
|
||||
- **In a CI pipeline**: same pattern — exit code gates, JSON is the evidence.
|
||||
|
||||
## What to do with the output
|
||||
|
||||
### Happy path (`healthy: true`)
|
||||
|
||||
Surface the summary in the agent's output only if asked. Nothing else.
|
||||
|
||||
### Action needed (`healthy: false`)
|
||||
|
||||
The `actions[]` array contains the commands to run, in order. Execute them:
|
||||
|
||||
```bash
|
||||
for cmd in $(echo "$REPORT" | jq -r '.actions[]'); do
|
||||
eval "$cmd"
|
||||
done
|
||||
```
|
||||
|
||||
Common `actions[]` entries and what they mean:
|
||||
|
||||
- `gbrain apply-migrations --yes` — A migration is pending or half-finished.
|
||||
Run this (it's idempotent). If it exits `status: "partial"`, the host has
|
||||
non-builtin cron handlers that need plugin registration — follow
|
||||
`skills/migrations/v0.11.0.md`.
|
||||
- `gbrain embed --stale` — Embeddings are stale.
|
||||
- `gbrain check-backlinks --fix` — Dead links or missing back-links.
|
||||
- Free-text action (no `Run:` prefix in the source message) — agent judgment
|
||||
needed. Quote it in the report for the user.
|
||||
|
||||
### Determine failure (`exit 2`)
|
||||
|
||||
Treat as urgent. Probably means the gbrain binary is missing from `$PATH` or
|
||||
a required subcommand crashed. Check:
|
||||
|
||||
1. `which gbrain` returns a path
|
||||
2. `gbrain --version` exits 0
|
||||
3. `~/.gbrain/` is accessible
|
||||
|
||||
## Output format
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "0.11.1",
|
||||
"ts": "2026-04-18T12:34:56.789Z",
|
||||
"healthy": false,
|
||||
"summary": "gbrain skillpack needs attention: 1 action(s) — gbrain apply-migrations --yes",
|
||||
"actions": ["gbrain apply-migrations --yes"],
|
||||
"doctor": {
|
||||
"exit_code": 1,
|
||||
"checks": [
|
||||
{ "name": "minions_migration", "status": "fail", "message": "MINIONS HALF-INSTALLED (partial migration: 0.11.0). Run: gbrain apply-migrations --yes" }
|
||||
]
|
||||
},
|
||||
"migrations": {
|
||||
"applied_count": 0,
|
||||
"pending_count": 0,
|
||||
"partial_count": 1,
|
||||
"stdout": "..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- ❌ Running without `--quiet` in a cron that emails its output — you'll get
|
||||
the full JSON blob in every daily email. Use `--quiet` in crons.
|
||||
- ❌ Ignoring exit code 2. A crashed doctor is worse than a failing check
|
||||
because you don't even know what's wrong.
|
||||
- ❌ Running on every chat turn. Once per hour (or on user request) is plenty.
|
||||
- ❌ Treating warnings as failures. Only `fail` status needs action;
|
||||
`warn` is informational.
|
||||
|
||||
## Output Format
|
||||
|
||||
The skill itself doesn't write files; it reports the CLI output verbatim to
|
||||
the user (or to the agent's briefing pipeline). One-line summary first,
|
||||
then the action list, then (only if relevant) the full JSON for debugging.
|
||||
|
||||
## Related
|
||||
|
||||
- `gbrain doctor` — the underlying filesystem + DB check. skillpack-check
|
||||
composes this.
|
||||
- `gbrain apply-migrations --list` — the migration status view.
|
||||
- `skills/migrations/v0.11.0.md` — the host-agent instruction manual for
|
||||
resolving `pending-host-work.jsonl` items.
|
||||
- `docs/guides/minions-fix.md` — troubleshooting a half-migrated install.
|
||||
@@ -1,160 +0,0 @@
|
||||
---
|
||||
name: smoke-test
|
||||
description: |
|
||||
Post-restart smoke tests + auto-fix for gbrain and OpenClaw environments.
|
||||
Tests critical services, auto-fixes known issues, extensible via user-defined
|
||||
test scripts in ~/.gbrain/smoke-tests.d/*.sh.
|
||||
triggers:
|
||||
- "smoke test"
|
||||
- "run smoke tests"
|
||||
- "container restart check"
|
||||
- "health check"
|
||||
- "did the restart break anything"
|
||||
- "did the container restart break anything"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Smoke Test Skillpack
|
||||
|
||||
> Run `gbrain smoke-test` or `bash scripts/smoke-test.sh` after any container restart.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
- 8 core tests verify gbrain + OpenClaw health after restart
|
||||
- Known failures are auto-fixed before reporting
|
||||
- User-extensible via `~/.gbrain/smoke-tests.d/*.sh` drop-in scripts
|
||||
- Results logged to `/tmp/gbrain-smoke-test.log`
|
||||
- Exit code = number of unfixed failures (0 = all pass)
|
||||
|
||||
## Built-in Tests
|
||||
|
||||
| # | Test | Auto-Fix |
|
||||
|---|------|----------|
|
||||
| 1 | Bun runtime | Install from bun.sh |
|
||||
| 2 | GBrain CLI loads | Reinstall deps |
|
||||
| 3 | GBrain database (doctor) | — |
|
||||
| 4 | GBrain worker process | Start worker |
|
||||
| 5 | OpenClaw Codex plugin (Zod CJS) | `npm install zod@4 --force` |
|
||||
| 6 | OpenClaw gateway | — (may not be started yet) |
|
||||
| 7 | Embedding API key | — (check .env) |
|
||||
| 8 | Brain repo exists | — |
|
||||
|
||||
## Usage
|
||||
|
||||
### CLI
|
||||
```bash
|
||||
gbrain smoke-test
|
||||
```
|
||||
|
||||
### Direct
|
||||
```bash
|
||||
bash scripts/smoke-test.sh
|
||||
```
|
||||
|
||||
### From OpenClaw bootstrap
|
||||
Add to your `ensure-services.sh` or equivalent:
|
||||
```bash
|
||||
bash /path/to/gbrain/scripts/smoke-test.sh >> /tmp/bootstrap.log 2>&1
|
||||
```
|
||||
|
||||
### From an agent
|
||||
```
|
||||
exec: bash /data/gbrain/scripts/smoke-test.sh
|
||||
```
|
||||
|
||||
## Adding Custom Tests
|
||||
|
||||
Create executable scripts in `~/.gbrain/smoke-tests.d/`:
|
||||
|
||||
```bash
|
||||
# ~/.gbrain/smoke-tests.d/check-redis.sh
|
||||
#!/bin/bash
|
||||
redis-cli ping | grep -q PONG
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Exit 0 = pass, non-zero = fail
|
||||
- Filename becomes the test name (e.g. `check-redis` from `check-redis.sh`)
|
||||
- Keep tests fast (< 10s each)
|
||||
- Tests run in alphabetical order
|
||||
|
||||
## Adding Built-in Tests
|
||||
|
||||
Edit `scripts/smoke-test.sh`. Follow this pattern:
|
||||
|
||||
```bash
|
||||
# ── N. [Service Name] ──────────────────────────────────────
|
||||
if [test condition]; then
|
||||
pass "[Service Name]"
|
||||
else
|
||||
# Auto-fix attempt
|
||||
[fix command]
|
||||
if [re-test condition]; then
|
||||
fixed "[What was fixed]"
|
||||
pass "[Service Name] (after fix)"
|
||||
else
|
||||
fail "[Service Name] — [error detail]"
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
### Design rules:
|
||||
1. **Test first** — never fix without confirming broken
|
||||
2. **Re-test after fix** — verify the fix worked
|
||||
3. **Timeout everything** — `timeout N` on any command that could hang
|
||||
4. **Use helpers** — `pass()`, `fail()`, `fixed()`, `skip()`
|
||||
5. **Idempotent fixes** — safe to run repeatedly
|
||||
6. **Skip gracefully** — `skip()` when a prerequisite is missing, don't fail
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Var | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `GBRAIN_SMOKE_LOG` | `/tmp/gbrain-smoke-test.log` | Log file path |
|
||||
| `GBRAIN_DIR_OVERRIDE` | (auto-detect) | Force gbrain install path |
|
||||
| `GBRAIN_DATABASE_URL` | (from .env) | Database connection URL |
|
||||
| `OPENCLAW_GATEWAY_PORT` | `18789` | Gateway port to test |
|
||||
| `GBRAIN_BRAIN_PATH` | `/data/brain` | Brain repo path |
|
||||
|
||||
## Known Issues & Their Auto-Fixes
|
||||
|
||||
### Codex Zod core.cjs Missing (discovered 2026-04-23)
|
||||
- **Symptom:** `Cannot find module './core.cjs'` → all Codex ACP sessions fail
|
||||
- **Cause:** Zod v4 npm package ships without `core.cjs` in some installs
|
||||
- **Auto-fix:** `npm install zod@4 --force` in the codex extension's zod dir
|
||||
- **Persistence:** Does NOT survive container restart (gateway reinstalls deps)
|
||||
- This is why smoke tests must run on every restart
|
||||
|
||||
### GBrain Worker Auth Failure
|
||||
- **Symptom:** Worker can't connect to DB
|
||||
- **Cause:** `GBRAIN_DATABASE_URL` not propagated to worker subprocess
|
||||
- **Auto-fix:** Script explicitly passes both `DATABASE_URL` and `GBRAIN_DATABASE_URL`
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- ❌ Running smoke tests on every chat turn. Once per container restart (or
|
||||
on user request) is plenty. The script is cheap but it's not free.
|
||||
- ❌ Writing a user drop-in without `timeout N` around any command that
|
||||
could hang. A single hung drop-in stalls every subsequent run.
|
||||
- ❌ Auto-fixing without confirming the check is actually broken first.
|
||||
The `pass → fail-detected → fix → re-test` loop is the contract; fixes
|
||||
that skip the re-test can report success on a still-broken state.
|
||||
- ❌ Treating `skip` as `fail`. Missing prerequisites (no OpenClaw installed,
|
||||
no brain repo configured) are skips, not failures. Exit code = count of
|
||||
real failures, not skipped checks.
|
||||
- ❌ Hardcoding paths in a user drop-in. Read env vars
|
||||
(`GBRAIN_DATABASE_URL`, `HOME`, etc.) so the script travels across
|
||||
container rebuilds.
|
||||
|
||||
## Output Format
|
||||
|
||||
The script writes a one-line status per check to stdout (✅/❌/🔧/⏭️) plus a
|
||||
final summary line: `Results: N/M passed, F auto-fixed, S skipped`. A
|
||||
structured timestamped log appends to `$GBRAIN_SMOKE_LOG`
|
||||
(default `/tmp/gbrain-smoke-test.log`) for post-run forensics. Exit code
|
||||
equals the count of unfixed failures (0 = all pass, positive integer =
|
||||
count of remaining failures).
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user