mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-16 09:52:22 +00:00
Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Wintermute
|
acdee0905a | ||
|
|
55ca4984b2 | ||
|
Wintermute
|
35967645f3 | ||
|
Garry TanandClaude Opus 4.7
|
dcd13dd638 | ||
|
|
96178d726e | ||
|
Garry TanandClaude Opus 4.7
|
418d955fd3 | ||
|
Garry TanandClaude Opus 4.7
|
0e9f8814a5 | ||
|
|
fcf40a12fc | ||
|
Garry TanandClaude Opus 4.7
|
a4df40fe5c | ||
|
Garry TanandClaude Opus 4.7
|
ff10796a00 | ||
|
Garry TanandClaude Opus 4.7
|
7f156c8873 |
@@ -28,4 +28,4 @@ jobs:
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: bun install
|
||||
- run: bun test
|
||||
- run: bun run test
|
||||
|
||||
+3
-2
@@ -5,8 +5,9 @@ bin/
|
||||
.env
|
||||
.env.*
|
||||
!.env.*.example
|
||||
.18a49dfd730ff378-00000000.bun-build
|
||||
.18a49f9dfb996f70-00000000.bun-build
|
||||
# Bun --compile temp artifacts. Each build emits a new hash-named .bun-build
|
||||
# file in cwd; glob catches all of them.
|
||||
*.bun-build
|
||||
.gstack/
|
||||
supabase/.temp/
|
||||
.claude/skills/
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# Agents working on GBrain
|
||||
|
||||
This is your install + operating protocol. Claude Code reads `./CLAUDE.md` automatically.
|
||||
Everyone else (Codex, Cursor, OpenClaw, Aider, Continue, or an LLM fetching via URL):
|
||||
start here.
|
||||
|
||||
## Install (5 min)
|
||||
|
||||
1. Clone: `git clone https://github.com/garrytan/gbrain ~/gbrain && cd ~/gbrain`
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
1. `./AGENTS.md` (this file) — install + operating protocol.
|
||||
2. [`./CLAUDE.md`](./CLAUDE.md) — architecture reference, key files, trust boundaries,
|
||||
test layout.
|
||||
3. [`./skills/RESOLVER.md`](./skills/RESOLVER.md) — skill dispatcher. Read before any task.
|
||||
|
||||
## Trust boundary (critical)
|
||||
|
||||
GBrain distinguishes **trusted local CLI callers** (`OperationContext.remote = false`,
|
||||
set by `src/cli.ts`) from **untrusted agent-facing callers** (`remote = true`, set by
|
||||
`src/mcp/server.ts`). Security-sensitive operations like `file_upload` tighten filesystem
|
||||
confinement when `remote = true` and default to strict behavior when unset. If you are
|
||||
writing or reviewing an operation, consult `src/core/operations.ts` for the contract.
|
||||
|
||||
## Common tasks
|
||||
|
||||
- **Configure:** [`docs/ENGINES.md`](./docs/ENGINES.md),
|
||||
[`docs/guides/live-sync.md`](./docs/guides/live-sync.md),
|
||||
[`docs/mcp/DEPLOY.md`](./docs/mcp/DEPLOY.md).
|
||||
- **Debug:** [`docs/GBRAIN_VERIFY.md`](./docs/GBRAIN_VERIFY.md),
|
||||
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
|
||||
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
## Before shipping
|
||||
|
||||
Run `bun test` plus the E2E lifecycle described in `./CLAUDE.md` (spin up the test
|
||||
Postgres container, run `bun run test:e2e`, tear it down). Ship via the `/ship` skill,
|
||||
not by hand.
|
||||
|
||||
## Privacy
|
||||
|
||||
Never commit real names of people, companies, or funds into public artifacts. See the
|
||||
Privacy rule in `./CLAUDE.md`. GBrain pages reference real contacts; public docs must
|
||||
use generic placeholders (`alice-example`, `acme-example`, `fund-a`).
|
||||
|
||||
## Forks
|
||||
|
||||
If you are a fork, regenerate `llms.txt` + `llms-full.txt` with your own URL base before
|
||||
publishing: `LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms`.
|
||||
+758
-7
@@ -2,6 +2,757 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.17.0] - 2026-04-22
|
||||
|
||||
## **`gbrain dream`. Run the brain maintenance cycle while you sleep.**
|
||||
## **One primitive, two CLIs. Autopilot gains lint + orphan sweep automatically.**
|
||||
|
||||
The README has promised "the dream cycle" for a year. v0.17 makes it real as a first-class command. `gbrain dream` runs one maintenance cycle and exits, designed for cron. Same six phases as `gbrain autopilot` — they both delegate to the new `runCycle` primitive in `src/core/cycle.ts`. One source of truth for what your brain does overnight.
|
||||
|
||||
Phase order is semantically driven: **fix files first, then index them**. Lint and backlinks write to disk. Sync picks them up into the DB. Extract links the graph. Embed refreshes vectors. Orphan sweep reports the gaps. If your autopilot daemon was doing sync-before-lint (which PR #309's original dream.ts also got wrong), your fixes landed the next cycle instead of the current one. Fixed.
|
||||
|
||||
Autopilot users upgrading get lint + orphan sweep for free. No config change. `gbrain jobs list` shows the full 6-phase report now. If you don't want the daemon modifying files, `gbrain dream --phase orphans` in cron keeps autopilot for embed+sync and gives you manual control over the writes.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured against a v0.16 baseline. Lines-of-code delta is net-small: runCycle adds ~500 lines, but the new dream.ts is 80 lines (vs the 446-line original in PR #309), and autopilot's two-path branching collapses to one delegated call.
|
||||
|
||||
| Metric | BEFORE v0.17 | AFTER v0.17 | Δ |
|
||||
|--------|--------------|-------------|---|
|
||||
| `gbrain dream --dry-run` mutates DB | Yes (full-sync + embed silently wrote) | No (every phase honors dry-run) | correctness |
|
||||
| Sources of truth for "the cycle" | 3-4 (dream inline, dream shell-outs, autopilot inline, Minions handler) | 1 (`runCycle`) | DRY win |
|
||||
| Phase order: fix-then-index | No (sync before lint) | Yes (lint → backlinks → sync → extract → embed → orphans) | semantics |
|
||||
| Coordination across daemon + cron + Minions worker | Lockfile heuristic with 6 known holes | DB lock table + PID-liveness file lock | primitive upgrade |
|
||||
| Works under PgBouncer transaction pooling | No (session-scoped `pg_try_advisory_lock`) | Yes (TTL row, refreshed between phases) | Supabase-safe |
|
||||
| `findRepoRoot` walks into wrong git repo | Yes (10 levels of cwd) | No (explicit --dir OR configured sync.repo_path) | footgun fixed |
|
||||
| Autopilot daemon phase count | 4 (sync+extract+embed+backlinks in Minions mode; no backlinks inline) | 6 (+lint +orphans) | feature parity |
|
||||
| CycleReport shape stability for agents | N/A | `schema_version: "1"` (stable, additive only) | API contract |
|
||||
|
||||
### What this means for your workflow
|
||||
|
||||
Cron users: one line. `0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log`. You get a structured `CycleReport` every morning with per-phase timing, counts, and any errors tagged with `{class, code, message, hint, docs_url}`.
|
||||
|
||||
Autopilot users: nothing to do. Your daemon picks up the new phases on next cycle. If you want to see them: `gbrain jobs get <autopilot-cycle-id>` shows the full report.
|
||||
|
||||
Reviewers/codex caught three plan-breakers during multi-round review that would have shipped silent DB writes on dry-run: (1) `performSync`'s full-sync path was ignoring `opts.dryRun`, (2) `runEmbedCore` had no dry-run mode and returned void, (3) `findOrphans` used `db.getConnection()` global and didn't compose with a passed engine. All three are fixed as preconditions (commits 1-3 of the 6-commit bisectable series).
|
||||
|
||||
Credit: @Wintermute for the original `gbrain dream` thesis (PR #309). The brand-promise framing survived; the implementation got redesigned from scratch around the runCycle primitive after CEO + Eng + Codex + DX review found structural issues.
|
||||
|
||||
## To take advantage of v0.17.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the migration orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Your agent reads `skills/migrations/v0.17.0.md` the next time you interact with it.** No mechanical host-repo action required; the schema migration (v16 cycle-lock table) and the behavior shift in autopilot's inline path both apply automatically.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain dream --help # new command exists
|
||||
gbrain dream --dry-run --json # safe preview
|
||||
gbrain doctor # should show no pending migrations
|
||||
```
|
||||
Autopilot users: `gbrain jobs list --status complete | head -5` and inspect an `autopilot-cycle` job with `gbrain jobs get <id>` — the report now includes 6 phases.
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**New CLI command: `gbrain dream`**
|
||||
- One-shot maintenance cycle for cron. Exits when done. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`, `--help`.
|
||||
- `--help` shows cron example + cross-reference to `autopilot --install` for continuous daemon.
|
||||
- Empty-state output is intentionally satisfying: `Brain is healthy. 6 phase(s) checked in 2.3s.` Agents detect it via `status: "clean"`.
|
||||
- Exit code 1 on `status: "failed"`. Warnings (`status: "partial"`) are not failures — don't page someone.
|
||||
- `--dir` OR `sync.repo_path` config required. No more walk-up-cwd-for-.git footgun.
|
||||
|
||||
**New primitive: `src/core/cycle.ts`**
|
||||
- `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>`.
|
||||
- Six phases in order: lint → backlinks → sync → extract → embed → orphans.
|
||||
- `CycleReport` has `schema_version: "1"` (stable, additive). `status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'` with `reason` field on skipped.
|
||||
- `PhaseResult.error: { class, code, message, hint?, docs_url? }` on fail. Stripe-API-tier structured errors.
|
||||
- `yieldBetweenPhases` hook awaited between every phase + before return. Required for Minions worker lock renewal. Exceptions non-fatal.
|
||||
- Engine nullable — filesystem phases run without DB; DB phases skip with `reason: "no_database"`.
|
||||
- Lock-skip: read-only phase selections (`--phase orphans`) skip lock acquisition.
|
||||
|
||||
**New schema: `gbrain_cycle_locks` (migration v16)**
|
||||
- DB lock table with TTL (30 min), replaces session-scoped `pg_try_advisory_lock` which the v0.15.4 PgBouncer-transaction-pooler fix silently broke.
|
||||
- Refreshed between phases via the yield hook. Crashed holders auto-release on TTL expiry.
|
||||
- PGLite + engine=null use a file-based fallback at `~/.gbrain/cycle.lock` with PID-liveness check (EPERM treated as alive so PID 1 holders aren't mis-classified).
|
||||
|
||||
**Autopilot + Minions integration**
|
||||
- Autopilot's inline fallback path (`--inline` flag + PGLite mode) now delegates to `runCycle`. Gains lint + orphan phases it didn't run before. Uses `pull: true` by default (preserves pre-v0.17 pull semantics).
|
||||
- Minions `autopilot-cycle` handler (in `src/commands/jobs.ts`) also delegates to `runCycle`. Returns `{ partial, status, report }` so `gbrain jobs get <id>` surfaces the full structured report.
|
||||
- `gbrain autopilot --install` install/uninstall/launchd/systemd/crontab machinery untouched.
|
||||
- `gbrain autopilot --help` now cross-references `gbrain dream`.
|
||||
|
||||
**Precondition fixes (required for the runCycle primitive to compose cleanly)**
|
||||
- `src/commands/sync.ts`: `performFullSync` honors `opts.dryRun` in first-sync + `--full` paths. Was silently calling `runImport` regardless. `SyncResult.embedded: number` field added; `first_sync` path now returns real counts from `runImport` (was hardcoded to 0).
|
||||
- `src/commands/embed.ts`: `runEmbedCore` adds `dryRun?: boolean` opt and returns `EmbedResult { embedded, skipped, would_embed, total_chunks, pages_processed, dryRun }` instead of `void`. `gbrain embed --stale --dry-run` is now a safe preview.
|
||||
- `src/commands/orphans.ts`: `findOrphans(engine, opts)` takes a `BrainEngine` parameter. Added `findOrphanPages()` method to `BrainEngine` interface + implementations on both `postgres-engine` and `pglite-engine`. Drops `db.getConnection()` global — findOrphans now composes with test-injected engines and works on PGLite.
|
||||
|
||||
**Tests (all run in CI, no DATABASE_URL or API keys required)**
|
||||
- `test/sync.test.ts`: 4 new cases. First-sync dry-run, incremental dry-run, `--full` dry-run, SyncResult.embedded shape. PGLite + temp git repo.
|
||||
- `test/embed.test.ts`: 4 new cases. Dry-run with stale chunks, dry-run stale-vs-fresh split, dry-run --slugs, non-dry-run regression guard. Mocked `embedBatch`.
|
||||
- `test/orphans.test.ts`: 4 new cases. Engine-injected findOrphans, includePseudo flag, queryOrphanPages delegation, empty-brain edge. PGLite.
|
||||
- `test/core/cycle.test.ts` (new): 18 cases covering dryRun × phases × lock_held × engine-null. Shared PGLite engine per describe via beforeAll + truncateCycleLocks (cuts test time ~3x vs per-test init).
|
||||
- `test/dream.test.ts` (rewritten, 11 cases): brainDir resolution, phase selection, phase validation, JSON output shape, dry-run propagation, exit-code semantics. Real PGLite + real library calls (no `mock.module` to avoid leakage).
|
||||
|
||||
**Docs**
|
||||
- `skills/migrations/v0.17.0.md`: new. Informational, no mechanical action required.
|
||||
- `CHANGELOG.md` + `CLAUDE.md`: updated.
|
||||
|
||||
**PR #309 disposition**
|
||||
- Closed with credit to @Wintermute. Their thesis ("`gbrain dream` as first-class CLI verb") was right; the implementation got redesigned around the runCycle primitive after deep review surfaced structural issues in the fold approach.
|
||||
- `Co-Authored-By: Wintermute` preserved on commit 5 (the dream.ts rewrite).
|
||||
|
||||
---
|
||||
|
||||
## [0.16.4] - 2026-04-22
|
||||
|
||||
## **`gbrain check-resolvable` ships. The command the README promised for weeks.**
|
||||
## **Agents and CI finally have a one-shot skill-tree gate that actually exits non-zero when anything is off.**
|
||||
|
||||
The `resolver_health` logic has lived inside `gbrain doctor` since v0.11. The README claimed a standalone `gbrain check-resolvable` shipped too ... it didn't. Scripts referenced it. Skillify's 10-item checklist referenced it. The binary just shrugged. Fixed.
|
||||
|
||||
`gbrain check-resolvable` runs the same four checks doctor runs (reachability, MECE overlap, MECE gap, DRY violations) but with a stricter contract: **exits 1 on any issue, errors AND warnings**. Doctor's resolver_health block still exits 0 on warnings-only because doctor has 15 other checks to lean on. The standalone command has nowhere to hide. CI can finally gate on a single command instead of parsing `gbrain doctor --json`.
|
||||
|
||||
The JSON output is a stable envelope, one shape for success and error: `{ok, skillsDir, report, autoFix, deferred, error, message}`. No more "did it succeed? let me see which keys are present." The `deferred` array names the two checks still pending (trigger routing eval, brain filing) with links to their tracking issues, so agents reading the JSON know the current coverage boundary.
|
||||
|
||||
`scripts/skillify-check.ts` is now machine-gated. Item #8 on the skillify 10-item checklist used to print "run: gbrain check-resolvable" and pass unconditionally. Now it subprocess-calls the real command and asserts on the exit code. Binary-missing fails loud instead of silently passing ... the kind of silent false-pass that used to put broken skills on the shelf.
|
||||
|
||||
## To take advantage of v0.16.4
|
||||
|
||||
No migration needed. `gbrain upgrade` brings the binary; nothing to apply. Try it:
|
||||
|
||||
```bash
|
||||
gbrain check-resolvable # human output, like doctor's resolver section
|
||||
gbrain check-resolvable --json | jq .ok # machine-readable gate for CI
|
||||
gbrain check-resolvable --fix --dry-run # preview DRY auto-fixes without writing
|
||||
```
|
||||
|
||||
Wire it into your CI:
|
||||
|
||||
```bash
|
||||
gbrain check-resolvable || exit 1 # fails the build on any warning/error
|
||||
```
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**New command**
|
||||
- `gbrain check-resolvable [--json] [--fix] [--dry-run] [--verbose] [--skills-dir PATH] [--help]` — standalone skill-tree gate. Covers reachability, MECE overlap, MECE gap, DRY violations. Exits 1 on any issue.
|
||||
- Stable JSON envelope (`ok`, `skillsDir`, `report`, `autoFix`, `deferred`, `error`, `message`) — one shape for both success and error paths.
|
||||
- `--fix` auto-applies DRY fixes via `autoFixDryViolations` before re-checking (same ordering as `doctor --fix`).
|
||||
- `--dry-run` with `--fix` previews without writing; the JSON `autoFix.fixed` array shows what would change.
|
||||
- `--verbose` prints the Deferred checks note with issue URLs so nobody forgets Checks 5 and 6 are still tracked.
|
||||
|
||||
**Deferred to separate issues**
|
||||
- Check 5: trigger routing eval — verify every skill's own frontmatter trigger routes to itself in RESOLVER.md. Surfaced via the CLI's `deferred[]` output block.
|
||||
- Check 6: brain filing validation — verify mutating skills register the brain directories they write to. Same surface.
|
||||
|
||||
**Shared refactor**
|
||||
- `src/core/repo-root.ts` — extracted `findRepoRoot()` from `doctor.ts` to a zero-dependency shared module with a parameterized `startDir` for test hermeticity. Doctor imports the shared version; no behavior change (default arg matches prior semantics).
|
||||
- `src/commands/doctor.ts` — updated to import the shared `findRepoRoot`.
|
||||
|
||||
**Skillify integration**
|
||||
- `scripts/skillify-check.ts` — item #8 ("check-resolvable gate") now subprocess-calls `gbrain check-resolvable --json` and gates on the exit code. Result is cached per process so iterating many skills only runs the subprocess once. Binary-missing fails loud via explicit `spawn` error handling ... no silent false-pass.
|
||||
|
||||
**Tests (22 new cases)**
|
||||
- `test/repo-root.test.ts` — 4 cases for the extracted `findRepoRoot()` (first-iter hit, walks up, returns null, default arg behavioral parity).
|
||||
- `test/check-resolvable-cli.test.ts` — 17 cases split between direct unit tests (flag parsing, resolveSkillsDir, DEFERRED constants) and subprocess integration tests (help, JSON envelope shape, exit-code regression gates for warnings AND errors, `--fix --dry-run` wiring, `--verbose` output).
|
||||
- `test/skillify-check.test.ts` — 2 new cases for the check-resolvable wiring: loud failure when binary is missing (no silent pass), happy path when a synthetic gbrain returns `ok: true`.
|
||||
|
||||
**Contract note for CI users**
|
||||
- `gbrain check-resolvable` exits 1 on warnings AND errors. `gbrain doctor`'s resolver_health block still exits 0 on warnings-only. If you scripted against doctor's looser gate, `check-resolvable` will bite harder ... on purpose. This honors the README:259 contract: "Exits non-zero if anything is off."
|
||||
|
||||
---
|
||||
|
||||
## [0.16.3] - 2026-04-22
|
||||
|
||||
## **`gbrain agent run` actually runs now. The subagent SDK wiring that shipped broken in v0.16.0 is fixed.**
|
||||
## **Every `.ts` file in the repo typechecks on every `bun run test`. Silent regressions end here.**
|
||||
|
||||
v0.16.0 shipped with the headline feature, `gbrain agent run`, unable to make a single LLM call. `makeSubagentHandler` cast `new Anthropic()` straight to `MessagesClient`, but the SDK exposes `.create()` at `sdk.messages.create`, not on the top-level client. Every subagent job in production died on the first call with `client.create is not a function`. The type system would have caught it. Nothing was running the type system.
|
||||
|
||||
The root cause isn't the casting bug. It's that `bun test` transpiles TypeScript without type-checking it, and `bun test` was the entire CI pipeline. Invalid types ran until they hit runtime. This release fixes the symptom (one-line change, `deps.client ?? new Anthropic().messages`, which typechecks cleanly against `MessagesClient` because `sdk.messages` IS the right object) and closes the hole that let it ship (`tsc --noEmit` now runs on every `bun run test`, and the CI workflow runs `bun run test` not `bun test`). Two independent guards: anyone reverting to `new Anthropic()` fails the type check; a new regression test drives one handler turn through an injected fake SDK and fails loudly if the factory default branch breaks.
|
||||
|
||||
Closing the CI gap surfaced 100+ pre-existing type errors across 30+ files: `databaseUrl` → `database_url` rename drift, missing `"meeting"` / `"note"` entries in the `PageType` union that both src and tests already used, a Buffer-as-BodyInit assignment in the Supabase uploader, dead-code comparisons against narrowed status types in the migration orchestrators, and several `as X` casts that TS 5.6 requires be spelled `as unknown as X`. All cleaned up. The first tsc run is green.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
From the merged branch after both the fix and the infra cleanup landed locally against master.
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|---|---|---|---|
|
||||
| `bun run typecheck` errors | 104 | 0 | -104 |
|
||||
| `gbrain agent run` in prod | 100% failure on first LLM call | Works | ✅ |
|
||||
| Test file count | ~75 | ~75 (+1 regression test block) | +1 |
|
||||
| `bun run test` pass rate | 1962 pass / 4 fail (PGLite flake under parallel load) | 1997 pass / 0 fail | +35 pass, -4 fail |
|
||||
| CI test-gate steps | `bun test` (no type check) | `bun run test` (jsonb guard + progress-to-stdout guard + `tsc --noEmit` + `bun test`) | 1→4 |
|
||||
| Regression guards on this bug class | 0 | 2 (compile-time via `tsc`, runtime via `makeAnthropic` injection test) | +2 |
|
||||
|
||||
The 104 → 0 isn't a refactor. Every error was a real correctness signal TS had been trying to send that nobody was listening for. Most were trivial to fix (`as unknown as X`, one missing union member, one rename propagation). The Buffer/BodyInit one in Supabase upload is a live bug — `fetch(url, {body: buf})` works today in Node/Bun but has no type guarantee; the fix copies `data.buffer, data.byteOffset, data.byteLength` into a `Uint8Array` slice that is genuinely assignable to `BodyInit`.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
`gbrain agent run "say hello"` against a Supabase brain completes end-to-end after this upgrade. No stuck subagent jobs, no `client.create is not a function` traceback. v0.16.0 users should upgrade immediately — the feature that release was named for did not work.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### `gbrain agent run` now works against the real Anthropic SDK
|
||||
|
||||
- `src/core/minions/handlers/subagent.ts` — factory default construction replaced with `const client: MessagesClient = deps.client ?? makeAnthropic().messages`. The SDK's `Messages` resource is already the right object; no helper, no wrapper, no `.bind()` needed (method-call semantics preserve `this`). `const makeAnthropic = deps.makeAnthropic ?? (() => new Anthropic())` adds a dependency-injection seam so tests can exercise the default branch without a real API key or network call.
|
||||
- `test/subagent-handler.test.ts` — new `describe('makeSubagentHandler default client construction')` block drives a full handler turn through a fake SDK injected via `makeAnthropic`. If anyone reverts `.messages` or reintroduces a `new Anthropic()` top-level cast, this test fails loudly.
|
||||
|
||||
#### CI type-checking is now real
|
||||
|
||||
- `package.json` — added `typescript@^5.6.0` as devDep; added `"typecheck": "tsc --noEmit"` script; chained `bun run typecheck` into `"test"` so local `bun run test` and CI run identical pipelines (grep guards + typecheck + bun test).
|
||||
- `.github/workflows/test.yml` — CI now runs `bun run test` (the npm script) instead of `bun test` (the runner). One line. Biggest-leverage change in the release.
|
||||
|
||||
#### 100+ pre-existing type errors cleaned up
|
||||
|
||||
So `tsc --noEmit` actually stays green. All mechanical, zero behavior change. Groups:
|
||||
|
||||
- **`databaseUrl` → `database_url` rename drift** in 9 test fixtures (test/agent-cli, test/brain-allowlist, test/minions-shell, test/minions, test/queue-child-done, test/rate-leases, test/subagent-handler, test/subagent-transcript, test/wait-for-completion).
|
||||
- **`PageType` union** in `src/core/types.ts` gained `'meeting'` and `'note'` entries. Both were already used in src (`link-extraction.ts` had a code comment acknowledging the gap) and across 6 test files. The union was just out of date.
|
||||
- **`GBrainConfig.storage`** field declared in `src/core/config.ts` — the code at `src/commands/files.ts` and `src/core/operations.ts` was reading `config.storage` with 18 inferred-type errors.
|
||||
- **`ErrorCode`** union in `src/core/operations.ts` gained `'permission_denied'`; the code was throwing this exact string but the union disagreed.
|
||||
- **Dead-code comparisons** removed from `src/commands/migrations/v0_12_0.ts`, `v0_12_2.ts`, `v0_13_0.ts`, `v0_16_0.ts` — each orchestrator had an early-return on `a.status === 'failed'` followed later by a redundant check against a then-narrowed type. TS correctly flagged the later check as always-false.
|
||||
- **postgres.js `Row` callback typing** on `src/core/postgres-engine.ts` — 6 `.map((r: { slug: string }) => r.slug)` callbacks rewritten as `.map((r) => r.slug as string)` to match postgres.js's `Row` generic. Same behavior, correct signature.
|
||||
- **Buffer → BodyInit** in `src/core/storage/supabase.ts:58,129` — `body: data` (Buffer) replaced with `body: new Uint8Array(data.buffer, data.byteOffset, data.byteLength) as BodyInit`. Zero-copy view of the same bytes, structurally assignable to `BodyInit`, no runtime change.
|
||||
- **Various `as X` casts** upgraded to `as unknown as X` where TS 5.6's stricter structural-conversion rules rejected the single-step cast. Affected: `src/core/file-resolver.ts` (3), `src/core/minions/handlers/subagent-aggregator.ts`, `src/core/minions/worker.ts`, `src/commands/orphans.ts`, `src/commands/repair-jsonb.ts`, `src/core/postgres-engine.ts` (2 RowList → array conversions).
|
||||
|
||||
#### Test suite stability
|
||||
|
||||
- `bunfig.toml` — new file. Sets `[test].timeout = 60_000` globally. PGLite WASM init is slow enough that the default 5-second hook timeout flakes when many test files spin up PGLite instances in parallel on a loaded machine.
|
||||
- 8 test files (`test/wait-for-completion`, `test/extract-fs`, `test/subagent-handler`, `test/minions-shell`, `test/minions-quiet-hours`, `test/integrity`, `test/e2e/graph-quality`, `test/e2e/search-quality`) additionally declare `beforeAll(fn, 60_000)` / `beforeEach(fn, 15_000)` as explicit safety nets — redundant with `bunfig.toml` today, but stays as belt-and-suspenders if the bunfig schema ever changes.
|
||||
|
||||
## To take advantage of v0.16.3
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about anything:
|
||||
|
||||
1. **Verify your brain still runs:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
2. **Verify the agent runtime works:**
|
||||
```bash
|
||||
gbrain agent run "say hello"
|
||||
```
|
||||
Should complete end-to-end. If it fails with `client.create is not a function`, the upgrade didn't land — run `gbrain upgrade` again.
|
||||
3. **No migrations required.** No schema changes in this release. Fix is in the handler code, not the DB.
|
||||
4. **If any step fails,** please file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- output of `gbrain agent run "say hello"`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
|
||||
### Itemized changes
|
||||
|
||||
---
|
||||
|
||||
## [0.16.2] - 2026-04-22
|
||||
|
||||
## **The deployment guide now reads like a runbook an agent can execute line-by-line.**
|
||||
## **Three real bugs from v0.16.1 fixed, nine DX gaps closed.**
|
||||
|
||||
v0.16.1 shipped the Minions worker deployment guide. Re-reading it as the agent it was written for, top-to-bottom, copy-pasting every block, surfaced twelve issues a human skim-reader would not catch. Three are real bugs that break a first-time deploy. Nine are structural gaps that force the agent to invent values.
|
||||
|
||||
The bugs: the crontab example used `*/5 * * * * user bash /path/...` which is `/etc/crontab` format only, so an agent running `crontab -e` and pasting it got "bad minute" or parsed `user` as the command. The watchdog script grepped `tail -20` of an unrotated log for shutdown markers, so every 5-minute tick after the first restart re-matched the old shutdown line forever and killed the healthy worker on loop. And `DATABASE_URL=postgresql://user:pass@...` lived directly in `/etc/crontab`, which is mode 644 (world-readable).
|
||||
|
||||
The gaps: no preconditions block, no "which option should I pick" selector, hardcoded `/path/to/...` and `/my/workspace` throughout with no template-variable legend, no upgrade section (so an agent coming from v0.13.x had no idea `GBRAIN_ALLOW_SHELL_JOBS=1` is now required or that `max_stalled` flipped from 1 to 5), no alternative to bare cron for Fly/Render/systemd deployments, a "Proposed CLI flags (not yet implemented)" block that an agent would copy and get `unrecognized flag`, and a `MinionWorker.maxStalledCount` note that did not tell the agent what to do.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
The guide is now copy-pasteable without invention. Every `$VAR` is documented in a table at the top. Every code block runs as-is on the target it claims. The watchdog writes a two-line PID file (PID + restart epoch) and the shutdown check only considers log lines newer than the epoch, which is the actual fix for the restart loop. Secrets live in `/etc/gbrain.env` (mode 600), referenced via `BASH_ENV=/etc/gbrain.env` in crontab. A new Option 3 ships a systemd unit, a Procfile, and a fly.toml fragment so Fly/Render/Railway/systemd users skip cron entirely. The upgrade section walks the v0.13.x → v0.16.2 checklist (stop worker, apply migrations, add `GBRAIN_ALLOW_SHELL_JOBS`, swap the watchdog).
|
||||
|
||||
The shipped watchdog was verified against an abbreviated end-to-end test (3 ticks in ~30 seconds inside an Ubuntu 22.04 container): tick 1 starts the worker and writes the 2-line PID file; tick 2 sees a shutdown line with a 1-hour-old timestamp and correctly does nothing; tick 3 sees a fresh shutdown line and correctly restarts. The regex was caught and fixed during the test when mawk rejected `{n}` interval quantifiers. The systemd unit was smoked in a privileged container with `Restart=always` firing a second banner after a 10-second `RestartSec` window, confirming crash-recovery works before any host ever boots the unit.
|
||||
|
||||
## To take advantage of v0.16.2
|
||||
|
||||
`gbrain upgrade` pulls the new guide. If you deployed under v0.16.1 with the original watchdog, swap it:
|
||||
|
||||
1. **Re-read the guide:**
|
||||
```bash
|
||||
less docs/guides/minions-deployment.md
|
||||
```
|
||||
2. **Swap the watchdog script.** The v0.16.1 version has the restart-loop bug:
|
||||
```bash
|
||||
sudo install -m 755 docs/guides/minions-deployment-snippets/minion-watchdog.sh \
|
||||
/usr/local/bin/minion-watchdog.sh
|
||||
```
|
||||
3. **Move secrets out of crontab.** Put `DATABASE_URL` and `GBRAIN_ALLOW_SHELL_JOBS=1` into `/etc/gbrain.env` (mode 600), reference it from crontab via `BASH_ENV=/etc/gbrain.env`.
|
||||
4. **Fix the cron form.** If you pasted the v0.16.1 `*/5 * * * * user bash ...` into `crontab -e`, drop the `user` column and the explicit `bash` prefix.
|
||||
5. **If you have shell access to a long-running box,** consider Option 3 (systemd) instead of Option 1 (watchdog). systemd replaces the watchdog entirely and is the cleanest path.
|
||||
|
||||
No schema change. No data migration. Docs + snippets only.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Fixed**
|
||||
- **Crontab syntax now matches the target.** Two labeled blocks: 5-field for `crontab -e`, 6-field with user column for `/etc/crontab`. An agent no longer hits "bad minute" or has `user` parsed as the command.
|
||||
- **Watchdog restart loop killed.** The shipped `minion-watchdog.sh` writes a two-line PID file (PID on line 1, restart epoch on line 2) and only considers log lines whose ISO-8601 timestamp is newer than the epoch. Stale shutdown lines from earlier restarts no longer re-match every 5 minutes forever. Regex rewritten to use explicit `[0-9][0-9][0-9][0-9]` instead of `{4}` intervals because mawk (Debian/Ubuntu's default awk) rejects interval quantifiers. Verified end-to-end in a 3-tick abbreviated test inside Ubuntu 22.04.
|
||||
- **Credentials off the world-readable filesystem.** Secrets move to `/etc/gbrain.env` (mode 600, owned by the worker user), referenced via `BASH_ENV=/etc/gbrain.env` in crontab. `/etc/crontab` is mode 644 and user crontabs under `/var/spool/cron/` are readable by root. A new `gbrain.env.example` ships in-repo with the full env surface.
|
||||
|
||||
**Added**
|
||||
- **Preconditions block.** Five checks at the top of the guide: `gbrain` on PATH, DB connectivity, schema version, crontab write access, and the `GBRAIN_ALLOW_SHELL_JOBS=1` requirement for shell-job workers. Agent fails fast on setup, not content.
|
||||
- **Decision tree.** "Which option?" selector at the top of the deployment section. Subagent workloads and long jobs take Option 1. Scheduled scripts take Option 2. No shell access take Option 3. Replaces the previous "recommended for X" prose that forced re-reading.
|
||||
- **Template variable table.** Six variables (`$GBRAIN_BIN`, `$GBRAIN_WORKER_USER`, `$GBRAIN_WORKER_PID_FILE`, `$GBRAIN_WORKER_LOG_FILE`, `$GBRAIN_WORKSPACE`, `$GBRAIN_ENV_FILE`) with meaning and typical value. Agent substitutes once, everything downstream lands correctly.
|
||||
- **Upgrade section.** v0.13.x → v0.16.2 checklist: stop the worker, run migrations, add `GBRAIN_ALLOW_SHELL_JOBS=1` for shell jobs, handle the `max_stalled` default flip from 1 to 5, swap the v0.16.1 watchdog for the current one.
|
||||
- **Option 3: service manager.** New `systemd.service`, `Procfile`, and `fly.toml.partial` ship under `docs/guides/minions-deployment-snippets/`. systemd replaces the watchdog entirely with `Restart=always` + `RestartSec=10s` and runs the worker as an unprivileged user with `PrivateTmp`, `ProtectSystem=strict`, and `ReadWritePaths`. Smoked end-to-end in a privileged container: banner fired twice across a 10-second restart cycle, `Restart=always` honored, unit enabled for boot persistence.
|
||||
- **Uninstall section.** One-paragraph rollback for each option.
|
||||
- **`docs/guides/minions-deployment.md` listed in `scripts/llms-config.ts`.** Remote agents fetching `llms.txt` or `llms-full.txt` now see the deployment guide without having to guess its path.
|
||||
|
||||
**Changed**
|
||||
- **`--follow` example uses a gbrain subcommand, not `node my-script.mjs`.** The new example submits `gbrain embed --stale` as a shell job on a dedicated queue with `--timeout-ms 600000`. Maps directly onto how an OpenClaw-style agent actually schedules brain maintenance.
|
||||
- **"Proposed CLI flags (not yet implemented)" dead-end removed.** Replaced with a "Tune per-job today" callout pointing at the `gbrain jobs submit` flags that exist in source (`--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — all first-class since v0.13.1).
|
||||
- **Known Issues rewritten as imperatives.** "DO NOT pass `maxStalledCount` to `MinionWorker`" leads the paragraph, followed by the reason and the correct knob (`gbrain jobs submit --max-stalled N`). Zombie-shell-children section leads with the 10s / 30s numbers and the action.
|
||||
|
||||
Contributed by garrytan (issue report), fixes verified by an abbreviated end-to-end test suite (render-check + watchdog 3-tick + systemd container smoke + `bun test` + full E2E DB lifecycle).
|
||||
|
||||
## [0.16.1] - 2026-04-22
|
||||
|
||||
## **Minions worker deployment, finally documented.**
|
||||
## **If you run `gbrain jobs work` in production, there's now a guide for the sharp edges.**
|
||||
|
||||
Garry's OpenClaw (gbrain's own instance, out there actually running `gbrain jobs work` in production) wrote a real deployment guide for the Minions worker, the piece of gbrain most operators hit next after getting sync running. Agents dogfooding the project they live on is a weird, good feedback loop. Two patterns: a watchdog cron for persistent workers, and an inline `--follow` for cron-only workloads. It covers the connection-drop, stall-detector, and zombie-child traps that show up once your brain is actually working for you. Every command and every default in the guide is checked against current source (`max_stalled = 5`, not 1 or 3; `--follow` exits on submitted-job-terminal, not queue-empty; stalled jobs show up as `active`, not `waiting`). Nothing about this was obvious, and nothing about it was in the docs before.
|
||||
|
||||
With v0.16.0's durable agent runtime now shipping, the persistent worker is load-bearing for a lot more (`subagent` + `subagent_aggregator` handlers run there too). A supervised deployment story is the sharp end of the stick.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
If you have been running the Minions worker under `nohup` with no restart story, this guide is the missing manual. Copy the watchdog script, paste the crontab env lines (`SHELL=/bin/bash`, `PATH`, `DATABASE_URL`, `GBRAIN_ALLOW_SHELL_JOBS=1`), and wire the cron to run every 5 minutes. You get a restart loop that handles the three silent-death modes: DB connection blip, lock-renewal stall, event loop wedge.
|
||||
|
||||
If you are running scheduled shell jobs only, skip the persistent worker and use `--follow`. 2-3 seconds of startup overhead is trivial when your job runs for a minute.
|
||||
|
||||
Docs-only release. No code changed. Zero migration required.
|
||||
|
||||
## To take advantage of v0.16.1
|
||||
|
||||
`gbrain upgrade` pulls the new guide. Read it:
|
||||
|
||||
1. **Open the guide:**
|
||||
```bash
|
||||
less docs/guides/minions-deployment.md
|
||||
```
|
||||
Or browse it on GitHub.
|
||||
2. **Persistent worker:** copy `minion-watchdog.sh`, set crontab env lines, wire a `*/5 * * * *` cron.
|
||||
3. **Scheduled shell jobs only:** rewrite your cron as `gbrain jobs submit shell ... --follow --timeout-ms N` and drop the persistent worker entirely.
|
||||
4. **The "Proposed CLI flags" section** (`--lock-duration` / `--max-stalled` / `--stall-interval` on `gbrain jobs work`): those are on the roadmap. Per-job `--max-stalled` on `gbrain jobs submit` is already real and writes to the row's column directly.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Added**
|
||||
- **Minions worker deployment guide** — new `docs/guides/minions-deployment.md` covering watchdog cron patterns, inline `--follow` for cron-only workloads, and the sharp edges of running `gbrain jobs work` against Supabase in production. Addresses a real gap: existing Minions docs (`minions-fix.md`, `minions-shell-jobs.md`) cover schema repair and shell-job security, not deploy patterns. Contributed by your OpenClaw via #287. Pre-landing accuracy pass corrected five factual bugs against current source: the `max_stalled` column default (5, not 1 or 3), the stalled-jobs smoke-test query (`active`, not `waiting`), the SIGTERM-to-SIGKILL grace window (10s minimum, not 2s), the cron env pattern (crontab env lines, not `source ~/.bashrc`), and the `--follow` exit semantics (blocks until submitted job is terminal, not until queue is empty).
|
||||
## [0.16.0] - 2026-04-20
|
||||
|
||||
## **Durable agents land. Your LLM loops survive crashes, timeouts, and worker restarts now.**
|
||||
## **OpenClaw died mid-run? Come back, resume from the last committed turn.**
|
||||
|
||||
Your OpenClaw crashes daily. Not "sometimes." Daily. An 8-turn OpenClaw subagent fires a tool call, the worker dies on a memory blip, all eight turns of context are gone, and there's nothing to do but start over from turn zero. This release kills that. `gbrain agent run` submits an Anthropic Messages API conversation as a first-class Minion job: every turn persists to `subagent_messages`, every tool call is a two-phase ledger row (`pending` → `complete | failed`), and replay on worker restart picks up from exactly the last committed turn. Crash-safe by construction, not by hope.
|
||||
|
||||
Fan-out works the same way. `--fanout-manifest` splits N prompts across N subagent children plus one aggregator. Children run `on_child_fail: 'continue'` so one failing run doesn't cascade, and the aggregator claims after all children reach ANY terminal state (complete, failed, dead, cancelled, timeout) and writes a mixed-outcome summary. No polling loop, no dead parents stranded in `waiting-children`.
|
||||
|
||||
Plugins work. Host repos drop a `gbrain.plugin.json` + `subagents/*.md` dir somewhere on `GBRAIN_PLUGIN_PATH`, and their custom subagent defs load at worker startup. Your OpenClaw ships its meeting-ingestion, signal-detector, and daily-task-prep subagents in its own repo now; gbrain discovers them day one. Collision rule is deterministic (left-wins with a loud warning). Trust boundary is strict on purpose: plugins ship DEFS, not tools. Tool allow-list stays here.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on the v0.15 branch against real Postgres via `bun run test:e2e`, plus the 159 new unit tests across 10 new test files. Coverage: 12 new runtime modules, 53+ code paths + user flows traced, 3 critical regression tests for the shell-jobs queue surface.
|
||||
|
||||
| Metric | BEFORE v0.15 | AFTER v0.15 | Δ |
|
||||
|----------------------------------------------------------|------------------------------------|---------------------------------------------|--------------------------------------|
|
||||
| Your OpenClaw run survives worker kill mid-tool-call | No (start over) | Yes (resume from last committed turn) | crash-recovery unlocked |
|
||||
| Fan-out run with 1 failed child out of N | Aggregator fails | Aggregator still claims + summarizes | mixed-outcome aggregation works |
|
||||
| `gbrain agent logs --follow` during long Anthropic call | Silent (looks frozen) | Heartbeat line per turn boundary | visible progress |
|
||||
| Tool-use replay on resume | N/A (no resume) | Idempotent re-run, non-idempotent aborts | two-phase protocol |
|
||||
| `put_page` exposure to agent-driven writes | Full write surface | Namespace-scoped `wiki/agents/<id>/…` | fail-closed, server-enforced |
|
||||
| Plugin subagent defs for downstream hosts | Not supported | `GBRAIN_PLUGIN_PATH` + validated at startup | OpenClaw day-1 usable |
|
||||
| Rate-lease capacity leaks on worker crash | Counter-based (leaks) | Lease-based (auto-prune on next acquire) | no starvation after SIGKILL |
|
||||
| Anthropic prompt cache on 40-turn agent | Per-turn cold | `cache_control: ephemeral` on system + tools | ~10x cost reduction (best-case) |
|
||||
|
||||
### What this means for your OpenClaw
|
||||
|
||||
You stop rerunning from zero. A crash at 3am that used to lose two hours of turns now costs you whatever fraction of one turn was in-flight when the worker died. The rest of the conversation is rows in `subagent_messages` and `subagent_tool_executions`, and the next worker claim replays from there. `gbrain agent logs <job>` shows you where it died, which tool it was running, and what came back from the last successful call. Real debugging, not guessing.
|
||||
|
||||
Credit: shell-jobs (v0.14) established every pattern v0.15 reuses — handler signature, dual-signal abort, ctx.updateTokens, protected-names, trusted-submit, JSONL audit log, timeout_ms. Codex caught the Mode A "transparent Agent() interception" impossibility during plan review and saved the shape of this work. The v0.15 handler is what survives on the other side of that review.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**New capability: `gbrain agent` CLI**
|
||||
- `gbrain agent run <prompt> [--subagent-def|--model|--max-turns|--tools|--timeout-ms|--fanout-manifest|--follow|--detach]` — submits a subagent job (or fan-out of N subagents + aggregator) under the trusted-submit flag. Follow mode tails status + logs until terminal; detach prints the job id and exits. Ctrl-C detaches (job keeps running), does not cancel.
|
||||
- `gbrain agent logs <job_id> [--follow] [--since ISO-or-relative]` — merges the JSONL heartbeat audit with persisted `subagent_messages` into one chronological timeline. `--since 5m` / `1h` / `2d` shorthand supported. Transcript tail renders the full message + tool tree only after the job is terminal.
|
||||
- Always registered on the worker (no separate env flag). `ANTHROPIC_API_KEY` is the natural cost gate — no key, the SDK call fails immediately. Who-can-submit is already gated by `PROTECTED_JOB_NAMES` + `TrustedSubmitOpts` so only the trusted-CLI path can insert `subagent` / `subagent_aggregator` rows.
|
||||
|
||||
**New durability primitives**
|
||||
- `src/core/minions/handlers/subagent.ts` — the LLM-loop handler. Two-phase tool persistence, replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs, injectable `MessagesClient` for mocking.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` — claims AFTER all children resolve (Lane 1B's queue changes guarantee each terminal child posts a `child_done` inbox message), produces deterministic mixed-outcome markdown summary.
|
||||
- `src/core/minions/rate-leases.ts` — lease-based concurrency cap for outbound providers. Owner-tagged rows with `expires_at` auto-prune on acquire, so a crashed worker can't strand capacity. `pg_advisory_xact_lock` guards the check-then-insert.
|
||||
- `src/core/minions/wait-for-completion.ts` — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; AbortSignal exits cleanly. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` — JSONL audit + heartbeat writer. Rotates weekly via ISO week. `readSubagentAuditForJob` is the readback path for `gbrain agent logs`.
|
||||
- `src/core/minions/transcript.ts` — messages + tool executions → markdown renderer. UTF-8-safe truncation; unknown block types fall through to JSON for diagnostics.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` — derives the subagent tool registry from `src/core/operations.ts`. 11-name allow-list (read-only + deterministic `put_page`). `put_page` schema is namespace-wrapped per subagent so the model writes correct slugs first-try; the server-side check in `put_page` is the authoritative gate.
|
||||
- `src/core/minions/plugin-loader.ts` — `GBRAIN_PLUGIN_PATH` (colon-separated absolute paths like `PATH`) + `gbrain.plugin.json` manifest + `subagents/*.md` defs. Strict path policy, left-wins collision, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time.
|
||||
- `src/mcp/tool-defs.ts` — extracted from an inline `operations.map(...)` block in the MCP server so subagent + MCP use the same source of truth. Byte-for-byte equivalence pinned by regression test.
|
||||
|
||||
**Schema (3 new tables + OperationContext fields + migration orchestrator)**
|
||||
- `subagent_messages` — Anthropic message-block persistence. `(job_id, message_idx)` UNIQUE; `content_blocks JSONB` holds parallel tool_use blocks in one assistant message.
|
||||
- `subagent_tool_executions` — two-phase ledger. `(job_id, tool_use_id)` UNIQUE; status: `pending | complete | failed`.
|
||||
- `subagent_rate_leases` — lease-based concurrency control. CASCADE deletes on owning job removal so no leaked rows.
|
||||
- `OperationContext` gains `jobId?`, `subagentId?`, and `viaSubagent?` (fail-closed signal for agent-path gating). Added to `src/core/operations.ts`.
|
||||
- `src/commands/migrations/v0_15_0.ts` — post-upgrade orchestrator (phases: schema → verify → record). `v0_14_0.ts` noop stub keeps the registry version sequence gapless.
|
||||
|
||||
**Queue correctness fixes**
|
||||
- `failJob`, `cancelJob`, and `handleTimeouts` all emit `child_done` inbox messages with `outcome: 'complete' | 'failed' | 'dead' | 'cancelled' | 'timeout'`. Pre-v0.15 only `completeJob` emitted; failed/cancelled/timed-out children silently stranded aggregator-style parents.
|
||||
- Parent-resolution terminal set expanded from `{completed, dead, cancelled}` to include `'failed'` everywhere parent-state is checked. A failed child with `on_child_fail: 'continue'` now correctly unblocks the parent.
|
||||
- `failJob` emits `child_done` BEFORE the parent-terminal UPDATE. Without insertion ordering, the EXISTS guard on the inbox INSERT would skip the row on `fail_parent` paths (caught by codex iteration 3).
|
||||
- `MinionJobInput.max_stalled` threads through `MinionQueue.add()` as INSERT param (not UPDATE on idempotency replay — that would mutate first-submitter state).
|
||||
|
||||
**Trust model**
|
||||
- `subagent` and `subagent_aggregator` join `PROTECTED_JOB_NAMES`. MCP `submit_job` returns `permission_denied`; only `gbrain agent run` (with `allowProtectedSubmit`) can insert these rows.
|
||||
- `put_page` gains a server-side fail-closed namespace check: when `ctx.viaSubagent === true`, `slug` MUST match `^wiki/agents/<subagentId>/.+` — even if `subagentId` is undefined (dispatcher bug must not open a hole).
|
||||
|
||||
**Docs**
|
||||
- `docs/guides/plugin-authors.md` — downstream-OpenClaw-facing walkthrough (minimum viable plugin, path + collision + trust policies, frontmatter fields, caveats).
|
||||
- 12 bisectable commits on `garrytan/minions-seam`, each PR-worthy on its own; the full series lands v0.15.0 end-to-end.
|
||||
|
||||
**Tests**
|
||||
- 159 new unit tests across 10 new files: `mcp-tool-defs`, `put-page-namespace`, `migrations-v0_15_0`, `queue-child-done`, `rate-leases`, `wait-for-completion`, `brain-allowlist`, `subagent-audit`, `subagent-transcript`, `subagent-handler`, `subagent-aggregator`, `plugin-loader`, `agent-cli`.
|
||||
- 3 critical regression tests pin the shell-jobs queue surface: `failJob` child_done behavior, `put_page` namespace path for non-subagent callers, MCP `buildToolDefs` byte-equivalence.
|
||||
- E2E `minions-resilience.test.ts` updated: the max_children test renames its spawned children off the now-protected `subagent` name.
|
||||
|
||||
## [0.15.4] - 2026-04-21
|
||||
|
||||
## **PgBouncer transaction-mode prepared statements, fixed at the pool.**
|
||||
## **`gbrain jobs work` against Supabase pooler stops silently dropping rows.**
|
||||
|
||||
Three separate PRs (#284, #286, #270) were all trying to fix the same bug: on a Supabase transaction-mode pooler (port 6543), `postgres.js`'s per-client prepared-statement cache goes stale every time PgBouncer recycles the backend connection. The symptom under sustained gbrain load is `prepared statement "xyz" does not exist` in the logs and silently dropped rows during sync. v0.15.4 lands the combined fix: the `resolvePrepare()` helper from #284, the both-connection-paths coverage from @notjbg's community PR #270, a new doctor check, and real tests against `bun:test`. The one-liner in #286 is dominated by this.
|
||||
|
||||
### The one number that matters
|
||||
|
||||
There isn't a benchmark, there's a correctness gate. On a Supabase pooler at port 6543 with a 4,500-page sync:
|
||||
|
||||
| | Before v0.15.4 | After v0.15.4 |
|
||||
|---|---|---|
|
||||
| `prepared statement ... does not exist` errors | Dozens per sync | Zero |
|
||||
| Rows inserted vs. manifest count | Short by 50-200 rows (silent) | 1:1 parity |
|
||||
| `gbrain jobs work` crash under load | Yes | No |
|
||||
|
||||
The silent-drop is the dangerous half. You run `gbrain sync`, the exit code is 0, the logs have a few noise lines you scroll past, and three weeks later you notice your brain is missing pages. `resolvePrepare(url)` disables prepared statements when the URL targets port 6543, and the doctor check flags the misconfiguration if you've manually forced `GBRAIN_PREPARE=true` on that port.
|
||||
|
||||
### What this means for pooler users
|
||||
|
||||
If you connect via `aws-0-REGION.pooler.supabase.com:6543`, do nothing. The upgrade disables prepared statements automatically and `gbrain doctor` confirms it with `pgbouncer_prepare: ok`. If you're on session mode (port 5432 on the pooler host) or direct Postgres, nothing changes: prepared statements stay on, plan caching stays intact. If your PgBouncer runs in session mode on a non-standard port, set `GBRAIN_PREPARE=true` explicitly.
|
||||
|
||||
## To take advantage of v0.15.4
|
||||
|
||||
`gbrain upgrade` handles this automatically. If you're not sure whether the fix is live:
|
||||
|
||||
1. **Run the doctor check:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for `pgbouncer_prepare`. On a `:6543` URL you should see `ok` (prepared statements disabled). On a direct URL the check silently passes.
|
||||
2. **Verify on sustained load:**
|
||||
```bash
|
||||
gbrain sync
|
||||
```
|
||||
Zero `prepared statement ... does not exist` log lines. Row count inserted matches the source manifest.
|
||||
3. **If something looks wrong,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- the connection URL shape (port and pooler hostname — redact credentials)
|
||||
- whether `GBRAIN_PREPARE` is set
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Fixed**
|
||||
- **Supabase PgBouncer port-6543 prepared statements no longer break sync.** New `resolvePrepare(url)` helper in `src/core/db.ts` with 4-level precedence: `GBRAIN_PREPARE` env var → `?prepare=` query param → port-6543 auto-detect → default. Wired into both the module-singleton `connect()` in `db.ts` AND the worker-instance `PostgresEngine.connect({poolSize})` in `src/core/postgres-engine.ts` so `gbrain jobs work` gets the same treatment as the main CLI. The second path was the gap #284 missed; community PR #270 caught it. Contributed by @notjbg.
|
||||
- **`gbrain doctor` surfaces the misconfiguration.** New `pgbouncer_prepare` check reads the configured URL via `loadConfig()` and reports `ok` when prepared statements are safely disabled, `warn` when the URL points at port 6543 but prepared statements are still enabled (the footgun that caused silent row drops).
|
||||
|
||||
**Tests**
|
||||
- New `test/resolve-prepare.test.ts` — 11 cases covering the full precedence matrix: env override, URL query param, port auto-detect, malformed URLs, `postgres://` vs `postgresql://` schemes, URL-encoded credentials. Uses `bun:test` (not vitest — #284's original tests were in the wrong framework and would never have run).
|
||||
- Extended `test/postgres-engine.test.ts` — new source-level grep assertion that the worker-instance `connect({poolSize})` branch calls `db.resolvePrepare(url)` and conditionally includes the `prepare` key in the options literal. Mirrors the existing `SET LOCAL statement_timeout` guardrail in the same file. If anyone rips out the wiring, the build fails before a shipping brain drops rows.
|
||||
|
||||
**Supersedes**
|
||||
- Closes #284 (ours, Wintermute): architecture landed as-is (port-only detection, no hostname expansion). Tests rewritten from vitest to bun:test.
|
||||
- Closes #286 (ours, Codex one-liner): dominated; unconditional `prepare: false` would have cost direct-Postgres users plan caching for no reason.
|
||||
- Closes #270 (@notjbg): the critical both-connection-paths insight landed; credit preserved in commit trailer and this CHANGELOG entry.
|
||||
|
||||
## [0.15.3] - 2026-04-21
|
||||
|
||||
## **Two upgrade-night bugs that crashed v0.13 → v0.14, now fixed with regression guards.**
|
||||
## **Migrations find the right binary. Autopilot spawns its worker. `gbrain upgrade` survives.**
|
||||
|
||||
Tonight's production upgrade surfaced eleven bugs. Two of them — Bug 1 (the migration shell-out) and Bug 4 (the autopilot resolver) — survived two eng-review passes AND nine Codex reviews with correct diagnoses and implementable fixes. The other nine had wrong root causes or unimplementable architectures (documented in `~/.claude/plans/` as deferred work with grounded starting context for future `/investigate` sessions). This release ships the two clean fixes so the next `gbrain upgrade` actually lands.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Fixed**
|
||||
- **`gbrain upgrade` no longer crashes mid-migration on bun installs.** The v0.13.0 migration orchestrator used to shell out via `process.execPath`, which on bun-installed trees is the `bun` runtime itself. `${bun} extract links --source db …` got reinterpreted as `bun run extract` and crashed with "script not found." The fix drops the execPath detour and shells out to the bare `gbrain` string, letting the canonical shim on PATH (`/usr/local/bin/gbrain` by default) win. Regression test in `test/migrations-v0_13_0.test.ts` greps the source for `process.execPath` and fails the build if anyone reintroduces the pattern. Contributed by @garrytan.
|
||||
- **Autopilot spawns its Minions worker again.** `resolveGbrainCliPath` checked `argv[1]` first and happily returned `/path/to/src/cli.ts` on bun-source installs. `spawn()` then failed with `EACCES` because TypeScript source isn't executable, and autopilot silently lost its worker. The fix reorders the probe: `which gbrain` (shim on PATH) wins first, then compiled `process.execPath`, then an `argv[1]=/gbrain` fallback. The `.ts` branch is deleted entirely. A critical regression test enforces that the resolver NEVER returns a `.ts` path across any combination of `argv[1]` + `process.execPath` + shim availability.
|
||||
|
||||
**Tests**
|
||||
- New `test/migrations-v0_13_0.test.ts` — 7 cases covering registry wiring, dry-run semantics, and three regression guards against the Bug 1 re-introduction (no `process.execPath`, no `GBRAIN` constant, no `bun` or `.ts` in `execSync` calls).
|
||||
- Rewrote `test/autopilot-resolve-cli.test.ts` — the old test enshrined the buggy `.ts` return path. New test parameterizes argv/execPath combinations and asserts the resolver never returns a `.ts` path. This is the test that would have caught Bug 4 before it shipped.
|
||||
|
||||
**Deferred (tracked for follow-up `/investigate` sessions)**
|
||||
- Bug 2 (pooler MaxClients), Bug 3 (partial-migration retry loop), Bug 5 (v0.14.0 registry gap), Bug 6/10 (duplicate graph edges), Bug 7 (doctor --fast), Bug 8 (autopilot-cycle stalls), Bug 9 (YAML colons), Bug 11 (brain_score breakdown). Each has grounded Codex findings documenting the real root cause and where prior diagnoses went wrong. Landing target: subsequent PR waves.
|
||||
|
||||
## [0.15.2] - 2026-04-21
|
||||
|
||||
## **Silent binaries are dead. Every bulk action now heartbeats.**
|
||||
## **Agents can tell the difference between "working" and "hung."**
|
||||
|
||||
`gbrain doctor` on a 52K-page brain used to sit silent for 10+ minutes and then get killed by an agent timeout. The checks always completed when run by hand, but stdout buffered and agents saw nothing. The same pattern hit `embed`, `sync`, `import`, `extract`, `migrate`, and every orchestrator that shelled out to them — progress either went to stdout with `\r` rewrites that collapse when piped, or nowhere at all. v0.15.2 routes every bulk action through one shared reporter. Non-TTY default is plain human lines on stderr, one line per event. Agents that want structured progress flip `--progress-json` and get one JSON object per line.
|
||||
|
||||
Progress events never touch stdout. Data and final summaries still go there. Script you wrote six months ago that parses `gbrain embed` output? Still works. Agent that captures stdout to JSON.parse the result? Now gets clean JSON instead of `\r\r\r1234/52000 pages...` mixed in.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on this repo (80 unit test files, 14 E2E test files, real Postgres+pgvector, 141 E2E cases incl. 3 new doctor-progress tests):
|
||||
|
||||
| Metric | BEFORE v0.15.2 | AFTER v0.15.2 | Δ |
|
||||
|---------------------------------------------------|------------------------|----------------------------------------|----------------|
|
||||
| Commands that stream progress | 3 (ad-hoc `\r` stdout) | **14** (reporter, stderr, rate-gated) | **+11** |
|
||||
| Progress observable when stdout is piped | **0 of 3** | **14 of 14** | always visible |
|
||||
| Canonical JSON event schema | none | **locked in `docs/progress-events.md`** | stable |
|
||||
| `doctor` silence window on 52K pages | 10+ min then killed | **heartbeat every 1s** | observable |
|
||||
| `jsonb_integrity` scan targets | 4 (missed `page_versions.frontmatter`) | **5** | matches `repair-jsonb` |
|
||||
| Minion jobs that update `job.progress` | 0 bulk cores | **embed** wired (import/sync/extract ready via callbacks) | DB-backed |
|
||||
| Unit tests for progress/CLI plumbing | 0 | **37** (progress + cli-options) | +37 |
|
||||
| E2E tests for agent-visible progress | 0 | **3** (doctor-progress Tier 1) | +3 |
|
||||
|
||||
| Bulk command | Progress today | Progress after v0.15.2 |
|
||||
|-----------------------|-----------------|----------------------------------------------------------------|
|
||||
| `doctor` | None (blocks) | Per-check heartbeat, 1s on slow queries |
|
||||
| `orphans` | Final summary | Heartbeat while `NOT EXISTS` scan runs |
|
||||
| `embed` | `\r` stdout | Per-page stderr, `job.updateProgress` from Minions |
|
||||
| `files sync` | `\r` stdout | Per-file stderr |
|
||||
| `export` | `\r` stdout | Per-page stderr (newly in scope) |
|
||||
| `import` | Per-100 stdout | Per-file stderr, rate-gated |
|
||||
| `extract` (fs + db) | Ad-hoc stderr | Canonical event schema, all paths |
|
||||
| `sync` | Final summary | Per-file ticks across delete/rename/import phases |
|
||||
| `migrate --to ...` | Per-50 stdout | `migrate.copy_pages` + `migrate.copy_links` phases |
|
||||
| `repair-jsonb` | Final summary | Per-column heartbeat (stdout stays JSON-clean for orchestrator)|
|
||||
| `check-backlinks` | Final summary | Heartbeat during the double-walk |
|
||||
| `lint` | Per-file stdout | Per-file stderr, issues still on stdout |
|
||||
| `integrity auto` | Own progress file | Unified reporter (file kept as resume marker) |
|
||||
| `eval` | None | Per-query tick in single + A/B modes |
|
||||
| `apply-migrations` | Inherited child output | Explicit flag propagation + stdio discipline |
|
||||
|
||||
Concrete agent win: on a 52K-page brain, `gbrain --progress-json doctor` emits ~10 events per second on stderr (start per check, heartbeats during the slow scan, finish per check) while `gbrain doctor --json` keeps stdout clean and JSON-parseable. The agent never sees silence longer than 1 second, and its stdout parser doesn't need to scrub progress garbage.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you run `gbrain` in CI, through a Minion worker, or inside any agent that captures stdout, this release means your downstream consumers stop guessing. Slow migrations announce themselves. Long imports name each file. `gbrain jobs get <id>` returns live `progress` for Minion-queued bulk work. The `gbrain doctor` warning you've been ignoring because it fires silently and then 10 minutes later tells you nothing is wrong becomes a 1-second heartbeat that proves it's working. If you're reading logs from a shell pipeline and prefer plain human lines, you don't need to do anything, that's the default for non-TTY stderr. Only add `--progress-json` when you want structured events.
|
||||
|
||||
## To take advantage of v0.15.2
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Nothing mechanical is required.** v0.15.2 is purely additive to the CLI surface — no schema changes, no migration orchestrator, no data rewrites. Progress events start flowing the next time you invoke a bulk command.
|
||||
2. **To stream structured events to your agent:**
|
||||
```bash
|
||||
gbrain --progress-json sync 2> progress.log
|
||||
# or
|
||||
gbrain doctor --progress-json --json > doctor.json 2> doctor.progress
|
||||
```
|
||||
3. **For Minion-queued jobs:**
|
||||
```bash
|
||||
gbrain jobs submit embed
|
||||
# while it runs:
|
||||
gbrain jobs get <id> # .progress is live-updated by the worker
|
||||
```
|
||||
4. **If `gbrain doctor` still looks hung** on a very large brain, check the CLI output for heartbeat lines. If they're missing, file an issue at https://github.com/garrytan/gbrain/issues with the command you ran, stdout/stderr samples, and output of `gbrain doctor --fast`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Reporter (new, `src/core/progress.ts`)
|
||||
- Dependency-free. Modes: `auto` (TTY → `\r`-rewriting; non-TTY → plain lines), `human`, `json` (JSONL on stderr), `quiet`.
|
||||
- Rate gating: emits on whichever fires first: `minIntervalMs` (default 1000) or `minItems` (default `max(10, ceil(total/100))`). Final `tick` where `done === total` always emits.
|
||||
- `startHeartbeat(reporter, note)` helper for single long-running queries (doctor's `markdown_body_completeness`, `orphans` anti-join, `repair-jsonb` per-column UPDATE).
|
||||
- `child()` composes phase paths, `sync.import.<slug>`, not flat `<slug>`.
|
||||
- EPIPE defense on both sync throws and stream `'error'` events. Singleton module-level SIGINT/SIGTERM handler emits `abort` events for every live phase, one handler no matter how many reporters exist.
|
||||
|
||||
#### CLI plumbing (`src/core/cli-options.ts`, `src/cli.ts`)
|
||||
- Global flags `--quiet`, `--progress-json`, `--progress-interval=<ms>` parsed before command dispatch.
|
||||
- `CliOptions` singleton (`getCliOptions`) reachable from every command without threading a new parameter through 20 handlers.
|
||||
- `OperationContext.cliOpts` extends shared-op dispatch, MCP callers see defaults, CLI callers see parsed flags.
|
||||
- `childGlobalFlags()` helper: appends the parent's flags to every `execSync('gbrain ...')` call in the migration orchestrators, so child progress matches parent mode.
|
||||
|
||||
#### JSON event schema
|
||||
- Stable from v0.15.2, documented in `docs/progress-events.md`.
|
||||
- `{event, phase, ts}` always present. Optional: `total`, `done`, `pct`, `eta_ms`, `note`, `elapsed_ms`, `reason`. No fake totals when a query has no count.
|
||||
- Phases use `snake_case.dot.path`. Machine-stable. Agent parsers can group by phase prefix (all `doctor.*` events belong to one run).
|
||||
|
||||
#### Backward-compat warnings
|
||||
Progress for `embed`, `files`, `export`, `extract`, `import`, `migrate-engine` moved from stdout to stderr. Stdout now carries only final summaries and `--json` payloads. Scripts that parsed `process.stdout` for progress lines (`\r 1234/52000 pages...`) see empty stdout for those counters; the data they actually want (the final "Embedded N chunks" summary) is still there. Point anything grepping stdout for progress at stderr instead.
|
||||
|
||||
#### Minion handlers (`src/commands/jobs.ts`)
|
||||
- `embed` handler passes `job.updateProgress({done, total, embedded, phase})` as the `onProgress` callback. Primary Minion progress channel is DB-backed, readable via `gbrain jobs get <id>` or the `get_job_progress` MCP op. Stderr from `jobs work` stays coarse for daemon liveness.
|
||||
- Other handlers (`sync`, `extract`, `backlinks`, `autopilot-cycle`, `import`) have the callback plumbing ready from the core functions; wiring the remaining handlers is a follow-up.
|
||||
|
||||
#### `gbrain doctor`
|
||||
- `jsonb_integrity` now scans 5 targets (adds `page_versions.frontmatter`), matching `repair-jsonb`'s surface. The old 4-target check missed one of the repair sites.
|
||||
- Per-check heartbeats so agents see `doctor.db_checks` starting, which check is in-flight, and `doctor.markdown_body_completeness` scanning.
|
||||
- No false totals: the `LIMIT 100` truncation check reports `heartbeat`, not `tick` with a fake count.
|
||||
|
||||
#### Upgrade (`src/commands/upgrade.ts`)
|
||||
- Post-upgrade timeout bumped 300s → 1800s (30 min). Override via `GBRAIN_POST_UPGRADE_TIMEOUT_MS`. The old 300s cap killed v0.12.0 graph-backfill migrations on 50K+ brains; heartbeat wiring in v0.15.2 makes the long wait observable.
|
||||
|
||||
#### CI guard
|
||||
- `scripts/check-progress-to-stdout.sh` greps `src/` for `process.stdout.write('\r...')` and fails `bun run test` if any regression lands.
|
||||
|
||||
#### Tests
|
||||
- New: `test/progress.test.ts` (17 cases — mode resolution, rate gating, EPIPE paths, SIGINT singleton, child phase composition), `test/cli-options.test.ts` (18 cases — flag parsing, `--quiet` skillpack-check collision regression, global-flag strip-and-dispatch), `test/e2e/doctor-progress.test.ts` (3 cases, Tier 1 — spawns the real CLI against a real Postgres, asserts stderr JSONL matches the schema and stdout stays clean).
|
||||
## [0.15.1] - 2026-04-21
|
||||
|
||||
## **Fix wave: 4 hot issues that blocked real brains, landed together.**
|
||||
## **PGLite survives macOS 26.3. Minions actually rescues SIGKILL'd jobs. Autopilot dashboards stop the 14.6s seqscan. `bun install -g` tells you when it's broken.**
|
||||
|
||||
v0.15.1 is the hotfix wave on top of the v0.14.x stack (shell job type in v0.14.0, doctor DRY + `--fix` in v0.14.1, 8 deferred bug fixes in v0.14.2) plus v0.15.0 (llms.txt + AGENTS.md): four user-filed issues against v0.13.x, fixed and verified together, plus three scope expansions that close adjacent footguns. Upgrade is automatic. If `gbrain upgrade` runs clean, your brain gets faster and more reliable on the next sync cycle.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
The four issues this release closes, with measured impact:
|
||||
|
||||
| Issue | Before v0.15.1 | After v0.15.1 | Δ |
|
||||
|-------|----------------|----------------|---|
|
||||
| #170 `SELECT * FROM pages ORDER BY updated_at DESC` on 31k rows (Postgres) | ~14.6s seqscan | <20ms index scan | ~700x |
|
||||
| #219 `max_stalled` default on `minion_jobs` | 3 (three rescues before dead, v0.14.2 set this) | 5 (four rescues before dead) | extra headroom for flaky deploys |
|
||||
| #219 existing waiting/active jobs with `max_stalled<5` | would still dead-letter earlier than expected | backfilled to 5 on upgrade | closes the pain today |
|
||||
| #218 `bun install -g github:garrytan/gbrain` postinstall failure | silent `|| true` | visible stderr warning with recovery URL | users know it's broken |
|
||||
| #223 PGLite WASM crash on macOS 26.3 | raw `Aborted()`, no hint | pinned `@electric-sql/pglite` to `0.4.3` + actionable error message naming the issue | users can route to #223 |
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you run autopilot against a Supabase brain with 30k+ pages, your health/dashboard cycle was silently burning 14.6 seconds on every iteration. The new index drops that to single-digit milliseconds without locking writes (Postgres gets `CREATE INDEX CONCURRENTLY` with an invalid-index cleanup DO block; PGLite gets plain `CREATE INDEX` since it has no concurrent writers). Your agent stops blocking on list-pages-by-date queries.
|
||||
|
||||
If you use Minions, the "SIGKILL mid-flight, 10/10 rescued" claim is now actually true out-of-the-box with generous headroom. Default `max_stalled=5` means a kill -9'd worker gets picked up by the next worker instead of dead-lettered early. v15 migration backfills existing non-terminal rows (`waiting/active/delayed/waiting-children/paused`) so upgrading doesn't leave a queue full of doomed jobs.
|
||||
|
||||
If you install via `bun install -g github:...` (not recommended but people try it), you'll now see a loud stderr warning with a link to #218 instead of a broken CLI that fails on next invocation. The real fix is `git clone + bun link`, documented in README and INSTALL_FOR_AGENTS.md.
|
||||
|
||||
If you're on macOS 26.3 and PGLite was crashing with `Aborted()`, the pin to 0.4.3 gives us the best shot at avoiding the WASM regression (noting: 0.4.3 is unverified against 26.3 in CI — the error-wrap at `pglite-engine.ts connect()` is the safety net if the pin doesn't hold). Any PGLite init failure now shows the #223 link instead of a raw runtime error.
|
||||
|
||||
## To take advantage of v0.15.1
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Verify the outcome:**
|
||||
```bash
|
||||
psql "$DATABASE_URL" -c "\d minion_jobs" | grep max_stalled # DEFAULT should be 5
|
||||
psql "$DATABASE_URL" -c "\d pages" | grep idx_pages_updated_at_desc # index should exist
|
||||
gbrain doctor
|
||||
```
|
||||
3. **If any step fails or the numbers look wrong,** file an issue with `gbrain doctor` output and the contents of `~/.gbrain/upgrade-errors.jsonl` if it exists. https://github.com/garrytan/gbrain/issues
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- Schema migration **v14** — `CREATE INDEX [CONCURRENTLY] IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC)` (engine-aware; Postgres uses CONCURRENTLY with an invalid-index DO-block cleanup, PGLite uses plain CREATE). Closes #170. Contributed by @fuleinist (#215).
|
||||
- Schema migration **v15** — `ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 5` (bumps v0.14.2's default of 3 to 5 for extra flaky-deploy headroom) + `UPDATE` backfill scoped to non-terminal statuses (`waiting/active/delayed/waiting-children/paused`) so existing queued work benefits on upgrade. Closes #219. Reported by @macbotmini-eng.
|
||||
- `MinionJobInput.max_stalled` — new optional field, plumbed through `queue.add()` with `[1, 100]` clamp.
|
||||
- `gbrain jobs submit --max-stalled N` — CLI flag to set per-job stall tolerance.
|
||||
- `gbrain jobs submit --backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — scope-expansion audit exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case that simulates a killed worker and asserts the v0.15.1 default actually rescues.
|
||||
- `gbrain doctor --index-audit` — new opt-in Postgres check that reports zero-scan indexes from `pg_stat_user_indexes`. Informational only (no auto-drop). PGLite no-ops.
|
||||
- `BrainEngine.kind` readonly discriminator (`'postgres' | 'pglite'`) — lets migrations and consumers branch on engine without `instanceof` + dynamic imports.
|
||||
- `package.json trustedDependencies: ["@electric-sql/pglite"]` — lets Bun run PGLite's dep postinstall on global installs.
|
||||
|
||||
#### Changed
|
||||
- `@electric-sql/pglite` pinned to exactly `0.4.3` (was `^0.4.4`) — best-available mitigation for the macOS 26.3 WASM abort. Reported by @AndreLYL (#223). Flagged as unverified; reproduce on a 26.3 machine and file a follow-up if it still aborts.
|
||||
- `package.json postinstall` — now warns loudly on stderr with a recovery URL instead of silencing errors with `2>/dev/null || true`. `bun install -g` hitting a migration failure now tells you what to do. Reported by @gopalpatel (#218).
|
||||
- `src/core/pglite-engine.ts connect()` — wraps `PGlite.create()` with a friendly error pointing at #223 and `gbrain doctor`. Nests the original error for debuggability.
|
||||
- `doctor` `schema_version` check — now fails loudly when `version=0` (migrations never ran), linking #218.
|
||||
- `README.md` + `INSTALL_FOR_AGENTS.md` — explicit warning against `bun install -g github:garrytan/gbrain`.
|
||||
|
||||
#### Fixed
|
||||
- **The "SIGKILL mid-flight, 10/10 rescued" claim is now accurate** out-of-the-box with headroom (#219). Schema default 3 → 5.
|
||||
- **Autopilot dashboards stop blocking on list-pages queries** on 30k+ row Postgres brains (#170).
|
||||
- **PGLite error on macOS 26.3** is now actionable instead of a raw `Aborted()` (#223).
|
||||
- **`bun install -g` no longer produces a silently broken CLI** (#218) — postinstall surfaces failures.
|
||||
|
||||
#### Internal
|
||||
- `Migration` interface extended with `sqlFor: { postgres?, pglite? }` + `transaction: boolean` fields. Runner picks the engine-specific SQL branch and (on Postgres only) bypasses `engine.transaction()` when `transaction: false` (required for CONCURRENTLY).
|
||||
- `scripts/check-jsonb-pattern.sh` extended with a CI guard against `max_stalled DEFAULT 1` regressing.
|
||||
- ~15 new unit tests covering max_stalled default/clamp/backfill/v14/v15 semantics. 3 regression tests pinned by IRON RULE.
|
||||
- `test/e2e/` now runs test files sequentially via `scripts/run-e2e.sh` to eliminate shared-DB races that caused ~3/5 runs to have 4-10 flaky fails. Every run post-fix: 13 files, 138 tests, 0 fails.
|
||||
|
||||
## [0.15.0] - 2026-04-21
|
||||
|
||||
## **GBrain now talks to LLMs the way modern docs sites do.**
|
||||
## **One URL, full context. Three files, zero drift.**
|
||||
|
||||
Three new artifacts ship at the repo root: `llms.txt` (llmstxt.org-spec index), `llms-full.txt` (same map with core docs inlined, ~225KB, fits well under a 150k-token context window), and `AGENTS.md` (the non-Claude-agent operating protocol). All three are generator-driven. `scripts/build-llms.ts` reads a curated `scripts/llms-config.ts` and emits `llms.txt` + `llms-full.txt` deterministically; `AGENTS.md` is hand-written and uses relative links so it survives forks and rename. Every agent that clones GBrain now has a one-screen answer to "I just got here, what do I do?"
|
||||
|
||||
README and `INSTALL_FOR_AGENTS.md` now point agents at `AGENTS.md` first. The old install prompt still works, but the leverage point, Codex's read of the plan, was that these files are invisible unless the install path references them. Fixed.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on this release:
|
||||
|
||||
| Metric | BEFORE | AFTER | Δ |
|
||||
|-------------------------------------------------|----------------------------------|-----------------------------------|----------------------------|
|
||||
| Agent entry points with clear install protocol | 1 (CLAUDE.md, Claude Code only) | 3 (CLAUDE.md + AGENTS.md + llms.txt) | +non-Claude coverage |
|
||||
| Docs referenced at a single canonical URL | 0 | 20 (across 5 H2 sections) | index exists |
|
||||
| Full-context fetch round-trips | ~20 (one per doc) | 1 (`llms-full.txt`, 224 KB) | ~20x fewer fetches |
|
||||
| Tests guarding the doc index | 0 | 7 (paths resolve, idempotent, spec shape, regen-drift, content contract, AGENTS mirror, size budget) | +7 |
|
||||
| Pre-existing repo bugs found and fixed | — | 1 (`git pull origin main` → `master`) | drive-by |
|
||||
|
||||
The 7 tests enforce content contract: removing `skills/RESOLVER.md` or the Debugging H2 from the config fails `bun test`. Forgetting to rerun `bun run build:llms` after adding a new doc fails `bun test`. The size budget (600KB) fails `bun test` if `llms-full.txt` balloons.
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you're running GBrain: nothing to do. Your agent already has CLAUDE.md. But next time you install GBrain on Codex, Cursor, or OpenClaw, the agent lands on `AGENTS.md` and walks the install without hunting. If you run a fork, regenerate with `LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms` to rewrite URLs. If you publish GBrain docs alongside your own, `llms.txt` is the index; `llms-full.txt` is the drop-into-a-context-window bundle.
|
||||
|
||||
Credit to Codex for catching that the original plan's AGENTS.md was underpowered, that the eng review missed a content-contract test, and that the install prompt was the real leverage point. Seven of the fifteen Codex findings landed directly in the plan; three went to user decision; five stayed as intentional NOT-in-scope.
|
||||
|
||||
## To take advantage of this release
|
||||
|
||||
`gbrain upgrade` does not need to do anything. These are new public files; existing installs pick them up on their next pull.
|
||||
|
||||
1. **If you wrote a downstream fork:** regenerate with your URL base.
|
||||
```bash
|
||||
LLMS_REPO_BASE=https://raw.githubusercontent.com/your-org/your-fork/main bun run build:llms
|
||||
git add llms.txt llms-full.txt && git commit
|
||||
```
|
||||
2. **If you add a new doc under `docs/`:** add it to `scripts/llms-config.ts`, then
|
||||
```bash
|
||||
bun run build:llms
|
||||
bun test test/build-llms.test.ts
|
||||
```
|
||||
CI blocks ship if these drift.
|
||||
3. **Verify it actually works:** ask a fresh LLM
|
||||
```
|
||||
Fetch https://raw.githubusercontent.com/garrytan/gbrain/master/llms.txt and tell me
|
||||
how I'd debug a broken live sync.
|
||||
```
|
||||
Answer should cite `docs/GBRAIN_VERIFY.md`, `docs/guides/live-sync.md`, and `gbrain doctor`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Added
|
||||
- `AGENTS.md` at repo root — ~45-line non-Claude-agent operating protocol. Install, read order, trust boundary, config/debug/migration pointers, fork instructions. Uses relative links so it survives renames.
|
||||
- `llms.txt` at repo root — llmstxt.org-spec index. H1 + blockquote + 5 required H2 sections (Core entry points, Configuration, Debugging, Migrations) plus an Operational tips block with `gbrain doctor`, `gbrain orphans`, `gbrain repair-jsonb`. ~4KB.
|
||||
- `llms-full.txt` at repo root — same index with core docs inlined under `## {path}` headings for single-fetch ingestion. ~225KB, under the 600KB `FULL_SIZE_BUDGET`.
|
||||
- `scripts/llms-config.ts` — curated TS config. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `includeInFull: false` flags entries that should appear in `llms.txt` but not be inlined in `llms-full.txt` (Philosophy, Optional, CHANGELOG).
|
||||
- `scripts/build-llms.ts` — the generator. Deterministic, no timestamps, sorted by config order. Warns (does not fail) if `llms-full.txt` exceeds `FULL_SIZE_BUDGET` with the biggest entries listed.
|
||||
- `test/build-llms.test.ts` — 7 cases: paths resolve on disk, generator idempotent, llms.txt spec shape, checked-in files match generator output (drift guard), content contract (RESOLVER / AGENTS / INSTALL_FOR_AGENTS referenced), AGENTS mirrors README+INSTALL install path, size budget enforcement.
|
||||
- `bun run build:llms` script in `package.json`.
|
||||
|
||||
#### Changed
|
||||
- `README.md` — adds a one-line LLMs/Agents pointer above the install CTA and a follow-up paragraph under the agent paste block naming `AGENTS.md` + `llms.txt` as fallback entry points for non-Claude agents.
|
||||
- `INSTALL_FOR_AGENTS.md` — new "Step 0: If you are not Claude Code" prelude points agents at `AGENTS.md` first.
|
||||
- `CLAUDE.md` — adds `scripts/llms-config.ts`, `scripts/build-llms.ts`, and `AGENTS.md` to Key files. Explicitly notes that committed generator output is NOT analogous to `schema-embedded.ts` (no runtime consumer; committed for GitHub browsing + fork safety).
|
||||
|
||||
- `INSTALL_FOR_AGENTS.md:136` — `git pull origin main` → `git pull origin master`. Pre-existing drift: README and CI use `master`, `origin/HEAD -> master`, but the upgrade instructions told users to pull from a branch that doesn't exist. Folded into this release as a drive-by fix.
|
||||
|
||||
## [0.14.2] - 2026-04-20
|
||||
|
||||
## **Eight deferred bugs, root-cause fixes, one clean wave.**
|
||||
@@ -56,7 +807,7 @@ Your agent's feedback loops tighten. When sync blocks, doctor surfaces the exact
|
||||
#### Reliability
|
||||
- **Bug 2: `GBRAIN_POOL_SIZE` env knob** (`src/core/db.ts`, `src/commands/import.ts`). Honored by both the singleton pool and the parallel-import worker pool. Defaults to 10; lower for Supabase transaction pooler. `initPostgres` / `initPGLite` now wrap lifecycle in `try { ... } finally { await engine.disconnect() }`.
|
||||
- **Bug 3: Migration ledger centralization + wedge cap** (`src/commands/apply-migrations.ts`, `src/core/preferences.ts`). Runner owns all ledger writes. 3 consecutive partials = wedged, skipped with a loud message. New `--force-retry <version>` flag writes a `'retry'` marker without faking success. `complete` status never regresses. `appendCompletedMigration` is idempotent on double-complete.
|
||||
- **Bug 8: `max_stalled` default 1 → 3** (`src/core/schema-embedded.ts`, `src/core/pglite-schema.ts`, `src/schema.sql`). First lock-lost tick no longer dead-letters. `v0_14_0` Phase A ALTERs existing installs. `autopilot-cycle` handler yields to the event loop between phases so the worker's lock-renewal timer fires.
|
||||
- **Bug 8: `max_stalled` default 1 → 3** (`src/core/schema-embedded.ts`, `src/core/pglite-schema.ts`, `src/schema.sql`). First lock-lost tick no longer dead-letters. `v0_14_0` Phase A ALTERs existing installs. `autopilot-cycle` handler yields to the event loop between phases so the worker's lock-renewal timer fires. (v0.15.1 further bumps this to 5 and adds a non-terminal row backfill — see #219.)
|
||||
- **Bug 9: Sync gate + acknowledge mechanism** (`src/commands/sync.ts`, `src/commands/import.ts`, `src/core/sync.ts`). All 3 sync paths (incremental, full via `runImport`, `gbrain import` git continuity) gate `sync.last_commit` on no-failures. Failures append to `~/.gbrain/sync-failures.jsonl` with dedup key. New `gbrain sync --skip-failed` + `--retry-failed` flags. Doctor surfaces unacknowledged failures.
|
||||
|
||||
#### Observability
|
||||
@@ -67,7 +818,7 @@ Your agent's feedback loops tighten. When sync blocks, doctor surfaces the exact
|
||||
- **Bug 6/10: `jsonb_agg(DISTINCT ...)` in legacy `traverseGraph`** (`src/core/postgres-engine.ts`, `src/core/pglite-engine.ts`). Presentation-level dedup only — the schema continues to preserve per-`origin_page_id` / per-`link_source` provenance rows. Fixes duplicate edges like `works_at → companies/brex` appearing twice in `gbrain graph`.
|
||||
|
||||
#### New migration
|
||||
- **Bug 5: `v0_14_0` migration registered** (`src/commands/migrations/v0_14_0.ts`). Phase A: `ALTER minion_jobs.max_stalled SET DEFAULT 3` (idempotent). Phase B: emits `pending-host-work.jsonl` entry pointing at `skills/migrations/v0.14.0.md` for shell-jobs adoption. Registered in `src/commands/migrations/index.ts`. `package.json` bumped to 0.14.2 (0.14.0 and 0.14.1 were taken by upstream during this branch's work).
|
||||
- **Bug 5: `v0_14_0` migration registered** (`src/commands/migrations/v0_14_0.ts`). Phase A: `ALTER minion_jobs.max_stalled SET DEFAULT 3` (idempotent). Phase B: emits `pending-host-work.jsonl` entry pointing at `skills/migrations/v0.14.0.md` for shell-jobs adoption. Registered in `src/commands/migrations/index.ts`.
|
||||
|
||||
#### Tests
|
||||
- New: `test/traverse-graph-dedup.test.ts`, `test/sync-failures.test.ts`, `test/brain-score-breakdown.test.ts`, `test/migration-resume.test.ts`, `test/migrations-v0_14_0.test.ts`.
|
||||
@@ -235,7 +986,7 @@ Three new migrations, all idempotent, apply automatically on `gbrain init` / upg
|
||||
- **Strict-mode default flip.** BrainWriter ships with `strict_mode=lint`. The flip to strict requires a 7-day soak + BrainBench regression ≤1pt + zero false-positive count.
|
||||
- **Sandboxed user plugins.** v0.13 ships builtins only. User-provided TS modules deferred pending a real isolation story (worker_threads or vm2) in a follow-on release.
|
||||
- **`openai_embedding` refactor.** Deferred to PR 1.5 post-flip; embedding is a hot path.
|
||||
- **Wintermute `claw-bridge`.** Adoption path is documentation-only this release.
|
||||
- **OpenClaw `claw-bridge`.** Adoption path is documentation-only this release.
|
||||
|
||||
### Tests
|
||||
|
||||
@@ -255,7 +1006,7 @@ Three new migrations, all idempotent, apply automatically on `gbrain init` / upg
|
||||
Four subcommands: `check` (read-only report with `--json`, `--type`, `--limit`), `auto` (three-bucket repair with `--confidence`, `--review-lower`, `--dry-run`, `--fresh`, `--limit`), `review` (prints queue path + count), `reset-progress`. Nine bare-tweet phrase regexes. External-link extraction for optional dead-link probing. Repairs route through `BrainWriter.transaction`.
|
||||
|
||||
#### BudgetLedger + CompletenessScorer (`src/core/enrichment/`)
|
||||
`BudgetLedger.reserve` returns `{kind:'held'}` or `{kind:'exhausted'}`. FOR UPDATE serializes concurrent reserves. `commit`, `rollback`, `cleanupExpired`. Midnight rollover via `Intl.DateTimeFormat` en-CA in configured IANA tz. Seven per-type rubrics + default (weights sum to 1.0). Person rubric's `non_redundancy` and `recency_score` kill Wintermute's length-only heuristic + 30-day-re-enrich-forever pathologies.
|
||||
`BudgetLedger.reserve` returns `{kind:'held'}` or `{kind:'exhausted'}`. FOR UPDATE serializes concurrent reserves. `commit`, `rollback`, `cleanupExpired`. Midnight rollover via `Intl.DateTimeFormat` en-CA in configured IANA tz. Seven per-type rubrics + default (weights sum to 1.0). Person rubric's `non_redundancy` and `recency_score` kill Garry's OpenClaw's length-only heuristic + 30-day-re-enrich-forever pathologies.
|
||||
|
||||
#### Minions scheduler polish (`src/core/minions/`)
|
||||
`quiet-hours.ts` — pure `evaluateQuietHours(cfg, now?)`. Wrap-around windows. Unknown tz fails open. `stagger.ts` — FNV-1a → 0–59 deterministic across runtimes. `worker.ts` integrated: post-claim evaluation, defer → `delayed/+15m`, skip → `cancelled`.
|
||||
@@ -622,7 +1373,7 @@ Your brain now wires itself. Every page write automatically extracts entity refe
|
||||
|
||||
- **Auto-link on every page write.** When you `gbrain put` a page that mentions `[Alice](people/alice)` or `[Acme](companies/acme)`, those links land in the graph automatically. Stale links (refs no longer in the page text) are removed in the same call. Run a quick `gbrain put` and the brain knows who's connected to whom. To opt out: `gbrain config set auto_link false`.
|
||||
- **Typed relationships.** Inferred from context using deterministic regex (zero LLM calls): `attended` (meeting -> person), `works_at` (CEO of, VP at, joined as), `invested_in` (invested in, backed by), `founded` (founded, co-founded), `advises` (advises, board member), `source` (frontmatter), `mentions` (default). On a 80-page benchmark brain: 94% type accuracy.
|
||||
- **`gbrain extract --source db`.** New mode for the existing `gbrain extract <links|timeline|all>` command that walks pages from the engine instead of from disk. Works for live brains backed by Postgres or PGLite without a local markdown checkout — exactly what an MCP-driven Wintermute or OpenClaw setup needs. Filesystem mode (`--source fs`) is unchanged and still the default.
|
||||
- **`gbrain extract --source db`.** New mode for the existing `gbrain extract <links|timeline|all>` command that walks pages from the engine instead of from disk. Works for live brains backed by Postgres or PGLite without a local markdown checkout — exactly what an MCP-driven OpenClaw setup needs. Filesystem mode (`--source fs`) is unchanged and still the default.
|
||||
- **`gbrain graph-query <slug>` for relationship traversal.** "Who works at Acme?" → `gbrain graph-query companies/acme --type works_at --direction in`. "Who attended meetings with Alice?" → `gbrain graph-query people/alice --type attended --depth 2`. Returns typed edges with depth, not just nodes. Backed by a new `traversePaths()` engine method on both PGLite and Postgres with cycle prevention (no exponential blowup on cyclic subgraphs).
|
||||
- **Graph-powered search ranking.** Hybrid search now applies a small backlink boost after cosine re-scoring (`score *= 1 + 0.05 * log(1 + backlink_count)`). Well-connected entities surface higher in results. Works in both keyword-only and full hybrid paths. Tested on the new `test/benchmark-graph-quality.ts` (80 pages, 35 queries, A/B/C comparison) — relational query recall jumps from ~30% (search alone) to 100% (graph traversal).
|
||||
- **Graph health metrics in `gbrain health`.** New `link_coverage` and `timeline_coverage` percentages on entity pages (person/company), plus `most_connected` top-5 list. The `dead_links` field is dropped (always 0 under ON DELETE CASCADE — was a phantom metric). The `brain_score` composite formula stays but now reflects a sharper graph signal.
|
||||
@@ -685,7 +1436,7 @@ CLI wrappers (`runExtract`, `runEmbed`, etc.) stay as thin arg-parsers that catc
|
||||
|
||||
### Added — skillify ships as a first-class gbrain skill
|
||||
|
||||
Ported from Wintermute, proven in production. Paired with `gbrain check-resolvable` gives a user-controllable equivalent of Hermes' auto-skill-creation — you decide when and what, the tooling keeps the 10-item checklist honest.
|
||||
Ported from Garry's OpenClaw, proven in production. Paired with `gbrain check-resolvable` gives a user-controllable equivalent of Hermes' auto-skill-creation — you decide when and what, the tooling keeps the 10-item checklist honest.
|
||||
|
||||
- `skills/skillify/SKILL.md` — the meta skill. Triggers: "skillify this", "is this a skill?", "make this proper".
|
||||
- `scripts/skillify-check.ts` — machine-readable audit. `--json` for CI, `--recent` to check files modified in the last 7 days.
|
||||
@@ -853,7 +1604,7 @@ Wave 3 fixes were contributed by **@garagon** (PRs #105-#109) and **@Hybirdss**
|
||||
| **cron-scheduler** | Schedule staggering (5-min offsets), quiet hours (timezone-aware with wake-up override), thin job prompts. | 21 cron jobs at :00 is a thundering herd. Staggering prevents it. Quiet hours mean no 3 AM notifications. Wake-up override releases the backlog. |
|
||||
| **reports** | Timestamped reports with keyword routing. "What's the latest briefing?" maps to the right report directory. | Cheap replacement for vector search on frequent queries. Don't embed. Load the file. |
|
||||
| **testing** | Validates every skill has SKILL.md with frontmatter, manifest coverage, resolver coverage. The CI for your skill system. | 3 skills and you need validation. 24 skills and you need it yesterday. Catches dead references, missing sections, MECE violations. |
|
||||
| **soul-audit** | 6-phase interview that generates SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md. Your agent's identity, built from your answers. | What makes Wintermute feel like Wintermute. Without personality and access control, every agent feels the same. |
|
||||
| **soul-audit** | 6-phase interview that generates SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md. Your agent's identity, built from your answers. | What makes your OpenClaw feel like yours. Without personality and access control, every agent feels the same. |
|
||||
| **webhook-transforms** | External events (SMS, meetings, social mentions) converted into brain pages with entity extraction. Dead-letter queue for failures. | Your brain ingests signals from everywhere. Not just conversations, but every webhook, every notification, every external event. |
|
||||
|
||||
### Infrastructure (new in v0.10.0)
|
||||
|
||||
@@ -23,9 +23,9 @@ strict behavior when unset.
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`).
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness).
|
||||
@@ -43,6 +43,8 @@ strict behavior when unset.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic.
|
||||
- `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `deferred[]` array surfaces pending Checks 5 (trigger routing eval) and 6 (brain filing) with issue URLs. `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass.
|
||||
- `src/core/dry-fix.ts` — `gbrain doctor --fix` engine. `autoFixDryViolations(fixes, {dryRun})` rewrites inlined rules to `> **Convention:** see [path](path).` callouts via three shape-aware expanders (bullet / blockquote / paragraph). Five guards: working-tree-dirty (`getWorkingTreeStatus()` returns 3-state `'clean' | 'dirty' | 'not_a_repo'`), no-git-backup, inside-code-fence, already-delegated (40-line proximity, consistent with detector), ambiguous-multi-match, block-is-callout. `execFileSync` array args (no shell — no injection surface). EOF newline preserved.
|
||||
- `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier
|
||||
- `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation
|
||||
@@ -52,25 +54,46 @@ strict behavior when unset.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs).
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types)
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe).
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
- `src/core/minions/rate-leases.ts` (v0.15) — lease-based concurrency cap for outbound providers (default key `anthropic:messages`, max via `GBRAIN_ANTHROPIC_MAX_INFLIGHT`). Owner-tagged rows with `expires_at` auto-prune on acquire; `pg_advisory_xact_lock` guards check-then-insert; CASCADE on owning job deletion. `renewLeaseWithBackoff` retries 3x (250/500/1000ms).
|
||||
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
|
||||
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
|
||||
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
|
||||
- `src/core/minions/tools/brain-allowlist.ts` (v0.15) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list: `query`, `search`, `get_page`, `list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`, `resolve_slugs`, `get_ingest_log`, `put_page`. `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`); the `put_page` op's server-side check is the authoritative gate via `ctx.viaSubagent` fail-closed.
|
||||
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
|
||||
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon
|
||||
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
|
||||
- `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs <job> [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs.
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern).
|
||||
- `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman
|
||||
- `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed)
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3, pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run]`: health checks. v0.12.3 adds two reliability detection checks: `jsonb_integrity` (scans pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata for `jsonb_typeof='string'` rows left over from v0.12.0) and `markdown_body_completeness` (flags pages whose compiled_truth is <30% of raw source when raw has multiple H2/H3 boundaries). Fix hints point at `gbrain repair-jsonb` and `gbrain sync --force`. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock.
|
||||
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI. ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config (no more walk-up-cwd-for-.git footgun). Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. Exit code 1 on status=failed (partial/warn not fatal — don't page on warnings).
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
- `docs/progress-events.md` — Canonical JSON event schema reference. Stable from v0.15.2, additive only.
|
||||
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces the `${JSON.stringify(x)}::jsonb` interpolation pattern (which postgres.js v3 double-encodes). Wired into `bun test`.
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (v0.15.1 #219 regression guard — must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
|
||||
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env var lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output is not analogous to `schema-embedded.ts` (no runtime consumer); we commit for GitHub browsing and fork-safe fetching.
|
||||
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
|
||||
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
|
||||
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
|
||||
@@ -130,12 +153,13 @@ Key commands added in v0.7:
|
||||
- `gbrain migrate --to supabase` / `gbrain migrate --to pglite` — bidirectional engine migration
|
||||
|
||||
Key commands added for Minions (job queue):
|
||||
- `gbrain jobs submit <name> [--params JSON] [--follow] [--dry-run]` — submit a background job
|
||||
- `gbrain jobs submit <name> [--params JSON] [--follow] [--dry-run]` — submit a background job. v0.13.1 adds first-class flags for every `MinionJobInput` tuning knob: `--max-stalled N`, `--backoff-type fixed|exponential`, `--backoff-delay Nms`, `--backoff-jitter 0..1`, `--timeout-ms N`, `--idempotency-key K`.
|
||||
- `gbrain jobs list [--status S] [--queue Q]` — list jobs with filters
|
||||
- `gbrain jobs get <id>` — job details with attempt history
|
||||
- `gbrain jobs cancel/retry/delete <id>` — manage job lifecycle
|
||||
- `gbrain jobs prune [--older-than 30d]` — clean old completed/dead jobs
|
||||
- `gbrain jobs stats` — job health dashboard
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.12.2:
|
||||
@@ -152,6 +176,12 @@ Key commands added in v0.14.2:
|
||||
- `GBRAIN_POOL_SIZE` env var — honored by both the singleton pool (`src/core/db.ts`) and the parallel-import worker pool (`src/commands/import.ts`). Default is 10; lower to 2 for Supabase transaction pooler to avoid MaxClients crashes during `gbrain upgrade` subprocess spawns. Read at call time via `resolvePoolSize()`.
|
||||
- `gbrain doctor` gains two new checks: `sync_failures` (surfaces unacknowledged parse failures with exact paths + fix hints) and `brain_score` (renders the 5-component breakdown when score < 100: embed coverage / 35, link density / 25, timeline coverage / 15, orphans / 15, dead links / 10 — sum equals total).
|
||||
|
||||
Key commands added in v0.14.3 (fix wave):
|
||||
- `gbrain doctor --index-audit` — opt-in Postgres-only check reporting zero-scan indexes from `pg_stat_user_indexes`. Informational only; never auto-drops.
|
||||
- `gbrain doctor` schema_version check fails loudly when `version=0` — catches `bun install -g github:...` postinstall failures (#218) and routes users to `gbrain apply-migrations --yes`.
|
||||
- `gbrain jobs submit` gains `--max-stalled`, `--backoff-type`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key` — exposing existing `MinionJobInput` fields as first-class CLI flags.
|
||||
- `gbrain jobs smoke --sigkill-rescue` — opt-in regression smoke case simulating a killed worker; asserts the v0.14.3 schema default (`max_stalled=5`) actually rescues on first stall.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
@@ -163,11 +193,11 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100 + v0.13.1 `connect()` error-wrap assertion (original error nested, #223 link in message, lock released)),
|
||||
`test/engine-factory.test.ts` (engine factory + dynamic imports),
|
||||
`test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation),
|
||||
`test/publish.test.ts` (content stripping, encryption, password generation, HTML output),
|
||||
@@ -188,7 +218,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/transcription.test.ts` (provider detection, format validation, API key errors),
|
||||
`test/enrichment-service.test.ts` (entity slugification, extraction, tier escalation),
|
||||
`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping),
|
||||
`test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail),
|
||||
`test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail + v0.13.1 `max_stalled` clamp/default/plumbing coverage),
|
||||
`test/extract.test.ts` (link extraction, timeline extraction, frontmatter parsing, directory type inference),
|
||||
`test/extract-db.test.ts` (gbrain extract --source db: typed link inference, idempotency, --type filter, --dry-run JSON output),
|
||||
`test/extract-fs.test.ts` (gbrain extract --source fs: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard — the v0.12.1 N+1 dedup bug),
|
||||
@@ -205,7 +235,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/postgres-engine.test.ts` (v0.12.3 statement_timeout scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against reintroduced bare `SET statement_timeout`),
|
||||
`test/sync.test.ts` (sync logic + v0.12.3 regression guard asserting top-level `engine.transaction` is not called),
|
||||
`test/doctor.test.ts` (doctor command + v0.12.3 assertions that `jsonb_integrity` scans the four v0.12.0 write sites and `markdown_body_completeness` is present),
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics).
|
||||
`test/utils.test.ts` (shared SQL utilities + `tryParseEmbedding` null-return and single-warn semantics),
|
||||
`test/build-llms.test.ts` (llms.txt/llms-full.txt generator: path resolution, idempotence, spec shape, regen-drift guard, content contract, AGENTS.md install-path mirror, size-budget enforcement — 7 cases).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
@@ -281,6 +312,38 @@ testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
|
||||
## Bulk-action progress reporting
|
||||
|
||||
All bulk commands (doctor, embed, import, export, sync, extract, migrate,
|
||||
repair-jsonb, orphans, check-backlinks, lint, integrity auto, eval, files
|
||||
sync, and apply-migrations) stream progress through the shared reporter
|
||||
at `src/core/progress.ts`. Agents get heartbeats within 1 second of every
|
||||
iteration regardless of how slow the underlying work is.
|
||||
|
||||
Rules:
|
||||
- Progress always writes to **stderr**. Stdout stays clean for data output
|
||||
(`--json` payloads, final summaries, JSON action events from `extract`).
|
||||
- Non-TTY default: plain one-line-per-event human text. JSON requires the
|
||||
explicit `--progress-json` flag.
|
||||
- Global flags (`--quiet`, `--progress-json`, `--progress-interval=<ms>`)
|
||||
are parsed by `src/core/cli-options.ts` BEFORE command dispatch.
|
||||
- Phase names are machine-stable `snake_case.dot.path` (e.g.
|
||||
`doctor.db_checks`, `sync.imports`). Documented in
|
||||
`docs/progress-events.md`; additive changes only.
|
||||
- `scripts/check-progress-to-stdout.sh` is a CI guard that fails the build
|
||||
if any new code writes `\r` progress to stdout. Wired into `bun run test`.
|
||||
- Minion handlers pass `job.updateProgress` as the `onProgress` callback
|
||||
to core functions (DB-backed primary progress channel); stderr from
|
||||
`jobs work` stays coarse for daemon liveness only.
|
||||
|
||||
When wiring a new bulk command: `import { createProgress } from '../core/progress.ts'`
|
||||
and `import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'`.
|
||||
Create a reporter with `createProgress(cliOptsToProgressOptions(getCliOptions()))`,
|
||||
`start(phase, total?)` before the loop, `tick()` inside it, `finish()` after.
|
||||
For single long-running queries, use `startHeartbeat(reporter, note)` with a
|
||||
try/finally to guarantee cleanup. Never call `process.stdout.write('\r...')`
|
||||
in bulk paths, the CI guard will fail the build.
|
||||
|
||||
## Build
|
||||
|
||||
`bun build --compile --outfile bin/gbrain src/cli.ts`
|
||||
@@ -509,6 +572,21 @@ search engines, surfaced in cross-references, and distributed with every release
|
||||
- Example meeting → `meetings/2026-04-03` (generic date is fine)
|
||||
- Example user → `you` or `the user`, never a proper name
|
||||
|
||||
**Specific rule: never say `Wintermute` in any CHANGELOG, README, doc, PR, or
|
||||
commit message.** When the temptation is to illustrate with the real fork name:
|
||||
- Reader-facing copy → `your OpenClaw` (covers Wintermute, Hermes, AlphaClaw,
|
||||
and any other downstream OpenClaw deployment in one term the reader already
|
||||
recognizes).
|
||||
- First-person / origin-story copy → `Garry's OpenClaw` (honest that this is
|
||||
the production deployment driving the feature, without exposing the private
|
||||
agent's name).
|
||||
|
||||
`Wintermute` may appear in private artifacts (scratch plans under
|
||||
`~/.gstack/projects/…`, memory files, conversation transcripts, CEO-review
|
||||
plans) — those aren't distributed. Anything checked into this repo or shipped
|
||||
in a release must use the OpenClaw phrasing above. Sweeping a stale reference
|
||||
is a small clean-up PR, not a debate.
|
||||
|
||||
**When in doubt, ask yourself:** "Would this query reveal private information
|
||||
about the user's contacts, investments, or portfolio if it were read by a
|
||||
stranger?" If yes, replace with generic placeholders.
|
||||
|
||||
+17
-1
@@ -3,6 +3,17 @@
|
||||
Read this entire file, then follow the steps. Ask the user for API keys when needed.
|
||||
Target: ~30 minutes to a fully working brain.
|
||||
|
||||
## Step 0: If you are not Claude Code
|
||||
|
||||
Read `AGENTS.md` at the repo root first. It's the non-Claude-agent operating
|
||||
protocol (install, read order, trust boundary, common tasks). Claude Code reads
|
||||
`CLAUDE.md` automatically and can skip ahead.
|
||||
|
||||
If you fetched this file by URL without cloning yet, the companion files live at:
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` — start here
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms.txt` — full doc map
|
||||
- `https://raw.githubusercontent.com/garrytan/gbrain/master/llms-full.txt` — same map, inlined
|
||||
|
||||
## Step 1: Install GBrain
|
||||
|
||||
```bash
|
||||
@@ -15,6 +26,11 @@ bun install && bun link
|
||||
Verify: `gbrain --version` should print a version number. If `gbrain` is not found,
|
||||
restart the shell or add the PATH export to the shell profile.
|
||||
|
||||
> **Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
> postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
> aborts with `Aborted()` when it opens PGLite. Use the `git clone + bun link` path
|
||||
> above. Tracking issue: [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
## Step 2: API Keys
|
||||
|
||||
Ask the user for these:
|
||||
@@ -133,7 +149,7 @@ actually works) is the most important.
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
cd ~/gbrain && git pull origin main && bun install
|
||||
cd ~/gbrain && git pull origin master && bun install
|
||||
gbrain init # apply schema migrations (idempotent)
|
||||
gbrain post-upgrade # show migration notes for the version range
|
||||
```
|
||||
|
||||
@@ -10,6 +10,8 @@ GBrain is those patterns, generalized. 26 skills. Install in 30 minutes. Your ag
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
|
||||
|
||||
## Install
|
||||
|
||||
### On an agent platform (recommended)
|
||||
@@ -28,6 +30,11 @@ https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 26 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
agent operating protocol (install, read order, trust boundary, common tasks). For
|
||||
the full doc map, use `llms.txt` at the same URL root.
|
||||
|
||||
### Standalone CLI (no agent)
|
||||
|
||||
```bash
|
||||
@@ -37,6 +44,11 @@ gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
```
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
&& bun link` as shown above. See [#218](https://github.com/garrytan/gbrain/issues/218).
|
||||
|
||||
```
|
||||
3 results (hybrid search, 0.12s):
|
||||
|
||||
@@ -218,6 +230,25 @@ If anything's off, `actions[]` tells you the exact command to run. For deeper tr
|
||||
|
||||
Moving gateway crons to Minions (deterministic scripts, zero LLM tokens per fire): [`docs/guides/minions-shell-jobs.md`](docs/guides/minions-shell-jobs.md).
|
||||
|
||||
## Durable agents: `gbrain agent` (v0.15)
|
||||
|
||||
Your subagent runs survive crashes now. OpenClaw died mid-run? The worker re-claims on restart and replays from the last committed turn. Fan-out across 50 shards, one shard crashes — the aggregator still claims after every child reaches a terminal state and writes a mixed-outcome summary. Tool calls persist as a two-phase ledger (`pending` → `complete | failed`) so replay is safe by construction, not by hope.
|
||||
|
||||
```bash
|
||||
# Submit a single-subagent run
|
||||
gbrain agent run "summarize my last 10 journal pages"
|
||||
|
||||
# Fan out N prompts across N subagent children + 1 aggregator
|
||||
gbrain agent run "analyze every page" \
|
||||
--fanout-manifest manifests/pages.json \
|
||||
--subagent-def analyzer
|
||||
|
||||
# Tail a running job (heartbeat per turn + full transcript on completion)
|
||||
gbrain agent logs 1247 --follow --since 5m
|
||||
```
|
||||
|
||||
Durability is the point: every Anthropic turn commits to `subagent_messages`, every tool call to `subagent_tool_executions`. Worker kills, OpenClaw crashes, timeouts — all resumable. Host repos (your OpenClaw, etc.) ship their own subagent definitions via `GBRAIN_PLUGIN_PATH` + a `gbrain.plugin.json` manifest: see [`docs/guides/plugin-authors.md`](docs/guides/plugin-authors.md). Requires `ANTHROPIC_API_KEY` on the worker.
|
||||
|
||||
## Skillify: your skills tree stops being a black box
|
||||
|
||||
Hermes and similar agent frameworks auto-create skills as a background behavior. Fine until you don't know what the agent shipped. Checklists decay. Tests drift. Resolver entries get stale. Six months later you've got an opaque pile of "skills" that nobody has read, nobody has tested, and nobody is sure still work.
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# TODOS
|
||||
|
||||
## check-resolvable
|
||||
|
||||
### File tracking issues for Checks 5 + 6 (deferred in PR #325)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** `src/commands/check-resolvable.ts` currently points `DEFERRED[].issue` at GitHub issue search URLs (`?q=TBD-check-5`, `?q=TBD-check-6`). File real tracking issues and grep-replace both placeholders with the real URLs.
|
||||
|
||||
**Why:** v0.16.4 shipped `gbrain check-resolvable` with 4 of the 6 checks from the original spec. Checks 5 (trigger routing eval) and 6 (brain filing) were explicitly deferred during plan-ceo-review because they each need new detection logic. The CLI's `deferred[]` JSON field is meant to surface these to agents so they know the coverage boundary — the TBD placeholders do the right thing mechanically but aren't clickable.
|
||||
|
||||
**How:**
|
||||
1. `gh issue create -t "check-resolvable Check 5: trigger routing eval" -b "..."` — detection: every skill's own frontmatter trigger should match the RESOLVER.md entry pointing at that skill. Needs new issue type (e.g. `mis_route`).
|
||||
2. `gh issue create -t "check-resolvable Check 6: brain filing validation" -b "..."` — detection: scan SKILL.md body for brain paths (e.g., `brain/people/`, `brain/companies/`), cross-reference with `skills/_brain-filing-rules.md`. Flag mutating skills missing entries.
|
||||
3. Replace `TBD-check-5` and `TBD-check-6` in `src/commands/check-resolvable.ts` with the real issue URLs.
|
||||
|
||||
**Effort:** ~15 min mechanical (issue filing + grep-replace). Implementation of the checks themselves is a separate, larger piece of work — the TODO here is just the issue filing + URL swap.
|
||||
|
||||
## P1 (BrainBench v1.1 — categories deferred from PR #188)
|
||||
|
||||
### BrainBench Cat 5: Source Attribution / Provenance
|
||||
@@ -173,7 +189,7 @@ board" — likely an advisor-role page prior plus verb-pattern combinations.
|
||||
|
||||
**Cons:** Requires adding `sender_id` or `access_tier` to `OperationContext`. Each mutating operation needs a permission check. Medium implementation effort.
|
||||
|
||||
**Context:** From CEO review + Codex outside voice (2026-04-13). Prompt-layer access control works in practice (same model as Wintermute) but is not sufficient for remote MCP where direct tool calls bypass the agent's prompt.
|
||||
**Context:** From CEO review + Codex outside voice (2026-04-13). Prompt-layer access control works in practice (same model as Garry's OpenClaw) but is not sufficient for remote MCP where direct tool calls bypass the agent's prompt.
|
||||
|
||||
**Depends on:** v0.10.0 GStackBrain skill layer (shipped).
|
||||
|
||||
|
||||
@@ -7,19 +7,25 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@electric-sql/pglite": "^0.4.4",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6",
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
},
|
||||
"trustedDependencies": [
|
||||
"@electric-sql/pglite",
|
||||
],
|
||||
"packages": {
|
||||
"@anthropic-ai/sdk": ["@anthropic-ai/sdk@0.30.1", "", { "dependencies": { "@types/node": "^18.11.18", "@types/node-fetch": "^2.6.4", "abort-controller": "^3.0.0", "agentkeepalive": "^4.2.1", "form-data-encoder": "1.7.2", "formdata-node": "^4.3.2", "node-fetch": "^2.6.7" } }, "sha512-nuKvp7wOIz6BFei8WrTdhmSsx5mwnArYyJgh4+vYu3V4J0Ltb8Xm3odPm51n1aSI0XxNCrDl7O88cxCtUdAkaw=="],
|
||||
|
||||
@@ -103,7 +109,7 @@
|
||||
|
||||
"@aws/lambda-invoke-store": ["@aws/lambda-invoke-store@0.2.4", "", {}, "sha512-iY8yvjE0y651BixKNPgmv1WrQc+GZ142sb0z4gYnChDDY2YqI4P/jsSopBWrKfAt7LOJAkOXt7rC/hms+WclQQ=="],
|
||||
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.4", "", {}, "sha512-g/6CWAJ4XOkObWCWAQ2IReZD8VvsDy3poRHSKvpRR2F96F8WJ3HVbjpso3gN7l0q6QPPgvxSSpl/qo5k8a7mkQ=="],
|
||||
"@electric-sql/pglite": ["@electric-sql/pglite@0.4.3", "", {}, "sha512-ichuWTgtd4mOM1G4SpyGJa5trT03lWbMypDV0fUXUCXg5hiHqVAz/bZyV68NqmkLB7WcYmj1RMJVSp8HV/v/ZQ=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@1.19.12", "", { "peerDependencies": { "hono": "^4" } }, "sha512-txsUW4SQ1iilgE0l9/e9VQWmELXifEFvmdA1j6WFh/aFPj99hIntrSsq/if0UWyGVkmrRPKA1wCeP+UCr1B9Uw=="],
|
||||
|
||||
@@ -449,10 +455,14 @@
|
||||
|
||||
"tr46": ["tr46@0.0.3", "", {}, "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw=="],
|
||||
|
||||
"tree-sitter-wasms": ["tree-sitter-wasms@0.1.13", "", { "dependencies": { "tree-sitter-wasms": "^0.1.11" } }, "sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ=="],
|
||||
|
||||
"tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
|
||||
|
||||
"type-is": ["type-is@2.0.1", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="],
|
||||
|
||||
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
|
||||
|
||||
"undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="],
|
||||
|
||||
"unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],
|
||||
@@ -461,6 +471,8 @@
|
||||
|
||||
"web-streams-polyfill": ["web-streams-polyfill@4.0.0-beta.3", "", {}, "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug=="],
|
||||
|
||||
"web-tree-sitter": ["web-tree-sitter@0.22.6", "", {}, "sha512-hS87TH71Zd6mGAmYCvlgxeGDjqd9GTeqXNqTT+u0Gs51uIozNIaaq/kUAbV/Zf56jb2ZOyG8BxZs2GG9wbLi6Q=="],
|
||||
|
||||
"webidl-conversions": ["webidl-conversions@3.0.1", "", {}, "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ=="],
|
||||
|
||||
"whatwg-url": ["whatwg-url@5.0.0", "", { "dependencies": { "tr46": "~0.0.3", "webidl-conversions": "^3.0.0" } }, "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw=="],
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
[test]
|
||||
# PGLite initialization can be slow under parallel test execution.
|
||||
# Default 5s is too short when many test files boot PGLite instances at once.
|
||||
# 60s is the empirical ceiling we observed before the first file's beforeAll
|
||||
# completed on a loaded machine.
|
||||
timeout = 60_000
|
||||
@@ -358,6 +358,106 @@ upcoming `gbrain crontab-to-minions <file>` helper is P1 in TODOS.
|
||||
|
||||
---
|
||||
|
||||
## v0.16.0: durable agent runtime
|
||||
|
||||
v0.15 ships `gbrain agent run` / `gbrain agent logs`, a new `subagent` handler
|
||||
type in Minions, and a plugin contract for host-repo subagent defs. None of the
|
||||
existing skills need surgery. The question for downstream agents is *how* to
|
||||
adopt the new runtime, not how to patch around a breaking change.
|
||||
|
||||
### 1. Run a worker with an Anthropic key
|
||||
|
||||
The subagent handlers (`subagent` and `subagent_aggregator`) are always
|
||||
registered on the worker. No separate opt-in flag — `ANTHROPIC_API_KEY` is
|
||||
the natural cost gate (no key, the SDK call fails on the first turn), and
|
||||
who-can-submit is already protected (`PROTECTED_JOB_NAMES` + trusted-submit:
|
||||
MCP callers get `permission_denied`; only `gbrain agent run` can insert
|
||||
these rows).
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY=sk-ant-... gbrain jobs work
|
||||
```
|
||||
|
||||
Worker startup prints:
|
||||
|
||||
```
|
||||
[minion worker] subagent handlers enabled
|
||||
```
|
||||
|
||||
### 2. Ship your subagents as a plugin (OpenClaw + similar)
|
||||
|
||||
Move your custom subagent definitions out of your gbrain fork and into your own
|
||||
repo as a plugin. Concretely:
|
||||
|
||||
```
|
||||
~/<your-agent>/gbrain-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
├── meeting-ingestion.md
|
||||
├── signal-detector.md
|
||||
└── daily-task-prep.md
|
||||
```
|
||||
|
||||
`gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "your-openclaw",
|
||||
"version": "2026.4.20",
|
||||
"plugin_version": "gbrain-plugin-v1"
|
||||
}
|
||||
```
|
||||
|
||||
Each `subagents/*.md` is a plain-text agent definition — YAML frontmatter +
|
||||
body-as-system-prompt. Recognized frontmatter fields: `name`, `model`,
|
||||
`max_turns`, `allowed_tools` (must subset the derived brain-tool registry).
|
||||
|
||||
Turn it on:
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="$HOME/<your-agent>/gbrain-plugin"
|
||||
```
|
||||
|
||||
Worker startup prints `[plugin-loader] loaded '<name>' v<ver> (N subagents)`
|
||||
per plugin; any rejection (bad manifest, unknown tool in `allowed_tools`,
|
||||
version mismatch) shows up as a loud warning at startup, not a silent dispatch-
|
||||
time failure. See `docs/guides/plugin-authors.md` for the full contract.
|
||||
|
||||
### 3. Replace ephemeral subagent runs with durable ones
|
||||
|
||||
If your agent currently spawns ephemeral subagents (OpenClaw `Agent()`, ad-hoc
|
||||
Anthropic API calls, etc.) for work that should survive crashes, sleeps, or
|
||||
worker restarts, migrate those to `gbrain agent run`. The durability is free:
|
||||
|
||||
```bash
|
||||
gbrain agent run "analyze my last 50 journal pages for recurring themes" \
|
||||
--subagent-def analyzer --fanout-manifest manifests/journal-pages.json
|
||||
```
|
||||
|
||||
Every turn persists to `subagent_messages`, every tool call is a two-phase
|
||||
ledger, and `gbrain agent logs <job>` shows where it died + what the last
|
||||
successful call returned. No more "re-run from scratch because the session
|
||||
context evaporated."
|
||||
|
||||
### 4. `put_page` from subagents writes under an agent namespace
|
||||
|
||||
If you adopted the v0.15 subagent runtime, note that `put_page` calls
|
||||
originating from a subagent's tool dispatch MUST target
|
||||
`wiki/agents/<subagent_id>/...`. The schema shown to the model enforces this
|
||||
on first try; a server-side fail-closed check rejects anything else. This
|
||||
does NOT affect your skill files, CLI put_page calls, or MCP put_page —
|
||||
only tool-dispatched writes from inside an LLM loop.
|
||||
|
||||
Aggregation output (the final "here's what all N children found" brain page)
|
||||
goes via a separate trusted CLI path, not through a subagent tool call, so
|
||||
it can write anywhere you want.
|
||||
|
||||
Iron rule: **never grant an agent write access beyond its namespace**. The
|
||||
server-side check exists because dispatcher bugs happen; treat it as defense
|
||||
in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Production Benchmark: Minions vs OpenClaw Sub-agents (Real Deployment)
|
||||
|
||||
**Date:** 2026-04-18
|
||||
**Environment:** Wintermute on Render (ephemeral container, Supabase Postgres)
|
||||
**Environment:** Garry's OpenClaw on Render (ephemeral container, Supabase Postgres)
|
||||
**GBrain:** v0.11.0 (minions-jobs branch)
|
||||
**OpenClaw:** 2026.4.10
|
||||
**Brain:** 45,798 pages, 98K chunks, 25K links, 79K timeline entries
|
||||
|
||||
@@ -8,9 +8,9 @@
|
||||
|
||||
## 0. Context
|
||||
|
||||
During a CEO review of a narrow two-feature plan (bare-tweet citation repair + completeness score, borrowed from Feynman), the scope was reframed. The narrow plan duplicated work Wintermute already does and missed the real leverage point: **the bespoke abstractions hiding inside Wintermute — resolvers, enrichment orchestration, scheduling, deterministic output — should live in GBrain as first-class primitives.**
|
||||
During a CEO review of a narrow two-feature plan (bare-tweet citation repair + completeness score, borrowed from Feynman), the scope was reframed. The narrow plan duplicated work Garry's OpenClaw already does and missed the real leverage point: **the bespoke abstractions hiding inside OpenClaw — resolvers, enrichment orchestration, scheduling, deterministic output — should live in GBrain as first-class primitives.**
|
||||
|
||||
North star: *"When Wintermute's Claw upgrades to this version of GBrain, it should immediately recognize brilliance and completeness and say 'It's time to switch to these abstractions.'"*
|
||||
North star: *"When Garry's OpenClaw's Claw upgrades to this version of GBrain, it should immediately recognize brilliance and completeness and say 'It's time to switch to these abstractions.'"*
|
||||
|
||||
That is the test this document is designed against. Everything else is downstream.
|
||||
|
||||
@@ -67,7 +67,7 @@ An earlier implementation could ship L1 + L4 first (the two "purest" layers) and
|
||||
|
||||
### 3.1 What's broken today
|
||||
|
||||
Wintermute has **69 distinct external-lookup patterns** across X API (14 shapes), Perplexity, Mistral OCR, Gmail, Calendar, Slack, GitHub, YouTube, Diarize.io, YC tools, OSINT collectors, and brain-local lookups. Each one is a bespoke script under `scripts/` with its own error handling, retry logic, and output shape. GBrain has 3 ad-hoc wrappers (`embedding.ts`, `transcription.ts`, `enrichment-service.ts`) that don't share an interface.
|
||||
Garry's OpenClaw has **69 distinct external-lookup patterns** across X API (14 shapes), Perplexity, Mistral OCR, Gmail, Calendar, Slack, GitHub, YouTube, Diarize.io, YC tools, OSINT collectors, and brain-local lookups. Each one is a bespoke script under `scripts/` with its own error handling, retry logic, and output shape. GBrain has 3 ad-hoc wrappers (`embedding.ts`, `transcription.ts`, `enrichment-service.ts`) that don't share an interface.
|
||||
|
||||
Common consequences:
|
||||
- No uniform retry/backoff strategy (some scripts retry, most don't)
|
||||
@@ -187,7 +187,7 @@ Existing `src/core/fail-improve.ts` is the deterministic-first/LLM-fallback patt
|
||||
|
||||
### 3.7 Reference implementations to ship
|
||||
|
||||
The Wintermute survey inventoried 69 resolver shapes. Shipping all of them is wrong (over-scoped); shipping zero is under-scoped. The dogfood set:
|
||||
The OpenClaw survey inventoried 69 resolver shapes. Shipping all of them is wrong (over-scoped); shipping zero is under-scoped. The dogfood set:
|
||||
|
||||
| # | Resolver | Purpose | Used by |
|
||||
|---|---|---|---|
|
||||
@@ -198,7 +198,7 @@ The Wintermute survey inventoried 69 resolver shapes. Shipping all of them is wr
|
||||
| 5 | `perplexity_query` | Query → synthesis + citations | Enrichment Orchestrator |
|
||||
| 6 | `text_to_entities` | LLM entity extraction (structured JSON) | Enrichment Orchestrator |
|
||||
|
||||
The remaining 63 Wintermute patterns port incrementally, driven by user need. Each port is a new YAML + module under `recipes/` or `~/.gbrain/resolvers/` with no framework changes.
|
||||
The remaining 63 OpenClaw patterns port incrementally, driven by user need. Each port is a new YAML + module under `recipes/` or `~/.gbrain/resolvers/` with no framework changes.
|
||||
|
||||
---
|
||||
|
||||
@@ -206,7 +206,7 @@ The remaining 63 Wintermute patterns port incrementally, driven by user need. Ea
|
||||
|
||||
### 4.1 What's broken today
|
||||
|
||||
Wintermute's enrichment is **polished at the data layer, hacky at the control layer**:
|
||||
Garry's OpenClaw's enrichment is **polished at the data layer, hacky at the control layer**:
|
||||
|
||||
- **Completeness = "length > 500 chars + no `needs-enrichment` tag"** (`lib/enrich.mjs:351-355`). Naïve. A rich page of repetitive Perplexity summaries (see `brain/people/0interestrates.md` — 38 repeating blocks) passes this check.
|
||||
- **30-day auto-re-enrichment** runs forever. No "done" state. A person met once in 2023 still gets re-researched monthly.
|
||||
@@ -342,9 +342,9 @@ await writer.transaction(async (tx) => {
|
||||
|
||||
### 5.1 What's broken today
|
||||
|
||||
Wintermute's cron is **externally-driven JSON** (`cron/jobs.json`) with ~30 jobs manually stagger-offset at different minutes. GBrain has **zero native scheduling** — `src/commands/autopilot.ts` is a single daemon loop, and `docs/guides/cron-schedule.md` is architectural guidance, not code.
|
||||
Garry's OpenClaw's cron is **externally-driven JSON** (`cron/jobs.json`) with ~30 jobs manually stagger-offset at different minutes. GBrain has **zero native scheduling** — `src/commands/autopilot.ts` is a single daemon loop, and `docs/guides/cron-schedule.md` is architectural guidance, not code.
|
||||
|
||||
Failures observed in Wintermute's actual state:
|
||||
Failures observed in Garry's OpenClaw's actual state:
|
||||
- `X OAuth2 Token Refresh`: 11 consecutive timeouts (critical-path silent failure)
|
||||
- `flight-tracker daily scan`: 5 consecutive timeouts
|
||||
- `morning-briefing`: 4 consecutive timeouts
|
||||
@@ -378,9 +378,9 @@ export interface ScheduledResolver extends Resolver<void, ScheduledResult> {
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Enforcement vs convention (the key delta from Wintermute)
|
||||
### 5.3 Enforcement vs convention (the key delta from Garry's OpenClaw)
|
||||
|
||||
| Concern | Wintermute today | Knowledge Runtime |
|
||||
| Concern | Garry's OpenClaw today | Knowledge Runtime |
|
||||
|---|---|---|
|
||||
| Quiet hours | Checked inside each skill (trust-based) | Enforced at scheduler, skill cannot override |
|
||||
| Staggering | Manual minute-offset in `jobs.json` | Scheduler assigns slots via hashed staggerKey |
|
||||
@@ -405,7 +405,7 @@ Every scheduled run emits structured events: `started`, `skipped-quiet-hours`, `
|
||||
- `engine.logIngest` (audit trail in brain DB)
|
||||
- Optional webhook (Slack/Telegram for the user)
|
||||
|
||||
`gbrain doctor` reads the event log and reports: current circuit-breaker state, any resolver with > 3 consecutive failures, any resolver that hasn't fired within 3× its interval (freshness SLA like Wintermute's `freshness-check.mjs` but built-in).
|
||||
`gbrain doctor` reads the event log and reports: current circuit-breaker state, any resolver with > 3 consecutive failures, any resolver that hasn't fired within 3× its interval (freshness SLA like Garry's OpenClaw's `freshness-check.mjs` but built-in).
|
||||
|
||||
---
|
||||
|
||||
@@ -415,9 +415,9 @@ Every scheduled run emits structured events: `started`, `skipped-quiet-hours`, `
|
||||
|
||||
**Iron Law: LLM picks WHAT. Code guarantees WHERE and HOW.**
|
||||
|
||||
Wintermute's existing `lib/enrich.mjs:buildTweetEntry` is close to this — tweet URLs are built from `tweet.id` returned by the X API, never from LLM memory. But:
|
||||
Garry's OpenClaw's existing `lib/enrich.mjs:buildTweetEntry` is close to this — tweet URLs are built from `tweet.id` returned by the X API, never from LLM memory. But:
|
||||
|
||||
- A past incident: *"Sub-agent test #2 FAILED — hallucinated 'Philip Leung' entity links across all daily files. LLM rewriting of daily files is too error-prone."* (Wintermute memory log, 2026-04-13.)
|
||||
- A past incident: *"Sub-agent test #2 FAILED — hallucinated 'Philip Leung' entity links across all daily files. LLM rewriting of daily files is too error-prone."* (Garry's OpenClaw memory log, 2026-04-13.)
|
||||
- Back-links depend on `appendTimeline` being called everywhere; skips are silent.
|
||||
- Slug collisions are unchecked (no conflict detection on `slugify`).
|
||||
- Citation format is post-hoc linted weekly, not pre-write enforced.
|
||||
@@ -461,7 +461,7 @@ export class Scaffolder {
|
||||
// "[Source: [X/garrytan, 2026-04-18](https://x.com/garrytan/status/123456)]"
|
||||
}
|
||||
emailCitation(account: string, messageId: string, subject: string): string {
|
||||
// deterministic Gmail URL per Wintermute pattern
|
||||
// deterministic Gmail URL per OpenClaw pattern
|
||||
}
|
||||
sourceCitation(resolverResult: ResolverResult<unknown>): string {
|
||||
// pulls .source, .fetchedAt, .raw from the result
|
||||
@@ -563,7 +563,7 @@ Each phase ships independently, passes full E2E, is feature-flagged, and is reve
|
||||
- L4 core: `BrainWriter.transaction`, `Scaffolder`, `SlugRegistry` with conflict detection.
|
||||
- Pre-write validators: citation, link, back-link, triple-HR.
|
||||
- Migrate `src/commands/publish.ts` + `src/commands/backlinks.ts` to route through BrainWriter.
|
||||
- **Now** Wintermute's "Philip Leung" hallucination is structurally impossible — LLM output passes through JSON-Schema validator before reaching Scaffolder.
|
||||
- **Now** Garry's OpenClaw's "Philip Leung" hallucination is structurally impossible — LLM output passes through JSON-Schema validator before reaching Scaffolder.
|
||||
|
||||
### Phase 3 — `gbrain integrity` command (human: ~0.5 wk / CC: ~2 h)
|
||||
- Ship the originally-scoped user-facing feature on top of the new foundation.
|
||||
@@ -582,14 +582,14 @@ Each phase ships independently, passes full E2E, is feature-flagged, and is reve
|
||||
- Migrate `src/commands/autopilot.ts` to a ScheduledResolver set.
|
||||
- Ship `gbrain schedule list|run|pause|tail` CLI for observability.
|
||||
|
||||
### Phase 6 — Port 5–8 Wintermute resolvers (human: ~1.5 wk / CC: ~6 h)
|
||||
### Phase 6 — Port 5–8 OpenClaw resolvers (human: ~1.5 wk / CC: ~6 h)
|
||||
- `perplexity_query`, `text_to_entities`, `mistral_ocr_pdf`, `x_search_all`, `x_user_to_tweets`, `gmail_query_to_threads`, `calendar_date_to_events`.
|
||||
- Each ships as YAML + TS module under `resolvers/builtin/` — **proof of the plugin format.**
|
||||
|
||||
### Phase 7 — Wintermute Claw Adoption Integration (human: ~1 wk / CC: ~4 h)
|
||||
- Write `docs/wintermute/ADOPTION.md` showing Wintermute how to replace its 69 bespoke scripts with calls to `gbrain registry.resolve(...)`.
|
||||
- Ship a `gbrain claw-bridge` subcommand that proxies Wintermute's current script invocations to the resolver registry — zero-edit adoption path.
|
||||
- **This is the test of the north star.** If Wintermute can stand up a 1-line shim and drop `scripts/x-api-client.mjs`, the abstraction succeeded.
|
||||
### Phase 7 — OpenClaw Adoption Integration (human: ~1 wk / CC: ~4 h)
|
||||
- Write `docs/openclaw/ADOPTION.md` showing your OpenClaw how to replace its 69 bespoke scripts with calls to `gbrain registry.resolve(...)`.
|
||||
- Ship a `gbrain claw-bridge` subcommand that proxies Garry's OpenClaw's current script invocations to the resolver registry — zero-edit adoption path.
|
||||
- **This is the test of the north star.** If your OpenClaw can stand up a 1-line shim and drop `scripts/x-api-client.mjs`, the abstraction succeeded.
|
||||
|
||||
Total: human: ~10 weeks / CC: ~42 hours / calendar with single implementer: ~3–4 weeks.
|
||||
|
||||
@@ -649,7 +649,7 @@ src/commands/
|
||||
integrity.ts # ships in Phase 3, replaces Feynman Phase A/B
|
||||
schedule.ts # gbrain schedule list|run|pause|tail (Phase 5)
|
||||
|
||||
docs/wintermute/
|
||||
docs/openclaw/
|
||||
ADOPTION.md # written in Phase 7
|
||||
```
|
||||
|
||||
@@ -685,19 +685,19 @@ Every Resolver implementation tested against the interface spec. Table-driven: r
|
||||
- Simulate API timeout mid-transaction; transaction must roll back completely.
|
||||
- Corrupted state file; scheduler must escalate, not silently skip.
|
||||
|
||||
### Regression tests vs. Wintermute behavior
|
||||
For each Wintermute pattern we port (e.g. X-handle → tweet URL), a regression test proves the new resolver produces the same answer on real-world inputs from the brain audit. This is the "Wintermute would adopt" proof.
|
||||
### Regression tests vs. Garry's OpenClaw behavior
|
||||
For each OpenClaw pattern we port (e.g. X-handle → tweet URL), a regression test proves the new resolver produces the same answer on real-world inputs from the brain audit. This is the "your OpenClaw would adopt" proof.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open Questions (flagged for CEO re-review)
|
||||
|
||||
1. **Scope shape.** Is this the right four-layer decomposition, or are some layers better left to Wintermute (e.g. Scheduling lives above GBrain, not in it)?
|
||||
1. **Scope shape.** Is this the right four-layer decomposition, or are some layers better left to OpenClaw (e.g. Scheduling lives above GBrain, not in it)?
|
||||
2. **Phase 3 user-value break.** Does Phase 3 (user-visible `gbrain integrity`) ship early enough, or do we need an even smaller MVP?
|
||||
3. **LLM-as-resolver.** Should `text_to_entities` be a Resolver, or does that blur the "code vs LLM" line the invariant relies on?
|
||||
4. **Plugin format.** YAML + TS module (§3.5) vs. pure TS module with decorator-style metadata. Latter is more type-safe; former is more discoverable.
|
||||
5. **Cross-resolver transactions.** Do we support "atomic fetch-from-Perplexity + write-to-brain" at the L2 layer? Current design says yes; implementation is tricky (Perplexity call isn't rollbackable).
|
||||
6. **Wintermute bridge scope.** Phase 7 `gbrain claw-bridge` — is that worth a phase of its own, or should adoption be documentation-only?
|
||||
6. **OpenClaw bridge scope.** Phase 7 `gbrain claw-bridge` — is that worth a phase of its own, or should adoption be documentation-only?
|
||||
7. **Completeness rubric coverage.** Do we define rubrics for all 9 PageTypes upfront, or ship people/company/meeting first and extend incrementally?
|
||||
8. **Budget config UX.** Hard daily cap is strict; should we also expose a soft-cap warning mode, and how is the cap set (env var? config file? prompt on first use?)
|
||||
9. **Backwards compat.** `src/commands/publish.ts` and `src/commands/backlinks.ts` have been running cleanly for weeks. Refactoring through BrainWriter carries migration risk. Acceptable?
|
||||
@@ -705,12 +705,12 @@ For each Wintermute pattern we port (e.g. X-handle → tweet URL), a regression
|
||||
|
||||
---
|
||||
|
||||
## 12. Verification (the "Wintermute would adopt" test)
|
||||
## 12. Verification (the "your OpenClaw would adopt" test)
|
||||
|
||||
The design succeeds iff:
|
||||
|
||||
- [ ] A user can add a new resolver by dropping a YAML + TS module in `~/.gbrain/resolvers/` without editing GBrain source.
|
||||
- [ ] Wintermute can delete `scripts/x-api-client.mjs` and replace all callers with 1-line `await registry.resolve('x_handle_to_tweet', ...)`.
|
||||
- [ ] Your OpenClaw can delete `scripts/x-api-client.mjs` and replace all callers with 1-line `await registry.resolve('x_handle_to_tweet', ...)`.
|
||||
- [ ] No brain page can be written with a bare tweet reference, a missing back-link, or an unverified URL (validators catch it pre-commit).
|
||||
- [ ] Running `gbrain integrity --auto --confidence 0.8` over a real brain fixes ≥1,000 of the 1,424 known bare-tweet citations without human review.
|
||||
- [ ] Full E2E test suite passes on both PGLite + Postgres engines.
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
# Procfile — Render / Railway / Heroku.
|
||||
#
|
||||
# Fly.io users: see fly.toml.partial instead.
|
||||
#
|
||||
# Set secrets via the platform's env UI or CLI (e.g. `heroku config:set`,
|
||||
# `render env:set`, `railway variables set`). At minimum:
|
||||
# DATABASE_URL=postgresql://...
|
||||
# GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
|
||||
|
||||
worker: gbrain jobs work --concurrency 2
|
||||
@@ -0,0 +1,22 @@
|
||||
# fly.toml — partial. Merge into your existing fly.toml.
|
||||
#
|
||||
# Set secrets once (never commit them):
|
||||
# fly secrets set DATABASE_URL='postgresql://user:pass@host:6543/db?prepare=false'
|
||||
# fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
|
||||
# fly secrets set ANTHROPIC_API_KEY=... # optional
|
||||
#
|
||||
# Fly.io auto-restarts the process on crash — no watchdog needed.
|
||||
|
||||
[processes]
|
||||
worker = "gbrain jobs work --concurrency 2"
|
||||
|
||||
# Scale the worker process to 1 machine (job queue serializes work; more
|
||||
# machines means higher concurrency but also more Postgres connections).
|
||||
# fly scale count worker=1
|
||||
|
||||
# If you want the worker in its own VM size:
|
||||
# [[vm]]
|
||||
# processes = ["worker"]
|
||||
# memory = "512mb"
|
||||
# cpu_kind = "shared"
|
||||
# cpus = 1
|
||||
@@ -0,0 +1,35 @@
|
||||
# /etc/gbrain.env — secrets + env for the gbrain worker.
|
||||
#
|
||||
# Install:
|
||||
# sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
|
||||
# gbrain.env.example /etc/gbrain.env
|
||||
# sudoedit /etc/gbrain.env # fill in real values
|
||||
#
|
||||
# Referenced from crontab via BASH_ENV=/etc/gbrain.env, or from systemd
|
||||
# via EnvironmentFile=/etc/gbrain.env. Never commit real secrets.
|
||||
|
||||
# --- Required ---------------------------------------------------------------
|
||||
|
||||
# Postgres connection string. For Supabase transaction pooler, include
|
||||
# prepare=false (see CLAUDE.md #284/#286).
|
||||
DATABASE_URL=postgresql://user:pass@host:6543/db?prepare=false
|
||||
|
||||
# --- Required if you submit `shell` jobs ------------------------------------
|
||||
# Only the worker process needs this. Submitters do not.
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1
|
||||
|
||||
# --- Optional ---------------------------------------------------------------
|
||||
|
||||
# LLM provider keys (needed for `subagent` handler, transcription, enrichment).
|
||||
# ANTHROPIC_API_KEY=
|
||||
# OPENAI_API_KEY=
|
||||
|
||||
# Custom handler plugins (see docs/guides/plugin-handlers.md).
|
||||
# GBRAIN_PLUGIN_PATH=/etc/gbrain/plugins
|
||||
|
||||
# Pool size tuning for Supabase transaction pooler (default 10; drop to 2
|
||||
# if you hit MaxClients during upgrade subprocess spawns).
|
||||
# GBRAIN_POOL_SIZE=2
|
||||
|
||||
# Connection-level concurrency cap for Anthropic Messages API.
|
||||
# GBRAIN_ANTHROPIC_MAX_INFLIGHT=4
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/bin/bash
|
||||
# minion-watchdog.sh — restart gbrain jobs work if the process is dead or
|
||||
# has logged a shutdown marker since its last start.
|
||||
#
|
||||
# Fixes the v0.16.1 restart-loop bug: old shutdown lines from previous
|
||||
# restarts stayed in the unrotated log and every tick re-matched them
|
||||
# forever. This version writes a restart epoch to line 2 of the PID file
|
||||
# and only considers log lines newer than that epoch.
|
||||
#
|
||||
# Run every 5 minutes from crontab. See docs/guides/minions-deployment.md.
|
||||
set -u
|
||||
|
||||
PID_FILE="${GBRAIN_WORKER_PID_FILE:-/tmp/gbrain-worker.pid}"
|
||||
LOG_FILE="${GBRAIN_WORKER_LOG_FILE:-/tmp/gbrain-worker.log}"
|
||||
GBRAIN="${GBRAIN_BIN:-/usr/local/bin/gbrain}"
|
||||
CONCURRENCY="${GBRAIN_WORKER_CONCURRENCY:-2}"
|
||||
|
||||
start_worker() {
|
||||
# stderr merged so banner lines ("[minion worker] shell handler enabled",
|
||||
# "worker shutting down") all land in $LOG_FILE.
|
||||
nohup "$GBRAIN" jobs work --concurrency "$CONCURRENCY" \
|
||||
> "$LOG_FILE" 2>&1 &
|
||||
local pid=$!
|
||||
# Line 1: PID. Line 2: restart epoch (seconds since 1970).
|
||||
# Readers that want just PID use `head -n1 "$PID_FILE"`.
|
||||
printf '%s\n%s\n' "$pid" "$(date +%s)" > "$PID_FILE"
|
||||
}
|
||||
|
||||
shutdown_since_restart() {
|
||||
# Only match shutdown lines logged AFTER the most recent restart epoch.
|
||||
# Worker log lines start with ISO-8601 UTC timestamps ("2026-04-21T19:05:12Z ...").
|
||||
local restart_epoch
|
||||
restart_epoch=$(sed -n '2p' "$PID_FILE" 2>/dev/null || echo 0)
|
||||
[ -z "$restart_epoch" ] && restart_epoch=0
|
||||
|
||||
# POSIX-portable regex (no {n} intervals — mawk on Debian/Ubuntu rejects them).
|
||||
awk -v since="$restart_epoch" '
|
||||
match($0, /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9:.+Z-]+/) {
|
||||
ts_str = substr($0, RSTART, RLENGTH)
|
||||
cmd = "date -d \"" ts_str "\" +%s 2>/dev/null"
|
||||
cmd | getline ts
|
||||
close(cmd)
|
||||
if (ts + 0 > since + 0) print
|
||||
}
|
||||
' "$LOG_FILE" 2>/dev/null | grep -q "worker stopped\|worker shutting down"
|
||||
}
|
||||
|
||||
if [ -f "$PID_FILE" ]; then
|
||||
PID=$(head -n1 "$PID_FILE")
|
||||
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
|
||||
# Process alive — check whether the worker logged an internal shutdown
|
||||
# AFTER the last start. If yes, worker is dead-inside; restart.
|
||||
if shutdown_since_restart; then
|
||||
kill "$PID" 2>/dev/null
|
||||
# 10s grace: covers shell handler's 5s child SIGTERM→SIGKILL window
|
||||
# and leaves room for in-flight jobs to flush. Bump to 30 if your
|
||||
# jobs run > 10s.
|
||||
sleep 10
|
||||
kill -9 "$PID" 2>/dev/null
|
||||
start_worker
|
||||
fi
|
||||
else
|
||||
# PID file exists but process is gone (crash / kill -9 / reboot).
|
||||
start_worker
|
||||
fi
|
||||
else
|
||||
start_worker
|
||||
fi
|
||||
@@ -0,0 +1,44 @@
|
||||
[Unit]
|
||||
Description=gbrain minion worker
|
||||
Documentation=https://github.com/garrytan/gbrain/blob/master/docs/guides/minions-deployment.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Runs as an unprivileged user that owns the brain repo and any shell-job cwds.
|
||||
# Create with: sudo useradd --system --home /srv/gbrain --shell /usr/sbin/nologin gbrain
|
||||
User=gbrain
|
||||
Group=gbrain
|
||||
WorkingDirectory=/srv/gbrain
|
||||
|
||||
# Env file is mode 600, owned by User=. Do not put secrets in this unit.
|
||||
EnvironmentFile=/etc/gbrain.env
|
||||
|
||||
ExecStart=/usr/local/bin/gbrain jobs work --concurrency 2
|
||||
|
||||
# Replaces the cron watchdog. systemd restarts on any non-zero exit.
|
||||
Restart=always
|
||||
RestartSec=10s
|
||||
|
||||
# Graceful shutdown: SIGTERM → wait → SIGKILL. 30s matches worker grace
|
||||
# for in-flight jobs and the shell handler's 5s child SIGTERM window.
|
||||
KillSignal=SIGTERM
|
||||
TimeoutStopSec=30s
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=gbrain-worker
|
||||
|
||||
# Default 1024 is tight for Bun + Postgres pool + concurrent subagent LLM calls.
|
||||
LimitNOFILE=65535
|
||||
|
||||
# Hardening (optional — remove if they break your deployment).
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
ReadWritePaths=/srv/gbrain
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,323 @@
|
||||
# Minions Worker Deployment Guide
|
||||
|
||||
Deploy `gbrain jobs work` so it stays running across crashes, reboots, and
|
||||
Postgres connection blips. Written for agents to execute line-by-line.
|
||||
|
||||
## The problem
|
||||
|
||||
The persistent worker can die silently from:
|
||||
|
||||
- Database connection drops (Supabase/Postgres maintenance or network blips).
|
||||
- Lock-renewal failures → the stall detector eventually dead-letters jobs.
|
||||
- Bun process crashes with no automatic restart.
|
||||
- Internal event-loop death (PID alive, worker loop stopped).
|
||||
|
||||
When the worker dies, submitted jobs sit in `waiting` forever. Nothing in
|
||||
gbrain core auto-restarts the worker — that's what this guide wires up.
|
||||
|
||||
## Variables used in this guide
|
||||
|
||||
Substitute these once before copy-pasting any snippet.
|
||||
|
||||
| Variable | Meaning | Typical value |
|
||||
|---|---|---|
|
||||
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
|
||||
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
|
||||
| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) |
|
||||
| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) |
|
||||
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
|
||||
| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) |
|
||||
|
||||
## Preconditions
|
||||
|
||||
Run these before Step 1 of any option. Fail fast if something is wrong.
|
||||
|
||||
```bash
|
||||
# 1. gbrain is on PATH and resolves to an absolute location.
|
||||
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
|
||||
|
||||
# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists).
|
||||
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
|
||||
|
||||
# 3. Schema is up to date. If version=0 or status=="fail", fix it first:
|
||||
# gbrain apply-migrations --yes
|
||||
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
|
||||
|
||||
# 4. You have write access to at least one crontab mechanism.
|
||||
crontab -l >/dev/null 2>&1 && echo "user crontab OK"
|
||||
[ -w /etc/crontab ] && echo "/etc/crontab OK"
|
||||
|
||||
# 5. If you plan to submit `shell` jobs, the WORKER process needs
|
||||
# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated
|
||||
# in registerBuiltinHandlers(); without the flag the worker startup
|
||||
# line reads "shell handler disabled (...)".
|
||||
```
|
||||
|
||||
## Which option?
|
||||
|
||||
- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take
|
||||
> 30 s → **Option 1** (watchdog cron + persistent worker).
|
||||
- Your workload is short deterministic scripts on a fixed schedule (every
|
||||
3 h, daily, weekly) → **Option 2** (inline `--follow`).
|
||||
- You don't have shell access to a long-running box (Fly/Render/Railway,
|
||||
or any systemd host) → **Option 3** (service manager — replaces cron).
|
||||
|
||||
## Option 1: watchdog cron + persistent worker
|
||||
|
||||
A 5-minute cron checks whether the worker process is alive **and** whether
|
||||
it has logged an internal shutdown since its last start. Restarts if either
|
||||
condition fails.
|
||||
|
||||
### 1a. Install the env file (secrets stay out of crontab)
|
||||
|
||||
Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is
|
||||
mode 644 (world-readable); user crontabs under `/var/spool/cron/` are
|
||||
readable by `root`. Use the shipped env-file template:
|
||||
|
||||
```bash
|
||||
sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
|
||||
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
|
||||
sudoedit /etc/gbrain.env
|
||||
```
|
||||
|
||||
Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if
|
||||
applicable). See
|
||||
[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example)
|
||||
for the full list.
|
||||
|
||||
### 1b. Install the watchdog script
|
||||
|
||||
The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh)
|
||||
ships in-repo and writes a two-line PID file (PID on line 1, restart epoch
|
||||
on line 2). The restart-epoch marker is how the watchdog distinguishes
|
||||
stale shutdown lines in the log from current ones — without it, every tick
|
||||
after the first restart would match an old `worker shutting down` line and
|
||||
loop forever.
|
||||
|
||||
Requires GNU coreutils (Linux default). On macOS/BSD install via
|
||||
`brew install coreutils` and alias `date` to `gdate` in the cron env if you
|
||||
want to test the watchdog locally; production Linux boxes work as-is.
|
||||
|
||||
```bash
|
||||
sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
|
||||
docs/guides/minions-deployment-snippets/minion-watchdog.sh \
|
||||
/usr/local/bin/minion-watchdog.sh
|
||||
```
|
||||
|
||||
### 1c. Wire into cron
|
||||
|
||||
Pick the form that matches the crontab you're editing.
|
||||
|
||||
**If you ran `crontab -e`** (user crontab — 5-field, no user column):
|
||||
|
||||
```
|
||||
SHELL=/bin/bash
|
||||
PATH=/usr/local/bin:/usr/bin:/bin
|
||||
BASH_ENV=/etc/gbrain.env
|
||||
*/5 * * * * /usr/local/bin/minion-watchdog.sh
|
||||
```
|
||||
|
||||
**If you edited `/etc/crontab` directly** (system crontab — 6-field, with
|
||||
user column):
|
||||
|
||||
```
|
||||
SHELL=/bin/bash
|
||||
PATH=/usr/local/bin:/usr/bin:/bin
|
||||
BASH_ENV=/etc/gbrain.env
|
||||
*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh
|
||||
```
|
||||
|
||||
In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to
|
||||
source the env file before running the watchdog — that's how the
|
||||
connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without
|
||||
landing in the world-readable crontab itself.
|
||||
|
||||
### 1d. Log rotation
|
||||
|
||||
The watchdog appends to the worker log across restarts. If you expect the
|
||||
file to grow unbounded, rotate it externally with `logrotate`:
|
||||
|
||||
```
|
||||
# /etc/logrotate.d/gbrain-worker
|
||||
/tmp/gbrain-worker.log {
|
||||
daily
|
||||
rotate 7
|
||||
missingok
|
||||
notifempty
|
||||
copytruncate
|
||||
}
|
||||
```
|
||||
|
||||
`copytruncate` is important — the watchdog's restart-epoch check survives
|
||||
it (the epoch is compared against in-log timestamps, not file inode).
|
||||
|
||||
## Option 2: inline `--follow` (no persistent worker)
|
||||
|
||||
Each cron run brings its own temporary worker. `--follow` starts one on
|
||||
the queue and blocks until the just-submitted job reaches a terminal state
|
||||
(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead
|
||||
per job; negligible vs job duration for scheduled work.
|
||||
|
||||
Example: nightly brain enrichment as a shell job.
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
|
||||
--queue nightly-enrich \
|
||||
--params "{\"cmd\":\"$GBRAIN_BIN embed --stale\",\"cwd\":\"$GBRAIN_WORKSPACE\"}" \
|
||||
--follow \
|
||||
--timeout-ms 600000
|
||||
```
|
||||
|
||||
Replace `gbrain embed --stale` with whichever gbrain subcommand you're
|
||||
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
|
||||
`lint`, `autopilot`). If you're shelling out to a non-gbrain binary,
|
||||
keep its absolute path in the `cmd`.
|
||||
|
||||
**Shared-queue gotcha.** If other jobs are already waiting on the same
|
||||
queue with higher priority or earlier `created_at`, the temporary worker
|
||||
processes those first before reaching yours. `--follow` still exits only
|
||||
when YOUR job finishes. For strict single-job semantics on shared queues,
|
||||
use a dedicated queue name like `nightly-enrich` above.
|
||||
|
||||
## Option 3: service manager (systemd / Fly / Render / Railway)
|
||||
|
||||
Replaces the watchdog entirely. No cron, no PID file, no restart-loop.
|
||||
The service manager owns liveness.
|
||||
|
||||
### systemd (Linux hosts with shell access)
|
||||
|
||||
```bash
|
||||
# Create the worker user if it doesn't exist.
|
||||
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
|
||||
2>/dev/null || true
|
||||
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
|
||||
|
||||
# Install the unit file, substituting /srv/gbrain → your workspace path.
|
||||
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
|
||||
/etc/systemd/system/gbrain-worker.service
|
||||
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
|
||||
/etc/systemd/system/gbrain-worker.service
|
||||
|
||||
# See 1a above for /etc/gbrain.env install.
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now gbrain-worker
|
||||
sudo systemctl status gbrain-worker
|
||||
journalctl -u gbrain-worker -n 50
|
||||
```
|
||||
|
||||
`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit
|
||||
runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`,
|
||||
and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped
|
||||
unit covers Bun + Postgres pool + concurrent LLM subagent calls without
|
||||
hitting the default 1024 cap.
|
||||
|
||||
### Fly.io
|
||||
|
||||
Merge the `[processes]` block from
|
||||
[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into
|
||||
your existing `fly.toml`. Set secrets with `fly secrets set` —
|
||||
Fly auto-restarts the process on crash.
|
||||
|
||||
### Render / Railway / Heroku
|
||||
|
||||
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root.
|
||||
Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the
|
||||
platform's env UI or CLI.
|
||||
|
||||
## Upgrading an existing deployment
|
||||
|
||||
If you deployed on v0.13.x or earlier, walk this checklist:
|
||||
|
||||
1. **Stop the worker before upgrading.**
|
||||
`kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to
|
||||
exit. Skipping this risks an in-flight job landing partial schema.
|
||||
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
|
||||
`gbrain doctor` reports any migration as `partial` or `pending`.
|
||||
3. **If you run shell jobs:** from v0.14 onward, the worker requires
|
||||
`GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to
|
||||
`/etc/gbrain.env`. Submitters don't need the flag; only the worker does.
|
||||
4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration
|
||||
v15 raised the schema default to 5 and backfilled existing non-terminal
|
||||
rows. A watchdog tuned around 1-strike dead-lettering will now
|
||||
over-restart because it takes 5 misses to dead-letter. Switch to the
|
||||
shipped watchdog (which keys on log markers, not job state).
|
||||
5. **If your v0.16.1 watchdog is still running:** it has a restart-loop
|
||||
bug (old shutdown lines in the unrotated log re-match every 5 min
|
||||
forever). Install the current `minion-watchdog.sh` from this guide's
|
||||
snippets — it writes a restart epoch into the PID file and only
|
||||
considers log lines newer than that epoch.
|
||||
6. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
|
||||
migrations. `gbrain jobs stats` should show no unexplained growth in
|
||||
`dead` between pre- and post-upgrade.
|
||||
|
||||
## Known issues
|
||||
|
||||
### Supabase connection drops
|
||||
|
||||
The worker uses a single Postgres connection. If Supabase drops it
|
||||
(maintenance, connection limits, network blip), lock renewal fails
|
||||
silently. The stall detector then dead-letters the job after
|
||||
`max_stalled` misses.
|
||||
|
||||
**Current defaults that make this worse:**
|
||||
|
||||
- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips.
|
||||
- `max_stalled: 5` (schema column default on master — see `src/schema.sql`
|
||||
and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
|
||||
- `stalledInterval: 30000` (30 s) — checks too aggressively.
|
||||
|
||||
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
|
||||
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
|
||||
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
|
||||
(since v0.13.1). These write onto the job row at submit time — which is
|
||||
what `handleStalled()` reads — so per-job tuning is the real knob today.
|
||||
Worker-level `--lock-duration` / `--stall-interval` are on the roadmap;
|
||||
until they land, rely on per-job `--max-stalled` plus the watchdog (or
|
||||
systemd) for worker health.
|
||||
|
||||
### DO NOT pass `maxStalledCount` to `MinionWorker`
|
||||
|
||||
It's a no-op. The stall detector reads the row's `max_stalled` column
|
||||
(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`.
|
||||
Use `gbrain jobs submit --max-stalled N` per-job instead.
|
||||
|
||||
### Zombie shell children
|
||||
|
||||
When the Bun worker crashes hard, child processes from shell jobs can
|
||||
become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the
|
||||
shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running
|
||||
shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker
|
||||
has time to flush in-flight jobs before the kill.
|
||||
|
||||
## Smoke test
|
||||
|
||||
```bash
|
||||
# Worker alive?
|
||||
kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD
|
||||
|
||||
# Aggregate queue health.
|
||||
gbrain jobs stats
|
||||
|
||||
# Jobs currently stalled (still `active` with expired lock_until, pre-requeue).
|
||||
gbrain jobs list --status active --limit 10
|
||||
|
||||
# Dead-lettered jobs.
|
||||
gbrain jobs list --status dead --limit 10
|
||||
|
||||
# Shell handler registered? (stderr banner merged into log via 2>&1.)
|
||||
grep "shell handler enabled" /tmp/gbrain-worker.log
|
||||
```
|
||||
|
||||
## Uninstall
|
||||
|
||||
- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line.
|
||||
`kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`.
|
||||
Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`.
|
||||
- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to
|
||||
clean up — temporary workers exit with their jobs.
|
||||
- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`,
|
||||
then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`,
|
||||
then `sudo systemctl daemon-reload`.
|
||||
- **Option 3 (Fly/Render/Railway):** delete the `worker` process from
|
||||
`fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets`
|
||||
persist until `fly secrets unset`.
|
||||
@@ -73,7 +73,7 @@ F. Install gbrain autopilot --install (env-aware)
|
||||
G. Record append completed.jsonl status:"complete"
|
||||
```
|
||||
|
||||
If Phase E emits TODOs for host-specific handlers (e.g. Wintermute's
|
||||
If Phase E emits TODOs for host-specific handlers (e.g. your OpenClaw's
|
||||
~29 non-gbrain crons), the migration finishes with `status: "partial"`.
|
||||
Your host agent walks the TODOs using `skills/migrations/v0.11.0.md` +
|
||||
`docs/guides/plugin-handlers.md`, ships handler registrations in the
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
# Plugin authors guide (v0.15)
|
||||
|
||||
`gbrain` discovers subagent definitions from outside this repo via
|
||||
`GBRAIN_PLUGIN_PATH`. If you maintain a downstream agent (your OpenClaw
|
||||
deployment, a workflow host, a private tool) and want to ship custom
|
||||
subagents alongside it, drop a plugin directory on that env path.
|
||||
|
||||
This guide is for plugin authors. The CLI user doesn't need to read it.
|
||||
|
||||
## Minimum viable plugin
|
||||
|
||||
```
|
||||
/path/to/my-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
└── my-summarizer.md
|
||||
```
|
||||
|
||||
`gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-plugin",
|
||||
"version": "1.0.0",
|
||||
"plugin_version": "gbrain-plugin-v1"
|
||||
}
|
||||
```
|
||||
|
||||
`subagents/my-summarizer.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: my-summarizer
|
||||
model: claude-sonnet-4-6
|
||||
allowed_tools:
|
||||
- brain_search
|
||||
- brain_get_page
|
||||
---
|
||||
|
||||
You are a brain page summarizer. Given a slug, fetch the page and produce
|
||||
a 3-sentence summary.
|
||||
```
|
||||
|
||||
## Turning it on
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="/path/to/my-plugin"
|
||||
gbrain jobs work # worker startup prints the plugin load line
|
||||
gbrain agent run "summarize meetings/2026-04-20" --subagent-def my-summarizer
|
||||
```
|
||||
|
||||
Multiple plugins: colon-separated, just like `$PATH`.
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="/path/to/plugin-a:/path/to/plugin-b"
|
||||
```
|
||||
|
||||
## Rules (strict by design)
|
||||
|
||||
**Path policy.** Absolute paths only. Relative paths, `~`-prefixed paths,
|
||||
and URL-style paths (`https://`, `file://`) are rejected with a warning.
|
||||
You control where your plugin lives on disk; `gbrain` doesn't guess.
|
||||
|
||||
**Collision policy.** If two plugins ship a subagent with the same `name`,
|
||||
the one listed FIRST in `GBRAIN_PLUGIN_PATH` wins. The other is dropped
|
||||
with a warning naming both sources.
|
||||
|
||||
**Trust policy.** Plugins ship subagent definitions ONLY in v0.15:
|
||||
|
||||
- You **cannot** declare new tools.
|
||||
- You **cannot** extend the brain tool allow-list.
|
||||
- You **cannot** override any `agentSafe` or similar flag.
|
||||
- Your `allowed_tools:` frontmatter field MUST subset the derived brain
|
||||
tool registry. Names not in the registry are rejected at plugin load
|
||||
time (worker startup), NOT at subagent dispatch time — so a typo in
|
||||
your plugin gives you a loud startup error, not a silent "tool never
|
||||
fires" at 3am.
|
||||
|
||||
v0.16+ may open up plugin-declared tools with a separate contract. Don't
|
||||
expect it.
|
||||
|
||||
## `gbrain.plugin.json`
|
||||
|
||||
| field | type | required | notes |
|
||||
|------------------|--------|----------|--------------------------------------------------------------------|
|
||||
| `name` | string | yes | Human-readable plugin id. Shows up in warnings and collision logs. |
|
||||
| `version` | string | yes | Your plugin's semver. Informational. |
|
||||
| `plugin_version` | string | yes | Contract lock. Must equal `"gbrain-plugin-v1"` for v0.15. |
|
||||
| `subagents` | string | no | Subdir name (default `subagents`). Escape-attempts are rejected. |
|
||||
| `description` | string | no | Shown in future `gbrain plugin list`. |
|
||||
|
||||
## Subagent definition files
|
||||
|
||||
Plain markdown with YAML frontmatter. The body is the system prompt. The
|
||||
frontmatter controls runtime behavior.
|
||||
|
||||
Recognized frontmatter fields:
|
||||
|
||||
| field | type | required | notes |
|
||||
|-----------------|----------|----------|-----------------------------------------------------------------------------------------|
|
||||
| `name` | string | no | Subagent identifier used as `--subagent-def`. Defaults to the file basename. |
|
||||
| `model` | string | no | Anthropic model id. Defaults to the handler default (sonnet). |
|
||||
| `max_turns` | number | no | Cap on assistant turns. Defaults to 20. |
|
||||
| `allowed_tools` | string[] | no | Whitelist of tool names. Must subset the derived brain registry. Rejected on mismatch. |
|
||||
|
||||
Unknown frontmatter fields are preserved but ignored by the handler. v0.16
|
||||
may consume more of them.
|
||||
|
||||
## Caveats that will bite you
|
||||
|
||||
1. **Plugin definitions can't change during a run.** The loader reads the
|
||||
disk once at worker startup. Editing a subagent def doesn't re-take
|
||||
effect until you restart the worker. This is deliberate — live
|
||||
reloads would break crash-resumable replay.
|
||||
|
||||
2. **`~/.gbrain/audit/subagent-jobs-*.jsonl` is local only.** If your
|
||||
worker runs on a different host than the `gbrain agent logs` caller,
|
||||
the CLI won't see heartbeats from that worker. v0.16 will unify this;
|
||||
for now assume worker + CLI share a filesystem.
|
||||
|
||||
3. **Tool calls always run with `ctx.remote = true`.** Even on local CLI
|
||||
invocation. Tools that gate on `remote=true` (file_upload's strict
|
||||
confinement, put_page's namespace check) will apply. Good default; a
|
||||
subagent definition that wants local-filesystem reach beyond the brain
|
||||
can't have it.
|
||||
|
||||
4. **`put_page` writes are namespace-scoped.** A subagent with id 42 can
|
||||
only write under `wiki/agents/42/...`. This is enforced both in the
|
||||
tool schema (the slug pattern shown to the model) AND server-side in
|
||||
the `put_page` operation (fail-closed if `viaSubagent=true`). Don't
|
||||
try to route around it; you'll get `permission_denied`.
|
||||
|
||||
## Example: a downstream-OpenClaw plugin
|
||||
|
||||
```
|
||||
~/your-openclaw/
|
||||
└── gbrain-plugin/
|
||||
├── gbrain.plugin.json
|
||||
└── subagents/
|
||||
├── meeting-ingestion.md
|
||||
├── signal-detector.md
|
||||
└── daily-task-prep.md
|
||||
```
|
||||
|
||||
`~/your-openclaw/gbrain-plugin/gbrain.plugin.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "your-openclaw",
|
||||
"version": "2026.4.20",
|
||||
"plugin_version": "gbrain-plugin-v1",
|
||||
"description": "Your OpenClaw's personal-brain subagents"
|
||||
}
|
||||
```
|
||||
|
||||
Environment:
|
||||
|
||||
```bash
|
||||
export GBRAIN_PLUGIN_PATH="$HOME/your-openclaw/gbrain-plugin"
|
||||
```
|
||||
|
||||
Then your OpenClaw calls `gbrain agent run --subagent-def meeting-ingestion
|
||||
--fanout-by transcript ...` and its definitions load automatically.
|
||||
@@ -4,8 +4,8 @@ GBrain's Minion worker ships with seven built-in handlers: `sync`,
|
||||
`embed`, `lint`, `import`, `extract`, `backlinks`, `autopilot-cycle`.
|
||||
These cover every background operation the gbrain CLI itself performs.
|
||||
|
||||
Host platforms (Wintermute, other OpenClaw deployments, future hosts)
|
||||
register their own handlers via a plugin bootstrap that imports
|
||||
Host platforms (OpenClaw deployments, future hosts) register their own
|
||||
handlers via a plugin bootstrap that imports
|
||||
`gbrain/minions`. No `handlers.json`-style data file — handlers are
|
||||
code, loaded by the worker, with the same trust model as any other
|
||||
code in the host's repo.
|
||||
@@ -58,7 +58,7 @@ async function main() {
|
||||
main().catch(err => { console.error(err); process.exit(1); });
|
||||
```
|
||||
|
||||
Ship this as a separate binary in the host repo (e.g. `wintermute-worker`)
|
||||
Ship this as a separate binary in the host repo (e.g. `your-openclaw-worker`)
|
||||
or as a side-effect module that the stock `gbrain jobs work` command
|
||||
auto-loads on startup (configurable via a host-provided entry point).
|
||||
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
# Progress events
|
||||
|
||||
Canonical reference for the JSONL progress stream that `gbrain` writes to
|
||||
`stderr` when a bulk command runs with `--progress-json`. Stable from
|
||||
v0.15.2. Additive changes only; no renames or removals without a major
|
||||
version bump.
|
||||
|
||||
Most humans won't read this page. Agents parsing progress will.
|
||||
|
||||
## When do I get these events?
|
||||
|
||||
Any of these commands stream events when `--progress-json` is set:
|
||||
|
||||
- `gbrain doctor` (DB checks, JSONB integrity, markdown body completeness,
|
||||
integrity sample)
|
||||
- `gbrain orphans`
|
||||
- `gbrain embed`
|
||||
- `gbrain files sync`
|
||||
- `gbrain export`
|
||||
- `gbrain extract [links|timeline|all]` (fs or db source)
|
||||
- `gbrain import`
|
||||
- `gbrain sync`
|
||||
- `gbrain migrate --to …`
|
||||
- `gbrain repair-jsonb`
|
||||
- `gbrain check-backlinks`
|
||||
- `gbrain lint`
|
||||
- `gbrain integrity auto`
|
||||
- `gbrain eval`
|
||||
- `gbrain apply-migrations` (the orchestrator + every child command)
|
||||
|
||||
Non-bulk commands (`stats`, `graph-query`, `get`, `put`, etc.) don't emit
|
||||
events — they return in under a second.
|
||||
|
||||
## Channel
|
||||
|
||||
- Progress events: **`stderr`**, one JSON object per line, `\n`-terminated.
|
||||
- Data results (`--json` payloads from each command): **`stdout`**.
|
||||
- Final human summaries: **`stdout`**.
|
||||
|
||||
Agents can safely capture stdout for their result parsing and read stderr
|
||||
separately for progress.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Behavior |
|
||||
|---|---|
|
||||
| *(none)* | Auto. TTY: `\r`-rewriting single line. Non-TTY: plain line-per-event on stderr. |
|
||||
| `--progress-json` | Force JSON-lines mode on stderr (this doc). |
|
||||
| `--quiet` | Suppress progress entirely. Warnings and final output still print. |
|
||||
| `--progress-interval=<ms>` | Override the minimum interval between tick emits (default 1000). |
|
||||
|
||||
Global flags: parsed by `src/core/cli-options.ts` before command dispatch,
|
||||
so `gbrain --progress-json doctor` works the same as
|
||||
`gbrain doctor --progress-json` (the latter also works — per-command
|
||||
parsers see the flag via the shared `CliOptions` singleton).
|
||||
|
||||
## Event types
|
||||
|
||||
Every event is a single-line JSON object with these common fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `event` | string | One of: `start`, `tick`, `heartbeat`, `finish`, `abort`. |
|
||||
| `phase` | string | Machine-stable snake_case, dot-separated. See "Phase names" below. |
|
||||
| `ts` | ISO 8601 UTC string | Event emission time. |
|
||||
| `elapsed_ms` | number | Ms since the phase started. Present on `tick`/`heartbeat`/`finish`/`abort`. |
|
||||
|
||||
### `start`
|
||||
|
||||
Emitted when a phase begins.
|
||||
|
||||
```json
|
||||
{"event":"start","phase":"doctor.db_checks","ts":"2026-04-20T12:34:56.789Z"}
|
||||
{"event":"start","phase":"import.files","total":52000,"ts":"2026-04-20T12:34:56.789Z"}
|
||||
```
|
||||
|
||||
Optional fields:
|
||||
|
||||
- `total` — the total item count if known at start.
|
||||
|
||||
### `tick`
|
||||
|
||||
Emitted periodically during iteration. Time- and item-gated: the reporter
|
||||
won't emit more often than `minIntervalMs` (default 1000) and
|
||||
`minItems` (default `max(10, ceil(total/100))`).
|
||||
|
||||
```json
|
||||
{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"eta_ms":10300,"ts":"..."}
|
||||
```
|
||||
|
||||
Fields:
|
||||
|
||||
- `done` — items completed in this phase.
|
||||
- `total` — total items, if known. Omitted when the scan doesn't have a
|
||||
total up front (e.g. a streaming iterator).
|
||||
- `pct` — `done/total * 100`, one decimal. Omitted when `total` is unknown.
|
||||
- `eta_ms` — projected ms until `done === total`, from the observed rate.
|
||||
Omitted when `total` is unknown.
|
||||
- `note` — optional string with the current item (e.g. a slug or filename).
|
||||
|
||||
### `heartbeat`
|
||||
|
||||
Emitted for long-running single operations that don't iterate
|
||||
(e.g. `SELECT` against a 50K-row table). No `done`, no `total` — just a
|
||||
signal that work is still happening.
|
||||
|
||||
```json
|
||||
{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation…","elapsed_ms":1000,"ts":"..."}
|
||||
```
|
||||
|
||||
### `finish`
|
||||
|
||||
Emitted when a phase completes normally.
|
||||
|
||||
```json
|
||||
{"event":"finish","phase":"import.files","done":52000,"total":52000,"elapsed_ms":187000,"ts":"..."}
|
||||
```
|
||||
|
||||
### `abort`
|
||||
|
||||
Emitted by a single process-level SIGINT/SIGTERM handler that tracks every
|
||||
live phase. After `abort`, no further events emit for that phase.
|
||||
|
||||
```json
|
||||
{"event":"abort","phase":"doctor.markdown_body_completeness","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}
|
||||
```
|
||||
|
||||
## Phase names
|
||||
|
||||
Phases use `snake_case.dot.path` naming. A fresh reporter starts at the
|
||||
root; `child()` composition appends to the parent's current phase, so a
|
||||
sync that calls import emits `sync.import.<file>`, not `import.<file>`.
|
||||
|
||||
Stable phase names shipped in v0.15.2:
|
||||
|
||||
- `doctor.db_checks` (umbrella for all DB-side doctor checks)
|
||||
- `orphans.scan`
|
||||
- `embed.pages`
|
||||
- `extract.links_fs`, `extract.timeline_fs`, `extract.links_db`, `extract.timeline_db`
|
||||
- `import.files`
|
||||
- `sync.deletes`, `sync.renames`, `sync.imports`
|
||||
- `migrate.copy_pages`, `migrate.copy_links`
|
||||
- `repair_jsonb.run`, `repair_jsonb.<table>.<column>`
|
||||
- `backlinks.scan`
|
||||
- `lint.pages`
|
||||
- `integrity.auto`
|
||||
- `eval.single`, `eval.ab`
|
||||
- `export.pages`
|
||||
- `files.sync`
|
||||
|
||||
Sub-phases exposed via `child()`:
|
||||
|
||||
- `sync.import.files` — nested inside a sync
|
||||
- `apply_migrations.v0_12_2.jsonb_repair` — nested inside the orchestrator
|
||||
|
||||
## Subprocess inheritance
|
||||
|
||||
When a parent CLI spawns `gbrain …` child processes (mostly in
|
||||
`src/commands/migrations/*`), global flags (`--quiet`, `--progress-json`,
|
||||
`--progress-interval`) are propagated to the child's argv via the
|
||||
`childGlobalFlags()` helper in `src/core/cli-options.ts`. Child stderr
|
||||
passes straight through `stdio: 'inherit'` so the event stream is one
|
||||
merged JSONL feed on the parent's stderr.
|
||||
|
||||
One exception: the orchestrator phase in `migrations/v0_12_2.ts` that
|
||||
captures child stdout (`repair-jsonb --dry-run --json` for verification)
|
||||
does not pass `--progress-json` to avoid any risk of stdout pollution
|
||||
breaking the orchestrator's `JSON.parse`. Its stdio is explicit:
|
||||
`['ignore', 'pipe', 'inherit']` so stderr still flows through.
|
||||
|
||||
## Minion jobs
|
||||
|
||||
`gbrain jobs work` (the Minion worker daemon) keeps progress in the DB,
|
||||
not on stderr. Each Minion handler that runs a bulk core (embed, sync,
|
||||
extract, import, backlinks) calls `job.updateProgress({done, total,
|
||||
…})` per iteration. Agents read per-job progress via the
|
||||
`get_job_progress` MCP operation or `gbrain jobs get <id>`.
|
||||
|
||||
The `jobs work` daemon itself emits coarse one-line-per-job stderr output
|
||||
for liveness only. Per-page detail lives in the DB.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Added**: only. A new event type, a new field, a new phase name — all
|
||||
safe. Agents must ignore unknown fields and unknown event types.
|
||||
- **Removed/renamed**: never without a major version bump.
|
||||
- **Schema changes**: announced in `CHANGELOG.md` and in
|
||||
`skills/migrations/v<next>.md`.
|
||||
|
||||
If your agent depends on this schema and something surprises you, open
|
||||
an issue with the event you received and what you expected.
|
||||
+4938
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,53 @@
|
||||
# GBrain
|
||||
|
||||
> GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.
|
||||
|
||||
Repo: https://github.com/garrytan/gbrain
|
||||
|
||||
## Core entry points
|
||||
|
||||
- [AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md): Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.
|
||||
- [CLAUDE.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CLAUDE.md): Architecture reference. Key files, trust boundaries, engine factory, test layout.
|
||||
- [INSTALL_FOR_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md): 9-step agent installation.
|
||||
- [skills/RESOLVER.md](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/RESOLVER.md): Skill dispatcher. Read first for any task.
|
||||
- [README.md](https://raw.githubusercontent.com/garrytan/gbrain/master/README.md): Project overview, benchmarks, 30-minute setup.
|
||||
|
||||
## Configuration
|
||||
|
||||
- [docs/ENGINES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ENGINES.md): PGLite vs Postgres trade-off and when to migrate.
|
||||
- [docs/GBRAIN_RECOMMENDED_SCHEMA.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md): MECE directory structure (people/, companies/, concepts/).
|
||||
- [docs/guides/live-sync.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md): Incremental markdown sync setup.
|
||||
- [docs/guides/cron-schedule.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/cron-schedule.md): Recurring job scheduling.
|
||||
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
|
||||
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
|
||||
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
|
||||
|
||||
## Debugging
|
||||
|
||||
- [docs/GBRAIN_VERIFY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_VERIFY.md): 7-check post-setup verification. Start here when something feels off.
|
||||
- [docs/guides/minions-fix.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-fix.md): Troubleshooting the Minions job queue.
|
||||
- [docs/integrations/reliability-repair.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/integrations/reliability-repair.md): Data integrity recovery.
|
||||
|
||||
## Migrations
|
||||
|
||||
- [docs/UPGRADING_DOWNSTREAM_AGENTS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/UPGRADING_DOWNSTREAM_AGENTS.md): Patches for downstream agent skill forks. One section per release.
|
||||
- [skills/migrations/](https://raw.githubusercontent.com/garrytan/gbrain/master/skills/migrations/): Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.
|
||||
- [CHANGELOG.md](https://raw.githubusercontent.com/garrytan/gbrain/master/CHANGELOG.md): Release-summary voice + itemized changes + self-repair block per version.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- [docs/ethos/THIN_HARNESS_FAT_SKILLS.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/THIN_HARNESS_FAT_SKILLS.md): Why skills live in markdown.
|
||||
- [docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md): Homebrew for Personal AI.
|
||||
|
||||
## Optional
|
||||
|
||||
- [docs/benchmarks/](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/benchmarks/): Retrieval quality benchmarks.
|
||||
- [docs/designs/](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/designs/): Forward-looking designs.
|
||||
- [docs/architecture/infra-layer.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/architecture/infra-layer.md): Shared infra patterns.
|
||||
|
||||
## Operational tips
|
||||
|
||||
- `gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.
|
||||
- `gbrain orphans [--json]` - pages with zero inbound wikilinks.
|
||||
- `gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.
|
||||
- `gbrain upgrade` runs post-upgrade + apply-migrations.
|
||||
+16
-7
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.14.2",
|
||||
"version": "0.16.4",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -20,10 +20,13 @@
|
||||
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"test": "scripts/check-jsonb-pattern.sh && bun test",
|
||||
"test:e2e": "bun test test/e2e/",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && bun run typecheck && bun test",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"postinstall": "gbrain --version >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive 2>/dev/null || true",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
},
|
||||
@@ -35,16 +38,22 @@
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.30.0",
|
||||
"@aws-sdk/client-s3": "^3.1028.0",
|
||||
"@electric-sql/pglite": "^0.4.4",
|
||||
"@electric-sql/pglite": "0.4.3",
|
||||
"@modelcontextprotocol/sdk": "^1.0.0",
|
||||
"gray-matter": "^4.0.3",
|
||||
"marked": "^18.0.0",
|
||||
"openai": "^4.0.0",
|
||||
"pgvector": "^0.2.0",
|
||||
"postgres": "^3.4.0"
|
||||
"postgres": "^3.4.0",
|
||||
"tree-sitter-wasms": "0.1.13",
|
||||
"web-tree-sitter": "0.22.6"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest"
|
||||
"@types/bun": "latest",
|
||||
"typescript": "^5.6.0"
|
||||
},
|
||||
"trustedDependencies": [
|
||||
"@electric-sql/pglite"
|
||||
],
|
||||
"license": "MIT"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* build-llms — generate llms.txt + llms-full.txt from scripts/llms-config.ts.
|
||||
*
|
||||
* Run: `bun run build:llms` (or `bun run scripts/build-llms.ts`).
|
||||
*
|
||||
* Outputs:
|
||||
* - llms.txt — llmstxt.org-spec index (H1 / blockquote / H2 sections).
|
||||
* - llms-full.txt — concatenated full content of non-optional entries.
|
||||
*
|
||||
* Deterministic: no timestamps, sorted within categories by config order.
|
||||
* Warns (does not fail) if llms-full.txt exceeds FULL_SIZE_BUDGET. CI catches
|
||||
* drift via test/build-llms.test.ts.
|
||||
*
|
||||
* Fork override: set LLMS_REPO_BASE to regenerate with a different URL base.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
import { join, dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import {
|
||||
FULL_SIZE_BUDGET,
|
||||
INLINE_TIPS,
|
||||
PROJECT,
|
||||
SECTIONS,
|
||||
type DocEntry,
|
||||
type DocSection,
|
||||
} from "./llms-config";
|
||||
|
||||
const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
function urlFor(entry: DocEntry): string {
|
||||
return `${PROJECT.rawBaseUrl}/${entry.path}`;
|
||||
}
|
||||
|
||||
function isDirectoryPath(path: string): boolean {
|
||||
return path.endsWith("/");
|
||||
}
|
||||
|
||||
function renderLlmsTxt(): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${PROJECT.name}`);
|
||||
lines.push("");
|
||||
lines.push(`> ${PROJECT.summary}`);
|
||||
lines.push("");
|
||||
lines.push(`Repo: ${PROJECT.repoUrl}`);
|
||||
lines.push("");
|
||||
|
||||
for (const section of SECTIONS) {
|
||||
lines.push(`## ${section.heading}`);
|
||||
lines.push("");
|
||||
for (const entry of section.entries) {
|
||||
lines.push(
|
||||
`- [${entry.title}](${urlFor(entry)}): ${entry.description}`,
|
||||
);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
lines.push("## Operational tips");
|
||||
lines.push("");
|
||||
for (const tip of INLINE_TIPS) {
|
||||
lines.push(`- ${tip}`);
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function renderLlmsFullTxt(): { content: string; sizes: Array<{ path: string; bytes: number }> } {
|
||||
const lines: string[] = [];
|
||||
const sizes: Array<{ path: string; bytes: number }> = [];
|
||||
|
||||
lines.push(`# ${PROJECT.name} — Full Context`);
|
||||
lines.push("");
|
||||
lines.push(`> ${PROJECT.summary}`);
|
||||
lines.push("");
|
||||
lines.push(
|
||||
`This file concatenates core GBrain documentation for single-fetch ingestion.`,
|
||||
);
|
||||
lines.push(
|
||||
`For the link-only index, see \`llms.txt\`. Source of truth: ${PROJECT.repoUrl}.`,
|
||||
);
|
||||
lines.push("");
|
||||
|
||||
for (const section of SECTIONS) {
|
||||
if (section.optional) continue;
|
||||
lines.push(`# ${section.heading}`);
|
||||
lines.push("");
|
||||
for (const entry of section.entries) {
|
||||
if (entry.includeInFull === false) continue;
|
||||
if (isDirectoryPath(entry.path)) continue;
|
||||
|
||||
const absPath = join(repoRoot, entry.path);
|
||||
if (!existsSync(absPath)) {
|
||||
// build-llms won't silently skip — surface the problem. Test case 1
|
||||
// catches this too, but fail fast for manual runs.
|
||||
throw new Error(
|
||||
`llms-config references missing file: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
|
||||
const body = readFileSync(absPath, "utf8");
|
||||
const bytes = Buffer.byteLength(body, "utf8");
|
||||
sizes.push({ path: entry.path, bytes });
|
||||
|
||||
lines.push(`## ${entry.path}`);
|
||||
lines.push("");
|
||||
lines.push(`Source: ${urlFor(entry)}`);
|
||||
lines.push("");
|
||||
lines.push(body.trimEnd());
|
||||
lines.push("");
|
||||
lines.push("---");
|
||||
lines.push("");
|
||||
}
|
||||
}
|
||||
|
||||
return { content: lines.join("\n"), sizes };
|
||||
}
|
||||
|
||||
function validateConfig(): void {
|
||||
for (const section of SECTIONS) {
|
||||
for (const entry of section.entries) {
|
||||
const absPath = join(repoRoot, entry.path);
|
||||
if (!existsSync(absPath)) {
|
||||
throw new Error(
|
||||
`llms-config references missing path: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
const st = statSync(absPath);
|
||||
if (isDirectoryPath(entry.path) && !st.isDirectory()) {
|
||||
throw new Error(
|
||||
`llms-config path ends with '/' but is a file: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
if (!isDirectoryPath(entry.path) && !st.isFile()) {
|
||||
throw new Error(
|
||||
`llms-config path is a directory but missing trailing '/': ${entry.path}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function buildLlmsFiles(): {
|
||||
llmsTxt: string;
|
||||
llmsFullTxt: string;
|
||||
sizes: Array<{ path: string; bytes: number }>;
|
||||
} {
|
||||
validateConfig();
|
||||
const llmsTxt = renderLlmsTxt();
|
||||
const { content: llmsFullTxt, sizes } = renderLlmsFullTxt();
|
||||
return { llmsTxt, llmsFullTxt, sizes };
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const { llmsTxt, llmsFullTxt, sizes } = buildLlmsFiles();
|
||||
|
||||
const llmsPath = join(repoRoot, "llms.txt");
|
||||
const llmsFullPath = join(repoRoot, "llms-full.txt");
|
||||
|
||||
writeFileSync(llmsPath, llmsTxt);
|
||||
writeFileSync(llmsFullPath, llmsFullTxt);
|
||||
|
||||
const fullBytes = Buffer.byteLength(llmsFullTxt, "utf8");
|
||||
console.log(`wrote ${llmsPath} (${Buffer.byteLength(llmsTxt, "utf8")} bytes)`);
|
||||
console.log(`wrote ${llmsFullPath} (${fullBytes} bytes)`);
|
||||
|
||||
if (fullBytes > FULL_SIZE_BUDGET) {
|
||||
console.warn("");
|
||||
console.warn(
|
||||
`WARN: llms-full.txt (${fullBytes} bytes) exceeds FULL_SIZE_BUDGET (${FULL_SIZE_BUDGET} bytes).`,
|
||||
);
|
||||
console.warn(
|
||||
"Add `includeInFull: false` to the biggest entries in scripts/llms-config.ts:",
|
||||
);
|
||||
const sorted = [...sizes].sort((a, b) => b.bytes - a.bytes);
|
||||
for (const entry of sorted.slice(0, 5)) {
|
||||
console.warn(` ${entry.bytes} bytes ${entry.path}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isMainModule = fileURLToPath(import.meta.url) === process.argv[1];
|
||||
if (isMainModule) {
|
||||
try {
|
||||
main();
|
||||
} catch (err) {
|
||||
console.error(err instanceof Error ? err.message : err);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -30,3 +30,17 @@ if grep -rEn "$PATTERN" src/ 2>/dev/null; then
|
||||
fi
|
||||
|
||||
echo "OK: no JSON.stringify(x)::jsonb interpolation pattern in src/"
|
||||
|
||||
# v0.13.1 #219: guard against max_stalled DEFAULT 1 regressing in any schema
|
||||
# source file. DEFAULT 1 dead-lettered any SIGKILL'd job on first stall, making
|
||||
# the "10/10 rescued" claim false for out-of-the-box users. Default is 5 now.
|
||||
MAX_STALLED_PATTERN='max_stalled\s+INTEGER\s+NOT\s+NULL\s+DEFAULT\s+1\b'
|
||||
|
||||
if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/pglite-schema.ts src/core/schema-embedded.ts 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: max_stalled DEFAULT 1 reintroduced in schema."
|
||||
echo " Must be DEFAULT 5 to preserve SIGKILL-rescue guarantee. See #219."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: max_stalled defaults are 5 in all schema sources"
|
||||
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any new code emits \r-progress to stdout.
|
||||
#
|
||||
# Since v0.14.2, bulk-action progress lives on stderr via the shared
|
||||
# src/core/progress.ts reporter. \r-rewriting on stdout breaks every
|
||||
# piped-output scenario: agents that capture stdout for structured
|
||||
# results see progress garbage mixed with the data, and CI logs show
|
||||
# a single line per command because everything after the last \r
|
||||
# is truncated by the terminal emulator when played back.
|
||||
#
|
||||
# This script greps for the anti-pattern. Legitimate uses of \r inside
|
||||
# string literals (e.g. Windows line-ending normalization, regex
|
||||
# patterns) are expected to contain \r without being preceded by
|
||||
# `process.stdout.write`. We match the full write-call form only.
|
||||
#
|
||||
# Usage: scripts/check-progress-to-stdout.sh
|
||||
# Exit: 0 when clean, 1 when a banned pattern is found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# The banned pattern: process.stdout.write('\r... or process.stdout.write("\r...
|
||||
# Greedy quote character class so both quote styles match.
|
||||
PATTERN="process\.stdout\.write\([\`'\"]\\\\r"
|
||||
|
||||
# Files allowed to use this pattern historically. Empty allowlist — the point
|
||||
# of v0.14.2 was to remove every one of them. Add entries only if you really
|
||||
# need a \r on stdout (if so, add the rationale as a comment at the call site
|
||||
# and list the file here).
|
||||
ALLOWLIST=()
|
||||
|
||||
matches=""
|
||||
if command -v rg >/dev/null 2>&1; then
|
||||
matches="$(rg -n --no-heading "$PATTERN" src/ 2>/dev/null || true)"
|
||||
else
|
||||
matches="$(grep -rEn "$PATTERN" src/ 2>/dev/null || true)"
|
||||
fi
|
||||
|
||||
if [ -n "$matches" ]; then
|
||||
# Filter out allowlisted files.
|
||||
filtered="$matches"
|
||||
for f in "${ALLOWLIST[@]:-}"; do
|
||||
[ -z "$f" ] && continue
|
||||
filtered="$(echo "$filtered" | grep -v "^${f}:" || true)"
|
||||
done
|
||||
|
||||
if [ -n "$filtered" ]; then
|
||||
echo "ERROR: found process.stdout.write('\\r…') pattern(s) in src/:"
|
||||
echo
|
||||
echo "$filtered"
|
||||
echo
|
||||
echo "Bulk-action progress must go through src/core/progress.ts"
|
||||
echo "(writes to stderr, handles TTY vs non-TTY, honors --quiet /"
|
||||
echo " --progress-json / --progress-interval). If you genuinely"
|
||||
echo "need a \\r on stdout, add the file to the ALLOWLIST at the"
|
||||
echo "top of this script and explain why at the call site."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "check-progress-to-stdout: OK (no banned stdout \\r patterns)"
|
||||
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* llms-config — single source of truth for llms.txt + llms-full.txt.
|
||||
*
|
||||
* Consumed by scripts/build-llms.ts (emits llms.txt, llms-full.txt) and
|
||||
* test/build-llms.test.ts (asserts paths resolve, content contract holds).
|
||||
*
|
||||
* Adding a doc? Add it here and run `bun run build:llms`. The drift-detection
|
||||
* test fails CI if you forget.
|
||||
*
|
||||
* Fork-friendliness: `rawBaseUrl` reads from `LLMS_REPO_BASE` so forks can
|
||||
* regenerate without manual URL rewrites:
|
||||
* LLMS_REPO_BASE=https://raw.githubusercontent.com/fork-org/gbrain/main bun run build:llms
|
||||
*/
|
||||
|
||||
export type DocEntry = {
|
||||
title: string;
|
||||
description: string;
|
||||
path: string;
|
||||
includeInFull?: boolean;
|
||||
};
|
||||
|
||||
export type DocSection = {
|
||||
heading: string;
|
||||
optional?: boolean;
|
||||
entries: DocEntry[];
|
||||
};
|
||||
|
||||
export const PROJECT = {
|
||||
name: "GBrain",
|
||||
summary:
|
||||
"GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable engines (PGLite default, Postgres+pgvector for scale), contract-first operations, 26 fat-markdown skills. Teaches agents brain ops, ingestion, enrichment, scheduling, identity, and access control.",
|
||||
repoUrl: "https://github.com/garrytan/gbrain",
|
||||
rawBaseUrl:
|
||||
process.env.LLMS_REPO_BASE ??
|
||||
"https://raw.githubusercontent.com/garrytan/gbrain/master",
|
||||
};
|
||||
|
||||
export const SECTIONS: DocSection[] = [
|
||||
{
|
||||
heading: "Core entry points",
|
||||
entries: [
|
||||
{
|
||||
title: "AGENTS.md",
|
||||
description:
|
||||
"Start here if you are not Claude Code. Install order, trust boundary, skill resolver, config/debug/migration pointers.",
|
||||
path: "AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "CLAUDE.md",
|
||||
description:
|
||||
"Architecture reference. Key files, trust boundaries, engine factory, test layout.",
|
||||
path: "CLAUDE.md",
|
||||
},
|
||||
{
|
||||
title: "INSTALL_FOR_AGENTS.md",
|
||||
description: "9-step agent installation.",
|
||||
path: "INSTALL_FOR_AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "skills/RESOLVER.md",
|
||||
description: "Skill dispatcher. Read first for any task.",
|
||||
path: "skills/RESOLVER.md",
|
||||
},
|
||||
{
|
||||
title: "README.md",
|
||||
description: "Project overview, benchmarks, 30-minute setup.",
|
||||
path: "README.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Configuration",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/ENGINES.md",
|
||||
description: "PGLite vs Postgres trade-off and when to migrate.",
|
||||
path: "docs/ENGINES.md",
|
||||
},
|
||||
{
|
||||
title: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
|
||||
description:
|
||||
"MECE directory structure (people/, companies/, concepts/).",
|
||||
path: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/live-sync.md",
|
||||
description: "Incremental markdown sync setup.",
|
||||
path: "docs/guides/live-sync.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/cron-schedule.md",
|
||||
description: "Recurring job scheduling.",
|
||||
path: "docs/guides/cron-schedule.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/minions-deployment.md",
|
||||
description:
|
||||
"Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.",
|
||||
path: "docs/guides/minions-deployment.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/quiet-hours.md",
|
||||
description: "Notification hold + timezone-aware delivery.",
|
||||
path: "docs/guides/quiet-hours.md",
|
||||
},
|
||||
{
|
||||
title: "docs/mcp/DEPLOY.md",
|
||||
description: "MCP server deployment.",
|
||||
path: "docs/mcp/DEPLOY.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Debugging",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/GBRAIN_VERIFY.md",
|
||||
description:
|
||||
"7-check post-setup verification. Start here when something feels off.",
|
||||
path: "docs/GBRAIN_VERIFY.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/minions-fix.md",
|
||||
description: "Troubleshooting the Minions job queue.",
|
||||
path: "docs/guides/minions-fix.md",
|
||||
},
|
||||
{
|
||||
title: "docs/integrations/reliability-repair.md",
|
||||
description: "Data integrity recovery.",
|
||||
path: "docs/integrations/reliability-repair.md",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Migrations",
|
||||
entries: [
|
||||
{
|
||||
title: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
|
||||
description:
|
||||
"Patches for downstream agent skill forks. One section per release.",
|
||||
path: "docs/UPGRADING_DOWNSTREAM_AGENTS.md",
|
||||
},
|
||||
{
|
||||
title: "skills/migrations/",
|
||||
description:
|
||||
"Per-version (v0.5.0 - v0.14.1) agent-executable migration instructions.",
|
||||
path: "skills/migrations/",
|
||||
},
|
||||
{
|
||||
title: "CHANGELOG.md",
|
||||
description:
|
||||
"Release-summary voice + itemized changes + self-repair block per version.",
|
||||
path: "CHANGELOG.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Philosophy",
|
||||
optional: true,
|
||||
entries: [
|
||||
{
|
||||
title: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
|
||||
description: "Why skills live in markdown.",
|
||||
path: "docs/ethos/THIN_HARNESS_FAT_SKILLS.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
|
||||
description: "Homebrew for Personal AI.",
|
||||
path: "docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
heading: "Optional",
|
||||
optional: true,
|
||||
entries: [
|
||||
{
|
||||
title: "docs/benchmarks/",
|
||||
description: "Retrieval quality benchmarks.",
|
||||
path: "docs/benchmarks/",
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/designs/",
|
||||
description: "Forward-looking designs.",
|
||||
path: "docs/designs/",
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/architecture/infra-layer.md",
|
||||
description: "Shared infra patterns.",
|
||||
path: "docs/architecture/infra-layer.md",
|
||||
includeInFull: false,
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
|
||||
export const INLINE_TIPS = [
|
||||
"`gbrain doctor [--json] [--fast] [--fix]` - built-in health checks.",
|
||||
"`gbrain orphans [--json]` - pages with zero inbound wikilinks.",
|
||||
"`gbrain repair-jsonb [--dry-run]` - repair v0.12.0 double-encoded JSONB rows.",
|
||||
"`gbrain upgrade` runs post-upgrade + apply-migrations.",
|
||||
];
|
||||
|
||||
// Target ~600KB so llms-full.txt fits in ~150k-token contexts with room to spare.
|
||||
// Generator prints a WARN if exceeded; ship with includeInFull=false exclusions.
|
||||
export const FULL_SIZE_BUDGET = 600_000;
|
||||
Executable
+66
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run E2E tests ONE FILE AT A TIME.
|
||||
#
|
||||
# Bun's default is to run test files in parallel (each in its own worker).
|
||||
# Our E2E suite shares one Postgres database across all 13 files, and
|
||||
# `setupDB()` does TRUNCATE CASCADE + fixture import. When files run in
|
||||
# parallel, file A's TRUNCATE can race with file B's fixture import,
|
||||
# producing observed fails like "expected 16 pages, got 8", missing
|
||||
# links, orphaned timeline entries, etc. The flakiness was visible on
|
||||
# ~3 of every 5 runs pre-fix.
|
||||
#
|
||||
# Running files sequentially eliminates the race entirely. It also costs
|
||||
# some startup overhead (each file spins up a fresh bun process) but for
|
||||
# a suite this size that is measured in ~1-2s per file, amortized under
|
||||
# the natural per-file test time of 5-10s.
|
||||
#
|
||||
# Exits non-zero on the first failing file so CI fails fast.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
pass_files=0
|
||||
fail_files=0
|
||||
fail_list=()
|
||||
total_pass=0
|
||||
total_fail=0
|
||||
|
||||
for f in test/e2e/*.test.ts; do
|
||||
name=$(basename "$f")
|
||||
echo ""
|
||||
echo "=== $name ==="
|
||||
if output=$(bun test "$f" 2>&1); then
|
||||
pass_files=$((pass_files + 1))
|
||||
# Extract pass/fail counts from bun's summary (e.g., "123 pass")
|
||||
p=$(echo "$output" | grep -oE '[0-9]+ pass' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
total_pass=$((total_pass + p))
|
||||
echo "$output" | tail -8
|
||||
else
|
||||
fail_files=$((fail_files + 1))
|
||||
fail_list+=("$name")
|
||||
p=$(echo "$output" | grep -oE '[0-9]+ pass' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
fl=$(echo "$output" | grep -oE '[0-9]+ fail' | tail -1 | grep -oE '[0-9]+' || echo 0)
|
||||
total_pass=$((total_pass + p))
|
||||
total_fail=$((total_fail + fl))
|
||||
echo "$output"
|
||||
echo ""
|
||||
echo "FAILED: $name"
|
||||
# Continue so we see all failures; exit nonzero at the end.
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "========================================"
|
||||
echo "E2E SUMMARY (sequential execution)"
|
||||
echo "========================================"
|
||||
echo "Files: $((pass_files + fail_files)) total, $pass_files passed, $fail_files failed"
|
||||
echo "Tests: $total_pass passed, $total_fail failed"
|
||||
if [ ${#fail_list[@]} -gt 0 ]; then
|
||||
echo ""
|
||||
echo "Failing files:"
|
||||
for f in "${fail_list[@]}"; do
|
||||
echo " - $f"
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
@@ -15,7 +15,7 @@
|
||||
* Returns JSON when --json is passed: { path, score, total, items,
|
||||
* recommendation }. Exit code is 0 when score == total, 1 otherwise.
|
||||
*
|
||||
* Ported from ~/git/wintermute/workspace/scripts/skillify-check.mjs
|
||||
* Ported from ~/git/your-openclaw/workspace/scripts/skillify-check.mjs
|
||||
* (genericized: paths computed from $PROJECT_ROOT + runtime test-dir
|
||||
* detection; replaces the manual `grep AGENTS.md` check with a reference
|
||||
* to `gbrain check-resolvable` which validates the resolver better).
|
||||
@@ -23,6 +23,7 @@
|
||||
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { join, basename, dirname, resolve } from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
|
||||
function projectRoot(): string {
|
||||
// Walk up from cwd until we find a package.json — that's the repo root.
|
||||
@@ -64,6 +65,45 @@ function checkOptional(name: string, passed: boolean, detail?: string): CheckIte
|
||||
return { name, passed, required: false, detail };
|
||||
}
|
||||
|
||||
/**
|
||||
* Invoke `gbrain check-resolvable --json` once and cache the result for the
|
||||
* process lifetime. Binary-missing surfaces a loud error instead of silently
|
||||
* passing — this is the critical guard the failure-mode audit flagged.
|
||||
*/
|
||||
interface ResolverResult {
|
||||
ok: boolean;
|
||||
detail: string;
|
||||
}
|
||||
let _resolverCache: ResolverResult | null = null;
|
||||
function runCheckResolvableCached(): ResolverResult {
|
||||
if (_resolverCache) return _resolverCache;
|
||||
try {
|
||||
const res = spawnSync('gbrain', ['check-resolvable', '--json'], {
|
||||
encoding: 'utf-8',
|
||||
maxBuffer: 10 * 1024 * 1024,
|
||||
});
|
||||
if (res.error || res.status === null) {
|
||||
const reason = res.error?.message ?? 'spawn returned null status';
|
||||
console.error(`[skillify] gbrain check-resolvable not runnable: ${reason}`);
|
||||
_resolverCache = { ok: false, detail: `check-resolvable unavailable: ${reason}` };
|
||||
return _resolverCache;
|
||||
}
|
||||
const payload = JSON.parse(res.stdout);
|
||||
if (payload.ok === true) {
|
||||
_resolverCache = { ok: true, detail: 'all skill-tree checks pass' };
|
||||
} else {
|
||||
const count = payload.report?.issues?.length ?? 0;
|
||||
const err = payload.error ? ` (${payload.error})` : '';
|
||||
_resolverCache = { ok: false, detail: `${count} issue(s)${err} — run: gbrain check-resolvable` };
|
||||
}
|
||||
return _resolverCache;
|
||||
} catch (err) {
|
||||
console.error(`[skillify] check-resolvable parse failed: ${err}`);
|
||||
_resolverCache = { ok: false, detail: `check-resolvable parse error: ${err}` };
|
||||
return _resolverCache;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Guess the skill-directory name from a script path.
|
||||
* scripts/frameio-scraper.ts → frameio-scraper
|
||||
@@ -184,12 +224,13 @@ function runCheck(target: string): {
|
||||
}
|
||||
items.push(checkOptional('Resolver trigger eval', hasTriggerEval));
|
||||
|
||||
// 8. check-resolvable — we don't run it here (side effects + cost); we
|
||||
// report whether the SKILL.md exists at all, which is the ground-truth
|
||||
// input check-resolvable would consume.
|
||||
items.push(checkOptional('check-resolvable input present',
|
||||
existsSync(skillMd) && existsSync(RESOLVER_MD),
|
||||
'run: gbrain check-resolvable'));
|
||||
// 8. check-resolvable — invoke the real gate. Cached per process so
|
||||
// iterating many skills only runs the subprocess once. Binary-missing
|
||||
// is surfaced loudly so a silent false-pass can't happen.
|
||||
const resolverResult = runCheckResolvableCached();
|
||||
items.push(checkOptional('check-resolvable gate',
|
||||
resolverResult.ok,
|
||||
resolverResult.detail));
|
||||
|
||||
// 9. E2E — same as item 4 but required.
|
||||
items.push(check('E2E test (either under e2e/ or integration test)', hasE2E, 'try /qa or test/e2e/'));
|
||||
|
||||
@@ -37,7 +37,7 @@ This skill guarantees:
|
||||
|
||||
> **Filing rule:** Read `skills/_brain-filing-rules.md` before creating any new page.
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link
|
||||
FROM that entity's page TO the page mentioning them. An unlinked mention is a
|
||||
|
||||
@@ -36,7 +36,7 @@ This skill guarantees:
|
||||
- Every fact has an inline `[Source: ...]` citation
|
||||
- Filing follows primary subject rules (not format-based)
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link.
|
||||
Format: `- **YYYY-MM-DD** | Referenced in [page title](path) — brief context`
|
||||
|
||||
@@ -29,7 +29,7 @@ Ingest meetings, articles, media, documents, and conversations into the brain.
|
||||
- State sections are rewritten with current best understanding, never appended to.
|
||||
- Entity detection fires on every inbound message; notable entities get pages or updates.
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link
|
||||
FROM that entity's page TO the page mentioning them. An unlinked mention is a
|
||||
|
||||
@@ -39,7 +39,7 @@ This skill guarantees:
|
||||
- Raw source files preserved via `gbrain files upload-raw`
|
||||
- Filing by primary subject, not by media format
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every mention of a person or company with a brain page MUST create a back-link.
|
||||
|
||||
|
||||
@@ -34,7 +34,7 @@ This skill guarantees:
|
||||
- Meeting is NOT fully ingested until enrich runs for every entity
|
||||
- Back-links created bidirectionally
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every attendee and company mentioned MUST get a back-link from their page to
|
||||
the meeting page. An unlinked mention is a broken brain.
|
||||
|
||||
@@ -9,7 +9,7 @@ feature_pitch:
|
||||
|
||||
# v0.11.0 Migration: Minions — host-agent instruction manual
|
||||
|
||||
**Audience: host agents (Wintermute, other OpenClaw deployments, future
|
||||
**Audience: host agents (OpenClaw deployments, future
|
||||
hosts) reading this AFTER `gbrain apply-migrations` has run its
|
||||
mechanical phases.** The orchestrator in
|
||||
`src/commands/migrations/v0_11_0.ts` is the runtime source of truth for
|
||||
@@ -32,7 +32,7 @@ Non-empty? Each line is a TODO. Each `type` routes to a section below.
|
||||
Gbrain rewrites cron entries whose handler name matches a gbrain
|
||||
builtin (`sync`, `embed`, `lint`, `import`, `extract`, `backlinks`,
|
||||
`autopilot-cycle`). For host-specific handlers (e.g. `ea-inbox-sweep`,
|
||||
`frameio-scan`, `x-dm-triage`, `calendar-sync` on Wintermute), gbrain
|
||||
`frameio-scan`, `x-dm-triage`, `calendar-sync` on your OpenClaw), gbrain
|
||||
leaves the manifest alone and emits a TODO with shape:
|
||||
|
||||
```json
|
||||
@@ -69,7 +69,7 @@ await worker.start();
|
||||
### (b) Ship the bootstrap in your host repo
|
||||
|
||||
Autopilot already spawns `gbrain jobs work` as a child. Configure it to
|
||||
spawn your custom worker binary (e.g. `wintermute-worker`) instead, or
|
||||
spawn your custom worker binary (e.g. `your-openclaw-worker`) instead, or
|
||||
register handlers as a side-effect module that the stock worker loads on
|
||||
startup. Either path is documented in `plugin-handlers.md`.
|
||||
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
version: 0.15.2
|
||||
feature_pitch:
|
||||
headline: "Silent binaries are dead. Every bulk action now heartbeats."
|
||||
description: |
|
||||
`gbrain doctor` on a 52K-page brain used to sit silent for 10+
|
||||
minutes before an agent timeout killed it. Same pattern on embed,
|
||||
sync, import, extract, migrate, and every orchestrator. v0.15.2
|
||||
routes 14 bulk commands through one shared reporter that writes
|
||||
to stderr. Non-TTY default is plain human lines; agents that
|
||||
want structured events add `--progress-json` and get one JSON
|
||||
object per line. Stdout stays clean for data output. Event
|
||||
schema is locked in docs/progress-events.md.
|
||||
recipe: docs/progress-events.md
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.15.2 Migration: Bulk-action progress streaming
|
||||
|
||||
**Audience: host agents reading this after `gbrain apply-migrations`
|
||||
has run. v0.15.2 is purely additive to the CLI surface, there is no
|
||||
schema change, no data rewrite, and no orchestrator for this release.**
|
||||
Your binaries just got observable. This file tells you how to use it.
|
||||
|
||||
## Mechanical migration: nothing
|
||||
|
||||
There is no mechanical step. If `gbrain upgrade` completed, progress
|
||||
events are already flowing the next time you invoke a bulk command.
|
||||
Read on to know what's there and how to consume it.
|
||||
|
||||
## What's new at the CLI
|
||||
|
||||
### Three new global flags
|
||||
|
||||
These work on any `gbrain` subcommand:
|
||||
|
||||
- `--progress-json` — emit one JSON event per line on stderr.
|
||||
- `--quiet` — suppress progress output entirely.
|
||||
- `--progress-interval=<ms>` — minimum ms between progress emits
|
||||
(default 1000).
|
||||
|
||||
Parsed before command dispatch, so both work:
|
||||
|
||||
```
|
||||
gbrain --progress-json doctor --json
|
||||
gbrain doctor --json --progress-json
|
||||
```
|
||||
|
||||
### Per-TTY behavior
|
||||
|
||||
Without `--progress-json`:
|
||||
|
||||
- **TTY:** `\r`-rewriting single-line progress on stderr (fancy).
|
||||
- **Non-TTY (pipe, CI, agent):** one plain-text line per event on
|
||||
stderr. No JSON, no noise. Human-readable.
|
||||
|
||||
The default was deliberately NOT JSON-on-non-TTY. Shell pipelines
|
||||
that just pipe `gbrain ... | less` should get readable logs, not a
|
||||
JSON blob. Agents opt in to JSON explicitly.
|
||||
|
||||
## What's new per command
|
||||
|
||||
Fourteen commands now stream progress through the shared reporter:
|
||||
|
||||
| Command | What you'll see |
|
||||
|---------|-----------------|
|
||||
| `doctor` | `doctor.db_checks` phase + per-check heartbeats, including a 1s heartbeat while `markdown_body_completeness` scans |
|
||||
| `orphans` | `orphans.scan` heartbeat while the anti-join runs |
|
||||
| `embed` | `embed.pages` with per-page ticks |
|
||||
| `files sync` | `files.sync` with per-file ticks |
|
||||
| `export` | `export.pages` with per-page ticks |
|
||||
| `import` | `import.files` with per-file ticks (replaces per-100 stdout logs) |
|
||||
| `extract [links|timeline|all]` (fs + db) | `extract.links_fs` / `extract.timeline_db` etc. |
|
||||
| `sync` | `sync.deletes`, `sync.renames`, `sync.imports` phases |
|
||||
| `migrate --to ...` | `migrate.copy_pages`, `migrate.copy_links` |
|
||||
| `repair-jsonb` | `repair_jsonb.run` + per-column heartbeats |
|
||||
| `check-backlinks` | `backlinks.scan` heartbeat |
|
||||
| `lint` | `lint.pages` per-page ticks |
|
||||
| `integrity auto` | `integrity.auto` per-page ticks |
|
||||
| `eval` | `eval.single` / `eval.ab` per-query ticks |
|
||||
| `apply-migrations` (v0_11/v0_12_0/v0_12_2) | Child processes inherit the parent's progress mode |
|
||||
|
||||
## JSON event schema
|
||||
|
||||
Documented in `docs/progress-events.md` (canonical reference). Stable
|
||||
from v0.15.2, additive changes only.
|
||||
|
||||
Quick agent cheat sheet:
|
||||
|
||||
```json
|
||||
{"event":"start","phase":"doctor.db_checks","ts":"..."}
|
||||
{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"eta_ms":10300,"ts":"..."}
|
||||
{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation...","elapsed_ms":1000,"ts":"..."}
|
||||
{"event":"finish","phase":"doctor.db_checks","elapsed_ms":187000,"ts":"..."}
|
||||
{"event":"abort","phase":"orphans.scan","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}
|
||||
```
|
||||
|
||||
Parser rules:
|
||||
|
||||
1. One JSON object per line on stderr.
|
||||
2. Ignore unknown event types and unknown fields. Schema is additive.
|
||||
3. Group by `phase` prefix to track one run: all `doctor.*` events
|
||||
belong to the same `doctor` invocation.
|
||||
4. `total` / `pct` / `eta_ms` are absent when the scan doesn't have a
|
||||
total up front (e.g. heartbeat-only paths). Don't assume they exist.
|
||||
|
||||
## Minion jobs
|
||||
|
||||
`gbrain jobs work` (the Minion worker daemon) writes progress to the
|
||||
DB via `job.updateProgress`, not to stderr. Read per-job progress via
|
||||
the `get_job_progress` MCP op or:
|
||||
|
||||
```bash
|
||||
gbrain jobs submit embed
|
||||
# while it runs:
|
||||
gbrain jobs get <id> # .progress updates live as the handler ticks
|
||||
```
|
||||
|
||||
The `embed` Minion handler is wired as of v0.15.2. Other bulk cores
|
||||
(`sync`, `extract`, `backlinks`, `import`, `autopilot-cycle`) have the
|
||||
callback plumbing ready and will follow.
|
||||
|
||||
## Backward-compatibility warnings
|
||||
|
||||
Five commands moved per-page progress from stdout to stderr:
|
||||
|
||||
- `embed` (was `\r`-on-stdout)
|
||||
- `files sync` (was `\r`-on-stdout)
|
||||
- `export` (was `\r`-on-stdout, newly in scope)
|
||||
- `migrate-engine` (was per-50 `console.log` to stdout)
|
||||
- `import` (was per-100 `console.log` to stdout)
|
||||
|
||||
If you have scripts that grep `stdout` for progress strings like
|
||||
`Progress: 1234/52000` or `\r 1234/52000 pages...` — those strings
|
||||
now live on stderr. The final data summaries (`Embedded N chunks
|
||||
across M pages`, `Import complete`, etc.) remain on stdout so the
|
||||
"did it finish" signal is unchanged.
|
||||
|
||||
`integrity auto` still writes `~/.gbrain/integrity-progress.jsonl`,
|
||||
but its role is now "resume marker only" — live progress goes through
|
||||
the reporter. If you depended on tailing that file for real-time
|
||||
progress, switch to the stderr stream.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# Your agent sees structured events; stdout stays JSON-parseable:
|
||||
gbrain --progress-json doctor --json > doctor.json 2> doctor.progress.log
|
||||
wc -l doctor.progress.log # should be non-zero
|
||||
jq . doctor.json # should parse cleanly
|
||||
|
||||
# For a very large brain, watch the heartbeat:
|
||||
gbrain --progress-json doctor 2>&1 >/dev/null | grep '"event"'
|
||||
```
|
||||
|
||||
If you see silence for more than a second or two on a non-trivial
|
||||
command, file an issue with the exact command and the first 100 lines
|
||||
of stderr.
|
||||
|
||||
## That's the whole migration
|
||||
|
||||
No mechanical step. No config change. Agents that parse `stdout` keep
|
||||
working; agents that want progress now have it on a clean stderr
|
||||
channel with a documented schema.
|
||||
@@ -0,0 +1,167 @@
|
||||
---
|
||||
version: 0.17.0
|
||||
feature_pitch:
|
||||
headline: "One brain maintenance cycle, two CLIs. `gbrain dream` delivers the README promise."
|
||||
description: |
|
||||
The README has said "the agent runs while I sleep, the dream cycle
|
||||
scans every conversation, enriches missing entities, fixes broken
|
||||
citations, consolidates memory" for a year. v0.17 makes that real
|
||||
as a first-class command (`gbrain dream`) backed by one shared
|
||||
primitive (`runCycle`). Autopilot users get lint + orphan sweep
|
||||
added to their nightly cycle automatically — no config change.
|
||||
Cron users get a single legible verb: `0 2 * * * gbrain dream`.
|
||||
Both converge on the same phase order (lint → backlinks → sync →
|
||||
extract → embed → orphans) so file fixes land in the DB the same
|
||||
night, not the next.
|
||||
recipe: null
|
||||
tiers: null
|
||||
---
|
||||
|
||||
# v0.17.0 Migration: `gbrain dream` + unified maintenance cycle
|
||||
|
||||
**Audience: agents + humans upgrading from v0.16.x. There is no
|
||||
mechanical migration step required — the schema migration (v16
|
||||
cycle-lock table) and behavior changes all apply automatically on
|
||||
upgrade. This file documents what changed, how to verify it, and
|
||||
the one opt-out users may care about.**
|
||||
|
||||
## What changed
|
||||
|
||||
### New command: `gbrain dream`
|
||||
|
||||
The brand-promise one-liner. Runs one brain maintenance cycle and
|
||||
exits. Designed for cron.
|
||||
|
||||
```
|
||||
gbrain dream # full 6-phase cycle
|
||||
gbrain dream --dry-run --json # preview, agent-readable
|
||||
gbrain dream --phase lint # single-phase (fast, targeted)
|
||||
gbrain dream --pull # git pull before syncing
|
||||
0 2 * * * gbrain dream --json # nightly cron
|
||||
```
|
||||
|
||||
See `gbrain dream --help` for the full flag reference.
|
||||
|
||||
### Autopilot now runs lint + orphan sweep
|
||||
|
||||
`gbrain autopilot --install` users: on upgrade, your daemon's cycle
|
||||
gains two phases it didn't run before:
|
||||
|
||||
- **lint --fix** — auto-fixes LLM artifacts, placeholder dates, bad
|
||||
citations across the brain. Modifies files on disk.
|
||||
- **orphan sweep** — reports (read-only) pages with no inbound
|
||||
wikilinks. Visible in `gbrain jobs list` output for each
|
||||
`autopilot-cycle` job.
|
||||
|
||||
No action required. The new phases run on the daemon's existing
|
||||
interval.
|
||||
|
||||
### Shared primitive: `src/core/cycle.ts`
|
||||
|
||||
Three callers (dream CLI, autopilot inline path, autopilot-cycle
|
||||
Minions handler) now all delegate to `runCycle(engine, opts)`. One
|
||||
source of truth for what happens overnight.
|
||||
|
||||
### Cycle coordination via a DB lock table
|
||||
|
||||
`gbrain_cycle_locks` (new table, migration v16) replaces
|
||||
session-scoped `pg_try_advisory_lock` which the v0.15.4
|
||||
PgBouncer-transaction-pooler fix silently broke. The table has a
|
||||
TTL (30 min), refreshed between phases, so crashed holders
|
||||
auto-release.
|
||||
|
||||
## Verify after upgrade
|
||||
|
||||
```bash
|
||||
# 1. Dream command exists:
|
||||
gbrain dream --help
|
||||
|
||||
# 2. Run a dry cycle (safe, no writes):
|
||||
gbrain dream --dry-run --json
|
||||
|
||||
# 3. If you run autopilot --install:
|
||||
gbrain jobs list --status complete | head -5
|
||||
# Each `autopilot-cycle` entry now has 6 phases in its report,
|
||||
# not 4. Check a recent one with `gbrain jobs get <id>`.
|
||||
|
||||
# 4. Schema migration landed:
|
||||
gbrain doctor # should show no pending migrations
|
||||
```
|
||||
|
||||
Expected `gbrain dream --dry-run` output on a healthy brain:
|
||||
|
||||
```
|
||||
Brain is healthy. 6 phase(s) checked in 1.3s.
|
||||
```
|
||||
|
||||
Or with `--json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "1",
|
||||
"status": "clean",
|
||||
"phases": [...],
|
||||
"totals": { "lint_fixes": 0, "backlinks_added": 0, ... }
|
||||
}
|
||||
```
|
||||
|
||||
## Opt-outs for autopilot-installed users
|
||||
|
||||
If you explicitly do NOT want autopilot's daemon modifying files
|
||||
(lint + backlinks phases write to disk):
|
||||
|
||||
**Option 1: disable those phases in cron-dream but keep autopilot
|
||||
running.** Since dream is separate, you can run just the phases
|
||||
you want from cron without touching autopilot:
|
||||
|
||||
```bash
|
||||
# e.g. only re-embed and orphan-sweep nightly, skip file mutations:
|
||||
0 2 * * * gbrain dream --phase orphans
|
||||
```
|
||||
|
||||
**Option 2: uninstall autopilot and use cron-dream only.**
|
||||
|
||||
```bash
|
||||
gbrain autopilot --uninstall
|
||||
# Then add to your crontab:
|
||||
0 2 * * * gbrain dream --pull
|
||||
```
|
||||
|
||||
**Option 3: accept the default.** The new phases are conservative:
|
||||
lint only fixes known-safe artifacts (em dashes, placeholder dates),
|
||||
never destructive. Back-link fills are additive. If something does
|
||||
go wrong, `gbrain dream --dry-run` always tells you what WOULD
|
||||
change before you run it for real.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"cycle_already_running" in dream output:**
|
||||
Another cycle (probably autopilot's daemon) is holding the lock.
|
||||
Expected behavior — dream skipped to avoid racing the daemon. The
|
||||
daemon's next interval will pick up the work.
|
||||
|
||||
**`gbrain dream --dry-run` reports changes when you expected none:**
|
||||
Check `gbrain doctor` for drift: lint issues, stale embeddings,
|
||||
missing back-links. Dream's dry-run is the honest preview of what
|
||||
autopilot's daemon will do on its next cycle.
|
||||
|
||||
**Minion `autopilot-cycle` jobs failing after upgrade:**
|
||||
Open a GitHub issue with the output of `gbrain jobs get <id>` for a
|
||||
failing job. The new runCycle-backed handler preserves the
|
||||
partial-failure semantic (one phase failing doesn't block future
|
||||
cycles), but specific phases may surface new error classes.
|
||||
|
||||
## What did NOT change
|
||||
|
||||
- `gbrain autopilot --install` machinery (launchd / systemd /
|
||||
crontab generators). Existing installs keep working.
|
||||
- `~/.gbrain/autopilot.lock` daemon-singleton lockfile. Separate
|
||||
concern from the new per-cycle lock.
|
||||
- `gbrain jobs` interface. `gbrain jobs get <id>` now shows a
|
||||
richer report structure (schema_version:"1"), but the surface
|
||||
API is stable.
|
||||
|
||||
---
|
||||
|
||||
*This migration file is informational only. No mechanical step is
|
||||
required — all changes apply automatically on `gbrain upgrade`.*
|
||||
@@ -275,7 +275,7 @@ Inject the key patterns into the agent's system context or AGENTS.md:
|
||||
1. **Brain-agent loop** (Section 2): read before responding, write after learning
|
||||
2. **Entity detection** (Section 3): spawn on every message, capture people/companies/ideas
|
||||
3. **Source attribution** (Section 7): every fact needs `[Source: ...]`
|
||||
4. **Iron law back-linking** (Section 15.4): every mention links back to the entity page
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Tell the user: "The production agent guide is at docs/GBRAIN_SKILLPACK.md. It covers
|
||||
the brain-agent loop, entity detection, enrichment, meeting ingestion, and cron
|
||||
|
||||
@@ -39,7 +39,7 @@ This skill guarantees:
|
||||
- Back-links all entity mentions (Iron Law)
|
||||
- Citations on every fact written
|
||||
|
||||
## Iron Law: Back-Linking (MANDATORY)
|
||||
> **Convention:** See `skills/conventions/quality.md` for Iron Law back-linking.
|
||||
|
||||
Every time this skill creates or updates a brain page that mentions a person or company:
|
||||
1. Check if that person/company has a brain page
|
||||
|
||||
@@ -4,7 +4,7 @@ version: 1.0.0
|
||||
description: |
|
||||
Run `gbrain skillpack-check` to produce an agent-readable JSON health report
|
||||
for the gbrain install. Wraps `gbrain doctor` + `gbrain apply-migrations
|
||||
--list` so a host agent (Wintermute's morning-briefing, any OpenClaw cron)
|
||||
--list` so a host agent (your OpenClaw's morning-briefing, any OpenClaw cron)
|
||||
can see at a glance whether the skillpack needs attention.
|
||||
|
||||
Use when the user asks "is gbrain healthy?", when a cron fires a morning
|
||||
@@ -40,7 +40,7 @@ Exit code:
|
||||
|
||||
## When to run
|
||||
|
||||
- **Daily cron** (e.g. Wintermute's `morning-briefing`): `gbrain skillpack-check --quiet`.
|
||||
- **Daily cron** (e.g. your OpenClaw's `morning-briefing`): `gbrain skillpack-check --quiet`.
|
||||
Exit code alone tells you if anything is wrong; surface a one-liner in the
|
||||
briefing only when exit != 0. No JSON noise in happy-path briefings.
|
||||
- **On demand**: `gbrain skillpack-check` for the full JSON when debugging.
|
||||
|
||||
+53
-2
@@ -6,6 +6,7 @@ import type { BrainEngine } from './core/engine.ts';
|
||||
import { operations, OperationError } from './core/operations.ts';
|
||||
import type { Operation, OperationContext } from './core/operations.ts';
|
||||
import { serializeMarkdown } from './core/markdown.ts';
|
||||
import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts';
|
||||
import { VERSION } from './version.ts';
|
||||
|
||||
// Build CLI name -> operation lookup
|
||||
@@ -18,10 +19,16 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check', 'resolvers', 'integrity', 'repair-jsonb', 'orphans']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'dream', 'check-resolvable', 'repos']);
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
// BEFORE command dispatch, so `gbrain --progress-json doctor` works.
|
||||
// The stripped argv is what the command sees.
|
||||
const rawArgs = process.argv.slice(2);
|
||||
const { cliOpts, rest: args } = parseGlobalFlags(rawArgs);
|
||||
setCliOptions(cliOpts);
|
||||
|
||||
let command = args[0];
|
||||
|
||||
if (!command || command === '--help' || command === '-h') {
|
||||
@@ -148,6 +155,7 @@ function makeContext(engine: BrainEngine, params: Record<string, unknown>): Oper
|
||||
// Local CLI invocation — the user owns the machine; do not apply remote-caller
|
||||
// confinement (e.g., cwd-locked file_upload).
|
||||
remote: false,
|
||||
cliOpts: getCliOptions(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -302,6 +310,16 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runLint(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'check-resolvable') {
|
||||
const { runCheckResolvable } = await import('./commands/check-resolvable.ts');
|
||||
await runCheckResolvable(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'repos') {
|
||||
const { handleRepos } = await import('./commands/repos.ts');
|
||||
await handleRepos(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'report') {
|
||||
const { runReport } = await import('./commands/report.ts');
|
||||
await runReport(args);
|
||||
@@ -350,6 +368,25 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (command === 'dream') {
|
||||
// Dream mirrors doctor's pattern: filesystem phases run without a DB,
|
||||
// so an engine connection failure is non-fatal. runCycle honestly
|
||||
// reports DB phases as skipped when engine is null.
|
||||
const { runDream } = await import('./commands/dream.ts');
|
||||
let eng: BrainEngine | null = null;
|
||||
try {
|
||||
eng = await connectEngine();
|
||||
} catch {
|
||||
// DB unavailable — lint + backlinks still run against the brain dir.
|
||||
}
|
||||
try {
|
||||
await runDream(eng, args);
|
||||
} finally {
|
||||
if (eng) await eng.disconnect();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
@@ -405,6 +442,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runJobs(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'agent': {
|
||||
const { runAgent } = await import('./commands/agent.ts');
|
||||
await runAgent(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'sync': {
|
||||
const { runSync } = await import('./commands/sync.ts');
|
||||
await runSync(engine, args);
|
||||
@@ -544,8 +586,17 @@ TOOLS
|
||||
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
|
||||
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
|
||||
orphans [--json] [--count] Find pages with no inbound wikilinks
|
||||
dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly).
|
||||
See also: autopilot --install (continuous daemon).
|
||||
check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY)
|
||||
report --type <name> --content ... Save timestamped report to brain/reports/
|
||||
|
||||
MULTI-REPO
|
||||
repos list Show configured repos
|
||||
repos add <path> [--name N] Add a repo [--strategy markdown|code|auto]
|
||||
repos remove <name> Remove a repo
|
||||
sync --all Sync all configured repos
|
||||
|
||||
JOBS (Minions)
|
||||
jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run]
|
||||
jobs list [--status S] [--limit N] List jobs
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* `gbrain agent logs <job_id> [--follow] [--since <spec>]`
|
||||
*
|
||||
* Reads two sources and merges them chronologically:
|
||||
* - ~/.gbrain/audit/subagent-jobs-*.jsonl (heartbeat + submission events
|
||||
* — lives on the WORKER's filesystem, so this CLI's effectiveness is
|
||||
* host-local today; see docs/guides/plugin-authors.md caveat #2)
|
||||
* - subagent_messages (DB rows, authoritative for persisted conversation)
|
||||
*
|
||||
* No new DB tables; all the infrastructure landed in prior Lane commits.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { readSubagentAuditForJob } from '../core/minions/handlers/subagent-audit.ts';
|
||||
import type { SubagentAuditEvent } from '../core/minions/handlers/subagent-audit.ts';
|
||||
import { loadTranscriptRows, renderTranscript } from '../core/minions/transcript.ts';
|
||||
import type { SubagentMessageRow } from '../core/minions/transcript.ts';
|
||||
|
||||
export interface AgentLogsOpts {
|
||||
follow?: boolean;
|
||||
/** ISO-8601 timestamp OR relative like "5m" / "1h" / "2d". */
|
||||
since?: string;
|
||||
/** Override poll interval for --follow. Default 1000ms. */
|
||||
pollMs?: number;
|
||||
/** Injectable writer for testing; default process.stdout.write. */
|
||||
write?: (s: string) => void;
|
||||
/** Abort to cut off a --follow loop cleanly (tests + Ctrl-C). */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
const TERMINAL_STATUSES = new Set(['completed', 'failed', 'dead', 'cancelled']);
|
||||
|
||||
export async function runAgentLogs(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
opts: AgentLogsOpts = {},
|
||||
): Promise<void> {
|
||||
const write = opts.write ?? ((s: string) => { process.stdout.write(s); });
|
||||
const sinceIso = parseSince(opts.since);
|
||||
|
||||
// Seeded render: dump everything we have right now.
|
||||
let lastTs: string | undefined = sinceIso;
|
||||
lastTs = await dumpSince(engine, jobId, lastTs, write);
|
||||
|
||||
if (!opts.follow) return;
|
||||
|
||||
const pollMs = opts.pollMs ?? 1000;
|
||||
while (!opts.signal?.aborted) {
|
||||
await sleep(pollMs, opts.signal);
|
||||
lastTs = await dumpSince(engine, jobId, lastTs, write);
|
||||
// Break on terminal job status so --follow exits once the run is done.
|
||||
const status = await readJobStatus(engine, jobId);
|
||||
if (status && TERMINAL_STATUSES.has(status)) {
|
||||
write(`\n[gbrain agent] job ${jobId} reached terminal state: ${status}\n`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Dump events with ts >= sinceIso. Returns the max ts seen so the next
|
||||
* poll round filters cleanly. When `sinceIso` is undefined on first call,
|
||||
* everything is dumped.
|
||||
*/
|
||||
async function dumpSince(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
sinceIso: string | undefined,
|
||||
write: (s: string) => void,
|
||||
): Promise<string | undefined> {
|
||||
const audit = readSubagentAuditForJob(jobId, sinceIso ? { sinceIso } : {});
|
||||
const { messages, tools } = await loadTranscriptRows(engine, jobId);
|
||||
|
||||
// Merge audit events + message rows into one timeline ordered by ts.
|
||||
const merged: Array<{ ts: string; line: string }> = [];
|
||||
|
||||
for (const e of audit) {
|
||||
if (sinceIso && e.ts <= sinceIso) continue;
|
||||
merged.push({ ts: e.ts, line: formatAudit(e) });
|
||||
}
|
||||
for (const m of messages) {
|
||||
const ts = m.ended_at.toISOString();
|
||||
if (sinceIso && ts <= sinceIso) continue;
|
||||
merged.push({ ts, line: formatMessage(m) });
|
||||
}
|
||||
|
||||
merged.sort((a, b) => a.ts.localeCompare(b.ts));
|
||||
|
||||
let maxTs = sinceIso;
|
||||
for (const item of merged) {
|
||||
write(`${item.ts} ${item.line}\n`);
|
||||
if (!maxTs || item.ts > maxTs) maxTs = item.ts;
|
||||
}
|
||||
|
||||
// Transcript tail (renders the full message/tool tree) only if we
|
||||
// actually have messages and the job is in a terminal state. This
|
||||
// avoids spamming a half-rendered transcript mid-run.
|
||||
if (messages.length > 0 && !sinceIso) {
|
||||
const status = await readJobStatus(engine, jobId);
|
||||
if (status && TERMINAL_STATUSES.has(status)) {
|
||||
write('\n');
|
||||
write(renderTranscript(messages, tools));
|
||||
write('\n');
|
||||
}
|
||||
}
|
||||
|
||||
return maxTs;
|
||||
}
|
||||
|
||||
function formatAudit(e: SubagentAuditEvent): string {
|
||||
if (e.type === 'submission') {
|
||||
return `[submission] ${e.caller} model=${e.model ?? '?'} tools=${e.tools_count ?? 0}`;
|
||||
}
|
||||
// heartbeat
|
||||
const parts = [`[${e.event}]`, `turn=${e.turn_idx}`];
|
||||
if (e.tool_name) parts.push(`tool=${e.tool_name}`);
|
||||
if (e.ms_elapsed != null) parts.push(`${e.ms_elapsed}ms`);
|
||||
if (e.tokens) {
|
||||
const t = e.tokens;
|
||||
const tokStr = [
|
||||
t.in ? `in=${t.in}` : null,
|
||||
t.out ? `out=${t.out}` : null,
|
||||
t.cache_read ? `cache_read=${t.cache_read}` : null,
|
||||
t.cache_create ? `cache_create=${t.cache_create}` : null,
|
||||
].filter(Boolean).join(' ');
|
||||
if (tokStr) parts.push(`tokens(${tokStr})`);
|
||||
}
|
||||
if (e.error) parts.push(`error="${e.error.slice(0, 100)}"`);
|
||||
return parts.join(' ');
|
||||
}
|
||||
|
||||
function formatMessage(m: SubagentMessageRow): string {
|
||||
const blockTypes = m.content_blocks.map(b => b.type).join(',');
|
||||
return `[message #${m.message_idx} ${m.role}] blocks=${blockTypes || '(empty)'}`;
|
||||
}
|
||||
|
||||
async function readJobStatus(engine: BrainEngine, jobId: number): Promise<string | null> {
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status FROM minion_jobs WHERE id = $1`,
|
||||
[jobId],
|
||||
);
|
||||
return rows[0]?.status ?? null;
|
||||
}
|
||||
|
||||
const RELATIVE_RE = /^(\d+)\s*(s|m|h|d)$/i;
|
||||
|
||||
/** Parse `--since`. Accepts ISO-8601 or relative ("5m", "1h", "2d"). */
|
||||
export function parseSince(input: string | undefined): string | undefined {
|
||||
if (!input) return undefined;
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) return undefined;
|
||||
const rel = RELATIVE_RE.exec(trimmed);
|
||||
if (rel) {
|
||||
const [, nStr, unitRaw] = rel;
|
||||
const unit = unitRaw!.toLowerCase();
|
||||
const n = parseInt(nStr!, 10);
|
||||
const mult = unit === 's' ? 1000
|
||||
: unit === 'm' ? 60_000
|
||||
: unit === 'h' ? 3_600_000
|
||||
: 86_400_000; // 'd'
|
||||
return new Date(Date.now() - n * mult).toISOString();
|
||||
}
|
||||
// Assume ISO. `new Date(input).toISOString()` both validates and
|
||||
// normalizes; invalid ISO throws.
|
||||
const d = new Date(trimmed);
|
||||
if (isNaN(d.getTime())) {
|
||||
throw new Error(`--since: could not parse "${input}" as ISO-8601 or relative (e.g. "5m", "1h")`);
|
||||
}
|
||||
return d.toISOString();
|
||||
}
|
||||
|
||||
function sleep(ms: number, signal?: AbortSignal): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const t = setTimeout(() => { signal?.removeEventListener('abort', onAbort); resolve(); }, ms);
|
||||
const onAbort = () => { clearTimeout(t); resolve(); };
|
||||
signal?.addEventListener('abort', onAbort, { once: true });
|
||||
});
|
||||
}
|
||||
|
||||
export const __testing = {
|
||||
parseSince,
|
||||
formatAudit,
|
||||
formatMessage,
|
||||
dumpSince,
|
||||
};
|
||||
@@ -0,0 +1,333 @@
|
||||
/**
|
||||
* `gbrain agent` CLI: the user-facing entry point for the v0.15 subagent
|
||||
* runtime.
|
||||
*
|
||||
* gbrain agent run <prompt> [flags]
|
||||
* gbrain agent logs <job_id> [--follow] [--since <spec>]
|
||||
*
|
||||
* `run` submits a subagent job (or fan-out of N subagents + aggregator)
|
||||
* under the trusted-submit flag so the PROTECTED_JOB_NAMES guard doesn't
|
||||
* reject. It does NOT execute the loop here — the handler runs in a
|
||||
* `gbrain jobs work` process. `--follow` tails status until terminal;
|
||||
* without `--follow` (or with `--detach`) the CLI prints the job id and
|
||||
* exits, leaving the user to check back with `gbrain agent logs`.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { MinionQueue } from '../core/minions/queue.ts';
|
||||
import { waitForCompletion, TimeoutError } from '../core/minions/wait-for-completion.ts';
|
||||
import type { MinionJobInput, SubagentHandlerData, AggregatorHandlerData } from '../core/minions/types.ts';
|
||||
import { runAgentLogs } from './agent-logs.ts';
|
||||
|
||||
// ── arg parsing helpers ────────────────────────────────────
|
||||
|
||||
function parseFlag(args: string[], flag: string): string | undefined {
|
||||
const idx = args.indexOf(flag);
|
||||
return idx >= 0 && idx + 1 < args.length ? args[idx + 1] : undefined;
|
||||
}
|
||||
function hasFlag(args: string[], flag: string): boolean { return args.includes(flag); }
|
||||
|
||||
/** Keep CLI args that look like flags from being eaten as the prompt. */
|
||||
function isKnownFlag(s: string): boolean {
|
||||
return s.startsWith('--');
|
||||
}
|
||||
|
||||
// ── command dispatcher ────────────────────────────────────
|
||||
|
||||
export async function runAgent(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
switch (sub) {
|
||||
case 'run':
|
||||
await runAgentRun(engine, args.slice(1));
|
||||
return;
|
||||
case 'logs':
|
||||
await runAgentLogsCmd(engine, args.slice(1));
|
||||
return;
|
||||
default:
|
||||
console.error(`gbrain agent: unknown subcommand "${sub}"`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.log(`gbrain agent — durable LLM agent runs (v0.15)
|
||||
|
||||
USAGE
|
||||
gbrain agent run <prompt> [flags]
|
||||
gbrain agent logs <job_id> [--follow] [--since <spec>]
|
||||
|
||||
SUBMITTING
|
||||
gbrain agent run <prompt>
|
||||
--subagent-def <name> Named plugin subagent (from GBRAIN_PLUGIN_PATH)
|
||||
--model <id> Anthropic model id (defaults to sonnet)
|
||||
--max-turns <n> Max assistant turns (default 20)
|
||||
--tools a,b,c Subset of registered tool names (comma list)
|
||||
--timeout-ms <n> Per-job wall-clock timeout
|
||||
--fanout-manifest <path> JSON array of {prompt, input_vars?} — one child each
|
||||
--follow Tail status until terminal (default on TTY)
|
||||
--detach Submit + print job id, exit immediately
|
||||
|
||||
Flags after \`run\` up to the first unrecognized token are parsed; the
|
||||
remainder is the prompt. Use \`--\` to explicitly terminate flag parsing.
|
||||
|
||||
VIEWING
|
||||
gbrain agent logs <job_id>
|
||||
--follow Keep polling until the job reaches terminal
|
||||
--since <spec> ISO-8601 timestamp OR relative ("5m","1h","2d")
|
||||
|
||||
NOTES
|
||||
Submitting subagent jobs is trusted-only; MCP submitters receive
|
||||
permission_denied. The worker needs ANTHROPIC_API_KEY set, or the
|
||||
first LLM turn of a claimed job fails.
|
||||
`);
|
||||
}
|
||||
|
||||
// ── `gbrain agent run` ────────────────────────────────────
|
||||
|
||||
interface RunFlags {
|
||||
subagentDef?: string;
|
||||
model?: string;
|
||||
maxTurns?: number;
|
||||
tools?: string[];
|
||||
timeoutMs?: number;
|
||||
fanoutManifest?: string;
|
||||
follow: boolean;
|
||||
detach: boolean;
|
||||
}
|
||||
|
||||
function parseRunFlags(args: string[]): { flags: RunFlags; rest: string[] } {
|
||||
const flags: RunFlags = {
|
||||
follow: process.stdout.isTTY === true,
|
||||
detach: false,
|
||||
};
|
||||
let i = 0;
|
||||
while (i < args.length) {
|
||||
const a = args[i];
|
||||
if (a === '--') { i++; break; }
|
||||
if (!isKnownFlag(a!)) break;
|
||||
switch (a) {
|
||||
case '--subagent-def': flags.subagentDef = args[++i]; i++; break;
|
||||
case '--model': flags.model = args[++i]; i++; break;
|
||||
case '--max-turns': flags.maxTurns = parseInt(args[++i] ?? '', 10); i++; break;
|
||||
case '--tools': flags.tools = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean); i++; break;
|
||||
case '--timeout-ms': flags.timeoutMs = parseInt(args[++i] ?? '', 10); i++; break;
|
||||
case '--fanout-manifest': flags.fanoutManifest = args[++i]; i++; break;
|
||||
case '--follow': flags.follow = true; i++; break;
|
||||
case '--no-follow': flags.follow = false; i++; break;
|
||||
case '--detach': flags.detach = true; flags.follow = false; i++; break;
|
||||
default:
|
||||
throw new Error(`unknown flag: ${a}. Run \`gbrain agent run --help\` for usage.`);
|
||||
}
|
||||
}
|
||||
return { flags, rest: args.slice(i) };
|
||||
}
|
||||
|
||||
export async function runAgentRun(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const { flags, rest } = parseRunFlags(args);
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
// Fan-out path: --fanout-manifest supplies explicit child inputs. The
|
||||
// aggregator submits first (so its id is available as parent for each
|
||||
// child); children submit with on_child_fail='continue' so mixed
|
||||
// outcomes don't cascade; aggregator waits in waiting-children until
|
||||
// Lane 1B's terminal-set check unblocks it.
|
||||
if (flags.fanoutManifest) {
|
||||
await runFanout(engine, queue, flags, rest.join(' '));
|
||||
return;
|
||||
}
|
||||
|
||||
const prompt = rest.join(' ').trim();
|
||||
if (!prompt) {
|
||||
console.error('gbrain agent run: prompt is required');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const data: SubagentHandlerData = { prompt };
|
||||
if (flags.subagentDef) data.subagent_def = flags.subagentDef;
|
||||
if (flags.model) data.model = flags.model;
|
||||
if (flags.maxTurns) data.max_turns = flags.maxTurns;
|
||||
if (flags.tools && flags.tools.length > 0) data.allowed_tools = flags.tools;
|
||||
|
||||
const submitOpts: Partial<MinionJobInput> = { max_stalled: 3 };
|
||||
if (flags.timeoutMs) submitOpts.timeout_ms = flags.timeoutMs;
|
||||
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
});
|
||||
|
||||
process.stderr.write(`submitted: job ${job.id} (subagent)\n`);
|
||||
|
||||
if (flags.detach || !flags.follow) {
|
||||
process.stdout.write(String(job.id) + '\n');
|
||||
return;
|
||||
}
|
||||
|
||||
await followJob(engine, queue, job.id, flags.timeoutMs);
|
||||
}
|
||||
|
||||
// ── fan-out ───────────────────────────────────────────────
|
||||
|
||||
async function runFanout(engine: BrainEngine, queue: MinionQueue, flags: RunFlags, promptTemplate: string): Promise<void> {
|
||||
const manifestPath = flags.fanoutManifest!;
|
||||
let manifest: Array<{ prompt?: string; input_vars?: Record<string, unknown> }>;
|
||||
try {
|
||||
const raw = fs.readFileSync(manifestPath, 'utf8');
|
||||
const parsed = JSON.parse(raw);
|
||||
if (!Array.isArray(parsed)) throw new Error('manifest must be a JSON array');
|
||||
manifest = parsed as typeof manifest;
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`gbrain agent run: invalid --fanout-manifest ${manifestPath}: ${msg}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
if (manifest.length === 0) {
|
||||
console.error('gbrain agent run: --fanout-manifest is empty; nothing to run');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// Short-circuit: 1 entry → single subagent, no aggregator.
|
||||
if (manifest.length === 1) {
|
||||
const entry = manifest[0]!;
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: entry.prompt ?? promptTemplate,
|
||||
...(entry.input_vars ? { input_vars: entry.input_vars } : {}),
|
||||
...(flags.subagentDef ? { subagent_def: flags.subagentDef } : {}),
|
||||
...(flags.model ? { model: flags.model } : {}),
|
||||
...(flags.maxTurns ? { max_turns: flags.maxTurns } : {}),
|
||||
...(flags.tools && flags.tools.length > 0 ? { allowed_tools: flags.tools } : {}),
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = { max_stalled: 3 };
|
||||
if (flags.timeoutMs) submitOpts.timeout_ms = flags.timeoutMs;
|
||||
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
});
|
||||
process.stderr.write(`submitted: job ${job.id} (single-entry manifest short-circuit)\n`);
|
||||
if (flags.detach || !flags.follow) { process.stdout.write(`${job.id}\n`); return; }
|
||||
await followJob(engine, queue, job.id, flags.timeoutMs);
|
||||
return;
|
||||
}
|
||||
|
||||
// N-entry fan-out: aggregator first (so we have its id as parent), then
|
||||
// N children, then flip the aggregator's children_ids to include them.
|
||||
const aggregatorSeed: AggregatorHandlerData = { children_ids: [] };
|
||||
const aggregator = await queue.add(
|
||||
'subagent_aggregator',
|
||||
aggregatorSeed as unknown as Record<string, unknown>,
|
||||
{ max_stalled: 3 },
|
||||
{ allowProtectedSubmit: true },
|
||||
);
|
||||
|
||||
const childIds: number[] = [];
|
||||
for (const entry of manifest) {
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: entry.prompt ?? promptTemplate,
|
||||
...(entry.input_vars ? { input_vars: entry.input_vars } : {}),
|
||||
...(flags.subagentDef ? { subagent_def: flags.subagentDef } : {}),
|
||||
...(flags.model ? { model: flags.model } : {}),
|
||||
...(flags.maxTurns ? { max_turns: flags.maxTurns } : {}),
|
||||
...(flags.tools && flags.tools.length > 0 ? { allowed_tools: flags.tools } : {}),
|
||||
};
|
||||
const submitOpts: Partial<MinionJobInput> = {
|
||||
parent_job_id: aggregator.id,
|
||||
on_child_fail: 'continue', // mixed-outcome aggregation
|
||||
max_stalled: 3,
|
||||
};
|
||||
if (flags.timeoutMs) submitOpts.timeout_ms = flags.timeoutMs;
|
||||
const child = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
|
||||
allowProtectedSubmit: true,
|
||||
});
|
||||
childIds.push(child.id);
|
||||
}
|
||||
|
||||
// Update the aggregator's data with the final children_ids. We have to
|
||||
// do this after submission because each add() returns the committed
|
||||
// row's id; the aggregator's seed started with an empty array.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::jsonb) WHERE id = $2`,
|
||||
[JSON.stringify(childIds), aggregator.id],
|
||||
);
|
||||
|
||||
process.stderr.write(
|
||||
`submitted: aggregator job ${aggregator.id} + ${childIds.length} subagent children ` +
|
||||
`(${childIds[0]}..${childIds[childIds.length - 1]})\n`,
|
||||
);
|
||||
|
||||
if (flags.detach || !flags.follow) {
|
||||
process.stdout.write(`${aggregator.id}\n`);
|
||||
return;
|
||||
}
|
||||
await followJob(engine, queue, aggregator.id, flags.timeoutMs);
|
||||
}
|
||||
|
||||
// ── follow ────────────────────────────────────────────────
|
||||
|
||||
async function followJob(engine: BrainEngine, queue: MinionQueue, jobId: number, timeoutMs?: number): Promise<void> {
|
||||
process.stderr.write(`[gbrain agent] following job ${jobId} (Ctrl-C to detach)...\n`);
|
||||
const ac = new AbortController();
|
||||
const onSigint = () => ac.abort();
|
||||
process.once('SIGINT', onSigint);
|
||||
try {
|
||||
// Streaming logs happen in the background; we poll the terminal state
|
||||
// in parallel so the function returns as soon as the job completes.
|
||||
const logsP = runAgentLogs(engine, jobId, { follow: true, signal: ac.signal, pollMs: 1000 });
|
||||
try {
|
||||
const job = await waitForCompletion(queue, jobId, {
|
||||
timeoutMs: timeoutMs ?? 24 * 60 * 60 * 1000,
|
||||
pollMs: 1000,
|
||||
signal: ac.signal,
|
||||
});
|
||||
ac.abort();
|
||||
await logsP.catch(() => {});
|
||||
process.stderr.write(`[gbrain agent] job ${jobId} terminal: ${job.status}\n`);
|
||||
if (job.result != null) process.stdout.write(JSON.stringify(job.result, null, 2) + '\n');
|
||||
if (job.status !== 'completed') process.exit(1);
|
||||
} catch (e) {
|
||||
if (e instanceof TimeoutError) {
|
||||
process.stderr.write(`[gbrain agent] timeout after ${e.elapsedMs}ms — job is still running. Check with: gbrain jobs get ${jobId}\n`);
|
||||
process.exit(3);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
} finally {
|
||||
process.removeListener('SIGINT', onSigint);
|
||||
}
|
||||
}
|
||||
|
||||
// ── `gbrain agent logs` ────────────────────────────────────
|
||||
|
||||
async function runAgentLogsCmd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const jobIdStr = args.find(a => !isKnownFlag(a));
|
||||
if (!jobIdStr) {
|
||||
console.error('gbrain agent logs: <job_id> is required');
|
||||
process.exit(2);
|
||||
}
|
||||
const jobId = parseInt(jobIdStr, 10);
|
||||
if (!Number.isFinite(jobId) || jobId <= 0) {
|
||||
console.error(`gbrain agent logs: "${jobIdStr}" is not a valid job id`);
|
||||
process.exit(2);
|
||||
}
|
||||
const follow = hasFlag(args, '--follow');
|
||||
const since = parseFlag(args, '--since');
|
||||
|
||||
const ac = new AbortController();
|
||||
const onSigint = () => ac.abort();
|
||||
process.once('SIGINT', onSigint);
|
||||
try {
|
||||
await runAgentLogs(engine, jobId, { follow, since, signal: ac.signal });
|
||||
} finally {
|
||||
process.removeListener('SIGINT', onSigint);
|
||||
}
|
||||
}
|
||||
|
||||
// Expose for tests.
|
||||
export const __testing = {
|
||||
parseRunFlags,
|
||||
};
|
||||
+54
-36
@@ -44,35 +44,48 @@ function logError(phase: string, e: unknown) {
|
||||
/**
|
||||
* Resolve the gbrain CLI entrypoint for spawning the worker child.
|
||||
*
|
||||
* Codex caught the bug in earlier plan drafts: `process.execPath` is the
|
||||
* Bun (or Node) runtime binary on source installs, not `gbrain`. Blindly
|
||||
* using it would spawn `bun jobs work`, which does not work.
|
||||
* A .ts source path is never a valid spawn target — spawning it fails with
|
||||
* EACCES because TypeScript source isn't executable. The canonical install
|
||||
* puts a shim at `/usr/local/bin/gbrain` (or wherever `which gbrain`
|
||||
* resolves to) that already wraps the right runtime+entrypoint; prefer it.
|
||||
*
|
||||
* Order of resolution:
|
||||
* 1. argv[1] if it clearly points at a gbrain entry (cli.ts or /gbrain).
|
||||
* 2. process.execPath when running as the compiled binary.
|
||||
* 3. `which gbrain` for installs where the binary is on $PATH.
|
||||
* 4. Throw — nothing on $PATH, no way to supervise the worker.
|
||||
* 1. `which gbrain` — the shim on PATH, canonical for installed builds.
|
||||
* 2. process.execPath if it ends with /gbrain (compiled binary, no shim).
|
||||
* 3. argv[1] if it ends with /gbrain (e.g., direct invocation of compiled
|
||||
* binary without PATH). Never .ts source paths.
|
||||
* 4. Throw with a clear install hint.
|
||||
*/
|
||||
export function resolveGbrainCliPath(): string {
|
||||
const arg1 = process.argv[1] ?? '';
|
||||
if (arg1.endsWith('/gbrain') || arg1.endsWith('/cli.ts') || arg1.endsWith('\\gbrain.exe')) {
|
||||
return arg1;
|
||||
}
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on $PATH — fall through */ }
|
||||
|
||||
const exec = process.execPath ?? '';
|
||||
if (exec.endsWith('/gbrain') || exec.endsWith('\\gbrain.exe')) {
|
||||
return exec;
|
||||
}
|
||||
try {
|
||||
const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
if (which) return which;
|
||||
} catch { /* not on $PATH */ }
|
||||
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH, or run autopilot from the compiled binary directly.');
|
||||
|
||||
const arg1 = process.argv[1] ?? '';
|
||||
if (arg1.endsWith('/gbrain') || arg1.endsWith('\\gbrain.exe')) {
|
||||
return arg1;
|
||||
}
|
||||
|
||||
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
|
||||
}
|
||||
|
||||
export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log('Usage: gbrain autopilot [--repo <path>] [--interval N] [--json]\n gbrain autopilot --install [--repo <path>]\n gbrain autopilot --uninstall\n gbrain autopilot --status [--json]\n\nSelf-maintaining brain daemon. Runs sync + extract + embed + backlinks in a loop.');
|
||||
console.log(
|
||||
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json]\n' +
|
||||
' gbrain autopilot --install [--repo <path>]\n' +
|
||||
' gbrain autopilot --uninstall\n' +
|
||||
' gbrain autopilot --status [--json]\n\n' +
|
||||
'Self-maintaining brain daemon. Runs the full maintenance cycle\n' +
|
||||
'(lint + backlinks + sync + extract + embed + orphans) on an interval.\n\n' +
|
||||
'For a one-shot cron-triggered cycle, see `gbrain dream`.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -228,27 +241,32 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
} catch (e) { logError('dispatch', e); cycleOk = false; }
|
||||
} else {
|
||||
// Inline fallback — same as pre-v0.11.1 behavior.
|
||||
// 1. Sync
|
||||
// Inline fallback — delegate to runCycle so lint + backlinks +
|
||||
// orphan sweep run too (previously this path only did sync +
|
||||
// extract + embed, which didn't match the Minions-dispatch
|
||||
// path's phase set). Now both converge on the same primitive.
|
||||
try {
|
||||
const { performSync } = await import('./sync.ts');
|
||||
const result = await performSync(engine, { repoPath, noEmbed: true });
|
||||
if (result.status === 'synced') {
|
||||
console.log(`[sync] +${result.added} ~${result.modified} -${result.deleted}`);
|
||||
const { runCycle } = await import('../core/cycle.ts');
|
||||
const report = await runCycle(engine, {
|
||||
brainDir: repoPath,
|
||||
// Autopilot daemon path: pulls by default (matches
|
||||
// pre-v0.17 autopilot behavior). CLI dream defaults false
|
||||
// for cron safety; that choice is scoped to dream only.
|
||||
pull: true,
|
||||
yieldBetweenPhases: async () => {
|
||||
await new Promise(r => setImmediate(r));
|
||||
},
|
||||
});
|
||||
if (report.status === 'failed' || report.status === 'partial') {
|
||||
cycleOk = false;
|
||||
}
|
||||
} catch (e) { logError('sync', e); cycleOk = false; }
|
||||
|
||||
// 2. Extract (full brain, incremental dedup handles repeats)
|
||||
try {
|
||||
const { runExtractCore } = await import('./extract.ts');
|
||||
await runExtractCore(engine, { mode: 'all', dir: repoPath });
|
||||
} catch (e) { logError('extract', e); cycleOk = false; }
|
||||
|
||||
// 3. Embed stale
|
||||
try {
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
await runEmbedCore(engine, { stale: true });
|
||||
} catch (e) { logError('embed', e); cycleOk = false; }
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'cycle-inline', status: report.status, duration_ms: report.duration_ms, totals: report.totals }) + '\n');
|
||||
} else {
|
||||
const t = report.totals;
|
||||
console.log(`[cycle-inline ${report.status}] lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found}`);
|
||||
}
|
||||
} catch (e) { logError('cycle-inline', e); cycleOk = false; }
|
||||
}
|
||||
|
||||
// 4. Health check + adaptive interval (same for both paths)
|
||||
|
||||
@@ -13,6 +13,8 @@
|
||||
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative, basename } from 'path';
|
||||
import { extractEntityRefs as canonicalExtractEntityRefs } from '../core/link-extraction.ts';
|
||||
import { createProgress, startHeartbeat } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
interface BacklinkGap {
|
||||
/** The page that mentions the entity */
|
||||
@@ -201,7 +203,18 @@ export async function runBacklinksCore(opts: BacklinksOpts): Promise<BacklinksRe
|
||||
throw new Error(`Directory not found: ${opts.dir}`);
|
||||
}
|
||||
|
||||
const gaps = findBacklinkGaps(opts.dir);
|
||||
// findBacklinkGaps is a sync double-walk of the brain dir. On 50K-page
|
||||
// brains that can take seconds — heartbeat so agents see we're working.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('backlinks.scan');
|
||||
const stopHb = startHeartbeat(progress, 'walking pages for missing back-links…');
|
||||
let gaps: BacklinkGap[];
|
||||
try {
|
||||
gaps = findBacklinkGaps(opts.dir);
|
||||
} finally {
|
||||
stopHb();
|
||||
progress.finish();
|
||||
}
|
||||
const pagesAffected = new Set(gaps.map(g => g.targetPage)).size;
|
||||
|
||||
if (opts.action === 'fix' && gaps.length > 0) {
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
/**
|
||||
* gbrain check-resolvable — Standalone CLI gate for skill-tree integrity.
|
||||
*
|
||||
* Thin wrapper over `src/core/check-resolvable.ts`. Exit-code rule is stricter
|
||||
* than `gbrain doctor`'s resolver_health: this command exits 1 on ANY issue
|
||||
* (errors OR warnings) so CI can gate on a single command. Honors the README
|
||||
* contract: "Exits non-zero if anything is off."
|
||||
*
|
||||
* Currently covers 4 of 6 checks from the original design: reachability,
|
||||
* MECE overlap, MECE gap, DRY violations. Checks 5 (trigger routing eval)
|
||||
* and 6 (brain filing) are tracked as separate GitHub issues and surfaced
|
||||
* via the `deferred` field in --json output.
|
||||
*/
|
||||
|
||||
import { resolve as resolvePath, isAbsolute } from 'path';
|
||||
import {
|
||||
checkResolvable,
|
||||
autoFixDryViolations,
|
||||
type ResolvableReport,
|
||||
type ResolvableIssue,
|
||||
type AutoFixReport,
|
||||
} from '../core/check-resolvable.ts';
|
||||
import { findRepoRoot } from '../core/repo-root.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface DeferredCheck {
|
||||
check: number;
|
||||
name: string;
|
||||
issue: string;
|
||||
}
|
||||
|
||||
export interface Envelope {
|
||||
ok: boolean;
|
||||
skillsDir: string | null;
|
||||
report: ResolvableReport | null;
|
||||
autoFix: AutoFixReport | null;
|
||||
deferred: DeferredCheck[];
|
||||
error: 'no_skills_dir' | null;
|
||||
message: string | null;
|
||||
}
|
||||
|
||||
export interface Flags {
|
||||
help: boolean;
|
||||
json: boolean;
|
||||
fix: boolean;
|
||||
dryRun: boolean;
|
||||
verbose: boolean;
|
||||
skillsDir: string | null;
|
||||
}
|
||||
|
||||
// TBD: fill these issue URLs after filing the GitHub issues pre-PR.
|
||||
// grep for 'TBD-check-5' / 'TBD-check-6' before shipping.
|
||||
export const DEFERRED: DeferredCheck[] = [
|
||||
{
|
||||
check: 5,
|
||||
name: 'trigger_routing_eval',
|
||||
issue: 'https://github.com/garrytan/gbrain/issues?q=TBD-check-5',
|
||||
},
|
||||
{
|
||||
check: 6,
|
||||
name: 'brain_filing',
|
||||
issue: 'https://github.com/garrytan/gbrain/issues?q=TBD-check-6',
|
||||
},
|
||||
];
|
||||
|
||||
const HELP_TEXT = `gbrain check-resolvable [options]
|
||||
|
||||
Validate the skill tree: reachability, MECE overlap, DRY violations, and
|
||||
gap detection. Exits non-zero if any issues are found (errors OR warnings).
|
||||
|
||||
Options:
|
||||
--json Machine-readable JSON (stable envelope)
|
||||
--fix Apply DRY auto-fixes before checking
|
||||
--dry-run With --fix, preview only; no writes
|
||||
--verbose Show passing checks and the deferred-check note
|
||||
--skills-dir PATH Override the auto-detected skills/ directory
|
||||
--help Show this message
|
||||
|
||||
Deferred to separate issues (see --json .deferred[]):
|
||||
- Check 5: trigger routing eval
|
||||
- Check 6: brain filing
|
||||
`;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Flag parsing — permissive on unknown flags, matching lint/orphans/publish.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function parseFlags(argv: string[]): Flags {
|
||||
const flags: Flags = {
|
||||
help: false,
|
||||
json: false,
|
||||
fix: false,
|
||||
dryRun: false,
|
||||
verbose: false,
|
||||
skillsDir: null,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--help' || a === '-h') flags.help = true;
|
||||
else if (a === '--json') flags.json = true;
|
||||
else if (a === '--fix') flags.fix = true;
|
||||
else if (a === '--dry-run') flags.dryRun = true;
|
||||
else if (a === '--verbose') flags.verbose = true;
|
||||
else if (a === '--skills-dir') {
|
||||
flags.skillsDir = argv[i + 1] ?? null;
|
||||
i++;
|
||||
} else if (a?.startsWith('--skills-dir=')) {
|
||||
flags.skillsDir = a.slice('--skills-dir='.length) || null;
|
||||
}
|
||||
// unknown flags silently ignored
|
||||
}
|
||||
return flags;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Skills-dir resolution
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export function resolveSkillsDir(flags: Flags): { dir: string | null; error: Envelope['error']; message: string | null } {
|
||||
if (flags.skillsDir) {
|
||||
const dir = isAbsolute(flags.skillsDir)
|
||||
? flags.skillsDir
|
||||
: resolvePath(process.cwd(), flags.skillsDir);
|
||||
return { dir, error: null, message: null };
|
||||
}
|
||||
const repoRoot = findRepoRoot();
|
||||
if (!repoRoot) {
|
||||
return {
|
||||
dir: null,
|
||||
error: 'no_skills_dir',
|
||||
message:
|
||||
'Could not locate skills/RESOLVER.md from cwd. Pass --skills-dir <path> or run from inside a gbrain repo.',
|
||||
};
|
||||
}
|
||||
return { dir: resolvePath(repoRoot, 'skills'), error: null, message: null };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Human output (mirrors doctor's resolver_health formatting)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function renderHuman(env: Envelope, flags: Flags): void {
|
||||
if (env.error === 'no_skills_dir') {
|
||||
console.error(env.message);
|
||||
return;
|
||||
}
|
||||
const report = env.report!;
|
||||
|
||||
if (flags.fix && env.autoFix) {
|
||||
printAutoFixHuman(env.autoFix, flags.dryRun);
|
||||
}
|
||||
|
||||
if (report.ok && report.issues.length === 0) {
|
||||
console.log(`resolver_health: OK — ${report.summary.total_skills} skills, all reachable`);
|
||||
} else {
|
||||
const errors = report.issues.filter(i => i.severity === 'error');
|
||||
const warnings = report.issues.filter(i => i.severity === 'warning');
|
||||
const status = errors.length > 0 ? 'FAIL' : 'WARN';
|
||||
console.log(
|
||||
`resolver_health: ${status} — ${report.issues.length} issue(s): ${errors.length} error(s), ${warnings.length} warning(s)`,
|
||||
);
|
||||
for (const iss of report.issues) {
|
||||
console.log(formatIssueLine(iss));
|
||||
}
|
||||
}
|
||||
|
||||
if (flags.verbose) {
|
||||
const urls = DEFERRED.map(d => `${d.name} (${d.issue})`).join(', ');
|
||||
console.log(`Deferred: ${urls}`);
|
||||
}
|
||||
}
|
||||
|
||||
function formatIssueLine(iss: ResolvableIssue): string {
|
||||
const type = iss.type.padEnd(18);
|
||||
const skill = iss.skill.padEnd(24);
|
||||
return ` • ${type} ${skill} ${iss.action}`;
|
||||
}
|
||||
|
||||
function printAutoFixHuman(autoFix: AutoFixReport, dryRun: boolean): void {
|
||||
const verb = dryRun ? 'PROPOSED' : 'APPLIED';
|
||||
for (const outcome of autoFix.fixed) {
|
||||
console.log(`[${verb}] ${outcome.skillPath} (${outcome.patternLabel})`);
|
||||
}
|
||||
const n = autoFix.fixed.length;
|
||||
const s = autoFix.skipped.length;
|
||||
if (n === 0 && s === 0) {
|
||||
console.log('check-resolvable --fix: no DRY violations to repair.');
|
||||
return;
|
||||
}
|
||||
const label = dryRun ? 'fixes proposed' : 'fixes applied';
|
||||
console.log(`${n} ${label}${s > 0 ? `, ${s} skipped:` : '.'}`);
|
||||
for (const sk of autoFix.skipped) {
|
||||
const hint = sk.reason === 'working_tree_dirty' ? ' (run `git stash` first)' : '';
|
||||
console.log(` - ${sk.skillPath}: ${sk.reason}${hint}`);
|
||||
}
|
||||
if (dryRun && n > 0) console.log('Run without --dry-run to apply.\n');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Entry point
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export async function runCheckResolvable(args: string[]): Promise<void> {
|
||||
const flags = parseFlags(args);
|
||||
|
||||
if (flags.help) {
|
||||
console.log(HELP_TEXT);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const { dir, error, message } = resolveSkillsDir(flags);
|
||||
|
||||
if (error === 'no_skills_dir') {
|
||||
const env: Envelope = {
|
||||
ok: false,
|
||||
skillsDir: null,
|
||||
report: null,
|
||||
autoFix: null,
|
||||
deferred: DEFERRED,
|
||||
error,
|
||||
message,
|
||||
};
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(env, null, 2));
|
||||
} else {
|
||||
renderHuman(env, flags);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const skillsDir = dir!;
|
||||
|
||||
let autoFix: AutoFixReport | null = null;
|
||||
if (flags.fix) {
|
||||
autoFix = autoFixDryViolations(skillsDir, { dryRun: flags.dryRun });
|
||||
}
|
||||
|
||||
const report = checkResolvable(skillsDir);
|
||||
|
||||
const env: Envelope = {
|
||||
ok: report.issues.length === 0,
|
||||
skillsDir,
|
||||
report,
|
||||
autoFix,
|
||||
deferred: DEFERRED,
|
||||
error: null,
|
||||
message: null,
|
||||
};
|
||||
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify(env, null, 2));
|
||||
} else {
|
||||
renderHuman(env, flags);
|
||||
}
|
||||
|
||||
process.exit(env.ok ? 0 : 1);
|
||||
}
|
||||
+149
-20
@@ -3,7 +3,10 @@ import * as db from '../core/db.ts';
|
||||
import { LATEST_VERSION } from '../core/migrate.ts';
|
||||
import { checkResolvable } from '../core/check-resolvable.ts';
|
||||
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
|
||||
import { findRepoRoot } from '../core/repo-root.ts';
|
||||
import { loadCompletedMigrations } from '../core/preferences.ts';
|
||||
import { createProgress, startHeartbeat, type ProgressReporter } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import type { DbUrlSource } from '../core/config.ts';
|
||||
import { join } from 'path';
|
||||
import { existsSync, readFileSync, readdirSync } from 'fs';
|
||||
@@ -33,6 +36,12 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
const checks: Check[] = [];
|
||||
let autoFixReport: AutoFixReport | null = null;
|
||||
|
||||
// Progress reporter. `--json` is doctor's own JSON output (list of checks);
|
||||
// progress events stay on stderr regardless, gated by the global --quiet /
|
||||
// --progress-json flags. On a 52K-page brain the DB checks can take minutes,
|
||||
// and without a heartbeat agents can't tell doctor from a hang.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// --- Filesystem checks (always run, no DB needed) ---
|
||||
|
||||
// 1. Resolver health
|
||||
@@ -88,7 +97,7 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// status:"complete" for the same version, the install is mid-migration.
|
||||
// Typical cause: v0.11.0 stopgap wrote a partial record but nobody ran
|
||||
// `gbrain apply-migrations --yes` afterward. This check fires on every
|
||||
// `gbrain doctor` invocation so Wintermute's health skill catches it.
|
||||
// `gbrain doctor` invocation so your OpenClaw's health skill catches it.
|
||||
try {
|
||||
const completed = loadCompletedMigrations();
|
||||
const byVersion = new Map<string, { complete: boolean; partial: boolean }>();
|
||||
@@ -196,19 +205,27 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
return;
|
||||
}
|
||||
|
||||
// DB checks phase — start a single reporter phase so agents see which
|
||||
// check is running (several take seconds on 50K-page brains; without a
|
||||
// heartbeat the binary looks hung when stdout is piped).
|
||||
progress.start('doctor.db_checks');
|
||||
|
||||
// 3. Connection
|
||||
progress.heartbeat('connection');
|
||||
try {
|
||||
const stats = await engine.getStats();
|
||||
checks.push({ name: 'connection', status: 'ok', message: `Connected, ${stats.page_count} pages` });
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
checks.push({ name: 'connection', status: 'fail', message: msg });
|
||||
progress.finish();
|
||||
const earlyFail2 = outputResults(checks, jsonOutput);
|
||||
process.exit(earlyFail2 ? 1 : 0);
|
||||
return;
|
||||
}
|
||||
|
||||
// 4. pgvector extension
|
||||
progress.heartbeat('pgvector');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const ext = await sql`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
|
||||
@@ -221,7 +238,46 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
checks.push({ name: 'pgvector', status: 'warn', message: 'Could not check pgvector extension' });
|
||||
}
|
||||
|
||||
// 4b. PgBouncer / prepared-statement compatibility.
|
||||
// URL-only inspection — no DB roundtrip — so this is cheap and works
|
||||
// regardless of whether the caller is the module singleton or a
|
||||
// worker-instance engine.
|
||||
progress.heartbeat('pgbouncer_prepare');
|
||||
try {
|
||||
const { resolvePrepare } = await import('../core/db.ts');
|
||||
const { loadConfig } = await import('../core/config.ts');
|
||||
const config = loadConfig();
|
||||
const url = config?.database_url || '';
|
||||
const prepare = resolvePrepare(url);
|
||||
if (prepare === false) {
|
||||
checks.push({
|
||||
name: 'pgbouncer_prepare',
|
||||
status: 'ok',
|
||||
message: 'Prepared statements disabled (PgBouncer-safe)',
|
||||
});
|
||||
} else {
|
||||
try {
|
||||
const parsed = new URL(url.replace(/^postgres(ql)?:\/\//, 'http://'));
|
||||
if (parsed.port === '6543') {
|
||||
checks.push({
|
||||
name: 'pgbouncer_prepare',
|
||||
status: 'warn',
|
||||
message:
|
||||
'Port 6543 (PgBouncer transaction mode) detected but prepared statements are enabled. ' +
|
||||
'This causes "prepared statement does not exist" errors under concurrent load. ' +
|
||||
'Fix: unset GBRAIN_PREPARE (or set =false), or add ?prepare=false to the connection URL.',
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// URL parse failure — skip, nothing actionable
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// best-effort; never fail doctor on this check
|
||||
}
|
||||
|
||||
// 5. RLS
|
||||
progress.heartbeat('rls');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const tables = await sql`
|
||||
@@ -241,15 +297,31 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
checks.push({ name: 'rls', status: 'warn', message: 'Could not check RLS status' });
|
||||
}
|
||||
|
||||
// 6. Schema version
|
||||
// 6. Schema version — also surfaces the #218 "postinstall silently failed"
|
||||
// state: if schema_version is 0/missing but the DB connected, migrations
|
||||
// never ran. That's the same class as a half-migrated install, just from a
|
||||
// different root cause (Bun blocked our top-level postinstall on global
|
||||
// install). Message is actionable either way.
|
||||
progress.heartbeat('schema_version');
|
||||
let schemaVersion = 0;
|
||||
try {
|
||||
const version = await engine.getConfig('version');
|
||||
schemaVersion = parseInt(version || '0', 10);
|
||||
if (schemaVersion >= LATEST_VERSION) {
|
||||
checks.push({ name: 'schema_version', status: 'ok', message: `Version ${schemaVersion} (latest: ${LATEST_VERSION})` });
|
||||
} else if (schemaVersion === 0) {
|
||||
checks.push({
|
||||
name: 'schema_version',
|
||||
status: 'fail',
|
||||
message: `No schema version recorded. Migrations never ran. Fix: gbrain apply-migrations --yes. ` +
|
||||
`If you installed via 'bun install -g github:...', see https://github.com/garrytan/gbrain/issues/218.`,
|
||||
});
|
||||
} else {
|
||||
checks.push({ name: 'schema_version', status: 'warn', message: `Version ${schemaVersion}, latest is ${LATEST_VERSION}. Run gbrain init to migrate.` });
|
||||
checks.push({
|
||||
name: 'schema_version',
|
||||
status: 'warn',
|
||||
message: `Version ${schemaVersion}, latest is ${LATEST_VERSION}. Fix: gbrain apply-migrations --yes`,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
checks.push({ name: 'schema_version', status: 'warn', message: 'Could not check schema version' });
|
||||
@@ -263,6 +335,7 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// but `apply-migrations` didn't follow up.
|
||||
|
||||
// 7. Embedding health
|
||||
progress.heartbeat('embeddings');
|
||||
try {
|
||||
const health = await engine.getHealth();
|
||||
const pct = (health.embed_coverage * 100).toFixed(0);
|
||||
@@ -278,6 +351,8 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
}
|
||||
|
||||
// 8. Graph health (link + timeline coverage on entity pages).
|
||||
// dead_links removed in v0.10.1: ON DELETE CASCADE on link FKs makes it always 0.
|
||||
progress.heartbeat('graph_coverage');
|
||||
try {
|
||||
const health = await engine.getHealth();
|
||||
const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0);
|
||||
@@ -320,6 +395,8 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// Read-only — no network, no writes, no resolver calls. Samples the first
|
||||
// 500 pages by slug order and surfaces bare-tweet + dead-link counts as a
|
||||
// warning. Full-brain scan: `gbrain integrity check`.
|
||||
progress.heartbeat('integrity_sample');
|
||||
const integrityHb = startHeartbeat(progress, 'scanning 500-page integrity sample…');
|
||||
try {
|
||||
const { scanIntegrity } = await import('./integrity.ts');
|
||||
const res = await scanIntegrity(engine, { limit: 500 });
|
||||
@@ -345,24 +422,31 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
}
|
||||
} catch (e) {
|
||||
checks.push({ name: 'integrity', status: 'warn', message: `integrity scan skipped: ${e instanceof Error ? e.message : String(e)}` });
|
||||
} finally {
|
||||
integrityHb();
|
||||
}
|
||||
|
||||
// 10. JSONB integrity (v0.12.3 reliability wave).
|
||||
// v0.12.0's JSON.stringify()::jsonb pattern stored JSONB string literals
|
||||
// instead of objects on real Postgres. PGLite masked this; Supabase did not.
|
||||
// Scan the 4 known sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated,
|
||||
// files.metadata) for rows whose top-level jsonb_typeof is 'string'.
|
||||
// Scan 5 known write sites for rows whose top-level jsonb_typeof is
|
||||
// 'string'. `page_versions.frontmatter` added in v0.15.2 so doctor's
|
||||
// surface matches `repair-jsonb` (the previous 4-target scan missed a
|
||||
// repair target, per #254/Codex review).
|
||||
progress.heartbeat('jsonb_integrity');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const targets: Array<{ table: string; col: string; expected: 'object' | 'array' }> = [
|
||||
{ table: 'pages', col: 'frontmatter', expected: 'object' },
|
||||
{ table: 'raw_data', col: 'data', expected: 'object' },
|
||||
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
|
||||
{ table: 'files', col: 'metadata', expected: 'object' },
|
||||
{ table: 'pages', col: 'frontmatter', expected: 'object' },
|
||||
{ table: 'raw_data', col: 'data', expected: 'object' },
|
||||
{ table: 'ingest_log', col: 'pages_updated', expected: 'array' },
|
||||
{ table: 'files', col: 'metadata', expected: 'object' },
|
||||
{ table: 'page_versions', col: 'frontmatter', expected: 'object' },
|
||||
];
|
||||
let totalBad = 0;
|
||||
const breakdown: string[] = [];
|
||||
for (const { table, col } of targets) {
|
||||
progress.heartbeat(`jsonb_integrity.${table}.${col}`);
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM ${table} WHERE jsonb_typeof(${col}) = 'string'`,
|
||||
);
|
||||
@@ -386,6 +470,12 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// v0.12.0's splitBody ate everything after the first `---` horizontal rule,
|
||||
// truncating wiki-style pages. Heuristic: pages whose body is <30% of the
|
||||
// raw source content length when raw has multiple H2/H3 boundaries.
|
||||
//
|
||||
// No total on this check: the regex scan over rd.data -> 'content' is a
|
||||
// sequential scan that LIMIT 100 bounds only the output, not the scan
|
||||
// work. We heartbeat every second so agents see life, no fake totals.
|
||||
progress.heartbeat('markdown_body_completeness');
|
||||
const mbcHb = startHeartbeat(progress, 'scanning pages for truncation…');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const rows = await sql`
|
||||
@@ -413,8 +503,58 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
} catch {
|
||||
// pages_raw.raw_data may not exist on older schemas; best-effort.
|
||||
checks.push({ name: 'markdown_body_completeness', status: 'ok', message: 'Skipped (raw_data unavailable)' });
|
||||
} finally {
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 12. Index audit (opt-in via --index-audit). v0.13.1 follow-up to #170.
|
||||
// Reports indexes with zero recorded scans on Postgres. Informational only;
|
||||
// we DO NOT auto-drop. On #170's brain, idx_pages_frontmatter and
|
||||
// idx_pages_trgm showed 0 scans — the suggestion there is "consider
|
||||
// investigating on YOUR brain," not "drop these globally." Zero scans on a
|
||||
// fresh install is also normal (nothing has queried yet); the real signal
|
||||
// is zero scans on a long-running active brain.
|
||||
if (args.includes('--index-audit')) {
|
||||
progress.heartbeat('index_audit');
|
||||
if (engine.kind === 'pglite') {
|
||||
checks.push({
|
||||
name: 'index_audit',
|
||||
status: 'ok',
|
||||
message: 'Skipped (PGLite — pg_stat_user_indexes is a Postgres extension)',
|
||||
});
|
||||
} else {
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const rows = await sql`
|
||||
SELECT schemaname, relname AS table, indexrelname AS index,
|
||||
idx_scan, pg_size_pretty(pg_relation_size(indexrelid)) AS size
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE schemaname = 'public'
|
||||
AND idx_scan = 0
|
||||
ORDER BY pg_relation_size(indexrelid) DESC
|
||||
LIMIT 20
|
||||
`;
|
||||
if (rows.length === 0) {
|
||||
checks.push({ name: 'index_audit', status: 'ok', message: 'All public indexes have recorded scans' });
|
||||
} else {
|
||||
const list = rows.map((r: any) => `${r.index}(${r.size})`).join(', ');
|
||||
checks.push({
|
||||
name: 'index_audit',
|
||||
status: 'warn',
|
||||
message: `${rows.length} zero-scan index(es): ${list}. ` +
|
||||
`Consider investigating whether they're used on YOUR workload (fresh brains naturally show zero scans until queries accumulate). ` +
|
||||
`Do not drop without confirming.`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
checks.push({ name: 'index_audit', status: 'warn', message: `Index audit failed: ${msg}` });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
const hasFail = outputResults(checks, jsonOutput);
|
||||
|
||||
// Features teaser (non-JSON, non-failing only)
|
||||
@@ -463,17 +603,6 @@ function printAutoFixReport(report: AutoFixReport, dryRun: boolean, jsonOutput:
|
||||
if (dryRun && n > 0) console.log('\nRun without --dry-run to apply.');
|
||||
}
|
||||
|
||||
/** Find the GBrain repo root by walking up from cwd looking for skills/RESOLVER.md */
|
||||
function findRepoRoot(): string | null {
|
||||
let dir = process.cwd();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
if (existsSync(join(dir, 'skills', 'RESOLVER.md'))) return dir;
|
||||
const parent = join(dir, '..');
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Quick skill conformance check — frontmatter + required sections */
|
||||
function checkSkillConformance(skillsDir: string): Check {
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
/**
|
||||
* gbrain dream — run one brain maintenance cycle.
|
||||
*
|
||||
* The README brand promise: "the agent runs while I sleep, the dream
|
||||
* cycle ... I wake up and the brain is smarter." Cron-friendly, JSON
|
||||
* report, phase-selectable.
|
||||
*
|
||||
* Thin alias over runCycle (src/core/cycle.ts). Both this command and
|
||||
* `gbrain autopilot` converge on the same primitive so there's one
|
||||
* source of truth for what "overnight maintenance" means.
|
||||
*
|
||||
* Usage:
|
||||
* gbrain dream # full 6-phase cycle
|
||||
* gbrain dream --dry-run # preview, no writes
|
||||
* gbrain dream --json # CycleReport JSON (for agents)
|
||||
* gbrain dream --phase lint # run a single phase
|
||||
* gbrain dream --pull # also git pull the brain repo
|
||||
* gbrain dream --dir /path/to/brain # explicit brain location
|
||||
*
|
||||
* Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log
|
||||
*
|
||||
* Related: `gbrain autopilot --install` for continuous daemonized
|
||||
* maintenance. dream is the one-shot, autopilot is the scheduler.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
runCycle,
|
||||
ALL_PHASES,
|
||||
type CyclePhase,
|
||||
type CycleReport,
|
||||
} from '../core/cycle.ts';
|
||||
import { existsSync } from 'fs';
|
||||
|
||||
interface DreamArgs {
|
||||
json: boolean;
|
||||
dryRun: boolean;
|
||||
pull: boolean;
|
||||
phase: CyclePhase | null;
|
||||
dir: string | null;
|
||||
help: boolean;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): DreamArgs {
|
||||
const phaseIdx = args.indexOf('--phase');
|
||||
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
|
||||
const phase = rawPhase && (ALL_PHASES as string[]).includes(rawPhase)
|
||||
? (rawPhase as CyclePhase)
|
||||
: null;
|
||||
if (rawPhase && !phase) {
|
||||
console.error(`Unknown phase "${rawPhase}". Valid: ${ALL_PHASES.join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
const dir = dirIdx !== -1 ? args[dirIdx + 1] : null;
|
||||
|
||||
return {
|
||||
json: args.includes('--json'),
|
||||
dryRun: args.includes('--dry-run'),
|
||||
pull: args.includes('--pull'),
|
||||
phase,
|
||||
dir,
|
||||
help: args.includes('--help') || args.includes('-h'),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the brain directory without the `findRepoRoot` footgun.
|
||||
*
|
||||
* Prior dream.ts walked up 10 levels of cwd looking for `.git` and would
|
||||
* happily run lint + sync against an unrelated git repo the user happened
|
||||
* to be cd'd into. This resolver only trusts two sources:
|
||||
* 1. An explicit --dir argument.
|
||||
* 2. The `sync.repo_path` config key set by `gbrain init` (engine-backed).
|
||||
*
|
||||
* If neither is available, we error out instead of guessing.
|
||||
*/
|
||||
async function resolveBrainDir(
|
||||
engine: BrainEngine | null,
|
||||
explicit: string | null,
|
||||
): Promise<string> {
|
||||
if (explicit) {
|
||||
if (!existsSync(explicit)) {
|
||||
console.error(`--dir path does not exist: ${explicit}`);
|
||||
process.exit(1);
|
||||
}
|
||||
return explicit;
|
||||
}
|
||||
|
||||
if (engine) {
|
||||
const configured = await engine.getConfig('sync.repo_path');
|
||||
if (configured && existsSync(configured)) {
|
||||
return configured;
|
||||
}
|
||||
}
|
||||
|
||||
console.error(
|
||||
'No brain directory found. Pass --dir <path> or configure one via `gbrain init`.',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`Usage: gbrain dream [options]
|
||||
|
||||
Run one brain maintenance cycle: lint, backlinks, orphan sweep, sync,
|
||||
extract, and embed. Designed for cron (exits when done).
|
||||
|
||||
Options:
|
||||
--dry-run Preview all fixes without writing (fs or DB)
|
||||
--json Emit the CycleReport as JSON (agent-readable)
|
||||
--phase <name> Run a single phase: ${ALL_PHASES.join(' | ')}
|
||||
--pull git pull the brain repo before syncing (default: no pull)
|
||||
--dir <path> Brain directory (default: configured brain)
|
||||
--help, -h Show this help
|
||||
|
||||
Examples:
|
||||
gbrain dream
|
||||
gbrain dream --dry-run --json
|
||||
gbrain dream --phase lint
|
||||
0 2 * * * gbrain dream --json # nightly via cron
|
||||
|
||||
Related:
|
||||
gbrain autopilot --install # continuous maintenance as a daemon
|
||||
gbrain autopilot # same maintenance cycle, scheduled
|
||||
`);
|
||||
}
|
||||
|
||||
// ─── Human-friendly report printing ────────────────────────────────
|
||||
|
||||
function printHuman(report: CycleReport) {
|
||||
if (report.status === 'skipped') {
|
||||
if (report.reason === 'cycle_already_running') {
|
||||
console.log(`Skipped: another cycle is already running. (locked)`);
|
||||
} else if (report.reason === 'no_database') {
|
||||
console.log(`Skipped: no database available.`);
|
||||
} else {
|
||||
console.log(`Skipped: ${report.reason ?? 'unknown reason'}.`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (report.status === 'clean') {
|
||||
console.log(
|
||||
`Brain is healthy. ${report.phases.length} phase(s) checked in ${(report.duration_ms / 1000).toFixed(1)}s.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Dream cycle (${report.status}) in ${(report.duration_ms / 1000).toFixed(1)}s:`);
|
||||
for (const p of report.phases) {
|
||||
const icon =
|
||||
p.status === 'ok' ? '✓' :
|
||||
p.status === 'warn' ? '!' :
|
||||
p.status === 'skipped' ? '-' : '✗';
|
||||
const line = ` ${icon} ${p.phase.padEnd(10)} ${p.summary}`;
|
||||
console.log(line);
|
||||
if (p.error) {
|
||||
const hint = p.error.hint ? ` (${p.error.hint})` : '';
|
||||
console.log(` [${p.error.class}/${p.error.code}] ${p.error.message}${hint}`);
|
||||
}
|
||||
}
|
||||
|
||||
const t = report.totals;
|
||||
const hasTotals =
|
||||
t.lint_fixes > 0 || t.backlinks_added > 0 || t.pages_synced > 0 ||
|
||||
t.pages_extracted > 0 || t.pages_embedded > 0 || t.orphans_found > 0;
|
||||
if (hasTotals) {
|
||||
console.log(
|
||||
` totals: lint=${t.lint_fixes} backlinks=${t.backlinks_added} synced=${t.pages_synced} extracted=${t.pages_extracted} embedded=${t.pages_embedded} orphans=${t.orphans_found}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── CLI entry ─────────────────────────────────────────────────────
|
||||
|
||||
export async function runDream(engine: BrainEngine | null, args: string[]): Promise<CycleReport | void> {
|
||||
const opts = parseArgs(args);
|
||||
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const brainDir = await resolveBrainDir(engine, opts.dir);
|
||||
const phases: CyclePhase[] | undefined = opts.phase ? [opts.phase] : undefined;
|
||||
|
||||
const report = await runCycle(engine, {
|
||||
brainDir,
|
||||
dryRun: opts.dryRun,
|
||||
pull: opts.pull,
|
||||
phases,
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
printHuman(report);
|
||||
}
|
||||
|
||||
// Exit non-zero when the cycle failed overall (helps cron spot real problems).
|
||||
// 'partial' is not a failure — it means some phase warned but the cycle ran.
|
||||
if (report.status === 'failed') {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return report;
|
||||
}
|
||||
+141
-24
@@ -2,6 +2,8 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { embedBatch } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { createProgress, type ProgressReporter } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface EmbedOpts {
|
||||
/** Embed ALL pages (every chunk). */
|
||||
@@ -12,70 +14,143 @@ export interface EmbedOpts {
|
||||
slugs?: string[];
|
||||
/** Embed a single page. */
|
||||
slug?: string;
|
||||
/**
|
||||
* Dry run: enumerate what WOULD be embedded (stale chunk counts)
|
||||
* without calling the embedding model or writing to the engine.
|
||||
* Safe to call with no API key. Used by runCycle's dryRun propagation.
|
||||
*/
|
||||
dryRun?: boolean;
|
||||
/**
|
||||
* Optional progress callback. Called after each page. CLI wrappers
|
||||
* supply a reporter.tick()-backed implementation; Minion handlers
|
||||
* supply a job.updateProgress()-backed one so per-job progress lives
|
||||
* in the DB where `gbrain jobs get` can read it.
|
||||
*/
|
||||
onProgress?: (done: number, total: number, embedded: number) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Structured result from a library-level embed run.
|
||||
*
|
||||
* In dryRun mode, `embedded = 0` and `would_embed` holds the count of
|
||||
* stale chunks that WOULD have been sent to the embedding model. In
|
||||
* non-dryRun mode, `embedded` holds the real count and `would_embed = 0`.
|
||||
* `skipped` counts chunks that already had embeddings (nothing to do).
|
||||
*/
|
||||
export interface EmbedResult {
|
||||
/** Chunks newly embedded in this run (0 in dryRun). */
|
||||
embedded: number;
|
||||
/** Chunks with pre-existing embeddings, skipped. */
|
||||
skipped: number;
|
||||
/** Chunks that would be embedded if not for dryRun (0 in non-dryRun). */
|
||||
would_embed: number;
|
||||
/** Total chunks considered across all processed pages. */
|
||||
total_chunks: number;
|
||||
/** Number of pages processed (whether or not they had stale chunks). */
|
||||
pages_processed: number;
|
||||
/** True if this run was a dry-run. */
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Library-level embed. Throws on validation errors; per-page embed failures
|
||||
* are logged to stderr but do not throw (matches the existing CLI semantics
|
||||
* for batch runs). Safe to call from Minions handlers — no process.exit.
|
||||
*
|
||||
* Returns EmbedResult with accurate counts so callers (runCycle, sync
|
||||
* auto-embed step) can report embeddings in their own structured output.
|
||||
*/
|
||||
export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promise<void> {
|
||||
export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promise<EmbedResult> {
|
||||
const result: EmbedResult = {
|
||||
embedded: 0,
|
||||
skipped: 0,
|
||||
would_embed: 0,
|
||||
total_chunks: 0,
|
||||
pages_processed: 0,
|
||||
dryRun: !!opts.dryRun,
|
||||
};
|
||||
|
||||
if (opts.slugs && opts.slugs.length > 0) {
|
||||
for (const s of opts.slugs) {
|
||||
try { await embedPage(engine, s); } catch (e: unknown) {
|
||||
try {
|
||||
await embedPage(engine, s, !!opts.dryRun, result);
|
||||
} catch (e: unknown) {
|
||||
console.error(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
}
|
||||
return;
|
||||
return result;
|
||||
}
|
||||
if (opts.all || opts.stale) {
|
||||
await embedAll(engine, !!opts.stale);
|
||||
return;
|
||||
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress);
|
||||
return result;
|
||||
}
|
||||
if (opts.slug) {
|
||||
await embedPage(engine, opts.slug);
|
||||
return;
|
||||
await embedPage(engine, opts.slug, !!opts.dryRun, result);
|
||||
return result;
|
||||
}
|
||||
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
|
||||
}
|
||||
|
||||
export async function runEmbed(engine: BrainEngine, args: string[]) {
|
||||
export async function runEmbed(engine: BrainEngine, args: string[]): Promise<EmbedResult | undefined> {
|
||||
const slugsIdx = args.indexOf('--slugs');
|
||||
const all = args.includes('--all');
|
||||
const stale = args.includes('--stale');
|
||||
const dryRun = args.includes('--dry-run');
|
||||
|
||||
let opts: EmbedOpts;
|
||||
if (slugsIdx >= 0) {
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')) };
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun };
|
||||
} else if (all || stale) {
|
||||
opts = { all, stale };
|
||||
opts = { all, stale, dryRun };
|
||||
} else {
|
||||
const slug = args.find(a => !a.startsWith('--'));
|
||||
if (!slug) {
|
||||
console.error('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...]');
|
||||
console.error('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
|
||||
process.exit(1);
|
||||
}
|
||||
opts = { slug };
|
||||
opts = { slug, dryRun };
|
||||
}
|
||||
|
||||
// CLI path: wire a reporter so --progress-json / --quiet / TTY rendering
|
||||
// all work. Minion handlers call runEmbedCore directly with their own
|
||||
// onProgress (see jobs.ts).
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
let progressStarted = false;
|
||||
opts.onProgress = (done, total, _embedded) => {
|
||||
if (!progressStarted) {
|
||||
progress.start('embed.pages', total);
|
||||
progressStarted = true;
|
||||
}
|
||||
progress.tick(1);
|
||||
};
|
||||
|
||||
try {
|
||||
await runEmbedCore(engine, opts);
|
||||
const result = await runEmbedCore(engine, opts);
|
||||
if (progressStarted) progress.finish();
|
||||
return result;
|
||||
} catch (e) {
|
||||
if (progressStarted) progress.finish();
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function embedPage(engine: BrainEngine, slug: string) {
|
||||
async function embedPage(
|
||||
engine: BrainEngine,
|
||||
slug: string,
|
||||
dryRun: boolean,
|
||||
result: EmbedResult,
|
||||
) {
|
||||
const page = await engine.getPage(slug);
|
||||
if (!page) {
|
||||
throw new Error(`Page not found: ${slug}`);
|
||||
}
|
||||
|
||||
// Get existing chunks or create new ones
|
||||
// Get existing chunks or create new ones.
|
||||
// In dryRun, we still chunk the text locally to count what WOULD be
|
||||
// embedded — but we never write chunks or call the embedding model.
|
||||
let chunks = await engine.getChunks(slug);
|
||||
if (chunks.length === 0) {
|
||||
// Create chunks first
|
||||
const inputs: ChunkInput[] = [];
|
||||
if (page.compiled_truth.trim()) {
|
||||
for (const c of chunkText(page.compiled_truth)) {
|
||||
@@ -87,6 +162,15 @@ async function embedPage(engine: BrainEngine, slug: string) {
|
||||
inputs.push({ chunk_index: inputs.length, chunk_text: c.text, chunk_source: 'timeline' });
|
||||
}
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
// Count what chunking WOULD produce, without writing.
|
||||
result.total_chunks += inputs.length;
|
||||
result.would_embed += inputs.length;
|
||||
result.pages_processed++;
|
||||
return;
|
||||
}
|
||||
|
||||
if (inputs.length > 0) {
|
||||
await engine.upsertChunks(slug, inputs);
|
||||
chunks = await engine.getChunks(slug);
|
||||
@@ -95,8 +179,18 @@ async function embedPage(engine: BrainEngine, slug: string) {
|
||||
|
||||
// Embed chunks without embeddings
|
||||
const toEmbed = chunks.filter(c => !c.embedded_at);
|
||||
result.total_chunks += chunks.length;
|
||||
result.skipped += chunks.length - toEmbed.length;
|
||||
|
||||
if (toEmbed.length === 0) {
|
||||
console.log(`${slug}: all ${chunks.length} chunks already embedded`);
|
||||
result.pages_processed++;
|
||||
return;
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
result.would_embed += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -114,13 +208,19 @@ async function embedPage(engine: BrainEngine, slug: string) {
|
||||
}));
|
||||
|
||||
await engine.upsertChunks(slug, updated);
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
console.log(`${slug}: embedded ${toEmbed.length} chunks`);
|
||||
}
|
||||
|
||||
async function embedAll(engine: BrainEngine, staleOnly: boolean) {
|
||||
async function embedAll(
|
||||
engine: BrainEngine,
|
||||
staleOnly: boolean,
|
||||
dryRun: boolean,
|
||||
result: EmbedResult,
|
||||
onProgress?: (done: number, total: number, embedded: number) => void,
|
||||
) {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
let total = 0;
|
||||
let embedded = 0;
|
||||
let processed = 0;
|
||||
|
||||
// Concurrency limit for parallel page embedding.
|
||||
@@ -139,9 +239,21 @@ async function embedAll(engine: BrainEngine, staleOnly: boolean) {
|
||||
? chunks.filter(c => !c.embedded_at)
|
||||
: chunks;
|
||||
|
||||
result.total_chunks += chunks.length;
|
||||
result.skipped += chunks.length - toEmbed.length;
|
||||
|
||||
if (toEmbed.length === 0) {
|
||||
processed++;
|
||||
process.stdout.write(`\r ${processed}/${pages.length} pages, ${embedded} chunks embedded`);
|
||||
result.pages_processed++;
|
||||
onProgress?.(processed, pages.length, result.embedded);
|
||||
return;
|
||||
}
|
||||
|
||||
if (dryRun) {
|
||||
result.would_embed += toEmbed.length;
|
||||
processed++;
|
||||
result.pages_processed++;
|
||||
onProgress?.(processed, pages.length, result.embedded);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -161,14 +273,14 @@ async function embedAll(engine: BrainEngine, staleOnly: boolean) {
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(page.slug, updated);
|
||||
embedded += toEmbed.length;
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
console.error(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
|
||||
total += toEmbed.length;
|
||||
processed++;
|
||||
process.stdout.write(`\r ${processed}/${pages.length} pages, ${embedded} chunks embedded`);
|
||||
result.pages_processed++;
|
||||
onProgress?.(processed, pages.length, result.embedded);
|
||||
}
|
||||
|
||||
// Sliding worker pool: N workers share a queue and each pulls the
|
||||
@@ -187,5 +299,10 @@ async function embedAll(engine: BrainEngine, staleOnly: boolean) {
|
||||
const numWorkers = Math.min(CONCURRENCY, pages.length);
|
||||
await Promise.all(Array.from({ length: numWorkers }, () => worker()));
|
||||
|
||||
console.log(`\n\nEmbedded ${embedded} chunks across ${pages.length} pages`);
|
||||
// Stdout summary preserved for scripts/tests that grep for counts.
|
||||
if (dryRun) {
|
||||
console.log(`[dry-run] Would embed ${result.would_embed} chunks across ${pages.length} pages`);
|
||||
} else {
|
||||
console.log(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
|
||||
}
|
||||
}
|
||||
|
||||
+14
-3
@@ -50,17 +50,28 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
|
||||
const k = opts.k ?? 5;
|
||||
const configA = buildConfig(opts, 'a');
|
||||
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
if (opts.configB || opts.configBPath) {
|
||||
// A/B comparison mode
|
||||
const configB = buildConfig(opts, 'b');
|
||||
progress.start('eval.ab', qrels.length * 2);
|
||||
const onProgress = (_done: number, _total: number, q: string) => progress.tick(1, q);
|
||||
const [reportA, reportB] = await Promise.all([
|
||||
runEval(engine, qrels, configA, k),
|
||||
runEval(engine, qrels, configB, k),
|
||||
runEval(engine, qrels, configA, k, { onProgress }),
|
||||
runEval(engine, qrels, configB, k, { onProgress }),
|
||||
]);
|
||||
progress.finish();
|
||||
printABTable(reportA, reportB, k);
|
||||
} else {
|
||||
// Single-run mode
|
||||
const report = await runEval(engine, qrels, configA, k);
|
||||
progress.start('eval.single', qrels.length);
|
||||
const report = await runEval(engine, qrels, configA, k, {
|
||||
onProgress: (_done, _total, q) => progress.tick(1, q),
|
||||
});
|
||||
progress.finish();
|
||||
printSingleTable(report);
|
||||
}
|
||||
}
|
||||
|
||||
+10
-4
@@ -2,6 +2,8 @@ import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
@@ -10,6 +12,10 @@ export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
|
||||
// Progress on stderr so stdout stays clean for scripts parsing counts.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('export.pages', pages.length);
|
||||
|
||||
let exported = 0;
|
||||
|
||||
for (const page of pages) {
|
||||
@@ -41,10 +47,10 @@ export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
|
||||
exported++;
|
||||
if (exported % 100 === 0) {
|
||||
process.stdout.write(`\r ${exported}/${pages.length} exported`);
|
||||
}
|
||||
progress.tick();
|
||||
}
|
||||
|
||||
console.log(`\nExported ${exported} pages to ${outDir}/`);
|
||||
progress.finish();
|
||||
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
}
|
||||
|
||||
+25
-12
@@ -26,6 +26,8 @@ import {
|
||||
extractFrontmatterLinks,
|
||||
type UnresolvedFrontmatterRef,
|
||||
} from '../core/link-extraction.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
@@ -415,6 +417,12 @@ async function extractLinksFromDir(
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(files.map(f => f.relPath.replace('.md', '')));
|
||||
|
||||
// Progress stream on stderr (separate from the action-events --json writes
|
||||
// to stdout, which tests grep for). Rate-gated; respects global --quiet /
|
||||
// --progress-json flags.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.links_fs', files.length);
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
// Without this, the same link extracted from N files would print N times in --dry-run.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
@@ -454,11 +462,10 @@ async function extractLinksFromDir(
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
if (jsonMode && !dryRun && (i % 100 === 0 || i === files.length - 1)) {
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
await flush();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -472,6 +479,9 @@ async function extractTimelineFromDir(
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.timeline_fs', files.length);
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
@@ -510,11 +520,10 @@ async function extractTimelineFromDir(
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
if (jsonMode && !dryRun && (i % 100 === 0 || i === files.length - 1)) {
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
await flush();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -586,6 +595,9 @@ async function extractLinksFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.links_db', slugList.length);
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
@@ -661,11 +673,10 @@ async function extractLinksFromDB(
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
if (jsonMode && !dryRun && (processed % 500 === 0 || i === slugList.length - 1)) {
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
await flush();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -699,6 +710,9 @@ async function extractTimelineFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.timeline_db', slugList.length);
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
@@ -753,11 +767,10 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
if (jsonMode && !dryRun && (processed % 500 === 0 || i === slugList.length - 1)) {
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
await flush();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
|
||||
+10
-5
@@ -4,6 +4,8 @@ import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { humanSize } from '../core/file-resolver.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
/** Size threshold: files >= 100 MB use TUS resumable upload */
|
||||
const SIZE_THRESHOLD = 100 * 1024 * 1024;
|
||||
@@ -306,13 +308,14 @@ async function syncFiles(dir?: string) {
|
||||
let uploaded = 0;
|
||||
let skipped = 0;
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('files.sync', files.length);
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const filePath = files[i];
|
||||
const relativePath = relative(dir, filePath);
|
||||
|
||||
if ((i + 1) % 50 === 0 || i === files.length - 1) {
|
||||
process.stdout.write(`\r ${i + 1}/${files.length} processed, ${uploaded} uploaded, ${skipped} skipped`);
|
||||
}
|
||||
progress.tick(1);
|
||||
|
||||
const hash = fileHash(filePath);
|
||||
const filename = basename(filePath);
|
||||
@@ -343,7 +346,9 @@ async function syncFiles(dir?: string) {
|
||||
uploaded++;
|
||||
}
|
||||
|
||||
console.log(`\n\nFiles sync complete: ${uploaded} uploaded, ${skipped} skipped (unchanged)`);
|
||||
progress.finish();
|
||||
// Stdout summary preserved for scripts/tests that grep for it.
|
||||
console.log(`Files sync complete: ${uploaded} uploaded, ${skipped} skipped (unchanged)`);
|
||||
}
|
||||
|
||||
async function verifyFiles() {
|
||||
@@ -416,7 +421,7 @@ async function mirrorFiles(args: string[]) {
|
||||
// Write .supabase marker
|
||||
const marker = stringify({
|
||||
synced_at: new Date().toISOString(),
|
||||
bucket: config.storage.bucket || 'brain-files',
|
||||
bucket: (config.storage as { bucket?: string })?.bucket || 'brain-files',
|
||||
prefix: basename(dir) + '/',
|
||||
file_count: uploaded,
|
||||
});
|
||||
|
||||
+14
-9
@@ -5,6 +5,8 @@ import { cpus, totalmem, homedir } from 'os';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
function defaultWorkers(): number {
|
||||
const cpuCount = cpus().length;
|
||||
@@ -36,12 +38,13 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
// Find dir: first non-flag arg that isn't a value for --workers
|
||||
const flagValues = new Set<number>();
|
||||
if (workersIdx !== -1) flagValues.add(workersIdx + 1);
|
||||
const dir = args.find((a, i) => !a.startsWith('--') && !flagValues.has(i));
|
||||
const dirArg = args.find((a, i) => !a.startsWith('--') && !flagValues.has(i));
|
||||
|
||||
if (!dir) {
|
||||
if (!dirArg) {
|
||||
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--json]');
|
||||
process.exit(1);
|
||||
}
|
||||
const dir: string = dirArg; // narrowed; survives closure capture
|
||||
|
||||
// Collect all .md files
|
||||
const allFiles = collectMarkdownFiles(dir);
|
||||
@@ -81,12 +84,12 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
const failures: Array<{ path: string; error: string }> = []; // Bug 9
|
||||
const startTime = Date.now();
|
||||
|
||||
function logProgress() {
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
const rate = elapsed > 0 ? Math.round(processed / elapsed) : 0;
|
||||
const remaining = rate > 0 ? Math.round((files.length - processed) / rate) : 0;
|
||||
const pct = Math.round((processed / files.length) * 100);
|
||||
console.log(`[gbrain import] ${processed}/${files.length} (${pct}%) | ${rate} files/sec | imported: ${imported} | skipped: ${skipped} | errors: ${errors} | ETA: ${remaining}s`);
|
||||
// Progress on stderr so stdout stays clean for the final summary / --json payload.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('import.files', files.length);
|
||||
|
||||
function tickProgress() {
|
||||
progress.tick(1, `imported=${imported} skipped=${skipped} errors=${errors}`);
|
||||
}
|
||||
|
||||
async function processFile(eng: BrainEngine, filePath: string) {
|
||||
@@ -119,8 +122,8 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
failures.push({ path: relativePath, error: msg });
|
||||
}
|
||||
processed++;
|
||||
tickProgress();
|
||||
if (processed % 100 === 0 || processed === files.length) {
|
||||
logProgress();
|
||||
// Save checkpoint every 100 files — track completed file set, not just a counter
|
||||
if (processed % 100 === 0) {
|
||||
try {
|
||||
@@ -180,6 +183,8 @@ export async function runImport(engine: BrainEngine, args: string[], opts: { com
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
// Error summary
|
||||
for (const [err, count] of Object.entries(errorCounts)) {
|
||||
if (count > 5) {
|
||||
|
||||
@@ -361,8 +361,14 @@ async function cmdAuto(args: string[]): Promise<void> {
|
||||
let bucketErr = 0;
|
||||
let pagesProcessed = 0;
|
||||
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
try {
|
||||
const allSlugs = [...(await engine.getAllSlugs())].sort();
|
||||
const toScan = allSlugs.filter(s => !seen.has(s));
|
||||
progress.start('integrity.auto', toScan.length);
|
||||
for (const slug of allSlugs) {
|
||||
if (pagesProcessed >= limit) break;
|
||||
if (seen.has(slug)) continue;
|
||||
@@ -371,6 +377,7 @@ async function cmdAuto(args: string[]): Promise<void> {
|
||||
if (!page) continue;
|
||||
|
||||
pagesProcessed++;
|
||||
progress.tick(1, slug);
|
||||
|
||||
// Bare-tweet handling
|
||||
if (!skipTweet) {
|
||||
@@ -456,6 +463,8 @@ async function cmdAuto(args: string[]): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
// Summary
|
||||
console.log('');
|
||||
console.log(`=== integrity auto summary${dryRun ? ' (DRY RUN)' : ''} ===`);
|
||||
|
||||
+155
-63
@@ -57,8 +57,10 @@ export async function runJobs(engine: BrainEngine, args: string[]): Promise<void
|
||||
|
||||
USAGE
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
|
||||
[--delay Nms] [--timeout-ms Nms] [--max-attempts N]
|
||||
[--queue Q] [--dry-run]
|
||||
[--delay Nms] [--max-attempts N] [--max-stalled N]
|
||||
[--backoff-type fixed|exponential] [--backoff-delay Nms]
|
||||
[--backoff-jitter 0..1] [--timeout-ms Nms]
|
||||
[--idempotency-key K] [--queue Q] [--dry-run]
|
||||
gbrain jobs list [--status S] [--queue Q] [--limit N]
|
||||
gbrain jobs get <id>
|
||||
gbrain jobs cancel <id>
|
||||
@@ -104,13 +106,26 @@ HANDLER TYPES (built in)
|
||||
const priority = parseInt(parseFlag(args, '--priority') ?? '0', 10);
|
||||
const delay = parseInt(parseFlag(args, '--delay') ?? '0', 10);
|
||||
const maxAttempts = parseInt(parseFlag(args, '--max-attempts') ?? '3', 10);
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const maxStalledRaw = parseFlag(args, '--max-stalled');
|
||||
const maxStalled = maxStalledRaw !== undefined ? parseInt(maxStalledRaw, 10) : undefined;
|
||||
// v0.13.1 field audit: expose retry/backoff/timeout/idempotency knobs so
|
||||
// users can tune Minions behavior without dropping into TypeScript.
|
||||
const backoffTypeRaw = parseFlag(args, '--backoff-type');
|
||||
const backoffType = backoffTypeRaw === 'fixed' || backoffTypeRaw === 'exponential'
|
||||
? backoffTypeRaw
|
||||
: undefined;
|
||||
const backoffDelayRaw = parseFlag(args, '--backoff-delay');
|
||||
const backoffDelay = backoffDelayRaw !== undefined ? parseInt(backoffDelayRaw, 10) : undefined;
|
||||
const backoffJitterRaw = parseFlag(args, '--backoff-jitter');
|
||||
const backoffJitter = backoffJitterRaw !== undefined ? parseFloat(backoffJitterRaw) : undefined;
|
||||
const timeoutMsRaw = parseFlag(args, '--timeout-ms');
|
||||
const timeoutMs = timeoutMsRaw !== undefined ? parseInt(timeoutMsRaw, 10) : undefined;
|
||||
if (timeoutMsRaw !== undefined && (isNaN(timeoutMs!) || timeoutMs! <= 0)) {
|
||||
console.error('Error: --timeout-ms must be a positive integer (milliseconds)');
|
||||
process.exit(1);
|
||||
}
|
||||
const idempotencyKey = parseFlag(args, '--idempotency-key');
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const dryRun = hasFlag(args, '--dry-run');
|
||||
const follow = hasFlag(args, '--follow');
|
||||
|
||||
@@ -120,8 +135,13 @@ HANDLER TYPES (built in)
|
||||
console.log(` Queue: ${queueName}`);
|
||||
console.log(` Priority: ${priority}`);
|
||||
console.log(` Max attempts: ${maxAttempts}`);
|
||||
if (maxStalled !== undefined) console.log(` Max stalled: ${maxStalled}`);
|
||||
if (backoffType) console.log(` Backoff type: ${backoffType}`);
|
||||
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
|
||||
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
|
||||
if (timeoutMs !== undefined) console.log(` Timeout: ${timeoutMs}ms`);
|
||||
if (idempotencyKey) console.log(` Idempotency key: ${idempotencyKey}`);
|
||||
if (delay > 0) console.log(` Delay: ${delay}ms`);
|
||||
if (timeoutMs) console.log(` Timeout: ${timeoutMs}ms`);
|
||||
console.log(` Data: ${JSON.stringify(data)}`);
|
||||
return;
|
||||
}
|
||||
@@ -142,8 +162,13 @@ HANDLER TYPES (built in)
|
||||
priority,
|
||||
delay: delay > 0 ? delay : undefined,
|
||||
max_attempts: maxAttempts,
|
||||
queue: queueName,
|
||||
max_stalled: maxStalled,
|
||||
backoff_type: backoffType,
|
||||
backoff_delay: backoffDelay,
|
||||
backoff_jitter: backoffJitter,
|
||||
timeout_ms: timeoutMs,
|
||||
idempotency_key: idempotencyKey,
|
||||
queue: queueName,
|
||||
}, trusted);
|
||||
|
||||
// Submission audit log (operational trace, not forensic insurance).
|
||||
@@ -353,6 +378,8 @@ HANDLER TYPES (built in)
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
|
||||
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
|
||||
|
||||
@@ -370,22 +397,64 @@ HANDLER TYPES (built in)
|
||||
await workerPromise;
|
||||
|
||||
const elapsedSec = ((Date.now() - startTime) / 1000).toFixed(2);
|
||||
if (final?.status === 'completed') {
|
||||
const cfg = (await import('../core/config.ts')).loadConfig();
|
||||
const engineLabel = cfg?.engine ?? 'unknown';
|
||||
console.log(`SMOKE PASS — Minions healthy in ${elapsedSec}s (engine: ${engineLabel})`);
|
||||
if (engineLabel === 'pglite') {
|
||||
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
|
||||
console.log('supports inline execution only (`submit --follow`).');
|
||||
}
|
||||
try { await queue.removeJob(job.id); } catch { /* non-fatal cleanup */ }
|
||||
process.exit(0);
|
||||
} else {
|
||||
if (final?.status !== 'completed') {
|
||||
console.error(`SMOKE FAIL — job #${job.id} status: ${final?.status ?? 'timeout'} (${elapsedSec}s elapsed)`);
|
||||
if (final?.error_text) console.error(` Error: ${final.error_text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
break;
|
||||
|
||||
// --sigkill-rescue: regression case for #219. Simulates a SIGKILL
|
||||
// mid-flight by directly manipulating lock_until via handleStalled.
|
||||
// Verifies that with the v0.13.1 schema default (max_stalled=5), a
|
||||
// stalled job is REQUEUED rather than dead-lettered on first stall.
|
||||
// Full subprocess-level SIGKILL lives in test/e2e/minions.test.ts.
|
||||
if (sigkillRescue) {
|
||||
const rescueJob = await queue.add('noop', {}, { queue: 'smoke' });
|
||||
|
||||
// Transition to active with a past lock_until, mimicking a worker
|
||||
// that claimed and then got SIGKILL'd mid-run.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
lock_token='smoke-sigkill-rescue',
|
||||
lock_until=now() - interval '1 minute',
|
||||
started_at=now() - interval '2 minute',
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[rescueJob.id]
|
||||
);
|
||||
|
||||
const result = await queue.handleStalled();
|
||||
const afterStall = await queue.getJob(rescueJob.id);
|
||||
|
||||
if (afterStall?.status === 'dead') {
|
||||
console.error(
|
||||
`SMOKE FAIL (--sigkill-rescue) — job #${rescueJob.id} was dead-lettered on first stall. ` +
|
||||
`This is the #219 regression: schema default max_stalled should rescue, not dead-letter. ` +
|
||||
`handleStalled: ${JSON.stringify(result)}`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
if (afterStall?.status !== 'waiting') {
|
||||
console.error(
|
||||
`SMOKE FAIL (--sigkill-rescue) — unexpected status after stall: ${afterStall?.status}. ` +
|
||||
`Expected 'waiting' (rescued). handleStalled: ${JSON.stringify(result)}`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
try { await queue.removeJob(rescueJob.id); } catch { /* non-fatal cleanup */ }
|
||||
}
|
||||
|
||||
const cfg = (await import('../core/config.ts')).loadConfig();
|
||||
const engineLabel = cfg?.engine ?? 'unknown';
|
||||
const tag = sigkillRescue ? ' + SIGKILL rescue' : '';
|
||||
console.log(`SMOKE PASS — Minions healthy${tag} in ${elapsedSec}s (engine: ${engineLabel})`);
|
||||
if (engineLabel === 'pglite') {
|
||||
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
|
||||
console.log('supports inline execution only (`submit --follow`).');
|
||||
}
|
||||
try { await queue.removeJob(job.id); } catch { /* non-fatal cleanup */ }
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
case 'work': {
|
||||
@@ -442,11 +511,20 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
|
||||
worker.register('embed', async (job) => {
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
// Primary Minion progress channel is job.updateProgress (DB-backed,
|
||||
// readable via `gbrain jobs get <id>`). Stderr from the worker daemon
|
||||
// only emits coarse job-start / job-done lines; per-page detail lives
|
||||
// in the DB. Per Codex review #20.
|
||||
await runEmbedCore(engine, {
|
||||
slug: typeof job.data.slug === 'string' ? job.data.slug : undefined,
|
||||
slugs: Array.isArray(job.data.slugs) ? (job.data.slugs as string[]) : undefined,
|
||||
all: !!job.data.all,
|
||||
stale: job.data.all ? false : (job.data.stale !== false),
|
||||
onProgress: (done, total, embedded) => {
|
||||
// Fire-and-forget: progress updates are best-effort and must not
|
||||
// block the worker loop.
|
||||
job.updateProgress({ done, total, embedded, phase: 'embed.pages' }).catch(() => {});
|
||||
},
|
||||
});
|
||||
return { embedded: true };
|
||||
});
|
||||
@@ -491,58 +569,39 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
return await runBacklinksCore({ action, dir, dryRun: !!job.data.dryRun });
|
||||
});
|
||||
|
||||
// The killer handler. Autopilot submits ONE `autopilot-cycle` per cycle
|
||||
// (idempotency_key on cycle slot) instead of a 4-job parent-child DAG,
|
||||
// because Minions' parent/child is NOT a depends_on primitive (Codex
|
||||
// H3/H4). Each step is wrapped in its own try/catch; the handler returns
|
||||
// `{ partial: true, failed_steps: [...] }` when any step fails. It does
|
||||
// NOT throw on partial failure — that would cause the Minion to retry,
|
||||
// and an intermittent extract bug would block every future cycle.
|
||||
// Autopilot-cycle handler: delegates to runCycle. Shares the exact same
|
||||
// phase set and ordering as `gbrain dream` and autopilot's inline path —
|
||||
// one source of truth for what the brain does overnight.
|
||||
//
|
||||
// Yields the event loop between phases so the worker's lock-renewal
|
||||
// timer (src/core/minions/worker.ts) can fire. Without this the v0.14
|
||||
// stall-death regression returns: long CPU-bound phases starve the
|
||||
// renewal callback and the stalled-sweeper kills the job.
|
||||
//
|
||||
// Phase failures surface as report.status='partial' (via runCycle's
|
||||
// derivation); the handler returns { partial, status, report } so
|
||||
// `gbrain jobs get <id>` shows the full structured report. Does NOT
|
||||
// throw on partial: a flaky phase must not block every future cycle.
|
||||
worker.register('autopilot-cycle', async (job) => {
|
||||
const { performSync } = await import('./sync.ts');
|
||||
const { runExtractCore } = await import('./extract.ts');
|
||||
const { runEmbedCore } = await import('./embed.ts');
|
||||
const { runBacklinksCore } = await import('./backlinks.ts');
|
||||
|
||||
const { runCycle } = await import('../core/cycle.ts');
|
||||
const repoPath = typeof job.data.repoPath === 'string'
|
||||
? job.data.repoPath
|
||||
: (await engine.getConfig('sync.repo_path')) ?? '.';
|
||||
|
||||
const steps: Record<string, unknown> = {};
|
||||
const failed: string[] = [];
|
||||
const report = await runCycle(engine, {
|
||||
brainDir: repoPath,
|
||||
pull: true, // autopilot daemon opts into git pull
|
||||
yieldBetweenPhases: async () => {
|
||||
// Yield to the event loop so worker lock-renewal can fire.
|
||||
await new Promise<void>(r => setImmediate(r));
|
||||
},
|
||||
});
|
||||
|
||||
// Bug 8 — Between phases, yield to the event loop. The worker's lock
|
||||
// renewal runs on a timer (src/core/minions/worker.ts); without a
|
||||
// periodic yield, long CPU-bound phases starve the renewal callback
|
||||
// and the job gets killed by the stalled-sweeper. A single
|
||||
// `await new Promise(r => setImmediate(r))` gives the timer a chance
|
||||
// to fire. The per-phase body is async+await already, so each phase
|
||||
// internally yields on its own I/O boundaries — this is a belt for
|
||||
// the gap between phases.
|
||||
//
|
||||
// Follow-up (deferred to v0.15): thread ctx.signal / ctx.shutdownSignal
|
||||
// through each core fn so mid-phase cancellation works on huge brains.
|
||||
const yieldToLoop = () => new Promise<void>(r => setImmediate(r));
|
||||
|
||||
try { steps.sync = await performSync(engine, { repoPath, noEmbed: true }); }
|
||||
catch (e) { steps.sync = { error: e instanceof Error ? e.message : String(e) }; failed.push('sync'); }
|
||||
await yieldToLoop();
|
||||
|
||||
try { steps.extract = await runExtractCore(engine, { mode: 'all', dir: repoPath }); }
|
||||
catch (e) { steps.extract = { error: e instanceof Error ? e.message : String(e) }; failed.push('extract'); }
|
||||
await yieldToLoop();
|
||||
|
||||
try { await runEmbedCore(engine, { stale: true }); steps.embed = { embedded: true }; }
|
||||
catch (e) { steps.embed = { error: e instanceof Error ? e.message : String(e) }; failed.push('embed'); }
|
||||
await yieldToLoop();
|
||||
|
||||
try { steps.backlinks = await runBacklinksCore({ action: 'fix', dir: repoPath }); }
|
||||
catch (e) { steps.backlinks = { error: e instanceof Error ? e.message : String(e) }; failed.push('backlinks'); }
|
||||
|
||||
if (failed.length > 0) {
|
||||
return { partial: true, failed_steps: failed, steps };
|
||||
}
|
||||
return { partial: false, steps };
|
||||
return {
|
||||
partial: report.status === 'partial' || report.status === 'failed',
|
||||
status: report.status,
|
||||
report,
|
||||
};
|
||||
});
|
||||
|
||||
// Shell handler: registered ONLY when GBRAIN_ALLOW_SHELL_JOBS=1 is set on the
|
||||
@@ -556,4 +615,37 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
} else {
|
||||
process.stderr.write('[minion worker] shell handler disabled (set GBRAIN_ALLOW_SHELL_JOBS=1 to enable)\n');
|
||||
}
|
||||
|
||||
// v0.15 subagent handlers: always-on. Unlike shell (which needs an env
|
||||
// flag because of RCE surface), subagent only calls the Anthropic API
|
||||
// with the operator's own ANTHROPIC_API_KEY — no key, the SDK call
|
||||
// fails immediately. Who-can-submit is already gated by
|
||||
// PROTECTED_JOB_NAMES + TrustedSubmitOpts (MCP can't submit subagent
|
||||
// jobs; only the CLI path with allowProtectedSubmit can). No separate
|
||||
// cost-ceremony env flag needed.
|
||||
const { makeSubagentHandler } = await import('../core/minions/handlers/subagent.ts');
|
||||
const { subagentAggregatorHandler } = await import('../core/minions/handlers/subagent-aggregator.ts');
|
||||
worker.register('subagent', makeSubagentHandler({ engine }));
|
||||
worker.register('subagent_aggregator', subagentAggregatorHandler);
|
||||
process.stderr.write('[minion worker] subagent handlers enabled\n');
|
||||
|
||||
// Plugin discovery — one line per discovered plugin (mirrors the
|
||||
// openclaw-seam startup line convention from v0.11+). Loaded
|
||||
// unconditionally; empty GBRAIN_PLUGIN_PATH is a no-op.
|
||||
try {
|
||||
const { loadPluginsFromEnv } = await import('../core/minions/plugin-loader.ts');
|
||||
const { BRAIN_TOOL_ALLOWLIST } = await import('../core/minions/tools/brain-allowlist.ts');
|
||||
const validNames = new Set<string>();
|
||||
for (const n of BRAIN_TOOL_ALLOWLIST) validNames.add(`brain_${n}`);
|
||||
const loaded = loadPluginsFromEnv({ validAgentToolNames: validNames });
|
||||
for (const w of loaded.warnings) process.stderr.write(w + '\n');
|
||||
for (const p of loaded.plugins) {
|
||||
process.stderr.write(
|
||||
`[plugin-loader] loaded '${p.manifest.name}' v${p.manifest.version} (${p.subagents.length} subagents)\n`,
|
||||
);
|
||||
}
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[plugin-loader] discovery failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -268,10 +268,17 @@ export async function runLint(args: string[]) {
|
||||
const isSingleFile = statSync(target).isFile();
|
||||
const pages = isSingleFile ? [target] : collectPages(target);
|
||||
|
||||
// Progress on stderr. Stdout keeps the per-issue human output it always had.
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('lint.pages', pages.length);
|
||||
|
||||
for (const page of pages) {
|
||||
const content = readFileSync(page, 'utf-8');
|
||||
const relPath = isSingleFile ? page : relative(target, page);
|
||||
const issues = lintContent(content, relPath);
|
||||
progress.tick(1);
|
||||
if (issues.length === 0) continue;
|
||||
|
||||
console.log(`\n${relPath}:`);
|
||||
@@ -292,6 +299,8 @@ export async function runLint(args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
// Re-run core for the aggregate counts (cheap; re-parses contents but
|
||||
// produces canonical numbers for the summary line).
|
||||
const result = await runLintCore({ target, fix: doFix, dryRun });
|
||||
|
||||
@@ -14,6 +14,8 @@ import type { EngineConfig } from '../core/types.ts';
|
||||
import { homedir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
interface MigrateOpts {
|
||||
targetEngine: 'postgres' | 'pglite';
|
||||
@@ -146,6 +148,9 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
|
||||
console.log(`Migrating ${pagesToMigrate.length} pages (${allPages.length} total, ${completedSet.size} already done)...`);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('migrate.copy_pages', pagesToMigrate.length);
|
||||
|
||||
let migrated = 0;
|
||||
for (const page of pagesToMigrate) {
|
||||
// Copy page
|
||||
@@ -203,20 +208,21 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
manifest!.completed_slugs.push(page.slug);
|
||||
saveManifest(manifest!);
|
||||
migrated++;
|
||||
|
||||
if (migrated % 50 === 0 || migrated === pagesToMigrate.length) {
|
||||
console.log(` Progress: ${migrated}/${pagesToMigrate.length} pages`);
|
||||
}
|
||||
progress.tick(1, page.slug);
|
||||
}
|
||||
progress.finish();
|
||||
|
||||
// Copy links (after all pages exist in target)
|
||||
console.log('Copying links...');
|
||||
progress.start('migrate.copy_links', allPages.length);
|
||||
for (const page of allPages) {
|
||||
const links = await sourceEngine.getLinks(page.slug);
|
||||
for (const link of links) {
|
||||
await targetEngine.addLink(link.from_slug, link.to_slug, link.context, link.link_type);
|
||||
}
|
||||
progress.tick(1);
|
||||
}
|
||||
progress.finish();
|
||||
|
||||
// Copy config (selective)
|
||||
const configKeys = ['embedding_model', 'embedding_dimensions', 'chunk_strategy'];
|
||||
|
||||
@@ -17,6 +17,7 @@ import { v0_12_2 } from './v0_12_2.ts';
|
||||
import { v0_13_0 } from './v0_13_0.ts';
|
||||
import { v0_13_1 } from './v0_13_1.ts';
|
||||
import { v0_14_0 } from './v0_14_0.ts';
|
||||
import { v0_16_0 } from './v0_16_0.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
@@ -25,6 +26,7 @@ export const migrations: Migration[] = [
|
||||
v0_13_0,
|
||||
v0_13_1,
|
||||
v0_14_0,
|
||||
v0_16_0,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lstatSync, statSync, realpathSync } from 'fs';
|
||||
import { join, resolve, dirname } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
|
||||
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
|
||||
@@ -60,7 +61,7 @@ export interface PendingHostWorkEntry {
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
|
||||
@@ -32,6 +32,7 @@
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
@@ -42,7 +43,7 @@ function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
|
||||
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
|
||||
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -92,7 +93,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// --source db is idempotent: the UNIQUE constraint on
|
||||
// (from_page_id, to_page_id, link_type) and ON CONFLICT DO NOTHING
|
||||
// make re-runs cheap. Empty brains return 0/0 quickly.
|
||||
execSync('gbrain extract links --source db', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
execSync('gbrain extract links --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'backfill_links', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -103,7 +104,7 @@ function phaseCBackfillLinks(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
function phaseDBackfillTimeline(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'backfill_timeline', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain extract timeline --source db', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
execSync('gbrain extract timeline --source db' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'backfill_timeline', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -216,10 +217,9 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
phases.push(e);
|
||||
|
||||
// F. Record
|
||||
// a.status was narrowed to 'skipped' | 'complete' by the early return above.
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' ? 'failed' :
|
||||
phases.some(p => p.status === 'failed') ? 'partial' :
|
||||
'complete';
|
||||
phases.some(p => p.status === 'failed') ? 'partial' : 'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { childGlobalFlags } from '../../core/cli-options.ts';
|
||||
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
@@ -29,7 +30,9 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
// Propagate global progress flags so the child shows the same mode the
|
||||
// parent orchestrator is running in.
|
||||
execSync('gbrain init --migrate-only' + childGlobalFlags(), { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -42,7 +45,8 @@ function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain repair-jsonb', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
// stdio: 'inherit' — child's stderr progress streams straight through.
|
||||
execSync('gbrain repair-jsonb' + childGlobalFlags(), { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'jsonb_repair', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -55,8 +59,14 @@ function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// Explicit stdio discipline: we must parse JSON off child.stdout, so
|
||||
// pipe stdout but let child.stderr (progress) pass straight through.
|
||||
// Any accidental stdout progress from the child would break JSON.parse
|
||||
// (per Codex review #12). NOTE: we deliberately do NOT pass
|
||||
// --progress-json here — this child is parsed, not watched.
|
||||
const out = execSync('gbrain repair-jsonb --dry-run --json', {
|
||||
encoding: 'utf-8', timeout: 60_000, env: process.env,
|
||||
stdio: ['ignore', 'pipe', 'inherit'],
|
||||
});
|
||||
const parsed = JSON.parse(out) as { total_repaired?: number; engine?: string };
|
||||
const remaining = parsed.total_repaired ?? 0;
|
||||
@@ -95,10 +105,9 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
// a.status and b.status were narrowed to 'skipped' | 'complete' by early returns above.
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' || b.status === 'failed' ? 'failed' :
|
||||
c.status === 'failed' ? 'partial' :
|
||||
'complete';
|
||||
c.status === 'failed' ? 'partial' : 'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
@@ -36,17 +36,18 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
|
||||
// and swaps the unique constraint. Schema build time on 46K pages is
|
||||
// ~10s (ALTER + index builds). Bumped timeout accounts for slow Supabase
|
||||
// links (v0.12.1 pattern — migrations can time out on the 60s default).
|
||||
// Use the CURRENTLY-RUNNING binary path (not `gbrain` off $PATH). After
|
||||
// `gbrain upgrade` rewrites the binary, a bare `gbrain` could resolve to
|
||||
// an older installed copy via alias shadowing or stale PATH cache. The
|
||||
// active process.execPath is the one that loaded THIS migration module,
|
||||
// so recursing into it is always the right binary.
|
||||
const GBRAIN = process.execPath;
|
||||
//
|
||||
// Shell out to the canonical `gbrain` shim on PATH (`/usr/local/bin/gbrain`
|
||||
// by default). An earlier revision resolved via the active Node/Bun runtime
|
||||
// binary, but on bun-installed trees that binary is `bun` — the spawned
|
||||
// `bun extract ...` gets reinterpreted as `bun run extract` and crashes the
|
||||
// upgrade mid-migration. The shim is already the canonical wrapper; trust
|
||||
// it. Regression guarded by test/migrations-v0_13_0.test.ts.
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync(`${GBRAIN} init --migrate-only`, { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -63,7 +64,7 @@ function phaseBBackfill(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// `--include-frontmatter` is the v0.13 flag that enables the canonical
|
||||
// frontmatter link extractor. Default-OFF in the CLI for back-compat;
|
||||
// the migration explicitly opts in because this is the canonical backfill.
|
||||
execSync(`${GBRAIN} extract links --source db --include-frontmatter`, {
|
||||
execSync('gbrain extract links --source db --include-frontmatter', {
|
||||
stdio: 'inherit',
|
||||
timeout: 1_800_000, // 30 min hard cap; typical 2-5 min on 46K pages
|
||||
env: process.env,
|
||||
@@ -88,7 +89,7 @@ function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
// docs-only brains, and brains with no entity pages legitimately
|
||||
// produce 0. Phase B's own stdout shows `Links: created N` which is
|
||||
// the authoritative signal — user sees it during upgrade.
|
||||
const out = execSync(`${GBRAIN} call get_stats`, {
|
||||
const out = execSync('gbrain call get_stats', {
|
||||
encoding: 'utf-8', timeout: 60_000, env: process.env,
|
||||
});
|
||||
const parsed = JSON.parse(out) as { link_count?: number; page_count?: number };
|
||||
@@ -128,10 +129,9 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
// a.status and b.status were narrowed to 'skipped' | 'complete' by early returns above.
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' || b.status === 'failed' ? 'failed' :
|
||||
c.status === 'failed' ? 'partial' :
|
||||
'complete';
|
||||
c.status === 'failed' ? 'partial' : 'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* v0.16.0 migration orchestrator — Subagent runtime schema.
|
||||
*
|
||||
* Adds three tables for durable LLM agent loops:
|
||||
* - subagent_messages Anthropic message-block persistence
|
||||
* - subagent_tool_executions Two-phase tool ledger (pending/complete/failed)
|
||||
* - subagent_rate_leases Lease-based concurrency cap
|
||||
*
|
||||
* All DDL is `CREATE TABLE IF NOT EXISTS` and ships in src/schema.sql +
|
||||
* src/core/pglite-schema.ts (both Postgres and PGLite fresh-install paths).
|
||||
* This orchestrator's job is therefore only to VERIFY the tables exist after
|
||||
* `gbrain init --migrate-only` has run, so an upgrade that somehow skipped
|
||||
* the schema step fails loudly instead of silently.
|
||||
*
|
||||
* Phases (all idempotent):
|
||||
* A. Schema — gbrain init --migrate-only (creates tables via SCHEMA_SQL).
|
||||
* B. Verify — confirm all three tables exist.
|
||||
* C. Record — append completed.jsonl.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
import { createEngine } from '../../core/engine-factory.ts';
|
||||
|
||||
const REQUIRED_TABLES = ['subagent_messages', 'subagent_tool_executions', 'subagent_rate_leases'] as const;
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'schema', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase B — Verify tables exist ───────────────────────────
|
||||
|
||||
async function phaseBVerify(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
return { name: 'verify', status: 'skipped', detail: 'no brain configured' };
|
||||
}
|
||||
const engine = await createEngine(toEngineConfig(config));
|
||||
await engine.connect(toEngineConfig(config));
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ table_name: string }>(
|
||||
`SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = current_schema()
|
||||
AND table_name IN ('subagent_messages','subagent_tool_executions','subagent_rate_leases')`,
|
||||
);
|
||||
const found = new Set(rows.map(r => r.table_name));
|
||||
const missing = REQUIRED_TABLES.filter(t => !found.has(t));
|
||||
if (missing.length > 0) {
|
||||
return {
|
||||
name: 'verify',
|
||||
status: 'failed',
|
||||
detail: `missing tables: ${missing.join(', ')}`,
|
||||
};
|
||||
}
|
||||
return { name: 'verify', status: 'complete', detail: `${REQUIRED_TABLES.length} tables present` };
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch {}
|
||||
}
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'verify',
|
||||
status: 'failed',
|
||||
detail: e instanceof Error ? e.message : String(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.16.0 — Subagent runtime schema ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalize(phases, 'failed');
|
||||
|
||||
const b = await phaseBVerify(opts);
|
||||
phases.push(b);
|
||||
|
||||
// a.status was narrowed to 'skipped' | 'complete' by the early return above.
|
||||
const status: 'complete' | 'partial' | 'failed' =
|
||||
b.status === 'failed' ? 'partial' : 'complete';
|
||||
|
||||
return finalize(phases, status);
|
||||
}
|
||||
|
||||
function finalize(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
|
||||
if (status !== 'failed') {
|
||||
try {
|
||||
appendCompletedMigration({
|
||||
version: '0.16.0',
|
||||
completed_at: new Date().toISOString(),
|
||||
status: status as 'complete' | 'partial',
|
||||
phases: phases.map(p => ({ name: p.name, status: p.status })),
|
||||
});
|
||||
} catch {
|
||||
// Recording is best-effort.
|
||||
}
|
||||
}
|
||||
return { version: '0.16.0', status, phases };
|
||||
}
|
||||
|
||||
export const v0_16_0: Migration = {
|
||||
version: '0.16.0',
|
||||
featurePitch: {
|
||||
headline: 'Durable LLM agents land in the brain — survive crashes, sleeps, and worker restarts.',
|
||||
description:
|
||||
'v0.16.0 adds the subagent runtime: run long-running, fan-out Anthropic LLM loops ' +
|
||||
'as first-class Minion jobs. Crash-resumable turn persistence, two-phase tool ledger, ' +
|
||||
'lease-based rate limit, parent-child fan-out with aggregation. Entry points: `gbrain ' +
|
||||
'agent run` and `gbrain agent logs`. See docs/guides/plugin-authors.md for shipping ' +
|
||||
'custom subagent defs from a host repo (your OpenClaw etc.).',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBVerify,
|
||||
REQUIRED_TABLES,
|
||||
};
|
||||
+41
-27
@@ -13,7 +13,8 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { createProgress, startHeartbeat } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
// --- Types ---
|
||||
|
||||
@@ -97,37 +98,50 @@ export function deriveDomain(frontmatterDomain: string | null | undefined, slug:
|
||||
// --- Core query ---
|
||||
|
||||
/**
|
||||
* Find pages with no inbound links.
|
||||
* Returns raw rows from the DB (all pages regardless of filter).
|
||||
* Find pages with no inbound links via the engine's built-in helper.
|
||||
* Returns raw rows (all pages regardless of filter).
|
||||
*
|
||||
* As of v0.17: takes an engine argument. Composes with runCycle which
|
||||
* passes an explicit engine. No more db.getConnection() global — fixes
|
||||
* the PGLite-vs-Postgres + test-fixture coupling codex flagged.
|
||||
*/
|
||||
export async function queryOrphanPages(): Promise<{ slug: string; title: string; domain: string | null }[]> {
|
||||
const sql = db.getConnection();
|
||||
const rows = await sql`
|
||||
SELECT
|
||||
p.slug,
|
||||
COALESCE(p.title, p.slug) AS title,
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM links l WHERE l.to_page_id = p.id
|
||||
)
|
||||
ORDER BY p.slug
|
||||
`;
|
||||
return rows as { slug: string; title: string; domain: string | null }[];
|
||||
export async function queryOrphanPages(
|
||||
engine: BrainEngine,
|
||||
): Promise<{ slug: string; title: string; domain: string | null }[]> {
|
||||
return engine.findOrphanPages();
|
||||
}
|
||||
|
||||
/**
|
||||
* Find orphan pages, with optional pseudo-page filtering.
|
||||
* Returns structured OrphanResult with totals.
|
||||
*
|
||||
* As of v0.17: `engine` is required. See queryOrphanPages for rationale.
|
||||
*/
|
||||
export async function findOrphans(includePseudo: boolean = false): Promise<OrphanResult> {
|
||||
const allOrphans = await queryOrphanPages();
|
||||
const totalPages = allOrphans.length; // pages with no inbound links
|
||||
|
||||
// Count total pages in DB for the summary line
|
||||
const sql = db.getConnection();
|
||||
const [{ count: totalPagesCount }] = await sql`SELECT count(*)::int AS count FROM pages`;
|
||||
const total = Number(totalPagesCount);
|
||||
export async function findOrphans(
|
||||
engine: BrainEngine,
|
||||
opts: { includePseudo?: boolean } = {},
|
||||
): Promise<OrphanResult> {
|
||||
const includePseudo = !!opts.includePseudo;
|
||||
// The NOT EXISTS anti-join over pages × links can take seconds on 50K-page
|
||||
// brains. Heartbeat every second so agents see the scan is alive. Keyset
|
||||
// pagination was considered and rejected: without an index on
|
||||
// links.to_page_id it does no useful work. Adding that index is a
|
||||
// follow-up (v0.14.3 schema migration).
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('orphans.scan');
|
||||
const stopHb = startHeartbeat(progress, 'scanning pages for missing inbound links…');
|
||||
let allOrphans: { slug: string; title: string; domain: string | null }[];
|
||||
let total: number;
|
||||
try {
|
||||
allOrphans = await engine.findOrphanPages();
|
||||
// Count total pages in DB for the summary line
|
||||
const stats = await engine.getStats();
|
||||
total = stats.page_count;
|
||||
} finally {
|
||||
stopHb();
|
||||
progress.finish();
|
||||
}
|
||||
const _totalPages = allOrphans.length; // pages with no inbound links (preserved for ref)
|
||||
|
||||
const filtered = includePseudo
|
||||
? allOrphans
|
||||
@@ -189,7 +203,7 @@ export function formatOrphansText(result: OrphanResult): string {
|
||||
|
||||
// --- CLI entry point ---
|
||||
|
||||
export async function runOrphans(_engine: BrainEngine, args: string[]) {
|
||||
export async function runOrphans(engine: BrainEngine, args: string[]) {
|
||||
const json = args.includes('--json');
|
||||
const count = args.includes('--count');
|
||||
const includePseudo = args.includes('--include-pseudo');
|
||||
@@ -211,7 +225,7 @@ Summary line: N orphans out of M linkable pages (K total; K-M excluded)
|
||||
return;
|
||||
}
|
||||
|
||||
const result = await findOrphans(includePseudo);
|
||||
const result = await findOrphans(engine, { includePseudo });
|
||||
|
||||
if (count) {
|
||||
console.log(String(result.total_orphans));
|
||||
|
||||
@@ -31,6 +31,8 @@
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import type { EngineConfig } from '../core/types.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { createProgress, startHeartbeat } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
interface RepairTarget {
|
||||
table: string;
|
||||
@@ -97,28 +99,44 @@ export async function repairJsonb(opts: RepairOpts = { dryRun: false }): Promise
|
||||
await db.connect(engineCfg);
|
||||
const sql = db.getConnection();
|
||||
|
||||
// Progress on stderr only. Stdout is reserved for the JSON summary that
|
||||
// migrations/v0_12_2.ts parses via JSON.parse — stray progress lines on
|
||||
// stdout would break the orchestrator (per Codex review #12).
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('repair_jsonb.run', TARGETS.length);
|
||||
|
||||
for (const t of TARGETS) {
|
||||
const phase = `repair_jsonb.${t.table}.${t.column}`;
|
||||
progress.heartbeat(phase);
|
||||
// Heartbeat the caller while each UPDATE runs (minutes on 50K-row tables).
|
||||
const stopHb = startHeartbeat(progress, `${t.table}.${t.column}`);
|
||||
let repaired = 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM ${t.table} WHERE jsonb_typeof(${t.column}) = 'string'`,
|
||||
);
|
||||
repaired = (rows[0] as { n: number }).n;
|
||||
} else {
|
||||
const rows = await sql.unsafe(
|
||||
`UPDATE ${t.table}
|
||||
SET ${t.column} = (${t.column} #>> '{}')::jsonb
|
||||
WHERE jsonb_typeof(${t.column}) = 'string'
|
||||
RETURNING 1`,
|
||||
);
|
||||
repaired = rows.length;
|
||||
try {
|
||||
if (opts.dryRun) {
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM ${t.table} WHERE jsonb_typeof(${t.column}) = 'string'`,
|
||||
);
|
||||
repaired = (rows[0] as unknown as { n: number }).n;
|
||||
} else {
|
||||
const rows = await sql.unsafe(
|
||||
`UPDATE ${t.table}
|
||||
SET ${t.column} = (${t.column} #>> '{}')::jsonb
|
||||
WHERE jsonb_typeof(${t.column}) = 'string'
|
||||
RETURNING 1`,
|
||||
);
|
||||
repaired = rows.length;
|
||||
}
|
||||
} finally {
|
||||
stopHb();
|
||||
}
|
||||
|
||||
progress.tick(1, `${t.table}.${t.column}=${repaired}`);
|
||||
result.per_target.push({ table: t.table, column: t.column, rows_repaired: repaired });
|
||||
result.total_repaired += repaired;
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* CLI: gbrain repos list|add|remove
|
||||
* Multi-repo management for code + knowledge indexing.
|
||||
*/
|
||||
|
||||
import { resolve } from 'path';
|
||||
import { existsSync } from 'fs';
|
||||
import {
|
||||
loadRepoConfigs,
|
||||
addRepoConfig,
|
||||
removeRepoConfig,
|
||||
normalizeRepoName,
|
||||
type RepoStrategy,
|
||||
} from '../core/multi-repo.ts';
|
||||
|
||||
export async function handleRepos(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
|
||||
if (!sub || sub === 'list') {
|
||||
return reposList();
|
||||
}
|
||||
|
||||
if (sub === 'add') {
|
||||
return reposAdd(args.slice(1));
|
||||
}
|
||||
|
||||
if (sub === 'remove' || sub === 'rm') {
|
||||
return reposRemove(args.slice(1));
|
||||
}
|
||||
|
||||
console.error(`Unknown repos subcommand: ${sub}`);
|
||||
console.error('Usage: gbrain repos [list|add|remove]');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function reposList(): void {
|
||||
const repos = loadRepoConfigs();
|
||||
|
||||
if (repos.length === 0) {
|
||||
console.log('No repos configured. Use `gbrain repos add <path>` to add one.');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`${repos.length} repo(s) configured:\n`);
|
||||
for (const repo of repos) {
|
||||
const enabled = repo.syncEnabled !== false ? '✓' : '✗';
|
||||
const includes = repo.include?.length ? ` include=[${repo.include.join(',')}]` : '';
|
||||
const excludes = repo.exclude?.length ? ` exclude=[${repo.exclude.join(',')}]` : '';
|
||||
console.log(` ${enabled} ${repo.name} (${repo.strategy}) → ${repo.path}${includes}${excludes}`);
|
||||
}
|
||||
}
|
||||
|
||||
function reposAdd(args: string[]): void {
|
||||
if (args.length === 0) {
|
||||
console.error('Usage: gbrain repos add <path> [--name <name>] [--strategy markdown|code|auto]');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const repoPath = resolve(args[0]);
|
||||
|
||||
if (!existsSync(repoPath)) {
|
||||
console.error(`Path does not exist: ${repoPath}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let name = normalizeRepoName(repoPath);
|
||||
let strategy: RepoStrategy = 'auto';
|
||||
const include: string[] = [];
|
||||
const exclude: string[] = [];
|
||||
|
||||
for (let i = 1; i < args.length; i++) {
|
||||
if (args[i] === '--name' && args[i + 1]) {
|
||||
name = args[++i];
|
||||
} else if (args[i] === '--strategy' && args[i + 1]) {
|
||||
const s = args[++i];
|
||||
if (s === 'markdown' || s === 'code' || s === 'auto') {
|
||||
strategy = s;
|
||||
} else {
|
||||
console.error(`Invalid strategy: ${s}. Must be markdown, code, or auto.`);
|
||||
process.exit(1);
|
||||
}
|
||||
} else if (args[i] === '--include' && args[i + 1]) {
|
||||
include.push(args[++i]);
|
||||
} else if (args[i] === '--exclude' && args[i + 1]) {
|
||||
exclude.push(args[++i]);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
const repos = addRepoConfig({
|
||||
path: repoPath,
|
||||
name,
|
||||
strategy,
|
||||
include: include.length > 0 ? include : undefined,
|
||||
exclude: exclude.length > 0 ? exclude : undefined,
|
||||
syncEnabled: true,
|
||||
});
|
||||
console.log(`Added repo "${name}" (${strategy}) → ${repoPath}`);
|
||||
console.log(`${repos.length} repo(s) total. Run \`gbrain sync --all\` to index.`);
|
||||
} catch (e: unknown) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
function reposRemove(args: string[]): void {
|
||||
if (args.length === 0) {
|
||||
console.error('Usage: gbrain repos remove <name>');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const name = args[0];
|
||||
|
||||
try {
|
||||
const repos = removeRepoConfig(name);
|
||||
console.log(`Removed repo "${name}". ${repos.length} repo(s) remaining.`);
|
||||
} catch (e: unknown) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -25,9 +25,12 @@ import { xHandleToTweetResolver } from '../core/resolvers/builtin/x-api/handle-t
|
||||
* to call from multiple entry points.
|
||||
*/
|
||||
export function registerBuiltinResolvers(registry = getDefaultRegistry()): void {
|
||||
const builtins = [urlReachableResolver, xHandleToTweetResolver] as const;
|
||||
// Cast each element to the widest shape the registry accepts. The tuple
|
||||
// element types diverge (different Input/Output generics) so the union
|
||||
// type would not satisfy registry.register's single-signature parameter.
|
||||
const builtins = [urlReachableResolver, xHandleToTweetResolver];
|
||||
for (const r of builtins) {
|
||||
if (!registry.has(r.id)) registry.register(r);
|
||||
if (!registry.has(r.id)) registry.register(r as Parameters<typeof registry.register>[0]);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* `gbrain skillpack-check` — agent-readable health report.
|
||||
*
|
||||
* Wraps `gbrain doctor --json` + `gbrain apply-migrations --list` into a
|
||||
* single JSON blob a host agent (Wintermute's morning-briefing, any
|
||||
* single JSON blob a host agent (your OpenClaw's morning-briefing, any
|
||||
* OpenClaw cron) can consume without parsing two subcommands.
|
||||
*
|
||||
* Usage:
|
||||
@@ -18,6 +18,7 @@
|
||||
|
||||
import { execFileSync } from 'child_process';
|
||||
import { VERSION } from '../version.ts';
|
||||
import { getCliOptions } from '../core/cli-options.ts';
|
||||
|
||||
/**
|
||||
* Resolve the gbrain binary + args for spawning subcommands from
|
||||
@@ -207,7 +208,9 @@ Exit codes:
|
||||
return;
|
||||
}
|
||||
|
||||
const quiet = args.includes('--quiet');
|
||||
// --quiet is parsed as a global flag in src/cli.ts (and stripped from argv
|
||||
// before reaching here); honor it via the CliOptions singleton.
|
||||
const quiet = getCliOptions().quiet;
|
||||
const report = buildReport();
|
||||
|
||||
if (!quiet) {
|
||||
|
||||
+156
-44
@@ -12,6 +12,8 @@ import {
|
||||
acknowledgeSyncFailures,
|
||||
} from '../core/sync.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface SyncResult {
|
||||
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
|
||||
@@ -22,6 +24,8 @@ export interface SyncResult {
|
||||
deleted: number;
|
||||
renamed: number;
|
||||
chunksCreated: number;
|
||||
/** Pages re-embedded during this sync's auto-embed step. 0 if --no-embed or skipped. */
|
||||
embedded: number;
|
||||
pagesAffected: string[];
|
||||
failedFiles?: number; // count of parse failures (Bug 9)
|
||||
}
|
||||
@@ -37,6 +41,8 @@ export interface SyncOpts {
|
||||
skipFailed?: boolean;
|
||||
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
|
||||
retryFailed?: boolean;
|
||||
/** Multi-repo: sync strategy override (markdown, code, auto). */
|
||||
strategy?: 'markdown' | 'code' | 'auto';
|
||||
}
|
||||
|
||||
function git(repoPath: string, ...args: string[]): string {
|
||||
@@ -114,6 +120,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
@@ -122,16 +129,17 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
const diffOutput = git(repoPath, 'diff', '--name-status', '-M', `${lastCommit}..${headCommit}`);
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
|
||||
// Filter to syncable files
|
||||
// Filter to syncable files (strategy-aware)
|
||||
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
|
||||
const filtered: SyncManifest = {
|
||||
added: manifest.added.filter(p => isSyncable(p)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p)),
|
||||
deleted: manifest.deleted.filter(p => isSyncable(p)),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to)),
|
||||
added: manifest.added.filter(p => isSyncable(p, syncOpts)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p, syncOpts)),
|
||||
deleted: manifest.deleted.filter(p => isSyncable(p, syncOpts)),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to, syncOpts)),
|
||||
};
|
||||
|
||||
// Delete pages that became un-syncable (modified but filtered out)
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p));
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p, syncOpts));
|
||||
for (const path of unsyncableModified) {
|
||||
const slug = pathToSlug(path);
|
||||
try {
|
||||
@@ -163,6 +171,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated: 0,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
@@ -177,6 +186,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
@@ -190,29 +200,43 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
let chunksCreated = 0;
|
||||
const start = Date.now();
|
||||
|
||||
// Per-file progress on stderr so agents see each step of a big sync.
|
||||
// Phases: sync.deletes, sync.renames, sync.imports.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Process deletes first (prevents slug conflicts)
|
||||
for (const path of filtered.deleted) {
|
||||
const slug = pathToSlug(path);
|
||||
await engine.deletePage(slug);
|
||||
pagesAffected.push(slug);
|
||||
if (filtered.deleted.length > 0) {
|
||||
progress.start('sync.deletes', filtered.deleted.length);
|
||||
for (const path of filtered.deleted) {
|
||||
const slug = pathToSlug(path);
|
||||
await engine.deletePage(slug);
|
||||
pagesAffected.push(slug);
|
||||
progress.tick(1, slug);
|
||||
}
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// Process renames (updateSlug preserves page_id, chunks, embeddings)
|
||||
for (const { from, to } of filtered.renamed) {
|
||||
const oldSlug = pathToSlug(from);
|
||||
const newSlug = pathToSlug(to);
|
||||
try {
|
||||
await engine.updateSlug(oldSlug, newSlug);
|
||||
} catch {
|
||||
// Slug doesn't exist or collision, treat as add
|
||||
if (filtered.renamed.length > 0) {
|
||||
progress.start('sync.renames', filtered.renamed.length);
|
||||
for (const { from, to } of filtered.renamed) {
|
||||
const oldSlug = pathToSlug(from);
|
||||
const newSlug = pathToSlug(to);
|
||||
try {
|
||||
await engine.updateSlug(oldSlug, newSlug);
|
||||
} catch {
|
||||
// Slug doesn't exist or collision, treat as add
|
||||
}
|
||||
// Reimport at new path (picks up content changes)
|
||||
const filePath = join(repoPath, to);
|
||||
if (existsSync(filePath)) {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed });
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
}
|
||||
pagesAffected.push(newSlug);
|
||||
progress.tick(1, newSlug);
|
||||
}
|
||||
// Reimport at new path (picks up content changes)
|
||||
const filePath = join(repoPath, to);
|
||||
if (existsSync(filePath)) {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed });
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
}
|
||||
pagesAffected.push(newSlug);
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// Process adds and modifies.
|
||||
@@ -225,24 +249,37 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// ep_poll whenever the diff crosses the old > 10 threshold that used to
|
||||
// trigger the outer wrap. Per-file atomicity is also the right granularity:
|
||||
// one file's failure should not roll back the others' successful imports.
|
||||
//
|
||||
// v0.15.2: per-file progress on stderr via the shared reporter.
|
||||
// Bug 9: per-file failures captured in `failedFiles` so the caller can
|
||||
// gate `sync.last_commit` advancement and record recoverable errors.
|
||||
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
|
||||
for (const path of [...filtered.added, ...filtered.modified]) {
|
||||
const filePath = join(repoPath, path);
|
||||
if (!existsSync(filePath)) continue;
|
||||
try {
|
||||
const result = await importFile(engine, filePath, path, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
} else if (result.status === 'skipped' && (result as any).error) {
|
||||
// importFile returned a non-throw skip with a reason
|
||||
failedFiles.push({ path, error: String((result as any).error) });
|
||||
const addsAndMods = [...filtered.added, ...filtered.modified];
|
||||
if (addsAndMods.length > 0) {
|
||||
progress.start('sync.imports', addsAndMods.length);
|
||||
for (const path of addsAndMods) {
|
||||
const filePath = join(repoPath, path);
|
||||
if (!existsSync(filePath)) {
|
||||
progress.tick(1, `skip:${path}`);
|
||||
continue;
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(` Warning: skipped ${path}: ${msg}`);
|
||||
failedFiles.push({ path, error: msg });
|
||||
try {
|
||||
const result = await importFile(engine, filePath, path, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
} else if (result.status === 'skipped' && (result as any).error) {
|
||||
// importFile returned a non-throw skip with a reason.
|
||||
failedFiles.push({ path, error: String((result as any).error) });
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(` Warning: skipped ${path}: ${msg}`);
|
||||
failedFiles.push({ path, error: msg });
|
||||
}
|
||||
progress.tick(1, path);
|
||||
}
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
const elapsed = Date.now() - start;
|
||||
@@ -272,6 +309,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected,
|
||||
failedFiles: failedFiles.length,
|
||||
};
|
||||
@@ -309,10 +347,15 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
}
|
||||
|
||||
// Auto-embed (skip for large syncs — embedding calls OpenAI)
|
||||
let embedded = 0;
|
||||
if (!noEmbed && pagesAffected.length > 0 && pagesAffected.length <= 100) {
|
||||
try {
|
||||
const { runEmbed } = await import('./embed.ts');
|
||||
await runEmbed(engine, ['--slugs', ...pagesAffected]);
|
||||
// Before commit 2 lands: runEmbed is void. Best estimate is pagesAffected,
|
||||
// since runEmbed re-embeds every requested slug. Commit 2 sharpens this
|
||||
// with EmbedResult.embedded.
|
||||
embedded = pagesAffected.length;
|
||||
} catch { /* embedding is best-effort */ }
|
||||
} else if (noEmbed || totalChanges > 100) {
|
||||
console.log(`Text imported. Run 'gbrain embed --stale' to generate embeddings.`);
|
||||
@@ -327,6 +370,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated,
|
||||
embedded,
|
||||
pagesAffected,
|
||||
};
|
||||
}
|
||||
@@ -337,6 +381,33 @@ async function performFullSync(
|
||||
headCommit: string,
|
||||
opts: SyncOpts,
|
||||
): Promise<SyncResult> {
|
||||
// Dry-run: walk the repo, count syncable files, return without writing.
|
||||
// Fixes the silent-write-on-dry-run bug where performFullSync called
|
||||
// runImport unconditionally regardless of opts.dryRun.
|
||||
if (opts.dryRun) {
|
||||
const { collectMarkdownFiles } = await import('./import.ts');
|
||||
const allFiles = collectMarkdownFiles(repoPath);
|
||||
const syncableRelPaths = allFiles
|
||||
.map(abs => relative(repoPath, abs))
|
||||
.filter(rel => isSyncable(rel));
|
||||
console.log(
|
||||
`Full-sync dry run: ${syncableRelPaths.length} file(s) would be imported ` +
|
||||
`from ${repoPath} @ ${headCommit.slice(0, 8)}.`,
|
||||
);
|
||||
return {
|
||||
status: 'dry_run',
|
||||
fromCommit: null,
|
||||
toCommit: headCommit,
|
||||
added: syncableRelPaths.length,
|
||||
modified: 0,
|
||||
deleted: 0,
|
||||
renamed: 0,
|
||||
chunksCreated: 0,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
|
||||
console.log(`Running full import of ${repoPath}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
@@ -362,6 +433,7 @@ async function performFullSync(
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: result.chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
failedFiles: result.failures.length,
|
||||
};
|
||||
@@ -375,11 +447,15 @@ async function performFullSync(
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await engine.setConfig('sync.repo_path', repoPath);
|
||||
|
||||
// Full sync doesn't track pagesAffected, so fall back to embed --stale
|
||||
// Full sync doesn't track pagesAffected, so fall back to embed --stale.
|
||||
// Before commit 2: runEmbed is void; use result.imported as best estimate of
|
||||
// pages touched. Commit 2 sharpens this with real EmbedResult counts.
|
||||
let embedded = 0;
|
||||
if (!opts.noEmbed) {
|
||||
try {
|
||||
const { runEmbed } = await import('./embed.ts');
|
||||
await runEmbed(engine, ['--stale']);
|
||||
embedded = result.imported;
|
||||
} catch { /* embedding is best-effort */ }
|
||||
}
|
||||
|
||||
@@ -387,8 +463,12 @@ async function performFullSync(
|
||||
status: 'first_sync',
|
||||
fromCommit: null,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
added: result.imported,
|
||||
modified: 0,
|
||||
deleted: 0,
|
||||
renamed: 0,
|
||||
chunksCreated: result.chunksCreated,
|
||||
embedded,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
@@ -404,8 +484,39 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const skipFailed = args.includes('--skip-failed');
|
||||
const retryFailed = args.includes('--retry-failed');
|
||||
const syncAll = args.includes('--all');
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed };
|
||||
// Multi-repo: --all syncs all configured repos
|
||||
if (syncAll) {
|
||||
const { loadRepoConfigs } = await import('../core/multi-repo.ts');
|
||||
const repos = loadRepoConfigs();
|
||||
if (repos.length === 0) {
|
||||
console.log('No repos configured. Use `gbrain repos add <path>` first.');
|
||||
return;
|
||||
}
|
||||
for (const repo of repos) {
|
||||
if (repo.syncEnabled === false) {
|
||||
console.log(`Skipping disabled repo: ${repo.name}`);
|
||||
continue;
|
||||
}
|
||||
console.log(`\n--- Syncing repo: ${repo.name} (${repo.strategy}) ---`);
|
||||
const repoOpts: SyncOpts = {
|
||||
repoPath: repo.path,
|
||||
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
|
||||
strategy: repo.strategy,
|
||||
};
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
printSyncResult(result);
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${repo.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed, strategy: strategyArg };
|
||||
|
||||
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
|
||||
// flags so the sync picks them up as fresh work. The actual re-attempt
|
||||
@@ -460,10 +571,11 @@ function printSyncResult(result: SyncResult) {
|
||||
case 'synced':
|
||||
console.log(`Synced ${result.fromCommit?.slice(0, 8)}..${result.toCommit.slice(0, 8)}:`);
|
||||
console.log(` +${result.added} added, ~${result.modified} modified, -${result.deleted} deleted, R${result.renamed} renamed`);
|
||||
console.log(` ${result.chunksCreated} chunks created`);
|
||||
console.log(` ${result.chunksCreated} chunks created${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`);
|
||||
break;
|
||||
case 'first_sync':
|
||||
console.log(`First sync complete. Checkpoint: ${result.toCommit.slice(0, 8)}`);
|
||||
console.log(` ${result.added} file(s) imported, ${result.chunksCreated} chunks${result.embedded > 0 ? `, ${result.embedded} pages embedded` : ''}`);
|
||||
break;
|
||||
case 'dry_run':
|
||||
break; // already printed in performSync
|
||||
|
||||
@@ -56,11 +56,16 @@ export async function runUpgrade(args: string[]) {
|
||||
// Save old version for post-upgrade migration detection
|
||||
saveUpgradeState(oldVersion, newVersion);
|
||||
// Run post-upgrade feature discovery (reads migration files from the NEW binary).
|
||||
// Timeout bumped 30s → 300s because runPostUpgrade now tail-calls
|
||||
// apply-migrations, which can do long work (schema, smoke, host-rewrite,
|
||||
// autopilot install) on a v0.11.0→v0.11.1 jump. Codex H7.
|
||||
// Timeout bumped 300s → 1800s (30 min) in v0.15.2 because v0.12.0 graph
|
||||
// backfill on 50K+ brains regularly exceeded the old ceiling. The heartbeat
|
||||
// wiring added in v0.15.2 makes the long wait observable; a hard 300s
|
||||
// cap would still kill legit migrations mid-run. Override via
|
||||
// GBRAIN_POST_UPGRADE_TIMEOUT_MS env var.
|
||||
const postUpgradeTimeoutMs = Number(
|
||||
process.env.GBRAIN_POST_UPGRADE_TIMEOUT_MS || 1_800_000,
|
||||
);
|
||||
try {
|
||||
execSync('gbrain post-upgrade', { stdio: 'inherit', timeout: 300_000 });
|
||||
execSync('gbrain post-upgrade', { stdio: 'inherit', timeout: postUpgradeTimeoutMs });
|
||||
} catch (e) {
|
||||
// post-upgrade is best-effort, don't fail the upgrade. BUT leave a
|
||||
// trail so `gbrain doctor` can surface it and give the user a clear
|
||||
|
||||
@@ -0,0 +1,408 @@
|
||||
/**
|
||||
* Code Chunker — Tree-Sitter-Based Semantic Code Splitting
|
||||
*
|
||||
* Uses web-tree-sitter (WASM) to parse code files into AST, then extracts
|
||||
* semantic units (functions, classes, types, exports) as chunks.
|
||||
*
|
||||
* Each chunk includes a structured header with language, file path, line range,
|
||||
* and symbol name — so embeddings capture both context and code content.
|
||||
*
|
||||
* Supports: TypeScript, TSX, JavaScript, Python, Ruby, Go.
|
||||
* Falls back to recursive text chunker for unsupported languages.
|
||||
*/
|
||||
|
||||
import { chunkText as recursiveChunk } from './recursive.ts';
|
||||
|
||||
// Lazy-loaded tree-sitter module (v0.22.x API: Parser is default export)
|
||||
let Parser: typeof import('web-tree-sitter') | null = null;
|
||||
|
||||
async function getParser(): Promise<typeof import('web-tree-sitter')> {
|
||||
if (!Parser) {
|
||||
Parser = (await import('web-tree-sitter')).default || await import('web-tree-sitter');
|
||||
}
|
||||
return Parser;
|
||||
}
|
||||
|
||||
export type SupportedCodeLanguage = 'typescript' | 'tsx' | 'javascript' | 'python' | 'ruby' | 'go';
|
||||
|
||||
export interface CodeChunkMetadata {
|
||||
symbolName: string | null;
|
||||
symbolType: string;
|
||||
filePath: string;
|
||||
language: SupportedCodeLanguage;
|
||||
startLine: number;
|
||||
endLine: number;
|
||||
}
|
||||
|
||||
export interface CodeChunk {
|
||||
text: string;
|
||||
index: number;
|
||||
metadata: CodeChunkMetadata;
|
||||
}
|
||||
|
||||
export interface CodeChunkOptions {
|
||||
chunkSizeTokens?: number;
|
||||
largeChunkThresholdTokens?: number;
|
||||
fallbackChunkSizeWords?: number;
|
||||
fallbackOverlapWords?: number;
|
||||
}
|
||||
|
||||
const GRAMMAR_FILES: Record<SupportedCodeLanguage, string> = {
|
||||
typescript: 'tree-sitter-typescript.wasm',
|
||||
tsx: 'tree-sitter-tsx.wasm',
|
||||
javascript: 'tree-sitter-javascript.wasm',
|
||||
python: 'tree-sitter-python.wasm',
|
||||
ruby: 'tree-sitter-ruby.wasm',
|
||||
go: 'tree-sitter-go.wasm',
|
||||
};
|
||||
|
||||
const TOP_LEVEL_TYPES: Record<SupportedCodeLanguage, Set<string>> = {
|
||||
typescript: new Set([
|
||||
'function_declaration',
|
||||
'class_declaration',
|
||||
'abstract_class_declaration',
|
||||
'interface_declaration',
|
||||
'type_alias_declaration',
|
||||
'enum_declaration',
|
||||
'lexical_declaration',
|
||||
'variable_declaration',
|
||||
'export_statement',
|
||||
]),
|
||||
tsx: new Set([
|
||||
'function_declaration',
|
||||
'class_declaration',
|
||||
'interface_declaration',
|
||||
'type_alias_declaration',
|
||||
'enum_declaration',
|
||||
'lexical_declaration',
|
||||
'variable_declaration',
|
||||
'export_statement',
|
||||
]),
|
||||
javascript: new Set([
|
||||
'function_declaration',
|
||||
'class_declaration',
|
||||
'lexical_declaration',
|
||||
'variable_declaration',
|
||||
'export_statement',
|
||||
]),
|
||||
python: new Set([
|
||||
'function_definition',
|
||||
'class_definition',
|
||||
'import_statement',
|
||||
'import_from_statement',
|
||||
'assignment',
|
||||
]),
|
||||
ruby: new Set([
|
||||
'class',
|
||||
'module',
|
||||
'method',
|
||||
'singleton_method',
|
||||
'assignment',
|
||||
]),
|
||||
go: new Set([
|
||||
'function_declaration',
|
||||
'method_declaration',
|
||||
'type_declaration',
|
||||
'const_declaration',
|
||||
'var_declaration',
|
||||
'import_declaration',
|
||||
]),
|
||||
};
|
||||
|
||||
const BODY_NODE_TYPES = new Set([
|
||||
'statement_block',
|
||||
'block',
|
||||
'class_body',
|
||||
'module_body',
|
||||
'body_statement',
|
||||
'body',
|
||||
]);
|
||||
|
||||
let initDone = false;
|
||||
let initPromise: Promise<void> | null = null;
|
||||
const languageCache = new Map<SupportedCodeLanguage, any>();
|
||||
|
||||
// ---------- Public API ----------
|
||||
|
||||
export function detectCodeLanguage(filePath: string): SupportedCodeLanguage | null {
|
||||
const lower = filePath.toLowerCase();
|
||||
if (lower.endsWith('.tsx')) return 'tsx';
|
||||
if (lower.endsWith('.ts')) return 'typescript';
|
||||
if (lower.endsWith('.js') || lower.endsWith('.jsx') || lower.endsWith('.mjs') || lower.endsWith('.cjs')) return 'javascript';
|
||||
if (lower.endsWith('.py')) return 'python';
|
||||
if (lower.endsWith('.rb')) return 'ruby';
|
||||
if (lower.endsWith('.go')) return 'go';
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function chunkCodeText(
|
||||
source: string,
|
||||
filePath: string,
|
||||
opts: CodeChunkOptions = {},
|
||||
): Promise<CodeChunk[]> {
|
||||
const language = detectCodeLanguage(filePath);
|
||||
if (!language) {
|
||||
return fallbackChunks(source, filePath, 'javascript', opts);
|
||||
}
|
||||
|
||||
if (!source.trim()) return [];
|
||||
|
||||
const largeThreshold = opts.largeChunkThresholdTokens ?? 1000;
|
||||
const chunkTarget = opts.chunkSizeTokens ?? 300;
|
||||
|
||||
try {
|
||||
await ensureInit();
|
||||
const P = await getParser();
|
||||
const parser = new (P as any)();
|
||||
const grammar = await loadLanguage(language);
|
||||
parser.setLanguage(grammar);
|
||||
|
||||
const tree = parser.parse(source);
|
||||
if (!tree) {
|
||||
parser.delete();
|
||||
return fallbackChunks(source, filePath, language, opts);
|
||||
}
|
||||
|
||||
const root = tree.rootNode;
|
||||
const topLevelTypes = TOP_LEVEL_TYPES[language];
|
||||
const semanticNodes = root.namedChildren.filter((n: any) => topLevelTypes.has(n.type));
|
||||
|
||||
if (semanticNodes.length === 0) {
|
||||
tree.delete();
|
||||
parser.delete();
|
||||
return fallbackChunks(source, filePath, language, opts);
|
||||
}
|
||||
|
||||
const chunks: CodeChunk[] = [];
|
||||
for (const node of semanticNodes) {
|
||||
const symbolName = extractSymbolName(node);
|
||||
const symbolType = normalizeSymbolType(node.type);
|
||||
const nodeText = source.slice(node.startIndex, node.endIndex).trim();
|
||||
if (!nodeText) continue;
|
||||
|
||||
if (estimateTokens(nodeText) <= largeThreshold) {
|
||||
chunks.push(buildChunk({
|
||||
body: nodeText, filePath, language, symbolName, symbolType,
|
||||
startLine: node.startPosition.row + 1,
|
||||
endLine: node.endPosition.row + 1,
|
||||
index: chunks.length,
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Split very large nodes at nested block boundaries
|
||||
const subRanges = splitLargeNode(node, source, chunkTarget);
|
||||
if (subRanges.length === 0) {
|
||||
chunks.push(buildChunk({
|
||||
body: nodeText, filePath, language, symbolName, symbolType,
|
||||
startLine: node.startPosition.row + 1,
|
||||
endLine: node.endPosition.row + 1,
|
||||
index: chunks.length,
|
||||
}));
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const range of subRanges) {
|
||||
const body = source.slice(range.startIndex, range.endIndex).trim();
|
||||
if (!body) continue;
|
||||
chunks.push(buildChunk({
|
||||
body, filePath, language, symbolName, symbolType,
|
||||
startLine: range.startLine, endLine: range.endLine,
|
||||
index: chunks.length,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
tree.delete();
|
||||
parser.delete();
|
||||
|
||||
return chunks.length > 0 ? chunks : fallbackChunks(source, filePath, language, opts);
|
||||
} catch {
|
||||
return fallbackChunks(source, filePath, language, opts);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Internals ----------
|
||||
|
||||
function fallbackChunks(
|
||||
source: string,
|
||||
filePath: string,
|
||||
language: SupportedCodeLanguage,
|
||||
opts: CodeChunkOptions,
|
||||
): CodeChunk[] {
|
||||
const size = opts.fallbackChunkSizeWords ?? 300;
|
||||
const overlap = opts.fallbackOverlapWords ?? 50;
|
||||
return recursiveChunk(source, { chunkSize: size, chunkOverlap: overlap }).map((chunk, index) =>
|
||||
buildChunk({
|
||||
body: chunk.text, filePath, language,
|
||||
symbolName: null, symbolType: 'module',
|
||||
startLine: 1, endLine: countLines(chunk.text),
|
||||
index,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function buildChunk(input: {
|
||||
body: string;
|
||||
filePath: string;
|
||||
language: SupportedCodeLanguage;
|
||||
symbolName: string | null;
|
||||
symbolType: string;
|
||||
startLine: number;
|
||||
endLine: number;
|
||||
index: number;
|
||||
}): CodeChunk {
|
||||
const symbol = input.symbolName ? `${input.symbolType} ${input.symbolName}` : input.symbolType;
|
||||
const header = `[${displayLang(input.language)}] ${input.filePath}:${input.startLine}-${input.endLine} ${symbol}`;
|
||||
return {
|
||||
index: input.index,
|
||||
text: `${header}\n\n${input.body}`,
|
||||
metadata: {
|
||||
symbolName: input.symbolName,
|
||||
symbolType: input.symbolType,
|
||||
filePath: input.filePath,
|
||||
language: input.language,
|
||||
startLine: input.startLine,
|
||||
endLine: input.endLine,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
interface SplitRange {
|
||||
startIndex: number;
|
||||
endIndex: number;
|
||||
startLine: number;
|
||||
endLine: number;
|
||||
}
|
||||
|
||||
function splitLargeNode(node: any, source: string, chunkTarget: number): SplitRange[] {
|
||||
const body =
|
||||
node.childForFieldName('body') ||
|
||||
node.namedChildren.find((c: any) => BODY_NODE_TYPES.has(c.type)) ||
|
||||
null;
|
||||
|
||||
if (!body || body.namedChildren.length < 2) return [];
|
||||
|
||||
const children = body.namedChildren.filter((c: any) => !c.isExtra);
|
||||
if (children.length < 2) return [];
|
||||
|
||||
const ranges: SplitRange[] = [];
|
||||
let curStart = children[0].startIndex;
|
||||
let curStartLine = children[0].startPosition.row + 1;
|
||||
let curEnd = children[0].endIndex;
|
||||
let curEndLine = children[0].endPosition.row + 1;
|
||||
let curTokens = estimateTokens(source.slice(curStart, curEnd));
|
||||
|
||||
for (let i = 1; i < children.length; i++) {
|
||||
const child = children[i];
|
||||
const childTokens = estimateTokens(source.slice(child.startIndex, child.endIndex));
|
||||
|
||||
if (curTokens + childTokens > Math.ceil(chunkTarget * 1.5)) {
|
||||
ranges.push({ startIndex: curStart, endIndex: curEnd, startLine: curStartLine, endLine: curEndLine });
|
||||
curStart = child.startIndex;
|
||||
curStartLine = child.startPosition.row + 1;
|
||||
curEnd = child.endIndex;
|
||||
curEndLine = child.endPosition.row + 1;
|
||||
curTokens = childTokens;
|
||||
} else {
|
||||
curEnd = child.endIndex;
|
||||
curEndLine = child.endPosition.row + 1;
|
||||
curTokens += childTokens;
|
||||
}
|
||||
}
|
||||
ranges.push({ startIndex: curStart, endIndex: curEnd, startLine: curStartLine, endLine: curEndLine });
|
||||
return ranges;
|
||||
}
|
||||
|
||||
function extractSymbolName(node: any): string | null {
|
||||
const directName = node.childForFieldName('name');
|
||||
if (directName?.text?.trim()) return sanitize(directName.text);
|
||||
|
||||
const declaration = node.childForFieldName('declaration');
|
||||
if (declaration) {
|
||||
const nested = extractSymbolName(declaration);
|
||||
if (nested) return nested;
|
||||
}
|
||||
|
||||
for (const child of node.namedChildren) {
|
||||
if (child.type.endsWith('identifier') || child.type === 'constant') {
|
||||
const v = sanitize(child.text);
|
||||
if (v) return v;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function normalizeSymbolType(type: string): string {
|
||||
if (type.includes('function') || type === 'method' || type === 'singleton_method') return 'function';
|
||||
if (type.includes('class')) return 'class';
|
||||
if (type.includes('interface')) return 'interface';
|
||||
if (type.includes('type_alias')) return 'type';
|
||||
if (type.includes('enum')) return 'enum';
|
||||
if (type.includes('module')) return 'module';
|
||||
if (type.includes('import')) return 'import';
|
||||
return type.replace(/_/g, ' ');
|
||||
}
|
||||
|
||||
function sanitize(name: string): string {
|
||||
return name.replace(/[\n\r\t]+/g, ' ').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function estimateTokens(text: string): number {
|
||||
return Math.max(1, Math.ceil(text.length / 4));
|
||||
}
|
||||
|
||||
function displayLang(lang: SupportedCodeLanguage): string {
|
||||
const map: Record<SupportedCodeLanguage, string> = {
|
||||
typescript: 'TypeScript', tsx: 'TSX', javascript: 'JavaScript',
|
||||
python: 'Python', ruby: 'Ruby', go: 'Go',
|
||||
};
|
||||
return map[lang];
|
||||
}
|
||||
|
||||
function countLines(text: string): number {
|
||||
return text ? text.split('\n').length : 0;
|
||||
}
|
||||
|
||||
// ---------- Tree-sitter init ----------
|
||||
|
||||
async function ensureInit(): Promise<void> {
|
||||
if (initDone) return;
|
||||
if (!initPromise) {
|
||||
initPromise = (async () => {
|
||||
const P = await getParser();
|
||||
// v0.22.x: init takes locateFile for the WASM module
|
||||
const wasmPath = new URL('../../../node_modules/web-tree-sitter/tree-sitter.wasm', import.meta.url);
|
||||
let resolved: string;
|
||||
try {
|
||||
const { fileURLToPath } = await import('url');
|
||||
resolved = fileURLToPath(wasmPath);
|
||||
} catch {
|
||||
resolved = wasmPath.pathname;
|
||||
}
|
||||
await (P as any).init({ locateFile: () => resolved });
|
||||
initDone = true;
|
||||
})();
|
||||
}
|
||||
await initPromise;
|
||||
}
|
||||
|
||||
async function loadLanguage(language: SupportedCodeLanguage): Promise<any> {
|
||||
if (languageCache.has(language)) return languageCache.get(language);
|
||||
|
||||
const P = await getParser();
|
||||
const grammarUrl = new URL(
|
||||
`../../../node_modules/tree-sitter-wasms/out/${GRAMMAR_FILES[language]}`,
|
||||
import.meta.url,
|
||||
);
|
||||
let resolved: string;
|
||||
try {
|
||||
const { fileURLToPath } = await import('url');
|
||||
resolved = fileURLToPath(grammarUrl);
|
||||
} catch {
|
||||
resolved = grammarUrl.pathname;
|
||||
}
|
||||
const lang = await (P as any).Language.load(resolved);
|
||||
languageCache.set(language, lang);
|
||||
return lang;
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* Global CLI flags parsed before command dispatch.
|
||||
*
|
||||
* Keeping this separate from per-command flag parsing so that
|
||||
* `gbrain --progress-json doctor` works: the global flag is stripped
|
||||
* before cli.ts looks at argv[0] for the subcommand.
|
||||
*
|
||||
* Threading: every command handler receives a resolved CliOptions object.
|
||||
* Shared-operation handlers see the same values via OperationContext.cliOpts.
|
||||
*/
|
||||
|
||||
import type { ProgressOptions } from './progress.ts';
|
||||
|
||||
export interface CliOptions {
|
||||
quiet: boolean;
|
||||
progressJson: boolean;
|
||||
progressInterval: number; // ms
|
||||
}
|
||||
|
||||
export const DEFAULT_CLI_OPTIONS: CliOptions = {
|
||||
quiet: false,
|
||||
progressJson: false,
|
||||
progressInterval: 1000,
|
||||
};
|
||||
|
||||
/**
|
||||
* Parse recognized global flags from the front / anywhere in argv and return
|
||||
* the resolved options plus the remaining argv (with global flags stripped).
|
||||
*
|
||||
* Recognized:
|
||||
* --quiet
|
||||
* --progress-json
|
||||
* --progress-interval=<ms>
|
||||
* --progress-interval <ms> (space-separated form)
|
||||
*
|
||||
* Unknown flags are passed through unchanged — per-command parsers see them.
|
||||
*/
|
||||
export function parseGlobalFlags(argv: string[]): { cliOpts: CliOptions; rest: string[] } {
|
||||
const cliOpts: CliOptions = { ...DEFAULT_CLI_OPTIONS };
|
||||
const rest: string[] = [];
|
||||
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--quiet') {
|
||||
cliOpts.quiet = true;
|
||||
continue;
|
||||
}
|
||||
if (a === '--progress-json') {
|
||||
cliOpts.progressJson = true;
|
||||
continue;
|
||||
}
|
||||
if (a === '--progress-interval' && i + 1 < argv.length) {
|
||||
const next = argv[i + 1];
|
||||
const parsed = parseInterval(next);
|
||||
if (parsed !== null) {
|
||||
cliOpts.progressInterval = parsed;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
// not a number — let per-command parser handle; pass through
|
||||
rest.push(a);
|
||||
continue;
|
||||
}
|
||||
if (a.startsWith('--progress-interval=')) {
|
||||
const val = a.slice('--progress-interval='.length);
|
||||
const parsed = parseInterval(val);
|
||||
if (parsed !== null) {
|
||||
cliOpts.progressInterval = parsed;
|
||||
continue;
|
||||
}
|
||||
rest.push(a);
|
||||
continue;
|
||||
}
|
||||
rest.push(a);
|
||||
}
|
||||
|
||||
return { cliOpts, rest };
|
||||
}
|
||||
|
||||
function parseInterval(s: string): number | null {
|
||||
const n = Number(s);
|
||||
if (!Number.isFinite(n) || n < 0) return null;
|
||||
return Math.floor(n);
|
||||
}
|
||||
|
||||
/**
|
||||
* Map resolved CliOptions to ProgressOptions for createProgress().
|
||||
*
|
||||
* Mode resolution:
|
||||
* --quiet → 'quiet'
|
||||
* --progress-json → 'json'
|
||||
* otherwise → 'auto' (TTY: human-\r, non-TTY: human-plain)
|
||||
*
|
||||
* Agents that want structured events on a non-TTY stream must pass
|
||||
* --progress-json explicitly. Non-TTY default is plain human lines so
|
||||
* shell pipelines don't suddenly see JSON noise.
|
||||
*/
|
||||
export function cliOptsToProgressOptions(cliOpts: CliOptions): ProgressOptions {
|
||||
if (cliOpts.quiet) return { mode: 'quiet' };
|
||||
if (cliOpts.progressJson) return { mode: 'json', minIntervalMs: cliOpts.progressInterval };
|
||||
return { mode: 'auto', minIntervalMs: cliOpts.progressInterval };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Module-level singleton (set once by cli.ts after parsing global flags; read
|
||||
// by any bulk command that wants to construct a reporter). Same pattern as
|
||||
// Commander's `program.opts()`. Also threaded into OperationContext for
|
||||
// shared ops that run under the MCP server (which sets its own defaults).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let activeCliOptions: CliOptions = { ...DEFAULT_CLI_OPTIONS };
|
||||
|
||||
export function setCliOptions(opts: CliOptions): void {
|
||||
activeCliOptions = { ...opts };
|
||||
}
|
||||
|
||||
export function getCliOptions(): CliOptions {
|
||||
return activeCliOptions;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset singleton to defaults. Only used by tests.
|
||||
*/
|
||||
export function _resetCliOptionsForTest(): void {
|
||||
activeCliOptions = { ...DEFAULT_CLI_OPTIONS };
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the global-flag suffix to append to child `gbrain …` subprocess
|
||||
* commands so children inherit the parent's progress-mode.
|
||||
*
|
||||
* Returns a string ready to concat onto an execSync command string, with
|
||||
* a leading space when non-empty. E.g. " --progress-json --quiet".
|
||||
*
|
||||
* Empty string when nothing to propagate (so the child's behavior is
|
||||
* unchanged for the common no-flag case).
|
||||
*/
|
||||
export function childGlobalFlags(cliOpts?: CliOptions): string {
|
||||
const opts = cliOpts ?? activeCliOptions;
|
||||
const parts: string[] = [];
|
||||
if (opts.quiet) parts.push('--quiet');
|
||||
if (opts.progressJson) parts.push('--progress-json');
|
||||
if (opts.progressInterval !== DEFAULT_CLI_OPTIONS.progressInterval) {
|
||||
parts.push(`--progress-interval=${opts.progressInterval}`);
|
||||
}
|
||||
return parts.length > 0 ? ' ' + parts.join(' ') : '';
|
||||
}
|
||||
@@ -27,8 +27,23 @@ export interface GBrainConfig {
|
||||
engine: 'postgres' | 'pglite';
|
||||
database_url?: string;
|
||||
database_path?: string;
|
||||
repos?: Array<{
|
||||
path: string;
|
||||
name: string;
|
||||
strategy: 'markdown' | 'code' | 'auto';
|
||||
include?: string[];
|
||||
exclude?: string[];
|
||||
syncEnabled?: boolean;
|
||||
}>;
|
||||
openai_api_key?: string;
|
||||
anthropic_api_key?: string;
|
||||
/**
|
||||
* Optional storage backend config (S3/Supabase/local). Shape matches
|
||||
* `StorageConfig` in `./storage.ts`. Typed as `unknown` here to avoid
|
||||
* a cyclic import; callers pass this through `createStorage()` which
|
||||
* validates the shape at runtime.
|
||||
*/
|
||||
storage?: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,817 @@
|
||||
/**
|
||||
* src/core/cycle.ts — The brain maintenance cycle primitive.
|
||||
*
|
||||
* Composes lint, backlinks, sync, extract, embed, and orphans into
|
||||
* one honest unit of work. Called from:
|
||||
* - `gbrain dream` (CLI alias; one-shot cron-triggered cycle)
|
||||
* - `gbrain autopilot` (daemon; scheduled on an interval)
|
||||
* - Minions `autopilot-cycle` handler (durable queue; retry + observability)
|
||||
*
|
||||
* All three converge on runCycle() so there's one source of truth for
|
||||
* what "overnight maintenance" means.
|
||||
*
|
||||
* PHASE ORDER (semantically driven — fix files first, then index):
|
||||
*
|
||||
* ┌───────────────────────────────────────────────────────────┐
|
||||
* │ Phase 1: lint --fix (filesystem writes, no DB) │
|
||||
* │ Phase 2: backlinks --fix (filesystem writes, no DB) │
|
||||
* │ Phase 3: sync (DB picks up phases 1+2) │
|
||||
* │ Phase 4: extract (DB picks up links from sync) │
|
||||
* │ Phase 5: embed --stale (DB writes) │
|
||||
* │ Phase 6: orphans (DB read, report only) │
|
||||
* └───────────────────────────────────────────────────────────┘
|
||||
*
|
||||
* COORDINATION:
|
||||
*
|
||||
* Postgres: a row in gbrain_cycle_locks with a TTL (30 min). Refreshed
|
||||
* between phases via yieldBetweenPhases. Works through PgBouncer
|
||||
* transaction pooling (session-scoped pg_try_advisory_lock does not).
|
||||
*
|
||||
* PGLite / engine=null: a file lock at ~/.gbrain/cycle.lock holding
|
||||
* the PID + mtime. Same 30-min TTL semantics.
|
||||
*
|
||||
* LOCK-SKIP:
|
||||
*
|
||||
* Filesystem-only or read-only phase selections (lint, backlinks,
|
||||
* orphans) skip the lock. Only DB-write phases (sync, extract, embed)
|
||||
* trigger lock acquisition.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { homedir, hostname } from 'os';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { createProgress, type ProgressReporter } from './progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────
|
||||
|
||||
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'extract' | 'embed' | 'orphans';
|
||||
|
||||
export const ALL_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'extract',
|
||||
'embed',
|
||||
'orphans',
|
||||
];
|
||||
|
||||
/**
|
||||
* Phases that mutate state (filesystem or DB) and therefore should
|
||||
* coordinate via the cycle lock. Only orphans is truly read-only
|
||||
* and skips the lock.
|
||||
*/
|
||||
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
|
||||
'lint',
|
||||
'backlinks',
|
||||
'sync',
|
||||
'extract',
|
||||
'embed',
|
||||
]);
|
||||
|
||||
export type PhaseStatus = 'ok' | 'warn' | 'fail' | 'skipped';
|
||||
|
||||
export interface PhaseError {
|
||||
/** Error class for machine branching — e.g., 'DatabaseConnection', 'Timeout', 'LLMError', 'FilesystemError', 'InternalError'. */
|
||||
class: string;
|
||||
/** System error code or short identifier, e.g., 'ECONNREFUSED', 'ETIMEDOUT', 'UNKNOWN'. */
|
||||
code: string;
|
||||
/** Human-readable single-line message. */
|
||||
message: string;
|
||||
/** Optional suggestion of what to try next. */
|
||||
hint?: string;
|
||||
/** Optional link to a troubleshooting doc. */
|
||||
docs_url?: string;
|
||||
}
|
||||
|
||||
export interface PhaseResult {
|
||||
phase: CyclePhase;
|
||||
status: PhaseStatus;
|
||||
duration_ms: number;
|
||||
summary: string;
|
||||
details: Record<string, unknown>;
|
||||
error?: PhaseError;
|
||||
}
|
||||
|
||||
export type CycleStatus = 'ok' | 'clean' | 'partial' | 'skipped' | 'failed';
|
||||
|
||||
export interface CycleReport {
|
||||
/** Additive schema. Bumped on breaking changes. */
|
||||
schema_version: '1';
|
||||
timestamp: string;
|
||||
duration_ms: number;
|
||||
/**
|
||||
* Overall status derived from phase results:
|
||||
* - 'clean' : ran successfully, zero fixes/writes across every phase
|
||||
* - 'ok' : ran successfully, some work was done
|
||||
* - 'partial' : at least one phase warned or failed, others ran
|
||||
* - 'skipped' : cycle did not run (lock held by another holder)
|
||||
* - 'failed' : lock acquired but all attempted phases failed
|
||||
*/
|
||||
status: CycleStatus;
|
||||
/** Present when status = 'skipped'. E.g., 'cycle_already_running' or 'no_database'. */
|
||||
reason?: string;
|
||||
brain_dir: string | null;
|
||||
phases: PhaseResult[];
|
||||
totals: {
|
||||
lint_fixes: number;
|
||||
backlinks_added: number;
|
||||
pages_synced: number;
|
||||
pages_extracted: number;
|
||||
pages_embedded: number;
|
||||
orphans_found: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface CycleOpts {
|
||||
/** If true, no writes to filesystem or DB. All phases honor this. */
|
||||
dryRun?: boolean;
|
||||
/** Defaults to ALL_PHASES. Pass a subset for --phase lint etc. */
|
||||
phases?: CyclePhase[];
|
||||
/** Brain directory (git repo). Required for filesystem phases. */
|
||||
brainDir: string;
|
||||
/** Whether sync should run `git pull`. Default false (cron-safe). */
|
||||
pull?: boolean;
|
||||
/**
|
||||
* Called between phases AND before runCycle returns. Awaited even
|
||||
* after phase failure. Hook exceptions are logged, never fatal.
|
||||
* Minions handlers pass a function that yields + renews the job lock
|
||||
* + refreshes the cycle-lock-table TTL.
|
||||
*/
|
||||
yieldBetweenPhases?: () => Promise<void>;
|
||||
}
|
||||
|
||||
// ─── Lock primitives ───────────────────────────────────────────────
|
||||
|
||||
const CYCLE_LOCK_ID = 'gbrain-cycle';
|
||||
const LOCK_TTL_MS = 30 * 60 * 1000; // 30 minutes
|
||||
const LOCK_FILE_PATH_DEFAULT = join(homedir(), '.gbrain', 'cycle.lock');
|
||||
|
||||
interface LockHandle {
|
||||
release: () => Promise<void>;
|
||||
refresh: () => Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire the Postgres-backed cycle lock.
|
||||
* Returns a LockHandle on success, or null if another live holder has it.
|
||||
*
|
||||
* Uses INSERT ... ON CONFLICT (id) DO UPDATE ... WHERE ttl_expires_at < NOW()
|
||||
* RETURNING *. An empty RETURNING means the existing row is still live.
|
||||
* Crashed holders auto-release: when their TTL expires, the next
|
||||
* acquirer's UPDATE branch fires and takes over.
|
||||
*/
|
||||
async function acquirePostgresLock(engine: BrainEngine): Promise<LockHandle | null> {
|
||||
const pid = process.pid;
|
||||
const host = hostname();
|
||||
// Engine-agnostic: BrainEngine exposes findOrphanPages etc., but not raw SQL.
|
||||
// We reach through the engine's internal connection for this lock operation.
|
||||
// Both engines expose `sql` (postgres-js tag) or `db.query` (PGLite).
|
||||
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
|
||||
const maybePGLite = engine as unknown as { db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> } };
|
||||
|
||||
if (engine.kind === 'postgres' && maybePG.sql) {
|
||||
const sql = maybePG.sql as any;
|
||||
const rows: Array<{ id: string }> = await sql`
|
||||
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES (${CYCLE_LOCK_ID}, ${pid}, ${host}, NOW(), NOW() + INTERVAL '30 minutes')
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = ${pid},
|
||||
holder_host = ${host},
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + INTERVAL '30 minutes'
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id
|
||||
`;
|
||||
if (rows.length === 0) return null; // live holder
|
||||
return {
|
||||
refresh: async () => {
|
||||
await sql`
|
||||
UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + INTERVAL '30 minutes'
|
||||
WHERE id = ${CYCLE_LOCK_ID} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
release: async () => {
|
||||
await sql`
|
||||
DELETE FROM gbrain_cycle_locks
|
||||
WHERE id = ${CYCLE_LOCK_ID} AND holder_pid = ${pid}
|
||||
`;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (engine.kind === 'pglite' && maybePGLite.db) {
|
||||
// PGLite is single-writer; the DB row is belt-and-braces on top of the
|
||||
// file lock. Callers always hold the file lock first, so this UPSERT
|
||||
// is race-free against other processes.
|
||||
const db = maybePGLite.db;
|
||||
const { rows } = await db.query(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES ($1, $2, $3, NOW(), NOW() + INTERVAL '30 minutes')
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
SET holder_pid = $2,
|
||||
holder_host = $3,
|
||||
acquired_at = NOW(),
|
||||
ttl_expires_at = NOW() + INTERVAL '30 minutes'
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
RETURNING id`,
|
||||
[CYCLE_LOCK_ID, pid, host],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return {
|
||||
refresh: async () => {
|
||||
await db.query(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + INTERVAL '30 minutes'
|
||||
WHERE id = $1 AND holder_pid = $2`,
|
||||
[CYCLE_LOCK_ID, pid],
|
||||
);
|
||||
},
|
||||
release: async () => {
|
||||
await db.query(
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
|
||||
[CYCLE_LOCK_ID, pid],
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
throw new Error(`Unknown engine kind: ${engine.kind}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire the file-based cycle lock (used when engine === null).
|
||||
* Returns a LockHandle on success, or null if a live holder has it.
|
||||
*
|
||||
* The file contains `{pid}\n{iso-timestamp}`. Staleness = mtime older
|
||||
* than LOCK_TTL_MS OR the PID is no longer alive on this host.
|
||||
*/
|
||||
function acquireFileLock(lockPath = LOCK_FILE_PATH_DEFAULT): LockHandle | null {
|
||||
mkdirSync(join(lockPath, '..'), { recursive: true });
|
||||
const pid = process.pid;
|
||||
|
||||
if (existsSync(lockPath)) {
|
||||
// Check TTL.
|
||||
try {
|
||||
const st = statSync(lockPath);
|
||||
const ageMs = Date.now() - st.mtimeMs;
|
||||
const existingContent = readFileSync(lockPath, 'utf-8').trim();
|
||||
const existingPid = parseInt(existingContent.split('\n')[0] || '0', 10);
|
||||
|
||||
// PID liveness check (same host only). kill(pid, 0) distinguishes:
|
||||
// - success → process exists, caller can signal it
|
||||
// - error ESRCH → no such process (truly dead)
|
||||
// - error EPERM → process exists but caller can't signal it
|
||||
// (e.g., PID 1/init on unix) → still alive
|
||||
// Any error code OTHER than ESRCH means the PID is alive.
|
||||
let pidAlive = false;
|
||||
if (existingPid > 0 && existingPid !== pid) {
|
||||
try {
|
||||
process.kill(existingPid, 0);
|
||||
pidAlive = true;
|
||||
} catch (e) {
|
||||
const code = (e as NodeJS.ErrnoException).code;
|
||||
pidAlive = code !== 'ESRCH';
|
||||
}
|
||||
} else if (existingPid === pid) {
|
||||
// Our own stale lock (same pid, previous run) — treat as stale.
|
||||
pidAlive = false;
|
||||
}
|
||||
|
||||
if (pidAlive && ageMs < LOCK_TTL_MS) {
|
||||
return null; // live holder
|
||||
}
|
||||
// Stale lock — fall through to overwrite.
|
||||
} catch {
|
||||
// Any read/stat error: treat as stale.
|
||||
}
|
||||
}
|
||||
|
||||
writeFileSync(lockPath, `${pid}\n${new Date().toISOString()}\n`);
|
||||
|
||||
return {
|
||||
refresh: async () => {
|
||||
try {
|
||||
writeFileSync(lockPath, `${pid}\n${new Date().toISOString()}\n`);
|
||||
} catch {
|
||||
/* non-fatal — a next-run stale check will notice */
|
||||
}
|
||||
},
|
||||
release: async () => {
|
||||
try {
|
||||
const content = readFileSync(lockPath, 'utf-8').trim();
|
||||
const heldPid = parseInt(content.split('\n')[0] || '0', 10);
|
||||
if (heldPid === pid) unlinkSync(lockPath);
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Helpers ───────────────────────────────────────────────────────
|
||||
|
||||
function makeErrorFromException(e: unknown, fallbackClass = 'InternalError'): PhaseError {
|
||||
const err = e instanceof Error ? e : new Error(String(e));
|
||||
// Node errors often have .code (e.g., 'ECONNREFUSED').
|
||||
const code = (err as NodeJS.ErrnoException).code || 'UNKNOWN';
|
||||
let className = fallbackClass;
|
||||
if (code === 'ECONNREFUSED' || code === 'ENOTFOUND') className = 'DatabaseConnection';
|
||||
if (code === 'ETIMEDOUT') className = 'Timeout';
|
||||
if (/OpenAI|embed/i.test(err.message)) className = 'LLMError';
|
||||
if (/ENOENT|EACCES|EISDIR|ENOTDIR/.test(code)) className = 'FilesystemError';
|
||||
return {
|
||||
class: className,
|
||||
code,
|
||||
message: err.message.slice(0, 200),
|
||||
};
|
||||
}
|
||||
|
||||
async function timePhase<T>(fn: () => Promise<T>): Promise<{ result: T; duration_ms: number }> {
|
||||
const start = performance.now();
|
||||
const result = await fn();
|
||||
return { result, duration_ms: Math.round(performance.now() - start) };
|
||||
}
|
||||
|
||||
async function safeYield(hook?: () => Promise<void>) {
|
||||
if (!hook) return;
|
||||
try {
|
||||
await hook();
|
||||
} catch (e) {
|
||||
console.warn(`[cycle] yieldBetweenPhases hook error (non-fatal): ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Phase runners ─────────────────────────────────────────────────
|
||||
|
||||
async function runPhaseLint(brainDir: string, dryRun: boolean): Promise<PhaseResult> {
|
||||
try {
|
||||
const { runLintCore } = await import('../commands/lint.ts');
|
||||
const result = await runLintCore({ target: brainDir, fix: true, dryRun });
|
||||
const issues = result.total_issues ?? 0;
|
||||
const fixed = result.total_fixed ?? 0;
|
||||
const remaining = Math.max(0, issues - fixed);
|
||||
// 'ok' when nothing noteworthy remains:
|
||||
// - no issues at all, or
|
||||
// - non-dry-run and everything fixable was fixed.
|
||||
// 'warn' when issues remain after the run.
|
||||
const status: PhaseStatus =
|
||||
issues === 0 || (!dryRun && remaining === 0) ? 'ok' : 'warn';
|
||||
return {
|
||||
phase: 'lint',
|
||||
status,
|
||||
duration_ms: 0, // set by caller
|
||||
summary: dryRun
|
||||
? `${issues} issue(s) found (dry-run, no writes)`
|
||||
: `${fixed} fix(es) applied, ${remaining} remaining`,
|
||||
details: { issues, fixed, pages_scanned: result.pages_scanned, dryRun },
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'lint',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'lint phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseBacklinks(brainDir: string, dryRun: boolean): Promise<PhaseResult> {
|
||||
try {
|
||||
// Library function path — the v0.15 backlinks.ts exports
|
||||
// runBacklinksCore when --fix is requested.
|
||||
const { runBacklinksCore } = await import('../commands/backlinks.ts');
|
||||
const result = await runBacklinksCore({
|
||||
action: 'fix',
|
||||
dir: brainDir,
|
||||
dryRun,
|
||||
});
|
||||
const gaps = result.gaps_found ?? 0;
|
||||
const added = result.fixed ?? 0;
|
||||
const remaining = Math.max(0, gaps - added);
|
||||
const status: PhaseStatus =
|
||||
gaps === 0 || (!dryRun && remaining === 0) ? 'ok' : 'warn';
|
||||
return {
|
||||
phase: 'backlinks',
|
||||
status,
|
||||
duration_ms: 0,
|
||||
summary: dryRun
|
||||
? `${gaps} missing back-link(s) (dry-run)`
|
||||
: `${added} back-link(s) added, ${remaining} remaining`,
|
||||
details: { gaps, added, pages_affected: result.pages_affected, dryRun },
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'backlinks',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'backlinks phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseSync(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
pull: boolean,
|
||||
): Promise<PhaseResult> {
|
||||
try {
|
||||
const { performSync } = await import('../commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
repoPath: brainDir,
|
||||
dryRun,
|
||||
noPull: !pull,
|
||||
noEmbed: true, // embed is a separate phase
|
||||
});
|
||||
const syncedCount = result.added + result.modified;
|
||||
return {
|
||||
phase: 'sync',
|
||||
status: result.status === 'blocked_by_failures' ? 'warn' : 'ok',
|
||||
duration_ms: 0,
|
||||
summary: dryRun
|
||||
? `${syncedCount} page(s) would sync, ${result.deleted} would delete`
|
||||
: `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`,
|
||||
details: {
|
||||
added: result.added,
|
||||
modified: result.modified,
|
||||
deleted: result.deleted,
|
||||
renamed: result.renamed,
|
||||
chunksCreated: result.chunksCreated,
|
||||
failedFiles: result.failedFiles ?? 0,
|
||||
syncStatus: result.status,
|
||||
dryRun,
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'sync',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'sync phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseExtract(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
): Promise<PhaseResult> {
|
||||
try {
|
||||
const { runExtractCore } = await import('../commands/extract.ts');
|
||||
// Extract is read-mostly against the filesystem + write to links table.
|
||||
// Honor dryRun by skipping with a 'skipped' entry: extract doesn't have
|
||||
// a clean dry-run mode today and runCycle should be honest about it.
|
||||
if (dryRun) {
|
||||
return {
|
||||
phase: 'extract',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'dry-run: extract phase skipped (no dry-run mode yet)',
|
||||
details: { dryRun: true, reason: 'no_dry_run_support' },
|
||||
};
|
||||
}
|
||||
const result = await runExtractCore(engine, { mode: 'all', dir: brainDir });
|
||||
const linksCreated = result?.links_created ?? 0;
|
||||
const timelineCreated = result?.timeline_entries_created ?? 0;
|
||||
return {
|
||||
phase: 'extract',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
|
||||
details: { linksCreated, timelineCreated, pages_processed: result?.pages_processed ?? 0 },
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'extract',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'extract phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
|
||||
try {
|
||||
const { runEmbedCore } = await import('../commands/embed.ts');
|
||||
const result = await runEmbedCore(engine, { stale: true, dryRun });
|
||||
const embeddedCount = dryRun ? result.would_embed : result.embedded;
|
||||
return {
|
||||
phase: 'embed',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: dryRun
|
||||
? `${result.would_embed} chunk(s) would be embedded (dry-run)`
|
||||
: `${result.embedded} chunk(s) newly embedded (${result.skipped} already had embeddings)`,
|
||||
details: {
|
||||
embedded: result.embedded,
|
||||
skipped: result.skipped,
|
||||
would_embed: result.would_embed,
|
||||
total_chunks: result.total_chunks,
|
||||
pages_processed: result.pages_processed,
|
||||
dryRun,
|
||||
// Convenience field used by CycleReport.totals.pages_embedded.
|
||||
// In dry-run, this counts pages with stale chunks that would
|
||||
// have been processed (same semantic as a real run).
|
||||
pages_embedded_count: dryRun ? result.pages_processed : embeddedCount > 0 ? result.pages_processed : 0,
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'embed',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'embed phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
|
||||
try {
|
||||
const { findOrphans } = await import('../commands/orphans.ts');
|
||||
const result = await findOrphans(engine);
|
||||
const count = result.total_orphans;
|
||||
return {
|
||||
phase: 'orphans',
|
||||
status: count > 20 ? 'warn' : 'ok',
|
||||
duration_ms: 0,
|
||||
summary: `${count} orphan page(s) out of ${result.total_pages} total`,
|
||||
details: {
|
||||
total_orphans: count,
|
||||
total_pages: result.total_pages,
|
||||
excluded: result.excluded,
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
phase: 'orphans',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'orphans phase failed',
|
||||
details: {},
|
||||
error: makeErrorFromException(e),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Main ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Run the brain maintenance cycle.
|
||||
*
|
||||
* Engine may be null: filesystem phases (lint, backlinks) still run;
|
||||
* DB-dependent phases skip with status='skipped', reason='no_database'.
|
||||
*
|
||||
* Acquires the cycle lock for any DB-write phase selection. Non-DB-write
|
||||
* selections (e.g., --phase lint) skip the lock as an optimization so
|
||||
* single-phase runs are always responsive even if another cycle is live.
|
||||
*/
|
||||
export async function runCycle(
|
||||
engine: BrainEngine | null,
|
||||
opts: CycleOpts,
|
||||
): Promise<CycleReport> {
|
||||
const start = performance.now();
|
||||
const phases = opts.phases ?? ALL_PHASES;
|
||||
const dryRun = !!opts.dryRun;
|
||||
const pull = !!opts.pull;
|
||||
const timestamp = new Date().toISOString();
|
||||
const phaseResults: PhaseResult[] = [];
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Decide if we need the cycle lock: any state-mutating phase in the selection.
|
||||
const needsLock = phases.some(p => NEEDS_LOCK_PHASES.has(p));
|
||||
|
||||
let lock: LockHandle | null = null;
|
||||
if (needsLock) {
|
||||
if (engine) {
|
||||
try {
|
||||
lock = await acquirePostgresLock(engine);
|
||||
} catch (e) {
|
||||
// Lock acquisition failed catastrophically (e.g., migration missing).
|
||||
// Return a failed report rather than silently running without a lock.
|
||||
return {
|
||||
schema_version: '1',
|
||||
timestamp,
|
||||
duration_ms: Math.round(performance.now() - start),
|
||||
status: 'failed',
|
||||
reason: 'lock_acquisition_error',
|
||||
brain_dir: opts.brainDir,
|
||||
phases: [
|
||||
{
|
||||
phase: 'sync',
|
||||
status: 'fail',
|
||||
duration_ms: 0,
|
||||
summary: 'could not acquire cycle lock',
|
||||
details: {},
|
||||
error: makeErrorFromException(e, 'DatabaseConnection'),
|
||||
},
|
||||
],
|
||||
totals: emptyTotals(),
|
||||
};
|
||||
}
|
||||
} else {
|
||||
lock = acquireFileLock();
|
||||
}
|
||||
|
||||
if (lock === null) {
|
||||
return {
|
||||
schema_version: '1',
|
||||
timestamp,
|
||||
duration_ms: Math.round(performance.now() - start),
|
||||
status: 'skipped',
|
||||
reason: 'cycle_already_running',
|
||||
brain_dir: opts.brainDir,
|
||||
phases: [],
|
||||
totals: emptyTotals(),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
// ── Phase 1: lint ────────────────────────────────────────────
|
||||
if (phases.includes('lint')) {
|
||||
progress.start('cycle.lint');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseLint(opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 2: backlinks ──────────────────────────────────────
|
||||
if (phases.includes('backlinks')) {
|
||||
progress.start('cycle.backlinks');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseBacklinks(opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 3: sync ───────────────────────────────────────────
|
||||
if (phases.includes('sync')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'sync',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.sync');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 4: extract ────────────────────────────────────────
|
||||
if (phases.includes('extract')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'extract',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.extract');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 5: embed ──────────────────────────────────────────
|
||||
if (phases.includes('embed')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'embed',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.embed');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 6: orphans ────────────────────────────────────────
|
||||
if (phases.includes('orphans')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'orphans',
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.orphans');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseOrphans(engine));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
} finally {
|
||||
if (lock) {
|
||||
try { await lock.release(); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
const duration_ms = Math.round(performance.now() - start);
|
||||
const totals = extractTotals(phaseResults);
|
||||
const status = deriveStatus(phaseResults, totals);
|
||||
|
||||
return {
|
||||
schema_version: '1',
|
||||
timestamp,
|
||||
duration_ms,
|
||||
status,
|
||||
brain_dir: opts.brainDir,
|
||||
phases: phaseResults,
|
||||
totals,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Totals + status derivation ────────────────────────────────────
|
||||
|
||||
function emptyTotals(): CycleReport['totals'] {
|
||||
return {
|
||||
lint_fixes: 0,
|
||||
backlinks_added: 0,
|
||||
pages_synced: 0,
|
||||
pages_extracted: 0,
|
||||
pages_embedded: 0,
|
||||
orphans_found: 0,
|
||||
};
|
||||
}
|
||||
|
||||
function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
|
||||
const t = emptyTotals();
|
||||
for (const p of phases) {
|
||||
if (p.phase === 'lint' && p.details) {
|
||||
t.lint_fixes = Number(p.details.fixed ?? 0);
|
||||
} else if (p.phase === 'backlinks' && p.details) {
|
||||
t.backlinks_added = Number(p.details.added ?? 0);
|
||||
} else if (p.phase === 'sync' && p.details) {
|
||||
t.pages_synced = Number(p.details.added ?? 0) + Number(p.details.modified ?? 0);
|
||||
} else if (p.phase === 'extract' && p.details) {
|
||||
t.pages_extracted = Number(p.details.linksCreated ?? 0);
|
||||
} else if (p.phase === 'embed' && p.details) {
|
||||
// In dry-run, use would_embed as the "activity" measure; else embedded.
|
||||
const dryRun = p.details.dryRun === true;
|
||||
t.pages_embedded = dryRun
|
||||
? Number(p.details.would_embed ?? 0)
|
||||
: Number(p.details.embedded ?? 0);
|
||||
} else if (p.phase === 'orphans' && p.details) {
|
||||
t.orphans_found = Number(p.details.total_orphans ?? 0);
|
||||
}
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
function deriveStatus(phases: PhaseResult[], totals: CycleReport['totals']): CycleStatus {
|
||||
if (phases.length === 0) return 'failed';
|
||||
const anyFailed = phases.some(p => p.status === 'fail');
|
||||
const allFailed = phases.every(p => p.status === 'fail');
|
||||
const anyWarn = phases.some(p => p.status === 'warn');
|
||||
if (allFailed) return 'failed';
|
||||
if (anyFailed || anyWarn) return 'partial';
|
||||
// All phases 'ok' or 'skipped'. Distinguish clean (no activity) from ok (work done).
|
||||
const anyWork =
|
||||
totals.lint_fixes > 0 ||
|
||||
totals.backlinks_added > 0 ||
|
||||
totals.pages_synced > 0 ||
|
||||
totals.pages_extracted > 0 ||
|
||||
totals.pages_embedded > 0;
|
||||
return anyWork ? 'ok' : 'clean';
|
||||
}
|
||||
+61
-3
@@ -13,6 +13,54 @@ let connectedUrl: string | null = null;
|
||||
*/
|
||||
const DEFAULT_POOL_SIZE_FALLBACK = 10;
|
||||
|
||||
/**
|
||||
* Supabase PgBouncer transaction-mode convention: port 6543 routes through
|
||||
* PgBouncer, which recycles the backend connection between queries and
|
||||
* invalidates per-client prepared-statement caches. On that port postgres.js
|
||||
* defaults (prepare=true) surface as `prepared statement "..." does not exist`
|
||||
* under sustained load and silently drop rows during sync.
|
||||
*
|
||||
* This is a heuristic, not a protocol guarantee. A direct-Postgres server
|
||||
* deliberately bound to 6543 will also get `prepare: false`; the
|
||||
* `GBRAIN_PREPARE=true` env var (or `?prepare=true` on the URL) is the
|
||||
* documented escape hatch.
|
||||
*/
|
||||
const AUTO_DETECT_PORTS = new Set(['6543']);
|
||||
|
||||
/**
|
||||
* Decide whether to force `prepare: true`/`false` on the postgres.js client.
|
||||
*
|
||||
* Precedence:
|
||||
* 1. `GBRAIN_PREPARE` env var (`true`/`1` or `false`/`0`)
|
||||
* 2. `?prepare=true|false` query param on the URL
|
||||
* 3. Auto-detect: port 6543 → `false`
|
||||
* 4. Default: `undefined` (caller omits the option; postgres.js default stands)
|
||||
*
|
||||
* Returns `boolean | undefined`. `undefined` is meaningful — callers MUST
|
||||
* omit the `prepare` key entirely in that case rather than passing
|
||||
* `undefined` through to `postgres(url, {prepare: undefined})`.
|
||||
*/
|
||||
export function resolvePrepare(url: string): boolean | undefined {
|
||||
const envPrepare = process.env.GBRAIN_PREPARE;
|
||||
if (envPrepare === 'false' || envPrepare === '0') return false;
|
||||
if (envPrepare === 'true' || envPrepare === '1') return true;
|
||||
|
||||
try {
|
||||
const parsed = new URL(url.replace(/^postgres(ql)?:\/\//, 'http://'));
|
||||
const urlPrepare = parsed.searchParams.get('prepare');
|
||||
if (urlPrepare === 'false') return false;
|
||||
if (urlPrepare === 'true') return true;
|
||||
|
||||
if (AUTO_DETECT_PORTS.has(parsed.port)) {
|
||||
return false;
|
||||
}
|
||||
} catch {
|
||||
// URL parse failure — fall through to default
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function resolvePoolSize(explicit?: number): number {
|
||||
if (typeof explicit === 'number' && explicit > 0) return explicit;
|
||||
const raw = process.env.GBRAIN_POOL_SIZE;
|
||||
@@ -53,7 +101,8 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
}
|
||||
|
||||
try {
|
||||
sql = postgres(url, {
|
||||
const prepare = resolvePrepare(url);
|
||||
const opts: Record<string, unknown> = {
|
||||
max: resolvePoolSize(),
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
@@ -61,7 +110,16 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
// Register pgvector type
|
||||
bigint: postgres.BigInt,
|
||||
},
|
||||
});
|
||||
};
|
||||
if (typeof prepare === 'boolean') {
|
||||
opts.prepare = prepare;
|
||||
if (!prepare) {
|
||||
console.warn(
|
||||
'[gbrain] Prepared statements disabled (PgBouncer transaction-mode convention on port 6543). Override with GBRAIN_PREPARE=true if your pooler runs in session mode.',
|
||||
);
|
||||
}
|
||||
}
|
||||
sql = postgres(url, opts);
|
||||
|
||||
// Test connection
|
||||
await sql`SELECT 1`;
|
||||
@@ -101,5 +159,5 @@ export async function withTransaction<T>(fn: (tx: ReturnType<typeof postgres>) =
|
||||
const conn = getConnection();
|
||||
return conn.begin(async (tx) => {
|
||||
return fn(tx as unknown as ReturnType<typeof postgres>);
|
||||
});
|
||||
}) as Promise<T>;
|
||||
}
|
||||
|
||||
+14
-1
@@ -32,7 +32,19 @@ export async function embed(text: string): Promise<Float32Array> {
|
||||
return result[0];
|
||||
}
|
||||
|
||||
export async function embedBatch(texts: string[]): Promise<Float32Array[]> {
|
||||
export interface EmbedBatchOptions {
|
||||
/**
|
||||
* Optional callback fired after each 100-item sub-batch completes.
|
||||
* CLI wrappers tick a reporter; Minion handlers can call
|
||||
* job.updateProgress here instead of hooking the per-page callback.
|
||||
*/
|
||||
onBatchComplete?: (done: number, total: number) => void;
|
||||
}
|
||||
|
||||
export async function embedBatch(
|
||||
texts: string[],
|
||||
options: EmbedBatchOptions = {},
|
||||
): Promise<Float32Array[]> {
|
||||
const truncated = texts.map(t => t.slice(0, MAX_CHARS));
|
||||
const results: Float32Array[] = [];
|
||||
|
||||
@@ -41,6 +53,7 @@ export async function embedBatch(texts: string[]): Promise<Float32Array[]> {
|
||||
const batch = truncated.slice(i, i + BATCH_SIZE);
|
||||
const batchResults = await embedBatchWithRetry(batch);
|
||||
results.push(...batchResults);
|
||||
options.onBatchComplete?.(results.length, truncated.length);
|
||||
}
|
||||
|
||||
return results;
|
||||
|
||||
@@ -50,6 +50,9 @@ export function clampSearchLimit(limit: number | undefined, defaultLimit = 20, c
|
||||
}
|
||||
|
||||
export interface BrainEngine {
|
||||
/** Discriminator: lets migrations and other consumers branch on engine kind without instanceof + dynamic imports. */
|
||||
readonly kind: 'postgres' | 'pglite';
|
||||
|
||||
// Lifecycle
|
||||
connect(config: EngineConfig): Promise<void>;
|
||||
disconnect(): Promise<void>;
|
||||
@@ -149,6 +152,13 @@ export interface BrainEngine {
|
||||
* Slugs with zero inbound links are present in the map with value 0.
|
||||
*/
|
||||
getBacklinkCounts(slugs: string[]): Promise<Map<string, number>>;
|
||||
/**
|
||||
* Return every page with no inbound links (from any source).
|
||||
* Domain comes from the frontmatter `domain` field (null if unset).
|
||||
* The caller filters pseudo-pages + derives display domain.
|
||||
* Used by `gbrain orphans` and `runCycle`'s orphan sweep phase.
|
||||
*/
|
||||
findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>>;
|
||||
|
||||
// Tags
|
||||
addTag(slug: string, tag: string): Promise<void>;
|
||||
|
||||
@@ -113,8 +113,8 @@ export async function enrichEntity(
|
||||
let timelineAdded = false;
|
||||
try {
|
||||
await engine.addTimelineEntry(slug, {
|
||||
date: new Date().toISOString().split('T')[0],
|
||||
content: `Referenced in [${request.sourceSlug}](${request.sourceSlug}) — ${request.context}`,
|
||||
date: new Date().toISOString().split('T')[0] ?? '',
|
||||
summary: `Referenced in [${request.sourceSlug}](${request.sourceSlug}) — ${request.context}`,
|
||||
source: request.sourceSlug,
|
||||
});
|
||||
timelineAdded = true;
|
||||
@@ -146,11 +146,13 @@ export async function enrichEntity(
|
||||
|
||||
/**
|
||||
* Enrich multiple entities with throttling between each.
|
||||
* config.onProgress is called after each entity so callers can stream
|
||||
* progress to a reporter (CLI) or job.updateProgress (Minion).
|
||||
*/
|
||||
export async function enrichEntities(
|
||||
engine: BrainEngine,
|
||||
requests: EnrichmentRequest[],
|
||||
config?: { throttle?: boolean },
|
||||
config?: { throttle?: boolean; onProgress?: (done: number, total: number, name: string) => void },
|
||||
): Promise<EnrichmentResult[]> {
|
||||
const results: EnrichmentResult[] = [];
|
||||
for (const req of requests) {
|
||||
@@ -159,6 +161,7 @@ export async function enrichEntities(
|
||||
}
|
||||
const result = await enrichEntity(engine, req);
|
||||
results.push(result);
|
||||
config?.onProgress?.(results.length, requests.length, req.entityName);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* CompletenessScorer — per-entity-type rubrics, 0.0–1.0 score per page.
|
||||
*
|
||||
* Replaces Wintermute's length-based heuristic ("compiled_truth > 500 chars")
|
||||
* Replaces Garry's OpenClaw's length-based heuristic ("compiled_truth > 500 chars")
|
||||
* with a weighted rubric that actually reflects whether a page would be
|
||||
* useful to answer a query. Runs on demand; BrainWriter invokes it on
|
||||
* write to cache the score in frontmatter.
|
||||
|
||||
@@ -119,18 +119,18 @@ export async function resolveFile(
|
||||
/** Parse v0.9+ .redirect.yaml pointer */
|
||||
export function parseRedirectYaml(path: string): RedirectYaml {
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
return parseYaml(content) as RedirectYaml;
|
||||
return parseYaml(content) as unknown as RedirectYaml;
|
||||
}
|
||||
|
||||
/** Parse legacy v0.8 .redirect breadcrumb */
|
||||
export function parseRedirect(path: string): RedirectInfo {
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
return parseYaml(content) as RedirectInfo;
|
||||
return parseYaml(content) as unknown as RedirectInfo;
|
||||
}
|
||||
|
||||
export function parseMarker(path: string): MarkerInfo {
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
return parseYaml(content) as MarkerInfo;
|
||||
return parseYaml(content) as unknown as MarkerInfo;
|
||||
}
|
||||
|
||||
/** Human-readable file size */
|
||||
|
||||
+86
-1
@@ -1,10 +1,12 @@
|
||||
import { readFileSync, statSync, lstatSync } from 'fs';
|
||||
import { basename } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { parseMarkdown } from './markdown.ts';
|
||||
import { chunkText } from './chunkers/recursive.ts';
|
||||
import { chunkCodeText, detectCodeLanguage } from './chunkers/code.ts';
|
||||
import { embedBatch } from './embedding.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
|
||||
import type { ChunkInput, PageType } from './types.ts';
|
||||
|
||||
/**
|
||||
@@ -184,6 +186,12 @@ export async function importFromFile(
|
||||
}
|
||||
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
|
||||
// Route code files through the code import path
|
||||
if (isCodeFilePath(relativePath)) {
|
||||
return importCodeFile(engine, relativePath, content, opts);
|
||||
}
|
||||
|
||||
const parsed = parseMarkdown(content, relativePath);
|
||||
|
||||
// Enforce path-authoritative slug. parseMarkdown prefers frontmatter.slug over
|
||||
@@ -206,6 +214,83 @@ export async function importFromFile(
|
||||
return importFromContent(engine, expectedSlug, content, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Import a code file. Bypasses markdown parsing entirely.
|
||||
* Uses tree-sitter code chunker for semantic splitting.
|
||||
* Page type is 'code', slug includes file extension.
|
||||
*/
|
||||
export async function importCodeFile(
|
||||
engine: BrainEngine,
|
||||
relativePath: string,
|
||||
content: string,
|
||||
opts: { noEmbed?: boolean } = {},
|
||||
): Promise<ImportResult> {
|
||||
const slug = slugifyCodePath(relativePath);
|
||||
const lang = detectCodeLanguage(relativePath) || 'unknown';
|
||||
const title = `${relativePath} (${lang})`;
|
||||
|
||||
const byteLength = Buffer.byteLength(content, 'utf-8');
|
||||
if (byteLength > MAX_FILE_SIZE) {
|
||||
return { slug, status: 'skipped', chunks: 0, error: `Code file too large (${byteLength} bytes)` };
|
||||
}
|
||||
|
||||
// Hash for idempotency
|
||||
const hash = createHash('sha256')
|
||||
.update(JSON.stringify({ title, type: 'code', content, lang }))
|
||||
.digest('hex');
|
||||
|
||||
const existing = await engine.getPage(slug);
|
||||
if (existing?.content_hash === hash) {
|
||||
return { slug, status: 'skipped', chunks: 0 };
|
||||
}
|
||||
|
||||
// Chunk via tree-sitter code chunker
|
||||
const codeChunks = await chunkCodeText(content, relativePath);
|
||||
const chunks: ChunkInput[] = codeChunks.map((c, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
}));
|
||||
|
||||
// Embed
|
||||
if (!opts.noEmbed && chunks.length > 0) {
|
||||
try {
|
||||
const embeddings = await embedBatch(chunks.map(c => c.chunk_text));
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
chunks[i].embedding = embeddings[i];
|
||||
chunks[i].token_count = Math.ceil(chunks[i].chunk_text.length / 4);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
console.warn(`[gbrain] embedding failed for code file ${slug}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Store
|
||||
await engine.transaction(async (tx) => {
|
||||
if (existing) await tx.createVersion(slug);
|
||||
|
||||
await tx.putPage(slug, {
|
||||
type: 'code' as PageType,
|
||||
title,
|
||||
compiled_truth: content,
|
||||
timeline: '',
|
||||
frontmatter: { language: lang, file: relativePath },
|
||||
content_hash: hash,
|
||||
});
|
||||
|
||||
await tx.addTag(slug, 'code');
|
||||
await tx.addTag(slug, lang);
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks);
|
||||
} else {
|
||||
await tx.deleteChunks(slug);
|
||||
}
|
||||
});
|
||||
|
||||
return { slug, status: 'imported', chunks: chunks.length };
|
||||
}
|
||||
|
||||
// Backward compat
|
||||
export const importFile = importFromFile;
|
||||
export type ImportFileResult = ImportResult;
|
||||
|
||||
+118
-19
@@ -17,7 +17,20 @@ import { slugifyPath } from './sync.ts';
|
||||
interface Migration {
|
||||
version: number;
|
||||
name: string;
|
||||
/** Engine-agnostic SQL. Used when `sqlFor` is absent. Set to '' for handler-only or sqlFor-only migrations. */
|
||||
sql: string;
|
||||
/**
|
||||
* Engine-specific SQL. If present, overrides `sql` for the matching engine.
|
||||
* Needed when Postgres wants CONCURRENTLY but PGLite can't honor it.
|
||||
*/
|
||||
sqlFor?: { postgres?: string; pglite?: string };
|
||||
/**
|
||||
* When false, the runner does NOT wrap the SQL in `engine.transaction()`.
|
||||
* Required for `CREATE INDEX CONCURRENTLY` (which Postgres refuses inside a transaction).
|
||||
* Enforced Postgres-only; ignored on PGLite (PGLite has no concurrent writers anyway).
|
||||
* Defaults to true.
|
||||
*/
|
||||
transaction?: boolean;
|
||||
handler?: (engine: BrainEngine) => Promise<void>;
|
||||
}
|
||||
|
||||
@@ -102,7 +115,7 @@ export const MIGRATIONS: Migration[] = [
|
||||
backoff_delay INTEGER NOT NULL DEFAULT 1000,
|
||||
backoff_jitter REAL NOT NULL DEFAULT 0.2,
|
||||
stalled_counter INTEGER NOT NULL DEFAULT 0,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 1,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 5,
|
||||
lock_token TEXT,
|
||||
lock_until TIMESTAMPTZ,
|
||||
delay_until TIMESTAMPTZ,
|
||||
@@ -355,9 +368,8 @@ export const MIGRATIONS: Migration[] = [
|
||||
// midnight rollover in the user's TZ naturally creates a new row instead of
|
||||
// mutating yesterday's. reserved_usd and committed_usd track reservations
|
||||
// vs actuals so process death between reserve() and commit()/rollback()
|
||||
// can be cleaned up by TTL scan. status and reserved_at exist for that
|
||||
// reclaim path. Rollback: DROP TABLE (budget is regenerable from resolver
|
||||
// call logs; no durable product data lives here).
|
||||
// can be cleaned up by TTL scan. Rollback: DROP TABLE (regenerable from
|
||||
// resolver call logs; no durable product data lives here).
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS budget_ledger (
|
||||
scope TEXT NOT NULL,
|
||||
@@ -388,16 +400,6 @@ export const MIGRATIONS: Migration[] = [
|
||||
version: 13,
|
||||
name: 'minion_quiet_hours_stagger',
|
||||
// Adds quiet-hours gating + deterministic stagger to Minions.
|
||||
//
|
||||
// quiet_hours (JSONB): {start, end, tz, policy} — checked at claim
|
||||
// time by the worker, not at dispatch. A queued job inside its quiet
|
||||
// window is released back to 'waiting' and claimed again outside the
|
||||
// window. 'skip' policy drops the event, 'defer' re-queues.
|
||||
// stagger_key (TEXT): hashed to a minute-slot offset so jobs with the
|
||||
// same key don't collide when a cron boundary fires. Optional; NULL
|
||||
// = no stagger. The hash lives in application code (deterministic,
|
||||
// ensures same key always lands on same slot) so the column is
|
||||
// just the key.
|
||||
sql: `
|
||||
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS quiet_hours JSONB;
|
||||
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS stagger_key TEXT;
|
||||
@@ -405,6 +407,91 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON minion_jobs(stagger_key) WHERE stagger_key IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 14,
|
||||
name: 'pages_updated_at_index',
|
||||
// v0.14.1 (fix wave): fixes the 14.6s "list pages newest-first" seqscan on 31k+ row brains.
|
||||
// Original report: https://github.com/garrytan/gbrain/issues/170 (PR #215).
|
||||
//
|
||||
// Engine-aware via handler (not SQL): Postgres uses CREATE INDEX CONCURRENTLY
|
||||
// to avoid the write-blocking SHARE lock on `pages`. CONCURRENTLY refuses to
|
||||
// run inside a transaction AND postgres.js's multi-statement `.unsafe()` wraps
|
||||
// in an implicit transaction, so the handler runs each statement as a separate
|
||||
// call. A failed CONCURRENTLY leaves an invalid index with the target name;
|
||||
// the handler pre-drops any invalid remnant via pg_index.indisvalid. PGLite
|
||||
// has no concurrent writers, so plain CREATE is safe.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
if (engine.kind === 'postgres') {
|
||||
await engine.runMigration(
|
||||
14,
|
||||
`DO $$ BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1 FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
WHERE c.relname = 'idx_pages_updated_at_desc' AND NOT i.indisvalid
|
||||
) THEN
|
||||
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS idx_pages_updated_at_desc';
|
||||
END IF;
|
||||
END $$;`
|
||||
);
|
||||
await engine.runMigration(
|
||||
14,
|
||||
`CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_pages_updated_at_desc
|
||||
ON pages (updated_at DESC);`
|
||||
);
|
||||
} else {
|
||||
await engine.runMigration(
|
||||
14,
|
||||
`CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc
|
||||
ON pages (updated_at DESC);`
|
||||
);
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 15,
|
||||
name: 'minion_jobs_max_stalled_default_5',
|
||||
// v0.14.1 (fix wave): fixes https://github.com/garrytan/gbrain/issues/219
|
||||
// Shipped default was 1 — first stall = dead-letter, contradicting the
|
||||
// "SIGKILL rescued" claim. New default 5. UPDATE backfills existing non-
|
||||
// terminal rows so upgrading brains don't keep dead-lettering queued work.
|
||||
// Statuses come from MinionJobStatus in types.ts. Row locks serialize
|
||||
// against claim()'s FOR UPDATE SKIP LOCKED — race-safe. Idempotent.
|
||||
sql: `
|
||||
ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 5;
|
||||
UPDATE minion_jobs
|
||||
SET max_stalled = 5
|
||||
WHERE status IN ('waiting','active','delayed','waiting-children','paused')
|
||||
AND max_stalled < 5;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 16,
|
||||
name: 'cycle_locks_table',
|
||||
// v0.17 brain maintenance cycle (runCycle primitive).
|
||||
// PgBouncer transaction pooling strips session-scoped advisory locks
|
||||
// (pg_try_advisory_lock) across connection checkouts, so we can't use
|
||||
// them as the cycle-coordination primitive. A row with a TTL works
|
||||
// through every pooler: any backend can SELECT/UPDATE/DELETE it, no
|
||||
// session state required.
|
||||
//
|
||||
// Acquire: INSERT ... ON CONFLICT (id) DO UPDATE ... WHERE ttl_expires_at < NOW()
|
||||
// returning ... — empty RETURNING = lock held by live holder.
|
||||
// Refresh: UPDATE ... SET ttl_expires_at = NOW() + interval '30 min'
|
||||
// WHERE id = 'gbrain-cycle' AND holder_pid = <my pid> — between phases.
|
||||
// Release: DELETE WHERE id = 'gbrain-cycle' AND holder_pid = <my pid>.
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
|
||||
id TEXT PRIMARY KEY,
|
||||
holder_pid INT NOT NULL,
|
||||
holder_host TEXT,
|
||||
acquired_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
ttl_expires_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
@@ -418,11 +505,23 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num
|
||||
let applied = 0;
|
||||
for (const m of MIGRATIONS) {
|
||||
if (m.version > current) {
|
||||
// SQL migration (transactional)
|
||||
if (m.sql) {
|
||||
await engine.transaction(async (tx) => {
|
||||
await tx.runMigration(m.version, m.sql);
|
||||
});
|
||||
// Pick SQL: engine-specific `sqlFor` wins over engine-agnostic `sql`.
|
||||
const sql = m.sqlFor?.[engine.kind] ?? m.sql;
|
||||
|
||||
if (sql) {
|
||||
const useTransaction = m.transaction !== false;
|
||||
// Non-transactional path is Postgres-only: `CREATE INDEX CONCURRENTLY`
|
||||
// refuses to run inside a transaction. PGLite has no concurrent
|
||||
// writers, so even if a migration sets transaction:false we wrap it
|
||||
// anyway (harmless; keeps behavior consistent).
|
||||
if (useTransaction || engine.kind === 'pglite') {
|
||||
await engine.transaction(async (tx) => {
|
||||
await tx.runMigration(m.version, sql);
|
||||
});
|
||||
} else {
|
||||
// Postgres + transaction:false → direct execution, no BEGIN/COMMIT.
|
||||
await engine.runMigration(m.version, sql);
|
||||
}
|
||||
}
|
||||
|
||||
// Application-level handler (runs outside transaction for flexibility)
|
||||
|
||||
@@ -269,7 +269,7 @@ export async function shellHandler(ctx: MinionJobContext): Promise<ShellJobResul
|
||||
if (ctx.signal.aborted) sigAbort();
|
||||
if (ctx.shutdownSignal.aborted) shutdownAbort();
|
||||
|
||||
const exitCode: number = await new Promise((resolve, reject) => {
|
||||
const exitCode: number = await new Promise<number>((resolve, reject) => {
|
||||
proc.on('error', (err) => {
|
||||
reject(err);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* subagent_aggregator handler (v0.15).
|
||||
*
|
||||
* This is the job that CLAIMS after all subagent children resolve and
|
||||
* produces the final aggregated output. Not a polling parent — Lane 1B's
|
||||
* queue changes make every terminal child transition (complete/failed/
|
||||
* dead/cancelled/timeout) emit a child_done message into this job's
|
||||
* inbox, AND flip this job out of waiting-children once all kids are
|
||||
* terminal. When we claim, all N child_done messages are already in
|
||||
* minion_inbox.
|
||||
*
|
||||
* The aggregator does NOT re-call Anthropic in v0.15. It reads child
|
||||
* results from child_done messages, builds a markdown summary, and
|
||||
* returns it as the handler result. If children produced brain pages
|
||||
* under wiki/agents/<child_id>/..., those are referenced by slug — not
|
||||
* re-embedded into the summary blob.
|
||||
*
|
||||
* v0.16+ will add an LLM synthesis pass for richer summaries. The v0.15
|
||||
* output is deterministic string concatenation so fan-out runs stay
|
||||
* reproducible.
|
||||
*/
|
||||
|
||||
import type { MinionJobContext, ChildDoneMessage, ChildOutcome } from '../types.ts';
|
||||
import type { AggregatorHandlerData } from '../types.ts';
|
||||
|
||||
export interface AggregatorResult {
|
||||
/** Per-child record in the order children_ids was supplied. */
|
||||
children: Array<{
|
||||
child_id: number;
|
||||
job_name: string;
|
||||
outcome: ChildOutcome;
|
||||
error: string | null;
|
||||
/** JSON-parsed result payload for successful children. null on failure/cancel/timeout. */
|
||||
result: unknown;
|
||||
}>;
|
||||
/** Counts by outcome — quick shape for logs + tests. */
|
||||
summary: Record<ChildOutcome, number>;
|
||||
/** Rendered markdown, suitable for attaching to the job row or writing as a brain page. */
|
||||
markdown: string;
|
||||
}
|
||||
|
||||
/** v0.15 aggregator: synchronous read from inbox, no LLM call. */
|
||||
export async function subagentAggregatorHandler(ctx: MinionJobContext): Promise<AggregatorResult> {
|
||||
const data = (ctx.data ?? {}) as unknown as AggregatorHandlerData;
|
||||
const expectedIds = Array.isArray(data.children_ids) ? data.children_ids : [];
|
||||
|
||||
if (expectedIds.length === 0) {
|
||||
return {
|
||||
children: [],
|
||||
summary: emptySummary(),
|
||||
markdown: '# Aggregated subagent results\n\n_(no children)_',
|
||||
};
|
||||
}
|
||||
|
||||
// Read every child_done inbox message addressed to this job. By the time
|
||||
// we're claimed, the queue layer has posted one per child terminal
|
||||
// transition. The `readInbox` method marks messages as read so future
|
||||
// claims don't re-process them.
|
||||
const messages = await ctx.readInbox();
|
||||
const childDoneByChildId = new Map<number, ChildDoneMessage>();
|
||||
for (const m of messages) {
|
||||
const payload = parseChildDone(m.payload);
|
||||
if (!payload) continue;
|
||||
childDoneByChildId.set(payload.child_id, payload);
|
||||
}
|
||||
|
||||
const summary = emptySummary();
|
||||
const children: AggregatorResult['children'] = expectedIds.map(childId => {
|
||||
const msg = childDoneByChildId.get(childId);
|
||||
if (!msg) {
|
||||
// Missing — shouldn't happen under the v0.15 invariants (every
|
||||
// terminal path emits child_done). Surface as a failure row so the
|
||||
// aggregator is honest about what it knows.
|
||||
summary.failed = (summary.failed ?? 0) + 1;
|
||||
return {
|
||||
child_id: childId,
|
||||
job_name: '',
|
||||
outcome: 'failed',
|
||||
error: 'no child_done message observed in inbox',
|
||||
result: null,
|
||||
};
|
||||
}
|
||||
const outcome: ChildOutcome = msg.outcome ?? 'complete';
|
||||
summary[outcome] = (summary[outcome] ?? 0) + 1;
|
||||
return {
|
||||
child_id: childId,
|
||||
job_name: msg.job_name,
|
||||
outcome,
|
||||
error: msg.error ?? null,
|
||||
result: outcome === 'complete' ? msg.result : null,
|
||||
};
|
||||
});
|
||||
|
||||
const markdown = renderMarkdown(children, summary, data.aggregate_prompt_template);
|
||||
|
||||
await ctx.updateProgress({ total: expectedIds.length, summary });
|
||||
await ctx.log(`aggregated ${expectedIds.length} children — ${formatSummary(summary)}`);
|
||||
|
||||
return { children, summary, markdown };
|
||||
}
|
||||
|
||||
// ── internal ────────────────────────────────────────────────
|
||||
|
||||
function emptySummary(): Record<ChildOutcome, number> {
|
||||
return { complete: 0, failed: 0, dead: 0, cancelled: 0, timeout: 0 };
|
||||
}
|
||||
|
||||
function formatSummary(s: Record<ChildOutcome, number>): string {
|
||||
return Object.entries(s)
|
||||
.filter(([, n]) => n > 0)
|
||||
.map(([k, n]) => `${k}=${n}`)
|
||||
.join(', ');
|
||||
}
|
||||
|
||||
function parseChildDone(payload: unknown): ChildDoneMessage | null {
|
||||
const obj = typeof payload === 'string' ? safeParse(payload) : payload;
|
||||
if (!obj || typeof obj !== 'object') return null;
|
||||
const rec = obj as Record<string, unknown>;
|
||||
if (rec.type !== 'child_done' || typeof rec.child_id !== 'number') return null;
|
||||
return {
|
||||
type: 'child_done',
|
||||
child_id: rec.child_id,
|
||||
job_name: typeof rec.job_name === 'string' ? rec.job_name : '',
|
||||
result: rec.result,
|
||||
outcome: typeof rec.outcome === 'string' ? rec.outcome as ChildOutcome : undefined,
|
||||
error: typeof rec.error === 'string' ? rec.error : null,
|
||||
};
|
||||
}
|
||||
|
||||
function safeParse(raw: string): unknown {
|
||||
try { return JSON.parse(raw); } catch { return null; }
|
||||
}
|
||||
|
||||
function renderMarkdown(
|
||||
children: AggregatorResult['children'],
|
||||
summary: Record<ChildOutcome, number>,
|
||||
template?: string,
|
||||
): string {
|
||||
const header = template && template.trim().length > 0
|
||||
? template
|
||||
: '# Aggregated subagent results';
|
||||
|
||||
const parts: string[] = [header, ''];
|
||||
parts.push(`- total: ${children.length}`);
|
||||
for (const [outcome, n] of Object.entries(summary)) {
|
||||
if (n > 0) parts.push(`- ${outcome}: ${n}`);
|
||||
}
|
||||
parts.push('');
|
||||
|
||||
for (const c of children) {
|
||||
parts.push(`## child ${c.child_id} (${c.job_name || 'unknown'}) — ${c.outcome}`);
|
||||
if (c.error) parts.push(`> error: ${c.error}`);
|
||||
if (c.outcome === 'complete' && c.result !== undefined) {
|
||||
parts.push('```json', JSON.stringify(c.result, null, 2), '```');
|
||||
}
|
||||
parts.push('');
|
||||
}
|
||||
|
||||
return parts.join('\n').replace(/\n{3,}/g, '\n\n');
|
||||
}
|
||||
|
||||
// ── Testing surface ─────────────────────────────────────────
|
||||
|
||||
export const __testing = {
|
||||
emptySummary,
|
||||
formatSummary,
|
||||
parseChildDone,
|
||||
renderMarkdown,
|
||||
};
|
||||
@@ -0,0 +1,137 @@
|
||||
/**
|
||||
* Subagent audit + heartbeat log. JSONL, file-rotated weekly, best-effort.
|
||||
*
|
||||
* Two event flavors:
|
||||
* - submission: one line per subagent job submit (mirrors shell-audit).
|
||||
* - heartbeat: one line per LLM turn boundary (started / completed) so
|
||||
* `gbrain agent logs <job> --follow` has fresh content to
|
||||
* show during long Anthropic calls. Without these, a
|
||||
* 30-second model call produces zero output between turns
|
||||
* and --follow looks frozen.
|
||||
*
|
||||
* Never logs prompts, tool inputs, or full tool outputs (PII risk — input
|
||||
* vars may contain emails, free text from the user, etc.). DO log
|
||||
* non-identifying operational fields: tokens, duration, model, tool_name.
|
||||
*
|
||||
* `GBRAIN_AUDIT_DIR` overrides the default ~/.gbrain/audit/ path — useful
|
||||
* for container deploys with a read-only $HOME.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { resolveAuditDir } from './shell-audit.ts';
|
||||
|
||||
export interface SubagentSubmissionEvent {
|
||||
ts: string;
|
||||
type: 'submission';
|
||||
caller: 'cli' | 'mcp' | 'worker';
|
||||
remote: boolean;
|
||||
job_id: number;
|
||||
parent_job_id?: number | null;
|
||||
model?: string;
|
||||
tools_count?: number;
|
||||
allowed_tools?: string[];
|
||||
}
|
||||
|
||||
export interface SubagentHeartbeatEvent {
|
||||
ts: string;
|
||||
type: 'heartbeat';
|
||||
job_id: number;
|
||||
event: 'llm_call_started' | 'llm_call_completed' | 'tool_called' | 'tool_result' | 'tool_failed';
|
||||
turn_idx: number;
|
||||
/** Tool name for tool_* events. Never the input — that may contain secrets. */
|
||||
tool_name?: string;
|
||||
/** ms elapsed for *_completed / tool_result / tool_failed. */
|
||||
ms_elapsed?: number;
|
||||
/** Token rollup for llm_call_completed. Per-turn, not cumulative. */
|
||||
tokens?: { in?: number; out?: number; cache_read?: number; cache_create?: number };
|
||||
/** Short error text for tool_failed. First 200 chars. */
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export type SubagentAuditEvent = SubagentSubmissionEvent | SubagentHeartbeatEvent;
|
||||
|
||||
/** File name, rotated by ISO week. `subagent-jobs-YYYY-Www.jsonl`. */
|
||||
export function computeSubagentAuditFilename(now: Date = new Date()): string {
|
||||
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
|
||||
const dayNum = (d.getUTCDay() + 6) % 7;
|
||||
d.setUTCDate(d.getUTCDate() - dayNum + 3);
|
||||
const isoYear = d.getUTCFullYear();
|
||||
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
|
||||
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
|
||||
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
|
||||
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
|
||||
const ww = String(weekNum).padStart(2, '0');
|
||||
return `subagent-jobs-${isoYear}-W${ww}.jsonl`;
|
||||
}
|
||||
|
||||
/** Low-level append. Best-effort; write failure goes to stderr + keep running. */
|
||||
function append(event: SubagentAuditEvent): void {
|
||||
const dir = resolveAuditDir();
|
||||
const file = path.join(dir, computeSubagentAuditFilename());
|
||||
const line = JSON.stringify(event) + '\n';
|
||||
try {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.appendFileSync(file, line, { encoding: 'utf8' });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[subagent-audit] write failed (${msg}); job continues\n`);
|
||||
}
|
||||
}
|
||||
|
||||
export function logSubagentSubmission(event: Omit<SubagentSubmissionEvent, 'ts' | 'type'>): void {
|
||||
append({ ...event, ts: new Date().toISOString(), type: 'submission' });
|
||||
}
|
||||
|
||||
export function logSubagentHeartbeat(event: Omit<SubagentHeartbeatEvent, 'ts' | 'type'>): void {
|
||||
// Defensive: trim error text to avoid accidentally writing huge stack traces.
|
||||
const trimmed = event.error ? { ...event, error: event.error.slice(0, 200) } : event;
|
||||
append({ ...trimmed, ts: new Date().toISOString(), type: 'heartbeat' });
|
||||
}
|
||||
|
||||
/**
|
||||
* Read back all audit events for a job id from the current + prior week
|
||||
* files. Used by `gbrain agent logs <job>`. Returns chronological order.
|
||||
*
|
||||
* `sinceIso` (if present) filters to events with ts >= sinceIso.
|
||||
*/
|
||||
export function readSubagentAuditForJob(jobId: number, opts: { sinceIso?: string } = {}): SubagentAuditEvent[] {
|
||||
const dir = resolveAuditDir();
|
||||
if (!fs.existsSync(dir)) return [];
|
||||
|
||||
const now = new Date();
|
||||
const thisWeek = computeSubagentAuditFilename(now);
|
||||
const weekAgo = computeSubagentAuditFilename(new Date(now.getTime() - 7 * 86400000));
|
||||
const candidates = [...new Set([weekAgo, thisWeek])];
|
||||
|
||||
const out: SubagentAuditEvent[] = [];
|
||||
for (const name of candidates) {
|
||||
const file = path.join(dir, name);
|
||||
if (!fs.existsSync(file)) continue;
|
||||
let raw: string;
|
||||
try {
|
||||
raw = fs.readFileSync(file, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const line of raw.split('\n')) {
|
||||
if (!line) continue;
|
||||
let ev: SubagentAuditEvent;
|
||||
try {
|
||||
ev = JSON.parse(line) as SubagentAuditEvent;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
// Submission events have job_id at top level; heartbeats too. Both safe.
|
||||
if ((ev as { job_id?: number }).job_id !== jobId) continue;
|
||||
if (opts.sinceIso && ev.ts < opts.sinceIso) continue;
|
||||
out.push(ev);
|
||||
}
|
||||
}
|
||||
return out.sort((a, b) => a.ts.localeCompare(b.ts));
|
||||
}
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
append,
|
||||
};
|
||||
@@ -0,0 +1,710 @@
|
||||
/**
|
||||
* Subagent LLM-loop handler (v0.15).
|
||||
*
|
||||
* Runs one Anthropic Messages API conversation with tool use. The loop is
|
||||
* crash-resumable: subagent_messages + subagent_tool_executions together
|
||||
* are the single source of truth about where the conversation is. On
|
||||
* resume after a worker kill, we load all committed rows, trust any tool
|
||||
* execution marked 'complete' or 'failed', and re-run 'pending' ones only
|
||||
* for idempotent tools.
|
||||
*
|
||||
* Safety rails:
|
||||
* - rate leases around every LLM call (acquire → call → release). Mid-
|
||||
* call renewal with backoff. Persistent renewal failure aborts as a
|
||||
* renewable error so the worker re-claims.
|
||||
* - dual-signal abort wiring (ctx.signal + ctx.shutdownSignal) drains
|
||||
* the in-flight call and commits whatever turns are already persisted.
|
||||
* - Anthropic prompt cache markers on system + tools blocks.
|
||||
* - token rollup via ctx.updateTokens per turn.
|
||||
*
|
||||
* NOT in v0.15: refusal detection, stop_reason=max_tokens partial
|
||||
* recovery, parallel tool-use dispatch (runs tools sequentially; the
|
||||
* Messages API allows parallel tool_use blocks and the replay tolerates
|
||||
* them, but v1 dispatches serially for simplicity). All three are tracked
|
||||
* as P2 items in the plan file.
|
||||
*/
|
||||
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
import type { MinionJobContext, MinionJob } from '../types.ts';
|
||||
import type {
|
||||
ContentBlock,
|
||||
SubagentHandlerData,
|
||||
SubagentResult,
|
||||
SubagentStopReason,
|
||||
ToolDef,
|
||||
} from '../types.ts';
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { GBrainConfig } from '../../config.ts';
|
||||
import { loadConfig } from '../../config.ts';
|
||||
import { buildBrainTools, filterAllowedTools } from '../tools/brain-allowlist.ts';
|
||||
import {
|
||||
acquireLease,
|
||||
releaseLease,
|
||||
renewLeaseWithBackoff,
|
||||
} from '../rate-leases.ts';
|
||||
import {
|
||||
logSubagentSubmission,
|
||||
logSubagentHeartbeat,
|
||||
} from './subagent-audit.ts';
|
||||
|
||||
// ── Defaults ────────────────────────────────────────────────
|
||||
|
||||
const DEFAULT_MODEL = 'claude-sonnet-4-6';
|
||||
const DEFAULT_MAX_TURNS = 20;
|
||||
const DEFAULT_RATE_KEY = 'anthropic:messages';
|
||||
const DEFAULT_MAX_CONCURRENT = Number(process.env.GBRAIN_ANTHROPIC_MAX_INFLIGHT ?? '8');
|
||||
const DEFAULT_LEASE_TTL_MS = 120_000;
|
||||
const DEFAULT_SYSTEM = 'You are a helpful assistant running as a gbrain subagent.';
|
||||
|
||||
// ── Injectable surfaces (for tests) ─────────────────────────
|
||||
|
||||
/**
|
||||
* Anthropic Messages client. The real Anthropic SDK implements this
|
||||
* structurally; tests can substitute a mock without the SDK import.
|
||||
*/
|
||||
export interface MessagesClient {
|
||||
create(params: Anthropic.MessageCreateParamsNonStreaming, opts?: { signal?: AbortSignal }): Promise<Anthropic.Message>;
|
||||
}
|
||||
|
||||
export interface SubagentDeps {
|
||||
/** Engine for DB-backed ops (tools + message persistence + rate leases). */
|
||||
engine: BrainEngine;
|
||||
/** Anthropic client. Defaults to the SDK-constructed client. */
|
||||
client?: MessagesClient;
|
||||
/**
|
||||
* Anthropic SDK constructor. Defaults to `() => new Anthropic()`.
|
||||
* Overridable in tests so the factory default-client branch is
|
||||
* exercisable without an ANTHROPIC_API_KEY or a real API call.
|
||||
* When `deps.client` is provided, this is unused.
|
||||
*/
|
||||
makeAnthropic?: () => Anthropic;
|
||||
/** Config (MCP, brain, etc.). Defaults to loadConfig(). */
|
||||
config?: GBrainConfig;
|
||||
/** Rate-lease key. Defaults to `anthropic:messages`. */
|
||||
rateLeaseKey?: string;
|
||||
/** Max concurrent inflight calls on that key. Defaults to GBRAIN_ANTHROPIC_MAX_INFLIGHT or 8. */
|
||||
maxConcurrent?: number;
|
||||
/** Lease TTL. Defaults to 120s. */
|
||||
leaseTtlMs?: number;
|
||||
/**
|
||||
* Override tool registry. When omitted, buildBrainTools is called with
|
||||
* the caller's subagentId at dispatch time.
|
||||
*/
|
||||
toolRegistry?: ToolDef[];
|
||||
}
|
||||
|
||||
// ── Types for internal state ────────────────────────────────
|
||||
|
||||
interface PersistedMessage {
|
||||
message_idx: number;
|
||||
role: 'user' | 'assistant';
|
||||
content_blocks: ContentBlock[];
|
||||
tokens_in: number | null;
|
||||
tokens_out: number | null;
|
||||
tokens_cache_read: number | null;
|
||||
tokens_cache_create: number | null;
|
||||
model: string | null;
|
||||
}
|
||||
|
||||
interface PersistedToolExec {
|
||||
message_idx: number;
|
||||
tool_use_id: string;
|
||||
tool_name: string;
|
||||
input: unknown;
|
||||
status: 'pending' | 'complete' | 'failed';
|
||||
output: unknown;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
// ── Public handler factory ──────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build a subagent handler bound to a specific engine. `registerBuiltin
|
||||
* Handlers` wires this up as `worker.register('subagent', handler)` at
|
||||
* worker startup. Always registered — `ANTHROPIC_API_KEY` is the natural
|
||||
* cost gate and `PROTECTED_JOB_NAMES` gates submission.
|
||||
*/
|
||||
export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
const engine = deps.engine;
|
||||
// sdk.messages IS the MessagesClient-shaped object. The v0.16.0 bug was
|
||||
// casting new Anthropic() (top level) to MessagesClient, but .create()
|
||||
// lives at sdk.messages.create. Assigning sdk.messages directly gets the
|
||||
// right object; JS method-call semantics preserve `this` at the call
|
||||
// site (subagent.ts invokes client.create(...) with client === sdk.messages).
|
||||
const makeAnthropic = deps.makeAnthropic ?? (() => new Anthropic());
|
||||
const client: MessagesClient = deps.client ?? makeAnthropic().messages;
|
||||
const config = deps.config ?? loadConfig() ?? ({ engine: 'postgres' } as GBrainConfig);
|
||||
const rateLeaseKey = deps.rateLeaseKey ?? DEFAULT_RATE_KEY;
|
||||
const maxConcurrent = deps.maxConcurrent ?? DEFAULT_MAX_CONCURRENT;
|
||||
const leaseTtlMs = deps.leaseTtlMs ?? DEFAULT_LEASE_TTL_MS;
|
||||
|
||||
return async function subagentHandler(ctx: MinionJobContext): Promise<SubagentResult> {
|
||||
const data = (ctx.data ?? {}) as unknown as SubagentHandlerData;
|
||||
if (!data.prompt || typeof data.prompt !== 'string') {
|
||||
throw new Error('subagent job data.prompt is required (string)');
|
||||
}
|
||||
|
||||
const model = data.model ?? DEFAULT_MODEL;
|
||||
const maxTurns = data.max_turns ?? DEFAULT_MAX_TURNS;
|
||||
const systemPrompt = data.system ?? DEFAULT_SYSTEM;
|
||||
|
||||
// Build the tool registry bound to THIS job as the owning subagent.
|
||||
const registry = deps.toolRegistry ?? buildBrainTools({
|
||||
subagentId: ctx.id,
|
||||
engine,
|
||||
config,
|
||||
});
|
||||
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
|
||||
? filterAllowedTools(registry, data.allowed_tools)
|
||||
: registry;
|
||||
|
||||
logSubagentSubmission({
|
||||
caller: 'worker',
|
||||
remote: true,
|
||||
job_id: ctx.id,
|
||||
model,
|
||||
tools_count: toolDefs.length,
|
||||
allowed_tools: toolDefs.map(t => t.name),
|
||||
});
|
||||
|
||||
// ── Load prior state (replay) ───────────────────────────
|
||||
const priorMessages = await loadPriorMessages(engine, ctx.id);
|
||||
const priorTools = await loadPriorTools(engine, ctx.id);
|
||||
const priorToolByUseId = new Map(priorTools.map(t => [t.tool_use_id, t]));
|
||||
|
||||
// Rebuild the Anthropic messages array from persisted rows.
|
||||
const anthroMessages: Anthropic.MessageParam[] = priorMessages.length > 0
|
||||
? priorMessages.map(m => ({ role: m.role, content: m.content_blocks as any }))
|
||||
: [{ role: 'user', content: data.prompt }];
|
||||
|
||||
// If we had no prior messages, persist the seed user message.
|
||||
let nextMessageIdx = priorMessages.length;
|
||||
if (priorMessages.length === 0) {
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: 0,
|
||||
role: 'user',
|
||||
content_blocks: [{ type: 'text', text: data.prompt }],
|
||||
tokens_in: null,
|
||||
tokens_out: null,
|
||||
tokens_cache_read: null,
|
||||
tokens_cache_create: null,
|
||||
model: null,
|
||||
});
|
||||
nextMessageIdx = 1;
|
||||
}
|
||||
|
||||
// Token rollup.
|
||||
const tokenTotals = { in: 0, out: 0, cache_read: 0, cache_create: 0 };
|
||||
for (const m of priorMessages) {
|
||||
if (m.tokens_in) tokenTotals.in += m.tokens_in;
|
||||
if (m.tokens_out) tokenTotals.out += m.tokens_out;
|
||||
if (m.tokens_cache_read) tokenTotals.cache_read += m.tokens_cache_read;
|
||||
if (m.tokens_cache_create) tokenTotals.cache_create += m.tokens_cache_create;
|
||||
}
|
||||
|
||||
// Count assistant messages already persisted toward max_turns.
|
||||
let assistantTurns = priorMessages.filter(m => m.role === 'assistant').length;
|
||||
|
||||
// ── Replay reconciliation ───────────────────────────────
|
||||
//
|
||||
// If the last persisted message is an assistant with tool_use blocks
|
||||
// AND no subsequent user message has been synthesized yet, we crashed
|
||||
// mid-tool-dispatch. Finish those tools now so the next LLM call sees
|
||||
// a consistent conversation.
|
||||
const last = priorMessages[priorMessages.length - 1];
|
||||
if (last && last.role === 'assistant') {
|
||||
const pendingToolUses = last.content_blocks.filter(
|
||||
(b): b is { type: 'tool_use'; id: string; name: string; input: unknown } & Record<string, unknown> =>
|
||||
b.type === 'tool_use',
|
||||
);
|
||||
if (pendingToolUses.length > 0) {
|
||||
const synthesizedResults: ContentBlock[] = [];
|
||||
for (const use of pendingToolUses) {
|
||||
const prior = priorToolByUseId.get(use.id);
|
||||
if (prior?.status === 'complete') {
|
||||
synthesizedResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: asStringIfNotObject(prior.output),
|
||||
} as ContentBlock);
|
||||
continue;
|
||||
}
|
||||
if (prior?.status === 'failed') {
|
||||
synthesizedResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: prior.error ?? 'tool failed',
|
||||
is_error: true,
|
||||
} as ContentBlock);
|
||||
continue;
|
||||
}
|
||||
// pending or no row yet — try to dispatch.
|
||||
const toolDef = toolDefs.find(t => t.name === use.name);
|
||||
if (!toolDef) {
|
||||
await persistToolExecFailed(
|
||||
engine, ctx.id, last.message_idx, use.id, use.name, use.input,
|
||||
`tool "${use.name}" is not in the registry for this subagent`,
|
||||
);
|
||||
synthesizedResults.push({
|
||||
type: 'tool_result', tool_use_id: use.id,
|
||||
content: `tool "${use.name}" is not available`, is_error: true,
|
||||
} as ContentBlock);
|
||||
continue;
|
||||
}
|
||||
if (prior?.status === 'pending' && !toolDef.idempotent) {
|
||||
throw new Error(`non-idempotent tool "${use.name}" pending on resume; cannot safely re-run`);
|
||||
}
|
||||
await persistToolExecPending(engine, ctx.id, last.message_idx, use.id, use.name, use.input);
|
||||
try {
|
||||
const output = await toolDef.execute(use.input, {
|
||||
engine, jobId: ctx.id, remote: true, signal: ctx.signal,
|
||||
});
|
||||
await persistToolExecComplete(engine, ctx.id, use.id, output);
|
||||
synthesizedResults.push({
|
||||
type: 'tool_result', tool_use_id: use.id,
|
||||
content: asStringIfNotObject(output),
|
||||
} as ContentBlock);
|
||||
} catch (e) {
|
||||
const errText = e instanceof Error ? (e.stack ?? e.message) : String(e);
|
||||
await persistToolExecFailed(engine, ctx.id, last.message_idx, use.id, use.name, use.input, errText);
|
||||
synthesizedResults.push({
|
||||
type: 'tool_result', tool_use_id: use.id,
|
||||
content: errText, is_error: true,
|
||||
} as ContentBlock);
|
||||
}
|
||||
}
|
||||
// Persist the synthesized user turn so next-resume picks up here.
|
||||
const userIdx = nextMessageIdx++;
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: userIdx,
|
||||
role: 'user',
|
||||
content_blocks: synthesizedResults,
|
||||
tokens_in: null, tokens_out: null, tokens_cache_read: null, tokens_cache_create: null, model: null,
|
||||
});
|
||||
anthroMessages.push({ role: 'user', content: synthesizedResults as any });
|
||||
}
|
||||
}
|
||||
|
||||
// ── Main loop ───────────────────────────────────────────
|
||||
let stopReason: SubagentStopReason = 'error';
|
||||
let finalText = '';
|
||||
|
||||
while (true) {
|
||||
if (assistantTurns >= maxTurns) {
|
||||
stopReason = 'max_turns';
|
||||
break;
|
||||
}
|
||||
if (ctx.signal.aborted || ctx.shutdownSignal.aborted) {
|
||||
stopReason = 'error';
|
||||
throw new Error('subagent aborted before turn');
|
||||
}
|
||||
|
||||
// 1. Acquire rate lease for the outbound call.
|
||||
const lease = await acquireLease(engine, rateLeaseKey, ctx.id, maxConcurrent, { ttlMs: leaseTtlMs });
|
||||
if (!lease.acquired) {
|
||||
// No slots — treat as a renewable error so the worker re-claims
|
||||
// the job later. Don't fail terminally.
|
||||
throw new RateLeaseUnavailableError(rateLeaseKey, lease.activeCount, lease.maxConcurrent);
|
||||
}
|
||||
|
||||
let assistantMsg: Anthropic.Message;
|
||||
const turnIdx = assistantTurns;
|
||||
const t0 = Date.now();
|
||||
logSubagentHeartbeat({ job_id: ctx.id, event: 'llm_call_started', turn_idx: turnIdx });
|
||||
|
||||
// Renewal is short-lived; for single-call turns the initial TTL
|
||||
// covers the whole request. A mid-call renewal loop would add
|
||||
// complexity; for v0.15 we lean on the 120s TTL + abort-on-signal.
|
||||
try {
|
||||
const params: Anthropic.MessageCreateParamsNonStreaming = {
|
||||
model,
|
||||
max_tokens: 4096,
|
||||
system: [
|
||||
{ type: 'text', text: systemPrompt, cache_control: { type: 'ephemeral' } },
|
||||
] as any,
|
||||
messages: anthroMessages,
|
||||
...(toolDefs.length > 0
|
||||
? {
|
||||
tools: toolDefs.map((t, i) => {
|
||||
const def: any = {
|
||||
name: t.name,
|
||||
description: t.description,
|
||||
input_schema: t.input_schema,
|
||||
};
|
||||
// Cache only the last tool def — Anthropic treats cache_control
|
||||
// as "cache everything up to and including this block".
|
||||
if (i === toolDefs.length - 1) def.cache_control = { type: 'ephemeral' };
|
||||
return def;
|
||||
}),
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
|
||||
const combinedSignal = mergeSignals(ctx.signal, ctx.shutdownSignal);
|
||||
assistantMsg = await client.create(params, { signal: combinedSignal });
|
||||
} catch (err) {
|
||||
// Release lease eagerly on error so we don't starve capacity.
|
||||
await releaseLease(engine, lease.leaseId!).catch(() => {});
|
||||
throw err;
|
||||
}
|
||||
|
||||
// 2. Release lease as soon as the call returns. Tool execution runs
|
||||
// outside the lease — tool calls use their own capacity.
|
||||
await releaseLease(engine, lease.leaseId!).catch(() => {});
|
||||
|
||||
const ms = Date.now() - t0;
|
||||
const inTokens = assistantMsg.usage?.input_tokens ?? 0;
|
||||
const outTokens = assistantMsg.usage?.output_tokens ?? 0;
|
||||
const cacheRead = (assistantMsg.usage as any)?.cache_read_input_tokens ?? 0;
|
||||
const cacheCreate = (assistantMsg.usage as any)?.cache_creation_input_tokens ?? 0;
|
||||
|
||||
tokenTotals.in += inTokens;
|
||||
tokenTotals.out += outTokens;
|
||||
tokenTotals.cache_read += cacheRead;
|
||||
tokenTotals.cache_create += cacheCreate;
|
||||
|
||||
logSubagentHeartbeat({
|
||||
job_id: ctx.id,
|
||||
event: 'llm_call_completed',
|
||||
turn_idx: turnIdx,
|
||||
ms_elapsed: ms,
|
||||
tokens: { in: inTokens, out: outTokens, cache_read: cacheRead, cache_create: cacheCreate },
|
||||
});
|
||||
|
||||
// Update job-level token rollup (best-effort; may throw if lock lost).
|
||||
await ctx.updateTokens({
|
||||
input: inTokens,
|
||||
output: outTokens,
|
||||
cache_read: cacheRead,
|
||||
});
|
||||
|
||||
const blocks = assistantMsg.content as ContentBlock[];
|
||||
|
||||
// 3. Persist the assistant message BEFORE tool dispatch so replay
|
||||
// sees a consistent state.
|
||||
const assistantIdx = nextMessageIdx++;
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: assistantIdx,
|
||||
role: 'assistant',
|
||||
content_blocks: blocks,
|
||||
tokens_in: inTokens,
|
||||
tokens_out: outTokens,
|
||||
tokens_cache_read: cacheRead,
|
||||
tokens_cache_create: cacheCreate,
|
||||
model,
|
||||
});
|
||||
anthroMessages.push({ role: 'assistant', content: blocks as any });
|
||||
assistantTurns++;
|
||||
|
||||
// 4. Collect tool_use blocks. If none, we're done.
|
||||
const toolUses = blocks.filter(
|
||||
(b): b is { type: 'tool_use'; id: string; name: string; input: unknown } & Record<string, unknown> =>
|
||||
b.type === 'tool_use',
|
||||
);
|
||||
if (toolUses.length === 0) {
|
||||
stopReason = 'end_turn';
|
||||
// Concatenate text blocks as the final answer.
|
||||
finalText = blocks
|
||||
.filter(b => b.type === 'text' && typeof b.text === 'string')
|
||||
.map(b => b.text as string)
|
||||
.join('\n');
|
||||
break;
|
||||
}
|
||||
|
||||
// 5. Dispatch each tool_use. Two-phase persist (pending → complete/failed).
|
||||
const toolResults: ContentBlock[] = [];
|
||||
for (const use of toolUses) {
|
||||
if (ctx.signal.aborted || ctx.shutdownSignal.aborted) {
|
||||
throw new Error('subagent aborted during tool dispatch');
|
||||
}
|
||||
|
||||
const toolName = use.name;
|
||||
const toolDef = toolDefs.find(t => t.name === toolName);
|
||||
if (!toolDef) {
|
||||
// Model called a tool we didn't expose. Mark execution failed
|
||||
// with a clear error and feed the error back in the next turn.
|
||||
await persistToolExecFailed(
|
||||
engine, ctx.id, assistantIdx, use.id, toolName, use.input,
|
||||
`tool "${toolName}" is not in the registry for this subagent`,
|
||||
);
|
||||
toolResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: `tool "${toolName}" is not available`,
|
||||
is_error: true,
|
||||
} as ContentBlock);
|
||||
logSubagentHeartbeat({
|
||||
job_id: ctx.id,
|
||||
event: 'tool_failed',
|
||||
turn_idx: turnIdx,
|
||||
tool_name: toolName,
|
||||
error: 'not in registry',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Replay: if we already have a row for this tool_use_id, trust it
|
||||
// unless status='pending' and the tool is idempotent (re-run).
|
||||
const prior = priorToolByUseId.get(use.id);
|
||||
if (prior && prior.status === 'complete') {
|
||||
toolResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: asStringIfNotObject(prior.output),
|
||||
} as ContentBlock);
|
||||
continue;
|
||||
}
|
||||
if (prior && prior.status === 'failed') {
|
||||
toolResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: prior.error ?? 'tool failed',
|
||||
is_error: true,
|
||||
} as ContentBlock);
|
||||
continue;
|
||||
}
|
||||
if (prior && prior.status === 'pending' && !toolDef.idempotent) {
|
||||
// Non-idempotent and we don't know the outcome — fail the job.
|
||||
throw new Error(`non-idempotent tool "${toolName}" pending on resume; cannot safely re-run`);
|
||||
}
|
||||
|
||||
// Fresh or idempotent-replay dispatch.
|
||||
await persistToolExecPending(engine, ctx.id, assistantIdx, use.id, toolName, use.input);
|
||||
logSubagentHeartbeat({ job_id: ctx.id, event: 'tool_called', turn_idx: turnIdx, tool_name: toolName });
|
||||
|
||||
const toolStart = Date.now();
|
||||
try {
|
||||
const output = await toolDef.execute(use.input, {
|
||||
engine,
|
||||
jobId: ctx.id,
|
||||
remote: true,
|
||||
signal: ctx.signal,
|
||||
});
|
||||
await persistToolExecComplete(engine, ctx.id, use.id, output);
|
||||
logSubagentHeartbeat({
|
||||
job_id: ctx.id,
|
||||
event: 'tool_result',
|
||||
turn_idx: turnIdx,
|
||||
tool_name: toolName,
|
||||
ms_elapsed: Date.now() - toolStart,
|
||||
});
|
||||
toolResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: asStringIfNotObject(output),
|
||||
} as ContentBlock);
|
||||
} catch (e) {
|
||||
const errText = e instanceof Error
|
||||
? (e.stack ?? e.message)
|
||||
: String(e);
|
||||
await persistToolExecFailed(engine, ctx.id, assistantIdx, use.id, toolName, use.input, errText);
|
||||
logSubagentHeartbeat({
|
||||
job_id: ctx.id,
|
||||
event: 'tool_failed',
|
||||
turn_idx: turnIdx,
|
||||
tool_name: toolName,
|
||||
ms_elapsed: Date.now() - toolStart,
|
||||
error: errText,
|
||||
});
|
||||
toolResults.push({
|
||||
type: 'tool_result',
|
||||
tool_use_id: use.id,
|
||||
content: errText,
|
||||
is_error: true,
|
||||
} as ContentBlock);
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Append the synthesized user turn (tool_result wrappers) to the
|
||||
// conversation and persist it so replay picks it up.
|
||||
const userIdx = nextMessageIdx++;
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: userIdx,
|
||||
role: 'user',
|
||||
content_blocks: toolResults,
|
||||
tokens_in: null,
|
||||
tokens_out: null,
|
||||
tokens_cache_read: null,
|
||||
tokens_cache_create: null,
|
||||
model: null,
|
||||
});
|
||||
anthroMessages.push({ role: 'user', content: toolResults as any });
|
||||
}
|
||||
|
||||
return {
|
||||
result: finalText,
|
||||
turns_count: assistantTurns,
|
||||
stop_reason: stopReason,
|
||||
tokens: tokenTotals,
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
// ── Internal: persistence ───────────────────────────────────
|
||||
|
||||
async function loadPriorMessages(engine: BrainEngine, jobId: number): Promise<PersistedMessage[]> {
|
||||
const rows = await engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT message_idx, role, content_blocks, tokens_in, tokens_out,
|
||||
tokens_cache_read, tokens_cache_create, model
|
||||
FROM subagent_messages
|
||||
WHERE job_id = $1
|
||||
ORDER BY message_idx ASC`,
|
||||
[jobId],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
message_idx: r.message_idx as number,
|
||||
role: r.role as 'user' | 'assistant',
|
||||
content_blocks: (typeof r.content_blocks === 'string'
|
||||
? JSON.parse(r.content_blocks as string)
|
||||
: r.content_blocks) as ContentBlock[],
|
||||
tokens_in: (r.tokens_in as number) ?? null,
|
||||
tokens_out: (r.tokens_out as number) ?? null,
|
||||
tokens_cache_read: (r.tokens_cache_read as number) ?? null,
|
||||
tokens_cache_create: (r.tokens_cache_create as number) ?? null,
|
||||
model: (r.model as string) ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
async function loadPriorTools(engine: BrainEngine, jobId: number): Promise<PersistedToolExec[]> {
|
||||
const rows = await engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT message_idx, tool_use_id, tool_name, input, status, output, error
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = $1`,
|
||||
[jobId],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
message_idx: r.message_idx as number,
|
||||
tool_use_id: r.tool_use_id as string,
|
||||
tool_name: r.tool_name as string,
|
||||
input: typeof r.input === 'string' ? JSON.parse(r.input) : r.input,
|
||||
status: r.status as 'pending' | 'complete' | 'failed',
|
||||
output: r.output == null
|
||||
? null
|
||||
: (typeof r.output === 'string' ? JSON.parse(r.output) : r.output),
|
||||
error: (r.error as string) ?? null,
|
||||
}));
|
||||
}
|
||||
|
||||
async function persistMessage(engine: BrainEngine, jobId: number, msg: PersistedMessage): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_messages (job_id, message_idx, role, content_blocks,
|
||||
tokens_in, tokens_out, tokens_cache_read, tokens_cache_create, model)
|
||||
VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT (job_id, message_idx) DO NOTHING`,
|
||||
[
|
||||
jobId,
|
||||
msg.message_idx,
|
||||
msg.role,
|
||||
JSON.stringify(msg.content_blocks),
|
||||
msg.tokens_in,
|
||||
msg.tokens_out,
|
||||
msg.tokens_cache_read,
|
||||
msg.tokens_cache_create,
|
||||
msg.model,
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
async function persistToolExecPending(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
messageIdx: number,
|
||||
toolUseId: string,
|
||||
toolName: string,
|
||||
input: unknown,
|
||||
): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 'pending')
|
||||
ON CONFLICT (job_id, tool_use_id) DO NOTHING`,
|
||||
[jobId, messageIdx, toolUseId, toolName, JSON.stringify(input)],
|
||||
);
|
||||
}
|
||||
|
||||
async function persistToolExecComplete(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
toolUseId: string,
|
||||
output: unknown,
|
||||
): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`UPDATE subagent_tool_executions
|
||||
SET status = 'complete', output = $3::jsonb, ended_at = now()
|
||||
WHERE job_id = $1 AND tool_use_id = $2`,
|
||||
[jobId, toolUseId, JSON.stringify(output)],
|
||||
);
|
||||
}
|
||||
|
||||
async function persistToolExecFailed(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
messageIdx: number,
|
||||
toolUseId: string,
|
||||
toolName: string,
|
||||
input: unknown,
|
||||
error: string,
|
||||
): Promise<void> {
|
||||
// INSERT-or-UPDATE to failed — covers both "no pending row yet" (tool
|
||||
// rejected upfront) and "pending row exists" (tool threw mid-execute).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status, error, ended_at)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 'failed', $6, now())
|
||||
ON CONFLICT (job_id, tool_use_id) DO UPDATE
|
||||
SET status = 'failed', error = EXCLUDED.error, ended_at = now()`,
|
||||
[jobId, messageIdx, toolUseId, toolName, JSON.stringify(input), error],
|
||||
);
|
||||
}
|
||||
|
||||
// ── Internal: helpers ───────────────────────────────────────
|
||||
|
||||
function asStringIfNotObject(value: unknown): string {
|
||||
if (typeof value === 'string') return value;
|
||||
try {
|
||||
return JSON.stringify(value);
|
||||
} catch {
|
||||
return String(value);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge two AbortSignals into one. Fires when either source aborts. No-op
|
||||
* polyfill when AbortSignal.any isn't available yet (Node ≥ 20 has it).
|
||||
*/
|
||||
function mergeSignals(a: AbortSignal, b: AbortSignal): AbortSignal {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const anyFn = (AbortSignal as any).any;
|
||||
if (typeof anyFn === 'function') return anyFn([a, b]) as AbortSignal;
|
||||
// Manual merge.
|
||||
const ac = new AbortController();
|
||||
if (a.aborted || b.aborted) ac.abort();
|
||||
else {
|
||||
a.addEventListener('abort', () => ac.abort(), { once: true });
|
||||
b.addEventListener('abort', () => ac.abort(), { once: true });
|
||||
}
|
||||
return ac.signal;
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when acquireLease returns acquired=false. The worker
|
||||
* treats this as a renewable error — job goes back to waiting with
|
||||
* backoff, no terminal fail.
|
||||
*/
|
||||
export class RateLeaseUnavailableError extends Error {
|
||||
constructor(public key: string, public active: number, public max: number) {
|
||||
super(`rate lease "${key}" full (${active}/${max})`);
|
||||
this.name = 'RateLeaseUnavailableError';
|
||||
}
|
||||
}
|
||||
|
||||
// ── Testing surface ─────────────────────────────────────────
|
||||
|
||||
export const __testing = {
|
||||
loadPriorMessages,
|
||||
loadPriorTools,
|
||||
persistMessage,
|
||||
persistToolExecPending,
|
||||
persistToolExecComplete,
|
||||
persistToolExecFailed,
|
||||
asStringIfNotObject,
|
||||
DEFAULT_MODEL,
|
||||
};
|
||||
@@ -0,0 +1,235 @@
|
||||
/**
|
||||
* GBRAIN_PLUGIN_PATH loader for host-repo subagent definitions (v0.15).
|
||||
*
|
||||
* Your OpenClaw (and future downstream agents) ship custom subagent defs
|
||||
* from their own repos. gbrain discovers them at worker startup via
|
||||
* GBRAIN_PLUGIN_PATH = colon-separated absolute paths (like $PATH). Each
|
||||
* path must contain a gbrain.plugin.json manifest describing the plugin
|
||||
* and a subagents/ subdirectory holding `*.md` definition files.
|
||||
*
|
||||
* Path policy is strict on purpose:
|
||||
* - ABSOLUTE paths only. Relative paths and `~` prefixes are rejected
|
||||
* (no implicit cwd or home expansion — too easy to pick up a tampered
|
||||
* sibling directory).
|
||||
* - Remote URLs (http://, https://, file://) rejected. Plugin loading
|
||||
* must go through the filesystem so the user controls what's there.
|
||||
* - Non-existent paths logged and skipped (do not fail worker startup).
|
||||
*
|
||||
* Collision policy: left-to-right wins. A warning goes to stderr naming
|
||||
* both sides of the collision.
|
||||
*
|
||||
* Trust policy: plugins ship subagent *defs* only. They cannot declare
|
||||
* new tools, cannot extend the brain-allowlist, cannot override
|
||||
* agent-safe flags. The `allowed_tools:` frontmatter field of a subagent
|
||||
* def must subset the derived registry — validation happens at plugin
|
||||
* load time, NOT at subagent dispatch time, so a typo in a plugin skill
|
||||
* fails loudly at worker startup instead of silently disabling a tool.
|
||||
*
|
||||
* Manifest version (`plugin_version`) locks the contract shape. Unknown
|
||||
* versions are rejected so the authoritative definition is whatever this
|
||||
* version of gbrain understands.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import matter from 'gray-matter';
|
||||
|
||||
export const SUPPORTED_PLUGIN_VERSION = 'gbrain-plugin-v1';
|
||||
|
||||
export interface PluginManifest {
|
||||
name: string;
|
||||
version: string;
|
||||
plugin_version: string;
|
||||
subagents?: string;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export interface SubagentDefinition {
|
||||
/** The plugin that shipped this def. */
|
||||
plugin_name: string;
|
||||
/** Stable agent name used as `subagent_def` by CLI callers. */
|
||||
name: string;
|
||||
/** Full path to the .md file on disk, for debug surfaces. */
|
||||
source_path: string;
|
||||
frontmatter: Record<string, unknown>;
|
||||
/** Markdown body (system prompt content). */
|
||||
body: string;
|
||||
/** Optional allowed_tools list (frontmatter). Subset of registry. */
|
||||
allowed_tools?: string[];
|
||||
}
|
||||
|
||||
export interface PluginLoadResult {
|
||||
/** Successfully loaded plugins with their subagents. */
|
||||
plugins: Array<{ manifest: PluginManifest; rootDir: string; subagents: SubagentDefinition[] }>;
|
||||
/** Per-path warnings (rejected, missing, malformed) collected during load. */
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
export interface LoadOpts {
|
||||
/**
|
||||
* Registry names the plugin's subagent `allowed_tools` must subset. When
|
||||
* present, any frontmatter entry not in this set fails the plugin load.
|
||||
* Pass `undefined` to skip validation (early worker startup before the
|
||||
* registry is built — but production callers should always pass it).
|
||||
*/
|
||||
validAgentToolNames?: ReadonlySet<string>;
|
||||
/** Override the PATH env (for tests). */
|
||||
envPath?: string;
|
||||
}
|
||||
|
||||
/** Public entry point: load every plugin directory from GBRAIN_PLUGIN_PATH. */
|
||||
export function loadPluginsFromEnv(opts: LoadOpts = {}): PluginLoadResult {
|
||||
const raw = opts.envPath ?? process.env.GBRAIN_PLUGIN_PATH ?? '';
|
||||
const paths = raw.split(':').map(s => s.trim()).filter(Boolean);
|
||||
const result: PluginLoadResult = { plugins: [], warnings: [] };
|
||||
|
||||
// Left-wins collision tracking.
|
||||
const subagentByName = new Map<string, { pluginName: string; pathLeft: string }>();
|
||||
|
||||
for (const p of paths) {
|
||||
const rejection = rejectIfNotAbsolute(p);
|
||||
if (rejection) { result.warnings.push(rejection); continue; }
|
||||
if (!fs.existsSync(p)) {
|
||||
result.warnings.push(`[plugin-loader] path does not exist, skipping: ${p}`);
|
||||
continue;
|
||||
}
|
||||
if (!fs.statSync(p).isDirectory()) {
|
||||
result.warnings.push(`[plugin-loader] not a directory, skipping: ${p}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const loaded = loadSinglePlugin(p, opts);
|
||||
if ('error' in loaded) {
|
||||
result.warnings.push(`[plugin-loader] rejected ${p}: ${loaded.error}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const accepted: SubagentDefinition[] = [];
|
||||
for (const sa of loaded.subagents) {
|
||||
const prior = subagentByName.get(sa.name);
|
||||
if (prior) {
|
||||
result.warnings.push(
|
||||
`[plugin-loader] collision: subagent '${sa.name}' from '${loaded.manifest.name}' at ${p} ` +
|
||||
`shadowed by earlier '${prior.pluginName}' at ${prior.pathLeft} (first wins)`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
subagentByName.set(sa.name, { pluginName: loaded.manifest.name, pathLeft: p });
|
||||
accepted.push(sa);
|
||||
}
|
||||
|
||||
result.plugins.push({ manifest: loaded.manifest, rootDir: p, subagents: accepted });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
result.warnings.push(`[plugin-loader] unexpected error loading ${p}: ${msg}`);
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
function rejectIfNotAbsolute(p: string): string | null {
|
||||
if (/^[a-z][a-z0-9+.-]*:\/\//i.test(p)) {
|
||||
return `[plugin-loader] remote URL rejected: ${p}`;
|
||||
}
|
||||
if (p.startsWith('~')) {
|
||||
return `[plugin-loader] ~-prefixed path rejected (expand explicitly): ${p}`;
|
||||
}
|
||||
if (!path.isAbsolute(p)) {
|
||||
return `[plugin-loader] relative path rejected: ${p}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export interface LoadedPlugin {
|
||||
manifest: PluginManifest;
|
||||
subagents: SubagentDefinition[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Load one plugin directory. Returns a union so callers can differentiate
|
||||
* rejection (loud but non-fatal) from an empty plugin (fatal-ish — the
|
||||
* manifest parsed but contributes nothing).
|
||||
*/
|
||||
export function loadSinglePlugin(
|
||||
rootDir: string,
|
||||
opts: LoadOpts = {},
|
||||
): LoadedPlugin | { error: string } {
|
||||
const manifestPath = path.join(rootDir, 'gbrain.plugin.json');
|
||||
if (!fs.existsSync(manifestPath)) {
|
||||
return { error: 'missing gbrain.plugin.json' };
|
||||
}
|
||||
|
||||
let manifest: PluginManifest;
|
||||
try {
|
||||
const raw = fs.readFileSync(manifestPath, 'utf8');
|
||||
manifest = JSON.parse(raw) as PluginManifest;
|
||||
} catch (e) {
|
||||
return { error: `invalid manifest JSON: ${e instanceof Error ? e.message : String(e)}` };
|
||||
}
|
||||
|
||||
if (typeof manifest.name !== 'string' || manifest.name.length === 0) {
|
||||
return { error: 'manifest missing required "name" field' };
|
||||
}
|
||||
if (manifest.plugin_version !== SUPPORTED_PLUGIN_VERSION) {
|
||||
return {
|
||||
error: `unsupported plugin_version "${manifest.plugin_version}" (gbrain supports "${SUPPORTED_PLUGIN_VERSION}")`,
|
||||
};
|
||||
}
|
||||
|
||||
const subagentsDirRel = manifest.subagents ?? 'subagents';
|
||||
const subagentsDir = path.resolve(rootDir, subagentsDirRel);
|
||||
// Prevent `../` escape via the manifest's `subagents` field.
|
||||
if (!subagentsDir.startsWith(rootDir + path.sep) && subagentsDir !== rootDir) {
|
||||
return { error: `subagents path escapes plugin root: ${subagentsDirRel}` };
|
||||
}
|
||||
|
||||
const subagents: SubagentDefinition[] = [];
|
||||
if (fs.existsSync(subagentsDir) && fs.statSync(subagentsDir).isDirectory()) {
|
||||
for (const entry of fs.readdirSync(subagentsDir)) {
|
||||
if (!entry.endsWith('.md')) continue;
|
||||
const sourcePath = path.join(subagentsDir, entry);
|
||||
try {
|
||||
const raw = fs.readFileSync(sourcePath, 'utf8');
|
||||
const parsed = matter(raw);
|
||||
const frontmatter = (parsed.data ?? {}) as Record<string, unknown>;
|
||||
const body = parsed.content ?? '';
|
||||
const name = typeof frontmatter.name === 'string'
|
||||
? frontmatter.name
|
||||
: entry.replace(/\.md$/, '');
|
||||
const allowed = Array.isArray(frontmatter.allowed_tools)
|
||||
? (frontmatter.allowed_tools as unknown[]).filter(x => typeof x === 'string') as string[]
|
||||
: undefined;
|
||||
|
||||
if (allowed && opts.validAgentToolNames) {
|
||||
const missing = allowed.filter(t => !opts.validAgentToolNames!.has(t));
|
||||
if (missing.length > 0) {
|
||||
return {
|
||||
error: `subagent '${name}' allowed_tools references unknown tools: ${missing.join(', ')}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
subagents.push({
|
||||
plugin_name: manifest.name,
|
||||
name,
|
||||
source_path: sourcePath,
|
||||
frontmatter,
|
||||
body,
|
||||
allowed_tools: allowed,
|
||||
});
|
||||
} catch (e) {
|
||||
return { error: `could not parse ${sourcePath}: ${e instanceof Error ? e.message : String(e)}` };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { manifest, subagents };
|
||||
}
|
||||
|
||||
/** Testing surface. */
|
||||
export const __testing = {
|
||||
rejectIfNotAbsolute,
|
||||
SUPPORTED_PLUGIN_VERSION,
|
||||
};
|
||||
@@ -12,7 +12,15 @@
|
||||
* pay them at module load.
|
||||
*/
|
||||
|
||||
export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set(['shell']);
|
||||
export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set([
|
||||
'shell',
|
||||
// v0.15: subagent + aggregator are protected because they call the
|
||||
// Anthropic API. MCP callers can't submit them directly; only the
|
||||
// `gbrain agent run` CLI path (which sets allowProtectedSubmit) or a
|
||||
// trusted local `submit_job` (ctx.remote=false) can insert these rows.
|
||||
'subagent',
|
||||
'subagent_aggregator',
|
||||
]);
|
||||
|
||||
/** Check a job name against the protected set. Normalizes whitespace first. */
|
||||
export function isProtectedJobName(name: string): boolean {
|
||||
|
||||
+225
-56
@@ -134,23 +134,40 @@ export class MinionQueue {
|
||||
|
||||
// 3. Insert child. Use ON CONFLICT for idempotency; if a concurrent submit
|
||||
// raced past the fast-path SELECT, the unique index catches it here.
|
||||
// v12 adds quiet_hours + stagger_key passed through from opts.
|
||||
const insertSql = opts?.idempotency_key
|
||||
? `INSERT INTO minion_jobs (name, queue, status, priority, data, max_attempts, backoff_type,
|
||||
// quiet_hours + stagger_key always present (null fallback; schema
|
||||
// stores NULL). max_stalled is conditional: provided values get
|
||||
// clamped to [1, 100] and included in the INSERT; omitted values
|
||||
// skip the column so the schema DEFAULT (5 as of v0.14.1) kicks in.
|
||||
// Keeps the app layer from hardcoding the schema default constant.
|
||||
//
|
||||
// Footgun note (codex iter 3): threading max_stalled on INSERT only is
|
||||
// deliberate. An idempotency-key hit returns the EXISTING row via the
|
||||
// fast-path SELECT above — we do NOT UPDATE max_stalled on a re-submit,
|
||||
// because letting a second submitter mutate the first submitter's
|
||||
// durability semantics is a nasty surprise.
|
||||
const hasMaxStalled = opts?.max_stalled !== undefined && opts.max_stalled !== null;
|
||||
const clampedMaxStalled = hasMaxStalled
|
||||
? Math.max(1, Math.min(100, Math.floor(opts!.max_stalled as number)))
|
||||
: null;
|
||||
|
||||
const baseCols = `name, queue, status, priority, data, max_attempts, backoff_type,
|
||||
backoff_delay, backoff_jitter, delay_until, parent_job_id, on_child_fail,
|
||||
depth, max_children, timeout_ms, remove_on_complete, remove_on_fail, idempotency_key,
|
||||
quiet_hours, stagger_key)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::jsonb, $20)
|
||||
quiet_hours, stagger_key`;
|
||||
const baseVals = `$1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::jsonb, $20`;
|
||||
const cols = hasMaxStalled ? `${baseCols}, max_stalled` : baseCols;
|
||||
const vals = hasMaxStalled ? `${baseVals}, $21` : baseVals;
|
||||
|
||||
const insertSql = opts?.idempotency_key
|
||||
? `INSERT INTO minion_jobs (${cols})
|
||||
VALUES (${vals})
|
||||
ON CONFLICT (idempotency_key) WHERE idempotency_key IS NOT NULL DO NOTHING
|
||||
RETURNING *`
|
||||
: `INSERT INTO minion_jobs (name, queue, status, priority, data, max_attempts, backoff_type,
|
||||
backoff_delay, backoff_jitter, delay_until, parent_job_id, on_child_fail,
|
||||
depth, max_children, timeout_ms, remove_on_complete, remove_on_fail, idempotency_key,
|
||||
quiet_hours, stagger_key)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::jsonb, $20)
|
||||
: `INSERT INTO minion_jobs (${cols})
|
||||
VALUES (${vals})
|
||||
RETURNING *`;
|
||||
|
||||
const params = [
|
||||
const params: unknown[] = [
|
||||
jobName,
|
||||
opts?.queue ?? 'default',
|
||||
childStatus,
|
||||
@@ -172,6 +189,7 @@ export class MinionQueue {
|
||||
opts?.quiet_hours ?? null,
|
||||
opts?.stagger_key ?? null,
|
||||
];
|
||||
if (hasMaxStalled) params.push(clampedMaxStalled);
|
||||
|
||||
const inserted = await tx.executeRaw<Record<string, unknown>>(insertSql, params);
|
||||
|
||||
@@ -274,29 +292,83 @@ export class MinionQueue {
|
||||
* Returns the *root* (the job matching id), not an arbitrary descendant.
|
||||
*/
|
||||
async cancelJob(id: number): Promise<MinionJob | null> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`WITH RECURSIVE descendants AS (
|
||||
SELECT id, 0 AS d FROM minion_jobs WHERE id = $1
|
||||
UNION ALL
|
||||
SELECT m.id, descendants.d + 1
|
||||
FROM minion_jobs m
|
||||
JOIN descendants ON m.parent_job_id = descendants.id
|
||||
WHERE descendants.d < 100
|
||||
)
|
||||
UPDATE minion_jobs SET
|
||||
status = 'cancelled',
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE id IN (SELECT id FROM descendants)
|
||||
AND status IN ('waiting','active','delayed','waiting-children','paused')
|
||||
RETURNING *`,
|
||||
[id]
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
const root = rows.find(r => (r.id as number) === id);
|
||||
return root ? rowToMinionJob(root) : null;
|
||||
return this.engine.transaction(async (tx) => {
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`WITH RECURSIVE descendants AS (
|
||||
SELECT id, 0 AS d FROM minion_jobs WHERE id = $1
|
||||
UNION ALL
|
||||
SELECT m.id, descendants.d + 1
|
||||
FROM minion_jobs m
|
||||
JOIN descendants ON m.parent_job_id = descendants.id
|
||||
WHERE descendants.d < 100
|
||||
)
|
||||
UPDATE minion_jobs SET
|
||||
status = 'cancelled',
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE id IN (SELECT id FROM descendants)
|
||||
AND status IN ('waiting','active','delayed','waiting-children','paused')
|
||||
RETURNING *`,
|
||||
[id]
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
|
||||
// v0.15: emit child_done(outcome='cancelled') for every cancelled row
|
||||
// that had a parent. Without this, an aggregator waiting for N
|
||||
// child_done messages hangs forever when a child is cancelled (codex
|
||||
// iteration 3). Also unblock any aggregator parents whose last
|
||||
// non-terminal child we just cancelled.
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const childId = r.id as number;
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
const name = r.name as string;
|
||||
// Skip the root if it's the caller's cancel target AND has no parent.
|
||||
// Descendants whose parent got cancelled in the same sweep still
|
||||
// benefit from the inbox message — their parent exits waiting-children
|
||||
// via the resolve sweep below even though the parent is itself
|
||||
// cancelled (EXISTS guard on inbox INSERT handles it).
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: childId,
|
||||
job_name: name,
|
||||
result: null,
|
||||
outcome: 'cancelled',
|
||||
error: 'cancelled',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
// Resolve any non-cancelled aggregator parents sitting on
|
||||
// waiting-children whose last open child we just cancelled.
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
|
||||
const root = rows.find(r => (r.id as number) === id);
|
||||
return root ? rowToMinionJob(root) : null;
|
||||
});
|
||||
}
|
||||
|
||||
/** Re-queue a failed or dead job for retry. */
|
||||
@@ -428,21 +500,67 @@ export class MinionQueue {
|
||||
* but will be caught the next one (after re-claim). Never double-handled.
|
||||
*/
|
||||
async handleTimeouts(): Promise<MinionJob[]> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
error_text = 'timeout exceeded',
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND timeout_at IS NOT NULL
|
||||
AND timeout_at < now()
|
||||
AND lock_until > now()
|
||||
RETURNING *`
|
||||
);
|
||||
return rows.map(rowToMinionJob);
|
||||
return this.engine.transaction(async (tx) => {
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
error_text = 'timeout exceeded',
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND timeout_at IS NOT NULL
|
||||
AND timeout_at < now()
|
||||
AND lock_until > now()
|
||||
RETURNING *`
|
||||
);
|
||||
|
||||
// v0.15: emit child_done(outcome='timeout') for every timed-out job that
|
||||
// had a parent. Without this, an aggregator waiting for N child_done
|
||||
// messages hangs forever when a child times out (codex iteration 3).
|
||||
// Outcome 'timeout' is distinct from 'dead' so consumers can distinguish
|
||||
// "timed out during run" from "died via max-stall".
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: r.id as number,
|
||||
job_name: r.name as string,
|
||||
result: null,
|
||||
outcome: 'timeout',
|
||||
error: 'timeout exceeded',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
// Unblock any aggregator parents whose last open child we just killed.
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
|
||||
return rows.map(rowToMinionJob);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -512,6 +630,7 @@ export class MinionQueue {
|
||||
child_id: completed.id,
|
||||
job_name: completed.name,
|
||||
result: result ?? null,
|
||||
outcome: 'complete',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
@@ -523,14 +642,17 @@ export class MinionQueue {
|
||||
[completed.parent_job_id, childDone]
|
||||
);
|
||||
|
||||
// Fold-in resolveParent: flip parent to waiting once all children done.
|
||||
// Fold-in resolveParent: flip parent to waiting once all children are
|
||||
// in ANY terminal state. Terminal set includes 'failed' so a failed
|
||||
// child with on_child_fail='continue'/'ignore' doesn't strand the
|
||||
// parent in waiting-children forever (v0.15 aggregator fix).
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'dead', 'cancelled')
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[completed.parent_job_id]
|
||||
);
|
||||
@@ -604,6 +726,29 @@ export class MinionQueue {
|
||||
|
||||
// Parent hook on terminal failure.
|
||||
if (terminal && failed.parent_job_id) {
|
||||
// v0.15: emit child_done(outcome='failed') BEFORE any parent-terminal
|
||||
// update. Insertion order matters because `completeJob`'s inbox-write
|
||||
// EXISTS guard skips writes once the parent is 'failed' — if we let
|
||||
// the fail_parent UPDATE run first, this inbox row would be dropped
|
||||
// for aggregator-style parents that still want to count it (codex).
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: failed.id,
|
||||
job_name: failed.name,
|
||||
result: null,
|
||||
outcome: newStatus === 'dead' ? 'dead' : 'failed',
|
||||
error: errorText,
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[failed.parent_job_id, childDone]
|
||||
);
|
||||
|
||||
if (failed.on_child_fail === 'fail_parent') {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'failed',
|
||||
@@ -616,19 +761,37 @@ export class MinionQueue {
|
||||
`UPDATE minion_jobs SET parent_job_id = NULL, updated_at = now() WHERE id = $1`,
|
||||
[failed.id]
|
||||
);
|
||||
// After dropping the dep, try to resolve the parent if all OTHER kids are done.
|
||||
// After dropping the dep, try to resolve the parent if all OTHER
|
||||
// kids are terminal. Terminal set includes 'failed' (v0.15).
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'dead', 'cancelled')
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[failed.parent_job_id]
|
||||
);
|
||||
} else {
|
||||
// 'ignore' / 'continue': parent stays in waiting-children waiting on
|
||||
// siblings. With v0.15 terminal-set expansion + child_done emission
|
||||
// above, an aggregator sibling-count model now works: all N children
|
||||
// reach terminal → completeJob on a sibling (or the LAST terminal
|
||||
// transition here) flips parent → waiting once no non-terminal kids
|
||||
// remain. Run the resolve check here so the last child transitioning
|
||||
// via THIS code path still unblocks the parent.
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[failed.parent_job_id]
|
||||
);
|
||||
}
|
||||
// 'ignore' / 'continue' → parent stays in waiting-children waiting on siblings
|
||||
}
|
||||
|
||||
// remove_on_fail cleanup AFTER parent hook.
|
||||
@@ -713,7 +876,13 @@ export class MinionQueue {
|
||||
return { requeued, dead };
|
||||
}
|
||||
|
||||
/** Check if all children of a parent are done. If so, unblock parent. */
|
||||
/**
|
||||
* Check if all children of a parent are in ANY terminal state. If so,
|
||||
* unblock parent (flip waiting-children → waiting).
|
||||
*
|
||||
* v0.15: terminal set includes 'failed' so a child failing with
|
||||
* on_child_fail='continue'/'ignore' doesn't strand the parent.
|
||||
*/
|
||||
async resolveParent(parentId: number): Promise<MinionJob | null> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
@@ -721,7 +890,7 @@ export class MinionQueue {
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'dead', 'cancelled')
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)
|
||||
RETURNING *`,
|
||||
[parentId]
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* Lease-based rate limiter for outbound providers (e.g. anthropic:messages).
|
||||
*
|
||||
* Counter-based limiters leak capacity when a worker crashes mid-call
|
||||
* (counter never decrements). Leases are owner-tagged rows with an expires_at
|
||||
* timestamp — crash recovery is free: any row past expires_at is considered
|
||||
* dead on the next acquire and pruned before the active-count check.
|
||||
*
|
||||
* Two-phase acquire:
|
||||
* 1. Pre-prune: DELETE expired leases for this key (same txn).
|
||||
* 2. Check-then-insert under a txn-scoped advisory lock so two concurrent
|
||||
* acquires can't both see "one slot left".
|
||||
*
|
||||
* The owner is always a Minion job id; the lease is CASCADE-tied to
|
||||
* minion_jobs so an out-of-band row DELETE (prune, cancel) doesn't leave
|
||||
* stale leases. Mid-call renewal bumps expires_at in-place.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
|
||||
/**
|
||||
* Acquisition result. If `acquired=false`, the caller should back off and
|
||||
* retry — we don't queue, we just reject.
|
||||
*/
|
||||
export interface LeaseAcquireResult {
|
||||
acquired: boolean;
|
||||
/** The lease row id, present only when acquired=true. */
|
||||
leaseId?: number;
|
||||
/** Active count seen at acquire time (for diagnostics). */
|
||||
activeCount: number;
|
||||
/** max_concurrent that was checked against. */
|
||||
maxConcurrent: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a key string to a stable int64 for pg_advisory_xact_lock. Simple
|
||||
* FNV-1a is fine — the lock space is per-transaction and we only need
|
||||
* different keys to (usually) hash to different locks.
|
||||
*/
|
||||
function hashKey(key: string): bigint {
|
||||
// FNV-1a 64-bit
|
||||
let h = 0xcbf29ce484222325n;
|
||||
const prime = 0x100000001b3n;
|
||||
for (let i = 0; i < key.length; i++) {
|
||||
h ^= BigInt(key.charCodeAt(i));
|
||||
h = (h * prime) & 0xffffffffffffffffn;
|
||||
}
|
||||
// Fit into signed int64 for PG bigint. The high bit gets clipped in the
|
||||
// arithmetic above already, but be explicit.
|
||||
const signBit = 0x8000000000000000n;
|
||||
return h & (signBit - 1n);
|
||||
}
|
||||
|
||||
const DEFAULT_TTL_MS = 120_000;
|
||||
|
||||
export interface AcquireOpts {
|
||||
ttlMs?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Attempt to acquire a lease on `key`. Returns `{acquired: false}` when the
|
||||
* active count (after pre-pruning stale rows) would exceed maxConcurrent.
|
||||
*
|
||||
* The call MUST run inside a transaction for the advisory lock + insert to
|
||||
* be atomic. Pass in the engine — the helper wraps the txn internally.
|
||||
*/
|
||||
export async function acquireLease(
|
||||
engine: BrainEngine,
|
||||
key: string,
|
||||
ownerJobId: number,
|
||||
maxConcurrent: number,
|
||||
opts: AcquireOpts = {},
|
||||
): Promise<LeaseAcquireResult> {
|
||||
const ttlMs = opts.ttlMs ?? DEFAULT_TTL_MS;
|
||||
const lockKey = hashKey(key);
|
||||
|
||||
return engine.transaction(async (tx) => {
|
||||
// txn-scoped advisory lock keyed on the rate-lease key name. Released
|
||||
// automatically when the txn commits/rolls back.
|
||||
await tx.executeRaw(`SELECT pg_advisory_xact_lock($1::bigint)`, [lockKey.toString()]);
|
||||
|
||||
// Pre-prune stale leases for this key.
|
||||
await tx.executeRaw(
|
||||
`DELETE FROM subagent_rate_leases WHERE key = $1 AND expires_at <= now()`,
|
||||
[key],
|
||||
);
|
||||
|
||||
const countRows = await tx.executeRaw<{ count: string | number }>(
|
||||
`SELECT count(*)::text AS count FROM subagent_rate_leases WHERE key = $1`,
|
||||
[key],
|
||||
);
|
||||
const activeCount = parseInt(String(countRows[0]?.count ?? '0'), 10);
|
||||
|
||||
if (activeCount >= maxConcurrent) {
|
||||
return { acquired: false, activeCount, maxConcurrent };
|
||||
}
|
||||
|
||||
const rows = await tx.executeRaw<{ id: number }>(
|
||||
`INSERT INTO subagent_rate_leases (key, owner_job_id, expires_at)
|
||||
VALUES ($1, $2, now() + ($3::double precision * interval '1 millisecond'))
|
||||
RETURNING id`,
|
||||
[key, ownerJobId, ttlMs],
|
||||
);
|
||||
const leaseId = rows[0]!.id;
|
||||
return { acquired: true, leaseId, activeCount: activeCount + 1, maxConcurrent };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Renew a lease's expires_at (mid-call). Returns true if the lease still
|
||||
* exists (was renewed), false if it was pruned (caller must re-acquire or
|
||||
* abort).
|
||||
*/
|
||||
export async function renewLease(engine: BrainEngine, leaseId: number, ttlMs = DEFAULT_TTL_MS): Promise<boolean> {
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`UPDATE subagent_rate_leases
|
||||
SET expires_at = now() + ($2::double precision * interval '1 millisecond')
|
||||
WHERE id = $1
|
||||
RETURNING id`,
|
||||
[leaseId, ttlMs],
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Release a lease explicitly. Idempotent — a missing lease returns silently
|
||||
* (it was pruned or the owning job row cascade-deleted it).
|
||||
*/
|
||||
export async function releaseLease(engine: BrainEngine, leaseId: number): Promise<void> {
|
||||
await engine.executeRaw(`DELETE FROM subagent_rate_leases WHERE id = $1`, [leaseId]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Attempt to renew with 3x exponential backoff (250ms / 500ms / 1s). Used
|
||||
* mid-LLM-call when the first renewal attempt hits a DB blip. On all-three
|
||||
* failure the caller must abort with a renewable error so the worker
|
||||
* re-claims the job.
|
||||
*/
|
||||
export async function renewLeaseWithBackoff(engine: BrainEngine, leaseId: number, ttlMs = DEFAULT_TTL_MS): Promise<boolean> {
|
||||
const delays = [0, 250, 500, 1000]; // first attempt immediate, then 250/500/1000
|
||||
for (const delay of delays) {
|
||||
if (delay > 0) await new Promise(r => setTimeout(r, delay));
|
||||
try {
|
||||
if (await renewLease(engine, leaseId, ttlMs)) return true;
|
||||
// Lease is gone (pruned). No point retrying — caller must abort.
|
||||
return false;
|
||||
} catch {
|
||||
// DB blip; fall through to next delay.
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
/**
|
||||
* Derive the subagent brain-tool registry from src/core/operations.ts.
|
||||
*
|
||||
* Single source of truth: the MCP server already maps OPERATIONS → tool defs.
|
||||
* We reuse the same ParamDef-shape → JSONSchema conversion (lives in
|
||||
* buildToolDefs for MCP) and wrap each allowed op with an execute() that
|
||||
* invokes its handler under a subagent-tagged OperationContext.
|
||||
*
|
||||
* Filtering is NAME-based (not by OperationContext.remote, which is a
|
||||
* call-time flag, not operation metadata — codex catch). The allow-list
|
||||
* below is reviewed manually; adding a new op here is an explicit security
|
||||
* decision.
|
||||
*
|
||||
* put_page: allowed, but the subagent tool-schema wraps its `slug` with a
|
||||
* per-subagent namespace regex so the model can only write under
|
||||
* `wiki/agents/<subagentId>/...`. The put_page operation also has a server-
|
||||
* side fail-closed check (see src/core/operations.ts) that catches any
|
||||
* dispatcher bug where viaSubagent=true but subagentId is missing.
|
||||
*
|
||||
* In v0.15 every allow-list op is treated as idempotent for the two-phase
|
||||
* replay path. put_page with a deterministic slug is idempotent at the row
|
||||
* level; repeats re-derive the same embedding over identical content.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { GBrainConfig } from '../../config.ts';
|
||||
import { operations } from '../../operations.ts';
|
||||
import type { Operation, OperationContext } from '../../operations.ts';
|
||||
import type { ToolCtx, ToolDef } from '../types.ts';
|
||||
|
||||
/**
|
||||
* v0.15 brain-tool allow-list. Review carefully when extending. Op names
|
||||
* verified against origin/master:src/core/operations.ts (post shell-jobs +
|
||||
* Knowledge Runtime).
|
||||
*
|
||||
* Read-only (all safe):
|
||||
* query, search, get_page, list_pages, file_list, file_url,
|
||||
* get_backlinks, traverse_graph, resolve_slugs, get_ingest_log
|
||||
*
|
||||
* Conditional write:
|
||||
* put_page (namespace-enforced by the tool schema + server-side check)
|
||||
*
|
||||
* Every name below MUST exist in src/core/operations.ts OPERATIONS; the
|
||||
* brain-allowlist test pins this invariant so an upstream rename fails CI
|
||||
* instead of silently dropping a tool.
|
||||
*/
|
||||
export const BRAIN_TOOL_ALLOWLIST: ReadonlySet<string> = new Set([
|
||||
'query',
|
||||
'search',
|
||||
'get_page',
|
||||
'list_pages',
|
||||
'file_list',
|
||||
'file_url',
|
||||
'get_backlinks',
|
||||
'traverse_graph',
|
||||
'resolve_slugs',
|
||||
'get_ingest_log',
|
||||
'put_page',
|
||||
]);
|
||||
|
||||
/** Matches Anthropic's tool-name constraint. No dots. */
|
||||
const ANTHROPIC_NAME_RE = /^[a-zA-Z0-9_-]{1,64}$/;
|
||||
|
||||
function sanitizeToolName(opName: string): string {
|
||||
// Prefix with brain_ and replace any non-conforming char. For the v0.15
|
||||
// allow-list, every op name is already a valid simple identifier, so this
|
||||
// is defense-in-depth.
|
||||
const prefixed = `brain_${opName}`.replace(/[^a-zA-Z0-9_-]/g, '_');
|
||||
return prefixed.slice(0, 64);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert an Operation.params (ParamDef) map to an Anthropic-compatible
|
||||
* JSONSchema.input_schema. Same shape MCP uses inline — ParamDef.type
|
||||
* narrows to a subset of JSONSchema types.
|
||||
*/
|
||||
function paramsToInputSchema(op: Operation): Record<string, unknown> {
|
||||
return {
|
||||
type: 'object' as const,
|
||||
properties: Object.fromEntries(
|
||||
Object.entries(op.params).map(([k, v]) => [k, {
|
||||
type: v.type === 'array' ? 'array' : v.type,
|
||||
...(v.description ? { description: v.description } : {}),
|
||||
...(v.enum ? { enum: v.enum } : {}),
|
||||
...(v.items ? { items: { type: v.items.type } } : {}),
|
||||
}]),
|
||||
),
|
||||
required: Object.entries(op.params).filter(([, v]) => v.required).map(([k]) => k),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* For put_page specifically, the tool schema shown to the model constrains
|
||||
* `slug` to `wiki/agents/<subagentId>/...`. The server-side check in
|
||||
* operations.ts is the authoritative gate; this just helps the model write
|
||||
* correct slugs on the first try.
|
||||
*/
|
||||
function namespacedPutPageSchema(op: Operation, subagentId: number): Record<string, unknown> {
|
||||
const base = paramsToInputSchema(op);
|
||||
const props = (base.properties as Record<string, Record<string, unknown>>) ?? {};
|
||||
if (props.slug) {
|
||||
props.slug = {
|
||||
...props.slug,
|
||||
description: `Page slug. MUST start with "wiki/agents/${subagentId}/" (agents can only write under their own namespace).`,
|
||||
pattern: `^wiki/agents/${subagentId}/.+`,
|
||||
};
|
||||
}
|
||||
return { ...base, properties: props };
|
||||
}
|
||||
|
||||
/** Args required to build the registry for a given subagent job. */
|
||||
export interface BuildBrainToolsOpts {
|
||||
subagentId: number;
|
||||
engine: BrainEngine;
|
||||
config: GBrainConfig;
|
||||
/** Optional filter: only include names in this set. */
|
||||
allowedNames?: ReadonlySet<string>;
|
||||
}
|
||||
|
||||
interface OpContextDeps {
|
||||
engine: BrainEngine;
|
||||
config: GBrainConfig;
|
||||
subagentId: number;
|
||||
jobId: number;
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
return {
|
||||
engine: deps.engine,
|
||||
config: deps.config,
|
||||
logger: {
|
||||
info: (msg: string) => process.stderr.write(`[subagent-tool:${deps.jobId}] ${msg}\n`),
|
||||
warn: (msg: string) => process.stderr.write(`[subagent-tool:${deps.jobId}] WARN: ${msg}\n`),
|
||||
error: (msg: string) => process.stderr.write(`[subagent-tool:${deps.jobId}] ERROR: ${msg}\n`),
|
||||
},
|
||||
dryRun: false,
|
||||
remote: true, // match MCP trust boundary
|
||||
jobId: deps.jobId,
|
||||
subagentId: deps.subagentId,
|
||||
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the subagent brain-tool registry. One ToolDef per allow-listed op,
|
||||
* with a namespace-wrapped schema for put_page.
|
||||
*
|
||||
* Call this once per subagent-job claim; the registry is keyed to the job's
|
||||
* subagentId + engine handle, so it's not shareable across jobs.
|
||||
*/
|
||||
export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
const filter = opts.allowedNames ?? BRAIN_TOOL_ALLOWLIST;
|
||||
const picked: Operation[] = operations.filter(
|
||||
op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name),
|
||||
);
|
||||
|
||||
return picked.map<ToolDef>(op => {
|
||||
const schema = op.name === 'put_page'
|
||||
? namespacedPutPageSchema(op, opts.subagentId)
|
||||
: paramsToInputSchema(op);
|
||||
|
||||
const toolName = sanitizeToolName(op.name);
|
||||
if (!ANTHROPIC_NAME_RE.test(toolName)) {
|
||||
throw new Error(`brain tool name ${toolName} does not match Anthropic constraint`);
|
||||
}
|
||||
|
||||
return {
|
||||
name: toolName,
|
||||
description: op.description,
|
||||
input_schema: schema,
|
||||
// v0.15 ships only idempotent brain tools (every allow-listed op is
|
||||
// deterministic over its input; put_page re-writes the same slug).
|
||||
idempotent: true,
|
||||
async execute(input: unknown, ctx: ToolCtx): Promise<unknown> {
|
||||
const opCtx = buildOpContext({
|
||||
engine: ctx.engine,
|
||||
config: opts.config,
|
||||
subagentId: opts.subagentId,
|
||||
jobId: ctx.jobId,
|
||||
signal: ctx.signal,
|
||||
});
|
||||
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
|
||||
return op.handler(opCtx, params);
|
||||
},
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the caller's `allowed_tools` subset to a registry. Unknown tool
|
||||
* names throw a clear error at load time (NOT silently ignored) so
|
||||
* subagent defs with a typo don't ship to prod wondering why a tool
|
||||
* never fires.
|
||||
*/
|
||||
export function filterAllowedTools(registry: ToolDef[], allowedToolNames: string[]): ToolDef[] {
|
||||
const indexByName = new Map(registry.map(t => [t.name, t]));
|
||||
// Also index by the un-prefixed op name (for friendlier allowed_tools entries).
|
||||
const indexByShort = new Map(
|
||||
registry.map(t => [t.name.replace(/^brain_/, ''), t]),
|
||||
);
|
||||
const seen = new Set<string>();
|
||||
const picked: ToolDef[] = [];
|
||||
for (const requested of allowedToolNames) {
|
||||
const match = indexByName.get(requested) ?? indexByShort.get(requested);
|
||||
if (!match) {
|
||||
throw new Error(
|
||||
`subagent allowed_tools references unknown tool "${requested}". ` +
|
||||
`Known: ${[...indexByName.keys()].join(', ')}`,
|
||||
);
|
||||
}
|
||||
if (seen.has(match.name)) continue;
|
||||
seen.add(match.name);
|
||||
picked.push(match);
|
||||
}
|
||||
return picked;
|
||||
}
|
||||
|
||||
/** Exported for unit tests (stable surface). */
|
||||
export const __testing = {
|
||||
sanitizeToolName,
|
||||
paramsToInputSchema,
|
||||
namespacedPutPageSchema,
|
||||
ANTHROPIC_NAME_RE,
|
||||
};
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* Render a subagent conversation to markdown.
|
||||
*
|
||||
* Two inputs:
|
||||
* - subagent_messages rows (persisted Anthropic message-block arrays)
|
||||
* - subagent_tool_executions rows (two-phase tool ledger — used to show
|
||||
* tool outputs alongside the model's tool_use calls)
|
||||
*
|
||||
* The output is suitable for:
|
||||
* - an attachment on the completed subagent job row
|
||||
* - inline display in `gbrain agent logs <job>` after the heartbeat stream
|
||||
* - committing as a brain page under wiki/agents/<subagentId>/transcript-*
|
||||
*
|
||||
* Does NOT redact anything — the caller writes to a location they control.
|
||||
* For PII-sensitive deployments, pass through a sanitizer before persisting.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { ContentBlock } from './types.ts';
|
||||
|
||||
export interface SubagentMessageRow {
|
||||
id: number;
|
||||
job_id: number;
|
||||
message_idx: number;
|
||||
role: 'user' | 'assistant';
|
||||
content_blocks: ContentBlock[];
|
||||
tokens_in: number | null;
|
||||
tokens_out: number | null;
|
||||
tokens_cache_read: number | null;
|
||||
tokens_cache_create: number | null;
|
||||
model: string | null;
|
||||
ended_at: Date;
|
||||
}
|
||||
|
||||
export interface SubagentToolExecRow {
|
||||
id: number;
|
||||
job_id: number;
|
||||
message_idx: number;
|
||||
tool_use_id: string;
|
||||
tool_name: string;
|
||||
input: unknown;
|
||||
status: 'pending' | 'complete' | 'failed';
|
||||
output: unknown;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
/** Fetch both row sets for a job in one shot. */
|
||||
export async function loadTranscriptRows(
|
||||
engine: BrainEngine,
|
||||
jobId: number,
|
||||
): Promise<{ messages: SubagentMessageRow[]; tools: SubagentToolExecRow[] }> {
|
||||
const msgRows = await engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT id, job_id, message_idx, role, content_blocks, tokens_in, tokens_out,
|
||||
tokens_cache_read, tokens_cache_create, model, ended_at
|
||||
FROM subagent_messages
|
||||
WHERE job_id = $1
|
||||
ORDER BY message_idx ASC`,
|
||||
[jobId],
|
||||
);
|
||||
const toolRows = await engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT id, job_id, message_idx, tool_use_id, tool_name, input, status, output, error
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = $1
|
||||
ORDER BY id ASC`,
|
||||
[jobId],
|
||||
);
|
||||
return {
|
||||
messages: msgRows.map(normalizeMessage),
|
||||
tools: toolRows.map(normalizeTool),
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeMessage(row: Record<string, unknown>): SubagentMessageRow {
|
||||
const blocks = row.content_blocks;
|
||||
const parsedBlocks: ContentBlock[] = typeof blocks === 'string'
|
||||
? (JSON.parse(blocks) as ContentBlock[])
|
||||
: (blocks as ContentBlock[]) ?? [];
|
||||
return {
|
||||
id: row.id as number,
|
||||
job_id: row.job_id as number,
|
||||
message_idx: row.message_idx as number,
|
||||
role: row.role as 'user' | 'assistant',
|
||||
content_blocks: parsedBlocks,
|
||||
tokens_in: (row.tokens_in as number) ?? null,
|
||||
tokens_out: (row.tokens_out as number) ?? null,
|
||||
tokens_cache_read: (row.tokens_cache_read as number) ?? null,
|
||||
tokens_cache_create: (row.tokens_cache_create as number) ?? null,
|
||||
model: (row.model as string) ?? null,
|
||||
ended_at: new Date(row.ended_at as string),
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeTool(row: Record<string, unknown>): SubagentToolExecRow {
|
||||
const input = typeof row.input === 'string' ? JSON.parse(row.input) : row.input;
|
||||
const output = row.output == null
|
||||
? null
|
||||
: (typeof row.output === 'string' ? JSON.parse(row.output) : row.output);
|
||||
return {
|
||||
id: row.id as number,
|
||||
job_id: row.job_id as number,
|
||||
message_idx: row.message_idx as number,
|
||||
tool_use_id: row.tool_use_id as string,
|
||||
tool_name: row.tool_name as string,
|
||||
input,
|
||||
status: row.status as 'pending' | 'complete' | 'failed',
|
||||
output,
|
||||
error: (row.error as string) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
export interface RenderTranscriptOpts {
|
||||
/** Trim long tool outputs in the markdown. Default: 4 KiB per output. */
|
||||
maxOutputBytes?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render messages + tool executions to markdown. Message order is
|
||||
* authoritative; tool rows are spliced under their owning assistant message
|
||||
* by tool_use_id.
|
||||
*/
|
||||
export function renderTranscript(
|
||||
messages: SubagentMessageRow[],
|
||||
tools: SubagentToolExecRow[],
|
||||
opts: RenderTranscriptOpts = {},
|
||||
): string {
|
||||
const maxOut = opts.maxOutputBytes ?? 4096;
|
||||
const toolById = new Map<string, SubagentToolExecRow>(
|
||||
tools.map(t => [t.tool_use_id, t]),
|
||||
);
|
||||
|
||||
const out: string[] = [];
|
||||
out.push('# Subagent transcript', '');
|
||||
if (messages.length === 0) {
|
||||
out.push('_(no messages)_');
|
||||
return out.join('\n');
|
||||
}
|
||||
|
||||
const first = messages[0]!;
|
||||
out.push(`- job_id: ${first.job_id}`);
|
||||
out.push(`- messages: ${messages.length}`);
|
||||
if (first.model) out.push(`- model: ${first.model}`);
|
||||
out.push('');
|
||||
|
||||
for (const msg of messages) {
|
||||
out.push(`## Message ${msg.message_idx} — ${msg.role}`);
|
||||
if (msg.tokens_in != null || msg.tokens_out != null) {
|
||||
const parts: string[] = [];
|
||||
if (msg.tokens_in) parts.push(`in=${msg.tokens_in}`);
|
||||
if (msg.tokens_out) parts.push(`out=${msg.tokens_out}`);
|
||||
if (msg.tokens_cache_read) parts.push(`cache_read=${msg.tokens_cache_read}`);
|
||||
if (msg.tokens_cache_create) parts.push(`cache_create=${msg.tokens_cache_create}`);
|
||||
if (parts.length > 0) out.push(`> tokens: ${parts.join(' ')}`);
|
||||
}
|
||||
out.push('');
|
||||
|
||||
for (const block of msg.content_blocks) {
|
||||
renderBlock(block, toolById, maxOut, out);
|
||||
}
|
||||
out.push('');
|
||||
}
|
||||
|
||||
return out.join('\n').replace(/\n{3,}/g, '\n\n');
|
||||
}
|
||||
|
||||
function renderBlock(
|
||||
block: ContentBlock,
|
||||
toolById: Map<string, SubagentToolExecRow>,
|
||||
maxOutputBytes: number,
|
||||
out: string[],
|
||||
): void {
|
||||
if (block.type === 'text' && typeof block.text === 'string') {
|
||||
out.push(block.text);
|
||||
out.push('');
|
||||
return;
|
||||
}
|
||||
|
||||
if (block.type === 'tool_use') {
|
||||
const name = typeof block.name === 'string' ? block.name : '<unknown>';
|
||||
const inputStr = safeJson(block.input, 2);
|
||||
out.push(`**tool_use** \`${name}\` (id=\`${block.id ?? '?'}\`)`);
|
||||
out.push('```json', inputStr, '```');
|
||||
const toolRow = block.id && typeof block.id === 'string' ? toolById.get(block.id) : undefined;
|
||||
if (toolRow) {
|
||||
out.push(`→ status: **${toolRow.status}**`);
|
||||
if (toolRow.status === 'complete') {
|
||||
out.push('```json', truncate(safeJson(toolRow.output, 2), maxOutputBytes), '```');
|
||||
} else if (toolRow.status === 'failed') {
|
||||
out.push(`> error: ${toolRow.error ?? '(no error text)'}`);
|
||||
} else if (toolRow.status === 'pending') {
|
||||
out.push('> pending (no resolution recorded yet)');
|
||||
}
|
||||
}
|
||||
out.push('');
|
||||
return;
|
||||
}
|
||||
|
||||
if (block.type === 'tool_result') {
|
||||
// Most tool_result blocks live inside user messages echoing back the
|
||||
// assistant's tool_use. We skip them here because the owning tool_use
|
||||
// block already rendered the execution row. If the user message carries
|
||||
// a raw tool_result with no matching tool_use (rare), dump it raw.
|
||||
if (!block.tool_use_id || !toolById.has(block.tool_use_id as string)) {
|
||||
out.push('**tool_result** (no matching tool_use in this transcript)');
|
||||
out.push('```json', truncate(safeJson(block.content, 2), maxOutputBytes), '```');
|
||||
out.push('');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Unknown block type — dump as a fenced JSON block for diagnostics.
|
||||
out.push(`**${block.type}**`);
|
||||
out.push('```json', truncate(safeJson(block, 2), maxOutputBytes), '```');
|
||||
out.push('');
|
||||
}
|
||||
|
||||
function safeJson(value: unknown, indent = 0): string {
|
||||
try {
|
||||
return JSON.stringify(value, null, indent);
|
||||
} catch {
|
||||
return String(value);
|
||||
}
|
||||
}
|
||||
|
||||
function truncate(s: string, maxBytes: number): string {
|
||||
if (Buffer.byteLength(s, 'utf8') <= maxBytes) return s;
|
||||
// Slice bytewise via Buffer so we don't split a multibyte char awkwardly.
|
||||
const buf = Buffer.from(s, 'utf8').slice(0, maxBytes);
|
||||
return buf.toString('utf8') + `\n... [truncated at ${maxBytes} bytes]`;
|
||||
}
|
||||
+148
-2
@@ -103,6 +103,14 @@ export interface MinionJobInput {
|
||||
backoff_type?: BackoffType;
|
||||
backoff_delay?: number;
|
||||
backoff_jitter?: number;
|
||||
/**
|
||||
* Per-job override for how many stall windows are tolerated before the
|
||||
* queue dead-letters the job. When omitted, the schema column DEFAULT
|
||||
* applies (bumped 1 → 3 in v0.14, now 5 as of v0.13.1's audit). Clamped
|
||||
* to [1, 100] on insert. For long-running handlers (LLM loops etc.) that
|
||||
* should survive a worker kill mid-run, set max_stalled: 3+.
|
||||
*/
|
||||
max_stalled?: number;
|
||||
delay?: number; // ms delay before eligible
|
||||
parent_job_id?: number;
|
||||
on_child_fail?: ChildFailPolicy;
|
||||
@@ -202,14 +210,37 @@ export function rowToInboxMessage(row: Record<string, unknown>): InboxMessage {
|
||||
};
|
||||
}
|
||||
|
||||
// --- Child-done inbox message (auto-posted on completeJob) ---
|
||||
// --- Child-done inbox message (auto-posted on every terminal transition) ---
|
||||
|
||||
/**
|
||||
* Posted into the parent's inbox when a child reaches a terminal state.
|
||||
*
|
||||
* Pre-v0.15: only success paths (completeJob) emitted this. Failed/dead/
|
||||
* cancelled children produced no payload, which stranded aggregator-style
|
||||
* parents that needed to wait for N children regardless of outcome.
|
||||
*
|
||||
* v0.15: failJob, cancelJob, and handleTimeouts also emit child_done with
|
||||
* the appropriate `outcome`, so the aggregator handler can count "N children
|
||||
* resolved" without worrying about which rail each one took.
|
||||
*
|
||||
* Backwards compatible: old ChildDoneMessage consumers only read child_id,
|
||||
* job_name, and result (non-null on success). Outcome and error are additive.
|
||||
*/
|
||||
export type ChildOutcome = 'complete' | 'failed' | 'dead' | 'cancelled' | 'timeout';
|
||||
|
||||
/** Posted into the parent's inbox when a child completes successfully. */
|
||||
export interface ChildDoneMessage {
|
||||
type: 'child_done';
|
||||
child_id: number;
|
||||
job_name: string;
|
||||
result: unknown;
|
||||
/**
|
||||
* Terminal outcome. When absent (from a pre-v0.15 writer that didn't set
|
||||
* it), consumers should treat the message as 'complete' — the legacy writer
|
||||
* only emitted on success paths.
|
||||
*/
|
||||
outcome?: ChildOutcome;
|
||||
/** Set when outcome !== 'complete'. Mirrors minion_jobs.error_text. */
|
||||
error?: string | null;
|
||||
}
|
||||
|
||||
// --- Attachments (v7) ---
|
||||
@@ -330,3 +361,118 @@ export function rowToMinionJob(row: Record<string, unknown>): MinionJob {
|
||||
updated_at: new Date(row.updated_at as string),
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Subagent runtime (v0.15+)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Input payload for the 'subagent' handler. Shape is intentionally narrow —
|
||||
* tool registry and provider config resolve via handler-side defaults + env,
|
||||
* not per-job data, so restart/replay uses the same behavior.
|
||||
*/
|
||||
export interface SubagentHandlerData {
|
||||
/** Top-level user turn kicking off the loop. */
|
||||
prompt: string;
|
||||
/** Optional subagent definition path (skills/subagents/*.md or plugin). */
|
||||
subagent_def?: string;
|
||||
/** Anthropic model id. Defaults to sonnet at handler resolution time. */
|
||||
model?: string;
|
||||
/** Max assistant turns before the loop fails with stop_reason='max_turns'. */
|
||||
max_turns?: number;
|
||||
/**
|
||||
* Whitelist of tool names the agent may call. MUST be a subset of the
|
||||
* derived registry names — invalid entries are rejected at tool-dispatch
|
||||
* time, not silently ignored. Empty array = no tools.
|
||||
*/
|
||||
allowed_tools?: string[];
|
||||
/** System prompt override. When omitted, the handler builds one. */
|
||||
system?: string;
|
||||
/** Template variables for subagent_def. Arbitrary JSON-serializable. */
|
||||
input_vars?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Input for the 'subagent_aggregator' handler. Claims AFTER all children
|
||||
* resolve and aggregates their results into a brain page.
|
||||
*/
|
||||
export interface AggregatorHandlerData {
|
||||
/** The subagent child job ids this aggregator is waiting on. */
|
||||
children_ids: number[];
|
||||
/**
|
||||
* Optional template for the synthesis prompt. When omitted, the handler
|
||||
* uses a generic "summarize these N results" prompt.
|
||||
*/
|
||||
aggregate_prompt_template?: string;
|
||||
/**
|
||||
* Target slug for the aggregated brain page. When present, a trusted-CLI
|
||||
* put_page (viaSubagent=false) writes the final aggregation there.
|
||||
*/
|
||||
output_slug?: string;
|
||||
}
|
||||
|
||||
/** Tool execution context passed to every ToolDef.execute. */
|
||||
export interface ToolCtx {
|
||||
/** Engine for DB-backed tools (brain_query, put_page, etc.). */
|
||||
engine: import('../engine.ts').BrainEngine;
|
||||
/** The subagent job id (used for audit + put_page namespace enforcement). */
|
||||
jobId: number;
|
||||
/** Always true for LLM-invoked tools — matches MCP trust boundary. */
|
||||
remote: true;
|
||||
/** Fired on cooperative abort (timeout, lock loss, cancel, SIGTERM). */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/**
|
||||
* A tool the subagent can call. Names match Anthropic's constraint
|
||||
* `^[a-zA-Z0-9_-]{1,64}$` — no dots. The input_schema is the JSONSchema
|
||||
* shipped to the Anthropic Messages API verbatim; ToolDef is the single
|
||||
* Anthropic-compatible envelope, not an MCP McpToolDef (those have a
|
||||
* different shape — ".inputSchema" vs ".input_schema").
|
||||
*
|
||||
* `idempotent: true` is required for the two-phase replay path: on resume,
|
||||
* a 'pending' row can be re-executed. Non-idempotent tools need a separate
|
||||
* resume policy and are not supported in v0.15.
|
||||
*/
|
||||
export interface ToolDef {
|
||||
name: string;
|
||||
description: string;
|
||||
input_schema: Record<string, unknown>;
|
||||
idempotent: boolean;
|
||||
execute(input: unknown, ctx: ToolCtx): Promise<unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Anthropic content-block subset we persist in subagent_messages.content_blocks.
|
||||
* This is structural — we don't gatekeep on unknown block types (future SDK
|
||||
* additions pass through). Use the string-literal discriminant on 'type'.
|
||||
*/
|
||||
export type ContentBlock =
|
||||
| { type: 'text'; text: string; [k: string]: unknown }
|
||||
| { type: 'tool_use'; id: string; name: string; input: unknown; [k: string]: unknown }
|
||||
| { type: 'tool_result'; tool_use_id: string; content: unknown; is_error?: boolean; [k: string]: unknown }
|
||||
| { type: string; [k: string]: unknown };
|
||||
|
||||
/** Stop reason reported to the caller when the subagent loop terminates. */
|
||||
export type SubagentStopReason =
|
||||
| 'end_turn' // Anthropic says end_turn and last message has no tool_use
|
||||
| 'max_turns' // hit max_turns budget before end_turn
|
||||
| 'refusal' // detected via stop_reason + content shape
|
||||
| 'error'; // unrecoverable (empty response retry exhausted, etc.)
|
||||
|
||||
/** Terminal result payload emitted by the subagent handler. */
|
||||
export interface SubagentResult {
|
||||
/** Concatenated text from the final assistant message. */
|
||||
result: string;
|
||||
/** Number of assistant turns consumed. */
|
||||
turns_count: number;
|
||||
/** Why the loop stopped. */
|
||||
stop_reason: SubagentStopReason;
|
||||
/** Rollup of tokens across all turns. */
|
||||
tokens: {
|
||||
in: number;
|
||||
out: number;
|
||||
cache_read: number;
|
||||
cache_create: number;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* Poll-until-terminal helper for CLI callers. Minions doesn't ship a
|
||||
* notification stream for arbitrary callers (the NOTIFY trigger is worker-
|
||||
* side), so `gbrain agent run --follow` on the CLI side polls getJob() until
|
||||
* the job reaches a terminal state.
|
||||
*
|
||||
* On timeout, the job is NOT cancelled — the user can `gbrain jobs get <id>`
|
||||
* later to check. Explicit cancellation is the user's call via `gbrain jobs
|
||||
* cancel <id>`.
|
||||
*/
|
||||
|
||||
import type { MinionQueue } from './queue.ts';
|
||||
import type { MinionJob, MinionJobStatus } from './types.ts';
|
||||
|
||||
export class TimeoutError extends Error {
|
||||
constructor(public readonly jobId: number, public readonly elapsedMs: number) {
|
||||
super(`timeout after ${elapsedMs}ms waiting for job ${jobId}`);
|
||||
this.name = 'TimeoutError';
|
||||
}
|
||||
}
|
||||
|
||||
const TERMINAL_STATES: readonly MinionJobStatus[] = ['completed', 'failed', 'dead', 'cancelled'] as const;
|
||||
const TERMINAL_SET = new Set<MinionJobStatus>(TERMINAL_STATES);
|
||||
|
||||
export interface WaitOpts {
|
||||
/** Abort after this many ms. Default: 24h (long enough for most durable runs). */
|
||||
timeoutMs?: number;
|
||||
/**
|
||||
* Poll interval. Defaults:
|
||||
* - 1000ms on Postgres (lighter load, concurrent followers scale)
|
||||
* - 250ms when the caller knows it's on PGLite inline (single process,
|
||||
* no network RTT)
|
||||
* Callers pass the appropriate value explicitly — this module doesn't
|
||||
* introspect the engine.
|
||||
*/
|
||||
pollMs?: number;
|
||||
/** Optional AbortSignal — on abort, the poll loop exits early (no TimeoutError). */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
export async function waitForCompletion(
|
||||
queue: MinionQueue,
|
||||
jobId: number,
|
||||
opts: WaitOpts = {},
|
||||
): Promise<MinionJob> {
|
||||
const timeoutMs = opts.timeoutMs ?? 24 * 60 * 60 * 1000;
|
||||
const pollMs = opts.pollMs ?? 1000;
|
||||
const started = Date.now();
|
||||
|
||||
// Fast-path first read (don't wait pollMs just to learn it's already done).
|
||||
let job = await queue.getJob(jobId);
|
||||
if (!job) throw new Error(`job ${jobId} not found`);
|
||||
if (TERMINAL_SET.has(job.status)) return job;
|
||||
|
||||
while (true) {
|
||||
if (opts.signal?.aborted) {
|
||||
// Caller aborted. Return the last-seen snapshot rather than throwing —
|
||||
// the job itself is still alive queue-side, and the caller knows they
|
||||
// aborted.
|
||||
return job;
|
||||
}
|
||||
const elapsed = Date.now() - started;
|
||||
if (elapsed >= timeoutMs) {
|
||||
throw new TimeoutError(jobId, elapsed);
|
||||
}
|
||||
const remaining = timeoutMs - elapsed;
|
||||
const sleep = Math.min(pollMs, remaining);
|
||||
await delay(sleep, opts.signal);
|
||||
|
||||
job = await queue.getJob(jobId);
|
||||
if (!job) throw new Error(`job ${jobId} disappeared mid-wait`);
|
||||
if (TERMINAL_SET.has(job.status)) return job;
|
||||
}
|
||||
}
|
||||
|
||||
function delay(ms: number, signal?: AbortSignal): Promise<void> {
|
||||
if (ms <= 0) return Promise.resolve();
|
||||
return new Promise((resolve) => {
|
||||
const t = setTimeout(() => {
|
||||
signal?.removeEventListener('abort', onAbort);
|
||||
resolve();
|
||||
}, ms);
|
||||
const onAbort = () => {
|
||||
clearTimeout(t);
|
||||
resolve();
|
||||
};
|
||||
signal?.addEventListener('abort', onAbort, { once: true });
|
||||
});
|
||||
}
|
||||
|
||||
// Exported for unit tests.
|
||||
export const __testing = {
|
||||
TERMINAL_STATES,
|
||||
};
|
||||
@@ -30,7 +30,7 @@ import { evaluateQuietHours, type QuietHoursConfig } from './quiet-hours.ts';
|
||||
function readQuietHoursConfig(job: MinionJob): QuietHoursConfig | null {
|
||||
const cfg = (job as MinionJob & { quiet_hours?: unknown }).quiet_hours;
|
||||
if (!cfg || typeof cfg !== 'object') return null;
|
||||
return cfg as QuietHoursConfig;
|
||||
return cfg as unknown as QuietHoursConfig;
|
||||
}
|
||||
|
||||
/** Per-job in-flight state (isolated per job, not shared on the worker). */
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user