Compare commits

...
Author SHA1 Message Date
root c21180c972 perf: incremental extract — only process slugs that sync touched
The autopilot-cycle runs every 5 min. Its extract phase was doing a full
filesystem walk of ALL markdown files (54K+) — twice (links + timeline).
On a brain this size, extract alone exceeded the 600s job timeout,
producing zero useful writes.

Fix: sync already returns pagesAffected (the slugs it added/modified).
Pipe that list through to extract. When provided, extract reads ONLY
those files instead of walking the entire brain directory.

- Add ExtractOpts.slugs for targeted extraction
- Add extractForSlugs() — single-pass links + timeline for specific slugs
- cycle.ts: capture sync's pagesAffected, pass to runPhaseExtract
- If sync didn't run or failed, extract falls back to full walk (safe)
- If pagesAffected is empty (nothing changed), extract returns instantly

Expected improvement: 54K file reads → ~10-50 per cycle. The full walk
is still available via CLI `gbrain extract` and on first-run.
2026-04-25 00:46:35 +00:00
11abb24ddd v0.20.4 feat: merge gbrain-jobs into minion-orchestrator — single unified minions skill (#381)
* feat: merge gbrain-jobs into minion-orchestrator — single unified minions skill

* fix(skill/minion-orchestrator): correct MCP boundary, real handler names, PGLite path

The initial merge commit a51c737 documented `submit_job name="shell"` as
agent-callable, but src/core/operations.ts:1106 rejects protected names
from MCP callers (shell is in src/core/minions/protected-names.ts:16) —
shell-job submission is CLI-only. Subagent examples referenced non-existent
handler names (`research`, `orchestrate`) instead of the real `subagent` /
`subagent_aggregator` handlers. PGLite section wrongly told users to
migrate to Supabase when `gbrain jobs submit ... --follow` inline mode
works per docs/guides/minions-shell-jobs.md:15. Contract section canonized
"every task through Minions" against the `pain_triggered` default in
skills/conventions/subagent-routing.md:16,27.

Rewrite addresses all four:
- Shell Jobs section is explicit about CLI-only submission; agents observe
  via get_job / list_jobs / get_job_progress (non-protected).
- Subagent examples route through `gbrain agent run` (user-facing CLI)
  with raw handler names documented as the power-user path.
- PGLite gets --follow inline execution, not migration friction.
- Contract softened to point at subagent-routing.md convention.

Also adds a Preconditions block for Shell Jobs (env gate, RCE warning,
execution-mode choice, verification command), narrows the frontmatter
"gbrain jobs" trigger to "gbrain jobs submit" + "submit a gbrain job"
(bare was too broad — CLI namespace covers 9 subcommands), inlines a
"replaces older gbrain-jobs routing intent" note in the description, and
removes non-existent `get_job_stats` from the tools list (CLI is
`gbrain jobs stats`; no MCP equivalent).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(resolver): narrow "gbrain jobs" trigger to specific intents

Replace bare "gbrain jobs" in the routing table with "gbrain jobs submit"
+ "submit a gbrain job". The bare phrase was too broad — the CLI namespace
covers 9 subcommands (submit, list, get, retry, delete, prune, stats,
smoke, work). Users asking about stats/prune/retry now fall through to
`gbrain --help` instead of getting misrouted to minion-orchestrator, which
only documents shell execution and subagent orchestration.

Matches the frontmatter trigger narrow in minion-orchestrator/SKILL.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(resolver): add round-trip + skill-example-name validator

Two new assertion blocks in test/resolver.test.ts:

1. RESOLVER.md trigger round-trip: every quoted phrase in a routing-table
   row has a fuzzy match in the target skill's frontmatter `triggers:` list.
   Catches RESOLVER ↔ frontmatter drift that checkResolvable's reachability
   check doesn't. Fuzzy match is case-insensitive, trailing-punctuation-
   insensitive, and splits on "/" for compound phrases like
   "pause/resume agent" — accommodates RESOLVER.md's natural-language
   summary style without allowing real drift through.

2. Skill example-name validator: every `name="<word>"` reference in any
   SKILL.md body must resolve to either a declared operation in
   src/core/operations.ts or a known Minions handler in
   PROTECTED_JOB_NAMES. Would have caught the `name="research"` /
   `name="orchestrate"` drift that slipped through the first review
   — nothing in CI caught those handler names referencing non-existent
   handlers until a Codex cold-read found them. This test closes that
   class of regression gap.

51 / 51 tests pass locally. Full E2E suite (bun run test:e2e) still
passes 197 / 197 across 19 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): PGLite shell-job --follow inline path

Closes the T4 coverage gap surfaced during PR #381 eng review. The sibling
test/e2e/minions-shell.test.ts covers Postgres + persistent-daemon; this
file covers the PGLite + --follow path the minion-orchestrator skill now
documents.

Two assertions:

1. submit → registerBuiltinHandlers → worker.start → shell runs → completes
   with exit_code 0 and stdout_tail "hello\n". Exercises the exact dispatch
   path src/commands/jobs.ts:207 takes when --follow is set, including the
   GBRAIN_ALLOW_SHELL_JOBS=1 gate.

2. With GBRAIN_ALLOW_SHELL_JOBS unset, registerBuiltinHandlers leaves the
   shell handler unregistered. Confirms the env gate from
   src/commands/jobs.ts:611 works.

Runs in-memory against PGLiteEngine — no DATABASE_URL, no Docker, runs in
CI unconditionally. Completes in ~1.2s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: pre-landing review fixes

Pre-landing review caught 4 doc bugs + 2 test fragilities + 2 pre-existing
drift cases. All auto-fix category (clear correct answer, single obvious fix).

minion-orchestrator/SKILL.md:
- Shell submit examples used nonexistent `--cmd`/`--argv`/`--cwd` flags. Real
  CLI takes `--params '{"cmd":"...","cwd":"..."}'` (src/commands/jobs.ts:55-85).
  Examples now match `gbrain jobs submit --help` output.
- `--tools "search,web_search"` referenced `web_search` which isn't in
  BRAIN_TOOL_ALLOWLIST (src/core/minions/tools/brain-allowlist.ts:47-59).
  Swapped to `search,query`. Added a full allowlist enumeration so
  readers don't have to grep.
- `gbrain agent run` flags section listed `--queue`, `--priority`,
  `--max-attempts`, `--delay` — none of these exist on that command
  (src/commands/agent.ts:105-129). Replaced with the real flag set
  (`--subagent-def`, `--model`, `--max-turns`, `--tools`, `--timeout-ms`,
  `--fanout-manifest`, `--follow`, `--no-follow`, `--detach`) and a note
  about using `gbrain jobs submit` for queue tuning.
- MCP boundary claim "returns permission_denied" was imprecise. Reworded:
  throws an OperationError with code permission_denied.

test/resolver.test.ts:
- D5/C row regex required the backtick-quoted skill path to be followed
  immediately by `|`, silently skipping rows with trailing parentheticals
  (e.g., `` `skills/maintain/SKILL.md` (extraction sections) |``). Broadened
  to `[^|]*\|` so every row gets audited.

test/e2e/minions-shell-pglite.test.ts:
- Shared engine across both tests with no per-test reset. Future test
  additions would hit order-dependency. Added beforeEach TRUNCATE on
  minion_jobs / minion_inbox / minion_attachments, matching the Postgres
  sibling at test/e2e/minions-shell.test.ts:55-58.

skills/query/SKILL.md:
- Added 4 triggers RESOLVER.md routes to this skill but the frontmatter
  never declared: "who knows who", "relationship between", "connections",
  "graph query". Pre-existing drift — the broadened D5/C regex surfaced it.

skills/maintain/SKILL.md:
- Added 6 triggers with the same pre-existing drift: "extract links",
  "build link graph", "populate timeline", "populate links", "backfill graph",
  "extract timeline entries".

57/57 tests pass on the fixed tree.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: second-pass review fixes — stale CLI flag + handler name

Two more stale references caught by specialist re-dispatch on the fixed tree:

skills/minion-orchestrator/SKILL.md:72 — Routing table row described shell
  jobs as taking `--cmd` or `--argv` as CLI flags. Same class of bug as M1
  from the prior fix commit but in a different location. Now says `--params`
  with `cmd` or `argv`, matching the corrected submit examples (lines 112-120).

skills/conventions/subagent-routing.md:82 — "Check `get_job_stats`
  queue_health.active" referenced an MCP operation that doesn't exist in
  src/core/operations.ts. The new minion-orchestrator skill cross-references
  this convention file, so agents following the routing pointer would hit a
  non-existent op. Replaced with the real ops: `list_jobs --status active`
  (MCP) or `gbrain jobs stats` (CLI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: adversarial pass cleanups — manifest.json + anti-pattern scope

Claude adversarial subagent caught two last consistency gaps:

skills/manifest.json:135 — Skill description still read "Manage background
  agents via Minions job queue" (subagent-only framing), out of sync with
  the reframed SKILL.md frontmatter. Manifest is what the skill registry
  indexes; leaving this stale meant shell-job-intent routers would miss it.
  Updated to match the unified wording.

skills/minion-orchestrator/SKILL.md:288 — Anti-pattern line "Don't use
  sessions_spawn with runtime: subagent when Minions is available" was
  subagent-lane-specific inside the now-consolidated skill, reading like
  the one rule in the skill but only addressing one lane. Scoped to
  "For subagent work" and pointed at `gbrain agent run` so the rule
  doesn't confuse shell-job readers.

Two investigate-class items deferred to follow-up:
- D13 regex could false-positive on future skills with unrelated `name="..."`
  usage. Today clean; scope to backtick-fenced snippets if it bites.
- PGLite E2E env-var race if bun:test ever goes file-parallel. Today isolated
  per file; add helper + comment when needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.19.2)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README + CLAUDE.md for v0.19.2 Minions consolidation

- Skill count 28 -> 29 across README and CLAUDE.md (adds smoke-test from
  v0.19.1 to the Skills section, closes a prior drift).
- README minion-orchestrator row rewritten to name both lanes (shell jobs
  via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`)
  so the surface matches the consolidated skill file.
- README Operational table gains a smoke-test row.
- CLAUDE.md key-files entry for minion-orchestrator now describes the
  v0.19.2 consolidation, trust boundary (MCP permission_denied on
  protected names), and the narrowed trigger set.
- CLAUDE.md Skills section notes the consolidation and the new v0.19.1
  smoke-test skill.
- CLAUDE.md test inventory picks up `test/e2e/minions-shell-pglite.test.ts`
  and the v0.19.2 round-trip + name-validator additions in
  `test/resolver.test.ts`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): update PGLite test for new env-gate behavior + regenerate llms-full.txt

CI caught two issues:

1. `test/e2e/minions-shell-pglite.test.ts` — the "GBRAIN_ALLOW_SHELL_JOBS
   unset → shell handler not registered" test was written against pre-v0.20.3
   `registerBuiltinHandlers` behavior (env gate at registration time). Master's
   queue-resilience merge moved the gate from registration to execution:
   shell handler is now always registered so claimed jobs emit a clear rejection
   log, and `shellHandler` itself throws UnrecoverableError when
   GBRAIN_ALLOW_SHELL_JOBS != '1' (see src/core/minions/handlers/shell.ts:210).
   Updated the test to invoke shellHandler directly with a minimal ctx and
   assert the throw. Preserves the test's intent (prove the guard works) under
   the new control flow.

2. `llms-full.txt` drift — README.md + CLAUDE.md updates in v0.19.2 and v0.20.4
   updated the skill count to 29 and rewrote the minion-orchestrator
   description, but the committed `llms-full.txt` bundle still reflected the
   pre-consolidation content. Regenerated via `bun run build:llms`.

The third CI failure (`planInstall + applyInstall D-CX-11`) passes cleanly
locally (26/26 in test/skillpack-install.test.ts). The 1ms runtime in CI
suggests a filesystem-mtime flake, not a real regression from this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(skillpack): treat future-mtime lock as stale (CI race fix)

D-CX-11 ("--force-unlock overrides a stale lock") flaked in CI with a 1ms
runtime. Root cause: on fast CI filesystems (ext4 with high-resolution
mtimes on GitHub runners), `writeFileSync` can set a lock's mtime a few
microseconds ahead of the subsequent `Date.now()`, making `age` negative.

Old logic:
  const stale = age >= staleMs;

With `staleMs: 0` and `age = -0.3ms`: `-0.3 >= 0` is false → NOT stale →
the `!stale` branch throws `lock_held` before reaching the force-unlock
path. Test failed at the first ms, never exercised the actual unlock logic.

Fix (src/core/skillpack/installer.ts:189):
  const stale = age < 0 || age >= staleMs;

Treats negative age (future mtime) as stale. Safe: if the lock's mtime is
in the future, either the filesystem clock just jumped forward or the
lock was written by a racing process; either way it's not a live,
healthy lock and the stale path is the correct branch.

Passes locally (26/26 in test/skillpack-install.test.ts).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:39:58 -07:00
d838d4792b feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency (#379)
* feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard

Prevents stall-induced queue blockage discovered in production (OpenClaw):

1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms
   (or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB
   connections where FOR UPDATE SKIP LOCKED stall detection skips them.

2. Submission backpressure (maxWaiting): caps waiting jobs per name at
   submission time. Prevents autopilot-cycle flood when the queue is blocked.

3. --no-worker flag for autopilot: skips spawning the built-in worker child.
   For environments where the worker lifecycle is managed externally (systemd,
   Docker, OpenClaw service-manager).

4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the
   worker is spawned by autopilot (which can't pass CLI flags to the child).

5. Shell job env guard with clear logging: shell handler is always registered
   but throws UnrecoverableError with a clear message when
   GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering.

* feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI

Addresses three production-hardening findings from the CEO + Eng + Codex
adversarial review of PR #379:

D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both
see waitingCount < max and both insert. Wrap the count+select+insert in
pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent
decisions for the SAME key while leaving different keys fully parallel.
Lock auto-releases on txn commit/rollback — no cleanup path to leak.
Also fix the missing queue-scope bug: count and select now filter on
(name, queue) not name alone, so cross-queue same-name jobs don't
suppress each other.

D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from
parseInt. `inFlight.size < NaN` is always false → worker claims nothing →
silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr
warning naming the bad value.

D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the
MinionJobInput field. Wire the flag with clamp [1, 100], mirror
`--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing.

Q1: Silent coalesce was invisible by design. New
src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week
JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce
events write one JSONL line with (queue, name, waiting_count, max_waiting,
returned_job_id, ts). Best-effort — disk-full never blocks submission.

A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case.
Forges a wedged-worker row state, invokes handleStalled + handleTimeouts
+ handleWallClockTimeouts in order, asserts only wall-clock evicts.
Mirrors the v0.14.3 `--sigkill-rescue` shape.

Tests: 23 new unit cases in test/minions.test.ts covering wall-clock
timeout (3 cases + non-interference with handleTimeouts), maxWaiting
(coalesce, clamp 0, floor, concurrent-submitter race via Promise.all,
cross-queue isolation, unset fallthrough), concurrency clamp (7 cases
incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure
audit file write.

Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook

A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite
has no multi-process worker surface). Two subchecks, both cheap (single
SELECT each, status-index-covered):

- stalled-forever: any active job with started_at > 1h. Surfaces the
  worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel`
  fix hints. The incident that motivated v0.19.1 ran 90+ min before the
  operator noticed.
- waiting-depth: per-name waiting count exceeds threshold. Default 10,
  overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a
  submitter probably needs maxWaiting set.

Worker-heartbeat subcheck from the original plan dropped (D4/H4): no
minion_workers table exists, and lock_until-on-active-jobs is a lossy
proxy that can't distinguish idle-worker from dead-worker. Tracked as
follow-up B7.

A4: --no-worker peer-liveness probe in autopilot. When --no-worker is
set, every cycle runs a cheap SELECT checking for any active job whose
lock_until was refreshed in the last 2 minutes. After 3 consecutive
idle ticks, logs a loud WARNING naming the silent-wedge vector and
referencing B7 as the ground-truth follow-up. Re-arms on next live
signal so the warning doesn't spam every cycle.

A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60
lines). "My queue looks wedged — what do I run?" in order of
escalation. What each doctor subcheck means. Self-check for the
--no-worker / no-worker-running footgun.

CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer
3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health
doctor check (v0.19.1 D4), and backpressure-audit.ts.

Tests: all 143 minions + 13 doctor unit tests pass. No new test cases
required in Lane B; the doctor queue_health exercise is in the E2E
verification step (needs real PG to produce meaningful stalled-forever
rows). The --no-worker probe is exercised by the smoke case's wedge
setup in Lane A.

README: unchanged. Existing `gbrain jobs submit` examples don't show
--max-stalled, so no --max-waiting precedent to extend per A6 conditional.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md

VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change,
no new user-facing vocabulary).

CHANGELOG: new v0.19.1 entry at the top with the full release-summary
template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers
that matter" before/after table measured against the real incident,
"what this means for OpenClaw users" closer, required "To take
advantage of v0.19.1" block naming the worker-restart requirement,
itemized changes by area, and "For contributors" section closing the
loop on the stale autopilot-idempotency narrative the CEO review was
based on.

Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by
missing idempotency (autopilot already passes
`idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The
18 jobs were 18 DIFFERENT slots stacking up behind the wedged one.
`maxWaiting` still caps the pile; the incident just wasn't about
idempotency. Adversarial review caught this before ship.

SPEC.md: deleted from repo root. It was Wintermute's planning artifact
for the original PR, not a shipped spec. Design docs belong under
docs/designs/ per repo convention; leaving one at repo root set a
precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file
at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are
the durable artifacts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip

Smoke case was setting lock_until in the past, so handleStalled's
requeue path fired before handleWallClockTimeouts had a chance to
evict. Production scenario is "lock_until still live (worker
renewing) + timeout_at disqualified" — only wall-clock matches.

Single-connection smoke can't simulate a row lock held by another
txn, so we force the equivalent outcome:
- lock_until = now() + 30s → handleStalled skips (not a stall)
- timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
- started_at = now() - 10s, timeout_ms=1000 → wall-clock matches
  (2 × timeout_ms = 2000ms threshold exceeded)

Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: CI failures — shell-handler tests + llms-full.txt drift

Two CI failure clusters, both pre-existing but surfaced by the v0.20.3
merge:

1) test/minions-shell.test.ts — 12 failing cases. The shell handler
   throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the
   production RCE guard at shell.ts:210). The unit tests exercise
   handler mechanics, not the guard, but never set the env var — so
   every invocation exits through the guard path instead of the code
   being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll,
   restore in afterAll. The env-guard IS still tested separately via
   the test/minions.test.ts case added in v0.20.3 Lane A which toggles
   the var itself.

2) llms-full.txt — stale against CLAUDE.md. Key-files entries for
   queue.ts, doctor.ts, and the new backpressure-audit.ts updated in
   v0.20.3 Lane B triggered the build-llms drift guard. Regenerated
   via `bun run build:llms`; no behavior change, just the inlined-docs
   bundle catching up to source.

Full test run: 2367 pass, 0 fail across 137 files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:09:28 -07:00
e3f704229b v0.20.2 feat: gbrain jobs supervisor — self-healing worker process manager (#364)
* feat: add `gbrain jobs supervisor` — self-healing worker process manager

Adds a first-class supervisor command that:
- Spawns `gbrain jobs work` as a child process
- Restarts on crash with exponential backoff (1s→60s cap)
- Resets crash counter after 5min of stable operation
- PID file locking prevents duplicate supervisors
- Periodic health checks (stalled jobs, completion gaps)
- Graceful shutdown (SIGTERM→35s→SIGKILL)

Usage:
  gbrain jobs supervisor --concurrency 4

Replaces ad-hoc nohup patterns in bootstrap scripts.
The autopilot command's internal supervisor can be migrated
to use this in a follow-up.

Tests: 7 pass (backoff calc, PID management, crash tracking)

* supervisor: atomic PID lock, queue-scoped health, env safety, unified exit

Lane A of PR #364 review fixes (20-item multi-lane plan). Addresses the
codex-tier + CEO + Eng findings on src/core/minions/supervisor.ts:

Safety + correctness:
- Atomic O_CREAT|O_EXCL PID lock via openSync('wx') with stale-file
  liveness check. Prevents two supervisors racing on the same PID file.
  (codex #1)
- Health check now queries status='active' AND lock_until < now()
  matching queue.ts:848's authoritative stalled definition. The prior
  `status = 'stalled'` predicate returned zero rows forever because
  'stalled' is not a persisted value in the schema. (codex #2)
- All health queries scoped to WHERE queue = $1 via opts.queue binding.
  Multi-queue installs no longer see cross-queue false positives.
  (codex #3)
- Class default allowShellJobs flipped true→false AND explicit
  `delete env.GBRAIN_ALLOW_SHELL_JOBS` when false, so child workers
  don't silently inherit the var from the parent shell. (eng #8, codex #9)
- Unified shutdown(reason, exitCode) — max-crashes now routes through
  the same drain path as SIGTERM. Single source of truth for lifecycle
  cleanup; prerequisite for trustworthy audit events (Lane C). (eng #1)
- Default PID path moves from /tmp to ~/.gbrain/supervisor.pid with
  mkdirSync recursive + GBRAIN_SUPERVISOR_PID_FILE env override.
  Matches the rest of the product's ~/.gbrain/ convention; fresh
  installs no longer hit ENOENT. (CEO #2 + codex #6)

Refinements:
- crashCount = 1 after 5-min stable-run reset (was 0, produced
  calculateBackoffMs(-1) = 500ms by accident). Now reads as 'first
  crash of a new cycle' with a clean 1s backoff. (Nit 1)
- Top-of-file POSTGRES-ONLY docstring documenting why the supervisor
  can't run against PGLite. (Nit 2)
- inBackoff flag suppresses 'worker not alive' warn during the
  expected null-child window (crash → sleep → next spawn). (eng #2)
- Tracked listener refs for SIGTERM/SIGINT removed in shutdown() so
  integration tests spinning up/tearing down multiple supervisors on
  one process don't leak handlers. (eng #3)
- Single FILTER query replaces two SELECT counts — one round-trip
  instead of two, three metrics in one pass. (eng #10)
- child.on('error') listener emits worker_spawn_failed event for
  ENOENT/EACCES; exit handler still increments crashCount as usual
  so max-crashes bounds permanent misconfigurations. (codex #7)
- healthInFlight boolean guard with try/finally prevents overlapping
  health checks from stacking on a hung DB. (codex #8)

Documented exit codes (ExitCodes const):
  0 CLEAN, 1 MAX_CRASHES, 2 LOCK_HELD, 3 PID_UNWRITABLE
  Agent can branch on exit=2 ('another supervisor, I'm fine') vs
  exit=1 ('escalate to human').

Event emitter surface:
  - started / worker_spawned / worker_exited / worker_spawn_failed
  - backoff / health_warn / health_error / max_crashes_exceeded
  - shutting_down / stopped
  Plumbed through emit() with an onEvent callback hook for Lane C's
  audit writer. json:false is the default; Lane C's --json mode
  flips it and writes JSONL to stderr.

CLI changes (src/commands/jobs.ts):
- `gbrain jobs supervisor` gains --allow-shell-jobs (explicit opt-in
  mirroring the env-var gate), --cli-path (override auto-resolution
  for exotic setups), and --json (JSONL lifecycle events on stderr).
- Expanded --help body with description, 3 examples, and exit-code
  table. (DX Fix A per review)
- Three-tier PID path resolution: --pid-file > GBRAIN_SUPERVISOR_PID_FILE
  > ~/.gbrain/supervisor.pid (via exported DEFAULT_PID_FILE).
- Removed the catch-fallback to process.argv[1] — resolveGbrainCliPath()
  throws its own actionable install-hint error, which is what dev users
  need instead of a cryptic spawn failure on a .ts path. (codex #5)

Tests: existing 7 supervisor.test.ts cases continue to pass.
Integration tests (crash-restart, max-crashes, SIGTERM-during-backoff,
env-inheritance regression) land in Lane E.

Out of scope for this lane (tracked in follow-up lanes):
- Audit file writer at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl (Lane C)
- Documentation pass (Lane B)
- supervisor start/status/stop subcommands (Lane C)
- gbrain doctor supervisor check (Lane D)
- /ship release hygiene (Lane F)
- autopilot.ts migration to MinionSupervisor (deferred to follow-up PR
  per codex — requires non-blocking start() API redesign, not ~30 lines)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: supervisor as canonical worker deployment pattern

Lane B of PR #364 review fixes. Reframes docs/guides/minions-deployment.md
around `gbrain jobs supervisor` as the default answer (blocker 7), deletes
the 68-line legacy bash watchdog (F10), and updates README + deployment
snippets to match.

docs/guides/minions-deployment.md:
- New 'Worker supervision' section at the top with the canonical 3-command
  agent pattern (start --detach / status --json / stop) and a documented
  exit-code table (0 clean, 1 max-crashes, 2 lock-held, 3 PID-unwritable).
- 'Which supervisor when?' decision table: container = supervisor as
  PID 1, Linux VM = systemd-over-supervisor, dev laptop = bare terminal.
- New 'Agent usage' section for OpenClaw / Hermes / Cursor / Codex — the
  3-turn discover-start-maintain workflow that replaces shell archaeology
  with machine-parseable JSON events + an audit file at
  ~/.gbrain/audit/supervisor-YYYY-Www.jsonl.
- Demoted the 'Option 1: watchdog cron' path entirely; replaced with a
  straightforward upgrade migration block (stop script, remove cron line,
  start supervisor, verify via doctor).
- Preconditions now check Postgres connectivity directly (supervisor is
  Postgres-only; the CLI rejects PGLite with a clear error).

Snippets:
- systemd.service: ExecStart now invokes `gbrain jobs supervisor` instead
  of raw `gbrain jobs work`. Two-layer supervision (systemd → supervisor
  → worker) buys automatic restart on reboot plus fast crash recovery.
  ReadWritePaths expanded to cover $HOME/.gbrain (supervisor PID + audit).
- Procfile + fly.toml.partial: same change — platform restarts the
  container on host events, supervisor restarts the worker on crashes.
- minion-watchdog.sh: deleted (git history retains it for anyone in an
  exotic deployment). Supervisor subsumes every capability it had plus
  atomic PID locking, structured audit events, queue-scoped health
  checks, and graceful drain on SIGTERM.

README.md:
- Added a paragraph under the Minions section pointing `gbrain jobs
  supervisor` as canonical, noting the --detach / status / stop surface
  and the audit file path, with a link to the full deployment guide.
  Kept `gbrain jobs work` documented for direct raw invocation but
  flagged 'prefer supervisor' for any long-running use.

The supervisor `--help` body itself (3 examples + exit-code table in
src/commands/jobs.ts) landed with Lane A — this lane finishes the
discoverability story by making the supervisor findable via doc grep,
README landing, and deployment-guide landing paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* supervisor: daemon-manager subcommands + JSONL audit writer

Lane C of PR #364 review fixes. Adds the daemon-manager CLI surface so
agents can drive `gbrain jobs supervisor` in 3 turns instead of 10, and
the audit writer that makes lifecycle events inspectable across process
restarts. (Blocker 8, closes DX Fix A/B/C.)

New: src/core/minions/handlers/supervisor-audit.ts
  - writeSupervisorEvent(emission, supervisorPid) appends JSONL to
    `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`.
    ISO-week rotation via a `computeSupervisorAuditFilename()` helper
    that mirrors `shell-audit.ts` exactly (year-boundary ISO week math,
    Thursday anchor, etc).
  - readSupervisorEvents({sinceMs}) returns parsed events from the
    current week's file, oldest-first, for Lane D's doctor check.
    Malformed lines are skipped silently (disk-full truncation is
    already best-effort at write time).
  - Reuses `resolveAuditDir()` from shell-audit.ts so the
    `GBRAIN_AUDIT_DIR` env var override works identically across all
    gbrain audit trails.

src/commands/jobs.ts: supervisor subcommand dispatcher
  - `gbrain jobs supervisor [start] [--detach] [--json] ...` — default
    subcommand. Without --detach, runs foreground as before. With
    --detach, forks a background child (inheriting stderr so the caller
    can still tail JSONL events), writes a stdout payload:
      {"event":"started","supervisor_pid":N,"pid_file":"...","detached":true}
    and exits 0. Stdin/stdout on the detached child are /dev/null so
    the parent shell isn't held open.
  - `gbrain jobs supervisor status [--json]` — reads the PID file,
    checks liveness via `kill -0`, then reads the last 24h from the
    supervisor audit file to compute crashes_24h / last_start /
    max_crashes_exceeded. Exits 0 if running, 1 if not. JSON output
    is machine-parseable; human output is a 5-line ASCII report.
  - `gbrain jobs supervisor stop [--json]` — reads PID, sends SIGTERM,
    polls `kill -0` every 250ms for up to 40s (supervisor's own 35s
    worker-drain + 5s slack). Reports outcome: drained / timeout_40s
    / pid_file_missing / pid_file_corrupt / process_gone. Exit 0 on
    clean stop.
  - `--json` flag is already plumbed through to the supervisor opts
    from Lane A — this lane adds the onEvent audit-writer callback
    so every supervisor emission (started, worker_spawned,
    worker_exited, worker_spawn_failed, backoff, health_warn,
    health_error, max_crashes_exceeded, shutting_down, stopped) lands
    in the JSONL file with the supervisor's PID attached.

--help body updated:
  - Three separate usage lines (start / status / stop).
  - SUBCOMMANDS block with one-line summaries each.
  - EXIT CODES block (unchanged from Lane A, moved under SUBCOMMANDS).
  - EXAMPLES block updated with status --json + stop + --detach forms.

Tests: existing 127 supervisor + minions tests continue to pass.
Integration tests for the new subcommands + audit writer land with
Lane E.

Follow-up (Lane D): `gbrain doctor` will read readSupervisorEvents()
from this module to surface a `supervisor` health check alongside its
existing checks (DB connectivity, schema version, queue health).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doctor: add supervisor health check

Lane D of PR #364 review fixes. Closes the observability loop: now that
Lane C writes supervisor lifecycle events to
`${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`,
`gbrain doctor` surfaces a `supervisor` check alongside its existing
health indicators.

Implementation (src/commands/doctor.ts, filesystem-only block 3b-bis):
- Resolves DEFAULT_PID_FILE via the same three-tier logic as the start
  path (--pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid).
- Reads the PID file + `kill -0 <pid>` for liveness.
- Calls readSupervisorEvents({sinceMs: 24h}) from the audit module to
  derive last_start / crashes_24h / max_crashes_exceeded.
- Suppresses the check entirely when the user has never invoked the
  supervisor (no PID file AND no audit events) — avoids noise on
  installs that don't use the feature.

Status thresholds:
  fail   max_crashes_exceeded event seen in last 24h
         (supervisor gave up; operator needs to restart or triage)
  warn   supervisor not running but audit shows prior use
         (unexpected stop — likely crash or manual kill)
  warn   running but > 3 crashes in last 24h
         (supervisor recovering but worker is unstable)
  ok     running + ≤ 3 crashes + no max_crashes event

All failure paths emit a paste-ready recovery command. Read/import
errors are swallowed (best-effort like the other doctor checks).

Tests: all 127 supervisor + minions tests still green; 13 existing
doctor tests unaffected.

F3 done. All four lanes A/B/C/D are now committed; Lane E (integration
tests) and Lane F (/ship v0.20.2) remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: 4 critical integration tests for supervisor lifecycle

Lane E of PR #364 review fixes (blocker 10). Fills the ~15% coverage
gap flagged in the eng review by actually exercising the code paths
that will break in production — crash-restart loop, max-crashes exit,
SIGTERM-during-backoff, env-var inheritance — via real spawn() calls
against fake shell-script workers. No mocks: real fork, real signals,
real env propagation, real audit file writes.

test/fixtures/supervisor-runner.ts (new, 55 lines):
  A standalone bun script that constructs a MinionSupervisor from env
  vars (SUP_PID_FILE / SUP_CLI_PATH / SUP_MAX_CRASHES / SUP_BACKOFF_FLOOR_MS
  / SUP_HEALTH_INTERVAL_MS / SUP_ALLOW_SHELL_JOBS / SUP_AUDIT_DIR) and
  calls start(). Mock engine returns empty rows for executeRaw (health
  check path still exercised without Postgres). Tests spawn this as a
  subprocess because MinionSupervisor.start() calls process.exit() on
  shutdown — can't run it in the test runner's own process.

test/supervisor.test.ts (existing; 91 → 300 lines):
  - Added IntegrationHarness helper: creates a unique tmpdir per test,
    a fake worker shell script, a PID-file path, and an audit-dir path;
    cleanup runs in finally.
  - spawnSupervisor() forks bun on the runner with env vars set.
  - readAudit() reads the supervisor-YYYY-Www.jsonl file via the
    existing readSupervisorEvents() helper (Lane C), threading
    GBRAIN_AUDIT_DIR through so tests don't collide on ~/.gbrain.
  - waitFor(pred, timeoutMs) polls helper for event-driven tests.

Four integration tests (with _backoffFloorMs=5 for <1s suite runs):

  1. "respawns the worker after a crash and eventually exits with
     max-crashes code=1"
     Worker always `exit 1`. maxCrashes=3. Asserts: exit code 1, PID
     file cleaned up, audit contains started + 3x worker_spawned +
     3x worker_exited + max_crashes_exceeded + shutting_down + stopped,
     and the stopped event carries {reason:'max_crashes', exit_code:1}.
     Locks in blockers 1 (PID lock), 2+3+6 (health SQL doesn't 500),
     5 (unified shutdown emits right events), F8 (spawn errors counted).

  2. "receives SIGTERM while sleeping between crashes and exits 0 cleanly"
     Worker always `exit 1`, backoff floor 800ms to catch the sleep.
     Asserts: SIGTERM during backoff → exit code 0 (not 1) in <5s,
     no signal kill (process.exit via shutdown), audit contains
     shutting_down {reason:'SIGTERM'} + stopped, PID file cleaned up.
     Locks in eng Issue 1 (unified exit path), eng Issue 3 (signal
     handlers don't accumulate across shutdowns).

  3. "strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false,
     even if parent has it set"  ⚠ CRITICAL regression test
     Parent env has GBRAIN_ALLOW_SHELL_JOBS=1. SUP_ALLOW_SHELL_JOBS=0.
     Worker writes $GBRAIN_ALLOW_SHELL_JOBS (or 'UNSET' if absent) to
     an OUT_FILE. Asserts child sees 'UNSET'. Locks in codex #9 + eng
     #8: the `else delete env.GBRAIN_ALLOW_SHELL_JOBS` branch from
     Lane A is load-bearing for the supervisor's security posture;
     this test prevents a future refactor silently re-opening the
     inheritance hole.

  4. "DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs=true"
     Positive-path companion to #3. SUP_ALLOW_SHELL_JOBS=1 → worker
     sees '1'. Confirms the else-branch doesn't over-strip and that
     operators who explicitly opt in still get shell-exec enabled.

Plus two audit-format unit tests:
  - computeSupervisorAuditFilename format (regex match)
  - Year-boundary ISO week: 2027-01-01 → supervisor-2026-W53.jsonl
    (matches the shell-audit.ts pattern exactly)

Before: 7 tests covering backoff math + PID helpers (~15% behavioral
coverage per eng review).
After: 13 tests across all critical lifecycle paths (crash-restart,
max-crashes, SIGTERM, env-inheritance, audit rotation).

All 146 tests in supervisor + minions + doctor suites green in ~8s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.20.2)

Lane F of PR #364 review fixes. Closes the multi-lane plan with release
hygiene: VERSION bump 0.19.0 → 0.20.2, package.json sync, CHANGELOG entry
in GStack voice with release summary + "numbers that matter" table +
"To take advantage of v0.20.2" migration block + itemized changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: escape template-literal interpolation in supervisor --help

The --help body in src/commands/jobs.ts is one big backtick template
literal. The supervisor subcommand description I added in Lane B used
both `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}` (parsed as a template
interpolation into an undefined variable) and inline `code` backticks
(parsed as nested template literals). CI caught it with ~200 tsc parse
errors across the file.

Fix:
- Escape `${...}` → `\${...}` so the audit-file path renders literally.
- Replace prose inline-code backticks with plain single-quote fences
  (`gbrain jobs work` → 'gbrain jobs work', `~/.gbrain/supervisor.pid`
  → ~/.gbrain/supervisor.pid). `--help` output is human prose; the
  single-quote form reads cleanly in a terminal without needing to
  smuggle nested backticks through a template literal.

`bunx tsc --noEmit` is clean. 146 tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt after Lane B doc rewrite

CI drift guard caught that `llms-full.txt` didn't match the current
generator output. Root cause: the Lane B rewrite of
`docs/guides/minions-deployment.md` (supervisor as canonical, watchdog
deleted) changed content that gets inlined into `llms-full.txt`, but I
didn't run `bun run build:llms` to regenerate.

`bun test test/build-llms.test.ts` now clean (7/7 pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:24:10 -07:00
38 changed files with 3578 additions and 574 deletions
+263
View File
@@ -2,6 +2,269 @@
All notable changes to GBrain will be documented in this file.
## [0.20.4] - 2026-04-24
**Minions skill consolidation, now honest about what the CLI actually does.**
One skill for background work instead of two. Shell jobs and LLM subagents land under `skills/minion-orchestrator/` with a shared Preconditions block, accurate CLI examples, and a trigger set narrowed to what the skill actually covers. Corrects four documentation bugs the prior merge shipped ... `submit_job name="shell"` isn't MCP-callable, `research`/`orchestrate` aren't real handler names, PGLite users don't need to migrate to Supabase, and "every background task goes through Minions" contradicts the `pain_triggered` default in `skills/conventions/subagent-routing.md`. The skill now matches the code.
Two new tests guard this surface going forward. `test/resolver.test.ts` gets a round-trip check (every quoted RESOLVER.md trigger must resolve to a frontmatter `triggers:` entry in the target skill) and a name validator (every `name="<word>"` reference in any SKILL.md must resolve to either a declared operation in `src/core/operations.ts` or a known Minions handler). The validator would have caught the `research`/`orchestrate` drift in CI instead of from a Codex cold-read. One new E2E test (`test/e2e/minions-shell-pglite.test.ts`) exercises the PGLite `--follow` inline path, previously documented but untested.
### For users
- Shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only ... MCP returns `permission_denied` for protected names). Subagent jobs via `gbrain agent run` (user-facing entrypoint). Both lanes route through one skill.
- PGLite shell-job guidance now correctly points at `--follow` for inline execution. The persistent daemon mode is still Postgres-only, but you do not need to migrate.
- `gbrain jobs submit` and `submit a gbrain job` now route to the skill; bare "gbrain jobs" no longer does (it was too broad ... the CLI namespace covers 9 subcommands, and questions about `stats`/`prune`/`retry` fall through to `gbrain --help`).
### Added
- New E2E test `test/e2e/minions-shell-pglite.test.ts` covering the PGLite `--follow` inline shell-job path. Runs in-memory, no DATABASE_URL required.
- Resolver round-trip test in `test/resolver.test.ts`: every quoted RESOLVER.md trigger must have a fuzzy match in the target skill's frontmatter `triggers:` list.
- Skill-example-name validator in `test/resolver.test.ts`: every `name="<word>"` reference in any `SKILL.md` body must resolve to an op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`.
### Fixed
- `skills/minion-orchestrator/SKILL.md` shell-job examples use the real `--params` JSON form instead of nonexistent `--cmd`/`--argv`/`--cwd` flags.
- `gbrain agent run` flag list now matches `src/commands/agent.ts` (removed `--queue`/`--priority`/`--max-attempts`/`--delay` which aren't parsed by that command).
- `--tools` example uses `search,query` instead of `web_search` (the latter isn't in `BRAIN_TOOL_ALLOWLIST`, would throw at submit time).
- MCP boundary wording says `submit_job name="shell"` throws an `OperationError` with code `permission_denied`, instead of the earlier "returns permission_denied" (not a return, a throw).
- `skills/conventions/subagent-routing.md` stale reference to `get_job_stats` (no such op) replaced with `list_jobs --status active` or `gbrain jobs stats`.
- `skills/query/SKILL.md` + `skills/maintain/SKILL.md` frontmatter `triggers:` lists closed gaps the new round-trip test surfaced (RESOLVER.md was routing 10 triggers to these skills that their frontmatter never declared).
- `skills/manifest.json` minion-orchestrator description updated to match the unified SKILL.md framing.
### Changed
- Trigger `"gbrain jobs"` narrowed to `"gbrain jobs submit"` + `"submit a gbrain job"` in both `skills/RESOLVER.md` and the skill's frontmatter.
- Anti-pattern about `sessions_spawn` scoped to the subagent lane (was ambiguous in the consolidated skill).
### For contributors
- Code-to-doc drift is now partially machine-checkable. The skill-example-name validator catches T2-class bugs (docs referencing handler/op names that don't exist). CLI flag validation is a remaining gap ... a future PR could extend the test to validate `--flag-name` patterns in SKILL.md against actual CLI flag parsers.
## To take advantage of v0.20.4
Any gbrain user whose agent routes on "minions" work gets the corrected skill on the next `gbrain upgrade`. No manual migration required ... the renamed trigger is additive (old trigger gone, new triggers cover the same intent), and the doc corrections don't change runtime behavior.
1. **Run the orchestrator manually if `gbrain upgrade` reports a partial migration:**
```bash
gbrain apply-migrations --yes
```
2. **Your agent picks up the new skill content** next time it consults `skills/minion-orchestrator/SKILL.md`. No action required on your side.
3. **Verify the outcome:**
```bash
gbrain check-resolvable --json | python3 -c "import json,sys;d=json.load(sys.stdin);print('ok:',d['ok'])"
```
Should print `ok: True`.
4. **If any step fails,** file an issue at https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
## [0.20.3] - 2026-04-24
## **Your queue now rescues itself when a wedged worker holds a row lock. Wall-clock sweep kills the job that stall detection can't see.**
## **`maxWaiting` is race-proof, observable, and reachable from the CLI — three bugs in one patch.**
A production autopilot-cycle job wedged for over an hour on a single OpenClaw deployment because the worker's handler got stuck mid-transaction holding a row lock. Both eviction paths were blocked: the stall detector's `FOR UPDATE SKIP LOCKED` pass skipped the row-locked candidate, and the timeout sweep's `lock_until > now()` predicate disqualified the job once lock-renewal had been blocked. Neither could see the job. The shell-job pipeline starved completely behind the wedge.
v0.19.0 shipped the wall-clock sweep as the third-layer kill shot: drop both constraints, evict on `started_at` alone, worst case at `2 × timeout_ms + stalledInterval`. This release locks down three correctness holes the v0.19.0 PR introduced — then closes the observability gap that let the incident run to minute 90 in the first place.
### The queue-resilience numbers that matter
Measured against the real incident on 2026-04-23 (OpenClaw autopilot + shell-job pipeline, Postgres engine, concurrency=1 worker).
| Behavior | Before v0.20.3 | After v0.20.3 |
|---|---|---|
| Wedged worker escape window | 90+ minutes (manual kill) | `~2 × timeout_ms + 30s` sweep interval |
| Per-name waiting pile during wedge | 18 deferred per-slot jobs | capped at `maxWaiting` |
| `maxWaiting` under concurrent submit (2 submitters, cap=2) | up to 3 rows (TOCTOU race) | exactly 2 rows (advisory-lock serialization) |
| Same name across queues | cross-queue bleed — `shell` suppressed by `default` | isolated per `(name, queue)` |
| `GBRAIN_WORKER_CONCURRENCY=foo` | silent wedge (`inFlight < NaN` false) | clamped to 1, loud stderr warning |
| `gbrain jobs submit --max-waiting 2` | flag didn't exist | wired through to MinionJobInput |
| Silent coalesce events | invisible | JSONL audit at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` |
| `gbrain doctor` visibility into wedge | no check | new `queue_health` with 2 subchecks |
The two big shifts: (1) every silent-failure vector the v0.19.0 patches introduced now has a loud signal — JSONL audit files, doctor check, stderr warnings, peer-liveness probe. (2) `maxWaiting` is now actually a cap under concurrency, not a soft suggestion. A future multi-submitter pattern (parallel workspaces, dispatched children, OpenClaw + ycli cron) doesn't walk through it.
### What this means for OpenClaw users
If you're running `gbrain autopilot` on a daily-driver deployment, the wall-clock sweep is the difference between a 90-minute outage and a 30-second one. The `queue_health` doctor check means the next time your queue wedges, you notice in minute 2 instead of minute 90. If you've been writing programmatic Minion submitters and setting `maxWaiting`, it's worth re-reading the JSONL audit file the next time your agent does anything "interesting" — you'll see exactly which submission coalesces into which returned job.
## To take advantage of v0.20.3
`gbrain upgrade` handles the binary. You MUST restart long-running worker daemons so the new sweep runs in-process — the wall-clock eviction is a method on `MinionQueue`, not a cron job, so it only fires inside a worker loop.
1. **Upgrade the binary:**
```bash
gbrain upgrade
```
2. **Restart autopilot + workers:**
```bash
# systemd / launchd / OpenClaw service-manager: restart the unit.
# Manual: kill the old `gbrain autopilot` and `gbrain jobs work`, start new ones.
```
3. **Verify:**
```bash
gbrain jobs smoke --wedge-rescue # exercises the new wall-clock path
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
4. **If `gbrain doctor` flags anything unexpected,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/audit/backpressure-*.jsonl` (redact freely)
- what commands you ran leading up to the wedge
### Itemized changes
**Queue core** (`src/core/minions/queue.ts`)
- `maxWaiting` coalesce path wraps `count → select → insert` in `pg_advisory_xact_lock` keyed on `(name, queue)`. Concurrent submitters for the SAME key serialize; different keys stay parallel. Lock auto-releases on transaction commit/rollback — no cleanup path to leak. Fixes TOCTOU race caught by adversarial review.
- `maxWaiting` count and select now filter on `queue` in addition to `name`. Pre-v0.20.3 code filtered on name alone, so a waiting `autopilot-cycle` in `queue=default` would suppress submissions to `queue=shell` with the same name. Cross-queue bleed is gone.
**Backpressure observability** (new `src/core/minions/backpressure-audit.ts`)
- Every coalesce event writes one JSONL line to `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (ISO-week rotation, override dir via `GBRAIN_AUDIT_DIR`, mirrors the v0.14 shell-audit pattern).
- Fields: `ts, queue, name, waiting_count, max_waiting, decision='coalesced', returned_job_id`.
- Best-effort: write failures log to stderr but never block submission.
**CLI** (`src/commands/jobs.ts`)
- New `--max-waiting N` flag on `gbrain jobs submit`. Clamps to `[1, 100]`, mirrors the existing `--max-stalled` wiring. The `MinionJobInput.maxWaiting` field was programmatic-only before; now it's reachable from the command line too.
- `resolveWorkerConcurrency` clamps against invalid input. `parseInt` returns `NaN` for `"foo"`, `0` for `"0"`, negatives for `"-5"` — all of which silently wedge a worker (`inFlight.size < NaN/0/negative` is always false). Now clamped to ≥1 with a loud stderr warning naming the bad value. One typo in a systemd unit no longer reproduces the 90-minute outage.
- New `gbrain jobs smoke --wedge-rescue` opt-in case. Forges a wedged-worker row state, invokes `handleStalled` + `handleTimeouts` + `handleWallClockTimeouts` in sequence, asserts only the wall-clock sweep evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape.
**Doctor** (`src/commands/doctor.ts`)
- New `queue_health` check (Postgres-only; PGLite skips with `Skipped (PGLite — no multi-process worker surface)`).
- Subcheck 1 — **stalled-forever**: flags active jobs whose `started_at` is older than 1 hour. Reports the top 5 by start time with `gbrain jobs get/cancel <id>` fix hints.
- Subcheck 2 — **waiting-depth**: flags per-name queues whose waiting count exceeds threshold. Default 10, overridable via `GBRAIN_QUEUE_WAITING_THRESHOLD` env. Reports the top 5 by depth with "consider setting maxWaiting on the submitter" fix hint.
- Worker-heartbeat staleness subcheck intentionally deferred to follow-up because `lock_until`-on-active-jobs is a lossy proxy. A check that cries wolf erodes trust in every other doctor subcheck. Needs a `minion_workers` table to produce ground-truth signal.
**Autopilot** (`src/commands/autopilot.ts`)
- `--no-worker` mode gains a peer-worker-liveness probe. Every cycle runs a cheap `SELECT count(*)` checking for active jobs with `lock_until` refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud `WARNING` naming the silent-wedge vector (`--no-worker` set but no worker running). Re-arms once a live signal returns, so a healthy-but-idle worker doesn't trigger spam.
- Probe is documented as a proxy, not ground truth — idle worker with no active jobs reads as "no worker." The ground-truth fix needs a `minion_workers` heartbeat table (tracked as follow-up).
**Docs**
- New `docs/guides/queue-operations-runbook.md`: the "my queue looks wedged — what do I run?" reference. One viewport, in order of escalation. What each `queue_health` subcheck means. Self-check for the `--no-worker + no-worker-running` footgun.
- `CLAUDE.md` Key-files section updated for the new `handleWallClockTimeouts` method (v0.19.0, described here for the first time), the new `backpressure-audit.ts` module, the updated `maxWaiting` semantics, and the new `queue_health` doctor check.
**Tests** (`test/minions.test.ts`)
- 23 new unit cases. Wall-clock sweep (3 cases + non-interference with `handleTimeouts`). `maxWaiting` (coalesce, clamp 0 → 1, floor 1.7 → 1, concurrent-submitter race via `Promise.all`, cross-queue isolation, unset fallthrough). Concurrency clamp (7 cases including `NaN`/`0`/negative). `parseMaxWaitingFlag` (5 cases). Backpressure audit file write. All 143 minions tests pass.
- E2E wall-clock case against real Postgres is next on the roadmap (needs a second-connection row-lock helper; the unit-level coverage above exercises the sweep mechanics directly).
### For contributors
- The v0.19.0 PR's narrative framed the 18-job pileup as "duplicate submissions from a cron loop with no idempotency key." That framing was wrong. Autopilot already sets `idempotency_key: autopilot-cycle:${slot}` where slot is a 5-minute tick boundary — within-slot duplicates are structurally impossible. The 18 jobs were 18 different slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before v0.20.3 shipped.
- Follow-up issues tracked: B2 (autopilot heartbeat file), B3 (doctor `--fix` learns queue rescue), B4 (backpressure counts surfaced in `jobs stats`), B5 (cross-cutting "health-delivery-agent" pattern), B7 (`minion_workers` heartbeat table — unblocks both the dropped `queue_health` subcheck and a ground-truth `--no-worker` probe), P1 (composite indexes `(status, started_at)` and `(status, name)` on `minion_jobs` — currently the new sweeps fall back to `idx_minion_jobs_status`, selective enough on healthy queues, worth tightening in v0.20.4).
Full plan with CEO + Eng + Codex adversarial decisions lives at `~/.claude/plans/` for the operators who care about how this release was reviewed.
## [0.20.2] - 2026-04-24
## **`gbrain jobs supervisor` is now a self-healing daemon you can actually drive. The Minions worker stops dying silently.**
## **Three commands an agent can run: `start --detach`, `status --json`, `stop`. Crash loops are bounded, audit events are JSONL, and the health check finally reports real data.**
`gbrain jobs work` has always been the worker that drains your Minions queue. Problem: it dies (OOM, connection blip, panic) and nobody notices until jobs pile up. The old answer was `nohup` plus a 68-line bash watchdog script from the deployment guide, and it shipped its own bugs (restart-loop traps, log-parsing stall detection, zero audit trail).
v0.20.2 ships the replacement: `gbrain jobs supervisor` is a first-class CLI with atomic PID locking, exponential backoff, structured audit events at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`, and three subcommands that make it drivable by an OpenClaw or Hermes agent in three turns. The old bash watchdog is gone.
### The numbers that matter
Before v0.20.2, an agent driving the supervisor needed ~10 turns of shell archaeology (PID file scraping, `pgrep -f`, `kill -0`, log grep) just to start and stop the worker reliably. After v0.20.2, it's three commands with machine-parseable output.
| Capability | Before v0.20.2 | After v0.20.2 |
|---|---|---|
| Keeping the worker alive | `nohup` + `minion-watchdog.sh` (68 lines of bash, restart-loop bug, log-scrape health) | `gbrain jobs supervisor` (first-class CLI with atomic PID lock, exponential backoff, JSONL audit) |
| PID file locking | `existsSync + readFileSync + writeFileSync` TOCTOU race | Atomic `O_CREAT|O_EXCL` via `openSync('wx')` — kernel-atomic mutex |
| Stalled-jobs health alert | Queried `status='stalled'` — returned 0 rows forever (dead code) | Queries `status='active' AND lock_until < now()`, scoped to the supervised queue |
| Shell-exec env inheritance | Child inherited `GBRAIN_ALLOW_SHELL_JOBS=1` from parent shell regardless of CLI flag | Explicit `else delete env.GBRAIN_ALLOW_SHELL_JOBS` when not opted in + regression test |
| Agent discovery TTHW | ~10 turns of shell-scraping (cat PID / pgrep / kill -0 / log grep) | 3 turns: `start --detach``status --json``stop` |
| Lifecycle observability | `console.log` with human prefixes, zero audit trail | JSONL events on stderr + `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` + `gbrain doctor` integration |
| Exit codes | undocumented; agent couldn't distinguish "already running" from "gave up" | Four documented codes: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable |
| Test coverage of the supervisor itself | ~15% (backoff math + PID helpers only) | Integration tests covering crash-restart, max-crashes drain, SIGTERM-during-backoff, env-inheritance regression |
The supervisor's own reliability claims are now testable. Every lifecycle event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`) lands in a weekly-rotated JSONL file that `gbrain doctor` reads to surface a `supervisor` health check.
### What this means for your deployment
If you were using the old `nohup`/`minion-watchdog.sh` pattern:
1. **Stop the old watchdog:** `sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && crontab -e` and delete the watchdog cron line.
2. **Delete the script:** `sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid /tmp/gbrain-worker.log`.
3. **Start the supervisor:** `gbrain jobs supervisor start --detach --json` — or on systemd, reinstall the unit (now calls `gbrain jobs supervisor`).
4. **Verify:** `gbrain doctor` reports a `supervisor` check; `gbrain jobs supervisor status --json` returns `running:true`.
For containers (Fly / Railway / Render / Heroku): the shipped `Procfile` and `fly.toml.partial` now call `gbrain jobs supervisor`. The platform restarts the container on host events, the supervisor restarts the worker on in-process crashes. Two-layer supervision with clean separation.
For OpenClaw / Hermes / Cursor agents driving the supervisor: you no longer need a shell skill to drive the worker. Every piece of state — liveness, crash history, max-crashes exhaustion — is a machine-parseable JSON response. Start with `gbrain jobs supervisor status --json | jq`.
## To take advantage of v0.20.2
`gbrain upgrade` pulls the binary. Nothing else is required if you're currently running `gbrain jobs work` directly or using systemd — the new supervisor is opt-in. To migrate:
1. **Verify the binary:**
```bash
gbrain --version # should say 0.20.2
gbrain jobs supervisor --help | head -20
```
2. **Start the supervisor (detached, agent-friendly):**
```bash
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"pid_file":"/Users/you/.gbrain/supervisor.pid","detached":true}
```
3. **Check health:**
```bash
gbrain jobs supervisor status --json
gbrain doctor | grep supervisor
```
4. **Stop when done:**
```bash
gbrain jobs supervisor stop
```
5. **(Optional) Migrate off the old watchdog:** see `docs/guides/minions-deployment.md` "Upgrading from an older deployment" for the cron-to-supervisor migration.
If `gbrain jobs supervisor status` reports `running:false` unexpectedly, or `gbrain doctor` flags a `supervisor` failure, file an issue at https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- the last ~50 lines of `~/.gbrain/audit/supervisor-*.jsonl`
- which step broke
### Itemized changes
**`gbrain jobs supervisor`:**
- New subcommands: `start [--detach] [--json]`, `status [--json]`, `stop [--json]`. Foreground use is unchanged (back-compat).
- New flags: `--allow-shell-jobs` (explicit opt-in, replaces env-var sniffing), `--cli-path PATH` (override auto-resolution), `--json` (JSONL lifecycle events on stderr), `GBRAIN_SUPERVISOR_PID_FILE` env var (overrides default PID path).
- Exit codes documented in `--help`: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable.
- Default PID path moved from `/tmp/gbrain-supervisor.pid` to `~/.gbrain/supervisor.pid` with automatic parent-directory creation.
**Safety fixes (codex adversarial review + eng review):**
- Atomic PID lock via `openSync(path, 'wx')` — two supervisors starting simultaneously can no longer both win the race.
- `stalled` health check query rewritten from unreachable `status='stalled'` to `status='active' AND lock_until < now()` matching `queue.ts:848 handleStalled()`.
- Health queries now scoped to `WHERE queue = $1` — multi-queue deployments see the right queue.
- Unified exit path via `shutdown(reason, exitCode)` — max-crashes drains gracefully instead of bypassing cleanup via `process.exit(1)`.
- Listener ref tracking: `SIGTERM`/`SIGINT` handlers removed on shutdown for clean test lifecycle.
**Security hardening:**
- `allowShellJobs` class default flipped `true``false`.
- Child env now has `GBRAIN_ALLOW_SHELL_JOBS` explicitly deleted when `allowShellJobs:false` (was: silently inherited from parent shell).
- Integration regression test locks this against future refactors.
**Observability:**
- New `src/core/minions/handlers/supervisor-audit.ts` with ISO-week rotation (mirrors `shell-audit.ts` / `subagent-audit.ts` pattern).
- Every supervisor emission (started, worker_spawned, worker_exited, worker_spawn_failed, backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, stopped) written to `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`.
- `gbrain doctor` gains a `supervisor` check that reads the audit file and reports `running` / `last_start` / `crashes_24h` / `max_crashes_exceeded` with thresholds (ok / warn at 3+ crashes / fail on max-crashes event).
**Documentation:**
- `docs/guides/minions-deployment.md` rewritten: supervisor is the canonical answer; which-supervisor-when decision table (container / systemd / dev laptop); three-command agent pattern; migration block from the old watchdog.
- `README.md` Operations section gains a paragraph on `gbrain jobs supervisor`.
- `docs/guides/minions-deployment-snippets/{systemd.service,Procfile,fly.toml.partial}` now invoke `gbrain jobs supervisor` instead of raw `gbrain jobs work`.
- `docs/guides/minions-deployment-snippets/minion-watchdog.sh` deleted — subsumed by the supervisor.
**Tests:**
- `test/supervisor.test.ts`: 7 → 13 tests. Four new integration tests exercise real `spawn()` lifecycles via shell-script fakes (crash-restart happy path, max-crashes-via-shutdown with audit assertions, SIGTERM-during-backoff clean exit, `GBRAIN_ALLOW_SHELL_JOBS` inheritance regression — positive + negative).
- `test/fixtures/supervisor-runner.ts`: new standalone runner that constructs a supervisor from env vars so integration tests can observe `process.exit` without killing the test runner.
**For contributors:**
- The `MinionSupervisor` class has a test-only `_backoffFloorMs` override for fast crash-loop tests. Not exposed via CLI.
- `onEvent: (emission) => void` is an injectable hook on `SupervisorOpts` — Lane C's audit writer uses it; future observability integrations can too.
- `autopilot.ts` migration to `MinionSupervisor` is explicitly deferred (follow-up PR): the current `start()` API blocks, which deadlocks autopilot's interval loop. Codex's review flagged this; the fix is a non-blocking-start API redesign, not a drop-in substitution.
Credit: original supervisor feature built by OpenClaw (PR #364 initial commit). Review wave + code-level fixes + daemon-manager CLI + observability boomerang + integration tests shipped via /autoplan (CEO + DX + Eng + Codex adversarial) followed by a 20-item multi-lane implementation plan.
## [0.20.0] - 2026-04-23
## **BrainBench moves out. gbrain gets its install surface back.**
+16 -6
View File
@@ -62,12 +62,13 @@ strict behavior when unset.
- `src/commands/graph-query.ts``gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
- `src/core/minions/types.ts``MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
- `src/core/minions/handlers/shell.ts``shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
@@ -89,7 +90,7 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts``gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
@@ -142,7 +143,7 @@ strict behavior when unset.
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
- `skills/minion-orchestrator/SKILL.md`Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
- `skills/minion-orchestrator/SKILL.md`Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
@@ -227,7 +228,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
`test/report.test.ts` (report format, directory structure),
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
@@ -275,6 +276,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
@@ -327,7 +329,7 @@ stop and remove it before starting a new one.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
@@ -337,11 +339,19 @@ briefing, migrate, setup, publish.
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
routing is narrowed to what the skill actually covers.
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
(agent-readable health report).
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
`~/.gbrain/smoke-tests.d/*.sh`).
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
`skills/_output-rules.md` are shared references.
+11 -7
View File
@@ -6,7 +6,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
@@ -28,7 +28,7 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
@@ -87,9 +87,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
## The 28 Skills
## The 29 Skills
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
@@ -135,7 +135,8 @@ GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
### Identity and setup
@@ -211,9 +212,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis
gbrain jobs smoke # verify install
gbrain jobs submit sync --params '{}' # fire a background job
gbrain jobs stats # health dashboard
gbrain jobs work --concurrency 4 # start a worker (Postgres only)
gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
```
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.
@@ -374,7 +378,7 @@ Run `gbrain integrations` to see status.
│ Brain Repo │ │ GBrain │ │ AI Agent │
│ (git) │ │ (retrieval) │ │ (read/write) │
│ │ │ │ │ │
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
│ = source of │ │ pgvector │ │ define HOW to │
│ truth │ │ │ │ use the brain │
│ │<───│ hybrid │ │ │
+1 -1
View File
@@ -1 +1 @@
0.20.0
0.20.4
@@ -7,4 +7,7 @@
# DATABASE_URL=postgresql://...
# GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
worker: gbrain jobs work --concurrency 2
# Two-layer supervision: the platform restarts the container on host
# events (OOM, deploy); `gbrain jobs supervisor` restarts the worker
# on in-process crashes with exponential backoff.
worker: gbrain jobs supervisor --concurrency 2
@@ -5,10 +5,12 @@
# fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
# fly secrets set ANTHROPIC_API_KEY=... # optional
#
# Fly.io auto-restarts the process on crash — no watchdog needed.
# Two-layer supervision: Fly restarts the VM on host events; the
# `gbrain jobs supervisor` process restarts the worker on in-process
# crashes with exponential backoff and a structured audit trail.
[processes]
worker = "gbrain jobs work --concurrency 2"
worker = "gbrain jobs supervisor --concurrency 2"
# Scale the worker process to 1 machine (job queue serializes work; more
# machines means higher concurrency but also more Postgres connections).
@@ -1,68 +0,0 @@
#!/bin/bash
# minion-watchdog.sh — restart gbrain jobs work if the process is dead or
# has logged a shutdown marker since its last start.
#
# Fixes the v0.16.1 restart-loop bug: old shutdown lines from previous
# restarts stayed in the unrotated log and every tick re-matched them
# forever. This version writes a restart epoch to line 2 of the PID file
# and only considers log lines newer than that epoch.
#
# Run every 5 minutes from crontab. See docs/guides/minions-deployment.md.
set -u
PID_FILE="${GBRAIN_WORKER_PID_FILE:-/tmp/gbrain-worker.pid}"
LOG_FILE="${GBRAIN_WORKER_LOG_FILE:-/tmp/gbrain-worker.log}"
GBRAIN="${GBRAIN_BIN:-/usr/local/bin/gbrain}"
CONCURRENCY="${GBRAIN_WORKER_CONCURRENCY:-2}"
start_worker() {
# stderr merged so banner lines ("[minion worker] shell handler enabled",
# "worker shutting down") all land in $LOG_FILE.
nohup "$GBRAIN" jobs work --concurrency "$CONCURRENCY" \
> "$LOG_FILE" 2>&1 &
local pid=$!
# Line 1: PID. Line 2: restart epoch (seconds since 1970).
# Readers that want just PID use `head -n1 "$PID_FILE"`.
printf '%s\n%s\n' "$pid" "$(date +%s)" > "$PID_FILE"
}
shutdown_since_restart() {
# Only match shutdown lines logged AFTER the most recent restart epoch.
# Worker log lines start with ISO-8601 UTC timestamps ("2026-04-21T19:05:12Z ...").
local restart_epoch
restart_epoch=$(sed -n '2p' "$PID_FILE" 2>/dev/null || echo 0)
[ -z "$restart_epoch" ] && restart_epoch=0
# POSIX-portable regex (no {n} intervals — mawk on Debian/Ubuntu rejects them).
awk -v since="$restart_epoch" '
match($0, /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9:.+Z-]+/) {
ts_str = substr($0, RSTART, RLENGTH)
cmd = "date -d \"" ts_str "\" +%s 2>/dev/null"
cmd | getline ts
close(cmd)
if (ts + 0 > since + 0) print
}
' "$LOG_FILE" 2>/dev/null | grep -q "worker stopped\|worker shutting down"
}
if [ -f "$PID_FILE" ]; then
PID=$(head -n1 "$PID_FILE")
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
# Process alive — check whether the worker logged an internal shutdown
# AFTER the last start. If yes, worker is dead-inside; restart.
if shutdown_since_restart; then
kill "$PID" 2>/dev/null
# 10s grace: covers shell handler's 5s child SIGTERM→SIGKILL window
# and leaves room for in-flight jobs to flush. Bump to 30 if your
# jobs run > 10s.
sleep 10
kill -9 "$PID" 2>/dev/null
start_worker
fi
else
# PID file exists but process is gone (crash / kill -9 / reboot).
start_worker
fi
else
start_worker
fi
@@ -15,9 +15,13 @@ WorkingDirectory=/srv/gbrain
# Env file is mode 600, owned by User=. Do not put secrets in this unit.
EnvironmentFile=/etc/gbrain.env
ExecStart=/usr/local/bin/gbrain jobs work --concurrency 2
# Two-layer supervision: systemd restarts `gbrain jobs supervisor` on host
# events (reboot, unit crash); the supervisor restarts `gbrain jobs work`
# on in-process crashes with exponential backoff + structured audit.
ExecStart=/usr/local/bin/gbrain jobs supervisor --concurrency 2
# Replaces the cron watchdog. systemd restarts on any non-zero exit.
# systemd restarts the supervisor on any non-zero exit. The supervisor
# itself handles worker-level crash recovery.
Restart=always
RestartSec=10s
@@ -38,7 +42,9 @@ NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/srv/gbrain
# ReadWritePaths must include the brain workspace AND ~/.gbrain (PID file +
# audit log written by the supervisor).
ReadWritePaths=/srv/gbrain /home/gbrain/.gbrain
[Install]
WantedBy=multi-user.target
+208 -199
View File
@@ -1,7 +1,7 @@
# Minions Worker Deployment Guide
Deploy `gbrain jobs work` so it stays running across crashes, reboots, and
Postgres connection blips. Written for agents to execute line-by-line.
Keep `gbrain jobs work` running across crashes, reboots, and Postgres
connection blips. Written for agents to execute line-by-line.
## The problem
@@ -12,10 +12,61 @@ The persistent worker can die silently from:
- Bun process crashes with no automatic restart.
- Internal event-loop death (PID alive, worker loop stopped).
When the worker dies, submitted jobs sit in `waiting` forever. Nothing in
gbrain core auto-restarts the worker — that's what this guide wires up.
When the worker dies, submitted jobs sit in `waiting` forever. The
canonical answer is `gbrain jobs supervisor` — a first-class CLI that
spawns `gbrain jobs work` as a child and auto-restarts it on crash.
## Variables used in this guide
## Worker supervision
### The canonical pattern
`gbrain jobs supervisor` is an auto-restarting wrapper around
`gbrain jobs work`. It writes a PID file, restarts the worker on crash
with exponential backoff (1s → 60s cap), emits lifecycle events to an
audit file, and drains gracefully on SIGTERM (35s worker-drain window
before SIGKILL). Exit codes are documented so agents can branch on them.
**Typical commands:**
```bash
# Start in the foreground (blocks; Ctrl-C to stop).
gbrain jobs supervisor --concurrency 4
# Start detached — returns {"event":"started","supervisor_pid":…} on stdout.
gbrain jobs supervisor start --detach --json
# Check liveness without reading log files.
gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
```
**Exit codes:**
| Code | Meaning |
|---|---|
| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) |
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=1 should page a human.
### Which supervisor when?
The supervisor solves in-process crash recovery. Platform-level
supervision (systemd, Fly, Render) handles host-level failures. You
usually want both.
| Environment | Recommendation |
|---|---|
| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). |
| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). |
| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. |
### Variables used in this guide
Substitute these once before copy-pasting any snippet.
@@ -23,142 +74,122 @@ Substitute these once before copy-pasting any snippet.
|---|---|---|
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) |
| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) |
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) |
## Preconditions
### Preconditions
Run these before Step 1 of any option. Fail fast if something is wrong.
Run these before any deployment step.
```bash
# 1. gbrain is on PATH and resolves to an absolute location.
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists).
# 2. DATABASE_URL points at reachable Postgres.
# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the
# separate worker process. If `config.engine === 'pglite'` the CLI rejects
# with a clear error.)
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
# 3. Schema is up to date. If version=0 or status=="fail", fix it first:
# 3. Schema is up to date. If version=0 or status=="fail":
# gbrain apply-migrations --yes
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
# 4. You have write access to at least one crontab mechanism.
crontab -l >/dev/null 2>&1 && echo "user crontab OK"
[ -w /etc/crontab ] && echo "/etc/crontab OK"
# 5. If you plan to submit `shell` jobs, the WORKER process needs
# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated
# in registerBuiltinHandlers(); without the flag the worker startup
# line reads "shell handler disabled (...)".
# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the
# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting).
# Without the flag, the shell handler is disabled at worker startup.
```
## Which option?
## Agent usage (OpenClaw / Hermes / Cursor / Codex)
- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take
> 30 s → **Option 1** (watchdog cron + persistent worker).
- Your workload is short deterministic scripts on a fixed schedule (every
3 h, daily, weekly) → **Option 2** (inline `--follow`).
- You don't have shell access to a long-running box (Fly/Render/Railway,
or any systemd host) → **Option 3** (service manager — replaces cron).
## Option 1: watchdog cron + persistent worker
A 5-minute cron checks whether the worker process is alive **and** whether
it has logged an internal shutdown since its last start. Restarts if either
condition fails.
### 1a. Install the env file (secrets stay out of crontab)
Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is
mode 644 (world-readable); user crontabs under `/var/spool/cron/` are
readable by `root`. Use the shipped env-file template:
Three-command pattern an agent can drive without shell archaeology:
```bash
sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
# Start (returns PIDs + pid_file on stdout as JSON, then detaches)
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
# Check health (machine-parseable JSON, no log scraping)
gbrain jobs supervisor status --json
# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...}
# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback)
gbrain jobs supervisor stop
```
Every lifecycle event (spawn, crash, backoff, health warning, max-crashes,
shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
for historical inspection. `gbrain doctor` reads that file and surfaces
a `supervisor` check in its health report.
## Deployment: systemd
For long-running Linux VMs with shell access.
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# Install the env file (secrets stay out of the unit file).
sudo install -m 600 -o gbrain -g gbrain \
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
sudoedit /etc/gbrain.env
# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1.
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
```
Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if
applicable). See
[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example)
for the full list.
The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work`
directly) so you get two-layer supervision: systemd restarts the supervisor
on host reboot, supervisor restarts the worker on in-process crash.
### 1b. Install the watchdog script
`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery.
The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and
audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent
LLM subagent calls without hitting the default 1024 cap.
The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh)
ships in-repo and writes a two-line PID file (PID on line 1, restart epoch
on line 2). The restart-epoch marker is how the watchdog distinguishes
stale shutdown lines in the log from current ones — without it, every tick
after the first restart would match an old `worker shutting down` line and
loop forever.
Requires GNU coreutils (Linux default). On macOS/BSD install via
`brew install coreutils` and alias `date` to `gdate` in the cron env if you
want to test the watchdog locally; production Linux boxes work as-is.
## Deployment: Fly.io
```bash
sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
docs/guides/minions-deployment-snippets/minion-watchdog.sh \
/usr/local/bin/minion-watchdog.sh
# Merge the [processes] block from fly.toml.partial into your fly.toml.
cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml
# Review + edit as needed.
# Set secrets (Fly handles restart on crash).
fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1
```
### 1c. Wire into cron
The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly
restarts the container on host failure; the supervisor restarts the
worker on in-process crash.
Pick the form that matches the crontab you're editing.
## Deployment: Render / Railway / Heroku
**If you ran `crontab -e`** (user crontab — 5-field, no user column):
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo
root. The shipped Procfile calls `gbrain jobs supervisor`. Set
`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's
env UI or CLI.
```
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * /usr/local/bin/minion-watchdog.sh
```
## Deployment: inline `--follow` (no persistent worker)
**If you edited `/etc/crontab` directly** (system crontab — 6-field, with
user column):
```
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh
```
In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to
source the env file before running the watchdog — that's how the
connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without
landing in the world-readable crontab itself.
### 1d. Log rotation
The watchdog appends to the worker log across restarts. If you expect the
file to grow unbounded, rotate it externally with `logrotate`:
```
# /etc/logrotate.d/gbrain-worker
/tmp/gbrain-worker.log {
daily
rotate 7
missingok
notifempty
copytruncate
}
```
`copytruncate` is important — the watchdog's restart-epoch check survives
it (the epoch is compared against in-log timestamps, not file inode).
## Option 2: inline `--follow` (no persistent worker)
Each cron run brings its own temporary worker. `--follow` starts one on
the queue and blocks until the just-submitted job reaches a terminal state
(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead
per job; negligible vs job duration for scheduled work.
Example: nightly brain enrichment as a shell job.
For short deterministic scripts on a fixed schedule where you don't need
a persistent worker between runs. Each cron run brings its own temporary
worker. `--follow` starts one on the queue and blocks until the
just-submitted job reaches a terminal state (`completed` / `failed` /
`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job
duration for scheduled work.
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
@@ -170,85 +201,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
Replace `gbrain embed --stale` with whichever gbrain subcommand you're
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
`lint`, `autopilot`). If you're shelling out to a non-gbrain binary,
keep its absolute path in the `cmd`.
**Shared-queue gotcha.** If other jobs are already waiting on the same
queue with higher priority or earlier `created_at`, the temporary worker
processes those first before reaching yours. `--follow` still exits only
when YOUR job finishes. For strict single-job semantics on shared queues,
`lint`, `autopilot`). For strict single-job semantics on shared queues,
use a dedicated queue name like `nightly-enrich` above.
## Option 3: service manager (systemd / Fly / Render / Railway)
## Upgrading from an older deployment
Replaces the watchdog entirely. No cron, no PID file, no restart-loop.
The service manager owns liveness.
### From `minion-watchdog.sh` (pre-v0.20)
### systemd (Linux hosts with shell access)
Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
which handles everything the script did, plus atomic PID locking,
structured audit events, queue-scoped health checks, and graceful
drain on SIGTERM.
**Migration:**
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# 1. Stop and remove the old watchdog.
sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null
sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \
/tmp/gbrain-worker.log
crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
# 2. Start the supervisor (systemd users: reinstall the unit from
# docs/guides/minions-deployment-snippets/systemd.service, which
# now calls `gbrain jobs supervisor`).
gbrain jobs supervisor start --detach --json
# Or: sudo systemctl restart gbrain-worker
# See 1a above for /etc/gbrain.env install.
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
# 3. Verify.
gbrain jobs supervisor status --json
gbrain doctor # 'supervisor' check should report running=true
```
`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit
runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped
unit covers Bun + Postgres pool + concurrent LLM subagent calls without
hitting the default 1024 cap.
### Schema / migration hygiene
### Fly.io
Regardless of which deployment path you're upgrading from:
Merge the `[processes]` block from
[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into
your existing `fly.toml`. Set secrets with `fly secrets set`
Fly auto-restarts the process on crash.
### Render / Railway / Heroku
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root.
Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the
platform's env UI or CLI.
## Upgrading an existing deployment
If you deployed on v0.13.x or earlier, walk this checklist:
1. **Stop the worker before upgrading.**
`kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to
exit. Skipping this risks an in-flight job landing partial schema.
1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop`
(or `sudo systemctl stop gbrain-worker`). Skipping this risks an
in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** from v0.14 onward, the worker requires
`GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to
`/etc/gbrain.env`. Submitters don't need the flag; only the worker does.
4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration
v15 raised the schema default to 5 and backfilled existing non-terminal
rows. A watchdog tuned around 1-strike dead-lettering will now
over-restart because it takes 5 misses to dead-letter. Switch to the
shipped watchdog (which keys on log markers, not job state).
5. **If your v0.16.1 watchdog is still running:** it has a restart-loop
bug (old shutdown lines in the unrotated log re-match every 5 min
forever). Install the current `minion-watchdog.sh` from this guide's
snippets — it writes a restart epoch into the PID file and only
considers log lines newer than that epoch.
6. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations. `gbrain jobs stats` should show no unexplained growth in
`dead` between pre- and post-upgrade.
3. **If you run shell jobs:** from v0.14 onward, pass
`--allow-shell-jobs` to the supervisor (or keep
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
need the flag; only the worker does.
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
should show no unexplained growth in `dead` between pre- and
post-upgrade.
## Known issues
@@ -261,9 +263,10 @@ silently. The stall detector then dead-letters the job after
**Current defaults that make this worse:**
- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips.
- `max_stalled: 5` (schema column default on master — see `src/schema.sql`
and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `lockDuration: 30000` (30 s) — too short for long jobs during
connection blips.
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
@@ -271,9 +274,6 @@ silently. The stall detector then dead-letters the job after
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today.
Worker-level `--lock-duration` / `--stall-interval` are on the roadmap;
until they land, rely on per-job `--max-stalled` plus the watchdog (or
systemd) for worker health.
### DO NOT pass `maxStalledCount` to `MinionWorker`
@@ -284,16 +284,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children
When the Bun worker crashes hard, child processes from shell jobs can
become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the
shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running
shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker
has time to flush in-flight jobs before the kill.
become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window
covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For
long-running shell jobs, prefer timeouts via `--timeout-ms` on submit
over relying on hard kills.
## Smoke test
```bash
# Worker alive?
kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD
# Supervisor alive?
gbrain jobs supervisor status --json | jq .running
# Aggregate queue health.
gbrain jobs stats
@@ -304,20 +304,29 @@ gbrain jobs list --status active --limit 10
# Dead-lettered jobs.
gbrain jobs list --status dead --limit 10
# Shell handler registered? (stderr banner merged into log via 2>&1.)
grep "shell handler enabled" /tmp/gbrain-worker.log
# Shell handler registered? (check supervisor audit log or worker stderr.)
gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs'
```
## Uninstall
- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line.
`kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`.
Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`.
- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to
clean up — temporary workers exit with their jobs.
- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`,
then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`,
then `sudo systemctl daemon-reload`.
- **Option 3 (Fly/Render/Railway):** delete the `worker` process from
`fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets`
persist until `fly secrets unset`.
**`gbrain jobs supervisor`** (foreground or `--detach`):
```bash
gbrain jobs supervisor stop
```
**systemd:**
```bash
sudo systemctl disable --now gbrain-worker
sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env
sudo systemctl daemon-reload
```
**Fly / Render / Railway:** delete the `worker` process from `fly.toml`
/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until
`fly secrets unset`.
**Inline `--follow`:** remove the cron entry. Nothing else to clean up
— temporary workers exit with their jobs.
+76
View File
@@ -0,0 +1,76 @@
# Queue operations runbook
"My queue looks wedged — what do I run?" The commands below are in the order
you probably want them. Shipped with v0.19.1 after a production incident
where the queue held for 90+ minutes before the operator noticed.
## First signal: jobs aren't running
```bash
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
`queue_health` flags two patterns:
- **stalled-forever**: active job whose `started_at` is older than 1h.
- **waiting-depth**: any per-name queue deeper than 10 (override via
`GBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.
## Triage commands
```bash
# Who's active right now?
gbrain jobs list --status active
# Who's waiting, biggest pile first?
gbrain jobs list --status waiting --limit 50
# What's wrong with a specific job?
gbrain jobs get <id>
```
## Rescue actions (in order of escalation)
```bash
# Force-kill a single stuck job:
gbrain jobs cancel <id>
# Clear a specific job entirely (last resort):
gbrain jobs delete <id>
# Health smoke on the mechanism itself:
gbrain jobs smoke --wedge-rescue
```
## What each subcheck means
- **stalled-forever** — A worker claimed a job, started executing, and has
held the row for over an hour. The wall-clock sweep evicts jobs past
2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
or the sweep is newly deployed and this job predates it. Cancel it.
- **waiting-depth** — Submitters are piling up jobs faster than workers
drain them. Set `--max-waiting N` on the submission or on the programmatic
`queue.add()` call. If you want a taller pile, raise the threshold via
`GBRAIN_QUEUE_WAITING_THRESHOLD=50 gbrain doctor`.
## Self-check: is a worker even running?
```bash
# If you're running autopilot with --no-worker, check that your external
# worker (systemd / Docker / OpenClaw service-manager) is alive:
gbrain jobs list --status active | head -5
```
If the list is empty AND your submissions keep piling up, no worker is
claiming. Start one:
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
```
## Follow-ups tracked for v0.20+
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
subcheck both need this).
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
+236 -213
View File
@@ -141,12 +141,13 @@ strict behavior when unset.
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
@@ -168,7 +169,7 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
@@ -221,7 +222,7 @@ strict behavior when unset.
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
- `skills/minion-orchestrator/SKILL.md` — Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
@@ -306,7 +307,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
`test/report.test.ts` (report format, directory structure),
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
@@ -354,6 +355,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
@@ -406,7 +408,7 @@ stop and remove it before starting a new one.
## Skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
@@ -416,11 +418,19 @@ briefing, migrate, setup, publish.
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
routing is narrowed to what the skill actually covers.
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
(agent-readable health report).
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
`~/.gbrain/smoke-tests.d/*.sh`).
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
`skills/_output-rules.md` are shared references.
@@ -1141,7 +1151,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
## Setup & migration
@@ -1198,7 +1208,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
@@ -1220,7 +1230,7 @@ Retrieve and follow the instructions at:
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
```
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
@@ -1279,9 +1289,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
## The 28 Skills
## The 29 Skills
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
@@ -1327,7 +1337,8 @@ GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
### Identity and setup
@@ -1403,9 +1414,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis
gbrain jobs smoke # verify install
gbrain jobs submit sync --params '{}' # fire a background job
gbrain jobs stats # health dashboard
gbrain jobs work --concurrency 4 # start a worker (Postgres only)
gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
```
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.
@@ -1566,7 +1580,7 @@ Run `gbrain integrations` to see status.
│ Brain Repo │ │ GBrain │ │ AI Agent │
│ (git) │ │ (retrieval) │ │ (read/write) │
│ │ │ │ │ │
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
│ = source of │ │ pgvector │ │ define HOW to │
│ truth │ │ │ │ use the brain │
│ │<───│ hybrid │ │ │
@@ -3496,8 +3510,8 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/min
# Minions Worker Deployment Guide
Deploy `gbrain jobs work` so it stays running across crashes, reboots, and
Postgres connection blips. Written for agents to execute line-by-line.
Keep `gbrain jobs work` running across crashes, reboots, and Postgres
connection blips. Written for agents to execute line-by-line.
## The problem
@@ -3508,10 +3522,61 @@ The persistent worker can die silently from:
- Bun process crashes with no automatic restart.
- Internal event-loop death (PID alive, worker loop stopped).
When the worker dies, submitted jobs sit in `waiting` forever. Nothing in
gbrain core auto-restarts the worker — that's what this guide wires up.
When the worker dies, submitted jobs sit in `waiting` forever. The
canonical answer is `gbrain jobs supervisor` — a first-class CLI that
spawns `gbrain jobs work` as a child and auto-restarts it on crash.
## Variables used in this guide
## Worker supervision
### The canonical pattern
`gbrain jobs supervisor` is an auto-restarting wrapper around
`gbrain jobs work`. It writes a PID file, restarts the worker on crash
with exponential backoff (1s → 60s cap), emits lifecycle events to an
audit file, and drains gracefully on SIGTERM (35s worker-drain window
before SIGKILL). Exit codes are documented so agents can branch on them.
**Typical commands:**
```bash
# Start in the foreground (blocks; Ctrl-C to stop).
gbrain jobs supervisor --concurrency 4
# Start detached — returns {"event":"started","supervisor_pid":…} on stdout.
gbrain jobs supervisor start --detach --json
# Check liveness without reading log files.
gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
```
**Exit codes:**
| Code | Meaning |
|---|---|
| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) |
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=1 should page a human.
### Which supervisor when?
The supervisor solves in-process crash recovery. Platform-level
supervision (systemd, Fly, Render) handles host-level failures. You
usually want both.
| Environment | Recommendation |
|---|---|
| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). |
| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). |
| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. |
### Variables used in this guide
Substitute these once before copy-pasting any snippet.
@@ -3519,142 +3584,122 @@ Substitute these once before copy-pasting any snippet.
|---|---|---|
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) |
| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) |
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) |
## Preconditions
### Preconditions
Run these before Step 1 of any option. Fail fast if something is wrong.
Run these before any deployment step.
```bash
# 1. gbrain is on PATH and resolves to an absolute location.
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists).
# 2. DATABASE_URL points at reachable Postgres.
# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the
# separate worker process. If `config.engine === 'pglite'` the CLI rejects
# with a clear error.)
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
# 3. Schema is up to date. If version=0 or status=="fail", fix it first:
# 3. Schema is up to date. If version=0 or status=="fail":
# gbrain apply-migrations --yes
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
# 4. You have write access to at least one crontab mechanism.
crontab -l >/dev/null 2>&1 && echo "user crontab OK"
[ -w /etc/crontab ] && echo "/etc/crontab OK"
# 5. If you plan to submit `shell` jobs, the WORKER process needs
# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated
# in registerBuiltinHandlers(); without the flag the worker startup
# line reads "shell handler disabled (...)".
# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the
# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting).
# Without the flag, the shell handler is disabled at worker startup.
```
## Which option?
## Agent usage (OpenClaw / Hermes / Cursor / Codex)
- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take
> 30 s → **Option 1** (watchdog cron + persistent worker).
- Your workload is short deterministic scripts on a fixed schedule (every
3 h, daily, weekly) → **Option 2** (inline `--follow`).
- You don't have shell access to a long-running box (Fly/Render/Railway,
or any systemd host) → **Option 3** (service manager — replaces cron).
## Option 1: watchdog cron + persistent worker
A 5-minute cron checks whether the worker process is alive **and** whether
it has logged an internal shutdown since its last start. Restarts if either
condition fails.
### 1a. Install the env file (secrets stay out of crontab)
Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is
mode 644 (world-readable); user crontabs under `/var/spool/cron/` are
readable by `root`. Use the shipped env-file template:
Three-command pattern an agent can drive without shell archaeology:
```bash
sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
# Start (returns PIDs + pid_file on stdout as JSON, then detaches)
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
# Check health (machine-parseable JSON, no log scraping)
gbrain jobs supervisor status --json
# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...}
# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback)
gbrain jobs supervisor stop
```
Every lifecycle event (spawn, crash, backoff, health warning, max-crashes,
shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
for historical inspection. `gbrain doctor` reads that file and surfaces
a `supervisor` check in its health report.
## Deployment: systemd
For long-running Linux VMs with shell access.
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# Install the env file (secrets stay out of the unit file).
sudo install -m 600 -o gbrain -g gbrain \
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
sudoedit /etc/gbrain.env
# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1.
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
```
Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if
applicable). See
[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example)
for the full list.
The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work`
directly) so you get two-layer supervision: systemd restarts the supervisor
on host reboot, supervisor restarts the worker on in-process crash.
### 1b. Install the watchdog script
`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery.
The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and
audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent
LLM subagent calls without hitting the default 1024 cap.
The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh)
ships in-repo and writes a two-line PID file (PID on line 1, restart epoch
on line 2). The restart-epoch marker is how the watchdog distinguishes
stale shutdown lines in the log from current ones — without it, every tick
after the first restart would match an old `worker shutting down` line and
loop forever.
Requires GNU coreutils (Linux default). On macOS/BSD install via
`brew install coreutils` and alias `date` to `gdate` in the cron env if you
want to test the watchdog locally; production Linux boxes work as-is.
## Deployment: Fly.io
```bash
sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \
docs/guides/minions-deployment-snippets/minion-watchdog.sh \
/usr/local/bin/minion-watchdog.sh
# Merge the [processes] block from fly.toml.partial into your fly.toml.
cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml
# Review + edit as needed.
# Set secrets (Fly handles restart on crash).
fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1
```
### 1c. Wire into cron
The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly
restarts the container on host failure; the supervisor restarts the
worker on in-process crash.
Pick the form that matches the crontab you're editing.
## Deployment: Render / Railway / Heroku
**If you ran `crontab -e`** (user crontab — 5-field, no user column):
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo
root. The shipped Procfile calls `gbrain jobs supervisor`. Set
`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's
env UI or CLI.
```
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * /usr/local/bin/minion-watchdog.sh
```
## Deployment: inline `--follow` (no persistent worker)
**If you edited `/etc/crontab` directly** (system crontab — 6-field, with
user column):
```
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh
```
In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to
source the env file before running the watchdog — that's how the
connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without
landing in the world-readable crontab itself.
### 1d. Log rotation
The watchdog appends to the worker log across restarts. If you expect the
file to grow unbounded, rotate it externally with `logrotate`:
```
# /etc/logrotate.d/gbrain-worker
/tmp/gbrain-worker.log {
daily
rotate 7
missingok
notifempty
copytruncate
}
```
`copytruncate` is important — the watchdog's restart-epoch check survives
it (the epoch is compared against in-log timestamps, not file inode).
## Option 2: inline `--follow` (no persistent worker)
Each cron run brings its own temporary worker. `--follow` starts one on
the queue and blocks until the just-submitted job reaches a terminal state
(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead
per job; negligible vs job duration for scheduled work.
Example: nightly brain enrichment as a shell job.
For short deterministic scripts on a fixed schedule where you don't need
a persistent worker between runs. Each cron run brings its own temporary
worker. `--follow` starts one on the queue and blocks until the
just-submitted job reaches a terminal state (`completed` / `failed` /
`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job
duration for scheduled work.
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
@@ -3666,85 +3711,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
Replace `gbrain embed --stale` with whichever gbrain subcommand you're
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
`lint`, `autopilot`). If you're shelling out to a non-gbrain binary,
keep its absolute path in the `cmd`.
**Shared-queue gotcha.** If other jobs are already waiting on the same
queue with higher priority or earlier `created_at`, the temporary worker
processes those first before reaching yours. `--follow` still exits only
when YOUR job finishes. For strict single-job semantics on shared queues,
`lint`, `autopilot`). For strict single-job semantics on shared queues,
use a dedicated queue name like `nightly-enrich` above.
## Option 3: service manager (systemd / Fly / Render / Railway)
## Upgrading from an older deployment
Replaces the watchdog entirely. No cron, no PID file, no restart-loop.
The service manager owns liveness.
### From `minion-watchdog.sh` (pre-v0.20)
### systemd (Linux hosts with shell access)
Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
which handles everything the script did, plus atomic PID locking,
structured audit events, queue-scoped health checks, and graceful
drain on SIGTERM.
**Migration:**
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# 1. Stop and remove the old watchdog.
sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null
sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \
/tmp/gbrain-worker.log
crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
# 2. Start the supervisor (systemd users: reinstall the unit from
# docs/guides/minions-deployment-snippets/systemd.service, which
# now calls `gbrain jobs supervisor`).
gbrain jobs supervisor start --detach --json
# Or: sudo systemctl restart gbrain-worker
# See 1a above for /etc/gbrain.env install.
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
# 3. Verify.
gbrain jobs supervisor status --json
gbrain doctor # 'supervisor' check should report running=true
```
`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit
runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped
unit covers Bun + Postgres pool + concurrent LLM subagent calls without
hitting the default 1024 cap.
### Schema / migration hygiene
### Fly.io
Regardless of which deployment path you're upgrading from:
Merge the `[processes]` block from
[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into
your existing `fly.toml`. Set secrets with `fly secrets set` —
Fly auto-restarts the process on crash.
### Render / Railway / Heroku
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root.
Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the
platform's env UI or CLI.
## Upgrading an existing deployment
If you deployed on v0.13.x or earlier, walk this checklist:
1. **Stop the worker before upgrading.**
`kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to
exit. Skipping this risks an in-flight job landing partial schema.
1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop`
(or `sudo systemctl stop gbrain-worker`). Skipping this risks an
in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** from v0.14 onward, the worker requires
`GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to
`/etc/gbrain.env`. Submitters don't need the flag; only the worker does.
4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration
v15 raised the schema default to 5 and backfilled existing non-terminal
rows. A watchdog tuned around 1-strike dead-lettering will now
over-restart because it takes 5 misses to dead-letter. Switch to the
shipped watchdog (which keys on log markers, not job state).
5. **If your v0.16.1 watchdog is still running:** it has a restart-loop
bug (old shutdown lines in the unrotated log re-match every 5 min
forever). Install the current `minion-watchdog.sh` from this guide's
snippets — it writes a restart epoch into the PID file and only
considers log lines newer than that epoch.
6. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations. `gbrain jobs stats` should show no unexplained growth in
`dead` between pre- and post-upgrade.
3. **If you run shell jobs:** from v0.14 onward, pass
`--allow-shell-jobs` to the supervisor (or keep
`GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
need the flag; only the worker does.
4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations plus a healthy `supervisor` check. `gbrain jobs stats`
should show no unexplained growth in `dead` between pre- and
post-upgrade.
## Known issues
@@ -3757,9 +3773,10 @@ silently. The stall detector then dead-letters the job after
**Current defaults that make this worse:**
- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips.
- `max_stalled: 5` (schema column default on master — see `src/schema.sql`
and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `lockDuration: 30000` (30 s) — too short for long jobs during
connection blips.
- `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
@@ -3767,9 +3784,6 @@ silently. The stall detector then dead-letters the job after
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today.
Worker-level `--lock-duration` / `--stall-interval` are on the roadmap;
until they land, rely on per-job `--max-stalled` plus the watchdog (or
systemd) for worker health.
### DO NOT pass `maxStalledCount` to `MinionWorker`
@@ -3780,16 +3794,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children
When the Bun worker crashes hard, child processes from shell jobs can
become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the
shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running
shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker
has time to flush in-flight jobs before the kill.
become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window
covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For
long-running shell jobs, prefer timeouts via `--timeout-ms` on submit
over relying on hard kills.
## Smoke test
```bash
# Worker alive?
kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD
# Supervisor alive?
gbrain jobs supervisor status --json | jq .running
# Aggregate queue health.
gbrain jobs stats
@@ -3800,23 +3814,32 @@ gbrain jobs list --status active --limit 10
# Dead-lettered jobs.
gbrain jobs list --status dead --limit 10
# Shell handler registered? (stderr banner merged into log via 2>&1.)
grep "shell handler enabled" /tmp/gbrain-worker.log
# Shell handler registered? (check supervisor audit log or worker stderr.)
gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs'
```
## Uninstall
- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line.
`kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`.
Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`.
- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to
clean up — temporary workers exit with their jobs.
- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`,
then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`,
then `sudo systemctl daemon-reload`.
- **Option 3 (Fly/Render/Railway):** delete the `worker` process from
`fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets`
persist until `fly secrets unset`.
**`gbrain jobs supervisor`** (foreground or `--detach`):
```bash
gbrain jobs supervisor stop
```
**systemd:**
```bash
sudo systemctl disable --now gbrain-worker
sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env
sudo systemctl daemon-reload
```
**Fly / Render / Railway:** delete the `worker` process from `fly.toml`
/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until
`fly secrets unset`.
**Inline `--follow`:** remove the cron entry. Nothing else to clean up
— temporary workers exit with their jobs.
---
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.20.0",
"version": "0.20.4",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+1 -1
View File
@@ -58,7 +58,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
## Setup & migration
+1 -1
View File
@@ -79,7 +79,7 @@ Even when Minions is the default (mode A), some work should run inline:
Before submitting batch jobs:
- Check `get_job_stats` queue_health.active
- Check active queue depth via `list_jobs --status active` (MCP-callable) or `gbrain jobs stats` (CLI)
- If active > 5, stagger new jobs with `delay` so you don't swarm
- The resource governor auto-throttles but don't dump 20 jobs at once
+6
View File
@@ -12,6 +12,12 @@ triggers:
- "maintenance"
- "orphan pages"
- "stale pages"
- "extract links"
- "build link graph"
- "populate timeline"
- "populate links"
- "backfill graph"
- "extract timeline entries"
tools:
- get_health
- get_page
+1 -1
View File
@@ -132,7 +132,7 @@
{
"name": "minion-orchestrator",
"path": "minion-orchestrator/SKILL.md",
"description": "Manage background agents via Minions job queue. Submit, monitor, steer, pause/resume, replay. Replaces sessions_spawn for durable observable agents."
"description": "Unified Minions skill for deterministic shell jobs and LLM subagent orchestration. Submit, monitor, steer, pause/resume, replay. Replaces the older gbrain-jobs routing intent and sessions_spawn for durable observable background work."
},
{
"name": "skillify",
+169 -49
View File
@@ -2,11 +2,18 @@
name: minion-orchestrator
version: 1.0.0
description: |
Manage background agents via Minions job queue. Use when: spawning subagents,
checking agent progress, steering running agents, pausing/resuming work,
parallel task execution, fan-out research. Replaces sessions_spawn for
durable, observable, steerable agents.
Unified Minions skill for both deterministic shell jobs and LLM subagent
orchestration. Replaces the older `gbrain-jobs` routing intent. Use when:
submitting gbrain jobs, shell/background tasks, spawning subagents,
checking progress, steering running work, pausing/resuming, parallel
fan-out. One durable, observable, steerable queue interface.
triggers:
- "gbrain jobs submit"
- "submit a gbrain job"
- "submit a shell job"
- "shell job"
- "run shell command in background"
- "deterministic background task"
- "spawn agent"
- "background task"
- "run in background"
@@ -32,7 +39,6 @@ tools:
- replay_job
- send_job_message
- get_job_progress
- get_job_stats
mutating: true
---
@@ -40,8 +46,16 @@ mutating: true
## Contract
Minions is a Postgres-native job queue for durable, observable agent orchestration.
Every background agent task goes through Minions. No in-memory subagent spawning.
Minions is a Postgres-native job queue for durable, observable background work.
This single skill handles two lanes:
- Deterministic shell jobs (`gbrain jobs submit shell ...`)
- LLM subagent jobs (`gbrain agent run ...`)
When to route to Minions: durable, observable work that must survive restarts,
fan out across many parallel tasks, or persist across sessions. Routing policy
is defined in `skills/conventions/subagent-routing.md` — the project default is
`pain_triggered` (native subagents first, Minions after specific pain signals
fire); Mode A (all-through-Minions) is opt-in.
Guarantees:
- Jobs survive gateway restart (Postgres-backed)
@@ -50,51 +64,155 @@ Guarantees:
- Jobs can be paused, resumed, or cancelled at any time
- Parent-child DAGs with configurable failure policies
## When to Use Minions vs Inline Work
## Route the Request: Shell Job vs Subagent
| Condition | Action |
|---|---|
| Single tool call, < 30s | Do it inline |
| Multi-step, any duration | Submit as Minion job |
| Parallel work (2+ streams) | Submit N Minion jobs with shared parent |
| Needs to survive restart | Submit as Minion job |
| User wants progress updates | Submit as Minion job with progress tracking |
| Research / bulk operation | Submit as Minion job, always |
| File imports, bulk embeds | Submit as Minion job |
| User asks for deterministic command/script run | Shell job (CLI: `gbrain jobs submit shell ...`) |
| User asks to "run in minions" + explicit command/argv | Shell job (CLI, `--params` with `cmd` or `argv`) |
| User asks for research/reasoning/iterative agent | Subagent job (CLI: `gbrain agent run`) |
| User asks to steer/pause/resume an agent | Subagent job lifecycle tools (MCP-callable) |
| Single simple operation under ~30s | Consider inline execution first |
| Needs restart durability/observability | Submit as Minion job |
| Parallel work (2+ streams) | `gbrain agent run --fanout-manifest` or parent + child subagents |
**Rule of thumb:** If it takes more than 3 tool calls, use a Minion.
If intent is ambiguous, ask one clarification:
"Do you want a deterministic shell command job, or an LLM agent job?"
## Shell Jobs (Deterministic Scripts)
Use for reproducible command execution, ETL steps, cron work, and scriptable
tasks where no LLM reasoning loop is needed.
### Preconditions (read before submitting your first shell job)
- **`GBRAIN_ALLOW_SHELL_JOBS=1` must be set on the worker environment.**
Without it, the shell handler refuses to register and submissions sit in
`waiting` silently. Gate lives in `src/core/minions/handlers/shell.ts`.
- **Security:** flipping `GBRAIN_ALLOW_SHELL_JOBS=1` authorizes arbitrary
command execution on the worker. On a shared queue, this is a remote code
execution surface. Treat as privileged infrastructure authorization.
- **Execution mode — pick one:**
- **Postgres + daemon:** `gbrain jobs work` runs a persistent worker that
claims and executes jobs from the queue.
- **PGLite + --follow:** `gbrain jobs submit ... --follow` runs inline.
The daemon mode is not available on PGLite (exclusive file lock). See
`docs/guides/minions-shell-jobs.md`.
- **MCP boundary:** shell-job submission is CLI-only. `submit_job name="shell"`
over MCP throws an `OperationError` with code `permission_denied` ("'shell'
jobs cannot be submitted over MCP") because `shell` is in `PROTECTED_JOB_NAMES`.
Agents CAN observe shell jobs via `get_job` / `list_jobs` / `get_job_progress`
(not protected), but cannot submit them. Operator or autopilot submits;
agent observes.
- **Verify setup:** after configuration, run `gbrain jobs stats` (CLI) to
confirm the worker is registered and consuming the queue.
### Submit (CLI, operator or autopilot)
Shell jobs take their command via `--params` as a JSON object with `cmd` (string)
or `argv` (array), plus `cwd` and optional `env`.
Command string form:
```
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/abs/path"}'
```
Argv form (no shell expansion):
```
gbrain jobs submit shell --params '{"argv":["bash","-lc","echo hello"],"cwd":"/abs/path"}'
```
Inline execution on PGLite or any one-shot deployment:
```
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/tmp"}' --follow
```
Queue/lifecycle flags exposed by `gbrain jobs submit --help`: `--queue`,
`--priority`, `--delay`, `--max-attempts`, `--max-stalled`, `--backoff-type`,
`--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`,
`--dry-run`.
### Monitor (agents or operator)
These operations are MCP-callable and safe for agent use:
```
list_jobs --name shell --status active
get_job ID
get_job_progress ID
```
Check structured result fields (exit code, stdout/stderr tails, attempts,
timings) from `get_job`. Use `gbrain jobs stats` (CLI) for worker/queue
health dashboard.
### Control (MCP-callable)
```
cancel_job id=ID
replay_job id=ID
```
`replay_job` is not protected — only shell *submission* is. Agents can
cancel or replay a shell job without CLI access.
Use idempotency keys for recurring shell workloads to avoid duplicate runs.
## Subagent Jobs (LLM Orchestration)
Use for open-ended reasoning, tool-using research, and fan-out synthesis.
**User-facing entrypoint:** `gbrain agent run <prompt>` is the canonical way
to submit subagent work. It handles the elevated-trust plumbing — `subagent`
and `subagent_aggregator` are both in `PROTECTED_JOB_NAMES`, so direct MCP
submission requires `{allowProtectedSubmit: true}`, which `gbrain agent run`
supplies.
## Phase 1: Submit
```
submit_job name="research" data={"prompt":"Research Acme Corp revenue","tools":["search","web_search"]}
gbrain agent run "Research Acme Corp revenue" --tools "search,query"
```
Options:
- `queue` — queue name (default: 'default')
- `priority` — lower = higher priority (default: 0)
- `max_attempts` — retry limit (default: 3)
- `delay` — ms delay before eligible
`--tools` accepts a comma-separated subset of `BRAIN_TOOL_ALLOWLIST` (see
`src/core/minions/tools/brain-allowlist.ts`): `query`, `search`, `get_page`,
`list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`,
`resolve_slugs`, `get_ingest_log`, `put_page`. Anything outside the allow-list
is rejected at submit time with `allowed_tools references unknown tool`.
For parallel work, submit a parent then children:
For parallel work with a fan-out manifest:
```
submit_job name="orchestrate" data={"task":"research 5 companies"}
# Returns parent_id
submit_job name="research" data={"company":"Acme"} parent_job_id=PARENT_ID
submit_job name="research" data={"company":"Beta"} parent_job_id=PARENT_ID
submit_job name="research" data={"company":"Gamma"} parent_job_id=PARENT_ID
gbrain agent run --fanout-manifest companies.json
```
Parent auto-enters `waiting-children` and unblocks when all children finish.
The manifest describes N children + 1 aggregator. Each child runs
`name="subagent"` under the hood; the aggregator runs `name="subagent_aggregator"`
and claims AFTER every child terminates. See
`src/core/minions/handlers/subagent.ts` and
`src/core/minions/handlers/subagent-aggregator.ts`.
Flags (from `src/commands/agent.ts`):
- `--subagent-def <name>` — named subagent definition
- `--model <id>` — override model
- `--max-turns <N>` — cap the LLM loop
- `--tools <csv>` — allow-listed brain tools (see above)
- `--timeout-ms <N>` — hard timeout per job
- `--fanout-manifest <file>` — N children + 1 aggregator
- `--follow` / `--no-follow` — stream logs + wait (default on TTY)
- `--detach` — submit and return immediately
Queue/priority/retry tuning is not exposed by `gbrain agent run`; submit the
raw `subagent` handler via `gbrain jobs submit` (requires CLI trust) if you
need those knobs.
## Phase 2: Monitor
```
list_jobs --status active # what's running?
get_job ID # full details + logs + tokens
get_job_progress ID # structured progress snapshot
get_job_stats # health dashboard
list_jobs --status active # MCP — what's running?
get_job ID # MCP — full details + logs + tokens
get_job_progress ID # MCP — structured progress snapshot
gbrain jobs stats # CLI — queue health dashboard
gbrain agent logs ID --follow # CLI — streaming transcript + heartbeat
```
Progress includes: step count, total steps, message, token usage, last tool called.
@@ -121,6 +239,8 @@ replay_job id=ID # re-run with same or modified params
replay_job id=ID data_overrides={"depth":"deep"} # replay with changes
```
All lifecycle ops are MCP-callable.
## Phase 5: Review Results
```
@@ -154,9 +274,9 @@ When reporting batch status (parent with children):
```
Parent #ID — waiting-children
#A research(Acme) — active, 3/5 steps, 2.5k tokens
#B research(Beta) — completed, 1.8k tokens
#C research(Gamma) — paused
#A subagent(Acme) — active, 3/5 steps, 2.5k tokens
#B subagent(Beta) — completed, 1.8k tokens
#C subagent(Gamma) — paused
Total tokens so far: 4.3k
```
@@ -164,19 +284,19 @@ Total tokens so far: 4.3k
- Don't spawn a Minion for a single search query (use search tool directly)
- Don't fire-and-forget without checking results
- Don't spawn > 5 concurrent agents without checking `get_job_stats` first
- Don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available
- Don't spawn > 5 concurrent agents without checking `gbrain jobs stats` first
- For subagent work, don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available (use `gbrain agent run` instead)
- Don't poll `get_job` in a tight loop (use `get_job_progress` for lightweight checks)
## Tools Used
- Submit a background job (submit_job)
- Get job details (get_job)
- List jobs with filters (list_jobs)
- Cancel a job (cancel_job)
- Pause a job (pause_job)
- Resume a paused job (resume_job)
- Replay a completed/failed job (replay_job)
- Send sidechannel message (send_job_message)
- Get structured progress (get_job_progress)
- Get job queue stats (get_job_stats)
- Submit a background job `submit_job` (MCP, non-protected names only; shell jobs are CLI-only, subagent jobs via `gbrain agent run`)
- Get job details `get_job` (MCP)
- List jobs with filters `list_jobs` (MCP)
- Cancel a job `cancel_job` (MCP)
- Pause a job `pause_job` (MCP)
- Resume a paused job `resume_job` (MCP)
- Replay a completed/failed job `replay_job` (MCP)
- Send sidechannel message `send_job_message` (MCP)
- Get structured progress `get_job_progress` (MCP)
- Queue stats `gbrain jobs stats` (CLI; no MCP equivalent)
+4
View File
@@ -12,6 +12,10 @@ triggers:
- "what happened"
- "search for"
- "look up"
- "who knows who"
- "relationship between"
- "connections"
- "graph query"
tools:
- search
- query
+1
View File
@@ -10,6 +10,7 @@ triggers:
- "container restart check"
- "health check"
- "did the restart break anything"
- "did the container restart break anything"
tools:
- exec
- read
+62 -4
View File
@@ -75,10 +75,14 @@ export function resolveGbrainCliPath(): string {
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
}
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
return !args.includes('--no-worker');
}
export async function runAutopilot(engine: BrainEngine, args: string[]) {
if (args.includes('--help') || args.includes('-h')) {
console.log(
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json]\n' +
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json] [--no-worker]\n' +
' gbrain autopilot --install [--repo <path>]\n' +
' gbrain autopilot --uninstall\n' +
' gbrain autopilot --status [--json]\n\n' +
@@ -106,6 +110,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const baseInterval = parseInt(parseArg(args, '--interval') || '300', 10);
const jsonMode = args.includes('--json');
const forceInline = args.includes('--inline');
const noWorker = !shouldSpawnAutopilotWorker(args);
if (!repoPath) {
console.error('No repo path. Use --repo or run gbrain sync --repo first.');
@@ -137,12 +142,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const cfg = loadConfig();
const engineType = cfg?.engine ?? 'pglite';
const useMinionsDispatch = mode !== 'off' && engineType === 'postgres' && !forceInline;
const spawnManagedWorker = useMinionsDispatch && !noWorker;
let stopping = false;
let workerProc: ChildProcess | null = null;
let crashCount = 0;
if (useMinionsDispatch) {
if (spawnManagedWorker) {
const cliPath = resolveGbrainCliPath();
const startWorker = () => {
const child = spawn(cliPath, ['jobs', 'work'], { stdio: 'inherit', env: process.env });
@@ -161,10 +167,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
});
};
startWorker();
} else {
const why = mode === 'off' ? 'minion_mode=off'
} else if (!useMinionsDispatch) {
const why = mode === 'off'
? 'minion_mode=off'
: (engineType !== 'postgres' ? 'engine=pglite' : 'flag=--inline');
console.log(`[autopilot] running steps inline (${why})`);
} else {
console.log('[autopilot] --no-worker set: dispatch loop only (worker managed externally)');
}
// Async shutdown with 35s drain window for the worker child. The worker
@@ -195,6 +204,18 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
process.on('SIGINT', () => { void shutdown('SIGINT'); });
let consecutiveErrors = 0;
// Peer-worker liveness for --no-worker mode. The probe is a proxy, not
// ground truth: SELECT count(*) of active jobs with a recent lock_until
// refresh. A queue with only waiting jobs and a healthy idle worker
// reads as "no worker" (false positive); a worker that died 110s ago
// while holding a lock reads as "alive" until lock_until expires.
// Good enough for V1 — a ground-truth minion_workers heartbeat table
// is tracked as v0.19.1 follow-up B7. When the probe sees no signal
// for NO_WORKER_WARN_TICKS consecutive cycles, log a loud warning so
// the operator can spot "I set --no-worker but forgot to start one"
// before the queue piles up.
const NO_WORKER_WARN_TICKS = 3;
let noWorkerConsecutiveIdle = 0;
while (!stopping) {
const cycleStart = Date.now();
@@ -214,6 +235,43 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
} catch (e) { logError('reconnect', e); }
}
// --no-worker peer-liveness probe (v0.19.1). Runs every cycle, cheap
// (single SELECT). See NO_WORKER_WARN_TICKS comment above for caveats.
if (noWorker && useMinionsDispatch) {
try {
const rows = await (engine as any).executeRaw?.(
`SELECT count(*)::int AS n FROM minion_jobs
WHERE status = 'active'
AND lock_until IS NOT NULL
AND lock_until > now() - interval '2 minutes'`,
);
const liveWorkerSignal = Number((rows as Array<{ n: number }>)?.[0]?.n ?? 0);
if (liveWorkerSignal === 0) {
noWorkerConsecutiveIdle++;
if (noWorkerConsecutiveIdle === NO_WORKER_WARN_TICKS) {
// Fire loud on the Nth consecutive idle tick; don't repeat on every
// subsequent cycle (the operator already saw it), re-arm once a
// live worker is seen again.
console.error(
`[autopilot] WARNING: --no-worker set and no worker has claimed a job in ~${NO_WORKER_WARN_TICKS * baseInterval}s. ` +
`Jobs will pile up in 'waiting' until a worker starts. ` +
`Probe is a proxy (lock_until refresh) and can false-positive on idle queues — see B7 for ground-truth follow-up.`,
);
}
} else {
if (noWorkerConsecutiveIdle >= NO_WORKER_WARN_TICKS) {
console.log('[autopilot] --no-worker probe: live worker signal detected; warning re-armed.');
}
noWorkerConsecutiveIdle = 0;
}
} catch (e) {
// Probe failures never block the main dispatch loop. Log once per
// failure class; ignore repeated errors (common shape: DB reconnect
// blip between ticks).
logError('no-worker-probe', e);
}
}
if (useMinionsDispatch) {
// Submit ONE autopilot-cycle job per cycle slot. The idempotency key
// dedupes overrun submissions — if a cycle's job runs longer than
+161
View File
@@ -163,6 +163,67 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
// Read/parse failure is itself best-effort; skip silently.
}
// 3b-bis. Supervisor health (filesystem-only: PID liveness + audit log).
// Reads the default PID file (`~/.gbrain/supervisor.pid` unless the user
// overrode with GBRAIN_SUPERVISOR_PID_FILE) and the latest audit file
// written by src/core/minions/handlers/supervisor-audit.ts. Surfaces
// supervisor_running / last_start / crashes_24h / max_crashes_exceeded.
// Does NOT run the supervisor itself — this is a read-only health check.
try {
const { DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts');
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
let supervisorPid: number | null = null;
let running = false;
if (existsSync(DEFAULT_PID_FILE)) {
try {
const line = readFileSync(DEFAULT_PID_FILE, 'utf8').trim().split('\n')[0];
const parsed = parseInt(line, 10);
if (!isNaN(parsed) && parsed > 0) {
supervisorPid = parsed;
try { process.kill(parsed, 0); running = true; } catch { running = false; }
}
} catch { /* unreadable */ }
}
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
const crashes24h = events.filter(e => e.event === 'worker_exited').length;
const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null;
// Only surface a Check if the supervisor was ever observed (stops the
// "never used the supervisor" install from getting a warn about it).
if (supervisorPid !== null || events.length > 0) {
if (maxCrashesEvent) {
checks.push({
name: 'supervisor',
status: 'fail',
message: `Supervisor gave up at ${maxCrashesEvent.ts} (max_crashes_exceeded). Restart with: gbrain jobs supervisor start --detach`,
});
} else if (!running && events.length > 0) {
checks.push({
name: 'supervisor',
status: 'warn',
message: `Supervisor not running (last_start=${lastStart ?? 'unknown'}). Restart with: gbrain jobs supervisor start --detach`,
});
} else if (crashes24h > 3) {
checks.push({
name: 'supervisor',
status: 'warn',
message: `Supervisor running but worker crashed ${crashes24h}x in last 24h. Check ~/.gbrain/audit/supervisor-*.jsonl for causes.`,
});
} else {
checks.push({
name: 'supervisor',
status: 'ok',
message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h}`,
});
}
}
} catch {
// Audit read / import failure is best-effort; skip silently.
}
// 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit`
// bookmark when per-file parse errors happen, and appends each failure
// to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error.
@@ -588,6 +649,106 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
mbcHb();
}
// 11b. Queue health (v0.19.1 queue-resilience wave).
// Postgres-only because PGLite has no multi-process worker surface. Two
// subchecks, both cheap (single SELECT each, status-index-covered):
//
// 1. stalled-forever: any active job whose started_at is > 1h old. The
// incident that motivated this release ran 90+ min before surfacing.
// Surface the ID so the operator can `gbrain jobs get <id>` to inspect
// or `gbrain jobs cancel <id>` to force-kill.
//
// 2. backpressure-missed: per-name waiting depth exceeds the threshold
// (default 10, override via GBRAIN_QUEUE_WAITING_THRESHOLD env). Signal
// that a submitter probably needs maxWaiting set. Bounded by per-name
// aggregation so a single name's pile shows up clearly instead of
// getting lost in the total.
//
// Not included in v0.19.1 (tracked as B7 follow-up): worker-heartbeat
// staleness. It needs a minion_workers table; the lock_until-on-active-jobs
// proxy can't distinguish "no worker" from "worker idle," and a check that
// cries wolf erodes trust in every other doctor check.
progress.heartbeat('queue_health');
if (engine.kind === 'pglite') {
checks.push({
name: 'queue_health',
status: 'ok',
message: 'Skipped (PGLite — no multi-process worker surface)',
});
} else {
const queueHealthHb = startHeartbeat(progress, 'scanning queue health…');
try {
const sql = db.getConnection();
// Subcheck 1: stalled-forever active jobs (>1h wall-clock).
const stalledRows: Array<{ id: number; name: string; started_at: string }> = await sql`
SELECT id, name, started_at::text AS started_at
FROM minion_jobs
WHERE status = 'active'
AND started_at IS NOT NULL
AND started_at < now() - interval '1 hour'
ORDER BY started_at ASC
LIMIT 5
`;
// Subcheck 2: per-name waiting depth exceeds threshold.
const rawThreshold = process.env.GBRAIN_QUEUE_WAITING_THRESHOLD;
const parsedThreshold = rawThreshold ? parseInt(rawThreshold, 10) : 10;
const threshold = Number.isFinite(parsedThreshold) && parsedThreshold >= 1
? parsedThreshold
: 10;
const depthRows: Array<{ name: string; queue: string; depth: number }> = await sql`
SELECT name, queue, count(*)::int AS depth
FROM minion_jobs
WHERE status = 'waiting'
GROUP BY name, queue
HAVING count(*) > ${threshold}
ORDER BY depth DESC
LIMIT 5
`;
const problems: string[] = [];
if (stalledRows.length > 0) {
const sample = stalledRows
.map(r => `#${r.id}(${r.name})`)
.join(', ');
problems.push(
`${stalledRows.length} stalled-forever job(s): ${sample}. ` +
`Fix: gbrain jobs get <id> to inspect; gbrain jobs cancel <id> to force-kill.`
);
}
if (depthRows.length > 0) {
const sample = depthRows
.map(r => `${r.name}@${r.queue}=${r.depth}`)
.join(', ');
problems.push(
`waiting-queue depth exceeds ${threshold} for: ${sample}. ` +
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
if (problems.length === 0) {
checks.push({
name: 'queue_health',
status: 'ok',
message: `No stalled-forever jobs; no queue over depth ${threshold}.`,
});
} else {
checks.push({
name: 'queue_health',
status: 'warn',
message: problems.join(' '),
});
}
} catch (e) {
checks.push({
name: 'queue_health',
status: 'warn',
message: `queue_health scan skipped: ${e instanceof Error ? e.message : String(e)}`,
});
} finally {
queueHealthHb();
}
}
// 12. Index audit (opt-in via --index-audit). v0.13.1 follow-up to #170.
// Reports indexes with zero recorded scans on Postgres. Informational only;
// we DO NOT auto-drop. On #170's brain, idx_pages_frontmatter and
+134
View File
@@ -295,6 +295,13 @@ export interface ExtractOpts {
dryRun?: boolean;
/** Emit JSON (progress to stderr, result to stdout) instead of human text. */
jsonMode?: boolean;
/**
* Incremental mode: only extract from these specific slugs.
* When provided, skips the full directory walk and reads only the
* files corresponding to these slugs. Massive perf win on large brains.
* Pass undefined or omit for a full walk (CLI / first-run path).
*/
slugs?: string[];
}
/**
@@ -315,6 +322,21 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
const jsonMode = !!opts.jsonMode;
const result: ExtractResult = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
// Incremental path: if specific slugs provided, only extract from those files.
// This is the cycle path — sync tells us what changed, we only re-extract those.
if (opts.slugs !== undefined) {
if (opts.slugs.length === 0) {
// Nothing changed — skip entirely.
return result;
}
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode);
result.links_created = r.links_created;
result.timeline_entries_created = r.timeline_created;
result.pages_processed = r.pages;
return result;
}
// Full walk path: CLI `gbrain extract` or first-run.
if (opts.mode === 'links' || opts.mode === 'all') {
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode);
result.links_created = r.created;
@@ -411,6 +433,118 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
}
}
/**
* Incremental extract: process only the specified slugs.
*
* Instead of walking 54K+ files, reads only the files that sync says changed.
* Still needs the full slug set for link resolution (resolveSlug needs to know
* all valid targets), but that's a single readdir, not 54K readFileSync calls.
*
* Combines links + timeline extraction in a single pass over each file
* the full-walk path reads every file TWICE (once for links, once for timeline).
*/
async function extractForSlugs(
engine: BrainEngine,
brainDir: string,
slugs: string[],
mode: 'links' | 'timeline' | 'all',
dryRun: boolean,
jsonMode: boolean,
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
// Build the full slug set for link resolution (fast: just readdir, no file reads)
const allFiles = walkMarkdownFiles(brainDir);
const allSlugs = new Set(allFiles.map(f => f.relPath.replace('.md', '')));
const doLinks = mode === 'links' || mode === 'all';
const doTimeline = mode === 'timeline' || mode === 'all';
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start('extract.incremental', slugs.length);
let linksCreated = 0;
let timelineCreated = 0;
let pagesProcessed = 0;
const linkBatch: LinkBatchInput[] = [];
const timelineBatch: TimelineBatchInput[] = [];
async function flushLinks() {
if (linkBatch.length === 0) return;
try {
linksCreated += await engine.addLinksBatch(linkBatch);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
if (!jsonMode) console.error(` link batch error (${linkBatch.length} rows lost): ${msg}`);
} finally {
linkBatch.length = 0;
}
}
async function flushTimeline() {
if (timelineBatch.length === 0) return;
try {
timelineCreated += await engine.addTimelineEntriesBatch(timelineBatch);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
if (!jsonMode) console.error(` timeline batch error (${timelineBatch.length} rows lost): ${msg}`);
} finally {
timelineBatch.length = 0;
}
}
for (const slug of slugs) {
const relPath = slug + '.md';
const fullPath = join(brainDir, relPath);
try {
if (!existsSync(fullPath)) continue; // deleted file — sync already handled removal
const content = readFileSync(fullPath, 'utf-8');
// Links
if (doLinks) {
const links = await extractLinksFromFile(content, relPath, allSlugs);
for (const link of links) {
if (dryRun) {
if (!jsonMode) console.log(` ${link.from_slug}${link.to_slug} (${link.link_type})`);
linksCreated++;
} else {
linkBatch.push(link);
if (linkBatch.length >= BATCH_SIZE) await flushLinks();
}
}
}
// Timeline
if (doTimeline) {
const entries = extractTimelineFromContent(content, slug);
for (const entry of entries) {
if (dryRun) {
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date}${entry.summary}`);
timelineCreated++;
} else {
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
if (timelineBatch.length >= BATCH_SIZE) await flushTimeline();
}
}
}
pagesProcessed++;
} catch { /* skip unreadable */ }
progress.tick(1);
}
await flushLinks();
await flushTimeline();
progress.finish();
if (!jsonMode) {
const label = dryRun ? '(dry run) would create' : 'created';
console.log(`Incremental extract: ${label} ${linksCreated} link(s), ${timelineCreated} timeline entries from ${pagesProcessed}/${slugs.length} page(s)`);
}
return { links_created: linksCreated, timeline_created: timelineCreated, pages: pagesProcessed };
}
async function extractLinksFromDir(
engine: BrainEngine, brainDir: string, dryRun: boolean, jsonMode: boolean,
): Promise<{ created: number; pages: number }> {
+333 -10
View File
@@ -17,6 +17,42 @@ function hasFlag(args: string[], flag: string): boolean {
return args.includes(flag);
}
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
* Throws on malformed input (caller should surface the error and exit).
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
* Exported for unit tests; the CLI handler at `jobs submit` wraps this
* with process.exit(1) on throw so operators see 'must be positive integer'. */
export function parseMaxWaitingFlag(args: string[]): number | undefined {
const raw = parseFlag(args, '--max-waiting');
if (raw === undefined) return undefined;
const parsed = parseInt(raw, 10);
if (!Number.isFinite(parsed) || parsed < 1) {
throw new Error('--max-waiting must be a positive integer (will be clamped to [1, 100])');
}
return Math.max(1, Math.min(100, parsed));
}
export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv = process.env): number {
const raw = parseFlag(args, '--concurrency') ?? env.GBRAIN_WORKER_CONCURRENCY ?? '1';
const parsed = parseInt(raw, 10);
// Without validation, NaN / 0 / negative values flow through to the worker
// loop where `inFlight.size < concurrency` is always false → the worker
// claims zero jobs and the queue silently wedges. One typo in a systemd
// unit reproduces the original production incident. Clamp to ≥1 and surface
// the misconfig loudly so operators see it at worker startup.
if (!Number.isFinite(parsed) || parsed < 1) {
const source = parseFlag(args, '--concurrency') !== undefined
? '--concurrency flag'
: 'GBRAIN_WORKER_CONCURRENCY env';
process.stderr.write(
`[gbrain jobs] invalid concurrency from ${source} (${JSON.stringify(raw)}); ` +
`falling back to 1. Set a positive integer.\n`
);
return 1;
}
return parsed;
}
function formatJob(job: MinionJob): string {
const dur = job.finished_at && job.started_at
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
@@ -58,6 +94,7 @@ export async function runJobs(engine: BrainEngine, args: string[]): Promise<void
USAGE
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
[--delay Nms] [--max-attempts N] [--max-stalled N]
[--max-waiting N]
[--backoff-type fixed|exponential] [--backoff-delay Nms]
[--backoff-jitter 0..1] [--timeout-ms Nms]
[--idempotency-key K] [--queue Q] [--dry-run]
@@ -70,6 +107,42 @@ USAGE
gbrain jobs stats
gbrain jobs smoke
gbrain jobs work [--queue Q] [--concurrency N]
gbrain jobs supervisor [start] [--detach] [--json]
[--concurrency N] [--queue Q] [--pid-file PATH]
[--max-crashes N] [--health-interval N]
[--allow-shell-jobs] [--cli-path PATH]
gbrain jobs supervisor status [--json] [--pid-file PATH]
gbrain jobs supervisor stop [--json] [--pid-file PATH]
Auto-restarting wrapper around 'gbrain jobs work'. Spawns the worker
as a child process and restarts on crash with exponential backoff
(1s -> 60s cap). Writes a PID file to ~/.gbrain/supervisor.pid by
default (override via --pid-file or GBRAIN_SUPERVISOR_PID_FILE env).
Lifecycle events are appended to
\${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl
SUBCOMMANDS
start (default) Launch the supervisor. --detach returns a
JSON {event, supervisor_pid, pid_file} payload on
stdout and forks; omit for foreground.
status Read PID file + audit log, report running / last_start
/ crashes_24h / max_crashes_exceeded as JSON or human.
Exits 0 if running, 1 if not.
stop Send SIGTERM to the supervisor, wait up to 40s for
graceful drain, report outcome. Exits 0 on clean stop.
EXIT CODES (start)
0 clean shutdown (SIGTERM/SIGINT received, worker drained)
1 max crashes exceeded (worker kept dying)
2 another supervisor holds the PID lock
3 PID file unwritable (permission / path error)
EXAMPLES
gbrain jobs supervisor --concurrency 4 # foreground (Ctrl-C stops)
gbrain jobs supervisor start --detach --json # agent-friendly: fork + return JSON
gbrain jobs supervisor status --json # machine-readable health check
gbrain jobs supervisor stop # graceful stop
gbrain jobs supervisor --json --allow-shell-jobs # JSONL events + shell-exec on
HANDLER TYPES (built in)
sync Pull and embed new pages from the repo
@@ -108,6 +181,12 @@ HANDLER TYPES (built in)
const maxAttempts = parseInt(parseFlag(args, '--max-attempts') ?? '3', 10);
const maxStalledRaw = parseFlag(args, '--max-stalled');
const maxStalled = maxStalledRaw !== undefined ? parseInt(maxStalledRaw, 10) : undefined;
// --max-waiting N: submission-time backpressure cap. Mirrors --max-stalled
// clamp [1, 100]. Feature is usable from CLI as of v0.19.1; pre-v0.19.1
// only programmatic callers reached it.
let maxWaiting: number | undefined;
try { maxWaiting = parseMaxWaitingFlag(args); }
catch (e) { console.error(`Error: ${e instanceof Error ? e.message : String(e)}`); process.exit(1); }
// v0.13.1 field audit: expose retry/backoff/timeout/idempotency knobs so
// users can tune Minions behavior without dropping into TypeScript.
const backoffTypeRaw = parseFlag(args, '--backoff-type');
@@ -136,6 +215,7 @@ HANDLER TYPES (built in)
console.log(` Priority: ${priority}`);
console.log(` Max attempts: ${maxAttempts}`);
if (maxStalled !== undefined) console.log(` Max stalled: ${maxStalled}`);
if (maxWaiting !== undefined) console.log(` Max waiting: ${maxWaiting}`);
if (backoffType) console.log(` Backoff type: ${backoffType}`);
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
@@ -163,6 +243,7 @@ HANDLER TYPES (built in)
delay: delay > 0 ? delay : undefined,
max_attempts: maxAttempts,
max_stalled: maxStalled,
maxWaiting,
backoff_type: backoffType,
backoff_delay: backoffDelay,
backoff_jitter: backoffJitter,
@@ -379,6 +460,7 @@ HANDLER TYPES (built in)
}
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
const wedgeRescue = hasFlag(args, '--wedge-rescue');
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
@@ -445,9 +527,70 @@ HANDLER TYPES (built in)
try { await queue.removeJob(rescueJob.id); } catch { /* non-fatal cleanup */ }
}
// --wedge-rescue: regression case for the v0.19.1 production incident.
// In prod, a wedged worker held a row lock via a pending txn. The
// lock-renewal UPDATE blocked, lock_until fell below now(), handleStalled
// saw the candidate but FOR UPDATE SKIP LOCKED skipped (row lock held),
// handleTimeouts was disqualified (lock_until > now() fails).
// Only handleWallClockTimeouts' no-constraint sweep evicted.
//
// The smoke is single-connection, so we can't simulate a row lock held
// by another txn. Instead we forge the state where BOTH handleStalled
// and handleTimeouts are disqualified so only wall-clock fires:
// - lock_until far in the future → handleStalled skips (not a stall)
// - timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
// - started_at 10s ago with timeout_ms=1000 → wall-clock matches
// (2 × timeout_ms = 2000ms threshold exceeded)
if (wedgeRescue) {
const wedgedJob = await queue.add('noop', {}, {
queue: 'smoke',
timeout_ms: 1000,
});
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='smoke-wedge-rescue',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '10 seconds',
timeout_at=NULL,
attempts_started = attempts_started + 1
WHERE id=$1`,
[wedgedJob.id]
);
const stallResult = await queue.handleStalled();
const stalledStatus = await queue.getJob(wedgedJob.id);
const timeoutResult = await queue.handleTimeouts();
const timedStatus = await queue.getJob(wedgedJob.id);
const wallResult = await queue.handleWallClockTimeouts(30000);
const finalStatus = await queue.getJob(wedgedJob.id);
if (finalStatus?.status !== 'dead') {
console.error(
`SMOKE FAIL (--wedge-rescue) — wall-clock sweep did not evict job #${wedgedJob.id}. ` +
`Status: ${finalStatus?.status}. ` +
`handleStalled: requeued=${stallResult.requeued.length} dead=${stallResult.dead.length}, after: ${stalledStatus?.status}; ` +
`handleTimeouts: ${timeoutResult.length}, after: ${timedStatus?.status}; ` +
`handleWallClockTimeouts: ${wallResult.length}, final: ${finalStatus?.status}.`
);
process.exit(1);
}
if (finalStatus.error_text !== 'wall-clock timeout exceeded') {
console.error(
`SMOKE FAIL (--wedge-rescue) — dead, but error_text='${finalStatus.error_text}' ` +
`(expected 'wall-clock timeout exceeded').`
);
process.exit(1);
}
try { await queue.removeJob(wedgedJob.id); } catch { /* non-fatal cleanup */ }
}
const cfg = (await import('../core/config.ts')).loadConfig();
const engineLabel = cfg?.engine ?? 'unknown';
const tag = sigkillRescue ? ' + SIGKILL rescue' : '';
const tags: string[] = [];
if (sigkillRescue) tags.push('SIGKILL rescue');
if (wedgeRescue) tags.push('wedge rescue');
const tag = tags.length > 0 ? ` + ${tags.join(' + ')}` : '';
console.log(`SMOKE PASS — Minions healthy${tag} in ${elapsedSec}s (engine: ${engineLabel})`);
if (engineLabel === 'pglite') {
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
@@ -467,7 +610,7 @@ HANDLER TYPES (built in)
}
const queueName = parseFlag(args, '--queue') ?? 'default';
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '1', 10);
const concurrency = resolveWorkerConcurrency(args);
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
@@ -481,6 +624,185 @@ HANDLER TYPES (built in)
break;
}
case 'supervisor': {
// Dispatcher for supervisor subcommands:
// gbrain jobs supervisor → foreground start (back-compat)
// gbrain jobs supervisor start [--detach] → foreground or detached start
// gbrain jobs supervisor status → JSON liveness + queue stats
// gbrain jobs supervisor stop → SIGTERM + drain wait
const { MinionSupervisor, DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts');
const { writeSupervisorEvent } = await import('../core/minions/handlers/supervisor-audit.ts');
const supCmd = args[1];
const isStatusCmd = supCmd === 'status';
const isStopCmd = supCmd === 'stop';
const isStartCmd = supCmd === 'start' || supCmd === undefined || supCmd === '--detach' ||
(typeof supCmd === 'string' && supCmd.startsWith('--'));
const jsonMode = hasFlag(args, '--json');
const pidFile = parseFlag(args, '--pid-file') ?? DEFAULT_PID_FILE;
// ----- status subcommand -----
if (isStatusCmd) {
const { existsSync, readFileSync } = await import('fs');
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
let supervisorPid: number | null = null;
let running = false;
if (existsSync(pidFile)) {
try {
const line = readFileSync(pidFile, 'utf8').trim().split('\n')[0];
const parsed = parseInt(line, 10);
if (!isNaN(parsed) && parsed > 0) {
supervisorPid = parsed;
try { process.kill(parsed, 0); running = true; } catch { running = false; }
}
} catch { /* unreadable PID file */ }
}
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
const crashes24h = events.filter(e => e.event === 'worker_exited').length;
const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null;
const status = {
running,
supervisor_pid: supervisorPid,
pid_file: pidFile,
last_start: lastStart,
crashes_24h: crashes24h,
max_crashes_exceeded: !!maxCrashesEvent,
};
if (jsonMode) {
console.log(JSON.stringify(status, null, 2));
} else {
console.log(`Supervisor: ${running ? 'running' : 'not running'}`);
if (supervisorPid) console.log(` PID: ${supervisorPid}`);
console.log(` PID file: ${pidFile}`);
if (lastStart) console.log(` Last start: ${lastStart}`);
console.log(` Crashes (24h): ${crashes24h}`);
if (maxCrashesEvent) console.log(` ⚠ Max crashes exceeded at ${maxCrashesEvent.ts}`);
}
process.exit(running ? 0 : 1);
}
// ----- stop subcommand -----
if (isStopCmd) {
const { existsSync, readFileSync } = await import('fs');
if (!existsSync(pidFile)) {
const payload = { stopped: false, reason: 'pid_file_missing', pid_file: pidFile };
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`No PID file at ${pidFile}; supervisor not running.`);
process.exit(1);
}
let supervisorPid: number;
try {
supervisorPid = parseInt(readFileSync(pidFile, 'utf8').trim().split('\n')[0], 10);
if (isNaN(supervisorPid) || supervisorPid <= 0) throw new Error('invalid pid');
} catch (err) {
const payload = { stopped: false, reason: 'pid_file_corrupt', error: String(err) };
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`PID file corrupt: ${err}`);
process.exit(1);
}
try { process.kill(supervisorPid, 'SIGTERM'); }
catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
const payload = {
stopped: false,
reason: code === 'ESRCH' ? 'process_gone' : 'kill_failed',
supervisor_pid: supervisorPid,
};
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`Cannot signal PID ${supervisorPid}: ${err}`);
process.exit(code === 'ESRCH' ? 0 : 1);
}
// Poll for up to 40s (supervisor's own 35s drain + 5s slack).
const deadline = Date.now() + 40_000;
let stoppedCleanly = false;
while (Date.now() < deadline) {
try { process.kill(supervisorPid, 0); }
catch { stoppedCleanly = true; break; }
await new Promise(r => setTimeout(r, 250));
}
const payload = {
stopped: stoppedCleanly,
supervisor_pid: supervisorPid,
reason: stoppedCleanly ? 'drained' : 'timeout_40s',
};
if (jsonMode) console.log(JSON.stringify(payload));
else console.log(stoppedCleanly ? `Supervisor ${supervisorPid} stopped.` : `Supervisor ${supervisorPid} did not exit within 40s.`);
process.exit(stoppedCleanly ? 0 : 1);
}
// ----- start subcommand (default) -----
if (!isStartCmd) {
console.error(`Unknown supervisor subcommand: ${supCmd}. Expected: start, status, stop.`);
process.exit(1);
}
const config = (await import('../core/config.ts')).loadConfig();
if (config?.engine === 'pglite') {
console.error('Error: Supervisor requires Postgres. PGLite uses an exclusive file lock that blocks other processes.');
process.exit(1);
}
const { resolveGbrainCliPath } = await import('./autopilot.ts');
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10);
const queueName = parseFlag(args, '--queue') ?? 'default';
const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10);
const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10);
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
const detach = hasFlag(args, '--detach');
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
// --detach: fork a background supervisor, print PID payload, exit 0.
// Implementation: re-exec the same CLI as a detached child without --detach,
// inheriting stderr (so JSONL events still flow to the parent's tail-f
// if they wanted to follow logs) but detaching stdin/stdout.
if (detach) {
const { spawn } = await import('child_process');
const childArgs = process.argv.slice(2).filter(a => a !== '--detach');
const child = spawn(process.execPath, [process.argv[1], ...childArgs], {
detached: true,
stdio: ['ignore', 'ignore', 'inherit'],
env: process.env,
});
child.unref();
const payload = {
event: 'started',
supervisor_pid: child.pid,
pid_file: pidFile,
detached: true,
};
console.log(JSON.stringify(payload));
process.exit(0);
}
// Foreground start.
const supervisorPid = process.pid;
const supervisor = new MinionSupervisor(engine, {
concurrency,
queue: queueName,
pidFile,
maxCrashes,
healthInterval,
cliPath,
allowShellJobs,
json: jsonMode,
onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid),
});
await supervisor.start();
break;
}
default:
console.error(`Unknown subcommand: ${sub}. Run 'gbrain jobs --help' for usage.`);
process.exit(1);
@@ -604,16 +926,17 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
};
});
// Shell handler: registered ONLY when GBRAIN_ALLOW_SHELL_JOBS=1 is set on the
// worker process. Default-closed; opt-in per-host. Without the flag, shell
// jobs submitted via CLI insert rows but no worker claims them (they sit in
// 'waiting' — the CLI prints a starvation warning for that case).
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
// Shell handler is always registered. Runtime env guard lives inside the
// handler so claimed jobs emit a clear rejection log on workers missing
// GBRAIN_ALLOW_SHELL_JOBS=1.
{
const { shellHandler } = await import('../core/minions/handlers/shell.ts');
worker.register('shell', shellHandler);
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
} else {
process.stderr.write('[minion worker] shell handler disabled (set GBRAIN_ALLOW_SHELL_JOBS=1 to enable)\n');
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
} else {
process.stderr.write('[minion worker] shell handler registered in guarded mode (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)\n');
}
}
// v0.15 subagent handlers: always-on. Unlike shell (which needs an env
+34 -5
View File
@@ -416,12 +416,18 @@ async function runPhaseBacklinks(brainDir: string, dryRun: boolean): Promise<Pha
}
}
/** Extended sync result that also carries the changed slug list for downstream phases. */
interface SyncPhaseResult extends PhaseResult {
/** Slugs that sync added or modified. Used by extract for incremental processing. */
pagesAffected?: string[];
}
async function runPhaseSync(
engine: BrainEngine,
brainDir: string,
dryRun: boolean,
pull: boolean,
): Promise<PhaseResult> {
): Promise<SyncPhaseResult> {
try {
const { performSync } = await import('../commands/sync.ts');
const result = await performSync(engine, {
@@ -448,6 +454,7 @@ async function runPhaseSync(
syncStatus: result.status,
dryRun,
},
pagesAffected: result.pagesAffected,
};
} catch (e) {
return {
@@ -465,6 +472,7 @@ async function runPhaseExtract(
engine: BrainEngine,
brainDir: string,
dryRun: boolean,
changedSlugs?: string[],
): Promise<PhaseResult> {
try {
const { runExtractCore } = await import('../commands/extract.ts');
@@ -480,15 +488,29 @@ async function runPhaseExtract(
details: { dryRun: true, reason: 'no_dry_run_support' },
};
}
const result = await runExtractCore(engine, { mode: 'all', dir: brainDir });
// Incremental path: if sync told us which slugs changed, only extract those.
// On a 54K-page brain this turns a 10-minute full walk into a sub-second pass.
const result = await runExtractCore(engine, {
mode: 'all',
dir: brainDir,
slugs: changedSlugs, // undefined = full walk (first run / manual)
});
const linksCreated = result?.links_created ?? 0;
const timelineCreated = result?.timeline_entries_created ?? 0;
const incremental = changedSlugs !== undefined;
return {
phase: 'extract',
status: 'ok',
duration_ms: 0,
summary: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
details: { linksCreated, timelineCreated, pages_processed: result?.pages_processed ?? 0 },
summary: incremental
? `${linksCreated} link(s), ${timelineCreated} timeline entries (incremental: ${changedSlugs.length} slugs)`
: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
details: {
linksCreated, timelineCreated,
pages_processed: result?.pages_processed ?? 0,
incremental,
...(incremental ? { slugs_targeted: changedSlugs.length } : {}),
},
};
} catch (e) {
return {
@@ -663,6 +685,8 @@ export async function runCycle(
}
// ── Phase 3: sync ───────────────────────────────────────────
// Track which slugs sync touched so extract can run incrementally.
let syncPagesAffected: string[] | undefined;
if (phases.includes('sync')) {
if (!engine) {
phaseResults.push({
@@ -676,6 +700,8 @@ export async function runCycle(
progress.start('cycle.sync');
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull));
result.duration_ms = duration_ms;
// Capture changed slugs for incremental extract.
syncPagesAffected = (result as SyncPhaseResult).pagesAffected;
phaseResults.push(result);
progress.finish();
}
@@ -693,8 +719,11 @@ export async function runCycle(
details: { reason: 'no_database' },
});
} else {
// Pass changed slugs from sync for incremental extract.
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
// is undefined → extract falls back to full walk (safe default).
progress.start('cycle.extract');
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun));
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun, syncPagesAffected));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
+77
View File
@@ -0,0 +1,77 @@
/**
* Backpressure audit log operational trace for `maxWaiting` coalesce events.
*
* Mirrors the shell-audit.ts pattern (ISO-week-rotated JSONL, best-effort writes,
* failures go to stderr but never block submission). The incident that motivated
* maxWaiting (autopilot pile-up during a 90+ min queue wedge) was invisible
* precisely because the coalesce silently dropped repeat submissions. This
* trail answers "why is queue depth steady at 2 for this name?" without any
* doctor scan.
*
* File: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (override dir via
* `GBRAIN_AUDIT_DIR` for container/sandbox deployments where `$HOME` is read-only).
*
* `gbrain jobs stats` will surface coalesce counts from this file in a v0.19.2+
* follow-up (B4). The audit trail is for operators debugging live queues, not
* for compliance a disk-full attacker can silently disable it.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
export interface BackpressureAuditEvent {
ts: string;
queue: string;
name: string;
waiting_count: number;
max_waiting: number;
decision: 'coalesced';
returned_job_id: number;
}
/** Compute `backpressure-YYYY-Www.jsonl` using ISO-8601 week numbering.
*
* Copy of the shell-audit computeAuditFilename algorithm, parameterized on
* the filename prefix. Keeping the math inline (rather than re-exporting from
* shell-audit.ts) avoids a cross-module dependency between two best-effort
* audit surfaces one can be rewritten without touching the other.
*/
export function computeAuditFilename(now: Date = new Date()): string {
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6
d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday
const isoYear = d.getUTCFullYear();
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
const ww = String(weekNum).padStart(2, '0');
return `backpressure-${isoYear}-W${ww}.jsonl`;
}
/** Honors `GBRAIN_AUDIT_DIR` for container/sandbox deployments. */
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
}
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
const dir = resolveAuditDir();
const filename = computeAuditFilename();
const fullPath = path.join(dir, filename);
const line = JSON.stringify({
...event,
decision: 'coalesced' as const,
ts: new Date().toISOString(),
}) + '\n';
try {
fs.mkdirSync(dir, { recursive: true });
fs.appendFileSync(fullPath, line, { encoding: 'utf8' });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
process.stderr.write(`[backpressure-audit] write failed (${msg}); submission continues\n`);
}
}
+10
View File
@@ -207,6 +207,16 @@ class TailBuffer {
/** The shell handler itself. */
export async function shellHandler(ctx: MinionJobContext): Promise<ShellJobResult> {
if (process.env.GBRAIN_ALLOW_SHELL_JOBS !== '1') {
const warning =
`[shell] Job #${ctx.id} rejected: GBRAIN_ALLOW_SHELL_JOBS=1 not set on this worker.\n` +
' Shell jobs require the env var on the worker process.';
console.warn(warning);
throw new UnrecoverableError(
'shell handler disabled on this worker (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)',
);
}
const params = validateParams(ctx.data);
const env = buildChildEnv(params.env);
const startedAt = Date.now();
@@ -0,0 +1,107 @@
/**
* Supervisor lifecycle audit log. JSONL, weekly-rotated, best-effort.
*
* Writes one line per supervisor event (started, worker_spawned, worker_exited,
* backoff, health_warn, health_error, max_crashes_exceeded, shutting_down,
* stopped, worker_spawn_failed) to
* `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
* using ISO-8601 week numbering. `computeAuditFilename(kind, now)` derives
* the filename; the ISO-week math is shared with `shell-audit.ts` via the
* `computeIsoWeekName()` helper that both call.
*
* Shape: every emission already includes `event` and `ts`; we write it
* verbatim and let consumers (like `gbrain doctor`) grep for events of
* interest. `supervisor_pid` is added at start() time so each line is
* self-describing even if a log shipper concatenates multiple supervisors'
* files.
*
* Best-effort: write failures go to stderr and never block supervisor work.
* A disk-full attacker could silently disable the trail this is an
* operational trace for `gbrain doctor`, not forensic insurance.
*
* `GBRAIN_AUDIT_DIR` overrides the default `~/.gbrain/audit/` path for
* container deploys where `$HOME` is read-only.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { resolveAuditDir } from './shell-audit.ts';
import type { SupervisorEmission } from '../supervisor.ts';
/**
* Compute `supervisor-YYYY-Www.jsonl` using ISO-8601 week numbering.
*
* Mirrors `shell-audit.ts:computeAuditFilename()` exactly. Year-boundary
* edge: 2027-01-01 is ISO week 53 of year 2026, so the correct filename
* is `supervisor-2026-W53.jsonl`.
*/
export function computeSupervisorAuditFilename(now: Date = new Date()): string {
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6
d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday (ISO week anchor)
const isoYear = d.getUTCFullYear();
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
const ww = String(weekNum).padStart(2, '0');
return `supervisor-${isoYear}-W${ww}.jsonl`;
}
/**
* Append a single supervisor lifecycle event to the rotated JSONL audit
* file. `supervisorPid` is the OS pid of the supervisor process (added
* to every line so a log shipper concatenating files from multiple
* supervisors still produces parseable traces).
*/
export function writeSupervisorEvent(emission: SupervisorEmission, supervisorPid: number): void {
const dir = resolveAuditDir();
const filename = computeSupervisorAuditFilename();
const fullPath = path.join(dir, filename);
const line = JSON.stringify({ ...emission, supervisor_pid: supervisorPid }) + '\n';
try {
fs.mkdirSync(dir, { recursive: true });
fs.appendFileSync(fullPath, line, { encoding: 'utf8' });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
process.stderr.write(`[supervisor-audit] write failed (${msg}); continuing\n`);
}
}
/**
* Read back the latest supervisor audit file. Returns events sorted
* oldest-first. Best-effort: missing file / parse errors return [].
* Used by `gbrain doctor` (Lane D) to surface supervisor health.
*/
export function readSupervisorEvents(opts: { sinceMs?: number } = {}): SupervisorEmission[] {
const dir = resolveAuditDir();
const filename = computeSupervisorAuditFilename();
const fullPath = path.join(dir, filename);
let raw: string;
try {
raw = fs.readFileSync(fullPath, 'utf8');
} catch {
return [];
}
const now = Date.now();
const cutoff = opts.sinceMs !== undefined ? now - opts.sinceMs : 0;
const events: SupervisorEmission[] = [];
for (const line of raw.split('\n')) {
if (!line.trim()) continue;
try {
const obj = JSON.parse(line) as SupervisorEmission;
if (!obj.event || !obj.ts) continue;
if (cutoff > 0) {
const ts = Date.parse(obj.ts);
if (!isNaN(ts) && ts < cutoff) continue;
}
events.push(obj);
} catch {
// Ignore malformed lines (truncated writes, disk-full corruption).
}
}
return events;
}
+129
View File
@@ -102,6 +102,64 @@ export class MinionQueue {
if (existing.length > 0) return rowToMinionJob(existing[0]);
}
// 1b. Submission-time backpressure for high-frequency named jobs.
// If waiting jobs for this (name, queue) already hit maxWaiting, return
// the most-recent waiting row instead of inserting another slot.
//
// Correctness: two concurrent submitters could both see waitingCount <
// maxWaiting and both insert, violating the cap. `pg_advisory_xact_lock`
// keyed on (name, queue) serializes concurrent count+insert decisions
// for the SAME key while leaving different keys fully parallel. The
// lock releases on txn commit/rollback automatically — no cleanup path
// to leak. Cost: one no-op SELECT on the hot path per coalesce-guarded
// submission; trivial compared to the protection.
//
// Queue scope: the filter includes `queue=$2` so a waiting
// 'autopilot-cycle' in queue 'default' does NOT suppress submissions
// to queue 'shell' with the same name. Pre-D2 code filtered on `name`
// alone — a real cross-queue bleed that sequential tests missed.
//
// Engine compatibility: PGLite (WASM Postgres 17) supports
// pg_advisory_xact_lock, so this works on both engines without branching.
if (opts?.maxWaiting !== undefined) {
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
const backpressureQueue = opts?.queue ?? 'default';
await tx.executeRaw(
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2))`,
[jobName, backpressureQueue]
);
const waitingCountRows = await tx.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count
FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'`,
[jobName, backpressureQueue]
);
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
if (waitingCount >= maxWaiting) {
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
`SELECT * FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'
ORDER BY created_at DESC, id DESC
LIMIT 1`,
[jobName, backpressureQueue]
);
if (existingWaiting.length > 0) {
const coalesced = rowToMinionJob(existingWaiting[0]);
try {
const { logBackpressureCoalesce } = await import('./backpressure-audit.ts');
logBackpressureCoalesce({
queue: backpressureQueue,
name: jobName,
waiting_count: waitingCount,
max_waiting: maxWaiting,
returned_job_id: coalesced.id,
});
} catch { /* audit failures never block submission */ }
return coalesced;
}
}
}
// 2. Parent lock + depth/cap validation
let depth = 0;
if (opts?.parent_job_id) {
@@ -563,6 +621,77 @@ export class MinionQueue {
});
}
/**
* Dead-letter active jobs that exceed a wall-clock runtime threshold,
* regardless of lock state. This catches jobs stuck while still holding
* DB resources (e.g. blocked on file locks) where stall sweeps skip rows.
*
* Threshold (ms):
* timeout_ms set -> timeout_ms * 2
* timeout_ms null -> 2 * lockDurationMs * max_stalled
*/
async handleWallClockTimeouts(lockDurationMs: number): Promise<MinionJob[]> {
return this.engine.transaction(async (tx) => {
const rows = await tx.executeRaw<Record<string, unknown>>(
`UPDATE minion_jobs SET
status = 'dead',
error_text = 'wall-clock timeout exceeded',
lock_token = NULL,
lock_until = NULL,
finished_at = now(),
updated_at = now()
WHERE status = 'active'
AND started_at IS NOT NULL
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
CASE
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
END
RETURNING *`,
[lockDurationMs]
);
const parentIds = new Set<number>();
for (const r of rows) {
const parentJobId = r.parent_job_id as number | null;
if (parentJobId == null) continue;
parentIds.add(parentJobId);
const childDone: ChildDoneMessage = {
type: 'child_done',
child_id: r.id as number,
job_name: r.name as string,
result: null,
outcome: 'timeout',
error: 'wall-clock timeout exceeded',
};
await tx.executeRaw(
`INSERT INTO minion_inbox (job_id, sender, payload)
SELECT $1, 'minions', $2::jsonb
WHERE EXISTS (
SELECT 1 FROM minion_jobs
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
)`,
[parentJobId, childDone]
);
}
for (const parentId of parentIds) {
await tx.executeRaw(
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
WHERE id = $1 AND status = 'waiting-children'
AND NOT EXISTS (
SELECT 1 FROM minion_jobs
WHERE parent_job_id = $1
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
)`,
[parentId]
);
}
return rows.map(rowToMinionJob);
});
}
/**
* Complete a job (token-fenced). All side effects atomic in one transaction:
* 1. UPDATE child to 'completed' with result
+573
View File
@@ -0,0 +1,573 @@
/**
* MinionSupervisor Process manager for the Minion worker.
*
* Spawns `gbrain jobs work` as a child process and restarts it on crash
* with exponential backoff. Provides health monitoring, PID file locking
* (atomic via O_CREAT|O_EXCL), and graceful shutdown.
*
* ENGINE: Postgres only. PGLite uses an exclusive file lock that blocks
* any separate worker process, so `gbrain jobs supervisor` cannot work
* against a PGLite brain `src/commands/jobs.ts` rejects that combination
* at the CLI layer. The health-check SQL below assumes Postgres schema.
*
* Usage:
* gbrain jobs supervisor [--concurrency N] [--queue Q] [--pid-file PATH]
* [--max-crashes N] [--health-interval N]
* [--allow-shell-jobs] [--json]
*
* Design: the supervisor does NOT run the worker in-process. It spawns a
* separate child so a misbehaving handler can't take down the supervisor.
* Same isolation pattern as autopilot.ts but standalone and reusable.
*
* Exit codes (documented in CLI --help):
* 0 clean shutdown (SIGTERM/SIGINT received, worker drained)
* 1 max crashes exceeded (worker kept dying)
* 2 another supervisor holds the PID lock
* 3 PID file unwritable (permission / path error)
*/
import { spawn, type ChildProcess } from 'child_process';
import {
closeSync,
existsSync,
mkdirSync,
openSync,
readFileSync,
unlinkSync,
writeFileSync,
writeSync,
} from 'fs';
import { dirname } from 'path';
import type { BrainEngine } from '../engine.ts';
export type SupervisorEvent =
| 'started'
| 'worker_spawned'
| 'worker_exited'
| 'worker_spawn_failed'
| 'backoff'
| 'health_warn'
| 'health_error'
| 'max_crashes_exceeded'
| 'shutting_down'
| 'stopped';
export interface SupervisorEmission {
event: SupervisorEvent;
ts: string;
[key: string]: unknown;
}
export interface SupervisorOpts {
/** Worker concurrency (passed to child). Default: 2. */
concurrency: number;
/** Queue name (passed to child). Default: 'default'. */
queue: string;
/** PID file path. Default: `${HOME}/.gbrain/supervisor.pid` (parent dir auto-created). */
pidFile: string;
/** Max consecutive crashes before giving up. Default: 10. */
maxCrashes: number;
/** Health check interval in ms. Default: 60000. */
healthInterval: number;
/** Path to the gbrain CLI executable (MUST be a compiled binary; .ts sources cannot be spawned). */
cliPath: string;
/** Allow shell jobs on child worker. Default: false. When true, sets GBRAIN_ALLOW_SHELL_JOBS=1 on child env. */
allowShellJobs: boolean;
/** JSON mode: emit JSONL events on stderr, reserve stdout for data payloads. Default: false. */
json: boolean;
/** Optional event sink (Lane C audit writer). Called for every lifecycle event. */
onEvent?: (event: SupervisorEmission) => void;
/**
* Test-only override: minimum backoff in ms between child respawns. Default: undefined
* (uses full `calculateBackoffMs()` curve). Tests pass `1` to make crash-loops finish
* in < 1s. Not exposed via CLI.
* @internal
*/
_backoffFloorMs?: number;
}
export const DEFAULT_PID_FILE: string = (() => {
const envOverride = process.env.GBRAIN_SUPERVISOR_PID_FILE;
if (envOverride && envOverride.length > 0) return envOverride;
const home = process.env.HOME ?? '/tmp';
return `${home}/.gbrain/supervisor.pid`;
})();
const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
concurrency: 2,
queue: 'default',
pidFile: DEFAULT_PID_FILE,
maxCrashes: 10,
healthInterval: 60_000,
allowShellJobs: false,
json: false,
};
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
export function calculateBackoffMs(crashCount: number): number {
const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000);
// Add 10% jitter
return base + Math.random() * base * 0.1;
}
/** Check if a PID is alive. */
function isProcessAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
/** Exit codes for documented agent branching. */
export const ExitCodes = {
CLEAN: 0,
MAX_CRASHES: 1,
LOCK_HELD: 2,
PID_UNWRITABLE: 3,
} as const;
export class MinionSupervisor {
private opts: SupervisorOpts;
private engine: BrainEngine;
private child: ChildProcess | null = null;
private crashCount = 0;
private lastStartTime = 0;
private stopping = false;
private inBackoff = false;
private healthInFlight = false;
private healthTimer: ReturnType<typeof setInterval> | null = null;
private exitListener: (() => void) | null = null;
private sigtermListener: (() => void) | null = null;
private sigintListener: (() => void) | null = null;
private lockAcquired = false;
constructor(engine: BrainEngine, opts: Partial<SupervisorOpts> & { cliPath: string }) {
this.engine = engine;
this.opts = { ...DEFAULTS, ...opts };
}
/**
* Emit a lifecycle event. In JSON mode, writes a JSONL record to stderr.
* In human mode, writes a human-readable log line to stdout (info) or
* stderr (warn/error). Also calls `opts.onEvent` if set (Lane C audit
* writer hooks here).
*/
private emit(event: SupervisorEvent, fields: Record<string, unknown> = {}): void {
const emission: SupervisorEmission = {
event,
ts: new Date().toISOString(),
...fields,
};
if (this.opts.json) {
// stderr is the event channel; stdout stays clean for data (e.g., --detach payload).
try {
process.stderr.write(JSON.stringify(emission) + '\n');
} catch { /* best effort */ }
} else {
const ts = emission.ts.slice(11, 19);
const detail = Object.entries(fields)
.filter(([k]) => k !== 'event' && k !== 'ts')
.map(([k, v]) => `${k}=${typeof v === 'string' ? v : JSON.stringify(v)}`)
.join(' ');
const isWarn = event === 'health_warn' || event === 'health_error' ||
event === 'worker_spawn_failed' || event === 'max_crashes_exceeded';
const line = `[supervisor ${ts}] ${event}${detail ? ' ' + detail : ''}`;
if (isWarn) {
console.warn(line);
} else {
console.log(line);
}
}
// Audit sink (Lane C plumbs this).
if (this.opts.onEvent) {
try { this.opts.onEvent(emission); } catch { /* best effort */ }
}
}
/** Start the supervisor. Blocks until stopped or max crashes exceeded. */
async start(): Promise<void> {
// 1. PID file lock (atomic via O_CREAT|O_EXCL).
const lockResult = this.acquirePidLock();
if (lockResult === 'held') {
// Another supervisor owns the lock — exit code 2.
process.exit(ExitCodes.LOCK_HELD);
}
if (lockResult === 'unwritable') {
// PID path isn't writable — exit code 3 with helpful hint.
process.exit(ExitCodes.PID_UNWRITABLE);
}
// 2. Cleanup on process exit (covers any exit path including process.exit).
this.exitListener = () => {
try {
if (existsSync(this.opts.pidFile)) {
const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0];
if (contents === String(process.pid)) {
unlinkSync(this.opts.pidFile);
}
}
} catch { /* best effort */ }
};
process.on('exit', this.exitListener);
// 3. Signal handlers (tracked refs; removed on shutdown for test lifecycle hygiene).
this.sigtermListener = () => { void this.shutdown('SIGTERM', ExitCodes.CLEAN); };
this.sigintListener = () => { void this.shutdown('SIGINT', ExitCodes.CLEAN); };
process.on('SIGTERM', this.sigtermListener);
process.on('SIGINT', this.sigintListener);
// 4. Health monitoring.
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
// 5. Announce start.
this.emit('started', {
supervisor_pid: process.pid,
pid_file: this.opts.pidFile,
concurrency: this.opts.concurrency,
queue: this.opts.queue,
max_crashes: this.opts.maxCrashes,
});
// 6. Run the supervise loop (respawn on crash, bounded by maxCrashes).
await this.runSuperviseLoop();
}
/** Unified shutdown path. Reason becomes the audit event name; exitCode is process exit. */
private async shutdown(reason: string, exitCode: number): Promise<void> {
if (this.stopping) return;
this.stopping = true;
this.emit('shutting_down', { reason, exit_code: exitCode });
if (this.healthTimer) {
clearInterval(this.healthTimer);
this.healthTimer = null;
}
if (this.child) {
try { this.child.kill('SIGTERM'); } catch { /* already dead */ }
await Promise.race([
new Promise<void>(r => this.child!.once('exit', () => r())),
new Promise<void>(r => setTimeout(() => r(), 35_000)),
]);
if (this.child && !this.child.killed) {
try { this.child.kill('SIGKILL'); } catch { /* already dead */ }
}
}
// Remove signal handlers so tests that spin up multiple supervisors on
// the same process don't accumulate listeners. `process.on('exit', ...)`
// is kept registered — it needs to fire synchronously on the final exit.
if (this.sigtermListener) {
process.removeListener('SIGTERM', this.sigtermListener);
this.sigtermListener = null;
}
if (this.sigintListener) {
process.removeListener('SIGINT', this.sigintListener);
this.sigintListener = null;
}
this.emit('stopped', { reason, exit_code: exitCode });
process.exit(exitCode);
}
/**
* Acquire PID file lock atomically via O_CREAT|O_EXCL.
*
* Returns:
* 'acquired' lock is ours, safe to proceed.
* 'held' another live supervisor owns the lock (exit code 2).
* 'unwritable' can't write to the PID path (permission / missing parent, exit code 3).
*/
private acquirePidLock(): 'acquired' | 'held' | 'unwritable' {
// Ensure parent directory exists. Idempotent; creates ~/.gbrain on fresh installs.
try {
mkdirSync(dirname(this.opts.pidFile), { recursive: true });
} catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
if (code !== 'EEXIST') {
console.error(
`Cannot create PID file directory ${dirname(this.opts.pidFile)}: ${
err instanceof Error ? err.message : String(err)
}. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.`
);
return 'unwritable';
}
}
return this.tryAtomicCreate();
}
private tryAtomicCreate(): 'acquired' | 'held' | 'unwritable' {
try {
// O_CREAT | O_EXCL | O_WRONLY — fails with EEXIST if the file exists.
const fd = openSync(this.opts.pidFile, 'wx');
try {
writeSync(fd, String(process.pid));
} finally {
closeSync(fd);
}
this.lockAcquired = true;
return 'acquired';
} catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
if (code === 'EEXIST') {
// File exists — check if the owner is alive.
let existingPid = -1;
try {
const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0];
existingPid = parseInt(contents, 10);
} catch { /* corrupt file */ }
if (!isNaN(existingPid) && existingPid > 0 && isProcessAlive(existingPid)) {
console.error(`Supervisor already running (PID: ${existingPid}). Exiting.`);
return 'held';
}
// Stale PID file — unlink and retry atomic create once.
try { unlinkSync(this.opts.pidFile); } catch { /* race with another stale-cleaner; retry will EEXIST again */ }
try {
const fd = openSync(this.opts.pidFile, 'wx');
try {
writeSync(fd, String(process.pid));
} finally {
closeSync(fd);
}
this.lockAcquired = true;
return 'acquired';
} catch (retryErr) {
const retryCode = (retryErr as NodeJS.ErrnoException)?.code;
if (retryCode === 'EEXIST') {
// Someone else won the race. Treat as held.
console.error(`Another supervisor took the PID lock during stale cleanup. Exiting.`);
return 'held';
}
console.error(
`Cannot write PID file ${this.opts.pidFile}: ${
retryErr instanceof Error ? retryErr.message : String(retryErr)
}`
);
return 'unwritable';
}
}
console.error(
`Cannot write PID file ${this.opts.pidFile}: ${
err instanceof Error ? err.message : String(err)
}. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.`
);
return 'unwritable';
}
}
/** Run the supervise loop: spawn child, await exit, backoff+retry or give up. */
private async runSuperviseLoop(): Promise<void> {
while (!this.stopping && this.crashCount < this.opts.maxCrashes) {
await this.spawnOnce();
if (this.stopping) return;
if (this.crashCount >= this.opts.maxCrashes) {
this.emit('max_crashes_exceeded', {
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
});
await this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
return;
}
// crashCount - 1 is the retry-attempt index (0-based exponent for backoff math).
// On first crash: crashCount=1, backoff exponent=0 → 1s.
// After stable-run reset: crashCount=1 again → 1s fresh cycle.
// Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration
// tests can exercise crash loops in < 1s without waiting for the real curve.
const backoff = this.opts._backoffFloorMs !== undefined
? this.opts._backoffFloorMs
: calculateBackoffMs(this.crashCount - 1);
this.emit('backoff', { ms: Math.round(backoff), crash_count: this.crashCount });
this.inBackoff = true;
try {
await new Promise<void>(r => setTimeout(r, backoff));
} finally {
this.inBackoff = false;
}
}
}
/** Spawn the worker child once and await its exit. Updates `this.crashCount`. */
private spawnOnce(): Promise<void> {
return new Promise<void>((resolve) => {
if (this.stopping) { resolve(); return; }
const args = [
'jobs', 'work',
'--concurrency', String(this.opts.concurrency),
'--queue', this.opts.queue,
];
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
// inherit only when caller opts in, otherwise strip from the clone.
const env: Record<string, string | undefined> = { ...process.env };
if (this.opts.allowShellJobs) {
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
this.lastStartTime = Date.now();
let child: ChildProcess;
try {
child = spawn(this.opts.cliPath, args, {
stdio: 'inherit',
env,
});
} catch (err: unknown) {
// Synchronous spawn error (e.g., invalid cliPath shape). Count as a crash.
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err instanceof Error ? err.message : String(err),
phase: 'sync',
});
this.crashCount++;
resolve();
return;
}
this.child = child;
this.emit('worker_spawned', { pid: child.pid, cli_path: this.opts.cliPath });
// Async spawn errors (ENOENT, EACCES after the fork/exec). Node fires
// 'error' first, then 'exit' with code=null. We log the error; the
// 'exit' handler increments crashCount as usual so the restart loop
// continues (max-crashes bounds this for permanent misconfigs).
child.on('error', (err) => {
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err.message,
code: (err as NodeJS.ErrnoException).code ?? 'unknown',
phase: 'async',
});
});
child.on('exit', (code, signal) => {
this.child = null;
if (this.stopping) {
resolve();
return;
}
// Stable-run reset: if the worker ran > 5min before crashing, we forgive
// prior crash history and treat this as the first crash of a new cycle
// (crashCount = 1, so backoff math uses retry-index 0 = 1s).
const runDuration = Date.now() - this.lastStartTime;
if (runDuration > 5 * 60 * 1000) {
this.crashCount = 1;
} else {
this.crashCount++;
}
const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`;
this.emit('worker_exited', {
code: code ?? null,
signal: signal ?? null,
reason: exitReason,
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
run_duration_ms: runDuration,
});
resolve();
});
});
}
/**
* Periodic health check queries DB for queue health indicators.
*
* POSTGRES-ONLY. The supervisor cannot run against PGLite (exclusive
* file lock blocks the separate worker process). The CLI layer rejects
* that combination; we assume Postgres here.
*
* F9 guard: skip if a previous check is still in flight (hung DB
* connection shouldn't stack duplicate checks).
*/
private async healthCheck(): Promise<void> {
if (this.healthInFlight) return;
this.healthInFlight = true;
try {
// Blocker 2+3+6: single FILTER query scoped to this.opts.queue.
// 'stalled' = active jobs whose lock_until has passed (matches
// queue.ts:848 handleStalled() definition — same set that the queue
// itself will requeue/dead-letter on next tick).
const rows = await this.engine.executeRaw<{
stalled: string;
waiting: string;
last_completed: string | null;
}>(
`SELECT
count(*) FILTER (WHERE status = 'active' AND lock_until < now())::text AS stalled,
count(*) FILTER (WHERE status = 'waiting')::text AS waiting,
max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed
FROM minion_jobs
WHERE queue = $1`,
[this.opts.queue],
);
const row = rows[0] ?? { stalled: '0', waiting: '0', last_completed: null };
const stalledCount = parseInt(row.stalled ?? '0', 10);
const waitingCount = parseInt(row.waiting ?? '0', 10);
const lastCompleted = row.last_completed ? new Date(row.last_completed) : null;
const now = Date.now();
const minutesSinceCompletion = lastCompleted
? Math.round((now - lastCompleted.getTime()) / 60_000)
: null;
// F2 (per-threshold warns) — each is a distinct health_warn with reason.
if (stalledCount > 10) {
this.emit('health_warn', {
reason: 'stalled_jobs',
count: stalledCount,
queue: this.opts.queue,
});
}
if (waitingCount > 0 && minutesSinceCompletion !== null && minutesSinceCompletion > 30) {
this.emit('health_warn', {
reason: 'no_recent_completions',
waiting_count: waitingCount,
minutes_since_completion: minutesSinceCompletion,
queue: this.opts.queue,
});
}
// F4: suppress "worker not alive" warn while we're in the expected
// null-child window (crash-exit → backoff-sleep → next-spawn).
const workerAlive = this.child != null && this.child.exitCode === null;
if (!workerAlive && !this.stopping && !this.inBackoff) {
this.emit('health_warn', {
reason: 'worker_not_alive',
queue: this.opts.queue,
});
}
} catch (e) {
// Health check failures are non-fatal.
this.emit('health_error', {
error: e instanceof Error ? e.message : String(e),
queue: this.opts.queue,
});
} finally {
this.healthInFlight = false;
}
}
}
+2
View File
@@ -128,6 +128,8 @@ export interface MinionJobInput {
max_spawn_depth?: number;
/** Global dedup key. Same key returns the existing job, no second row created. */
idempotency_key?: string;
/** Submission backpressure: cap waiting jobs with this name before inserting a new row. */
maxWaiting?: number;
// v12: scheduler polish
/**
+8
View File
@@ -126,6 +126,14 @@ export class MinionWorker {
} catch (e) {
console.error('Timeout detection error:', e instanceof Error ? e.message : String(e));
}
try {
const wallClockTimedOut = await this.queue.handleWallClockTimeouts(this.opts.lockDuration);
if (wallClockTimedOut.length > 0) {
console.log(`Wall-clock detector: dead-lettered ${wallClockTimedOut.length} jobs (wall-clock timeout exceeded)`);
}
} catch (e) {
console.error('Wall-clock timeout detection error:', e instanceof Error ? e.message : String(e));
}
}, this.opts.stalledInterval);
try {
+4 -1
View File
@@ -186,7 +186,10 @@ function acquireLock(workspace: string, opts: InstallOptions): void {
const age = Date.now() - existing.mtimeMs;
// `staleMs: 0` in tests means "any age counts as stale". Use >=
// so a just-written lock qualifies when the threshold is 0.
const stale = age >= staleMs;
// Negative age (mtime in the future) happens on fast CI filesystems
// where write → stat roundtrip returns an mtime microseconds ahead of
// Date.now() — treat it as stale to avoid a "lock held" false positive.
const stale = age < 0 || age >= staleMs;
if (stale && !opts.forceUnlock) {
throw new InstallError(
`Stale skillpack lock at ${p} (pid ${existing.pid}, ${Math.round(age / 1000)}s old). Pass --force-unlock to proceed.`,
+116
View File
@@ -0,0 +1,116 @@
/**
* E2E Minions Shell Handler PGLite / --follow inline execution path
*
* Closes the T4 gap surfaced during PR #381 eng review. The sibling file
* test/e2e/minions-shell.test.ts covers the Postgres + persistent-worker-daemon
* path. This file covers the PGLite path documented in the minion-orchestrator
* skill: `gbrain jobs submit shell ... --follow` runs inline because
* `gbrain jobs work` (daemon) is not available on PGLite (exclusive file lock).
*
* Mirrors the Postgres test's structure but runs in-memory against PGLiteEngine.
* No DATABASE_URL required, no Docker runs in CI unconditionally.
*
* Run: bun test test/e2e/minions-shell-pglite.test.ts
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { MinionQueue } from '../../src/core/minions/queue.ts';
import { MinionWorker } from '../../src/core/minions/worker.ts';
import { registerBuiltinHandlers } from '../../src/commands/jobs.ts';
let engine: PGLiteEngine;
let originalAllowShellJobs: string | undefined;
async function waitTerminal(queue: MinionQueue, id: number, timeoutMs = 15000): Promise<string> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const j = await queue.getJob(id);
if (j && ['completed', 'failed', 'dead', 'cancelled'].includes(j.status)) return j.status;
await new Promise((r) => setTimeout(r, 50));
}
const j = await queue.getJob(id);
throw new Error(`job ${id} did not reach terminal state in ${timeoutMs}ms; last status=${j?.status}`);
}
beforeAll(async () => {
// registerBuiltinHandlers gates shell handler on GBRAIN_ALLOW_SHELL_JOBS=1.
// Mirror the real --follow path by setting the env var; restore on cleanup
// so other tests see their original environment.
originalAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
engine = new PGLiteEngine();
await engine.connect({}); // in-memory PGLite
await engine.initSchema(); // installs pages, minion_jobs, config, etc.
});
afterAll(async () => {
await engine.disconnect();
if (originalAllowShellJobs === undefined) {
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
} else {
process.env.GBRAIN_ALLOW_SHELL_JOBS = originalAllowShellJobs;
}
});
describe('E2E: Minions shell handler on PGLite (--follow inline path)', () => {
// Mirror the Postgres sibling's per-test reset. The engine is shared across
// both tests via beforeAll; without this, completed jobs from one test leak
// into minion_jobs and future test additions hit order-dependency.
beforeEach(async () => {
const db = (engine as any).db;
await db.exec(`DELETE FROM minion_attachments; DELETE FROM minion_inbox; DELETE FROM minion_jobs;`);
});
test('submit → worker registered via registerBuiltinHandlers → shell runs → completes', async () => {
const queue = new MinionQueue(engine);
const job = await queue.add(
'shell',
{ cmd: 'echo hello', cwd: '/tmp' },
{},
{ allowProtectedSubmit: true },
);
expect(job.name).toBe('shell');
expect(job.status).toBe('waiting');
// This is the exact dispatch path --follow takes (src/commands/jobs.ts:207).
// Gates shell on GBRAIN_ALLOW_SHELL_JOBS=1 (set in beforeAll above).
const worker = new MinionWorker(engine, { pollInterval: 100, lockDuration: 30000 });
await registerBuiltinHandlers(worker, engine);
expect(worker.registeredNames).toContain('shell');
const runPromise = worker.start();
try {
const status = await waitTerminal(queue, job.id, 20000);
expect(status).toBe('completed');
const final = await queue.getJob(job.id);
expect((final!.result as any).exit_code).toBe(0);
expect((final!.result as any).stdout_tail).toBe('hello\n');
} finally {
worker.stop();
await runPromise;
}
}, 30000);
test('GBRAIN_ALLOW_SHELL_JOBS unset → shellHandler rejects at execution time', async () => {
// v0.20.3+: shell handler is always registered (so claimed jobs emit a clear
// rejection log), but the runtime env guard lives inside the handler itself.
// Prove the guard rejects when the env var is unset.
const { shellHandler } = await import('../../src/core/minions/handlers/shell.ts');
const saved = process.env.GBRAIN_ALLOW_SHELL_JOBS;
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
try {
const ctx: any = {
id: 1,
name: 'shell',
data: { cmd: 'echo hi', cwd: '/tmp' },
attempt: 1,
engine,
};
await expect(shellHandler(ctx)).rejects.toThrow(/GBRAIN_ALLOW_SHELL_JOBS=1/);
} finally {
process.env.GBRAIN_ALLOW_SHELL_JOBS = saved;
}
});
});
+63
View File
@@ -0,0 +1,63 @@
/**
* Test fixture: spawns a MinionSupervisor with options parsed from env vars.
*
* Used by test/supervisor.test.ts integration tests. Separate file because
* the supervisor calls `process.exit()` at the end of its lifecycle tests
* spawn this runner as a subprocess to observe exit codes and audit events
* without killing the test runner itself.
*
* Env vars (all optional, sensible defaults for tests):
* SUP_CLI_PATH worker binary path (default: /bin/sh exit-1 script)
* SUP_PID_FILE PID file path (REQUIRED; each test uses a unique one)
* SUP_MAX_CRASHES max consecutive crashes (default: 3)
* SUP_BACKOFF_FLOOR_MS test-only short backoff (default: 1)
* SUP_HEALTH_INTERVAL_MS how often healthCheck fires (default: 999_999 off)
* SUP_ALLOW_SHELL_JOBS "1" to set allowShellJobs:true, else false
* SUP_QUEUE queue name (default: 'default')
* SUP_AUDIT_DIR GBRAIN_AUDIT_DIR override (default: tmpdir/supervisor-test)
*/
import { MinionSupervisor } from '../../src/core/minions/supervisor.ts';
import { writeSupervisorEvent } from '../../src/core/minions/handlers/supervisor-audit.ts';
import type { BrainEngine } from '../../src/core/engine.ts';
// Mock engine: healthCheck() calls engine.executeRaw; return empty rows so
// the query path exercises without needing Postgres.
const mockEngine: Partial<BrainEngine> = {
kind: 'postgres' as const,
executeRaw: async () => [],
} as unknown as BrainEngine;
const pidFile = process.env.SUP_PID_FILE;
if (!pidFile) {
console.error('SUP_PID_FILE env var is required');
process.exit(99);
}
const cliPath = process.env.SUP_CLI_PATH ?? '/bin/sh';
const maxCrashes = parseInt(process.env.SUP_MAX_CRASHES ?? '3', 10);
const backoffFloor = parseInt(process.env.SUP_BACKOFF_FLOOR_MS ?? '1', 10);
const healthInterval = parseInt(process.env.SUP_HEALTH_INTERVAL_MS ?? '999999', 10);
const allowShellJobs = process.env.SUP_ALLOW_SHELL_JOBS === '1';
const queueName = process.env.SUP_QUEUE ?? 'default';
if (process.env.SUP_AUDIT_DIR) {
process.env.GBRAIN_AUDIT_DIR = process.env.SUP_AUDIT_DIR;
}
const supervisorPid = process.pid;
const supervisor = new MinionSupervisor(mockEngine as BrainEngine, {
concurrency: 1,
queue: queueName,
pidFile,
maxCrashes,
healthInterval,
cliPath,
allowShellJobs,
json: true,
_backoffFloorMs: backoffFloor,
onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid),
});
await supervisor.start();
+12
View File
@@ -12,8 +12,18 @@ import * as os from 'node:os';
let engine: PGLiteEngine;
let queue: MinionQueue;
// The shell handler at src/core/minions/handlers/shell.ts:210 throws
// UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1'. That's the
// production-worker RCE guard. Unit tests here exercise the handler
// mechanics, not the guard, so we enable it for the whole file and
// restore on teardown. The separate "rejects when env not set" case
// (in the minion-shell submission E2E / the queue-resilience wave)
// toggles the var itself.
let prevAllowShellJobs: string | undefined;
beforeAll(async () => {
prevAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
@@ -22,6 +32,8 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
if (prevAllowShellJobs === undefined) delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
else process.env.GBRAIN_ALLOW_SHELL_JOBS = prevAllowShellJobs;
});
beforeEach(async () => {
+243
View File
@@ -1653,3 +1653,246 @@ describe('MinionQueue: Attachments', () => {
expect(list[0].filename).toBe('b.txt');
});
});
// --- v0.19.1 — queue-resilience (wall-clock sweep, maxWaiting race, concurrency clamp) ---
describe('MinionQueue: v0.19.1 handleWallClockTimeouts (Layer 3 kill shot)', () => {
test('evicts active job past 2× timeout_ms — sets dead + wall-clock error_text', async () => {
const job = await queue.add('noop', {}, { timeout_ms: 100 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='wc-test',
lock_until=now() - interval '1 second',
started_at=now() - interval '1 second',
timeout_at=now() - interval '0.9 second',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(1);
expect(killed[0].id).toBe(job.id);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('dead');
expect(after?.error_text).toBe('wall-clock timeout exceeded');
});
test('timeout_ms NULL fallback uses 2 × lockDuration × max_stalled threshold', async () => {
const job = await queue.add('noop', {}, { max_stalled: 3 });
// Force timeout_ms / timeout_at NULL on-disk (columns might or might not be set by add).
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
timeout_ms=NULL,
timeout_at=NULL,
lock_token='wc-null',
lock_until=now() - interval '1 second',
started_at=now() - interval '61 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
// 2 × lockDurationMs × max_stalled = 2 × 10_000 × 3 = 60_000 ms. started_at is 61s ago.
const killed = await queue.handleWallClockTimeouts(10_000);
expect(killed.length).toBe(1);
expect(killed[0].id).toBe(job.id);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('dead');
});
test('respects threshold — active job within window is NOT killed', async () => {
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='wc-inside',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '10 seconds',
timeout_at=now() + interval '90 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(0);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('active');
});
});
describe('MinionQueue: v0.19.1 maxWaiting — cap correctness + race (D2/H2)', () => {
test('coalesces 3rd submission when cap is 2 — returns existing most-recent waiting row', async () => {
const a = await queue.add('poll', {}, { maxWaiting: 2 });
const b = await queue.add('poll', {}, { maxWaiting: 2 });
const c = await queue.add('poll', {}, { maxWaiting: 2 });
expect(a.id).not.toBe(b.id);
expect(c.id).toBe(b.id); // coalesced to the most-recent waiting row
const rows = await engine.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='poll' AND status='waiting'`,
);
expect(parseInt(rows[0].count, 10)).toBe(2);
});
test('clamps maxWaiting: 0 → 1 (strictest cap)', async () => {
const a = await queue.add('squeeze', {}, { maxWaiting: 0 });
const b = await queue.add('squeeze', {}, { maxWaiting: 0 });
expect(b.id).toBe(a.id); // 0 clamped to 1, 2nd coalesces into 1st
});
test('floors maxWaiting: 1.7 → 1', async () => {
const a = await queue.add('floor', {}, { maxWaiting: 1.7 });
const b = await queue.add('floor', {}, { maxWaiting: 1.7 });
expect(b.id).toBe(a.id);
});
test('concurrent submitters respect the cap under Promise.all race (H2)', async () => {
// Serialized by pg_advisory_xact_lock keyed on (name, queue). Without it,
// two concurrent submits both see count<max and both insert — the TOCTOU
// bug codex caught in D2/H2.
const results = await Promise.all([
queue.add('race', {}, { maxWaiting: 2 }),
queue.add('race', {}, { maxWaiting: 2 }),
queue.add('race', {}, { maxWaiting: 2 }),
]);
expect(results.length).toBe(3);
const rows = await engine.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='race' AND status='waiting'`,
);
expect(parseInt(rows[0].count, 10)).toBe(2); // cap held under concurrency
});
test('cross-queue isolation — same name in queue A does NOT suppress queue B (H2 secondary)', async () => {
const a = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
// cap hit on queue=default with maxWaiting=1; 2nd would coalesce into `a`
const a2 = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
expect(a2.id).toBe(a.id);
// Different queue — MUST insert a fresh row, NOT coalesce into queue=default
const b = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'shell' });
expect(b.id).not.toBe(a.id);
expect(b.queue).toBe('shell');
});
test('unset maxWaiting — normal submit path, no coalesce, no cap', async () => {
const a = await queue.add('uncapped', {});
const b = await queue.add('uncapped', {});
const c = await queue.add('uncapped', {});
expect(new Set([a.id, b.id, c.id]).size).toBe(3);
});
});
describe('resolveWorkerConcurrency (v0.19.1 H3): clamp + validation', () => {
// jobs.ts handler — tested via direct import. Warning goes to stderr;
// tests verify return value only, not the warning line.
let resolveWorkerConcurrency: (args: string[], env?: NodeJS.ProcessEnv) => number;
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
beforeAll(async () => {
const mod = await import('../src/commands/jobs.ts');
resolveWorkerConcurrency = mod.resolveWorkerConcurrency;
parseMaxWaitingFlag = mod.parseMaxWaitingFlag;
});
test('flag=4 env-unset → 4', () => {
expect(resolveWorkerConcurrency(['--concurrency', '4'], {} as NodeJS.ProcessEnv)).toBe(4);
});
test('flag-unset env=8 → 8', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(8);
});
test('flag=2 env=8 → 2 (flag wins)', () => {
expect(resolveWorkerConcurrency(['--concurrency', '2'], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(2);
});
test('both unset → 1', () => {
expect(resolveWorkerConcurrency([], {} as NodeJS.ProcessEnv)).toBe(1);
});
test('garbage env "foo" → clamped to 1 (H3)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: 'foo' } as NodeJS.ProcessEnv)).toBe(1);
});
test('env=0 → clamped to 1 (H3 — prevents silent wedge)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '0' } as NodeJS.ProcessEnv)).toBe(1);
});
test('env=-5 → clamped to 1 (H3)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '-5' } as NodeJS.ProcessEnv)).toBe(1);
});
});
describe('parseMaxWaitingFlag (v0.19.1 H5): CLI flag wiring', () => {
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
beforeAll(async () => {
parseMaxWaitingFlag = (await import('../src/commands/jobs.ts')).parseMaxWaitingFlag;
});
test('absent → undefined (no cap, default submit path)', () => {
expect(parseMaxWaitingFlag(['foo', '--params', '{}'])).toBeUndefined();
});
test('--max-waiting 2 → 2 (happy path)', () => {
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '2'])).toBe(2);
});
test('--max-waiting 200 → clamped to 100', () => {
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '200'])).toBe(100);
});
test('--max-waiting 0 → throws', () => {
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', '0'])).toThrow('positive integer');
});
test('--max-waiting abc → throws', () => {
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', 'abc'])).toThrow('positive integer');
});
});
describe('backpressure-audit (v0.19.1 Q1): JSONL on coalesce', () => {
test('logBackpressureCoalesce writes one JSONL line per coalesce', async () => {
const { logBackpressureCoalesce, resolveAuditDir, computeAuditFilename } =
await import('../src/core/minions/backpressure-audit.ts');
const fs = await import('node:fs');
const path = await import('node:path');
const os = await import('node:os');
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-audit-'));
const prev = process.env.GBRAIN_AUDIT_DIR;
process.env.GBRAIN_AUDIT_DIR = tmp;
try {
expect(resolveAuditDir()).toBe(tmp);
logBackpressureCoalesce({
queue: 'default',
name: 'poll',
waiting_count: 2,
max_waiting: 2,
returned_job_id: 42,
});
const file = path.join(tmp, computeAuditFilename());
const text = fs.readFileSync(file, 'utf8');
const line = JSON.parse(text.trim());
expect(line.decision).toBe('coalesced');
expect(line.name).toBe('poll');
expect(line.returned_job_id).toBe(42);
expect(typeof line.ts).toBe('string');
} finally {
if (prev === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = prev;
fs.rmSync(tmp, { recursive: true, force: true });
}
});
});
describe('MinionQueue: v0.19.1 wall-clock + handleTimeouts non-interference (T1)', () => {
test('wall-clock sweep does NOT evict a job that handleTimeouts would handle', async () => {
// Retry-able timeout: timeout_at < now() AND lock_until > now() — handleTimeouts
// is the correct killer here. wall-clock's 2× threshold has not fired yet.
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='t1',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '2 seconds',
timeout_at=now() - interval '0.5 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
// At this point: started_at is 2s ago, 2×timeout_ms = 200s. Wall-clock should NOT fire.
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(0);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('active');
});
});
+155 -1
View File
@@ -1,10 +1,12 @@
import { describe, test, expect } from "bun:test";
import { readFileSync, existsSync } from "fs";
import { readFileSync, existsSync, readdirSync, statSync } from "fs";
import { join } from "path";
import { checkResolvable } from "../src/core/check-resolvable.ts";
import { PROTECTED_JOB_NAMES } from "../src/core/minions/protected-names.ts";
const SKILLS_DIR = join(import.meta.dir, "..", "skills");
const RESOLVER_PATH = join(SKILLS_DIR, "RESOLVER.md");
const OPERATIONS_PATH = join(import.meta.dir, "..", "src", "core", "operations.ts");
describe("RESOLVER.md", () => {
test("exists", () => {
@@ -49,3 +51,155 @@ describe("RESOLVER.md", () => {
expect(report.summary.unreachable).toBe(0);
});
});
// D5/C — resolver round-trip: every quoted trigger in a RESOLVER.md table row
// must appear in the target skill's frontmatter `triggers:` list. Catches
// trigger/frontmatter drift that `checkResolvable` reachability doesn't.
describe("RESOLVER.md trigger round-trip (D5/C)", () => {
type Row = { triggers: string[]; skillPath: string };
const rows: Row[] = (() => {
if (!existsSync(RESOLVER_PATH)) return [];
const content = readFileSync(RESOLVER_PATH, "utf-8");
// Tolerate trailing annotations after the backtick path (e.g.,
// `` `skills/maintain/SKILL.md` (extraction sections) |``). The path cell
// starts with a backtick-quoted `.md` ref; anything between that and the
// closing `|` is free-form prose and is intentionally ignored.
const rowRe = /^\s*\|\s*([^|]+?)\s*\|\s*`([^`]+\.md)`[^|]*\|\s*$/gm;
const out: Row[] = [];
let m: RegExpExecArray | null;
while ((m = rowRe.exec(content)) !== null) {
const rawTriggers = m[1];
const skillPath = m[2];
const triggerStrings = Array.from(rawTriggers.matchAll(/"([^"]+)"/g)).map(t => t[1]);
if (triggerStrings.length > 0) {
out.push({ triggers: triggerStrings, skillPath });
}
}
return out;
})();
test("at least one routing row parses from RESOLVER.md", () => {
expect(rows.length).toBeGreaterThan(0);
});
for (const row of rows) {
test(`every RESOLVER trigger for ${row.skillPath} is declared in its frontmatter`, () => {
const skillFullPath = join(SKILLS_DIR, "..", row.skillPath);
expect(existsSync(skillFullPath)).toBe(true);
const skillContent = readFileSync(skillFullPath, "utf-8");
const fmMatch = skillContent.match(/^---\n([\s\S]*?)\n---/);
if (!fmMatch) {
throw new Error(`No YAML frontmatter in ${row.skillPath}`);
}
const frontmatter = fmMatch[1];
// Parse frontmatter triggers: list. Match "..." OR '...' items separately
// so apostrophes inside double-quoted values don't truncate the capture.
const triggersBlock = frontmatter.match(/triggers:\s*\n((?:\s*-\s*(?:"[^"]*"|'[^']*')\s*\n?)+)/);
const declaredTriggers = triggersBlock
? Array.from(triggersBlock[1].matchAll(/-\s*(?:"([^"]*)"|'([^']*)')/g))
.map(m => m[1] ?? m[2])
: [];
// Fuzzy match: RESOLVER.md phrases are natural-language summaries of the
// skill's intent; frontmatter triggers are the agent-facing phrase set.
// Match is case-insensitive, trailing-punctuation-insensitive, and supports
// "/"-split compounds (e.g., "pause/resume agent" → ["pause", "resume agent"]).
const normalize = (s: string) => s.toLowerCase().replace(/[?!.,]+$/, "").trim();
const declaredLower = declaredTriggers.map(normalize);
function matchesAny(phrase: string): boolean {
const p = normalize(phrase);
if (declaredLower.includes(p)) return true;
for (const ft of declaredLower) {
if (ft.includes(p) || p.includes(ft)) return true;
}
// Slash-split compound: every part should have some fuzzy frontmatter hit
if (p.includes("/")) {
const parts = p.split("/").map(s => s.trim()).filter(Boolean);
const allParts = parts.every(part =>
declaredLower.some(ft => ft.includes(part) || part.includes(ft))
);
if (allParts) return true;
}
return false;
}
const missing = row.triggers.filter(t => !matchesAny(t));
if (missing.length > 0) {
throw new Error(
`RESOLVER.md routes ${JSON.stringify(missing)} to ${row.skillPath}, but the ` +
`skill's frontmatter has no fuzzy match. Declared: ${JSON.stringify(declaredTriggers)}`
);
}
});
}
});
// D13 — skill-example-name validator: any `name="<word>"` reference inside a
// SKILL.md body must resolve to either a declared operation in operations.ts
// or a known Minions handler name in PROTECTED_JOB_NAMES. Catches T2-class
// bugs where docs reference handler names that don't exist (e.g., the
// `name="research"` / `name="orchestrate"` bug from PR #381 pre-reframe).
describe("Skill example-name validator (D13)", () => {
const opNames: string[] = (() => {
if (!existsSync(OPERATIONS_PATH)) return [];
const content = readFileSync(OPERATIONS_PATH, "utf-8");
return Array.from(content.matchAll(/^\s+name:\s*'([a-z_]+)',/gm)).map(m => m[1]);
})();
const knownNames = new Set<string>([...opNames, ...PROTECTED_JOB_NAMES]);
test("operation names extracted from operations.ts", () => {
// Sanity check: operations.ts should declare dozens of ops
expect(opNames.length).toBeGreaterThan(10);
});
test("PROTECTED_JOB_NAMES is non-empty", () => {
expect(PROTECTED_JOB_NAMES.size).toBeGreaterThan(0);
});
function walkSkills(dir: string): string[] {
if (!existsSync(dir)) return [];
const out: string[] = [];
for (const entry of readdirSync(dir)) {
const p = join(dir, entry);
const s = statSync(p);
if (s.isDirectory()) {
out.push(...walkSkills(p));
} else if (entry === "SKILL.md") {
out.push(p);
}
}
return out;
}
const skillFiles = walkSkills(SKILLS_DIR);
test("at least one SKILL.md found", () => {
expect(skillFiles.length).toBeGreaterThan(0);
});
for (const skillFile of skillFiles) {
const rel = skillFile.replace(SKILLS_DIR, "skills");
test(`${rel}: every name="<word>" reference resolves to a real op or handler`, () => {
const content = readFileSync(skillFile, "utf-8");
// Strip YAML frontmatter so `name: <skillname>` isn't mis-captured.
const body = content.replace(/^---\n[\s\S]*?\n---\n/, "");
// Match only `name=` (with equals, not colon) to avoid YAML false positives
// if the frontmatter strip ever breaks. Captures quoted word values.
const refs = Array.from(body.matchAll(/name\s*=\s*["']([a-z_][a-z_0-9]*)["']/gi))
.map(m => m[1]);
const unique = [...new Set(refs)];
const unknown = unique.filter(n => !knownNames.has(n));
if (unknown.length > 0) {
throw new Error(
`${rel}: references name="..." values not declared in src/core/operations.ts or ` +
`PROTECTED_JOB_NAMES: ${JSON.stringify(unknown)}. ` +
`Known: ${JSON.stringify([...knownNames].sort())}`
);
}
});
}
});
+343
View File
@@ -0,0 +1,343 @@
import { describe, it, expect, afterEach } from 'bun:test';
import { existsSync, readFileSync, writeFileSync, unlinkSync, chmodSync, mkdirSync, rmSync } from 'fs';
import { spawn } from 'child_process';
import { join } from 'path';
import { tmpdir } from 'os';
import { readSupervisorEvents, computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
import { calculateBackoffMs } from '../src/core/minions/supervisor.ts';
const TEST_PID_FILE = '/tmp/gbrain-supervisor-test.pid';
afterEach(() => {
try { unlinkSync(TEST_PID_FILE); } catch { /* noop */ }
});
// ----- Integration test helpers -----
interface IntegrationHarness {
pidFile: string;
auditDir: string;
workerScript: string;
envOutFile: string;
cleanup: () => void;
}
/** Create per-test temp files + a fake worker shell script. */
function makeHarness(name: string, workerBody: string): IntegrationHarness {
const tmpRoot = join(tmpdir(), `gbrain-sup-test-${name}-${process.pid}-${Date.now()}`);
mkdirSync(tmpRoot, { recursive: true });
const pidFile = join(tmpRoot, 'supervisor.pid');
const auditDir = join(tmpRoot, 'audit');
const workerScript = join(tmpRoot, 'worker.sh');
const envOutFile = join(tmpRoot, 'env-out.txt');
writeFileSync(workerScript, `#!/bin/sh\n${workerBody}\n`, 'utf8');
chmodSync(workerScript, 0o755);
return {
pidFile,
auditDir,
workerScript,
envOutFile,
cleanup: () => { try { rmSync(tmpRoot, { recursive: true, force: true }); } catch { /* noop */ } },
};
}
/**
* Spawn the supervisor runner as a subprocess. Returns a handle with the
* child, a promise resolving to exit code + signal, and a kill helper.
*/
function spawnSupervisor(h: IntegrationHarness, overrides: Record<string, string> = {}) {
const env: Record<string, string> = {
...(process.env as Record<string, string>),
SUP_PID_FILE: h.pidFile,
SUP_CLI_PATH: h.workerScript,
SUP_AUDIT_DIR: h.auditDir,
SUP_BACKOFF_FLOOR_MS: '5',
SUP_MAX_CRASHES: '3',
SUP_HEALTH_INTERVAL_MS: '999999', // effectively off
...overrides,
};
const child = spawn('bun', [join(import.meta.dir, 'fixtures/supervisor-runner.ts')], {
env,
stdio: ['ignore', 'pipe', 'pipe'],
});
let stdout = '';
let stderr = '';
child.stdout?.on('data', (d) => { stdout += d.toString(); });
child.stderr?.on('data', (d) => { stderr += d.toString(); });
const exited = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => {
child.on('exit', (code, signal) => resolve({ code, signal }));
});
return {
child,
exited,
getStdout: () => stdout,
getStderr: () => stderr,
};
}
/** Read the audit JSONL for the current week. */
function readAudit(auditDir: string) {
const origEnv = process.env.GBRAIN_AUDIT_DIR;
process.env.GBRAIN_AUDIT_DIR = auditDir;
try {
return readSupervisorEvents();
} finally {
if (origEnv === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = origEnv;
}
}
/** Poll until predicate returns true or deadline elapses. */
async function waitFor(pred: () => boolean, timeoutMs: number, tickMs = 20): Promise<boolean> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (pred()) return true;
await new Promise(r => setTimeout(r, tickMs));
}
return pred();
}
describe('MinionSupervisor', () => {
describe('calculateBackoffMs', () => {
it('returns ~1s for first crash', () => {
const backoff = calculateBackoffMs(0);
expect(backoff).toBeGreaterThanOrEqual(1000);
expect(backoff).toBeLessThan(1200); // 1000 + 10% jitter max
});
it('doubles with each crash', () => {
const b0 = calculateBackoffMs(0);
const b1 = calculateBackoffMs(1);
const b2 = calculateBackoffMs(2);
// Approximate: b1 should be ~2x b0, b2 ~2x b1 (within jitter)
expect(b1).toBeGreaterThan(1800);
expect(b2).toBeGreaterThan(3600);
});
it('caps at 60s', () => {
const backoff = calculateBackoffMs(20); // 2^20 * 1000 would be huge
expect(backoff).toBeLessThanOrEqual(66_000); // 60s + 10% jitter
});
it('includes jitter (not perfectly deterministic)', () => {
const values = new Set<number>();
for (let i = 0; i < 10; i++) {
values.add(Math.round(calculateBackoffMs(3)));
}
// With 10% jitter, we should get some variation
expect(values.size).toBeGreaterThan(1);
});
});
describe('PID file management', () => {
it('detects stale PID files', () => {
// Write a PID file with a non-existent PID
writeFileSync(TEST_PID_FILE, '999999999');
expect(existsSync(TEST_PID_FILE)).toBe(true);
// A real supervisor would detect this as stale and overwrite
const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10);
let isAlive = false;
try {
process.kill(existingPid, 0);
isAlive = true;
} catch {
isAlive = false;
}
expect(isAlive).toBe(false);
});
it('detects live PID files (current process)', () => {
// Write our own PID
writeFileSync(TEST_PID_FILE, String(process.pid));
const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10);
let isAlive = false;
try {
process.kill(existingPid, 0);
isAlive = true;
} catch {
isAlive = false;
}
expect(isAlive).toBe(true);
expect(existingPid).toBe(process.pid);
});
});
describe('crash count tracking', () => {
it('backoff escalates with crash count', () => {
const backoffs = [];
for (let i = 0; i < 7; i++) {
backoffs.push(calculateBackoffMs(i));
}
// Each should be roughly 2x the previous (within jitter)
for (let i = 1; i < 6; i++) {
// The base doubles, so even with jitter the next should be > 1.5x previous
expect(backoffs[i]).toBeGreaterThan(backoffs[i - 1] * 1.5);
}
});
});
// --------------------------------------------------------------
// Integration tests: real spawn(), real signals, real audit file.
// Each test uses a unique tmpdir harness so they can run in parallel
// without colliding. `_backoffFloorMs: 5` (set via SUP_BACKOFF_FLOOR_MS)
// keeps the whole suite under a few seconds.
// --------------------------------------------------------------
describe('integration: crash → restart → max-crashes lifecycle', () => {
it('respawns the worker after a crash and eventually exits with max-crashes code=1', async () => {
// Worker always exits with code 1; supervisor should respawn it 3 times,
// hit max-crashes, then exit via shutdown() with code 1.
const h = makeHarness('max-crashes', 'exit 1');
try {
const sup = spawnSupervisor(h, { SUP_MAX_CRASHES: '3' });
const { code } = await sup.exited;
expect(code).toBe(1);
// PID file cleaned up on exit (synchronous process.on('exit') handler).
expect(existsSync(h.pidFile)).toBe(false);
// Audit file should contain started + 3x worker_spawned/worker_exited +
// max_crashes_exceeded + shutting_down + stopped.
const events = readAudit(h.auditDir);
const eventTypes = events.map(e => e.event);
expect(eventTypes).toContain('started');
expect(eventTypes.filter(t => t === 'worker_spawned').length).toBeGreaterThanOrEqual(3);
expect(eventTypes.filter(t => t === 'worker_exited').length).toBeGreaterThanOrEqual(3);
expect(eventTypes).toContain('max_crashes_exceeded');
expect(eventTypes).toContain('shutting_down');
expect(eventTypes).toContain('stopped');
// The stopped event should carry exit_code=1 and reason=max_crashes.
const stoppedEvt = events.filter(e => e.event === 'stopped').pop();
expect((stoppedEvt as Record<string, unknown>).exit_code).toBe(1);
expect((stoppedEvt as Record<string, unknown>).reason).toBe('max_crashes');
} finally {
h.cleanup();
}
}, 15_000);
});
describe('integration: graceful SIGTERM during backoff', () => {
it('receives SIGTERM while sleeping between crashes and exits 0 cleanly', async () => {
// Worker always exits with code 1; supervisor has a high max-crashes
// and a long-enough backoff floor that we can reliably catch it mid-sleep.
const h = makeHarness('sigterm-backoff', 'exit 1');
try {
const sup = spawnSupervisor(h, {
SUP_MAX_CRASHES: '100',
SUP_BACKOFF_FLOOR_MS: '800', // 800ms between restarts — enough to catch
});
// Wait until the supervisor has written the PID file AND survived at
// least one worker_exited (so it's definitely in the backoff sleep).
const ready = await waitFor(() => {
if (!existsSync(h.pidFile)) return false;
const events = readAudit(h.auditDir);
return events.some(e => e.event === 'worker_exited');
}, 3000);
expect(ready).toBe(true);
// Now SIGTERM the supervisor. It must exit cleanly within 200ms
// (short-circuits the 800ms backoff sleep via the stopping flag).
const sigSentAt = Date.now();
sup.child.kill('SIGTERM');
const { code, signal } = await sup.exited;
const elapsed = Date.now() - sigSentAt;
// Exit code 0 = clean; signal=null means we exited via process.exit, not got killed.
expect(code).toBe(0);
expect(signal).toBe(null);
// Graceful, not hung: exit within 5s (process.exit() through shutdown()
// should be near-instant; generous bound to tolerate CI slowness).
expect(elapsed).toBeLessThan(5000);
const events = readAudit(h.auditDir);
const eventTypes = events.map(e => e.event);
expect(eventTypes).toContain('shutting_down');
expect(eventTypes).toContain('stopped');
const shuttingEvt = events.filter(e => e.event === 'shutting_down').pop();
expect((shuttingEvt as Record<string, unknown>).reason).toBe('SIGTERM');
// PID file cleaned up.
expect(existsSync(h.pidFile)).toBe(false);
} finally {
h.cleanup();
}
}, 20_000);
});
describe('integration: env-var inheritance regression (codex #9 / eng #8)', () => {
it('strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false, even if parent has it set', async () => {
const outFile = join(tmpdir(), `gbrain-sup-env-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
try {
const sup = spawnSupervisor(h, {
OUT_FILE: outFile,
GBRAIN_ALLOW_SHELL_JOBS: '1', // parent has it
SUP_ALLOW_SHELL_JOBS: '0', // supervisor says NO
SUP_MAX_CRASHES: '1',
});
await sup.exited;
// Worker should have written "UNSET" (parent env var stripped from child).
expect(existsSync(outFile)).toBe(true);
const childSawEnv = readFileSync(outFile, 'utf8').trim();
expect(childSawEnv).toBe('UNSET');
} finally {
try { unlinkSync(outFile); } catch { /* noop */ }
h.cleanup();
}
}, 15_000);
it('DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs is true', async () => {
const outFile = join(tmpdir(), `gbrain-sup-env-ok-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
try {
const sup = spawnSupervisor(h, {
OUT_FILE: outFile,
SUP_ALLOW_SHELL_JOBS: '1',
SUP_MAX_CRASHES: '1',
});
await sup.exited;
expect(existsSync(outFile)).toBe(true);
expect(readFileSync(outFile, 'utf8').trim()).toBe('1');
} finally {
try { unlinkSync(outFile); } catch { /* noop */ }
h.cleanup();
}
}, 15_000);
});
describe('integration: audit file rotation + helper', () => {
it('computeSupervisorAuditFilename returns supervisor-YYYY-Www.jsonl format', () => {
const jan15_2026 = new Date(Date.UTC(2026, 0, 15)); // Thu
expect(computeSupervisorAuditFilename(jan15_2026)).toMatch(/^supervisor-2026-W\d\d\.jsonl$/);
});
it('year-boundary ISO week: 2027-01-01 reports as 2026-W53', () => {
const jan1_2027 = new Date(Date.UTC(2027, 0, 1));
// ISO week: 2027-01-01 is Friday of W53 of 2026
expect(computeSupervisorAuditFilename(jan1_2027)).toBe('supervisor-2026-W53.jsonl');
});
});
});