diff --git a/CHANGELOG.md b/CHANGELOG.md index 559c33f45..e9b4ff2e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,117 @@ All notable changes to GBrain will be documented in this file. +## [0.20.2] - 2026-04-24 + +## **`gbrain jobs supervisor` is now a self-healing daemon you can actually drive. The Minions worker stops dying silently.** +## **Three commands an agent can run: `start --detach`, `status --json`, `stop`. Crash loops are bounded, audit events are JSONL, and the health check finally reports real data.** + +`gbrain jobs work` has always been the worker that drains your Minions queue. Problem: it dies (OOM, connection blip, panic) and nobody notices until jobs pile up. The old answer was `nohup` plus a 68-line bash watchdog script from the deployment guide, and it shipped its own bugs (restart-loop traps, log-parsing stall detection, zero audit trail). + +v0.20.2 ships the replacement: `gbrain jobs supervisor` is a first-class CLI with atomic PID locking, exponential backoff, structured audit events at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`, and three subcommands that make it drivable by an OpenClaw or Hermes agent in three turns. The old bash watchdog is gone. + +### The numbers that matter + +Before v0.20.2, an agent driving the supervisor needed ~10 turns of shell archaeology (PID file scraping, `pgrep -f`, `kill -0`, log grep) just to start and stop the worker reliably. After v0.20.2, it's three commands with machine-parseable output. + +| Capability | Before v0.20.2 | After v0.20.2 | +|---|---|---| +| Keeping the worker alive | `nohup` + `minion-watchdog.sh` (68 lines of bash, restart-loop bug, log-scrape health) | `gbrain jobs supervisor` (first-class CLI with atomic PID lock, exponential backoff, JSONL audit) | +| PID file locking | `existsSync + readFileSync + writeFileSync` TOCTOU race | Atomic `O_CREAT|O_EXCL` via `openSync('wx')` — kernel-atomic mutex | +| Stalled-jobs health alert | Queried `status='stalled'` — returned 0 rows forever (dead code) | Queries `status='active' AND lock_until < now()`, scoped to the supervised queue | +| Shell-exec env inheritance | Child inherited `GBRAIN_ALLOW_SHELL_JOBS=1` from parent shell regardless of CLI flag | Explicit `else delete env.GBRAIN_ALLOW_SHELL_JOBS` when not opted in + regression test | +| Agent discovery TTHW | ~10 turns of shell-scraping (cat PID / pgrep / kill -0 / log grep) | 3 turns: `start --detach` → `status --json` → `stop` | +| Lifecycle observability | `console.log` with human prefixes, zero audit trail | JSONL events on stderr + `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` + `gbrain doctor` integration | +| Exit codes | undocumented; agent couldn't distinguish "already running" from "gave up" | Four documented codes: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable | +| Test coverage of the supervisor itself | ~15% (backoff math + PID helpers only) | Integration tests covering crash-restart, max-crashes drain, SIGTERM-during-backoff, env-inheritance regression | + +The supervisor's own reliability claims are now testable. Every lifecycle event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`) lands in a weekly-rotated JSONL file that `gbrain doctor` reads to surface a `supervisor` health check. + +### What this means for your deployment + +If you were using the old `nohup`/`minion-watchdog.sh` pattern: + +1. **Stop the old watchdog:** `sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && crontab -e` and delete the watchdog cron line. +2. **Delete the script:** `sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid /tmp/gbrain-worker.log`. +3. **Start the supervisor:** `gbrain jobs supervisor start --detach --json` — or on systemd, reinstall the unit (now calls `gbrain jobs supervisor`). +4. **Verify:** `gbrain doctor` reports a `supervisor` check; `gbrain jobs supervisor status --json` returns `running:true`. + +For containers (Fly / Railway / Render / Heroku): the shipped `Procfile` and `fly.toml.partial` now call `gbrain jobs supervisor`. The platform restarts the container on host events, the supervisor restarts the worker on in-process crashes. Two-layer supervision with clean separation. + +For OpenClaw / Hermes / Cursor agents driving the supervisor: you no longer need a shell skill to drive the worker. Every piece of state — liveness, crash history, max-crashes exhaustion — is a machine-parseable JSON response. Start with `gbrain jobs supervisor status --json | jq`. + +## To take advantage of v0.20.2 + +`gbrain upgrade` pulls the binary. Nothing else is required if you're currently running `gbrain jobs work` directly or using systemd — the new supervisor is opt-in. To migrate: + +1. **Verify the binary:** + ```bash + gbrain --version # should say 0.20.2 + gbrain jobs supervisor --help | head -20 + ``` +2. **Start the supervisor (detached, agent-friendly):** + ```bash + gbrain jobs supervisor start --detach --json + # → {"event":"started","supervisor_pid":1234,"pid_file":"/Users/you/.gbrain/supervisor.pid","detached":true} + ``` +3. **Check health:** + ```bash + gbrain jobs supervisor status --json + gbrain doctor | grep supervisor + ``` +4. **Stop when done:** + ```bash + gbrain jobs supervisor stop + ``` +5. **(Optional) Migrate off the old watchdog:** see `docs/guides/minions-deployment.md` "Upgrading from an older deployment" for the cron-to-supervisor migration. + +If `gbrain jobs supervisor status` reports `running:false` unexpectedly, or `gbrain doctor` flags a `supervisor` failure, file an issue at https://github.com/garrytan/gbrain/issues with: +- output of `gbrain doctor` +- the last ~50 lines of `~/.gbrain/audit/supervisor-*.jsonl` +- which step broke + +### Itemized changes + +**`gbrain jobs supervisor`:** +- New subcommands: `start [--detach] [--json]`, `status [--json]`, `stop [--json]`. Foreground use is unchanged (back-compat). +- New flags: `--allow-shell-jobs` (explicit opt-in, replaces env-var sniffing), `--cli-path PATH` (override auto-resolution), `--json` (JSONL lifecycle events on stderr), `GBRAIN_SUPERVISOR_PID_FILE` env var (overrides default PID path). +- Exit codes documented in `--help`: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable. +- Default PID path moved from `/tmp/gbrain-supervisor.pid` to `~/.gbrain/supervisor.pid` with automatic parent-directory creation. + +**Safety fixes (codex adversarial review + eng review):** +- Atomic PID lock via `openSync(path, 'wx')` — two supervisors starting simultaneously can no longer both win the race. +- `stalled` health check query rewritten from unreachable `status='stalled'` to `status='active' AND lock_until < now()` matching `queue.ts:848 handleStalled()`. +- Health queries now scoped to `WHERE queue = $1` — multi-queue deployments see the right queue. +- Unified exit path via `shutdown(reason, exitCode)` — max-crashes drains gracefully instead of bypassing cleanup via `process.exit(1)`. +- Listener ref tracking: `SIGTERM`/`SIGINT` handlers removed on shutdown for clean test lifecycle. + +**Security hardening:** +- `allowShellJobs` class default flipped `true` → `false`. +- Child env now has `GBRAIN_ALLOW_SHELL_JOBS` explicitly deleted when `allowShellJobs:false` (was: silently inherited from parent shell). +- Integration regression test locks this against future refactors. + +**Observability:** +- New `src/core/minions/handlers/supervisor-audit.ts` with ISO-week rotation (mirrors `shell-audit.ts` / `subagent-audit.ts` pattern). +- Every supervisor emission (started, worker_spawned, worker_exited, worker_spawn_failed, backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, stopped) written to `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`. +- `gbrain doctor` gains a `supervisor` check that reads the audit file and reports `running` / `last_start` / `crashes_24h` / `max_crashes_exceeded` with thresholds (ok / warn at 3+ crashes / fail on max-crashes event). + +**Documentation:** +- `docs/guides/minions-deployment.md` rewritten: supervisor is the canonical answer; which-supervisor-when decision table (container / systemd / dev laptop); three-command agent pattern; migration block from the old watchdog. +- `README.md` Operations section gains a paragraph on `gbrain jobs supervisor`. +- `docs/guides/minions-deployment-snippets/{systemd.service,Procfile,fly.toml.partial}` now invoke `gbrain jobs supervisor` instead of raw `gbrain jobs work`. +- `docs/guides/minions-deployment-snippets/minion-watchdog.sh` deleted — subsumed by the supervisor. + +**Tests:** +- `test/supervisor.test.ts`: 7 → 13 tests. Four new integration tests exercise real `spawn()` lifecycles via shell-script fakes (crash-restart happy path, max-crashes-via-shutdown with audit assertions, SIGTERM-during-backoff clean exit, `GBRAIN_ALLOW_SHELL_JOBS` inheritance regression — positive + negative). +- `test/fixtures/supervisor-runner.ts`: new standalone runner that constructs a supervisor from env vars so integration tests can observe `process.exit` without killing the test runner. + +**For contributors:** +- The `MinionSupervisor` class has a test-only `_backoffFloorMs` override for fast crash-loop tests. Not exposed via CLI. +- `onEvent: (emission) => void` is an injectable hook on `SupervisorOpts` — Lane C's audit writer uses it; future observability integrations can too. +- `autopilot.ts` migration to `MinionSupervisor` is explicitly deferred (follow-up PR): the current `start()` API blocks, which deadlocks autopilot's interval loop. Codex's review flagged this; the fix is a non-blocking-start API redesign, not a drop-in substitution. + +Credit: original supervisor feature built by OpenClaw (PR #364 initial commit). Review wave + code-level fixes + daemon-manager CLI + observability boomerang + integration tests shipped via /autoplan (CEO + DX + Eng + Codex adversarial) followed by a 20-item multi-lane implementation plan. + ## [0.20.0] - 2026-04-23 ## **BrainBench moves out. gbrain gets its install surface back.** diff --git a/README.md b/README.md index 047cd9cf1..bdeb75569 100644 --- a/README.md +++ b/README.md @@ -211,9 +211,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis gbrain jobs smoke # verify install gbrain jobs submit sync --params '{}' # fire a background job gbrain jobs stats # health dashboard -gbrain jobs work --concurrency 4 # start a worker (Postgres only) +gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only) +gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`) ``` +`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md). + Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox. **Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now. diff --git a/VERSION b/VERSION index 5a03fb737..727d97b9b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.20.0 +0.20.2 diff --git a/docs/guides/minions-deployment-snippets/Procfile b/docs/guides/minions-deployment-snippets/Procfile index d76f93d37..972e8c198 100644 --- a/docs/guides/minions-deployment-snippets/Procfile +++ b/docs/guides/minions-deployment-snippets/Procfile @@ -7,4 +7,7 @@ # DATABASE_URL=postgresql://... # GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs -worker: gbrain jobs work --concurrency 2 +# Two-layer supervision: the platform restarts the container on host +# events (OOM, deploy); `gbrain jobs supervisor` restarts the worker +# on in-process crashes with exponential backoff. +worker: gbrain jobs supervisor --concurrency 2 diff --git a/docs/guides/minions-deployment-snippets/fly.toml.partial b/docs/guides/minions-deployment-snippets/fly.toml.partial index ee65b533d..0f796092b 100644 --- a/docs/guides/minions-deployment-snippets/fly.toml.partial +++ b/docs/guides/minions-deployment-snippets/fly.toml.partial @@ -5,10 +5,12 @@ # fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs # fly secrets set ANTHROPIC_API_KEY=... # optional # -# Fly.io auto-restarts the process on crash — no watchdog needed. +# Two-layer supervision: Fly restarts the VM on host events; the +# `gbrain jobs supervisor` process restarts the worker on in-process +# crashes with exponential backoff and a structured audit trail. [processes] - worker = "gbrain jobs work --concurrency 2" + worker = "gbrain jobs supervisor --concurrency 2" # Scale the worker process to 1 machine (job queue serializes work; more # machines means higher concurrency but also more Postgres connections). diff --git a/docs/guides/minions-deployment-snippets/minion-watchdog.sh b/docs/guides/minions-deployment-snippets/minion-watchdog.sh deleted file mode 100755 index ac504e208..000000000 --- a/docs/guides/minions-deployment-snippets/minion-watchdog.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/bin/bash -# minion-watchdog.sh — restart gbrain jobs work if the process is dead or -# has logged a shutdown marker since its last start. -# -# Fixes the v0.16.1 restart-loop bug: old shutdown lines from previous -# restarts stayed in the unrotated log and every tick re-matched them -# forever. This version writes a restart epoch to line 2 of the PID file -# and only considers log lines newer than that epoch. -# -# Run every 5 minutes from crontab. See docs/guides/minions-deployment.md. -set -u - -PID_FILE="${GBRAIN_WORKER_PID_FILE:-/tmp/gbrain-worker.pid}" -LOG_FILE="${GBRAIN_WORKER_LOG_FILE:-/tmp/gbrain-worker.log}" -GBRAIN="${GBRAIN_BIN:-/usr/local/bin/gbrain}" -CONCURRENCY="${GBRAIN_WORKER_CONCURRENCY:-2}" - -start_worker() { - # stderr merged so banner lines ("[minion worker] shell handler enabled", - # "worker shutting down") all land in $LOG_FILE. - nohup "$GBRAIN" jobs work --concurrency "$CONCURRENCY" \ - > "$LOG_FILE" 2>&1 & - local pid=$! - # Line 1: PID. Line 2: restart epoch (seconds since 1970). - # Readers that want just PID use `head -n1 "$PID_FILE"`. - printf '%s\n%s\n' "$pid" "$(date +%s)" > "$PID_FILE" -} - -shutdown_since_restart() { - # Only match shutdown lines logged AFTER the most recent restart epoch. - # Worker log lines start with ISO-8601 UTC timestamps ("2026-04-21T19:05:12Z ..."). - local restart_epoch - restart_epoch=$(sed -n '2p' "$PID_FILE" 2>/dev/null || echo 0) - [ -z "$restart_epoch" ] && restart_epoch=0 - - # POSIX-portable regex (no {n} intervals — mawk on Debian/Ubuntu rejects them). - awk -v since="$restart_epoch" ' - match($0, /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9:.+Z-]+/) { - ts_str = substr($0, RSTART, RLENGTH) - cmd = "date -d \"" ts_str "\" +%s 2>/dev/null" - cmd | getline ts - close(cmd) - if (ts + 0 > since + 0) print - } - ' "$LOG_FILE" 2>/dev/null | grep -q "worker stopped\|worker shutting down" -} - -if [ -f "$PID_FILE" ]; then - PID=$(head -n1 "$PID_FILE") - if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then - # Process alive — check whether the worker logged an internal shutdown - # AFTER the last start. If yes, worker is dead-inside; restart. - if shutdown_since_restart; then - kill "$PID" 2>/dev/null - # 10s grace: covers shell handler's 5s child SIGTERM→SIGKILL window - # and leaves room for in-flight jobs to flush. Bump to 30 if your - # jobs run > 10s. - sleep 10 - kill -9 "$PID" 2>/dev/null - start_worker - fi - else - # PID file exists but process is gone (crash / kill -9 / reboot). - start_worker - fi -else - start_worker -fi diff --git a/docs/guides/minions-deployment-snippets/systemd.service b/docs/guides/minions-deployment-snippets/systemd.service index 591c39753..bb6282a2a 100644 --- a/docs/guides/minions-deployment-snippets/systemd.service +++ b/docs/guides/minions-deployment-snippets/systemd.service @@ -15,9 +15,13 @@ WorkingDirectory=/srv/gbrain # Env file is mode 600, owned by User=. Do not put secrets in this unit. EnvironmentFile=/etc/gbrain.env -ExecStart=/usr/local/bin/gbrain jobs work --concurrency 2 +# Two-layer supervision: systemd restarts `gbrain jobs supervisor` on host +# events (reboot, unit crash); the supervisor restarts `gbrain jobs work` +# on in-process crashes with exponential backoff + structured audit. +ExecStart=/usr/local/bin/gbrain jobs supervisor --concurrency 2 -# Replaces the cron watchdog. systemd restarts on any non-zero exit. +# systemd restarts the supervisor on any non-zero exit. The supervisor +# itself handles worker-level crash recovery. Restart=always RestartSec=10s @@ -38,7 +42,9 @@ NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=read-only -ReadWritePaths=/srv/gbrain +# ReadWritePaths must include the brain workspace AND ~/.gbrain (PID file + +# audit log written by the supervisor). +ReadWritePaths=/srv/gbrain /home/gbrain/.gbrain [Install] WantedBy=multi-user.target diff --git a/docs/guides/minions-deployment.md b/docs/guides/minions-deployment.md index 1b9e9d92f..f6bd20d98 100644 --- a/docs/guides/minions-deployment.md +++ b/docs/guides/minions-deployment.md @@ -1,7 +1,7 @@ # Minions Worker Deployment Guide -Deploy `gbrain jobs work` so it stays running across crashes, reboots, and -Postgres connection blips. Written for agents to execute line-by-line. +Keep `gbrain jobs work` running across crashes, reboots, and Postgres +connection blips. Written for agents to execute line-by-line. ## The problem @@ -12,10 +12,61 @@ The persistent worker can die silently from: - Bun process crashes with no automatic restart. - Internal event-loop death (PID alive, worker loop stopped). -When the worker dies, submitted jobs sit in `waiting` forever. Nothing in -gbrain core auto-restarts the worker — that's what this guide wires up. +When the worker dies, submitted jobs sit in `waiting` forever. The +canonical answer is `gbrain jobs supervisor` — a first-class CLI that +spawns `gbrain jobs work` as a child and auto-restarts it on crash. -## Variables used in this guide +## Worker supervision + +### The canonical pattern + +`gbrain jobs supervisor` is an auto-restarting wrapper around +`gbrain jobs work`. It writes a PID file, restarts the worker on crash +with exponential backoff (1s → 60s cap), emits lifecycle events to an +audit file, and drains gracefully on SIGTERM (35s worker-drain window +before SIGKILL). Exit codes are documented so agents can branch on them. + +**Typical commands:** + +```bash +# Start in the foreground (blocks; Ctrl-C to stop). +gbrain jobs supervisor --concurrency 4 + +# Start detached — returns {"event":"started","supervisor_pid":…} on stdout. +gbrain jobs supervisor start --detach --json + +# Check liveness without reading log files. +gbrain jobs supervisor status --json + +# Graceful stop (SIGTERM + drain wait + SIGKILL fallback). +gbrain jobs supervisor stop +``` + +**Exit codes:** + +| Code | Meaning | +|---|---| +| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) | +| 1 | Max crashes exceeded (worker kept dying) | +| 2 | Another supervisor holds the PID lock | +| 3 | PID file unwritable (permission / path error) | + +An agent seeing exit=2 can safely treat it as "one is already running"; +exit=1 should page a human. + +### Which supervisor when? + +The supervisor solves in-process crash recovery. Platform-level +supervision (systemd, Fly, Render) handles host-level failures. You +usually want both. + +| Environment | Recommendation | +|---|---| +| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). | +| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). | +| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. | + +### Variables used in this guide Substitute these once before copy-pasting any snippet. @@ -23,142 +74,122 @@ Substitute these once before copy-pasting any snippet. |---|---|---| | `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` | | `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` | -| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) | -| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) | | `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` | -| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) | +| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) | -## Preconditions +### Preconditions -Run these before Step 1 of any option. Fail fast if something is wrong. +Run these before any deployment step. ```bash # 1. gbrain is on PATH and resolves to an absolute location. command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; } -# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists). +# 2. DATABASE_URL points at reachable Postgres. +# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the +# separate worker process. If `config.engine === 'pglite'` the CLI rejects +# with a clear error.) gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")' -# 3. Schema is up to date. If version=0 or status=="fail", fix it first: +# 3. Schema is up to date. If version=0 or status=="fail": # gbrain apply-migrations --yes gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")' -# 4. You have write access to at least one crontab mechanism. -crontab -l >/dev/null 2>&1 && echo "user crontab OK" -[ -w /etc/crontab ] && echo "/etc/crontab OK" - -# 5. If you plan to submit `shell` jobs, the WORKER process needs -# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated -# in registerBuiltinHandlers(); without the flag the worker startup -# line reads "shell handler disabled (...)". +# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the +# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting). +# Without the flag, the shell handler is disabled at worker startup. ``` -## Which option? +## Agent usage (OpenClaw / Hermes / Cursor / Codex) -- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take - > 30 s → **Option 1** (watchdog cron + persistent worker). -- Your workload is short deterministic scripts on a fixed schedule (every - 3 h, daily, weekly) → **Option 2** (inline `--follow`). -- You don't have shell access to a long-running box (Fly/Render/Railway, - or any systemd host) → **Option 3** (service manager — replaces cron). - -## Option 1: watchdog cron + persistent worker - -A 5-minute cron checks whether the worker process is alive **and** whether -it has logged an internal shutdown since its last start. Restarts if either -condition fails. - -### 1a. Install the env file (secrets stay out of crontab) - -Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is -mode 644 (world-readable); user crontabs under `/var/spool/cron/` are -readable by `root`. Use the shipped env-file template: +Three-command pattern an agent can drive without shell archaeology: ```bash -sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ +# Start (returns PIDs + pid_file on stdout as JSON, then detaches) +gbrain jobs supervisor start --detach --json +# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"} + +# Check health (machine-parseable JSON, no log scraping) +gbrain jobs supervisor status --json +# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...} + +# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback) +gbrain jobs supervisor stop +``` + +Every lifecycle event (spawn, crash, backoff, health warning, max-crashes, +shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl` +for historical inspection. `gbrain doctor` reads that file and surfaces +a `supervisor` check in its health report. + +## Deployment: systemd + +For long-running Linux VMs with shell access. + +```bash +# Create the worker user if it doesn't exist. +sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ + 2>/dev/null || true +sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" + +# Install the env file (secrets stay out of the unit file). +sudo install -m 600 -o gbrain -g gbrain \ docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env sudoedit /etc/gbrain.env +# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1. + +# Install the unit file, substituting /srv/gbrain → your workspace path. +sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ + /etc/systemd/system/gbrain-worker.service +sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ + /etc/systemd/system/gbrain-worker.service + +sudo systemctl daemon-reload +sudo systemctl enable --now gbrain-worker +sudo systemctl status gbrain-worker +journalctl -u gbrain-worker -n 50 ``` -Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if -applicable). See -[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example) -for the full list. +The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work` +directly) so you get two-layer supervision: systemd restarts the supervisor +on host reboot, supervisor restarts the worker on in-process crash. -### 1b. Install the watchdog script +`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery. +The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`, +and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and +audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent +LLM subagent calls without hitting the default 1024 cap. -The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh) -ships in-repo and writes a two-line PID file (PID on line 1, restart epoch -on line 2). The restart-epoch marker is how the watchdog distinguishes -stale shutdown lines in the log from current ones — without it, every tick -after the first restart would match an old `worker shutting down` line and -loop forever. - -Requires GNU coreutils (Linux default). On macOS/BSD install via -`brew install coreutils` and alias `date` to `gdate` in the cron env if you -want to test the watchdog locally; production Linux boxes work as-is. +## Deployment: Fly.io ```bash -sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ - docs/guides/minions-deployment-snippets/minion-watchdog.sh \ - /usr/local/bin/minion-watchdog.sh +# Merge the [processes] block from fly.toml.partial into your fly.toml. +cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml +# Review + edit as needed. + +# Set secrets (Fly handles restart on crash). +fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1 ``` -### 1c. Wire into cron +The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly +restarts the container on host failure; the supervisor restarts the +worker on in-process crash. -Pick the form that matches the crontab you're editing. +## Deployment: Render / Railway / Heroku -**If you ran `crontab -e`** (user crontab — 5-field, no user column): +Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo +root. The shipped Procfile calls `gbrain jobs supervisor`. Set +`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's +env UI or CLI. -``` -SHELL=/bin/bash -PATH=/usr/local/bin:/usr/bin:/bin -BASH_ENV=/etc/gbrain.env -*/5 * * * * /usr/local/bin/minion-watchdog.sh -``` +## Deployment: inline `--follow` (no persistent worker) -**If you edited `/etc/crontab` directly** (system crontab — 6-field, with -user column): - -``` -SHELL=/bin/bash -PATH=/usr/local/bin:/usr/bin:/bin -BASH_ENV=/etc/gbrain.env -*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh -``` - -In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to -source the env file before running the watchdog — that's how the -connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without -landing in the world-readable crontab itself. - -### 1d. Log rotation - -The watchdog appends to the worker log across restarts. If you expect the -file to grow unbounded, rotate it externally with `logrotate`: - -``` -# /etc/logrotate.d/gbrain-worker -/tmp/gbrain-worker.log { - daily - rotate 7 - missingok - notifempty - copytruncate -} -``` - -`copytruncate` is important — the watchdog's restart-epoch check survives -it (the epoch is compared against in-log timestamps, not file inode). - -## Option 2: inline `--follow` (no persistent worker) - -Each cron run brings its own temporary worker. `--follow` starts one on -the queue and blocks until the just-submitted job reaches a terminal state -(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead -per job; negligible vs job duration for scheduled work. - -Example: nightly brain enrichment as a shell job. +For short deterministic scripts on a fixed schedule where you don't need +a persistent worker between runs. Each cron run brings its own temporary +worker. `--follow` starts one on the queue and blocks until the +just-submitted job reaches a terminal state (`completed` / `failed` / +`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job +duration for scheduled work. ```bash GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ @@ -170,85 +201,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ Replace `gbrain embed --stale` with whichever gbrain subcommand you're scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`, -`lint`, `autopilot`). If you're shelling out to a non-gbrain binary, -keep its absolute path in the `cmd`. - -**Shared-queue gotcha.** If other jobs are already waiting on the same -queue with higher priority or earlier `created_at`, the temporary worker -processes those first before reaching yours. `--follow` still exits only -when YOUR job finishes. For strict single-job semantics on shared queues, +`lint`, `autopilot`). For strict single-job semantics on shared queues, use a dedicated queue name like `nightly-enrich` above. -## Option 3: service manager (systemd / Fly / Render / Railway) +## Upgrading from an older deployment -Replaces the watchdog entirely. No cron, no PID file, no restart-loop. -The service manager owns liveness. +### From `minion-watchdog.sh` (pre-v0.20) -### systemd (Linux hosts with shell access) +Earlier versions of this guide shipped a 68-line bash watchdog +(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor` +which handles everything the script did, plus atomic PID locking, +structured audit events, queue-scoped health checks, and graceful +drain on SIGTERM. + +**Migration:** ```bash -# Create the worker user if it doesn't exist. -sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ - 2>/dev/null || true -sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" +# 1. Stop and remove the old watchdog. +sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null +sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \ + /tmp/gbrain-worker.log +crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line -# Install the unit file, substituting /srv/gbrain → your workspace path. -sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ - /etc/systemd/system/gbrain-worker.service -sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ - /etc/systemd/system/gbrain-worker.service +# 2. Start the supervisor (systemd users: reinstall the unit from +# docs/guides/minions-deployment-snippets/systemd.service, which +# now calls `gbrain jobs supervisor`). +gbrain jobs supervisor start --detach --json +# Or: sudo systemctl restart gbrain-worker -# See 1a above for /etc/gbrain.env install. -sudo systemctl daemon-reload -sudo systemctl enable --now gbrain-worker -sudo systemctl status gbrain-worker -journalctl -u gbrain-worker -n 50 +# 3. Verify. +gbrain jobs supervisor status --json +gbrain doctor # 'supervisor' check should report running=true ``` -`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit -runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`, -and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped -unit covers Bun + Postgres pool + concurrent LLM subagent calls without -hitting the default 1024 cap. +### Schema / migration hygiene -### Fly.io +Regardless of which deployment path you're upgrading from: -Merge the `[processes]` block from -[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into -your existing `fly.toml`. Set secrets with `fly secrets set` — -Fly auto-restarts the process on crash. - -### Render / Railway / Heroku - -Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root. -Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the -platform's env UI or CLI. - -## Upgrading an existing deployment - -If you deployed on v0.13.x or earlier, walk this checklist: - -1. **Stop the worker before upgrading.** - `kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to - exit. Skipping this risks an in-flight job landing partial schema. +1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop` + (or `sudo systemctl stop gbrain-worker`). Skipping this risks an + in-flight job landing partial schema. 2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if `gbrain doctor` reports any migration as `partial` or `pending`. -3. **If you run shell jobs:** from v0.14 onward, the worker requires - `GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to - `/etc/gbrain.env`. Submitters don't need the flag; only the worker does. -4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration - v15 raised the schema default to 5 and backfilled existing non-terminal - rows. A watchdog tuned around 1-strike dead-lettering will now - over-restart because it takes 5 misses to dead-letter. Switch to the - shipped watchdog (which keys on log markers, not job state). -5. **If your v0.16.1 watchdog is still running:** it has a restart-loop - bug (old shutdown lines in the unrotated log re-match every 5 min - forever). Install the current `minion-watchdog.sh` from this guide's - snippets — it writes a restart epoch into the PID file and only - considers log lines newer than that epoch. -6. **Verify.** `gbrain doctor` should report zero `pending` or `partial` - migrations. `gbrain jobs stats` should show no unexplained growth in - `dead` between pre- and post-upgrade. +3. **If you run shell jobs:** from v0.14 onward, pass + `--allow-shell-jobs` to the supervisor (or keep + `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't + need the flag; only the worker does. +4. **Verify.** `gbrain doctor` should report zero `pending` or `partial` + migrations plus a healthy `supervisor` check. `gbrain jobs stats` + should show no unexplained growth in `dead` between pre- and + post-upgrade. ## Known issues @@ -261,9 +263,10 @@ silently. The stall detector then dead-letters the job after **Current defaults that make this worse:** -- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips. -- `max_stalled: 5` (schema column default on master — see `src/schema.sql` - and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. +- `lockDuration: 30000` (30 s) — too short for long jobs during + connection blips. +- `max_stalled: 5` (schema column default — see `src/schema.sql` and + `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. - `stalledInterval: 30000` (30 s) — checks too aggressively. **Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, @@ -271,9 +274,6 @@ silently. The stall detector then dead-letters the job after `--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags (since v0.13.1). These write onto the job row at submit time — which is what `handleStalled()` reads — so per-job tuning is the real knob today. -Worker-level `--lock-duration` / `--stall-interval` are on the roadmap; -until they land, rely on per-job `--max-stalled` plus the watchdog (or -systemd) for worker health. ### DO NOT pass `maxStalledCount` to `MinionWorker` @@ -284,16 +284,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead. ### Zombie shell children When the Bun worker crashes hard, child processes from shell jobs can -become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the -shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running -shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker -has time to flush in-flight jobs before the kill. +become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window +covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For +long-running shell jobs, prefer timeouts via `--timeout-ms` on submit +over relying on hard kills. ## Smoke test ```bash -# Worker alive? -kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD +# Supervisor alive? +gbrain jobs supervisor status --json | jq .running # Aggregate queue health. gbrain jobs stats @@ -304,20 +304,29 @@ gbrain jobs list --status active --limit 10 # Dead-lettered jobs. gbrain jobs list --status dead --limit 10 -# Shell handler registered? (stderr banner merged into log via 2>&1.) -grep "shell handler enabled" /tmp/gbrain-worker.log +# Shell handler registered? (check supervisor audit log or worker stderr.) +gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs' ``` ## Uninstall -- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line. - `kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`. - Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`. -- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to - clean up — temporary workers exit with their jobs. -- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`, - then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`, - then `sudo systemctl daemon-reload`. -- **Option 3 (Fly/Render/Railway):** delete the `worker` process from - `fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets` - persist until `fly secrets unset`. +**`gbrain jobs supervisor`** (foreground or `--detach`): + +```bash +gbrain jobs supervisor stop +``` + +**systemd:** + +```bash +sudo systemctl disable --now gbrain-worker +sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env +sudo systemctl daemon-reload +``` + +**Fly / Render / Railway:** delete the `worker` process from `fly.toml` +/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until +`fly secrets unset`. + +**Inline `--follow`:** remove the cron entry. Nothing else to clean up +— temporary workers exit with their jobs. diff --git a/llms-full.txt b/llms-full.txt index abbe04514..5292c5916 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1403,9 +1403,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis gbrain jobs smoke # verify install gbrain jobs submit sync --params '{}' # fire a background job gbrain jobs stats # health dashboard -gbrain jobs work --concurrency 4 # start a worker (Postgres only) +gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only) +gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`) ``` +`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md). + Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox. **Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now. @@ -3496,8 +3499,8 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/min # Minions Worker Deployment Guide -Deploy `gbrain jobs work` so it stays running across crashes, reboots, and -Postgres connection blips. Written for agents to execute line-by-line. +Keep `gbrain jobs work` running across crashes, reboots, and Postgres +connection blips. Written for agents to execute line-by-line. ## The problem @@ -3508,10 +3511,61 @@ The persistent worker can die silently from: - Bun process crashes with no automatic restart. - Internal event-loop death (PID alive, worker loop stopped). -When the worker dies, submitted jobs sit in `waiting` forever. Nothing in -gbrain core auto-restarts the worker — that's what this guide wires up. +When the worker dies, submitted jobs sit in `waiting` forever. The +canonical answer is `gbrain jobs supervisor` — a first-class CLI that +spawns `gbrain jobs work` as a child and auto-restarts it on crash. -## Variables used in this guide +## Worker supervision + +### The canonical pattern + +`gbrain jobs supervisor` is an auto-restarting wrapper around +`gbrain jobs work`. It writes a PID file, restarts the worker on crash +with exponential backoff (1s → 60s cap), emits lifecycle events to an +audit file, and drains gracefully on SIGTERM (35s worker-drain window +before SIGKILL). Exit codes are documented so agents can branch on them. + +**Typical commands:** + +```bash +# Start in the foreground (blocks; Ctrl-C to stop). +gbrain jobs supervisor --concurrency 4 + +# Start detached — returns {"event":"started","supervisor_pid":…} on stdout. +gbrain jobs supervisor start --detach --json + +# Check liveness without reading log files. +gbrain jobs supervisor status --json + +# Graceful stop (SIGTERM + drain wait + SIGKILL fallback). +gbrain jobs supervisor stop +``` + +**Exit codes:** + +| Code | Meaning | +|---|---| +| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) | +| 1 | Max crashes exceeded (worker kept dying) | +| 2 | Another supervisor holds the PID lock | +| 3 | PID file unwritable (permission / path error) | + +An agent seeing exit=2 can safely treat it as "one is already running"; +exit=1 should page a human. + +### Which supervisor when? + +The supervisor solves in-process crash recovery. Platform-level +supervision (systemd, Fly, Render) handles host-level failures. You +usually want both. + +| Environment | Recommendation | +|---|---| +| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). | +| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). | +| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. | + +### Variables used in this guide Substitute these once before copy-pasting any snippet. @@ -3519,142 +3573,122 @@ Substitute these once before copy-pasting any snippet. |---|---|---| | `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` | | `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` | -| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) | -| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) | | `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` | -| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) | +| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) | -## Preconditions +### Preconditions -Run these before Step 1 of any option. Fail fast if something is wrong. +Run these before any deployment step. ```bash # 1. gbrain is on PATH and resolves to an absolute location. command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; } -# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists). +# 2. DATABASE_URL points at reachable Postgres. +# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the +# separate worker process. If `config.engine === 'pglite'` the CLI rejects +# with a clear error.) gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")' -# 3. Schema is up to date. If version=0 or status=="fail", fix it first: +# 3. Schema is up to date. If version=0 or status=="fail": # gbrain apply-migrations --yes gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")' -# 4. You have write access to at least one crontab mechanism. -crontab -l >/dev/null 2>&1 && echo "user crontab OK" -[ -w /etc/crontab ] && echo "/etc/crontab OK" - -# 5. If you plan to submit `shell` jobs, the WORKER process needs -# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated -# in registerBuiltinHandlers(); without the flag the worker startup -# line reads "shell handler disabled (...)". +# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the +# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting). +# Without the flag, the shell handler is disabled at worker startup. ``` -## Which option? +## Agent usage (OpenClaw / Hermes / Cursor / Codex) -- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take - > 30 s → **Option 1** (watchdog cron + persistent worker). -- Your workload is short deterministic scripts on a fixed schedule (every - 3 h, daily, weekly) → **Option 2** (inline `--follow`). -- You don't have shell access to a long-running box (Fly/Render/Railway, - or any systemd host) → **Option 3** (service manager — replaces cron). - -## Option 1: watchdog cron + persistent worker - -A 5-minute cron checks whether the worker process is alive **and** whether -it has logged an internal shutdown since its last start. Restarts if either -condition fails. - -### 1a. Install the env file (secrets stay out of crontab) - -Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is -mode 644 (world-readable); user crontabs under `/var/spool/cron/` are -readable by `root`. Use the shipped env-file template: +Three-command pattern an agent can drive without shell archaeology: ```bash -sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ +# Start (returns PIDs + pid_file on stdout as JSON, then detaches) +gbrain jobs supervisor start --detach --json +# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"} + +# Check health (machine-parseable JSON, no log scraping) +gbrain jobs supervisor status --json +# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...} + +# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback) +gbrain jobs supervisor stop +``` + +Every lifecycle event (spawn, crash, backoff, health warning, max-crashes, +shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl` +for historical inspection. `gbrain doctor` reads that file and surfaces +a `supervisor` check in its health report. + +## Deployment: systemd + +For long-running Linux VMs with shell access. + +```bash +# Create the worker user if it doesn't exist. +sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ + 2>/dev/null || true +sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" + +# Install the env file (secrets stay out of the unit file). +sudo install -m 600 -o gbrain -g gbrain \ docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env sudoedit /etc/gbrain.env +# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1. + +# Install the unit file, substituting /srv/gbrain → your workspace path. +sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ + /etc/systemd/system/gbrain-worker.service +sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ + /etc/systemd/system/gbrain-worker.service + +sudo systemctl daemon-reload +sudo systemctl enable --now gbrain-worker +sudo systemctl status gbrain-worker +journalctl -u gbrain-worker -n 50 ``` -Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if -applicable). See -[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example) -for the full list. +The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work` +directly) so you get two-layer supervision: systemd restarts the supervisor +on host reboot, supervisor restarts the worker on in-process crash. -### 1b. Install the watchdog script +`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery. +The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`, +and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and +audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent +LLM subagent calls without hitting the default 1024 cap. -The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh) -ships in-repo and writes a two-line PID file (PID on line 1, restart epoch -on line 2). The restart-epoch marker is how the watchdog distinguishes -stale shutdown lines in the log from current ones — without it, every tick -after the first restart would match an old `worker shutting down` line and -loop forever. - -Requires GNU coreutils (Linux default). On macOS/BSD install via -`brew install coreutils` and alias `date` to `gdate` in the cron env if you -want to test the watchdog locally; production Linux boxes work as-is. +## Deployment: Fly.io ```bash -sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ - docs/guides/minions-deployment-snippets/minion-watchdog.sh \ - /usr/local/bin/minion-watchdog.sh +# Merge the [processes] block from fly.toml.partial into your fly.toml. +cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml +# Review + edit as needed. + +# Set secrets (Fly handles restart on crash). +fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1 ``` -### 1c. Wire into cron +The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly +restarts the container on host failure; the supervisor restarts the +worker on in-process crash. -Pick the form that matches the crontab you're editing. +## Deployment: Render / Railway / Heroku -**If you ran `crontab -e`** (user crontab — 5-field, no user column): +Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo +root. The shipped Procfile calls `gbrain jobs supervisor`. Set +`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's +env UI or CLI. -``` -SHELL=/bin/bash -PATH=/usr/local/bin:/usr/bin:/bin -BASH_ENV=/etc/gbrain.env -*/5 * * * * /usr/local/bin/minion-watchdog.sh -``` +## Deployment: inline `--follow` (no persistent worker) -**If you edited `/etc/crontab` directly** (system crontab — 6-field, with -user column): - -``` -SHELL=/bin/bash -PATH=/usr/local/bin:/usr/bin:/bin -BASH_ENV=/etc/gbrain.env -*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh -``` - -In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to -source the env file before running the watchdog — that's how the -connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without -landing in the world-readable crontab itself. - -### 1d. Log rotation - -The watchdog appends to the worker log across restarts. If you expect the -file to grow unbounded, rotate it externally with `logrotate`: - -``` -# /etc/logrotate.d/gbrain-worker -/tmp/gbrain-worker.log { - daily - rotate 7 - missingok - notifempty - copytruncate -} -``` - -`copytruncate` is important — the watchdog's restart-epoch check survives -it (the epoch is compared against in-log timestamps, not file inode). - -## Option 2: inline `--follow` (no persistent worker) - -Each cron run brings its own temporary worker. `--follow` starts one on -the queue and blocks until the just-submitted job reaches a terminal state -(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead -per job; negligible vs job duration for scheduled work. - -Example: nightly brain enrichment as a shell job. +For short deterministic scripts on a fixed schedule where you don't need +a persistent worker between runs. Each cron run brings its own temporary +worker. `--follow` starts one on the queue and blocks until the +just-submitted job reaches a terminal state (`completed` / `failed` / +`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job +duration for scheduled work. ```bash GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ @@ -3666,85 +3700,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ Replace `gbrain embed --stale` with whichever gbrain subcommand you're scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`, -`lint`, `autopilot`). If you're shelling out to a non-gbrain binary, -keep its absolute path in the `cmd`. - -**Shared-queue gotcha.** If other jobs are already waiting on the same -queue with higher priority or earlier `created_at`, the temporary worker -processes those first before reaching yours. `--follow` still exits only -when YOUR job finishes. For strict single-job semantics on shared queues, +`lint`, `autopilot`). For strict single-job semantics on shared queues, use a dedicated queue name like `nightly-enrich` above. -## Option 3: service manager (systemd / Fly / Render / Railway) +## Upgrading from an older deployment -Replaces the watchdog entirely. No cron, no PID file, no restart-loop. -The service manager owns liveness. +### From `minion-watchdog.sh` (pre-v0.20) -### systemd (Linux hosts with shell access) +Earlier versions of this guide shipped a 68-line bash watchdog +(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor` +which handles everything the script did, plus atomic PID locking, +structured audit events, queue-scoped health checks, and graceful +drain on SIGTERM. + +**Migration:** ```bash -# Create the worker user if it doesn't exist. -sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ - 2>/dev/null || true -sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" +# 1. Stop and remove the old watchdog. +sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null +sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \ + /tmp/gbrain-worker.log +crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line -# Install the unit file, substituting /srv/gbrain → your workspace path. -sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ - /etc/systemd/system/gbrain-worker.service -sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ - /etc/systemd/system/gbrain-worker.service +# 2. Start the supervisor (systemd users: reinstall the unit from +# docs/guides/minions-deployment-snippets/systemd.service, which +# now calls `gbrain jobs supervisor`). +gbrain jobs supervisor start --detach --json +# Or: sudo systemctl restart gbrain-worker -# See 1a above for /etc/gbrain.env install. -sudo systemctl daemon-reload -sudo systemctl enable --now gbrain-worker -sudo systemctl status gbrain-worker -journalctl -u gbrain-worker -n 50 +# 3. Verify. +gbrain jobs supervisor status --json +gbrain doctor # 'supervisor' check should report running=true ``` -`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit -runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`, -and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped -unit covers Bun + Postgres pool + concurrent LLM subagent calls without -hitting the default 1024 cap. +### Schema / migration hygiene -### Fly.io +Regardless of which deployment path you're upgrading from: -Merge the `[processes]` block from -[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into -your existing `fly.toml`. Set secrets with `fly secrets set` — -Fly auto-restarts the process on crash. - -### Render / Railway / Heroku - -Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root. -Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the -platform's env UI or CLI. - -## Upgrading an existing deployment - -If you deployed on v0.13.x or earlier, walk this checklist: - -1. **Stop the worker before upgrading.** - `kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to - exit. Skipping this risks an in-flight job landing partial schema. +1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop` + (or `sudo systemctl stop gbrain-worker`). Skipping this risks an + in-flight job landing partial schema. 2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if `gbrain doctor` reports any migration as `partial` or `pending`. -3. **If you run shell jobs:** from v0.14 onward, the worker requires - `GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to - `/etc/gbrain.env`. Submitters don't need the flag; only the worker does. -4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration - v15 raised the schema default to 5 and backfilled existing non-terminal - rows. A watchdog tuned around 1-strike dead-lettering will now - over-restart because it takes 5 misses to dead-letter. Switch to the - shipped watchdog (which keys on log markers, not job state). -5. **If your v0.16.1 watchdog is still running:** it has a restart-loop - bug (old shutdown lines in the unrotated log re-match every 5 min - forever). Install the current `minion-watchdog.sh` from this guide's - snippets — it writes a restart epoch into the PID file and only - considers log lines newer than that epoch. -6. **Verify.** `gbrain doctor` should report zero `pending` or `partial` - migrations. `gbrain jobs stats` should show no unexplained growth in - `dead` between pre- and post-upgrade. +3. **If you run shell jobs:** from v0.14 onward, pass + `--allow-shell-jobs` to the supervisor (or keep + `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't + need the flag; only the worker does. +4. **Verify.** `gbrain doctor` should report zero `pending` or `partial` + migrations plus a healthy `supervisor` check. `gbrain jobs stats` + should show no unexplained growth in `dead` between pre- and + post-upgrade. ## Known issues @@ -3757,9 +3762,10 @@ silently. The stall detector then dead-letters the job after **Current defaults that make this worse:** -- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips. -- `max_stalled: 5` (schema column default on master — see `src/schema.sql` - and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. +- `lockDuration: 30000` (30 s) — too short for long jobs during + connection blips. +- `max_stalled: 5` (schema column default — see `src/schema.sql` and + `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. - `stalledInterval: 30000` (30 s) — checks too aggressively. **Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, @@ -3767,9 +3773,6 @@ silently. The stall detector then dead-letters the job after `--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags (since v0.13.1). These write onto the job row at submit time — which is what `handleStalled()` reads — so per-job tuning is the real knob today. -Worker-level `--lock-duration` / `--stall-interval` are on the roadmap; -until they land, rely on per-job `--max-stalled` plus the watchdog (or -systemd) for worker health. ### DO NOT pass `maxStalledCount` to `MinionWorker` @@ -3780,16 +3783,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead. ### Zombie shell children When the Bun worker crashes hard, child processes from shell jobs can -become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the -shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running -shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker -has time to flush in-flight jobs before the kill. +become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window +covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For +long-running shell jobs, prefer timeouts via `--timeout-ms` on submit +over relying on hard kills. ## Smoke test ```bash -# Worker alive? -kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD +# Supervisor alive? +gbrain jobs supervisor status --json | jq .running # Aggregate queue health. gbrain jobs stats @@ -3800,23 +3803,32 @@ gbrain jobs list --status active --limit 10 # Dead-lettered jobs. gbrain jobs list --status dead --limit 10 -# Shell handler registered? (stderr banner merged into log via 2>&1.) -grep "shell handler enabled" /tmp/gbrain-worker.log +# Shell handler registered? (check supervisor audit log or worker stderr.) +gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs' ``` ## Uninstall -- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line. - `kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`. - Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`. -- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to - clean up — temporary workers exit with their jobs. -- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`, - then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`, - then `sudo systemctl daemon-reload`. -- **Option 3 (Fly/Render/Railway):** delete the `worker` process from - `fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets` - persist until `fly secrets unset`. +**`gbrain jobs supervisor`** (foreground or `--detach`): + +```bash +gbrain jobs supervisor stop +``` + +**systemd:** + +```bash +sudo systemctl disable --now gbrain-worker +sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env +sudo systemctl daemon-reload +``` + +**Fly / Render / Railway:** delete the `worker` process from `fly.toml` +/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until +`fly secrets unset`. + +**Inline `--follow`:** remove the cron entry. Nothing else to clean up +— temporary workers exit with their jobs. --- diff --git a/package.json b/package.json index 7011e391d..a34415c76 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.20.0", + "version": "0.20.2", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index e132ca736..f8f53849f 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -163,6 +163,67 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo // Read/parse failure is itself best-effort; skip silently. } + // 3b-bis. Supervisor health (filesystem-only: PID liveness + audit log). + // Reads the default PID file (`~/.gbrain/supervisor.pid` unless the user + // overrode with GBRAIN_SUPERVISOR_PID_FILE) and the latest audit file + // written by src/core/minions/handlers/supervisor-audit.ts. Surfaces + // supervisor_running / last_start / crashes_24h / max_crashes_exceeded. + // Does NOT run the supervisor itself — this is a read-only health check. + try { + const { DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts'); + const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts'); + + let supervisorPid: number | null = null; + let running = false; + if (existsSync(DEFAULT_PID_FILE)) { + try { + const line = readFileSync(DEFAULT_PID_FILE, 'utf8').trim().split('\n')[0]; + const parsed = parseInt(line, 10); + if (!isNaN(parsed) && parsed > 0) { + supervisorPid = parsed; + try { process.kill(parsed, 0); running = true; } catch { running = false; } + } + } catch { /* unreadable */ } + } + + const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 }); + const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null; + const crashes24h = events.filter(e => e.event === 'worker_exited').length; + const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null; + + // Only surface a Check if the supervisor was ever observed (stops the + // "never used the supervisor" install from getting a warn about it). + if (supervisorPid !== null || events.length > 0) { + if (maxCrashesEvent) { + checks.push({ + name: 'supervisor', + status: 'fail', + message: `Supervisor gave up at ${maxCrashesEvent.ts} (max_crashes_exceeded). Restart with: gbrain jobs supervisor start --detach`, + }); + } else if (!running && events.length > 0) { + checks.push({ + name: 'supervisor', + status: 'warn', + message: `Supervisor not running (last_start=${lastStart ?? 'unknown'}). Restart with: gbrain jobs supervisor start --detach`, + }); + } else if (crashes24h > 3) { + checks.push({ + name: 'supervisor', + status: 'warn', + message: `Supervisor running but worker crashed ${crashes24h}x in last 24h. Check ~/.gbrain/audit/supervisor-*.jsonl for causes.`, + }); + } else { + checks.push({ + name: 'supervisor', + status: 'ok', + message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h}`, + }); + } + } + } catch { + // Audit read / import failure is best-effort; skip silently. + } + // 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit` // bookmark when per-file parse errors happen, and appends each failure // to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error. diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index 8148664d3..d8c87d0f3 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -70,6 +70,42 @@ USAGE gbrain jobs stats gbrain jobs smoke gbrain jobs work [--queue Q] [--concurrency N] + gbrain jobs supervisor [start] [--detach] [--json] + [--concurrency N] [--queue Q] [--pid-file PATH] + [--max-crashes N] [--health-interval N] + [--allow-shell-jobs] [--cli-path PATH] + gbrain jobs supervisor status [--json] [--pid-file PATH] + gbrain jobs supervisor stop [--json] [--pid-file PATH] + + Auto-restarting wrapper around 'gbrain jobs work'. Spawns the worker + as a child process and restarts on crash with exponential backoff + (1s -> 60s cap). Writes a PID file to ~/.gbrain/supervisor.pid by + default (override via --pid-file or GBRAIN_SUPERVISOR_PID_FILE env). + Lifecycle events are appended to + \${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl + + SUBCOMMANDS + start (default) Launch the supervisor. --detach returns a + JSON {event, supervisor_pid, pid_file} payload on + stdout and forks; omit for foreground. + status Read PID file + audit log, report running / last_start + / crashes_24h / max_crashes_exceeded as JSON or human. + Exits 0 if running, 1 if not. + stop Send SIGTERM to the supervisor, wait up to 40s for + graceful drain, report outcome. Exits 0 on clean stop. + + EXIT CODES (start) + 0 clean shutdown (SIGTERM/SIGINT received, worker drained) + 1 max crashes exceeded (worker kept dying) + 2 another supervisor holds the PID lock + 3 PID file unwritable (permission / path error) + + EXAMPLES + gbrain jobs supervisor --concurrency 4 # foreground (Ctrl-C stops) + gbrain jobs supervisor start --detach --json # agent-friendly: fork + return JSON + gbrain jobs supervisor status --json # machine-readable health check + gbrain jobs supervisor stop # graceful stop + gbrain jobs supervisor --json --allow-shell-jobs # JSONL events + shell-exec on HANDLER TYPES (built in) sync Pull and embed new pages from the repo @@ -481,6 +517,185 @@ HANDLER TYPES (built in) break; } + case 'supervisor': { + // Dispatcher for supervisor subcommands: + // gbrain jobs supervisor → foreground start (back-compat) + // gbrain jobs supervisor start [--detach] → foreground or detached start + // gbrain jobs supervisor status → JSON liveness + queue stats + // gbrain jobs supervisor stop → SIGTERM + drain wait + const { MinionSupervisor, DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts'); + const { writeSupervisorEvent } = await import('../core/minions/handlers/supervisor-audit.ts'); + + const supCmd = args[1]; + const isStatusCmd = supCmd === 'status'; + const isStopCmd = supCmd === 'stop'; + const isStartCmd = supCmd === 'start' || supCmd === undefined || supCmd === '--detach' || + (typeof supCmd === 'string' && supCmd.startsWith('--')); + const jsonMode = hasFlag(args, '--json'); + const pidFile = parseFlag(args, '--pid-file') ?? DEFAULT_PID_FILE; + + // ----- status subcommand ----- + if (isStatusCmd) { + const { existsSync, readFileSync } = await import('fs'); + const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts'); + + let supervisorPid: number | null = null; + let running = false; + if (existsSync(pidFile)) { + try { + const line = readFileSync(pidFile, 'utf8').trim().split('\n')[0]; + const parsed = parseInt(line, 10); + if (!isNaN(parsed) && parsed > 0) { + supervisorPid = parsed; + try { process.kill(parsed, 0); running = true; } catch { running = false; } + } + } catch { /* unreadable PID file */ } + } + + const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 }); + const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null; + const crashes24h = events.filter(e => e.event === 'worker_exited').length; + const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null; + + const status = { + running, + supervisor_pid: supervisorPid, + pid_file: pidFile, + last_start: lastStart, + crashes_24h: crashes24h, + max_crashes_exceeded: !!maxCrashesEvent, + }; + + if (jsonMode) { + console.log(JSON.stringify(status, null, 2)); + } else { + console.log(`Supervisor: ${running ? 'running' : 'not running'}`); + if (supervisorPid) console.log(` PID: ${supervisorPid}`); + console.log(` PID file: ${pidFile}`); + if (lastStart) console.log(` Last start: ${lastStart}`); + console.log(` Crashes (24h): ${crashes24h}`); + if (maxCrashesEvent) console.log(` ⚠ Max crashes exceeded at ${maxCrashesEvent.ts}`); + } + process.exit(running ? 0 : 1); + } + + // ----- stop subcommand ----- + if (isStopCmd) { + const { existsSync, readFileSync } = await import('fs'); + if (!existsSync(pidFile)) { + const payload = { stopped: false, reason: 'pid_file_missing', pid_file: pidFile }; + if (jsonMode) console.log(JSON.stringify(payload)); + else console.error(`No PID file at ${pidFile}; supervisor not running.`); + process.exit(1); + } + let supervisorPid: number; + try { + supervisorPid = parseInt(readFileSync(pidFile, 'utf8').trim().split('\n')[0], 10); + if (isNaN(supervisorPid) || supervisorPid <= 0) throw new Error('invalid pid'); + } catch (err) { + const payload = { stopped: false, reason: 'pid_file_corrupt', error: String(err) }; + if (jsonMode) console.log(JSON.stringify(payload)); + else console.error(`PID file corrupt: ${err}`); + process.exit(1); + } + + try { process.kill(supervisorPid, 'SIGTERM'); } + catch (err: unknown) { + const code = (err as NodeJS.ErrnoException)?.code; + const payload = { + stopped: false, + reason: code === 'ESRCH' ? 'process_gone' : 'kill_failed', + supervisor_pid: supervisorPid, + }; + if (jsonMode) console.log(JSON.stringify(payload)); + else console.error(`Cannot signal PID ${supervisorPid}: ${err}`); + process.exit(code === 'ESRCH' ? 0 : 1); + } + + // Poll for up to 40s (supervisor's own 35s drain + 5s slack). + const deadline = Date.now() + 40_000; + let stoppedCleanly = false; + while (Date.now() < deadline) { + try { process.kill(supervisorPid, 0); } + catch { stoppedCleanly = true; break; } + await new Promise(r => setTimeout(r, 250)); + } + + const payload = { + stopped: stoppedCleanly, + supervisor_pid: supervisorPid, + reason: stoppedCleanly ? 'drained' : 'timeout_40s', + }; + if (jsonMode) console.log(JSON.stringify(payload)); + else console.log(stoppedCleanly ? `Supervisor ${supervisorPid} stopped.` : `Supervisor ${supervisorPid} did not exit within 40s.`); + process.exit(stoppedCleanly ? 0 : 1); + } + + // ----- start subcommand (default) ----- + if (!isStartCmd) { + console.error(`Unknown supervisor subcommand: ${supCmd}. Expected: start, status, stop.`); + process.exit(1); + } + + const config = (await import('../core/config.ts')).loadConfig(); + if (config?.engine === 'pglite') { + console.error('Error: Supervisor requires Postgres. PGLite uses an exclusive file lock that blocks other processes.'); + process.exit(1); + } + + const { resolveGbrainCliPath } = await import('./autopilot.ts'); + + const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10); + const queueName = parseFlag(args, '--queue') ?? 'default'; + const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10); + const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10); + const allowShellJobs = hasFlag(args, '--allow-shell-jobs') || + !!process.env.GBRAIN_ALLOW_SHELL_JOBS; + const detach = hasFlag(args, '--detach'); + + const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath(); + + // --detach: fork a background supervisor, print PID payload, exit 0. + // Implementation: re-exec the same CLI as a detached child without --detach, + // inheriting stderr (so JSONL events still flow to the parent's tail-f + // if they wanted to follow logs) but detaching stdin/stdout. + if (detach) { + const { spawn } = await import('child_process'); + const childArgs = process.argv.slice(2).filter(a => a !== '--detach'); + const child = spawn(process.execPath, [process.argv[1], ...childArgs], { + detached: true, + stdio: ['ignore', 'ignore', 'inherit'], + env: process.env, + }); + child.unref(); + const payload = { + event: 'started', + supervisor_pid: child.pid, + pid_file: pidFile, + detached: true, + }; + console.log(JSON.stringify(payload)); + process.exit(0); + } + + // Foreground start. + const supervisorPid = process.pid; + const supervisor = new MinionSupervisor(engine, { + concurrency, + queue: queueName, + pidFile, + maxCrashes, + healthInterval, + cliPath, + allowShellJobs, + json: jsonMode, + onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid), + }); + + await supervisor.start(); + break; + } + default: console.error(`Unknown subcommand: ${sub}. Run 'gbrain jobs --help' for usage.`); process.exit(1); diff --git a/src/core/minions/handlers/supervisor-audit.ts b/src/core/minions/handlers/supervisor-audit.ts new file mode 100644 index 000000000..49138a328 --- /dev/null +++ b/src/core/minions/handlers/supervisor-audit.ts @@ -0,0 +1,107 @@ +/** + * Supervisor lifecycle audit log. JSONL, weekly-rotated, best-effort. + * + * Writes one line per supervisor event (started, worker_spawned, worker_exited, + * backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, + * stopped, worker_spawn_failed) to + * `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl` + * using ISO-8601 week numbering. `computeAuditFilename(kind, now)` derives + * the filename; the ISO-week math is shared with `shell-audit.ts` via the + * `computeIsoWeekName()` helper that both call. + * + * Shape: every emission already includes `event` and `ts`; we write it + * verbatim and let consumers (like `gbrain doctor`) grep for events of + * interest. `supervisor_pid` is added at start() time so each line is + * self-describing even if a log shipper concatenates multiple supervisors' + * files. + * + * Best-effort: write failures go to stderr and never block supervisor work. + * A disk-full attacker could silently disable the trail — this is an + * operational trace for `gbrain doctor`, not forensic insurance. + * + * `GBRAIN_AUDIT_DIR` overrides the default `~/.gbrain/audit/` path for + * container deploys where `$HOME` is read-only. + */ + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { resolveAuditDir } from './shell-audit.ts'; +import type { SupervisorEmission } from '../supervisor.ts'; + +/** + * Compute `supervisor-YYYY-Www.jsonl` using ISO-8601 week numbering. + * + * Mirrors `shell-audit.ts:computeAuditFilename()` exactly. Year-boundary + * edge: 2027-01-01 is ISO week 53 of year 2026, so the correct filename + * is `supervisor-2026-W53.jsonl`. + */ +export function computeSupervisorAuditFilename(now: Date = new Date()): string { + const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate())); + const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6 + d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday (ISO week anchor) + const isoYear = d.getUTCFullYear(); + const firstThursday = new Date(Date.UTC(isoYear, 0, 4)); + const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7; + firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3); + const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1; + const ww = String(weekNum).padStart(2, '0'); + return `supervisor-${isoYear}-W${ww}.jsonl`; +} + +/** + * Append a single supervisor lifecycle event to the rotated JSONL audit + * file. `supervisorPid` is the OS pid of the supervisor process (added + * to every line so a log shipper concatenating files from multiple + * supervisors still produces parseable traces). + */ +export function writeSupervisorEvent(emission: SupervisorEmission, supervisorPid: number): void { + const dir = resolveAuditDir(); + const filename = computeSupervisorAuditFilename(); + const fullPath = path.join(dir, filename); + const line = JSON.stringify({ ...emission, supervisor_pid: supervisorPid }) + '\n'; + + try { + fs.mkdirSync(dir, { recursive: true }); + fs.appendFileSync(fullPath, line, { encoding: 'utf8' }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[supervisor-audit] write failed (${msg}); continuing\n`); + } +} + +/** + * Read back the latest supervisor audit file. Returns events sorted + * oldest-first. Best-effort: missing file / parse errors return []. + * Used by `gbrain doctor` (Lane D) to surface supervisor health. + */ +export function readSupervisorEvents(opts: { sinceMs?: number } = {}): SupervisorEmission[] { + const dir = resolveAuditDir(); + const filename = computeSupervisorAuditFilename(); + const fullPath = path.join(dir, filename); + + let raw: string; + try { + raw = fs.readFileSync(fullPath, 'utf8'); + } catch { + return []; + } + + const now = Date.now(); + const cutoff = opts.sinceMs !== undefined ? now - opts.sinceMs : 0; + const events: SupervisorEmission[] = []; + for (const line of raw.split('\n')) { + if (!line.trim()) continue; + try { + const obj = JSON.parse(line) as SupervisorEmission; + if (!obj.event || !obj.ts) continue; + if (cutoff > 0) { + const ts = Date.parse(obj.ts); + if (!isNaN(ts) && ts < cutoff) continue; + } + events.push(obj); + } catch { + // Ignore malformed lines (truncated writes, disk-full corruption). + } + } + return events; +} diff --git a/src/core/minions/supervisor.ts b/src/core/minions/supervisor.ts new file mode 100644 index 000000000..4da80b660 --- /dev/null +++ b/src/core/minions/supervisor.ts @@ -0,0 +1,573 @@ +/** + * MinionSupervisor — Process manager for the Minion worker. + * + * Spawns `gbrain jobs work` as a child process and restarts it on crash + * with exponential backoff. Provides health monitoring, PID file locking + * (atomic via O_CREAT|O_EXCL), and graceful shutdown. + * + * ENGINE: Postgres only. PGLite uses an exclusive file lock that blocks + * any separate worker process, so `gbrain jobs supervisor` cannot work + * against a PGLite brain — `src/commands/jobs.ts` rejects that combination + * at the CLI layer. The health-check SQL below assumes Postgres schema. + * + * Usage: + * gbrain jobs supervisor [--concurrency N] [--queue Q] [--pid-file PATH] + * [--max-crashes N] [--health-interval N] + * [--allow-shell-jobs] [--json] + * + * Design: the supervisor does NOT run the worker in-process. It spawns a + * separate child so a misbehaving handler can't take down the supervisor. + * Same isolation pattern as autopilot.ts but standalone and reusable. + * + * Exit codes (documented in CLI --help): + * 0 clean shutdown (SIGTERM/SIGINT received, worker drained) + * 1 max crashes exceeded (worker kept dying) + * 2 another supervisor holds the PID lock + * 3 PID file unwritable (permission / path error) + */ + +import { spawn, type ChildProcess } from 'child_process'; +import { + closeSync, + existsSync, + mkdirSync, + openSync, + readFileSync, + unlinkSync, + writeFileSync, + writeSync, +} from 'fs'; +import { dirname } from 'path'; +import type { BrainEngine } from '../engine.ts'; + +export type SupervisorEvent = + | 'started' + | 'worker_spawned' + | 'worker_exited' + | 'worker_spawn_failed' + | 'backoff' + | 'health_warn' + | 'health_error' + | 'max_crashes_exceeded' + | 'shutting_down' + | 'stopped'; + +export interface SupervisorEmission { + event: SupervisorEvent; + ts: string; + [key: string]: unknown; +} + +export interface SupervisorOpts { + /** Worker concurrency (passed to child). Default: 2. */ + concurrency: number; + /** Queue name (passed to child). Default: 'default'. */ + queue: string; + /** PID file path. Default: `${HOME}/.gbrain/supervisor.pid` (parent dir auto-created). */ + pidFile: string; + /** Max consecutive crashes before giving up. Default: 10. */ + maxCrashes: number; + /** Health check interval in ms. Default: 60000. */ + healthInterval: number; + /** Path to the gbrain CLI executable (MUST be a compiled binary; .ts sources cannot be spawned). */ + cliPath: string; + /** Allow shell jobs on child worker. Default: false. When true, sets GBRAIN_ALLOW_SHELL_JOBS=1 on child env. */ + allowShellJobs: boolean; + /** JSON mode: emit JSONL events on stderr, reserve stdout for data payloads. Default: false. */ + json: boolean; + /** Optional event sink (Lane C audit writer). Called for every lifecycle event. */ + onEvent?: (event: SupervisorEmission) => void; + /** + * Test-only override: minimum backoff in ms between child respawns. Default: undefined + * (uses full `calculateBackoffMs()` curve). Tests pass `1` to make crash-loops finish + * in < 1s. Not exposed via CLI. + * @internal + */ + _backoffFloorMs?: number; +} + +export const DEFAULT_PID_FILE: string = (() => { + const envOverride = process.env.GBRAIN_SUPERVISOR_PID_FILE; + if (envOverride && envOverride.length > 0) return envOverride; + const home = process.env.HOME ?? '/tmp'; + return `${home}/.gbrain/supervisor.pid`; +})(); + +const DEFAULTS: Omit = { + concurrency: 2, + queue: 'default', + pidFile: DEFAULT_PID_FILE, + maxCrashes: 10, + healthInterval: 60_000, + allowShellJobs: false, + json: false, +}; + +/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */ +export function calculateBackoffMs(crashCount: number): number { + const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000); + // Add 10% jitter + return base + Math.random() * base * 0.1; +} + +/** Check if a PID is alive. */ +function isProcessAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +/** Exit codes for documented agent branching. */ +export const ExitCodes = { + CLEAN: 0, + MAX_CRASHES: 1, + LOCK_HELD: 2, + PID_UNWRITABLE: 3, +} as const; + +export class MinionSupervisor { + private opts: SupervisorOpts; + private engine: BrainEngine; + private child: ChildProcess | null = null; + private crashCount = 0; + private lastStartTime = 0; + private stopping = false; + private inBackoff = false; + private healthInFlight = false; + private healthTimer: ReturnType | null = null; + private exitListener: (() => void) | null = null; + private sigtermListener: (() => void) | null = null; + private sigintListener: (() => void) | null = null; + private lockAcquired = false; + + constructor(engine: BrainEngine, opts: Partial & { cliPath: string }) { + this.engine = engine; + this.opts = { ...DEFAULTS, ...opts }; + } + + /** + * Emit a lifecycle event. In JSON mode, writes a JSONL record to stderr. + * In human mode, writes a human-readable log line to stdout (info) or + * stderr (warn/error). Also calls `opts.onEvent` if set (Lane C audit + * writer hooks here). + */ + private emit(event: SupervisorEvent, fields: Record = {}): void { + const emission: SupervisorEmission = { + event, + ts: new Date().toISOString(), + ...fields, + }; + + if (this.opts.json) { + // stderr is the event channel; stdout stays clean for data (e.g., --detach payload). + try { + process.stderr.write(JSON.stringify(emission) + '\n'); + } catch { /* best effort */ } + } else { + const ts = emission.ts.slice(11, 19); + const detail = Object.entries(fields) + .filter(([k]) => k !== 'event' && k !== 'ts') + .map(([k, v]) => `${k}=${typeof v === 'string' ? v : JSON.stringify(v)}`) + .join(' '); + const isWarn = event === 'health_warn' || event === 'health_error' || + event === 'worker_spawn_failed' || event === 'max_crashes_exceeded'; + const line = `[supervisor ${ts}] ${event}${detail ? ' ' + detail : ''}`; + if (isWarn) { + console.warn(line); + } else { + console.log(line); + } + } + + // Audit sink (Lane C plumbs this). + if (this.opts.onEvent) { + try { this.opts.onEvent(emission); } catch { /* best effort */ } + } + } + + /** Start the supervisor. Blocks until stopped or max crashes exceeded. */ + async start(): Promise { + // 1. PID file lock (atomic via O_CREAT|O_EXCL). + const lockResult = this.acquirePidLock(); + if (lockResult === 'held') { + // Another supervisor owns the lock — exit code 2. + process.exit(ExitCodes.LOCK_HELD); + } + if (lockResult === 'unwritable') { + // PID path isn't writable — exit code 3 with helpful hint. + process.exit(ExitCodes.PID_UNWRITABLE); + } + + // 2. Cleanup on process exit (covers any exit path including process.exit). + this.exitListener = () => { + try { + if (existsSync(this.opts.pidFile)) { + const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0]; + if (contents === String(process.pid)) { + unlinkSync(this.opts.pidFile); + } + } + } catch { /* best effort */ } + }; + process.on('exit', this.exitListener); + + // 3. Signal handlers (tracked refs; removed on shutdown for test lifecycle hygiene). + this.sigtermListener = () => { void this.shutdown('SIGTERM', ExitCodes.CLEAN); }; + this.sigintListener = () => { void this.shutdown('SIGINT', ExitCodes.CLEAN); }; + process.on('SIGTERM', this.sigtermListener); + process.on('SIGINT', this.sigintListener); + + // 4. Health monitoring. + this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval); + + // 5. Announce start. + this.emit('started', { + supervisor_pid: process.pid, + pid_file: this.opts.pidFile, + concurrency: this.opts.concurrency, + queue: this.opts.queue, + max_crashes: this.opts.maxCrashes, + }); + + // 6. Run the supervise loop (respawn on crash, bounded by maxCrashes). + await this.runSuperviseLoop(); + } + + /** Unified shutdown path. Reason becomes the audit event name; exitCode is process exit. */ + private async shutdown(reason: string, exitCode: number): Promise { + if (this.stopping) return; + this.stopping = true; + + this.emit('shutting_down', { reason, exit_code: exitCode }); + + if (this.healthTimer) { + clearInterval(this.healthTimer); + this.healthTimer = null; + } + + if (this.child) { + try { this.child.kill('SIGTERM'); } catch { /* already dead */ } + await Promise.race([ + new Promise(r => this.child!.once('exit', () => r())), + new Promise(r => setTimeout(() => r(), 35_000)), + ]); + if (this.child && !this.child.killed) { + try { this.child.kill('SIGKILL'); } catch { /* already dead */ } + } + } + + // Remove signal handlers so tests that spin up multiple supervisors on + // the same process don't accumulate listeners. `process.on('exit', ...)` + // is kept registered — it needs to fire synchronously on the final exit. + if (this.sigtermListener) { + process.removeListener('SIGTERM', this.sigtermListener); + this.sigtermListener = null; + } + if (this.sigintListener) { + process.removeListener('SIGINT', this.sigintListener); + this.sigintListener = null; + } + + this.emit('stopped', { reason, exit_code: exitCode }); + process.exit(exitCode); + } + + /** + * Acquire PID file lock atomically via O_CREAT|O_EXCL. + * + * Returns: + * 'acquired' — lock is ours, safe to proceed. + * 'held' — another live supervisor owns the lock (exit code 2). + * 'unwritable' — can't write to the PID path (permission / missing parent, exit code 3). + */ + private acquirePidLock(): 'acquired' | 'held' | 'unwritable' { + // Ensure parent directory exists. Idempotent; creates ~/.gbrain on fresh installs. + try { + mkdirSync(dirname(this.opts.pidFile), { recursive: true }); + } catch (err: unknown) { + const code = (err as NodeJS.ErrnoException)?.code; + if (code !== 'EEXIST') { + console.error( + `Cannot create PID file directory ${dirname(this.opts.pidFile)}: ${ + err instanceof Error ? err.message : String(err) + }. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.` + ); + return 'unwritable'; + } + } + + return this.tryAtomicCreate(); + } + + private tryAtomicCreate(): 'acquired' | 'held' | 'unwritable' { + try { + // O_CREAT | O_EXCL | O_WRONLY — fails with EEXIST if the file exists. + const fd = openSync(this.opts.pidFile, 'wx'); + try { + writeSync(fd, String(process.pid)); + } finally { + closeSync(fd); + } + this.lockAcquired = true; + return 'acquired'; + } catch (err: unknown) { + const code = (err as NodeJS.ErrnoException)?.code; + if (code === 'EEXIST') { + // File exists — check if the owner is alive. + let existingPid = -1; + try { + const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0]; + existingPid = parseInt(contents, 10); + } catch { /* corrupt file */ } + + if (!isNaN(existingPid) && existingPid > 0 && isProcessAlive(existingPid)) { + console.error(`Supervisor already running (PID: ${existingPid}). Exiting.`); + return 'held'; + } + + // Stale PID file — unlink and retry atomic create once. + try { unlinkSync(this.opts.pidFile); } catch { /* race with another stale-cleaner; retry will EEXIST again */ } + try { + const fd = openSync(this.opts.pidFile, 'wx'); + try { + writeSync(fd, String(process.pid)); + } finally { + closeSync(fd); + } + this.lockAcquired = true; + return 'acquired'; + } catch (retryErr) { + const retryCode = (retryErr as NodeJS.ErrnoException)?.code; + if (retryCode === 'EEXIST') { + // Someone else won the race. Treat as held. + console.error(`Another supervisor took the PID lock during stale cleanup. Exiting.`); + return 'held'; + } + console.error( + `Cannot write PID file ${this.opts.pidFile}: ${ + retryErr instanceof Error ? retryErr.message : String(retryErr) + }` + ); + return 'unwritable'; + } + } + + console.error( + `Cannot write PID file ${this.opts.pidFile}: ${ + err instanceof Error ? err.message : String(err) + }. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.` + ); + return 'unwritable'; + } + } + + /** Run the supervise loop: spawn child, await exit, backoff+retry or give up. */ + private async runSuperviseLoop(): Promise { + while (!this.stopping && this.crashCount < this.opts.maxCrashes) { + await this.spawnOnce(); + + if (this.stopping) return; + + if (this.crashCount >= this.opts.maxCrashes) { + this.emit('max_crashes_exceeded', { + crash_count: this.crashCount, + max_crashes: this.opts.maxCrashes, + }); + await this.shutdown('max_crashes', ExitCodes.MAX_CRASHES); + return; + } + + // crashCount - 1 is the retry-attempt index (0-based exponent for backoff math). + // On first crash: crashCount=1, backoff exponent=0 → 1s. + // After stable-run reset: crashCount=1 again → 1s fresh cycle. + // Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration + // tests can exercise crash loops in < 1s without waiting for the real curve. + const backoff = this.opts._backoffFloorMs !== undefined + ? this.opts._backoffFloorMs + : calculateBackoffMs(this.crashCount - 1); + + this.emit('backoff', { ms: Math.round(backoff), crash_count: this.crashCount }); + + this.inBackoff = true; + try { + await new Promise(r => setTimeout(r, backoff)); + } finally { + this.inBackoff = false; + } + } + } + + /** Spawn the worker child once and await its exit. Updates `this.crashCount`. */ + private spawnOnce(): Promise { + return new Promise((resolve) => { + if (this.stopping) { resolve(); return; } + + const args = [ + 'jobs', 'work', + '--concurrency', String(this.opts.concurrency), + '--queue', this.opts.queue, + ]; + + // Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS: + // inherit only when caller opts in, otherwise strip from the clone. + const env: Record = { ...process.env }; + if (this.opts.allowShellJobs) { + env.GBRAIN_ALLOW_SHELL_JOBS = '1'; + } else { + delete env.GBRAIN_ALLOW_SHELL_JOBS; + } + + this.lastStartTime = Date.now(); + + let child: ChildProcess; + try { + child = spawn(this.opts.cliPath, args, { + stdio: 'inherit', + env, + }); + } catch (err: unknown) { + // Synchronous spawn error (e.g., invalid cliPath shape). Count as a crash. + this.emit('worker_spawn_failed', { + cli_path: this.opts.cliPath, + error: err instanceof Error ? err.message : String(err), + phase: 'sync', + }); + this.crashCount++; + resolve(); + return; + } + + this.child = child; + + this.emit('worker_spawned', { pid: child.pid, cli_path: this.opts.cliPath }); + + // Async spawn errors (ENOENT, EACCES after the fork/exec). Node fires + // 'error' first, then 'exit' with code=null. We log the error; the + // 'exit' handler increments crashCount as usual so the restart loop + // continues (max-crashes bounds this for permanent misconfigs). + child.on('error', (err) => { + this.emit('worker_spawn_failed', { + cli_path: this.opts.cliPath, + error: err.message, + code: (err as NodeJS.ErrnoException).code ?? 'unknown', + phase: 'async', + }); + }); + + child.on('exit', (code, signal) => { + this.child = null; + + if (this.stopping) { + resolve(); + return; + } + + // Stable-run reset: if the worker ran > 5min before crashing, we forgive + // prior crash history and treat this as the first crash of a new cycle + // (crashCount = 1, so backoff math uses retry-index 0 = 1s). + const runDuration = Date.now() - this.lastStartTime; + if (runDuration > 5 * 60 * 1000) { + this.crashCount = 1; + } else { + this.crashCount++; + } + + const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`; + this.emit('worker_exited', { + code: code ?? null, + signal: signal ?? null, + reason: exitReason, + crash_count: this.crashCount, + max_crashes: this.opts.maxCrashes, + run_duration_ms: runDuration, + }); + + resolve(); + }); + }); + } + + /** + * Periodic health check — queries DB for queue health indicators. + * + * POSTGRES-ONLY. The supervisor cannot run against PGLite (exclusive + * file lock blocks the separate worker process). The CLI layer rejects + * that combination; we assume Postgres here. + * + * F9 guard: skip if a previous check is still in flight (hung DB + * connection shouldn't stack duplicate checks). + */ + private async healthCheck(): Promise { + if (this.healthInFlight) return; + this.healthInFlight = true; + + try { + // Blocker 2+3+6: single FILTER query scoped to this.opts.queue. + // 'stalled' = active jobs whose lock_until has passed (matches + // queue.ts:848 handleStalled() definition — same set that the queue + // itself will requeue/dead-letter on next tick). + const rows = await this.engine.executeRaw<{ + stalled: string; + waiting: string; + last_completed: string | null; + }>( + `SELECT + count(*) FILTER (WHERE status = 'active' AND lock_until < now())::text AS stalled, + count(*) FILTER (WHERE status = 'waiting')::text AS waiting, + max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed + FROM minion_jobs + WHERE queue = $1`, + [this.opts.queue], + ); + + const row = rows[0] ?? { stalled: '0', waiting: '0', last_completed: null }; + const stalledCount = parseInt(row.stalled ?? '0', 10); + const waitingCount = parseInt(row.waiting ?? '0', 10); + const lastCompleted = row.last_completed ? new Date(row.last_completed) : null; + + const now = Date.now(); + const minutesSinceCompletion = lastCompleted + ? Math.round((now - lastCompleted.getTime()) / 60_000) + : null; + + // F2 (per-threshold warns) — each is a distinct health_warn with reason. + if (stalledCount > 10) { + this.emit('health_warn', { + reason: 'stalled_jobs', + count: stalledCount, + queue: this.opts.queue, + }); + } + + if (waitingCount > 0 && minutesSinceCompletion !== null && minutesSinceCompletion > 30) { + this.emit('health_warn', { + reason: 'no_recent_completions', + waiting_count: waitingCount, + minutes_since_completion: minutesSinceCompletion, + queue: this.opts.queue, + }); + } + + // F4: suppress "worker not alive" warn while we're in the expected + // null-child window (crash-exit → backoff-sleep → next-spawn). + const workerAlive = this.child != null && this.child.exitCode === null; + if (!workerAlive && !this.stopping && !this.inBackoff) { + this.emit('health_warn', { + reason: 'worker_not_alive', + queue: this.opts.queue, + }); + } + } catch (e) { + // Health check failures are non-fatal. + this.emit('health_error', { + error: e instanceof Error ? e.message : String(e), + queue: this.opts.queue, + }); + } finally { + this.healthInFlight = false; + } + } +} diff --git a/test/fixtures/supervisor-runner.ts b/test/fixtures/supervisor-runner.ts new file mode 100644 index 000000000..8387f6bbf --- /dev/null +++ b/test/fixtures/supervisor-runner.ts @@ -0,0 +1,63 @@ +/** + * Test fixture: spawns a MinionSupervisor with options parsed from env vars. + * + * Used by test/supervisor.test.ts integration tests. Separate file because + * the supervisor calls `process.exit()` at the end of its lifecycle — tests + * spawn this runner as a subprocess to observe exit codes and audit events + * without killing the test runner itself. + * + * Env vars (all optional, sensible defaults for tests): + * SUP_CLI_PATH — worker binary path (default: /bin/sh exit-1 script) + * SUP_PID_FILE — PID file path (REQUIRED; each test uses a unique one) + * SUP_MAX_CRASHES — max consecutive crashes (default: 3) + * SUP_BACKOFF_FLOOR_MS — test-only short backoff (default: 1) + * SUP_HEALTH_INTERVAL_MS — how often healthCheck fires (default: 999_999 off) + * SUP_ALLOW_SHELL_JOBS — "1" to set allowShellJobs:true, else false + * SUP_QUEUE — queue name (default: 'default') + * SUP_AUDIT_DIR — GBRAIN_AUDIT_DIR override (default: tmpdir/supervisor-test) + */ + +import { MinionSupervisor } from '../../src/core/minions/supervisor.ts'; +import { writeSupervisorEvent } from '../../src/core/minions/handlers/supervisor-audit.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; + +// Mock engine: healthCheck() calls engine.executeRaw; return empty rows so +// the query path exercises without needing Postgres. +const mockEngine: Partial = { + kind: 'postgres' as const, + executeRaw: async () => [], +} as unknown as BrainEngine; + +const pidFile = process.env.SUP_PID_FILE; +if (!pidFile) { + console.error('SUP_PID_FILE env var is required'); + process.exit(99); +} + +const cliPath = process.env.SUP_CLI_PATH ?? '/bin/sh'; +const maxCrashes = parseInt(process.env.SUP_MAX_CRASHES ?? '3', 10); +const backoffFloor = parseInt(process.env.SUP_BACKOFF_FLOOR_MS ?? '1', 10); +const healthInterval = parseInt(process.env.SUP_HEALTH_INTERVAL_MS ?? '999999', 10); +const allowShellJobs = process.env.SUP_ALLOW_SHELL_JOBS === '1'; +const queueName = process.env.SUP_QUEUE ?? 'default'; + +if (process.env.SUP_AUDIT_DIR) { + process.env.GBRAIN_AUDIT_DIR = process.env.SUP_AUDIT_DIR; +} + +const supervisorPid = process.pid; + +const supervisor = new MinionSupervisor(mockEngine as BrainEngine, { + concurrency: 1, + queue: queueName, + pidFile, + maxCrashes, + healthInterval, + cliPath, + allowShellJobs, + json: true, + _backoffFloorMs: backoffFloor, + onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid), +}); + +await supervisor.start(); diff --git a/test/supervisor.test.ts b/test/supervisor.test.ts new file mode 100644 index 000000000..272b5ed5c --- /dev/null +++ b/test/supervisor.test.ts @@ -0,0 +1,343 @@ +import { describe, it, expect, afterEach } from 'bun:test'; +import { existsSync, readFileSync, writeFileSync, unlinkSync, chmodSync, mkdirSync, rmSync } from 'fs'; +import { spawn } from 'child_process'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { readSupervisorEvents, computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts'; +import { calculateBackoffMs } from '../src/core/minions/supervisor.ts'; + +const TEST_PID_FILE = '/tmp/gbrain-supervisor-test.pid'; + +afterEach(() => { + try { unlinkSync(TEST_PID_FILE); } catch { /* noop */ } +}); + +// ----- Integration test helpers ----- + +interface IntegrationHarness { + pidFile: string; + auditDir: string; + workerScript: string; + envOutFile: string; + cleanup: () => void; +} + +/** Create per-test temp files + a fake worker shell script. */ +function makeHarness(name: string, workerBody: string): IntegrationHarness { + const tmpRoot = join(tmpdir(), `gbrain-sup-test-${name}-${process.pid}-${Date.now()}`); + mkdirSync(tmpRoot, { recursive: true }); + const pidFile = join(tmpRoot, 'supervisor.pid'); + const auditDir = join(tmpRoot, 'audit'); + const workerScript = join(tmpRoot, 'worker.sh'); + const envOutFile = join(tmpRoot, 'env-out.txt'); + + writeFileSync(workerScript, `#!/bin/sh\n${workerBody}\n`, 'utf8'); + chmodSync(workerScript, 0o755); + + return { + pidFile, + auditDir, + workerScript, + envOutFile, + cleanup: () => { try { rmSync(tmpRoot, { recursive: true, force: true }); } catch { /* noop */ } }, + }; +} + +/** + * Spawn the supervisor runner as a subprocess. Returns a handle with the + * child, a promise resolving to exit code + signal, and a kill helper. + */ +function spawnSupervisor(h: IntegrationHarness, overrides: Record = {}) { + const env: Record = { + ...(process.env as Record), + SUP_PID_FILE: h.pidFile, + SUP_CLI_PATH: h.workerScript, + SUP_AUDIT_DIR: h.auditDir, + SUP_BACKOFF_FLOOR_MS: '5', + SUP_MAX_CRASHES: '3', + SUP_HEALTH_INTERVAL_MS: '999999', // effectively off + ...overrides, + }; + + const child = spawn('bun', [join(import.meta.dir, 'fixtures/supervisor-runner.ts')], { + env, + stdio: ['ignore', 'pipe', 'pipe'], + }); + + let stdout = ''; + let stderr = ''; + child.stdout?.on('data', (d) => { stdout += d.toString(); }); + child.stderr?.on('data', (d) => { stderr += d.toString(); }); + + const exited = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => { + child.on('exit', (code, signal) => resolve({ code, signal })); + }); + + return { + child, + exited, + getStdout: () => stdout, + getStderr: () => stderr, + }; +} + +/** Read the audit JSONL for the current week. */ +function readAudit(auditDir: string) { + const origEnv = process.env.GBRAIN_AUDIT_DIR; + process.env.GBRAIN_AUDIT_DIR = auditDir; + try { + return readSupervisorEvents(); + } finally { + if (origEnv === undefined) delete process.env.GBRAIN_AUDIT_DIR; + else process.env.GBRAIN_AUDIT_DIR = origEnv; + } +} + +/** Poll until predicate returns true or deadline elapses. */ +async function waitFor(pred: () => boolean, timeoutMs: number, tickMs = 20): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (pred()) return true; + await new Promise(r => setTimeout(r, tickMs)); + } + return pred(); +} + +describe('MinionSupervisor', () => { + describe('calculateBackoffMs', () => { + it('returns ~1s for first crash', () => { + const backoff = calculateBackoffMs(0); + expect(backoff).toBeGreaterThanOrEqual(1000); + expect(backoff).toBeLessThan(1200); // 1000 + 10% jitter max + }); + + it('doubles with each crash', () => { + const b0 = calculateBackoffMs(0); + const b1 = calculateBackoffMs(1); + const b2 = calculateBackoffMs(2); + // Approximate: b1 should be ~2x b0, b2 ~2x b1 (within jitter) + expect(b1).toBeGreaterThan(1800); + expect(b2).toBeGreaterThan(3600); + }); + + it('caps at 60s', () => { + const backoff = calculateBackoffMs(20); // 2^20 * 1000 would be huge + expect(backoff).toBeLessThanOrEqual(66_000); // 60s + 10% jitter + }); + + it('includes jitter (not perfectly deterministic)', () => { + const values = new Set(); + for (let i = 0; i < 10; i++) { + values.add(Math.round(calculateBackoffMs(3))); + } + // With 10% jitter, we should get some variation + expect(values.size).toBeGreaterThan(1); + }); + }); + + describe('PID file management', () => { + it('detects stale PID files', () => { + // Write a PID file with a non-existent PID + writeFileSync(TEST_PID_FILE, '999999999'); + expect(existsSync(TEST_PID_FILE)).toBe(true); + + // A real supervisor would detect this as stale and overwrite + const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10); + let isAlive = false; + try { + process.kill(existingPid, 0); + isAlive = true; + } catch { + isAlive = false; + } + expect(isAlive).toBe(false); + }); + + it('detects live PID files (current process)', () => { + // Write our own PID + writeFileSync(TEST_PID_FILE, String(process.pid)); + + const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10); + let isAlive = false; + try { + process.kill(existingPid, 0); + isAlive = true; + } catch { + isAlive = false; + } + expect(isAlive).toBe(true); + expect(existingPid).toBe(process.pid); + }); + }); + + describe('crash count tracking', () => { + it('backoff escalates with crash count', () => { + const backoffs = []; + for (let i = 0; i < 7; i++) { + backoffs.push(calculateBackoffMs(i)); + } + // Each should be roughly 2x the previous (within jitter) + for (let i = 1; i < 6; i++) { + // The base doubles, so even with jitter the next should be > 1.5x previous + expect(backoffs[i]).toBeGreaterThan(backoffs[i - 1] * 1.5); + } + }); + }); + + // -------------------------------------------------------------- + // Integration tests: real spawn(), real signals, real audit file. + // Each test uses a unique tmpdir harness so they can run in parallel + // without colliding. `_backoffFloorMs: 5` (set via SUP_BACKOFF_FLOOR_MS) + // keeps the whole suite under a few seconds. + // -------------------------------------------------------------- + + describe('integration: crash → restart → max-crashes lifecycle', () => { + it('respawns the worker after a crash and eventually exits with max-crashes code=1', async () => { + // Worker always exits with code 1; supervisor should respawn it 3 times, + // hit max-crashes, then exit via shutdown() with code 1. + const h = makeHarness('max-crashes', 'exit 1'); + try { + const sup = spawnSupervisor(h, { SUP_MAX_CRASHES: '3' }); + const { code } = await sup.exited; + + expect(code).toBe(1); + + // PID file cleaned up on exit (synchronous process.on('exit') handler). + expect(existsSync(h.pidFile)).toBe(false); + + // Audit file should contain started + 3x worker_spawned/worker_exited + + // max_crashes_exceeded + shutting_down + stopped. + const events = readAudit(h.auditDir); + const eventTypes = events.map(e => e.event); + expect(eventTypes).toContain('started'); + expect(eventTypes.filter(t => t === 'worker_spawned').length).toBeGreaterThanOrEqual(3); + expect(eventTypes.filter(t => t === 'worker_exited').length).toBeGreaterThanOrEqual(3); + expect(eventTypes).toContain('max_crashes_exceeded'); + expect(eventTypes).toContain('shutting_down'); + expect(eventTypes).toContain('stopped'); + + // The stopped event should carry exit_code=1 and reason=max_crashes. + const stoppedEvt = events.filter(e => e.event === 'stopped').pop(); + expect((stoppedEvt as Record).exit_code).toBe(1); + expect((stoppedEvt as Record).reason).toBe('max_crashes'); + } finally { + h.cleanup(); + } + }, 15_000); + }); + + describe('integration: graceful SIGTERM during backoff', () => { + it('receives SIGTERM while sleeping between crashes and exits 0 cleanly', async () => { + // Worker always exits with code 1; supervisor has a high max-crashes + // and a long-enough backoff floor that we can reliably catch it mid-sleep. + const h = makeHarness('sigterm-backoff', 'exit 1'); + try { + const sup = spawnSupervisor(h, { + SUP_MAX_CRASHES: '100', + SUP_BACKOFF_FLOOR_MS: '800', // 800ms between restarts — enough to catch + }); + + // Wait until the supervisor has written the PID file AND survived at + // least one worker_exited (so it's definitely in the backoff sleep). + const ready = await waitFor(() => { + if (!existsSync(h.pidFile)) return false; + const events = readAudit(h.auditDir); + return events.some(e => e.event === 'worker_exited'); + }, 3000); + expect(ready).toBe(true); + + // Now SIGTERM the supervisor. It must exit cleanly within 200ms + // (short-circuits the 800ms backoff sleep via the stopping flag). + const sigSentAt = Date.now(); + sup.child.kill('SIGTERM'); + + const { code, signal } = await sup.exited; + const elapsed = Date.now() - sigSentAt; + + // Exit code 0 = clean; signal=null means we exited via process.exit, not got killed. + expect(code).toBe(0); + expect(signal).toBe(null); + // Graceful, not hung: exit within 5s (process.exit() through shutdown() + // should be near-instant; generous bound to tolerate CI slowness). + expect(elapsed).toBeLessThan(5000); + + const events = readAudit(h.auditDir); + const eventTypes = events.map(e => e.event); + expect(eventTypes).toContain('shutting_down'); + expect(eventTypes).toContain('stopped'); + + const shuttingEvt = events.filter(e => e.event === 'shutting_down').pop(); + expect((shuttingEvt as Record).reason).toBe('SIGTERM'); + + // PID file cleaned up. + expect(existsSync(h.pidFile)).toBe(false); + } finally { + h.cleanup(); + } + }, 20_000); + }); + + describe('integration: env-var inheritance regression (codex #9 / eng #8)', () => { + it('strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false, even if parent has it set', async () => { + const outFile = join(tmpdir(), `gbrain-sup-env-${process.pid}-${Date.now()}.txt`); + try { unlinkSync(outFile); } catch { /* may not exist */ } + + const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`); + + try { + const sup = spawnSupervisor(h, { + OUT_FILE: outFile, + GBRAIN_ALLOW_SHELL_JOBS: '1', // parent has it + SUP_ALLOW_SHELL_JOBS: '0', // supervisor says NO + SUP_MAX_CRASHES: '1', + }); + + await sup.exited; + + // Worker should have written "UNSET" (parent env var stripped from child). + expect(existsSync(outFile)).toBe(true); + const childSawEnv = readFileSync(outFile, 'utf8').trim(); + expect(childSawEnv).toBe('UNSET'); + } finally { + try { unlinkSync(outFile); } catch { /* noop */ } + h.cleanup(); + } + }, 15_000); + + it('DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs is true', async () => { + const outFile = join(tmpdir(), `gbrain-sup-env-ok-${process.pid}-${Date.now()}.txt`); + try { unlinkSync(outFile); } catch { /* may not exist */ } + + const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`); + + try { + const sup = spawnSupervisor(h, { + OUT_FILE: outFile, + SUP_ALLOW_SHELL_JOBS: '1', + SUP_MAX_CRASHES: '1', + }); + + await sup.exited; + + expect(existsSync(outFile)).toBe(true); + expect(readFileSync(outFile, 'utf8').trim()).toBe('1'); + } finally { + try { unlinkSync(outFile); } catch { /* noop */ } + h.cleanup(); + } + }, 15_000); + }); + + describe('integration: audit file rotation + helper', () => { + it('computeSupervisorAuditFilename returns supervisor-YYYY-Www.jsonl format', () => { + const jan15_2026 = new Date(Date.UTC(2026, 0, 15)); // Thu + expect(computeSupervisorAuditFilename(jan15_2026)).toMatch(/^supervisor-2026-W\d\d\.jsonl$/); + }); + + it('year-boundary ISO week: 2027-01-01 reports as 2026-W53', () => { + const jan1_2027 = new Date(Date.UTC(2027, 0, 1)); + // ISO week: 2027-01-01 is Friday of W53 of 2026 + expect(computeSupervisorAuditFilename(jan1_2027)).toBe('supervisor-2026-W53.jsonl'); + }); + }); +});