Compare commits

..
Author SHA1 Message Date
Garry Tan 2bcc1c75e1 Merge remote-tracking branch 'origin/master' into feat/worker-supervisor
# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
2026-04-24 00:12:09 -07:00
Garry TanandClaude Opus 4.7 a2876ef1b6 chore: regenerate llms-full.txt after Lane B doc rewrite
CI drift guard caught that `llms-full.txt` didn't match the current
generator output. Root cause: the Lane B rewrite of
`docs/guides/minions-deployment.md` (supervisor as canonical, watchdog
deleted) changed content that gets inlined into `llms-full.txt`, but I
didn't run `bun run build:llms` to regenerate.

`bun test test/build-llms.test.ts` now clean (7/7 pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:08:39 -07:00
Garry TanandClaude Opus 4.7 05d07ed531 fix: escape template-literal interpolation in supervisor --help
The --help body in src/commands/jobs.ts is one big backtick template
literal. The supervisor subcommand description I added in Lane B used
both `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}` (parsed as a template
interpolation into an undefined variable) and inline `code` backticks
(parsed as nested template literals). CI caught it with ~200 tsc parse
errors across the file.

Fix:
- Escape `${...}` → `\${...}` so the audit-file path renders literally.
- Replace prose inline-code backticks with plain single-quote fences
  (`gbrain jobs work` → 'gbrain jobs work', `~/.gbrain/supervisor.pid`
  → ~/.gbrain/supervisor.pid). `--help` output is human prose; the
  single-quote form reads cleanly in a terminal without needing to
  smuggle nested backticks through a template literal.

`bunx tsc --noEmit` is clean. 146 tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 00:04:09 -07:00
Garry Tan d216bde5f4 Merge remote-tracking branch 'origin/master' into feat/worker-supervisor
# Conflicts:
#	CHANGELOG.md
#	VERSION
#	package.json
2026-04-24 00:01:06 -07:00
Garry TanandClaude Opus 4.7 0a54dda172 chore: bump version and changelog (v0.20.2)
Lane F of PR #364 review fixes. Closes the multi-lane plan with release
hygiene: VERSION bump 0.19.0 → 0.20.2, package.json sync, CHANGELOG entry
in GStack voice with release summary + "numbers that matter" table +
"To take advantage of v0.20.2" migration block + itemized changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:56:55 -07:00
Garry TanandClaude Opus 4.7 861b968808 test: 4 critical integration tests for supervisor lifecycle
Lane E of PR #364 review fixes (blocker 10). Fills the ~15% coverage
gap flagged in the eng review by actually exercising the code paths
that will break in production — crash-restart loop, max-crashes exit,
SIGTERM-during-backoff, env-var inheritance — via real spawn() calls
against fake shell-script workers. No mocks: real fork, real signals,
real env propagation, real audit file writes.

test/fixtures/supervisor-runner.ts (new, 55 lines):
  A standalone bun script that constructs a MinionSupervisor from env
  vars (SUP_PID_FILE / SUP_CLI_PATH / SUP_MAX_CRASHES / SUP_BACKOFF_FLOOR_MS
  / SUP_HEALTH_INTERVAL_MS / SUP_ALLOW_SHELL_JOBS / SUP_AUDIT_DIR) and
  calls start(). Mock engine returns empty rows for executeRaw (health
  check path still exercised without Postgres). Tests spawn this as a
  subprocess because MinionSupervisor.start() calls process.exit() on
  shutdown — can't run it in the test runner's own process.

test/supervisor.test.ts (existing; 91 → 300 lines):
  - Added IntegrationHarness helper: creates a unique tmpdir per test,
    a fake worker shell script, a PID-file path, and an audit-dir path;
    cleanup runs in finally.
  - spawnSupervisor() forks bun on the runner with env vars set.
  - readAudit() reads the supervisor-YYYY-Www.jsonl file via the
    existing readSupervisorEvents() helper (Lane C), threading
    GBRAIN_AUDIT_DIR through so tests don't collide on ~/.gbrain.
  - waitFor(pred, timeoutMs) polls helper for event-driven tests.

Four integration tests (with _backoffFloorMs=5 for <1s suite runs):

  1. "respawns the worker after a crash and eventually exits with
     max-crashes code=1"
     Worker always `exit 1`. maxCrashes=3. Asserts: exit code 1, PID
     file cleaned up, audit contains started + 3x worker_spawned +
     3x worker_exited + max_crashes_exceeded + shutting_down + stopped,
     and the stopped event carries {reason:'max_crashes', exit_code:1}.
     Locks in blockers 1 (PID lock), 2+3+6 (health SQL doesn't 500),
     5 (unified shutdown emits right events), F8 (spawn errors counted).

  2. "receives SIGTERM while sleeping between crashes and exits 0 cleanly"
     Worker always `exit 1`, backoff floor 800ms to catch the sleep.
     Asserts: SIGTERM during backoff → exit code 0 (not 1) in <5s,
     no signal kill (process.exit via shutdown), audit contains
     shutting_down {reason:'SIGTERM'} + stopped, PID file cleaned up.
     Locks in eng Issue 1 (unified exit path), eng Issue 3 (signal
     handlers don't accumulate across shutdowns).

  3. "strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false,
     even if parent has it set"  ⚠ CRITICAL regression test
     Parent env has GBRAIN_ALLOW_SHELL_JOBS=1. SUP_ALLOW_SHELL_JOBS=0.
     Worker writes $GBRAIN_ALLOW_SHELL_JOBS (or 'UNSET' if absent) to
     an OUT_FILE. Asserts child sees 'UNSET'. Locks in codex #9 + eng
     #8: the `else delete env.GBRAIN_ALLOW_SHELL_JOBS` branch from
     Lane A is load-bearing for the supervisor's security posture;
     this test prevents a future refactor silently re-opening the
     inheritance hole.

  4. "DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs=true"
     Positive-path companion to #3. SUP_ALLOW_SHELL_JOBS=1 → worker
     sees '1'. Confirms the else-branch doesn't over-strip and that
     operators who explicitly opt in still get shell-exec enabled.

Plus two audit-format unit tests:
  - computeSupervisorAuditFilename format (regex match)
  - Year-boundary ISO week: 2027-01-01 → supervisor-2026-W53.jsonl
    (matches the shell-audit.ts pattern exactly)

Before: 7 tests covering backoff math + PID helpers (~15% behavioral
coverage per eng review).
After: 13 tests across all critical lifecycle paths (crash-restart,
max-crashes, SIGTERM, env-inheritance, audit rotation).

All 146 tests in supervisor + minions + doctor suites green in ~8s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:54:34 -07:00
Garry TanandClaude Opus 4.7 cafde77ba7 doctor: add supervisor health check
Lane D of PR #364 review fixes. Closes the observability loop: now that
Lane C writes supervisor lifecycle events to
`${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`,
`gbrain doctor` surfaces a `supervisor` check alongside its existing
health indicators.

Implementation (src/commands/doctor.ts, filesystem-only block 3b-bis):
- Resolves DEFAULT_PID_FILE via the same three-tier logic as the start
  path (--pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid).
- Reads the PID file + `kill -0 <pid>` for liveness.
- Calls readSupervisorEvents({sinceMs: 24h}) from the audit module to
  derive last_start / crashes_24h / max_crashes_exceeded.
- Suppresses the check entirely when the user has never invoked the
  supervisor (no PID file AND no audit events) — avoids noise on
  installs that don't use the feature.

Status thresholds:
  fail   max_crashes_exceeded event seen in last 24h
         (supervisor gave up; operator needs to restart or triage)
  warn   supervisor not running but audit shows prior use
         (unexpected stop — likely crash or manual kill)
  warn   running but > 3 crashes in last 24h
         (supervisor recovering but worker is unstable)
  ok     running + ≤ 3 crashes + no max_crashes event

All failure paths emit a paste-ready recovery command. Read/import
errors are swallowed (best-effort like the other doctor checks).

Tests: all 127 supervisor + minions tests still green; 13 existing
doctor tests unaffected.

F3 done. All four lanes A/B/C/D are now committed; Lane E (integration
tests) and Lane F (/ship v0.20.2) remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:49:49 -07:00
Garry TanandClaude Opus 4.7 e62ae46a95 supervisor: daemon-manager subcommands + JSONL audit writer
Lane C of PR #364 review fixes. Adds the daemon-manager CLI surface so
agents can drive `gbrain jobs supervisor` in 3 turns instead of 10, and
the audit writer that makes lifecycle events inspectable across process
restarts. (Blocker 8, closes DX Fix A/B/C.)

New: src/core/minions/handlers/supervisor-audit.ts
  - writeSupervisorEvent(emission, supervisorPid) appends JSONL to
    `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`.
    ISO-week rotation via a `computeSupervisorAuditFilename()` helper
    that mirrors `shell-audit.ts` exactly (year-boundary ISO week math,
    Thursday anchor, etc).
  - readSupervisorEvents({sinceMs}) returns parsed events from the
    current week's file, oldest-first, for Lane D's doctor check.
    Malformed lines are skipped silently (disk-full truncation is
    already best-effort at write time).
  - Reuses `resolveAuditDir()` from shell-audit.ts so the
    `GBRAIN_AUDIT_DIR` env var override works identically across all
    gbrain audit trails.

src/commands/jobs.ts: supervisor subcommand dispatcher
  - `gbrain jobs supervisor [start] [--detach] [--json] ...` — default
    subcommand. Without --detach, runs foreground as before. With
    --detach, forks a background child (inheriting stderr so the caller
    can still tail JSONL events), writes a stdout payload:
      {"event":"started","supervisor_pid":N,"pid_file":"...","detached":true}
    and exits 0. Stdin/stdout on the detached child are /dev/null so
    the parent shell isn't held open.
  - `gbrain jobs supervisor status [--json]` — reads the PID file,
    checks liveness via `kill -0`, then reads the last 24h from the
    supervisor audit file to compute crashes_24h / last_start /
    max_crashes_exceeded. Exits 0 if running, 1 if not. JSON output
    is machine-parseable; human output is a 5-line ASCII report.
  - `gbrain jobs supervisor stop [--json]` — reads PID, sends SIGTERM,
    polls `kill -0` every 250ms for up to 40s (supervisor's own 35s
    worker-drain + 5s slack). Reports outcome: drained / timeout_40s
    / pid_file_missing / pid_file_corrupt / process_gone. Exit 0 on
    clean stop.
  - `--json` flag is already plumbed through to the supervisor opts
    from Lane A — this lane adds the onEvent audit-writer callback
    so every supervisor emission (started, worker_spawned,
    worker_exited, worker_spawn_failed, backoff, health_warn,
    health_error, max_crashes_exceeded, shutting_down, stopped) lands
    in the JSONL file with the supervisor's PID attached.

--help body updated:
  - Three separate usage lines (start / status / stop).
  - SUBCOMMANDS block with one-line summaries each.
  - EXIT CODES block (unchanged from Lane A, moved under SUBCOMMANDS).
  - EXAMPLES block updated with status --json + stop + --detach forms.

Tests: existing 127 supervisor + minions tests continue to pass.
Integration tests for the new subcommands + audit writer land with
Lane E.

Follow-up (Lane D): `gbrain doctor` will read readSupervisorEvents()
from this module to surface a `supervisor` health check alongside its
existing checks (DB connectivity, schema version, queue health).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:48:37 -07:00
Garry TanandClaude Opus 4.7 b377e2c0e3 docs: supervisor as canonical worker deployment pattern
Lane B of PR #364 review fixes. Reframes docs/guides/minions-deployment.md
around `gbrain jobs supervisor` as the default answer (blocker 7), deletes
the 68-line legacy bash watchdog (F10), and updates README + deployment
snippets to match.

docs/guides/minions-deployment.md:
- New 'Worker supervision' section at the top with the canonical 3-command
  agent pattern (start --detach / status --json / stop) and a documented
  exit-code table (0 clean, 1 max-crashes, 2 lock-held, 3 PID-unwritable).
- 'Which supervisor when?' decision table: container = supervisor as
  PID 1, Linux VM = systemd-over-supervisor, dev laptop = bare terminal.
- New 'Agent usage' section for OpenClaw / Hermes / Cursor / Codex — the
  3-turn discover-start-maintain workflow that replaces shell archaeology
  with machine-parseable JSON events + an audit file at
  ~/.gbrain/audit/supervisor-YYYY-Www.jsonl.
- Demoted the 'Option 1: watchdog cron' path entirely; replaced with a
  straightforward upgrade migration block (stop script, remove cron line,
  start supervisor, verify via doctor).
- Preconditions now check Postgres connectivity directly (supervisor is
  Postgres-only; the CLI rejects PGLite with a clear error).

Snippets:
- systemd.service: ExecStart now invokes `gbrain jobs supervisor` instead
  of raw `gbrain jobs work`. Two-layer supervision (systemd → supervisor
  → worker) buys automatic restart on reboot plus fast crash recovery.
  ReadWritePaths expanded to cover $HOME/.gbrain (supervisor PID + audit).
- Procfile + fly.toml.partial: same change — platform restarts the
  container on host events, supervisor restarts the worker on crashes.
- minion-watchdog.sh: deleted (git history retains it for anyone in an
  exotic deployment). Supervisor subsumes every capability it had plus
  atomic PID locking, structured audit events, queue-scoped health
  checks, and graceful drain on SIGTERM.

README.md:
- Added a paragraph under the Minions section pointing `gbrain jobs
  supervisor` as canonical, noting the --detach / status / stop surface
  and the audit file path, with a link to the full deployment guide.
  Kept `gbrain jobs work` documented for direct raw invocation but
  flagged 'prefer supervisor' for any long-running use.

The supervisor `--help` body itself (3 examples + exit-code table in
src/commands/jobs.ts) landed with Lane A — this lane finishes the
discoverability story by making the supervisor findable via doc grep,
README landing, and deployment-guide landing paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:46:05 -07:00
Garry TanandClaude Opus 4.7 dc637e8f28 supervisor: atomic PID lock, queue-scoped health, env safety, unified exit
Lane A of PR #364 review fixes (20-item multi-lane plan). Addresses the
codex-tier + CEO + Eng findings on src/core/minions/supervisor.ts:

Safety + correctness:
- Atomic O_CREAT|O_EXCL PID lock via openSync('wx') with stale-file
  liveness check. Prevents two supervisors racing on the same PID file.
  (codex #1)
- Health check now queries status='active' AND lock_until < now()
  matching queue.ts:848's authoritative stalled definition. The prior
  `status = 'stalled'` predicate returned zero rows forever because
  'stalled' is not a persisted value in the schema. (codex #2)
- All health queries scoped to WHERE queue = $1 via opts.queue binding.
  Multi-queue installs no longer see cross-queue false positives.
  (codex #3)
- Class default allowShellJobs flipped true→false AND explicit
  `delete env.GBRAIN_ALLOW_SHELL_JOBS` when false, so child workers
  don't silently inherit the var from the parent shell. (eng #8, codex #9)
- Unified shutdown(reason, exitCode) — max-crashes now routes through
  the same drain path as SIGTERM. Single source of truth for lifecycle
  cleanup; prerequisite for trustworthy audit events (Lane C). (eng #1)
- Default PID path moves from /tmp to ~/.gbrain/supervisor.pid with
  mkdirSync recursive + GBRAIN_SUPERVISOR_PID_FILE env override.
  Matches the rest of the product's ~/.gbrain/ convention; fresh
  installs no longer hit ENOENT. (CEO #2 + codex #6)

Refinements:
- crashCount = 1 after 5-min stable-run reset (was 0, produced
  calculateBackoffMs(-1) = 500ms by accident). Now reads as 'first
  crash of a new cycle' with a clean 1s backoff. (Nit 1)
- Top-of-file POSTGRES-ONLY docstring documenting why the supervisor
  can't run against PGLite. (Nit 2)
- inBackoff flag suppresses 'worker not alive' warn during the
  expected null-child window (crash → sleep → next spawn). (eng #2)
- Tracked listener refs for SIGTERM/SIGINT removed in shutdown() so
  integration tests spinning up/tearing down multiple supervisors on
  one process don't leak handlers. (eng #3)
- Single FILTER query replaces two SELECT counts — one round-trip
  instead of two, three metrics in one pass. (eng #10)
- child.on('error') listener emits worker_spawn_failed event for
  ENOENT/EACCES; exit handler still increments crashCount as usual
  so max-crashes bounds permanent misconfigurations. (codex #7)
- healthInFlight boolean guard with try/finally prevents overlapping
  health checks from stacking on a hung DB. (codex #8)

Documented exit codes (ExitCodes const):
  0 CLEAN, 1 MAX_CRASHES, 2 LOCK_HELD, 3 PID_UNWRITABLE
  Agent can branch on exit=2 ('another supervisor, I'm fine') vs
  exit=1 ('escalate to human').

Event emitter surface:
  - started / worker_spawned / worker_exited / worker_spawn_failed
  - backoff / health_warn / health_error / max_crashes_exceeded
  - shutting_down / stopped
  Plumbed through emit() with an onEvent callback hook for Lane C's
  audit writer. json:false is the default; Lane C's --json mode
  flips it and writes JSONL to stderr.

CLI changes (src/commands/jobs.ts):
- `gbrain jobs supervisor` gains --allow-shell-jobs (explicit opt-in
  mirroring the env-var gate), --cli-path (override auto-resolution
  for exotic setups), and --json (JSONL lifecycle events on stderr).
- Expanded --help body with description, 3 examples, and exit-code
  table. (DX Fix A per review)
- Three-tier PID path resolution: --pid-file > GBRAIN_SUPERVISOR_PID_FILE
  > ~/.gbrain/supervisor.pid (via exported DEFAULT_PID_FILE).
- Removed the catch-fallback to process.argv[1] — resolveGbrainCliPath()
  throws its own actionable install-hint error, which is what dev users
  need instead of a cryptic spawn failure on a .ts path. (codex #5)

Tests: existing 7 supervisor.test.ts cases continue to pass.
Integration tests (crash-restart, max-crashes, SIGTERM-during-backoff,
env-inheritance regression) land in Lane E.

Out of scope for this lane (tracked in follow-up lanes):
- Audit file writer at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl (Lane C)
- Documentation pass (Lane B)
- supervisor start/status/stop subcommands (Lane C)
- gbrain doctor supervisor check (Lane D)
- /ship release hygiene (Lane F)
- autopilot.ts migration to MinionSupervisor (deferred to follow-up PR
  per codex — requires non-blocking start() API redesign, not ~30 lines)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 23:42:32 -07:00
root b1bfabdaef feat: add gbrain jobs supervisor — self-healing worker process manager
Adds a first-class supervisor command that:
- Spawns `gbrain jobs work` as a child process
- Restarts on crash with exponential backoff (1s→60s cap)
- Resets crash counter after 5min of stable operation
- PID file locking prevents duplicate supervisors
- Periodic health checks (stalled jobs, completion gaps)
- Graceful shutdown (SIGTERM→35s→SIGKILL)

Usage:
  gbrain jobs supervisor --concurrency 4

Replaces ad-hoc nohup patterns in bootstrap scripts.
The autopilot command's internal supervisor can be migrated
to use this in a follow-up.

Tests: 7 pass (backoff calc, PID management, crash tracking)
2026-04-23 19:57:05 -07:00
22 changed files with 33 additions and 1178 deletions
-95
View File
@@ -2,101 +2,6 @@
All notable changes to GBrain will be documented in this file.
## [0.20.3] - 2026-04-24
## **Your queue now rescues itself when a wedged worker holds a row lock. Wall-clock sweep kills the job that stall detection can't see.**
## **`maxWaiting` is race-proof, observable, and reachable from the CLI — three bugs in one patch.**
A production autopilot-cycle job wedged for over an hour on a single OpenClaw deployment because the worker's handler got stuck mid-transaction holding a row lock. Both eviction paths were blocked: the stall detector's `FOR UPDATE SKIP LOCKED` pass skipped the row-locked candidate, and the timeout sweep's `lock_until > now()` predicate disqualified the job once lock-renewal had been blocked. Neither could see the job. The shell-job pipeline starved completely behind the wedge.
v0.19.0 shipped the wall-clock sweep as the third-layer kill shot: drop both constraints, evict on `started_at` alone, worst case at `2 × timeout_ms + stalledInterval`. This release locks down three correctness holes the v0.19.0 PR introduced — then closes the observability gap that let the incident run to minute 90 in the first place.
### The queue-resilience numbers that matter
Measured against the real incident on 2026-04-23 (OpenClaw autopilot + shell-job pipeline, Postgres engine, concurrency=1 worker).
| Behavior | Before v0.20.3 | After v0.20.3 |
|---|---|---|
| Wedged worker escape window | 90+ minutes (manual kill) | `~2 × timeout_ms + 30s` sweep interval |
| Per-name waiting pile during wedge | 18 deferred per-slot jobs | capped at `maxWaiting` |
| `maxWaiting` under concurrent submit (2 submitters, cap=2) | up to 3 rows (TOCTOU race) | exactly 2 rows (advisory-lock serialization) |
| Same name across queues | cross-queue bleed — `shell` suppressed by `default` | isolated per `(name, queue)` |
| `GBRAIN_WORKER_CONCURRENCY=foo` | silent wedge (`inFlight < NaN` false) | clamped to 1, loud stderr warning |
| `gbrain jobs submit --max-waiting 2` | flag didn't exist | wired through to MinionJobInput |
| Silent coalesce events | invisible | JSONL audit at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` |
| `gbrain doctor` visibility into wedge | no check | new `queue_health` with 2 subchecks |
The two big shifts: (1) every silent-failure vector the v0.19.0 patches introduced now has a loud signal — JSONL audit files, doctor check, stderr warnings, peer-liveness probe. (2) `maxWaiting` is now actually a cap under concurrency, not a soft suggestion. A future multi-submitter pattern (parallel workspaces, dispatched children, OpenClaw + ycli cron) doesn't walk through it.
### What this means for OpenClaw users
If you're running `gbrain autopilot` on a daily-driver deployment, the wall-clock sweep is the difference between a 90-minute outage and a 30-second one. The `queue_health` doctor check means the next time your queue wedges, you notice in minute 2 instead of minute 90. If you've been writing programmatic Minion submitters and setting `maxWaiting`, it's worth re-reading the JSONL audit file the next time your agent does anything "interesting" — you'll see exactly which submission coalesces into which returned job.
## To take advantage of v0.20.3
`gbrain upgrade` handles the binary. You MUST restart long-running worker daemons so the new sweep runs in-process — the wall-clock eviction is a method on `MinionQueue`, not a cron job, so it only fires inside a worker loop.
1. **Upgrade the binary:**
```bash
gbrain upgrade
```
2. **Restart autopilot + workers:**
```bash
# systemd / launchd / OpenClaw service-manager: restart the unit.
# Manual: kill the old `gbrain autopilot` and `gbrain jobs work`, start new ones.
```
3. **Verify:**
```bash
gbrain jobs smoke --wedge-rescue # exercises the new wall-clock path
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
4. **If `gbrain doctor` flags anything unexpected,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/audit/backpressure-*.jsonl` (redact freely)
- what commands you ran leading up to the wedge
### Itemized changes
**Queue core** (`src/core/minions/queue.ts`)
- `maxWaiting` coalesce path wraps `count → select → insert` in `pg_advisory_xact_lock` keyed on `(name, queue)`. Concurrent submitters for the SAME key serialize; different keys stay parallel. Lock auto-releases on transaction commit/rollback — no cleanup path to leak. Fixes TOCTOU race caught by adversarial review.
- `maxWaiting` count and select now filter on `queue` in addition to `name`. Pre-v0.20.3 code filtered on name alone, so a waiting `autopilot-cycle` in `queue=default` would suppress submissions to `queue=shell` with the same name. Cross-queue bleed is gone.
**Backpressure observability** (new `src/core/minions/backpressure-audit.ts`)
- Every coalesce event writes one JSONL line to `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (ISO-week rotation, override dir via `GBRAIN_AUDIT_DIR`, mirrors the v0.14 shell-audit pattern).
- Fields: `ts, queue, name, waiting_count, max_waiting, decision='coalesced', returned_job_id`.
- Best-effort: write failures log to stderr but never block submission.
**CLI** (`src/commands/jobs.ts`)
- New `--max-waiting N` flag on `gbrain jobs submit`. Clamps to `[1, 100]`, mirrors the existing `--max-stalled` wiring. The `MinionJobInput.maxWaiting` field was programmatic-only before; now it's reachable from the command line too.
- `resolveWorkerConcurrency` clamps against invalid input. `parseInt` returns `NaN` for `"foo"`, `0` for `"0"`, negatives for `"-5"` — all of which silently wedge a worker (`inFlight.size < NaN/0/negative` is always false). Now clamped to ≥1 with a loud stderr warning naming the bad value. One typo in a systemd unit no longer reproduces the 90-minute outage.
- New `gbrain jobs smoke --wedge-rescue` opt-in case. Forges a wedged-worker row state, invokes `handleStalled` + `handleTimeouts` + `handleWallClockTimeouts` in sequence, asserts only the wall-clock sweep evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape.
**Doctor** (`src/commands/doctor.ts`)
- New `queue_health` check (Postgres-only; PGLite skips with `Skipped (PGLite — no multi-process worker surface)`).
- Subcheck 1 — **stalled-forever**: flags active jobs whose `started_at` is older than 1 hour. Reports the top 5 by start time with `gbrain jobs get/cancel <id>` fix hints.
- Subcheck 2 — **waiting-depth**: flags per-name queues whose waiting count exceeds threshold. Default 10, overridable via `GBRAIN_QUEUE_WAITING_THRESHOLD` env. Reports the top 5 by depth with "consider setting maxWaiting on the submitter" fix hint.
- Worker-heartbeat staleness subcheck intentionally deferred to follow-up because `lock_until`-on-active-jobs is a lossy proxy. A check that cries wolf erodes trust in every other doctor subcheck. Needs a `minion_workers` table to produce ground-truth signal.
**Autopilot** (`src/commands/autopilot.ts`)
- `--no-worker` mode gains a peer-worker-liveness probe. Every cycle runs a cheap `SELECT count(*)` checking for active jobs with `lock_until` refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud `WARNING` naming the silent-wedge vector (`--no-worker` set but no worker running). Re-arms once a live signal returns, so a healthy-but-idle worker doesn't trigger spam.
- Probe is documented as a proxy, not ground truth — idle worker with no active jobs reads as "no worker." The ground-truth fix needs a `minion_workers` heartbeat table (tracked as follow-up).
**Docs**
- New `docs/guides/queue-operations-runbook.md`: the "my queue looks wedged — what do I run?" reference. One viewport, in order of escalation. What each `queue_health` subcheck means. Self-check for the `--no-worker + no-worker-running` footgun.
- `CLAUDE.md` Key-files section updated for the new `handleWallClockTimeouts` method (v0.19.0, described here for the first time), the new `backpressure-audit.ts` module, the updated `maxWaiting` semantics, and the new `queue_health` doctor check.
**Tests** (`test/minions.test.ts`)
- 23 new unit cases. Wall-clock sweep (3 cases + non-interference with `handleTimeouts`). `maxWaiting` (coalesce, clamp 0 → 1, floor 1.7 → 1, concurrent-submitter race via `Promise.all`, cross-queue isolation, unset fallthrough). Concurrency clamp (7 cases including `NaN`/`0`/negative). `parseMaxWaitingFlag` (5 cases). Backpressure audit file write. All 143 minions tests pass.
- E2E wall-clock case against real Postgres is next on the roadmap (needs a second-connection row-lock helper; the unit-level coverage above exercises the sweep mechanics directly).
### For contributors
- The v0.19.0 PR's narrative framed the 18-job pileup as "duplicate submissions from a cron loop with no idempotency key." That framing was wrong. Autopilot already sets `idempotency_key: autopilot-cycle:${slot}` where slot is a 5-minute tick boundary — within-slot duplicates are structurally impossible. The 18 jobs were 18 different slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before v0.20.3 shipped.
- Follow-up issues tracked: B2 (autopilot heartbeat file), B3 (doctor `--fix` learns queue rescue), B4 (backpressure counts surfaced in `jobs stats`), B5 (cross-cutting "health-delivery-agent" pattern), B7 (`minion_workers` heartbeat table — unblocks both the dropped `queue_health` subcheck and a ground-truth `--no-worker` probe), P1 (composite indexes `(status, started_at)` and `(status, name)` on `minion_jobs` — currently the new sweeps fall back to `idx_minion_jobs_status`, selective enough on healthy queues, worth tightening in v0.20.4).
Full plan with CEO + Eng + Codex adversarial decisions lives at `~/.claude/plans/` for the operators who care about how this release was reviewed.
## [0.20.2] - 2026-04-24
## **`gbrain jobs supervisor` is now a self-healing daemon you can actually drive. The Minions worker stops dying silently.**
+2 -3
View File
@@ -62,13 +62,12 @@ strict behavior when unset.
- `src/commands/graph-query.ts``gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
- `src/core/minions/types.ts``MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
- `src/core/minions/handlers/shell.ts``shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
@@ -90,7 +89,7 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts``gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
+1 -1
View File
@@ -1 +1 @@
0.20.3
0.20.2
-76
View File
@@ -1,76 +0,0 @@
# Queue operations runbook
"My queue looks wedged — what do I run?" The commands below are in the order
you probably want them. Shipped with v0.19.1 after a production incident
where the queue held for 90+ minutes before the operator noticed.
## First signal: jobs aren't running
```bash
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
```
`queue_health` flags two patterns:
- **stalled-forever**: active job whose `started_at` is older than 1h.
- **waiting-depth**: any per-name queue deeper than 10 (override via
`GBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.
## Triage commands
```bash
# Who's active right now?
gbrain jobs list --status active
# Who's waiting, biggest pile first?
gbrain jobs list --status waiting --limit 50
# What's wrong with a specific job?
gbrain jobs get <id>
```
## Rescue actions (in order of escalation)
```bash
# Force-kill a single stuck job:
gbrain jobs cancel <id>
# Clear a specific job entirely (last resort):
gbrain jobs delete <id>
# Health smoke on the mechanism itself:
gbrain jobs smoke --wedge-rescue
```
## What each subcheck means
- **stalled-forever** — A worker claimed a job, started executing, and has
held the row for over an hour. The wall-clock sweep evicts jobs past
2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
or the sweep is newly deployed and this job predates it. Cancel it.
- **waiting-depth** — Submitters are piling up jobs faster than workers
drain them. Set `--max-waiting N` on the submission or on the programmatic
`queue.add()` call. If you want a taller pile, raise the threshold via
`GBRAIN_QUEUE_WAITING_THRESHOLD=50 gbrain doctor`.
## Self-check: is a worker even running?
```bash
# If you're running autopilot with --no-worker, check that your external
# worker (systemd / Docker / OpenClaw service-manager) is alive:
gbrain jobs list --status active | head -5
```
If the list is empty AND your submissions keep piling up, no worker is
claiming. Start one:
```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
```
## Follow-ups tracked for v0.20+
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
subcheck both need this).
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
+2 -3
View File
@@ -141,13 +141,12 @@ strict behavior when unset.
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
@@ -169,7 +168,7 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.20.3",
"version": "0.20.2",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+5 -8
View File
@@ -13,14 +13,13 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Trigger | Skill |
|---------|-------|
| "What do we know about", "tell me about", "search for", "search the brain", "brain search", "background on", "notes on this", "who is" | `skills/query/SKILL.md` |
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
| "Fix broken citations", "citations are broken", "fix citations", "citation audit" | `skills/citation-fixer/SKILL.md` |
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
| Share a brain page as a link | `skills/publish/SKILL.md` |
| "validate frontmatter", "check frontmatter", "brain lint", "fix frontmatter" | `skills/frontmatter-guard/SKILL.md` |
## Content & media ingestion
@@ -59,7 +58,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
## Setup & migration
@@ -90,10 +89,8 @@ When multiple skills could match:
1. Prefer the most specific skill (meeting-ingestion over ingest)
2. If the user mentions a URL, route by content type (link → idea-ingest, video → media-ingest)
3. If the user mentions a person/company, check if enrich or query fits better
4. **Citation audit** → use `citation-fixer` (targeted fix). `maintain` includes citation checking as one step of a broader health sweep — use `maintain` only for full brain health runs.
5. **Background task / spawn agent** → use `minion-orchestrator` for spawning and steering agents. `gbrain-jobs` is the lower-level queue CLI.
6. Chaining is explicit in each skill's Phases section
7. When in doubt, ask the user
4. Chaining is explicit in each skill's Phases section
5. When in doubt, ask the user
## Conventions (cross-cutting)
-2
View File
@@ -8,8 +8,6 @@ triggers:
- "fix citations"
- "citation audit"
- "check citations"
- "citations are broken"
- "fix broken citations"
tools:
- search
- get_page
+7 -4
View File
@@ -55,9 +55,14 @@ they building, what makes them tick, where are they headed.
## Citation Requirements (MANDATORY)
> **Convention:** See `skills/conventions/quality.md` for full citation format rules.
Every fact must carry an inline `[Source: ...]` citation.
Every fact must carry an inline `[Source: ...]` citation. Source precedence (highest to lowest):
Three formats:
- **Direct attribution:** `[Source: User, {context}, YYYY-MM-DD]`
- **API/external:** `[Source: {provider} enrichment, YYYY-MM-DD]`
- **Synthesis:** `[Source: compiled from {list of sources}]`
Source precedence (highest to lowest):
1. User's direct statements
2. Compiled truth (pre-existing brain synthesis)
3. Timeline entries (raw evidence)
@@ -84,8 +89,6 @@ When sources conflict, note the contradiction with both citations.
Scale enrichment to importance. Don't waste API calls on low-value entities.
> **Convention:** See `skills/_brain-filing-rules.md` for the notability gate and filing rules.
| Tier | Who | Effort | Sources |
|------|-----|--------|---------|
| 1 (key) | Inner circle, close collaborators, key contacts | Full pipeline | All available APIs + deep web research |
-218
View File
@@ -1,218 +0,0 @@
---
name: frontmatter-guard
version: 1.0.0
description: |
Validates and auto-repairs frontmatter YAML on every brain page write.
Gate that prevents malformed pages from entering the brain. Import
writeBrainPage() instead of raw writeFileSync for any /data/brain/ write.
triggers:
- "validate frontmatter"
- "check frontmatter"
- "brain lint"
- "fix frontmatter"
tools:
- exec
- read
- write
mutating: true
---
# Frontmatter Guard
> Every brain write goes through the guard. No exceptions.
## Why This Exists
On 2026-04-24, a brain health audit found 203 pages with malformed frontmatter:
- 111 people pages missing closing `---` (entity detector bug)
- 43 meeting pages with unstructured YAML (ingestion bug)
- 16 files with slug mismatches
- 11 with binary corruption
- 4 with nested quote escaping
All written by our own agents. The guard prevents this class of error.
## The Library
**Location:** `lib/brain-writer.mjs` (in the OpenClaw workspace)
### Core API
```javascript
import { writeBrainPage, validateFrontmatter, autoFixFrontmatter } from '../lib/brain-writer.mjs';
// 1. Validated write (throws on bad frontmatter)
writeBrainPage('/data/brain/people/jane-doe.md', content);
// 2. Validated write with auto-repair
writeBrainPage('/data/brain/people/jane-doe.md', content, { autoFix: true });
// 3. Validate only (no write)
const result = validateFrontmatter(content, { filePath: '/data/brain/people/jane-doe.md' });
// → { ok: true/false, errors: [{ code, message }] }
// 4. Auto-fix only (returns fixed content)
const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath });
```
### What It Validates
| Check | Error Code | Description |
|-------|-----------|-------------|
| Opening `---` | `MISSING_OPEN` | File doesn't start with frontmatter |
| Closing `---` | `MISSING_CLOSE` | No closing delimiter (heading found inside YAML zone) |
| YAML parse | `YAML_PARSE` | js-yaml can't parse the frontmatter block |
| Slug match | `SLUG_MISMATCH` | `slug:` field doesn't match file path |
| Null bytes | `NULL_BYTES` | Binary corruption in content |
| Nested quotes | `NESTED_QUOTES` | `title: "Name "Nick" Last"` pattern |
| Empty frontmatter | `EMPTY_FRONTMATTER` | Frontmatter block is empty |
### What It Auto-Fixes
| Fix | Description |
|-----|-------------|
| Missing `---` | Inserts closing delimiter before first heading |
| Nested quotes in title | `"Name "Nick" Last"``'Name "Nick" Last'` |
| Nested quotes in lists | Investor notes with inner quotes → inner singles |
| Bracket titles | `title: [Name``title: "Name"` |
| Slug removal | Removes `slug:` field (gbrain derives from path) |
| Null bytes | Strips `\x00` characters |
### Path Guard
```javascript
// This THROWS — path is not under /data/brain/
writeBrainPage('/data/.openclaw/workspace/brain/people/test.md', content);
// Error: writeBrainPage: path is not under /data/brain/
```
This prevents the #1 brain write bug: writing to the workspace `brain/` subdirectory instead of the actual brain repo.
## Pre-Commit Hook
**Location:** `/data/brain/.githooks/pre-commit`
Runs on every `git commit` in the brain repo. Checks staged `.md` files for:
1. Missing closing `---`
2. YAML parse errors (via js-yaml from workspace node_modules)
3. Null bytes
Blocks the commit with actionable errors. Bypass: `git commit --no-verify`.
## Integration Rules for Agents
### When writing a brain page directly (writeFileSync)
**ALWAYS** use `writeBrainPage()` instead:
```javascript
// ❌ BAD — no validation, silent corruption
import { writeFileSync } from 'node:fs';
writeFileSync('/data/brain/people/jane-doe.md', content);
// ✅ GOOD — validates, blocks bad writes
import { writeBrainPage } from '../lib/brain-writer.mjs';
writeBrainPage('/data/brain/people/jane-doe.md', content);
```
### When generating frontmatter in a prompt
Always include the closing `---`:
```markdown
---
title: "Person Name"
type: person
created: 2026-04-24
---
# Person Name
```
### When titles contain special characters
Use single quotes for titles with inner double quotes:
```yaml
# ❌ BAD
title: "Phil Libin's Journey to Finding a "Life's Work""
# ✅ GOOD
title: 'Phil Libin''s Journey to Finding a "Life''s Work"'
# ✅ ALSO GOOD
title: "Phil Libin's Journey to Finding a Life's Work"
```
### When values contain colons
Always quote values with colons:
```yaml
# ❌ BAD — YAML thinks everything after the colon is a new key
garry_context: Fucking sick coding song — one of Garry's favorites
# ✅ GOOD
garry_context: "Fucking sick coding song — one of Garry's favorites"
```
## Running a Brain-Wide Audit
```bash
cd /data/.openclaw/workspace && node -e "
import { validateFrontmatter } from './lib/brain-writer.mjs';
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';
function walk(dir, files = []) {
for (const f of readdirSync(dir)) {
if (f === '.git') continue;
const p = join(dir, f);
if (statSync(p).isDirectory()) walk(p, files);
else if (f.endsWith('.md')) files.push(p);
}
return files;
}
let valid = 0, invalid = 0;
for (const file of walk('/data/brain')) {
const content = readFileSync(file, 'utf8');
if (!content.startsWith('---')) continue;
const r = validateFrontmatter(content, { filePath: file });
if (r.ok) valid++; else invalid++;
}
console.log('Valid:', valid, '| Invalid:', invalid, '| Rate:', (valid*100/(valid+invalid)).toFixed(1) + '%');
"
```
## Batch Auto-Fix
```bash
cd /data/.openclaw/workspace && node -e "
import { validateFrontmatter, autoFixFrontmatter } from './lib/brain-writer.mjs';
import { readFileSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';
// ... walk function ...
let fixed = 0;
for (const file of walk('/data/brain')) {
const content = readFileSync(file, 'utf8');
if (!content.startsWith('---')) continue;
if (validateFrontmatter(content).ok) continue;
const result = autoFixFrontmatter(content, { filePath: file });
if (result.fixes.length > 0 && validateFrontmatter(result.content).ok) {
writeFileSync(file, result.content);
fixed++;
}
}
console.log('Fixed:', fixed, 'files');
"
```
## Upstream Path
Once battle-tested here, the validator moves into gbrain's core:
1. `src/core/frontmatter.ts` — the validation + auto-fix logic
2. Integrated into `putPage()` / `upsertPage()` — every DB write validates
3. `gbrain lint` CLI command — runs the audit
4. `gbrain lint --fix` — runs auto-repair
5. Pre-commit hook ships with `gbrain init`
@@ -1,5 +0,0 @@
// Routing eval fixtures for skills/frontmatter-guard. Check 5 (W2, v0.19).
{"intent": "can you validate the frontmatter on these brain pages I just wrote", "expected_skill": "frontmatter-guard"}
{"intent": "run a brain lint to find broken frontmatter across the repo", "expected_skill": "frontmatter-guard"}
// Negative: general brain health is maintain, not frontmatter-guard
{"intent": "check overall brain health and run maintenance", "expected_skill": null, "ambiguous_with": []}
+1 -1
View File
@@ -8,7 +8,7 @@ description: |
triggers:
- "brain health"
- "check backlinks"
- "maintenance audit"
- "citation audit"
- "maintenance"
- "orphan pages"
- "stale pages"
+4 -62
View File
@@ -75,14 +75,10 @@ export function resolveGbrainCliPath(): string {
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
}
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
return !args.includes('--no-worker');
}
export async function runAutopilot(engine: BrainEngine, args: string[]) {
if (args.includes('--help') || args.includes('-h')) {
console.log(
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json] [--no-worker]\n' +
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json]\n' +
' gbrain autopilot --install [--repo <path>]\n' +
' gbrain autopilot --uninstall\n' +
' gbrain autopilot --status [--json]\n\n' +
@@ -110,7 +106,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const baseInterval = parseInt(parseArg(args, '--interval') || '300', 10);
const jsonMode = args.includes('--json');
const forceInline = args.includes('--inline');
const noWorker = !shouldSpawnAutopilotWorker(args);
if (!repoPath) {
console.error('No repo path. Use --repo or run gbrain sync --repo first.');
@@ -142,13 +137,12 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
const cfg = loadConfig();
const engineType = cfg?.engine ?? 'pglite';
const useMinionsDispatch = mode !== 'off' && engineType === 'postgres' && !forceInline;
const spawnManagedWorker = useMinionsDispatch && !noWorker;
let stopping = false;
let workerProc: ChildProcess | null = null;
let crashCount = 0;
if (spawnManagedWorker) {
if (useMinionsDispatch) {
const cliPath = resolveGbrainCliPath();
const startWorker = () => {
const child = spawn(cliPath, ['jobs', 'work'], { stdio: 'inherit', env: process.env });
@@ -167,13 +161,10 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
});
};
startWorker();
} else if (!useMinionsDispatch) {
const why = mode === 'off'
? 'minion_mode=off'
} else {
const why = mode === 'off' ? 'minion_mode=off'
: (engineType !== 'postgres' ? 'engine=pglite' : 'flag=--inline');
console.log(`[autopilot] running steps inline (${why})`);
} else {
console.log('[autopilot] --no-worker set: dispatch loop only (worker managed externally)');
}
// Async shutdown with 35s drain window for the worker child. The worker
@@ -204,18 +195,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
process.on('SIGINT', () => { void shutdown('SIGINT'); });
let consecutiveErrors = 0;
// Peer-worker liveness for --no-worker mode. The probe is a proxy, not
// ground truth: SELECT count(*) of active jobs with a recent lock_until
// refresh. A queue with only waiting jobs and a healthy idle worker
// reads as "no worker" (false positive); a worker that died 110s ago
// while holding a lock reads as "alive" until lock_until expires.
// Good enough for V1 — a ground-truth minion_workers heartbeat table
// is tracked as v0.19.1 follow-up B7. When the probe sees no signal
// for NO_WORKER_WARN_TICKS consecutive cycles, log a loud warning so
// the operator can spot "I set --no-worker but forgot to start one"
// before the queue piles up.
const NO_WORKER_WARN_TICKS = 3;
let noWorkerConsecutiveIdle = 0;
while (!stopping) {
const cycleStart = Date.now();
@@ -235,43 +214,6 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
} catch (e) { logError('reconnect', e); }
}
// --no-worker peer-liveness probe (v0.19.1). Runs every cycle, cheap
// (single SELECT). See NO_WORKER_WARN_TICKS comment above for caveats.
if (noWorker && useMinionsDispatch) {
try {
const rows = await (engine as any).executeRaw?.(
`SELECT count(*)::int AS n FROM minion_jobs
WHERE status = 'active'
AND lock_until IS NOT NULL
AND lock_until > now() - interval '2 minutes'`,
);
const liveWorkerSignal = Number((rows as Array<{ n: number }>)?.[0]?.n ?? 0);
if (liveWorkerSignal === 0) {
noWorkerConsecutiveIdle++;
if (noWorkerConsecutiveIdle === NO_WORKER_WARN_TICKS) {
// Fire loud on the Nth consecutive idle tick; don't repeat on every
// subsequent cycle (the operator already saw it), re-arm once a
// live worker is seen again.
console.error(
`[autopilot] WARNING: --no-worker set and no worker has claimed a job in ~${NO_WORKER_WARN_TICKS * baseInterval}s. ` +
`Jobs will pile up in 'waiting' until a worker starts. ` +
`Probe is a proxy (lock_until refresh) and can false-positive on idle queues — see B7 for ground-truth follow-up.`,
);
}
} else {
if (noWorkerConsecutiveIdle >= NO_WORKER_WARN_TICKS) {
console.log('[autopilot] --no-worker probe: live worker signal detected; warning re-armed.');
}
noWorkerConsecutiveIdle = 0;
}
} catch (e) {
// Probe failures never block the main dispatch loop. Log once per
// failure class; ignore repeated errors (common shape: DB reconnect
// blip between ticks).
logError('no-worker-probe', e);
}
}
if (useMinionsDispatch) {
// Submit ONE autopilot-cycle job per cycle slot. The idempotency key
// dedupes overrun submissions — if a cycle's job runs longer than
-100
View File
@@ -649,106 +649,6 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
mbcHb();
}
// 11b. Queue health (v0.19.1 queue-resilience wave).
// Postgres-only because PGLite has no multi-process worker surface. Two
// subchecks, both cheap (single SELECT each, status-index-covered):
//
// 1. stalled-forever: any active job whose started_at is > 1h old. The
// incident that motivated this release ran 90+ min before surfacing.
// Surface the ID so the operator can `gbrain jobs get <id>` to inspect
// or `gbrain jobs cancel <id>` to force-kill.
//
// 2. backpressure-missed: per-name waiting depth exceeds the threshold
// (default 10, override via GBRAIN_QUEUE_WAITING_THRESHOLD env). Signal
// that a submitter probably needs maxWaiting set. Bounded by per-name
// aggregation so a single name's pile shows up clearly instead of
// getting lost in the total.
//
// Not included in v0.19.1 (tracked as B7 follow-up): worker-heartbeat
// staleness. It needs a minion_workers table; the lock_until-on-active-jobs
// proxy can't distinguish "no worker" from "worker idle," and a check that
// cries wolf erodes trust in every other doctor check.
progress.heartbeat('queue_health');
if (engine.kind === 'pglite') {
checks.push({
name: 'queue_health',
status: 'ok',
message: 'Skipped (PGLite — no multi-process worker surface)',
});
} else {
const queueHealthHb = startHeartbeat(progress, 'scanning queue health…');
try {
const sql = db.getConnection();
// Subcheck 1: stalled-forever active jobs (>1h wall-clock).
const stalledRows: Array<{ id: number; name: string; started_at: string }> = await sql`
SELECT id, name, started_at::text AS started_at
FROM minion_jobs
WHERE status = 'active'
AND started_at IS NOT NULL
AND started_at < now() - interval '1 hour'
ORDER BY started_at ASC
LIMIT 5
`;
// Subcheck 2: per-name waiting depth exceeds threshold.
const rawThreshold = process.env.GBRAIN_QUEUE_WAITING_THRESHOLD;
const parsedThreshold = rawThreshold ? parseInt(rawThreshold, 10) : 10;
const threshold = Number.isFinite(parsedThreshold) && parsedThreshold >= 1
? parsedThreshold
: 10;
const depthRows: Array<{ name: string; queue: string; depth: number }> = await sql`
SELECT name, queue, count(*)::int AS depth
FROM minion_jobs
WHERE status = 'waiting'
GROUP BY name, queue
HAVING count(*) > ${threshold}
ORDER BY depth DESC
LIMIT 5
`;
const problems: string[] = [];
if (stalledRows.length > 0) {
const sample = stalledRows
.map(r => `#${r.id}(${r.name})`)
.join(', ');
problems.push(
`${stalledRows.length} stalled-forever job(s): ${sample}. ` +
`Fix: gbrain jobs get <id> to inspect; gbrain jobs cancel <id> to force-kill.`
);
}
if (depthRows.length > 0) {
const sample = depthRows
.map(r => `${r.name}@${r.queue}=${r.depth}`)
.join(', ');
problems.push(
`waiting-queue depth exceeds ${threshold} for: ${sample}. ` +
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
);
}
if (problems.length === 0) {
checks.push({
name: 'queue_health',
status: 'ok',
message: `No stalled-forever jobs; no queue over depth ${threshold}.`,
});
} else {
checks.push({
name: 'queue_health',
status: 'warn',
message: problems.join(' '),
});
}
} catch (e) {
checks.push({
name: 'queue_health',
status: 'warn',
message: `queue_health scan skipped: ${e instanceof Error ? e.message : String(e)}`,
});
} finally {
queueHealthHb();
}
}
// 12. Index audit (opt-in via --index-audit). v0.13.1 follow-up to #170.
// Reports indexes with zero recorded scans on Postgres. Informational only;
// we DO NOT auto-drop. On #170's brain, idx_pages_frontmatter and
+10 -118
View File
@@ -17,42 +17,6 @@ function hasFlag(args: string[], flag: string): boolean {
return args.includes(flag);
}
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
* Throws on malformed input (caller should surface the error and exit).
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
* Exported for unit tests; the CLI handler at `jobs submit` wraps this
* with process.exit(1) on throw so operators see 'must be positive integer'. */
export function parseMaxWaitingFlag(args: string[]): number | undefined {
const raw = parseFlag(args, '--max-waiting');
if (raw === undefined) return undefined;
const parsed = parseInt(raw, 10);
if (!Number.isFinite(parsed) || parsed < 1) {
throw new Error('--max-waiting must be a positive integer (will be clamped to [1, 100])');
}
return Math.max(1, Math.min(100, parsed));
}
export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv = process.env): number {
const raw = parseFlag(args, '--concurrency') ?? env.GBRAIN_WORKER_CONCURRENCY ?? '1';
const parsed = parseInt(raw, 10);
// Without validation, NaN / 0 / negative values flow through to the worker
// loop where `inFlight.size < concurrency` is always false → the worker
// claims zero jobs and the queue silently wedges. One typo in a systemd
// unit reproduces the original production incident. Clamp to ≥1 and surface
// the misconfig loudly so operators see it at worker startup.
if (!Number.isFinite(parsed) || parsed < 1) {
const source = parseFlag(args, '--concurrency') !== undefined
? '--concurrency flag'
: 'GBRAIN_WORKER_CONCURRENCY env';
process.stderr.write(
`[gbrain jobs] invalid concurrency from ${source} (${JSON.stringify(raw)}); ` +
`falling back to 1. Set a positive integer.\n`
);
return 1;
}
return parsed;
}
function formatJob(job: MinionJob): string {
const dur = job.finished_at && job.started_at
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
@@ -94,7 +58,6 @@ export async function runJobs(engine: BrainEngine, args: string[]): Promise<void
USAGE
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
[--delay Nms] [--max-attempts N] [--max-stalled N]
[--max-waiting N]
[--backoff-type fixed|exponential] [--backoff-delay Nms]
[--backoff-jitter 0..1] [--timeout-ms Nms]
[--idempotency-key K] [--queue Q] [--dry-run]
@@ -181,12 +144,6 @@ HANDLER TYPES (built in)
const maxAttempts = parseInt(parseFlag(args, '--max-attempts') ?? '3', 10);
const maxStalledRaw = parseFlag(args, '--max-stalled');
const maxStalled = maxStalledRaw !== undefined ? parseInt(maxStalledRaw, 10) : undefined;
// --max-waiting N: submission-time backpressure cap. Mirrors --max-stalled
// clamp [1, 100]. Feature is usable from CLI as of v0.19.1; pre-v0.19.1
// only programmatic callers reached it.
let maxWaiting: number | undefined;
try { maxWaiting = parseMaxWaitingFlag(args); }
catch (e) { console.error(`Error: ${e instanceof Error ? e.message : String(e)}`); process.exit(1); }
// v0.13.1 field audit: expose retry/backoff/timeout/idempotency knobs so
// users can tune Minions behavior without dropping into TypeScript.
const backoffTypeRaw = parseFlag(args, '--backoff-type');
@@ -215,7 +172,6 @@ HANDLER TYPES (built in)
console.log(` Priority: ${priority}`);
console.log(` Max attempts: ${maxAttempts}`);
if (maxStalled !== undefined) console.log(` Max stalled: ${maxStalled}`);
if (maxWaiting !== undefined) console.log(` Max waiting: ${maxWaiting}`);
if (backoffType) console.log(` Backoff type: ${backoffType}`);
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
@@ -243,7 +199,6 @@ HANDLER TYPES (built in)
delay: delay > 0 ? delay : undefined,
max_attempts: maxAttempts,
max_stalled: maxStalled,
maxWaiting,
backoff_type: backoffType,
backoff_delay: backoffDelay,
backoff_jitter: backoffJitter,
@@ -460,7 +415,6 @@ HANDLER TYPES (built in)
}
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
const wedgeRescue = hasFlag(args, '--wedge-rescue');
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
@@ -527,70 +481,9 @@ HANDLER TYPES (built in)
try { await queue.removeJob(rescueJob.id); } catch { /* non-fatal cleanup */ }
}
// --wedge-rescue: regression case for the v0.19.1 production incident.
// In prod, a wedged worker held a row lock via a pending txn. The
// lock-renewal UPDATE blocked, lock_until fell below now(), handleStalled
// saw the candidate but FOR UPDATE SKIP LOCKED skipped (row lock held),
// handleTimeouts was disqualified (lock_until > now() fails).
// Only handleWallClockTimeouts' no-constraint sweep evicted.
//
// The smoke is single-connection, so we can't simulate a row lock held
// by another txn. Instead we forge the state where BOTH handleStalled
// and handleTimeouts are disqualified so only wall-clock fires:
// - lock_until far in the future → handleStalled skips (not a stall)
// - timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
// - started_at 10s ago with timeout_ms=1000 → wall-clock matches
// (2 × timeout_ms = 2000ms threshold exceeded)
if (wedgeRescue) {
const wedgedJob = await queue.add('noop', {}, {
queue: 'smoke',
timeout_ms: 1000,
});
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='smoke-wedge-rescue',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '10 seconds',
timeout_at=NULL,
attempts_started = attempts_started + 1
WHERE id=$1`,
[wedgedJob.id]
);
const stallResult = await queue.handleStalled();
const stalledStatus = await queue.getJob(wedgedJob.id);
const timeoutResult = await queue.handleTimeouts();
const timedStatus = await queue.getJob(wedgedJob.id);
const wallResult = await queue.handleWallClockTimeouts(30000);
const finalStatus = await queue.getJob(wedgedJob.id);
if (finalStatus?.status !== 'dead') {
console.error(
`SMOKE FAIL (--wedge-rescue) — wall-clock sweep did not evict job #${wedgedJob.id}. ` +
`Status: ${finalStatus?.status}. ` +
`handleStalled: requeued=${stallResult.requeued.length} dead=${stallResult.dead.length}, after: ${stalledStatus?.status}; ` +
`handleTimeouts: ${timeoutResult.length}, after: ${timedStatus?.status}; ` +
`handleWallClockTimeouts: ${wallResult.length}, final: ${finalStatus?.status}.`
);
process.exit(1);
}
if (finalStatus.error_text !== 'wall-clock timeout exceeded') {
console.error(
`SMOKE FAIL (--wedge-rescue) — dead, but error_text='${finalStatus.error_text}' ` +
`(expected 'wall-clock timeout exceeded').`
);
process.exit(1);
}
try { await queue.removeJob(wedgedJob.id); } catch { /* non-fatal cleanup */ }
}
const cfg = (await import('../core/config.ts')).loadConfig();
const engineLabel = cfg?.engine ?? 'unknown';
const tags: string[] = [];
if (sigkillRescue) tags.push('SIGKILL rescue');
if (wedgeRescue) tags.push('wedge rescue');
const tag = tags.length > 0 ? ` + ${tags.join(' + ')}` : '';
const tag = sigkillRescue ? ' + SIGKILL rescue' : '';
console.log(`SMOKE PASS — Minions healthy${tag} in ${elapsedSec}s (engine: ${engineLabel})`);
if (engineLabel === 'pglite') {
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
@@ -610,7 +503,7 @@ HANDLER TYPES (built in)
}
const queueName = parseFlag(args, '--queue') ?? 'default';
const concurrency = resolveWorkerConcurrency(args);
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '1', 10);
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
@@ -926,17 +819,16 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
};
});
// Shell handler is always registered. Runtime env guard lives inside the
// handler so claimed jobs emit a clear rejection log on workers missing
// GBRAIN_ALLOW_SHELL_JOBS=1.
{
// Shell handler: registered ONLY when GBRAIN_ALLOW_SHELL_JOBS=1 is set on the
// worker process. Default-closed; opt-in per-host. Without the flag, shell
// jobs submitted via CLI insert rows but no worker claims them (they sit in
// 'waiting' — the CLI prints a starvation warning for that case).
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
const { shellHandler } = await import('../core/minions/handlers/shell.ts');
worker.register('shell', shellHandler);
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
} else {
process.stderr.write('[minion worker] shell handler registered in guarded mode (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)\n');
}
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
} else {
process.stderr.write('[minion worker] shell handler disabled (set GBRAIN_ALLOW_SHELL_JOBS=1 to enable)\n');
}
// v0.15 subagent handlers: always-on. Unlike shell (which needs an env
-77
View File
@@ -1,77 +0,0 @@
/**
* Backpressure audit log — operational trace for `maxWaiting` coalesce events.
*
* Mirrors the shell-audit.ts pattern (ISO-week-rotated JSONL, best-effort writes,
* failures go to stderr but never block submission). The incident that motivated
* maxWaiting (autopilot pile-up during a 90+ min queue wedge) was invisible
* precisely because the coalesce silently dropped repeat submissions. This
* trail answers "why is queue depth steady at 2 for this name?" without any
* doctor scan.
*
* File: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (override dir via
* `GBRAIN_AUDIT_DIR` for container/sandbox deployments where `$HOME` is read-only).
*
* `gbrain jobs stats` will surface coalesce counts from this file in a v0.19.2+
* follow-up (B4). The audit trail is for operators debugging live queues, not
* for compliance — a disk-full attacker can silently disable it.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
export interface BackpressureAuditEvent {
ts: string;
queue: string;
name: string;
waiting_count: number;
max_waiting: number;
decision: 'coalesced';
returned_job_id: number;
}
/** Compute `backpressure-YYYY-Www.jsonl` using ISO-8601 week numbering.
*
* Copy of the shell-audit computeAuditFilename algorithm, parameterized on
* the filename prefix. Keeping the math inline (rather than re-exporting from
* shell-audit.ts) avoids a cross-module dependency between two best-effort
* audit surfaces — one can be rewritten without touching the other.
*/
export function computeAuditFilename(now: Date = new Date()): string {
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6
d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday
const isoYear = d.getUTCFullYear();
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
const ww = String(weekNum).padStart(2, '0');
return `backpressure-${isoYear}-W${ww}.jsonl`;
}
/** Honors `GBRAIN_AUDIT_DIR` for container/sandbox deployments. */
export function resolveAuditDir(): string {
const override = process.env.GBRAIN_AUDIT_DIR;
if (override && override.trim().length > 0) return override;
return path.join(os.homedir(), '.gbrain', 'audit');
}
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
const dir = resolveAuditDir();
const filename = computeAuditFilename();
const fullPath = path.join(dir, filename);
const line = JSON.stringify({
...event,
decision: 'coalesced' as const,
ts: new Date().toISOString(),
}) + '\n';
try {
fs.mkdirSync(dir, { recursive: true });
fs.appendFileSync(fullPath, line, { encoding: 'utf8' });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
process.stderr.write(`[backpressure-audit] write failed (${msg}); submission continues\n`);
}
}
-10
View File
@@ -207,16 +207,6 @@ class TailBuffer {
/** The shell handler itself. */
export async function shellHandler(ctx: MinionJobContext): Promise<ShellJobResult> {
if (process.env.GBRAIN_ALLOW_SHELL_JOBS !== '1') {
const warning =
`[shell] Job #${ctx.id} rejected: GBRAIN_ALLOW_SHELL_JOBS=1 not set on this worker.\n` +
' Shell jobs require the env var on the worker process.';
console.warn(warning);
throw new UnrecoverableError(
'shell handler disabled on this worker (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)',
);
}
const params = validateParams(ctx.data);
const env = buildChildEnv(params.env);
const startedAt = Date.now();
-129
View File
@@ -102,64 +102,6 @@ export class MinionQueue {
if (existing.length > 0) return rowToMinionJob(existing[0]);
}
// 1b. Submission-time backpressure for high-frequency named jobs.
// If waiting jobs for this (name, queue) already hit maxWaiting, return
// the most-recent waiting row instead of inserting another slot.
//
// Correctness: two concurrent submitters could both see waitingCount <
// maxWaiting and both insert, violating the cap. `pg_advisory_xact_lock`
// keyed on (name, queue) serializes concurrent count+insert decisions
// for the SAME key while leaving different keys fully parallel. The
// lock releases on txn commit/rollback automatically — no cleanup path
// to leak. Cost: one no-op SELECT on the hot path per coalesce-guarded
// submission; trivial compared to the protection.
//
// Queue scope: the filter includes `queue=$2` so a waiting
// 'autopilot-cycle' in queue 'default' does NOT suppress submissions
// to queue 'shell' with the same name. Pre-D2 code filtered on `name`
// alone — a real cross-queue bleed that sequential tests missed.
//
// Engine compatibility: PGLite (WASM Postgres 17) supports
// pg_advisory_xact_lock, so this works on both engines without branching.
if (opts?.maxWaiting !== undefined) {
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
const backpressureQueue = opts?.queue ?? 'default';
await tx.executeRaw(
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2))`,
[jobName, backpressureQueue]
);
const waitingCountRows = await tx.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count
FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'`,
[jobName, backpressureQueue]
);
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
if (waitingCount >= maxWaiting) {
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
`SELECT * FROM minion_jobs
WHERE name = $1 AND queue = $2 AND status = 'waiting'
ORDER BY created_at DESC, id DESC
LIMIT 1`,
[jobName, backpressureQueue]
);
if (existingWaiting.length > 0) {
const coalesced = rowToMinionJob(existingWaiting[0]);
try {
const { logBackpressureCoalesce } = await import('./backpressure-audit.ts');
logBackpressureCoalesce({
queue: backpressureQueue,
name: jobName,
waiting_count: waitingCount,
max_waiting: maxWaiting,
returned_job_id: coalesced.id,
});
} catch { /* audit failures never block submission */ }
return coalesced;
}
}
}
// 2. Parent lock + depth/cap validation
let depth = 0;
if (opts?.parent_job_id) {
@@ -621,77 +563,6 @@ export class MinionQueue {
});
}
/**
* Dead-letter active jobs that exceed a wall-clock runtime threshold,
* regardless of lock state. This catches jobs stuck while still holding
* DB resources (e.g. blocked on file locks) where stall sweeps skip rows.
*
* Threshold (ms):
* timeout_ms set -> timeout_ms * 2
* timeout_ms null -> 2 * lockDurationMs * max_stalled
*/
async handleWallClockTimeouts(lockDurationMs: number): Promise<MinionJob[]> {
return this.engine.transaction(async (tx) => {
const rows = await tx.executeRaw<Record<string, unknown>>(
`UPDATE minion_jobs SET
status = 'dead',
error_text = 'wall-clock timeout exceeded',
lock_token = NULL,
lock_until = NULL,
finished_at = now(),
updated_at = now()
WHERE status = 'active'
AND started_at IS NOT NULL
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
CASE
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
END
RETURNING *`,
[lockDurationMs]
);
const parentIds = new Set<number>();
for (const r of rows) {
const parentJobId = r.parent_job_id as number | null;
if (parentJobId == null) continue;
parentIds.add(parentJobId);
const childDone: ChildDoneMessage = {
type: 'child_done',
child_id: r.id as number,
job_name: r.name as string,
result: null,
outcome: 'timeout',
error: 'wall-clock timeout exceeded',
};
await tx.executeRaw(
`INSERT INTO minion_inbox (job_id, sender, payload)
SELECT $1, 'minions', $2::jsonb
WHERE EXISTS (
SELECT 1 FROM minion_jobs
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
)`,
[parentJobId, childDone]
);
}
for (const parentId of parentIds) {
await tx.executeRaw(
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
WHERE id = $1 AND status = 'waiting-children'
AND NOT EXISTS (
SELECT 1 FROM minion_jobs
WHERE parent_job_id = $1
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
)`,
[parentId]
);
}
return rows.map(rowToMinionJob);
});
}
/**
* Complete a job (token-fenced). All side effects atomic in one transaction:
* 1. UPDATE child to 'completed' with result
-2
View File
@@ -128,8 +128,6 @@ export interface MinionJobInput {
max_spawn_depth?: number;
/** Global dedup key. Same key returns the existing job, no second row created. */
idempotency_key?: string;
/** Submission backpressure: cap waiting jobs with this name before inserting a new row. */
maxWaiting?: number;
// v12: scheduler polish
/**
-8
View File
@@ -126,14 +126,6 @@ export class MinionWorker {
} catch (e) {
console.error('Timeout detection error:', e instanceof Error ? e.message : String(e));
}
try {
const wallClockTimedOut = await this.queue.handleWallClockTimeouts(this.opts.lockDuration);
if (wallClockTimedOut.length > 0) {
console.log(`Wall-clock detector: dead-lettered ${wallClockTimedOut.length} jobs (wall-clock timeout exceeded)`);
}
} catch (e) {
console.error('Wall-clock timeout detection error:', e instanceof Error ? e.message : String(e));
}
}, this.opts.stalledInterval);
try {
-12
View File
@@ -12,18 +12,8 @@ import * as os from 'node:os';
let engine: PGLiteEngine;
let queue: MinionQueue;
// The shell handler at src/core/minions/handlers/shell.ts:210 throws
// UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1'. That's the
// production-worker RCE guard. Unit tests here exercise the handler
// mechanics, not the guard, so we enable it for the whole file and
// restore on teardown. The separate "rejects when env not set" case
// (in the minion-shell submission E2E / the queue-resilience wave)
// toggles the var itself.
let prevAllowShellJobs: string | undefined;
beforeAll(async () => {
prevAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
@@ -32,8 +22,6 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
if (prevAllowShellJobs === undefined) delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
else process.env.GBRAIN_ALLOW_SHELL_JOBS = prevAllowShellJobs;
});
beforeEach(async () => {
-243
View File
@@ -1653,246 +1653,3 @@ describe('MinionQueue: Attachments', () => {
expect(list[0].filename).toBe('b.txt');
});
});
// --- v0.19.1 — queue-resilience (wall-clock sweep, maxWaiting race, concurrency clamp) ---
describe('MinionQueue: v0.19.1 handleWallClockTimeouts (Layer 3 kill shot)', () => {
test('evicts active job past 2× timeout_ms — sets dead + wall-clock error_text', async () => {
const job = await queue.add('noop', {}, { timeout_ms: 100 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='wc-test',
lock_until=now() - interval '1 second',
started_at=now() - interval '1 second',
timeout_at=now() - interval '0.9 second',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(1);
expect(killed[0].id).toBe(job.id);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('dead');
expect(after?.error_text).toBe('wall-clock timeout exceeded');
});
test('timeout_ms NULL fallback uses 2 × lockDuration × max_stalled threshold', async () => {
const job = await queue.add('noop', {}, { max_stalled: 3 });
// Force timeout_ms / timeout_at NULL on-disk (columns might or might not be set by add).
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
timeout_ms=NULL,
timeout_at=NULL,
lock_token='wc-null',
lock_until=now() - interval '1 second',
started_at=now() - interval '61 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
// 2 × lockDurationMs × max_stalled = 2 × 10_000 × 3 = 60_000 ms. started_at is 61s ago.
const killed = await queue.handleWallClockTimeouts(10_000);
expect(killed.length).toBe(1);
expect(killed[0].id).toBe(job.id);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('dead');
});
test('respects threshold — active job within window is NOT killed', async () => {
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='wc-inside',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '10 seconds',
timeout_at=now() + interval '90 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(0);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('active');
});
});
describe('MinionQueue: v0.19.1 maxWaiting — cap correctness + race (D2/H2)', () => {
test('coalesces 3rd submission when cap is 2 — returns existing most-recent waiting row', async () => {
const a = await queue.add('poll', {}, { maxWaiting: 2 });
const b = await queue.add('poll', {}, { maxWaiting: 2 });
const c = await queue.add('poll', {}, { maxWaiting: 2 });
expect(a.id).not.toBe(b.id);
expect(c.id).toBe(b.id); // coalesced to the most-recent waiting row
const rows = await engine.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='poll' AND status='waiting'`,
);
expect(parseInt(rows[0].count, 10)).toBe(2);
});
test('clamps maxWaiting: 0 → 1 (strictest cap)', async () => {
const a = await queue.add('squeeze', {}, { maxWaiting: 0 });
const b = await queue.add('squeeze', {}, { maxWaiting: 0 });
expect(b.id).toBe(a.id); // 0 clamped to 1, 2nd coalesces into 1st
});
test('floors maxWaiting: 1.7 → 1', async () => {
const a = await queue.add('floor', {}, { maxWaiting: 1.7 });
const b = await queue.add('floor', {}, { maxWaiting: 1.7 });
expect(b.id).toBe(a.id);
});
test('concurrent submitters respect the cap under Promise.all race (H2)', async () => {
// Serialized by pg_advisory_xact_lock keyed on (name, queue). Without it,
// two concurrent submits both see count<max and both insert — the TOCTOU
// bug codex caught in D2/H2.
const results = await Promise.all([
queue.add('race', {}, { maxWaiting: 2 }),
queue.add('race', {}, { maxWaiting: 2 }),
queue.add('race', {}, { maxWaiting: 2 }),
]);
expect(results.length).toBe(3);
const rows = await engine.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='race' AND status='waiting'`,
);
expect(parseInt(rows[0].count, 10)).toBe(2); // cap held under concurrency
});
test('cross-queue isolation — same name in queue A does NOT suppress queue B (H2 secondary)', async () => {
const a = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
// cap hit on queue=default with maxWaiting=1; 2nd would coalesce into `a`
const a2 = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
expect(a2.id).toBe(a.id);
// Different queue — MUST insert a fresh row, NOT coalesce into queue=default
const b = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'shell' });
expect(b.id).not.toBe(a.id);
expect(b.queue).toBe('shell');
});
test('unset maxWaiting — normal submit path, no coalesce, no cap', async () => {
const a = await queue.add('uncapped', {});
const b = await queue.add('uncapped', {});
const c = await queue.add('uncapped', {});
expect(new Set([a.id, b.id, c.id]).size).toBe(3);
});
});
describe('resolveWorkerConcurrency (v0.19.1 H3): clamp + validation', () => {
// jobs.ts handler — tested via direct import. Warning goes to stderr;
// tests verify return value only, not the warning line.
let resolveWorkerConcurrency: (args: string[], env?: NodeJS.ProcessEnv) => number;
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
beforeAll(async () => {
const mod = await import('../src/commands/jobs.ts');
resolveWorkerConcurrency = mod.resolveWorkerConcurrency;
parseMaxWaitingFlag = mod.parseMaxWaitingFlag;
});
test('flag=4 env-unset → 4', () => {
expect(resolveWorkerConcurrency(['--concurrency', '4'], {} as NodeJS.ProcessEnv)).toBe(4);
});
test('flag-unset env=8 → 8', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(8);
});
test('flag=2 env=8 → 2 (flag wins)', () => {
expect(resolveWorkerConcurrency(['--concurrency', '2'], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(2);
});
test('both unset → 1', () => {
expect(resolveWorkerConcurrency([], {} as NodeJS.ProcessEnv)).toBe(1);
});
test('garbage env "foo" → clamped to 1 (H3)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: 'foo' } as NodeJS.ProcessEnv)).toBe(1);
});
test('env=0 → clamped to 1 (H3 — prevents silent wedge)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '0' } as NodeJS.ProcessEnv)).toBe(1);
});
test('env=-5 → clamped to 1 (H3)', () => {
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '-5' } as NodeJS.ProcessEnv)).toBe(1);
});
});
describe('parseMaxWaitingFlag (v0.19.1 H5): CLI flag wiring', () => {
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
beforeAll(async () => {
parseMaxWaitingFlag = (await import('../src/commands/jobs.ts')).parseMaxWaitingFlag;
});
test('absent → undefined (no cap, default submit path)', () => {
expect(parseMaxWaitingFlag(['foo', '--params', '{}'])).toBeUndefined();
});
test('--max-waiting 2 → 2 (happy path)', () => {
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '2'])).toBe(2);
});
test('--max-waiting 200 → clamped to 100', () => {
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '200'])).toBe(100);
});
test('--max-waiting 0 → throws', () => {
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', '0'])).toThrow('positive integer');
});
test('--max-waiting abc → throws', () => {
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', 'abc'])).toThrow('positive integer');
});
});
describe('backpressure-audit (v0.19.1 Q1): JSONL on coalesce', () => {
test('logBackpressureCoalesce writes one JSONL line per coalesce', async () => {
const { logBackpressureCoalesce, resolveAuditDir, computeAuditFilename } =
await import('../src/core/minions/backpressure-audit.ts');
const fs = await import('node:fs');
const path = await import('node:path');
const os = await import('node:os');
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-audit-'));
const prev = process.env.GBRAIN_AUDIT_DIR;
process.env.GBRAIN_AUDIT_DIR = tmp;
try {
expect(resolveAuditDir()).toBe(tmp);
logBackpressureCoalesce({
queue: 'default',
name: 'poll',
waiting_count: 2,
max_waiting: 2,
returned_job_id: 42,
});
const file = path.join(tmp, computeAuditFilename());
const text = fs.readFileSync(file, 'utf8');
const line = JSON.parse(text.trim());
expect(line.decision).toBe('coalesced');
expect(line.name).toBe('poll');
expect(line.returned_job_id).toBe(42);
expect(typeof line.ts).toBe('string');
} finally {
if (prev === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = prev;
fs.rmSync(tmp, { recursive: true, force: true });
}
});
});
describe('MinionQueue: v0.19.1 wall-clock + handleTimeouts non-interference (T1)', () => {
test('wall-clock sweep does NOT evict a job that handleTimeouts would handle', async () => {
// Retry-able timeout: timeout_at < now() AND lock_until > now() — handleTimeouts
// is the correct killer here. wall-clock's 2× threshold has not fired yet.
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
await engine.executeRaw(
`UPDATE minion_jobs
SET status='active',
lock_token='t1',
lock_until=now() + interval '30 seconds',
started_at=now() - interval '2 seconds',
timeout_at=now() - interval '0.5 seconds',
attempts_started = attempts_started + 1
WHERE id=$1`,
[job.id],
);
// At this point: started_at is 2s ago, 2×timeout_ms = 200s. Wall-clock should NOT fire.
const killed = await queue.handleWallClockTimeouts(30_000);
expect(killed.length).toBe(0);
const after = await queue.getJob(job.id);
expect(after?.status).toBe('active');
});
});