mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-16 18:02:30 +00:00
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c21180c972 | ||
|
|
11abb24ddd | ||
|
|
d838d4792b |
+152
@@ -2,6 +2,158 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.20.4] - 2026-04-24
|
||||
|
||||
**Minions skill consolidation, now honest about what the CLI actually does.**
|
||||
|
||||
One skill for background work instead of two. Shell jobs and LLM subagents land under `skills/minion-orchestrator/` with a shared Preconditions block, accurate CLI examples, and a trigger set narrowed to what the skill actually covers. Corrects four documentation bugs the prior merge shipped ... `submit_job name="shell"` isn't MCP-callable, `research`/`orchestrate` aren't real handler names, PGLite users don't need to migrate to Supabase, and "every background task goes through Minions" contradicts the `pain_triggered` default in `skills/conventions/subagent-routing.md`. The skill now matches the code.
|
||||
|
||||
Two new tests guard this surface going forward. `test/resolver.test.ts` gets a round-trip check (every quoted RESOLVER.md trigger must resolve to a frontmatter `triggers:` entry in the target skill) and a name validator (every `name="<word>"` reference in any SKILL.md must resolve to either a declared operation in `src/core/operations.ts` or a known Minions handler). The validator would have caught the `research`/`orchestrate` drift in CI instead of from a Codex cold-read. One new E2E test (`test/e2e/minions-shell-pglite.test.ts`) exercises the PGLite `--follow` inline path, previously documented but untested.
|
||||
|
||||
### For users
|
||||
|
||||
- Shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only ... MCP returns `permission_denied` for protected names). Subagent jobs via `gbrain agent run` (user-facing entrypoint). Both lanes route through one skill.
|
||||
- PGLite shell-job guidance now correctly points at `--follow` for inline execution. The persistent daemon mode is still Postgres-only, but you do not need to migrate.
|
||||
- `gbrain jobs submit` and `submit a gbrain job` now route to the skill; bare "gbrain jobs" no longer does (it was too broad ... the CLI namespace covers 9 subcommands, and questions about `stats`/`prune`/`retry` fall through to `gbrain --help`).
|
||||
|
||||
### Added
|
||||
|
||||
- New E2E test `test/e2e/minions-shell-pglite.test.ts` covering the PGLite `--follow` inline shell-job path. Runs in-memory, no DATABASE_URL required.
|
||||
- Resolver round-trip test in `test/resolver.test.ts`: every quoted RESOLVER.md trigger must have a fuzzy match in the target skill's frontmatter `triggers:` list.
|
||||
- Skill-example-name validator in `test/resolver.test.ts`: every `name="<word>"` reference in any `SKILL.md` body must resolve to an op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `skills/minion-orchestrator/SKILL.md` shell-job examples use the real `--params` JSON form instead of nonexistent `--cmd`/`--argv`/`--cwd` flags.
|
||||
- `gbrain agent run` flag list now matches `src/commands/agent.ts` (removed `--queue`/`--priority`/`--max-attempts`/`--delay` which aren't parsed by that command).
|
||||
- `--tools` example uses `search,query` instead of `web_search` (the latter isn't in `BRAIN_TOOL_ALLOWLIST`, would throw at submit time).
|
||||
- MCP boundary wording says `submit_job name="shell"` throws an `OperationError` with code `permission_denied`, instead of the earlier "returns permission_denied" (not a return, a throw).
|
||||
- `skills/conventions/subagent-routing.md` stale reference to `get_job_stats` (no such op) replaced with `list_jobs --status active` or `gbrain jobs stats`.
|
||||
- `skills/query/SKILL.md` + `skills/maintain/SKILL.md` frontmatter `triggers:` lists closed gaps the new round-trip test surfaced (RESOLVER.md was routing 10 triggers to these skills that their frontmatter never declared).
|
||||
- `skills/manifest.json` minion-orchestrator description updated to match the unified SKILL.md framing.
|
||||
|
||||
### Changed
|
||||
|
||||
- Trigger `"gbrain jobs"` narrowed to `"gbrain jobs submit"` + `"submit a gbrain job"` in both `skills/RESOLVER.md` and the skill's frontmatter.
|
||||
- Anti-pattern about `sessions_spawn` scoped to the subagent lane (was ambiguous in the consolidated skill).
|
||||
|
||||
### For contributors
|
||||
|
||||
- Code-to-doc drift is now partially machine-checkable. The skill-example-name validator catches T2-class bugs (docs referencing handler/op names that don't exist). CLI flag validation is a remaining gap ... a future PR could extend the test to validate `--flag-name` patterns in SKILL.md against actual CLI flag parsers.
|
||||
|
||||
## To take advantage of v0.20.4
|
||||
|
||||
Any gbrain user whose agent routes on "minions" work gets the corrected skill on the next `gbrain upgrade`. No manual migration required ... the renamed trigger is additive (old trigger gone, new triggers cover the same intent), and the doc corrections don't change runtime behavior.
|
||||
|
||||
1. **Run the orchestrator manually if `gbrain upgrade` reports a partial migration:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Your agent picks up the new skill content** next time it consults `skills/minion-orchestrator/SKILL.md`. No action required on your side.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain check-resolvable --json | python3 -c "import json,sys;d=json.load(sys.stdin);print('ok:',d['ok'])"
|
||||
```
|
||||
Should print `ok: True`.
|
||||
4. **If any step fails,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
|
||||
## [0.20.3] - 2026-04-24
|
||||
|
||||
## **Your queue now rescues itself when a wedged worker holds a row lock. Wall-clock sweep kills the job that stall detection can't see.**
|
||||
## **`maxWaiting` is race-proof, observable, and reachable from the CLI — three bugs in one patch.**
|
||||
|
||||
A production autopilot-cycle job wedged for over an hour on a single OpenClaw deployment because the worker's handler got stuck mid-transaction holding a row lock. Both eviction paths were blocked: the stall detector's `FOR UPDATE SKIP LOCKED` pass skipped the row-locked candidate, and the timeout sweep's `lock_until > now()` predicate disqualified the job once lock-renewal had been blocked. Neither could see the job. The shell-job pipeline starved completely behind the wedge.
|
||||
|
||||
v0.19.0 shipped the wall-clock sweep as the third-layer kill shot: drop both constraints, evict on `started_at` alone, worst case at `2 × timeout_ms + stalledInterval`. This release locks down three correctness holes the v0.19.0 PR introduced — then closes the observability gap that let the incident run to minute 90 in the first place.
|
||||
|
||||
### The queue-resilience numbers that matter
|
||||
|
||||
Measured against the real incident on 2026-04-23 (OpenClaw autopilot + shell-job pipeline, Postgres engine, concurrency=1 worker).
|
||||
|
||||
| Behavior | Before v0.20.3 | After v0.20.3 |
|
||||
|---|---|---|
|
||||
| Wedged worker escape window | 90+ minutes (manual kill) | `~2 × timeout_ms + 30s` sweep interval |
|
||||
| Per-name waiting pile during wedge | 18 deferred per-slot jobs | capped at `maxWaiting` |
|
||||
| `maxWaiting` under concurrent submit (2 submitters, cap=2) | up to 3 rows (TOCTOU race) | exactly 2 rows (advisory-lock serialization) |
|
||||
| Same name across queues | cross-queue bleed — `shell` suppressed by `default` | isolated per `(name, queue)` |
|
||||
| `GBRAIN_WORKER_CONCURRENCY=foo` | silent wedge (`inFlight < NaN` false) | clamped to 1, loud stderr warning |
|
||||
| `gbrain jobs submit --max-waiting 2` | flag didn't exist | wired through to MinionJobInput |
|
||||
| Silent coalesce events | invisible | JSONL audit at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` |
|
||||
| `gbrain doctor` visibility into wedge | no check | new `queue_health` with 2 subchecks |
|
||||
|
||||
The two big shifts: (1) every silent-failure vector the v0.19.0 patches introduced now has a loud signal — JSONL audit files, doctor check, stderr warnings, peer-liveness probe. (2) `maxWaiting` is now actually a cap under concurrency, not a soft suggestion. A future multi-submitter pattern (parallel workspaces, dispatched children, OpenClaw + ycli cron) doesn't walk through it.
|
||||
|
||||
### What this means for OpenClaw users
|
||||
|
||||
If you're running `gbrain autopilot` on a daily-driver deployment, the wall-clock sweep is the difference between a 90-minute outage and a 30-second one. The `queue_health` doctor check means the next time your queue wedges, you notice in minute 2 instead of minute 90. If you've been writing programmatic Minion submitters and setting `maxWaiting`, it's worth re-reading the JSONL audit file the next time your agent does anything "interesting" — you'll see exactly which submission coalesces into which returned job.
|
||||
|
||||
## To take advantage of v0.20.3
|
||||
|
||||
`gbrain upgrade` handles the binary. You MUST restart long-running worker daemons so the new sweep runs in-process — the wall-clock eviction is a method on `MinionQueue`, not a cron job, so it only fires inside a worker loop.
|
||||
|
||||
1. **Upgrade the binary:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
2. **Restart autopilot + workers:**
|
||||
```bash
|
||||
# systemd / launchd / OpenClaw service-manager: restart the unit.
|
||||
# Manual: kill the old `gbrain autopilot` and `gbrain jobs work`, start new ones.
|
||||
```
|
||||
3. **Verify:**
|
||||
```bash
|
||||
gbrain jobs smoke --wedge-rescue # exercises the new wall-clock path
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
|
||||
```
|
||||
4. **If `gbrain doctor` flags anything unexpected,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/audit/backpressure-*.jsonl` (redact freely)
|
||||
- what commands you ran leading up to the wedge
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Queue core** (`src/core/minions/queue.ts`)
|
||||
- `maxWaiting` coalesce path wraps `count → select → insert` in `pg_advisory_xact_lock` keyed on `(name, queue)`. Concurrent submitters for the SAME key serialize; different keys stay parallel. Lock auto-releases on transaction commit/rollback — no cleanup path to leak. Fixes TOCTOU race caught by adversarial review.
|
||||
- `maxWaiting` count and select now filter on `queue` in addition to `name`. Pre-v0.20.3 code filtered on name alone, so a waiting `autopilot-cycle` in `queue=default` would suppress submissions to `queue=shell` with the same name. Cross-queue bleed is gone.
|
||||
|
||||
**Backpressure observability** (new `src/core/minions/backpressure-audit.ts`)
|
||||
- Every coalesce event writes one JSONL line to `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (ISO-week rotation, override dir via `GBRAIN_AUDIT_DIR`, mirrors the v0.14 shell-audit pattern).
|
||||
- Fields: `ts, queue, name, waiting_count, max_waiting, decision='coalesced', returned_job_id`.
|
||||
- Best-effort: write failures log to stderr but never block submission.
|
||||
|
||||
**CLI** (`src/commands/jobs.ts`)
|
||||
- New `--max-waiting N` flag on `gbrain jobs submit`. Clamps to `[1, 100]`, mirrors the existing `--max-stalled` wiring. The `MinionJobInput.maxWaiting` field was programmatic-only before; now it's reachable from the command line too.
|
||||
- `resolveWorkerConcurrency` clamps against invalid input. `parseInt` returns `NaN` for `"foo"`, `0` for `"0"`, negatives for `"-5"` — all of which silently wedge a worker (`inFlight.size < NaN/0/negative` is always false). Now clamped to ≥1 with a loud stderr warning naming the bad value. One typo in a systemd unit no longer reproduces the 90-minute outage.
|
||||
- New `gbrain jobs smoke --wedge-rescue` opt-in case. Forges a wedged-worker row state, invokes `handleStalled` + `handleTimeouts` + `handleWallClockTimeouts` in sequence, asserts only the wall-clock sweep evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape.
|
||||
|
||||
**Doctor** (`src/commands/doctor.ts`)
|
||||
- New `queue_health` check (Postgres-only; PGLite skips with `Skipped (PGLite — no multi-process worker surface)`).
|
||||
- Subcheck 1 — **stalled-forever**: flags active jobs whose `started_at` is older than 1 hour. Reports the top 5 by start time with `gbrain jobs get/cancel <id>` fix hints.
|
||||
- Subcheck 2 — **waiting-depth**: flags per-name queues whose waiting count exceeds threshold. Default 10, overridable via `GBRAIN_QUEUE_WAITING_THRESHOLD` env. Reports the top 5 by depth with "consider setting maxWaiting on the submitter" fix hint.
|
||||
- Worker-heartbeat staleness subcheck intentionally deferred to follow-up because `lock_until`-on-active-jobs is a lossy proxy. A check that cries wolf erodes trust in every other doctor subcheck. Needs a `minion_workers` table to produce ground-truth signal.
|
||||
|
||||
**Autopilot** (`src/commands/autopilot.ts`)
|
||||
- `--no-worker` mode gains a peer-worker-liveness probe. Every cycle runs a cheap `SELECT count(*)` checking for active jobs with `lock_until` refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud `WARNING` naming the silent-wedge vector (`--no-worker` set but no worker running). Re-arms once a live signal returns, so a healthy-but-idle worker doesn't trigger spam.
|
||||
- Probe is documented as a proxy, not ground truth — idle worker with no active jobs reads as "no worker." The ground-truth fix needs a `minion_workers` heartbeat table (tracked as follow-up).
|
||||
|
||||
**Docs**
|
||||
- New `docs/guides/queue-operations-runbook.md`: the "my queue looks wedged — what do I run?" reference. One viewport, in order of escalation. What each `queue_health` subcheck means. Self-check for the `--no-worker + no-worker-running` footgun.
|
||||
- `CLAUDE.md` Key-files section updated for the new `handleWallClockTimeouts` method (v0.19.0, described here for the first time), the new `backpressure-audit.ts` module, the updated `maxWaiting` semantics, and the new `queue_health` doctor check.
|
||||
|
||||
**Tests** (`test/minions.test.ts`)
|
||||
- 23 new unit cases. Wall-clock sweep (3 cases + non-interference with `handleTimeouts`). `maxWaiting` (coalesce, clamp 0 → 1, floor 1.7 → 1, concurrent-submitter race via `Promise.all`, cross-queue isolation, unset fallthrough). Concurrency clamp (7 cases including `NaN`/`0`/negative). `parseMaxWaitingFlag` (5 cases). Backpressure audit file write. All 143 minions tests pass.
|
||||
- E2E wall-clock case against real Postgres is next on the roadmap (needs a second-connection row-lock helper; the unit-level coverage above exercises the sweep mechanics directly).
|
||||
|
||||
### For contributors
|
||||
|
||||
- The v0.19.0 PR's narrative framed the 18-job pileup as "duplicate submissions from a cron loop with no idempotency key." That framing was wrong. Autopilot already sets `idempotency_key: autopilot-cycle:${slot}` where slot is a 5-minute tick boundary — within-slot duplicates are structurally impossible. The 18 jobs were 18 different slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before v0.20.3 shipped.
|
||||
- Follow-up issues tracked: B2 (autopilot heartbeat file), B3 (doctor `--fix` learns queue rescue), B4 (backpressure counts surfaced in `jobs stats`), B5 (cross-cutting "health-delivery-agent" pattern), B7 (`minion_workers` heartbeat table — unblocks both the dropped `queue_health` subcheck and a ground-truth `--no-worker` probe), P1 (composite indexes `(status, started_at)` and `(status, name)` on `minion_jobs` — currently the new sweeps fall back to `idx_minion_jobs_status`, selective enough on healthy queues, worth tightening in v0.20.4).
|
||||
|
||||
Full plan with CEO + Eng + Codex adversarial decisions lives at `~/.claude/plans/` for the operators who care about how this release was reviewed.
|
||||
|
||||
## [0.20.2] - 2026-04-24
|
||||
|
||||
## **`gbrain jobs supervisor` is now a self-healing daemon you can actually drive. The Minions worker stops dying silently.**
|
||||
|
||||
@@ -62,12 +62,13 @@ strict behavior when unset.
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
@@ -89,7 +90,7 @@ strict behavior when unset.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
@@ -142,7 +143,7 @@ strict behavior when unset.
|
||||
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
|
||||
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
|
||||
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
|
||||
- `skills/minion-orchestrator/SKILL.md` — Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
|
||||
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
|
||||
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
|
||||
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
|
||||
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
|
||||
@@ -227,7 +228,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
|
||||
`test/report.test.ts` (report format, directory structure),
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
@@ -275,6 +276,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
@@ -327,7 +329,7 @@ stop and remove it before starting a new one.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
@@ -337,11 +339,19 @@ briefing, migrate, setup, publish.
|
||||
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
|
||||
|
||||
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
|
||||
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
|
||||
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
|
||||
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
|
||||
routing is narrowed to what the skill actually covers.
|
||||
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
|
||||
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
|
||||
@@ -6,7 +6,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
@@ -28,7 +28,7 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
@@ -87,9 +87,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
## The 28 Skills
|
||||
## The 29 Skills
|
||||
|
||||
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
@@ -135,7 +135,8 @@ GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
@@ -377,7 +378,7 @@ Run `gbrain integrations` to see status.
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# Queue operations runbook
|
||||
|
||||
"My queue looks wedged — what do I run?" The commands below are in the order
|
||||
you probably want them. Shipped with v0.19.1 after a production incident
|
||||
where the queue held for 90+ minutes before the operator noticed.
|
||||
|
||||
## First signal: jobs aren't running
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
|
||||
```
|
||||
|
||||
`queue_health` flags two patterns:
|
||||
|
||||
- **stalled-forever**: active job whose `started_at` is older than 1h.
|
||||
- **waiting-depth**: any per-name queue deeper than 10 (override via
|
||||
`GBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.
|
||||
|
||||
## Triage commands
|
||||
|
||||
```bash
|
||||
# Who's active right now?
|
||||
gbrain jobs list --status active
|
||||
|
||||
# Who's waiting, biggest pile first?
|
||||
gbrain jobs list --status waiting --limit 50
|
||||
|
||||
# What's wrong with a specific job?
|
||||
gbrain jobs get <id>
|
||||
```
|
||||
|
||||
## Rescue actions (in order of escalation)
|
||||
|
||||
```bash
|
||||
# Force-kill a single stuck job:
|
||||
gbrain jobs cancel <id>
|
||||
|
||||
# Clear a specific job entirely (last resort):
|
||||
gbrain jobs delete <id>
|
||||
|
||||
# Health smoke on the mechanism itself:
|
||||
gbrain jobs smoke --wedge-rescue
|
||||
```
|
||||
|
||||
## What each subcheck means
|
||||
|
||||
- **stalled-forever** — A worker claimed a job, started executing, and has
|
||||
held the row for over an hour. The wall-clock sweep evicts jobs past
|
||||
2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
|
||||
or the sweep is newly deployed and this job predates it. Cancel it.
|
||||
- **waiting-depth** — Submitters are piling up jobs faster than workers
|
||||
drain them. Set `--max-waiting N` on the submission or on the programmatic
|
||||
`queue.add()` call. If you want a taller pile, raise the threshold via
|
||||
`GBRAIN_QUEUE_WAITING_THRESHOLD=50 gbrain doctor`.
|
||||
|
||||
## Self-check: is a worker even running?
|
||||
|
||||
```bash
|
||||
# If you're running autopilot with --no-worker, check that your external
|
||||
# worker (systemd / Docker / OpenClaw service-manager) is alive:
|
||||
gbrain jobs list --status active | head -5
|
||||
```
|
||||
|
||||
If the list is empty AND your submissions keep piling up, no worker is
|
||||
claiming. Start one:
|
||||
|
||||
```bash
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
## Follow-ups tracked for v0.20+
|
||||
|
||||
- B7 — `minion_workers` heartbeat table for ground-truth liveness (the
|
||||
`--no-worker` probe and the dropped `queue_health` worker-heartbeat
|
||||
subcheck both need this).
|
||||
- B3 — `gbrain doctor --fix` learns to rescue queue wedges.
|
||||
+24
-13
@@ -141,12 +141,13 @@ strict behavior when unset.
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in.
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). v0.14.0 abort-path fix: aborted jobs now call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`) instead of returning silently. `shutdownAbort` (instance field) fires on process SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` — shell handler listens to it; non-shell handlers don't.
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` (new in v0.14.1) is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
- `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`).
|
||||
- `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values.
|
||||
- `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced.
|
||||
- `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full.
|
||||
- `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15.
|
||||
- `src/core/minions/handlers/subagent-audit.ts` (v0.15) — JSONL audit + heartbeat writer at `~/.gbrain/audit/subagent-jobs-YYYY-Www.jsonl`. Events: `submission` (one line per submit) + `heartbeat` (per turn boundary: `llm_call_started | llm_call_completed | tool_called | tool_result | tool_failed`). Never logs prompts or tool inputs. `readSubagentAuditForJob(jobId, {sinceIso})` is the readback path for `gbrain agent logs`.
|
||||
@@ -168,7 +169,7 @@ strict behavior when unset.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, and `gbrain apply-migrations`.
|
||||
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`.
|
||||
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows.
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
@@ -221,7 +222,7 @@ strict behavior when unset.
|
||||
- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
|
||||
- `skills/webhook-transforms/SKILL.md` — External events to brain signals
|
||||
- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes
|
||||
- `skills/minion-orchestrator/SKILL.md` — Background job orchestration: submit, fan out children with depth/cap/timeouts, collect results via child_done inbox
|
||||
- `skills/minion-orchestrator/SKILL.md` — Unified background-work skill (v0.20.4 consolidation of the former `minion-orchestrator` + `gbrain-jobs` split). Two lanes: shell jobs via `gbrain jobs submit shell --params '{"cmd":"..."}'` (operator/CLI only; MCP throws `permission_denied` for protected names) and LLM subagents via `gbrain agent run` (user-facing entrypoint). Shared Preconditions block, parent-child DAGs with depth/cap/timeouts, `child_done` inbox for fan-in, PGLite `--follow` inline path for dev. Triggers narrowed from bare `"gbrain jobs"` to `"gbrain jobs submit"` + `"submit a gbrain job"` so `stats`/`prune`/`retry` questions fall through to `gbrain --help`.
|
||||
- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates
|
||||
- `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter
|
||||
- `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls)
|
||||
@@ -306,7 +307,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation),
|
||||
`test/report.test.ts` (report format, directory structure),
|
||||
`test/skills-conformance.test.ts` (skill frontmatter + required sections validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation),
|
||||
`test/resolver.test.ts` (RESOLVER.md coverage, routing validation + v0.20.4 round-trip: every quoted RESOLVER.md trigger must match a frontmatter `triggers:` entry in the target skill, and every `name="<word>"` reference in any SKILL.md must resolve to a declared op in `src/core/operations.ts` or a Minions handler in `PROTECTED_JOB_NAMES`),
|
||||
`test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key),
|
||||
`test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions),
|
||||
`test/intent.test.ts` (query intent classification: entity/temporal/event/general),
|
||||
@@ -354,6 +355,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/jsonb-roundtrip.test.ts` — v0.12.3 companion regression against the 4 doctor-scanned JSONB sites. Assertion-level overlap with `postgres-jsonb.test.ts` is intentional defense-in-depth: if doctor's scan surface ever drifts from the actual write surface, one of these tests catches it.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- `test/e2e/minions-shell-pglite.test.ts` (v0.20.4) exercises the PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the consolidated minion-orchestrator skill documents for dev use
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` (v0.19) — exercises `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the 107-skill OpenClaw deployment shape
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
@@ -406,7 +408,7 @@ stop and remove it before starting a new one.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 28 skills
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
organized by `skills/RESOLVER.md` (`AGENTS.md` is also accepted as of v0.19):
|
||||
|
||||
**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich,
|
||||
@@ -416,11 +418,19 @@ briefing, migrate, setup, publish.
|
||||
meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager.
|
||||
|
||||
**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports,
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator.
|
||||
testing, soul-audit, webhook-transforms, data-research, minion-orchestrator. As of
|
||||
v0.20.4, `minion-orchestrator` is the single unified skill for both lanes of background
|
||||
work (shell jobs via `gbrain jobs submit shell`, LLM subagents via `gbrain agent run`) ...
|
||||
the prior `gbrain-jobs` skill was merged in, Preconditions are shared, and trigger
|
||||
routing is narrowed to what the skill actually covers.
|
||||
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
|
||||
**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first,
|
||||
model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and
|
||||
`skills/_output-rules.md` are shared references.
|
||||
@@ -1141,7 +1151,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
@@ -1198,7 +1208,7 @@ Built by the President and CEO of Y Combinator to run his actual AI agents. The
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side against the category: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its own graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. The graph layer plus v0.12 extract quality together carry the gap. Full BrainBench scorecards + corpus live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
GBrain is those patterns, generalized. 28 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
@@ -1220,7 +1230,7 @@ Retrieve and follow the instructions at:
|
||||
https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md
|
||||
```
|
||||
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 28 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 29 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes.
|
||||
|
||||
If your agent doesn't auto-read `AGENTS.md`, point it at that file first:
|
||||
`https://raw.githubusercontent.com/garrytan/gbrain/master/AGENTS.md` is the non-Claude
|
||||
@@ -1279,9 +1289,9 @@ claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization
|
||||
|
||||
Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented).
|
||||
|
||||
## The 28 Skills
|
||||
## The 29 Skills
|
||||
|
||||
GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
GBrain ships 29 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AGENTS.md` — both filenames are supported as of v0.19). The resolver tells your agent which skill to read for any task.
|
||||
|
||||
[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime.
|
||||
|
||||
@@ -1327,7 +1337,8 @@ GBrain ships 28 skills organized by `skills/RESOLVER.md` (or your OpenClaw's `AG
|
||||
| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. |
|
||||
| **skillify** | The "skillify it!" meta-skill. Orchestrates the 10-step loop so failures become durable skills: scaffold the stubs via `gbrain skillify scaffold`, write the real logic, gate with `gbrain skillify check` + `gbrain check-resolvable`. |
|
||||
| **skillpack-check** | Agent-readable gbrain health report. Exit code for CI; JSON for debugging. Cron-friendly. |
|
||||
| **minion-orchestrator** | Long-running agent work as background jobs. Submit, fan out children with depth/cap/timeouts, collect results via child_done inbox. |
|
||||
| **smoke-test** | 8 post-restart health checks with auto-fix (Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo). Drop-in user tests at `~/.gbrain/smoke-tests.d/*.sh`. |
|
||||
| **minion-orchestrator** | Background work in one skill. Shell jobs via `gbrain jobs submit shell` (operator/CLI, MCP blocks protected names) and LLM subagents via `gbrain agent run`. Parent-child DAGs, `child_done` inbox, durability across worker restarts. |
|
||||
|
||||
### Identity and setup
|
||||
|
||||
@@ -1569,7 +1580,7 @@ Run `gbrain integrations` to see status.
|
||||
│ Brain Repo │ │ GBrain │ │ AI Agent │
|
||||
│ (git) │ │ (retrieval) │ │ (read/write) │
|
||||
│ │ │ │ │ │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 28 skills │
|
||||
│ markdown files │───>│ Postgres + │<──>│ 29 skills │
|
||||
│ = source of │ │ pgvector │ │ define HOW to │
|
||||
│ truth │ │ │ │ use the brain │
|
||||
│ │<───│ hybrid │ │ │
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.20.2",
|
||||
"version": "0.20.4",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+1
-1
@@ -58,7 +58,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
| "Validate skills", skill health check | `skills/testing/SKILL.md` |
|
||||
| Webhook setup, external event processing | `skills/webhook-transforms/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent" | `skills/minion-orchestrator/SKILL.md` |
|
||||
| "Spawn agent", "background task", "parallel tasks", "steer agent", "pause/resume agent", "gbrain jobs submit", "submit a gbrain job", "submit a shell job", "shell job" | `skills/minion-orchestrator/SKILL.md` |
|
||||
|
||||
## Setup & migration
|
||||
|
||||
|
||||
@@ -79,7 +79,7 @@ Even when Minions is the default (mode A), some work should run inline:
|
||||
|
||||
Before submitting batch jobs:
|
||||
|
||||
- Check `get_job_stats` queue_health.active
|
||||
- Check active queue depth via `list_jobs --status active` (MCP-callable) or `gbrain jobs stats` (CLI)
|
||||
- If active > 5, stagger new jobs with `delay` so you don't swarm
|
||||
- The resource governor auto-throttles but don't dump 20 jobs at once
|
||||
|
||||
|
||||
@@ -12,6 +12,12 @@ triggers:
|
||||
- "maintenance"
|
||||
- "orphan pages"
|
||||
- "stale pages"
|
||||
- "extract links"
|
||||
- "build link graph"
|
||||
- "populate timeline"
|
||||
- "populate links"
|
||||
- "backfill graph"
|
||||
- "extract timeline entries"
|
||||
tools:
|
||||
- get_health
|
||||
- get_page
|
||||
|
||||
@@ -132,7 +132,7 @@
|
||||
{
|
||||
"name": "minion-orchestrator",
|
||||
"path": "minion-orchestrator/SKILL.md",
|
||||
"description": "Manage background agents via Minions job queue. Submit, monitor, steer, pause/resume, replay. Replaces sessions_spawn for durable observable agents."
|
||||
"description": "Unified Minions skill for deterministic shell jobs and LLM subagent orchestration. Submit, monitor, steer, pause/resume, replay. Replaces the older gbrain-jobs routing intent and sessions_spawn for durable observable background work."
|
||||
},
|
||||
{
|
||||
"name": "skillify",
|
||||
|
||||
@@ -2,11 +2,18 @@
|
||||
name: minion-orchestrator
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Manage background agents via Minions job queue. Use when: spawning subagents,
|
||||
checking agent progress, steering running agents, pausing/resuming work,
|
||||
parallel task execution, fan-out research. Replaces sessions_spawn for
|
||||
durable, observable, steerable agents.
|
||||
Unified Minions skill for both deterministic shell jobs and LLM subagent
|
||||
orchestration. Replaces the older `gbrain-jobs` routing intent. Use when:
|
||||
submitting gbrain jobs, shell/background tasks, spawning subagents,
|
||||
checking progress, steering running work, pausing/resuming, parallel
|
||||
fan-out. One durable, observable, steerable queue interface.
|
||||
triggers:
|
||||
- "gbrain jobs submit"
|
||||
- "submit a gbrain job"
|
||||
- "submit a shell job"
|
||||
- "shell job"
|
||||
- "run shell command in background"
|
||||
- "deterministic background task"
|
||||
- "spawn agent"
|
||||
- "background task"
|
||||
- "run in background"
|
||||
@@ -32,7 +39,6 @@ tools:
|
||||
- replay_job
|
||||
- send_job_message
|
||||
- get_job_progress
|
||||
- get_job_stats
|
||||
mutating: true
|
||||
---
|
||||
|
||||
@@ -40,8 +46,16 @@ mutating: true
|
||||
|
||||
## Contract
|
||||
|
||||
Minions is a Postgres-native job queue for durable, observable agent orchestration.
|
||||
Every background agent task goes through Minions. No in-memory subagent spawning.
|
||||
Minions is a Postgres-native job queue for durable, observable background work.
|
||||
This single skill handles two lanes:
|
||||
- Deterministic shell jobs (`gbrain jobs submit shell ...`)
|
||||
- LLM subagent jobs (`gbrain agent run ...`)
|
||||
|
||||
When to route to Minions: durable, observable work that must survive restarts,
|
||||
fan out across many parallel tasks, or persist across sessions. Routing policy
|
||||
is defined in `skills/conventions/subagent-routing.md` — the project default is
|
||||
`pain_triggered` (native subagents first, Minions after specific pain signals
|
||||
fire); Mode A (all-through-Minions) is opt-in.
|
||||
|
||||
Guarantees:
|
||||
- Jobs survive gateway restart (Postgres-backed)
|
||||
@@ -50,51 +64,155 @@ Guarantees:
|
||||
- Jobs can be paused, resumed, or cancelled at any time
|
||||
- Parent-child DAGs with configurable failure policies
|
||||
|
||||
## When to Use Minions vs Inline Work
|
||||
## Route the Request: Shell Job vs Subagent
|
||||
|
||||
| Condition | Action |
|
||||
|---|---|
|
||||
| Single tool call, < 30s | Do it inline |
|
||||
| Multi-step, any duration | Submit as Minion job |
|
||||
| Parallel work (2+ streams) | Submit N Minion jobs with shared parent |
|
||||
| Needs to survive restart | Submit as Minion job |
|
||||
| User wants progress updates | Submit as Minion job with progress tracking |
|
||||
| Research / bulk operation | Submit as Minion job, always |
|
||||
| File imports, bulk embeds | Submit as Minion job |
|
||||
| User asks for deterministic command/script run | Shell job (CLI: `gbrain jobs submit shell ...`) |
|
||||
| User asks to "run in minions" + explicit command/argv | Shell job (CLI, `--params` with `cmd` or `argv`) |
|
||||
| User asks for research/reasoning/iterative agent | Subagent job (CLI: `gbrain agent run`) |
|
||||
| User asks to steer/pause/resume an agent | Subagent job lifecycle tools (MCP-callable) |
|
||||
| Single simple operation under ~30s | Consider inline execution first |
|
||||
| Needs restart durability/observability | Submit as Minion job |
|
||||
| Parallel work (2+ streams) | `gbrain agent run --fanout-manifest` or parent + child subagents |
|
||||
|
||||
**Rule of thumb:** If it takes more than 3 tool calls, use a Minion.
|
||||
If intent is ambiguous, ask one clarification:
|
||||
"Do you want a deterministic shell command job, or an LLM agent job?"
|
||||
|
||||
## Shell Jobs (Deterministic Scripts)
|
||||
|
||||
Use for reproducible command execution, ETL steps, cron work, and scriptable
|
||||
tasks where no LLM reasoning loop is needed.
|
||||
|
||||
### Preconditions (read before submitting your first shell job)
|
||||
|
||||
- **`GBRAIN_ALLOW_SHELL_JOBS=1` must be set on the worker environment.**
|
||||
Without it, the shell handler refuses to register and submissions sit in
|
||||
`waiting` silently. Gate lives in `src/core/minions/handlers/shell.ts`.
|
||||
- **Security:** flipping `GBRAIN_ALLOW_SHELL_JOBS=1` authorizes arbitrary
|
||||
command execution on the worker. On a shared queue, this is a remote code
|
||||
execution surface. Treat as privileged infrastructure authorization.
|
||||
- **Execution mode — pick one:**
|
||||
- **Postgres + daemon:** `gbrain jobs work` runs a persistent worker that
|
||||
claims and executes jobs from the queue.
|
||||
- **PGLite + --follow:** `gbrain jobs submit ... --follow` runs inline.
|
||||
The daemon mode is not available on PGLite (exclusive file lock). See
|
||||
`docs/guides/minions-shell-jobs.md`.
|
||||
- **MCP boundary:** shell-job submission is CLI-only. `submit_job name="shell"`
|
||||
over MCP throws an `OperationError` with code `permission_denied` ("'shell'
|
||||
jobs cannot be submitted over MCP") because `shell` is in `PROTECTED_JOB_NAMES`.
|
||||
Agents CAN observe shell jobs via `get_job` / `list_jobs` / `get_job_progress`
|
||||
(not protected), but cannot submit them. Operator or autopilot submits;
|
||||
agent observes.
|
||||
- **Verify setup:** after configuration, run `gbrain jobs stats` (CLI) to
|
||||
confirm the worker is registered and consuming the queue.
|
||||
|
||||
### Submit (CLI, operator or autopilot)
|
||||
|
||||
Shell jobs take their command via `--params` as a JSON object with `cmd` (string)
|
||||
or `argv` (array), plus `cwd` and optional `env`.
|
||||
|
||||
Command string form:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Argv form (no shell expansion):
|
||||
```
|
||||
gbrain jobs submit shell --params '{"argv":["bash","-lc","echo hello"],"cwd":"/abs/path"}'
|
||||
```
|
||||
|
||||
Inline execution on PGLite or any one-shot deployment:
|
||||
```
|
||||
gbrain jobs submit shell --params '{"cmd":"echo hello","cwd":"/tmp"}' --follow
|
||||
```
|
||||
|
||||
Queue/lifecycle flags exposed by `gbrain jobs submit --help`: `--queue`,
|
||||
`--priority`, `--delay`, `--max-attempts`, `--max-stalled`, `--backoff-type`,
|
||||
`--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`,
|
||||
`--dry-run`.
|
||||
|
||||
### Monitor (agents or operator)
|
||||
|
||||
These operations are MCP-callable and safe for agent use:
|
||||
|
||||
```
|
||||
list_jobs --name shell --status active
|
||||
get_job ID
|
||||
get_job_progress ID
|
||||
```
|
||||
|
||||
Check structured result fields (exit code, stdout/stderr tails, attempts,
|
||||
timings) from `get_job`. Use `gbrain jobs stats` (CLI) for worker/queue
|
||||
health dashboard.
|
||||
|
||||
### Control (MCP-callable)
|
||||
|
||||
```
|
||||
cancel_job id=ID
|
||||
replay_job id=ID
|
||||
```
|
||||
|
||||
`replay_job` is not protected — only shell *submission* is. Agents can
|
||||
cancel or replay a shell job without CLI access.
|
||||
|
||||
Use idempotency keys for recurring shell workloads to avoid duplicate runs.
|
||||
|
||||
## Subagent Jobs (LLM Orchestration)
|
||||
|
||||
Use for open-ended reasoning, tool-using research, and fan-out synthesis.
|
||||
|
||||
**User-facing entrypoint:** `gbrain agent run <prompt>` is the canonical way
|
||||
to submit subagent work. It handles the elevated-trust plumbing — `subagent`
|
||||
and `subagent_aggregator` are both in `PROTECTED_JOB_NAMES`, so direct MCP
|
||||
submission requires `{allowProtectedSubmit: true}`, which `gbrain agent run`
|
||||
supplies.
|
||||
|
||||
## Phase 1: Submit
|
||||
|
||||
```
|
||||
submit_job name="research" data={"prompt":"Research Acme Corp revenue","tools":["search","web_search"]}
|
||||
gbrain agent run "Research Acme Corp revenue" --tools "search,query"
|
||||
```
|
||||
|
||||
Options:
|
||||
- `queue` — queue name (default: 'default')
|
||||
- `priority` — lower = higher priority (default: 0)
|
||||
- `max_attempts` — retry limit (default: 3)
|
||||
- `delay` — ms delay before eligible
|
||||
`--tools` accepts a comma-separated subset of `BRAIN_TOOL_ALLOWLIST` (see
|
||||
`src/core/minions/tools/brain-allowlist.ts`): `query`, `search`, `get_page`,
|
||||
`list_pages`, `file_list`, `file_url`, `get_backlinks`, `traverse_graph`,
|
||||
`resolve_slugs`, `get_ingest_log`, `put_page`. Anything outside the allow-list
|
||||
is rejected at submit time with `allowed_tools references unknown tool`.
|
||||
|
||||
For parallel work, submit a parent then children:
|
||||
For parallel work with a fan-out manifest:
|
||||
```
|
||||
submit_job name="orchestrate" data={"task":"research 5 companies"}
|
||||
# Returns parent_id
|
||||
|
||||
submit_job name="research" data={"company":"Acme"} parent_job_id=PARENT_ID
|
||||
submit_job name="research" data={"company":"Beta"} parent_job_id=PARENT_ID
|
||||
submit_job name="research" data={"company":"Gamma"} parent_job_id=PARENT_ID
|
||||
gbrain agent run --fanout-manifest companies.json
|
||||
```
|
||||
|
||||
Parent auto-enters `waiting-children` and unblocks when all children finish.
|
||||
The manifest describes N children + 1 aggregator. Each child runs
|
||||
`name="subagent"` under the hood; the aggregator runs `name="subagent_aggregator"`
|
||||
and claims AFTER every child terminates. See
|
||||
`src/core/minions/handlers/subagent.ts` and
|
||||
`src/core/minions/handlers/subagent-aggregator.ts`.
|
||||
|
||||
Flags (from `src/commands/agent.ts`):
|
||||
- `--subagent-def <name>` — named subagent definition
|
||||
- `--model <id>` — override model
|
||||
- `--max-turns <N>` — cap the LLM loop
|
||||
- `--tools <csv>` — allow-listed brain tools (see above)
|
||||
- `--timeout-ms <N>` — hard timeout per job
|
||||
- `--fanout-manifest <file>` — N children + 1 aggregator
|
||||
- `--follow` / `--no-follow` — stream logs + wait (default on TTY)
|
||||
- `--detach` — submit and return immediately
|
||||
|
||||
Queue/priority/retry tuning is not exposed by `gbrain agent run`; submit the
|
||||
raw `subagent` handler via `gbrain jobs submit` (requires CLI trust) if you
|
||||
need those knobs.
|
||||
|
||||
## Phase 2: Monitor
|
||||
|
||||
```
|
||||
list_jobs --status active # what's running?
|
||||
get_job ID # full details + logs + tokens
|
||||
get_job_progress ID # structured progress snapshot
|
||||
get_job_stats # health dashboard
|
||||
list_jobs --status active # MCP — what's running?
|
||||
get_job ID # MCP — full details + logs + tokens
|
||||
get_job_progress ID # MCP — structured progress snapshot
|
||||
gbrain jobs stats # CLI — queue health dashboard
|
||||
gbrain agent logs ID --follow # CLI — streaming transcript + heartbeat
|
||||
```
|
||||
|
||||
Progress includes: step count, total steps, message, token usage, last tool called.
|
||||
@@ -121,6 +239,8 @@ replay_job id=ID # re-run with same or modified params
|
||||
replay_job id=ID data_overrides={"depth":"deep"} # replay with changes
|
||||
```
|
||||
|
||||
All lifecycle ops are MCP-callable.
|
||||
|
||||
## Phase 5: Review Results
|
||||
|
||||
```
|
||||
@@ -154,9 +274,9 @@ When reporting batch status (parent with children):
|
||||
|
||||
```
|
||||
Parent #ID — waiting-children
|
||||
#A research(Acme) — active, 3/5 steps, 2.5k tokens
|
||||
#B research(Beta) — completed, 1.8k tokens
|
||||
#C research(Gamma) — paused
|
||||
#A subagent(Acme) — active, 3/5 steps, 2.5k tokens
|
||||
#B subagent(Beta) — completed, 1.8k tokens
|
||||
#C subagent(Gamma) — paused
|
||||
Total tokens so far: 4.3k
|
||||
```
|
||||
|
||||
@@ -164,19 +284,19 @@ Total tokens so far: 4.3k
|
||||
|
||||
- Don't spawn a Minion for a single search query (use search tool directly)
|
||||
- Don't fire-and-forget without checking results
|
||||
- Don't spawn > 5 concurrent agents without checking `get_job_stats` first
|
||||
- Don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available
|
||||
- Don't spawn > 5 concurrent agents without checking `gbrain jobs stats` first
|
||||
- For subagent work, don't use `sessions_spawn` with `runtime: "subagent"` when Minions is available (use `gbrain agent run` instead)
|
||||
- Don't poll `get_job` in a tight loop (use `get_job_progress` for lightweight checks)
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Submit a background job (submit_job)
|
||||
- Get job details (get_job)
|
||||
- List jobs with filters (list_jobs)
|
||||
- Cancel a job (cancel_job)
|
||||
- Pause a job (pause_job)
|
||||
- Resume a paused job (resume_job)
|
||||
- Replay a completed/failed job (replay_job)
|
||||
- Send sidechannel message (send_job_message)
|
||||
- Get structured progress (get_job_progress)
|
||||
- Get job queue stats (get_job_stats)
|
||||
- Submit a background job — `submit_job` (MCP, non-protected names only; shell jobs are CLI-only, subagent jobs via `gbrain agent run`)
|
||||
- Get job details — `get_job` (MCP)
|
||||
- List jobs with filters — `list_jobs` (MCP)
|
||||
- Cancel a job — `cancel_job` (MCP)
|
||||
- Pause a job — `pause_job` (MCP)
|
||||
- Resume a paused job — `resume_job` (MCP)
|
||||
- Replay a completed/failed job — `replay_job` (MCP)
|
||||
- Send sidechannel message — `send_job_message` (MCP)
|
||||
- Get structured progress — `get_job_progress` (MCP)
|
||||
- Queue stats — `gbrain jobs stats` (CLI; no MCP equivalent)
|
||||
|
||||
@@ -12,6 +12,10 @@ triggers:
|
||||
- "what happened"
|
||||
- "search for"
|
||||
- "look up"
|
||||
- "who knows who"
|
||||
- "relationship between"
|
||||
- "connections"
|
||||
- "graph query"
|
||||
tools:
|
||||
- search
|
||||
- query
|
||||
|
||||
@@ -10,6 +10,7 @@ triggers:
|
||||
- "container restart check"
|
||||
- "health check"
|
||||
- "did the restart break anything"
|
||||
- "did the container restart break anything"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
|
||||
@@ -75,10 +75,14 @@ export function resolveGbrainCliPath(): string {
|
||||
throw new Error('Could not resolve the gbrain CLI path. Install gbrain so it is on $PATH (e.g. /usr/local/bin/gbrain), or run autopilot from the compiled binary directly.');
|
||||
}
|
||||
|
||||
export function shouldSpawnAutopilotWorker(args: string[]): boolean {
|
||||
return !args.includes('--no-worker');
|
||||
}
|
||||
|
||||
export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log(
|
||||
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json]\n' +
|
||||
'Usage: gbrain autopilot [--repo <path>] [--interval N] [--json] [--no-worker]\n' +
|
||||
' gbrain autopilot --install [--repo <path>]\n' +
|
||||
' gbrain autopilot --uninstall\n' +
|
||||
' gbrain autopilot --status [--json]\n\n' +
|
||||
@@ -106,6 +110,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const baseInterval = parseInt(parseArg(args, '--interval') || '300', 10);
|
||||
const jsonMode = args.includes('--json');
|
||||
const forceInline = args.includes('--inline');
|
||||
const noWorker = !shouldSpawnAutopilotWorker(args);
|
||||
|
||||
if (!repoPath) {
|
||||
console.error('No repo path. Use --repo or run gbrain sync --repo first.');
|
||||
@@ -137,12 +142,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const cfg = loadConfig();
|
||||
const engineType = cfg?.engine ?? 'pglite';
|
||||
const useMinionsDispatch = mode !== 'off' && engineType === 'postgres' && !forceInline;
|
||||
const spawnManagedWorker = useMinionsDispatch && !noWorker;
|
||||
|
||||
let stopping = false;
|
||||
let workerProc: ChildProcess | null = null;
|
||||
let crashCount = 0;
|
||||
|
||||
if (useMinionsDispatch) {
|
||||
if (spawnManagedWorker) {
|
||||
const cliPath = resolveGbrainCliPath();
|
||||
const startWorker = () => {
|
||||
const child = spawn(cliPath, ['jobs', 'work'], { stdio: 'inherit', env: process.env });
|
||||
@@ -161,10 +167,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
});
|
||||
};
|
||||
startWorker();
|
||||
} else {
|
||||
const why = mode === 'off' ? 'minion_mode=off'
|
||||
} else if (!useMinionsDispatch) {
|
||||
const why = mode === 'off'
|
||||
? 'minion_mode=off'
|
||||
: (engineType !== 'postgres' ? 'engine=pglite' : 'flag=--inline');
|
||||
console.log(`[autopilot] running steps inline (${why})`);
|
||||
} else {
|
||||
console.log('[autopilot] --no-worker set: dispatch loop only (worker managed externally)');
|
||||
}
|
||||
|
||||
// Async shutdown with 35s drain window for the worker child. The worker
|
||||
@@ -195,6 +204,18 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
process.on('SIGINT', () => { void shutdown('SIGINT'); });
|
||||
|
||||
let consecutiveErrors = 0;
|
||||
// Peer-worker liveness for --no-worker mode. The probe is a proxy, not
|
||||
// ground truth: SELECT count(*) of active jobs with a recent lock_until
|
||||
// refresh. A queue with only waiting jobs and a healthy idle worker
|
||||
// reads as "no worker" (false positive); a worker that died 110s ago
|
||||
// while holding a lock reads as "alive" until lock_until expires.
|
||||
// Good enough for V1 — a ground-truth minion_workers heartbeat table
|
||||
// is tracked as v0.19.1 follow-up B7. When the probe sees no signal
|
||||
// for NO_WORKER_WARN_TICKS consecutive cycles, log a loud warning so
|
||||
// the operator can spot "I set --no-worker but forgot to start one"
|
||||
// before the queue piles up.
|
||||
const NO_WORKER_WARN_TICKS = 3;
|
||||
let noWorkerConsecutiveIdle = 0;
|
||||
|
||||
while (!stopping) {
|
||||
const cycleStart = Date.now();
|
||||
@@ -214,6 +235,43 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
} catch (e) { logError('reconnect', e); }
|
||||
}
|
||||
|
||||
// --no-worker peer-liveness probe (v0.19.1). Runs every cycle, cheap
|
||||
// (single SELECT). See NO_WORKER_WARN_TICKS comment above for caveats.
|
||||
if (noWorker && useMinionsDispatch) {
|
||||
try {
|
||||
const rows = await (engine as any).executeRaw?.(
|
||||
`SELECT count(*)::int AS n FROM minion_jobs
|
||||
WHERE status = 'active'
|
||||
AND lock_until IS NOT NULL
|
||||
AND lock_until > now() - interval '2 minutes'`,
|
||||
);
|
||||
const liveWorkerSignal = Number((rows as Array<{ n: number }>)?.[0]?.n ?? 0);
|
||||
if (liveWorkerSignal === 0) {
|
||||
noWorkerConsecutiveIdle++;
|
||||
if (noWorkerConsecutiveIdle === NO_WORKER_WARN_TICKS) {
|
||||
// Fire loud on the Nth consecutive idle tick; don't repeat on every
|
||||
// subsequent cycle (the operator already saw it), re-arm once a
|
||||
// live worker is seen again.
|
||||
console.error(
|
||||
`[autopilot] WARNING: --no-worker set and no worker has claimed a job in ~${NO_WORKER_WARN_TICKS * baseInterval}s. ` +
|
||||
`Jobs will pile up in 'waiting' until a worker starts. ` +
|
||||
`Probe is a proxy (lock_until refresh) and can false-positive on idle queues — see B7 for ground-truth follow-up.`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
if (noWorkerConsecutiveIdle >= NO_WORKER_WARN_TICKS) {
|
||||
console.log('[autopilot] --no-worker probe: live worker signal detected; warning re-armed.');
|
||||
}
|
||||
noWorkerConsecutiveIdle = 0;
|
||||
}
|
||||
} catch (e) {
|
||||
// Probe failures never block the main dispatch loop. Log once per
|
||||
// failure class; ignore repeated errors (common shape: DB reconnect
|
||||
// blip between ticks).
|
||||
logError('no-worker-probe', e);
|
||||
}
|
||||
}
|
||||
|
||||
if (useMinionsDispatch) {
|
||||
// Submit ONE autopilot-cycle job per cycle slot. The idempotency key
|
||||
// dedupes overrun submissions — if a cycle's job runs longer than
|
||||
|
||||
@@ -649,6 +649,106 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 11b. Queue health (v0.19.1 queue-resilience wave).
|
||||
// Postgres-only because PGLite has no multi-process worker surface. Two
|
||||
// subchecks, both cheap (single SELECT each, status-index-covered):
|
||||
//
|
||||
// 1. stalled-forever: any active job whose started_at is > 1h old. The
|
||||
// incident that motivated this release ran 90+ min before surfacing.
|
||||
// Surface the ID so the operator can `gbrain jobs get <id>` to inspect
|
||||
// or `gbrain jobs cancel <id>` to force-kill.
|
||||
//
|
||||
// 2. backpressure-missed: per-name waiting depth exceeds the threshold
|
||||
// (default 10, override via GBRAIN_QUEUE_WAITING_THRESHOLD env). Signal
|
||||
// that a submitter probably needs maxWaiting set. Bounded by per-name
|
||||
// aggregation so a single name's pile shows up clearly instead of
|
||||
// getting lost in the total.
|
||||
//
|
||||
// Not included in v0.19.1 (tracked as B7 follow-up): worker-heartbeat
|
||||
// staleness. It needs a minion_workers table; the lock_until-on-active-jobs
|
||||
// proxy can't distinguish "no worker" from "worker idle," and a check that
|
||||
// cries wolf erodes trust in every other doctor check.
|
||||
progress.heartbeat('queue_health');
|
||||
if (engine.kind === 'pglite') {
|
||||
checks.push({
|
||||
name: 'queue_health',
|
||||
status: 'ok',
|
||||
message: 'Skipped (PGLite — no multi-process worker surface)',
|
||||
});
|
||||
} else {
|
||||
const queueHealthHb = startHeartbeat(progress, 'scanning queue health…');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
// Subcheck 1: stalled-forever active jobs (>1h wall-clock).
|
||||
const stalledRows: Array<{ id: number; name: string; started_at: string }> = await sql`
|
||||
SELECT id, name, started_at::text AS started_at
|
||||
FROM minion_jobs
|
||||
WHERE status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND started_at < now() - interval '1 hour'
|
||||
ORDER BY started_at ASC
|
||||
LIMIT 5
|
||||
`;
|
||||
// Subcheck 2: per-name waiting depth exceeds threshold.
|
||||
const rawThreshold = process.env.GBRAIN_QUEUE_WAITING_THRESHOLD;
|
||||
const parsedThreshold = rawThreshold ? parseInt(rawThreshold, 10) : 10;
|
||||
const threshold = Number.isFinite(parsedThreshold) && parsedThreshold >= 1
|
||||
? parsedThreshold
|
||||
: 10;
|
||||
const depthRows: Array<{ name: string; queue: string; depth: number }> = await sql`
|
||||
SELECT name, queue, count(*)::int AS depth
|
||||
FROM minion_jobs
|
||||
WHERE status = 'waiting'
|
||||
GROUP BY name, queue
|
||||
HAVING count(*) > ${threshold}
|
||||
ORDER BY depth DESC
|
||||
LIMIT 5
|
||||
`;
|
||||
|
||||
const problems: string[] = [];
|
||||
if (stalledRows.length > 0) {
|
||||
const sample = stalledRows
|
||||
.map(r => `#${r.id}(${r.name})`)
|
||||
.join(', ');
|
||||
problems.push(
|
||||
`${stalledRows.length} stalled-forever job(s): ${sample}. ` +
|
||||
`Fix: gbrain jobs get <id> to inspect; gbrain jobs cancel <id> to force-kill.`
|
||||
);
|
||||
}
|
||||
if (depthRows.length > 0) {
|
||||
const sample = depthRows
|
||||
.map(r => `${r.name}@${r.queue}=${r.depth}`)
|
||||
.join(', ');
|
||||
problems.push(
|
||||
`waiting-queue depth exceeds ${threshold} for: ${sample}. ` +
|
||||
`Fix: set maxWaiting on the submitter (or raise GBRAIN_QUEUE_WAITING_THRESHOLD).`
|
||||
);
|
||||
}
|
||||
|
||||
if (problems.length === 0) {
|
||||
checks.push({
|
||||
name: 'queue_health',
|
||||
status: 'ok',
|
||||
message: `No stalled-forever jobs; no queue over depth ${threshold}.`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'queue_health',
|
||||
status: 'warn',
|
||||
message: problems.join(' '),
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
checks.push({
|
||||
name: 'queue_health',
|
||||
status: 'warn',
|
||||
message: `queue_health scan skipped: ${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
} finally {
|
||||
queueHealthHb();
|
||||
}
|
||||
}
|
||||
|
||||
// 12. Index audit (opt-in via --index-audit). v0.13.1 follow-up to #170.
|
||||
// Reports indexes with zero recorded scans on Postgres. Informational only;
|
||||
// we DO NOT auto-drop. On #170's brain, idx_pages_frontmatter and
|
||||
|
||||
@@ -295,6 +295,13 @@ export interface ExtractOpts {
|
||||
dryRun?: boolean;
|
||||
/** Emit JSON (progress to stderr, result to stdout) instead of human text. */
|
||||
jsonMode?: boolean;
|
||||
/**
|
||||
* Incremental mode: only extract from these specific slugs.
|
||||
* When provided, skips the full directory walk and reads only the
|
||||
* files corresponding to these slugs. Massive perf win on large brains.
|
||||
* Pass undefined or omit for a full walk (CLI / first-run path).
|
||||
*/
|
||||
slugs?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -315,6 +322,21 @@ export async function runExtractCore(engine: BrainEngine, opts: ExtractOpts): Pr
|
||||
const jsonMode = !!opts.jsonMode;
|
||||
const result: ExtractResult = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
|
||||
|
||||
// Incremental path: if specific slugs provided, only extract from those files.
|
||||
// This is the cycle path — sync tells us what changed, we only re-extract those.
|
||||
if (opts.slugs !== undefined) {
|
||||
if (opts.slugs.length === 0) {
|
||||
// Nothing changed — skip entirely.
|
||||
return result;
|
||||
}
|
||||
const r = await extractForSlugs(engine, opts.dir, opts.slugs, opts.mode, dryRun, jsonMode);
|
||||
result.links_created = r.links_created;
|
||||
result.timeline_entries_created = r.timeline_created;
|
||||
result.pages_processed = r.pages;
|
||||
return result;
|
||||
}
|
||||
|
||||
// Full walk path: CLI `gbrain extract` or first-run.
|
||||
if (opts.mode === 'links' || opts.mode === 'all') {
|
||||
const r = await extractLinksFromDir(engine, opts.dir, dryRun, jsonMode);
|
||||
result.links_created = r.created;
|
||||
@@ -411,6 +433,118 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Incremental extract: process only the specified slugs.
|
||||
*
|
||||
* Instead of walking 54K+ files, reads only the files that sync says changed.
|
||||
* Still needs the full slug set for link resolution (resolveSlug needs to know
|
||||
* all valid targets), but that's a single readdir, not 54K readFileSync calls.
|
||||
*
|
||||
* Combines links + timeline extraction in a single pass over each file —
|
||||
* the full-walk path reads every file TWICE (once for links, once for timeline).
|
||||
*/
|
||||
async function extractForSlugs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
slugs: string[],
|
||||
mode: 'links' | 'timeline' | 'all',
|
||||
dryRun: boolean,
|
||||
jsonMode: boolean,
|
||||
): Promise<{ links_created: number; timeline_created: number; pages: number }> {
|
||||
// Build the full slug set for link resolution (fast: just readdir, no file reads)
|
||||
const allFiles = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(allFiles.map(f => f.relPath.replace('.md', '')));
|
||||
|
||||
const doLinks = mode === 'links' || mode === 'all';
|
||||
const doTimeline = mode === 'timeline' || mode === 'all';
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('extract.incremental', slugs.length);
|
||||
|
||||
let linksCreated = 0;
|
||||
let timelineCreated = 0;
|
||||
let pagesProcessed = 0;
|
||||
|
||||
const linkBatch: LinkBatchInput[] = [];
|
||||
const timelineBatch: TimelineBatchInput[] = [];
|
||||
|
||||
async function flushLinks() {
|
||||
if (linkBatch.length === 0) return;
|
||||
try {
|
||||
linksCreated += await engine.addLinksBatch(linkBatch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (!jsonMode) console.error(` link batch error (${linkBatch.length} rows lost): ${msg}`);
|
||||
} finally {
|
||||
linkBatch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
async function flushTimeline() {
|
||||
if (timelineBatch.length === 0) return;
|
||||
try {
|
||||
timelineCreated += await engine.addTimelineEntriesBatch(timelineBatch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (!jsonMode) console.error(` timeline batch error (${timelineBatch.length} rows lost): ${msg}`);
|
||||
} finally {
|
||||
timelineBatch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (const slug of slugs) {
|
||||
const relPath = slug + '.md';
|
||||
const fullPath = join(brainDir, relPath);
|
||||
|
||||
try {
|
||||
if (!existsSync(fullPath)) continue; // deleted file — sync already handled removal
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
|
||||
// Links
|
||||
if (doLinks) {
|
||||
const links = await extractLinksFromFile(content, relPath, allSlugs);
|
||||
for (const link of links) {
|
||||
if (dryRun) {
|
||||
if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`);
|
||||
linksCreated++;
|
||||
} else {
|
||||
linkBatch.push(link);
|
||||
if (linkBatch.length >= BATCH_SIZE) await flushLinks();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Timeline
|
||||
if (doTimeline) {
|
||||
const entries = extractTimelineFromContent(content, slug);
|
||||
for (const entry of entries) {
|
||||
if (dryRun) {
|
||||
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date} — ${entry.summary}`);
|
||||
timelineCreated++;
|
||||
} else {
|
||||
timelineBatch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
if (timelineBatch.length >= BATCH_SIZE) await flushTimeline();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pagesProcessed++;
|
||||
} catch { /* skip unreadable */ }
|
||||
progress.tick(1);
|
||||
}
|
||||
|
||||
await flushLinks();
|
||||
await flushTimeline();
|
||||
progress.finish();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
console.log(`Incremental extract: ${label} ${linksCreated} link(s), ${timelineCreated} timeline entries from ${pagesProcessed}/${slugs.length} page(s)`);
|
||||
}
|
||||
|
||||
return { links_created: linksCreated, timeline_created: timelineCreated, pages: pagesProcessed };
|
||||
}
|
||||
|
||||
async function extractLinksFromDir(
|
||||
engine: BrainEngine, brainDir: string, dryRun: boolean, jsonMode: boolean,
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
|
||||
+118
-10
@@ -17,6 +17,42 @@ function hasFlag(args: string[], flag: string): boolean {
|
||||
return args.includes(flag);
|
||||
}
|
||||
|
||||
/** Parse `--max-waiting N` from CLI args. Returns undefined if absent.
|
||||
* Throws on malformed input (caller should surface the error and exit).
|
||||
* Clamps to [1, 100] to match the queue-layer clamp in MinionQueue.add.
|
||||
* Exported for unit tests; the CLI handler at `jobs submit` wraps this
|
||||
* with process.exit(1) on throw so operators see 'must be positive integer'. */
|
||||
export function parseMaxWaitingFlag(args: string[]): number | undefined {
|
||||
const raw = parseFlag(args, '--max-waiting');
|
||||
if (raw === undefined) return undefined;
|
||||
const parsed = parseInt(raw, 10);
|
||||
if (!Number.isFinite(parsed) || parsed < 1) {
|
||||
throw new Error('--max-waiting must be a positive integer (will be clamped to [1, 100])');
|
||||
}
|
||||
return Math.max(1, Math.min(100, parsed));
|
||||
}
|
||||
|
||||
export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv = process.env): number {
|
||||
const raw = parseFlag(args, '--concurrency') ?? env.GBRAIN_WORKER_CONCURRENCY ?? '1';
|
||||
const parsed = parseInt(raw, 10);
|
||||
// Without validation, NaN / 0 / negative values flow through to the worker
|
||||
// loop where `inFlight.size < concurrency` is always false → the worker
|
||||
// claims zero jobs and the queue silently wedges. One typo in a systemd
|
||||
// unit reproduces the original production incident. Clamp to ≥1 and surface
|
||||
// the misconfig loudly so operators see it at worker startup.
|
||||
if (!Number.isFinite(parsed) || parsed < 1) {
|
||||
const source = parseFlag(args, '--concurrency') !== undefined
|
||||
? '--concurrency flag'
|
||||
: 'GBRAIN_WORKER_CONCURRENCY env';
|
||||
process.stderr.write(
|
||||
`[gbrain jobs] invalid concurrency from ${source} (${JSON.stringify(raw)}); ` +
|
||||
`falling back to 1. Set a positive integer.\n`
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function formatJob(job: MinionJob): string {
|
||||
const dur = job.finished_at && job.started_at
|
||||
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
|
||||
@@ -58,6 +94,7 @@ export async function runJobs(engine: BrainEngine, args: string[]): Promise<void
|
||||
USAGE
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
|
||||
[--delay Nms] [--max-attempts N] [--max-stalled N]
|
||||
[--max-waiting N]
|
||||
[--backoff-type fixed|exponential] [--backoff-delay Nms]
|
||||
[--backoff-jitter 0..1] [--timeout-ms Nms]
|
||||
[--idempotency-key K] [--queue Q] [--dry-run]
|
||||
@@ -144,6 +181,12 @@ HANDLER TYPES (built in)
|
||||
const maxAttempts = parseInt(parseFlag(args, '--max-attempts') ?? '3', 10);
|
||||
const maxStalledRaw = parseFlag(args, '--max-stalled');
|
||||
const maxStalled = maxStalledRaw !== undefined ? parseInt(maxStalledRaw, 10) : undefined;
|
||||
// --max-waiting N: submission-time backpressure cap. Mirrors --max-stalled
|
||||
// clamp [1, 100]. Feature is usable from CLI as of v0.19.1; pre-v0.19.1
|
||||
// only programmatic callers reached it.
|
||||
let maxWaiting: number | undefined;
|
||||
try { maxWaiting = parseMaxWaitingFlag(args); }
|
||||
catch (e) { console.error(`Error: ${e instanceof Error ? e.message : String(e)}`); process.exit(1); }
|
||||
// v0.13.1 field audit: expose retry/backoff/timeout/idempotency knobs so
|
||||
// users can tune Minions behavior without dropping into TypeScript.
|
||||
const backoffTypeRaw = parseFlag(args, '--backoff-type');
|
||||
@@ -172,6 +215,7 @@ HANDLER TYPES (built in)
|
||||
console.log(` Priority: ${priority}`);
|
||||
console.log(` Max attempts: ${maxAttempts}`);
|
||||
if (maxStalled !== undefined) console.log(` Max stalled: ${maxStalled}`);
|
||||
if (maxWaiting !== undefined) console.log(` Max waiting: ${maxWaiting}`);
|
||||
if (backoffType) console.log(` Backoff type: ${backoffType}`);
|
||||
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
|
||||
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
|
||||
@@ -199,6 +243,7 @@ HANDLER TYPES (built in)
|
||||
delay: delay > 0 ? delay : undefined,
|
||||
max_attempts: maxAttempts,
|
||||
max_stalled: maxStalled,
|
||||
maxWaiting,
|
||||
backoff_type: backoffType,
|
||||
backoff_delay: backoffDelay,
|
||||
backoff_jitter: backoffJitter,
|
||||
@@ -415,6 +460,7 @@ HANDLER TYPES (built in)
|
||||
}
|
||||
|
||||
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
|
||||
const wedgeRescue = hasFlag(args, '--wedge-rescue');
|
||||
|
||||
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
|
||||
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
|
||||
@@ -481,9 +527,70 @@ HANDLER TYPES (built in)
|
||||
try { await queue.removeJob(rescueJob.id); } catch { /* non-fatal cleanup */ }
|
||||
}
|
||||
|
||||
// --wedge-rescue: regression case for the v0.19.1 production incident.
|
||||
// In prod, a wedged worker held a row lock via a pending txn. The
|
||||
// lock-renewal UPDATE blocked, lock_until fell below now(), handleStalled
|
||||
// saw the candidate but FOR UPDATE SKIP LOCKED skipped (row lock held),
|
||||
// handleTimeouts was disqualified (lock_until > now() fails).
|
||||
// Only handleWallClockTimeouts' no-constraint sweep evicted.
|
||||
//
|
||||
// The smoke is single-connection, so we can't simulate a row lock held
|
||||
// by another txn. Instead we forge the state where BOTH handleStalled
|
||||
// and handleTimeouts are disqualified so only wall-clock fires:
|
||||
// - lock_until far in the future → handleStalled skips (not a stall)
|
||||
// - timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
|
||||
// - started_at 10s ago with timeout_ms=1000 → wall-clock matches
|
||||
// (2 × timeout_ms = 2000ms threshold exceeded)
|
||||
if (wedgeRescue) {
|
||||
const wedgedJob = await queue.add('noop', {}, {
|
||||
queue: 'smoke',
|
||||
timeout_ms: 1000,
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
lock_token='smoke-wedge-rescue',
|
||||
lock_until=now() + interval '30 seconds',
|
||||
started_at=now() - interval '10 seconds',
|
||||
timeout_at=NULL,
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[wedgedJob.id]
|
||||
);
|
||||
|
||||
const stallResult = await queue.handleStalled();
|
||||
const stalledStatus = await queue.getJob(wedgedJob.id);
|
||||
const timeoutResult = await queue.handleTimeouts();
|
||||
const timedStatus = await queue.getJob(wedgedJob.id);
|
||||
const wallResult = await queue.handleWallClockTimeouts(30000);
|
||||
const finalStatus = await queue.getJob(wedgedJob.id);
|
||||
|
||||
if (finalStatus?.status !== 'dead') {
|
||||
console.error(
|
||||
`SMOKE FAIL (--wedge-rescue) — wall-clock sweep did not evict job #${wedgedJob.id}. ` +
|
||||
`Status: ${finalStatus?.status}. ` +
|
||||
`handleStalled: requeued=${stallResult.requeued.length} dead=${stallResult.dead.length}, after: ${stalledStatus?.status}; ` +
|
||||
`handleTimeouts: ${timeoutResult.length}, after: ${timedStatus?.status}; ` +
|
||||
`handleWallClockTimeouts: ${wallResult.length}, final: ${finalStatus?.status}.`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
if (finalStatus.error_text !== 'wall-clock timeout exceeded') {
|
||||
console.error(
|
||||
`SMOKE FAIL (--wedge-rescue) — dead, but error_text='${finalStatus.error_text}' ` +
|
||||
`(expected 'wall-clock timeout exceeded').`
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
try { await queue.removeJob(wedgedJob.id); } catch { /* non-fatal cleanup */ }
|
||||
}
|
||||
|
||||
const cfg = (await import('../core/config.ts')).loadConfig();
|
||||
const engineLabel = cfg?.engine ?? 'unknown';
|
||||
const tag = sigkillRescue ? ' + SIGKILL rescue' : '';
|
||||
const tags: string[] = [];
|
||||
if (sigkillRescue) tags.push('SIGKILL rescue');
|
||||
if (wedgeRescue) tags.push('wedge rescue');
|
||||
const tag = tags.length > 0 ? ` + ${tags.join(' + ')}` : '';
|
||||
console.log(`SMOKE PASS — Minions healthy${tag} in ${elapsedSec}s (engine: ${engineLabel})`);
|
||||
if (engineLabel === 'pglite') {
|
||||
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
|
||||
@@ -503,7 +610,7 @@ HANDLER TYPES (built in)
|
||||
}
|
||||
|
||||
const queueName = parseFlag(args, '--queue') ?? 'default';
|
||||
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '1', 10);
|
||||
const concurrency = resolveWorkerConcurrency(args);
|
||||
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
@@ -819,16 +926,17 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
};
|
||||
});
|
||||
|
||||
// Shell handler: registered ONLY when GBRAIN_ALLOW_SHELL_JOBS=1 is set on the
|
||||
// worker process. Default-closed; opt-in per-host. Without the flag, shell
|
||||
// jobs submitted via CLI insert rows but no worker claims them (they sit in
|
||||
// 'waiting' — the CLI prints a starvation warning for that case).
|
||||
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
|
||||
// Shell handler is always registered. Runtime env guard lives inside the
|
||||
// handler so claimed jobs emit a clear rejection log on workers missing
|
||||
// GBRAIN_ALLOW_SHELL_JOBS=1.
|
||||
{
|
||||
const { shellHandler } = await import('../core/minions/handlers/shell.ts');
|
||||
worker.register('shell', shellHandler);
|
||||
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
|
||||
} else {
|
||||
process.stderr.write('[minion worker] shell handler disabled (set GBRAIN_ALLOW_SHELL_JOBS=1 to enable)\n');
|
||||
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
|
||||
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
|
||||
} else {
|
||||
process.stderr.write('[minion worker] shell handler registered in guarded mode (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)\n');
|
||||
}
|
||||
}
|
||||
|
||||
// v0.15 subagent handlers: always-on. Unlike shell (which needs an env
|
||||
|
||||
+34
-5
@@ -416,12 +416,18 @@ async function runPhaseBacklinks(brainDir: string, dryRun: boolean): Promise<Pha
|
||||
}
|
||||
}
|
||||
|
||||
/** Extended sync result that also carries the changed slug list for downstream phases. */
|
||||
interface SyncPhaseResult extends PhaseResult {
|
||||
/** Slugs that sync added or modified. Used by extract for incremental processing. */
|
||||
pagesAffected?: string[];
|
||||
}
|
||||
|
||||
async function runPhaseSync(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
pull: boolean,
|
||||
): Promise<PhaseResult> {
|
||||
): Promise<SyncPhaseResult> {
|
||||
try {
|
||||
const { performSync } = await import('../commands/sync.ts');
|
||||
const result = await performSync(engine, {
|
||||
@@ -448,6 +454,7 @@ async function runPhaseSync(
|
||||
syncStatus: result.status,
|
||||
dryRun,
|
||||
},
|
||||
pagesAffected: result.pagesAffected,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
@@ -465,6 +472,7 @@ async function runPhaseExtract(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
dryRun: boolean,
|
||||
changedSlugs?: string[],
|
||||
): Promise<PhaseResult> {
|
||||
try {
|
||||
const { runExtractCore } = await import('../commands/extract.ts');
|
||||
@@ -480,15 +488,29 @@ async function runPhaseExtract(
|
||||
details: { dryRun: true, reason: 'no_dry_run_support' },
|
||||
};
|
||||
}
|
||||
const result = await runExtractCore(engine, { mode: 'all', dir: brainDir });
|
||||
// Incremental path: if sync told us which slugs changed, only extract those.
|
||||
// On a 54K-page brain this turns a 10-minute full walk into a sub-second pass.
|
||||
const result = await runExtractCore(engine, {
|
||||
mode: 'all',
|
||||
dir: brainDir,
|
||||
slugs: changedSlugs, // undefined = full walk (first run / manual)
|
||||
});
|
||||
const linksCreated = result?.links_created ?? 0;
|
||||
const timelineCreated = result?.timeline_entries_created ?? 0;
|
||||
const incremental = changedSlugs !== undefined;
|
||||
return {
|
||||
phase: 'extract',
|
||||
status: 'ok',
|
||||
duration_ms: 0,
|
||||
summary: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
|
||||
details: { linksCreated, timelineCreated, pages_processed: result?.pages_processed ?? 0 },
|
||||
summary: incremental
|
||||
? `${linksCreated} link(s), ${timelineCreated} timeline entries (incremental: ${changedSlugs.length} slugs)`
|
||||
: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
|
||||
details: {
|
||||
linksCreated, timelineCreated,
|
||||
pages_processed: result?.pages_processed ?? 0,
|
||||
incremental,
|
||||
...(incremental ? { slugs_targeted: changedSlugs.length } : {}),
|
||||
},
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
@@ -663,6 +685,8 @@ export async function runCycle(
|
||||
}
|
||||
|
||||
// ── Phase 3: sync ───────────────────────────────────────────
|
||||
// Track which slugs sync touched so extract can run incrementally.
|
||||
let syncPagesAffected: string[] | undefined;
|
||||
if (phases.includes('sync')) {
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
@@ -676,6 +700,8 @@ export async function runCycle(
|
||||
progress.start('cycle.sync');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull));
|
||||
result.duration_ms = duration_ms;
|
||||
// Capture changed slugs for incremental extract.
|
||||
syncPagesAffected = (result as SyncPhaseResult).pagesAffected;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
}
|
||||
@@ -693,8 +719,11 @@ export async function runCycle(
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
// Pass changed slugs from sync for incremental extract.
|
||||
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
|
||||
// is undefined → extract falls back to full walk (safe default).
|
||||
progress.start('cycle.extract');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun, syncPagesAffected));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* Backpressure audit log — operational trace for `maxWaiting` coalesce events.
|
||||
*
|
||||
* Mirrors the shell-audit.ts pattern (ISO-week-rotated JSONL, best-effort writes,
|
||||
* failures go to stderr but never block submission). The incident that motivated
|
||||
* maxWaiting (autopilot pile-up during a 90+ min queue wedge) was invisible
|
||||
* precisely because the coalesce silently dropped repeat submissions. This
|
||||
* trail answers "why is queue depth steady at 2 for this name?" without any
|
||||
* doctor scan.
|
||||
*
|
||||
* File: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (override dir via
|
||||
* `GBRAIN_AUDIT_DIR` for container/sandbox deployments where `$HOME` is read-only).
|
||||
*
|
||||
* `gbrain jobs stats` will surface coalesce counts from this file in a v0.19.2+
|
||||
* follow-up (B4). The audit trail is for operators debugging live queues, not
|
||||
* for compliance — a disk-full attacker can silently disable it.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import * as os from 'node:os';
|
||||
|
||||
export interface BackpressureAuditEvent {
|
||||
ts: string;
|
||||
queue: string;
|
||||
name: string;
|
||||
waiting_count: number;
|
||||
max_waiting: number;
|
||||
decision: 'coalesced';
|
||||
returned_job_id: number;
|
||||
}
|
||||
|
||||
/** Compute `backpressure-YYYY-Www.jsonl` using ISO-8601 week numbering.
|
||||
*
|
||||
* Copy of the shell-audit computeAuditFilename algorithm, parameterized on
|
||||
* the filename prefix. Keeping the math inline (rather than re-exporting from
|
||||
* shell-audit.ts) avoids a cross-module dependency between two best-effort
|
||||
* audit surfaces — one can be rewritten without touching the other.
|
||||
*/
|
||||
export function computeAuditFilename(now: Date = new Date()): string {
|
||||
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
|
||||
const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6
|
||||
d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday
|
||||
const isoYear = d.getUTCFullYear();
|
||||
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
|
||||
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
|
||||
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
|
||||
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
|
||||
const ww = String(weekNum).padStart(2, '0');
|
||||
return `backpressure-${isoYear}-W${ww}.jsonl`;
|
||||
}
|
||||
|
||||
/** Honors `GBRAIN_AUDIT_DIR` for container/sandbox deployments. */
|
||||
export function resolveAuditDir(): string {
|
||||
const override = process.env.GBRAIN_AUDIT_DIR;
|
||||
if (override && override.trim().length > 0) return override;
|
||||
return path.join(os.homedir(), '.gbrain', 'audit');
|
||||
}
|
||||
|
||||
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
|
||||
const dir = resolveAuditDir();
|
||||
const filename = computeAuditFilename();
|
||||
const fullPath = path.join(dir, filename);
|
||||
const line = JSON.stringify({
|
||||
...event,
|
||||
decision: 'coalesced' as const,
|
||||
ts: new Date().toISOString(),
|
||||
}) + '\n';
|
||||
|
||||
try {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.appendFileSync(fullPath, line, { encoding: 'utf8' });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[backpressure-audit] write failed (${msg}); submission continues\n`);
|
||||
}
|
||||
}
|
||||
@@ -207,6 +207,16 @@ class TailBuffer {
|
||||
|
||||
/** The shell handler itself. */
|
||||
export async function shellHandler(ctx: MinionJobContext): Promise<ShellJobResult> {
|
||||
if (process.env.GBRAIN_ALLOW_SHELL_JOBS !== '1') {
|
||||
const warning =
|
||||
`[shell] Job #${ctx.id} rejected: GBRAIN_ALLOW_SHELL_JOBS=1 not set on this worker.\n` +
|
||||
' Shell jobs require the env var on the worker process.';
|
||||
console.warn(warning);
|
||||
throw new UnrecoverableError(
|
||||
'shell handler disabled on this worker (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)',
|
||||
);
|
||||
}
|
||||
|
||||
const params = validateParams(ctx.data);
|
||||
const env = buildChildEnv(params.env);
|
||||
const startedAt = Date.now();
|
||||
|
||||
@@ -102,6 +102,64 @@ export class MinionQueue {
|
||||
if (existing.length > 0) return rowToMinionJob(existing[0]);
|
||||
}
|
||||
|
||||
// 1b. Submission-time backpressure for high-frequency named jobs.
|
||||
// If waiting jobs for this (name, queue) already hit maxWaiting, return
|
||||
// the most-recent waiting row instead of inserting another slot.
|
||||
//
|
||||
// Correctness: two concurrent submitters could both see waitingCount <
|
||||
// maxWaiting and both insert, violating the cap. `pg_advisory_xact_lock`
|
||||
// keyed on (name, queue) serializes concurrent count+insert decisions
|
||||
// for the SAME key while leaving different keys fully parallel. The
|
||||
// lock releases on txn commit/rollback automatically — no cleanup path
|
||||
// to leak. Cost: one no-op SELECT on the hot path per coalesce-guarded
|
||||
// submission; trivial compared to the protection.
|
||||
//
|
||||
// Queue scope: the filter includes `queue=$2` so a waiting
|
||||
// 'autopilot-cycle' in queue 'default' does NOT suppress submissions
|
||||
// to queue 'shell' with the same name. Pre-D2 code filtered on `name`
|
||||
// alone — a real cross-queue bleed that sequential tests missed.
|
||||
//
|
||||
// Engine compatibility: PGLite (WASM Postgres 17) supports
|
||||
// pg_advisory_xact_lock, so this works on both engines without branching.
|
||||
if (opts?.maxWaiting !== undefined) {
|
||||
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
|
||||
const backpressureQueue = opts?.queue ?? 'default';
|
||||
await tx.executeRaw(
|
||||
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2))`,
|
||||
[jobName, backpressureQueue]
|
||||
);
|
||||
const waitingCountRows = await tx.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count
|
||||
FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'`,
|
||||
[jobName, backpressureQueue]
|
||||
);
|
||||
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
|
||||
if (waitingCount >= maxWaiting) {
|
||||
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
|
||||
`SELECT * FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1`,
|
||||
[jobName, backpressureQueue]
|
||||
);
|
||||
if (existingWaiting.length > 0) {
|
||||
const coalesced = rowToMinionJob(existingWaiting[0]);
|
||||
try {
|
||||
const { logBackpressureCoalesce } = await import('./backpressure-audit.ts');
|
||||
logBackpressureCoalesce({
|
||||
queue: backpressureQueue,
|
||||
name: jobName,
|
||||
waiting_count: waitingCount,
|
||||
max_waiting: maxWaiting,
|
||||
returned_job_id: coalesced.id,
|
||||
});
|
||||
} catch { /* audit failures never block submission */ }
|
||||
return coalesced;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Parent lock + depth/cap validation
|
||||
let depth = 0;
|
||||
if (opts?.parent_job_id) {
|
||||
@@ -563,6 +621,77 @@ export class MinionQueue {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Dead-letter active jobs that exceed a wall-clock runtime threshold,
|
||||
* regardless of lock state. This catches jobs stuck while still holding
|
||||
* DB resources (e.g. blocked on file locks) where stall sweeps skip rows.
|
||||
*
|
||||
* Threshold (ms):
|
||||
* timeout_ms set -> timeout_ms * 2
|
||||
* timeout_ms null -> 2 * lockDurationMs * max_stalled
|
||||
*/
|
||||
async handleWallClockTimeouts(lockDurationMs: number): Promise<MinionJob[]> {
|
||||
return this.engine.transaction(async (tx) => {
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
error_text = 'wall-clock timeout exceeded',
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
|
||||
CASE
|
||||
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
|
||||
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
|
||||
END
|
||||
RETURNING *`,
|
||||
[lockDurationMs]
|
||||
);
|
||||
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: r.id as number,
|
||||
job_name: r.name as string,
|
||||
result: null,
|
||||
outcome: 'timeout',
|
||||
error: 'wall-clock timeout exceeded',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
|
||||
return rows.map(rowToMinionJob);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Complete a job (token-fenced). All side effects atomic in one transaction:
|
||||
* 1. UPDATE child to 'completed' with result
|
||||
|
||||
@@ -128,6 +128,8 @@ export interface MinionJobInput {
|
||||
max_spawn_depth?: number;
|
||||
/** Global dedup key. Same key returns the existing job, no second row created. */
|
||||
idempotency_key?: string;
|
||||
/** Submission backpressure: cap waiting jobs with this name before inserting a new row. */
|
||||
maxWaiting?: number;
|
||||
|
||||
// v12: scheduler polish
|
||||
/**
|
||||
|
||||
@@ -126,6 +126,14 @@ export class MinionWorker {
|
||||
} catch (e) {
|
||||
console.error('Timeout detection error:', e instanceof Error ? e.message : String(e));
|
||||
}
|
||||
try {
|
||||
const wallClockTimedOut = await this.queue.handleWallClockTimeouts(this.opts.lockDuration);
|
||||
if (wallClockTimedOut.length > 0) {
|
||||
console.log(`Wall-clock detector: dead-lettered ${wallClockTimedOut.length} jobs (wall-clock timeout exceeded)`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Wall-clock timeout detection error:', e instanceof Error ? e.message : String(e));
|
||||
}
|
||||
}, this.opts.stalledInterval);
|
||||
|
||||
try {
|
||||
|
||||
@@ -186,7 +186,10 @@ function acquireLock(workspace: string, opts: InstallOptions): void {
|
||||
const age = Date.now() - existing.mtimeMs;
|
||||
// `staleMs: 0` in tests means "any age counts as stale". Use >=
|
||||
// so a just-written lock qualifies when the threshold is 0.
|
||||
const stale = age >= staleMs;
|
||||
// Negative age (mtime in the future) happens on fast CI filesystems
|
||||
// where write → stat roundtrip returns an mtime microseconds ahead of
|
||||
// Date.now() — treat it as stale to avoid a "lock held" false positive.
|
||||
const stale = age < 0 || age >= staleMs;
|
||||
if (stale && !opts.forceUnlock) {
|
||||
throw new InstallError(
|
||||
`Stale skillpack lock at ${p} (pid ${existing.pid}, ${Math.round(age / 1000)}s old). Pass --force-unlock to proceed.`,
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* E2E Minions Shell Handler — PGLite / --follow inline execution path
|
||||
*
|
||||
* Closes the T4 gap surfaced during PR #381 eng review. The sibling file
|
||||
* test/e2e/minions-shell.test.ts covers the Postgres + persistent-worker-daemon
|
||||
* path. This file covers the PGLite path documented in the minion-orchestrator
|
||||
* skill: `gbrain jobs submit shell ... --follow` runs inline because
|
||||
* `gbrain jobs work` (daemon) is not available on PGLite (exclusive file lock).
|
||||
*
|
||||
* Mirrors the Postgres test's structure but runs in-memory against PGLiteEngine.
|
||||
* No DATABASE_URL required, no Docker — runs in CI unconditionally.
|
||||
*
|
||||
* Run: bun test test/e2e/minions-shell-pglite.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../../src/core/minions/queue.ts';
|
||||
import { MinionWorker } from '../../src/core/minions/worker.ts';
|
||||
import { registerBuiltinHandlers } from '../../src/commands/jobs.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let originalAllowShellJobs: string | undefined;
|
||||
|
||||
async function waitTerminal(queue: MinionQueue, id: number, timeoutMs = 15000): Promise<string> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const j = await queue.getJob(id);
|
||||
if (j && ['completed', 'failed', 'dead', 'cancelled'].includes(j.status)) return j.status;
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
const j = await queue.getJob(id);
|
||||
throw new Error(`job ${id} did not reach terminal state in ${timeoutMs}ms; last status=${j?.status}`);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
// registerBuiltinHandlers gates shell handler on GBRAIN_ALLOW_SHELL_JOBS=1.
|
||||
// Mirror the real --follow path by setting the env var; restore on cleanup
|
||||
// so other tests see their original environment.
|
||||
originalAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({}); // in-memory PGLite
|
||||
await engine.initSchema(); // installs pages, minion_jobs, config, etc.
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
if (originalAllowShellJobs === undefined) {
|
||||
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
} else {
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = originalAllowShellJobs;
|
||||
}
|
||||
});
|
||||
|
||||
describe('E2E: Minions shell handler on PGLite (--follow inline path)', () => {
|
||||
// Mirror the Postgres sibling's per-test reset. The engine is shared across
|
||||
// both tests via beforeAll; without this, completed jobs from one test leak
|
||||
// into minion_jobs and future test additions hit order-dependency.
|
||||
beforeEach(async () => {
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`DELETE FROM minion_attachments; DELETE FROM minion_inbox; DELETE FROM minion_jobs;`);
|
||||
});
|
||||
|
||||
test('submit → worker registered via registerBuiltinHandlers → shell runs → completes', async () => {
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add(
|
||||
'shell',
|
||||
{ cmd: 'echo hello', cwd: '/tmp' },
|
||||
{},
|
||||
{ allowProtectedSubmit: true },
|
||||
);
|
||||
expect(job.name).toBe('shell');
|
||||
expect(job.status).toBe('waiting');
|
||||
|
||||
// This is the exact dispatch path --follow takes (src/commands/jobs.ts:207).
|
||||
// Gates shell on GBRAIN_ALLOW_SHELL_JOBS=1 (set in beforeAll above).
|
||||
const worker = new MinionWorker(engine, { pollInterval: 100, lockDuration: 30000 });
|
||||
await registerBuiltinHandlers(worker, engine);
|
||||
expect(worker.registeredNames).toContain('shell');
|
||||
|
||||
const runPromise = worker.start();
|
||||
try {
|
||||
const status = await waitTerminal(queue, job.id, 20000);
|
||||
expect(status).toBe('completed');
|
||||
const final = await queue.getJob(job.id);
|
||||
expect((final!.result as any).exit_code).toBe(0);
|
||||
expect((final!.result as any).stdout_tail).toBe('hello\n');
|
||||
} finally {
|
||||
worker.stop();
|
||||
await runPromise;
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
test('GBRAIN_ALLOW_SHELL_JOBS unset → shellHandler rejects at execution time', async () => {
|
||||
// v0.20.3+: shell handler is always registered (so claimed jobs emit a clear
|
||||
// rejection log), but the runtime env guard lives inside the handler itself.
|
||||
// Prove the guard rejects when the env var is unset.
|
||||
const { shellHandler } = await import('../../src/core/minions/handlers/shell.ts');
|
||||
const saved = process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
try {
|
||||
const ctx: any = {
|
||||
id: 1,
|
||||
name: 'shell',
|
||||
data: { cmd: 'echo hi', cwd: '/tmp' },
|
||||
attempt: 1,
|
||||
engine,
|
||||
};
|
||||
await expect(shellHandler(ctx)).rejects.toThrow(/GBRAIN_ALLOW_SHELL_JOBS=1/);
|
||||
} finally {
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = saved;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -12,8 +12,18 @@ import * as os from 'node:os';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
// The shell handler at src/core/minions/handlers/shell.ts:210 throws
|
||||
// UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1'. That's the
|
||||
// production-worker RCE guard. Unit tests here exercise the handler
|
||||
// mechanics, not the guard, so we enable it for the whole file and
|
||||
// restore on teardown. The separate "rejects when env not set" case
|
||||
// (in the minion-shell submission E2E / the queue-resilience wave)
|
||||
// toggles the var itself.
|
||||
let prevAllowShellJobs: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
prevAllowShellJobs = process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
process.env.GBRAIN_ALLOW_SHELL_JOBS = '1';
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
@@ -22,6 +32,8 @@ beforeAll(async () => {
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
if (prevAllowShellJobs === undefined) delete process.env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
else process.env.GBRAIN_ALLOW_SHELL_JOBS = prevAllowShellJobs;
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
|
||||
@@ -1653,3 +1653,246 @@ describe('MinionQueue: Attachments', () => {
|
||||
expect(list[0].filename).toBe('b.txt');
|
||||
});
|
||||
});
|
||||
|
||||
// --- v0.19.1 — queue-resilience (wall-clock sweep, maxWaiting race, concurrency clamp) ---
|
||||
|
||||
describe('MinionQueue: v0.19.1 handleWallClockTimeouts (Layer 3 kill shot)', () => {
|
||||
test('evicts active job past 2× timeout_ms — sets dead + wall-clock error_text', async () => {
|
||||
const job = await queue.add('noop', {}, { timeout_ms: 100 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
lock_token='wc-test',
|
||||
lock_until=now() - interval '1 second',
|
||||
started_at=now() - interval '1 second',
|
||||
timeout_at=now() - interval '0.9 second',
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
const killed = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed.length).toBe(1);
|
||||
expect(killed[0].id).toBe(job.id);
|
||||
const after = await queue.getJob(job.id);
|
||||
expect(after?.status).toBe('dead');
|
||||
expect(after?.error_text).toBe('wall-clock timeout exceeded');
|
||||
});
|
||||
|
||||
test('timeout_ms NULL fallback uses 2 × lockDuration × max_stalled threshold', async () => {
|
||||
const job = await queue.add('noop', {}, { max_stalled: 3 });
|
||||
// Force timeout_ms / timeout_at NULL on-disk (columns might or might not be set by add).
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
timeout_ms=NULL,
|
||||
timeout_at=NULL,
|
||||
lock_token='wc-null',
|
||||
lock_until=now() - interval '1 second',
|
||||
started_at=now() - interval '61 seconds',
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
// 2 × lockDurationMs × max_stalled = 2 × 10_000 × 3 = 60_000 ms. started_at is 61s ago.
|
||||
const killed = await queue.handleWallClockTimeouts(10_000);
|
||||
expect(killed.length).toBe(1);
|
||||
expect(killed[0].id).toBe(job.id);
|
||||
const after = await queue.getJob(job.id);
|
||||
expect(after?.status).toBe('dead');
|
||||
});
|
||||
|
||||
test('respects threshold — active job within window is NOT killed', async () => {
|
||||
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
lock_token='wc-inside',
|
||||
lock_until=now() + interval '30 seconds',
|
||||
started_at=now() - interval '10 seconds',
|
||||
timeout_at=now() + interval '90 seconds',
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
const killed = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed.length).toBe(0);
|
||||
const after = await queue.getJob(job.id);
|
||||
expect(after?.status).toBe('active');
|
||||
});
|
||||
});
|
||||
|
||||
describe('MinionQueue: v0.19.1 maxWaiting — cap correctness + race (D2/H2)', () => {
|
||||
test('coalesces 3rd submission when cap is 2 — returns existing most-recent waiting row', async () => {
|
||||
const a = await queue.add('poll', {}, { maxWaiting: 2 });
|
||||
const b = await queue.add('poll', {}, { maxWaiting: 2 });
|
||||
const c = await queue.add('poll', {}, { maxWaiting: 2 });
|
||||
expect(a.id).not.toBe(b.id);
|
||||
expect(c.id).toBe(b.id); // coalesced to the most-recent waiting row
|
||||
const rows = await engine.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='poll' AND status='waiting'`,
|
||||
);
|
||||
expect(parseInt(rows[0].count, 10)).toBe(2);
|
||||
});
|
||||
|
||||
test('clamps maxWaiting: 0 → 1 (strictest cap)', async () => {
|
||||
const a = await queue.add('squeeze', {}, { maxWaiting: 0 });
|
||||
const b = await queue.add('squeeze', {}, { maxWaiting: 0 });
|
||||
expect(b.id).toBe(a.id); // 0 clamped to 1, 2nd coalesces into 1st
|
||||
});
|
||||
|
||||
test('floors maxWaiting: 1.7 → 1', async () => {
|
||||
const a = await queue.add('floor', {}, { maxWaiting: 1.7 });
|
||||
const b = await queue.add('floor', {}, { maxWaiting: 1.7 });
|
||||
expect(b.id).toBe(a.id);
|
||||
});
|
||||
|
||||
test('concurrent submitters respect the cap under Promise.all race (H2)', async () => {
|
||||
// Serialized by pg_advisory_xact_lock keyed on (name, queue). Without it,
|
||||
// two concurrent submits both see count<max and both insert — the TOCTOU
|
||||
// bug codex caught in D2/H2.
|
||||
const results = await Promise.all([
|
||||
queue.add('race', {}, { maxWaiting: 2 }),
|
||||
queue.add('race', {}, { maxWaiting: 2 }),
|
||||
queue.add('race', {}, { maxWaiting: 2 }),
|
||||
]);
|
||||
expect(results.length).toBe(3);
|
||||
const rows = await engine.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count FROM minion_jobs WHERE name='race' AND status='waiting'`,
|
||||
);
|
||||
expect(parseInt(rows[0].count, 10)).toBe(2); // cap held under concurrency
|
||||
});
|
||||
|
||||
test('cross-queue isolation — same name in queue A does NOT suppress queue B (H2 secondary)', async () => {
|
||||
const a = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
|
||||
// cap hit on queue=default with maxWaiting=1; 2nd would coalesce into `a`
|
||||
const a2 = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'default' });
|
||||
expect(a2.id).toBe(a.id);
|
||||
// Different queue — MUST insert a fresh row, NOT coalesce into queue=default
|
||||
const b = await queue.add('isolate', {}, { maxWaiting: 1, queue: 'shell' });
|
||||
expect(b.id).not.toBe(a.id);
|
||||
expect(b.queue).toBe('shell');
|
||||
});
|
||||
|
||||
test('unset maxWaiting — normal submit path, no coalesce, no cap', async () => {
|
||||
const a = await queue.add('uncapped', {});
|
||||
const b = await queue.add('uncapped', {});
|
||||
const c = await queue.add('uncapped', {});
|
||||
expect(new Set([a.id, b.id, c.id]).size).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveWorkerConcurrency (v0.19.1 H3): clamp + validation', () => {
|
||||
// jobs.ts handler — tested via direct import. Warning goes to stderr;
|
||||
// tests verify return value only, not the warning line.
|
||||
let resolveWorkerConcurrency: (args: string[], env?: NodeJS.ProcessEnv) => number;
|
||||
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
|
||||
beforeAll(async () => {
|
||||
const mod = await import('../src/commands/jobs.ts');
|
||||
resolveWorkerConcurrency = mod.resolveWorkerConcurrency;
|
||||
parseMaxWaitingFlag = mod.parseMaxWaitingFlag;
|
||||
});
|
||||
|
||||
test('flag=4 env-unset → 4', () => {
|
||||
expect(resolveWorkerConcurrency(['--concurrency', '4'], {} as NodeJS.ProcessEnv)).toBe(4);
|
||||
});
|
||||
test('flag-unset env=8 → 8', () => {
|
||||
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(8);
|
||||
});
|
||||
test('flag=2 env=8 → 2 (flag wins)', () => {
|
||||
expect(resolveWorkerConcurrency(['--concurrency', '2'], { GBRAIN_WORKER_CONCURRENCY: '8' } as NodeJS.ProcessEnv)).toBe(2);
|
||||
});
|
||||
test('both unset → 1', () => {
|
||||
expect(resolveWorkerConcurrency([], {} as NodeJS.ProcessEnv)).toBe(1);
|
||||
});
|
||||
test('garbage env "foo" → clamped to 1 (H3)', () => {
|
||||
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: 'foo' } as NodeJS.ProcessEnv)).toBe(1);
|
||||
});
|
||||
test('env=0 → clamped to 1 (H3 — prevents silent wedge)', () => {
|
||||
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '0' } as NodeJS.ProcessEnv)).toBe(1);
|
||||
});
|
||||
test('env=-5 → clamped to 1 (H3)', () => {
|
||||
expect(resolveWorkerConcurrency([], { GBRAIN_WORKER_CONCURRENCY: '-5' } as NodeJS.ProcessEnv)).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseMaxWaitingFlag (v0.19.1 H5): CLI flag wiring', () => {
|
||||
let parseMaxWaitingFlag: (args: string[]) => number | undefined;
|
||||
beforeAll(async () => {
|
||||
parseMaxWaitingFlag = (await import('../src/commands/jobs.ts')).parseMaxWaitingFlag;
|
||||
});
|
||||
|
||||
test('absent → undefined (no cap, default submit path)', () => {
|
||||
expect(parseMaxWaitingFlag(['foo', '--params', '{}'])).toBeUndefined();
|
||||
});
|
||||
test('--max-waiting 2 → 2 (happy path)', () => {
|
||||
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '2'])).toBe(2);
|
||||
});
|
||||
test('--max-waiting 200 → clamped to 100', () => {
|
||||
expect(parseMaxWaitingFlag(['foo', '--max-waiting', '200'])).toBe(100);
|
||||
});
|
||||
test('--max-waiting 0 → throws', () => {
|
||||
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', '0'])).toThrow('positive integer');
|
||||
});
|
||||
test('--max-waiting abc → throws', () => {
|
||||
expect(() => parseMaxWaitingFlag(['foo', '--max-waiting', 'abc'])).toThrow('positive integer');
|
||||
});
|
||||
});
|
||||
|
||||
describe('backpressure-audit (v0.19.1 Q1): JSONL on coalesce', () => {
|
||||
test('logBackpressureCoalesce writes one JSONL line per coalesce', async () => {
|
||||
const { logBackpressureCoalesce, resolveAuditDir, computeAuditFilename } =
|
||||
await import('../src/core/minions/backpressure-audit.ts');
|
||||
const fs = await import('node:fs');
|
||||
const path = await import('node:path');
|
||||
const os = await import('node:os');
|
||||
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-audit-'));
|
||||
const prev = process.env.GBRAIN_AUDIT_DIR;
|
||||
process.env.GBRAIN_AUDIT_DIR = tmp;
|
||||
try {
|
||||
expect(resolveAuditDir()).toBe(tmp);
|
||||
logBackpressureCoalesce({
|
||||
queue: 'default',
|
||||
name: 'poll',
|
||||
waiting_count: 2,
|
||||
max_waiting: 2,
|
||||
returned_job_id: 42,
|
||||
});
|
||||
const file = path.join(tmp, computeAuditFilename());
|
||||
const text = fs.readFileSync(file, 'utf8');
|
||||
const line = JSON.parse(text.trim());
|
||||
expect(line.decision).toBe('coalesced');
|
||||
expect(line.name).toBe('poll');
|
||||
expect(line.returned_job_id).toBe(42);
|
||||
expect(typeof line.ts).toBe('string');
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.GBRAIN_AUDIT_DIR;
|
||||
else process.env.GBRAIN_AUDIT_DIR = prev;
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('MinionQueue: v0.19.1 wall-clock + handleTimeouts non-interference (T1)', () => {
|
||||
test('wall-clock sweep does NOT evict a job that handleTimeouts would handle', async () => {
|
||||
// Retry-able timeout: timeout_at < now() AND lock_until > now() — handleTimeouts
|
||||
// is the correct killer here. wall-clock's 2× threshold has not fired yet.
|
||||
const job = await queue.add('noop', {}, { timeout_ms: 100_000 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status='active',
|
||||
lock_token='t1',
|
||||
lock_until=now() + interval '30 seconds',
|
||||
started_at=now() - interval '2 seconds',
|
||||
timeout_at=now() - interval '0.5 seconds',
|
||||
attempts_started = attempts_started + 1
|
||||
WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
// At this point: started_at is 2s ago, 2×timeout_ms = 200s. Wall-clock should NOT fire.
|
||||
const killed = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed.length).toBe(0);
|
||||
const after = await queue.getJob(job.id);
|
||||
expect(after?.status).toBe('active');
|
||||
});
|
||||
});
|
||||
|
||||
+155
-1
@@ -1,10 +1,12 @@
|
||||
import { describe, test, expect } from "bun:test";
|
||||
import { readFileSync, existsSync } from "fs";
|
||||
import { readFileSync, existsSync, readdirSync, statSync } from "fs";
|
||||
import { join } from "path";
|
||||
import { checkResolvable } from "../src/core/check-resolvable.ts";
|
||||
import { PROTECTED_JOB_NAMES } from "../src/core/minions/protected-names.ts";
|
||||
|
||||
const SKILLS_DIR = join(import.meta.dir, "..", "skills");
|
||||
const RESOLVER_PATH = join(SKILLS_DIR, "RESOLVER.md");
|
||||
const OPERATIONS_PATH = join(import.meta.dir, "..", "src", "core", "operations.ts");
|
||||
|
||||
describe("RESOLVER.md", () => {
|
||||
test("exists", () => {
|
||||
@@ -49,3 +51,155 @@ describe("RESOLVER.md", () => {
|
||||
expect(report.summary.unreachable).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// D5/C — resolver round-trip: every quoted trigger in a RESOLVER.md table row
|
||||
// must appear in the target skill's frontmatter `triggers:` list. Catches
|
||||
// trigger/frontmatter drift that `checkResolvable` reachability doesn't.
|
||||
describe("RESOLVER.md trigger round-trip (D5/C)", () => {
|
||||
type Row = { triggers: string[]; skillPath: string };
|
||||
|
||||
const rows: Row[] = (() => {
|
||||
if (!existsSync(RESOLVER_PATH)) return [];
|
||||
const content = readFileSync(RESOLVER_PATH, "utf-8");
|
||||
// Tolerate trailing annotations after the backtick path (e.g.,
|
||||
// `` `skills/maintain/SKILL.md` (extraction sections) |``). The path cell
|
||||
// starts with a backtick-quoted `.md` ref; anything between that and the
|
||||
// closing `|` is free-form prose and is intentionally ignored.
|
||||
const rowRe = /^\s*\|\s*([^|]+?)\s*\|\s*`([^`]+\.md)`[^|]*\|\s*$/gm;
|
||||
const out: Row[] = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = rowRe.exec(content)) !== null) {
|
||||
const rawTriggers = m[1];
|
||||
const skillPath = m[2];
|
||||
const triggerStrings = Array.from(rawTriggers.matchAll(/"([^"]+)"/g)).map(t => t[1]);
|
||||
if (triggerStrings.length > 0) {
|
||||
out.push({ triggers: triggerStrings, skillPath });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
})();
|
||||
|
||||
test("at least one routing row parses from RESOLVER.md", () => {
|
||||
expect(rows.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
for (const row of rows) {
|
||||
test(`every RESOLVER trigger for ${row.skillPath} is declared in its frontmatter`, () => {
|
||||
const skillFullPath = join(SKILLS_DIR, "..", row.skillPath);
|
||||
expect(existsSync(skillFullPath)).toBe(true);
|
||||
|
||||
const skillContent = readFileSync(skillFullPath, "utf-8");
|
||||
const fmMatch = skillContent.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!fmMatch) {
|
||||
throw new Error(`No YAML frontmatter in ${row.skillPath}`);
|
||||
}
|
||||
const frontmatter = fmMatch[1];
|
||||
// Parse frontmatter triggers: list. Match "..." OR '...' items separately
|
||||
// so apostrophes inside double-quoted values don't truncate the capture.
|
||||
const triggersBlock = frontmatter.match(/triggers:\s*\n((?:\s*-\s*(?:"[^"]*"|'[^']*')\s*\n?)+)/);
|
||||
const declaredTriggers = triggersBlock
|
||||
? Array.from(triggersBlock[1].matchAll(/-\s*(?:"([^"]*)"|'([^']*)')/g))
|
||||
.map(m => m[1] ?? m[2])
|
||||
: [];
|
||||
|
||||
// Fuzzy match: RESOLVER.md phrases are natural-language summaries of the
|
||||
// skill's intent; frontmatter triggers are the agent-facing phrase set.
|
||||
// Match is case-insensitive, trailing-punctuation-insensitive, and supports
|
||||
// "/"-split compounds (e.g., "pause/resume agent" → ["pause", "resume agent"]).
|
||||
const normalize = (s: string) => s.toLowerCase().replace(/[?!.,]+$/, "").trim();
|
||||
const declaredLower = declaredTriggers.map(normalize);
|
||||
|
||||
function matchesAny(phrase: string): boolean {
|
||||
const p = normalize(phrase);
|
||||
if (declaredLower.includes(p)) return true;
|
||||
for (const ft of declaredLower) {
|
||||
if (ft.includes(p) || p.includes(ft)) return true;
|
||||
}
|
||||
// Slash-split compound: every part should have some fuzzy frontmatter hit
|
||||
if (p.includes("/")) {
|
||||
const parts = p.split("/").map(s => s.trim()).filter(Boolean);
|
||||
const allParts = parts.every(part =>
|
||||
declaredLower.some(ft => ft.includes(part) || part.includes(ft))
|
||||
);
|
||||
if (allParts) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
const missing = row.triggers.filter(t => !matchesAny(t));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(
|
||||
`RESOLVER.md routes ${JSON.stringify(missing)} to ${row.skillPath}, but the ` +
|
||||
`skill's frontmatter has no fuzzy match. Declared: ${JSON.stringify(declaredTriggers)}`
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// D13 — skill-example-name validator: any `name="<word>"` reference inside a
|
||||
// SKILL.md body must resolve to either a declared operation in operations.ts
|
||||
// or a known Minions handler name in PROTECTED_JOB_NAMES. Catches T2-class
|
||||
// bugs where docs reference handler names that don't exist (e.g., the
|
||||
// `name="research"` / `name="orchestrate"` bug from PR #381 pre-reframe).
|
||||
describe("Skill example-name validator (D13)", () => {
|
||||
const opNames: string[] = (() => {
|
||||
if (!existsSync(OPERATIONS_PATH)) return [];
|
||||
const content = readFileSync(OPERATIONS_PATH, "utf-8");
|
||||
return Array.from(content.matchAll(/^\s+name:\s*'([a-z_]+)',/gm)).map(m => m[1]);
|
||||
})();
|
||||
|
||||
const knownNames = new Set<string>([...opNames, ...PROTECTED_JOB_NAMES]);
|
||||
|
||||
test("operation names extracted from operations.ts", () => {
|
||||
// Sanity check: operations.ts should declare dozens of ops
|
||||
expect(opNames.length).toBeGreaterThan(10);
|
||||
});
|
||||
|
||||
test("PROTECTED_JOB_NAMES is non-empty", () => {
|
||||
expect(PROTECTED_JOB_NAMES.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
function walkSkills(dir: string): string[] {
|
||||
if (!existsSync(dir)) return [];
|
||||
const out: string[] = [];
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const p = join(dir, entry);
|
||||
const s = statSync(p);
|
||||
if (s.isDirectory()) {
|
||||
out.push(...walkSkills(p));
|
||||
} else if (entry === "SKILL.md") {
|
||||
out.push(p);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const skillFiles = walkSkills(SKILLS_DIR);
|
||||
|
||||
test("at least one SKILL.md found", () => {
|
||||
expect(skillFiles.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
for (const skillFile of skillFiles) {
|
||||
const rel = skillFile.replace(SKILLS_DIR, "skills");
|
||||
test(`${rel}: every name="<word>" reference resolves to a real op or handler`, () => {
|
||||
const content = readFileSync(skillFile, "utf-8");
|
||||
// Strip YAML frontmatter so `name: <skillname>` isn't mis-captured.
|
||||
const body = content.replace(/^---\n[\s\S]*?\n---\n/, "");
|
||||
// Match only `name=` (with equals, not colon) to avoid YAML false positives
|
||||
// if the frontmatter strip ever breaks. Captures quoted word values.
|
||||
const refs = Array.from(body.matchAll(/name\s*=\s*["']([a-z_][a-z_0-9]*)["']/gi))
|
||||
.map(m => m[1]);
|
||||
const unique = [...new Set(refs)];
|
||||
const unknown = unique.filter(n => !knownNames.has(n));
|
||||
if (unknown.length > 0) {
|
||||
throw new Error(
|
||||
`${rel}: references name="..." values not declared in src/core/operations.ts or ` +
|
||||
`PROTECTED_JOB_NAMES: ${JSON.stringify(unknown)}. ` +
|
||||
`Known: ${JSON.stringify([...knownNames].sort())}`
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user