mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-15 17:32:37 +00:00
Compare commits
18
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ec3d7e79c7 | ||
|
|
802441223d | ||
|
|
f6aec5b0a0 | ||
|
|
2b1a0811b3 | ||
|
|
f870f1d3ef | ||
|
|
8adbf87443 | ||
|
|
2aac6390cb | ||
|
|
905aeb481d | ||
|
|
a2bbdb7c04 | ||
|
|
c2a6bc8078 | ||
|
|
12b89ef46f | ||
|
|
4a12fcc340 | ||
|
|
3a779b9752 | ||
|
|
04a319e311 | ||
|
|
a02bfe34ef | ||
|
|
8caa2b0c88 | ||
|
|
222023f65f | ||
|
|
2344821ecd |
@@ -96,7 +96,9 @@ jobs:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
- name: Run Tier 1 E2E tests
|
||||
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
|
||||
# job-isolation rides tier1 deliberately: e2e.yml runs only explicitly
|
||||
# NAMED files (no glob) — an unwired e2e file is silent coverage loss.
|
||||
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts test/e2e/job-isolation.test.ts
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
# #3485 preload guard: this job intentionally tests against a DB.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<!-- gbrain-runbook-stamp: 0.46.0.0 -->
|
||||
<!-- gbrain-runbook-stamp: 0.46.1.0 -->
|
||||
<!-- This stamp must equal the VERSION file at every release; CI enforces it
|
||||
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
|
||||
the installed binary and warns on skew. -->
|
||||
|
||||
@@ -2,6 +2,61 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.46.1.0] - 2026-08-15
|
||||
|
||||
**A stuck job can no longer take down your whole worker.** Field reports from
|
||||
a production deployment ([#5](https://github.com/garrytan-agents/gbrain/issues/5),
|
||||
[#6](https://github.com/garrytan-agents/gbrain/issues/6)) showed two
|
||||
compounding failure modes: a handler that ignored its abort signal could only
|
||||
be "force-evicted" (abandoned but still running, still holding connections),
|
||||
and abandoned probe/renewal queries starved the connection pool until the
|
||||
worker killed itself with a misleading "DB unreachable" — while the database
|
||||
sat at a fraction of capacity. This release fixes the starvation class and
|
||||
adds real per-job blast-radius control.
|
||||
|
||||
### Added
|
||||
- **`gbrain jobs work --job-isolation process`** (also
|
||||
`gbrain jobs supervisor --job-isolation process`, env
|
||||
`GBRAIN_JOB_ISOLATION`): each claimed job runs in its own child process.
|
||||
A stuck handler is group-SIGKILLed for real instead of abandoned, a crash
|
||||
or memory blowup takes one job instead of all N, and the OS reclaims every
|
||||
leaked resource when the child dies. The worker keeps claiming, renewing,
|
||||
and recording; handler-error semantics (unrecoverable → dead, rate-lease →
|
||||
no attempt burned, backoff otherwise) are preserved across the boundary.
|
||||
Worker shutdown gives children the drain window to finish and report — a
|
||||
routine deploy never burns a job attempt. Recommended for long-running
|
||||
LLM-bound handlers; see the new section in `docs/guides/minions-deployment.md`.
|
||||
- **Health-probe verdicts that name the failing layer.** When the worker's DB
|
||||
probe fails, it now disambiguates via the direct session lane and says
|
||||
`pool_starved` ("server IS reachable; the fault is in the
|
||||
transaction-pooler path") or `server_unreachable` — instead of the blanket
|
||||
"DB unreachable" that historically sent operators debugging database
|
||||
capacity while the real fault was client-side. A startup warning also makes
|
||||
single-pool mode (direct-lane kill switch) loud instead of silent, and
|
||||
`docs/guides/queue-operations-runbook.md` gains a verdict-interpretation
|
||||
table.
|
||||
- `GBRAIN_POOL_MAX_LIFETIME_S`: explicit client-pool connection max-lifetime
|
||||
knob (0 disables; default stays the per-connection 30–60min jitter).
|
||||
|
||||
### Fixed
|
||||
- **Timed-out DB probes and lock renewals are now cancelled, not abandoned.**
|
||||
Every place that raced a query against a timer (health probe, minion lock
|
||||
renewal, cycle-drain renewal, submit-time queue probes, DB-lock refresh)
|
||||
previously let the losing query keep running on a checked-out connection —
|
||||
under pool exhaustion each abandoned racer held a slot and made the
|
||||
exhaustion worse, starving the lock heartbeat first. All five sites now
|
||||
abort the query via its cancellation signal so the slot is released.
|
||||
- Long-running maintenance holds (index rebuilds, non-transactional
|
||||
migrations, backfill write batches) now reserve from the direct session
|
||||
lane instead of pinning the worker's shared pool — capped so reserved
|
||||
holds always leave a direct-lane slot for the claim/renewal heartbeats,
|
||||
and falling back to the previous behavior when the direct lane is
|
||||
unavailable.
|
||||
|
||||
Full operational detail: `docs/guides/minions-deployment.md` (isolation
|
||||
sizing: connections, memory, spawn cost) and
|
||||
`docs/guides/queue-operations-runbook.md` (probe verdicts).
|
||||
|
||||
## [0.46.0.0] - 2026-08-14
|
||||
|
||||
**Your other agents' sessions become brain knowledge.** Until now only Claude
|
||||
|
||||
@@ -311,7 +311,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Opt-in per-job process isolation (`gbrain jobs work --job-isolation process`) runs each claimed job in its own SIGKILL-able child process, so a stuck handler dies for real and a crash takes one job instead of the whole worker; when the worker's DB health probe fails, it names the failing layer (`pool_starved` vs `server_unreachable`) instead of a blanket "DB unreachable". Sizing and rollout guidance in [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md); probe-verdict triage in [`docs/guides/queue-operations-runbook.md`](docs/guides/queue-operations-runbook.md). Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
|
||||
@@ -1,5 +1,87 @@
|
||||
# TODOS
|
||||
|
||||
## Issues #5+#6 follow-ups (pool starvation + process isolation; plan: ~/.claude/plans/system-instruction-you-are-working-witty-moore.md)
|
||||
|
||||
- [ ] **P1-companion — nested-checkout audit + dev-mode detection.** **What:**
|
||||
`transaction()` callers that invoke parent-engine methods (or module helpers
|
||||
taking `engine` not `tx`) take a SECOND read-pool slot while holding the tx
|
||||
slot — e.g. the `operations.ts` advisory-lock loop around `tx.addLink`. Under
|
||||
a saturated pool this is a client-side self-deadlock class. Audit call sites;
|
||||
add a dev-mode warning (e.g. a tx-depth counter consulted by `runUnsafe`).
|
||||
**Why:** the #6 incident's exact 240s-idle sessions were never reproduced
|
||||
under a debugger; this is the strongest remaining candidate — the shipped
|
||||
wave mitigates the starvation class but does not close this path. **Effort:**
|
||||
M. **Priority:** P1-companion.
|
||||
- [ ] **P2 — per-handler isolation policy.** **What:** a per-handler-name set
|
||||
(e.g. long-running LLM-bound handlers isolate, sub-second `lint`/`backlinks`
|
||||
stay inline) instead of the all-or-nothing `--job-isolation process`.
|
||||
**Why:** spawn cost (~0.3–1s) is noise for 644s subagent jobs, meaningful
|
||||
for sub-second handlers; one worker should be able to mix. **Context:**
|
||||
`worker.ts` executeJob's `isolated` gate is the seam. **Effort:** M.
|
||||
**Priority:** P2.
|
||||
- [ ] **P2 — per-child --max-rss caps.** **What:** RSS watchdog for isolation
|
||||
children (the worker-level watchdog covers the worker only in process mode;
|
||||
a startup note ships today). **Context:** child-job-runner.ts owns the child
|
||||
lifecycle; a poll of the child's RSS + group-kill on breach mirrors the
|
||||
worker watchdog. **Effort:** M. **Priority:** P2.
|
||||
- [ ] **P2 — jobs-side connection-budget clamp for isolated workers.** **What:**
|
||||
warn/clamp concurrency when `concurrency × (child pool + 1) + parent pools`
|
||||
exceeds a configured budget (GBRAIN_MAX_CONNECTIONS-style; precedent
|
||||
`sync-concurrency.ts:clampWorkersForConnectionBudget`). **Why:** isolation
|
||||
multiplies pooler CLIENT connections (~73 at concurrency 15); today the
|
||||
budget lives only in docs math. **Effort:** S. **Priority:** P2.
|
||||
- [ ] **P3 — --job-isolation pass-through for the autopilot's embedded
|
||||
supervisor.** **What:** `autopilot.ts` builds its own worker args; add the
|
||||
conditional flag there (jobs supervisor already passes through). **Effort:**
|
||||
S. **Priority:** P3.
|
||||
- [ ] **P3 — runLockRenewalTick adoption in the cycle drain.** **What:**
|
||||
`synthesize.ts` now uses the minimal `runDrainRenewalTick` (per-call signal +
|
||||
guard); adopting the full tick would add the audit channel + bounded
|
||||
reconnect. **Effort:** S. **Priority:** P3.
|
||||
- [ ] **P3 — streaming child progress.** **What:** isolation children report
|
||||
progress via their own token-fenced DB writes today (identical to inline);
|
||||
an IPC stream would only add parent-side visibility (e.g. lifecycle events
|
||||
in `jobs watch`). **Effort:** M. **Priority:** P3.
|
||||
- [ ] **P3 — connection-audit release events + plain-idle visibility.**
|
||||
**What:** `logConnectionEvent` never emits `release`, so the JSONL cannot
|
||||
answer "who holds a slot"; and `getIdleBlockers` filters
|
||||
`state='idle in transaction'` only — the #6 incident's plain-`idle` sessions
|
||||
were invisible to it. **Effort:** M. **Priority:** P3.
|
||||
- [ ] **P3 — doctor connection_routing check.** **What:** wire
|
||||
`ConnectionManager.describeMode()` + `healthCheck()` (both currently
|
||||
zero-caller outside tests) into a doctor check naming the routing mode,
|
||||
kill-switch state, and per-pool probe latency. Comments in four files
|
||||
already reference this check as if it existed. **Effort:** S.
|
||||
**Priority:** P3.
|
||||
- [ ] **P3 — isolation test-gap follow-ups (pre-landing review).** **What:**
|
||||
(a) spawned-CLI negative tests for `jobs run-child` bootstrap guards (PGLite
|
||||
→ exit 13; missing job-id/env → exit 13) and for `jobs work` with
|
||||
isolation on + an unresolvable child CLI (fail-fast exit 1) — both need a
|
||||
real engine bootstrap so they live in the e2e lane; (b) a behavioral (not
|
||||
structural) test driving `withRefreshingLock` with a hung injected
|
||||
`handle.refresh` (signal aborted at timeout, no overlapping ticks); (c) a
|
||||
force-evict-skip test for isolation mode (needs the 30s evict window made
|
||||
injectable); (d) operator-flow message tests (verdict-tailored FATAL text,
|
||||
single-pool startup banner). **Why:** the ship coverage audit scored the
|
||||
wave 82% — these are the surviving gaps. **Effort:** M. **Priority:** P3.
|
||||
- [ ] **P3 — raceWithAbortTimeout shared helper.** **What:** the
|
||||
"Promise.race a query vs a setTimeout that aborts an AbortController,
|
||||
clearTimeout in finally" pattern now exists at five sites (db-probe
|
||||
withDeadline, synthesize runDrainRenewalTick, lock-renewal-tick callAbort,
|
||||
db-lock tickAbort, supervisor probeAbort), each re-deriving the same
|
||||
invariants. Extract one helper and adopt it. **Effort:** S. **Priority:** P3.
|
||||
- [ ] **P3 — lazy handler resolution in run-child.** **What:** every isolation
|
||||
child runs full registerBuiltinHandlers (incl. plugin discovery) to resolve
|
||||
ONE handler; the job name is known from the row — a resolve-by-name path
|
||||
would skip discovery for builtins. Matters only if isolation is ever used
|
||||
for short jobs (documented as not the target). **Effort:** S. **Priority:** P3.
|
||||
- [ ] **P3 — full checkout instrumentation via a Sql proxy.** **What:** the
|
||||
CheckoutGauge covers raw/direct/reserved/tx seams only; tagged-template
|
||||
traffic (most engine load) is untracked. A proxy around the postgres.js Sql
|
||||
callable could count real checkouts — investigate cost/fragility before
|
||||
building. **Why:** would turn the probe's "tracked subset" caveat into full
|
||||
coverage. **Effort:** M. **Priority:** P3.
|
||||
|
||||
## Security-process follow-ups (filed with Wave −1 of the fix-wave campaign, 2026-08-14)
|
||||
|
||||
- [ ] **P2 — Vulnerability disclosure policy.** **What:** a written disclosure
|
||||
@@ -254,14 +336,6 @@ fix-now findings landed on the branch; these four are the review-deferred tail.
|
||||
`requestToolsPersistLimiter`; the surface_change audit rows already give
|
||||
a DB-side count to enforce against if needed. **Effort:** medium.
|
||||
**Priority:** P3.
|
||||
- [ ] **P3 — cancel (not just abandon) timed-out submit-time queue probes.**
|
||||
**What:** the WP5 wedge/pause probes time-bound via Promise.race, but the
|
||||
losing query keeps running on the pool after the race resolves. Wire
|
||||
AbortSignal / statement_timeout so a slow probe releases its slot. **Why:**
|
||||
under pool exhaustion (the exact regime the probes exist to detect) an
|
||||
abandoned probe query holds a pooler slot and makes the exhaustion worse.
|
||||
**Context:** `src/core/minion/supervisor.ts` queryWedgeSignals callers in
|
||||
`src/core/operations.ts` submit paths. **Effort:** small. **Priority:** P3.
|
||||
- [ ] **P3 — document the status --json snapshot union under schema_version.**
|
||||
**What:** a short protocol note (docs/progress-events.md sibling) pinning
|
||||
the `get_status_snapshot` v2 shape as a discriminated union on
|
||||
|
||||
@@ -334,6 +334,18 @@ Unit tests and what they cover:
|
||||
- `test/enrichment-service.test.ts` — entity slugification, extraction, tier escalation.
|
||||
- `test/data-research.test.ts` — recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping.
|
||||
- `test/minions.test.ts` — Minions job queue: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, `child_done` inbox, attachments, removeOnComplete/Fail, `max_stalled` clamp/default/plumbing coverage.
|
||||
- `test/minion-queue-renewlock-signal.test.ts` — `renewLock` forwards its optional AbortSignal to `executeRawDirect` (stub-engine capture); legacy 3-arg calls unchanged; token-fence miss returns false.
|
||||
- `test/cycle-drain-renewal.test.ts` — `runDrainRenewalTick` (cycle drain): per-call signal aborted on timeout (slot released), onLost once on a lost fence, throws swallowed, hung renewal resolves at the deadline.
|
||||
- `test/queue-probe-cancellation.test.ts` — `probeQueueState`/`queryWedgeSignals` signal threading: the 1500ms budget CANCELS the losing probe query; fast-path signals never abort; throw still collapses to `{probe_failed: true}`.
|
||||
- `test/db-pool-max-lifetime.test.ts` — `resolveMaxLifetimeSeconds`: env forms, 0-disables, 30–60min jitter bounds, warn-once on invalid, per-call jitter variance.
|
||||
- `test/pool-gauge.test.ts` — `CheckoutGauge` pure semantics + the PostgresEngine seams with fake pools: counted while in flight, released on resolve, on REJECTED queries, and on the SYNCHRONOUS pre-aborted-signal throw (leak guards); `getPoolDiagnostics` fail-open.
|
||||
- `test/db-probe.test.ts` — `runDbProbe` verdict matrix (pool_starved / server_unreachable / unknown), honest-disjunction + no-waiter-arithmetic wording pins, hung probes cancelled via their signals, diagnostics absent/throwing fail open.
|
||||
- `test/postgres-engine-reserved-routing.test.ts` — `withReservedConnection` routing: direct pool when dual-pool active, read pool when kill-switched/in-tx, semaphore cap (directPoolSize−1) with read-pool overflow, permit released on fn throw and reserve failure.
|
||||
- `test/job-isolation-protocol.test.ts` — outcome-file codec round-trip + every decode failure path (missing/malformed/oversize→UnrecoverableError; byte counts, never content), handler-error instanceof reconstruction, child-CLI invocation resolution, and REAL detached-process `killProcessGroup` tests incl. the grandchild-death guarantee (exercises the Bun negative-pid `/bin/kill` fallback for real under `bun test`).
|
||||
- `test/run-child-entry.test.ts` — `runChildJobEntry` on real in-memory PGLite with a REAL claim-minted token: success (fenced updateProgress lands), handler-failure outcome (exit 0), token-mismatch never runs the handler (exit 14), missing job/handler, parent-death watchdog aborts a live handler.
|
||||
- `test/child-job-runner.test.ts` — `runJobInChild` against real .mjs children: success + full env contract (incl. `GBRAIN_DIRECT_POOL_SIZE=1`), error/lease outcome reconstruction, crash, SIGTERM-ignorer → group SIGKILL at the injected grace, pre-aborted signal, spawn ENOENT → `ChildSpawnInfraError`, worker-shutdown drain (report-during-drain completes; non-reporting kill → `ChildWorkerShutdownError`).
|
||||
- `test/worker-job-isolation.test.ts` — full parent path on PGLite with the `fake-run-child.mjs` fixture: claim → child → fenced completeJob (real token over env), error outcome → failJob, crash burns the attempt, spawn failure RELEASES with zero attempts burned, and the codex-2 #8 serialization-parity pin (unreportable results fail in BOTH modes, never falsely complete).
|
||||
- `test/jobs-isolation-flag.test.ts` — `parseJobIsolationFlag`: space/= forms, env fallback + flag-wins, empty-env default, other flags untouched.
|
||||
- `test/extract.test.ts` — link extraction, timeline extraction, frontmatter parsing, directory type inference.
|
||||
- `test/extract-db.test.ts` — `gbrain extract --source db`: typed link inference, idempotency, `--type` filter, `--dry-run` JSON output.
|
||||
- `test/extract-fs.test.ts` — `gbrain extract --source fs`: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard for the N+1 dedup bug.
|
||||
@@ -400,6 +412,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/sync.test.ts` — `--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
|
||||
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
|
||||
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
|
||||
- `test/e2e/job-isolation.test.ts` — process isolation on real Postgres (DATABASE_URL-gated, wired EXPLICITLY into `.github/workflows/e2e.yml` tier1 — the workflow runs only named files): a concurrency-3 isolated drain through real child processes (the `fake-run-child.mjs` fixture — real spawns, no child DB pools), and the REAL `jobs run-child` CLI entrypoint end-to-end (engine bootstrap incl. the child's own pools, quiet handler registry, token validation, outcome protocol).
|
||||
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
|
||||
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
|
||||
- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -87,6 +87,69 @@ check warns if what you asked for isn't what's actually running (e.g. a
|
||||
negative value denied without privilege, or an OS `RLIMIT_NICE` clamp). This
|
||||
is distinct from the concurrency / inflight cap and composes with it.
|
||||
|
||||
### Per-job process isolation (`--job-isolation process`)
|
||||
|
||||
By default all concurrency slots execute inside one worker process. A
|
||||
handler that ignores its abort signal can only be force-evicted — the
|
||||
promise is abandoned, still running, still holding connections and memory —
|
||||
and any worker exit destroys every in-flight job at once. With isolation on,
|
||||
each claimed job runs in its own child process: a stuck handler is
|
||||
group-SIGKILLed for real (group signaling under Bun falls back to POSIX
|
||||
`/bin/kill`; if that's unavailable the worker logs that isolation is
|
||||
degraded), a crash or OOM in a child takes that one job instead of all N,
|
||||
and the OS reclaims every leaked resource when the child dies:
|
||||
|
||||
```bash
|
||||
# Recommended for long-running LLM-bound handlers (subagent):
|
||||
gbrain jobs supervisor --concurrency 4 --job-isolation process
|
||||
|
||||
# Bare worker, or durably via env:
|
||||
GBRAIN_JOB_ISOLATION=process gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
How it works: the worker keeps claim, lock renewal, and all result
|
||||
recording; the child (an internal `run-child` entrypoint of the same gbrain
|
||||
binary) re-validates the claim, runs the handler with its own small engine
|
||||
pool, and reports one atomic outcome file. Handler-error semantics are
|
||||
preserved across the boundary (unrecoverable → dead, rate-lease → no attempt
|
||||
burned, everything else → normal backoff). On worker shutdown children get
|
||||
the drain window to finish and report; a child killed before reporting is
|
||||
released with no attempt burned. If the worker dies hard, the orphaned child
|
||||
self-terminates via a parent-liveness watchdog and the stall sweeper
|
||||
requeues the job after lock expiry — the lock token fences the orphan's
|
||||
queue writes (result recording, progress, state transitions) into no-ops.
|
||||
The handler's own side effects (page writes through its engine) can still
|
||||
land until the watchdog stops the child; that window is the watchdog's
|
||||
poll + grace, not unbounded.
|
||||
|
||||
Sizing notes:
|
||||
|
||||
- **Connections:** each child opens its own small pools (read 3 by default,
|
||||
override via `GBRAIN_JOB_CHILD_POOL_SIZE`; direct 1). Worked example at
|
||||
concurrency 15: 15×(3+1) + the worker's 10+3 ≈ **73 client connections**
|
||||
total — 55 ride the transaction-pooler lane (multiplexed, no extra server
|
||||
backends) and 18 are lazy direct session-lane connections, each holding a
|
||||
real server backend while open. Budget the pooler-lane count against your
|
||||
pooler's client limit and the session-lane count against
|
||||
`max_connections`.
|
||||
- **Memory:** `--max-rss` covers the WORKER process only in this mode
|
||||
(handler memory lives in the children; the worker prints a note when both
|
||||
are set). There is no per-child RSS cap yet — a runaway child is contained
|
||||
only by host/container limits. Size host memory for concurrency × handler
|
||||
footprint.
|
||||
- **Spawn cost:** ~0.3–1s per job (engine connect included) — noise for
|
||||
long-running handlers, meaningful for sub-second ones (`lint`,
|
||||
`backlinks`). Keep those inline or on a separate inline worker.
|
||||
- **Security note:** the child receives the job's lock token via env. It is
|
||||
a *fencing* token (split-brain protection), not a secret — same-user env
|
||||
already contains the database URL.
|
||||
- **Child CLI resolution:** the worker fail-fast validates the child CLI at
|
||||
startup (compiled `gbrain` binary, bun-dev fallback, or the
|
||||
`GBRAIN_JOB_CHILD_CLI` env override — the ops/test escape hatch). Three
|
||||
consecutive child spawn/bootstrap failures self-exit the worker as
|
||||
unhealthy (a deterministically broken child CLI) for process-manager
|
||||
restart instead of burning attempts across the queue.
|
||||
|
||||
### Which supervisor when?
|
||||
|
||||
The supervisor solves in-process crash recovery. Platform-level
|
||||
|
||||
@@ -71,7 +71,10 @@ gbrain jobs get <id>
|
||||
## Rescue actions (in order of escalation)
|
||||
|
||||
```bash
|
||||
# Force-kill a single stuck job:
|
||||
# Cancel a single stuck job (inline mode: cooperative — the handler must
|
||||
# observe its abort signal, and after 30s it is force-evicted from tracking
|
||||
# but the promise keeps running; with --job-isolation process the child is
|
||||
# actually SIGTERM→SIGKILLed once cancellation is detected):
|
||||
gbrain jobs cancel <id>
|
||||
|
||||
# Clear a specific job entirely (last resort):
|
||||
@@ -119,6 +122,29 @@ claiming. Start one:
|
||||
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
|
||||
```
|
||||
|
||||
## Reading the DB-probe verdicts (pool starved vs server unreachable)
|
||||
|
||||
When the worker's health probe fails repeatedly, the terminal
|
||||
`[health] DB probe failed N consecutive times (verdict: ...)` line — and the
|
||||
`unhealthy` payload the supervisor sees — carries a verdict that names the
|
||||
failing LAYER (the intermediate `(N/3)` lines log only the failure detail).
|
||||
Read it before touching anything — the historical failure mode here was
|
||||
hours spent evaluating a database instance upgrade while the server sat at
|
||||
10% of max_connections.
|
||||
|
||||
| Verdict | What it means | What to do |
|
||||
|---|---|---|
|
||||
| `pool_starved` | The read-pool probe failed but the DIRECT-lane probe succeeded — the database server is reachable; the fault is in the transaction-pooler path (client pool exhaustion or a pooler-layer fault; the probe deliberately does not distinguish the two). | Look at client-side load: long-running handler queries holding slots, `GBRAIN_POOL_SIZE` too small for the workload, or a pooler-layer incident. Do NOT resize the database. The worker exit is correct recovery — it frees every client-held slot. |
|
||||
| `server_unreachable` | Both the pooler lane and the direct lane failed. | Check connectivity/capacity first: network, DNS, the database itself. Both-lanes-failed is the evidence — credential/config errors or a saturated direct lane can also land here, so glance at the probe detail text before concluding the server is down. |
|
||||
| `unknown` | The read probe failed and no direct lane exists to disambiguate (single-pool mode: non-Supabase, kill switch active, or no derivable direct URL). | Check the startup log for the single-pool warning; consider `GBRAIN_DIRECT_DATABASE_URL` so future incidents self-diagnose. |
|
||||
|
||||
The `gbrain-tracked in flight` counts in the message are a tracked SUBSET
|
||||
(raw/direct/reserved/transaction seams only) — most template-path queries are
|
||||
untracked, so `0 in flight` next to a `pool_starved` verdict means the
|
||||
saturation lives in that untracked traffic or at the pooler layer itself,
|
||||
not that the pool is idle. The verdict, not the counts, is the
|
||||
authoritative signal.
|
||||
|
||||
## Related
|
||||
|
||||
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
|
||||
|
||||
+1
-1
@@ -1921,7 +1921,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Opt-in per-job process isolation (`gbrain jobs work --job-isolation process`) runs each claimed job in its own SIGKILL-able child process, so a stuck handler dies for real and a crash takes one job instead of the whole worker; when the worker's DB health probe fails, it names the failing layer (`pool_starved` vs `server_unreachable`) instead of a blanket "DB unreachable". Sizing and rollout guidance in [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md); probe-verdict triage in [`docs/guides/queue-operations-runbook.md`](docs/guides/queue-operations-runbook.md). Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "gbrain-context-engine",
|
||||
"name": "gbrain",
|
||||
"version": "0.46.0.0",
|
||||
"version": "0.46.1.0",
|
||||
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
|
||||
"family": "bundle-plugin",
|
||||
"configSchema": {
|
||||
|
||||
+1
-1
@@ -157,7 +157,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.46.0.0",
|
||||
"version": "0.46.1.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.5",
|
||||
|
||||
+200
-4
@@ -6,7 +6,12 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { MinionQueue } from '../core/minions/queue.ts';
|
||||
import { MinionWorker } from '../core/minions/worker.ts';
|
||||
import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts';
|
||||
import {
|
||||
WORKER_EXIT_RSS_WATCHDOG,
|
||||
JOB_CHILD_EXIT_USAGE,
|
||||
} from '../core/minions/worker-exit-codes.ts';
|
||||
import { CHILD_ENV, resolveChildCliInvocation } from '../core/minions/job-isolation.ts';
|
||||
import { runChildJobEntry } from '../core/minions/run-child.ts';
|
||||
import type { MinionHandler, MinionJob, MinionJobStatus } from '../core/minions/types.ts';
|
||||
import type { PaceKeyOverrides } from '../core/pace-mode.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
@@ -154,6 +159,32 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export type JobIsolationMode = 'inline' | 'process';
|
||||
|
||||
/**
|
||||
* issue #5: `--job-isolation <inline|process>` (space or `=` form), env
|
||||
* fallback GBRAIN_JOB_ISOLATION, default inline. `process` runs each claimed
|
||||
* job in a SIGKILL-able child process — blast radius 1 job instead of N.
|
||||
* Env injected as a param so tests never mutate process.env (rule R1).
|
||||
* Invalid values fail fast (parseMaxRssFlag convention).
|
||||
*/
|
||||
export function parseJobIsolationFlag(
|
||||
args: string[],
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): JobIsolationMode {
|
||||
let raw: string | undefined;
|
||||
const eqForm = args.find((a) => a.startsWith('--job-isolation='));
|
||||
if (eqForm !== undefined) raw = eqForm.slice('--job-isolation='.length);
|
||||
if (raw === undefined) raw = parseFlag(args, '--job-isolation');
|
||||
if (raw === undefined || raw === '') raw = env.GBRAIN_JOB_ISOLATION;
|
||||
if (raw === undefined || raw === '') return 'inline';
|
||||
if (raw === 'inline' || raw === 'process') return raw;
|
||||
console.error(
|
||||
`Error: invalid job isolation mode ${JSON.stringify(raw)}. Valid: inline, process.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
/**
|
||||
* #3026: the thin-client `list`/`get` branches receive jobs as parsed JSON
|
||||
* off the MCP wire, where every timestamp is an ISO string — but formatJob /
|
||||
@@ -269,11 +300,13 @@ USAGE
|
||||
gbrain jobs watch [--json] [--follow] [--refresh-ms=N]
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
[--health-interval MS] [--nice N]
|
||||
[--job-isolation inline|process]
|
||||
gbrain jobs supervisor [start] [--detach] [--json]
|
||||
[--concurrency N] [--queue Q] [--pid-file PATH]
|
||||
[--max-crashes N] [--health-interval N]
|
||||
[--allow-shell-jobs] [--cli-path PATH]
|
||||
[--max-rss MB] [--nice N]
|
||||
[--job-isolation inline|process]
|
||||
|
||||
--nice N OS scheduling priority, -20 (highest) to 19 (nicest). Lowers CPU
|
||||
priority without cutting concurrency — full throughput when the
|
||||
@@ -344,9 +377,18 @@ const JOBS_SUBCOMMAND_HELP: Record<string, string> = {
|
||||
USAGE
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
[--health-interval MS] [--nice N]
|
||||
[--job-isolation inline|process]
|
||||
|
||||
OPTIONS
|
||||
--queue Q Queue to claim from (default: default)
|
||||
--job-isolation M inline (default): handlers run in the worker process.
|
||||
process: each claimed job runs in its own child
|
||||
process — a stuck handler is group-SIGKILLed instead
|
||||
of abandoned, and a crash takes one job, not all N.
|
||||
Env fallback: GBRAIN_JOB_ISOLATION. Recommended for
|
||||
long-running LLM-bound handlers (subagent). Note:
|
||||
--max-rss then covers the worker only, and each child
|
||||
adds ~4 pooler client connections.
|
||||
--concurrency N Max jobs in flight. Resolution: flag, then
|
||||
GBRAIN_WORKER_CONCURRENCY env, then 1. Values < 1
|
||||
are clamped to 1 with a loud stderr note.
|
||||
@@ -376,6 +418,7 @@ USAGE
|
||||
[--max-crashes N] [--health-interval N]
|
||||
[--allow-shell-jobs] [--cli-path PATH]
|
||||
[--max-rss MB] [--nice N]
|
||||
[--job-isolation inline|process]
|
||||
gbrain jobs supervisor status [--json] [--pid-file PATH]
|
||||
gbrain jobs supervisor stop [--json] [--pid-file PATH]
|
||||
|
||||
@@ -397,6 +440,7 @@ OPTIONS (start)
|
||||
--cli-path PATH Explicit gbrain binary for the worker child
|
||||
--max-rss MB RSS watchdog for the worker (same rules as jobs work)
|
||||
--nice N OS priority for supervisor + worker children
|
||||
--job-isolation M Passed through to the worker (see jobs work --help)
|
||||
|
||||
EXIT CODES (start)
|
||||
0 clean shutdown 1 max crashes exceeded
|
||||
@@ -1185,6 +1229,59 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
case 'run-child': {
|
||||
// INTERNAL (issue #5 process isolation): spawned by `jobs work` with
|
||||
// process isolation enabled. One job, one process: validate the claim,
|
||||
// run the handler with the child's own engine, write ONE outcome file,
|
||||
// exit. Deliberately absent from user-facing help. The CLI layer owns
|
||||
// engine.disconnect() + process.exit() (engine-ownership invariant).
|
||||
{
|
||||
const config = loadConfig();
|
||||
if (config?.engine === 'pglite') {
|
||||
console.error('[run-child] process isolation requires the Postgres engine.');
|
||||
await engine.disconnect();
|
||||
process.exit(JOB_CHILD_EXIT_USAGE);
|
||||
}
|
||||
const jobIdRaw = parseFlag(args, '--job-id');
|
||||
const jobId = jobIdRaw != null ? parseInt(jobIdRaw, 10) : NaN;
|
||||
const lockToken = process.env[CHILD_ENV.lockToken];
|
||||
const resultPath = process.env[CHILD_ENV.resultPath];
|
||||
const parentPidRaw = parseInt(process.env[CHILD_ENV.parentPid] ?? '0', 10);
|
||||
if (!Number.isInteger(jobId) || jobId <= 0 || !lockToken || !resultPath) {
|
||||
console.error(
|
||||
'[run-child] internal command spawned by the jobs worker; requires ' +
|
||||
`a numeric job id plus ${CHILD_ENV.lockToken} and ${CHILD_ENV.resultPath} in env.`,
|
||||
);
|
||||
await engine.disconnect();
|
||||
process.exit(JOB_CHILD_EXIT_USAGE);
|
||||
}
|
||||
|
||||
// Same handler surface as the worker: registerBuiltinHandlers also
|
||||
// performs plugin discovery, so plugin subagent jobs isolate too.
|
||||
const throwaway = new MinionWorker(engine, { queue: 'default', concurrency: 1 });
|
||||
await registerBuiltinHandlers(throwaway, engine, { quiet: true });
|
||||
|
||||
let code: number;
|
||||
try {
|
||||
code = await runChildJobEntry(
|
||||
engine,
|
||||
{
|
||||
jobId,
|
||||
lockToken,
|
||||
resultPath,
|
||||
parentPid: Number.isInteger(parentPidRaw) && parentPidRaw > 0 ? parentPidRaw : 0,
|
||||
},
|
||||
{ resolveHandler: (name) => throwaway.getHandler(name) },
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(`[run-child] fatal: ${e instanceof Error ? e.message : String(e)}`);
|
||||
code = 1;
|
||||
}
|
||||
await engine.disconnect();
|
||||
process.exit(code);
|
||||
}
|
||||
}
|
||||
// eslint-disable-next-line no-fallthrough -- unreachable: the case above always exits
|
||||
case 'work': {
|
||||
// Check if PGLite
|
||||
const config = (await import('../core/config.ts')).loadConfig();
|
||||
@@ -1245,11 +1342,78 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
|
||||
}
|
||||
}
|
||||
|
||||
// issue #5: per-job process isolation. Resolve + validate the child CLI
|
||||
// invocation ONCE at startup and refuse to start on failure — a bad
|
||||
// path discovered per-job would release every claim as infra failures
|
||||
// (never dead-lettering, but never progressing either).
|
||||
const jobIsolation = parseJobIsolationFlag(args);
|
||||
let childCliInvocation: { cmd: string; argsPrefix: string[] } | null = null;
|
||||
let childTiniPath = '';
|
||||
if (jobIsolation === 'process') {
|
||||
const { resolveGbrainCliPath } = await import('./autopilot.ts');
|
||||
const inv = resolveChildCliInvocation(
|
||||
process.env,
|
||||
process.execPath,
|
||||
process.argv[1],
|
||||
() => resolveGbrainCliPath(),
|
||||
);
|
||||
if (!inv) {
|
||||
console.error(
|
||||
'Error: process isolation needs a resolvable gbrain CLI for job children ' +
|
||||
'(compiled binary on PATH, or GBRAIN_JOB_CHILD_CLI override).',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
// Canonicalize BEFORE validating: existsSync on a relative name checks
|
||||
// cwd while spawn() resolves via PATH — the validated file and the
|
||||
// executed binary could differ (security review). Resolving to an
|
||||
// absolute path makes the fail-fast check and the spawn agree.
|
||||
const { existsSync: childCliExists } = await import('node:fs');
|
||||
const { resolve: resolveCliPath } = await import('node:path');
|
||||
inv.cmd = resolveCliPath(inv.cmd);
|
||||
if (!childCliExists(inv.cmd)) {
|
||||
console.error(
|
||||
`Error: resolved child CLI does not exist: ${inv.cmd} ` +
|
||||
'(set GBRAIN_JOB_CHILD_CLI to a valid gbrain binary).',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
childCliInvocation = inv;
|
||||
const { detectTini } = await import('../core/minions/spawn-helpers.ts');
|
||||
childTiniPath = detectTini();
|
||||
if (maxRssMb > 0) {
|
||||
console.error(
|
||||
'[gbrain jobs] note: with process isolation on, the --max-rss watchdog covers the ' +
|
||||
'WORKER process only — handler memory now lives in job children. Per-child caps are ' +
|
||||
'a filed follow-up; size host memory for concurrency x handler footprint.',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
// issue #6: the direct-pool kill switch collapses lock renewal, health
|
||||
// probes, and handler workload onto ONE shared pool — silently. Make
|
||||
// the collapse loud at startup so a later 'pool_starved' incident has
|
||||
// an obvious prior warning instead of a mystery.
|
||||
{
|
||||
const { getConnectionRouting } = await import('../core/minions/db-probe.ts');
|
||||
const cm = getConnectionRouting(engine);
|
||||
if (cm?.isDualPoolActive && !cm.isDualPoolActive()) {
|
||||
const killSwitched = cm.describeMode?.().kill_switch_active === true;
|
||||
console.error(
|
||||
`[gbrain jobs] single-pool mode: lock renewal, health probes and handler workload share ` +
|
||||
`one connection pool${killSwitched ? ' (direct-lane kill switch is active)' : ''}. ` +
|
||||
`Under heavy handler load this pool can starve the lock heartbeat. For Supabase brains, ` +
|
||||
`ensure the direct (5432) host is reachable or set GBRAIN_DIRECT_DATABASE_URL.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: queueName, concurrency, maxRssMb, healthCheckInterval,
|
||||
jobIsolation, childCliInvocation, childTiniPath,
|
||||
});
|
||||
await registerBuiltinHandlers(worker, engine);
|
||||
|
||||
@@ -1259,9 +1423,37 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
|
||||
// the external PM (systemd, Docker, cron watchdog) restart cleanly.
|
||||
worker.on('unhealthy', (info) => {
|
||||
if (info.reason === 'db_dead') {
|
||||
// issue #6: name the failing LAYER, not just "DB unreachable" —
|
||||
// that message sent operators chasing database capacity while the
|
||||
// real fault was client-side pool exhaustion. Exiting is still
|
||||
// correct recovery either way (it frees every client-held slot).
|
||||
if (info.verdict === 'pool_starved') {
|
||||
console.error(
|
||||
`[health] FATAL: connection-pool path saturated after ${info.consecutiveFailures} probes — ` +
|
||||
`the database server itself is reachable. (${info.message}) ` +
|
||||
`Likely causes: long-running handler queries holding pool slots, or too-small GBRAIN_POOL_SIZE ` +
|
||||
`for this workload. Consider --job-isolation process for long-running handlers ` +
|
||||
`(handler connections then die with each job's child process). ` +
|
||||
`Exiting for process-manager restart (frees all client-held slots).`,
|
||||
);
|
||||
} else if (info.verdict === 'server_unreachable') {
|
||||
console.error(
|
||||
`[health] FATAL: database server unreachable after ${info.consecutiveFailures} probes ` +
|
||||
`(both pooler and direct lanes failed). (${info.message}) ` +
|
||||
`Exiting for process-manager restart.`,
|
||||
);
|
||||
} else {
|
||||
console.error(
|
||||
`[health] FATAL: DB probe failed ${info.consecutiveFailures} consecutive times (${info.message}). ` +
|
||||
`Exiting for process-manager restart.`,
|
||||
);
|
||||
}
|
||||
} else if (info.reason === 'child_spawn_failing') {
|
||||
console.error(
|
||||
`[health] FATAL: DB unreachable after ${info.consecutiveFailures} probes (${info.message}). ` +
|
||||
`Exiting for process-manager restart.`,
|
||||
`[health] FATAL: ${info.consecutiveFailures} consecutive job-child spawn/bootstrap ` +
|
||||
`failures (${info.message}). The child CLI is deterministically broken — fix the ` +
|
||||
`worker's child CLI configuration (or GBRAIN_JOB_CHILD_CLI). Exiting for ` +
|
||||
`process-manager restart.`,
|
||||
);
|
||||
} else {
|
||||
console.error(
|
||||
@@ -1290,7 +1482,10 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
|
||||
: `, health-check: ${Math.round(healthCheckInterval / 1000)}s`)
|
||||
: '';
|
||||
const niceNote = niceResult ? `, nice: ${formatNice(niceResult.effective ?? niceVal!)}` : '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote}${niceNote})`);
|
||||
const isolationNote = jobIsolation === 'process'
|
||||
? `, isolation: process (child cli: ${childCliInvocation?.cmd}${childTiniPath ? ', tini' : ''})`
|
||||
: '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote}${niceNote}${isolationNote})`);
|
||||
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
|
||||
|
||||
// Register in the live worker registry (issue #1815) so jobs stats / doctor
|
||||
@@ -1609,6 +1804,7 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
|
||||
allowShellJobs,
|
||||
json: jsonMode,
|
||||
maxRssMb,
|
||||
jobIsolation: parseJobIsolationFlag(args),
|
||||
...(supNice !== undefined ? { nice_requested: supNice } : {}),
|
||||
...(supNiceResult?.effective != null ? { nice_effective: supNiceResult.effective } : {}),
|
||||
...(supNiceResult?.error ? { nice_error: supNiceResult.error } : {}),
|
||||
|
||||
@@ -40,7 +40,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
'config': ['--aliases', '--all', '--brain', '--column', '--coverage-override', '--detail', '--embedding-dimensions', '--embedding-model', '--fast', '--federated-read', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--markdown', '--model', '--multimodal', '--no-embedding', '--no-extract', '--no-federated', '--pattern', '--pending', '--pglite', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--token-ttl', '--yes'],
|
||||
'connect': ['--agent', '--bearer-token-env-var', '--bind', '--brain', '--client-id', '--client-secret', '--force', '--grant-types', '--help', '--http', '--install', '--json', '--name', '--oauth', '--public-url', '--register', '--scope', '--scopes', '--show-token', '--source', '--timeout-ms', '--token', '--token-endpoint-auth-method', '--url', '--version', '--yes'],
|
||||
'conversation-parser': ['--aliases', '--all', '--brain', '--help', '--include-null-signature', '--json', '--no-extract', '--pattern', '--pending', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--thin'],
|
||||
'doctor': ['--ab', '--abbrev-ref', '--abi', '--abort', '--aliases', '--all', '--allow-shell-jobs', '--allow-unverified-remote', '--auto', '--auto-fix', '--auto-update', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--branch', '--break-lock', '--build-index', '--by-mention', '--by-type', '--cached', '--check', '--column', '--compile', '--concurrency', '--confidence', '--confirm', '--content-audit', '--count', '--days', '--delete-brain', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exclude-standard', '--exclusive', '--explain', '--fast', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--force-schema', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--get', '--git-dir', '--git-path', '--grant-types', '--harness', '--health-interval', '--help', '--history', '--home', '--http', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--include-pseudo', '--index-audit', '--init', '--input', '--is-inside-work-tree', '--jq', '--json', '--lang', '--limit', '--local', '--locks', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-crashes', '--max-jobs', '--max-rss', '--max-usd', '--mcp-only', '--migrate-only', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--nice', '--no', '--no-cron', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-mutate', '--no-verify', '--oauth-client-secret', '--older-than', '--once', '--others', '--overwrite', '--parallel', '--params', '--pat-file', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--priority', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--push-only', '--query', '--queue', '--quiet', '--rebase', '--rebuild-rollup', '--refresh', '--refresh-unqualified', '--regenerate', '--remediate', '--remediation-plan', '--remove', '--repo', '--reset', '--resolve', '--restore-only', '--resume', '--review-lower', '--rollback', '--scope', '--scopes', '--set', '--short', '--show-current', '--show-toplevel', '--since', '--skills-dir', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slugs', '--source', '--source-id', '--stale', '--stats', '--status', '--strategy', '--strict', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--target', '--target-score', '--thin', '--timeout', '--to', '--token', '--token-ttl', '--top-k', '--type', '--undo-wave', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--use-captured-snapshot', '--verbose', '--verify', '--version', '--window', '--with-calibration', '--workers', '--yes'],
|
||||
'doctor': ['--ab', '--abbrev-ref', '--abi', '--abort', '--aliases', '--all', '--allow-shell-jobs', '--allow-unverified-remote', '--auto', '--auto-fix', '--auto-update', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--branch', '--break-lock', '--build-index', '--by-mention', '--by-type', '--cached', '--check', '--column', '--compile', '--concurrency', '--confidence', '--confirm', '--content-audit', '--count', '--days', '--delete-brain', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exclude-standard', '--exclusive', '--explain', '--fast', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--force-schema', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--get', '--git-dir', '--git-path', '--grant-types', '--harness', '--health-interval', '--help', '--history', '--home', '--http', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--include-pseudo', '--index-audit', '--init', '--input', '--is-inside-work-tree', '--job-isolation', '--jq', '--json', '--lang', '--limit', '--local', '--locks', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-crashes', '--max-jobs', '--max-rss', '--max-usd', '--mcp-only', '--migrate-only', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--nice', '--no', '--no-cron', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-mutate', '--no-verify', '--oauth-client-secret', '--older-than', '--once', '--others', '--overwrite', '--parallel', '--params', '--pat-file', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--priority', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--push-only', '--query', '--queue', '--quiet', '--rebase', '--rebuild-rollup', '--refresh', '--refresh-unqualified', '--regenerate', '--remediate', '--remediation-plan', '--remove', '--repo', '--reset', '--resolve', '--restore-only', '--resume', '--review-lower', '--rollback', '--scope', '--scopes', '--set', '--short', '--show-current', '--show-toplevel', '--since', '--skills-dir', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slugs', '--source', '--source-id', '--stale', '--stats', '--status', '--strategy', '--strict', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--target', '--target-score', '--thin', '--timeout', '--to', '--token', '--token-ttl', '--top-k', '--type', '--undo-wave', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--use-captured-snapshot', '--verbose', '--verify', '--version', '--window', '--with-calibration', '--workers', '--yes'],
|
||||
'dream': ['--against', '--aliases', '--all', '--allow-regression', '--anchor', '--asof', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--by-type', '--by-type-floor', '--code', '--committed-baseline', '--compare', '--compile', '--concurrent', '--ctx-size', '--cycles', '--date', '--detail', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--embeddings', '--expansion', '--explain', '--fast', '--federated', '--fix', '--fixtures', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-retry', '--format', '--from', '--from-db', '--from-pages', '--gold', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--install', '--json', '--judge-model', '--justification', '--keyword-only', '--lang', '--limit', '--llm', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--max-tokens', '--max-usd', '--mcp-only', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--name-only', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-llm', '--no-mutate', '--no-trajectory', '--once', '--out', '--output', '--output-dir', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--phase', '--priority', '--progress-interval', '--progress-json', '--pull', '--quiet', '--receipt-dir', '--remediate', '--repo', '--reranking', '--reset', '--resolve', '--restore-only', '--resume-from', '--retrieval-only', '--rounds', '--rubric-version', '--save', '--seed', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--source-id', '--stale', '--suite', '--suites', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--take', '--task', '--thin', '--threshold', '--timeout', '--to', '--token-ttl', '--top-k', '--undo', '--unsafe-bypass-dream-guard', '--update-baseline', '--verify', '--version', '--window', '--yes'],
|
||||
'edges-backfill': ['--aliases', '--all', '--all-sources', '--brain', '--concurrency', '--federated', '--help', '--include-null-signature', '--json', '--max-age', '--max-chunks', '--max-cost-usd', '--no-extract', '--no-federated', '--older-than', '--path', '--pattern', '--pending', '--repo', '--reset', '--resolve', '--restore-only', '--source', '--stale', '--supersessions', '--thin', '--timeout', '--workers'],
|
||||
'embed': ['--aliases', '--all', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--catch-up', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--serial', '--slugs', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--token-ttl', '--version'],
|
||||
@@ -61,7 +61,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
|
||||
'init': ['--all', '--brain', '--chat-model', '--check', '--ctx-size', '--embedding-dimensions', '--embedding-model', '--entity', '--expansion-model', '--fast', '--flag', '--force', '--from-pages', '--grant-types', '--help', '--http', '--issuer-url', '--json', '--judge-model', '--key', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--path', '--pglite', '--provenance', '--schema-pack', '--scopes', '--skip-embed-check', '--source', '--stale', '--supabase', '--surface', '--to', '--token-ttl', '--touchpoint', '--url', '--version'],
|
||||
'integrations': ['--auto', '--brain', '--dry-run', '--embeddings', '--fast', '--force', '--from-pages', '--help', '--http', '--json', '--no-embedding', '--overwrite', '--refresh', '--reranking', '--source', '--surface', '--target', '--token-ttl'],
|
||||
'integrity': ['--aliases', '--all', '--auto', '--backend', '--background', '--brain', '--brain-wide-max-cost-usd', '--check', '--confidence', '--cost', '--dry-run', '--explain', '--fast', '--follow', '--force', '--fresh', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--limit', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--review-lower', '--skip-bare-tweet', '--skip-urls', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl', '--type', '--url'],
|
||||
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
|
||||
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--job-id', '--job-isolation', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
|
||||
'lint': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--exclude', '--explain', '--fast', '--fix', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl'],
|
||||
'lsd': ['--brain', '--force-resume', '--help', '--json', '--judge-model', '--limit', '--list-runs', '--max-cost', '--max-far-set', '--max-ideas-per-judge-call', '--no-save', '--resume', '--retry-judge', '--save', '--source', '--strict-budget', '--yes'],
|
||||
'maintain': ['--aliases', '--all', '--background', '--brain', '--break-lock', '--by-mention', '--catch-up', '--column', '--concurrency', '--content-audit', '--count', '--detach', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--force', '--force-retry', '--force-schema', '--from-meetings', '--full', '--help', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--index-audit', '--infer-dates', '--input', '--json', '--kind', '--lang', '--locks', '--markdown', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-usd', '--migrate-only', '--multimodal', '--near-symbol', '--ner', '--nice', '--no-extract', '--no-mutate', '--older-than', '--once', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--probe-pglite', '--progress-json', '--query', '--queue', '--quiet', '--rebuild-rollup', '--regenerate', '--remediate', '--remediation-plan', '--reset', '--resolve', '--restore-only', '--resume', '--run-id', '--safe', '--scope', '--since', '--skills-dir', '--skip-failed', '--slugs', '--source', '--source-id', '--stale', '--status', '--supabase', '--supersessions', '--symbol-kind', '--target', '--target-score', '--thin', '--to', '--top-k', '--type', '--unsafe-bypass-dream-guard', '--url', '--verbose', '--window', '--workers', '--yes'],
|
||||
|
||||
@@ -37,7 +37,7 @@
|
||||
*/
|
||||
|
||||
import postgres from 'postgres';
|
||||
import { resolvePrepare, resolveSessionTimeouts, resolvePoolSize, endPoolBounded } from './db.ts';
|
||||
import { resolvePrepare, resolveSessionTimeouts, resolvePoolSize, resolveMaxLifetimeSeconds, endPoolBounded } from './db.ts';
|
||||
import { redactPgUrl } from './url-redact.ts';
|
||||
import { logConnectionEvent } from './connection-audit.ts';
|
||||
|
||||
@@ -303,6 +303,8 @@ export class ConnectionManager {
|
||||
max: resolvePoolSize(this.opts.readPoolSize),
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
|
||||
max_lifetime: resolveMaxLifetimeSeconds(),
|
||||
types: { bigint: postgres.BigInt },
|
||||
};
|
||||
const timeouts = resolveSessionTimeouts();
|
||||
@@ -406,6 +408,8 @@ export class ConnectionManager {
|
||||
max: size,
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
|
||||
max_lifetime: resolveMaxLifetimeSeconds(),
|
||||
types: { bigint: postgres.BigInt },
|
||||
// Always use prepared statements on the direct pool — no PgBouncer
|
||||
// here, so the prepare-cache invalidation issue doesn't apply.
|
||||
|
||||
@@ -241,6 +241,55 @@ export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): st
|
||||
|
||||
// ── Public entry ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* One drain-loop lock-renewal tick, extracted for hermetic tests (worker.ts
|
||||
* parity is `runLockRenewalTick`; this is the deliberately simpler best-effort
|
||||
* variant — no audit channel, no reconnect, no time-based give-up).
|
||||
*
|
||||
* Fixes the issue #6 abandoned-racer class in the cycle drain: the previous
|
||||
* inline tick had no per-call timeout, so a hung renewLock stacked one
|
||||
* checked-out pool slot per interval firing forever. Now each call carries an
|
||||
* AbortSignal that is aborted when the timeout wins the race (the query is
|
||||
* cancelled and its slot released), and callers guard re-entrancy so at most
|
||||
* one renewal is in flight.
|
||||
*
|
||||
* Returns after the renewal settles or times out; a `false` renewal invokes
|
||||
* `onLost` (token fence lost — caller aborts the handler). Errors and
|
||||
* timeouts are swallowed: best-effort, the next tick retries.
|
||||
*/
|
||||
export async function runDrainRenewalTick(
|
||||
renewLock: (
|
||||
id: number,
|
||||
lockToken: string,
|
||||
lockMs: number,
|
||||
opts?: { signal?: AbortSignal },
|
||||
) => Promise<boolean>,
|
||||
jobId: number,
|
||||
lockToken: string,
|
||||
lockMs: number,
|
||||
onLost: () => void,
|
||||
callTimeoutMs: number,
|
||||
): Promise<void> {
|
||||
const callAbort = new AbortController();
|
||||
let timer: ReturnType<typeof setTimeout> | null = null;
|
||||
try {
|
||||
const ok = await Promise.race([
|
||||
renewLock(jobId, lockToken, lockMs, { signal: callAbort.signal }),
|
||||
new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
callAbort.abort();
|
||||
reject(new Error(`renewLock timed out after ${callTimeoutMs}ms`));
|
||||
}, callTimeoutMs);
|
||||
}),
|
||||
]);
|
||||
if (!ok) onLost();
|
||||
} catch {
|
||||
/* best-effort; next tick retries */
|
||||
} finally {
|
||||
if (timer != null) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
export interface SynthesizePhaseOpts {
|
||||
brainDir: string;
|
||||
dryRun: boolean;
|
||||
@@ -433,12 +482,25 @@ export async function runSubagentsInline(
|
||||
// just its own) can't requeue a live child. A false return means the row
|
||||
// was cancelled or reclaimed — abort the handler. Errors are swallowed
|
||||
// (best-effort; the next tick retries), never an unhandledRejection.
|
||||
// Re-entrancy guard + per-call cancellation via runDrainRenewalTick: a
|
||||
// hung renewLock no longer stacks a fresh checked-out pool slot per
|
||||
// interval firing (issue #6 abandoned-racer class).
|
||||
let drainTickInFlight = false;
|
||||
const renewTimer = setInterval(() => {
|
||||
queue.renewLock(job.id, lockToken, lockMs)
|
||||
.then((ok) => {
|
||||
if (!ok && !abort.signal.aborted) abort.abort(new Error('lock-renewal-failed'));
|
||||
})
|
||||
.catch(() => { /* best-effort; next tick retries */ });
|
||||
if (drainTickInFlight) return;
|
||||
drainTickInFlight = true;
|
||||
void runDrainRenewalTick(
|
||||
(id, tok, ms, opts) => queue.renewLock(id, tok, ms, opts),
|
||||
job.id,
|
||||
lockToken,
|
||||
lockMs,
|
||||
() => {
|
||||
if (!abort.signal.aborted) abort.abort(new Error('lock-renewal-failed'));
|
||||
},
|
||||
Math.max(1000, Math.floor(lockMs / 3)),
|
||||
).finally(() => {
|
||||
drainTickInFlight = false;
|
||||
});
|
||||
}, Math.max(50, Math.floor(lockMs / 3)));
|
||||
// Run, then record — separated so a completeJob connection error can't
|
||||
// masquerade as a handler failure, and a failJob connection error can't
|
||||
|
||||
+32
-6
@@ -43,8 +43,13 @@ export interface DbLockHandle {
|
||||
* false means the lock was stolen or released — the caller must stop
|
||||
* relying on mutual exclusion. Transient DB errors still THROW (they are
|
||||
* not evidence of a steal; the TTL is the backstop).
|
||||
*
|
||||
* `opts.signal` cancels the in-flight UPDATE when the caller's heartbeat
|
||||
* timeout gives up on it (issue #6 — an abandoned refresh otherwise holds
|
||||
* a checked-out pool slot for its full server-side duration). PGLite
|
||||
* ignores the signal (single embedded connection, no pool to starve).
|
||||
*/
|
||||
refresh: () => Promise<boolean>;
|
||||
refresh: (opts?: { signal?: AbortSignal }) => Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -268,7 +273,7 @@ export async function tryAcquireDbLock(
|
||||
return {
|
||||
id: lockId,
|
||||
acquiredAt: fence,
|
||||
refresh: async () => {
|
||||
refresh: async (refreshOpts?: { signal?: AbortSignal }) => {
|
||||
// v0.41.13.0: bump BOTH ttl_expires_at AND last_refreshed_at.
|
||||
// v0.42.x (#1794): route through the DIRECT session pool, not the
|
||||
// transaction pool, so a Supavisor pooler exhaustion (EMAXCONNSESSION)
|
||||
@@ -280,6 +285,7 @@ export async function tryAcquireDbLock(
|
||||
WHERE id = $2 AND holder_pid = $3 AND extract(epoch from acquired_at)::text = $4
|
||||
RETURNING id`,
|
||||
[ttl, lockId, pid, fence],
|
||||
refreshOpts,
|
||||
);
|
||||
return updated.length > 0;
|
||||
},
|
||||
@@ -882,8 +888,13 @@ export async function withRefreshingLock<T>(
|
||||
if (!handle) throw new LockUnavailableError(lockId);
|
||||
|
||||
let healthOk = true;
|
||||
// Re-entrancy guard: with a 15s minimum cadence and a 30s default timeout,
|
||||
// two ticks can overlap on a slow pool — one refresh in flight at a time.
|
||||
let refreshTickInFlight = false;
|
||||
|
||||
const interval = setInterval(() => {
|
||||
if (refreshTickInFlight) return;
|
||||
refreshTickInFlight = true;
|
||||
void (async () => {
|
||||
try {
|
||||
// v0.42.x (#1794, V1): the refresh IS the heartbeat. handle.refresh()
|
||||
@@ -896,10 +907,23 @@ export async function withRefreshingLock<T>(
|
||||
// health, and we do NOT clearInterval on a transient failure: a blip
|
||||
// self-heals on the next tick; the TTL is the backstop if the pool stays
|
||||
// genuinely dead (at which point a steal is correct).
|
||||
const timeout = new Promise<never>((_, reject) =>
|
||||
setTimeout(() => reject(new Error('refresh_timeout')), heartbeatTimeoutMs)
|
||||
);
|
||||
const stillOwned = await Promise.race([handle.refresh(), timeout]);
|
||||
// issue #6: abort the per-tick signal when the timeout wins so the
|
||||
// losing UPDATE is cancelled (slot released), not orphaned on the
|
||||
// direct pool for its full server-side duration.
|
||||
const tickAbort = new AbortController();
|
||||
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
const timeout = new Promise<never>((_, reject) => {
|
||||
timeoutTimer = setTimeout(() => {
|
||||
tickAbort.abort();
|
||||
reject(new Error('refresh_timeout'));
|
||||
}, heartbeatTimeoutMs);
|
||||
});
|
||||
let stillOwned: boolean;
|
||||
try {
|
||||
stillOwned = await Promise.race([handle.refresh({ signal: tickAbort.signal }), timeout]);
|
||||
} finally {
|
||||
if (timeoutTimer != null) clearTimeout(timeoutTimer);
|
||||
}
|
||||
if (stillOwned === false) {
|
||||
// W0 (D5.10): the fenced refresh matched 0 rows — the lock was
|
||||
// stolen or force-cleared. Further refreshes are pointless (and a
|
||||
@@ -918,6 +942,8 @@ export async function withRefreshingLock<T>(
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[lock-refresh] ${lockId}: ${msg}; will retry next tick\n`);
|
||||
healthOk = false;
|
||||
} finally {
|
||||
refreshTickInFlight = false;
|
||||
}
|
||||
})();
|
||||
}, refreshIntervalMs);
|
||||
|
||||
@@ -115,6 +115,55 @@ export function resolvePoolSize(explicit?: number): number {
|
||||
return DEFAULT_POOL_SIZE_FALLBACK;
|
||||
}
|
||||
|
||||
let warnedBadMaxLifetime = false;
|
||||
/** Test-only: reset the warn-once latch. */
|
||||
export function _resetMaxLifetimeWarningForTests(): void {
|
||||
warnedBadMaxLifetime = false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-pool connection max lifetime for every postgres() call site
|
||||
* (module singleton, engine instance pool, ConnectionManager read + direct
|
||||
* pools).
|
||||
*
|
||||
* postgres.js already defaults to `60 * (30 + Math.random() * 30)` — and
|
||||
* critically that built-in default is a FUNCTION, re-evaluated PER
|
||||
* CONNECTION (connection.js: `typeof seconds === 'function' ? seconds() :
|
||||
* seconds`), so each connection gets its own 30–60min deadline. A
|
||||
* pre-evaluated number would make every connection in a pool share ONE
|
||||
* recycle deadline — a warm-up burst then reconnects simultaneously
|
||||
* (data-migration specialist finding). The default here is therefore the
|
||||
* same per-connection jitter function; only the env override returns a
|
||||
* fixed number (the explicit escape hatch):
|
||||
*
|
||||
* GBRAIN_POOL_MAX_LIFETIME_S=900 # recycle after 15 min
|
||||
* GBRAIN_POOL_MAX_LIFETIME_S=0 # disable recycling entirely
|
||||
*
|
||||
* max_lifetime only recycles connections as they are RETURNED to the pool;
|
||||
* it cannot reclaim a leaked checkout — this is explicitness + a knob, not
|
||||
* a starvation fix. Invalid values warn once on stderr and fall back to the
|
||||
* default. The env param is injectable so tests never mutate process.env.
|
||||
*/
|
||||
export function resolveMaxLifetimeSeconds(
|
||||
env: Record<string, string | undefined> = process.env,
|
||||
): number | null | (() => number) {
|
||||
const raw = env.GBRAIN_POOL_MAX_LIFETIME_S;
|
||||
if (raw !== undefined && raw !== '') {
|
||||
const parsed = Number(raw);
|
||||
if (Number.isFinite(parsed) && Number.isInteger(parsed) && parsed >= 0) {
|
||||
return parsed === 0 ? null : parsed;
|
||||
}
|
||||
if (!warnedBadMaxLifetime) {
|
||||
warnedBadMaxLifetime = true;
|
||||
process.stderr.write(
|
||||
`[gbrain] Ignoring invalid GBRAIN_POOL_MAX_LIFETIME_S=${JSON.stringify(raw)} (want a non-negative integer of seconds; 0 disables); using the jittered 30-60min default\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// Per-connection jitter, matching the postgres.js built-in default shape.
|
||||
return () => Math.floor(60 * (30 + Math.random() * 30));
|
||||
}
|
||||
|
||||
/**
|
||||
* Session-level GUCs applied to every new backend connection. Prevents
|
||||
* orphan pgbouncer sessions from holding locks or running queries
|
||||
@@ -240,6 +289,8 @@ export async function connect(config: EngineConfig): Promise<boolean> {
|
||||
max: resolvePoolSize(),
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
|
||||
max_lifetime: resolveMaxLifetimeSeconds(),
|
||||
types: {
|
||||
// Register pgvector type
|
||||
bigint: postgres.BigInt,
|
||||
|
||||
@@ -0,0 +1,309 @@
|
||||
/**
|
||||
* Parent-side child-process runner for per-job isolation (issue #5).
|
||||
*
|
||||
* `runJobInChild` is the one-line seam executeJob swaps in for
|
||||
* `handler(context)` when isolation is on. The parent keeps claim, lock
|
||||
* renewal and ALL result recording; this module owns spawn → signal → reap →
|
||||
* decode:
|
||||
*
|
||||
* spawn — detached (own process group; group signals reach handler
|
||||
* grandchildren even under tini), tini-wrapped when available,
|
||||
* stdio ['ignore','inherit','inherit'] so handler logs stream
|
||||
* to the operator; results travel by outcome file, never stdout.
|
||||
* signal — per-job abort (timeout / cancel / lock-lost /
|
||||
* lock-renewal-failed) → group SIGTERM now, group SIGKILL at
|
||||
* +CHILD_KILL_GRACE_MS (25s — inside the worker's 30s
|
||||
* force-evict window, which stays as an untouched backstop).
|
||||
* Worker shutdown → same SIGTERM (the child's own handler fires
|
||||
* ctx.shutdownSignal, giving handlers the drain window to
|
||||
* finish AND write their outcome) with the SIGKILL backstop.
|
||||
* classify — outcome file presence rules (job-isolation.ts). No file:
|
||||
* per-job abort → generic throw (executeJob's catch reads
|
||||
* abort.signal.reason, so infra aborts still burn no attempt);
|
||||
* worker shutdown → ChildWorkerShutdownError (released, NO
|
||||
* attempt burned — a routine deploy must not burn attempts;
|
||||
* codex-2 #7); otherwise a crash (attempt burned, correct).
|
||||
* Pre-exec spawn failure → ChildSpawnInfraError (released, no
|
||||
* attempt burned: one bad CLI path must not dead-letter a
|
||||
* queue; the CLI layer also fail-fast validates at startup).
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcess } from 'node:child_process';
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { buildSpawnInvocation } from './spawn-helpers.ts';
|
||||
import {
|
||||
UnrecoverableError,
|
||||
ABORT_REASON_TIMEOUT,
|
||||
ABORT_REASON_LOCK_LOST,
|
||||
ABORT_REASON_LOCK_RENEWAL_FAILED,
|
||||
} from './types.ts';
|
||||
import {
|
||||
JOB_CHILD_EXIT_USAGE,
|
||||
JOB_CHILD_EXIT_NOT_CLAIMED,
|
||||
} from './worker-exit-codes.ts';
|
||||
import {
|
||||
CHILD_ENV,
|
||||
CHILD_KILL_GRACE_MS,
|
||||
CHILD_READ_POOL_MAX,
|
||||
buildChildArgs,
|
||||
decodeChildOutcomeFileAsync,
|
||||
killProcessGroup,
|
||||
reconstructHandlerError,
|
||||
unrefTimer,
|
||||
type ChildCliInvocation,
|
||||
} from './job-isolation.ts';
|
||||
|
||||
/** Pre-exec spawn failure — infrastructure, not a job defect. executeJob
|
||||
* releases the job with no attempt burned (stall sweeper requeues). */
|
||||
export class ChildSpawnInfraError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'ChildSpawnInfraError';
|
||||
}
|
||||
}
|
||||
|
||||
/** Child terminated by worker shutdown before it could report. Released with
|
||||
* no attempt burned — routine deploys must not burn attempts (codex-2 #7). */
|
||||
export class ChildWorkerShutdownError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'ChildWorkerShutdownError';
|
||||
}
|
||||
}
|
||||
|
||||
/** Child found the job reclaimed/cancelled (exit 14) — provably owned
|
||||
* elsewhere. The worker releases without failJob (the fenced failJob would
|
||||
* no-op anyway); definitely not an attempt against THIS claim. */
|
||||
export class ChildNotClaimedError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'ChildNotClaimedError';
|
||||
}
|
||||
}
|
||||
|
||||
/** Per-job abort reasons that mean THE JOB was targeted (timeout / lock
|
||||
* loss) rather than the worker winding down. gracefulShutdown('watchdog')
|
||||
* aborts BOTH the shutdown signal and every per-job signal — the shutdown
|
||||
* classification must win for those (adversarial-review P3: the watchdog
|
||||
* drain otherwise burns an attempt on innocent isolated jobs). Built from
|
||||
* the shared literals in types.ts so a rename at an abort site cannot
|
||||
* silently flip child classification (maintainability review — the
|
||||
* never-produced 'cancel'/'cancelled' entries were dropped: cancellation
|
||||
* surfaces as lock-lost via the fenced renewLock). */
|
||||
const PER_JOB_ABORT_REASONS = new Set<string>([
|
||||
ABORT_REASON_TIMEOUT,
|
||||
ABORT_REASON_LOCK_LOST,
|
||||
ABORT_REASON_LOCK_RENEWAL_FAILED,
|
||||
]);
|
||||
|
||||
export interface RunJobInChildOpts {
|
||||
jobId: number;
|
||||
jobName: string;
|
||||
lockToken: string;
|
||||
/** Per-job abort (timeout / cancel / lock-lost / lock-renewal-failed). */
|
||||
abortSignal: AbortSignal;
|
||||
/** Worker-process SIGTERM/SIGINT. */
|
||||
shutdownSignal: AbortSignal;
|
||||
/** Resolved once at worker startup (fail-fast); how to invoke the CLI. */
|
||||
invocation: ChildCliInvocation;
|
||||
/** tini path ('' when absent — direct spawn, same degradation as the supervisor). */
|
||||
tiniPath: string;
|
||||
/** Injectable for tests. Default CHILD_KILL_GRACE_MS. */
|
||||
killGraceMs?: number;
|
||||
/** Injectable base env for tests. Default process.env. */
|
||||
env?: Record<string, string | undefined>;
|
||||
}
|
||||
|
||||
interface ChildExit {
|
||||
code: number | null;
|
||||
signal: NodeJS.Signals | null;
|
||||
spawnErr?: Error;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one claimed job in a child process. Resolves with the handler result
|
||||
* (parent then runs the normal completeJob path); throws reconstructed
|
||||
* handler errors / classification errors (parent's existing catch handles
|
||||
* them verbatim).
|
||||
*/
|
||||
export async function runJobInChild(opts: RunJobInChildOpts): Promise<unknown> {
|
||||
const dir = mkdtempSync(join(tmpdir(), `gbrain-job-${opts.jobId}-`));
|
||||
const resultPath = join(dir, 'outcome.json');
|
||||
const graceMs = opts.killGraceMs ?? CHILD_KILL_GRACE_MS;
|
||||
const base = opts.env ?? process.env;
|
||||
|
||||
// Bound the child's pools: sockets die with the process (the isolation
|
||||
// win), but per-child footprint must stay small — read pool <= 3, direct
|
||||
// pool 1 (a child runs no claim/renewal heartbeats; codex-2 #6). An
|
||||
// operator's own GBRAIN_POOL_SIZE is respected when STRICTER than the
|
||||
// default (their pooler MaxClients tuning must not be silently raised);
|
||||
// GBRAIN_JOB_CHILD_POOL_SIZE, when valid, is the explicit per-child knob
|
||||
// and wins outright. Invalid values fall through to the default.
|
||||
const parsePoolSize = (v: string | undefined): number | null => {
|
||||
if (v === undefined || v === '') return null;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isInteger(n) && n > 0 ? n : null;
|
||||
};
|
||||
const childOverride = parsePoolSize(base[CHILD_ENV.childPoolSize]);
|
||||
const userPool = parsePoolSize(base.GBRAIN_POOL_SIZE);
|
||||
const childPoolSize = childOverride ?? Math.min(userPool ?? CHILD_READ_POOL_MAX, CHILD_READ_POOL_MAX);
|
||||
|
||||
const childEnv: Record<string, string | undefined> = {
|
||||
...base,
|
||||
[CHILD_ENV.lockToken]: opts.lockToken,
|
||||
[CHILD_ENV.resultPath]: resultPath,
|
||||
[CHILD_ENV.isChild]: '1',
|
||||
[CHILD_ENV.parentPid]: String(process.pid),
|
||||
GBRAIN_POOL_SIZE: String(childPoolSize),
|
||||
GBRAIN_DIRECT_POOL_SIZE: '1',
|
||||
};
|
||||
|
||||
const inv = buildSpawnInvocation(opts.tiniPath, opts.invocation.cmd, [
|
||||
...opts.invocation.argsPrefix,
|
||||
...buildChildArgs(opts.jobId),
|
||||
]);
|
||||
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(inv.cmd, inv.args, {
|
||||
stdio: ['ignore', 'inherit', 'inherit'],
|
||||
env: childEnv as NodeJS.ProcessEnv,
|
||||
detached: true,
|
||||
});
|
||||
} catch (e) {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
throw new ChildSpawnInfraError(`job child spawn failed (${inv.cmd}): ${msg}`);
|
||||
}
|
||||
|
||||
let killTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let termed = false;
|
||||
const terminate = (): void => {
|
||||
if (termed) return;
|
||||
termed = true;
|
||||
if (child.pid != null) {
|
||||
killProcessGroup(child.pid, 'SIGTERM');
|
||||
killTimer = setTimeout(() => {
|
||||
// Loud on failure (red-team finding): the /bin/kill fallback is the
|
||||
// NORMAL delivery path in Bun-compiled binaries, and a container
|
||||
// without /bin/kill (distroless) would otherwise silently void the
|
||||
// SIGKILL guarantee while the child runs to completion and the job
|
||||
// gets requeued elsewhere (duplicate side effects).
|
||||
if (child.pid != null && child.exitCode == null && child.signalCode == null) {
|
||||
const delivered = killProcessGroup(child.pid, 'SIGKILL');
|
||||
if (!delivered) {
|
||||
console.error(
|
||||
`[isolation] job ${opts.jobId} (${opts.jobName}): group SIGKILL was NOT delivered ` +
|
||||
`to pid ${child.pid} (platform=${process.platform}; is /bin/kill present?). ` +
|
||||
`The child may still be running — the SIGKILL guarantee is degraded on this host.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}, graceMs);
|
||||
unrefTimer(killTimer);
|
||||
}
|
||||
};
|
||||
const onAbort = (): void => terminate();
|
||||
const onShutdown = (): void => terminate();
|
||||
if (opts.abortSignal.aborted) onAbort();
|
||||
else opts.abortSignal.addEventListener('abort', onAbort, { once: true });
|
||||
if (opts.shutdownSignal.aborted) onShutdown();
|
||||
else opts.shutdownSignal.addEventListener('abort', onShutdown, { once: true });
|
||||
|
||||
console.log(
|
||||
`[isolation] job ${opts.jobId} (${opts.jobName}) child pid ${child.pid ?? '?'} spawned`,
|
||||
);
|
||||
|
||||
try {
|
||||
const exit = await new Promise<ChildExit>((resolve) => {
|
||||
child.once('error', (e) => resolve({ code: null, signal: null, spawnErr: e }));
|
||||
child.once('exit', (code, signal) => resolve({ code, signal }));
|
||||
});
|
||||
|
||||
if (exit.spawnErr && child.pid == null) {
|
||||
throw new ChildSpawnInfraError(
|
||||
`job child spawn failed (${inv.cmd}): ${exit.spawnErr.message}`,
|
||||
);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`[isolation] job ${opts.jobId} (${opts.jobName}) child pid ${child.pid ?? '?'} ` +
|
||||
`exited code=${exit.code ?? 'null'} signal=${exit.signal ?? 'null'}`,
|
||||
);
|
||||
|
||||
const abortReason = opts.abortSignal.aborted
|
||||
? (opts.abortSignal.reason instanceof Error
|
||||
? opts.abortSignal.reason.message
|
||||
: String(opts.abortSignal.reason ?? 'aborted'))
|
||||
: null;
|
||||
// Shutdown classification wins UNLESS the per-job abort names a
|
||||
// job-targeted reason. gracefulShutdown('watchdog') aborts BOTH signals —
|
||||
// checking abortSignal first would shadow the no-burn shutdown release
|
||||
// and dead-letter innocent isolated jobs (adversarial-review P3).
|
||||
const isShutdownClass =
|
||||
opts.shutdownSignal.aborted &&
|
||||
(abortReason === null || !PER_JOB_ABORT_REASONS.has(abortReason));
|
||||
|
||||
let outcome: Awaited<ReturnType<typeof decodeChildOutcomeFileAsync>>;
|
||||
try {
|
||||
// Async decode: a large-but-allowed outcome must not block the worker
|
||||
// event loop that runs lock-renewal ticks (performance review).
|
||||
outcome = await decodeChildOutcomeFileAsync(resultPath);
|
||||
} catch (decodeErr) {
|
||||
// No usable outcome. Classify by WHY the child died.
|
||||
if (decodeErr instanceof UnrecoverableError) throw decodeErr; // oversize cap — dead on attempt 1
|
||||
if (isShutdownClass) {
|
||||
throw new ChildWorkerShutdownError(
|
||||
`job child terminated by worker shutdown before reporting (exit code=${exit.code} signal=${exit.signal})`,
|
||||
);
|
||||
}
|
||||
if (opts.abortSignal.aborted) {
|
||||
// executeJob's catch reads abort.signal.reason first, so infra
|
||||
// reasons (lock-renewal-failed / lock-lost) still burn no attempt
|
||||
// and timeout/cancel keep their existing semantics.
|
||||
throw new Error(
|
||||
`job child terminated after abort without an outcome (exit code=${exit.code} signal=${exit.signal})`,
|
||||
);
|
||||
}
|
||||
// Bootstrap failures carry reserved exit codes and are NOT handler
|
||||
// defects: 13 = usage/config (ops misconfiguration — release like a
|
||||
// spawn failure), 14 = job reclaimed before the handler ran (owned
|
||||
// elsewhere — release; the fenced failJob would no-op regardless).
|
||||
if (exit.code === JOB_CHILD_EXIT_USAGE) {
|
||||
throw new ChildSpawnInfraError(
|
||||
`job child bootstrap failed (exit ${exit.code}) — check the worker's child CLI/engine configuration`,
|
||||
);
|
||||
}
|
||||
if (exit.code === JOB_CHILD_EXIT_NOT_CLAIMED) {
|
||||
throw new ChildNotClaimedError(
|
||||
`job child found the claim gone (exit ${exit.code}) — reclaimed or cancelled before the handler ran`,
|
||||
);
|
||||
}
|
||||
throw new Error(
|
||||
`${decodeErr instanceof Error ? decodeErr.message : String(decodeErr)} ` +
|
||||
`(exit code=${exit.code} signal=${exit.signal})`,
|
||||
);
|
||||
}
|
||||
|
||||
if (outcome.outcome === 'success') return outcome.result;
|
||||
// A handler-error outcome DURING worker shutdown is presumed
|
||||
// shutdown-induced (cooperative handlers that honor shutdownSignal bail
|
||||
// and report an error): release with no attempt burned rather than
|
||||
// punishing exactly the well-behaved handlers on every deploy
|
||||
// (adversarial-review P2). Worst case a genuinely-failing job that
|
||||
// coincided with a deploy gets one free retry — bounded and benign.
|
||||
if (isShutdownClass) {
|
||||
throw new ChildWorkerShutdownError(
|
||||
`job child reported an error during worker shutdown (${outcome.message}) — released, not burned`,
|
||||
);
|
||||
}
|
||||
throw reconstructHandlerError(outcome);
|
||||
} finally {
|
||||
if (killTimer != null) clearTimeout(killTimer);
|
||||
opts.abortSignal.removeEventListener('abort', onAbort);
|
||||
opts.shutdownSignal.removeEventListener('abort', onShutdown);
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
/**
|
||||
* DB liveness probe with pool-starvation disambiguation (issue #6).
|
||||
*
|
||||
* The incident this exists for: a worker's read pool was exhausted by
|
||||
* checked-out-and-abandoned queries, the `SELECT 1` probe couldn't get a slot
|
||||
* within its budget, and the worker exited with "DB unreachable" — while the
|
||||
* server sat at ~10% of max_connections. Operators spent hours on the wrong
|
||||
* layer (evaluating an instance upgrade that would have changed nothing).
|
||||
*
|
||||
* Mechanism: probe the read pool; on failure, probe the DIRECT session lane
|
||||
* (when dual-pool is active). Direct success proves the server is reachable
|
||||
* and narrows the fault to the transaction-pooler path — client pool
|
||||
* exhaustion or a pooler-layer fault; the probe deliberately does NOT claim
|
||||
* to distinguish those two (codex-2 #2). Either way the operator is pointed
|
||||
* away from "the database is down / too small".
|
||||
*
|
||||
* Verdicts:
|
||||
* pool_starved — read probe failed, direct probe succeeded.
|
||||
* server_unreachable — read AND direct probes failed.
|
||||
* unknown — read probe failed, no direct lane to disambiguate.
|
||||
*
|
||||
* Both probes carry an AbortSignal that fires when their deadline wins, so a
|
||||
* hung probe is cancelled (slot released), never abandoned — same contract
|
||||
* as the lock-renewal tick.
|
||||
*
|
||||
* Pure/hermetic: every effect is injected via `DbProbeDeps`; the worker's
|
||||
* adapter is a thin closure. (lock-renewal-tick.ts is the pattern.)
|
||||
*/
|
||||
|
||||
import type { PoolGaugeSnapshot } from '../pool-gauge.ts';
|
||||
|
||||
export type ProbeVerdict = 'pool_starved' | 'server_unreachable' | 'unknown';
|
||||
|
||||
/** Default budget for the direct-lane disambiguation probe. */
|
||||
export const DIRECT_PROBE_TIMEOUT_MS = 3_000;
|
||||
|
||||
export interface PoolDiagnostics {
|
||||
/** Gauge counts — a tracked SUBSET (see pool-gauge.ts honesty contract). */
|
||||
tracked: PoolGaugeSnapshot;
|
||||
/** Read-pool max, when the engine can report it; null otherwise. */
|
||||
poolMax: number | null;
|
||||
}
|
||||
|
||||
export interface DbProbeDeps {
|
||||
/** SELECT 1 on the read pool; MUST honor the signal (cancellation). */
|
||||
probeRead: (signal: AbortSignal) => Promise<void>;
|
||||
/**
|
||||
* SELECT 1 on the direct session lane. Present ONLY when dual-pool is
|
||||
* genuinely active — an executeRawDirect that would silently fall back to
|
||||
* the read pool (kill-switch collapse) must NOT be passed here, or the
|
||||
* "disambiguation" would probe the same starved pool twice.
|
||||
*/
|
||||
probeDirect?: (signal: AbortSignal) => Promise<void>;
|
||||
/** Optional gauge snapshot for supporting detail. Fail-open: may be absent or throw. */
|
||||
getDiagnostics?: () => PoolDiagnostics | null;
|
||||
/** Read-probe budget (worker default 10s). */
|
||||
timeoutMs: number;
|
||||
/** Direct-probe budget (default 3s). */
|
||||
directTimeoutMs: number;
|
||||
}
|
||||
|
||||
export type DbProbeResult =
|
||||
| { ok: true }
|
||||
| { ok: false; verdict: ProbeVerdict; detail: string };
|
||||
|
||||
/**
|
||||
* Narrow, shared view of the engine's ConnectionManager for routing-aware
|
||||
* callers (the worker's probe adapter, jobs.ts's single-pool startup
|
||||
* warning). One typed accessor instead of hand-rolled structural casts that
|
||||
* drift independently from the real class (maintainability review).
|
||||
*/
|
||||
export interface EngineConnectionRouting {
|
||||
isDualPoolActive?: () => boolean;
|
||||
describeMode?: () => { kill_switch_active?: boolean; direct_pool_size?: number };
|
||||
}
|
||||
|
||||
export function getConnectionRouting(engine: unknown): EngineConnectionRouting | null {
|
||||
const cm = (engine as { connectionManager?: EngineConnectionRouting }).connectionManager;
|
||||
return cm ?? null;
|
||||
}
|
||||
|
||||
async function withDeadline(
|
||||
run: (signal: AbortSignal) => Promise<void>,
|
||||
ms: number,
|
||||
label: string,
|
||||
): Promise<void> {
|
||||
const ac = new AbortController();
|
||||
let timer: ReturnType<typeof setTimeout> | null = null;
|
||||
try {
|
||||
await Promise.race([
|
||||
run(ac.signal),
|
||||
new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
ac.abort();
|
||||
reject(new Error(`${label} timeout after ${ms}ms`));
|
||||
}, ms);
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
if (timer != null) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/** Render gauge detail. Never throws; empty string when unavailable. */
|
||||
function renderDiagnostics(getDiagnostics?: () => PoolDiagnostics | null): string {
|
||||
try {
|
||||
const diag = getDiagnostics?.();
|
||||
if (!diag) return '';
|
||||
const { tracked, poolMax } = diag;
|
||||
const maxNote = poolMax != null ? ` (read pool max ${poolMax})` : '';
|
||||
const base =
|
||||
` gbrain-tracked in flight (subset — template-path queries untracked):` +
|
||||
` raw=${tracked.raw}, direct=${tracked.direct}, reserved=${tracked.reserved}, tx=${tracked.tx}${maxNote}.`;
|
||||
const total = tracked.raw + tracked.direct + tracked.reserved + tracked.tx;
|
||||
if (total === 0) {
|
||||
return (
|
||||
base +
|
||||
' Tracked subset shows 0 in flight — the saturation is in untracked' +
|
||||
' template-query traffic; see docs/guides/queue-operations-runbook.md.'
|
||||
);
|
||||
}
|
||||
return base;
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
export async function runDbProbe(deps: DbProbeDeps): Promise<DbProbeResult> {
|
||||
let readErrMsg: string;
|
||||
try {
|
||||
await withDeadline(deps.probeRead, deps.timeoutMs, 'probe');
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
readErrMsg = e instanceof Error ? e.message : String(e);
|
||||
}
|
||||
|
||||
if (!deps.probeDirect) {
|
||||
return {
|
||||
ok: false,
|
||||
verdict: 'unknown',
|
||||
detail: `${readErrMsg}; no direct lane available to disambiguate pool starvation from a dead server`,
|
||||
};
|
||||
}
|
||||
|
||||
const t0 = Date.now();
|
||||
try {
|
||||
await withDeadline(deps.probeDirect, deps.directTimeoutMs, 'direct probe');
|
||||
return {
|
||||
ok: false,
|
||||
verdict: 'pool_starved',
|
||||
detail:
|
||||
`read-pool probe failed (${readErrMsg}) but the direct-lane probe succeeded in ${Date.now() - t0}ms — ` +
|
||||
`the server IS reachable; the fault is in the transaction-pooler path ` +
|
||||
`(client pool exhaustion or a pooler-layer fault).` +
|
||||
renderDiagnostics(deps.getDiagnostics),
|
||||
};
|
||||
} catch (e) {
|
||||
const directErrMsg = e instanceof Error ? e.message : String(e);
|
||||
return {
|
||||
ok: false,
|
||||
verdict: 'server_unreachable',
|
||||
detail: `read probe: ${readErrMsg}; direct probe: ${directErrMsg} — server/network unreachable`,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Shared MinionJobContext builder (issue #5 — per-job process isolation).
|
||||
*
|
||||
* Extracted verbatim from MinionWorker.executeJob so the same DB-backed
|
||||
* context wiring serves BOTH execution modes:
|
||||
*
|
||||
* inline — the worker builds it against its own engine/queue;
|
||||
* process — `gbrain jobs run-child` builds it against the CHILD's engine
|
||||
* (child-owns-engine design: every write below is token-fenced,
|
||||
* so a reclaimed job's orphan child degrades to a no-op writer).
|
||||
*
|
||||
* Behavioral no-op for the inline path.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { MinionQueue } from './queue.ts';
|
||||
import type { MinionJob, MinionJobContext, TokenUpdate } from './types.ts';
|
||||
|
||||
export function buildJobContext(
|
||||
engine: BrainEngine,
|
||||
queue: MinionQueue,
|
||||
job: MinionJob,
|
||||
lockToken: string,
|
||||
signal: AbortSignal,
|
||||
shutdownSignal: AbortSignal,
|
||||
): MinionJobContext {
|
||||
return {
|
||||
id: job.id,
|
||||
name: job.name,
|
||||
data: job.data,
|
||||
attempts_made: job.attempts_made,
|
||||
signal,
|
||||
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
|
||||
shutdownSignal,
|
||||
updateProgress: async (progress: unknown) => {
|
||||
await queue.updateProgress(job.id, lockToken, progress);
|
||||
},
|
||||
updateTokens: async (tokens: TokenUpdate) => {
|
||||
await queue.updateTokens(job.id, lockToken, tokens);
|
||||
},
|
||||
log: async (message: string | Record<string, unknown>) => {
|
||||
const value = typeof message === 'string' ? message : JSON.stringify(message);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
|
||||
updated_at = now()
|
||||
WHERE id = $2 AND status = 'active' AND lock_token = $3`,
|
||||
[value, job.id, lockToken]
|
||||
);
|
||||
},
|
||||
isActive: async () => {
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM minion_jobs WHERE id = $1 AND status = 'active' AND lock_token = $2`,
|
||||
[job.id, lockToken]
|
||||
);
|
||||
return rows.length > 0;
|
||||
},
|
||||
readInbox: async () => {
|
||||
return queue.readInbox(job.id, lockToken);
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
/**
|
||||
* Per-job process isolation protocol (issue #5).
|
||||
*
|
||||
* The worker (parent) keeps claim / lock-renewal / completeJob / failJob;
|
||||
* `gbrain jobs run-child` (child) owns handler execution with its own small
|
||||
* engine pool. This module is the protocol between them:
|
||||
*
|
||||
* payload in — job id via argv (`jobs run-child --job-id N`, ps-visible
|
||||
* for ops); lock token via GBRAIN_JOB_LOCK_TOKEN env (off
|
||||
* argv — not a secret, it's a fencing token, but no reason
|
||||
* to put it in `ps` output); result path via
|
||||
* GBRAIN_JOB_RESULT_PATH.
|
||||
* result out — ONE JSON file, written atomically (tmp + rename), decoded
|
||||
* by the parent. stdout/stderr stay inherited for handler
|
||||
* logs (handlers print freely — no sentinel parsing), and
|
||||
* node-IPC is deliberately avoided (zero precedent in this
|
||||
* codebase; fd inheritance through a tini wrapper is
|
||||
* unproven here).
|
||||
* termination — killProcessGroup(): children are spawned detached (own
|
||||
* process group) because SIGKILL on the tini pid alone kills
|
||||
* tini, NOT the handler grandchild (tini cannot forward
|
||||
* SIGKILL). Bun rejects negative pids in process.kill()
|
||||
* (oven-sh/bun#15791) and gbrain ships as a Bun-compiled
|
||||
* binary, so the group signal falls back to POSIX
|
||||
* /bin/kill when needed.
|
||||
*
|
||||
* Handler-error semantics survive the boundary: the child encodes the two
|
||||
* error classes executeJob branches on (UnrecoverableError → 'dead',
|
||||
* RateLeaseUnavailableError → lease release, no attempt burned) and
|
||||
* `reconstructHandlerError` rebuilds real instances parent-side so the
|
||||
* existing `instanceof` branches work verbatim. Everything else degrades to
|
||||
* a generic Error → the normal delayed/dead backoff path, same as inline.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, renameSync, statSync } from 'node:fs';
|
||||
import { readFile, stat } from 'node:fs/promises';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { UnrecoverableError } from './types.ts';
|
||||
import { RateLeaseUnavailableError } from './handlers/subagent.ts';
|
||||
|
||||
/** Grace between group-SIGTERM and group-SIGKILL on abort. Deliberately
|
||||
* inside the worker's 30s force-evict window so the evict path stays a
|
||||
* nearly-unreachable backstop. */
|
||||
export const CHILD_KILL_GRACE_MS = 25_000;
|
||||
|
||||
/** Decode cap for the child's outcome file. Results already round-trip
|
||||
* through the completeJob JSONB column in inline mode, so anything near
|
||||
* this cap is pathological; oversize throws UnrecoverableError (loud dead
|
||||
* on attempt 1 — deterministic failure, retries would fail identically). */
|
||||
export const CHILD_OUTCOME_MAX_BYTES = 32 * 1024 * 1024;
|
||||
|
||||
/** Default read-pool cap for isolation children. Referenced by the
|
||||
* --job-isolation help copy ("~4 pooler client connections" = this + the
|
||||
* direct pool of 1) and the minions-deployment.md budget math. */
|
||||
export const CHILD_READ_POOL_MAX = 3;
|
||||
|
||||
/** Bun-compat timer unref (plain cast copy-pasted thrice before this helper). */
|
||||
export function unrefTimer(t: unknown): void {
|
||||
(t as { unref?: () => void }).unref?.();
|
||||
}
|
||||
|
||||
/** Env vars of the parent↔child contract. Spelled once here. */
|
||||
export const CHILD_ENV = {
|
||||
lockToken: 'GBRAIN_JOB_LOCK_TOKEN',
|
||||
resultPath: 'GBRAIN_JOB_RESULT_PATH',
|
||||
isChild: 'GBRAIN_JOB_CHILD',
|
||||
parentPid: 'GBRAIN_JOB_PARENT_PID',
|
||||
childCliOverride: 'GBRAIN_JOB_CHILD_CLI',
|
||||
childPoolSize: 'GBRAIN_JOB_CHILD_POOL_SIZE',
|
||||
} as const;
|
||||
|
||||
export type ChildErrorKind = 'unrecoverable' | 'rate_lease' | 'generic';
|
||||
|
||||
export type ChildOutcome =
|
||||
| { outcome: 'success'; result: unknown }
|
||||
| {
|
||||
outcome: 'error';
|
||||
errorKind: ChildErrorKind;
|
||||
message: string;
|
||||
stack?: string;
|
||||
lease?: { key: string; active: number; max: number };
|
||||
};
|
||||
|
||||
/** Child-side: classify a handler throw into the wire shape. */
|
||||
export function encodeHandlerError(err: unknown): ChildOutcome {
|
||||
if (err instanceof RateLeaseUnavailableError) {
|
||||
return {
|
||||
outcome: 'error',
|
||||
errorKind: 'rate_lease',
|
||||
message: err.message,
|
||||
lease: { key: err.key, active: err.active, max: err.max },
|
||||
};
|
||||
}
|
||||
if (err instanceof UnrecoverableError) {
|
||||
return {
|
||||
outcome: 'error',
|
||||
errorKind: 'unrecoverable',
|
||||
message: err.message,
|
||||
...(err.stack ? { stack: err.stack } : {}),
|
||||
};
|
||||
}
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
const stack = err instanceof Error ? err.stack : undefined;
|
||||
return { outcome: 'error', errorKind: 'generic', message, ...(stack ? { stack } : {}) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Parent-side: rebuild a real error instance so executeJob's existing
|
||||
* `instanceof` branches (dead / lease-release / delayed+backoff) work
|
||||
* verbatim. Unknown errorKind values degrade to generic (whitelist — the
|
||||
* file is same-user-written but a malformed kind must not crash the worker).
|
||||
*/
|
||||
export function reconstructHandlerError(o: Extract<ChildOutcome, { outcome: 'error' }>): Error {
|
||||
if (o.errorKind === 'rate_lease' && o.lease) {
|
||||
return new RateLeaseUnavailableError(o.lease.key, o.lease.active, o.lease.max);
|
||||
}
|
||||
if (o.errorKind === 'unrecoverable') {
|
||||
return new UnrecoverableError(o.message);
|
||||
}
|
||||
const err = new Error(o.message);
|
||||
if (o.stack) {
|
||||
(err as Error & { childStack?: string }).childStack = o.stack;
|
||||
}
|
||||
return err;
|
||||
}
|
||||
|
||||
/** Child-side: atomic outcome write (tmp + rename on the same filesystem). */
|
||||
export function writeChildOutcomeFile(path: string, outcome: ChildOutcome): void {
|
||||
const tmp = `${path}.tmp`;
|
||||
writeFileSync(tmp, JSON.stringify(outcome), 'utf8');
|
||||
renameSync(tmp, path);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure outcome parser (shared by the sync and async decode paths). Throws:
|
||||
* - UnrecoverableError when the file exceeds `maxBytes` (deterministic —
|
||||
* dead on attempt 1, no silent truncation);
|
||||
* - generic Error for malformed/unrecognized content (byte count only in
|
||||
* the message, NEVER file content — handler output may carry secrets).
|
||||
* The `lease` payload is shape-validated (security review): a corrupt file
|
||||
* must degrade to 'generic', not inject undefined fields into the parent's
|
||||
* lease-release accounting.
|
||||
*/
|
||||
export function parseChildOutcome(raw: string, size: number, maxBytes = CHILD_OUTCOME_MAX_BYTES): ChildOutcome {
|
||||
if (size > maxBytes) {
|
||||
throw new UnrecoverableError(
|
||||
`job child result exceeds the ${Math.floor(maxBytes / (1024 * 1024))}MiB outcome cap (${size} bytes); ` +
|
||||
`retries would fail identically — return a smaller result or persist large artifacts elsewhere`,
|
||||
);
|
||||
}
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
throw new Error(`job child outcome file is not valid JSON (${size} bytes)`);
|
||||
}
|
||||
const o = parsed as Partial<ChildOutcome> | null;
|
||||
if (o && o.outcome === 'success') return { outcome: 'success', result: (o as { result?: unknown }).result };
|
||||
if (o && o.outcome === 'error' && typeof (o as { message?: unknown }).message === 'string') {
|
||||
const kind = (o as { errorKind?: unknown }).errorKind;
|
||||
const rawLease = (o as { lease?: unknown }).lease as
|
||||
| { key?: unknown; active?: unknown; max?: unknown }
|
||||
| undefined;
|
||||
const leaseValid =
|
||||
rawLease != null &&
|
||||
typeof rawLease.key === 'string' &&
|
||||
Number.isFinite(rawLease.active as number) &&
|
||||
Number.isFinite(rawLease.max as number);
|
||||
// rate_lease without a valid lease payload degrades to generic — same
|
||||
// policy as the errorKind whitelist.
|
||||
const errorKind =
|
||||
kind === 'unrecoverable' ? 'unrecoverable'
|
||||
: kind === 'rate_lease' && leaseValid ? 'rate_lease'
|
||||
: 'generic';
|
||||
return {
|
||||
outcome: 'error',
|
||||
errorKind,
|
||||
message: (o as { message: string }).message,
|
||||
...((o as { stack?: unknown }).stack && typeof (o as { stack?: unknown }).stack === 'string'
|
||||
? { stack: (o as { stack: string }).stack }
|
||||
: {}),
|
||||
...(errorKind === 'rate_lease' && leaseValid
|
||||
? { lease: { key: rawLease.key as string, active: rawLease.active as number, max: rawLease.max as number } }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
throw new Error(`job child outcome file has an unrecognized shape (${size} bytes)`);
|
||||
}
|
||||
|
||||
const MISSING_OUTCOME_MESSAGE =
|
||||
'job child exited without writing its outcome file (crash, OOM, or kill before completion)';
|
||||
|
||||
/** Sync decode (tests + non-hot-path callers). */
|
||||
export function decodeChildOutcomeFile(path: string, maxBytes = CHILD_OUTCOME_MAX_BYTES): ChildOutcome {
|
||||
let size: number;
|
||||
try {
|
||||
size = statSync(path).size;
|
||||
} catch {
|
||||
throw new Error(MISSING_OUTCOME_MESSAGE);
|
||||
}
|
||||
if (size > maxBytes) return parseChildOutcome('', size, maxBytes); // throws the cap error
|
||||
return parseChildOutcome(readFileSync(path, 'utf8'), size, maxBytes);
|
||||
}
|
||||
|
||||
/**
|
||||
* Async decode for the WORKER's per-job path: a large-but-allowed outcome
|
||||
* (up to 32MiB) must not block the event loop that runs lock-renewal ticks
|
||||
* and the health-probe chain (performance review).
|
||||
*/
|
||||
export async function decodeChildOutcomeFileAsync(
|
||||
path: string,
|
||||
maxBytes = CHILD_OUTCOME_MAX_BYTES,
|
||||
): Promise<ChildOutcome> {
|
||||
let size: number;
|
||||
try {
|
||||
size = (await stat(path)).size;
|
||||
} catch {
|
||||
throw new Error(MISSING_OUTCOME_MESSAGE);
|
||||
}
|
||||
if (size > maxBytes) return parseChildOutcome('', size, maxBytes); // throws the cap error
|
||||
return parseChildOutcome(await readFile(path, 'utf8'), size, maxBytes);
|
||||
}
|
||||
|
||||
/** argv for the child invocation (appended after the resolved CLI). */
|
||||
export function buildChildArgs(jobId: number): string[] {
|
||||
return ['jobs', 'run-child', '--job-id', String(jobId)];
|
||||
}
|
||||
|
||||
export interface ChildCliInvocation {
|
||||
cmd: string;
|
||||
/** Args that come BEFORE buildChildArgs() output (e.g. the cli.ts path in bun-dev). */
|
||||
argsPrefix: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve how to invoke the gbrain CLI for a child process. Pure — all
|
||||
* inputs injected:
|
||||
*
|
||||
* 1. GBRAIN_JOB_CHILD_CLI env override (ops/test escape hatch)
|
||||
* 2. resolveBinary() — the compiled-binary resolver
|
||||
* (resolveGbrainCliPath; never returns a .ts path)
|
||||
* 3. bun-dev fallback: running from `bun src/cli.ts` → invoke
|
||||
* `<execPath> <argv1>` so dev and tests work without a compiled binary
|
||||
*
|
||||
* Returns null when nothing resolves — the caller must fail fast at worker
|
||||
* startup (one bad path must not dead-letter a queue job-by-job).
|
||||
*/
|
||||
export function resolveChildCliInvocation(
|
||||
env: Record<string, string | undefined>,
|
||||
execPath: string,
|
||||
argv1: string | undefined,
|
||||
resolveBinary: () => string | null,
|
||||
): ChildCliInvocation | null {
|
||||
const override = env[CHILD_ENV.childCliOverride];
|
||||
if (override && override.trim() !== '') {
|
||||
return { cmd: override, argsPrefix: [] };
|
||||
}
|
||||
try {
|
||||
const bin = resolveBinary();
|
||||
if (bin) return { cmd: bin, argsPrefix: [] };
|
||||
} catch {
|
||||
// fall through to the dev fallback
|
||||
}
|
||||
if (argv1 && (argv1.endsWith('/cli.ts') || argv1.endsWith('\\cli.ts') || argv1 === 'cli.ts')) {
|
||||
return { cmd: execPath, argsPrefix: [argv1] };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Signal an entire process GROUP.
|
||||
*
|
||||
* Children are spawned `detached: true` (own group) so this reaches the
|
||||
* handler grandchildren even under a tini wrapper — SIGKILL on the tini pid
|
||||
* alone kills tini and ORPHANS the still-running handler (tini cannot
|
||||
* forward SIGKILL; that failure mode would silently void issue #5's
|
||||
* headline guarantee exactly in container deployments).
|
||||
*
|
||||
* Bun's process.kill() rejects negative pids (oven-sh/bun#15791), so on any
|
||||
* throw other than ESRCH we fall back to POSIX /bin/kill, which
|
||||
* group-signals fine on darwin + linux. Returns true when the signal was
|
||||
* delivered to a live group; false when the group is already gone (ESRCH —
|
||||
* success for our purposes) or delivery failed.
|
||||
*/
|
||||
export function killProcessGroup(pid: number, signal: 'SIGTERM' | 'SIGKILL'): boolean {
|
||||
if (!Number.isInteger(pid) || pid <= 1) return false;
|
||||
try {
|
||||
process.kill(-pid, signal);
|
||||
return true;
|
||||
} catch (e) {
|
||||
const code = (e as NodeJS.ErrnoException).code;
|
||||
if (code === 'ESRCH') return false; // group already gone
|
||||
// RangeError on Bun (negative pid unsupported) or EPERM etc. — fall back
|
||||
// to /bin/kill by ABSOLUTE path (a PATH-resolved binary in a kill path is
|
||||
// an unnecessary indirection; security review). This is the NORMAL path
|
||||
// in Bun-compiled production binaries; the sync exec is ~1-3ms and only
|
||||
// runs on abort/shutdown, never in the claim/renewal hot loop.
|
||||
try {
|
||||
const sigName = signal.replace(/^SIG/, '');
|
||||
const res = spawnSync('/bin/kill', ['-s', sigName, '--', `-${pid}`], { stdio: 'ignore' });
|
||||
return res.status === 0;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -147,7 +147,19 @@ function warnAndFallback(name: string, raw: string, fallback: number): number {
|
||||
* `runLockRenewalTick` is pure and trivially testable.
|
||||
*/
|
||||
export interface LockRenewalDeps {
|
||||
renewLock: (jobId: number, lockToken: string, lockDurationMs: number) => Promise<boolean>;
|
||||
/**
|
||||
* The optional `opts.signal` is aborted when this call loses the tick's
|
||||
* timeout race, so the underlying UPDATE is CANCELLED (postgres.js
|
||||
* `.cancel()` via executeRawDirect) instead of orphaned on a checked-out
|
||||
* pool slot for its full server-side duration — the #6 starvation class.
|
||||
* Optional-param widening keeps the legacy 3-arg test mocks compiling.
|
||||
*/
|
||||
renewLock: (
|
||||
jobId: number,
|
||||
lockToken: string,
|
||||
lockDurationMs: number,
|
||||
opts?: { signal?: AbortSignal },
|
||||
) => Promise<boolean>;
|
||||
audit: LockRenewalAuditSinkLike;
|
||||
/** Injectable for hermetic time-based tests. Production: `Date.now`. */
|
||||
now: () => number;
|
||||
@@ -155,8 +167,9 @@ export interface LockRenewalDeps {
|
||||
* Injectable for hermetic Promise.race tests. Production:
|
||||
* `globalThis.setTimeout`. The function must return a value that
|
||||
* `clearTimeout` accepts, but this seam doesn't expose clearTimeout
|
||||
* because the timeout race fires-and-forgets (the lost race is
|
||||
* harmless — at worst we have a dangling reject that no one awaits).
|
||||
* because the timeout race fires-and-forgets. The losing renewLock is no
|
||||
* longer merely abandoned: the timeout callback also aborts the per-call
|
||||
* signal so the query releases its pool slot.
|
||||
*/
|
||||
setTimeout: (cb: () => void, ms: number) => unknown;
|
||||
/**
|
||||
@@ -228,11 +241,19 @@ export async function runLockRenewalTick(
|
||||
if (state.cancelled()) return { kind: 'cancelled' };
|
||||
|
||||
let renewed: boolean;
|
||||
// Per-call cancellation: when the timeout wins the race, abort the signal
|
||||
// so the losing UPDATE releases its pool slot instead of holding it until
|
||||
// the server finishes (issue #6 — an abandoned racer under a saturated
|
||||
// pooler pinned a checked-out connection for minutes). On the win path the
|
||||
// late-firing timer aborts an already-settled query, which runUnsafe
|
||||
// ignores (abort listener removed in its .finally).
|
||||
const callAbort = new AbortController();
|
||||
try {
|
||||
renewed = await Promise.race([
|
||||
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs),
|
||||
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs, { signal: callAbort.signal }),
|
||||
new Promise<never>((_, reject) => {
|
||||
deps.setTimeout(() => {
|
||||
callAbort.abort();
|
||||
reject(new Error(`renewLock timed out after ${state.knobs.callTimeoutMs}ms`));
|
||||
}, state.knobs.callTimeoutMs);
|
||||
}),
|
||||
|
||||
@@ -1291,8 +1291,18 @@ export class MinionQueue {
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
/** Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed). */
|
||||
async renewLock(id: number, lockToken: string, lockDurationMs: number): Promise<boolean> {
|
||||
/**
|
||||
* Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed).
|
||||
* `opts.signal` cancels the in-flight UPDATE (postgres.js `.cancel()`) when the
|
||||
* caller's timeout race gives up on it — otherwise the abandoned query holds a
|
||||
* checked-out pool slot for its full server-side duration (issue #6).
|
||||
*/
|
||||
async renewLock(
|
||||
id: number,
|
||||
lockToken: string,
|
||||
lockDurationMs: number,
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<boolean> {
|
||||
// Direct (session-mode) pool — see claim(). The heartbeat that keeps a job
|
||||
// alive for minutes cannot run on the transaction pooler without periodic
|
||||
// CONNECTION_ENDED drops that look like lock-expiry and orphan the job.
|
||||
@@ -1300,7 +1310,8 @@ export class MinionQueue {
|
||||
`UPDATE minion_jobs SET lock_until = now() + ($1::double precision * interval '1 millisecond'), updated_at = now()
|
||||
WHERE id = $2 AND lock_token = $3 AND status = 'active'
|
||||
RETURNING id`,
|
||||
[lockDurationMs, id, lockToken]
|
||||
[lockDurationMs, id, lockToken],
|
||||
opts
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* `gbrain jobs run-child` core (issue #5 — per-job process isolation).
|
||||
*
|
||||
* The child side of the isolation boundary. The PARENT worker owns claim,
|
||||
* lock renewal, completeJob/failJob and all attempt accounting; this process
|
||||
* only executes the handler and reports ONE outcome file (see
|
||||
* job-isolation.ts for the protocol). Child-owns-engine: every ctx callback
|
||||
* below is token-fenced, so if the job is reclaimed while we run, our writes
|
||||
* degrade to no-ops.
|
||||
*
|
||||
* Deliberately runs NONE of the worker machinery: no health probe, no stall
|
||||
* detection, no lock timer — the parent is the sole liveness owner. What it
|
||||
* does install:
|
||||
*
|
||||
* - SIGTERM handler → fires shutdownSignal ONLY (inline signal-separation
|
||||
* parity): cooperative handlers keep ctx.signal live, finish inside the
|
||||
* drain window, and write their outcome before the parent escalates to a
|
||||
* group SIGKILL. Handlers that watch shutdownSignal (shell) run their own
|
||||
* SIGTERM→grace→SIGKILL cleanup.
|
||||
* - Parent-liveness watchdog: polls `process.kill(parentPid, 0)` every 15s
|
||||
* (a ppid check is DEAD CODE under tini — the child's ppid is tini,
|
||||
* which outlives the worker). On parent death: abort both signals, and
|
||||
* hard-exit after a 30s grace so an orphaned LLM-bound handler doesn't
|
||||
* burn spend to completion. Lock expiry + the stall sweeper requeue the
|
||||
* job on the parent's side of the world.
|
||||
*
|
||||
* The CLI layer (jobs.ts case 'run-child') owns engine.disconnect() and
|
||||
* process.exit() — this module returns an exit code (engine-ownership
|
||||
* invariant, same as MinionWorker).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { MinionQueue } from './queue.ts';
|
||||
import type { MinionHandler } from './types.ts';
|
||||
import { buildJobContext } from './job-context.ts';
|
||||
import {
|
||||
JOB_CHILD_EXIT_NOT_CLAIMED,
|
||||
JOB_CHILD_EXIT_RESULT_WRITE_FAILED,
|
||||
} from './worker-exit-codes.ts';
|
||||
import { encodeHandlerError, unrefTimer, writeChildOutcomeFile } from './job-isolation.ts';
|
||||
|
||||
export interface RunChildOpts {
|
||||
jobId: number;
|
||||
lockToken: string;
|
||||
resultPath: string;
|
||||
/** Worker pid for the liveness watchdog; 0/absent disables the watchdog. */
|
||||
parentPid: number;
|
||||
}
|
||||
|
||||
export interface RunChildInjectables {
|
||||
/** Hermetic tests inject a handler map instead of registerBuiltinHandlers. */
|
||||
resolveHandler: (name: string) => MinionHandler | undefined;
|
||||
/** Watchdog cadence override (default 15s). */
|
||||
parentPollMs?: number;
|
||||
/** Orphan hard-exit grace override (default 30s). */
|
||||
orphanGraceMs?: number;
|
||||
/** Exit hook for tests (default process.exit for the orphan path only). */
|
||||
hardExit?: (code: number) => void;
|
||||
}
|
||||
|
||||
export async function runChildJobEntry(
|
||||
engine: BrainEngine,
|
||||
opts: RunChildOpts,
|
||||
injectables: RunChildInjectables,
|
||||
): Promise<number> {
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
// Ground truth is the DB row, re-read here (one SELECT) rather than a
|
||||
// serialized payload: the parent claimed it, but reclaim/cancel can race
|
||||
// our startup. A mismatch means we must not run the handler at all.
|
||||
const job = await queue.getJob(opts.jobId);
|
||||
if (!job || job.status !== 'active' || job.lock_token !== opts.lockToken) {
|
||||
process.stderr.write(
|
||||
`[run-child] job ${opts.jobId} is not claimed by this token ` +
|
||||
`(status=${job?.status ?? 'missing'}) — exiting without running the handler\n`,
|
||||
);
|
||||
return JOB_CHILD_EXIT_NOT_CLAIMED;
|
||||
}
|
||||
|
||||
const handler = injectables.resolveHandler(job.name);
|
||||
if (!handler) {
|
||||
// Parity with inline mode, which dead-letters a missing handler
|
||||
// immediately (failJob 'dead') — 'unrecoverable' reconstructs to
|
||||
// UnrecoverableError parent-side, so the parent dead-letters on attempt 1
|
||||
// instead of retrying a deterministic condition (adversarial-review P3).
|
||||
try {
|
||||
writeChildOutcomeFile(opts.resultPath, {
|
||||
outcome: 'error',
|
||||
errorKind: 'unrecoverable',
|
||||
message: `No handler for job type '${job.name}' in the isolation child`,
|
||||
});
|
||||
return 0;
|
||||
} catch {
|
||||
return JOB_CHILD_EXIT_RESULT_WRITE_FAILED;
|
||||
}
|
||||
}
|
||||
|
||||
const abort = new AbortController();
|
||||
const shutdown = new AbortController();
|
||||
|
||||
// SIGTERM = worker shutdown: fire ONLY shutdownSignal, preserving the
|
||||
// inline signal-separation contract (worker.ts aborts only shutdownAbort on
|
||||
// SIGTERM; per-job ctx.signal stays live so cooperative handlers finish +
|
||||
// report inside the drain window instead of aborting mid-deploy —
|
||||
// adversarial-review P2). The parent's group SIGKILL at drain end is the
|
||||
// backstop for handlers that keep running.
|
||||
const onSigterm = (): void => {
|
||||
if (!shutdown.signal.aborted) shutdown.abort(new Error('worker-shutdown'));
|
||||
};
|
||||
// Parent death (orphan) aborts BOTH: nobody will SIGKILL us, the lock will
|
||||
// expire and the job will be requeued — stop the handler outright.
|
||||
const onOrphaned = (): void => {
|
||||
onSigterm();
|
||||
if (!abort.signal.aborted) abort.abort(new Error('worker-shutdown'));
|
||||
};
|
||||
process.on('SIGTERM', onSigterm);
|
||||
|
||||
// Parent-liveness watchdog (unref'd — never keeps the child alive).
|
||||
const pollMs = injectables.parentPollMs ?? 15_000;
|
||||
const graceMs = injectables.orphanGraceMs ?? 30_000;
|
||||
const hardExit = injectables.hardExit ?? ((code: number) => process.exit(code));
|
||||
let watchdog: ReturnType<typeof setInterval> | null = null;
|
||||
if (opts.parentPid > 0) {
|
||||
watchdog = setInterval(() => {
|
||||
try {
|
||||
process.kill(opts.parentPid, 0);
|
||||
} catch {
|
||||
process.stderr.write(
|
||||
`[run-child] parent worker (pid ${opts.parentPid}) is gone — aborting handler; ` +
|
||||
`hard exit in ${Math.round(graceMs / 1000)}s\n`,
|
||||
);
|
||||
if (watchdog != null) clearInterval(watchdog);
|
||||
onOrphaned();
|
||||
const t = setTimeout(() => hardExit(1), graceMs);
|
||||
unrefTimer(t);
|
||||
}
|
||||
}, pollMs);
|
||||
unrefTimer(watchdog);
|
||||
}
|
||||
|
||||
try {
|
||||
const context = buildJobContext(
|
||||
engine,
|
||||
queue,
|
||||
job,
|
||||
opts.lockToken,
|
||||
abort.signal,
|
||||
shutdown.signal,
|
||||
);
|
||||
let outcome;
|
||||
try {
|
||||
const result = await handler(context);
|
||||
// completeJob's {value: x} wrap decision must run BEFORE JSON
|
||||
// serialization: a JSON round-trip changes typeof for Date /
|
||||
// toJSON-bearing results (object → string), which would flip the wrap
|
||||
// parent-side (adversarial-review P3, result-shape parity). Wrap here;
|
||||
// the parent's own wrap is then a no-op (object/undefined passthrough).
|
||||
const wrapped = result != null
|
||||
? (typeof result === 'object' ? (result as Record<string, unknown>) : { value: result })
|
||||
: undefined;
|
||||
outcome = { outcome: 'success' as const, result: wrapped };
|
||||
} catch (err) {
|
||||
outcome = encodeHandlerError(err);
|
||||
}
|
||||
try {
|
||||
writeChildOutcomeFile(opts.resultPath, outcome);
|
||||
} catch (writeErr) {
|
||||
const msg = writeErr instanceof Error ? writeErr.message : String(writeErr);
|
||||
process.stderr.write(`[run-child] failed to write outcome file: ${msg}\n`);
|
||||
return JOB_CHILD_EXIT_RESULT_WRITE_FAILED;
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
if (watchdog != null) clearInterval(watchdog);
|
||||
process.off('SIGTERM', onSigterm);
|
||||
}
|
||||
}
|
||||
@@ -98,6 +98,10 @@ export interface SupervisorOpts {
|
||||
nice_requested?: number;
|
||||
/** Effective niceness of the supervisor process after its own renice attempt. */
|
||||
nice_effective?: number;
|
||||
/** issue #5: when 'process', the spawned worker runs each claimed job in a
|
||||
* SIGKILL-able child process (passed through as `--job-isolation process`).
|
||||
* Omitted/inline: today's shared-process execution. */
|
||||
jobIsolation?: 'inline' | 'process';
|
||||
/** Error string if the supervisor's own renice failed (e.g. EPERM). */
|
||||
nice_error?: string;
|
||||
/**
|
||||
@@ -174,7 +178,7 @@ const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
|
||||
* niceness also inherits to the worker's own children automatically.
|
||||
*/
|
||||
export function buildWorkerArgs(
|
||||
opts: Pick<SupervisorOpts, 'concurrency' | 'queue' | 'maxRssMb' | 'nice_requested'>,
|
||||
opts: Pick<SupervisorOpts, 'concurrency' | 'queue' | 'maxRssMb' | 'nice_requested' | 'jobIsolation'>,
|
||||
): string[] {
|
||||
const args = [
|
||||
'jobs', 'work',
|
||||
@@ -187,6 +191,11 @@ export function buildWorkerArgs(
|
||||
if (opts.nice_requested !== undefined) {
|
||||
args.push('--nice', String(opts.nice_requested));
|
||||
}
|
||||
// Conditional push (issue #5): omitted for inline so existing deployments'
|
||||
// argv is byte-identical (pinned by supervisor-build-worker-args.test.ts).
|
||||
if (opts.jobIsolation === 'process') {
|
||||
args.push('--job-isolation', 'process');
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
@@ -250,6 +259,7 @@ export async function queryWedgeSignals(
|
||||
engine: BrainEngine,
|
||||
queue: string,
|
||||
handlerNames: string[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<WedgeSignals> {
|
||||
const rows = await engine.executeRaw<{
|
||||
stalled: string;
|
||||
@@ -272,6 +282,7 @@ export async function queryWedgeSignals(
|
||||
FROM minion_jobs
|
||||
WHERE queue = $1`,
|
||||
[queue, handlerNames],
|
||||
opts,
|
||||
);
|
||||
const row = rows[0] ?? {
|
||||
stalled: '0', active_healthy: '0', waiting: '0',
|
||||
@@ -339,11 +350,23 @@ export async function probeQueueState(
|
||||
): Promise<QueueSubmitState> {
|
||||
const timeoutMs = opts.timeoutMs ?? QUEUE_PROBE_TIMEOUT_MS;
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
// issue #6 / TODOS "cancel timed-out submit-time queue probes": the losing
|
||||
// inner probe used to keep running on the pool after the race resolved —
|
||||
// under pool exhaustion the abandoned query held a slot and made the
|
||||
// exhaustion worse. The timeout now aborts a per-probe signal so the
|
||||
// in-flight SQL is cancelled (postgres.js .cancel()) and its slot released.
|
||||
const probeAbort = new AbortController();
|
||||
const timeout = new Promise<QueueSubmitState>((resolveTimeout) => {
|
||||
timer = setTimeout(() => resolveTimeout({ probe_failed: true }), timeoutMs);
|
||||
timer = setTimeout(() => {
|
||||
probeAbort.abort();
|
||||
resolveTimeout({ probe_failed: true });
|
||||
}, timeoutMs);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([probeQueueStateInner(engine, queue, handlerNames), timeout]);
|
||||
return await Promise.race([
|
||||
probeQueueStateInner(engine, queue, handlerNames, { signal: probeAbort.signal }),
|
||||
timeout,
|
||||
]);
|
||||
} catch {
|
||||
return { probe_failed: true };
|
||||
} finally {
|
||||
@@ -355,8 +378,9 @@ async function probeQueueStateInner(
|
||||
engine: BrainEngine,
|
||||
queue: string,
|
||||
handlerNames: string[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<QueueSubmitState> {
|
||||
const sig = await queryWedgeSignals(engine, queue, handlerNames);
|
||||
const sig = await queryWedgeSignals(engine, queue, handlerNames, opts);
|
||||
|
||||
// Oldest-waiting age, same filter shape as the wedge signals. Perf: the
|
||||
// wave's migration (landing in another lane) adds the btree
|
||||
@@ -368,6 +392,7 @@ async function probeQueueStateInner(
|
||||
FROM minion_jobs
|
||||
WHERE queue = $1 AND status = 'waiting'`,
|
||||
[queue],
|
||||
opts,
|
||||
);
|
||||
const rawAge = ageRows[0]?.age ?? null;
|
||||
const oldestAge = rawAge === null ? null : Number(rawAge);
|
||||
|
||||
@@ -218,6 +218,16 @@ export interface MinionWorkerOpts {
|
||||
* hung probe would wedge the recursive setTimeout chain forever and
|
||||
* silently disable the health monitor. Default: 10000 (10 seconds). */
|
||||
dbProbeTimeoutMs?: number;
|
||||
/** issue #5: 'process' runs each claimed job in a SIGKILL-able child
|
||||
* process (blast radius = 1 job). Default 'inline' (today's behavior).
|
||||
* Requires childCliInvocation; the CLI layer resolves + validates it. */
|
||||
jobIsolation?: 'inline' | 'process';
|
||||
/** How to invoke the gbrain CLI for job children (resolved fail-fast at
|
||||
* worker startup by the CLI layer; structurally ChildCliInvocation from
|
||||
* job-isolation.ts — kept inline here to avoid an import cycle). */
|
||||
childCliInvocation?: { cmd: string; argsPrefix: string[] } | null;
|
||||
/** tini path for wrapping job children ('' = absent, direct spawn). */
|
||||
childTiniPath?: string;
|
||||
}
|
||||
|
||||
// --- Job Context (passed to handlers) ---
|
||||
@@ -372,6 +382,20 @@ export type TranscriptEntry =
|
||||
| { type: 'llm_turn'; model: string; tokens_in: number; tokens_out: number; ts: string }
|
||||
| { type: 'error'; message: string; stack?: string; ts: string };
|
||||
|
||||
// --- Abort-reason literals (single source of truth) ---
|
||||
//
|
||||
// Per-job abort sites construct `new Error(REASON)`; classification sites
|
||||
// (worker.ts INFRASTRUCTURE_ABORT_REASONS, child-job-runner.ts
|
||||
// PER_JOB_ABORT_REASONS) match on the message. Deriving both sets from these
|
||||
// constants keeps a rename at an abort site from silently flipping child
|
||||
// classification (maintainability review).
|
||||
|
||||
/** Infrastructure faults: released, no attempt burned; stall sweeper requeues. */
|
||||
export const ABORT_REASON_LOCK_RENEWAL_FAILED = 'lock-renewal-failed';
|
||||
export const ABORT_REASON_LOCK_LOST = 'lock-lost';
|
||||
/** Job-targeted aborts: keep their existing attempt semantics. */
|
||||
export const ABORT_REASON_TIMEOUT = 'timeout';
|
||||
|
||||
// --- Errors ---
|
||||
|
||||
/** Throw this from a handler to skip all retry logic and go straight to 'dead'. */
|
||||
|
||||
@@ -22,3 +22,17 @@
|
||||
|
||||
/** Worker drained itself because RSS crossed the watchdog cap. */
|
||||
export const WORKER_EXIT_RSS_WATCHDOG = 12;
|
||||
|
||||
// --- `gbrain jobs run-child` exit codes (issue #5 process isolation) -------
|
||||
//
|
||||
// The parent (child-job-runner.ts) classifies a child by RESULT-FILE PRESENCE
|
||||
// first: a written outcome file + exit 0 is the normal path even for handler
|
||||
// FAILURE (a reported error outcome is a successful report). Non-zero codes
|
||||
// mean "could not run or could not report":
|
||||
|
||||
/** run-child misuse: bad/missing argv or env, or a PGLite engine (no isolation there). */
|
||||
export const JOB_CHILD_EXIT_USAGE = 13;
|
||||
/** Job row validation failed: not 'active' or lock-token mismatch (reclaimed). */
|
||||
export const JOB_CHILD_EXIT_NOT_CLAIMED = 14;
|
||||
/** Handler finished but the outcome file could not be written. */
|
||||
export const JOB_CHILD_EXIT_RESULT_WRITE_FAILED = 15;
|
||||
|
||||
+201
-77
@@ -18,7 +18,11 @@ import type {
|
||||
MinionJob, MinionJobContext, MinionHandler, MinionWorkerOpts,
|
||||
MinionQueueOpts, TokenUpdate,
|
||||
} from './types.ts';
|
||||
import { UnrecoverableError } from './types.ts';
|
||||
import {
|
||||
UnrecoverableError,
|
||||
ABORT_REASON_LOCK_RENEWAL_FAILED,
|
||||
ABORT_REASON_LOCK_LOST,
|
||||
} from './types.ts';
|
||||
import { MinionQueue } from './queue.ts';
|
||||
import { calculateBackoff } from './backoff.ts';
|
||||
import { RateLeaseUnavailableError } from './handlers/subagent.ts';
|
||||
@@ -29,6 +33,20 @@ import {
|
||||
type LockRenewalDeps,
|
||||
type LockRenewalState,
|
||||
} from './lock-renewal-tick.ts';
|
||||
import {
|
||||
runDbProbe,
|
||||
getConnectionRouting,
|
||||
DIRECT_PROBE_TIMEOUT_MS,
|
||||
type DbProbeResult,
|
||||
type PoolDiagnostics,
|
||||
} from './db-probe.ts';
|
||||
import { buildJobContext } from './job-context.ts';
|
||||
import {
|
||||
runJobInChild,
|
||||
ChildSpawnInfraError,
|
||||
ChildWorkerShutdownError,
|
||||
ChildNotClaimedError,
|
||||
} from './child-job-runner.ts';
|
||||
import { lockRenewalAudit } from '../audit/lock-renewal-audit.ts';
|
||||
import { isRetryableConnError } from '../retry-matcher.ts';
|
||||
import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from './reconnect.ts';
|
||||
@@ -49,8 +67,8 @@ import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from '
|
||||
* to this set is a deliberate two-line change, not a silent regression).
|
||||
*/
|
||||
export const INFRASTRUCTURE_ABORT_REASONS = new Set<string>([
|
||||
'lock-renewal-failed',
|
||||
'lock-lost',
|
||||
ABORT_REASON_LOCK_RENEWAL_FAILED,
|
||||
ABORT_REASON_LOCK_LOST,
|
||||
]);
|
||||
import { randomUUID } from 'crypto';
|
||||
import { EventEmitter } from 'events';
|
||||
@@ -120,10 +138,19 @@ export function getAccurateRss(
|
||||
}
|
||||
|
||||
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
|
||||
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
|
||||
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit.
|
||||
* `verdict` (issue #6) distinguishes local pool starvation from a genuinely
|
||||
* unreachable server so operators stop debugging the wrong layer; absent on
|
||||
* engines without the probe's disambiguation lane. */
|
||||
export type UnhealthyReason =
|
||||
| { reason: 'db_dead'; consecutiveFailures: number; message: string }
|
||||
| { reason: 'stalled'; waitingCount: number; idleMinutes: number };
|
||||
| {
|
||||
reason: 'db_dead';
|
||||
consecutiveFailures: number;
|
||||
message: string;
|
||||
verdict?: 'pool_starved' | 'server_unreachable' | 'unknown';
|
||||
}
|
||||
| { reason: 'stalled'; waitingCount: number; idleMinutes: number }
|
||||
| { reason: 'child_spawn_failing'; consecutiveFailures: number; message: string };
|
||||
|
||||
/**
|
||||
* Read the quiet_hours JSONB column off a MinionJob, if present. The
|
||||
@@ -188,6 +215,18 @@ export class MinionWorker extends EventEmitter {
|
||||
private _peakRssMb = 0;
|
||||
/** Latch so the 80%-of-cap soft-warn fires once per crossing, not every check. */
|
||||
private _softWarnFired = false;
|
||||
/**
|
||||
* Circuit breaker for deterministic child-bootstrap failures (red-team
|
||||
* finding): a spawn failure releases the job with no attempt burned, the
|
||||
* stall sweeper requeues it, the same worker re-claims — an infinite
|
||||
* claim/release loop the stall detector cannot see (every settle refreshes
|
||||
* the progress clock). After CHILD_SPAWN_FAIL_EXIT_AFTER consecutive
|
||||
* spawn-class failures we emit 'unhealthy' so the process manager restarts
|
||||
* the worker (and the supervisor's crash budget takes over if the child
|
||||
* CLI stays broken).
|
||||
*/
|
||||
private _consecutiveChildSpawnFailures = 0;
|
||||
private static readonly CHILD_SPAWN_FAIL_EXIT_AFTER = 3;
|
||||
|
||||
private opts: Required<MinionWorkerOpts>;
|
||||
|
||||
@@ -215,7 +254,22 @@ export class MinionWorker extends EventEmitter {
|
||||
stallExitAfterMs: opts?.stallExitAfterMs ?? 10 * 60_000,
|
||||
dbFailExitAfter: opts?.dbFailExitAfter ?? 3,
|
||||
dbProbeTimeoutMs: opts?.dbProbeTimeoutMs ?? 10_000,
|
||||
jobIsolation: opts?.jobIsolation ?? 'inline',
|
||||
childCliInvocation: opts?.childCliInvocation ?? null,
|
||||
childTiniPath: opts?.childTiniPath ?? '',
|
||||
};
|
||||
// Process isolation contract: 'process' without a resolved child CLI
|
||||
// invocation would silently execute handlers INLINE while the evict path
|
||||
// believed it was isolated (predicate mismatch — red-team finding). The
|
||||
// CLI layer always resolves + validates the invocation; library callers
|
||||
// must too. Loud construction throw, same discipline as the stall
|
||||
// thresholds below.
|
||||
if (this.opts.jobIsolation === 'process' && this.opts.childCliInvocation == null) {
|
||||
throw new Error(
|
||||
"MinionWorkerOpts: jobIsolation 'process' requires childCliInvocation " +
|
||||
'(resolve it via resolveChildCliInvocation and validate it exists before constructing the worker).',
|
||||
);
|
||||
}
|
||||
// Stall thresholds contract: exit MUST be strictly greater than warn.
|
||||
// If exit <= warn, the warn-then-exit semantics break: a single tick at
|
||||
// idle > warn would set stallWarningSince and the subsequent tick at
|
||||
@@ -231,6 +285,15 @@ export class MinionWorker extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only handler lookup. `gbrain jobs run-child` registers the builtin
|
||||
* handlers against a throwaway worker (registerBuiltinHandlers's existing
|
||||
* contract) and resolves the one it needs through this accessor.
|
||||
*/
|
||||
getHandler(name: string): MinionHandler | undefined {
|
||||
return this.handlers.get(name);
|
||||
}
|
||||
|
||||
/** Register a handler for a job type. */
|
||||
register(name: string, handler: MinionHandler): void {
|
||||
this.handlers.set(name, handler);
|
||||
@@ -263,7 +326,9 @@ export class MinionWorker extends EventEmitter {
|
||||
if (this.listenerCount('unhealthy') === 0) {
|
||||
const detail = info.reason === 'db_dead'
|
||||
? `DB unreachable (${info.consecutiveFailures} probes): ${info.message}`
|
||||
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
|
||||
: info.reason === 'child_spawn_failing'
|
||||
? `job-child spawn failing (${info.consecutiveFailures} consecutive): ${info.message}`
|
||||
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
|
||||
console.error(
|
||||
`[health] FATAL: ${detail}. No 'unhealthy' listener registered; ` +
|
||||
`defaulting to process.exit(1) for process-manager restart.`,
|
||||
@@ -388,28 +453,36 @@ export class MinionWorker extends EventEmitter {
|
||||
let healthRunning = false;
|
||||
let healthExited = false;
|
||||
|
||||
// Race executeRaw against a wall-clock deadline. A hung connection
|
||||
// (network-partitioned PgBouncer, deadlocked backend) would otherwise
|
||||
// hold the await forever — the recursive setTimeout's next tick is only
|
||||
// scheduled in `finally`, so a hung probe would silently disable the
|
||||
// entire health monitor. The timeout treats hangs as failures and feeds
|
||||
// them into `dbFailExitAfter`.
|
||||
const probeWithTimeout = async (): Promise<void> => {
|
||||
const ac = new AbortController();
|
||||
const timeoutMs = this.opts.dbProbeTimeoutMs;
|
||||
const timer = setTimeout(() => ac.abort(), timeoutMs);
|
||||
try {
|
||||
await Promise.race([
|
||||
this.engine.executeRaw('SELECT 1'),
|
||||
new Promise<never>((_, reject) => {
|
||||
ac.signal.addEventListener('abort', () => {
|
||||
reject(new Error(`probe timeout after ${timeoutMs}ms`));
|
||||
});
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
// DB liveness probe with pool-starvation disambiguation (issue #6).
|
||||
// The probe body lives in db-probe.ts (hermetically tested,
|
||||
// lock-renewal-tick pattern); this is the thin adapter. Both probes
|
||||
// carry an AbortSignal — a hung probe is CANCELLED (slot released),
|
||||
// never abandoned. The direct-lane probe runs ONLY when dual-pool is
|
||||
// genuinely active (a kill-switched executeRawDirect would probe the
|
||||
// same starved read pool twice and fake a verdict).
|
||||
const runProbe = async (): Promise<DbProbeResult> => {
|
||||
const cm = getConnectionRouting(this.engine);
|
||||
const dualPool = cm?.isDualPoolActive?.() === true;
|
||||
const getDiag = (this.engine as {
|
||||
getPoolDiagnostics?: () => PoolDiagnostics | null;
|
||||
}).getPoolDiagnostics;
|
||||
return runDbProbe({
|
||||
probeRead: async (signal) => {
|
||||
await this.engine.executeRaw('SELECT 1', undefined, { signal });
|
||||
},
|
||||
...(dualPool
|
||||
? {
|
||||
probeDirect: async (signal: AbortSignal) => {
|
||||
await this.engine.executeRawDirect('SELECT 1', undefined, { signal });
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
...(typeof getDiag === 'function'
|
||||
? { getDiagnostics: () => getDiag.call(this.engine) }
|
||||
: {}),
|
||||
timeoutMs: this.opts.dbProbeTimeoutMs,
|
||||
directTimeoutMs: DIRECT_PROBE_TIMEOUT_MS,
|
||||
});
|
||||
};
|
||||
|
||||
const runHealthCheck = async (): Promise<void> => {
|
||||
@@ -417,25 +490,25 @@ export class MinionWorker extends EventEmitter {
|
||||
healthRunning = true;
|
||||
try {
|
||||
// --- 1. DB liveness probe ---
|
||||
try {
|
||||
await probeWithTimeout();
|
||||
const probe = await runProbe();
|
||||
if (probe.ok) {
|
||||
consecutiveDbFailures = 0;
|
||||
} catch (e) {
|
||||
} else {
|
||||
consecutiveDbFailures++;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(
|
||||
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${msg}`,
|
||||
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${probe.detail}`,
|
||||
);
|
||||
if (consecutiveDbFailures >= this.opts.dbFailExitAfter) {
|
||||
console.error(
|
||||
`[health] DB unreachable after ${this.opts.dbFailExitAfter} consecutive probes. ` +
|
||||
`Emitting 'unhealthy' for process-manager restart.`,
|
||||
`[health] DB probe failed ${this.opts.dbFailExitAfter} consecutive times ` +
|
||||
`(verdict: ${probe.verdict}). Emitting 'unhealthy' for process-manager restart.`,
|
||||
);
|
||||
healthExited = true;
|
||||
this.emitUnhealthy({
|
||||
reason: 'db_dead',
|
||||
consecutiveFailures: consecutiveDbFailures,
|
||||
message: msg,
|
||||
message: probe.detail,
|
||||
verdict: probe.verdict,
|
||||
});
|
||||
}
|
||||
return; // Skip stall check when DB is flaky
|
||||
@@ -876,7 +949,7 @@ export class MinionWorker extends EventEmitter {
|
||||
// and the tick keeps its legacy no-reconnect behavior.
|
||||
const engineReconnect = (this.engine as { reconnect?: (ctx?: { error?: unknown }) => Promise<void> }).reconnect;
|
||||
const renewalDeps: LockRenewalDeps = {
|
||||
renewLock: (id, tok, dur) => this.queue.renewLock(id, tok, dur),
|
||||
renewLock: (id, tok, dur, opts) => this.queue.renewLock(id, tok, dur, opts),
|
||||
audit: lockRenewalAudit,
|
||||
now: Date.now,
|
||||
setTimeout: (cb, ms) => globalThis.setTimeout(cb, ms),
|
||||
@@ -946,11 +1019,12 @@ export class MinionWorker extends EventEmitter {
|
||||
);
|
||||
clearInterval(lockTimer);
|
||||
this.inFlight.delete(job.id);
|
||||
// D8a: don't failJob if the abort was infrastructure. The
|
||||
// stall detector will reclaim the row cleanly because the
|
||||
// lock has expired (lock-renewal aborts only fire after
|
||||
// lockDuration - safetyMargin elapsed without renewal).
|
||||
if (!INFRASTRUCTURE_ABORT_REASONS.has(reason)) {
|
||||
// D8a: don't failJob on infrastructure aborts (stall detector
|
||||
// reclaims after lock expiry). Isolation mode: also skip — the
|
||||
// group SIGKILL already fired and executeJob's own recording
|
||||
// follows; a competing evict failJob('dead') could dead-letter a
|
||||
// job with attempts remaining (adversarial-review P3).
|
||||
if (!INFRASTRUCTURE_ABORT_REASONS.has(reason) && this.opts.jobIsolation !== 'process') {
|
||||
this.queue.failJob(
|
||||
job.id,
|
||||
lockToken,
|
||||
@@ -1024,48 +1098,48 @@ export class MinionWorker extends EventEmitter {
|
||||
return;
|
||||
}
|
||||
|
||||
// issue #5: 'process' isolation runs the handler in a SIGKILL-able child;
|
||||
// the parent keeps claim/renewal and ALL result recording below — this
|
||||
// branch swaps ONLY the execution engine. When isolated, the parent-side
|
||||
// context is skipped entirely (the child builds its own against its own
|
||||
// engine; building it here would be dead work holding closures). The
|
||||
// constructor guarantees childCliInvocation != null whenever
|
||||
// jobIsolation === 'process', so this predicate matches the evict guard.
|
||||
const isolated = this.opts.jobIsolation === 'process';
|
||||
|
||||
// Build job context with per-job AbortSignal + shared shutdown signal.
|
||||
// Most handlers only care about `signal` (timeout / cancel / lock-loss).
|
||||
// `shutdownSignal` is separate: fires only on worker process SIGTERM/SIGINT.
|
||||
// Handlers that need to run cleanup before worker exit (shell handler's
|
||||
// SIGTERM→5s→SIGKILL on its child) subscribe to shutdownSignal too.
|
||||
const context: MinionJobContext = {
|
||||
id: job.id,
|
||||
name: job.name,
|
||||
data: job.data,
|
||||
attempts_made: job.attempts_made,
|
||||
signal: abort.signal,
|
||||
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
|
||||
shutdownSignal: this.shutdownAbort.signal,
|
||||
updateProgress: async (progress: unknown) => {
|
||||
await this.queue.updateProgress(job.id, lockToken, progress);
|
||||
},
|
||||
updateTokens: async (tokens: TokenUpdate) => {
|
||||
await this.queue.updateTokens(job.id, lockToken, tokens);
|
||||
},
|
||||
log: async (message: string | Record<string, unknown>) => {
|
||||
const value = typeof message === 'string' ? message : JSON.stringify(message);
|
||||
await this.engine.executeRaw(
|
||||
`UPDATE minion_jobs SET stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
|
||||
updated_at = now()
|
||||
WHERE id = $2 AND status = 'active' AND lock_token = $3`,
|
||||
[value, job.id, lockToken]
|
||||
// Builder shared with `gbrain jobs run-child` (job-context.ts) so the
|
||||
// process-isolation child wires the exact same DB-backed callbacks.
|
||||
const context: MinionJobContext | null = isolated
|
||||
? null
|
||||
: buildJobContext(
|
||||
this.engine,
|
||||
this.queue,
|
||||
job,
|
||||
lockToken,
|
||||
abort.signal,
|
||||
this.shutdownAbort.signal,
|
||||
);
|
||||
},
|
||||
isActive: async () => {
|
||||
const rows = await this.engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM minion_jobs WHERE id = $1 AND status = 'active' AND lock_token = $2`,
|
||||
[job.id, lockToken]
|
||||
);
|
||||
return rows.length > 0;
|
||||
},
|
||||
readInbox: async () => {
|
||||
return this.queue.readInbox(job.id, lockToken);
|
||||
},
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await handler(context);
|
||||
const result = isolated
|
||||
? await runJobInChild({
|
||||
jobId: job.id,
|
||||
jobName: job.name,
|
||||
lockToken,
|
||||
abortSignal: abort.signal,
|
||||
shutdownSignal: this.shutdownAbort.signal,
|
||||
invocation: this.opts.childCliInvocation as { cmd: string; argsPrefix: string[] },
|
||||
tiniPath: this.opts.childTiniPath,
|
||||
})
|
||||
: await handler(context as MinionJobContext);
|
||||
|
||||
// The child spawned and ran — the spawn path is healthy again.
|
||||
this._consecutiveChildSpawnFailures = 0;
|
||||
|
||||
clearInterval(lockTimer);
|
||||
|
||||
@@ -1121,6 +1195,56 @@ export class MinionWorker extends EventEmitter {
|
||||
return;
|
||||
}
|
||||
|
||||
// Any error that ISN'T a spawn failure proves the spawn path works —
|
||||
// keep the breaker's "consecutive" semantics honest.
|
||||
if (!(err instanceof ChildSpawnInfraError)) {
|
||||
this._consecutiveChildSpawnFailures = 0;
|
||||
}
|
||||
|
||||
// issue #5 process isolation — two more infrastructure classes, same
|
||||
// release semantics as the block above (lock expires once launchJob's
|
||||
// finally clears the renewal timer; the stall sweeper requeues):
|
||||
// - spawn failure: an ops misconfiguration (bad child CLI path) must
|
||||
// not burn attempts job-by-job until the queue dead-letters. The
|
||||
// CLI layer also fail-fast validates the invocation at startup.
|
||||
// - worker shutdown: a routine deploy killed the child before it
|
||||
// could report (codex-2 #7); burning an attempt per deploy would
|
||||
// dead-letter long jobs after a few releases.
|
||||
if (err instanceof ChildSpawnInfraError) {
|
||||
console.error(
|
||||
`Job ${job.id} (${job.name}) released after child spawn failure — ` +
|
||||
`check the worker's child CLI configuration: ${errorText} (no attempt burned)`,
|
||||
);
|
||||
this._consecutiveChildSpawnFailures += 1;
|
||||
if (this._consecutiveChildSpawnFailures >= MinionWorker.CHILD_SPAWN_FAIL_EXIT_AFTER) {
|
||||
console.error(
|
||||
`[isolation] ${this._consecutiveChildSpawnFailures} consecutive child spawn/bootstrap ` +
|
||||
`failures — the child CLI is deterministically broken. Emitting 'unhealthy' for ` +
|
||||
`process-manager restart instead of looping claim/release forever.`,
|
||||
);
|
||||
this.emitUnhealthy({
|
||||
reason: 'child_spawn_failing',
|
||||
consecutiveFailures: this._consecutiveChildSpawnFailures,
|
||||
message: errorText,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (err instanceof ChildWorkerShutdownError) {
|
||||
console.log(
|
||||
`Job ${job.id} (${job.name}) released after worker shutdown (${errorText}); ` +
|
||||
`stall detector will requeue (no attempt burned)`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (err instanceof ChildNotClaimedError) {
|
||||
// The child proved the claim is gone (reclaimed/cancelled before the
|
||||
// handler ran). The token-fenced failJob would no-op anyway — return
|
||||
// without burning anything against a claim we no longer hold.
|
||||
console.log(`Job ${job.id} (${job.name}): ${errorText}`);
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.41 Bug 2: lease-full bounces don't burn attempts.
|
||||
//
|
||||
// Pre-v0.41 every non-`UnrecoverableError` routed to `delayed` with
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* CheckoutGauge — approximate in-flight query counters for the health probe's
|
||||
* pool diagnostics (issue #6).
|
||||
*
|
||||
* HONESTY CONTRACT (read before extending): this gauge counts calls through
|
||||
* the engine's raw/direct/reserved/transaction seams ONLY. The majority of
|
||||
* engine traffic — tagged-template queries on `this.sql` (getConfig, CRUD,
|
||||
* search) — is NOT tracked; postgres.js exposes no public checkout counters
|
||||
* and proxying the Sql template function is too invasive for a diagnostic.
|
||||
* Every consumer must label these numbers as a tracked SUBSET and must never
|
||||
* derive "available" or "waiting" figures from them (that arithmetic is
|
||||
* invented telemetry — outside-voice review, codex-2 #3). The authoritative
|
||||
* starvation signal is the direct-lane disambiguation probe in
|
||||
* `src/core/minions/db-probe.ts`; these counts are supporting detail.
|
||||
*
|
||||
* Fail-open by construction: plain integer bumps, no I/O, release() clamps
|
||||
* at zero so a missed acquire can never underflow into negative counts.
|
||||
*/
|
||||
|
||||
/** Which engine seam the in-flight call went through. */
|
||||
export type GaugeKind = 'raw' | 'direct' | 'reserved' | 'tx';
|
||||
|
||||
export interface PoolGaugeSnapshot {
|
||||
/** executeRaw on the read pool. */
|
||||
raw: number;
|
||||
/** executeRawDirect (direct session lane when dual-pool, read pool otherwise). */
|
||||
direct: number;
|
||||
/** withReservedConnection holders. */
|
||||
reserved: number;
|
||||
/** transaction() bodies. */
|
||||
tx: number;
|
||||
}
|
||||
|
||||
export class CheckoutGauge {
|
||||
private counts: PoolGaugeSnapshot = { raw: 0, direct: 0, reserved: 0, tx: 0 };
|
||||
|
||||
acquire(kind: GaugeKind): void {
|
||||
this.counts[kind] += 1;
|
||||
}
|
||||
|
||||
release(kind: GaugeKind): void {
|
||||
if (this.counts[kind] > 0) this.counts[kind] -= 1;
|
||||
}
|
||||
|
||||
snapshot(): PoolGaugeSnapshot {
|
||||
return { ...this.counts };
|
||||
}
|
||||
}
|
||||
+129
-13
@@ -25,6 +25,7 @@ import {
|
||||
type BatchAuditSite,
|
||||
} from './retry.ts';
|
||||
import { isConnectionEndedError } from './retry-matcher.ts';
|
||||
import { CheckoutGauge, type PoolGaugeSnapshot } from './pool-gauge.ts';
|
||||
import {
|
||||
valueHash,
|
||||
normalizeDimension,
|
||||
@@ -82,7 +83,7 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
|
||||
import { finalizeLastSeen } from './chronicle/last-seen.ts';
|
||||
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
|
||||
import * as db from './db.ts';
|
||||
import { ConnectionManager } from './connection-manager.ts';
|
||||
import { ConnectionManager, DEFAULT_DIRECT_POOL_SIZE } from './connection-manager.ts';
|
||||
import { logConnectionEvent } from './connection-audit.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToStalePage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, takeHitRowToHit, isUndefinedTableError, warnOncePerProcess } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
@@ -132,6 +133,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
private _savedConfig: (EngineConfig & { poolSize?: number; parentConnectionManager?: ConnectionManager }) | null = null;
|
||||
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
|
||||
private _reconnecting = false;
|
||||
/**
|
||||
* Approximate in-flight counters for the health probe's diagnostics
|
||||
* (issue #6). Tracks the raw/direct/reserved/tx seams ONLY — see the
|
||||
* honesty contract in pool-gauge.ts. Shared by tx-scoped engine clones
|
||||
* via the prototype chain (same process, same pools). Fail-open.
|
||||
*/
|
||||
private checkoutGauge = new CheckoutGauge();
|
||||
/**
|
||||
* #1471: module-singleton OWNERSHIP token. `true` only for the engine whose
|
||||
* connect() actually created the shared db.ts `sql` singleton (returned
|
||||
@@ -297,6 +305,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
max: size,
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
|
||||
max_lifetime: db.resolveMaxLifetimeSeconds(),
|
||||
types: { bigint: postgres.BigInt },
|
||||
// Silence postgres NOTICE-level messages by default. See db.ts for
|
||||
// rationale (stdout-parsing callers like jobs-submit --json break when
|
||||
@@ -1059,18 +1069,83 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
const conn = this.sql;
|
||||
return conn.begin(async (tx) => {
|
||||
// Create a scoped engine with tx as its connection, no shared state mutation
|
||||
const txEngine = Object.create(this) as PostgresEngine;
|
||||
Object.defineProperty(txEngine, 'sql', { get: () => tx });
|
||||
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
|
||||
return fn(txEngine);
|
||||
}) as Promise<T>;
|
||||
// try/finally, not .finally on the chained promise: begin() can throw
|
||||
// SYNCHRONOUSLY (e.g. nested transaction on a tx clone whose conn has no
|
||||
// .begin), which would skip a chained .finally and leak the counter.
|
||||
this.checkoutGauge.acquire('tx');
|
||||
try {
|
||||
return await (conn.begin(async (tx) => {
|
||||
// Create a scoped engine with tx as its connection, no shared state mutation
|
||||
const txEngine = Object.create(this) as PostgresEngine;
|
||||
Object.defineProperty(txEngine, 'sql', { get: () => tx });
|
||||
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
|
||||
return fn(txEngine);
|
||||
}) as Promise<T>);
|
||||
} finally {
|
||||
this.checkoutGauge.release('tx');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #6 (reserved-connection routing): concurrent DIRECT-pool reserves
|
||||
* are capped at directPoolSize - 1 so the claim/renewLock heartbeats always
|
||||
* keep >= 1 direct slot; overflow falls back to the READ pool — exactly the
|
||||
* pre-routing behavior, so this change is strictly never-worse than the
|
||||
* status quo (deliberate rejection of queue-for-a-permit: that would block
|
||||
* migrations behind multi-minute CREATE INDEX holds). Per-process by
|
||||
* design: each process owns its own direct pool, so a CLI migration's
|
||||
* reserves cannot starve a worker's heartbeats.
|
||||
*/
|
||||
private _reservedDirectInFlight = 0;
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
const pool = this.sql;
|
||||
const reserved = await pool.reserve();
|
||||
// Long-hold reserved work (CREATE INDEX CONCURRENTLY, transaction:false
|
||||
// migration DDL, backfill BEGIN..COMMIT batches) belongs on the DIRECT
|
||||
// session lane: 30-min statement_timeout + maintenance_work_mem GUCs and
|
||||
// it stops pinning the worker's shared read pool (the observed 353s
|
||||
// COMMIT in issue #6 was a reserved read-pool slot). Never reroute inside
|
||||
// an open transaction (same guard shape as executeRawDirect).
|
||||
const inTransaction = this._sql !== null && this.connectionManager?.peekReadPool() !== this._sql;
|
||||
let pool = this.sql;
|
||||
let fromDirect = false;
|
||||
if (!inTransaction && this.connectionManager?.isDualPoolActive()) {
|
||||
const size = this.connectionManager.describeMode().direct_pool_size ?? DEFAULT_DIRECT_POOL_SIZE;
|
||||
// NO floor on the cap (red-team finding): at direct_pool_size=1 a
|
||||
// Math.max(1, ...) floor would let a multi-minute reserve consume the
|
||||
// ONLY direct session and starve claim/renewLock heartbeats — the
|
||||
// exact #6 class, reintroduced on the direct pool. cap <= 0 means the
|
||||
// direct lane has no spare capacity for reserves: use the read pool
|
||||
// (the true status quo).
|
||||
const cap = size - 1;
|
||||
if (cap >= 1 && this._reservedDirectInFlight < cap) {
|
||||
// Take the permit in the SAME synchronous frame as the check — a
|
||||
// check-then-increment spanning `await ddl()` is a TOCTOU that lets
|
||||
// same-tick concurrent reserves overshoot the cap and starve the
|
||||
// heartbeat slot the cap exists to protect (adversarial-review P2).
|
||||
this._reservedDirectInFlight += 1;
|
||||
fromDirect = true;
|
||||
try {
|
||||
pool = await this.connectionManager.ddl();
|
||||
} catch {
|
||||
// ddl() failure flips its own kill switch; fall back to the read
|
||||
// pool (status quo) rather than failing the caller.
|
||||
this._reservedDirectInFlight -= 1;
|
||||
fromDirect = false;
|
||||
pool = this.sql;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Gauge BEFORE reserve(): a reserve() stuck waiting for a free slot is
|
||||
// exactly the in-flight pressure the probe diagnostics should surface.
|
||||
this.checkoutGauge.acquire('reserved');
|
||||
let reserved: Awaited<ReturnType<typeof pool.reserve>>;
|
||||
try {
|
||||
reserved = await pool.reserve();
|
||||
} catch (e) {
|
||||
this.checkoutGauge.release('reserved');
|
||||
if (fromDirect) this._reservedDirectInFlight -= 1;
|
||||
throw e;
|
||||
}
|
||||
try {
|
||||
const conn: ReservedConnection = {
|
||||
async executeRaw<R = Record<string, unknown>>(
|
||||
@@ -1091,7 +1166,34 @@ export class PostgresEngine implements BrainEngine {
|
||||
};
|
||||
return await fn(conn);
|
||||
} finally {
|
||||
reserved.release();
|
||||
// Counter/gauge decrements run regardless of release() throwing
|
||||
// (double-release or socket error must not permanently leak a permit
|
||||
// of the small direct-reserve budget — data-migration review).
|
||||
try {
|
||||
reserved.release();
|
||||
} catch {
|
||||
// best-effort; the pool's own lifecycle handles a broken reservation
|
||||
}
|
||||
this.checkoutGauge.release('reserved');
|
||||
if (fromDirect) this._reservedDirectInFlight -= 1;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Health-probe diagnostics (issue #6). Duck-typed — deliberately NOT on the
|
||||
* BrainEngine interface (PGLite has no pool to diagnose; the worker reads
|
||||
* it optionally, same pattern as `engine.reconnect`). Fail-open: returns
|
||||
* null instead of throwing.
|
||||
*/
|
||||
getPoolDiagnostics(): { tracked: PoolGaugeSnapshot; poolMax: number | null } | null {
|
||||
try {
|
||||
const max = (this.sql as unknown as { options?: { max?: number } }).options?.max;
|
||||
return {
|
||||
tracked: this.checkoutGauge.snapshot(),
|
||||
poolMax: typeof max === 'number' ? max : null,
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6150,7 +6252,15 @@ export class PostgresEngine implements BrainEngine {
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
): Promise<T[]> {
|
||||
return this.runUnsafe<T>(this.sql, sql, params, opts);
|
||||
// try/finally (not .finally on the promise): runUnsafe throws
|
||||
// SYNCHRONOUSLY on a pre-aborted signal, which would skip a chained
|
||||
// .finally and leak the counter.
|
||||
this.checkoutGauge.acquire('raw');
|
||||
try {
|
||||
return await this.runUnsafe<T>(this.sql, sql, params, opts);
|
||||
} finally {
|
||||
this.checkoutGauge.release('raw');
|
||||
}
|
||||
// Pre-#406 behavior: throw on any error including connection death.
|
||||
// Per-call auto-retry is not safe here because executeRaw is also used
|
||||
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
|
||||
@@ -6186,7 +6296,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
const conn = (!inTransaction && this.connectionManager?.isDualPoolActive())
|
||||
? await this.connectionManager.ddl()
|
||||
: this.sql;
|
||||
return this.runUnsafe<T>(conn, sql, params, opts);
|
||||
// try/finally, not .finally — see executeRaw (sync throw on pre-aborted signal).
|
||||
this.checkoutGauge.acquire('direct');
|
||||
try {
|
||||
return await this.runUnsafe<T>(conn, sql, params, opts);
|
||||
} finally {
|
||||
this.checkoutGauge.release('direct');
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# gbrain agent workspace — template
|
||||
|
||||
<!-- gbrain-template-stamp: 0.46.0.0 -->
|
||||
<!-- gbrain-template-stamp: 0.46.1.0 -->
|
||||
|
||||
This repository is the **"Use this template"** distribution artifact for a
|
||||
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
/**
|
||||
* issue #5 — `runJobInChild` against REAL child processes (.mjs harnesses run
|
||||
* by process.execPath — the child-worker-supervisor.test.ts pattern).
|
||||
*
|
||||
* Paths pinned:
|
||||
* - success outcome file → resolves with the handler result
|
||||
* - error outcome file → throws the reconstructed error class (exit 0!)
|
||||
* - exit 1 with no file → generic throw naming the exit (attempt burned)
|
||||
* - SIGTERM-ignoring child + aborted signal → group SIGKILL at the injected
|
||||
* grace; "terminated after abort" classification
|
||||
* - pre-aborted signal → child killed promptly
|
||||
* - spawn ENOENT → ChildSpawnInfraError (release, no attempt burned)
|
||||
* - worker-shutdown: child finishes + reports during the drain window →
|
||||
* normal success; child that can't report → ChildWorkerShutdownError
|
||||
* - child env contract (result path, lock token, parent pid, pool bounds)
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, existsSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
runJobInChild,
|
||||
ChildSpawnInfraError,
|
||||
ChildWorkerShutdownError,
|
||||
ChildNotClaimedError,
|
||||
} from '../src/core/minions/child-job-runner.ts';
|
||||
import { UnrecoverableError } from '../src/core/minions/types.ts';
|
||||
import { RateLeaseUnavailableError } from '../src/core/minions/handlers/subagent.ts';
|
||||
|
||||
const TEST_TIMEOUT_MS = 30_000;
|
||||
|
||||
let harnessDir: string;
|
||||
|
||||
function makeHarness(name: string, body: string): string {
|
||||
const path = join(harnessDir, `${name}.mjs`);
|
||||
writeFileSync(
|
||||
path,
|
||||
`import { writeFileSync, renameSync } from 'node:fs';\n` +
|
||||
`const RESULT = process.env.GBRAIN_JOB_RESULT_PATH;\n` +
|
||||
`const writeOutcome = (o) => { writeFileSync(RESULT + '.tmp', JSON.stringify(o)); renameSync(RESULT + '.tmp', RESULT); };\n` +
|
||||
body +
|
||||
// Readiness handshake LAST (after the body installed its signal
|
||||
// handlers): fixed sleeps raced child startup on loaded CI runners —
|
||||
// a SIGTERM landing before process.on('SIGTERM') installs takes the
|
||||
// default disposition and flips shutdown-semantics assertions
|
||||
// (testing specialist). The runner's outcome dir is internal, so the
|
||||
// ready path comes from a TEST-provided env var.
|
||||
`\nif (process.env.HARNESS_READY_PATH) writeFileSync(process.env.HARNESS_READY_PATH, '1');\n`,
|
||||
'utf8',
|
||||
);
|
||||
return path;
|
||||
}
|
||||
|
||||
/** Make a per-test ready path + the env to hand runJobInChild. */
|
||||
function readiness(name: string): { env: Record<string, string | undefined>; wait: () => Promise<void> } {
|
||||
const readyPath = join(harnessDir, `${name}.ready`);
|
||||
return {
|
||||
env: { ...process.env, HARNESS_READY_PATH: readyPath },
|
||||
wait: async () => {
|
||||
const deadline = Date.now() + 10_000;
|
||||
while (Date.now() < deadline) {
|
||||
if (existsSync(readyPath)) return;
|
||||
await new Promise((r) => setTimeout(r, 25));
|
||||
}
|
||||
throw new Error('harness never signalled readiness');
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
harnessDir = mkdtempSync(join(tmpdir(), 'gbrain-cjr-harness-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(harnessDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function baseOpts(harnessPath: string) {
|
||||
return {
|
||||
jobId: 77,
|
||||
jobName: 'subagent',
|
||||
lockToken: 'tok-cjr',
|
||||
abortSignal: new AbortController().signal,
|
||||
shutdownSignal: new AbortController().signal,
|
||||
invocation: { cmd: process.execPath, argsPrefix: [harnessPath] },
|
||||
tiniPath: '', // direct spawn in tests; group signaling covers both shapes
|
||||
};
|
||||
}
|
||||
|
||||
describe('runJobInChild (real children)', () => {
|
||||
test('success outcome resolves with the result; env contract honored', async () => {
|
||||
const harness = makeHarness(
|
||||
'success',
|
||||
`writeOutcome({ outcome: 'success', result: {
|
||||
echoedToken: process.env.GBRAIN_JOB_LOCK_TOKEN,
|
||||
isChild: process.env.GBRAIN_JOB_CHILD,
|
||||
parentPid: process.env.GBRAIN_JOB_PARENT_PID,
|
||||
poolSize: process.env.GBRAIN_POOL_SIZE,
|
||||
directPoolSize: process.env.GBRAIN_DIRECT_POOL_SIZE,
|
||||
argv: process.argv.slice(2),
|
||||
}});\n` +
|
||||
`process.exit(0);\n`,
|
||||
);
|
||||
const result = (await runJobInChild(baseOpts(harness))) as Record<string, unknown>;
|
||||
expect(result.echoedToken).toBe('tok-cjr');
|
||||
expect(result.isChild).toBe('1');
|
||||
expect(result.parentPid).toBe(String(process.pid));
|
||||
expect(result.poolSize).toBe('3');
|
||||
expect(result.directPoolSize).toBe('1'); // codex-2 #6: child direct pool bounded
|
||||
expect(result.argv).toEqual(['jobs', 'run-child', '--job-id', '77']);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('error outcome (exit 0) throws the reconstructed class', async () => {
|
||||
const harness = makeHarness(
|
||||
'error-unrecoverable',
|
||||
`writeOutcome({ outcome: 'error', errorKind: 'unrecoverable', message: 'bad schema, never retry' });\n` +
|
||||
`process.exit(0);\n`,
|
||||
);
|
||||
await expect(runJobInChild(baseOpts(harness))).rejects.toBeInstanceOf(UnrecoverableError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('rate-lease outcome rebuilds RateLeaseUnavailableError with fields', async () => {
|
||||
const harness = makeHarness(
|
||||
'error-lease',
|
||||
`writeOutcome({ outcome: 'error', errorKind: 'rate_lease', message: 'lease full', lease: { key: 'anthropic', active: 4, max: 4 } });\n` +
|
||||
`process.exit(0);\n`,
|
||||
);
|
||||
try {
|
||||
await runJobInChild(baseOpts(harness));
|
||||
throw new Error('should have thrown');
|
||||
} catch (e) {
|
||||
expect(e).toBeInstanceOf(RateLeaseUnavailableError);
|
||||
expect((e as RateLeaseUnavailableError).key).toBe('anthropic');
|
||||
}
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('exit 1 with no outcome file → generic throw naming the exit code', async () => {
|
||||
const harness = makeHarness('crash', `process.exit(1);\n`);
|
||||
await expect(runJobInChild(baseOpts(harness))).rejects.toThrow(/exit code=1/);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('SIGTERM-ignoring child: abort → group SIGKILL at the injected grace', async () => {
|
||||
const harness = makeHarness(
|
||||
'stubborn',
|
||||
`process.on('SIGTERM', () => {});\n` +
|
||||
`setInterval(() => {}, 1000);\n`, // never exits voluntarily
|
||||
);
|
||||
const abort = new AbortController();
|
||||
const ready = readiness('stubborn');
|
||||
const opts = { ...baseOpts(harness), abortSignal: abort.signal, killGraceMs: 400, env: ready.env };
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
abort.abort(new Error('timeout'));
|
||||
const started = Date.now();
|
||||
await expect(p).rejects.toThrow(/terminated after abort/);
|
||||
// Died via the SIGKILL escalation, not the 30s force-evict scale.
|
||||
expect(Date.now() - started).toBeLessThan(5_000);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('pre-aborted signal: child is terminated promptly', async () => {
|
||||
const harness = makeHarness(
|
||||
'prekilled',
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const abort = new AbortController();
|
||||
abort.abort(new Error('cancel'));
|
||||
const opts = { ...baseOpts(harness), abortSignal: abort.signal, killGraceMs: 400 };
|
||||
await expect(runJobInChild(opts)).rejects.toThrow(/terminated after abort/);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('spawn ENOENT → ChildSpawnInfraError (infra release, not a job defect)', async () => {
|
||||
const opts = {
|
||||
...baseOpts('/nonexistent'),
|
||||
invocation: { cmd: '/nonexistent/gbrain-binary', argsPrefix: [] },
|
||||
};
|
||||
await expect(runJobInChild(opts)).rejects.toBeInstanceOf(ChildSpawnInfraError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('worker shutdown: child finishes + reports during the drain window → normal success', async () => {
|
||||
const harness = makeHarness(
|
||||
'graceful-drain',
|
||||
`let done = false;\n` +
|
||||
`process.on('SIGTERM', () => {\n` +
|
||||
` writeOutcome({ outcome: 'success', result: { finishedDuringDrain: true } });\n` +
|
||||
` done = true; process.exit(0);\n` +
|
||||
`});\n` +
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const shutdown = new AbortController();
|
||||
const ready = readiness('graceful-drain');
|
||||
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 5_000, env: ready.env };
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
shutdown.abort(new Error('worker-shutdown'));
|
||||
const result = (await p) as Record<string, unknown>;
|
||||
expect(result.finishedDuringDrain).toBe(true);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('worker shutdown: child that cannot report → ChildWorkerShutdownError (no attempt burned)', async () => {
|
||||
const harness = makeHarness(
|
||||
'shutdown-stubborn',
|
||||
`process.on('SIGTERM', () => {});\n` +
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const shutdown = new AbortController();
|
||||
const ready = readiness('shutdown-stubborn');
|
||||
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 400, env: ready.env };
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
shutdown.abort(new Error('worker-shutdown'));
|
||||
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('ERROR outcome during worker shutdown → ChildWorkerShutdownError, not a burned attempt (adversarial P2)', async () => {
|
||||
// A cooperative handler that bails on shutdown and reports an error must
|
||||
// be RELEASED — punishing exactly the well-behaved handlers on every
|
||||
// deploy inverts the no-burn guarantee.
|
||||
const harness = makeHarness(
|
||||
'shutdown-error-report',
|
||||
`process.on('SIGTERM', () => {\n` +
|
||||
` writeOutcome({ outcome: 'error', errorKind: 'generic', message: 'aborted: shutdown' });\n` +
|
||||
` process.exit(0);\n` +
|
||||
`});\n` +
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const shutdown = new AbortController();
|
||||
const ready = readiness('shutdown-error-report');
|
||||
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 5_000, env: ready.env };
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
shutdown.abort(new Error('worker-shutdown'));
|
||||
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('watchdog drain (BOTH signals aborted, non-per-job reason) → shutdown class, not a burned attempt (adversarial P3)', async () => {
|
||||
const harness = makeHarness(
|
||||
'watchdog-stubborn',
|
||||
`process.on('SIGTERM', () => {});\n` +
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const abort = new AbortController();
|
||||
const shutdown = new AbortController();
|
||||
const ready = readiness('watchdog-stubborn');
|
||||
const opts = {
|
||||
...baseOpts(harness),
|
||||
abortSignal: abort.signal,
|
||||
shutdownSignal: shutdown.signal,
|
||||
killGraceMs: 400,
|
||||
env: ready.env,
|
||||
};
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
// gracefulShutdown('watchdog') aborts BOTH — shutdown classification must win.
|
||||
shutdown.abort(new Error('watchdog'));
|
||||
abort.abort(new Error('watchdog'));
|
||||
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('per-job reason (timeout) wins over a concurrent shutdown — attempt semantics preserved', async () => {
|
||||
const harness = makeHarness(
|
||||
'timeout-during-shutdown',
|
||||
`process.on('SIGTERM', () => {});\n` +
|
||||
`setInterval(() => {}, 1000);\n`,
|
||||
);
|
||||
const abort = new AbortController();
|
||||
const shutdown = new AbortController();
|
||||
const ready = readiness('timeout-during-shutdown');
|
||||
const opts = {
|
||||
...baseOpts(harness),
|
||||
abortSignal: abort.signal,
|
||||
shutdownSignal: shutdown.signal,
|
||||
killGraceMs: 400,
|
||||
env: ready.env,
|
||||
};
|
||||
const p = runJobInChild(opts);
|
||||
await ready.wait();
|
||||
shutdown.abort(new Error('worker-shutdown'));
|
||||
abort.abort(new Error('timeout')); // the JOB was targeted — not shutdown class
|
||||
await expect(p).rejects.toThrow(/terminated after abort/);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('bootstrap exit codes: 13 → ChildSpawnInfraError, 14 → ChildNotClaimedError (no burned attempts)', async () => {
|
||||
const usage = makeHarness('exit13', `process.exit(13);\n`);
|
||||
await expect(runJobInChild(baseOpts(usage))).rejects.toBeInstanceOf(ChildSpawnInfraError);
|
||||
|
||||
const notClaimed = makeHarness('exit14', `process.exit(14);\n`);
|
||||
await expect(runJobInChild(baseOpts(notClaimed))).rejects.toBeInstanceOf(ChildNotClaimedError);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
test('child pool-size env: a STRICTER user GBRAIN_POOL_SIZE is respected; explicit child override wins; invalid falls back', async () => {
|
||||
const harness = makeHarness(
|
||||
'pool-echo',
|
||||
`writeOutcome({ outcome: 'success', result: { poolSize: process.env.GBRAIN_POOL_SIZE } });\n` +
|
||||
`process.exit(0);\n`,
|
||||
);
|
||||
const run = (env: Record<string, string | undefined>) =>
|
||||
runJobInChild({ ...baseOpts(harness), env: { ...process.env, ...env } }) as Promise<{ poolSize: string }>;
|
||||
|
||||
expect((await run({ GBRAIN_POOL_SIZE: '2' })).poolSize).toBe('2'); // stricter user tuning respected
|
||||
expect((await run({ GBRAIN_POOL_SIZE: '10' })).poolSize).toBe('3'); // never raised above the child default
|
||||
expect((await run({ GBRAIN_POOL_SIZE: '2', GBRAIN_JOB_CHILD_POOL_SIZE: '5' })).poolSize).toBe('5'); // explicit knob wins
|
||||
expect((await run({ GBRAIN_JOB_CHILD_POOL_SIZE: 'abc' })).poolSize).toBe('3'); // invalid → default
|
||||
}, TEST_TIMEOUT_MS);
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* issue #6 — `runDrainRenewalTick` (cycle drain lock-renewal tick, extracted
|
||||
* from the inline setInterval in synthesize.ts). The previous inline tick had
|
||||
* no per-call timeout: a hung renewLock stacked one checked-out pool slot per
|
||||
* interval firing, forever. The extracted tick:
|
||||
*
|
||||
* - passes a per-call AbortSignal that is aborted when the timeout wins
|
||||
* (the losing UPDATE is cancelled, its slot released)
|
||||
* - invokes onLost exactly once when the token fence is lost (ok === false)
|
||||
* - swallows errors and timeouts (best-effort; the next tick retries)
|
||||
*
|
||||
* Hermetic: injected renewLock, no DB, no real cycle.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { runDrainRenewalTick } from '../src/core/cycle/synthesize.ts';
|
||||
|
||||
describe('drain-loop wiring (structural — the shape guard only covers worker.ts)', () => {
|
||||
const src = readFileSync(
|
||||
new URL('../src/core/cycle/synthesize.ts', import.meta.url),
|
||||
'utf-8',
|
||||
);
|
||||
test('the renewTimer interval is guarded and routes through runDrainRenewalTick', () => {
|
||||
expect(src).toContain('if (drainTickInFlight) return;');
|
||||
expect(src).toContain('void runDrainRenewalTick(');
|
||||
// The pre-fix inline shape (an unguarded queue.renewLock(...).then chain
|
||||
// inside the interval) must not come back.
|
||||
expect(src).not.toMatch(/setInterval\(\(\) => \{\s*\n\s*queue\.renewLock\(/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('db-lock heartbeat wiring (structural — issue #6 cancellation)', () => {
|
||||
const src = readFileSync(new URL('../src/core/db-lock.ts', import.meta.url), 'utf-8');
|
||||
test('withRefreshingLock aborts a per-tick signal into handle.refresh and guards re-entrancy', () => {
|
||||
expect(src).toContain('handle.refresh({ signal: tickAbort.signal })');
|
||||
expect(src).toContain('if (refreshTickInFlight) return;');
|
||||
// refresh() forwards the opts to executeRawDirect as the trailing arg.
|
||||
expect(src).toMatch(/executeRawDirect<\{ id: string \}>\([\s\S]*?refreshOpts,\s*\)/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runDrainRenewalTick (issue #6)', () => {
|
||||
test('successful renewal: signal not aborted, onLost not called', async () => {
|
||||
let seenSignal: AbortSignal | undefined;
|
||||
let lost = 0;
|
||||
await runDrainRenewalTick(
|
||||
async (_id, _tok, _ms, opts) => {
|
||||
seenSignal = opts?.signal;
|
||||
return true;
|
||||
},
|
||||
42,
|
||||
'tok',
|
||||
30_000,
|
||||
() => { lost += 1; },
|
||||
1_000,
|
||||
);
|
||||
expect(seenSignal).toBeInstanceOf(AbortSignal);
|
||||
expect(seenSignal!.aborted).toBe(false);
|
||||
expect(lost).toBe(0);
|
||||
});
|
||||
|
||||
test('lost token fence (ok=false): onLost called once', async () => {
|
||||
let lost = 0;
|
||||
await runDrainRenewalTick(
|
||||
async () => false,
|
||||
42,
|
||||
'tok',
|
||||
30_000,
|
||||
() => { lost += 1; },
|
||||
1_000,
|
||||
);
|
||||
expect(lost).toBe(1);
|
||||
});
|
||||
|
||||
test('hung renewLock: tick resolves at the timeout and aborts the per-call signal', async () => {
|
||||
let seenSignal: AbortSignal | undefined;
|
||||
let lost = 0;
|
||||
const started = Date.now();
|
||||
await runDrainRenewalTick(
|
||||
(_id, _tok, _ms, opts) => {
|
||||
seenSignal = opts?.signal;
|
||||
return new Promise<boolean>(() => { /* hangs forever */ });
|
||||
},
|
||||
42,
|
||||
'tok',
|
||||
30_000,
|
||||
() => { lost += 1; },
|
||||
50, // short timeout keeps the test fast
|
||||
);
|
||||
// Resolved via the timeout path (not the hung renewal).
|
||||
expect(Date.now() - started).toBeGreaterThanOrEqual(40);
|
||||
expect(seenSignal).toBeInstanceOf(AbortSignal);
|
||||
expect(seenSignal!.aborted).toBe(true);
|
||||
expect(lost).toBe(0); // timeout is NOT a lost fence
|
||||
});
|
||||
|
||||
test('throwing renewLock is swallowed (best-effort; next tick retries)', async () => {
|
||||
let lost = 0;
|
||||
await runDrainRenewalTick(
|
||||
async () => { throw new Error('CONNECTION_ENDED'); },
|
||||
42,
|
||||
'tok',
|
||||
30_000,
|
||||
() => { lost += 1; },
|
||||
1_000,
|
||||
);
|
||||
expect(lost).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -120,12 +120,17 @@ describe('claim stamps timeout_at (deadlineAtMs ground truth)', () => {
|
||||
|
||||
describe('deadline plumbing wiring (structural)', () => {
|
||||
const workerSrc = readFileSync(new URL('../src/core/minions/worker.ts', import.meta.url), 'utf-8');
|
||||
// The context builder was extracted from executeJob into job-context.ts
|
||||
// (shared with `jobs run-child` for process isolation) — the deadlineAtMs
|
||||
// derivation lives there now; worker.ts calls buildJobContext.
|
||||
const jobContextSrc = readFileSync(new URL('../src/core/minions/job-context.ts', import.meta.url), 'utf-8');
|
||||
const jobsSrc = readFileSync(new URL('../src/commands/jobs.ts', import.meta.url), 'utf-8');
|
||||
const cycleSrc = readFileSync(new URL('../src/core/cycle.ts', import.meta.url), 'utf-8');
|
||||
const patternsSrc = readFileSync(new URL('../src/core/cycle/patterns.ts', import.meta.url), 'utf-8');
|
||||
|
||||
test('worker exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
|
||||
expect(workerSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
|
||||
test('job context exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
|
||||
expect(jobContextSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
|
||||
expect(workerSrc).toContain('buildJobContext(');
|
||||
});
|
||||
|
||||
test('worker arms its abort timer from timeout_at when present (one absolute deadline)', () => {
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* issue #6 — `resolveMaxLifetimeSeconds`: explicit, env-overridable
|
||||
* max_lifetime for all four postgres() call sites.
|
||||
*
|
||||
* NOT a behavior change at default: postgres.js (verified against 3.4.9)
|
||||
* already defaults max_lifetime to `60 * (30 + Math.random() * 30)`. This
|
||||
* resolver makes the value explicit and adds the GBRAIN_POOL_MAX_LIFETIME_S
|
||||
* incident escape hatch (0 disables recycling; N = seconds).
|
||||
*
|
||||
* Hermetic: resolver only — env injected as a param (rule R1), no pools.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach } from 'bun:test';
|
||||
import {
|
||||
resolveMaxLifetimeSeconds,
|
||||
_resetMaxLifetimeWarningForTests,
|
||||
} from '../src/core/db.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
beforeEach(() => {
|
||||
_resetMaxLifetimeWarningForTests();
|
||||
});
|
||||
|
||||
describe('resolveMaxLifetimeSeconds', () => {
|
||||
test('default (no env): a per-CONNECTION jitter FUNCTION, 30-60 minutes', () => {
|
||||
// Must be a function, not a pre-evaluated number: postgres.js re-evaluates
|
||||
// a function default per connection, so connections in one pool get
|
||||
// independent recycle deadlines instead of a synchronized reconnect spike
|
||||
// (data-migration specialist).
|
||||
const v = resolveMaxLifetimeSeconds({});
|
||||
expect(typeof v).toBe('function');
|
||||
const fn = v as () => number;
|
||||
for (let i = 0; i < 20; i++) {
|
||||
const n = fn();
|
||||
expect(Number.isInteger(n)).toBe(true);
|
||||
expect(n).toBeGreaterThanOrEqual(1800);
|
||||
expect(n).toBeLessThanOrEqual(3600);
|
||||
}
|
||||
});
|
||||
|
||||
test('env override: positive integer seconds honored verbatim', () => {
|
||||
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '900' })).toBe(900);
|
||||
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '1' })).toBe(1);
|
||||
});
|
||||
|
||||
test('env 0 disables recycling (null — postgres.js accepts null)', () => {
|
||||
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '0' })).toBeNull();
|
||||
});
|
||||
|
||||
test('empty string falls through to the default (function)', () => {
|
||||
const v = resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '' });
|
||||
expect(typeof v).toBe('function');
|
||||
});
|
||||
|
||||
test('invalid values warn once on stderr and fall back to the default', () => {
|
||||
const writes: string[] = [];
|
||||
const realWrite = process.stderr.write.bind(process.stderr);
|
||||
(process.stderr as { write: unknown }).write = (chunk: string) => {
|
||||
writes.push(String(chunk));
|
||||
return true;
|
||||
};
|
||||
try {
|
||||
for (const bad of ['abc', '-5', '3.5', 'NaN']) {
|
||||
const v = resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: bad });
|
||||
expect(typeof v).toBe('function');
|
||||
}
|
||||
} finally {
|
||||
(process.stderr as { write: unknown }).write = realWrite;
|
||||
}
|
||||
// warn-once latch: 4 bad values, exactly 1 warning
|
||||
const warnings = writes.filter((w) => w.includes('GBRAIN_POOL_MAX_LIFETIME_S'));
|
||||
expect(warnings.length).toBe(1);
|
||||
});
|
||||
|
||||
test('jitter varies per CONNECTION (thundering-herd protection)', () => {
|
||||
const fn = resolveMaxLifetimeSeconds({}) as () => number;
|
||||
const values = new Set<number>();
|
||||
for (let i = 0; i < 30; i++) values.add(fn());
|
||||
expect(values.size).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
test('wiring: the env override reaches a REAL constructed pool (ConnectionManager read pool)', async () => {
|
||||
// postgres() is lazy — constructing the pool performs no I/O, so this
|
||||
// pins the construction seam without a database. Without this, the
|
||||
// resolver could be green while GBRAIN_POOL_MAX_LIFETIME_S is silently
|
||||
// dead at every call site (adversarial-review vacuity finding).
|
||||
const { ConnectionManager } = await import('../src/core/connection-manager.ts');
|
||||
const { endPoolBounded } = await import('../src/core/db.ts');
|
||||
await withEnv({ GBRAIN_POOL_MAX_LIFETIME_S: '900' }, async () => {
|
||||
const cm = new ConnectionManager({
|
||||
url: 'postgresql://user@127.0.0.1:5/never-connected',
|
||||
});
|
||||
const pool = await cm.getReadPool();
|
||||
try {
|
||||
expect(
|
||||
(pool as unknown as { options: { max_lifetime: number | null } }).options.max_lifetime,
|
||||
).toBe(900);
|
||||
} finally {
|
||||
await endPoolBounded(pool);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test('wiring: the default reaches a real pool as a FUNCTION (per-connection jitter)', async () => {
|
||||
const { ConnectionManager } = await import('../src/core/connection-manager.ts');
|
||||
const { endPoolBounded } = await import('../src/core/db.ts');
|
||||
await withEnv({ GBRAIN_POOL_MAX_LIFETIME_S: undefined }, async () => {
|
||||
const cm = new ConnectionManager({
|
||||
url: 'postgresql://user@127.0.0.1:5/never-connected',
|
||||
});
|
||||
const pool = await cm.getReadPool();
|
||||
try {
|
||||
const v = (pool as unknown as { options: { max_lifetime: unknown } }).options.max_lifetime;
|
||||
expect(typeof v).toBe('function');
|
||||
} finally {
|
||||
await endPoolBounded(pool);
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,136 @@
|
||||
/**
|
||||
* issue #6 — `runDbProbe` verdict matrix (hermetic; injected deps only).
|
||||
*
|
||||
* read OK → { ok: true }
|
||||
* read FAIL + direct OK → pool_starved (honest disjunction wording)
|
||||
* read FAIL + direct FAIL → server_unreachable
|
||||
* read FAIL + no direct lane → unknown
|
||||
* hung probes → cancelled via their AbortSignals
|
||||
* diagnostics absent/throwing → fail-open (verdict still produced)
|
||||
* tracked=0 subset → message points at untracked traffic
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { runDbProbe } from '../src/core/minions/db-probe.ts';
|
||||
|
||||
const FAST = { timeoutMs: 5_000, directTimeoutMs: 5_000 };
|
||||
const SHORT = { timeoutMs: 50, directTimeoutMs: 50 };
|
||||
|
||||
describe('runDbProbe verdict matrix', () => {
|
||||
test('read OK → ok:true, no verdict', async () => {
|
||||
const res = await runDbProbe({ probeRead: async () => {}, ...FAST });
|
||||
expect(res).toEqual({ ok: true });
|
||||
});
|
||||
|
||||
test('read FAIL + direct OK → pool_starved with honest-disjunction wording', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('probe timeout after 10000ms'); },
|
||||
probeDirect: async () => {},
|
||||
getDiagnostics: () => ({
|
||||
tracked: { raw: 9, direct: 0, reserved: 1, tx: 0 },
|
||||
poolMax: 10,
|
||||
}),
|
||||
...FAST,
|
||||
});
|
||||
expect(res.ok).toBe(false);
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('pool_starved');
|
||||
expect(res.detail).toContain('server IS reachable');
|
||||
// Codex-2 #2: the message must NOT claim to distinguish client pool
|
||||
// exhaustion from a pooler-layer fault.
|
||||
expect(res.detail).toContain('client pool exhaustion or a pooler-layer fault');
|
||||
expect(res.detail).toContain('raw=9');
|
||||
expect(res.detail).toContain('read pool max 10');
|
||||
// Codex-2 #3: no invented waiter arithmetic anywhere in the message.
|
||||
expect(res.detail).not.toMatch(/waiting/i);
|
||||
expect(res.detail).toContain('subset');
|
||||
});
|
||||
|
||||
test('read FAIL + direct FAIL → server_unreachable', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('probe timeout after 10000ms'); },
|
||||
probeDirect: async () => { throw new Error('connect ECONNREFUSED'); },
|
||||
...FAST,
|
||||
});
|
||||
expect(res.ok).toBe(false);
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('server_unreachable');
|
||||
expect(res.detail).toContain('ECONNREFUSED');
|
||||
});
|
||||
|
||||
test('read FAIL + no direct lane → unknown', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('boom'); },
|
||||
...FAST,
|
||||
});
|
||||
expect(res.ok).toBe(false);
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('unknown');
|
||||
expect(res.detail).toContain('no direct lane');
|
||||
});
|
||||
|
||||
test('hung read probe: cancelled via its signal at the deadline', async () => {
|
||||
let readSignal: AbortSignal | undefined;
|
||||
const res = await runDbProbe({
|
||||
probeRead: (signal) => {
|
||||
readSignal = signal;
|
||||
return new Promise<never>(() => {});
|
||||
},
|
||||
...SHORT,
|
||||
});
|
||||
expect(res.ok).toBe(false);
|
||||
expect(readSignal?.aborted).toBe(true);
|
||||
});
|
||||
|
||||
test('hung direct probe: cancelled at its own (shorter) deadline', async () => {
|
||||
let directSignal: AbortSignal | undefined;
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('read dead'); },
|
||||
probeDirect: (signal) => {
|
||||
directSignal = signal;
|
||||
return new Promise<never>(() => {});
|
||||
},
|
||||
...SHORT,
|
||||
});
|
||||
expect(res.ok).toBe(false);
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('server_unreachable');
|
||||
expect(directSignal?.aborted).toBe(true);
|
||||
});
|
||||
|
||||
test('tracked=0 subset: message names untracked template traffic + runbook', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('probe timeout'); },
|
||||
probeDirect: async () => {},
|
||||
getDiagnostics: () => ({
|
||||
tracked: { raw: 0, direct: 0, reserved: 0, tx: 0 },
|
||||
poolMax: 10,
|
||||
}),
|
||||
...FAST,
|
||||
});
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.detail).toContain('untracked');
|
||||
expect(res.detail).toContain('queue-operations-runbook');
|
||||
});
|
||||
|
||||
test('diagnostics absent → verdict still produced (fail-open)', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('probe timeout'); },
|
||||
probeDirect: async () => {},
|
||||
...FAST,
|
||||
});
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('pool_starved');
|
||||
});
|
||||
|
||||
test('diagnostics THROWING → verdict still produced (fail-open)', async () => {
|
||||
const res = await runDbProbe({
|
||||
probeRead: async () => { throw new Error('probe timeout'); },
|
||||
probeDirect: async () => {},
|
||||
getDiagnostics: () => { throw new Error('gauge exploded'); },
|
||||
...FAST,
|
||||
});
|
||||
if (res.ok) throw new Error('unreachable');
|
||||
expect(res.verdict).toBe('pool_starved');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,133 @@
|
||||
/**
|
||||
* issue #5 — process isolation against REAL Postgres (DATABASE_URL-gated;
|
||||
* wired into .github/workflows/e2e.yml tier1 — e2e.yml runs only explicitly
|
||||
* NAMED files, so an unwired e2e file is silent coverage loss).
|
||||
*
|
||||
* Legs:
|
||||
* 1. Worker-level concurrency (codex-2 #6): a real MinionWorker with
|
||||
* isolation on drains 6 jobs at concurrency 3 through real child
|
||||
* processes against the real pooler/DB — the child-pool topology the
|
||||
* per-process pool math describes.
|
||||
* 2. Real CLI entrypoint: `bun src/cli.ts jobs run-child` with the env
|
||||
* contract against a genuinely claimed job — proves engine bootstrap,
|
||||
* handler registry (quiet), token validation, and the outcome protocol
|
||||
* end-to-end. Uses the real 'orphans' handler (cheap on a fresh DB).
|
||||
* 3. Stuck-child kill: a SIGTERM-ignoring child is group-SIGKILLed and the
|
||||
* worker keeps claiming.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, rmSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import { MinionQueue } from '../../src/core/minions/queue.ts';
|
||||
import { MinionWorker } from '../../src/core/minions/worker.ts';
|
||||
import { decodeChildOutcomeFile, CHILD_ENV } from '../../src/core/minions/job-isolation.ts';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
|
||||
const FIXTURE = resolve(import.meta.dir, '..', 'fixtures', 'fake-run-child.mjs');
|
||||
const CLI = resolve(import.meta.dir, '..', '..', 'src', 'cli.ts');
|
||||
|
||||
const describeDb = hasDatabase() ? describe : describe.skip;
|
||||
|
||||
let queue: MinionQueue;
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!hasDatabase()) return;
|
||||
await setupDB();
|
||||
queue = new MinionQueue(getEngine());
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-isolation-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (!hasDatabase()) return;
|
||||
await teardownDB();
|
||||
if (tmp) rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function rowStatus(id: number): Promise<string> {
|
||||
const rows = await getEngine().executeRaw<{ status: string }>(
|
||||
'SELECT status FROM minion_jobs WHERE id = $1',
|
||||
[id],
|
||||
);
|
||||
return rows[0]?.status ?? 'missing';
|
||||
}
|
||||
|
||||
describeDb('process isolation on real Postgres', () => {
|
||||
test('concurrency 3: six isolated jobs drain through real children', async () => {
|
||||
await withEnv({ FAKE_RUN_CHILD_MODE: 'success' }, async () => {
|
||||
const jobs = [];
|
||||
for (let i = 0; i < 6; i++) jobs.push(await queue.add('isotest', { i }));
|
||||
|
||||
const worker = new MinionWorker(getEngine(), {
|
||||
queue: 'default',
|
||||
concurrency: 3,
|
||||
pollInterval: 50,
|
||||
healthCheckInterval: 0,
|
||||
maxRssMb: 0,
|
||||
jobIsolation: 'process',
|
||||
childCliInvocation: { cmd: process.execPath, argsPrefix: [FIXTURE] },
|
||||
});
|
||||
worker.register('isotest', async () => {
|
||||
throw new Error('parent handler must not run when isolated');
|
||||
});
|
||||
|
||||
const run = worker.start();
|
||||
const deadline = Date.now() + 30_000;
|
||||
try {
|
||||
while (Date.now() < deadline) {
|
||||
const statuses = await Promise.all(jobs.map((j) => rowStatus(j.id)));
|
||||
if (statuses.every((s) => s === 'completed')) break;
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
} finally {
|
||||
worker.stop();
|
||||
await run;
|
||||
}
|
||||
const statuses = await Promise.all(jobs.map((j) => rowStatus(j.id)));
|
||||
expect(statuses).toEqual(['completed', 'completed', 'completed', 'completed', 'completed', 'completed']);
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
test('real CLI run-child: engine bootstrap + protocol end-to-end', async () => {
|
||||
const job = await queue.add('orphans', {});
|
||||
const claimed = await queue.claim('e2e-cli-tok', 60_000, 'default', ['orphans']);
|
||||
expect(claimed?.id).toBe(job.id);
|
||||
|
||||
const resultPath = join(tmp, `cli-${job.id}.json`);
|
||||
const res = spawnSync(
|
||||
process.execPath,
|
||||
[CLI, 'jobs', 'run-child', '--job-id', String(job.id)],
|
||||
{
|
||||
env: {
|
||||
...process.env,
|
||||
GBRAIN_DATABASE_URL: process.env.DATABASE_URL,
|
||||
GBRAIN_TEST_ALLOW_DATABASE_URL: '1',
|
||||
[CHILD_ENV.lockToken]: 'e2e-cli-tok',
|
||||
[CHILD_ENV.resultPath]: resultPath,
|
||||
[CHILD_ENV.parentPid]: String(process.pid),
|
||||
},
|
||||
timeout: 60_000,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
},
|
||||
);
|
||||
if (res.status !== 0) {
|
||||
throw new Error(
|
||||
`run-child exited ${res.status}: ${String(res.stderr)}\n${String(res.stdout)}`,
|
||||
);
|
||||
}
|
||||
expect(existsSync(resultPath)).toBe(true);
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
// The protocol completed; the real handler's own verdict (success or a
|
||||
// reported error on a bare DB) is out of scope for this leg.
|
||||
expect(outcome.outcome === 'success' || outcome.outcome === 'error').toBe(true);
|
||||
}, 90_000);
|
||||
|
||||
// The stuck-child group-SIGKILL path is pinned with REAL processes in
|
||||
// test/child-job-runner.test.ts (SIGTERM-ignorer → SIGKILL at grace) and
|
||||
// the crash/burn semantics in test/worker-job-isolation.test.ts — this
|
||||
// lane deliberately doesn't duplicate them against the shared CI DB.
|
||||
});
|
||||
Vendored
+48
@@ -0,0 +1,48 @@
|
||||
// Fake `jobs run-child` for the isolation tests (worker-job-isolation.test.ts
|
||||
// + test/e2e/job-isolation.test.ts): honors the isolation env contract
|
||||
// without needing a compiled gbrain binary or a Postgres engine.
|
||||
// Mode via FAKE_RUN_CHILD_MODE: success | error | exit15 | crash.
|
||||
import { writeFileSync, renameSync } from 'node:fs';
|
||||
|
||||
const resultPath = process.env.GBRAIN_JOB_RESULT_PATH;
|
||||
const mode = process.env.FAKE_RUN_CHILD_MODE ?? 'success';
|
||||
|
||||
function writeOutcome(o) {
|
||||
writeFileSync(resultPath + '.tmp', JSON.stringify(o));
|
||||
renameSync(resultPath + '.tmp', resultPath);
|
||||
}
|
||||
|
||||
if (!resultPath) {
|
||||
process.stderr.write('[fake-run-child] missing GBRAIN_JOB_RESULT_PATH\n');
|
||||
process.exit(13);
|
||||
}
|
||||
|
||||
if (mode === 'success') {
|
||||
writeOutcome({
|
||||
outcome: 'success',
|
||||
result: {
|
||||
fromChild: true,
|
||||
token: process.env.GBRAIN_JOB_LOCK_TOKEN ?? null,
|
||||
argv: process.argv.slice(2),
|
||||
},
|
||||
});
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (mode === 'error') {
|
||||
writeOutcome({
|
||||
outcome: 'error',
|
||||
errorKind: 'generic',
|
||||
message: 'fake child handler failure',
|
||||
});
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (mode === 'exit15') {
|
||||
// Simulates a result-write failure (JOB_CHILD_EXIT_RESULT_WRITE_FAILED):
|
||||
// handler ran, outcome could not be persisted.
|
||||
process.exit(15);
|
||||
}
|
||||
|
||||
// crash: no outcome file
|
||||
process.exit(1);
|
||||
@@ -0,0 +1,256 @@
|
||||
/**
|
||||
* issue #5 — job-isolation protocol unit tests.
|
||||
*
|
||||
* Covers: outcome round-trip, missing/malformed/oversized decode paths
|
||||
* (oversize → UnrecoverableError: deterministic dead on attempt 1, never
|
||||
* silent truncation), instanceof reconstruction for the two error classes
|
||||
* executeJob branches on, child-CLI invocation resolution (env override /
|
||||
* binary / bun-dev fallback / fail-fast null), and killProcessGroup against
|
||||
* REAL detached processes — including the grandchild-death guarantee that
|
||||
* motivated group signaling (SIGKILL on a wrapper pid alone orphans the
|
||||
* handler; Bun rejects negative pids so the /bin/kill fallback is what
|
||||
* actually runs under `bun test`, making this a real-runtime regression
|
||||
* test for oven-sh/bun#15791).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
CHILD_OUTCOME_MAX_BYTES,
|
||||
buildChildArgs,
|
||||
decodeChildOutcomeFile,
|
||||
encodeHandlerError,
|
||||
killProcessGroup,
|
||||
reconstructHandlerError,
|
||||
resolveChildCliInvocation,
|
||||
writeChildOutcomeFile,
|
||||
type ChildOutcome,
|
||||
} from '../src/core/minions/job-isolation.ts';
|
||||
import { UnrecoverableError } from '../src/core/minions/types.ts';
|
||||
import { RateLeaseUnavailableError } from '../src/core/minions/handlers/subagent.ts';
|
||||
|
||||
function tmpFile(name: string): { dir: string; path: string } {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-isolation-test-'));
|
||||
return { dir, path: join(dir, name) };
|
||||
}
|
||||
|
||||
describe('outcome file round-trip', () => {
|
||||
test('success outcome survives write + decode; write is atomic (no .tmp left)', () => {
|
||||
const { dir, path } = tmpFile('outcome.json');
|
||||
try {
|
||||
writeChildOutcomeFile(path, { outcome: 'success', result: { pages: 3, ok: true } });
|
||||
expect(existsSync(`${path}.tmp`)).toBe(false);
|
||||
const decoded = decodeChildOutcomeFile(path);
|
||||
expect(decoded).toEqual({ outcome: 'success', result: { pages: 3, ok: true } });
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('missing file → generic Error naming the crash class', () => {
|
||||
const { dir, path } = tmpFile('never-written.json');
|
||||
try {
|
||||
expect(() => decodeChildOutcomeFile(path)).toThrow(/without writing its outcome file/);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('malformed JSON → generic Error with byte count, NEVER file content', () => {
|
||||
const { dir, path } = tmpFile('garbage.json');
|
||||
try {
|
||||
writeFileSync(path, 'sk-secret-key-do-not-leak {{{', 'utf8');
|
||||
try {
|
||||
decodeChildOutcomeFile(path);
|
||||
throw new Error('should have thrown');
|
||||
} catch (e) {
|
||||
const msg = (e as Error).message;
|
||||
expect(msg).toContain('not valid JSON');
|
||||
expect(msg).toContain('bytes');
|
||||
expect(msg).not.toContain('sk-secret');
|
||||
}
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('oversize file → UnrecoverableError naming the cap (dead on attempt 1)', () => {
|
||||
const { dir, path } = tmpFile('huge.json');
|
||||
try {
|
||||
writeFileSync(path, '{"outcome":"success","result":"xx"}', 'utf8');
|
||||
try {
|
||||
decodeChildOutcomeFile(path, 16); // tiny injected cap keeps the test fast
|
||||
throw new Error('should have thrown');
|
||||
} catch (e) {
|
||||
expect(e).toBeInstanceOf(UnrecoverableError);
|
||||
expect((e as Error).message).toContain('outcome cap');
|
||||
}
|
||||
// Default cap sanity.
|
||||
expect(CHILD_OUTCOME_MAX_BYTES).toBe(32 * 1024 * 1024);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('unrecognized shape → generic Error (byte count only)', () => {
|
||||
const { dir, path } = tmpFile('weird.json');
|
||||
try {
|
||||
writeFileSync(path, '{"totally":"unrelated"}', 'utf8');
|
||||
expect(() => decodeChildOutcomeFile(path)).toThrow(/unrecognized shape/);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('handler-error encode → reconstruct (instanceof parity with inline mode)', () => {
|
||||
test('UnrecoverableError survives the boundary', () => {
|
||||
const enc = encodeHandlerError(new UnrecoverableError('bad config, never retry'));
|
||||
expect(enc.outcome).toBe('error');
|
||||
const rebuilt = reconstructHandlerError(enc as Extract<ChildOutcome, { outcome: 'error' }>);
|
||||
expect(rebuilt).toBeInstanceOf(UnrecoverableError);
|
||||
expect(rebuilt.message).toBe('bad config, never retry');
|
||||
});
|
||||
|
||||
test('RateLeaseUnavailableError survives with lease fields', () => {
|
||||
const enc = encodeHandlerError(new RateLeaseUnavailableError('anthropic', 4, 4));
|
||||
const rebuilt = reconstructHandlerError(enc as Extract<ChildOutcome, { outcome: 'error' }>);
|
||||
expect(rebuilt).toBeInstanceOf(RateLeaseUnavailableError);
|
||||
const lease = rebuilt as RateLeaseUnavailableError;
|
||||
expect(lease.key).toBe('anthropic');
|
||||
expect(lease.active).toBe(4);
|
||||
expect(lease.max).toBe(4);
|
||||
});
|
||||
|
||||
test('generic Error carries message + child stack; unknown kinds degrade to generic', () => {
|
||||
const boom = new Error('handler exploded');
|
||||
const enc = encodeHandlerError(boom) as Extract<ChildOutcome, { outcome: 'error' }>;
|
||||
const rebuilt = reconstructHandlerError(enc);
|
||||
expect(rebuilt).toBeInstanceOf(Error);
|
||||
expect(rebuilt).not.toBeInstanceOf(UnrecoverableError);
|
||||
expect(rebuilt.message).toBe('handler exploded');
|
||||
expect((rebuilt as Error & { childStack?: string }).childStack).toContain('handler exploded');
|
||||
|
||||
const weird = reconstructHandlerError({
|
||||
outcome: 'error',
|
||||
errorKind: 'generic',
|
||||
message: 'from a hostile file',
|
||||
});
|
||||
expect(weird).toBeInstanceOf(Error);
|
||||
expect(weird).not.toBeInstanceOf(UnrecoverableError);
|
||||
});
|
||||
|
||||
test('non-Error throws (strings) encode without crashing', () => {
|
||||
const enc = encodeHandlerError('plain string throw');
|
||||
expect(enc.outcome).toBe('error');
|
||||
if (enc.outcome === 'error') expect(enc.message).toBe('plain string throw');
|
||||
});
|
||||
});
|
||||
|
||||
describe('child CLI invocation resolution', () => {
|
||||
test('env override wins', () => {
|
||||
const inv = resolveChildCliInvocation(
|
||||
{ GBRAIN_JOB_CHILD_CLI: '/opt/custom/gbrain' },
|
||||
'/usr/bin/bun',
|
||||
'/repo/src/cli.ts',
|
||||
() => '/usr/local/bin/gbrain',
|
||||
);
|
||||
expect(inv).toEqual({ cmd: '/opt/custom/gbrain', argsPrefix: [] });
|
||||
});
|
||||
|
||||
test('compiled binary next', () => {
|
||||
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => '/usr/local/bin/gbrain');
|
||||
expect(inv).toEqual({ cmd: '/usr/local/bin/gbrain', argsPrefix: [] });
|
||||
});
|
||||
|
||||
test('bun-dev fallback when no binary resolves', () => {
|
||||
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => null);
|
||||
expect(inv).toEqual({ cmd: '/usr/bin/bun', argsPrefix: ['/repo/src/cli.ts'] });
|
||||
});
|
||||
|
||||
test('nothing resolves → null (caller must fail fast at startup)', () => {
|
||||
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/dist/other.js', () => null);
|
||||
expect(inv).toBeNull();
|
||||
});
|
||||
|
||||
test('throwing binary resolver falls through to the dev fallback', () => {
|
||||
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => {
|
||||
throw new Error('which failed');
|
||||
});
|
||||
expect(inv).toEqual({ cmd: '/usr/bin/bun', argsPrefix: ['/repo/src/cli.ts'] });
|
||||
});
|
||||
|
||||
test('child argv shape', () => {
|
||||
expect(buildChildArgs(42)).toEqual(['jobs', 'run-child', '--job-id', '42']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('killProcessGroup (real detached processes; Bun negative-pid fallback)', () => {
|
||||
async function waitFor(cond: () => boolean, ms: number): Promise<boolean> {
|
||||
const deadline = Date.now() + ms;
|
||||
while (Date.now() < deadline) {
|
||||
if (cond()) return true;
|
||||
await new Promise((r) => setTimeout(r, 25));
|
||||
}
|
||||
return cond();
|
||||
}
|
||||
|
||||
function alive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
test('group SIGKILL kills the child AND its grandchild', async () => {
|
||||
const { dir, path: pidFile } = tmpFile('grandchild.pid');
|
||||
try {
|
||||
// Child shell spawns a long-lived grandchild and records its pid.
|
||||
const child = spawn('/bin/sh', ['-c', `sleep 300 & echo $! > ${pidFile}; wait`], {
|
||||
detached: true,
|
||||
stdio: 'ignore',
|
||||
});
|
||||
expect(child.pid).toBeGreaterThan(0);
|
||||
const gotPid = await waitFor(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 3_000);
|
||||
expect(gotPid).toBe(true);
|
||||
const grandchildPid = parseInt(readFileSync(pidFile, 'utf8').trim(), 10);
|
||||
expect(alive(child.pid!)).toBe(true);
|
||||
expect(alive(grandchildPid)).toBe(true);
|
||||
|
||||
killProcessGroup(child.pid!, 'SIGKILL');
|
||||
|
||||
const bothDead = await waitFor(() => !alive(child.pid!) && !alive(grandchildPid), 3_000);
|
||||
expect(bothDead).toBe(true);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}, 15_000);
|
||||
|
||||
test('SIGTERM delivery to a live group returns true; dead group returns false', async () => {
|
||||
const child = spawn('/bin/sh', ['-c', 'sleep 300'], { detached: true, stdio: 'ignore' });
|
||||
const delivered = killProcessGroup(child.pid!, 'SIGTERM');
|
||||
expect(delivered).toBe(true);
|
||||
await waitFor(() => {
|
||||
try {
|
||||
process.kill(child.pid!, 0);
|
||||
return false;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
}, 3_000);
|
||||
// Group is gone now — a second signal reports not-delivered.
|
||||
expect(killProcessGroup(child.pid!, 'SIGKILL')).toBe(false);
|
||||
}, 15_000);
|
||||
|
||||
test('nonsense pids are refused without throwing', () => {
|
||||
expect(killProcessGroup(0, 'SIGTERM')).toBe(false);
|
||||
expect(killProcessGroup(1, 'SIGTERM')).toBe(false);
|
||||
expect(killProcessGroup(-5, 'SIGTERM')).toBe(false);
|
||||
expect(killProcessGroup(1.5 as unknown as number, 'SIGTERM')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,38 @@
|
||||
/**
|
||||
* issue #5 — `parseJobIsolationFlag` (jobs-nice-flag.test.ts pattern: pure
|
||||
* parser, env injected as the 2nd param so tests never mutate process.env).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { parseJobIsolationFlag } from '../src/commands/jobs.ts';
|
||||
|
||||
describe('parseJobIsolationFlag', () => {
|
||||
test('default: inline', () => {
|
||||
expect(parseJobIsolationFlag([], {})).toBe('inline');
|
||||
});
|
||||
|
||||
test('space form', () => {
|
||||
expect(parseJobIsolationFlag(['--job-isolation', 'process'], {})).toBe('process');
|
||||
expect(parseJobIsolationFlag(['--job-isolation', 'inline'], {})).toBe('inline');
|
||||
});
|
||||
|
||||
test('= form', () => {
|
||||
expect(parseJobIsolationFlag(['--job-isolation=process'], {})).toBe('process');
|
||||
expect(parseJobIsolationFlag(['--job-isolation=inline'], {})).toBe('inline');
|
||||
});
|
||||
|
||||
test('env fallback, flag wins over env', () => {
|
||||
expect(parseJobIsolationFlag([], { GBRAIN_JOB_ISOLATION: 'process' })).toBe('process');
|
||||
expect(
|
||||
parseJobIsolationFlag(['--job-isolation', 'inline'], { GBRAIN_JOB_ISOLATION: 'process' }),
|
||||
).toBe('inline');
|
||||
});
|
||||
|
||||
test('empty env value falls through to the default', () => {
|
||||
expect(parseJobIsolationFlag([], { GBRAIN_JOB_ISOLATION: '' })).toBe('inline');
|
||||
});
|
||||
|
||||
test('other flags are untouched', () => {
|
||||
expect(parseJobIsolationFlag(['--queue', 'q', '--concurrency', '3'], {})).toBe('inline');
|
||||
});
|
||||
});
|
||||
@@ -74,6 +74,9 @@ describe('jobs --help and jobs <subcommand> --help print real help, never the st
|
||||
expect(out).toContain('--max-rss');
|
||||
expect(out).toContain('--health-interval');
|
||||
expect(out).toContain('GBRAIN_WORKER_CONCURRENCY');
|
||||
// issue #5 process isolation: flag + env fallback documented.
|
||||
expect(out).toContain('--job-isolation');
|
||||
expect(out).toContain('GBRAIN_JOB_ISOLATION');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
// A real `jobs work` on this fixture would refuse (thin-client/engine
|
||||
// path) or start a daemon; either output would differ from the help.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* issue #6 — `MinionQueue.renewLock` forwards its optional AbortSignal to
|
||||
* `engine.executeRawDirect` so the lock-renewal tick's timeout race can
|
||||
* CANCEL a hung UPDATE (postgres.js `.cancel()`) instead of abandoning it on
|
||||
* a checked-out pool slot.
|
||||
*
|
||||
* Hermetic: stub engine object literal (`as unknown as BrainEngine`), no DB —
|
||||
* the worker-conn-resilience-1720 pattern.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
function makeCaptureEngine() {
|
||||
const calls: Array<{
|
||||
sql: string;
|
||||
params: unknown[] | undefined;
|
||||
opts: { signal?: AbortSignal } | undefined;
|
||||
}> = [];
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRawDirect: async (
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
) => {
|
||||
calls.push({ sql, params, opts });
|
||||
return [{ id: 1 }];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
return { engine, calls };
|
||||
}
|
||||
|
||||
describe('MinionQueue.renewLock signal forwarding (issue #6)', () => {
|
||||
test('forwards opts.signal to executeRawDirect', async () => {
|
||||
const { engine, calls } = makeCaptureEngine();
|
||||
const queue = new MinionQueue(engine);
|
||||
const ac = new AbortController();
|
||||
|
||||
const ok = await queue.renewLock(7, 'tok-xyz', 30_000, { signal: ac.signal });
|
||||
|
||||
expect(ok).toBe(true);
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0].sql).toContain('UPDATE minion_jobs SET lock_until');
|
||||
expect(calls[0].params).toEqual([30_000, 7, 'tok-xyz']);
|
||||
expect(calls[0].opts?.signal).toBe(ac.signal);
|
||||
});
|
||||
|
||||
test('legacy 3-arg call still works (opts undefined)', async () => {
|
||||
const { engine, calls } = makeCaptureEngine();
|
||||
const queue = new MinionQueue(engine);
|
||||
|
||||
const ok = await queue.renewLock(7, 'tok-xyz', 30_000);
|
||||
|
||||
expect(ok).toBe(true);
|
||||
expect(calls[0].opts).toBeUndefined();
|
||||
});
|
||||
|
||||
test('token-fence miss returns false regardless of signal', async () => {
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRawDirect: async () => [],
|
||||
} as unknown as BrainEngine;
|
||||
const queue = new MinionQueue(engine);
|
||||
const ac = new AbortController();
|
||||
|
||||
const ok = await queue.renewLock(7, 'tok-stale', 30_000, { signal: ac.signal });
|
||||
expect(ok).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* issue #6 — CheckoutGauge (pure) + the PostgresEngine gauge seams.
|
||||
*
|
||||
* The leak guard matters most: a counter that isn't released on a THROWING
|
||||
* query drifts upward forever and turns the diagnostics into fiction. The
|
||||
* engine seams use the Object.create(PostgresEngine.prototype) + fake-pool
|
||||
* pattern (no real DB).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { CheckoutGauge } from '../src/core/pool-gauge.ts';
|
||||
import { PostgresEngine } from '../src/core/postgres-engine.ts';
|
||||
|
||||
describe('CheckoutGauge (pure)', () => {
|
||||
test('acquire/release round-trip per kind', () => {
|
||||
const g = new CheckoutGauge();
|
||||
g.acquire('raw');
|
||||
g.acquire('raw');
|
||||
g.acquire('direct');
|
||||
g.acquire('reserved');
|
||||
g.acquire('tx');
|
||||
expect(g.snapshot()).toEqual({ raw: 2, direct: 1, reserved: 1, tx: 1 });
|
||||
g.release('raw');
|
||||
g.release('direct');
|
||||
g.release('reserved');
|
||||
g.release('tx');
|
||||
expect(g.snapshot()).toEqual({ raw: 1, direct: 0, reserved: 0, tx: 0 });
|
||||
});
|
||||
|
||||
test('release clamps at zero (a missed acquire never underflows)', () => {
|
||||
const g = new CheckoutGauge();
|
||||
g.release('raw');
|
||||
g.release('tx');
|
||||
expect(g.snapshot()).toEqual({ raw: 0, direct: 0, reserved: 0, tx: 0 });
|
||||
});
|
||||
|
||||
test('snapshot is a copy, not a live reference', () => {
|
||||
const g = new CheckoutGauge();
|
||||
const snap = g.snapshot();
|
||||
g.acquire('raw');
|
||||
expect(snap.raw).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// --- engine seams (fake pool, no DB) ---------------------------------------
|
||||
|
||||
interface FakePoolBehavior {
|
||||
/** What conn.unsafe returns per call. */
|
||||
unsafe: (sql: string, params?: unknown[]) => unknown;
|
||||
}
|
||||
|
||||
function makeEngine(behavior: FakePoolBehavior): {
|
||||
engine: PostgresEngine;
|
||||
diagnostics: () => { tracked: Record<string, number>; poolMax: number | null } | null;
|
||||
} {
|
||||
const fakePool = {
|
||||
unsafe: behavior.unsafe,
|
||||
options: { max: 10 },
|
||||
};
|
||||
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
|
||||
Object.defineProperty(engine, 'sql', { get: () => fakePool });
|
||||
Object.defineProperty(engine, '_sql', { value: fakePool, writable: true });
|
||||
// Fresh gauge per fake engine (the real field initializer doesn't run for
|
||||
// Object.create instances).
|
||||
Object.defineProperty(engine, 'checkoutGauge', {
|
||||
value: new CheckoutGauge(),
|
||||
writable: true,
|
||||
});
|
||||
return {
|
||||
engine,
|
||||
diagnostics: () =>
|
||||
(engine as unknown as {
|
||||
getPoolDiagnostics: () => { tracked: Record<string, number>; poolMax: number | null } | null;
|
||||
}).getPoolDiagnostics(),
|
||||
};
|
||||
}
|
||||
|
||||
describe('PostgresEngine gauge seams (fake pool)', () => {
|
||||
test('executeRaw: counted while in flight, released on resolve', async () => {
|
||||
let resolveQuery: (rows: unknown[]) => void = () => {};
|
||||
const { engine, diagnostics } = makeEngine({
|
||||
unsafe: () => new Promise((r) => { resolveQuery = r; }),
|
||||
});
|
||||
|
||||
const p = engine.executeRaw('SELECT 42');
|
||||
expect(diagnostics()?.tracked.raw).toBe(1);
|
||||
expect(diagnostics()?.poolMax).toBe(10);
|
||||
resolveQuery([]);
|
||||
await p;
|
||||
expect(diagnostics()?.tracked.raw).toBe(0);
|
||||
});
|
||||
|
||||
test('executeRaw: released on REJECTED query (leak guard)', async () => {
|
||||
const { engine, diagnostics } = makeEngine({
|
||||
unsafe: () => Promise.reject(new Error('query was cancelled')),
|
||||
});
|
||||
await expect(engine.executeRaw('SELECT 42')).rejects.toThrow('query was cancelled');
|
||||
expect(diagnostics()?.tracked.raw).toBe(0);
|
||||
});
|
||||
|
||||
test('executeRaw: released on SYNCHRONOUS throw (pre-aborted signal)', async () => {
|
||||
const { engine, diagnostics } = makeEngine({
|
||||
unsafe: () => Promise.resolve([]),
|
||||
});
|
||||
const ac = new AbortController();
|
||||
ac.abort();
|
||||
await expect(
|
||||
engine.executeRaw('SELECT 42', undefined, { signal: ac.signal }),
|
||||
).rejects.toThrow();
|
||||
expect(diagnostics()?.tracked.raw).toBe(0);
|
||||
});
|
||||
|
||||
test('getPoolDiagnostics is fail-open (no pool → null, no throw)', () => {
|
||||
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
|
||||
// No sql defined — the getter on the prototype will throw internally.
|
||||
const diag = (engine as unknown as {
|
||||
getPoolDiagnostics: () => unknown;
|
||||
}).getPoolDiagnostics();
|
||||
expect(diag).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('PostgresEngine gauge seams: direct + tx (coverage-audit gaps)', () => {
|
||||
test('executeRawDirect: counted as direct while in flight, released on resolve and reject', async () => {
|
||||
let resolveQuery: (rows: unknown[]) => void = () => {};
|
||||
const { engine, diagnostics } = makeEngine({
|
||||
unsafe: () => new Promise((r) => { resolveQuery = r; }),
|
||||
});
|
||||
// No connectionManager on the fake -> executeRawDirect falls through to this.sql.
|
||||
const p = engine.executeRawDirect('SELECT 1');
|
||||
expect(diagnostics()?.tracked.direct).toBe(1);
|
||||
resolveQuery([]);
|
||||
await p;
|
||||
expect(diagnostics()?.tracked.direct).toBe(0);
|
||||
|
||||
const { engine: rejEngine, diagnostics: rejDiag } = makeEngine({
|
||||
unsafe: () => Promise.reject(new Error('boom')),
|
||||
});
|
||||
await expect(rejEngine.executeRawDirect('SELECT 1')).rejects.toThrow('boom');
|
||||
expect(rejDiag()?.tracked.direct).toBe(0);
|
||||
});
|
||||
|
||||
test('transaction: tx counter released on SYNCHRONOUS begin() throw (nested-tx clone class)', async () => {
|
||||
const { engine, diagnostics } = makeEngine({ unsafe: () => Promise.resolve([]) });
|
||||
// Fake pool has no .begin -> conn.begin(...) throws synchronously, the
|
||||
// exact shape a nested transaction on a tx clone produces.
|
||||
await expect(
|
||||
(engine as unknown as { transaction: (fn: unknown) => Promise<unknown> }).transaction(async () => 'x'),
|
||||
).rejects.toThrow();
|
||||
expect(diagnostics()?.tracked.tx).toBe(0);
|
||||
});
|
||||
|
||||
test('withReservedConnection: ddl() throw falls back to the READ pool and releases the permit', async () => {
|
||||
const log: string[] = [];
|
||||
const readPool = {
|
||||
unsafe: async () => [],
|
||||
options: { max: 10 },
|
||||
reserve: async () => {
|
||||
log.push('reserve:read');
|
||||
return { unsafe: async () => [], release: () => log.push('release:read') };
|
||||
},
|
||||
};
|
||||
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
|
||||
Object.defineProperty(engine, 'sql', { get: () => readPool });
|
||||
Object.defineProperty(engine, '_sql', { value: readPool, writable: true });
|
||||
Object.defineProperty(engine, 'checkoutGauge', { value: new CheckoutGauge(), writable: true });
|
||||
Object.defineProperty(engine, '_reservedDirectInFlight', { value: 0, writable: true });
|
||||
Object.defineProperty(engine, 'connectionManager', {
|
||||
value: {
|
||||
peekReadPool: () => readPool,
|
||||
isDualPoolActive: () => true,
|
||||
describeMode: () => ({ direct_pool_size: 3 }),
|
||||
ddl: async () => { throw new Error('EMAXCONNSESSION'); },
|
||||
},
|
||||
writable: true,
|
||||
});
|
||||
const out = await engine.withReservedConnection(async () => 'ok');
|
||||
expect(out).toBe('ok');
|
||||
expect(log).toEqual(['reserve:read', 'release:read']); // fell back, did not fail the caller
|
||||
expect((engine as unknown as { _reservedDirectInFlight: number })._reservedDirectInFlight).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,158 @@
|
||||
/**
|
||||
* issue #6 — `withReservedConnection` routing:
|
||||
*
|
||||
* dual-pool active + permit available → reserve from the DIRECT pool
|
||||
* (30-min statement_timeout lane; stops long DDL/backfill holds from
|
||||
* pinning the worker's shared read pool)
|
||||
* semaphore exhausted (directPoolSize-1 in flight) → READ pool (status quo;
|
||||
* >= 1 direct slot always stays free for claim/renewLock heartbeats)
|
||||
* kill-switched / single-pool → READ pool (status quo)
|
||||
* inside an open transaction → never rerouted
|
||||
* throw inside fn / reserve failure → semaphore released (leak guard)
|
||||
*
|
||||
* Hermetic: Object.create(PostgresEngine.prototype) + recording fake pools.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { PostgresEngine } from '../src/core/postgres-engine.ts';
|
||||
import { CheckoutGauge } from '../src/core/pool-gauge.ts';
|
||||
|
||||
interface FakeReserved {
|
||||
unsafe: (q: string, params?: unknown[]) => Promise<unknown[]>;
|
||||
release: () => void;
|
||||
}
|
||||
|
||||
function makeFakePool(label: string, log: string[]) {
|
||||
return {
|
||||
label,
|
||||
unsafe: async () => [],
|
||||
options: { max: 10 },
|
||||
reserve: async (): Promise<FakeReserved> => {
|
||||
log.push(`reserve:${label}`);
|
||||
return {
|
||||
unsafe: async () => [],
|
||||
release: () => log.push(`release:${label}`),
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function makeEngine(opts: {
|
||||
dualPool: boolean;
|
||||
directPoolSize?: number;
|
||||
inTransaction?: boolean;
|
||||
log: string[];
|
||||
}) {
|
||||
const readPool = makeFakePool('read', opts.log);
|
||||
const directPool = makeFakePool('direct', opts.log);
|
||||
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
|
||||
Object.defineProperty(engine, 'sql', { get: () => readPool });
|
||||
// inTransaction detection: _sql !== null && peekReadPool() !== _sql.
|
||||
// Simulate "in transaction" by making _sql differ from peekReadPool().
|
||||
Object.defineProperty(engine, '_sql', {
|
||||
value: opts.inTransaction ? { txMarker: true } : readPool,
|
||||
writable: true,
|
||||
});
|
||||
Object.defineProperty(engine, 'checkoutGauge', { value: new CheckoutGauge(), writable: true });
|
||||
Object.defineProperty(engine, '_reservedDirectInFlight', { value: 0, writable: true });
|
||||
Object.defineProperty(engine, 'connectionManager', {
|
||||
value: {
|
||||
peekReadPool: () => readPool,
|
||||
isDualPoolActive: () => opts.dualPool,
|
||||
describeMode: () => ({ direct_pool_size: opts.directPoolSize ?? 3 }),
|
||||
ddl: async () => directPool,
|
||||
},
|
||||
writable: true,
|
||||
});
|
||||
return engine;
|
||||
}
|
||||
|
||||
describe('withReservedConnection routing (issue #6)', () => {
|
||||
test('dual-pool active: reserves from the DIRECT pool', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, log });
|
||||
await engine.withReservedConnection(async () => 'ok');
|
||||
expect(log).toEqual(['reserve:direct', 'release:direct']);
|
||||
});
|
||||
|
||||
test('single-pool (kill-switched): reserves from the READ pool (status quo)', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: false, log });
|
||||
await engine.withReservedConnection(async () => 'ok');
|
||||
expect(log).toEqual(['reserve:read', 'release:read']);
|
||||
});
|
||||
|
||||
test('inside a transaction: never rerouted', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, inTransaction: true, log });
|
||||
await engine.withReservedConnection(async () => 'ok');
|
||||
expect(log).toEqual(['reserve:read', 'release:read']);
|
||||
});
|
||||
|
||||
test('semaphore: directPoolSize-1 concurrent direct reserves; overflow -> read pool', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, directPoolSize: 3, log });
|
||||
|
||||
let releaseFirst: () => void = () => {};
|
||||
let releaseSecond: () => void = () => {};
|
||||
const first = engine.withReservedConnection(
|
||||
() => new Promise((r) => { releaseFirst = () => r('one'); }),
|
||||
);
|
||||
const second = engine.withReservedConnection(
|
||||
() => new Promise((r) => { releaseSecond = () => r('two'); }),
|
||||
);
|
||||
// Let both reserves happen.
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
expect(log.filter((l) => l === 'reserve:direct').length).toBe(2); // cap = 3-1 = 2
|
||||
|
||||
// Third concurrent reserve overflows to the read pool (status quo, never blocks).
|
||||
const third = await engine.withReservedConnection(async () => 'three');
|
||||
expect(third).toBe('three');
|
||||
expect(log).toContain('reserve:read');
|
||||
|
||||
releaseFirst();
|
||||
releaseSecond();
|
||||
await Promise.all([first, second]);
|
||||
|
||||
// Permits released: the next reserve goes direct again.
|
||||
log.length = 0;
|
||||
await engine.withReservedConnection(async () => 'four');
|
||||
expect(log).toEqual(['reserve:direct', 'release:direct']);
|
||||
});
|
||||
|
||||
test('direct_pool_size=1: NEVER reserves from the direct pool (a reserve would starve ALL heartbeats)', async () => {
|
||||
// Red-team finding: a Math.max(1, size-1) floor made cap=1 at size=1,
|
||||
// letting a multi-minute reserve consume the ONLY direct session that
|
||||
// claim/renewLock heartbeats depend on — the #6 class reintroduced.
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, directPoolSize: 1, log });
|
||||
await engine.withReservedConnection(async () => 'ok');
|
||||
expect(log).toEqual(['reserve:read', 'release:read']);
|
||||
});
|
||||
|
||||
test('fn throw releases the semaphore permit (leak guard)', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, directPoolSize: 2, log }); // cap = 1
|
||||
await expect(
|
||||
engine.withReservedConnection(async () => { throw new Error('DDL boom'); }),
|
||||
).rejects.toThrow('DDL boom');
|
||||
// Permit released: next reserve still goes direct (cap would block if leaked).
|
||||
log.length = 0;
|
||||
await engine.withReservedConnection(async () => 'ok');
|
||||
expect(log).toEqual(['reserve:direct', 'release:direct']);
|
||||
});
|
||||
|
||||
test('reserve() failure releases the permit and rethrows', async () => {
|
||||
const log: string[] = [];
|
||||
const engine = makeEngine({ dualPool: true, directPoolSize: 2, log });
|
||||
const cm = (engine as unknown as { connectionManager: { ddl: () => Promise<unknown> } }).connectionManager;
|
||||
cm.ddl = async () => ({
|
||||
reserve: async () => { throw new Error('EMAXCONNSESSION'); },
|
||||
});
|
||||
await expect(engine.withReservedConnection(async () => 'x')).rejects.toThrow('EMAXCONNSESSION');
|
||||
// Permit released despite the failure.
|
||||
expect(
|
||||
(engine as unknown as { _reservedDirectInFlight: number })._reservedDirectInFlight,
|
||||
).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* issue #6 — timed-out probes are CANCELLED, not abandoned.
|
||||
*
|
||||
* Closes the TODOS entry "cancel (not just abandon) timed-out submit-time
|
||||
* queue probes": `probeQueueState` time-bounds via Promise.race, but the
|
||||
* losing query used to keep running on the pool after the race resolved —
|
||||
* under pool exhaustion (the exact regime the probe exists to detect) the
|
||||
* abandoned query held a slot and made the exhaustion worse. The timeout now
|
||||
* aborts a per-probe AbortSignal threaded through `queryWedgeSignals` into
|
||||
* `engine.executeRaw`, and `withRefreshingLock`'s heartbeat does the same for
|
||||
* `handle.refresh`.
|
||||
*
|
||||
* Hermetic: stub engines (`as unknown as BrainEngine`), no DB.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { probeQueueState, queryWedgeSignals } from '../src/core/minions/supervisor.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
const WEDGE_ROW = {
|
||||
stalled: '0',
|
||||
active_healthy: '1',
|
||||
waiting: '2',
|
||||
waiting_claimable: '2',
|
||||
last_completed: null,
|
||||
last_completed_claimable: null,
|
||||
};
|
||||
|
||||
describe('queryWedgeSignals signal threading (issue #6)', () => {
|
||||
test('forwards opts.signal to executeRaw', async () => {
|
||||
let seenOpts: { signal?: AbortSignal } | undefined;
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRaw: async (_sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
|
||||
seenOpts = opts;
|
||||
return [WEDGE_ROW];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const ac = new AbortController();
|
||||
|
||||
const sig = await queryWedgeSignals(engine, 'default', ['sync'], { signal: ac.signal });
|
||||
|
||||
expect(sig.waiting).toBe(2);
|
||||
expect(seenOpts?.signal).toBe(ac.signal);
|
||||
});
|
||||
});
|
||||
|
||||
describe('probeQueueState cancellation (issue #6)', () => {
|
||||
test('timeout aborts the per-probe signal so the losing query is cancelled', async () => {
|
||||
let seenSignal: AbortSignal | undefined;
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRaw: (_sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
|
||||
seenSignal = opts?.signal;
|
||||
// Hang forever — the abandoned-racer scenario.
|
||||
return new Promise<never>(() => {});
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 50 });
|
||||
|
||||
expect(state.probe_failed).toBe(true);
|
||||
expect(seenSignal).toBeInstanceOf(AbortSignal);
|
||||
expect(seenSignal!.aborted).toBe(true);
|
||||
});
|
||||
|
||||
test('fast probe: signal delivered but never aborted; result returned', async () => {
|
||||
const seenSignals: Array<AbortSignal | undefined> = [];
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRaw: async (sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
|
||||
seenSignals.push(opts?.signal);
|
||||
if (sql.includes('EXTRACT(EPOCH')) return [{ age: 12 }];
|
||||
return [WEDGE_ROW];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 5_000 });
|
||||
|
||||
expect(state.probe_failed).toBeUndefined();
|
||||
expect(state.depth).toBe(2);
|
||||
// The wedge + age queries carry the probe signal. Other engine calls made
|
||||
// by the probe (e.g. inspectLock) pass no opts — filter to the signals we
|
||||
// threaded and assert none were aborted on the fast path.
|
||||
const threaded = seenSignals.filter((s): s is AbortSignal => s instanceof AbortSignal);
|
||||
expect(threaded.length).toBeGreaterThanOrEqual(2);
|
||||
for (const s of threaded) {
|
||||
expect(s.aborted).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
test('probe throw still collapses to {probe_failed: true} (fail-open contract)', async () => {
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRaw: async () => {
|
||||
throw new Error('query was cancelled');
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 5_000 });
|
||||
expect(state.probe_failed).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,249 @@
|
||||
/**
|
||||
* issue #5 — `runChildJobEntry` (the `jobs run-child` core) against a real
|
||||
* in-memory PGLite engine with a REAL claim-minted lock token. (PGLite is
|
||||
* rejected at the CLI layer for actual isolation runs — Postgres-only — but
|
||||
* the core function is engine-agnostic, which is exactly what makes it
|
||||
* testable without a DATABASE_URL.)
|
||||
*
|
||||
* Paths pinned:
|
||||
* - success: handler result lands in the outcome file, exit 0
|
||||
* - handler failure: encoded error outcome, STILL exit 0 (a reported
|
||||
* failure is a successful report)
|
||||
* - token mismatch / reclaimed job: exit 14, handler NEVER runs
|
||||
* - missing handler: generic error outcome, exit 0
|
||||
* - SIGTERM semantics via direct signal wiring (ctx.signal + shutdown)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import { runChildJobEntry } from '../src/core/minions/run-child.ts';
|
||||
import { decodeChildOutcomeFile } from '../src/core/minions/job-isolation.ts';
|
||||
import {
|
||||
JOB_CHILD_EXIT_NOT_CLAIMED,
|
||||
} from '../src/core/minions/worker-exit-codes.ts';
|
||||
import { UnrecoverableError, type MinionHandler } from '../src/core/minions/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
let dir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' }); // in-memory
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
dir = mkdtempSync(join(tmpdir(), 'gbrain-run-child-test-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
/** Claim a freshly-added job the same way the parent worker does. */
|
||||
async function addAndClaim(name: string, data: Record<string, unknown> = {}) {
|
||||
await queue.add(name, data);
|
||||
const job = await queue.claim('parent-tok-1', 30_000, 'default', [name]);
|
||||
if (!job) throw new Error('claim failed in test setup');
|
||||
return job;
|
||||
}
|
||||
|
||||
function handlers(map: Record<string, MinionHandler>) {
|
||||
return { resolveHandler: (n: string) => map[n] };
|
||||
}
|
||||
|
||||
describe('runChildJobEntry', () => {
|
||||
test('success: outcome file carries the handler result; exit 0; ctx wired to the job row', async () => {
|
||||
const job = await addAndClaim('sync', { full: true });
|
||||
const resultPath = join(dir, `ok-${job.id}.json`);
|
||||
let sawData: unknown;
|
||||
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
|
||||
handlers({
|
||||
sync: async (ctx) => {
|
||||
sawData = ctx.data;
|
||||
await ctx.updateProgress({ step: 'half' });
|
||||
return { pages: 7 };
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(sawData).toEqual({ full: true });
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
expect(outcome).toEqual({ outcome: 'success', result: { pages: 7 } });
|
||||
// The token-fenced progress write really landed.
|
||||
const rows = await engine.executeRaw<{ progress: unknown }>(
|
||||
'SELECT progress FROM minion_jobs WHERE id = $1',
|
||||
[job.id],
|
||||
);
|
||||
const progress = rows[0]?.progress;
|
||||
expect(typeof progress === 'string' ? JSON.parse(progress) : progress).toEqual({ step: 'half' });
|
||||
});
|
||||
|
||||
test('handler failure: encoded error outcome, exit 0 (reported failure = successful report)', async () => {
|
||||
const job = await addAndClaim('sync');
|
||||
const resultPath = join(dir, `err-${job.id}.json`);
|
||||
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
|
||||
handlers({
|
||||
sync: async () => {
|
||||
throw new UnrecoverableError('schema mismatch, never retry');
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
expect(code).toBe(0);
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
expect(outcome.outcome).toBe('error');
|
||||
if (outcome.outcome === 'error') {
|
||||
expect(outcome.errorKind).toBe('unrecoverable');
|
||||
expect(outcome.message).toBe('schema mismatch, never retry');
|
||||
}
|
||||
});
|
||||
|
||||
test('token mismatch (job reclaimed): exit 14, handler never runs', async () => {
|
||||
const job = await addAndClaim('sync');
|
||||
const resultPath = join(dir, `mismatch-${job.id}.json`);
|
||||
let handlerRan = false;
|
||||
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'STALE-token', resultPath, parentPid: 0 },
|
||||
handlers({
|
||||
sync: async () => {
|
||||
handlerRan = true;
|
||||
return null;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
expect(code).toBe(JOB_CHILD_EXIT_NOT_CLAIMED);
|
||||
expect(handlerRan).toBe(false);
|
||||
expect(() => decodeChildOutcomeFile(resultPath)).toThrow(); // nothing written
|
||||
});
|
||||
|
||||
test('missing job id: exit 14', async () => {
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: 999_999, lockToken: 'parent-tok-1', resultPath: join(dir, 'none.json'), parentPid: 0 },
|
||||
handlers({}),
|
||||
);
|
||||
expect(code).toBe(JOB_CHILD_EXIT_NOT_CLAIMED);
|
||||
});
|
||||
|
||||
test('missing handler: UNRECOVERABLE error outcome (inline parity: dead on attempt 1), exit 0', async () => {
|
||||
const job = await addAndClaim('exotic-plugin-job');
|
||||
const resultPath = join(dir, `nohandler-${job.id}.json`);
|
||||
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
|
||||
handlers({}),
|
||||
);
|
||||
|
||||
expect(code).toBe(0);
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
expect(outcome.outcome).toBe('error');
|
||||
if (outcome.outcome === 'error') {
|
||||
expect(outcome.errorKind).toBe('unrecoverable');
|
||||
expect(outcome.message).toContain("No handler for job type 'exotic-plugin-job'");
|
||||
}
|
||||
});
|
||||
|
||||
test('SIGTERM aborts ONLY shutdownSignal — ctx.signal stays live (inline signal-separation parity)', async () => {
|
||||
const job = await addAndClaim('sync');
|
||||
const resultPath = join(dir, `sigterm-${job.id}.json`);
|
||||
let ctxSignalAbortedAtShutdown: boolean | null = null;
|
||||
|
||||
const entry = runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
|
||||
handlers({
|
||||
sync: (ctx) =>
|
||||
new Promise((resolve) => {
|
||||
ctx.shutdownSignal.addEventListener('abort', () => {
|
||||
// The whole point: a cooperative handler gets the drain window
|
||||
// with its per-job signal STILL LIVE, finishes, and reports.
|
||||
ctxSignalAbortedAtShutdown = ctx.signal.aborted;
|
||||
resolve('finished-during-drain');
|
||||
});
|
||||
}),
|
||||
}),
|
||||
);
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
// Trigger the entry's process.on('SIGTERM') handler in-process without
|
||||
// sending a real signal to the test runner.
|
||||
(process as unknown as { emit: (event: string) => boolean }).emit('SIGTERM');
|
||||
const code = await entry;
|
||||
|
||||
expect(code).toBe(0);
|
||||
// TS control-flow can't see the closure write; compare explicitly.
|
||||
expect(ctxSignalAbortedAtShutdown === false).toBe(true);
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
expect(outcome).toEqual({ outcome: 'success', result: { value: 'finished-during-drain' } });
|
||||
});
|
||||
|
||||
test('primitive results are wrapped {value: x} CHILD-side (completeJob shape parity across the JSON boundary)', async () => {
|
||||
const job = await addAndClaim('sync');
|
||||
const resultPath = join(dir, `wrap-${job.id}.json`);
|
||||
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
|
||||
handlers({ sync: async () => 42 }),
|
||||
);
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(decodeChildOutcomeFile(resultPath)).toEqual({ outcome: 'success', result: { value: 42 } });
|
||||
});
|
||||
|
||||
test('parent-death watchdog: aborts the handler and schedules the hard exit', async () => {
|
||||
const job = await addAndClaim('sync');
|
||||
const resultPath = join(dir, `orphan-${job.id}.json`);
|
||||
let sawAbort = false;
|
||||
const hardExits: number[] = [];
|
||||
|
||||
// A GUARANTEED-dead parent pid: spawn a real short-lived process, wait
|
||||
// for it to exit, and use its reaped pid. (A magic high pid is
|
||||
// allocatable on Linux — default kernel.pid_max is 4,194,304 — so a real
|
||||
// process could hold it and hang the test; testing specialist.)
|
||||
const { spawnSync: spawnDead } = await import('node:child_process');
|
||||
const deadPid = spawnDead('/bin/sh', ['-c', 'exit 0']).pid ?? 0;
|
||||
expect(deadPid).toBeGreaterThan(0);
|
||||
const code = await runChildJobEntry(
|
||||
engine,
|
||||
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: deadPid },
|
||||
{
|
||||
resolveHandler: () => async (ctx) =>
|
||||
new Promise((resolve) => {
|
||||
ctx.signal.addEventListener('abort', () => {
|
||||
sawAbort = true;
|
||||
resolve('aborted-cleanly');
|
||||
});
|
||||
}),
|
||||
parentPollMs: 30,
|
||||
orphanGraceMs: 60_000, // never fires in-test; we capture the schedule
|
||||
hardExit: (c) => { hardExits.push(c); },
|
||||
},
|
||||
);
|
||||
|
||||
expect(sawAbort).toBe(true);
|
||||
expect(code).toBe(0); // the handler resolved after abort; outcome written
|
||||
const outcome = decodeChildOutcomeFile(resultPath);
|
||||
expect(outcome.outcome).toBe('success');
|
||||
expect(hardExits.length).toBe(0); // grace not reached — clean wind-down
|
||||
});
|
||||
});
|
||||
@@ -36,4 +36,18 @@ describe('buildWorkerArgs', () => {
|
||||
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0 }))
|
||||
.not.toContain('--nice');
|
||||
});
|
||||
|
||||
// issue #5 — conditional pass-through: inline/omitted keeps existing
|
||||
// deployments' argv byte-identical (the pinned arrays above never change).
|
||||
test('appends --job-isolation process when set', () => {
|
||||
expect(buildWorkerArgs({ concurrency: 2, queue: 'default', maxRssMb: 0, jobIsolation: 'process' }))
|
||||
.toEqual(['jobs', 'work', '--concurrency', '2', '--queue', 'default', '--job-isolation', 'process']);
|
||||
});
|
||||
|
||||
test('omits --job-isolation when inline or undefined', () => {
|
||||
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0, jobIsolation: 'inline' }))
|
||||
.not.toContain('--job-isolation');
|
||||
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0 }))
|
||||
.not.toContain('--job-isolation');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
/**
|
||||
* issue #5 — full parent path with isolation on: claim → spawn child →
|
||||
* decode outcome → completeJob / failJob / release. Real in-memory PGLite
|
||||
* worker + the fake-run-child.mjs fixture standing in for the compiled CLI
|
||||
* (reads the isolation env contract, writes canned outcomes).
|
||||
*
|
||||
* Pins:
|
||||
* - success: job completes with the CHILD's result (fenced completeJob)
|
||||
* - error outcome: failJob path — attempt burned, delayed/dead per policy
|
||||
* - crash (exit 1, no file): attempt burned
|
||||
* - spawn failure (bad child CLI): RELEASED — status stays 'active',
|
||||
* attempts NOT burned (infra class; stall sweeper would requeue)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import { MinionWorker } from '../src/core/minions/worker.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const FIXTURE = join(import.meta.dir, 'fixtures', 'fake-run-child.mjs');
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
function makeWorker(invocationCmd = process.execPath, argsPrefix = [FIXTURE]) {
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
pollInterval: 25,
|
||||
healthCheckInterval: 0,
|
||||
maxRssMb: 0,
|
||||
jobIsolation: 'process',
|
||||
childCliInvocation: { cmd: invocationCmd, argsPrefix },
|
||||
});
|
||||
// Handler must exist in the parent registry (name-scoped claiming); its
|
||||
// body never runs in isolation mode.
|
||||
worker.register('isotest', async () => {
|
||||
throw new Error('parent-side handler must not run when isolated');
|
||||
});
|
||||
return worker;
|
||||
}
|
||||
|
||||
async function runWorkerUntil(
|
||||
worker: MinionWorker,
|
||||
done: () => Promise<boolean>,
|
||||
timeoutMs = 10_000,
|
||||
): Promise<void> {
|
||||
const run = worker.start();
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
try {
|
||||
while (Date.now() < deadline) {
|
||||
if (await done()) return;
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
throw new Error('runWorkerUntil timed out');
|
||||
} finally {
|
||||
worker.stop();
|
||||
await run;
|
||||
}
|
||||
}
|
||||
|
||||
async function jobRow(id: number): Promise<{ status: string; attempts_made: number; result: unknown; error_text: string | null }> {
|
||||
const rows = await engine.executeRaw<{
|
||||
status: string;
|
||||
attempts_made: number;
|
||||
result: unknown;
|
||||
error_text: string | null;
|
||||
}>('SELECT status, attempts_made, result, error_text FROM minion_jobs WHERE id = $1', [id]);
|
||||
if (!rows[0]) throw new Error('job row missing');
|
||||
return rows[0];
|
||||
}
|
||||
|
||||
describe('worker with jobIsolation=process (PGLite + fake child)', () => {
|
||||
test("construction throws for jobIsolation 'process' without childCliInvocation (predicate-mismatch guard)", () => {
|
||||
// Red-team finding: 'process' without an invocation silently executed
|
||||
// handlers INLINE while the evict path believed it was isolated.
|
||||
expect(
|
||||
() => new MinionWorker(engine, { queue: 'default', jobIsolation: 'process' }),
|
||||
).toThrow(/childCliInvocation/);
|
||||
});
|
||||
|
||||
test('spawn-failure circuit breaker: 3 consecutive bootstrap failures emit unhealthy(child_spawn_failing)', async () => {
|
||||
// Red-team finding: a deterministic child-bootstrap failure looped
|
||||
// claim/release forever, invisible to the stall detector (every settle
|
||||
// refreshes the progress clock).
|
||||
const j1 = await queue.add('isotest', {});
|
||||
const j2 = await queue.add('isotest', {});
|
||||
const j3 = await queue.add('isotest', {});
|
||||
const worker = makeWorker('/nonexistent/gbrain-binary', []);
|
||||
const unhealthy: unknown[] = [];
|
||||
worker.on('unhealthy', (i) => unhealthy.push(i));
|
||||
const run = worker.start();
|
||||
const deadline = Date.now() + 10_000;
|
||||
try {
|
||||
while (Date.now() < deadline && unhealthy.length === 0) {
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
} finally {
|
||||
worker.stop();
|
||||
await run;
|
||||
}
|
||||
expect(unhealthy.length).toBeGreaterThan(0);
|
||||
const info = unhealthy[0] as { reason: string; consecutiveFailures: number };
|
||||
expect(info.reason).toBe('child_spawn_failing');
|
||||
expect(info.consecutiveFailures).toBeGreaterThanOrEqual(3);
|
||||
// No attempts burned anywhere — all three rows released, not failed.
|
||||
for (const j of [j1, j2, j3]) {
|
||||
const row = await jobRow(j.id);
|
||||
expect(row.status).toBe('active');
|
||||
expect(row.attempts_made).toBe(0);
|
||||
}
|
||||
}, 20_000);
|
||||
|
||||
test('success: child result lands via the fenced completeJob', async () => {
|
||||
await withEnv({ FAKE_RUN_CHILD_MODE: 'success' }, async () => {
|
||||
const job = await queue.add('isotest', { prompt: 'hi' });
|
||||
const worker = makeWorker();
|
||||
await runWorkerUntil(worker, async () => (await jobRow(job.id)).status === 'completed');
|
||||
const row = await jobRow(job.id);
|
||||
expect(row.status).toBe('completed');
|
||||
const result = typeof row.result === 'string' ? JSON.parse(row.result) : row.result;
|
||||
expect((result as { fromChild?: boolean }).fromChild).toBe(true);
|
||||
// The child got the REAL claim token through the env contract.
|
||||
expect((result as { token?: string }).token).toMatch(/^.+:.+$/);
|
||||
});
|
||||
}, 20_000);
|
||||
|
||||
test('error outcome: failJob path, attempt burned', async () => {
|
||||
await withEnv({ FAKE_RUN_CHILD_MODE: 'error' }, async () => {
|
||||
const job = await queue.add('isotest', {}, { max_attempts: 1 });
|
||||
const worker = makeWorker();
|
||||
await runWorkerUntil(worker, async () => {
|
||||
const s = (await jobRow(job.id)).status;
|
||||
return s === 'dead' || s === 'failed';
|
||||
});
|
||||
const row = await jobRow(job.id);
|
||||
expect(row.status).toBe('dead'); // maxAttempts 1 → attempt burned → dead
|
||||
expect(row.error_text).toContain('fake child handler failure');
|
||||
});
|
||||
}, 20_000);
|
||||
|
||||
test('crash (exit 1, no outcome file): attempt burned', async () => {
|
||||
await withEnv({ FAKE_RUN_CHILD_MODE: 'crash' }, async () => {
|
||||
const job = await queue.add('isotest', {}, { max_attempts: 1 });
|
||||
const worker = makeWorker();
|
||||
await runWorkerUntil(worker, async () => {
|
||||
const s = (await jobRow(job.id)).status;
|
||||
return s === 'dead' || s === 'failed';
|
||||
});
|
||||
const row = await jobRow(job.id);
|
||||
expect(row.status).toBe('dead');
|
||||
expect(row.error_text).toContain('exit code=1');
|
||||
});
|
||||
}, 20_000);
|
||||
|
||||
test('serialization parity (codex-2 #8): unreportable results fail in BOTH modes, never complete', async () => {
|
||||
// Inline: a circular result blows up in completeJob's serialization →
|
||||
// failJob (attempt burned).
|
||||
const inlineWorker = new MinionWorker(engine, {
|
||||
queue: 'default', concurrency: 1, pollInterval: 25, healthCheckInterval: 0, maxRssMb: 0,
|
||||
});
|
||||
inlineWorker.register('isotest', async () => {
|
||||
const a: Record<string, unknown> = {};
|
||||
a.self = a; // circular — not JSONB-serializable
|
||||
return a;
|
||||
});
|
||||
const j1 = await queue.add('isotest', {}, { max_attempts: 1 });
|
||||
await runWorkerUntil(inlineWorker, async () => {
|
||||
const s = (await jobRow(j1.id)).status;
|
||||
return s !== 'waiting' && s !== 'active';
|
||||
});
|
||||
expect((await jobRow(j1.id)).status).not.toBe('completed');
|
||||
|
||||
// Process mode: a child that cannot persist its outcome (exit 15) lands
|
||||
// in the same terminal class — failed loudly, never falsely completed.
|
||||
await withEnv({ FAKE_RUN_CHILD_MODE: 'exit15' }, async () => {
|
||||
const j2 = await queue.add('isotest', {}, { max_attempts: 1 });
|
||||
const worker = makeWorker();
|
||||
await runWorkerUntil(worker, async () => (await jobRow(j2.id)).status === 'dead');
|
||||
expect((await jobRow(j2.id)).error_text).toContain('outcome file');
|
||||
});
|
||||
}, 30_000);
|
||||
|
||||
test('spawn failure: RELEASED — still active, attempts NOT burned (infra class)', async () => {
|
||||
const job = await queue.add('isotest', {});
|
||||
const worker = makeWorker('/nonexistent/gbrain-binary', []);
|
||||
// The claim happens, the spawn fails, the job is released (stays
|
||||
// 'active' until lock expiry — the stall sweeper's requeue territory).
|
||||
const run = worker.start();
|
||||
await new Promise((r) => setTimeout(r, 1_500));
|
||||
worker.stop();
|
||||
await run;
|
||||
const row = await jobRow(job.id);
|
||||
expect(row.status).toBe('active'); // NOT dead, NOT failed
|
||||
expect(row.attempts_made).toBe(0); // release = no failJob = no attempt burned
|
||||
}, 20_000);
|
||||
});
|
||||
@@ -569,3 +569,50 @@ describe('runLockRenewalTick: reconnect-once dep (issue #1678)', () => {
|
||||
expect(reconnectCalls).toBe(0); // pointless to reconnect when we're giving up the lock
|
||||
});
|
||||
});
|
||||
|
||||
describe('runLockRenewalTick: per-call cancellation signal (issue #6)', () => {
|
||||
test('renewLock receives a live AbortSignal; not aborted on the happy path', async () => {
|
||||
const audit = freshAudit();
|
||||
const timer = makeFakeTimer();
|
||||
let seenSignal: AbortSignal | undefined;
|
||||
const deps: LockRenewalDeps = {
|
||||
renewLock: async (_id, _tok, _dur, opts) => {
|
||||
seenSignal = opts?.signal;
|
||||
return true;
|
||||
},
|
||||
audit: audit.sink,
|
||||
now: () => 1000,
|
||||
setTimeout: timer.setTimeout,
|
||||
};
|
||||
const result = await runLockRenewalTick(deps, makeState());
|
||||
expect(result.kind).toBe('ok');
|
||||
expect(seenSignal).toBeInstanceOf(AbortSignal);
|
||||
expect(seenSignal!.aborted).toBe(false);
|
||||
});
|
||||
|
||||
test('hung renewLock: the timeout aborts the per-call signal so the query is cancelled, not orphaned', async () => {
|
||||
const audit = freshAudit();
|
||||
const timer = makeFakeTimer();
|
||||
let seenSignal: AbortSignal | undefined;
|
||||
const deps: LockRenewalDeps = {
|
||||
renewLock: (_id, _tok, _dur, opts) => {
|
||||
seenSignal = opts?.signal;
|
||||
return new Promise<boolean>(() => { /* hangs forever — abandoned racer */ });
|
||||
},
|
||||
audit: audit.sink,
|
||||
now: () => 1000,
|
||||
setTimeout: timer.setTimeout,
|
||||
};
|
||||
// Renewed recently: the timeout counts as failure #1, not a give-up.
|
||||
const state = makeState({ lastSuccessfulRenewalAt: 1000 });
|
||||
const tick = runLockRenewalTick(deps, state);
|
||||
// The race installs the timeout synchronously; firing it must abort the
|
||||
// per-call signal (this is what releases the checked-out pool slot).
|
||||
timer.runAll();
|
||||
const result = await tick;
|
||||
expect(result.kind).toBe('ok'); // counter bumped, deadline not yet crossed
|
||||
expect(state.consecutiveFailures).toBe(1);
|
||||
expect(seenSignal).toBeInstanceOf(AbortSignal);
|
||||
expect(seenSignal!.aborted).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -35,11 +35,19 @@ afterAll(async () => {
|
||||
|
||||
/** Make the DB-liveness probe (`SELECT 1`) throw; delegate every other query.
|
||||
* Returns a restore fn that removes the instance override (falls back to the
|
||||
* prototype method). */
|
||||
* prototype method). Also records the probe's `opts` so tests can pin that
|
||||
* the worker passes a cancellation signal (issue #6: a hung probe must be
|
||||
* cancelled, not abandoned on a checked-out pool slot). */
|
||||
const probeOptsSeen: Array<{ signal?: AbortSignal } | undefined> = [];
|
||||
function breakLivenessProbe(eng: PGLiteEngine): () => void {
|
||||
const real = eng.executeRaw.bind(eng);
|
||||
(eng as { executeRaw: unknown }).executeRaw = async (sql: string, params?: unknown[]) => {
|
||||
(eng as { executeRaw: unknown }).executeRaw = async (
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
) => {
|
||||
if (typeof sql === 'string' && sql.trim() === 'SELECT 1') {
|
||||
probeOptsSeen.push(opts);
|
||||
throw new Error('probe boom (simulated dead pool)');
|
||||
}
|
||||
return real(sql, params as never);
|
||||
@@ -95,6 +103,74 @@ describe('issue #1801 fix #2 — supervised DB self-defense', () => {
|
||||
expect(info?.reason).toBe('db_dead');
|
||||
}, 10_000);
|
||||
|
||||
it('the probe passes a cancellation AbortSignal so a hung SELECT 1 releases its slot (issue #6)', () => {
|
||||
// Populated by the two runUntilUnhealthy() calls above (>= 3 probes each).
|
||||
expect(probeOptsSeen.length).toBeGreaterThan(0);
|
||||
for (const opts of probeOptsSeen) {
|
||||
expect(opts?.signal).toBeInstanceOf(AbortSignal);
|
||||
}
|
||||
});
|
||||
|
||||
it('dual-pool probe gating: probeDirect is wired ONLY when isDualPoolActive() is true', async () => {
|
||||
// The load-bearing guard the adapter comments on: a kill-switched
|
||||
// executeRawDirect would probe the same starved read pool twice and
|
||||
// fabricate a verdict (testing specialist coverage gap).
|
||||
const directCalls: string[] = [];
|
||||
const runCase = async (dualPool: boolean): Promise<void> => {
|
||||
const cmStub = { isDualPoolActive: () => dualPool };
|
||||
(engine as unknown as { connectionManager?: unknown }).connectionManager = cmStub;
|
||||
const realDirect = engine.executeRawDirect.bind(engine);
|
||||
(engine as unknown as { executeRawDirect: unknown }).executeRawDirect = async (
|
||||
sql: string,
|
||||
params?: unknown[],
|
||||
opts?: { signal?: AbortSignal },
|
||||
) => {
|
||||
if (typeof sql === 'string' && sql.trim() === 'SELECT 1') {
|
||||
directCalls.push(`dual=${dualPool}`);
|
||||
return [];
|
||||
}
|
||||
return realDirect(sql, params as never, opts);
|
||||
};
|
||||
const restore = breakLivenessProbe(engine);
|
||||
try {
|
||||
await withEnv({ GBRAIN_SUPERVISED: '1' }, async () => {
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default', concurrency: 1, pollInterval: 25, maxRssMb: 0,
|
||||
healthCheckInterval: 20, dbFailExitAfter: 2, dbProbeTimeoutMs: 200,
|
||||
});
|
||||
worker.register('noop', async () => {});
|
||||
const got = new Promise<UnhealthyReason>((resolve) => {
|
||||
worker.on('unhealthy', (i) => resolve(i));
|
||||
});
|
||||
const runPromise = worker.start();
|
||||
const info = await Promise.race([
|
||||
got,
|
||||
new Promise<null>((r) => setTimeout(() => r(null), 3000)),
|
||||
]);
|
||||
worker.stop();
|
||||
await runPromise;
|
||||
expect(info).not.toBeNull();
|
||||
if (dualPool) {
|
||||
// Direct lane succeeded -> verdict names the pooler path.
|
||||
expect(info!.reason === 'db_dead' && info!.verdict).toBe('pool_starved');
|
||||
} else {
|
||||
expect(info!.reason === 'db_dead' && info!.verdict).toBe('unknown');
|
||||
}
|
||||
});
|
||||
} finally {
|
||||
restore();
|
||||
delete (engine as unknown as { executeRawDirect?: unknown }).executeRawDirect;
|
||||
delete (engine as unknown as { connectionManager?: unknown }).connectionManager;
|
||||
}
|
||||
};
|
||||
|
||||
await runCase(false);
|
||||
expect(directCalls.length).toBe(0); // single-pool: NEVER probes direct
|
||||
|
||||
await runCase(true);
|
||||
expect(directCalls.filter((c) => c === 'dual=true').length).toBeGreaterThan(0);
|
||||
}, 15_000);
|
||||
|
||||
it('structural: DB probe is NOT gated on !isSupervisedChild; stall detection IS', () => {
|
||||
const src = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'core', 'minions', 'worker.ts'),
|
||||
|
||||
Reference in New Issue
Block a user