Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 ec3d7e79c7 docs: cross-model doc-review precision fixes for v0.46.1.0
Codex review of the shipped docs vs the diff, each finding verified against
the code before applying:

- minions-deployment: group-SIGKILL platform caveat (Bun /bin/kill fallback),
  lock-token fencing scoped to queue writes (handler side effects bounded by
  the watchdog, not the token), connection math relabeled (pooler-lane vs
  direct session-lane split), no-per-child-RSS-cap note, GBRAIN_JOB_CHILD_CLI
  + the 3-consecutive-spawn-failure breaker documented.
- queue-operations-runbook: verdict rides the TERMINAL probe line (not every
  N/3 line), server_unreachable hedged (both-lanes-failed is the evidence),
  pooler-layer fault added to the 0-in-flight reading, jobs cancel described
  as cooperative inline vs real kill under isolation.
- KEY_FILES: run-child SIGTERM fires shutdownSignal ONLY (both only on
  parent death); third no-burn child class (ChildNotClaimedError).
- TESTING: e2e concurrency leg uses the fixture (no child DB pools); only
  the run-child leg boots real child pools.
- CHANGELOG: one wording precision fix (reserved holds leave a heartbeat
  slot, not "always keep a free slot").

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:54:50 -07:00
Garry TanandClaude Fable 5 802441223d docs: README Minions capability + KEY_FILES reserved-routing entry for v0.46.1.0
document-release sweep: the wave's docs covered the guides, TESTING, and the
new module entries but missed two spots — the README Job queue capability
paragraph (now names --job-isolation process and the probe verdicts, linking
both guides) and the KEY_FILES postgres-engine.ts entry (now carries the
withReservedConnection direct-lane routing invariants + getPoolDiagnostics
seam, pinned by test/postgres-engine-reserved-routing.test.ts). llms-full.txt
regenerated for the README edit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:45:51 -07:00
Garry TanandClaude Fable 5 f6aec5b0a0 test: passwordless fixture URLs in the pool-wiring tests
The pre-push credential guard (correctly) blocks any URL-with-password shape
in a pushed diff, including fake placeholders. The never-connected fixture
URLs don't need a password at construction time — drop it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:38:34 -07:00
Garry TanandClaude Fable 5 2b1a0811b3 chore: bump version and changelog (v0.46.1.0)
Issues #5 + #6 wave: pool-starvation cancellation + diagnostics, and opt-in
per-job process isolation. Version locations: VERSION, package.json,
CHANGELOG.md, openclaw.plugin.json, BOOTSTRAP_FOR_AGENTS.md stamp, and the
regenerated bootstrap template tree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:23:06 -07:00
Garry TanandClaude Fable 5 f870f1d3ef fix(minions,db): pre-landing review fixes — specialist + red-team findings (#5, #6)
Ship's review army (5 specialists + red team over the full diff; 0 critical
from specialists, 3 confirmed critical from red team) — all findings fixed:

Red team (critical):
- reserved-direct cap: removed the Math.max(1, size-1) floor — at
  direct_pool_size=1 it let a multi-minute reserve consume the ONLY direct
  session and starve claim/renewLock heartbeats (the #6 class reintroduced).
  cap = size - 1, direct routing only when cap >= 1; size<=1 uses the read
  pool (true status quo). Pinned by a size=1 routing test.
- silent group-kill failure: the SIGKILL escalation now logs loudly when
  delivery fails (distroless hosts without /bin/kill would otherwise void
  the kill guarantee with zero diagnostics while the job duplicated
  elsewhere), and skips the redundant signal when the child already exited.
- spawn-failure circuit breaker: a deterministically broken child CLI looped
  claim/release forever, invisible to the stall detector (every settle
  refreshes the progress clock). After 3 consecutive spawn/bootstrap
  failures the worker emits unhealthy(child_spawn_failing) for a
  process-manager restart; counter resets on any spawn that runs. Plus the
  predicate-mismatch guard: jobIsolation 'process' without childCliInvocation
  now throws at construction (it silently ran handlers inline while the
  evict path believed it was isolated).

Specialists (informational, all applied):
- performance: parent-side outcome decode is async (a 32MiB-capped file must
  not block the event loop running renewal ticks); /bin/kill by absolute
  path (also the security finding).
- security: lease payloads are shape-validated before reconstruction
  (corrupt outcome files degrade to generic); the child-CLI override is
  canonicalized to an absolute path so the fail-fast check validates the
  binary that actually spawns.
- data-migration: max_lifetime default is now a per-CONNECTION jitter
  FUNCTION (matching the postgres.js built-in shape — a pre-evaluated number
  synchronized every connection in a pool onto one recycle deadline);
  reserved.release() throws no longer leak the gauge or the direct permit.
- testing: child harnesses use a readiness handshake instead of fixed 400ms
  sleeps (CI-load flake); the orphan-watchdog test uses a real reaped pid
  (a magic high pid is allocatable under Linux pid_max); new pins for the
  dual-pool probe gating (probeDirect wired ONLY when isDualPoolActive),
  the executeRawDirect/transaction gauge seams incl. the sync begin()-throw
  leak guard, the ddl()-throw read-pool fallback, and the --job-isolation
  help text.
- maintainability: abort-reason literals shared via types.ts (dead
  'cancel'/'cancelled' entries dropped), DEFAULT_DIRECT_POOL_SIZE and
  CHILD_READ_POOL_MAX named, redundant dynamic imports removed, unrefTimer
  helper, getConnectionRouting shared accessor, docstring + fixture-header
  corrections.

Deferred with TODOS entries: raceWithAbortTimeout DRY helper (5 sites), lazy
handler resolution in run-child, e2e-lane negative tests for the run-child
bootstrap guards + operator-flow messages, behavioral withRefreshingLock test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 09:22:57 -07:00
Garry Tan 8adbf87443 Merge remote-tracking branch 'origin/master' into garrytan/fix-issue-5 2026-08-15 08:50:08 -07:00
Garry TanandClaude Fable 5 2aac6390cb fix(minions,db): adversarial-review hardening — shutdown/attempt semantics, TOCTOU semaphore, gauge + parity gaps (#5, #6)
A 23-agent adversarial review (5 dimension finders + per-finding refuters)
over the wave's diff confirmed 12 defects; all fixed here:

- [P2] run-child conflated worker SIGTERM with the per-job abort: cooperative
  handlers bailed mid-deploy, reported an error outcome, and the parent
  BURNED an attempt per routine deploy — while signal-ignoring handlers got
  the no-burn release (the exact inversion of the shutdown guarantee).
  SIGTERM now fires ONLY shutdownSignal (inline signal-separation parity —
  handlers finish + report inside the drain window), parent death still
  aborts both, and the parent classifies an ERROR outcome that arrives
  during shutdown as ChildWorkerShutdownError (released, not burned; a
  genuinely-failing job coinciding with a deploy gets one free retry).
- [P2] the reserved-direct semaphore was a check-then-increment spanning
  `await ddl()` — same-tick concurrent reserves could overshoot the cap and
  starve the heartbeat slot it exists to protect. The permit is now taken in
  the same synchronous frame as the check.
- [P3] RSS-watchdog drain (gracefulShutdown aborts BOTH signals, reason
  'watchdog') was classified as a per-job abort and burned attempts on
  innocent isolated jobs. Shutdown classification now wins unless the
  per-job reason is job-targeted (timeout/cancel/lock-*).
- [P3] force-evict's failJob('dead') could race executeJob's own recording
  in isolation mode (group SIGKILL at 25s + slow decode > 30s window) and
  dead-letter a job with attempts remaining — skipped when isolated (the
  inFlight eviction, which is what unblocks the worker, stays).
- [P3] child bootstrap exits were burned as handler crashes: exit 13 →
  ChildSpawnInfraError (release), exit 14 → new ChildNotClaimedError
  (release; the claim is provably owned elsewhere).
- [P3] missing handler in the child was 'generic' (retried to max_attempts)
  vs inline's immediate dead-letter — now 'unrecoverable' (parity).
- [P3] result-shape parity: the {value: x} wrap now happens CHILD-side,
  before JSON serialization, so Date/toJSON results can't flip the wrap
  decision across the boundary.
- [P3] child env no longer raises a stricter user GBRAIN_POOL_SIZE (pooler
  MaxClients tuning respected; explicit GBRAIN_JOB_CHILD_POOL_SIZE wins;
  invalid values fall back instead of flowing to the 10-conn fallback).
- [P3] transaction() gauge used a chained .finally that a synchronous
  begin() throw (nested tx on a clone) would skip — now try/finally.
- [P3 vacuity x3] new pins: db-lock heartbeat cancellation wiring +
  re-entrancy, the synthesize drain-loop guard + tick call (the shape guard
  only covers worker.ts), and GBRAIN_POOL_MAX_LIFETIME_S reaching a REAL
  constructed pool (postgres() is lazy — no I/O).

New tests: error-outcome-during-shutdown, watchdog double-abort,
timeout-beats-shutdown precedence, bootstrap exit codes, pool-size env
matrix, SIGTERM-only-aborts-shutdown (in-process emit), child-side wrap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:48:03 -07:00
Garry TanandClaude Fable 5 905aeb481d test: repoint the deadlineAtMs structural pin at the extracted job-context builder
The deadline-plumbing structural test grepped worker.ts for the literal
deadlineAtMs derivation, which moved verbatim into job-context.ts (the
builder shared by inline mode and 'jobs run-child'). The pin now checks the
derivation in job-context.ts AND that worker.ts calls buildJobContext — the
same contract, at its new home.

Full-suite triage note: an isolated A/B of the 22 files that failed in the
parallel full-suite run shows IDENTICAL results on this branch and on the
master base (290 pass / 5 fail — doctor-minions-check + unified-multimodal,
both env-dependent) — zero regression delta from this wave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:36:39 -07:00
Garry TanandClaude Fable 5 a2bbdb7c04 docs(minions,db): pool-starvation diagnostics + job isolation reference; file the follow-ups (#5, #6)
- KEY_FILES.md: entries for the six new modules (job-isolation,
  child-job-runner, run-child, job-context, db-probe, pool-gauge) and
  current-state updates for worker/queue/supervisor/jobs/db/db-lock/
  lock-renewal-tick/synthesize.
- minions-deployment.md: a --job-isolation section modeled on --nice — how
  the parent/child split works, preserved error semantics, orphan story, and
  the sizing notes (pooler CLIENT connection math: concurrency 15 ~ 73;
  --max-rss covers the worker only; spawn cost guidance; the lock token is a
  fencing token, not a secret).
- TESTING.md: inventory entries for the 12 new unit files + the e2e lane
  (which is wired EXPLICITLY into e2e.yml tier1 — no glob exists).
- TODOS.md: filed the 10 follow-ups, headlined by the P1-companion
  nested-checkout audit (the strongest remaining #6 root-cause candidate —
  this wave mitigates the starvation class and fixes the diagnostic; it does
  not claim to close every leak path), plus per-handler isolation policy,
  per-child RSS caps, the connection-budget clamp, autopilot pass-through,
  connection-audit release events, the doctor connection_routing check, and
  Sql-proxy checkout instrumentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:19:02 -07:00
Garry TanandClaude Fable 5 c2a6bc8078 feat(cli): --job-isolation flag, supervisor pass-through, e2e lane (#5)
The user-facing surface for per-job process isolation:

- jobs work --job-isolation <inline|process> (space or = form; env fallback
  GBRAIN_JOB_ISOLATION; default inline — fully opt-in). With 'process' the
  worker resolves the child CLI ONCE at startup (GBRAIN_JOB_CHILD_CLI ->
  compiled binary -> bun-dev fallback) and REFUSES to start on an
  unresolvable/nonexistent path — a bad path discovered per-job would stall
  the queue one released claim at a time. detectTini() wraps children when
  available. Startup banner names the mode + child CLI; combining with
  --max-rss prints a note that the watchdog now covers the worker only.
- jobs supervisor --job-isolation passes through via buildWorkerArgs as a
  CONDITIONAL push — inline/omitted keeps existing deployments' worker argv
  byte-identical (pinned arrays in supervisor-build-worker-args.test.ts are
  untouched; two new cases added).
- pool_starved fatal text now names the flag as a remedy (handler
  connections die with each job's child).
- help text for work + supervisor + the jobs index; flag registry
  regenerated.
- NEW test/e2e/job-isolation.test.ts, wired into e2e.yml tier1 EXPLICITLY —
  the workflow runs only named files (no glob), so an unwired e2e file would
  be silent coverage loss. Legs: concurrency-3 isolated drain through real
  children against real Postgres (the child-pool topology), and the REAL
  `jobs run-child` CLI entrypoint end-to-end (engine bootstrap, quiet
  handler registry, token validation, outcome protocol). Follows the #4128
  ambient-URL-guard conventions (explicit env in the e2e lane).
- serialization parity (codex-2 #8): a non-JSONB-serializable result fails
  loudly in BOTH modes (inline completeJob serialization vs child exit 15) —
  isolation never falsely completes a job inline mode would have failed.

Tests: jobs-isolation-flag.test.ts (parser matrix), extended
supervisor-build-worker-args + worker-job-isolation, cli-flag-validation
green via regen, jobs-subcommand-help.serial green (engine-free help path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:14:36 -07:00
Garry TanandClaude Fable 5 12b89ef46f feat(minions): process isolation — run claimed jobs in SIGKILL-able children (#5)
The parent-side seam. executeJob swaps ONE line — handler(context) vs
runJobInChild(...) — and every existing reporting branch (completeJob,
failJob dead/delayed, lease release, infra-abort no-burn) is reused verbatim
on the child's reconstructed outcome. Blast radius of a stuck or crashing
handler drops from N in-flight jobs to exactly one.

child-job-runner.ts:
- detached spawn (own process group) + tini wrap when available; stdio
  ['ignore','inherit','inherit'] so handler logs stream to the operator;
  per-job lifecycle log lines (spawned / exited code+signal);
- per-job abort -> group SIGTERM now, group SIGKILL at +25s (inside the 30s
  force-evict window, which stays as an untouched backstop) — force-eviction
  is now a real kill, not an abandonment;
- worker shutdown -> same SIGTERM so the child's handlers get the drain
  window to finish AND report; a child that reported before the kill
  completes normally; one that couldn't throws ChildWorkerShutdownError,
  which the worker RELEASES with no attempt burned — routine deploys must
  not burn attempts (codex-2 #7);
- pre-exec spawn failure -> ChildSpawnInfraError, also released with no
  attempt burned (one bad CLI path must not dead-letter a queue);
- child env contract: fenced lock token, outcome path, parent pid for the
  orphan watchdog, GBRAIN_POOL_SIZE=3 + GBRAIN_DIRECT_POOL_SIZE=1 bounds
  (children run no heartbeats; sockets die with the process — the point).

worker.ts: MinionWorkerOpts gains jobIsolation / childCliInvocation /
childTiniPath (defaults preserve inline behavior exactly); when isolated the
parent-side MinionJobContext is not built at all (the child builds its own).

Tests: child-job-runner.test.ts (real .mjs children: success + env contract,
error/lease outcome reconstruction, crash, SIGTERM-ignorer -> group SIGKILL,
pre-aborted, spawn ENOENT, both shutdown semantics);
worker-job-isolation.test.ts (real PGLite worker end-to-end: claim -> child
-> fenced completeJob with the REAL claim token, failJob on error outcome,
crash burns attempt, spawn failure releases with zero attempts burned).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:07:39 -07:00
Garry TanandClaude Fable 5 4a12fcc340 feat(minions): hidden 'jobs run-child' single-job entrypoint (#5)
The child side of process isolation. `gbrain jobs run-child --job-id N`
(internal; spawned by the worker, absent from user help):

- re-reads the job row and validates status='active' + lock-token match
  before running anything — a reclaimed/cancelled job exits 14 with the
  handler never invoked (the DB stays ground truth; no payload
  serialization across the boundary);
- registers the same handler surface as the worker via
  registerBuiltinHandlers({quiet}) — which includes plugin discovery, so
  plugin subagent jobs isolate identically — resolved through the new
  MinionWorker.getHandler() accessor;
- builds the shared token-fenced MinionJobContext against the CHILD's own
  engine, runs the handler, and writes ONE atomic outcome file: handler
  failure is an encoded error outcome with exit 0 (a reported failure is a
  successful report); only write-failure exits 15;
- runs NO worker machinery (no probe/stall/lock timers — the parent owns
  liveness). Installs a SIGTERM handler (fires ctx.signal + shutdownSignal
  so handlers get the drain window to finish and report) and a
  parent-liveness watchdog polling process.kill(parentPid, 0) — a ppid
  check is dead code under tini — that aborts the handler and hard-exits
  after a grace so orphaned LLM-bound work stops burning spend;
- CLI layer owns engine.disconnect() + process.exit() (engine-ownership
  invariant); PGLite exits 13 (isolation is Postgres-only, like jobs work).

Flag registry regenerated for the internal job-id flag.

Tests: run-child-entry.test.ts against real in-memory PGLite with a REAL
claim-minted token — success (incl. a fenced updateProgress landing),
handler-failure outcome, token-mismatch never-runs, missing job, missing
handler, and the parent-death watchdog aborting a live handler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 08:00:41 -07:00
Garry TanandClaude Fable 5 3a779b9752 feat(minions): job-isolation protocol, child exit codes, shared job-context builder (#5)
Foundation for per-job process isolation (no behavior change yet):

- job-isolation.ts — the parent<->child protocol: atomic outcome-file codec
  (tmp+rename; 32MiB decode cap that throws UnrecoverableError so oversize
  results die LOUDLY on attempt 1 instead of retrying identically or being
  silently truncated; decode errors report byte counts, never file content),
  handler-error encode/reconstruct preserving the two instanceof branches
  executeJob dispatches on (UnrecoverableError, RateLeaseUnavailableError),
  child argv/env contract, child-CLI resolution (env override -> compiled
  binary -> bun-dev fallback -> null for fail-fast), and killProcessGroup —
  children run detached in their own process group because SIGKILL on a tini
  pid alone kills tini and orphans the handler grandchild (tini cannot
  forward SIGKILL), and Bun rejects negative pids in process.kill()
  (oven-sh/bun#15791) so group signaling falls back to POSIX /bin/kill.
- worker-exit-codes.ts — reserved run-child codes 13 (usage/PGLite),
  14 (not claimed / token mismatch), 15 (result-write failed). Result-file
  presence, not the exit code, classifies the normal path: a reported
  handler FAILURE is still exit 0.
- job-context.ts — MinionJobContext builder extracted verbatim from
  executeJob so the child wires the exact same token-fenced DB callbacks;
  worker.ts now calls it (behavioral no-op, full minions suite green).

Tests: job-isolation-protocol.test.ts — codec round-trip + all decode
failure paths, instanceof reconstruction, invocation resolution, and REAL
detached-process group-kill tests incl. the grandchild-death guarantee
(runs under bun test, so the Bun negative-pid fallback is exercised for
real, not mocked).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:56:42 -07:00
Garry TanandClaude Fable 5 04a319e311 fix(db): route withReservedConnection to the direct session pool when dual-pool is active (#6)
Long-hold reserved work — CREATE INDEX CONCURRENTLY (vector-index), non-
transactional migration DDL, and backfill BEGIN..COMMIT batches (the observed
353s COMMIT session) — previously reserved from the worker's shared READ
pool, pinning slots under the 5-min pooler statement_timeout. It now reserves
from the DIRECT session lane, whose 30-min statement_timeout and
maintenance_work_mem GUCs are the right fit, and stops competing with handler
workload.

Heartbeat protection: concurrent direct reserves are capped at
directPoolSize - 1 (default 2 of 3) via a per-process semaphore so
claim/renewLock always keep >= 1 direct slot; overflow falls back to the
read pool — exactly the pre-change behavior, so this commit is strictly
never-worse than master. (Deliberate rejection of queue-for-a-permit: that
would block migrations behind multi-minute index builds. Per-process is the
correct scope: each process owns its own direct pool, so a CLI migration
cannot starve a worker's heartbeats.) Never rerouted inside an open
transaction (same guard shape as executeRawDirect); kill-switch collapse
degrades to status quo. Callers unchanged.

Tests: postgres-engine-reserved-routing.test.ts — direct when active, read
when kill-switched/in-tx, semaphore cap + overflow + permit release on fn
throw and on reserve() failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:52:31 -07:00
Garry TanandClaude Fable 5 a02bfe34ef feat(minions): pool-state health-probe diagnostics — pool starved vs server unreachable (#6)
"[health] DB unreachable" sent operators chasing database capacity while the
real fault was client-side: the server sat at ~10% of max_connections. The
probe now names the failing layer:

- New src/core/minions/db-probe.ts (hermetic, injected-deps — the
  lock-renewal-tick pattern): on read-pool probe failure, a 3s direct-lane
  SELECT 1 disambiguates. Direct OK -> verdict 'pool_starved' ("server IS
  reachable; the fault is in the transaction-pooler path — client pool
  exhaustion or a pooler-layer fault", deliberately an honest disjunction).
  Both fail -> 'server_unreachable'. No direct lane -> 'unknown'. Both
  probes carry AbortSignals — a hung probe is cancelled, never abandoned.
- New src/core/pool-gauge.ts: approximate in-flight counters at the engine's
  raw/direct/reserved/transaction seams, surfaced via a duck-typed
  PostgresEngine.getPoolDiagnostics() (no BrainEngine churn, no PGLite
  stub). Explicitly labeled a tracked SUBSET — template-path traffic is
  untracked and no waiter/available figures are derived (that would be
  invented telemetry). Counters use try/finally (runUnsafe throws
  synchronously on a pre-aborted signal) and clamp at zero.
- worker.ts probe adapter emits the verdict in every failure line and on the
  final unhealthy payload; exit semantics UNCHANGED (exiting on a starved
  pool is correct recovery — it frees all client-held slots).
- jobs.ts: verdict-aware fatal text, plus a startup warning when a
  Supabase-shaped engine is running single-pool (kill-switch collapse used
  to be silent — renewal + probes + workload all sharing one pool is the
  precondition for this incident class).
- Runbook: verdict interpretation table in queue-operations-runbook.md.

Tests: pool-gauge.test.ts (pure + engine seams incl. rejected-query and
sync-throw leak guards), db-probe.test.ts (full verdict matrix, signal
cancellation, fail-open diagnostics, no-waiter-wording pin).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:50:48 -07:00
Garry TanandClaude Fable 5 8caa2b0c88 fix(db): explicit jittered max_lifetime on all four client pools (#6)
Makes the pool connection lifetime explicit at every postgres() call site
(db.ts module singleton, engine instance pool, ConnectionManager read +
direct pools) and adds GBRAIN_POOL_MAX_LIFETIME_S as an incident escape
hatch (N seconds; 0 disables recycling).

NOT a behavior change at default: postgres.js (verified against the pinned
3.4.9) already defaults max_lifetime to 60*(30+rand*30) — 30-60 min,
jittered per pool — and max_lifetime only recycles connections as they
return to the pool; it cannot reclaim a leaked checkout. Framed accordingly:
explicitness + operator knob, not a fix for the starvation class (that is
the cancellation work in the two prior commits).

Tests: hermetic resolver suite (env forms, 0-disables, jitter bounds,
warn-once on invalid values, per-call jitter variance).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:43:55 -07:00
Garry TanandClaude Fable 5 222023f65f fix(minions,db): cancel timed-out wedge probes + db-lock refreshes (#6)
Same abandoned-racer class as the previous commit, in two more spots:

- probeQueueState raced probeQueueStateInner against its 1500ms budget but
  the losing wedge/age queries kept running on the pool after the race
  resolved — under pool exhaustion (the exact regime the probe exists to
  detect) the orphaned query held a slot and made the exhaustion worse. The
  timeout now aborts a per-probe signal threaded through queryWedgeSignals
  and the oldest-waiting age query. Closes the filed TODOS entry.
- withRefreshingLock raced handle.refresh() against heartbeatTimeoutMs the
  same way; DbLockHandle.refresh now accepts { signal } (Postgres forwards
  to executeRawDirect; PGLite ignores it — no pool to starve), the timeout
  aborts it, and a re-entrancy guard stops overlapping ticks (15s min
  cadence vs 30s default timeout could stack two).

Tests: new hermetic queue-probe-cancellation.test.ts (signal threading,
timeout-aborts, fast-path-not-aborted, fail-open contract).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:41:35 -07:00
Garry TanandClaude Fable 5 2344821ecd fix(minions): cancel abandoned probe + lock-renewal queries instead of orphaning pool slots (#6)
Three hot paths raced a live query against a timer and abandoned the loser,
leaving the query holding a checked-out pool slot for its full server-side
duration. Under a saturated transaction-mode pooler those orphaned slots
starve lock renewal ('lock-renewal-failed' cascades) and the health probe.

- Health probe: pass the deadline AbortController's signal into
  executeRaw('SELECT 1') so a hung probe is cancelled via postgres.js
  .cancel() (runUnsafe already wires signal -> pending.cancel()).
- Minion lock renewal: LockRenewalDeps.renewLock widened with optional
  { signal }; runLockRenewalTick aborts a per-call controller when the
  timeout wins the race; MinionQueue.renewLock forwards the signal to
  executeRawDirect. Optional-param widening keeps the 14 existing hermetic
  tests compiling untouched.
- Cycle drain renewal (synthesize.ts): the inline best-effort tick had no
  per-call timeout and no re-entrancy guard, so a hung renewLock stacked a
  fresh checked-out slot per interval firing. Extracted as exported
  runDrainRenewalTick (per-call signal + timeout + swallow) behind a
  tick-in-flight guard.

Tests: 2 new signal paths in worker-lock-renewal.test.ts, probe-signal
assertion in worker-supervised-db-probe.test.ts, new hermetic
minion-queue-renewlock-signal.test.ts + cycle-drain-renewal.test.ts.
scripts/check-worker-lock-renewal-shape.sh stays green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 07:38:12 -07:00
52 changed files with 4397 additions and 150 deletions
+3 -1
View File
@@ -96,7 +96,9 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Run Tier 1 E2E tests
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts
# job-isolation rides tier1 deliberately: e2e.yml runs only explicitly
# NAMED files (no glob) — an unwired e2e file is silent coverage loss.
run: bun test --timeout=60000 test/e2e/mechanical.test.ts test/e2e/mcp.test.ts test/e2e/job-isolation.test.ts
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
# #3485 preload guard: this job intentionally tests against a DB.
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- gbrain-runbook-stamp: 0.46.0.0 -->
<!-- gbrain-runbook-stamp: 0.46.1.0 -->
<!-- This stamp must equal the VERSION file at every release; CI enforces it
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
the installed binary and warns on skew. -->
+55
View File
@@ -2,6 +2,61 @@
All notable changes to GBrain will be documented in this file.
## [0.46.1.0] - 2026-08-15
**A stuck job can no longer take down your whole worker.** Field reports from
a production deployment ([#5](https://github.com/garrytan-agents/gbrain/issues/5),
[#6](https://github.com/garrytan-agents/gbrain/issues/6)) showed two
compounding failure modes: a handler that ignored its abort signal could only
be "force-evicted" (abandoned but still running, still holding connections),
and abandoned probe/renewal queries starved the connection pool until the
worker killed itself with a misleading "DB unreachable" — while the database
sat at a fraction of capacity. This release fixes the starvation class and
adds real per-job blast-radius control.
### Added
- **`gbrain jobs work --job-isolation process`** (also
`gbrain jobs supervisor --job-isolation process`, env
`GBRAIN_JOB_ISOLATION`): each claimed job runs in its own child process.
A stuck handler is group-SIGKILLed for real instead of abandoned, a crash
or memory blowup takes one job instead of all N, and the OS reclaims every
leaked resource when the child dies. The worker keeps claiming, renewing,
and recording; handler-error semantics (unrecoverable → dead, rate-lease →
no attempt burned, backoff otherwise) are preserved across the boundary.
Worker shutdown gives children the drain window to finish and report — a
routine deploy never burns a job attempt. Recommended for long-running
LLM-bound handlers; see the new section in `docs/guides/minions-deployment.md`.
- **Health-probe verdicts that name the failing layer.** When the worker's DB
probe fails, it now disambiguates via the direct session lane and says
`pool_starved` ("server IS reachable; the fault is in the
transaction-pooler path") or `server_unreachable` — instead of the blanket
"DB unreachable" that historically sent operators debugging database
capacity while the real fault was client-side. A startup warning also makes
single-pool mode (direct-lane kill switch) loud instead of silent, and
`docs/guides/queue-operations-runbook.md` gains a verdict-interpretation
table.
- `GBRAIN_POOL_MAX_LIFETIME_S`: explicit client-pool connection max-lifetime
knob (0 disables; default stays the per-connection 3060min jitter).
### Fixed
- **Timed-out DB probes and lock renewals are now cancelled, not abandoned.**
Every place that raced a query against a timer (health probe, minion lock
renewal, cycle-drain renewal, submit-time queue probes, DB-lock refresh)
previously let the losing query keep running on a checked-out connection —
under pool exhaustion each abandoned racer held a slot and made the
exhaustion worse, starving the lock heartbeat first. All five sites now
abort the query via its cancellation signal so the slot is released.
- Long-running maintenance holds (index rebuilds, non-transactional
migrations, backfill write batches) now reserve from the direct session
lane instead of pinning the worker's shared pool — capped so reserved
holds always leave a direct-lane slot for the claim/renewal heartbeats,
and falling back to the previous behavior when the direct lane is
unavailable.
Full operational detail: `docs/guides/minions-deployment.md` (isolation
sizing: connections, memory, spawn cost) and
`docs/guides/queue-operations-runbook.md` (probe verdicts).
## [0.46.0.0] - 2026-08-14
**Your other agents' sessions become brain knowledge.** Until now only Claude
+1 -1
View File
@@ -311,7 +311,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Opt-in per-job process isolation (`gbrain jobs work --job-isolation process`) runs each claimed job in its own SIGKILL-able child process, so a stuck handler dies for real and a crash takes one job instead of the whole worker; when the worker's DB health probe fails, it names the failing layer (`pool_starved` vs `server_unreachable`) instead of a blanket "DB unreachable". Sizing and rollout guidance in [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md); probe-verdict triage in [`docs/guides/queue-operations-runbook.md`](docs/guides/queue-operations-runbook.md). Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
+82 -8
View File
@@ -1,5 +1,87 @@
# TODOS
## Issues #5+#6 follow-ups (pool starvation + process isolation; plan: ~/.claude/plans/system-instruction-you-are-working-witty-moore.md)
- [ ] **P1-companion — nested-checkout audit + dev-mode detection.** **What:**
`transaction()` callers that invoke parent-engine methods (or module helpers
taking `engine` not `tx`) take a SECOND read-pool slot while holding the tx
slot — e.g. the `operations.ts` advisory-lock loop around `tx.addLink`. Under
a saturated pool this is a client-side self-deadlock class. Audit call sites;
add a dev-mode warning (e.g. a tx-depth counter consulted by `runUnsafe`).
**Why:** the #6 incident's exact 240s-idle sessions were never reproduced
under a debugger; this is the strongest remaining candidate — the shipped
wave mitigates the starvation class but does not close this path. **Effort:**
M. **Priority:** P1-companion.
- [ ] **P2 — per-handler isolation policy.** **What:** a per-handler-name set
(e.g. long-running LLM-bound handlers isolate, sub-second `lint`/`backlinks`
stay inline) instead of the all-or-nothing `--job-isolation process`.
**Why:** spawn cost (~0.31s) is noise for 644s subagent jobs, meaningful
for sub-second handlers; one worker should be able to mix. **Context:**
`worker.ts` executeJob's `isolated` gate is the seam. **Effort:** M.
**Priority:** P2.
- [ ] **P2 — per-child --max-rss caps.** **What:** RSS watchdog for isolation
children (the worker-level watchdog covers the worker only in process mode;
a startup note ships today). **Context:** child-job-runner.ts owns the child
lifecycle; a poll of the child's RSS + group-kill on breach mirrors the
worker watchdog. **Effort:** M. **Priority:** P2.
- [ ] **P2 — jobs-side connection-budget clamp for isolated workers.** **What:**
warn/clamp concurrency when `concurrency × (child pool + 1) + parent pools`
exceeds a configured budget (GBRAIN_MAX_CONNECTIONS-style; precedent
`sync-concurrency.ts:clampWorkersForConnectionBudget`). **Why:** isolation
multiplies pooler CLIENT connections (~73 at concurrency 15); today the
budget lives only in docs math. **Effort:** S. **Priority:** P2.
- [ ] **P3 — --job-isolation pass-through for the autopilot's embedded
supervisor.** **What:** `autopilot.ts` builds its own worker args; add the
conditional flag there (jobs supervisor already passes through). **Effort:**
S. **Priority:** P3.
- [ ] **P3 — runLockRenewalTick adoption in the cycle drain.** **What:**
`synthesize.ts` now uses the minimal `runDrainRenewalTick` (per-call signal +
guard); adopting the full tick would add the audit channel + bounded
reconnect. **Effort:** S. **Priority:** P3.
- [ ] **P3 — streaming child progress.** **What:** isolation children report
progress via their own token-fenced DB writes today (identical to inline);
an IPC stream would only add parent-side visibility (e.g. lifecycle events
in `jobs watch`). **Effort:** M. **Priority:** P3.
- [ ] **P3 — connection-audit release events + plain-idle visibility.**
**What:** `logConnectionEvent` never emits `release`, so the JSONL cannot
answer "who holds a slot"; and `getIdleBlockers` filters
`state='idle in transaction'` only — the #6 incident's plain-`idle` sessions
were invisible to it. **Effort:** M. **Priority:** P3.
- [ ] **P3 — doctor connection_routing check.** **What:** wire
`ConnectionManager.describeMode()` + `healthCheck()` (both currently
zero-caller outside tests) into a doctor check naming the routing mode,
kill-switch state, and per-pool probe latency. Comments in four files
already reference this check as if it existed. **Effort:** S.
**Priority:** P3.
- [ ] **P3 — isolation test-gap follow-ups (pre-landing review).** **What:**
(a) spawned-CLI negative tests for `jobs run-child` bootstrap guards (PGLite
→ exit 13; missing job-id/env → exit 13) and for `jobs work` with
isolation on + an unresolvable child CLI (fail-fast exit 1) — both need a
real engine bootstrap so they live in the e2e lane; (b) a behavioral (not
structural) test driving `withRefreshingLock` with a hung injected
`handle.refresh` (signal aborted at timeout, no overlapping ticks); (c) a
force-evict-skip test for isolation mode (needs the 30s evict window made
injectable); (d) operator-flow message tests (verdict-tailored FATAL text,
single-pool startup banner). **Why:** the ship coverage audit scored the
wave 82% — these are the surviving gaps. **Effort:** M. **Priority:** P3.
- [ ] **P3 — raceWithAbortTimeout shared helper.** **What:** the
"Promise.race a query vs a setTimeout that aborts an AbortController,
clearTimeout in finally" pattern now exists at five sites (db-probe
withDeadline, synthesize runDrainRenewalTick, lock-renewal-tick callAbort,
db-lock tickAbort, supervisor probeAbort), each re-deriving the same
invariants. Extract one helper and adopt it. **Effort:** S. **Priority:** P3.
- [ ] **P3 — lazy handler resolution in run-child.** **What:** every isolation
child runs full registerBuiltinHandlers (incl. plugin discovery) to resolve
ONE handler; the job name is known from the row — a resolve-by-name path
would skip discovery for builtins. Matters only if isolation is ever used
for short jobs (documented as not the target). **Effort:** S. **Priority:** P3.
- [ ] **P3 — full checkout instrumentation via a Sql proxy.** **What:** the
CheckoutGauge covers raw/direct/reserved/tx seams only; tagged-template
traffic (most engine load) is untracked. A proxy around the postgres.js Sql
callable could count real checkouts — investigate cost/fragility before
building. **Why:** would turn the probe's "tracked subset" caveat into full
coverage. **Effort:** M. **Priority:** P3.
## Security-process follow-ups (filed with Wave 1 of the fix-wave campaign, 2026-08-14)
- [ ] **P2 — Vulnerability disclosure policy.** **What:** a written disclosure
@@ -254,14 +336,6 @@ fix-now findings landed on the branch; these four are the review-deferred tail.
`requestToolsPersistLimiter`; the surface_change audit rows already give
a DB-side count to enforce against if needed. **Effort:** medium.
**Priority:** P3.
- [ ] **P3 — cancel (not just abandon) timed-out submit-time queue probes.**
**What:** the WP5 wedge/pause probes time-bound via Promise.race, but the
losing query keeps running on the pool after the race resolves. Wire
AbortSignal / statement_timeout so a slow probe releases its slot. **Why:**
under pool exhaustion (the exact regime the probes exist to detect) an
abandoned probe query holds a pooler slot and makes the exhaustion worse.
**Context:** `src/core/minion/supervisor.ts` queryWedgeSignals callers in
`src/core/operations.ts` submit paths. **Effort:** small. **Priority:** P3.
- [ ] **P3 — document the status --json snapshot union under schema_version.**
**What:** a short protocol note (docs/progress-events.md sibling) pinning
the `get_status_snapshot` v2 shape as a discriminated union on
+1 -1
View File
@@ -1 +1 @@
0.46.0.0
0.46.1.0
+13
View File
@@ -334,6 +334,18 @@ Unit tests and what they cover:
- `test/enrichment-service.test.ts` — entity slugification, extraction, tier escalation.
- `test/data-research.test.ts` — recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping.
- `test/minions.test.ts` — Minions job queue: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, `child_done` inbox, attachments, removeOnComplete/Fail, `max_stalled` clamp/default/plumbing coverage.
- `test/minion-queue-renewlock-signal.test.ts``renewLock` forwards its optional AbortSignal to `executeRawDirect` (stub-engine capture); legacy 3-arg calls unchanged; token-fence miss returns false.
- `test/cycle-drain-renewal.test.ts``runDrainRenewalTick` (cycle drain): per-call signal aborted on timeout (slot released), onLost once on a lost fence, throws swallowed, hung renewal resolves at the deadline.
- `test/queue-probe-cancellation.test.ts``probeQueueState`/`queryWedgeSignals` signal threading: the 1500ms budget CANCELS the losing probe query; fast-path signals never abort; throw still collapses to `{probe_failed: true}`.
- `test/db-pool-max-lifetime.test.ts``resolveMaxLifetimeSeconds`: env forms, 0-disables, 3060min jitter bounds, warn-once on invalid, per-call jitter variance.
- `test/pool-gauge.test.ts``CheckoutGauge` pure semantics + the PostgresEngine seams with fake pools: counted while in flight, released on resolve, on REJECTED queries, and on the SYNCHRONOUS pre-aborted-signal throw (leak guards); `getPoolDiagnostics` fail-open.
- `test/db-probe.test.ts``runDbProbe` verdict matrix (pool_starved / server_unreachable / unknown), honest-disjunction + no-waiter-arithmetic wording pins, hung probes cancelled via their signals, diagnostics absent/throwing fail open.
- `test/postgres-engine-reserved-routing.test.ts``withReservedConnection` routing: direct pool when dual-pool active, read pool when kill-switched/in-tx, semaphore cap (directPoolSize1) with read-pool overflow, permit released on fn throw and reserve failure.
- `test/job-isolation-protocol.test.ts` — outcome-file codec round-trip + every decode failure path (missing/malformed/oversize→UnrecoverableError; byte counts, never content), handler-error instanceof reconstruction, child-CLI invocation resolution, and REAL detached-process `killProcessGroup` tests incl. the grandchild-death guarantee (exercises the Bun negative-pid `/bin/kill` fallback for real under `bun test`).
- `test/run-child-entry.test.ts``runChildJobEntry` on real in-memory PGLite with a REAL claim-minted token: success (fenced updateProgress lands), handler-failure outcome (exit 0), token-mismatch never runs the handler (exit 14), missing job/handler, parent-death watchdog aborts a live handler.
- `test/child-job-runner.test.ts``runJobInChild` against real .mjs children: success + full env contract (incl. `GBRAIN_DIRECT_POOL_SIZE=1`), error/lease outcome reconstruction, crash, SIGTERM-ignorer → group SIGKILL at the injected grace, pre-aborted signal, spawn ENOENT → `ChildSpawnInfraError`, worker-shutdown drain (report-during-drain completes; non-reporting kill → `ChildWorkerShutdownError`).
- `test/worker-job-isolation.test.ts` — full parent path on PGLite with the `fake-run-child.mjs` fixture: claim → child → fenced completeJob (real token over env), error outcome → failJob, crash burns the attempt, spawn failure RELEASES with zero attempts burned, and the codex-2 #8 serialization-parity pin (unreportable results fail in BOTH modes, never falsely complete).
- `test/jobs-isolation-flag.test.ts``parseJobIsolationFlag`: space/= forms, env fallback + flag-wins, empty-env default, other flags untouched.
- `test/extract.test.ts` — link extraction, timeline extraction, frontmatter parsing, directory type inference.
- `test/extract-db.test.ts``gbrain extract --source db`: typed link inference, idempotency, `--type` filter, `--dry-run` JSON output.
- `test/extract-fs.test.ts``gbrain extract --source fs`: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard for the N+1 dedup bug.
@@ -400,6 +412,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/sync.test.ts``--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
- `test/e2e/job-isolation.test.ts` — process isolation on real Postgres (DATABASE_URL-gated, wired EXPLICITLY into `.github/workflows/e2e.yml` tier1 — the workflow runs only named files): a concurrency-3 isolated drain through real child processes (the `fake-run-child.mjs` fixture — real spawns, no child DB pools), and the REAL `jobs run-child` CLI entrypoint end-to-end (engine bootstrap incl. the child's own pools, quiet handler registry, token validation, outcome protocol).
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972#2015#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
- `test/e2e/volunteer-context-postgres.test.ts``volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
File diff suppressed because one or more lines are too long
+63
View File
@@ -87,6 +87,69 @@ check warns if what you asked for isn't what's actually running (e.g. a
negative value denied without privilege, or an OS `RLIMIT_NICE` clamp). This
is distinct from the concurrency / inflight cap and composes with it.
### Per-job process isolation (`--job-isolation process`)
By default all concurrency slots execute inside one worker process. A
handler that ignores its abort signal can only be force-evicted — the
promise is abandoned, still running, still holding connections and memory —
and any worker exit destroys every in-flight job at once. With isolation on,
each claimed job runs in its own child process: a stuck handler is
group-SIGKILLed for real (group signaling under Bun falls back to POSIX
`/bin/kill`; if that's unavailable the worker logs that isolation is
degraded), a crash or OOM in a child takes that one job instead of all N,
and the OS reclaims every leaked resource when the child dies:
```bash
# Recommended for long-running LLM-bound handlers (subagent):
gbrain jobs supervisor --concurrency 4 --job-isolation process
# Bare worker, or durably via env:
GBRAIN_JOB_ISOLATION=process gbrain jobs work --concurrency 4
```
How it works: the worker keeps claim, lock renewal, and all result
recording; the child (an internal `run-child` entrypoint of the same gbrain
binary) re-validates the claim, runs the handler with its own small engine
pool, and reports one atomic outcome file. Handler-error semantics are
preserved across the boundary (unrecoverable → dead, rate-lease → no attempt
burned, everything else → normal backoff). On worker shutdown children get
the drain window to finish and report; a child killed before reporting is
released with no attempt burned. If the worker dies hard, the orphaned child
self-terminates via a parent-liveness watchdog and the stall sweeper
requeues the job after lock expiry — the lock token fences the orphan's
queue writes (result recording, progress, state transitions) into no-ops.
The handler's own side effects (page writes through its engine) can still
land until the watchdog stops the child; that window is the watchdog's
poll + grace, not unbounded.
Sizing notes:
- **Connections:** each child opens its own small pools (read 3 by default,
override via `GBRAIN_JOB_CHILD_POOL_SIZE`; direct 1). Worked example at
concurrency 15: 15×(3+1) + the worker's 10+3 ≈ **73 client connections**
total — 55 ride the transaction-pooler lane (multiplexed, no extra server
backends) and 18 are lazy direct session-lane connections, each holding a
real server backend while open. Budget the pooler-lane count against your
pooler's client limit and the session-lane count against
`max_connections`.
- **Memory:** `--max-rss` covers the WORKER process only in this mode
(handler memory lives in the children; the worker prints a note when both
are set). There is no per-child RSS cap yet — a runaway child is contained
only by host/container limits. Size host memory for concurrency × handler
footprint.
- **Spawn cost:** ~0.31s per job (engine connect included) — noise for
long-running handlers, meaningful for sub-second ones (`lint`,
`backlinks`). Keep those inline or on a separate inline worker.
- **Security note:** the child receives the job's lock token via env. It is
a *fencing* token (split-brain protection), not a secret — same-user env
already contains the database URL.
- **Child CLI resolution:** the worker fail-fast validates the child CLI at
startup (compiled `gbrain` binary, bun-dev fallback, or the
`GBRAIN_JOB_CHILD_CLI` env override — the ops/test escape hatch). Three
consecutive child spawn/bootstrap failures self-exit the worker as
unhealthy (a deterministically broken child CLI) for process-manager
restart instead of burning attempts across the queue.
### Which supervisor when?
The supervisor solves in-process crash recovery. Platform-level
+27 -1
View File
@@ -71,7 +71,10 @@ gbrain jobs get <id>
## Rescue actions (in order of escalation)
```bash
# Force-kill a single stuck job:
# Cancel a single stuck job (inline mode: cooperative — the handler must
# observe its abort signal, and after 30s it is force-evicted from tracking
# but the promise keeps running; with --job-isolation process the child is
# actually SIGTERM→SIGKILLed once cancellation is detected):
gbrain jobs cancel <id>
# Clear a specific job entirely (last resort):
@@ -119,6 +122,29 @@ claiming. Start one:
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work --concurrency 4
```
## Reading the DB-probe verdicts (pool starved vs server unreachable)
When the worker's health probe fails repeatedly, the terminal
`[health] DB probe failed N consecutive times (verdict: ...)` line — and the
`unhealthy` payload the supervisor sees — carries a verdict that names the
failing LAYER (the intermediate `(N/3)` lines log only the failure detail).
Read it before touching anything — the historical failure mode here was
hours spent evaluating a database instance upgrade while the server sat at
10% of max_connections.
| Verdict | What it means | What to do |
|---|---|---|
| `pool_starved` | The read-pool probe failed but the DIRECT-lane probe succeeded — the database server is reachable; the fault is in the transaction-pooler path (client pool exhaustion or a pooler-layer fault; the probe deliberately does not distinguish the two). | Look at client-side load: long-running handler queries holding slots, `GBRAIN_POOL_SIZE` too small for the workload, or a pooler-layer incident. Do NOT resize the database. The worker exit is correct recovery — it frees every client-held slot. |
| `server_unreachable` | Both the pooler lane and the direct lane failed. | Check connectivity/capacity first: network, DNS, the database itself. Both-lanes-failed is the evidence — credential/config errors or a saturated direct lane can also land here, so glance at the probe detail text before concluding the server is down. |
| `unknown` | The read probe failed and no direct lane exists to disambiguate (single-pool mode: non-Supabase, kill switch active, or no derivable direct URL). | Check the startup log for the single-pool warning; consider `GBRAIN_DIRECT_DATABASE_URL` so future incidents self-diagnose. |
The `gbrain-tracked in flight` counts in the message are a tracked SUBSET
(raw/direct/reserved/transaction seams only) — most template-path queries are
untracked, so `0 in flight` next to a `pool_starved` verdict means the
saturation lives in that untracked traffic or at the pooler layer itself,
not that the pool is idle. The verdict, not the counts, is the
authoritative signal.
## Related
- [Minions worker deployment](minions-deployment.md) — supervisor lifecycle,
+1 -1
View File
@@ -1921,7 +1921,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
**Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. **Obsidian-style vaults:** bare `[[note-name]]` wikilinks that point across folders — you wrote `[[struktura]]` but the page lives at `projects/struktura.md` — resolve by basename once you opt in with `gbrain config set link_resolution.global_basename true`. Off by default; `gbrain doctor` tells you how many edges you'd gain before you flip it. See [migrating an Obsidian vault](INSTALL_FOR_AGENTS.md#step-45-wire-the-knowledge-graph).
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Opt-in per-job process isolation (`gbrain jobs work --job-isolation process`) runs each claimed job in its own SIGKILL-able child process, so a stuck handler dies for real and a crash takes one job instead of the whole worker; when the worker's DB health probe fails, it names the failing layer (`pool_starved` vs `server_unreachable`) instead of a blanket "DB unreachable". Sizing and rollout guidance in [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md); probe-verdict triage in [`docs/guides/queue-operations-runbook.md`](docs/guides/queue-operations-runbook.md). Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.46.0.0",
"version": "0.46.1.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
"family": "bundle-plugin",
"configSchema": {
+1 -1
View File
@@ -157,7 +157,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.46.0.0",
"version": "0.46.1.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.5",
+200 -4
View File
@@ -6,7 +6,12 @@
import type { BrainEngine } from '../core/engine.ts';
import { MinionQueue } from '../core/minions/queue.ts';
import { MinionWorker } from '../core/minions/worker.ts';
import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts';
import {
WORKER_EXIT_RSS_WATCHDOG,
JOB_CHILD_EXIT_USAGE,
} from '../core/minions/worker-exit-codes.ts';
import { CHILD_ENV, resolveChildCliInvocation } from '../core/minions/job-isolation.ts';
import { runChildJobEntry } from '../core/minions/run-child.ts';
import type { MinionHandler, MinionJob, MinionJobStatus } from '../core/minions/types.ts';
import type { PaceKeyOverrides } from '../core/pace-mode.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
@@ -154,6 +159,32 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
return parsed;
}
export type JobIsolationMode = 'inline' | 'process';
/**
* issue #5: `--job-isolation <inline|process>` (space or `=` form), env
* fallback GBRAIN_JOB_ISOLATION, default inline. `process` runs each claimed
* job in a SIGKILL-able child process blast radius 1 job instead of N.
* Env injected as a param so tests never mutate process.env (rule R1).
* Invalid values fail fast (parseMaxRssFlag convention).
*/
export function parseJobIsolationFlag(
args: string[],
env: NodeJS.ProcessEnv = process.env,
): JobIsolationMode {
let raw: string | undefined;
const eqForm = args.find((a) => a.startsWith('--job-isolation='));
if (eqForm !== undefined) raw = eqForm.slice('--job-isolation='.length);
if (raw === undefined) raw = parseFlag(args, '--job-isolation');
if (raw === undefined || raw === '') raw = env.GBRAIN_JOB_ISOLATION;
if (raw === undefined || raw === '') return 'inline';
if (raw === 'inline' || raw === 'process') return raw;
console.error(
`Error: invalid job isolation mode ${JSON.stringify(raw)}. Valid: inline, process.`,
);
process.exit(1);
}
/**
* #3026: the thin-client `list`/`get` branches receive jobs as parsed JSON
* off the MCP wire, where every timestamp is an ISO string but formatJob /
@@ -269,11 +300,13 @@ USAGE
gbrain jobs watch [--json] [--follow] [--refresh-ms=N]
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
[--health-interval MS] [--nice N]
[--job-isolation inline|process]
gbrain jobs supervisor [start] [--detach] [--json]
[--concurrency N] [--queue Q] [--pid-file PATH]
[--max-crashes N] [--health-interval N]
[--allow-shell-jobs] [--cli-path PATH]
[--max-rss MB] [--nice N]
[--job-isolation inline|process]
--nice N OS scheduling priority, -20 (highest) to 19 (nicest). Lowers CPU
priority without cutting concurrency full throughput when the
@@ -344,9 +377,18 @@ const JOBS_SUBCOMMAND_HELP: Record<string, string> = {
USAGE
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
[--health-interval MS] [--nice N]
[--job-isolation inline|process]
OPTIONS
--queue Q Queue to claim from (default: default)
--job-isolation M inline (default): handlers run in the worker process.
process: each claimed job runs in its own child
process a stuck handler is group-SIGKILLed instead
of abandoned, and a crash takes one job, not all N.
Env fallback: GBRAIN_JOB_ISOLATION. Recommended for
long-running LLM-bound handlers (subagent). Note:
--max-rss then covers the worker only, and each child
adds ~4 pooler client connections.
--concurrency N Max jobs in flight. Resolution: flag, then
GBRAIN_WORKER_CONCURRENCY env, then 1. Values < 1
are clamped to 1 with a loud stderr note.
@@ -376,6 +418,7 @@ USAGE
[--max-crashes N] [--health-interval N]
[--allow-shell-jobs] [--cli-path PATH]
[--max-rss MB] [--nice N]
[--job-isolation inline|process]
gbrain jobs supervisor status [--json] [--pid-file PATH]
gbrain jobs supervisor stop [--json] [--pid-file PATH]
@@ -397,6 +440,7 @@ OPTIONS (start)
--cli-path PATH Explicit gbrain binary for the worker child
--max-rss MB RSS watchdog for the worker (same rules as jobs work)
--nice N OS priority for supervisor + worker children
--job-isolation M Passed through to the worker (see jobs work --help)
EXIT CODES (start)
0 clean shutdown 1 max crashes exceeded
@@ -1185,6 +1229,59 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
process.exit(0);
}
case 'run-child': {
// INTERNAL (issue #5 process isolation): spawned by `jobs work` with
// process isolation enabled. One job, one process: validate the claim,
// run the handler with the child's own engine, write ONE outcome file,
// exit. Deliberately absent from user-facing help. The CLI layer owns
// engine.disconnect() + process.exit() (engine-ownership invariant).
{
const config = loadConfig();
if (config?.engine === 'pglite') {
console.error('[run-child] process isolation requires the Postgres engine.');
await engine.disconnect();
process.exit(JOB_CHILD_EXIT_USAGE);
}
const jobIdRaw = parseFlag(args, '--job-id');
const jobId = jobIdRaw != null ? parseInt(jobIdRaw, 10) : NaN;
const lockToken = process.env[CHILD_ENV.lockToken];
const resultPath = process.env[CHILD_ENV.resultPath];
const parentPidRaw = parseInt(process.env[CHILD_ENV.parentPid] ?? '0', 10);
if (!Number.isInteger(jobId) || jobId <= 0 || !lockToken || !resultPath) {
console.error(
'[run-child] internal command spawned by the jobs worker; requires ' +
`a numeric job id plus ${CHILD_ENV.lockToken} and ${CHILD_ENV.resultPath} in env.`,
);
await engine.disconnect();
process.exit(JOB_CHILD_EXIT_USAGE);
}
// Same handler surface as the worker: registerBuiltinHandlers also
// performs plugin discovery, so plugin subagent jobs isolate too.
const throwaway = new MinionWorker(engine, { queue: 'default', concurrency: 1 });
await registerBuiltinHandlers(throwaway, engine, { quiet: true });
let code: number;
try {
code = await runChildJobEntry(
engine,
{
jobId,
lockToken,
resultPath,
parentPid: Number.isInteger(parentPidRaw) && parentPidRaw > 0 ? parentPidRaw : 0,
},
{ resolveHandler: (name) => throwaway.getHandler(name) },
);
} catch (e) {
console.error(`[run-child] fatal: ${e instanceof Error ? e.message : String(e)}`);
code = 1;
}
await engine.disconnect();
process.exit(code);
}
}
// eslint-disable-next-line no-fallthrough -- unreachable: the case above always exits
case 'work': {
// Check if PGLite
const config = (await import('../core/config.ts')).loadConfig();
@@ -1245,11 +1342,78 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
}
}
// issue #5: per-job process isolation. Resolve + validate the child CLI
// invocation ONCE at startup and refuse to start on failure — a bad
// path discovered per-job would release every claim as infra failures
// (never dead-lettering, but never progressing either).
const jobIsolation = parseJobIsolationFlag(args);
let childCliInvocation: { cmd: string; argsPrefix: string[] } | null = null;
let childTiniPath = '';
if (jobIsolation === 'process') {
const { resolveGbrainCliPath } = await import('./autopilot.ts');
const inv = resolveChildCliInvocation(
process.env,
process.execPath,
process.argv[1],
() => resolveGbrainCliPath(),
);
if (!inv) {
console.error(
'Error: process isolation needs a resolvable gbrain CLI for job children ' +
'(compiled binary on PATH, or GBRAIN_JOB_CHILD_CLI override).',
);
process.exit(1);
}
// Canonicalize BEFORE validating: existsSync on a relative name checks
// cwd while spawn() resolves via PATH — the validated file and the
// executed binary could differ (security review). Resolving to an
// absolute path makes the fail-fast check and the spawn agree.
const { existsSync: childCliExists } = await import('node:fs');
const { resolve: resolveCliPath } = await import('node:path');
inv.cmd = resolveCliPath(inv.cmd);
if (!childCliExists(inv.cmd)) {
console.error(
`Error: resolved child CLI does not exist: ${inv.cmd} ` +
'(set GBRAIN_JOB_CHILD_CLI to a valid gbrain binary).',
);
process.exit(1);
}
childCliInvocation = inv;
const { detectTini } = await import('../core/minions/spawn-helpers.ts');
childTiniPath = detectTini();
if (maxRssMb > 0) {
console.error(
'[gbrain jobs] note: with process isolation on, the --max-rss watchdog covers the ' +
'WORKER process only — handler memory now lives in job children. Per-child caps are ' +
'a filed follow-up; size host memory for concurrency x handler footprint.',
);
}
}
try { await queue.ensureSchema(); }
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
// issue #6: the direct-pool kill switch collapses lock renewal, health
// probes, and handler workload onto ONE shared pool — silently. Make
// the collapse loud at startup so a later 'pool_starved' incident has
// an obvious prior warning instead of a mystery.
{
const { getConnectionRouting } = await import('../core/minions/db-probe.ts');
const cm = getConnectionRouting(engine);
if (cm?.isDualPoolActive && !cm.isDualPoolActive()) {
const killSwitched = cm.describeMode?.().kill_switch_active === true;
console.error(
`[gbrain jobs] single-pool mode: lock renewal, health probes and handler workload share ` +
`one connection pool${killSwitched ? ' (direct-lane kill switch is active)' : ''}. ` +
`Under heavy handler load this pool can starve the lock heartbeat. For Supabase brains, ` +
`ensure the direct (5432) host is reachable or set GBRAIN_DIRECT_DATABASE_URL.`,
);
}
}
const worker = new MinionWorker(engine, {
queue: queueName, concurrency, maxRssMb, healthCheckInterval,
jobIsolation, childCliInvocation, childTiniPath,
});
await registerBuiltinHandlers(worker, engine);
@@ -1259,9 +1423,37 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
// the external PM (systemd, Docker, cron watchdog) restart cleanly.
worker.on('unhealthy', (info) => {
if (info.reason === 'db_dead') {
// issue #6: name the failing LAYER, not just "DB unreachable" —
// that message sent operators chasing database capacity while the
// real fault was client-side pool exhaustion. Exiting is still
// correct recovery either way (it frees every client-held slot).
if (info.verdict === 'pool_starved') {
console.error(
`[health] FATAL: connection-pool path saturated after ${info.consecutiveFailures} probes — ` +
`the database server itself is reachable. (${info.message}) ` +
`Likely causes: long-running handler queries holding pool slots, or too-small GBRAIN_POOL_SIZE ` +
`for this workload. Consider --job-isolation process for long-running handlers ` +
`(handler connections then die with each job's child process). ` +
`Exiting for process-manager restart (frees all client-held slots).`,
);
} else if (info.verdict === 'server_unreachable') {
console.error(
`[health] FATAL: database server unreachable after ${info.consecutiveFailures} probes ` +
`(both pooler and direct lanes failed). (${info.message}) ` +
`Exiting for process-manager restart.`,
);
} else {
console.error(
`[health] FATAL: DB probe failed ${info.consecutiveFailures} consecutive times (${info.message}). ` +
`Exiting for process-manager restart.`,
);
}
} else if (info.reason === 'child_spawn_failing') {
console.error(
`[health] FATAL: DB unreachable after ${info.consecutiveFailures} probes (${info.message}). ` +
`Exiting for process-manager restart.`,
`[health] FATAL: ${info.consecutiveFailures} consecutive job-child spawn/bootstrap ` +
`failures (${info.message}). The child CLI is deterministically broken — fix the ` +
`worker's child CLI configuration (or GBRAIN_JOB_CHILD_CLI). Exiting for ` +
`process-manager restart.`,
);
} else {
console.error(
@@ -1290,7 +1482,10 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
: `, health-check: ${Math.round(healthCheckInterval / 1000)}s`)
: '';
const niceNote = niceResult ? `, nice: ${formatNice(niceResult.effective ?? niceVal!)}` : '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote}${niceNote})`);
const isolationNote = jobIsolation === 'process'
? `, isolation: process (child cli: ${childCliInvocation?.cmd}${childTiniPath ? ', tini' : ''})`
: '';
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote}${niceNote}${isolationNote})`);
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
// Register in the live worker registry (issue #1815) so jobs stats / doctor
@@ -1609,6 +1804,7 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
allowShellJobs,
json: jsonMode,
maxRssMb,
jobIsolation: parseJobIsolationFlag(args),
...(supNice !== undefined ? { nice_requested: supNice } : {}),
...(supNiceResult?.effective != null ? { nice_effective: supNiceResult.effective } : {}),
...(supNiceResult?.error ? { nice_error: supNiceResult.error } : {}),
+2 -2
View File
@@ -40,7 +40,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'config': ['--aliases', '--all', '--brain', '--column', '--coverage-override', '--detail', '--embedding-dimensions', '--embedding-model', '--fast', '--federated-read', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--markdown', '--model', '--multimodal', '--no-embedding', '--no-extract', '--no-federated', '--pattern', '--pending', '--pglite', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--token-ttl', '--yes'],
'connect': ['--agent', '--bearer-token-env-var', '--bind', '--brain', '--client-id', '--client-secret', '--force', '--grant-types', '--help', '--http', '--install', '--json', '--name', '--oauth', '--public-url', '--register', '--scope', '--scopes', '--show-token', '--source', '--timeout-ms', '--token', '--token-endpoint-auth-method', '--url', '--version', '--yes'],
'conversation-parser': ['--aliases', '--all', '--brain', '--help', '--include-null-signature', '--json', '--no-extract', '--pattern', '--pending', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--thin'],
'doctor': ['--ab', '--abbrev-ref', '--abi', '--abort', '--aliases', '--all', '--allow-shell-jobs', '--allow-unverified-remote', '--auto', '--auto-fix', '--auto-update', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--branch', '--break-lock', '--build-index', '--by-mention', '--by-type', '--cached', '--check', '--column', '--compile', '--concurrency', '--confidence', '--confirm', '--content-audit', '--count', '--days', '--delete-brain', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exclude-standard', '--exclusive', '--explain', '--fast', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--force-schema', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--get', '--git-dir', '--git-path', '--grant-types', '--harness', '--health-interval', '--help', '--history', '--home', '--http', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--include-pseudo', '--index-audit', '--init', '--input', '--is-inside-work-tree', '--jq', '--json', '--lang', '--limit', '--local', '--locks', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-crashes', '--max-jobs', '--max-rss', '--max-usd', '--mcp-only', '--migrate-only', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--nice', '--no', '--no-cron', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-mutate', '--no-verify', '--oauth-client-secret', '--older-than', '--once', '--others', '--overwrite', '--parallel', '--params', '--pat-file', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--priority', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--push-only', '--query', '--queue', '--quiet', '--rebase', '--rebuild-rollup', '--refresh', '--refresh-unqualified', '--regenerate', '--remediate', '--remediation-plan', '--remove', '--repo', '--reset', '--resolve', '--restore-only', '--resume', '--review-lower', '--rollback', '--scope', '--scopes', '--set', '--short', '--show-current', '--show-toplevel', '--since', '--skills-dir', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slugs', '--source', '--source-id', '--stale', '--stats', '--status', '--strategy', '--strict', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--target', '--target-score', '--thin', '--timeout', '--to', '--token', '--token-ttl', '--top-k', '--type', '--undo-wave', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--use-captured-snapshot', '--verbose', '--verify', '--version', '--window', '--with-calibration', '--workers', '--yes'],
'doctor': ['--ab', '--abbrev-ref', '--abi', '--abort', '--aliases', '--all', '--allow-shell-jobs', '--allow-unverified-remote', '--auto', '--auto-fix', '--auto-update', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--branch', '--break-lock', '--build-index', '--by-mention', '--by-type', '--cached', '--check', '--column', '--compile', '--concurrency', '--confidence', '--confirm', '--content-audit', '--count', '--days', '--delete-brain', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exclude-standard', '--exclusive', '--explain', '--fast', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--force-schema', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--get', '--git-dir', '--git-path', '--grant-types', '--harness', '--health-interval', '--help', '--history', '--home', '--http', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--include-pseudo', '--index-audit', '--init', '--input', '--is-inside-work-tree', '--job-isolation', '--jq', '--json', '--lang', '--limit', '--local', '--locks', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-crashes', '--max-jobs', '--max-rss', '--max-usd', '--mcp-only', '--migrate-only', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--nice', '--no', '--no-cron', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-mutate', '--no-verify', '--oauth-client-secret', '--older-than', '--once', '--others', '--overwrite', '--parallel', '--params', '--pat-file', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--priority', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--push-only', '--query', '--queue', '--quiet', '--rebase', '--rebuild-rollup', '--refresh', '--refresh-unqualified', '--regenerate', '--remediate', '--remediation-plan', '--remove', '--repo', '--reset', '--resolve', '--restore-only', '--resume', '--review-lower', '--rollback', '--scope', '--scopes', '--set', '--short', '--show-current', '--show-toplevel', '--since', '--skills-dir', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--slugs', '--source', '--source-id', '--stale', '--stats', '--status', '--strategy', '--strict', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--target', '--target-score', '--thin', '--timeout', '--to', '--token', '--token-ttl', '--top-k', '--type', '--undo-wave', '--unsafe-bypass-dream-guard', '--unset-all', '--untracked-files', '--url', '--use-captured-snapshot', '--verbose', '--verify', '--version', '--window', '--with-calibration', '--workers', '--yes'],
'dream': ['--against', '--aliases', '--all', '--allow-regression', '--anchor', '--asof', '--background', '--batch', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--by-type', '--by-type-floor', '--code', '--committed-baseline', '--compare', '--compile', '--concurrent', '--ctx-size', '--cycles', '--date', '--detail', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--embeddings', '--expansion', '--explain', '--fast', '--federated', '--fix', '--fixtures', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-retry', '--format', '--from', '--from-db', '--from-pages', '--gold', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--install', '--json', '--judge-model', '--justification', '--keyword-only', '--lang', '--limit', '--llm', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--max-tokens', '--max-usd', '--mcp-only', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--name-only', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--no-federated', '--no-llm', '--no-mutate', '--no-trajectory', '--once', '--out', '--output', '--output-dir', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--phase', '--priority', '--progress-interval', '--progress-json', '--pull', '--quiet', '--receipt-dir', '--remediate', '--repo', '--reranking', '--reset', '--resolve', '--restore-only', '--resume-from', '--retrieval-only', '--rounds', '--rubric-version', '--save', '--seed', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--source-id', '--stale', '--suite', '--suites', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--take', '--task', '--thin', '--threshold', '--timeout', '--to', '--token-ttl', '--top-k', '--undo', '--unsafe-bypass-dream-guard', '--update-baseline', '--verify', '--version', '--window', '--yes'],
'edges-backfill': ['--aliases', '--all', '--all-sources', '--brain', '--concurrency', '--federated', '--help', '--include-null-signature', '--json', '--max-age', '--max-chunks', '--max-cost-usd', '--no-extract', '--no-federated', '--older-than', '--path', '--pattern', '--pending', '--repo', '--reset', '--resolve', '--restore-only', '--source', '--stale', '--supersessions', '--thin', '--timeout', '--workers'],
'embed': ['--aliases', '--all', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--catch-up', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--serial', '--slugs', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--token-ttl', '--version'],
@@ -61,7 +61,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'init': ['--all', '--brain', '--chat-model', '--check', '--ctx-size', '--embedding-dimensions', '--embedding-model', '--entity', '--expansion-model', '--fast', '--flag', '--force', '--from-pages', '--grant-types', '--help', '--http', '--issuer-url', '--json', '--judge-model', '--key', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--path', '--pglite', '--provenance', '--schema-pack', '--scopes', '--skip-embed-check', '--source', '--stale', '--supabase', '--surface', '--to', '--token-ttl', '--touchpoint', '--url', '--version'],
'integrations': ['--auto', '--brain', '--dry-run', '--embeddings', '--fast', '--force', '--from-pages', '--help', '--http', '--json', '--no-embedding', '--overwrite', '--refresh', '--reranking', '--source', '--surface', '--target', '--token-ttl'],
'integrity': ['--aliases', '--all', '--auto', '--backend', '--background', '--brain', '--brain-wide-max-cost-usd', '--check', '--confidence', '--cost', '--dry-run', '--explain', '--fast', '--follow', '--force', '--fresh', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--limit', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--review-lower', '--skip-bare-tweet', '--skip-urls', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl', '--type', '--url'],
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--job-id', '--job-isolation', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
'lint': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--exclude', '--explain', '--fast', '--fix', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl'],
'lsd': ['--brain', '--force-resume', '--help', '--json', '--judge-model', '--limit', '--list-runs', '--max-cost', '--max-far-set', '--max-ideas-per-judge-call', '--no-save', '--resume', '--retry-judge', '--save', '--source', '--strict-budget', '--yes'],
'maintain': ['--aliases', '--all', '--background', '--brain', '--break-lock', '--by-mention', '--catch-up', '--column', '--concurrency', '--content-audit', '--count', '--detach', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--force', '--force-retry', '--force-schema', '--from-meetings', '--full', '--help', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--index-audit', '--infer-dates', '--input', '--json', '--kind', '--lang', '--locks', '--markdown', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-usd', '--migrate-only', '--multimodal', '--near-symbol', '--ner', '--nice', '--no-extract', '--no-mutate', '--older-than', '--once', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--probe-pglite', '--progress-json', '--query', '--queue', '--quiet', '--rebuild-rollup', '--regenerate', '--remediate', '--remediation-plan', '--reset', '--resolve', '--restore-only', '--resume', '--run-id', '--safe', '--scope', '--since', '--skills-dir', '--skip-failed', '--slugs', '--source', '--source-id', '--stale', '--status', '--supabase', '--supersessions', '--symbol-kind', '--target', '--target-score', '--thin', '--to', '--top-k', '--type', '--unsafe-bypass-dream-guard', '--url', '--verbose', '--window', '--workers', '--yes'],
+5 -1
View File
@@ -37,7 +37,7 @@
*/
import postgres from 'postgres';
import { resolvePrepare, resolveSessionTimeouts, resolvePoolSize, endPoolBounded } from './db.ts';
import { resolvePrepare, resolveSessionTimeouts, resolvePoolSize, resolveMaxLifetimeSeconds, endPoolBounded } from './db.ts';
import { redactPgUrl } from './url-redact.ts';
import { logConnectionEvent } from './connection-audit.ts';
@@ -303,6 +303,8 @@ export class ConnectionManager {
max: resolvePoolSize(this.opts.readPoolSize),
idle_timeout: 20,
connect_timeout: 10,
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
max_lifetime: resolveMaxLifetimeSeconds(),
types: { bigint: postgres.BigInt },
};
const timeouts = resolveSessionTimeouts();
@@ -406,6 +408,8 @@ export class ConnectionManager {
max: size,
idle_timeout: 20,
connect_timeout: 10,
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
max_lifetime: resolveMaxLifetimeSeconds(),
types: { bigint: postgres.BigInt },
// Always use prepared statements on the direct pool — no PgBouncer
// here, so the prepare-cache invalidation issue doesn't apply.
+67 -5
View File
@@ -241,6 +241,55 @@ export function rewriteChunkedSlug(slug: string, hash6: string, idx: number): st
// ── Public entry ──────────────────────────────────────────────────────
/**
* One drain-loop lock-renewal tick, extracted for hermetic tests (worker.ts
* parity is `runLockRenewalTick`; this is the deliberately simpler best-effort
* variant no audit channel, no reconnect, no time-based give-up).
*
* Fixes the issue #6 abandoned-racer class in the cycle drain: the previous
* inline tick had no per-call timeout, so a hung renewLock stacked one
* checked-out pool slot per interval firing forever. Now each call carries an
* AbortSignal that is aborted when the timeout wins the race (the query is
* cancelled and its slot released), and callers guard re-entrancy so at most
* one renewal is in flight.
*
* Returns after the renewal settles or times out; a `false` renewal invokes
* `onLost` (token fence lost caller aborts the handler). Errors and
* timeouts are swallowed: best-effort, the next tick retries.
*/
export async function runDrainRenewalTick(
renewLock: (
id: number,
lockToken: string,
lockMs: number,
opts?: { signal?: AbortSignal },
) => Promise<boolean>,
jobId: number,
lockToken: string,
lockMs: number,
onLost: () => void,
callTimeoutMs: number,
): Promise<void> {
const callAbort = new AbortController();
let timer: ReturnType<typeof setTimeout> | null = null;
try {
const ok = await Promise.race([
renewLock(jobId, lockToken, lockMs, { signal: callAbort.signal }),
new Promise<never>((_, reject) => {
timer = setTimeout(() => {
callAbort.abort();
reject(new Error(`renewLock timed out after ${callTimeoutMs}ms`));
}, callTimeoutMs);
}),
]);
if (!ok) onLost();
} catch {
/* best-effort; next tick retries */
} finally {
if (timer != null) clearTimeout(timer);
}
}
export interface SynthesizePhaseOpts {
brainDir: string;
dryRun: boolean;
@@ -433,12 +482,25 @@ export async function runSubagentsInline(
// just its own) can't requeue a live child. A false return means the row
// was cancelled or reclaimed — abort the handler. Errors are swallowed
// (best-effort; the next tick retries), never an unhandledRejection.
// Re-entrancy guard + per-call cancellation via runDrainRenewalTick: a
// hung renewLock no longer stacks a fresh checked-out pool slot per
// interval firing (issue #6 abandoned-racer class).
let drainTickInFlight = false;
const renewTimer = setInterval(() => {
queue.renewLock(job.id, lockToken, lockMs)
.then((ok) => {
if (!ok && !abort.signal.aborted) abort.abort(new Error('lock-renewal-failed'));
})
.catch(() => { /* best-effort; next tick retries */ });
if (drainTickInFlight) return;
drainTickInFlight = true;
void runDrainRenewalTick(
(id, tok, ms, opts) => queue.renewLock(id, tok, ms, opts),
job.id,
lockToken,
lockMs,
() => {
if (!abort.signal.aborted) abort.abort(new Error('lock-renewal-failed'));
},
Math.max(1000, Math.floor(lockMs / 3)),
).finally(() => {
drainTickInFlight = false;
});
}, Math.max(50, Math.floor(lockMs / 3)));
// Run, then record — separated so a completeJob connection error can't
// masquerade as a handler failure, and a failJob connection error can't
+32 -6
View File
@@ -43,8 +43,13 @@ export interface DbLockHandle {
* false means the lock was stolen or released the caller must stop
* relying on mutual exclusion. Transient DB errors still THROW (they are
* not evidence of a steal; the TTL is the backstop).
*
* `opts.signal` cancels the in-flight UPDATE when the caller's heartbeat
* timeout gives up on it (issue #6 an abandoned refresh otherwise holds
* a checked-out pool slot for its full server-side duration). PGLite
* ignores the signal (single embedded connection, no pool to starve).
*/
refresh: () => Promise<boolean>;
refresh: (opts?: { signal?: AbortSignal }) => Promise<boolean>;
}
/**
@@ -268,7 +273,7 @@ export async function tryAcquireDbLock(
return {
id: lockId,
acquiredAt: fence,
refresh: async () => {
refresh: async (refreshOpts?: { signal?: AbortSignal }) => {
// v0.41.13.0: bump BOTH ttl_expires_at AND last_refreshed_at.
// v0.42.x (#1794): route through the DIRECT session pool, not the
// transaction pool, so a Supavisor pooler exhaustion (EMAXCONNSESSION)
@@ -280,6 +285,7 @@ export async function tryAcquireDbLock(
WHERE id = $2 AND holder_pid = $3 AND extract(epoch from acquired_at)::text = $4
RETURNING id`,
[ttl, lockId, pid, fence],
refreshOpts,
);
return updated.length > 0;
},
@@ -882,8 +888,13 @@ export async function withRefreshingLock<T>(
if (!handle) throw new LockUnavailableError(lockId);
let healthOk = true;
// Re-entrancy guard: with a 15s minimum cadence and a 30s default timeout,
// two ticks can overlap on a slow pool — one refresh in flight at a time.
let refreshTickInFlight = false;
const interval = setInterval(() => {
if (refreshTickInFlight) return;
refreshTickInFlight = true;
void (async () => {
try {
// v0.42.x (#1794, V1): the refresh IS the heartbeat. handle.refresh()
@@ -896,10 +907,23 @@ export async function withRefreshingLock<T>(
// health, and we do NOT clearInterval on a transient failure: a blip
// self-heals on the next tick; the TTL is the backstop if the pool stays
// genuinely dead (at which point a steal is correct).
const timeout = new Promise<never>((_, reject) =>
setTimeout(() => reject(new Error('refresh_timeout')), heartbeatTimeoutMs)
);
const stillOwned = await Promise.race([handle.refresh(), timeout]);
// issue #6: abort the per-tick signal when the timeout wins so the
// losing UPDATE is cancelled (slot released), not orphaned on the
// direct pool for its full server-side duration.
const tickAbort = new AbortController();
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
const timeout = new Promise<never>((_, reject) => {
timeoutTimer = setTimeout(() => {
tickAbort.abort();
reject(new Error('refresh_timeout'));
}, heartbeatTimeoutMs);
});
let stillOwned: boolean;
try {
stillOwned = await Promise.race([handle.refresh({ signal: tickAbort.signal }), timeout]);
} finally {
if (timeoutTimer != null) clearTimeout(timeoutTimer);
}
if (stillOwned === false) {
// W0 (D5.10): the fenced refresh matched 0 rows — the lock was
// stolen or force-cleared. Further refreshes are pointless (and a
@@ -918,6 +942,8 @@ export async function withRefreshingLock<T>(
const msg = err instanceof Error ? err.message : String(err);
process.stderr.write(`[lock-refresh] ${lockId}: ${msg}; will retry next tick\n`);
healthOk = false;
} finally {
refreshTickInFlight = false;
}
})();
}, refreshIntervalMs);
+51
View File
@@ -115,6 +115,55 @@ export function resolvePoolSize(explicit?: number): number {
return DEFAULT_POOL_SIZE_FALLBACK;
}
let warnedBadMaxLifetime = false;
/** Test-only: reset the warn-once latch. */
export function _resetMaxLifetimeWarningForTests(): void {
warnedBadMaxLifetime = false;
}
/**
* Client-pool connection max lifetime for every postgres() call site
* (module singleton, engine instance pool, ConnectionManager read + direct
* pools).
*
* postgres.js already defaults to `60 * (30 + Math.random() * 30)` and
* critically that built-in default is a FUNCTION, re-evaluated PER
* CONNECTION (connection.js: `typeof seconds === 'function' ? seconds() :
* seconds`), so each connection gets its own 3060min deadline. A
* pre-evaluated number would make every connection in a pool share ONE
* recycle deadline a warm-up burst then reconnects simultaneously
* (data-migration specialist finding). The default here is therefore the
* same per-connection jitter function; only the env override returns a
* fixed number (the explicit escape hatch):
*
* GBRAIN_POOL_MAX_LIFETIME_S=900 # recycle after 15 min
* GBRAIN_POOL_MAX_LIFETIME_S=0 # disable recycling entirely
*
* max_lifetime only recycles connections as they are RETURNED to the pool;
* it cannot reclaim a leaked checkout this is explicitness + a knob, not
* a starvation fix. Invalid values warn once on stderr and fall back to the
* default. The env param is injectable so tests never mutate process.env.
*/
export function resolveMaxLifetimeSeconds(
env: Record<string, string | undefined> = process.env,
): number | null | (() => number) {
const raw = env.GBRAIN_POOL_MAX_LIFETIME_S;
if (raw !== undefined && raw !== '') {
const parsed = Number(raw);
if (Number.isFinite(parsed) && Number.isInteger(parsed) && parsed >= 0) {
return parsed === 0 ? null : parsed;
}
if (!warnedBadMaxLifetime) {
warnedBadMaxLifetime = true;
process.stderr.write(
`[gbrain] Ignoring invalid GBRAIN_POOL_MAX_LIFETIME_S=${JSON.stringify(raw)} (want a non-negative integer of seconds; 0 disables); using the jittered 30-60min default\n`,
);
}
}
// Per-connection jitter, matching the postgres.js built-in default shape.
return () => Math.floor(60 * (30 + Math.random() * 30));
}
/**
* Session-level GUCs applied to every new backend connection. Prevents
* orphan pgbouncer sessions from holding locks or running queries
@@ -240,6 +289,8 @@ export async function connect(config: EngineConfig): Promise<boolean> {
max: resolvePoolSize(),
idle_timeout: 20,
connect_timeout: 10,
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
max_lifetime: resolveMaxLifetimeSeconds(),
types: {
// Register pgvector type
bigint: postgres.BigInt,
+309
View File
@@ -0,0 +1,309 @@
/**
* Parent-side child-process runner for per-job isolation (issue #5).
*
* `runJobInChild` is the one-line seam executeJob swaps in for
* `handler(context)` when isolation is on. The parent keeps claim, lock
* renewal and ALL result recording; this module owns spawn signal reap
* decode:
*
* spawn detached (own process group; group signals reach handler
* grandchildren even under tini), tini-wrapped when available,
* stdio ['ignore','inherit','inherit'] so handler logs stream
* to the operator; results travel by outcome file, never stdout.
* signal per-job abort (timeout / cancel / lock-lost /
* lock-renewal-failed) group SIGTERM now, group SIGKILL at
* +CHILD_KILL_GRACE_MS (25s inside the worker's 30s
* force-evict window, which stays as an untouched backstop).
* Worker shutdown same SIGTERM (the child's own handler fires
* ctx.shutdownSignal, giving handlers the drain window to
* finish AND write their outcome) with the SIGKILL backstop.
* classify outcome file presence rules (job-isolation.ts). No file:
* per-job abort generic throw (executeJob's catch reads
* abort.signal.reason, so infra aborts still burn no attempt);
* worker shutdown ChildWorkerShutdownError (released, NO
* attempt burned a routine deploy must not burn attempts;
* codex-2 #7); otherwise a crash (attempt burned, correct).
* Pre-exec spawn failure ChildSpawnInfraError (released, no
* attempt burned: one bad CLI path must not dead-letter a
* queue; the CLI layer also fail-fast validates at startup).
*/
import { spawn, type ChildProcess } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { buildSpawnInvocation } from './spawn-helpers.ts';
import {
UnrecoverableError,
ABORT_REASON_TIMEOUT,
ABORT_REASON_LOCK_LOST,
ABORT_REASON_LOCK_RENEWAL_FAILED,
} from './types.ts';
import {
JOB_CHILD_EXIT_USAGE,
JOB_CHILD_EXIT_NOT_CLAIMED,
} from './worker-exit-codes.ts';
import {
CHILD_ENV,
CHILD_KILL_GRACE_MS,
CHILD_READ_POOL_MAX,
buildChildArgs,
decodeChildOutcomeFileAsync,
killProcessGroup,
reconstructHandlerError,
unrefTimer,
type ChildCliInvocation,
} from './job-isolation.ts';
/** Pre-exec spawn failure infrastructure, not a job defect. executeJob
* releases the job with no attempt burned (stall sweeper requeues). */
export class ChildSpawnInfraError extends Error {
constructor(message: string) {
super(message);
this.name = 'ChildSpawnInfraError';
}
}
/** Child terminated by worker shutdown before it could report. Released with
* no attempt burned routine deploys must not burn attempts (codex-2 #7). */
export class ChildWorkerShutdownError extends Error {
constructor(message: string) {
super(message);
this.name = 'ChildWorkerShutdownError';
}
}
/** Child found the job reclaimed/cancelled (exit 14) provably owned
* elsewhere. The worker releases without failJob (the fenced failJob would
* no-op anyway); definitely not an attempt against THIS claim. */
export class ChildNotClaimedError extends Error {
constructor(message: string) {
super(message);
this.name = 'ChildNotClaimedError';
}
}
/** Per-job abort reasons that mean THE JOB was targeted (timeout / lock
* loss) rather than the worker winding down. gracefulShutdown('watchdog')
* aborts BOTH the shutdown signal and every per-job signal the shutdown
* classification must win for those (adversarial-review P3: the watchdog
* drain otherwise burns an attempt on innocent isolated jobs). Built from
* the shared literals in types.ts so a rename at an abort site cannot
* silently flip child classification (maintainability review the
* never-produced 'cancel'/'cancelled' entries were dropped: cancellation
* surfaces as lock-lost via the fenced renewLock). */
const PER_JOB_ABORT_REASONS = new Set<string>([
ABORT_REASON_TIMEOUT,
ABORT_REASON_LOCK_LOST,
ABORT_REASON_LOCK_RENEWAL_FAILED,
]);
export interface RunJobInChildOpts {
jobId: number;
jobName: string;
lockToken: string;
/** Per-job abort (timeout / cancel / lock-lost / lock-renewal-failed). */
abortSignal: AbortSignal;
/** Worker-process SIGTERM/SIGINT. */
shutdownSignal: AbortSignal;
/** Resolved once at worker startup (fail-fast); how to invoke the CLI. */
invocation: ChildCliInvocation;
/** tini path ('' when absent — direct spawn, same degradation as the supervisor). */
tiniPath: string;
/** Injectable for tests. Default CHILD_KILL_GRACE_MS. */
killGraceMs?: number;
/** Injectable base env for tests. Default process.env. */
env?: Record<string, string | undefined>;
}
interface ChildExit {
code: number | null;
signal: NodeJS.Signals | null;
spawnErr?: Error;
}
/**
* Run one claimed job in a child process. Resolves with the handler result
* (parent then runs the normal completeJob path); throws reconstructed
* handler errors / classification errors (parent's existing catch handles
* them verbatim).
*/
export async function runJobInChild(opts: RunJobInChildOpts): Promise<unknown> {
const dir = mkdtempSync(join(tmpdir(), `gbrain-job-${opts.jobId}-`));
const resultPath = join(dir, 'outcome.json');
const graceMs = opts.killGraceMs ?? CHILD_KILL_GRACE_MS;
const base = opts.env ?? process.env;
// Bound the child's pools: sockets die with the process (the isolation
// win), but per-child footprint must stay small — read pool <= 3, direct
// pool 1 (a child runs no claim/renewal heartbeats; codex-2 #6). An
// operator's own GBRAIN_POOL_SIZE is respected when STRICTER than the
// default (their pooler MaxClients tuning must not be silently raised);
// GBRAIN_JOB_CHILD_POOL_SIZE, when valid, is the explicit per-child knob
// and wins outright. Invalid values fall through to the default.
const parsePoolSize = (v: string | undefined): number | null => {
if (v === undefined || v === '') return null;
const n = parseInt(v, 10);
return Number.isInteger(n) && n > 0 ? n : null;
};
const childOverride = parsePoolSize(base[CHILD_ENV.childPoolSize]);
const userPool = parsePoolSize(base.GBRAIN_POOL_SIZE);
const childPoolSize = childOverride ?? Math.min(userPool ?? CHILD_READ_POOL_MAX, CHILD_READ_POOL_MAX);
const childEnv: Record<string, string | undefined> = {
...base,
[CHILD_ENV.lockToken]: opts.lockToken,
[CHILD_ENV.resultPath]: resultPath,
[CHILD_ENV.isChild]: '1',
[CHILD_ENV.parentPid]: String(process.pid),
GBRAIN_POOL_SIZE: String(childPoolSize),
GBRAIN_DIRECT_POOL_SIZE: '1',
};
const inv = buildSpawnInvocation(opts.tiniPath, opts.invocation.cmd, [
...opts.invocation.argsPrefix,
...buildChildArgs(opts.jobId),
]);
let child: ChildProcess;
try {
child = spawn(inv.cmd, inv.args, {
stdio: ['ignore', 'inherit', 'inherit'],
env: childEnv as NodeJS.ProcessEnv,
detached: true,
});
} catch (e) {
rmSync(dir, { recursive: true, force: true });
const msg = e instanceof Error ? e.message : String(e);
throw new ChildSpawnInfraError(`job child spawn failed (${inv.cmd}): ${msg}`);
}
let killTimer: ReturnType<typeof setTimeout> | null = null;
let termed = false;
const terminate = (): void => {
if (termed) return;
termed = true;
if (child.pid != null) {
killProcessGroup(child.pid, 'SIGTERM');
killTimer = setTimeout(() => {
// Loud on failure (red-team finding): the /bin/kill fallback is the
// NORMAL delivery path in Bun-compiled binaries, and a container
// without /bin/kill (distroless) would otherwise silently void the
// SIGKILL guarantee while the child runs to completion and the job
// gets requeued elsewhere (duplicate side effects).
if (child.pid != null && child.exitCode == null && child.signalCode == null) {
const delivered = killProcessGroup(child.pid, 'SIGKILL');
if (!delivered) {
console.error(
`[isolation] job ${opts.jobId} (${opts.jobName}): group SIGKILL was NOT delivered ` +
`to pid ${child.pid} (platform=${process.platform}; is /bin/kill present?). ` +
`The child may still be running — the SIGKILL guarantee is degraded on this host.`,
);
}
}
}, graceMs);
unrefTimer(killTimer);
}
};
const onAbort = (): void => terminate();
const onShutdown = (): void => terminate();
if (opts.abortSignal.aborted) onAbort();
else opts.abortSignal.addEventListener('abort', onAbort, { once: true });
if (opts.shutdownSignal.aborted) onShutdown();
else opts.shutdownSignal.addEventListener('abort', onShutdown, { once: true });
console.log(
`[isolation] job ${opts.jobId} (${opts.jobName}) child pid ${child.pid ?? '?'} spawned`,
);
try {
const exit = await new Promise<ChildExit>((resolve) => {
child.once('error', (e) => resolve({ code: null, signal: null, spawnErr: e }));
child.once('exit', (code, signal) => resolve({ code, signal }));
});
if (exit.spawnErr && child.pid == null) {
throw new ChildSpawnInfraError(
`job child spawn failed (${inv.cmd}): ${exit.spawnErr.message}`,
);
}
console.log(
`[isolation] job ${opts.jobId} (${opts.jobName}) child pid ${child.pid ?? '?'} ` +
`exited code=${exit.code ?? 'null'} signal=${exit.signal ?? 'null'}`,
);
const abortReason = opts.abortSignal.aborted
? (opts.abortSignal.reason instanceof Error
? opts.abortSignal.reason.message
: String(opts.abortSignal.reason ?? 'aborted'))
: null;
// Shutdown classification wins UNLESS the per-job abort names a
// job-targeted reason. gracefulShutdown('watchdog') aborts BOTH signals —
// checking abortSignal first would shadow the no-burn shutdown release
// and dead-letter innocent isolated jobs (adversarial-review P3).
const isShutdownClass =
opts.shutdownSignal.aborted &&
(abortReason === null || !PER_JOB_ABORT_REASONS.has(abortReason));
let outcome: Awaited<ReturnType<typeof decodeChildOutcomeFileAsync>>;
try {
// Async decode: a large-but-allowed outcome must not block the worker
// event loop that runs lock-renewal ticks (performance review).
outcome = await decodeChildOutcomeFileAsync(resultPath);
} catch (decodeErr) {
// No usable outcome. Classify by WHY the child died.
if (decodeErr instanceof UnrecoverableError) throw decodeErr; // oversize cap — dead on attempt 1
if (isShutdownClass) {
throw new ChildWorkerShutdownError(
`job child terminated by worker shutdown before reporting (exit code=${exit.code} signal=${exit.signal})`,
);
}
if (opts.abortSignal.aborted) {
// executeJob's catch reads abort.signal.reason first, so infra
// reasons (lock-renewal-failed / lock-lost) still burn no attempt
// and timeout/cancel keep their existing semantics.
throw new Error(
`job child terminated after abort without an outcome (exit code=${exit.code} signal=${exit.signal})`,
);
}
// Bootstrap failures carry reserved exit codes and are NOT handler
// defects: 13 = usage/config (ops misconfiguration — release like a
// spawn failure), 14 = job reclaimed before the handler ran (owned
// elsewhere — release; the fenced failJob would no-op regardless).
if (exit.code === JOB_CHILD_EXIT_USAGE) {
throw new ChildSpawnInfraError(
`job child bootstrap failed (exit ${exit.code}) — check the worker's child CLI/engine configuration`,
);
}
if (exit.code === JOB_CHILD_EXIT_NOT_CLAIMED) {
throw new ChildNotClaimedError(
`job child found the claim gone (exit ${exit.code}) — reclaimed or cancelled before the handler ran`,
);
}
throw new Error(
`${decodeErr instanceof Error ? decodeErr.message : String(decodeErr)} ` +
`(exit code=${exit.code} signal=${exit.signal})`,
);
}
if (outcome.outcome === 'success') return outcome.result;
// A handler-error outcome DURING worker shutdown is presumed
// shutdown-induced (cooperative handlers that honor shutdownSignal bail
// and report an error): release with no attempt burned rather than
// punishing exactly the well-behaved handlers on every deploy
// (adversarial-review P2). Worst case a genuinely-failing job that
// coincided with a deploy gets one free retry — bounded and benign.
if (isShutdownClass) {
throw new ChildWorkerShutdownError(
`job child reported an error during worker shutdown (${outcome.message}) — released, not burned`,
);
}
throw reconstructHandlerError(outcome);
} finally {
if (killTimer != null) clearTimeout(killTimer);
opts.abortSignal.removeEventListener('abort', onAbort);
opts.shutdownSignal.removeEventListener('abort', onShutdown);
rmSync(dir, { recursive: true, force: true });
}
}
+165
View File
@@ -0,0 +1,165 @@
/**
* DB liveness probe with pool-starvation disambiguation (issue #6).
*
* The incident this exists for: a worker's read pool was exhausted by
* checked-out-and-abandoned queries, the `SELECT 1` probe couldn't get a slot
* within its budget, and the worker exited with "DB unreachable" while the
* server sat at ~10% of max_connections. Operators spent hours on the wrong
* layer (evaluating an instance upgrade that would have changed nothing).
*
* Mechanism: probe the read pool; on failure, probe the DIRECT session lane
* (when dual-pool is active). Direct success proves the server is reachable
* and narrows the fault to the transaction-pooler path client pool
* exhaustion or a pooler-layer fault; the probe deliberately does NOT claim
* to distinguish those two (codex-2 #2). Either way the operator is pointed
* away from "the database is down / too small".
*
* Verdicts:
* pool_starved read probe failed, direct probe succeeded.
* server_unreachable read AND direct probes failed.
* unknown read probe failed, no direct lane to disambiguate.
*
* Both probes carry an AbortSignal that fires when their deadline wins, so a
* hung probe is cancelled (slot released), never abandoned same contract
* as the lock-renewal tick.
*
* Pure/hermetic: every effect is injected via `DbProbeDeps`; the worker's
* adapter is a thin closure. (lock-renewal-tick.ts is the pattern.)
*/
import type { PoolGaugeSnapshot } from '../pool-gauge.ts';
export type ProbeVerdict = 'pool_starved' | 'server_unreachable' | 'unknown';
/** Default budget for the direct-lane disambiguation probe. */
export const DIRECT_PROBE_TIMEOUT_MS = 3_000;
export interface PoolDiagnostics {
/** Gauge counts — a tracked SUBSET (see pool-gauge.ts honesty contract). */
tracked: PoolGaugeSnapshot;
/** Read-pool max, when the engine can report it; null otherwise. */
poolMax: number | null;
}
export interface DbProbeDeps {
/** SELECT 1 on the read pool; MUST honor the signal (cancellation). */
probeRead: (signal: AbortSignal) => Promise<void>;
/**
* SELECT 1 on the direct session lane. Present ONLY when dual-pool is
* genuinely active an executeRawDirect that would silently fall back to
* the read pool (kill-switch collapse) must NOT be passed here, or the
* "disambiguation" would probe the same starved pool twice.
*/
probeDirect?: (signal: AbortSignal) => Promise<void>;
/** Optional gauge snapshot for supporting detail. Fail-open: may be absent or throw. */
getDiagnostics?: () => PoolDiagnostics | null;
/** Read-probe budget (worker default 10s). */
timeoutMs: number;
/** Direct-probe budget (default 3s). */
directTimeoutMs: number;
}
export type DbProbeResult =
| { ok: true }
| { ok: false; verdict: ProbeVerdict; detail: string };
/**
* Narrow, shared view of the engine's ConnectionManager for routing-aware
* callers (the worker's probe adapter, jobs.ts's single-pool startup
* warning). One typed accessor instead of hand-rolled structural casts that
* drift independently from the real class (maintainability review).
*/
export interface EngineConnectionRouting {
isDualPoolActive?: () => boolean;
describeMode?: () => { kill_switch_active?: boolean; direct_pool_size?: number };
}
export function getConnectionRouting(engine: unknown): EngineConnectionRouting | null {
const cm = (engine as { connectionManager?: EngineConnectionRouting }).connectionManager;
return cm ?? null;
}
async function withDeadline(
run: (signal: AbortSignal) => Promise<void>,
ms: number,
label: string,
): Promise<void> {
const ac = new AbortController();
let timer: ReturnType<typeof setTimeout> | null = null;
try {
await Promise.race([
run(ac.signal),
new Promise<never>((_, reject) => {
timer = setTimeout(() => {
ac.abort();
reject(new Error(`${label} timeout after ${ms}ms`));
}, ms);
}),
]);
} finally {
if (timer != null) clearTimeout(timer);
}
}
/** Render gauge detail. Never throws; empty string when unavailable. */
function renderDiagnostics(getDiagnostics?: () => PoolDiagnostics | null): string {
try {
const diag = getDiagnostics?.();
if (!diag) return '';
const { tracked, poolMax } = diag;
const maxNote = poolMax != null ? ` (read pool max ${poolMax})` : '';
const base =
` gbrain-tracked in flight (subset — template-path queries untracked):` +
` raw=${tracked.raw}, direct=${tracked.direct}, reserved=${tracked.reserved}, tx=${tracked.tx}${maxNote}.`;
const total = tracked.raw + tracked.direct + tracked.reserved + tracked.tx;
if (total === 0) {
return (
base +
' Tracked subset shows 0 in flight — the saturation is in untracked' +
' template-query traffic; see docs/guides/queue-operations-runbook.md.'
);
}
return base;
} catch {
return '';
}
}
export async function runDbProbe(deps: DbProbeDeps): Promise<DbProbeResult> {
let readErrMsg: string;
try {
await withDeadline(deps.probeRead, deps.timeoutMs, 'probe');
return { ok: true };
} catch (e) {
readErrMsg = e instanceof Error ? e.message : String(e);
}
if (!deps.probeDirect) {
return {
ok: false,
verdict: 'unknown',
detail: `${readErrMsg}; no direct lane available to disambiguate pool starvation from a dead server`,
};
}
const t0 = Date.now();
try {
await withDeadline(deps.probeDirect, deps.directTimeoutMs, 'direct probe');
return {
ok: false,
verdict: 'pool_starved',
detail:
`read-pool probe failed (${readErrMsg}) but the direct-lane probe succeeded in ${Date.now() - t0}ms — ` +
`the server IS reachable; the fault is in the transaction-pooler path ` +
`(client pool exhaustion or a pooler-layer fault).` +
renderDiagnostics(deps.getDiagnostics),
};
} catch (e) {
const directErrMsg = e instanceof Error ? e.message : String(e);
return {
ok: false,
verdict: 'server_unreachable',
detail: `read probe: ${readErrMsg}; direct probe: ${directErrMsg} — server/network unreachable`,
};
}
}
+61
View File
@@ -0,0 +1,61 @@
/**
* Shared MinionJobContext builder (issue #5 per-job process isolation).
*
* Extracted verbatim from MinionWorker.executeJob so the same DB-backed
* context wiring serves BOTH execution modes:
*
* inline the worker builds it against its own engine/queue;
* process `gbrain jobs run-child` builds it against the CHILD's engine
* (child-owns-engine design: every write below is token-fenced,
* so a reclaimed job's orphan child degrades to a no-op writer).
*
* Behavioral no-op for the inline path.
*/
import type { BrainEngine } from '../engine.ts';
import type { MinionQueue } from './queue.ts';
import type { MinionJob, MinionJobContext, TokenUpdate } from './types.ts';
export function buildJobContext(
engine: BrainEngine,
queue: MinionQueue,
job: MinionJob,
lockToken: string,
signal: AbortSignal,
shutdownSignal: AbortSignal,
): MinionJobContext {
return {
id: job.id,
name: job.name,
data: job.data,
attempts_made: job.attempts_made,
signal,
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
shutdownSignal,
updateProgress: async (progress: unknown) => {
await queue.updateProgress(job.id, lockToken, progress);
},
updateTokens: async (tokens: TokenUpdate) => {
await queue.updateTokens(job.id, lockToken, tokens);
},
log: async (message: string | Record<string, unknown>) => {
const value = typeof message === 'string' ? message : JSON.stringify(message);
await engine.executeRaw(
`UPDATE minion_jobs SET stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
updated_at = now()
WHERE id = $2 AND status = 'active' AND lock_token = $3`,
[value, job.id, lockToken]
);
},
isActive: async () => {
const rows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM minion_jobs WHERE id = $1 AND status = 'active' AND lock_token = $2`,
[job.id, lockToken]
);
return rows.length > 0;
},
readInbox: async () => {
return queue.readInbox(job.id, lockToken);
},
};
}
+306
View File
@@ -0,0 +1,306 @@
/**
* Per-job process isolation protocol (issue #5).
*
* The worker (parent) keeps claim / lock-renewal / completeJob / failJob;
* `gbrain jobs run-child` (child) owns handler execution with its own small
* engine pool. This module is the protocol between them:
*
* payload in job id via argv (`jobs run-child --job-id N`, ps-visible
* for ops); lock token via GBRAIN_JOB_LOCK_TOKEN env (off
* argv not a secret, it's a fencing token, but no reason
* to put it in `ps` output); result path via
* GBRAIN_JOB_RESULT_PATH.
* result out ONE JSON file, written atomically (tmp + rename), decoded
* by the parent. stdout/stderr stay inherited for handler
* logs (handlers print freely no sentinel parsing), and
* node-IPC is deliberately avoided (zero precedent in this
* codebase; fd inheritance through a tini wrapper is
* unproven here).
* termination killProcessGroup(): children are spawned detached (own
* process group) because SIGKILL on the tini pid alone kills
* tini, NOT the handler grandchild (tini cannot forward
* SIGKILL). Bun rejects negative pids in process.kill()
* (oven-sh/bun#15791) and gbrain ships as a Bun-compiled
* binary, so the group signal falls back to POSIX
* /bin/kill when needed.
*
* Handler-error semantics survive the boundary: the child encodes the two
* error classes executeJob branches on (UnrecoverableError 'dead',
* RateLeaseUnavailableError lease release, no attempt burned) and
* `reconstructHandlerError` rebuilds real instances parent-side so the
* existing `instanceof` branches work verbatim. Everything else degrades to
* a generic Error the normal delayed/dead backoff path, same as inline.
*/
import { readFileSync, writeFileSync, renameSync, statSync } from 'node:fs';
import { readFile, stat } from 'node:fs/promises';
import { spawnSync } from 'node:child_process';
import { UnrecoverableError } from './types.ts';
import { RateLeaseUnavailableError } from './handlers/subagent.ts';
/** Grace between group-SIGTERM and group-SIGKILL on abort. Deliberately
* inside the worker's 30s force-evict window so the evict path stays a
* nearly-unreachable backstop. */
export const CHILD_KILL_GRACE_MS = 25_000;
/** Decode cap for the child's outcome file. Results already round-trip
* through the completeJob JSONB column in inline mode, so anything near
* this cap is pathological; oversize throws UnrecoverableError (loud dead
* on attempt 1 deterministic failure, retries would fail identically). */
export const CHILD_OUTCOME_MAX_BYTES = 32 * 1024 * 1024;
/** Default read-pool cap for isolation children. Referenced by the
* --job-isolation help copy ("~4 pooler client connections" = this + the
* direct pool of 1) and the minions-deployment.md budget math. */
export const CHILD_READ_POOL_MAX = 3;
/** Bun-compat timer unref (plain cast copy-pasted thrice before this helper). */
export function unrefTimer(t: unknown): void {
(t as { unref?: () => void }).unref?.();
}
/** Env vars of the parent↔child contract. Spelled once here. */
export const CHILD_ENV = {
lockToken: 'GBRAIN_JOB_LOCK_TOKEN',
resultPath: 'GBRAIN_JOB_RESULT_PATH',
isChild: 'GBRAIN_JOB_CHILD',
parentPid: 'GBRAIN_JOB_PARENT_PID',
childCliOverride: 'GBRAIN_JOB_CHILD_CLI',
childPoolSize: 'GBRAIN_JOB_CHILD_POOL_SIZE',
} as const;
export type ChildErrorKind = 'unrecoverable' | 'rate_lease' | 'generic';
export type ChildOutcome =
| { outcome: 'success'; result: unknown }
| {
outcome: 'error';
errorKind: ChildErrorKind;
message: string;
stack?: string;
lease?: { key: string; active: number; max: number };
};
/** Child-side: classify a handler throw into the wire shape. */
export function encodeHandlerError(err: unknown): ChildOutcome {
if (err instanceof RateLeaseUnavailableError) {
return {
outcome: 'error',
errorKind: 'rate_lease',
message: err.message,
lease: { key: err.key, active: err.active, max: err.max },
};
}
if (err instanceof UnrecoverableError) {
return {
outcome: 'error',
errorKind: 'unrecoverable',
message: err.message,
...(err.stack ? { stack: err.stack } : {}),
};
}
const message = err instanceof Error ? err.message : String(err);
const stack = err instanceof Error ? err.stack : undefined;
return { outcome: 'error', errorKind: 'generic', message, ...(stack ? { stack } : {}) };
}
/**
* Parent-side: rebuild a real error instance so executeJob's existing
* `instanceof` branches (dead / lease-release / delayed+backoff) work
* verbatim. Unknown errorKind values degrade to generic (whitelist the
* file is same-user-written but a malformed kind must not crash the worker).
*/
export function reconstructHandlerError(o: Extract<ChildOutcome, { outcome: 'error' }>): Error {
if (o.errorKind === 'rate_lease' && o.lease) {
return new RateLeaseUnavailableError(o.lease.key, o.lease.active, o.lease.max);
}
if (o.errorKind === 'unrecoverable') {
return new UnrecoverableError(o.message);
}
const err = new Error(o.message);
if (o.stack) {
(err as Error & { childStack?: string }).childStack = o.stack;
}
return err;
}
/** Child-side: atomic outcome write (tmp + rename on the same filesystem). */
export function writeChildOutcomeFile(path: string, outcome: ChildOutcome): void {
const tmp = `${path}.tmp`;
writeFileSync(tmp, JSON.stringify(outcome), 'utf8');
renameSync(tmp, path);
}
/**
* Pure outcome parser (shared by the sync and async decode paths). Throws:
* - UnrecoverableError when the file exceeds `maxBytes` (deterministic
* dead on attempt 1, no silent truncation);
* - generic Error for malformed/unrecognized content (byte count only in
* the message, NEVER file content handler output may carry secrets).
* The `lease` payload is shape-validated (security review): a corrupt file
* must degrade to 'generic', not inject undefined fields into the parent's
* lease-release accounting.
*/
export function parseChildOutcome(raw: string, size: number, maxBytes = CHILD_OUTCOME_MAX_BYTES): ChildOutcome {
if (size > maxBytes) {
throw new UnrecoverableError(
`job child result exceeds the ${Math.floor(maxBytes / (1024 * 1024))}MiB outcome cap (${size} bytes); ` +
`retries would fail identically — return a smaller result or persist large artifacts elsewhere`,
);
}
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error(`job child outcome file is not valid JSON (${size} bytes)`);
}
const o = parsed as Partial<ChildOutcome> | null;
if (o && o.outcome === 'success') return { outcome: 'success', result: (o as { result?: unknown }).result };
if (o && o.outcome === 'error' && typeof (o as { message?: unknown }).message === 'string') {
const kind = (o as { errorKind?: unknown }).errorKind;
const rawLease = (o as { lease?: unknown }).lease as
| { key?: unknown; active?: unknown; max?: unknown }
| undefined;
const leaseValid =
rawLease != null &&
typeof rawLease.key === 'string' &&
Number.isFinite(rawLease.active as number) &&
Number.isFinite(rawLease.max as number);
// rate_lease without a valid lease payload degrades to generic — same
// policy as the errorKind whitelist.
const errorKind =
kind === 'unrecoverable' ? 'unrecoverable'
: kind === 'rate_lease' && leaseValid ? 'rate_lease'
: 'generic';
return {
outcome: 'error',
errorKind,
message: (o as { message: string }).message,
...((o as { stack?: unknown }).stack && typeof (o as { stack?: unknown }).stack === 'string'
? { stack: (o as { stack: string }).stack }
: {}),
...(errorKind === 'rate_lease' && leaseValid
? { lease: { key: rawLease.key as string, active: rawLease.active as number, max: rawLease.max as number } }
: {}),
};
}
throw new Error(`job child outcome file has an unrecognized shape (${size} bytes)`);
}
const MISSING_OUTCOME_MESSAGE =
'job child exited without writing its outcome file (crash, OOM, or kill before completion)';
/** Sync decode (tests + non-hot-path callers). */
export function decodeChildOutcomeFile(path: string, maxBytes = CHILD_OUTCOME_MAX_BYTES): ChildOutcome {
let size: number;
try {
size = statSync(path).size;
} catch {
throw new Error(MISSING_OUTCOME_MESSAGE);
}
if (size > maxBytes) return parseChildOutcome('', size, maxBytes); // throws the cap error
return parseChildOutcome(readFileSync(path, 'utf8'), size, maxBytes);
}
/**
* Async decode for the WORKER's per-job path: a large-but-allowed outcome
* (up to 32MiB) must not block the event loop that runs lock-renewal ticks
* and the health-probe chain (performance review).
*/
export async function decodeChildOutcomeFileAsync(
path: string,
maxBytes = CHILD_OUTCOME_MAX_BYTES,
): Promise<ChildOutcome> {
let size: number;
try {
size = (await stat(path)).size;
} catch {
throw new Error(MISSING_OUTCOME_MESSAGE);
}
if (size > maxBytes) return parseChildOutcome('', size, maxBytes); // throws the cap error
return parseChildOutcome(await readFile(path, 'utf8'), size, maxBytes);
}
/** argv for the child invocation (appended after the resolved CLI). */
export function buildChildArgs(jobId: number): string[] {
return ['jobs', 'run-child', '--job-id', String(jobId)];
}
export interface ChildCliInvocation {
cmd: string;
/** Args that come BEFORE buildChildArgs() output (e.g. the cli.ts path in bun-dev). */
argsPrefix: string[];
}
/**
* Resolve how to invoke the gbrain CLI for a child process. Pure all
* inputs injected:
*
* 1. GBRAIN_JOB_CHILD_CLI env override (ops/test escape hatch)
* 2. resolveBinary() the compiled-binary resolver
* (resolveGbrainCliPath; never returns a .ts path)
* 3. bun-dev fallback: running from `bun src/cli.ts` invoke
* `<execPath> <argv1>` so dev and tests work without a compiled binary
*
* Returns null when nothing resolves the caller must fail fast at worker
* startup (one bad path must not dead-letter a queue job-by-job).
*/
export function resolveChildCliInvocation(
env: Record<string, string | undefined>,
execPath: string,
argv1: string | undefined,
resolveBinary: () => string | null,
): ChildCliInvocation | null {
const override = env[CHILD_ENV.childCliOverride];
if (override && override.trim() !== '') {
return { cmd: override, argsPrefix: [] };
}
try {
const bin = resolveBinary();
if (bin) return { cmd: bin, argsPrefix: [] };
} catch {
// fall through to the dev fallback
}
if (argv1 && (argv1.endsWith('/cli.ts') || argv1.endsWith('\\cli.ts') || argv1 === 'cli.ts')) {
return { cmd: execPath, argsPrefix: [argv1] };
}
return null;
}
/**
* Signal an entire process GROUP.
*
* Children are spawned `detached: true` (own group) so this reaches the
* handler grandchildren even under a tini wrapper SIGKILL on the tini pid
* alone kills tini and ORPHANS the still-running handler (tini cannot
* forward SIGKILL; that failure mode would silently void issue #5's
* headline guarantee exactly in container deployments).
*
* Bun's process.kill() rejects negative pids (oven-sh/bun#15791), so on any
* throw other than ESRCH we fall back to POSIX /bin/kill, which
* group-signals fine on darwin + linux. Returns true when the signal was
* delivered to a live group; false when the group is already gone (ESRCH
* success for our purposes) or delivery failed.
*/
export function killProcessGroup(pid: number, signal: 'SIGTERM' | 'SIGKILL'): boolean {
if (!Number.isInteger(pid) || pid <= 1) return false;
try {
process.kill(-pid, signal);
return true;
} catch (e) {
const code = (e as NodeJS.ErrnoException).code;
if (code === 'ESRCH') return false; // group already gone
// RangeError on Bun (negative pid unsupported) or EPERM etc. — fall back
// to /bin/kill by ABSOLUTE path (a PATH-resolved binary in a kill path is
// an unnecessary indirection; security review). This is the NORMAL path
// in Bun-compiled production binaries; the sync exec is ~1-3ms and only
// runs on abort/shutdown, never in the claim/renewal hot loop.
try {
const sigName = signal.replace(/^SIG/, '');
const res = spawnSync('/bin/kill', ['-s', sigName, '--', `-${pid}`], { stdio: 'ignore' });
return res.status === 0;
} catch {
return false;
}
}
}
+25 -4
View File
@@ -147,7 +147,19 @@ function warnAndFallback(name: string, raw: string, fallback: number): number {
* `runLockRenewalTick` is pure and trivially testable.
*/
export interface LockRenewalDeps {
renewLock: (jobId: number, lockToken: string, lockDurationMs: number) => Promise<boolean>;
/**
* The optional `opts.signal` is aborted when this call loses the tick's
* timeout race, so the underlying UPDATE is CANCELLED (postgres.js
* `.cancel()` via executeRawDirect) instead of orphaned on a checked-out
* pool slot for its full server-side duration the #6 starvation class.
* Optional-param widening keeps the legacy 3-arg test mocks compiling.
*/
renewLock: (
jobId: number,
lockToken: string,
lockDurationMs: number,
opts?: { signal?: AbortSignal },
) => Promise<boolean>;
audit: LockRenewalAuditSinkLike;
/** Injectable for hermetic time-based tests. Production: `Date.now`. */
now: () => number;
@@ -155,8 +167,9 @@ export interface LockRenewalDeps {
* Injectable for hermetic Promise.race tests. Production:
* `globalThis.setTimeout`. The function must return a value that
* `clearTimeout` accepts, but this seam doesn't expose clearTimeout
* because the timeout race fires-and-forgets (the lost race is
* harmless at worst we have a dangling reject that no one awaits).
* because the timeout race fires-and-forgets. The losing renewLock is no
* longer merely abandoned: the timeout callback also aborts the per-call
* signal so the query releases its pool slot.
*/
setTimeout: (cb: () => void, ms: number) => unknown;
/**
@@ -228,11 +241,19 @@ export async function runLockRenewalTick(
if (state.cancelled()) return { kind: 'cancelled' };
let renewed: boolean;
// Per-call cancellation: when the timeout wins the race, abort the signal
// so the losing UPDATE releases its pool slot instead of holding it until
// the server finishes (issue #6 — an abandoned racer under a saturated
// pooler pinned a checked-out connection for minutes). On the win path the
// late-firing timer aborts an already-settled query, which runUnsafe
// ignores (abort listener removed in its .finally).
const callAbort = new AbortController();
try {
renewed = await Promise.race([
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs),
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs, { signal: callAbort.signal }),
new Promise<never>((_, reject) => {
deps.setTimeout(() => {
callAbort.abort();
reject(new Error(`renewLock timed out after ${state.knobs.callTimeoutMs}ms`));
}, state.knobs.callTimeoutMs);
}),
+14 -3
View File
@@ -1291,8 +1291,18 @@ export class MinionQueue {
return rows.length > 0;
}
/** Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed). */
async renewLock(id: number, lockToken: string, lockDurationMs: number): Promise<boolean> {
/**
* Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed).
* `opts.signal` cancels the in-flight UPDATE (postgres.js `.cancel()`) when the
* caller's timeout race gives up on it otherwise the abandoned query holds a
* checked-out pool slot for its full server-side duration (issue #6).
*/
async renewLock(
id: number,
lockToken: string,
lockDurationMs: number,
opts?: { signal?: AbortSignal },
): Promise<boolean> {
// Direct (session-mode) pool — see claim(). The heartbeat that keeps a job
// alive for minutes cannot run on the transaction pooler without periodic
// CONNECTION_ENDED drops that look like lock-expiry and orphan the job.
@@ -1300,7 +1310,8 @@ export class MinionQueue {
`UPDATE minion_jobs SET lock_until = now() + ($1::double precision * interval '1 millisecond'), updated_at = now()
WHERE id = $2 AND lock_token = $3 AND status = 'active'
RETURNING id`,
[lockDurationMs, id, lockToken]
[lockDurationMs, id, lockToken],
opts
);
return rows.length > 0;
}
+177
View File
@@ -0,0 +1,177 @@
/**
* `gbrain jobs run-child` core (issue #5 per-job process isolation).
*
* The child side of the isolation boundary. The PARENT worker owns claim,
* lock renewal, completeJob/failJob and all attempt accounting; this process
* only executes the handler and reports ONE outcome file (see
* job-isolation.ts for the protocol). Child-owns-engine: every ctx callback
* below is token-fenced, so if the job is reclaimed while we run, our writes
* degrade to no-ops.
*
* Deliberately runs NONE of the worker machinery: no health probe, no stall
* detection, no lock timer the parent is the sole liveness owner. What it
* does install:
*
* - SIGTERM handler fires shutdownSignal ONLY (inline signal-separation
* parity): cooperative handlers keep ctx.signal live, finish inside the
* drain window, and write their outcome before the parent escalates to a
* group SIGKILL. Handlers that watch shutdownSignal (shell) run their own
* SIGTERMgraceSIGKILL cleanup.
* - Parent-liveness watchdog: polls `process.kill(parentPid, 0)` every 15s
* (a ppid check is DEAD CODE under tini the child's ppid is tini,
* which outlives the worker). On parent death: abort both signals, and
* hard-exit after a 30s grace so an orphaned LLM-bound handler doesn't
* burn spend to completion. Lock expiry + the stall sweeper requeue the
* job on the parent's side of the world.
*
* The CLI layer (jobs.ts case 'run-child') owns engine.disconnect() and
* process.exit() this module returns an exit code (engine-ownership
* invariant, same as MinionWorker).
*/
import type { BrainEngine } from '../engine.ts';
import { MinionQueue } from './queue.ts';
import type { MinionHandler } from './types.ts';
import { buildJobContext } from './job-context.ts';
import {
JOB_CHILD_EXIT_NOT_CLAIMED,
JOB_CHILD_EXIT_RESULT_WRITE_FAILED,
} from './worker-exit-codes.ts';
import { encodeHandlerError, unrefTimer, writeChildOutcomeFile } from './job-isolation.ts';
export interface RunChildOpts {
jobId: number;
lockToken: string;
resultPath: string;
/** Worker pid for the liveness watchdog; 0/absent disables the watchdog. */
parentPid: number;
}
export interface RunChildInjectables {
/** Hermetic tests inject a handler map instead of registerBuiltinHandlers. */
resolveHandler: (name: string) => MinionHandler | undefined;
/** Watchdog cadence override (default 15s). */
parentPollMs?: number;
/** Orphan hard-exit grace override (default 30s). */
orphanGraceMs?: number;
/** Exit hook for tests (default process.exit for the orphan path only). */
hardExit?: (code: number) => void;
}
export async function runChildJobEntry(
engine: BrainEngine,
opts: RunChildOpts,
injectables: RunChildInjectables,
): Promise<number> {
const queue = new MinionQueue(engine);
// Ground truth is the DB row, re-read here (one SELECT) rather than a
// serialized payload: the parent claimed it, but reclaim/cancel can race
// our startup. A mismatch means we must not run the handler at all.
const job = await queue.getJob(opts.jobId);
if (!job || job.status !== 'active' || job.lock_token !== opts.lockToken) {
process.stderr.write(
`[run-child] job ${opts.jobId} is not claimed by this token ` +
`(status=${job?.status ?? 'missing'}) — exiting without running the handler\n`,
);
return JOB_CHILD_EXIT_NOT_CLAIMED;
}
const handler = injectables.resolveHandler(job.name);
if (!handler) {
// Parity with inline mode, which dead-letters a missing handler
// immediately (failJob 'dead') — 'unrecoverable' reconstructs to
// UnrecoverableError parent-side, so the parent dead-letters on attempt 1
// instead of retrying a deterministic condition (adversarial-review P3).
try {
writeChildOutcomeFile(opts.resultPath, {
outcome: 'error',
errorKind: 'unrecoverable',
message: `No handler for job type '${job.name}' in the isolation child`,
});
return 0;
} catch {
return JOB_CHILD_EXIT_RESULT_WRITE_FAILED;
}
}
const abort = new AbortController();
const shutdown = new AbortController();
// SIGTERM = worker shutdown: fire ONLY shutdownSignal, preserving the
// inline signal-separation contract (worker.ts aborts only shutdownAbort on
// SIGTERM; per-job ctx.signal stays live so cooperative handlers finish +
// report inside the drain window instead of aborting mid-deploy —
// adversarial-review P2). The parent's group SIGKILL at drain end is the
// backstop for handlers that keep running.
const onSigterm = (): void => {
if (!shutdown.signal.aborted) shutdown.abort(new Error('worker-shutdown'));
};
// Parent death (orphan) aborts BOTH: nobody will SIGKILL us, the lock will
// expire and the job will be requeued — stop the handler outright.
const onOrphaned = (): void => {
onSigterm();
if (!abort.signal.aborted) abort.abort(new Error('worker-shutdown'));
};
process.on('SIGTERM', onSigterm);
// Parent-liveness watchdog (unref'd — never keeps the child alive).
const pollMs = injectables.parentPollMs ?? 15_000;
const graceMs = injectables.orphanGraceMs ?? 30_000;
const hardExit = injectables.hardExit ?? ((code: number) => process.exit(code));
let watchdog: ReturnType<typeof setInterval> | null = null;
if (opts.parentPid > 0) {
watchdog = setInterval(() => {
try {
process.kill(opts.parentPid, 0);
} catch {
process.stderr.write(
`[run-child] parent worker (pid ${opts.parentPid}) is gone — aborting handler; ` +
`hard exit in ${Math.round(graceMs / 1000)}s\n`,
);
if (watchdog != null) clearInterval(watchdog);
onOrphaned();
const t = setTimeout(() => hardExit(1), graceMs);
unrefTimer(t);
}
}, pollMs);
unrefTimer(watchdog);
}
try {
const context = buildJobContext(
engine,
queue,
job,
opts.lockToken,
abort.signal,
shutdown.signal,
);
let outcome;
try {
const result = await handler(context);
// completeJob's {value: x} wrap decision must run BEFORE JSON
// serialization: a JSON round-trip changes typeof for Date /
// toJSON-bearing results (object → string), which would flip the wrap
// parent-side (adversarial-review P3, result-shape parity). Wrap here;
// the parent's own wrap is then a no-op (object/undefined passthrough).
const wrapped = result != null
? (typeof result === 'object' ? (result as Record<string, unknown>) : { value: result })
: undefined;
outcome = { outcome: 'success' as const, result: wrapped };
} catch (err) {
outcome = encodeHandlerError(err);
}
try {
writeChildOutcomeFile(opts.resultPath, outcome);
} catch (writeErr) {
const msg = writeErr instanceof Error ? writeErr.message : String(writeErr);
process.stderr.write(`[run-child] failed to write outcome file: ${msg}\n`);
return JOB_CHILD_EXIT_RESULT_WRITE_FAILED;
}
return 0;
} finally {
if (watchdog != null) clearInterval(watchdog);
process.off('SIGTERM', onSigterm);
}
}
+29 -4
View File
@@ -98,6 +98,10 @@ export interface SupervisorOpts {
nice_requested?: number;
/** Effective niceness of the supervisor process after its own renice attempt. */
nice_effective?: number;
/** issue #5: when 'process', the spawned worker runs each claimed job in a
* SIGKILL-able child process (passed through as `--job-isolation process`).
* Omitted/inline: today's shared-process execution. */
jobIsolation?: 'inline' | 'process';
/** Error string if the supervisor's own renice failed (e.g. EPERM). */
nice_error?: string;
/**
@@ -174,7 +178,7 @@ const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
* niceness also inherits to the worker's own children automatically.
*/
export function buildWorkerArgs(
opts: Pick<SupervisorOpts, 'concurrency' | 'queue' | 'maxRssMb' | 'nice_requested'>,
opts: Pick<SupervisorOpts, 'concurrency' | 'queue' | 'maxRssMb' | 'nice_requested' | 'jobIsolation'>,
): string[] {
const args = [
'jobs', 'work',
@@ -187,6 +191,11 @@ export function buildWorkerArgs(
if (opts.nice_requested !== undefined) {
args.push('--nice', String(opts.nice_requested));
}
// Conditional push (issue #5): omitted for inline so existing deployments'
// argv is byte-identical (pinned by supervisor-build-worker-args.test.ts).
if (opts.jobIsolation === 'process') {
args.push('--job-isolation', 'process');
}
return args;
}
@@ -250,6 +259,7 @@ export async function queryWedgeSignals(
engine: BrainEngine,
queue: string,
handlerNames: string[],
opts?: { signal?: AbortSignal },
): Promise<WedgeSignals> {
const rows = await engine.executeRaw<{
stalled: string;
@@ -272,6 +282,7 @@ export async function queryWedgeSignals(
FROM minion_jobs
WHERE queue = $1`,
[queue, handlerNames],
opts,
);
const row = rows[0] ?? {
stalled: '0', active_healthy: '0', waiting: '0',
@@ -339,11 +350,23 @@ export async function probeQueueState(
): Promise<QueueSubmitState> {
const timeoutMs = opts.timeoutMs ?? QUEUE_PROBE_TIMEOUT_MS;
let timer: ReturnType<typeof setTimeout> | undefined;
// issue #6 / TODOS "cancel timed-out submit-time queue probes": the losing
// inner probe used to keep running on the pool after the race resolved —
// under pool exhaustion the abandoned query held a slot and made the
// exhaustion worse. The timeout now aborts a per-probe signal so the
// in-flight SQL is cancelled (postgres.js .cancel()) and its slot released.
const probeAbort = new AbortController();
const timeout = new Promise<QueueSubmitState>((resolveTimeout) => {
timer = setTimeout(() => resolveTimeout({ probe_failed: true }), timeoutMs);
timer = setTimeout(() => {
probeAbort.abort();
resolveTimeout({ probe_failed: true });
}, timeoutMs);
});
try {
return await Promise.race([probeQueueStateInner(engine, queue, handlerNames), timeout]);
return await Promise.race([
probeQueueStateInner(engine, queue, handlerNames, { signal: probeAbort.signal }),
timeout,
]);
} catch {
return { probe_failed: true };
} finally {
@@ -355,8 +378,9 @@ async function probeQueueStateInner(
engine: BrainEngine,
queue: string,
handlerNames: string[],
opts?: { signal?: AbortSignal },
): Promise<QueueSubmitState> {
const sig = await queryWedgeSignals(engine, queue, handlerNames);
const sig = await queryWedgeSignals(engine, queue, handlerNames, opts);
// Oldest-waiting age, same filter shape as the wedge signals. Perf: the
// wave's migration (landing in another lane) adds the btree
@@ -368,6 +392,7 @@ async function probeQueueStateInner(
FROM minion_jobs
WHERE queue = $1 AND status = 'waiting'`,
[queue],
opts,
);
const rawAge = ageRows[0]?.age ?? null;
const oldestAge = rawAge === null ? null : Number(rawAge);
+24
View File
@@ -218,6 +218,16 @@ export interface MinionWorkerOpts {
* hung probe would wedge the recursive setTimeout chain forever and
* silently disable the health monitor. Default: 10000 (10 seconds). */
dbProbeTimeoutMs?: number;
/** issue #5: 'process' runs each claimed job in a SIGKILL-able child
* process (blast radius = 1 job). Default 'inline' (today's behavior).
* Requires childCliInvocation; the CLI layer resolves + validates it. */
jobIsolation?: 'inline' | 'process';
/** How to invoke the gbrain CLI for job children (resolved fail-fast at
* worker startup by the CLI layer; structurally ChildCliInvocation from
* job-isolation.ts kept inline here to avoid an import cycle). */
childCliInvocation?: { cmd: string; argsPrefix: string[] } | null;
/** tini path for wrapping job children ('' = absent, direct spawn). */
childTiniPath?: string;
}
// --- Job Context (passed to handlers) ---
@@ -372,6 +382,20 @@ export type TranscriptEntry =
| { type: 'llm_turn'; model: string; tokens_in: number; tokens_out: number; ts: string }
| { type: 'error'; message: string; stack?: string; ts: string };
// --- Abort-reason literals (single source of truth) ---
//
// Per-job abort sites construct `new Error(REASON)`; classification sites
// (worker.ts INFRASTRUCTURE_ABORT_REASONS, child-job-runner.ts
// PER_JOB_ABORT_REASONS) match on the message. Deriving both sets from these
// constants keeps a rename at an abort site from silently flipping child
// classification (maintainability review).
/** Infrastructure faults: released, no attempt burned; stall sweeper requeues. */
export const ABORT_REASON_LOCK_RENEWAL_FAILED = 'lock-renewal-failed';
export const ABORT_REASON_LOCK_LOST = 'lock-lost';
/** Job-targeted aborts: keep their existing attempt semantics. */
export const ABORT_REASON_TIMEOUT = 'timeout';
// --- Errors ---
/** Throw this from a handler to skip all retry logic and go straight to 'dead'. */
+14
View File
@@ -22,3 +22,17 @@
/** Worker drained itself because RSS crossed the watchdog cap. */
export const WORKER_EXIT_RSS_WATCHDOG = 12;
// --- `gbrain jobs run-child` exit codes (issue #5 process isolation) -------
//
// The parent (child-job-runner.ts) classifies a child by RESULT-FILE PRESENCE
// first: a written outcome file + exit 0 is the normal path even for handler
// FAILURE (a reported error outcome is a successful report). Non-zero codes
// mean "could not run or could not report":
/** run-child misuse: bad/missing argv or env, or a PGLite engine (no isolation there). */
export const JOB_CHILD_EXIT_USAGE = 13;
/** Job row validation failed: not 'active' or lock-token mismatch (reclaimed). */
export const JOB_CHILD_EXIT_NOT_CLAIMED = 14;
/** Handler finished but the outcome file could not be written. */
export const JOB_CHILD_EXIT_RESULT_WRITE_FAILED = 15;
+201 -77
View File
@@ -18,7 +18,11 @@ import type {
MinionJob, MinionJobContext, MinionHandler, MinionWorkerOpts,
MinionQueueOpts, TokenUpdate,
} from './types.ts';
import { UnrecoverableError } from './types.ts';
import {
UnrecoverableError,
ABORT_REASON_LOCK_RENEWAL_FAILED,
ABORT_REASON_LOCK_LOST,
} from './types.ts';
import { MinionQueue } from './queue.ts';
import { calculateBackoff } from './backoff.ts';
import { RateLeaseUnavailableError } from './handlers/subagent.ts';
@@ -29,6 +33,20 @@ import {
type LockRenewalDeps,
type LockRenewalState,
} from './lock-renewal-tick.ts';
import {
runDbProbe,
getConnectionRouting,
DIRECT_PROBE_TIMEOUT_MS,
type DbProbeResult,
type PoolDiagnostics,
} from './db-probe.ts';
import { buildJobContext } from './job-context.ts';
import {
runJobInChild,
ChildSpawnInfraError,
ChildWorkerShutdownError,
ChildNotClaimedError,
} from './child-job-runner.ts';
import { lockRenewalAudit } from '../audit/lock-renewal-audit.ts';
import { isRetryableConnError } from '../retry-matcher.ts';
import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from './reconnect.ts';
@@ -49,8 +67,8 @@ import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from '
* to this set is a deliberate two-line change, not a silent regression).
*/
export const INFRASTRUCTURE_ABORT_REASONS = new Set<string>([
'lock-renewal-failed',
'lock-lost',
ABORT_REASON_LOCK_RENEWAL_FAILED,
ABORT_REASON_LOCK_LOST,
]);
import { randomUUID } from 'crypto';
import { EventEmitter } from 'events';
@@ -120,10 +138,19 @@ export function getAccurateRss(
}
/** Reason payload emitted with `'unhealthy'` when self-health-check trips.
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit. */
* CLI layer (jobs.ts:work) subscribes and decides whether to call process.exit.
* `verdict` (issue #6) distinguishes local pool starvation from a genuinely
* unreachable server so operators stop debugging the wrong layer; absent on
* engines without the probe's disambiguation lane. */
export type UnhealthyReason =
| { reason: 'db_dead'; consecutiveFailures: number; message: string }
| { reason: 'stalled'; waitingCount: number; idleMinutes: number };
| {
reason: 'db_dead';
consecutiveFailures: number;
message: string;
verdict?: 'pool_starved' | 'server_unreachable' | 'unknown';
}
| { reason: 'stalled'; waitingCount: number; idleMinutes: number }
| { reason: 'child_spawn_failing'; consecutiveFailures: number; message: string };
/**
* Read the quiet_hours JSONB column off a MinionJob, if present. The
@@ -188,6 +215,18 @@ export class MinionWorker extends EventEmitter {
private _peakRssMb = 0;
/** Latch so the 80%-of-cap soft-warn fires once per crossing, not every check. */
private _softWarnFired = false;
/**
* Circuit breaker for deterministic child-bootstrap failures (red-team
* finding): a spawn failure releases the job with no attempt burned, the
* stall sweeper requeues it, the same worker re-claims an infinite
* claim/release loop the stall detector cannot see (every settle refreshes
* the progress clock). After CHILD_SPAWN_FAIL_EXIT_AFTER consecutive
* spawn-class failures we emit 'unhealthy' so the process manager restarts
* the worker (and the supervisor's crash budget takes over if the child
* CLI stays broken).
*/
private _consecutiveChildSpawnFailures = 0;
private static readonly CHILD_SPAWN_FAIL_EXIT_AFTER = 3;
private opts: Required<MinionWorkerOpts>;
@@ -215,7 +254,22 @@ export class MinionWorker extends EventEmitter {
stallExitAfterMs: opts?.stallExitAfterMs ?? 10 * 60_000,
dbFailExitAfter: opts?.dbFailExitAfter ?? 3,
dbProbeTimeoutMs: opts?.dbProbeTimeoutMs ?? 10_000,
jobIsolation: opts?.jobIsolation ?? 'inline',
childCliInvocation: opts?.childCliInvocation ?? null,
childTiniPath: opts?.childTiniPath ?? '',
};
// Process isolation contract: 'process' without a resolved child CLI
// invocation would silently execute handlers INLINE while the evict path
// believed it was isolated (predicate mismatch — red-team finding). The
// CLI layer always resolves + validates the invocation; library callers
// must too. Loud construction throw, same discipline as the stall
// thresholds below.
if (this.opts.jobIsolation === 'process' && this.opts.childCliInvocation == null) {
throw new Error(
"MinionWorkerOpts: jobIsolation 'process' requires childCliInvocation " +
'(resolve it via resolveChildCliInvocation and validate it exists before constructing the worker).',
);
}
// Stall thresholds contract: exit MUST be strictly greater than warn.
// If exit <= warn, the warn-then-exit semantics break: a single tick at
// idle > warn would set stallWarningSince and the subsequent tick at
@@ -231,6 +285,15 @@ export class MinionWorker extends EventEmitter {
}
}
/**
* Read-only handler lookup. `gbrain jobs run-child` registers the builtin
* handlers against a throwaway worker (registerBuiltinHandlers's existing
* contract) and resolves the one it needs through this accessor.
*/
getHandler(name: string): MinionHandler | undefined {
return this.handlers.get(name);
}
/** Register a handler for a job type. */
register(name: string, handler: MinionHandler): void {
this.handlers.set(name, handler);
@@ -263,7 +326,9 @@ export class MinionWorker extends EventEmitter {
if (this.listenerCount('unhealthy') === 0) {
const detail = info.reason === 'db_dead'
? `DB unreachable (${info.consecutiveFailures} probes): ${info.message}`
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
: info.reason === 'child_spawn_failing'
? `job-child spawn failing (${info.consecutiveFailures} consecutive): ${info.message}`
: `worker stalled (${info.waitingCount} waiting, ${info.idleMinutes}m idle)`;
console.error(
`[health] FATAL: ${detail}. No 'unhealthy' listener registered; ` +
`defaulting to process.exit(1) for process-manager restart.`,
@@ -388,28 +453,36 @@ export class MinionWorker extends EventEmitter {
let healthRunning = false;
let healthExited = false;
// Race executeRaw against a wall-clock deadline. A hung connection
// (network-partitioned PgBouncer, deadlocked backend) would otherwise
// hold the await forever — the recursive setTimeout's next tick is only
// scheduled in `finally`, so a hung probe would silently disable the
// entire health monitor. The timeout treats hangs as failures and feeds
// them into `dbFailExitAfter`.
const probeWithTimeout = async (): Promise<void> => {
const ac = new AbortController();
const timeoutMs = this.opts.dbProbeTimeoutMs;
const timer = setTimeout(() => ac.abort(), timeoutMs);
try {
await Promise.race([
this.engine.executeRaw('SELECT 1'),
new Promise<never>((_, reject) => {
ac.signal.addEventListener('abort', () => {
reject(new Error(`probe timeout after ${timeoutMs}ms`));
});
}),
]);
} finally {
clearTimeout(timer);
}
// DB liveness probe with pool-starvation disambiguation (issue #6).
// The probe body lives in db-probe.ts (hermetically tested,
// lock-renewal-tick pattern); this is the thin adapter. Both probes
// carry an AbortSignal — a hung probe is CANCELLED (slot released),
// never abandoned. The direct-lane probe runs ONLY when dual-pool is
// genuinely active (a kill-switched executeRawDirect would probe the
// same starved read pool twice and fake a verdict).
const runProbe = async (): Promise<DbProbeResult> => {
const cm = getConnectionRouting(this.engine);
const dualPool = cm?.isDualPoolActive?.() === true;
const getDiag = (this.engine as {
getPoolDiagnostics?: () => PoolDiagnostics | null;
}).getPoolDiagnostics;
return runDbProbe({
probeRead: async (signal) => {
await this.engine.executeRaw('SELECT 1', undefined, { signal });
},
...(dualPool
? {
probeDirect: async (signal: AbortSignal) => {
await this.engine.executeRawDirect('SELECT 1', undefined, { signal });
},
}
: {}),
...(typeof getDiag === 'function'
? { getDiagnostics: () => getDiag.call(this.engine) }
: {}),
timeoutMs: this.opts.dbProbeTimeoutMs,
directTimeoutMs: DIRECT_PROBE_TIMEOUT_MS,
});
};
const runHealthCheck = async (): Promise<void> => {
@@ -417,25 +490,25 @@ export class MinionWorker extends EventEmitter {
healthRunning = true;
try {
// --- 1. DB liveness probe ---
try {
await probeWithTimeout();
const probe = await runProbe();
if (probe.ok) {
consecutiveDbFailures = 0;
} catch (e) {
} else {
consecutiveDbFailures++;
const msg = e instanceof Error ? e.message : String(e);
console.error(
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${msg}`,
`[health] DB probe failed (${consecutiveDbFailures}/${this.opts.dbFailExitAfter}): ${probe.detail}`,
);
if (consecutiveDbFailures >= this.opts.dbFailExitAfter) {
console.error(
`[health] DB unreachable after ${this.opts.dbFailExitAfter} consecutive probes. ` +
`Emitting 'unhealthy' for process-manager restart.`,
`[health] DB probe failed ${this.opts.dbFailExitAfter} consecutive times ` +
`(verdict: ${probe.verdict}). Emitting 'unhealthy' for process-manager restart.`,
);
healthExited = true;
this.emitUnhealthy({
reason: 'db_dead',
consecutiveFailures: consecutiveDbFailures,
message: msg,
message: probe.detail,
verdict: probe.verdict,
});
}
return; // Skip stall check when DB is flaky
@@ -876,7 +949,7 @@ export class MinionWorker extends EventEmitter {
// and the tick keeps its legacy no-reconnect behavior.
const engineReconnect = (this.engine as { reconnect?: (ctx?: { error?: unknown }) => Promise<void> }).reconnect;
const renewalDeps: LockRenewalDeps = {
renewLock: (id, tok, dur) => this.queue.renewLock(id, tok, dur),
renewLock: (id, tok, dur, opts) => this.queue.renewLock(id, tok, dur, opts),
audit: lockRenewalAudit,
now: Date.now,
setTimeout: (cb, ms) => globalThis.setTimeout(cb, ms),
@@ -946,11 +1019,12 @@ export class MinionWorker extends EventEmitter {
);
clearInterval(lockTimer);
this.inFlight.delete(job.id);
// D8a: don't failJob if the abort was infrastructure. The
// stall detector will reclaim the row cleanly because the
// lock has expired (lock-renewal aborts only fire after
// lockDuration - safetyMargin elapsed without renewal).
if (!INFRASTRUCTURE_ABORT_REASONS.has(reason)) {
// D8a: don't failJob on infrastructure aborts (stall detector
// reclaims after lock expiry). Isolation mode: also skip — the
// group SIGKILL already fired and executeJob's own recording
// follows; a competing evict failJob('dead') could dead-letter a
// job with attempts remaining (adversarial-review P3).
if (!INFRASTRUCTURE_ABORT_REASONS.has(reason) && this.opts.jobIsolation !== 'process') {
this.queue.failJob(
job.id,
lockToken,
@@ -1024,48 +1098,48 @@ export class MinionWorker extends EventEmitter {
return;
}
// issue #5: 'process' isolation runs the handler in a SIGKILL-able child;
// the parent keeps claim/renewal and ALL result recording below — this
// branch swaps ONLY the execution engine. When isolated, the parent-side
// context is skipped entirely (the child builds its own against its own
// engine; building it here would be dead work holding closures). The
// constructor guarantees childCliInvocation != null whenever
// jobIsolation === 'process', so this predicate matches the evict guard.
const isolated = this.opts.jobIsolation === 'process';
// Build job context with per-job AbortSignal + shared shutdown signal.
// Most handlers only care about `signal` (timeout / cancel / lock-loss).
// `shutdownSignal` is separate: fires only on worker process SIGTERM/SIGINT.
// Handlers that need to run cleanup before worker exit (shell handler's
// SIGTERM→5s→SIGKILL on its child) subscribe to shutdownSignal too.
const context: MinionJobContext = {
id: job.id,
name: job.name,
data: job.data,
attempts_made: job.attempts_made,
signal: abort.signal,
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
shutdownSignal: this.shutdownAbort.signal,
updateProgress: async (progress: unknown) => {
await this.queue.updateProgress(job.id, lockToken, progress);
},
updateTokens: async (tokens: TokenUpdate) => {
await this.queue.updateTokens(job.id, lockToken, tokens);
},
log: async (message: string | Record<string, unknown>) => {
const value = typeof message === 'string' ? message : JSON.stringify(message);
await this.engine.executeRaw(
`UPDATE minion_jobs SET stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
updated_at = now()
WHERE id = $2 AND status = 'active' AND lock_token = $3`,
[value, job.id, lockToken]
// Builder shared with `gbrain jobs run-child` (job-context.ts) so the
// process-isolation child wires the exact same DB-backed callbacks.
const context: MinionJobContext | null = isolated
? null
: buildJobContext(
this.engine,
this.queue,
job,
lockToken,
abort.signal,
this.shutdownAbort.signal,
);
},
isActive: async () => {
const rows = await this.engine.executeRaw<{ id: number }>(
`SELECT id FROM minion_jobs WHERE id = $1 AND status = 'active' AND lock_token = $2`,
[job.id, lockToken]
);
return rows.length > 0;
},
readInbox: async () => {
return this.queue.readInbox(job.id, lockToken);
},
};
try {
const result = await handler(context);
const result = isolated
? await runJobInChild({
jobId: job.id,
jobName: job.name,
lockToken,
abortSignal: abort.signal,
shutdownSignal: this.shutdownAbort.signal,
invocation: this.opts.childCliInvocation as { cmd: string; argsPrefix: string[] },
tiniPath: this.opts.childTiniPath,
})
: await handler(context as MinionJobContext);
// The child spawned and ran — the spawn path is healthy again.
this._consecutiveChildSpawnFailures = 0;
clearInterval(lockTimer);
@@ -1121,6 +1195,56 @@ export class MinionWorker extends EventEmitter {
return;
}
// Any error that ISN'T a spawn failure proves the spawn path works —
// keep the breaker's "consecutive" semantics honest.
if (!(err instanceof ChildSpawnInfraError)) {
this._consecutiveChildSpawnFailures = 0;
}
// issue #5 process isolation — two more infrastructure classes, same
// release semantics as the block above (lock expires once launchJob's
// finally clears the renewal timer; the stall sweeper requeues):
// - spawn failure: an ops misconfiguration (bad child CLI path) must
// not burn attempts job-by-job until the queue dead-letters. The
// CLI layer also fail-fast validates the invocation at startup.
// - worker shutdown: a routine deploy killed the child before it
// could report (codex-2 #7); burning an attempt per deploy would
// dead-letter long jobs after a few releases.
if (err instanceof ChildSpawnInfraError) {
console.error(
`Job ${job.id} (${job.name}) released after child spawn failure — ` +
`check the worker's child CLI configuration: ${errorText} (no attempt burned)`,
);
this._consecutiveChildSpawnFailures += 1;
if (this._consecutiveChildSpawnFailures >= MinionWorker.CHILD_SPAWN_FAIL_EXIT_AFTER) {
console.error(
`[isolation] ${this._consecutiveChildSpawnFailures} consecutive child spawn/bootstrap ` +
`failures — the child CLI is deterministically broken. Emitting 'unhealthy' for ` +
`process-manager restart instead of looping claim/release forever.`,
);
this.emitUnhealthy({
reason: 'child_spawn_failing',
consecutiveFailures: this._consecutiveChildSpawnFailures,
message: errorText,
});
}
return;
}
if (err instanceof ChildWorkerShutdownError) {
console.log(
`Job ${job.id} (${job.name}) released after worker shutdown (${errorText}); ` +
`stall detector will requeue (no attempt burned)`,
);
return;
}
if (err instanceof ChildNotClaimedError) {
// The child proved the claim is gone (reclaimed/cancelled before the
// handler ran). The token-fenced failJob would no-op anyway — return
// without burning anything against a claim we no longer hold.
console.log(`Job ${job.id} (${job.name}): ${errorText}`);
return;
}
// v0.41 Bug 2: lease-full bounces don't burn attempts.
//
// Pre-v0.41 every non-`UnrecoverableError` routed to `delayed` with
+48
View File
@@ -0,0 +1,48 @@
/**
* CheckoutGauge approximate in-flight query counters for the health probe's
* pool diagnostics (issue #6).
*
* HONESTY CONTRACT (read before extending): this gauge counts calls through
* the engine's raw/direct/reserved/transaction seams ONLY. The majority of
* engine traffic tagged-template queries on `this.sql` (getConfig, CRUD,
* search) is NOT tracked; postgres.js exposes no public checkout counters
* and proxying the Sql template function is too invasive for a diagnostic.
* Every consumer must label these numbers as a tracked SUBSET and must never
* derive "available" or "waiting" figures from them (that arithmetic is
* invented telemetry outside-voice review, codex-2 #3). The authoritative
* starvation signal is the direct-lane disambiguation probe in
* `src/core/minions/db-probe.ts`; these counts are supporting detail.
*
* Fail-open by construction: plain integer bumps, no I/O, release() clamps
* at zero so a missed acquire can never underflow into negative counts.
*/
/** Which engine seam the in-flight call went through. */
export type GaugeKind = 'raw' | 'direct' | 'reserved' | 'tx';
export interface PoolGaugeSnapshot {
/** executeRaw on the read pool. */
raw: number;
/** executeRawDirect (direct session lane when dual-pool, read pool otherwise). */
direct: number;
/** withReservedConnection holders. */
reserved: number;
/** transaction() bodies. */
tx: number;
}
export class CheckoutGauge {
private counts: PoolGaugeSnapshot = { raw: 0, direct: 0, reserved: 0, tx: 0 };
acquire(kind: GaugeKind): void {
this.counts[kind] += 1;
}
release(kind: GaugeKind): void {
if (this.counts[kind] > 0) this.counts[kind] -= 1;
}
snapshot(): PoolGaugeSnapshot {
return { ...this.counts };
}
}
+129 -13
View File
@@ -25,6 +25,7 @@ import {
type BatchAuditSite,
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import { CheckoutGauge, type PoolGaugeSnapshot } from './pool-gauge.ts';
import {
valueHash,
normalizeDimension,
@@ -82,7 +83,7 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import * as db from './db.ts';
import { ConnectionManager } from './connection-manager.ts';
import { ConnectionManager, DEFAULT_DIRECT_POOL_SIZE } from './connection-manager.ts';
import { logConnectionEvent } from './connection-audit.ts';
import { validateSlug, contentHash, rowToPage, rowToStalePage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, takeHitRowToHit, isUndefinedTableError, warnOncePerProcess } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
@@ -132,6 +133,13 @@ export class PostgresEngine implements BrainEngine {
private _savedConfig: (EngineConfig & { poolSize?: number; parentConnectionManager?: ConnectionManager }) | null = null;
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
private _reconnecting = false;
/**
* Approximate in-flight counters for the health probe's diagnostics
* (issue #6). Tracks the raw/direct/reserved/tx seams ONLY see the
* honesty contract in pool-gauge.ts. Shared by tx-scoped engine clones
* via the prototype chain (same process, same pools). Fail-open.
*/
private checkoutGauge = new CheckoutGauge();
/**
* #1471: module-singleton OWNERSHIP token. `true` only for the engine whose
* connect() actually created the shared db.ts `sql` singleton (returned
@@ -297,6 +305,8 @@ export class PostgresEngine implements BrainEngine {
max: size,
idle_timeout: 20,
connect_timeout: 10,
// Explicit (matches the postgres.js implicit default; GBRAIN_POOL_MAX_LIFETIME_S overrides).
max_lifetime: db.resolveMaxLifetimeSeconds(),
types: { bigint: postgres.BigInt },
// Silence postgres NOTICE-level messages by default. See db.ts for
// rationale (stdout-parsing callers like jobs-submit --json break when
@@ -1059,18 +1069,83 @@ export class PostgresEngine implements BrainEngine {
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
const conn = this.sql;
return conn.begin(async (tx) => {
// Create a scoped engine with tx as its connection, no shared state mutation
const txEngine = Object.create(this) as PostgresEngine;
Object.defineProperty(txEngine, 'sql', { get: () => tx });
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
return fn(txEngine);
}) as Promise<T>;
// try/finally, not .finally on the chained promise: begin() can throw
// SYNCHRONOUSLY (e.g. nested transaction on a tx clone whose conn has no
// .begin), which would skip a chained .finally and leak the counter.
this.checkoutGauge.acquire('tx');
try {
return await (conn.begin(async (tx) => {
// Create a scoped engine with tx as its connection, no shared state mutation
const txEngine = Object.create(this) as PostgresEngine;
Object.defineProperty(txEngine, 'sql', { get: () => tx });
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
return fn(txEngine);
}) as Promise<T>);
} finally {
this.checkoutGauge.release('tx');
}
}
/**
* issue #6 (reserved-connection routing): concurrent DIRECT-pool reserves
* are capped at directPoolSize - 1 so the claim/renewLock heartbeats always
* keep >= 1 direct slot; overflow falls back to the READ pool exactly the
* pre-routing behavior, so this change is strictly never-worse than the
* status quo (deliberate rejection of queue-for-a-permit: that would block
* migrations behind multi-minute CREATE INDEX holds). Per-process by
* design: each process owns its own direct pool, so a CLI migration's
* reserves cannot starve a worker's heartbeats.
*/
private _reservedDirectInFlight = 0;
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
const pool = this.sql;
const reserved = await pool.reserve();
// Long-hold reserved work (CREATE INDEX CONCURRENTLY, transaction:false
// migration DDL, backfill BEGIN..COMMIT batches) belongs on the DIRECT
// session lane: 30-min statement_timeout + maintenance_work_mem GUCs and
// it stops pinning the worker's shared read pool (the observed 353s
// COMMIT in issue #6 was a reserved read-pool slot). Never reroute inside
// an open transaction (same guard shape as executeRawDirect).
const inTransaction = this._sql !== null && this.connectionManager?.peekReadPool() !== this._sql;
let pool = this.sql;
let fromDirect = false;
if (!inTransaction && this.connectionManager?.isDualPoolActive()) {
const size = this.connectionManager.describeMode().direct_pool_size ?? DEFAULT_DIRECT_POOL_SIZE;
// NO floor on the cap (red-team finding): at direct_pool_size=1 a
// Math.max(1, ...) floor would let a multi-minute reserve consume the
// ONLY direct session and starve claim/renewLock heartbeats — the
// exact #6 class, reintroduced on the direct pool. cap <= 0 means the
// direct lane has no spare capacity for reserves: use the read pool
// (the true status quo).
const cap = size - 1;
if (cap >= 1 && this._reservedDirectInFlight < cap) {
// Take the permit in the SAME synchronous frame as the check — a
// check-then-increment spanning `await ddl()` is a TOCTOU that lets
// same-tick concurrent reserves overshoot the cap and starve the
// heartbeat slot the cap exists to protect (adversarial-review P2).
this._reservedDirectInFlight += 1;
fromDirect = true;
try {
pool = await this.connectionManager.ddl();
} catch {
// ddl() failure flips its own kill switch; fall back to the read
// pool (status quo) rather than failing the caller.
this._reservedDirectInFlight -= 1;
fromDirect = false;
pool = this.sql;
}
}
}
// Gauge BEFORE reserve(): a reserve() stuck waiting for a free slot is
// exactly the in-flight pressure the probe diagnostics should surface.
this.checkoutGauge.acquire('reserved');
let reserved: Awaited<ReturnType<typeof pool.reserve>>;
try {
reserved = await pool.reserve();
} catch (e) {
this.checkoutGauge.release('reserved');
if (fromDirect) this._reservedDirectInFlight -= 1;
throw e;
}
try {
const conn: ReservedConnection = {
async executeRaw<R = Record<string, unknown>>(
@@ -1091,7 +1166,34 @@ export class PostgresEngine implements BrainEngine {
};
return await fn(conn);
} finally {
reserved.release();
// Counter/gauge decrements run regardless of release() throwing
// (double-release or socket error must not permanently leak a permit
// of the small direct-reserve budget — data-migration review).
try {
reserved.release();
} catch {
// best-effort; the pool's own lifecycle handles a broken reservation
}
this.checkoutGauge.release('reserved');
if (fromDirect) this._reservedDirectInFlight -= 1;
}
}
/**
* Health-probe diagnostics (issue #6). Duck-typed deliberately NOT on the
* BrainEngine interface (PGLite has no pool to diagnose; the worker reads
* it optionally, same pattern as `engine.reconnect`). Fail-open: returns
* null instead of throwing.
*/
getPoolDiagnostics(): { tracked: PoolGaugeSnapshot; poolMax: number | null } | null {
try {
const max = (this.sql as unknown as { options?: { max?: number } }).options?.max;
return {
tracked: this.checkoutGauge.snapshot(),
poolMax: typeof max === 'number' ? max : null,
};
} catch {
return null;
}
}
@@ -6150,7 +6252,15 @@ export class PostgresEngine implements BrainEngine {
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
return this.runUnsafe<T>(this.sql, sql, params, opts);
// try/finally (not .finally on the promise): runUnsafe throws
// SYNCHRONOUSLY on a pre-aborted signal, which would skip a chained
// .finally and leak the counter.
this.checkoutGauge.acquire('raw');
try {
return await this.runUnsafe<T>(this.sql, sql, params, opts);
} finally {
this.checkoutGauge.release('raw');
}
// Pre-#406 behavior: throw on any error including connection death.
// Per-call auto-retry is not safe here because executeRaw is also used
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
@@ -6186,7 +6296,13 @@ export class PostgresEngine implements BrainEngine {
const conn = (!inTransaction && this.connectionManager?.isDualPoolActive())
? await this.connectionManager.ddl()
: this.sql;
return this.runUnsafe<T>(conn, sql, params, opts);
// try/finally, not .finally — see executeRaw (sync throw on pre-aborted signal).
this.checkoutGauge.acquire('direct');
try {
return await this.runUnsafe<T>(conn, sql, params, opts);
} finally {
this.checkoutGauge.release('direct');
}
}
// ============================================================
+1 -1
View File
@@ -1,6 +1,6 @@
# gbrain agent workspace — template
<!-- gbrain-template-stamp: 0.46.0.0 -->
<!-- gbrain-template-stamp: 0.46.1.0 -->
This repository is the **"Use this template"** distribution artifact for a
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
+305
View File
@@ -0,0 +1,305 @@
/**
* issue #5 `runJobInChild` against REAL child processes (.mjs harnesses run
* by process.execPath the child-worker-supervisor.test.ts pattern).
*
* Paths pinned:
* - success outcome file resolves with the handler result
* - error outcome file throws the reconstructed error class (exit 0!)
* - exit 1 with no file generic throw naming the exit (attempt burned)
* - SIGTERM-ignoring child + aborted signal group SIGKILL at the injected
* grace; "terminated after abort" classification
* - pre-aborted signal child killed promptly
* - spawn ENOENT ChildSpawnInfraError (release, no attempt burned)
* - worker-shutdown: child finishes + reports during the drain window
* normal success; child that can't report ChildWorkerShutdownError
* - child env contract (result path, lock token, parent pid, pool bounds)
*/
import { describe, expect, test, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, writeFileSync, existsSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
runJobInChild,
ChildSpawnInfraError,
ChildWorkerShutdownError,
ChildNotClaimedError,
} from '../src/core/minions/child-job-runner.ts';
import { UnrecoverableError } from '../src/core/minions/types.ts';
import { RateLeaseUnavailableError } from '../src/core/minions/handlers/subagent.ts';
const TEST_TIMEOUT_MS = 30_000;
let harnessDir: string;
function makeHarness(name: string, body: string): string {
const path = join(harnessDir, `${name}.mjs`);
writeFileSync(
path,
`import { writeFileSync, renameSync } from 'node:fs';\n` +
`const RESULT = process.env.GBRAIN_JOB_RESULT_PATH;\n` +
`const writeOutcome = (o) => { writeFileSync(RESULT + '.tmp', JSON.stringify(o)); renameSync(RESULT + '.tmp', RESULT); };\n` +
body +
// Readiness handshake LAST (after the body installed its signal
// handlers): fixed sleeps raced child startup on loaded CI runners —
// a SIGTERM landing before process.on('SIGTERM') installs takes the
// default disposition and flips shutdown-semantics assertions
// (testing specialist). The runner's outcome dir is internal, so the
// ready path comes from a TEST-provided env var.
`\nif (process.env.HARNESS_READY_PATH) writeFileSync(process.env.HARNESS_READY_PATH, '1');\n`,
'utf8',
);
return path;
}
/** Make a per-test ready path + the env to hand runJobInChild. */
function readiness(name: string): { env: Record<string, string | undefined>; wait: () => Promise<void> } {
const readyPath = join(harnessDir, `${name}.ready`);
return {
env: { ...process.env, HARNESS_READY_PATH: readyPath },
wait: async () => {
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
if (existsSync(readyPath)) return;
await new Promise((r) => setTimeout(r, 25));
}
throw new Error('harness never signalled readiness');
},
};
}
beforeAll(() => {
harnessDir = mkdtempSync(join(tmpdir(), 'gbrain-cjr-harness-'));
});
afterAll(() => {
rmSync(harnessDir, { recursive: true, force: true });
});
function baseOpts(harnessPath: string) {
return {
jobId: 77,
jobName: 'subagent',
lockToken: 'tok-cjr',
abortSignal: new AbortController().signal,
shutdownSignal: new AbortController().signal,
invocation: { cmd: process.execPath, argsPrefix: [harnessPath] },
tiniPath: '', // direct spawn in tests; group signaling covers both shapes
};
}
describe('runJobInChild (real children)', () => {
test('success outcome resolves with the result; env contract honored', async () => {
const harness = makeHarness(
'success',
`writeOutcome({ outcome: 'success', result: {
echoedToken: process.env.GBRAIN_JOB_LOCK_TOKEN,
isChild: process.env.GBRAIN_JOB_CHILD,
parentPid: process.env.GBRAIN_JOB_PARENT_PID,
poolSize: process.env.GBRAIN_POOL_SIZE,
directPoolSize: process.env.GBRAIN_DIRECT_POOL_SIZE,
argv: process.argv.slice(2),
}});\n` +
`process.exit(0);\n`,
);
const result = (await runJobInChild(baseOpts(harness))) as Record<string, unknown>;
expect(result.echoedToken).toBe('tok-cjr');
expect(result.isChild).toBe('1');
expect(result.parentPid).toBe(String(process.pid));
expect(result.poolSize).toBe('3');
expect(result.directPoolSize).toBe('1'); // codex-2 #6: child direct pool bounded
expect(result.argv).toEqual(['jobs', 'run-child', '--job-id', '77']);
}, TEST_TIMEOUT_MS);
test('error outcome (exit 0) throws the reconstructed class', async () => {
const harness = makeHarness(
'error-unrecoverable',
`writeOutcome({ outcome: 'error', errorKind: 'unrecoverable', message: 'bad schema, never retry' });\n` +
`process.exit(0);\n`,
);
await expect(runJobInChild(baseOpts(harness))).rejects.toBeInstanceOf(UnrecoverableError);
}, TEST_TIMEOUT_MS);
test('rate-lease outcome rebuilds RateLeaseUnavailableError with fields', async () => {
const harness = makeHarness(
'error-lease',
`writeOutcome({ outcome: 'error', errorKind: 'rate_lease', message: 'lease full', lease: { key: 'anthropic', active: 4, max: 4 } });\n` +
`process.exit(0);\n`,
);
try {
await runJobInChild(baseOpts(harness));
throw new Error('should have thrown');
} catch (e) {
expect(e).toBeInstanceOf(RateLeaseUnavailableError);
expect((e as RateLeaseUnavailableError).key).toBe('anthropic');
}
}, TEST_TIMEOUT_MS);
test('exit 1 with no outcome file → generic throw naming the exit code', async () => {
const harness = makeHarness('crash', `process.exit(1);\n`);
await expect(runJobInChild(baseOpts(harness))).rejects.toThrow(/exit code=1/);
}, TEST_TIMEOUT_MS);
test('SIGTERM-ignoring child: abort → group SIGKILL at the injected grace', async () => {
const harness = makeHarness(
'stubborn',
`process.on('SIGTERM', () => {});\n` +
`setInterval(() => {}, 1000);\n`, // never exits voluntarily
);
const abort = new AbortController();
const ready = readiness('stubborn');
const opts = { ...baseOpts(harness), abortSignal: abort.signal, killGraceMs: 400, env: ready.env };
const p = runJobInChild(opts);
await ready.wait();
abort.abort(new Error('timeout'));
const started = Date.now();
await expect(p).rejects.toThrow(/terminated after abort/);
// Died via the SIGKILL escalation, not the 30s force-evict scale.
expect(Date.now() - started).toBeLessThan(5_000);
}, TEST_TIMEOUT_MS);
test('pre-aborted signal: child is terminated promptly', async () => {
const harness = makeHarness(
'prekilled',
`setInterval(() => {}, 1000);\n`,
);
const abort = new AbortController();
abort.abort(new Error('cancel'));
const opts = { ...baseOpts(harness), abortSignal: abort.signal, killGraceMs: 400 };
await expect(runJobInChild(opts)).rejects.toThrow(/terminated after abort/);
}, TEST_TIMEOUT_MS);
test('spawn ENOENT → ChildSpawnInfraError (infra release, not a job defect)', async () => {
const opts = {
...baseOpts('/nonexistent'),
invocation: { cmd: '/nonexistent/gbrain-binary', argsPrefix: [] },
};
await expect(runJobInChild(opts)).rejects.toBeInstanceOf(ChildSpawnInfraError);
}, TEST_TIMEOUT_MS);
test('worker shutdown: child finishes + reports during the drain window → normal success', async () => {
const harness = makeHarness(
'graceful-drain',
`let done = false;\n` +
`process.on('SIGTERM', () => {\n` +
` writeOutcome({ outcome: 'success', result: { finishedDuringDrain: true } });\n` +
` done = true; process.exit(0);\n` +
`});\n` +
`setInterval(() => {}, 1000);\n`,
);
const shutdown = new AbortController();
const ready = readiness('graceful-drain');
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 5_000, env: ready.env };
const p = runJobInChild(opts);
await ready.wait();
shutdown.abort(new Error('worker-shutdown'));
const result = (await p) as Record<string, unknown>;
expect(result.finishedDuringDrain).toBe(true);
}, TEST_TIMEOUT_MS);
test('worker shutdown: child that cannot report → ChildWorkerShutdownError (no attempt burned)', async () => {
const harness = makeHarness(
'shutdown-stubborn',
`process.on('SIGTERM', () => {});\n` +
`setInterval(() => {}, 1000);\n`,
);
const shutdown = new AbortController();
const ready = readiness('shutdown-stubborn');
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 400, env: ready.env };
const p = runJobInChild(opts);
await ready.wait();
shutdown.abort(new Error('worker-shutdown'));
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
}, TEST_TIMEOUT_MS);
test('ERROR outcome during worker shutdown → ChildWorkerShutdownError, not a burned attempt (adversarial P2)', async () => {
// A cooperative handler that bails on shutdown and reports an error must
// be RELEASED — punishing exactly the well-behaved handlers on every
// deploy inverts the no-burn guarantee.
const harness = makeHarness(
'shutdown-error-report',
`process.on('SIGTERM', () => {\n` +
` writeOutcome({ outcome: 'error', errorKind: 'generic', message: 'aborted: shutdown' });\n` +
` process.exit(0);\n` +
`});\n` +
`setInterval(() => {}, 1000);\n`,
);
const shutdown = new AbortController();
const ready = readiness('shutdown-error-report');
const opts = { ...baseOpts(harness), shutdownSignal: shutdown.signal, killGraceMs: 5_000, env: ready.env };
const p = runJobInChild(opts);
await ready.wait();
shutdown.abort(new Error('worker-shutdown'));
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
}, TEST_TIMEOUT_MS);
test('watchdog drain (BOTH signals aborted, non-per-job reason) → shutdown class, not a burned attempt (adversarial P3)', async () => {
const harness = makeHarness(
'watchdog-stubborn',
`process.on('SIGTERM', () => {});\n` +
`setInterval(() => {}, 1000);\n`,
);
const abort = new AbortController();
const shutdown = new AbortController();
const ready = readiness('watchdog-stubborn');
const opts = {
...baseOpts(harness),
abortSignal: abort.signal,
shutdownSignal: shutdown.signal,
killGraceMs: 400,
env: ready.env,
};
const p = runJobInChild(opts);
await ready.wait();
// gracefulShutdown('watchdog') aborts BOTH — shutdown classification must win.
shutdown.abort(new Error('watchdog'));
abort.abort(new Error('watchdog'));
await expect(p).rejects.toBeInstanceOf(ChildWorkerShutdownError);
}, TEST_TIMEOUT_MS);
test('per-job reason (timeout) wins over a concurrent shutdown — attempt semantics preserved', async () => {
const harness = makeHarness(
'timeout-during-shutdown',
`process.on('SIGTERM', () => {});\n` +
`setInterval(() => {}, 1000);\n`,
);
const abort = new AbortController();
const shutdown = new AbortController();
const ready = readiness('timeout-during-shutdown');
const opts = {
...baseOpts(harness),
abortSignal: abort.signal,
shutdownSignal: shutdown.signal,
killGraceMs: 400,
env: ready.env,
};
const p = runJobInChild(opts);
await ready.wait();
shutdown.abort(new Error('worker-shutdown'));
abort.abort(new Error('timeout')); // the JOB was targeted — not shutdown class
await expect(p).rejects.toThrow(/terminated after abort/);
}, TEST_TIMEOUT_MS);
test('bootstrap exit codes: 13 → ChildSpawnInfraError, 14 → ChildNotClaimedError (no burned attempts)', async () => {
const usage = makeHarness('exit13', `process.exit(13);\n`);
await expect(runJobInChild(baseOpts(usage))).rejects.toBeInstanceOf(ChildSpawnInfraError);
const notClaimed = makeHarness('exit14', `process.exit(14);\n`);
await expect(runJobInChild(baseOpts(notClaimed))).rejects.toBeInstanceOf(ChildNotClaimedError);
}, TEST_TIMEOUT_MS);
test('child pool-size env: a STRICTER user GBRAIN_POOL_SIZE is respected; explicit child override wins; invalid falls back', async () => {
const harness = makeHarness(
'pool-echo',
`writeOutcome({ outcome: 'success', result: { poolSize: process.env.GBRAIN_POOL_SIZE } });\n` +
`process.exit(0);\n`,
);
const run = (env: Record<string, string | undefined>) =>
runJobInChild({ ...baseOpts(harness), env: { ...process.env, ...env } }) as Promise<{ poolSize: string }>;
expect((await run({ GBRAIN_POOL_SIZE: '2' })).poolSize).toBe('2'); // stricter user tuning respected
expect((await run({ GBRAIN_POOL_SIZE: '10' })).poolSize).toBe('3'); // never raised above the child default
expect((await run({ GBRAIN_POOL_SIZE: '2', GBRAIN_JOB_CHILD_POOL_SIZE: '5' })).poolSize).toBe('5'); // explicit knob wins
expect((await run({ GBRAIN_JOB_CHILD_POOL_SIZE: 'abc' })).poolSize).toBe('3'); // invalid → default
}, TEST_TIMEOUT_MS);
});
+110
View File
@@ -0,0 +1,110 @@
/**
* issue #6 `runDrainRenewalTick` (cycle drain lock-renewal tick, extracted
* from the inline setInterval in synthesize.ts). The previous inline tick had
* no per-call timeout: a hung renewLock stacked one checked-out pool slot per
* interval firing, forever. The extracted tick:
*
* - passes a per-call AbortSignal that is aborted when the timeout wins
* (the losing UPDATE is cancelled, its slot released)
* - invokes onLost exactly once when the token fence is lost (ok === false)
* - swallows errors and timeouts (best-effort; the next tick retries)
*
* Hermetic: injected renewLock, no DB, no real cycle.
*/
import { describe, expect, test } from 'bun:test';
import { readFileSync } from 'node:fs';
import { runDrainRenewalTick } from '../src/core/cycle/synthesize.ts';
describe('drain-loop wiring (structural — the shape guard only covers worker.ts)', () => {
const src = readFileSync(
new URL('../src/core/cycle/synthesize.ts', import.meta.url),
'utf-8',
);
test('the renewTimer interval is guarded and routes through runDrainRenewalTick', () => {
expect(src).toContain('if (drainTickInFlight) return;');
expect(src).toContain('void runDrainRenewalTick(');
// The pre-fix inline shape (an unguarded queue.renewLock(...).then chain
// inside the interval) must not come back.
expect(src).not.toMatch(/setInterval\(\(\) => \{\s*\n\s*queue\.renewLock\(/);
});
});
describe('db-lock heartbeat wiring (structural — issue #6 cancellation)', () => {
const src = readFileSync(new URL('../src/core/db-lock.ts', import.meta.url), 'utf-8');
test('withRefreshingLock aborts a per-tick signal into handle.refresh and guards re-entrancy', () => {
expect(src).toContain('handle.refresh({ signal: tickAbort.signal })');
expect(src).toContain('if (refreshTickInFlight) return;');
// refresh() forwards the opts to executeRawDirect as the trailing arg.
expect(src).toMatch(/executeRawDirect<\{ id: string \}>\([\s\S]*?refreshOpts,\s*\)/);
});
});
describe('runDrainRenewalTick (issue #6)', () => {
test('successful renewal: signal not aborted, onLost not called', async () => {
let seenSignal: AbortSignal | undefined;
let lost = 0;
await runDrainRenewalTick(
async (_id, _tok, _ms, opts) => {
seenSignal = opts?.signal;
return true;
},
42,
'tok',
30_000,
() => { lost += 1; },
1_000,
);
expect(seenSignal).toBeInstanceOf(AbortSignal);
expect(seenSignal!.aborted).toBe(false);
expect(lost).toBe(0);
});
test('lost token fence (ok=false): onLost called once', async () => {
let lost = 0;
await runDrainRenewalTick(
async () => false,
42,
'tok',
30_000,
() => { lost += 1; },
1_000,
);
expect(lost).toBe(1);
});
test('hung renewLock: tick resolves at the timeout and aborts the per-call signal', async () => {
let seenSignal: AbortSignal | undefined;
let lost = 0;
const started = Date.now();
await runDrainRenewalTick(
(_id, _tok, _ms, opts) => {
seenSignal = opts?.signal;
return new Promise<boolean>(() => { /* hangs forever */ });
},
42,
'tok',
30_000,
() => { lost += 1; },
50, // short timeout keeps the test fast
);
// Resolved via the timeout path (not the hung renewal).
expect(Date.now() - started).toBeGreaterThanOrEqual(40);
expect(seenSignal).toBeInstanceOf(AbortSignal);
expect(seenSignal!.aborted).toBe(true);
expect(lost).toBe(0); // timeout is NOT a lost fence
});
test('throwing renewLock is swallowed (best-effort; next tick retries)', async () => {
let lost = 0;
await runDrainRenewalTick(
async () => { throw new Error('CONNECTION_ENDED'); },
42,
'tok',
30_000,
() => { lost += 1; },
1_000,
);
expect(lost).toBe(0);
});
});
+7 -2
View File
@@ -120,12 +120,17 @@ describe('claim stamps timeout_at (deadlineAtMs ground truth)', () => {
describe('deadline plumbing wiring (structural)', () => {
const workerSrc = readFileSync(new URL('../src/core/minions/worker.ts', import.meta.url), 'utf-8');
// The context builder was extracted from executeJob into job-context.ts
// (shared with `jobs run-child` for process isolation) — the deadlineAtMs
// derivation lives there now; worker.ts calls buildJobContext.
const jobContextSrc = readFileSync(new URL('../src/core/minions/job-context.ts', import.meta.url), 'utf-8');
const jobsSrc = readFileSync(new URL('../src/commands/jobs.ts', import.meta.url), 'utf-8');
const cycleSrc = readFileSync(new URL('../src/core/cycle.ts', import.meta.url), 'utf-8');
const patternsSrc = readFileSync(new URL('../src/core/cycle/patterns.ts', import.meta.url), 'utf-8');
test('worker exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
expect(workerSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
test('job context exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
expect(jobContextSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
expect(workerSrc).toContain('buildJobContext(');
});
test('worker arms its abort timer from timeout_at when present (one absolute deadline)', () => {
+120
View File
@@ -0,0 +1,120 @@
/**
* issue #6 `resolveMaxLifetimeSeconds`: explicit, env-overridable
* max_lifetime for all four postgres() call sites.
*
* NOT a behavior change at default: postgres.js (verified against 3.4.9)
* already defaults max_lifetime to `60 * (30 + Math.random() * 30)`. This
* resolver makes the value explicit and adds the GBRAIN_POOL_MAX_LIFETIME_S
* incident escape hatch (0 disables recycling; N = seconds).
*
* Hermetic: resolver only env injected as a param (rule R1), no pools.
*/
import { describe, expect, test, beforeEach } from 'bun:test';
import {
resolveMaxLifetimeSeconds,
_resetMaxLifetimeWarningForTests,
} from '../src/core/db.ts';
import { withEnv } from './helpers/with-env.ts';
beforeEach(() => {
_resetMaxLifetimeWarningForTests();
});
describe('resolveMaxLifetimeSeconds', () => {
test('default (no env): a per-CONNECTION jitter FUNCTION, 30-60 minutes', () => {
// Must be a function, not a pre-evaluated number: postgres.js re-evaluates
// a function default per connection, so connections in one pool get
// independent recycle deadlines instead of a synchronized reconnect spike
// (data-migration specialist).
const v = resolveMaxLifetimeSeconds({});
expect(typeof v).toBe('function');
const fn = v as () => number;
for (let i = 0; i < 20; i++) {
const n = fn();
expect(Number.isInteger(n)).toBe(true);
expect(n).toBeGreaterThanOrEqual(1800);
expect(n).toBeLessThanOrEqual(3600);
}
});
test('env override: positive integer seconds honored verbatim', () => {
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '900' })).toBe(900);
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '1' })).toBe(1);
});
test('env 0 disables recycling (null — postgres.js accepts null)', () => {
expect(resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '0' })).toBeNull();
});
test('empty string falls through to the default (function)', () => {
const v = resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: '' });
expect(typeof v).toBe('function');
});
test('invalid values warn once on stderr and fall back to the default', () => {
const writes: string[] = [];
const realWrite = process.stderr.write.bind(process.stderr);
(process.stderr as { write: unknown }).write = (chunk: string) => {
writes.push(String(chunk));
return true;
};
try {
for (const bad of ['abc', '-5', '3.5', 'NaN']) {
const v = resolveMaxLifetimeSeconds({ GBRAIN_POOL_MAX_LIFETIME_S: bad });
expect(typeof v).toBe('function');
}
} finally {
(process.stderr as { write: unknown }).write = realWrite;
}
// warn-once latch: 4 bad values, exactly 1 warning
const warnings = writes.filter((w) => w.includes('GBRAIN_POOL_MAX_LIFETIME_S'));
expect(warnings.length).toBe(1);
});
test('jitter varies per CONNECTION (thundering-herd protection)', () => {
const fn = resolveMaxLifetimeSeconds({}) as () => number;
const values = new Set<number>();
for (let i = 0; i < 30; i++) values.add(fn());
expect(values.size).toBeGreaterThan(1);
});
test('wiring: the env override reaches a REAL constructed pool (ConnectionManager read pool)', async () => {
// postgres() is lazy — constructing the pool performs no I/O, so this
// pins the construction seam without a database. Without this, the
// resolver could be green while GBRAIN_POOL_MAX_LIFETIME_S is silently
// dead at every call site (adversarial-review vacuity finding).
const { ConnectionManager } = await import('../src/core/connection-manager.ts');
const { endPoolBounded } = await import('../src/core/db.ts');
await withEnv({ GBRAIN_POOL_MAX_LIFETIME_S: '900' }, async () => {
const cm = new ConnectionManager({
url: 'postgresql://user@127.0.0.1:5/never-connected',
});
const pool = await cm.getReadPool();
try {
expect(
(pool as unknown as { options: { max_lifetime: number | null } }).options.max_lifetime,
).toBe(900);
} finally {
await endPoolBounded(pool);
}
});
});
test('wiring: the default reaches a real pool as a FUNCTION (per-connection jitter)', async () => {
const { ConnectionManager } = await import('../src/core/connection-manager.ts');
const { endPoolBounded } = await import('../src/core/db.ts');
await withEnv({ GBRAIN_POOL_MAX_LIFETIME_S: undefined }, async () => {
const cm = new ConnectionManager({
url: 'postgresql://user@127.0.0.1:5/never-connected',
});
const pool = await cm.getReadPool();
try {
const v = (pool as unknown as { options: { max_lifetime: unknown } }).options.max_lifetime;
expect(typeof v).toBe('function');
} finally {
await endPoolBounded(pool);
}
});
});
});
+136
View File
@@ -0,0 +1,136 @@
/**
* issue #6 `runDbProbe` verdict matrix (hermetic; injected deps only).
*
* read OK { ok: true }
* read FAIL + direct OK pool_starved (honest disjunction wording)
* read FAIL + direct FAIL server_unreachable
* read FAIL + no direct lane unknown
* hung probes cancelled via their AbortSignals
* diagnostics absent/throwing fail-open (verdict still produced)
* tracked=0 subset message points at untracked traffic
*/
import { describe, expect, test } from 'bun:test';
import { runDbProbe } from '../src/core/minions/db-probe.ts';
const FAST = { timeoutMs: 5_000, directTimeoutMs: 5_000 };
const SHORT = { timeoutMs: 50, directTimeoutMs: 50 };
describe('runDbProbe verdict matrix', () => {
test('read OK → ok:true, no verdict', async () => {
const res = await runDbProbe({ probeRead: async () => {}, ...FAST });
expect(res).toEqual({ ok: true });
});
test('read FAIL + direct OK → pool_starved with honest-disjunction wording', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('probe timeout after 10000ms'); },
probeDirect: async () => {},
getDiagnostics: () => ({
tracked: { raw: 9, direct: 0, reserved: 1, tx: 0 },
poolMax: 10,
}),
...FAST,
});
expect(res.ok).toBe(false);
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('pool_starved');
expect(res.detail).toContain('server IS reachable');
// Codex-2 #2: the message must NOT claim to distinguish client pool
// exhaustion from a pooler-layer fault.
expect(res.detail).toContain('client pool exhaustion or a pooler-layer fault');
expect(res.detail).toContain('raw=9');
expect(res.detail).toContain('read pool max 10');
// Codex-2 #3: no invented waiter arithmetic anywhere in the message.
expect(res.detail).not.toMatch(/waiting/i);
expect(res.detail).toContain('subset');
});
test('read FAIL + direct FAIL → server_unreachable', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('probe timeout after 10000ms'); },
probeDirect: async () => { throw new Error('connect ECONNREFUSED'); },
...FAST,
});
expect(res.ok).toBe(false);
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('server_unreachable');
expect(res.detail).toContain('ECONNREFUSED');
});
test('read FAIL + no direct lane → unknown', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('boom'); },
...FAST,
});
expect(res.ok).toBe(false);
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('unknown');
expect(res.detail).toContain('no direct lane');
});
test('hung read probe: cancelled via its signal at the deadline', async () => {
let readSignal: AbortSignal | undefined;
const res = await runDbProbe({
probeRead: (signal) => {
readSignal = signal;
return new Promise<never>(() => {});
},
...SHORT,
});
expect(res.ok).toBe(false);
expect(readSignal?.aborted).toBe(true);
});
test('hung direct probe: cancelled at its own (shorter) deadline', async () => {
let directSignal: AbortSignal | undefined;
const res = await runDbProbe({
probeRead: async () => { throw new Error('read dead'); },
probeDirect: (signal) => {
directSignal = signal;
return new Promise<never>(() => {});
},
...SHORT,
});
expect(res.ok).toBe(false);
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('server_unreachable');
expect(directSignal?.aborted).toBe(true);
});
test('tracked=0 subset: message names untracked template traffic + runbook', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('probe timeout'); },
probeDirect: async () => {},
getDiagnostics: () => ({
tracked: { raw: 0, direct: 0, reserved: 0, tx: 0 },
poolMax: 10,
}),
...FAST,
});
if (res.ok) throw new Error('unreachable');
expect(res.detail).toContain('untracked');
expect(res.detail).toContain('queue-operations-runbook');
});
test('diagnostics absent → verdict still produced (fail-open)', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('probe timeout'); },
probeDirect: async () => {},
...FAST,
});
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('pool_starved');
});
test('diagnostics THROWING → verdict still produced (fail-open)', async () => {
const res = await runDbProbe({
probeRead: async () => { throw new Error('probe timeout'); },
probeDirect: async () => {},
getDiagnostics: () => { throw new Error('gauge exploded'); },
...FAST,
});
if (res.ok) throw new Error('unreachable');
expect(res.verdict).toBe('pool_starved');
});
});
+133
View File
@@ -0,0 +1,133 @@
/**
* issue #5 process isolation against REAL Postgres (DATABASE_URL-gated;
* wired into .github/workflows/e2e.yml tier1 e2e.yml runs only explicitly
* NAMED files, so an unwired e2e file is silent coverage loss).
*
* Legs:
* 1. Worker-level concurrency (codex-2 #6): a real MinionWorker with
* isolation on drains 6 jobs at concurrency 3 through real child
* processes against the real pooler/DB the child-pool topology the
* per-process pool math describes.
* 2. Real CLI entrypoint: `bun src/cli.ts jobs run-child` with the env
* contract against a genuinely claimed job proves engine bootstrap,
* handler registry (quiet), token validation, and the outcome protocol
* end-to-end. Uses the real 'orphans' handler (cheap on a fresh DB).
* 3. Stuck-child kill: a SIGTERM-ignoring child is group-SIGKILLed and the
* worker keeps claiming.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, rmSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
import { MinionQueue } from '../../src/core/minions/queue.ts';
import { MinionWorker } from '../../src/core/minions/worker.ts';
import { decodeChildOutcomeFile, CHILD_ENV } from '../../src/core/minions/job-isolation.ts';
import { withEnv } from '../helpers/with-env.ts';
const FIXTURE = resolve(import.meta.dir, '..', 'fixtures', 'fake-run-child.mjs');
const CLI = resolve(import.meta.dir, '..', '..', 'src', 'cli.ts');
const describeDb = hasDatabase() ? describe : describe.skip;
let queue: MinionQueue;
let tmp: string;
beforeAll(async () => {
if (!hasDatabase()) return;
await setupDB();
queue = new MinionQueue(getEngine());
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-isolation-'));
});
afterAll(async () => {
if (!hasDatabase()) return;
await teardownDB();
if (tmp) rmSync(tmp, { recursive: true, force: true });
});
async function rowStatus(id: number): Promise<string> {
const rows = await getEngine().executeRaw<{ status: string }>(
'SELECT status FROM minion_jobs WHERE id = $1',
[id],
);
return rows[0]?.status ?? 'missing';
}
describeDb('process isolation on real Postgres', () => {
test('concurrency 3: six isolated jobs drain through real children', async () => {
await withEnv({ FAKE_RUN_CHILD_MODE: 'success' }, async () => {
const jobs = [];
for (let i = 0; i < 6; i++) jobs.push(await queue.add('isotest', { i }));
const worker = new MinionWorker(getEngine(), {
queue: 'default',
concurrency: 3,
pollInterval: 50,
healthCheckInterval: 0,
maxRssMb: 0,
jobIsolation: 'process',
childCliInvocation: { cmd: process.execPath, argsPrefix: [FIXTURE] },
});
worker.register('isotest', async () => {
throw new Error('parent handler must not run when isolated');
});
const run = worker.start();
const deadline = Date.now() + 30_000;
try {
while (Date.now() < deadline) {
const statuses = await Promise.all(jobs.map((j) => rowStatus(j.id)));
if (statuses.every((s) => s === 'completed')) break;
await new Promise((r) => setTimeout(r, 100));
}
} finally {
worker.stop();
await run;
}
const statuses = await Promise.all(jobs.map((j) => rowStatus(j.id)));
expect(statuses).toEqual(['completed', 'completed', 'completed', 'completed', 'completed', 'completed']);
});
}, 60_000);
test('real CLI run-child: engine bootstrap + protocol end-to-end', async () => {
const job = await queue.add('orphans', {});
const claimed = await queue.claim('e2e-cli-tok', 60_000, 'default', ['orphans']);
expect(claimed?.id).toBe(job.id);
const resultPath = join(tmp, `cli-${job.id}.json`);
const res = spawnSync(
process.execPath,
[CLI, 'jobs', 'run-child', '--job-id', String(job.id)],
{
env: {
...process.env,
GBRAIN_DATABASE_URL: process.env.DATABASE_URL,
GBRAIN_TEST_ALLOW_DATABASE_URL: '1',
[CHILD_ENV.lockToken]: 'e2e-cli-tok',
[CHILD_ENV.resultPath]: resultPath,
[CHILD_ENV.parentPid]: String(process.pid),
},
timeout: 60_000,
stdio: ['ignore', 'pipe', 'pipe'],
},
);
if (res.status !== 0) {
throw new Error(
`run-child exited ${res.status}: ${String(res.stderr)}\n${String(res.stdout)}`,
);
}
expect(existsSync(resultPath)).toBe(true);
const outcome = decodeChildOutcomeFile(resultPath);
// The protocol completed; the real handler's own verdict (success or a
// reported error on a bare DB) is out of scope for this leg.
expect(outcome.outcome === 'success' || outcome.outcome === 'error').toBe(true);
}, 90_000);
// The stuck-child group-SIGKILL path is pinned with REAL processes in
// test/child-job-runner.test.ts (SIGTERM-ignorer → SIGKILL at grace) and
// the crash/burn semantics in test/worker-job-isolation.test.ts — this
// lane deliberately doesn't duplicate them against the shared CI DB.
});
+48
View File
@@ -0,0 +1,48 @@
// Fake `jobs run-child` for the isolation tests (worker-job-isolation.test.ts
// + test/e2e/job-isolation.test.ts): honors the isolation env contract
// without needing a compiled gbrain binary or a Postgres engine.
// Mode via FAKE_RUN_CHILD_MODE: success | error | exit15 | crash.
import { writeFileSync, renameSync } from 'node:fs';
const resultPath = process.env.GBRAIN_JOB_RESULT_PATH;
const mode = process.env.FAKE_RUN_CHILD_MODE ?? 'success';
function writeOutcome(o) {
writeFileSync(resultPath + '.tmp', JSON.stringify(o));
renameSync(resultPath + '.tmp', resultPath);
}
if (!resultPath) {
process.stderr.write('[fake-run-child] missing GBRAIN_JOB_RESULT_PATH\n');
process.exit(13);
}
if (mode === 'success') {
writeOutcome({
outcome: 'success',
result: {
fromChild: true,
token: process.env.GBRAIN_JOB_LOCK_TOKEN ?? null,
argv: process.argv.slice(2),
},
});
process.exit(0);
}
if (mode === 'error') {
writeOutcome({
outcome: 'error',
errorKind: 'generic',
message: 'fake child handler failure',
});
process.exit(0);
}
if (mode === 'exit15') {
// Simulates a result-write failure (JOB_CHILD_EXIT_RESULT_WRITE_FAILED):
// handler ran, outcome could not be persisted.
process.exit(15);
}
// crash: no outcome file
process.exit(1);
+256
View File
@@ -0,0 +1,256 @@
/**
* issue #5 job-isolation protocol unit tests.
*
* Covers: outcome round-trip, missing/malformed/oversized decode paths
* (oversize UnrecoverableError: deterministic dead on attempt 1, never
* silent truncation), instanceof reconstruction for the two error classes
* executeJob branches on, child-CLI invocation resolution (env override /
* binary / bun-dev fallback / fail-fast null), and killProcessGroup against
* REAL detached processes including the grandchild-death guarantee that
* motivated group signaling (SIGKILL on a wrapper pid alone orphans the
* handler; Bun rejects negative pids so the /bin/kill fallback is what
* actually runs under `bun test`, making this a real-runtime regression
* test for oven-sh/bun#15791).
*/
import { describe, expect, test } from 'bun:test';
import { spawn } from 'node:child_process';
import { mkdtempSync, readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import {
CHILD_OUTCOME_MAX_BYTES,
buildChildArgs,
decodeChildOutcomeFile,
encodeHandlerError,
killProcessGroup,
reconstructHandlerError,
resolveChildCliInvocation,
writeChildOutcomeFile,
type ChildOutcome,
} from '../src/core/minions/job-isolation.ts';
import { UnrecoverableError } from '../src/core/minions/types.ts';
import { RateLeaseUnavailableError } from '../src/core/minions/handlers/subagent.ts';
function tmpFile(name: string): { dir: string; path: string } {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-isolation-test-'));
return { dir, path: join(dir, name) };
}
describe('outcome file round-trip', () => {
test('success outcome survives write + decode; write is atomic (no .tmp left)', () => {
const { dir, path } = tmpFile('outcome.json');
try {
writeChildOutcomeFile(path, { outcome: 'success', result: { pages: 3, ok: true } });
expect(existsSync(`${path}.tmp`)).toBe(false);
const decoded = decodeChildOutcomeFile(path);
expect(decoded).toEqual({ outcome: 'success', result: { pages: 3, ok: true } });
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('missing file → generic Error naming the crash class', () => {
const { dir, path } = tmpFile('never-written.json');
try {
expect(() => decodeChildOutcomeFile(path)).toThrow(/without writing its outcome file/);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('malformed JSON → generic Error with byte count, NEVER file content', () => {
const { dir, path } = tmpFile('garbage.json');
try {
writeFileSync(path, 'sk-secret-key-do-not-leak {{{', 'utf8');
try {
decodeChildOutcomeFile(path);
throw new Error('should have thrown');
} catch (e) {
const msg = (e as Error).message;
expect(msg).toContain('not valid JSON');
expect(msg).toContain('bytes');
expect(msg).not.toContain('sk-secret');
}
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('oversize file → UnrecoverableError naming the cap (dead on attempt 1)', () => {
const { dir, path } = tmpFile('huge.json');
try {
writeFileSync(path, '{"outcome":"success","result":"xx"}', 'utf8');
try {
decodeChildOutcomeFile(path, 16); // tiny injected cap keeps the test fast
throw new Error('should have thrown');
} catch (e) {
expect(e).toBeInstanceOf(UnrecoverableError);
expect((e as Error).message).toContain('outcome cap');
}
// Default cap sanity.
expect(CHILD_OUTCOME_MAX_BYTES).toBe(32 * 1024 * 1024);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test('unrecognized shape → generic Error (byte count only)', () => {
const { dir, path } = tmpFile('weird.json');
try {
writeFileSync(path, '{"totally":"unrelated"}', 'utf8');
expect(() => decodeChildOutcomeFile(path)).toThrow(/unrecognized shape/);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
describe('handler-error encode → reconstruct (instanceof parity with inline mode)', () => {
test('UnrecoverableError survives the boundary', () => {
const enc = encodeHandlerError(new UnrecoverableError('bad config, never retry'));
expect(enc.outcome).toBe('error');
const rebuilt = reconstructHandlerError(enc as Extract<ChildOutcome, { outcome: 'error' }>);
expect(rebuilt).toBeInstanceOf(UnrecoverableError);
expect(rebuilt.message).toBe('bad config, never retry');
});
test('RateLeaseUnavailableError survives with lease fields', () => {
const enc = encodeHandlerError(new RateLeaseUnavailableError('anthropic', 4, 4));
const rebuilt = reconstructHandlerError(enc as Extract<ChildOutcome, { outcome: 'error' }>);
expect(rebuilt).toBeInstanceOf(RateLeaseUnavailableError);
const lease = rebuilt as RateLeaseUnavailableError;
expect(lease.key).toBe('anthropic');
expect(lease.active).toBe(4);
expect(lease.max).toBe(4);
});
test('generic Error carries message + child stack; unknown kinds degrade to generic', () => {
const boom = new Error('handler exploded');
const enc = encodeHandlerError(boom) as Extract<ChildOutcome, { outcome: 'error' }>;
const rebuilt = reconstructHandlerError(enc);
expect(rebuilt).toBeInstanceOf(Error);
expect(rebuilt).not.toBeInstanceOf(UnrecoverableError);
expect(rebuilt.message).toBe('handler exploded');
expect((rebuilt as Error & { childStack?: string }).childStack).toContain('handler exploded');
const weird = reconstructHandlerError({
outcome: 'error',
errorKind: 'generic',
message: 'from a hostile file',
});
expect(weird).toBeInstanceOf(Error);
expect(weird).not.toBeInstanceOf(UnrecoverableError);
});
test('non-Error throws (strings) encode without crashing', () => {
const enc = encodeHandlerError('plain string throw');
expect(enc.outcome).toBe('error');
if (enc.outcome === 'error') expect(enc.message).toBe('plain string throw');
});
});
describe('child CLI invocation resolution', () => {
test('env override wins', () => {
const inv = resolveChildCliInvocation(
{ GBRAIN_JOB_CHILD_CLI: '/opt/custom/gbrain' },
'/usr/bin/bun',
'/repo/src/cli.ts',
() => '/usr/local/bin/gbrain',
);
expect(inv).toEqual({ cmd: '/opt/custom/gbrain', argsPrefix: [] });
});
test('compiled binary next', () => {
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => '/usr/local/bin/gbrain');
expect(inv).toEqual({ cmd: '/usr/local/bin/gbrain', argsPrefix: [] });
});
test('bun-dev fallback when no binary resolves', () => {
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => null);
expect(inv).toEqual({ cmd: '/usr/bin/bun', argsPrefix: ['/repo/src/cli.ts'] });
});
test('nothing resolves → null (caller must fail fast at startup)', () => {
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/dist/other.js', () => null);
expect(inv).toBeNull();
});
test('throwing binary resolver falls through to the dev fallback', () => {
const inv = resolveChildCliInvocation({}, '/usr/bin/bun', '/repo/src/cli.ts', () => {
throw new Error('which failed');
});
expect(inv).toEqual({ cmd: '/usr/bin/bun', argsPrefix: ['/repo/src/cli.ts'] });
});
test('child argv shape', () => {
expect(buildChildArgs(42)).toEqual(['jobs', 'run-child', '--job-id', '42']);
});
});
describe('killProcessGroup (real detached processes; Bun negative-pid fallback)', () => {
async function waitFor(cond: () => boolean, ms: number): Promise<boolean> {
const deadline = Date.now() + ms;
while (Date.now() < deadline) {
if (cond()) return true;
await new Promise((r) => setTimeout(r, 25));
}
return cond();
}
function alive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
test('group SIGKILL kills the child AND its grandchild', async () => {
const { dir, path: pidFile } = tmpFile('grandchild.pid');
try {
// Child shell spawns a long-lived grandchild and records its pid.
const child = spawn('/bin/sh', ['-c', `sleep 300 & echo $! > ${pidFile}; wait`], {
detached: true,
stdio: 'ignore',
});
expect(child.pid).toBeGreaterThan(0);
const gotPid = await waitFor(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 3_000);
expect(gotPid).toBe(true);
const grandchildPid = parseInt(readFileSync(pidFile, 'utf8').trim(), 10);
expect(alive(child.pid!)).toBe(true);
expect(alive(grandchildPid)).toBe(true);
killProcessGroup(child.pid!, 'SIGKILL');
const bothDead = await waitFor(() => !alive(child.pid!) && !alive(grandchildPid), 3_000);
expect(bothDead).toBe(true);
} finally {
rmSync(dir, { recursive: true, force: true });
}
}, 15_000);
test('SIGTERM delivery to a live group returns true; dead group returns false', async () => {
const child = spawn('/bin/sh', ['-c', 'sleep 300'], { detached: true, stdio: 'ignore' });
const delivered = killProcessGroup(child.pid!, 'SIGTERM');
expect(delivered).toBe(true);
await waitFor(() => {
try {
process.kill(child.pid!, 0);
return false;
} catch {
return true;
}
}, 3_000);
// Group is gone now — a second signal reports not-delivered.
expect(killProcessGroup(child.pid!, 'SIGKILL')).toBe(false);
}, 15_000);
test('nonsense pids are refused without throwing', () => {
expect(killProcessGroup(0, 'SIGTERM')).toBe(false);
expect(killProcessGroup(1, 'SIGTERM')).toBe(false);
expect(killProcessGroup(-5, 'SIGTERM')).toBe(false);
expect(killProcessGroup(1.5 as unknown as number, 'SIGTERM')).toBe(false);
});
});
+38
View File
@@ -0,0 +1,38 @@
/**
* issue #5 `parseJobIsolationFlag` (jobs-nice-flag.test.ts pattern: pure
* parser, env injected as the 2nd param so tests never mutate process.env).
*/
import { describe, expect, test } from 'bun:test';
import { parseJobIsolationFlag } from '../src/commands/jobs.ts';
describe('parseJobIsolationFlag', () => {
test('default: inline', () => {
expect(parseJobIsolationFlag([], {})).toBe('inline');
});
test('space form', () => {
expect(parseJobIsolationFlag(['--job-isolation', 'process'], {})).toBe('process');
expect(parseJobIsolationFlag(['--job-isolation', 'inline'], {})).toBe('inline');
});
test('= form', () => {
expect(parseJobIsolationFlag(['--job-isolation=process'], {})).toBe('process');
expect(parseJobIsolationFlag(['--job-isolation=inline'], {})).toBe('inline');
});
test('env fallback, flag wins over env', () => {
expect(parseJobIsolationFlag([], { GBRAIN_JOB_ISOLATION: 'process' })).toBe('process');
expect(
parseJobIsolationFlag(['--job-isolation', 'inline'], { GBRAIN_JOB_ISOLATION: 'process' }),
).toBe('inline');
});
test('empty env value falls through to the default', () => {
expect(parseJobIsolationFlag([], { GBRAIN_JOB_ISOLATION: '' })).toBe('inline');
});
test('other flags are untouched', () => {
expect(parseJobIsolationFlag(['--queue', 'q', '--concurrency', '3'], {})).toBe('inline');
});
});
+3
View File
@@ -74,6 +74,9 @@ describe('jobs --help and jobs <subcommand> --help print real help, never the st
expect(out).toContain('--max-rss');
expect(out).toContain('--health-interval');
expect(out).toContain('GBRAIN_WORKER_CONCURRENCY');
// issue #5 process isolation: flag + env fallback documented.
expect(out).toContain('--job-isolation');
expect(out).toContain('GBRAIN_JOB_ISOLATION');
expect(out).not.toContain(STUB_MARKER);
// A real `jobs work` on this fixture would refuse (thin-client/engine
// path) or start a daemon; either output would differ from the help.
@@ -0,0 +1,71 @@
/**
* issue #6 `MinionQueue.renewLock` forwards its optional AbortSignal to
* `engine.executeRawDirect` so the lock-renewal tick's timeout race can
* CANCEL a hung UPDATE (postgres.js `.cancel()`) instead of abandoning it on
* a checked-out pool slot.
*
* Hermetic: stub engine object literal (`as unknown as BrainEngine`), no DB
* the worker-conn-resilience-1720 pattern.
*/
import { describe, expect, test } from 'bun:test';
import { MinionQueue } from '../src/core/minions/queue.ts';
import type { BrainEngine } from '../src/core/engine.ts';
function makeCaptureEngine() {
const calls: Array<{
sql: string;
params: unknown[] | undefined;
opts: { signal?: AbortSignal } | undefined;
}> = [];
const engine = {
kind: 'postgres',
executeRawDirect: async (
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
) => {
calls.push({ sql, params, opts });
return [{ id: 1 }];
},
} as unknown as BrainEngine;
return { engine, calls };
}
describe('MinionQueue.renewLock signal forwarding (issue #6)', () => {
test('forwards opts.signal to executeRawDirect', async () => {
const { engine, calls } = makeCaptureEngine();
const queue = new MinionQueue(engine);
const ac = new AbortController();
const ok = await queue.renewLock(7, 'tok-xyz', 30_000, { signal: ac.signal });
expect(ok).toBe(true);
expect(calls.length).toBe(1);
expect(calls[0].sql).toContain('UPDATE minion_jobs SET lock_until');
expect(calls[0].params).toEqual([30_000, 7, 'tok-xyz']);
expect(calls[0].opts?.signal).toBe(ac.signal);
});
test('legacy 3-arg call still works (opts undefined)', async () => {
const { engine, calls } = makeCaptureEngine();
const queue = new MinionQueue(engine);
const ok = await queue.renewLock(7, 'tok-xyz', 30_000);
expect(ok).toBe(true);
expect(calls[0].opts).toBeUndefined();
});
test('token-fence miss returns false regardless of signal', async () => {
const engine = {
kind: 'postgres',
executeRawDirect: async () => [],
} as unknown as BrainEngine;
const queue = new MinionQueue(engine);
const ac = new AbortController();
const ok = await queue.renewLock(7, 'tok-stale', 30_000, { signal: ac.signal });
expect(ok).toBe(false);
});
});
+182
View File
@@ -0,0 +1,182 @@
/**
* issue #6 CheckoutGauge (pure) + the PostgresEngine gauge seams.
*
* The leak guard matters most: a counter that isn't released on a THROWING
* query drifts upward forever and turns the diagnostics into fiction. The
* engine seams use the Object.create(PostgresEngine.prototype) + fake-pool
* pattern (no real DB).
*/
import { describe, expect, test } from 'bun:test';
import { CheckoutGauge } from '../src/core/pool-gauge.ts';
import { PostgresEngine } from '../src/core/postgres-engine.ts';
describe('CheckoutGauge (pure)', () => {
test('acquire/release round-trip per kind', () => {
const g = new CheckoutGauge();
g.acquire('raw');
g.acquire('raw');
g.acquire('direct');
g.acquire('reserved');
g.acquire('tx');
expect(g.snapshot()).toEqual({ raw: 2, direct: 1, reserved: 1, tx: 1 });
g.release('raw');
g.release('direct');
g.release('reserved');
g.release('tx');
expect(g.snapshot()).toEqual({ raw: 1, direct: 0, reserved: 0, tx: 0 });
});
test('release clamps at zero (a missed acquire never underflows)', () => {
const g = new CheckoutGauge();
g.release('raw');
g.release('tx');
expect(g.snapshot()).toEqual({ raw: 0, direct: 0, reserved: 0, tx: 0 });
});
test('snapshot is a copy, not a live reference', () => {
const g = new CheckoutGauge();
const snap = g.snapshot();
g.acquire('raw');
expect(snap.raw).toBe(0);
});
});
// --- engine seams (fake pool, no DB) ---------------------------------------
interface FakePoolBehavior {
/** What conn.unsafe returns per call. */
unsafe: (sql: string, params?: unknown[]) => unknown;
}
function makeEngine(behavior: FakePoolBehavior): {
engine: PostgresEngine;
diagnostics: () => { tracked: Record<string, number>; poolMax: number | null } | null;
} {
const fakePool = {
unsafe: behavior.unsafe,
options: { max: 10 },
};
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
Object.defineProperty(engine, 'sql', { get: () => fakePool });
Object.defineProperty(engine, '_sql', { value: fakePool, writable: true });
// Fresh gauge per fake engine (the real field initializer doesn't run for
// Object.create instances).
Object.defineProperty(engine, 'checkoutGauge', {
value: new CheckoutGauge(),
writable: true,
});
return {
engine,
diagnostics: () =>
(engine as unknown as {
getPoolDiagnostics: () => { tracked: Record<string, number>; poolMax: number | null } | null;
}).getPoolDiagnostics(),
};
}
describe('PostgresEngine gauge seams (fake pool)', () => {
test('executeRaw: counted while in flight, released on resolve', async () => {
let resolveQuery: (rows: unknown[]) => void = () => {};
const { engine, diagnostics } = makeEngine({
unsafe: () => new Promise((r) => { resolveQuery = r; }),
});
const p = engine.executeRaw('SELECT 42');
expect(diagnostics()?.tracked.raw).toBe(1);
expect(diagnostics()?.poolMax).toBe(10);
resolveQuery([]);
await p;
expect(diagnostics()?.tracked.raw).toBe(0);
});
test('executeRaw: released on REJECTED query (leak guard)', async () => {
const { engine, diagnostics } = makeEngine({
unsafe: () => Promise.reject(new Error('query was cancelled')),
});
await expect(engine.executeRaw('SELECT 42')).rejects.toThrow('query was cancelled');
expect(diagnostics()?.tracked.raw).toBe(0);
});
test('executeRaw: released on SYNCHRONOUS throw (pre-aborted signal)', async () => {
const { engine, diagnostics } = makeEngine({
unsafe: () => Promise.resolve([]),
});
const ac = new AbortController();
ac.abort();
await expect(
engine.executeRaw('SELECT 42', undefined, { signal: ac.signal }),
).rejects.toThrow();
expect(diagnostics()?.tracked.raw).toBe(0);
});
test('getPoolDiagnostics is fail-open (no pool → null, no throw)', () => {
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
// No sql defined — the getter on the prototype will throw internally.
const diag = (engine as unknown as {
getPoolDiagnostics: () => unknown;
}).getPoolDiagnostics();
expect(diag).toBeNull();
});
});
describe('PostgresEngine gauge seams: direct + tx (coverage-audit gaps)', () => {
test('executeRawDirect: counted as direct while in flight, released on resolve and reject', async () => {
let resolveQuery: (rows: unknown[]) => void = () => {};
const { engine, diagnostics } = makeEngine({
unsafe: () => new Promise((r) => { resolveQuery = r; }),
});
// No connectionManager on the fake -> executeRawDirect falls through to this.sql.
const p = engine.executeRawDirect('SELECT 1');
expect(diagnostics()?.tracked.direct).toBe(1);
resolveQuery([]);
await p;
expect(diagnostics()?.tracked.direct).toBe(0);
const { engine: rejEngine, diagnostics: rejDiag } = makeEngine({
unsafe: () => Promise.reject(new Error('boom')),
});
await expect(rejEngine.executeRawDirect('SELECT 1')).rejects.toThrow('boom');
expect(rejDiag()?.tracked.direct).toBe(0);
});
test('transaction: tx counter released on SYNCHRONOUS begin() throw (nested-tx clone class)', async () => {
const { engine, diagnostics } = makeEngine({ unsafe: () => Promise.resolve([]) });
// Fake pool has no .begin -> conn.begin(...) throws synchronously, the
// exact shape a nested transaction on a tx clone produces.
await expect(
(engine as unknown as { transaction: (fn: unknown) => Promise<unknown> }).transaction(async () => 'x'),
).rejects.toThrow();
expect(diagnostics()?.tracked.tx).toBe(0);
});
test('withReservedConnection: ddl() throw falls back to the READ pool and releases the permit', async () => {
const log: string[] = [];
const readPool = {
unsafe: async () => [],
options: { max: 10 },
reserve: async () => {
log.push('reserve:read');
return { unsafe: async () => [], release: () => log.push('release:read') };
},
};
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
Object.defineProperty(engine, 'sql', { get: () => readPool });
Object.defineProperty(engine, '_sql', { value: readPool, writable: true });
Object.defineProperty(engine, 'checkoutGauge', { value: new CheckoutGauge(), writable: true });
Object.defineProperty(engine, '_reservedDirectInFlight', { value: 0, writable: true });
Object.defineProperty(engine, 'connectionManager', {
value: {
peekReadPool: () => readPool,
isDualPoolActive: () => true,
describeMode: () => ({ direct_pool_size: 3 }),
ddl: async () => { throw new Error('EMAXCONNSESSION'); },
},
writable: true,
});
const out = await engine.withReservedConnection(async () => 'ok');
expect(out).toBe('ok');
expect(log).toEqual(['reserve:read', 'release:read']); // fell back, did not fail the caller
expect((engine as unknown as { _reservedDirectInFlight: number })._reservedDirectInFlight).toBe(0);
});
});
@@ -0,0 +1,158 @@
/**
* issue #6 `withReservedConnection` routing:
*
* dual-pool active + permit available reserve from the DIRECT pool
* (30-min statement_timeout lane; stops long DDL/backfill holds from
* pinning the worker's shared read pool)
* semaphore exhausted (directPoolSize-1 in flight) READ pool (status quo;
* >= 1 direct slot always stays free for claim/renewLock heartbeats)
* kill-switched / single-pool READ pool (status quo)
* inside an open transaction never rerouted
* throw inside fn / reserve failure semaphore released (leak guard)
*
* Hermetic: Object.create(PostgresEngine.prototype) + recording fake pools.
*/
import { describe, expect, test } from 'bun:test';
import { PostgresEngine } from '../src/core/postgres-engine.ts';
import { CheckoutGauge } from '../src/core/pool-gauge.ts';
interface FakeReserved {
unsafe: (q: string, params?: unknown[]) => Promise<unknown[]>;
release: () => void;
}
function makeFakePool(label: string, log: string[]) {
return {
label,
unsafe: async () => [],
options: { max: 10 },
reserve: async (): Promise<FakeReserved> => {
log.push(`reserve:${label}`);
return {
unsafe: async () => [],
release: () => log.push(`release:${label}`),
};
},
};
}
function makeEngine(opts: {
dualPool: boolean;
directPoolSize?: number;
inTransaction?: boolean;
log: string[];
}) {
const readPool = makeFakePool('read', opts.log);
const directPool = makeFakePool('direct', opts.log);
const engine = Object.create(PostgresEngine.prototype) as PostgresEngine;
Object.defineProperty(engine, 'sql', { get: () => readPool });
// inTransaction detection: _sql !== null && peekReadPool() !== _sql.
// Simulate "in transaction" by making _sql differ from peekReadPool().
Object.defineProperty(engine, '_sql', {
value: opts.inTransaction ? { txMarker: true } : readPool,
writable: true,
});
Object.defineProperty(engine, 'checkoutGauge', { value: new CheckoutGauge(), writable: true });
Object.defineProperty(engine, '_reservedDirectInFlight', { value: 0, writable: true });
Object.defineProperty(engine, 'connectionManager', {
value: {
peekReadPool: () => readPool,
isDualPoolActive: () => opts.dualPool,
describeMode: () => ({ direct_pool_size: opts.directPoolSize ?? 3 }),
ddl: async () => directPool,
},
writable: true,
});
return engine;
}
describe('withReservedConnection routing (issue #6)', () => {
test('dual-pool active: reserves from the DIRECT pool', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: true, log });
await engine.withReservedConnection(async () => 'ok');
expect(log).toEqual(['reserve:direct', 'release:direct']);
});
test('single-pool (kill-switched): reserves from the READ pool (status quo)', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: false, log });
await engine.withReservedConnection(async () => 'ok');
expect(log).toEqual(['reserve:read', 'release:read']);
});
test('inside a transaction: never rerouted', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: true, inTransaction: true, log });
await engine.withReservedConnection(async () => 'ok');
expect(log).toEqual(['reserve:read', 'release:read']);
});
test('semaphore: directPoolSize-1 concurrent direct reserves; overflow -> read pool', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: true, directPoolSize: 3, log });
let releaseFirst: () => void = () => {};
let releaseSecond: () => void = () => {};
const first = engine.withReservedConnection(
() => new Promise((r) => { releaseFirst = () => r('one'); }),
);
const second = engine.withReservedConnection(
() => new Promise((r) => { releaseSecond = () => r('two'); }),
);
// Let both reserves happen.
await new Promise((r) => setTimeout(r, 10));
expect(log.filter((l) => l === 'reserve:direct').length).toBe(2); // cap = 3-1 = 2
// Third concurrent reserve overflows to the read pool (status quo, never blocks).
const third = await engine.withReservedConnection(async () => 'three');
expect(third).toBe('three');
expect(log).toContain('reserve:read');
releaseFirst();
releaseSecond();
await Promise.all([first, second]);
// Permits released: the next reserve goes direct again.
log.length = 0;
await engine.withReservedConnection(async () => 'four');
expect(log).toEqual(['reserve:direct', 'release:direct']);
});
test('direct_pool_size=1: NEVER reserves from the direct pool (a reserve would starve ALL heartbeats)', async () => {
// Red-team finding: a Math.max(1, size-1) floor made cap=1 at size=1,
// letting a multi-minute reserve consume the ONLY direct session that
// claim/renewLock heartbeats depend on — the #6 class reintroduced.
const log: string[] = [];
const engine = makeEngine({ dualPool: true, directPoolSize: 1, log });
await engine.withReservedConnection(async () => 'ok');
expect(log).toEqual(['reserve:read', 'release:read']);
});
test('fn throw releases the semaphore permit (leak guard)', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: true, directPoolSize: 2, log }); // cap = 1
await expect(
engine.withReservedConnection(async () => { throw new Error('DDL boom'); }),
).rejects.toThrow('DDL boom');
// Permit released: next reserve still goes direct (cap would block if leaked).
log.length = 0;
await engine.withReservedConnection(async () => 'ok');
expect(log).toEqual(['reserve:direct', 'release:direct']);
});
test('reserve() failure releases the permit and rethrows', async () => {
const log: string[] = [];
const engine = makeEngine({ dualPool: true, directPoolSize: 2, log });
const cm = (engine as unknown as { connectionManager: { ddl: () => Promise<unknown> } }).connectionManager;
cm.ddl = async () => ({
reserve: async () => { throw new Error('EMAXCONNSESSION'); },
});
await expect(engine.withReservedConnection(async () => 'x')).rejects.toThrow('EMAXCONNSESSION');
// Permit released despite the failure.
expect(
(engine as unknown as { _reservedDirectInFlight: number })._reservedDirectInFlight,
).toBe(0);
});
});
+103
View File
@@ -0,0 +1,103 @@
/**
* issue #6 timed-out probes are CANCELLED, not abandoned.
*
* Closes the TODOS entry "cancel (not just abandon) timed-out submit-time
* queue probes": `probeQueueState` time-bounds via Promise.race, but the
* losing query used to keep running on the pool after the race resolved
* under pool exhaustion (the exact regime the probe exists to detect) the
* abandoned query held a slot and made the exhaustion worse. The timeout now
* aborts a per-probe AbortSignal threaded through `queryWedgeSignals` into
* `engine.executeRaw`, and `withRefreshingLock`'s heartbeat does the same for
* `handle.refresh`.
*
* Hermetic: stub engines (`as unknown as BrainEngine`), no DB.
*/
import { describe, expect, test } from 'bun:test';
import { probeQueueState, queryWedgeSignals } from '../src/core/minions/supervisor.ts';
import type { BrainEngine } from '../src/core/engine.ts';
const WEDGE_ROW = {
stalled: '0',
active_healthy: '1',
waiting: '2',
waiting_claimable: '2',
last_completed: null,
last_completed_claimable: null,
};
describe('queryWedgeSignals signal threading (issue #6)', () => {
test('forwards opts.signal to executeRaw', async () => {
let seenOpts: { signal?: AbortSignal } | undefined;
const engine = {
kind: 'postgres',
executeRaw: async (_sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
seenOpts = opts;
return [WEDGE_ROW];
},
} as unknown as BrainEngine;
const ac = new AbortController();
const sig = await queryWedgeSignals(engine, 'default', ['sync'], { signal: ac.signal });
expect(sig.waiting).toBe(2);
expect(seenOpts?.signal).toBe(ac.signal);
});
});
describe('probeQueueState cancellation (issue #6)', () => {
test('timeout aborts the per-probe signal so the losing query is cancelled', async () => {
let seenSignal: AbortSignal | undefined;
const engine = {
kind: 'postgres',
executeRaw: (_sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
seenSignal = opts?.signal;
// Hang forever — the abandoned-racer scenario.
return new Promise<never>(() => {});
},
} as unknown as BrainEngine;
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 50 });
expect(state.probe_failed).toBe(true);
expect(seenSignal).toBeInstanceOf(AbortSignal);
expect(seenSignal!.aborted).toBe(true);
});
test('fast probe: signal delivered but never aborted; result returned', async () => {
const seenSignals: Array<AbortSignal | undefined> = [];
const engine = {
kind: 'postgres',
executeRaw: async (sql: string, _params?: unknown[], opts?: { signal?: AbortSignal }) => {
seenSignals.push(opts?.signal);
if (sql.includes('EXTRACT(EPOCH')) return [{ age: 12 }];
return [WEDGE_ROW];
},
} as unknown as BrainEngine;
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 5_000 });
expect(state.probe_failed).toBeUndefined();
expect(state.depth).toBe(2);
// The wedge + age queries carry the probe signal. Other engine calls made
// by the probe (e.g. inspectLock) pass no opts — filter to the signals we
// threaded and assert none were aborted on the fast path.
const threaded = seenSignals.filter((s): s is AbortSignal => s instanceof AbortSignal);
expect(threaded.length).toBeGreaterThanOrEqual(2);
for (const s of threaded) {
expect(s.aborted).toBe(false);
}
});
test('probe throw still collapses to {probe_failed: true} (fail-open contract)', async () => {
const engine = {
kind: 'postgres',
executeRaw: async () => {
throw new Error('query was cancelled');
},
} as unknown as BrainEngine;
const state = await probeQueueState(engine, 'default', ['sync'], { timeoutMs: 5_000 });
expect(state.probe_failed).toBe(true);
});
});
+249
View File
@@ -0,0 +1,249 @@
/**
* issue #5 `runChildJobEntry` (the `jobs run-child` core) against a real
* in-memory PGLite engine with a REAL claim-minted lock token. (PGLite is
* rejected at the CLI layer for actual isolation runs Postgres-only but
* the core function is engine-agnostic, which is exactly what makes it
* testable without a DATABASE_URL.)
*
* Paths pinned:
* - success: handler result lands in the outcome file, exit 0
* - handler failure: encoded error outcome, STILL exit 0 (a reported
* failure is a successful report)
* - token mismatch / reclaimed job: exit 14, handler NEVER runs
* - missing handler: generic error outcome, exit 0
* - SIGTERM semantics via direct signal wiring (ctx.signal + shutdown)
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionQueue } from '../src/core/minions/queue.ts';
import { runChildJobEntry } from '../src/core/minions/run-child.ts';
import { decodeChildOutcomeFile } from '../src/core/minions/job-isolation.ts';
import {
JOB_CHILD_EXIT_NOT_CLAIMED,
} from '../src/core/minions/worker-exit-codes.ts';
import { UnrecoverableError, type MinionHandler } from '../src/core/minions/types.ts';
let engine: PGLiteEngine;
let queue: MinionQueue;
let dir: string;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' }); // in-memory
await engine.initSchema();
queue = new MinionQueue(engine);
dir = mkdtempSync(join(tmpdir(), 'gbrain-run-child-test-'));
});
afterAll(async () => {
await engine.disconnect();
rmSync(dir, { recursive: true, force: true });
});
beforeEach(async () => {
await engine.executeRaw('DELETE FROM minion_jobs');
});
/** Claim a freshly-added job the same way the parent worker does. */
async function addAndClaim(name: string, data: Record<string, unknown> = {}) {
await queue.add(name, data);
const job = await queue.claim('parent-tok-1', 30_000, 'default', [name]);
if (!job) throw new Error('claim failed in test setup');
return job;
}
function handlers(map: Record<string, MinionHandler>) {
return { resolveHandler: (n: string) => map[n] };
}
describe('runChildJobEntry', () => {
test('success: outcome file carries the handler result; exit 0; ctx wired to the job row', async () => {
const job = await addAndClaim('sync', { full: true });
const resultPath = join(dir, `ok-${job.id}.json`);
let sawData: unknown;
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
handlers({
sync: async (ctx) => {
sawData = ctx.data;
await ctx.updateProgress({ step: 'half' });
return { pages: 7 };
},
}),
);
expect(code).toBe(0);
expect(sawData).toEqual({ full: true });
const outcome = decodeChildOutcomeFile(resultPath);
expect(outcome).toEqual({ outcome: 'success', result: { pages: 7 } });
// The token-fenced progress write really landed.
const rows = await engine.executeRaw<{ progress: unknown }>(
'SELECT progress FROM minion_jobs WHERE id = $1',
[job.id],
);
const progress = rows[0]?.progress;
expect(typeof progress === 'string' ? JSON.parse(progress) : progress).toEqual({ step: 'half' });
});
test('handler failure: encoded error outcome, exit 0 (reported failure = successful report)', async () => {
const job = await addAndClaim('sync');
const resultPath = join(dir, `err-${job.id}.json`);
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
handlers({
sync: async () => {
throw new UnrecoverableError('schema mismatch, never retry');
},
}),
);
expect(code).toBe(0);
const outcome = decodeChildOutcomeFile(resultPath);
expect(outcome.outcome).toBe('error');
if (outcome.outcome === 'error') {
expect(outcome.errorKind).toBe('unrecoverable');
expect(outcome.message).toBe('schema mismatch, never retry');
}
});
test('token mismatch (job reclaimed): exit 14, handler never runs', async () => {
const job = await addAndClaim('sync');
const resultPath = join(dir, `mismatch-${job.id}.json`);
let handlerRan = false;
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'STALE-token', resultPath, parentPid: 0 },
handlers({
sync: async () => {
handlerRan = true;
return null;
},
}),
);
expect(code).toBe(JOB_CHILD_EXIT_NOT_CLAIMED);
expect(handlerRan).toBe(false);
expect(() => decodeChildOutcomeFile(resultPath)).toThrow(); // nothing written
});
test('missing job id: exit 14', async () => {
const code = await runChildJobEntry(
engine,
{ jobId: 999_999, lockToken: 'parent-tok-1', resultPath: join(dir, 'none.json'), parentPid: 0 },
handlers({}),
);
expect(code).toBe(JOB_CHILD_EXIT_NOT_CLAIMED);
});
test('missing handler: UNRECOVERABLE error outcome (inline parity: dead on attempt 1), exit 0', async () => {
const job = await addAndClaim('exotic-plugin-job');
const resultPath = join(dir, `nohandler-${job.id}.json`);
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
handlers({}),
);
expect(code).toBe(0);
const outcome = decodeChildOutcomeFile(resultPath);
expect(outcome.outcome).toBe('error');
if (outcome.outcome === 'error') {
expect(outcome.errorKind).toBe('unrecoverable');
expect(outcome.message).toContain("No handler for job type 'exotic-plugin-job'");
}
});
test('SIGTERM aborts ONLY shutdownSignal — ctx.signal stays live (inline signal-separation parity)', async () => {
const job = await addAndClaim('sync');
const resultPath = join(dir, `sigterm-${job.id}.json`);
let ctxSignalAbortedAtShutdown: boolean | null = null;
const entry = runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
handlers({
sync: (ctx) =>
new Promise((resolve) => {
ctx.shutdownSignal.addEventListener('abort', () => {
// The whole point: a cooperative handler gets the drain window
// with its per-job signal STILL LIVE, finishes, and reports.
ctxSignalAbortedAtShutdown = ctx.signal.aborted;
resolve('finished-during-drain');
});
}),
}),
);
await new Promise((r) => setTimeout(r, 100));
// Trigger the entry's process.on('SIGTERM') handler in-process without
// sending a real signal to the test runner.
(process as unknown as { emit: (event: string) => boolean }).emit('SIGTERM');
const code = await entry;
expect(code).toBe(0);
// TS control-flow can't see the closure write; compare explicitly.
expect(ctxSignalAbortedAtShutdown === false).toBe(true);
const outcome = decodeChildOutcomeFile(resultPath);
expect(outcome).toEqual({ outcome: 'success', result: { value: 'finished-during-drain' } });
});
test('primitive results are wrapped {value: x} CHILD-side (completeJob shape parity across the JSON boundary)', async () => {
const job = await addAndClaim('sync');
const resultPath = join(dir, `wrap-${job.id}.json`);
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
handlers({ sync: async () => 42 }),
);
expect(code).toBe(0);
expect(decodeChildOutcomeFile(resultPath)).toEqual({ outcome: 'success', result: { value: 42 } });
});
test('parent-death watchdog: aborts the handler and schedules the hard exit', async () => {
const job = await addAndClaim('sync');
const resultPath = join(dir, `orphan-${job.id}.json`);
let sawAbort = false;
const hardExits: number[] = [];
// A GUARANTEED-dead parent pid: spawn a real short-lived process, wait
// for it to exit, and use its reaped pid. (A magic high pid is
// allocatable on Linux — default kernel.pid_max is 4,194,304 — so a real
// process could hold it and hang the test; testing specialist.)
const { spawnSync: spawnDead } = await import('node:child_process');
const deadPid = spawnDead('/bin/sh', ['-c', 'exit 0']).pid ?? 0;
expect(deadPid).toBeGreaterThan(0);
const code = await runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: deadPid },
{
resolveHandler: () => async (ctx) =>
new Promise((resolve) => {
ctx.signal.addEventListener('abort', () => {
sawAbort = true;
resolve('aborted-cleanly');
});
}),
parentPollMs: 30,
orphanGraceMs: 60_000, // never fires in-test; we capture the schedule
hardExit: (c) => { hardExits.push(c); },
},
);
expect(sawAbort).toBe(true);
expect(code).toBe(0); // the handler resolved after abort; outcome written
const outcome = decodeChildOutcomeFile(resultPath);
expect(outcome.outcome).toBe('success');
expect(hardExits.length).toBe(0); // grace not reached — clean wind-down
});
});
+14
View File
@@ -36,4 +36,18 @@ describe('buildWorkerArgs', () => {
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0 }))
.not.toContain('--nice');
});
// issue #5 — conditional pass-through: inline/omitted keeps existing
// deployments' argv byte-identical (the pinned arrays above never change).
test('appends --job-isolation process when set', () => {
expect(buildWorkerArgs({ concurrency: 2, queue: 'default', maxRssMb: 0, jobIsolation: 'process' }))
.toEqual(['jobs', 'work', '--concurrency', '2', '--queue', 'default', '--job-isolation', 'process']);
});
test('omits --job-isolation when inline or undefined', () => {
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0, jobIsolation: 'inline' }))
.not.toContain('--job-isolation');
expect(buildWorkerArgs({ concurrency: 1, queue: 'q', maxRssMb: 0 }))
.not.toContain('--job-isolation');
});
});
+214
View File
@@ -0,0 +1,214 @@
/**
* issue #5 full parent path with isolation on: claim spawn child
* decode outcome completeJob / failJob / release. Real in-memory PGLite
* worker + the fake-run-child.mjs fixture standing in for the compiled CLI
* (reads the isolation env contract, writes canned outcomes).
*
* Pins:
* - success: job completes with the CHILD's result (fenced completeJob)
* - error outcome: failJob path attempt burned, delayed/dead per policy
* - crash (exit 1, no file): attempt burned
* - spawn failure (bad child CLI): RELEASED status stays 'active',
* attempts NOT burned (infra class; stall sweeper would requeue)
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { join } from 'node:path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionQueue } from '../src/core/minions/queue.ts';
import { MinionWorker } from '../src/core/minions/worker.ts';
import { withEnv } from './helpers/with-env.ts';
const FIXTURE = join(import.meta.dir, 'fixtures', 'fake-run-child.mjs');
let engine: PGLiteEngine;
let queue: MinionQueue;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
queue = new MinionQueue(engine);
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw('DELETE FROM minion_jobs');
});
function makeWorker(invocationCmd = process.execPath, argsPrefix = [FIXTURE]) {
const worker = new MinionWorker(engine, {
queue: 'default',
concurrency: 1,
pollInterval: 25,
healthCheckInterval: 0,
maxRssMb: 0,
jobIsolation: 'process',
childCliInvocation: { cmd: invocationCmd, argsPrefix },
});
// Handler must exist in the parent registry (name-scoped claiming); its
// body never runs in isolation mode.
worker.register('isotest', async () => {
throw new Error('parent-side handler must not run when isolated');
});
return worker;
}
async function runWorkerUntil(
worker: MinionWorker,
done: () => Promise<boolean>,
timeoutMs = 10_000,
): Promise<void> {
const run = worker.start();
const deadline = Date.now() + timeoutMs;
try {
while (Date.now() < deadline) {
if (await done()) return;
await new Promise((r) => setTimeout(r, 50));
}
throw new Error('runWorkerUntil timed out');
} finally {
worker.stop();
await run;
}
}
async function jobRow(id: number): Promise<{ status: string; attempts_made: number; result: unknown; error_text: string | null }> {
const rows = await engine.executeRaw<{
status: string;
attempts_made: number;
result: unknown;
error_text: string | null;
}>('SELECT status, attempts_made, result, error_text FROM minion_jobs WHERE id = $1', [id]);
if (!rows[0]) throw new Error('job row missing');
return rows[0];
}
describe('worker with jobIsolation=process (PGLite + fake child)', () => {
test("construction throws for jobIsolation 'process' without childCliInvocation (predicate-mismatch guard)", () => {
// Red-team finding: 'process' without an invocation silently executed
// handlers INLINE while the evict path believed it was isolated.
expect(
() => new MinionWorker(engine, { queue: 'default', jobIsolation: 'process' }),
).toThrow(/childCliInvocation/);
});
test('spawn-failure circuit breaker: 3 consecutive bootstrap failures emit unhealthy(child_spawn_failing)', async () => {
// Red-team finding: a deterministic child-bootstrap failure looped
// claim/release forever, invisible to the stall detector (every settle
// refreshes the progress clock).
const j1 = await queue.add('isotest', {});
const j2 = await queue.add('isotest', {});
const j3 = await queue.add('isotest', {});
const worker = makeWorker('/nonexistent/gbrain-binary', []);
const unhealthy: unknown[] = [];
worker.on('unhealthy', (i) => unhealthy.push(i));
const run = worker.start();
const deadline = Date.now() + 10_000;
try {
while (Date.now() < deadline && unhealthy.length === 0) {
await new Promise((r) => setTimeout(r, 50));
}
} finally {
worker.stop();
await run;
}
expect(unhealthy.length).toBeGreaterThan(0);
const info = unhealthy[0] as { reason: string; consecutiveFailures: number };
expect(info.reason).toBe('child_spawn_failing');
expect(info.consecutiveFailures).toBeGreaterThanOrEqual(3);
// No attempts burned anywhere — all three rows released, not failed.
for (const j of [j1, j2, j3]) {
const row = await jobRow(j.id);
expect(row.status).toBe('active');
expect(row.attempts_made).toBe(0);
}
}, 20_000);
test('success: child result lands via the fenced completeJob', async () => {
await withEnv({ FAKE_RUN_CHILD_MODE: 'success' }, async () => {
const job = await queue.add('isotest', { prompt: 'hi' });
const worker = makeWorker();
await runWorkerUntil(worker, async () => (await jobRow(job.id)).status === 'completed');
const row = await jobRow(job.id);
expect(row.status).toBe('completed');
const result = typeof row.result === 'string' ? JSON.parse(row.result) : row.result;
expect((result as { fromChild?: boolean }).fromChild).toBe(true);
// The child got the REAL claim token through the env contract.
expect((result as { token?: string }).token).toMatch(/^.+:.+$/);
});
}, 20_000);
test('error outcome: failJob path, attempt burned', async () => {
await withEnv({ FAKE_RUN_CHILD_MODE: 'error' }, async () => {
const job = await queue.add('isotest', {}, { max_attempts: 1 });
const worker = makeWorker();
await runWorkerUntil(worker, async () => {
const s = (await jobRow(job.id)).status;
return s === 'dead' || s === 'failed';
});
const row = await jobRow(job.id);
expect(row.status).toBe('dead'); // maxAttempts 1 → attempt burned → dead
expect(row.error_text).toContain('fake child handler failure');
});
}, 20_000);
test('crash (exit 1, no outcome file): attempt burned', async () => {
await withEnv({ FAKE_RUN_CHILD_MODE: 'crash' }, async () => {
const job = await queue.add('isotest', {}, { max_attempts: 1 });
const worker = makeWorker();
await runWorkerUntil(worker, async () => {
const s = (await jobRow(job.id)).status;
return s === 'dead' || s === 'failed';
});
const row = await jobRow(job.id);
expect(row.status).toBe('dead');
expect(row.error_text).toContain('exit code=1');
});
}, 20_000);
test('serialization parity (codex-2 #8): unreportable results fail in BOTH modes, never complete', async () => {
// Inline: a circular result blows up in completeJob's serialization →
// failJob (attempt burned).
const inlineWorker = new MinionWorker(engine, {
queue: 'default', concurrency: 1, pollInterval: 25, healthCheckInterval: 0, maxRssMb: 0,
});
inlineWorker.register('isotest', async () => {
const a: Record<string, unknown> = {};
a.self = a; // circular — not JSONB-serializable
return a;
});
const j1 = await queue.add('isotest', {}, { max_attempts: 1 });
await runWorkerUntil(inlineWorker, async () => {
const s = (await jobRow(j1.id)).status;
return s !== 'waiting' && s !== 'active';
});
expect((await jobRow(j1.id)).status).not.toBe('completed');
// Process mode: a child that cannot persist its outcome (exit 15) lands
// in the same terminal class — failed loudly, never falsely completed.
await withEnv({ FAKE_RUN_CHILD_MODE: 'exit15' }, async () => {
const j2 = await queue.add('isotest', {}, { max_attempts: 1 });
const worker = makeWorker();
await runWorkerUntil(worker, async () => (await jobRow(j2.id)).status === 'dead');
expect((await jobRow(j2.id)).error_text).toContain('outcome file');
});
}, 30_000);
test('spawn failure: RELEASED — still active, attempts NOT burned (infra class)', async () => {
const job = await queue.add('isotest', {});
const worker = makeWorker('/nonexistent/gbrain-binary', []);
// The claim happens, the spawn fails, the job is released (stays
// 'active' until lock expiry — the stall sweeper's requeue territory).
const run = worker.start();
await new Promise((r) => setTimeout(r, 1_500));
worker.stop();
await run;
const row = await jobRow(job.id);
expect(row.status).toBe('active'); // NOT dead, NOT failed
expect(row.attempts_made).toBe(0); // release = no failJob = no attempt burned
}, 20_000);
});
+47
View File
@@ -569,3 +569,50 @@ describe('runLockRenewalTick: reconnect-once dep (issue #1678)', () => {
expect(reconnectCalls).toBe(0); // pointless to reconnect when we're giving up the lock
});
});
describe('runLockRenewalTick: per-call cancellation signal (issue #6)', () => {
test('renewLock receives a live AbortSignal; not aborted on the happy path', async () => {
const audit = freshAudit();
const timer = makeFakeTimer();
let seenSignal: AbortSignal | undefined;
const deps: LockRenewalDeps = {
renewLock: async (_id, _tok, _dur, opts) => {
seenSignal = opts?.signal;
return true;
},
audit: audit.sink,
now: () => 1000,
setTimeout: timer.setTimeout,
};
const result = await runLockRenewalTick(deps, makeState());
expect(result.kind).toBe('ok');
expect(seenSignal).toBeInstanceOf(AbortSignal);
expect(seenSignal!.aborted).toBe(false);
});
test('hung renewLock: the timeout aborts the per-call signal so the query is cancelled, not orphaned', async () => {
const audit = freshAudit();
const timer = makeFakeTimer();
let seenSignal: AbortSignal | undefined;
const deps: LockRenewalDeps = {
renewLock: (_id, _tok, _dur, opts) => {
seenSignal = opts?.signal;
return new Promise<boolean>(() => { /* hangs forever — abandoned racer */ });
},
audit: audit.sink,
now: () => 1000,
setTimeout: timer.setTimeout,
};
// Renewed recently: the timeout counts as failure #1, not a give-up.
const state = makeState({ lastSuccessfulRenewalAt: 1000 });
const tick = runLockRenewalTick(deps, state);
// The race installs the timeout synchronously; firing it must abort the
// per-call signal (this is what releases the checked-out pool slot).
timer.runAll();
const result = await tick;
expect(result.kind).toBe('ok'); // counter bumped, deadline not yet crossed
expect(state.consecutiveFailures).toBe(1);
expect(seenSignal).toBeInstanceOf(AbortSignal);
expect(seenSignal!.aborted).toBe(true);
});
});
+78 -2
View File
@@ -35,11 +35,19 @@ afterAll(async () => {
/** Make the DB-liveness probe (`SELECT 1`) throw; delegate every other query.
* Returns a restore fn that removes the instance override (falls back to the
* prototype method). */
* prototype method). Also records the probe's `opts` so tests can pin that
* the worker passes a cancellation signal (issue #6: a hung probe must be
* cancelled, not abandoned on a checked-out pool slot). */
const probeOptsSeen: Array<{ signal?: AbortSignal } | undefined> = [];
function breakLivenessProbe(eng: PGLiteEngine): () => void {
const real = eng.executeRaw.bind(eng);
(eng as { executeRaw: unknown }).executeRaw = async (sql: string, params?: unknown[]) => {
(eng as { executeRaw: unknown }).executeRaw = async (
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
) => {
if (typeof sql === 'string' && sql.trim() === 'SELECT 1') {
probeOptsSeen.push(opts);
throw new Error('probe boom (simulated dead pool)');
}
return real(sql, params as never);
@@ -95,6 +103,74 @@ describe('issue #1801 fix #2 — supervised DB self-defense', () => {
expect(info?.reason).toBe('db_dead');
}, 10_000);
it('the probe passes a cancellation AbortSignal so a hung SELECT 1 releases its slot (issue #6)', () => {
// Populated by the two runUntilUnhealthy() calls above (>= 3 probes each).
expect(probeOptsSeen.length).toBeGreaterThan(0);
for (const opts of probeOptsSeen) {
expect(opts?.signal).toBeInstanceOf(AbortSignal);
}
});
it('dual-pool probe gating: probeDirect is wired ONLY when isDualPoolActive() is true', async () => {
// The load-bearing guard the adapter comments on: a kill-switched
// executeRawDirect would probe the same starved read pool twice and
// fabricate a verdict (testing specialist coverage gap).
const directCalls: string[] = [];
const runCase = async (dualPool: boolean): Promise<void> => {
const cmStub = { isDualPoolActive: () => dualPool };
(engine as unknown as { connectionManager?: unknown }).connectionManager = cmStub;
const realDirect = engine.executeRawDirect.bind(engine);
(engine as unknown as { executeRawDirect: unknown }).executeRawDirect = async (
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
) => {
if (typeof sql === 'string' && sql.trim() === 'SELECT 1') {
directCalls.push(`dual=${dualPool}`);
return [];
}
return realDirect(sql, params as never, opts);
};
const restore = breakLivenessProbe(engine);
try {
await withEnv({ GBRAIN_SUPERVISED: '1' }, async () => {
const worker = new MinionWorker(engine, {
queue: 'default', concurrency: 1, pollInterval: 25, maxRssMb: 0,
healthCheckInterval: 20, dbFailExitAfter: 2, dbProbeTimeoutMs: 200,
});
worker.register('noop', async () => {});
const got = new Promise<UnhealthyReason>((resolve) => {
worker.on('unhealthy', (i) => resolve(i));
});
const runPromise = worker.start();
const info = await Promise.race([
got,
new Promise<null>((r) => setTimeout(() => r(null), 3000)),
]);
worker.stop();
await runPromise;
expect(info).not.toBeNull();
if (dualPool) {
// Direct lane succeeded -> verdict names the pooler path.
expect(info!.reason === 'db_dead' && info!.verdict).toBe('pool_starved');
} else {
expect(info!.reason === 'db_dead' && info!.verdict).toBe('unknown');
}
});
} finally {
restore();
delete (engine as unknown as { executeRawDirect?: unknown }).executeRawDirect;
delete (engine as unknown as { connectionManager?: unknown }).connectionManager;
}
};
await runCase(false);
expect(directCalls.length).toBe(0); // single-pool: NEVER probes direct
await runCase(true);
expect(directCalls.filter((c) => c === 'dual=true').length).toBeGreaterThan(0);
}, 15_000);
it('structural: DB probe is NOT gated on !isSupervisedChild; stall detection IS', () => {
const src = readFileSync(
join(import.meta.dir, '..', 'src', 'core', 'minions', 'worker.ts'),