gbrain

mirror of https://github.com/garrytan/gbrain.git synced 2026-08-15 17:32:37 +00:00

Author	SHA1	Message	Date
Garry TanandClaude Fable 5	ec3d7e79c7	docs: cross-model doc-review precision fixes for v0.46.1.0 Codex review of the shipped docs vs the diff, each finding verified against the code before applying: - minions-deployment: group-SIGKILL platform caveat (Bun /bin/kill fallback), lock-token fencing scoped to queue writes (handler side effects bounded by the watchdog, not the token), connection math relabeled (pooler-lane vs direct session-lane split), no-per-child-RSS-cap note, GBRAIN_JOB_CHILD_CLI + the 3-consecutive-spawn-failure breaker documented. - queue-operations-runbook: verdict rides the TERMINAL probe line (not every N/3 line), server_unreachable hedged (both-lanes-failed is the evidence), pooler-layer fault added to the 0-in-flight reading, jobs cancel described as cooperative inline vs real kill under isolation. - KEY_FILES: run-child SIGTERM fires shutdownSignal ONLY (both only on parent death); third no-burn child class (ChildNotClaimedError). - TESTING: e2e concurrency leg uses the fixture (no child DB pools); only the run-child leg boots real child pools. - CHANGELOG: one wording precision fix (reserved holds leave a heartbeat slot, not "always keep a free slot"). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 09:54:50 -07:00
Garry TanandClaude Fable 5	802441223d	docs: README Minions capability + KEY_FILES reserved-routing entry for v0.46.1.0 document-release sweep: the wave's docs covered the guides, TESTING, and the new module entries but missed two spots — the README Job queue capability paragraph (now names --job-isolation process and the probe verdicts, linking both guides) and the KEY_FILES postgres-engine.ts entry (now carries the withReservedConnection direct-lane routing invariants + getPoolDiagnostics seam, pinned by test/postgres-engine-reserved-routing.test.ts). llms-full.txt regenerated for the README edit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 09:45:51 -07:00
Garry TanandClaude Fable 5	f6aec5b0a0	test: passwordless fixture URLs in the pool-wiring tests The pre-push credential guard (correctly) blocks any URL-with-password shape in a pushed diff, including fake placeholders. The never-connected fixture URLs don't need a password at construction time — drop it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 09:38:34 -07:00
Garry TanandClaude Fable 5	2b1a0811b3	chore: bump version and changelog (v0.46.1.0) Issues #5 + #6 wave: pool-starvation cancellation + diagnostics, and opt-in per-job process isolation. Version locations: VERSION, package.json, CHANGELOG.md, openclaw.plugin.json, BOOTSTRAP_FOR_AGENTS.md stamp, and the regenerated bootstrap template tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 09:23:06 -07:00
Garry TanandClaude Fable 5	f870f1d3ef	fix(minions,db): pre-landing review fixes — specialist + red-team findings (#5 , #6 ) Ship's review army (5 specialists + red team over the full diff; 0 critical from specialists, 3 confirmed critical from red team) — all findings fixed: Red team (critical): - reserved-direct cap: removed the Math.max(1, size-1) floor — at direct_pool_size=1 it let a multi-minute reserve consume the ONLY direct session and starve claim/renewLock heartbeats (the #6 class reintroduced). cap = size - 1, direct routing only when cap >= 1; size<=1 uses the read pool (true status quo). Pinned by a size=1 routing test. - silent group-kill failure: the SIGKILL escalation now logs loudly when delivery fails (distroless hosts without /bin/kill would otherwise void the kill guarantee with zero diagnostics while the job duplicated elsewhere), and skips the redundant signal when the child already exited. - spawn-failure circuit breaker: a deterministically broken child CLI looped claim/release forever, invisible to the stall detector (every settle refreshes the progress clock). After 3 consecutive spawn/bootstrap failures the worker emits unhealthy(child_spawn_failing) for a process-manager restart; counter resets on any spawn that runs. Plus the predicate-mismatch guard: jobIsolation 'process' without childCliInvocation now throws at construction (it silently ran handlers inline while the evict path believed it was isolated). Specialists (informational, all applied): - performance: parent-side outcome decode is async (a 32MiB-capped file must not block the event loop running renewal ticks); /bin/kill by absolute path (also the security finding). - security: lease payloads are shape-validated before reconstruction (corrupt outcome files degrade to generic); the child-CLI override is canonicalized to an absolute path so the fail-fast check validates the binary that actually spawns. - data-migration: max_lifetime default is now a per-CONNECTION jitter FUNCTION (matching the postgres.js built-in shape — a pre-evaluated number synchronized every connection in a pool onto one recycle deadline); reserved.release() throws no longer leak the gauge or the direct permit. - testing: child harnesses use a readiness handshake instead of fixed 400ms sleeps (CI-load flake); the orphan-watchdog test uses a real reaped pid (a magic high pid is allocatable under Linux pid_max); new pins for the dual-pool probe gating (probeDirect wired ONLY when isDualPoolActive), the executeRawDirect/transaction gauge seams incl. the sync begin()-throw leak guard, the ddl()-throw read-pool fallback, and the --job-isolation help text. - maintainability: abort-reason literals shared via types.ts (dead 'cancel'/'cancelled' entries dropped), DEFAULT_DIRECT_POOL_SIZE and CHILD_READ_POOL_MAX named, redundant dynamic imports removed, unrefTimer helper, getConnectionRouting shared accessor, docstring + fixture-header corrections. Deferred with TODOS entries: raceWithAbortTimeout DRY helper (5 sites), lazy handler resolution in run-child, e2e-lane negative tests for the run-child bootstrap guards + operator-flow messages, behavioral withRefreshingLock test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 09:22:57 -07:00
Garry Tan	8adbf87443	Merge remote-tracking branch 'origin/master' into garrytan/fix-issue-5	2026-08-15 08:50:08 -07:00
Garry TanandClaude Fable 5	2aac6390cb	fix(minions,db): adversarial-review hardening — shutdown/attempt semantics, TOCTOU semaphore, gauge + parity gaps (#5 , #6 ) A 23-agent adversarial review (5 dimension finders + per-finding refuters) over the wave's diff confirmed 12 defects; all fixed here: - [P2] run-child conflated worker SIGTERM with the per-job abort: cooperative handlers bailed mid-deploy, reported an error outcome, and the parent BURNED an attempt per routine deploy — while signal-ignoring handlers got the no-burn release (the exact inversion of the shutdown guarantee). SIGTERM now fires ONLY shutdownSignal (inline signal-separation parity — handlers finish + report inside the drain window), parent death still aborts both, and the parent classifies an ERROR outcome that arrives during shutdown as ChildWorkerShutdownError (released, not burned; a genuinely-failing job coinciding with a deploy gets one free retry). - [P2] the reserved-direct semaphore was a check-then-increment spanning `await ddl()` — same-tick concurrent reserves could overshoot the cap and starve the heartbeat slot it exists to protect. The permit is now taken in the same synchronous frame as the check. - [P3] RSS-watchdog drain (gracefulShutdown aborts BOTH signals, reason 'watchdog') was classified as a per-job abort and burned attempts on innocent isolated jobs. Shutdown classification now wins unless the per-job reason is job-targeted (timeout/cancel/lock-*). - [P3] force-evict's failJob('dead') could race executeJob's own recording in isolation mode (group SIGKILL at 25s + slow decode > 30s window) and dead-letter a job with attempts remaining — skipped when isolated (the inFlight eviction, which is what unblocks the worker, stays). - [P3] child bootstrap exits were burned as handler crashes: exit 13 → ChildSpawnInfraError (release), exit 14 → new ChildNotClaimedError (release; the claim is provably owned elsewhere). - [P3] missing handler in the child was 'generic' (retried to max_attempts) vs inline's immediate dead-letter — now 'unrecoverable' (parity). - [P3] result-shape parity: the {value: x} wrap now happens CHILD-side, before JSON serialization, so Date/toJSON results can't flip the wrap decision across the boundary. - [P3] child env no longer raises a stricter user GBRAIN_POOL_SIZE (pooler MaxClients tuning respected; explicit GBRAIN_JOB_CHILD_POOL_SIZE wins; invalid values fall back instead of flowing to the 10-conn fallback). - [P3] transaction() gauge used a chained .finally that a synchronous begin() throw (nested tx on a clone) would skip — now try/finally. - [P3 vacuity x3] new pins: db-lock heartbeat cancellation wiring + re-entrancy, the synthesize drain-loop guard + tick call (the shape guard only covers worker.ts), and GBRAIN_POOL_MAX_LIFETIME_S reaching a REAL constructed pool (postgres() is lazy — no I/O). New tests: error-outcome-during-shutdown, watchdog double-abort, timeout-beats-shutdown precedence, bootstrap exit codes, pool-size env matrix, SIGTERM-only-aborts-shutdown (in-process emit), child-side wrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:48:03 -07:00
Garry TanandClaude Fable 5	905aeb481d	test: repoint the deadlineAtMs structural pin at the extracted job-context builder The deadline-plumbing structural test grepped worker.ts for the literal deadlineAtMs derivation, which moved verbatim into job-context.ts (the builder shared by inline mode and 'jobs run-child'). The pin now checks the derivation in job-context.ts AND that worker.ts calls buildJobContext — the same contract, at its new home. Full-suite triage note: an isolated A/B of the 22 files that failed in the parallel full-suite run shows IDENTICAL results on this branch and on the master base (290 pass / 5 fail — doctor-minions-check + unified-multimodal, both env-dependent) — zero regression delta from this wave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:36:39 -07:00
Garry TanandClaude Fable 5	a2bbdb7c04	docs(minions,db): pool-starvation diagnostics + job isolation reference; file the follow-ups (#5 , #6 ) - KEY_FILES.md: entries for the six new modules (job-isolation, child-job-runner, run-child, job-context, db-probe, pool-gauge) and current-state updates for worker/queue/supervisor/jobs/db/db-lock/ lock-renewal-tick/synthesize. - minions-deployment.md: a --job-isolation section modeled on --nice — how the parent/child split works, preserved error semantics, orphan story, and the sizing notes (pooler CLIENT connection math: concurrency 15 ~ 73; --max-rss covers the worker only; spawn cost guidance; the lock token is a fencing token, not a secret). - TESTING.md: inventory entries for the 12 new unit files + the e2e lane (which is wired EXPLICITLY into e2e.yml tier1 — no glob exists). - TODOS.md: filed the 10 follow-ups, headlined by the P1-companion nested-checkout audit (the strongest remaining #6 root-cause candidate — this wave mitigates the starvation class and fixes the diagnostic; it does not claim to close every leak path), plus per-handler isolation policy, per-child RSS caps, the connection-budget clamp, autopilot pass-through, connection-audit release events, the doctor connection_routing check, and Sql-proxy checkout instrumentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:19:02 -07:00
Garry TanandClaude Fable 5	c2a6bc8078	feat(cli): --job-isolation flag, supervisor pass-through, e2e lane (#5 ) The user-facing surface for per-job process isolation: - jobs work --job-isolation <inline\|process> (space or = form; env fallback GBRAIN_JOB_ISOLATION; default inline — fully opt-in). With 'process' the worker resolves the child CLI ONCE at startup (GBRAIN_JOB_CHILD_CLI -> compiled binary -> bun-dev fallback) and REFUSES to start on an unresolvable/nonexistent path — a bad path discovered per-job would stall the queue one released claim at a time. detectTini() wraps children when available. Startup banner names the mode + child CLI; combining with --max-rss prints a note that the watchdog now covers the worker only. - jobs supervisor --job-isolation passes through via buildWorkerArgs as a CONDITIONAL push — inline/omitted keeps existing deployments' worker argv byte-identical (pinned arrays in supervisor-build-worker-args.test.ts are untouched; two new cases added). - pool_starved fatal text now names the flag as a remedy (handler connections die with each job's child). - help text for work + supervisor + the jobs index; flag registry regenerated. - NEW test/e2e/job-isolation.test.ts, wired into e2e.yml tier1 EXPLICITLY — the workflow runs only named files (no glob), so an unwired e2e file would be silent coverage loss. Legs: concurrency-3 isolated drain through real children against real Postgres (the child-pool topology), and the REAL `jobs run-child` CLI entrypoint end-to-end (engine bootstrap, quiet handler registry, token validation, outcome protocol). Follows the #4128 ambient-URL-guard conventions (explicit env in the e2e lane). - serialization parity (codex-2 #8): a non-JSONB-serializable result fails loudly in BOTH modes (inline completeJob serialization vs child exit 15) — isolation never falsely completes a job inline mode would have failed. Tests: jobs-isolation-flag.test.ts (parser matrix), extended supervisor-build-worker-args + worker-job-isolation, cli-flag-validation green via regen, jobs-subcommand-help.serial green (engine-free help path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:14:36 -07:00
Garry TanandClaude Fable 5	12b89ef46f	feat(minions): process isolation — run claimed jobs in SIGKILL-able children (#5 ) The parent-side seam. executeJob swaps ONE line — handler(context) vs runJobInChild(...) — and every existing reporting branch (completeJob, failJob dead/delayed, lease release, infra-abort no-burn) is reused verbatim on the child's reconstructed outcome. Blast radius of a stuck or crashing handler drops from N in-flight jobs to exactly one. child-job-runner.ts: - detached spawn (own process group) + tini wrap when available; stdio ['ignore','inherit','inherit'] so handler logs stream to the operator; per-job lifecycle log lines (spawned / exited code+signal); - per-job abort -> group SIGTERM now, group SIGKILL at +25s (inside the 30s force-evict window, which stays as an untouched backstop) — force-eviction is now a real kill, not an abandonment; - worker shutdown -> same SIGTERM so the child's handlers get the drain window to finish AND report; a child that reported before the kill completes normally; one that couldn't throws ChildWorkerShutdownError, which the worker RELEASES with no attempt burned — routine deploys must not burn attempts (codex-2 #7); - pre-exec spawn failure -> ChildSpawnInfraError, also released with no attempt burned (one bad CLI path must not dead-letter a queue); - child env contract: fenced lock token, outcome path, parent pid for the orphan watchdog, GBRAIN_POOL_SIZE=3 + GBRAIN_DIRECT_POOL_SIZE=1 bounds (children run no heartbeats; sockets die with the process — the point). worker.ts: MinionWorkerOpts gains jobIsolation / childCliInvocation / childTiniPath (defaults preserve inline behavior exactly); when isolated the parent-side MinionJobContext is not built at all (the child builds its own). Tests: child-job-runner.test.ts (real .mjs children: success + env contract, error/lease outcome reconstruction, crash, SIGTERM-ignorer -> group SIGKILL, pre-aborted, spawn ENOENT, both shutdown semantics); worker-job-isolation.test.ts (real PGLite worker end-to-end: claim -> child -> fenced completeJob with the REAL claim token, failJob on error outcome, crash burns attempt, spawn failure releases with zero attempts burned). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:07:39 -07:00
Garry TanandClaude Fable 5	4a12fcc340	feat(minions): hidden 'jobs run-child' single-job entrypoint (#5 ) The child side of process isolation. `gbrain jobs run-child --job-id N` (internal; spawned by the worker, absent from user help): - re-reads the job row and validates status='active' + lock-token match before running anything — a reclaimed/cancelled job exits 14 with the handler never invoked (the DB stays ground truth; no payload serialization across the boundary); - registers the same handler surface as the worker via registerBuiltinHandlers({quiet}) — which includes plugin discovery, so plugin subagent jobs isolate identically — resolved through the new MinionWorker.getHandler() accessor; - builds the shared token-fenced MinionJobContext against the CHILD's own engine, runs the handler, and writes ONE atomic outcome file: handler failure is an encoded error outcome with exit 0 (a reported failure is a successful report); only write-failure exits 15; - runs NO worker machinery (no probe/stall/lock timers — the parent owns liveness). Installs a SIGTERM handler (fires ctx.signal + shutdownSignal so handlers get the drain window to finish and report) and a parent-liveness watchdog polling process.kill(parentPid, 0) — a ppid check is dead code under tini — that aborts the handler and hard-exits after a grace so orphaned LLM-bound work stops burning spend; - CLI layer owns engine.disconnect() + process.exit() (engine-ownership invariant); PGLite exits 13 (isolation is Postgres-only, like jobs work). Flag registry regenerated for the internal job-id flag. Tests: run-child-entry.test.ts against real in-memory PGLite with a REAL claim-minted token — success (incl. a fenced updateProgress landing), handler-failure outcome, token-mismatch never-runs, missing job, missing handler, and the parent-death watchdog aborting a live handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 08:00:41 -07:00
Garry TanandClaude Fable 5	3a779b9752	feat(minions): job-isolation protocol, child exit codes, shared job-context builder (#5 ) Foundation for per-job process isolation (no behavior change yet): - job-isolation.ts — the parent<->child protocol: atomic outcome-file codec (tmp+rename; 32MiB decode cap that throws UnrecoverableError so oversize results die LOUDLY on attempt 1 instead of retrying identically or being silently truncated; decode errors report byte counts, never file content), handler-error encode/reconstruct preserving the two instanceof branches executeJob dispatches on (UnrecoverableError, RateLeaseUnavailableError), child argv/env contract, child-CLI resolution (env override -> compiled binary -> bun-dev fallback -> null for fail-fast), and killProcessGroup — children run detached in their own process group because SIGKILL on a tini pid alone kills tini and orphans the handler grandchild (tini cannot forward SIGKILL), and Bun rejects negative pids in process.kill() (oven-sh/bun#15791) so group signaling falls back to POSIX /bin/kill. - worker-exit-codes.ts — reserved run-child codes 13 (usage/PGLite), 14 (not claimed / token mismatch), 15 (result-write failed). Result-file presence, not the exit code, classifies the normal path: a reported handler FAILURE is still exit 0. - job-context.ts — MinionJobContext builder extracted verbatim from executeJob so the child wires the exact same token-fenced DB callbacks; worker.ts now calls it (behavioral no-op, full minions suite green). Tests: job-isolation-protocol.test.ts — codec round-trip + all decode failure paths, instanceof reconstruction, invocation resolution, and REAL detached-process group-kill tests incl. the grandchild-death guarantee (runs under bun test, so the Bun negative-pid fallback is exercised for real, not mocked). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:56:42 -07:00
Garry TanandClaude Fable 5	04a319e311	fix(db): route withReservedConnection to the direct session pool when dual-pool is active (#6 ) Long-hold reserved work — CREATE INDEX CONCURRENTLY (vector-index), non- transactional migration DDL, and backfill BEGIN..COMMIT batches (the observed 353s COMMIT session) — previously reserved from the worker's shared READ pool, pinning slots under the 5-min pooler statement_timeout. It now reserves from the DIRECT session lane, whose 30-min statement_timeout and maintenance_work_mem GUCs are the right fit, and stops competing with handler workload. Heartbeat protection: concurrent direct reserves are capped at directPoolSize - 1 (default 2 of 3) via a per-process semaphore so claim/renewLock always keep >= 1 direct slot; overflow falls back to the read pool — exactly the pre-change behavior, so this commit is strictly never-worse than master. (Deliberate rejection of queue-for-a-permit: that would block migrations behind multi-minute index builds. Per-process is the correct scope: each process owns its own direct pool, so a CLI migration cannot starve a worker's heartbeats.) Never rerouted inside an open transaction (same guard shape as executeRawDirect); kill-switch collapse degrades to status quo. Callers unchanged. Tests: postgres-engine-reserved-routing.test.ts — direct when active, read when kill-switched/in-tx, semaphore cap + overflow + permit release on fn throw and on reserve() failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:52:31 -07:00
Garry TanandClaude Fable 5	a02bfe34ef	feat(minions): pool-state health-probe diagnostics — pool starved vs server unreachable (#6 ) "[health] DB unreachable" sent operators chasing database capacity while the real fault was client-side: the server sat at ~10% of max_connections. The probe now names the failing layer: - New src/core/minions/db-probe.ts (hermetic, injected-deps — the lock-renewal-tick pattern): on read-pool probe failure, a 3s direct-lane SELECT 1 disambiguates. Direct OK -> verdict 'pool_starved' ("server IS reachable; the fault is in the transaction-pooler path — client pool exhaustion or a pooler-layer fault", deliberately an honest disjunction). Both fail -> 'server_unreachable'. No direct lane -> 'unknown'. Both probes carry AbortSignals — a hung probe is cancelled, never abandoned. - New src/core/pool-gauge.ts: approximate in-flight counters at the engine's raw/direct/reserved/transaction seams, surfaced via a duck-typed PostgresEngine.getPoolDiagnostics() (no BrainEngine churn, no PGLite stub). Explicitly labeled a tracked SUBSET — template-path traffic is untracked and no waiter/available figures are derived (that would be invented telemetry). Counters use try/finally (runUnsafe throws synchronously on a pre-aborted signal) and clamp at zero. - worker.ts probe adapter emits the verdict in every failure line and on the final unhealthy payload; exit semantics UNCHANGED (exiting on a starved pool is correct recovery — it frees all client-held slots). - jobs.ts: verdict-aware fatal text, plus a startup warning when a Supabase-shaped engine is running single-pool (kill-switch collapse used to be silent — renewal + probes + workload all sharing one pool is the precondition for this incident class). - Runbook: verdict interpretation table in queue-operations-runbook.md. Tests: pool-gauge.test.ts (pure + engine seams incl. rejected-query and sync-throw leak guards), db-probe.test.ts (full verdict matrix, signal cancellation, fail-open diagnostics, no-waiter-wording pin). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:50:48 -07:00
Garry TanandClaude Fable 5	8caa2b0c88	fix(db): explicit jittered max_lifetime on all four client pools (#6 ) Makes the pool connection lifetime explicit at every postgres() call site (db.ts module singleton, engine instance pool, ConnectionManager read + direct pools) and adds GBRAIN_POOL_MAX_LIFETIME_S as an incident escape hatch (N seconds; 0 disables recycling). NOT a behavior change at default: postgres.js (verified against the pinned 3.4.9) already defaults max_lifetime to 60(30+rand30) — 30-60 min, jittered per pool — and max_lifetime only recycles connections as they return to the pool; it cannot reclaim a leaked checkout. Framed accordingly: explicitness + operator knob, not a fix for the starvation class (that is the cancellation work in the two prior commits). Tests: hermetic resolver suite (env forms, 0-disables, jitter bounds, warn-once on invalid values, per-call jitter variance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:43:55 -07:00
Garry TanandClaude Fable 5	222023f65f	fix(minions,db): cancel timed-out wedge probes + db-lock refreshes (#6 ) Same abandoned-racer class as the previous commit, in two more spots: - probeQueueState raced probeQueueStateInner against its 1500ms budget but the losing wedge/age queries kept running on the pool after the race resolved — under pool exhaustion (the exact regime the probe exists to detect) the orphaned query held a slot and made the exhaustion worse. The timeout now aborts a per-probe signal threaded through queryWedgeSignals and the oldest-waiting age query. Closes the filed TODOS entry. - withRefreshingLock raced handle.refresh() against heartbeatTimeoutMs the same way; DbLockHandle.refresh now accepts { signal } (Postgres forwards to executeRawDirect; PGLite ignores it — no pool to starve), the timeout aborts it, and a re-entrancy guard stops overlapping ticks (15s min cadence vs 30s default timeout could stack two). Tests: new hermetic queue-probe-cancellation.test.ts (signal threading, timeout-aborts, fast-path-not-aborted, fail-open contract). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:41:35 -07:00
Garry TanandClaude Fable 5	2344821ecd	fix(minions): cancel abandoned probe + lock-renewal queries instead of orphaning pool slots (#6 ) Three hot paths raced a live query against a timer and abandoned the loser, leaving the query holding a checked-out pool slot for its full server-side duration. Under a saturated transaction-mode pooler those orphaned slots starve lock renewal ('lock-renewal-failed' cascades) and the health probe. - Health probe: pass the deadline AbortController's signal into executeRaw('SELECT 1') so a hung probe is cancelled via postgres.js .cancel() (runUnsafe already wires signal -> pending.cancel()). - Minion lock renewal: LockRenewalDeps.renewLock widened with optional { signal }; runLockRenewalTick aborts a per-call controller when the timeout wins the race; MinionQueue.renewLock forwards the signal to executeRawDirect. Optional-param widening keeps the 14 existing hermetic tests compiling untouched. - Cycle drain renewal (synthesize.ts): the inline best-effort tick had no per-call timeout and no re-entrancy guard, so a hung renewLock stacked a fresh checked-out slot per interval firing. Extracted as exported runDrainRenewalTick (per-call signal + timeout + swallow) behind a tick-in-flight guard. Tests: 2 new signal paths in worker-lock-renewal.test.ts, probe-signal assertion in worker-supervised-db-probe.test.ts, new hermetic minion-queue-renewlock-signal.test.ts + cycle-drain-renewal.test.ts. scripts/check-worker-lock-renewal-shape.sh stays green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-15 07:38:12 -07:00

Compare commits

Diff Content Not Available