v0.42.24.0 fix(minions): route lock claim/renewLock through direct session pool (#1822)

* fix(minions): route lock claim/renewLock through direct session pool

The Minion lock heartbeat (claim + renewLock) ran every UPDATE through
engine.executeRaw(), which is hardcoded to the read pool. On Supabase that
is the transaction-mode pooler (6543), which recycles connections per
transaction. A lock is held for minutes, so the pooler periodically reaps
the socket mid-heartbeat -> CONNECTION_ENDED -> the lock looks expired ->
the worker force-evicts its own job and the claim loop wedges silently.

Add BrainEngine.executeRawDirect(): same contract as executeRaw, but routes
to the direct session-mode pool (5432, GBRAIN_DIRECT_DATABASE_URL) when
dual-pool is active. No-op delegation on PGLite / non-Supabase / kill-switch.
claim/renewLock now use it. Single-statement UPDATEs only, so the double-claim
guard and the renewLock no-inline-retry contract are preserved. Statements
inside an open transaction keep their tx connection (in-transaction guard keys
on peekReadPool() !== _sql); the lock hot-path never runs inside transaction().

The Postgres impl shares its cancellation plumbing with executeRaw via a
private runUnsafe helper. New test/postgres-execute-raw-direct.test.ts covers
the routing decision (dual-pool on/off x in-tx/not + abort short-circuit)
without a live Postgres; queue-lock-retry.test.ts gains a guard that claim
can never fall back to executeRaw.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.24.0 chore: bump version and changelog

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.42.24.0

Document executeRawDirect on the BrainEngine contract and the
claim/renewLock direct-session-pool routing in KEY_FILES.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: make D3 executeRaw-no-retry guard refactor-aware

The DRY refactor in this PR extracted executeRaw/executeRawDirect's shared
cancellation plumbing into a private runUnsafe(conn, ...) helper, so the single
conn.unsafe() call moved out of executeRaw's body. The D3 guard read
executeRaw's source and asserted conn.unsafe( appeared exactly once there,
which now fails (it's zero — executeRaw delegates).

The D3 invariant (no per-call retry wrapper) is unchanged; it just spans the
delegate now. Update the guard to check both public methods delegate to
runUnsafe without reconnect/retry, and assert the exactly-once conn.unsafe +
cancel-only catch in runUnsafe. Also extends coverage to executeRawDirect so
the lock hot-path can't reintroduce a retry wrapper either.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-03 22:40:12 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent f11d56cfca
commit f868257405
12 changed files with 279 additions and 31 deletions
+12
View File
@@ -2,6 +2,18 @@
All notable changes to GBrain will be documented in this file.
## [0.42.24.0] - 2026-06-03
**Minion workers no longer silently wedge mid-job on a Supabase brain.** The background worker that runs your cron jobs, enrich fan-out, and autopilot cycle holds a lock on each job and heartbeats it every couple of seconds to say "still working." On a Supabase brain that heartbeat was running on the transaction-mode pooler (the high-traffic 6543 port), which recycles its connections per transaction. A lock is held open for minutes, so the pooler would periodically drop the socket mid-heartbeat. The worker read that dropped socket as "the lock expired," force-evicted its own in-flight job, and then sat in a claim loop holding nothing — process alive, no errors in the log, just quietly doing no work. It showed up most under heavy `enrich` load.
The fix routes only the lock hot-path (`claim` and `renewLock`) to the direct **session-mode** pool (port 5432, `GBRAIN_DIRECT_DATABASE_URL`), which holds its connection open for the life of the worker so heartbeats survive. gbrain already shipped this dual-pool design for DDL and bulk work; the lock path just never used it. No new infrastructure — the direct URL was already in your config.
- **Nothing to configure.** `gbrain upgrade` and the worker uses the right pool automatically on any Supabase brain. PGLite (the zero-config default) has no pooler, so this is a no-op there — same behavior on both engines.
- **Atomicity is preserved.** Statements inside an open transaction still run on the transaction's own connection; only the standalone lock heartbeat (which never runs inside a transaction) gets rerouted. There's a kill-switch (`GBRAIN_DISABLE_DIRECT_POOL`) if you ever need the old behavior.
- **Empirically:** a heartbeat survived 4/4 beats over 8s on the session pool, vs. connection-drop storms on the transaction pooler.
### For contributors
New `BrainEngine.executeRawDirect()` — same contract as `executeRaw`, but routes to the direct pool when dual-pool is active (no-op delegation on PGLite / non-Supabase / kill-switch). `claim`/`renewLock` in `minions/queue.ts` point at it. The Postgres impl shares its cancellation plumbing with `executeRaw` via a private `runUnsafe` helper; the in-transaction guard keys on `peekReadPool() !== _sql` so a tx clone is detected and never rerouted. New `test/postgres-execute-raw-direct.test.ts` covers the routing decision (dual-pool on/off × in-tx/not, plus abort short-circuit) without a live Postgres; `test/queue-lock-retry.test.ts` gains a guard that `claim` can never fall back to `executeRaw`. Eng review cleared the plan; the four lock/pool guards stay green.
## [0.42.23.0] - 2026-06-03
**`gbrain jobs work` and `gbrain jobs supervisor` take a `--nice <n>` flag that lowers the background job tree's CPU scheduling priority without cutting concurrency.** When the Minions worker pool runs at full width (sync, embed, extract, subagent fans), it can drive a machine's load average high enough to starve your interactive shell. Dropping concurrency throws away throughput. Niceness is the right lever: keep full concurrency, run at low priority, and the work finishes just as fast when the box is idle while yielding politely when it's busy. In the real incident that drove this, reniceing the tree took load from ~7 to ~3 with no measurable throughput loss.
+21
View File
@@ -95,6 +95,27 @@ structural change). Plan + GSTACK REVIEW REPORT at
(`apply-edits.ts:311`) + `mkdirSync(recursive)` in
`runOptimizationLoop` (`src/core/skillopt/orchestrator.ts`). Small, own PR.
## Minion-lock direct-pool follow-up (v0.42+)
Filed from the eng-review of the lock-claim/renewLock → direct-session-pool fix
(PR #1816, now folded into `garrytan/minion-locks-session-pool`). Deliberately
scoped OUT of that change; not a regression.
- [ ] **P3 — Size the direct session pool for enrich fan-out.** The lock
hot-path (`claim`/`renewLock`) now routes through the direct session-mode pool
(port 5432) via `executeRawDirect`. Supabase's session-mode pool has a far
smaller connection ceiling than the transaction pooler (6543). `executeRawDirect`
checks out per-statement (not held open), so the risk is bounded by *concurrent
in-flight heartbeats*, not duration — but under heavy `enrich` fan-out (many
Minion workers each heartbeating at once) the smaller pool could contend or
exhaust. **Why:** a starved session pool would reintroduce the exact wedge class
the fix removes, just from a different cause. **Current state:** direct pool size
comes from `resolveDirectPoolSize` / `DEFAULT_DIRECT_POOL_SIZE`
(`src/core/connection-manager.ts`); no fan-out-aware tuning. **Where to start:**
measure concurrent heartbeat count under a realistic `enrich` burst, compare to
`DEFAULT_DIRECT_POOL_SIZE`, and either raise the default or add a
worker-count-aware knob. **Depends on:** PR #1816 landing first.
## v0.42.12.0 #1685 brain-health-as-solved follow-ups (v0.42+)
Deferred from the v0.42.12.0 wave (issue #1685, the posture umbrella over #1678/#1735).
+1 -1
View File
@@ -1 +1 @@
0.42.23.0
0.42.24.0
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -143,5 +143,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.23.0"
"version": "0.42.24.0"
}
+18
View File
@@ -1912,6 +1912,24 @@ export interface BrainEngine {
opts?: { signal?: AbortSignal },
): Promise<T[]>;
/**
* Like `executeRaw`, but routes through the DIRECT (session-mode) pool when
* dual-pool is active (Supabase: port 5432), falling back to the read pool
* otherwise. Use this for the Minion lock hot-path (`claim`/`renewLock`):
* those statements heartbeat a lock over many seconds, and the
* transaction-mode pooler (port 6543) recycles connections per-transaction,
* which surfaces as `CONNECTION_ENDED` mid-heartbeat → orphaned locks →
* silent worker wedge. The direct session pool holds the connection open for
* the life of the worker, so heartbeats survive. Single-statement UPDATEs
* only — same idempotency contract as `executeRaw`. On PGLite (no pooler)
* this is identical to `executeRaw`.
*/
executeRawDirect<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]>;
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 5 populates, Layer 7 consumes)
// ============================================================
+8 -2
View File
@@ -601,7 +601,10 @@ export class MinionQueue {
async claim(lockToken: string, lockDurationMs: number, queue: string, registeredNames: string[]): Promise<MinionJob | null> {
if (registeredNames.length === 0) return null;
const rows = await this.engine.executeRaw<Record<string, unknown>>(
// Direct (session-mode) pool: claim opens the lock that renewLock then
// heartbeats. Both must live on a connection the transaction-mode pooler
// won't recycle mid-hold, or the lock orphans and the worker wedges.
const rows = await this.engine.executeRawDirect<Record<string, unknown>>(
`UPDATE minion_jobs SET
status = 'active',
lock_token = $1,
@@ -1073,7 +1076,10 @@ export class MinionQueue {
/** Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed). */
async renewLock(id: number, lockToken: string, lockDurationMs: number): Promise<boolean> {
const rows = await this.engine.executeRaw<Record<string, unknown>>(
// Direct (session-mode) pool — see claim(). The heartbeat that keeps a job
// alive for minutes cannot run on the transaction pooler without periodic
// CONNECTION_ENDED drops that look like lock-expiry and orphan the job.
const rows = await this.engine.executeRawDirect<Record<string, unknown>>(
`UPDATE minion_jobs SET lock_until = now() + ($1::double precision * interval '1 millisecond'), updated_at = now()
WHERE id = $2 AND lock_token = $3 AND status = 'active'
RETURNING id`,
+14
View File
@@ -4756,6 +4756,20 @@ export class PGLiteEngine implements BrainEngine {
return Promise.race([queryPromise, abortPromise]);
}
/**
* PGLite is in-process WASM with no connection pooler, so the direct-pool
* routing that `executeRawDirect` provides on Postgres is a no-op here:
* delegate straight to `executeRaw`. Present so the BrainEngine contract is
* satisfied and the Minion lock hot-path works identically on both engines.
*/
async executeRawDirect<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
return this.executeRaw<T>(sql, params, opts);
}
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
// ============================================================
+52 -8
View File
@@ -4879,17 +4879,25 @@ export class PostgresEngine implements BrainEngine {
}
}
async executeRaw<T = Record<string, unknown>>(
/**
* Shared body for executeRaw / executeRawDirect: run a raw statement on the
* given connection and wire AbortSignal cancellation onto the pending query.
* The ONLY difference between the two public methods is which connection they
* pick (read pool vs direct session pool), so the cancellation plumbing lives
* here in one place rather than being copy-pasted.
*
* v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's .cancel()
* on the pending query. Init nudge (3s wallclock cap) is the first consumer;
* the AbortSignal fires when the timer trips. An already-aborted signal
* short-circuits before the network round-trip.
*/
private runUnsafe<T>(
conn: ReturnType<typeof postgres>,
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
const conn = this.sql;
const pending = conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]);
// v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's
// .cancel() on the pending query. Init nudge (3s wallclock cap) is the
// first consumer; the AbortSignal fires when the timer trips.
// Already-aborted signal short-circuits before the network round-trip.
if (opts?.signal) {
if (opts.signal.aborted) {
// .cancel() is fire-and-forget; the awaited query rejects with the
@@ -4905,7 +4913,7 @@ export class PostgresEngine implements BrainEngine {
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
} catch {
// best-effort; the .then below settles regardless
// best-effort; the .finally below settles regardless
}
};
opts.signal.addEventListener('abort', onAbort, { once: true });
@@ -4913,7 +4921,15 @@ export class PostgresEngine implements BrainEngine {
opts.signal?.removeEventListener('abort', onAbort);
});
}
return pending as unknown as T[];
return pending as unknown as Promise<T[]>;
}
async executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
return this.runUnsafe<T>(this.sql, sql, params, opts);
// Pre-#406 behavior: throw on any error including connection death.
// Per-call auto-retry is not safe here because executeRaw is also used
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
@@ -4924,6 +4940,34 @@ export class PostgresEngine implements BrainEngine {
// swap in a fresh pool. See db.ts setSessionDefaults / supervisor.ts.
}
/**
* Minion lock hot-path variant of executeRaw. Routes to the DIRECT
* session-mode pool (port 5432) when dual-pool is active so lock
* heartbeats survive the transaction-pooler's per-transaction connection
* recycling. See BrainEngine.executeRawDirect for the full rationale.
*
* When this engine is a transaction-scoped clone (txEngine from
* transaction()), `connectionManager` is inherited but `this.sql` is the tx
* connection; we intentionally honor the tx connection in that case by
* falling through to this.sql, because routing a statement inside an open
* transaction onto a different pool would break atomicity. The lock
* hot-path (claim/renewLock) does NOT run inside transaction(), so in
* practice this always reaches the direct pool there.
*/
async executeRawDirect<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
// Inside an open transaction, _sql is the reserved tx connection (set via
// defineProperty in transaction()); never reroute off it.
const inTransaction = this._sql !== null && this.connectionManager?.peekReadPool() !== this._sql;
const conn = (!inTransaction && this.connectionManager?.isDualPoolActive())
? await this.connectionManager.ddl()
: this.sql;
return this.runUnsafe<T>(conn, sql, params, opts);
}
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
// ============================================================
+34 -16
View File
@@ -262,28 +262,46 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => {
it('PostgresEngine.executeRaw is a single-statement passthrough (no try/catch on connection errors)', () => {
const src = readFileSync(resolve('src/core/postgres-engine.ts'), 'utf-8');
// Find the executeRaw method in the class (not the helper inside withReservedConnection)
// v0.41.18.0 (T5/A20): signature extended with optional `opts?: { signal?: AbortSignal }`
// 3rd arg + multi-line shape for real query cancellation. Regex updated to
// tolerate both the legacy single-line and the new multi-line signatures.
// v0.42.24.0 (eng-review D1): the cancellation plumbing shared by executeRaw
// and executeRawDirect was extracted into a private `runUnsafe(conn, ...)`
// helper. executeRaw / executeRawDirect now pick a connection and delegate;
// the single `conn.unsafe(` call lives in runUnsafe. The D3 invariant (no
// per-call retry wrapper) is unchanged — it just spans the delegate now, so
// this guard checks both the public methods AND the shared helper.
// Find executeRaw in the class (not the helper inside withReservedConnection).
// v0.41.18.0 (T5/A20): signature extended with optional `opts?: { signal?: AbortSignal }`.
const fnMatch = src.match(/async executeRaw<T = Record<string, unknown>>\(\s*sql: string,\s*params\?: unknown\[\][^)]*\):\s*Promise<T\[\]>\s*\{([\s\S]*?)\n \}/);
expect(fnMatch).not.toBeNull();
const body = fnMatch![1];
// Must not call reconnect() from this method (D3 intent: no per-call
// retry — recovery is supervisor-driven via reconnect()).
// executeRaw must not retry: no reconnect, no inline re-issue, and it must
// delegate to runUnsafe rather than re-implementing the query path.
expect(body).not.toContain('this.reconnect()');
// Must call conn.unsafe directly, exactly ONCE (no retry re-issue).
expect(body).toContain('conn.unsafe(');
const unsafeCallCount = (body.match(/conn\.unsafe\(/g) || []).length;
expect(unsafeCallCount).toBe(1);
// The try/catch present here is ONLY for AbortSignal cancellation
// swallow (v0.41.18.0 A20), NOT for connection retry. Confirm by checking
// the swallowed throws are .cancel() not network re-issue.
if (body.includes('catch')) {
expect(body).toContain('this.runUnsafe');
expect((body.match(/conn\.unsafe\(/g) || []).length).toBe(0);
// executeRawDirect (the Minion lock hot-path sibling) routes to the direct
// session pool but must NOT introduce a retry wrapper either — same delegate.
const directMatch = src.match(/async executeRawDirect<T = Record<string, unknown>>\(\s*sql: string,\s*params\?: unknown\[\][^)]*\):\s*Promise<T\[\]>\s*\{([\s\S]*?)\n \}/);
expect(directMatch).not.toBeNull();
const directBody = directMatch![1];
expect(directBody).not.toContain('this.reconnect()');
expect(directBody).toContain('this.runUnsafe');
expect((directBody.match(/conn\.unsafe\(/g) || []).length).toBe(0);
// The shared helper issues conn.unsafe EXACTLY ONCE (no retry re-issue) and
// never reconnects. Its try/catch is ONLY the AbortSignal cancellation
// swallow (v0.41.18.0 A20), NOT a connection retry.
const helperMatch = src.match(/private runUnsafe<T>\(\s*conn:[^)]*\):\s*Promise<T\[\]>\s*\{([\s\S]*?)\n \}/);
expect(helperMatch).not.toBeNull();
const helperBody = helperMatch![1];
expect(helperBody).not.toContain('this.reconnect()');
expect((helperBody.match(/conn\.unsafe\(/g) || []).length).toBe(1);
if (helperBody.includes('catch')) {
// If catch exists, it must be the cancel-swallow shape, NOT a retry shape.
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?conn\.unsafe/);
expect(body).not.toMatch(/catch[^{]*\{[\s\S]*?setTimeout/);
expect(helperBody).not.toMatch(/catch[^{]*\{[\s\S]*?conn\.unsafe/);
expect(helperBody).not.toMatch(/catch[^{]*\{[\s\S]*?setTimeout/);
}
});
+111
View File
@@ -0,0 +1,111 @@
import { describe, expect, test } from 'bun:test';
import { PostgresEngine } from '../src/core/postgres-engine.ts';
/**
* executeRawDirect routing decision (PR #1816 lock hot-path fix).
*
* The point of executeRawDirect is to send the Minion lock heartbeat
* (claim/renewLock) to the DIRECT session-mode pool (port 5432) instead of the
* transaction pooler (6543) that reaps connections mid-hold. That routing
* branch only fires against a real Supabase dual-pool, which CI doesn't have
* so the decision itself (which connection gets the statement) went untested.
*
* This exercises the pure routing logic with a stubbed ConnectionManager and
* fake Sql handles, covering all three shapes:
*
*
* engine shape dual-pool conn chosen
*
* worker, not in tx active ddl() direct
* tx clone (_sql = tx conn) active this.sql (tx) atomicity
* worker, not in tx inactive this.sql read
*
*/
type FakeSql = { unsafe: (sql: string, params?: unknown[]) => Promise<unknown[]> };
/** A fake postgres.js handle whose unsafe() tags rows with its label. */
function fakeSql(label: string): FakeSql {
return {
unsafe: async () => [{ via: label }],
};
}
/**
* Build a PostgresEngine with its connection internals stubbed so
* executeRawDirect can be driven without a live database.
*
* - readConn the read pool (what `this.sql` returns when _sql === readPool)
* - directConn what connectionManager.ddl() resolves to
* - sqlOverride forces the `get sql()` getter (used to model a tx clone whose
* `this.sql` is the tx connection, distinct from peekReadPool()).
*/
function makeEngine(opts: {
dualPoolActive: boolean;
readConn: FakeSql;
directConn: FakeSql;
// When set, models a tx clone: _sql is the tx conn (!== peekReadPool()).
txConn?: FakeSql;
}): PostgresEngine {
const engine = new PostgresEngine();
const e = engine as unknown as Record<string, unknown>;
// _sql: tx conn for a clone, otherwise the read pool itself (the worker case
// where connect() does setReadPool(this._sql)).
e._sql = opts.txConn ?? opts.readConn;
// The `get sql()` getter on the prototype returns _sql when set, so we don't
// need to override it — _sql already drives it. For a tx clone _sql is txConn,
// so this.sql === txConn, exactly as Object.defineProperty does in transaction().
e.connectionManager = {
isDualPoolActive: () => opts.dualPoolActive,
peekReadPool: () => opts.readConn,
ddl: async () => opts.directConn,
};
return engine;
}
describe('PostgresEngine.executeRawDirect — routing decision (PR #1816)', () => {
test('dual-pool active + not in tx → routes to direct (ddl) pool', async () => {
const readConn = fakeSql('read');
const directConn = fakeSql('direct');
const engine = makeEngine({ dualPoolActive: true, readConn, directConn });
const rows = await engine.executeRawDirect<{ via: string }>('UPDATE minion_jobs SET x=1');
expect(rows[0].via).toBe('direct');
});
test('inside a transaction → honors the tx connection (never reroutes off it)', async () => {
const readConn = fakeSql('read');
const directConn = fakeSql('direct');
const txConn = fakeSql('tx');
// dual-pool is active, but _sql (tx) !== peekReadPool() (read) → inTransaction.
const engine = makeEngine({ dualPoolActive: true, readConn, directConn, txConn });
const rows = await engine.executeRawDirect<{ via: string }>('UPDATE minion_jobs SET x=1');
expect(rows[0].via).toBe('tx');
});
test('dual-pool inactive → falls back to the read pool', async () => {
const readConn = fakeSql('read');
const directConn = fakeSql('direct');
const engine = makeEngine({ dualPoolActive: false, readConn, directConn });
const rows = await engine.executeRawDirect<{ via: string }>('UPDATE minion_jobs SET x=1');
expect(rows[0].via).toBe('read');
});
test('already-aborted signal short-circuits with AbortError before routing the query', async () => {
const readConn = fakeSql('read');
const directConn = fakeSql('direct');
const engine = makeEngine({ dualPoolActive: true, readConn, directConn });
const ac = new AbortController();
ac.abort();
await expect(
engine.executeRawDirect('UPDATE minion_jobs SET x=1', [], { signal: ac.signal }),
).rejects.toThrow(/abort/i);
});
});
+5 -1
View File
@@ -58,7 +58,11 @@ describe('MinionQueue lock-path recovery (issue #1678)', () => {
let calls = 0;
const engine = {
kind: 'postgres',
executeRaw: async () => { calls++; throw connEndedError(); },
// claim() routes through executeRawDirect (direct session pool) as of
// the lock-hot-path fix; executeRaw is kept as a throwing guard to
// prove claim never falls back to it.
executeRawDirect: async () => { calls++; throw connEndedError(); },
executeRaw: async () => { throw new Error('claim must not use executeRaw'); },
reconnect: async () => {},
} as unknown as ConstructorParameters<typeof MinionQueue>[0];