mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
55506d7037 | ||
|
|
d7783122b9 | ||
|
|
41a5a3f316 | ||
|
|
6ad91ff3bf | ||
|
|
21b9ba0fe1 | ||
|
|
2d54660775 | ||
|
|
7df55cfc9f | ||
|
|
d86464e0d7 | ||
|
|
1232ba2b8b | ||
|
|
bbde573ca6 | ||
|
|
f4a0f899ad | ||
|
|
f576205523 | ||
|
|
784e62c535 | ||
|
|
0979d1875b | ||
|
|
8fa606f09d | ||
|
|
180a776f13 | ||
|
|
7bde448f96 | ||
|
|
8f633dc79a | ||
|
|
0cbf3b85d1 | ||
|
|
b068e0a212 |
@@ -2,6 +2,27 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.52.0] - 2026-06-18
|
||||
|
||||
**Autopilot stops manufacturing dead jobs and wedging its own queue, plus four operational rough edges get fixed: minion attempt-accounting, `agent run` flag parsing, honest `sources status`, and a budgeted `gbrain status`.** On a multi-source Postgres brain, autopilot could fan out a continuous stream of dead `autopilot-cycle` jobs while the supervisor periodically wedged the very queue it exists to keep alive. The root cause was one disease with several interacting parts; this wave addresses all of them, then cleans up four smaller reliability bugs found alongside.
|
||||
|
||||
### Changed
|
||||
- **Autopilot runs one brain-wide maintenance pass, not one per source.** The cycle is split: per-source jobs run only source-scoped phases, and a single `autopilot-global-maintenance` job runs the brain-wide phases once per window. This removes the per-cycle memory blow-up that was the shared root cause of the dead-job storm and the queue wedge. Per-source filesystem phases bind to the source's own path, so the freshness stamp and the work agree on which source ran.
|
||||
- **The supervisor self-heals instead of giving up.** A transient database blip no longer trips the crash-budget breaker into a permanent stop; the supervisor degrades to capped-backoff retry and recovers, with a hard ceiling as the backstop. It detects a live sibling supervisor through the queue's database lock (not a `$HOME`-derived pidfile), so two supervisors under a split home directory can't both claim the queue.
|
||||
- **`gbrain sources status` tells a running sync apart from an idle source.** A source holding a live sync lock now reads as actively syncing instead of "idle," matching the honest-freshness signal `gbrain doctor` already shows.
|
||||
|
||||
### Added
|
||||
- **Per-source failure cooldown + fan-out clamp.** A source that fails backs off (bounded exponential) instead of being re-dispatched every tick; per-tick fan-out is clamped to the worker concurrency, with a `gbrain doctor` check that warns on a mismatch.
|
||||
- **`gbrain status --deadline-ms` / `--fast`.** A budgeted status snapshot returns whatever sections completed within the budget (marked partial) instead of hanging a poller; the JSON envelope also carries the CLI `version`.
|
||||
- **A sync stall watchdog.** If the import drain makes no forward progress for `GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900), the run aborts and releases its per-source lock so the next `gbrain sync` resumes from the checkpoint — no manual `pkill`. It keys off import progress (not the lock heartbeat) and reports a distinct `stall_timeout` reason. (Limit: a hang inside a single file's import is observed between files, not mid-file; the wall-clock deadline remains the backstop there.)
|
||||
|
||||
### Fixed
|
||||
- **A timed-out minion run counts as a spent attempt.** Wall-clock dead-lettering already did; the per-job timeout path didn't, so long-lane jobs (subagent / embed-backfill / autopilot-cycle) could read `attempts: 0/N (started: N)`. Accounting is now honest across all dead-letter paths.
|
||||
- **`gbrain agent run` no longer swallows flags after the prompt.** A trailing `--detach` / `--follow` is recognized instead of being captured into the prompt string; a `--word` inside the prompt stays verbatim, and an explicit `--` ends flag parsing anywhere.
|
||||
|
||||
### To take advantage of v0.42.52.0
|
||||
`gbrain upgrade`. Existing brains pick up the cycle split, supervisor backoff, and per-source cooldown on the next autopilot tick (one catch-up global-maintenance pass on the first tick) — no migration, all on by default. Tune the sync stall watchdog with `GBRAIN_SYNC_STALL_ABORT_SECONDS` if 900s doesn't fit your largest files; budget a status poller with `gbrain status --fast` or `--deadline-ms=<n>`.
|
||||
|
||||
## [0.42.51.0] - 2026-06-17
|
||||
|
||||
**`gbrain sync` stops bottlenecking all its workers on a single database row, a malformed checkpoint can no longer wedge a source, and `gbrain doctor` tells an actively-running sync apart from a stuck one.** A slow source that fell behind HEAD could read as permanently stale even while it imported every cycle: sync was single-core-bound at the database layer, so handing it more workers didn't help, and the freshness check couldn't see that a sync was in fact running.
|
||||
|
||||
@@ -390,7 +390,7 @@ Progress banks into the append-only `op_checkpoint_paths` table (one row per dra
|
||||
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
|
||||
run resumes from the checkpoint and `last_commit` only advances on true completion. The
|
||||
per-source lock heartbeats through the direct pool and refuses to steal a live,
|
||||
recently-refreshed holder. Five env knobs tune it (all env-only, incident-time escape
|
||||
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
|
||||
hatches — no config-dashboard surface by design):
|
||||
|
||||
| Env var | Default | What it does |
|
||||
@@ -400,6 +400,7 @@ hatches — no config-dashboard surface by design):
|
||||
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
|
||||
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
|
||||
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
|
||||
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
|
||||
|
||||
## Pace Mode (DB-contention-aware backfill pacing)
|
||||
|
||||
|
||||
@@ -1,5 +1,26 @@
|
||||
# TODOS
|
||||
|
||||
## reliability fix-wave follow-ups (filed v0.42.52.0)
|
||||
|
||||
Deferred from the autopilot/supervisor + sync/status/minion reliability wave
|
||||
(plan-eng-review + codex + adversarial diff review CLEARED). Both surfaced by the
|
||||
ship-stage pre-landing review; neither blocks the wave.
|
||||
|
||||
- [ ] **P2 — Thread a cancellation signal through `importFile` (#1950).** The sync
|
||||
stall watchdog aborts `opts.signal`, but the per-iteration abort checks observe
|
||||
it BETWEEN files — a hang inside one `importFile` call (e.g. a stuck embed
|
||||
network request) isn't interrupted until that call returns. Thread an
|
||||
`AbortSignal` into `importFromContent`/`importFromFile` and check it at the async
|
||||
phase boundaries (post-parse, pre-embed, pre-DB-write) so an in-flight wedge is
|
||||
reaped too. Core hot path (engine-parity + downstream-client surface) — scope it
|
||||
on its own. Where: `src/core/import-file.ts`, `src/commands/sync.ts`.
|
||||
- [ ] **P3 — Centralize live-sync liveness onto `liveSyncStatus` (#1950).**
|
||||
`gbrain sources status` now uses the shared `liveSyncStatus(engine, sourceId)`
|
||||
helper; retrofit `gbrain doctor` (its own inline lock probe) and `gbrain status`
|
||||
onto the same helper so there's one source of truth for "is this source
|
||||
syncing." Where: `src/core/db-lock.ts`, `src/commands/doctor.ts`,
|
||||
`src/commands/status.ts`.
|
||||
|
||||
## Pace Mode follow-ups (filed v0.42.49.0)
|
||||
|
||||
Deferred from the paced-backfill wave (CEO + eng review CLEARED). Core shipped:
|
||||
|
||||
File diff suppressed because one or more lines are too long
+2
-1
@@ -539,7 +539,7 @@ Progress banks into the append-only `op_checkpoint_paths` table (one row per dra
|
||||
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
|
||||
run resumes from the checkpoint and `last_commit` only advances on true completion. The
|
||||
per-source lock heartbeats through the direct pool and refuses to steal a live,
|
||||
recently-refreshed holder. Five env knobs tune it (all env-only, incident-time escape
|
||||
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
|
||||
hatches — no config-dashboard surface by design):
|
||||
|
||||
| Env var | Default | What it does |
|
||||
@@ -549,6 +549,7 @@ hatches — no config-dashboard surface by design):
|
||||
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
|
||||
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
|
||||
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
|
||||
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
|
||||
|
||||
## Pace Mode (DB-contention-aware backfill pacing)
|
||||
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.51.0"
|
||||
"version": "0.42.52.0"
|
||||
}
|
||||
|
||||
+77
-18
@@ -74,8 +74,12 @@ SUBMITTING
|
||||
--follow Tail status until terminal (default on TTY)
|
||||
--detach Submit + print job id, exit immediately
|
||||
|
||||
Flags after \`run\` up to the first unrecognized token are parsed; the
|
||||
remainder is the prompt. Use \`--\` to explicitly terminate flag parsing.
|
||||
Flags before the prompt are parsed normally. The no-value switches
|
||||
--detach, --follow and --no-follow are ALSO recognized when they trail
|
||||
the prompt, so \`gbrain agent run "do X" --detach\` detaches. Any other
|
||||
--word is treated as prompt text (no error). Use \`--\` to end flag
|
||||
parsing and pass the rest verbatim:
|
||||
gbrain agent run -- "literally --detach this, with --flags"
|
||||
|
||||
VIEWING
|
||||
gbrain agent logs <job_id>
|
||||
@@ -102,31 +106,86 @@ interface RunFlags {
|
||||
detach: boolean;
|
||||
}
|
||||
|
||||
/** No-value switches that may also trail the prompt and get hoisted out (#1738). */
|
||||
const BOOLEAN_TAIL_FLAGS = new Set(['--follow', '--no-follow', '--detach']);
|
||||
|
||||
function applyBooleanFlag(flags: RunFlags, a: string): void {
|
||||
if (a === '--follow') flags.follow = true;
|
||||
else if (a === '--no-follow') flags.follow = false;
|
||||
else { flags.detach = true; flags.follow = false; } // --detach
|
||||
}
|
||||
|
||||
/** Read the value for a value-flag, rejecting a missing or flag-shaped value. */
|
||||
function requireFlagValue(args: string[], i: number, flag: string): string {
|
||||
const v = args[i];
|
||||
if (v === undefined || v.startsWith('--')) {
|
||||
throw new Error(`gbrain agent run: ${flag} requires a value. Run \`gbrain agent run --help\`.`);
|
||||
}
|
||||
return v;
|
||||
}
|
||||
|
||||
function parseIntFlagValue(v: string, flag: string): number {
|
||||
const n = parseInt(v, 10);
|
||||
if (Number.isNaN(n)) {
|
||||
throw new Error(`gbrain agent run: ${flag} expects a number, got "${v}".`);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse `agent run` args into flags + prompt (#1738).
|
||||
*
|
||||
* args ──► [ leading flag zone ] [ ── ? ] [ prompt … (trailing booleans) ]
|
||||
*
|
||||
* Leading zone: known flags (value + boolean) are consumed left-to-right until
|
||||
* the first positional token, an UNKNOWN --flag, or an explicit `--`. An
|
||||
* unknown --flag is NOT an error — it begins the freeform prompt, so
|
||||
* `agent run "--note: do X"` works without `--`. Value-flags missing their
|
||||
* value throw a usage error instead of silently capturing `undefined`/`NaN`.
|
||||
*
|
||||
* Prompt zone: a trailing run of the no-value switches (--detach/--follow/
|
||||
* --no-follow) is hoisted out so `agent run "do X" --detach` detaches. Only
|
||||
* trailing switches are hoisted; a `--word` elsewhere in the prompt stays
|
||||
* verbatim. After an explicit `--`, nothing is hoisted.
|
||||
*/
|
||||
function parseRunFlags(args: string[]): { flags: RunFlags; rest: string[] } {
|
||||
const flags: RunFlags = {
|
||||
follow: process.stdout.isTTY === true,
|
||||
detach: false,
|
||||
};
|
||||
let i = 0;
|
||||
while (i < args.length) {
|
||||
const a = args[i];
|
||||
if (a === '--') { i++; break; }
|
||||
if (!isKnownFlag(a!)) break;
|
||||
let escaped = false;
|
||||
for (; i < args.length; i++) {
|
||||
const a = args[i]!;
|
||||
if (a === '--') { i++; escaped = true; break; }
|
||||
if (!a.startsWith('--')) break;
|
||||
let known = true;
|
||||
switch (a) {
|
||||
case '--subagent-def': flags.subagentDef = args[++i]; i++; break;
|
||||
case '--model': flags.model = args[++i]; i++; break;
|
||||
case '--max-turns': flags.maxTurns = parseInt(args[++i] ?? '', 10); i++; break;
|
||||
case '--tools': flags.tools = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean); i++; break;
|
||||
case '--timeout-ms': flags.timeoutMs = parseInt(args[++i] ?? '', 10); i++; break;
|
||||
case '--fanout-manifest': flags.fanoutManifest = args[++i]; i++; break;
|
||||
case '--follow': flags.follow = true; i++; break;
|
||||
case '--no-follow': flags.follow = false; i++; break;
|
||||
case '--detach': flags.detach = true; flags.follow = false; i++; break;
|
||||
default:
|
||||
throw new Error(`unknown flag: ${a}. Run \`gbrain agent run --help\` for usage.`);
|
||||
case '--subagent-def': flags.subagentDef = requireFlagValue(args, ++i, a); break;
|
||||
case '--model': flags.model = requireFlagValue(args, ++i, a); break;
|
||||
case '--max-turns': flags.maxTurns = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
|
||||
case '--tools': flags.tools = requireFlagValue(args, ++i, a).split(',').map(s => s.trim()).filter(Boolean); break;
|
||||
case '--timeout-ms': flags.timeoutMs = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
|
||||
case '--fanout-manifest': flags.fanoutManifest = requireFlagValue(args, ++i, a); break;
|
||||
case '--follow': flags.follow = true; break;
|
||||
case '--no-follow': flags.follow = false; break;
|
||||
case '--detach': flags.detach = true; flags.follow = false; break;
|
||||
default: known = false; break;
|
||||
}
|
||||
if (!known) break; // unknown --flag → first token of the (freeform) prompt
|
||||
}
|
||||
const rest = args.slice(i);
|
||||
// An explicit `--` terminates flag parsing wherever it appears — leading
|
||||
// zone (escaped) OR after a positional (the leading loop breaks before it,
|
||||
// so `escaped` stays false). Honor both: when the prompt carries a literal
|
||||
// `--`, hoist nothing, so `agent run note -- --detach` keeps `--detach`
|
||||
// verbatim instead of silently flipping detach mode.
|
||||
if (!escaped && !rest.includes('--')) {
|
||||
while (rest.length > 0 && BOOLEAN_TAIL_FLAGS.has(rest[rest.length - 1]!)) {
|
||||
applyBooleanFlag(flags, rest.pop()!);
|
||||
}
|
||||
}
|
||||
return { flags, rest: args.slice(i) };
|
||||
return { flags, rest };
|
||||
}
|
||||
|
||||
export async function runAgentRun(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
|
||||
@@ -32,9 +32,26 @@
|
||||
|
||||
import type { BrainEngine, SourceRow } from '../core/engine.ts';
|
||||
import type { MinionQueue } from '../core/minions/queue.ts';
|
||||
import { NON_GLOBAL_PHASES, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY } from '../core/cycle.ts';
|
||||
|
||||
const FULL_CYCLE_FLOOR_MIN = 60;
|
||||
|
||||
// #2194 fix #2: failure cooldown. A source whose autopilot-cycle keeps
|
||||
// failing/timing-out re-dispatches every tick today (only SUCCESS gates
|
||||
// dispatch), so the same handful of sources fail and re-fan-out forever — the
|
||||
// self-perpetuating dead-job storm. Back a failed source off with bounded
|
||||
// exponential cooldown so a chronically-slow source can't re-dispatch every
|
||||
// tick. Disabled with autopilot.failure_cooldown_min=0.
|
||||
const FAILURE_COOLDOWN_BASE_MIN = 10;
|
||||
const FAILURE_COOLDOWN_CAP_MIN = 120;
|
||||
const FAILURE_COOLDOWN_EXP_CAP = 4; // 2^4 = 16× base before the cap clamps
|
||||
|
||||
/** Recent-failure record for one source (from minion_jobs dead/failed rows). */
|
||||
export interface SourceFailure { count: number; lastFailedAt: Date; }
|
||||
|
||||
/** Resolved cooldown knobs. baseMin <= 0 means the cooldown is disabled. */
|
||||
export interface CooldownOpts { baseMin: number; capMin: number; }
|
||||
|
||||
export interface FanoutOpts {
|
||||
repoPath: string;
|
||||
slot: string;
|
||||
@@ -58,6 +75,8 @@ export interface FanoutResult {
|
||||
skipped_fresh: string[];
|
||||
/** Source ids beyond the fanoutMax cap (will retry next tick). */
|
||||
skipped_cap: string[];
|
||||
/** Source ids skipped because they're in failure cooldown (#2194 fix #2). */
|
||||
skipped_cooldown: string[];
|
||||
/** True when this tick fell back to the legacy single-job path
|
||||
* (no sources rows / engine empty). */
|
||||
legacy_fallback: boolean;
|
||||
@@ -83,6 +102,62 @@ export async function resolveFanoutMax(engine: BrainEngine): Promise<number> {
|
||||
return engine.kind === 'pglite' ? 1 : 4;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the worker concurrency the supervisor most recently STARTED with, from
|
||||
* its `started` audit event (the lowest-coupling source — no extra lock-row
|
||||
* column). Filesystem read; returns null when no supervisor has ever started
|
||||
* (or the event lacks concurrency). Filtered by queue so a `shell`-queue
|
||||
* supervisor's concurrency doesn't leak into the `default`-queue decision.
|
||||
*
|
||||
* ADVISORY use only (doctor warning). Behavior-changing callers (the fanout
|
||||
* clamp) must additionally gate on a LIVE supervisor — see
|
||||
* resolveEffectiveFanoutMax — because a stale `started` row can otherwise
|
||||
* shrink fan-out for a supervisor that isn't running that config (codex #9/D5).
|
||||
*/
|
||||
export async function readSupervisorConcurrency(queue = 'default'): Promise<number | null> {
|
||||
try {
|
||||
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
|
||||
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
|
||||
const started = events
|
||||
.filter((e) => e.event === 'started' && (e.queue === undefined || e.queue === queue))
|
||||
.pop();
|
||||
const c = started?.concurrency;
|
||||
return typeof c === 'number' && Number.isFinite(c) ? c : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve fanoutMax CLAMPED to the worker's effective concurrency (#2194 fix #1).
|
||||
*
|
||||
* Fanning out more cycles than the worker can run guarantees waiters that then
|
||||
* race the stalled-sweeper. Clamp to `max(1, concurrency - 1)` — reserving ≥1
|
||||
* slot for targeted sync/embed jobs that share the `default` queue.
|
||||
*
|
||||
* codex #9 / D5: the clamp is BEHAVIOR-changing, so it trusts only a
|
||||
* proven-alive supervisor (live DB-lock holder, `ttl_expires_at`-gated). With
|
||||
* no live holder the concurrency is UNKNOWN and we fall back to the unclamped
|
||||
* default (4 pg / 1 pglite) — the safe direction (never starve on stale data).
|
||||
* Operators can disable the clamp via `autopilot.fanout_clamp_to_concurrency`.
|
||||
*/
|
||||
export async function resolveEffectiveFanoutMax(engine: BrainEngine, queue = 'default'): Promise<number> {
|
||||
const base = await resolveFanoutMax(engine);
|
||||
const clampCfg = await engine.getConfig('autopilot.fanout_clamp_to_concurrency');
|
||||
if (clampCfg === 'false' || clampCfg === '0') return base; // operator opt-out
|
||||
try {
|
||||
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
|
||||
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
|
||||
const snap = await inspectLock(engine, supervisorLockId(queue));
|
||||
if (!snap || !isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) return base; // no live holder → unknown → no clamp
|
||||
const concurrency = await readSupervisorConcurrency(queue);
|
||||
if (concurrency === null) return base;
|
||||
return Math.max(1, Math.min(base, concurrency - 1));
|
||||
} catch {
|
||||
return base;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read `last_full_cycle_at` ISO string from a source's config JSONB.
|
||||
* Returns null when missing or unparseable. Pure function over the row
|
||||
@@ -111,6 +186,133 @@ export function isSourceStale(src: SourceRow, now = Date.now(), floorMin = FULL_
|
||||
return ageMin >= floorMin;
|
||||
}
|
||||
|
||||
/**
|
||||
* Most recent SUCCESSFUL cycle for a source. Prefers `last_source_cycle_at`
|
||||
* (per-source phases, written by the split cycle) and falls back to the legacy
|
||||
* `last_full_cycle_at`, so this works before AND after the cycle split.
|
||||
*/
|
||||
export function readLastSuccessAt(src: SourceRow): Date | null {
|
||||
const c = src.config ?? {};
|
||||
const raw = (typeof c.last_source_cycle_at === 'string' && c.last_source_cycle_at)
|
||||
|| (typeof c.last_full_cycle_at === 'string' && c.last_full_cycle_at)
|
||||
|| null;
|
||||
if (!raw) return null;
|
||||
const d = new Date(raw);
|
||||
return Number.isFinite(d.getTime()) ? d : null;
|
||||
}
|
||||
|
||||
/** Bounded exponential cooldown window (minutes) for a given failure count. */
|
||||
export function cooldownMinForCount(count: number, opts: CooldownOpts): number {
|
||||
if (count <= 0 || opts.baseMin <= 0) return 0;
|
||||
const mult = Math.pow(2, Math.min(count - 1, FAILURE_COOLDOWN_EXP_CAP));
|
||||
return Math.min(opts.baseMin * mult, opts.capMin);
|
||||
}
|
||||
|
||||
/**
|
||||
* Is a source currently in failure cooldown? Pure — drives both the dispatch
|
||||
* gate and the claim-time guard. A SUCCESS at-or-after the most recent failure
|
||||
* clears the cooldown (codex #7: operator repair / manual cycle re-eligibility),
|
||||
* so a recovered source is never suppressed by stale failure history.
|
||||
*/
|
||||
export function isInFailureCooldown(
|
||||
failure: SourceFailure | undefined,
|
||||
lastSuccessAt: Date | null,
|
||||
now: number,
|
||||
opts: CooldownOpts,
|
||||
): boolean {
|
||||
if (opts.baseMin <= 0) return false; // disabled
|
||||
if (!failure || failure.count <= 0) return false;
|
||||
if (lastSuccessAt && lastSuccessAt.getTime() >= failure.lastFailedAt.getTime()) return false;
|
||||
const cooldownMs = cooldownMinForCount(failure.count, opts) * 60_000;
|
||||
return (now - failure.lastFailedAt.getTime()) < cooldownMs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve cooldown knobs from config. `autopilot.failure_cooldown_min` overrides
|
||||
* the base (0 = disable entirely — exactly today's behavior);
|
||||
* `autopilot.failure_cooldown_cap_min` overrides the ceiling.
|
||||
*/
|
||||
export async function resolveFailureCooldownOpts(engine: BrainEngine): Promise<CooldownOpts> {
|
||||
let baseMin = FAILURE_COOLDOWN_BASE_MIN;
|
||||
let capMin = FAILURE_COOLDOWN_CAP_MIN;
|
||||
const baseCfg = await engine.getConfig('autopilot.failure_cooldown_min');
|
||||
if (baseCfg !== null && baseCfg !== undefined && baseCfg !== '') {
|
||||
const n = parseInt(baseCfg, 10);
|
||||
if (Number.isFinite(n) && n >= 0) baseMin = n;
|
||||
}
|
||||
const capCfg = await engine.getConfig('autopilot.failure_cooldown_cap_min');
|
||||
if (capCfg) {
|
||||
const n = parseInt(capCfg, 10);
|
||||
if (Number.isFinite(n) && n >= 1) capMin = n;
|
||||
}
|
||||
return { baseMin, capMin };
|
||||
}
|
||||
|
||||
/**
|
||||
* Read recent dead/failed autopilot-cycle jobs grouped by source. Read-at-
|
||||
* dispatch (NOT a write hook) because timeouts/RSS-kills/stalls dead-letter via
|
||||
* SQL in queue.ts and never run handler code — a write-only cooldown would miss
|
||||
* the exact failures that drive the storm. Engine-parity-safe via executeRaw
|
||||
* (one query, both engines); cutoff is precomputed in JS to avoid INTERVAL
|
||||
* portability concerns. codex #6: rows with a null source_id are excluded.
|
||||
*/
|
||||
export async function readRecentSourceFailures(
|
||||
engine: BrainEngine,
|
||||
opts: { sinceMin?: number; sourceId?: string } = {},
|
||||
): Promise<Map<string, SourceFailure>> {
|
||||
const sinceMin = opts.sinceMin ?? FAILURE_COOLDOWN_CAP_MIN;
|
||||
const cutoff = new Date(Date.now() - sinceMin * 60_000).toISOString();
|
||||
const map = new Map<string, SourceFailure>();
|
||||
try {
|
||||
const params: unknown[] = [cutoff];
|
||||
let sql =
|
||||
`SELECT data->>'source_id' AS source_id,
|
||||
count(*)::int AS fail_count,
|
||||
max(finished_at) AS last_failed_at
|
||||
FROM minion_jobs
|
||||
WHERE name = 'autopilot-cycle'
|
||||
AND status IN ('dead','failed')
|
||||
AND data->>'source_id' IS NOT NULL
|
||||
AND finished_at IS NOT NULL
|
||||
AND finished_at > $1`;
|
||||
if (opts.sourceId) { params.push(opts.sourceId); sql += ` AND data->>'source_id' = $${params.length}`; }
|
||||
sql += ` GROUP BY data->>'source_id'`;
|
||||
const rows = await engine.executeRaw<{ source_id: string | null; fail_count: number; last_failed_at: string | Date }>(sql, params);
|
||||
for (const r of rows) {
|
||||
if (!r.source_id) continue; // codex #6 null-source guard (defensive)
|
||||
const last = r.last_failed_at instanceof Date ? r.last_failed_at : new Date(r.last_failed_at);
|
||||
if (!Number.isFinite(last.getTime())) continue;
|
||||
map.set(r.source_id, { count: Number(r.fail_count) || 0, lastFailedAt: last });
|
||||
}
|
||||
} catch {
|
||||
// Pre-migration / transient DB error → no cooldown data (fail open: dispatch).
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
/**
|
||||
* Claim-time cooldown guard (codex #5 / D4): a job already queued or retrying
|
||||
* (max_attempts:2) can reach the worker after the dispatch gate decided. The
|
||||
* handler calls this immediately before runCycle; an in-cooldown claim becomes
|
||||
* a no-op skip (NOT a failure — it must not re-arm the cooldown). Shares the
|
||||
* exact cooldown math with the dispatch gate (DRY).
|
||||
*/
|
||||
export async function isSourceInCooldown(engine: BrainEngine, sourceId: string, now = Date.now()): Promise<boolean> {
|
||||
const opts = await resolveFailureCooldownOpts(engine);
|
||||
if (opts.baseMin <= 0) return false;
|
||||
const failures = await readRecentSourceFailures(engine, { sinceMin: opts.capMin, sourceId });
|
||||
const failure = failures.get(sourceId);
|
||||
if (!failure) return false;
|
||||
let lastSuccessAt: Date | null = null;
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ config: Record<string, unknown> | null }>(
|
||||
`SELECT config FROM sources WHERE id = $1`, [sourceId],
|
||||
);
|
||||
if (rows[0]) lastSuccessAt = readLastSuccessAt({ config: rows[0].config ?? {} } as SourceRow);
|
||||
} catch { /* treat as no success */ }
|
||||
return isInFailureCooldown(failure, lastSuccessAt, now, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide which sources to dispatch this tick. Pure function so tests can
|
||||
* exercise the freshness gate + cap math without an engine.
|
||||
@@ -126,11 +328,21 @@ export function selectSourcesForDispatch(
|
||||
fanoutMax: number,
|
||||
now = Date.now(),
|
||||
floorMin = FULL_CYCLE_FLOOR_MIN,
|
||||
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[] } {
|
||||
recentFailures: Map<string, SourceFailure> = new Map(),
|
||||
cooldownOpts: CooldownOpts = { baseMin: FAILURE_COOLDOWN_BASE_MIN, capMin: FAILURE_COOLDOWN_CAP_MIN },
|
||||
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[]; skippedCooldown: SourceRow[] } {
|
||||
const stale: SourceRow[] = [];
|
||||
const fresh: SourceRow[] = [];
|
||||
const cooldown: SourceRow[] = [];
|
||||
for (const s of sources) {
|
||||
(isSourceStale(s, now, floorMin) ? stale : fresh).push(s);
|
||||
if (!isSourceStale(s, now, floorMin)) { fresh.push(s); continue; }
|
||||
// #2194 fix #2: a stale source that recently failed is held in cooldown so
|
||||
// it can't re-dispatch every tick (the storm). Success clears it.
|
||||
if (isInFailureCooldown(recentFailures.get(s.id), readLastSuccessAt(s), now, cooldownOpts)) {
|
||||
cooldown.push(s);
|
||||
continue;
|
||||
}
|
||||
stale.push(s);
|
||||
}
|
||||
// Oldest-first ordering: NULL last_full_cycle_at sorts before any timestamp.
|
||||
stale.sort((a, b) => {
|
||||
@@ -141,7 +353,7 @@ export function selectSourcesForDispatch(
|
||||
});
|
||||
const dispatch = stale.slice(0, fanoutMax);
|
||||
const skippedCap = stale.slice(fanoutMax);
|
||||
return { dispatch, skippedFresh: fresh, skippedCap };
|
||||
return { dispatch, skippedFresh: fresh, skippedCap, skippedCooldown: cooldown };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -193,10 +405,27 @@ export async function dispatchPerSource(
|
||||
} else {
|
||||
log(`[dispatch] job #${job.id} autopilot-cycle (legacy single-source)`);
|
||||
}
|
||||
return { dispatched: [], skipped_fresh: [], skipped_cap: [], legacy_fallback: true };
|
||||
return { dispatched: [], skipped_fresh: [], skipped_cap: [], skipped_cooldown: [], legacy_fallback: true };
|
||||
}
|
||||
|
||||
const { dispatch, skippedFresh, skippedCap } = selectSourcesForDispatch(sources, opts.fanoutMax);
|
||||
// #2194 fix #2: load recent per-source failures + cooldown knobs so a
|
||||
// chronically-failing source is backed off instead of re-dispatched every
|
||||
// tick. Fail-open: cooldown is an optimization, not a correctness gate — if
|
||||
// config/job-history reads fail (or the engine lacks them), dispatch proceeds
|
||||
// with no cooldown rather than blocking.
|
||||
let cooldownOpts: CooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
|
||||
let recentFailures = new Map<string, SourceFailure>();
|
||||
try {
|
||||
cooldownOpts = await resolveFailureCooldownOpts(engine);
|
||||
if (cooldownOpts.baseMin > 0) {
|
||||
recentFailures = await readRecentSourceFailures(engine, { sinceMin: cooldownOpts.capMin });
|
||||
}
|
||||
} catch {
|
||||
cooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
|
||||
}
|
||||
|
||||
const { dispatch, skippedFresh, skippedCap, skippedCooldown } =
|
||||
selectSourcesForDispatch(sources, opts.fanoutMax, Date.now(), FULL_CYCLE_FLOOR_MIN, recentFailures, cooldownOpts);
|
||||
|
||||
const dispatched: string[] = [];
|
||||
for (const src of dispatch) {
|
||||
@@ -208,6 +437,11 @@ export async function dispatchPerSource(
|
||||
repoPath: opts.repoPath,
|
||||
source_id: src.id,
|
||||
pull: !!remoteUrl,
|
||||
// #2194 fix #3 (cycle split): per-source cycles run ONLY source-scoped
|
||||
// (+ mixed) phases. The brain-wide global phases (embed, orphans,
|
||||
// purge, …) run once in autopilot-global-maintenance, not N times
|
||||
// concurrently here — the fix for the 4→10GB RSS blowout.
|
||||
phases: NON_GLOBAL_PHASES,
|
||||
},
|
||||
{
|
||||
queue: 'default',
|
||||
@@ -261,10 +495,77 @@ export async function dispatchPerSource(
|
||||
}));
|
||||
}
|
||||
|
||||
if (skippedCooldown.length > 0 && opts.jsonMode) {
|
||||
emit(JSON.stringify({
|
||||
event: 'fanout_cooldown_skipped',
|
||||
sources: skippedCooldown.map(s => s.id),
|
||||
}));
|
||||
}
|
||||
|
||||
return {
|
||||
dispatched,
|
||||
skipped_fresh: skippedFresh.map(s => s.id),
|
||||
skipped_cap: skippedCap.map(s => s.id),
|
||||
skipped_cooldown: skippedCooldown.map(s => s.id),
|
||||
legacy_fallback: false,
|
||||
};
|
||||
}
|
||||
|
||||
const GLOBAL_FLOOR_MIN = 60;
|
||||
|
||||
/** Is the brain-wide maintenance overdue? Null/unparseable → overdue. */
|
||||
export function isGlobalMaintenanceStale(lastGlobalAtIso: string | null, now = Date.now(), floorMin = GLOBAL_FLOOR_MIN): boolean {
|
||||
if (!lastGlobalAtIso) return true;
|
||||
const d = new Date(lastGlobalAtIso);
|
||||
if (!Number.isFinite(d.getTime())) return true;
|
||||
return (now - d.getTime()) / 60_000 >= floorMin;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2194 fix #3 / #2227 bug #3 — dispatch the single brain-wide maintenance job
|
||||
* that runs the `global` cycle phases (embed, orphans, purge, …) ONCE per
|
||||
* window, instead of N per-source cycles each running them concurrently (the
|
||||
* RSS blowout). Single-flight is structural: one `idempotency_key` +
|
||||
* `maxWaiting:1`, so a slow run never stacks. Gated on `autopilot.last_global_at`
|
||||
* (stamped by the handler on success). Postgres-only fan-out concern; on PGLite
|
||||
* the file lock already serializes, but the job is still correct there.
|
||||
*/
|
||||
export async function dispatchGlobalMaintenance(
|
||||
engine: BrainEngine,
|
||||
queue: MinionQueue,
|
||||
opts: { repoPath: string; slot: string; timeoutMs: number; jsonMode: boolean; emit?: (l: string) => void; log?: (l: string) => void },
|
||||
): Promise<{ dispatched: boolean; reason: 'stale' | 'fresh' }> {
|
||||
const emit = opts.emit ?? ((line) => process.stderr.write(line + '\n'));
|
||||
const log = opts.log ?? ((line) => console.log(line));
|
||||
|
||||
let floorMin = GLOBAL_FLOOR_MIN;
|
||||
const floorCfg = await engine.getConfig('autopilot.global_floor_min');
|
||||
if (floorCfg) {
|
||||
const n = parseInt(floorCfg, 10);
|
||||
if (Number.isFinite(n) && n >= 1) floorMin = n;
|
||||
}
|
||||
const lastGlobalAt = await engine.getConfig(LAST_GLOBAL_AT_KEY);
|
||||
if (!isGlobalMaintenanceStale(lastGlobalAt, Date.now(), floorMin)) {
|
||||
return { dispatched: false, reason: 'fresh' };
|
||||
}
|
||||
|
||||
const job = await queue.add(
|
||||
'autopilot-global-maintenance',
|
||||
{ repoPath: opts.repoPath, phases: GLOBAL_PHASES },
|
||||
{
|
||||
queue: 'default',
|
||||
// Structural single-flight: one global job per slot; maxWaiting:1 coalesces
|
||||
// any surplus so a slow brain-wide pass never stacks duplicates.
|
||||
idempotency_key: `autopilot-global:${opts.slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: opts.timeoutMs,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'global_maintenance', slot: opts.slot }));
|
||||
} else {
|
||||
log(`[dispatch] job #${job.id} autopilot-global-maintenance (brain-wide phases)`);
|
||||
}
|
||||
return { dispatched: true, reason: 'stale' };
|
||||
}
|
||||
|
||||
@@ -871,8 +871,12 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// codex P1-3). Fresh-install brains with no sources rows fall
|
||||
// back to the legacy single autopilot-cycle so existing
|
||||
// behavior is preserved.
|
||||
const { dispatchPerSource, resolveFanoutMax } = await import('./autopilot-fanout.ts');
|
||||
const fanoutMax = await resolveFanoutMax(engine);
|
||||
const { dispatchPerSource, dispatchGlobalMaintenance, resolveEffectiveFanoutMax } = await import('./autopilot-fanout.ts');
|
||||
// #2194 fix #1: clamp fan-out to the worker's effective concurrency
|
||||
// (reserve ≥1 slot), gated on a LIVE supervisor so a stale audit row
|
||||
// can't shrink throughput (codex #9/D5). autopilot-cycle jobs run on
|
||||
// the 'default' queue, so that's the concurrency we compare against.
|
||||
const fanoutMax = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
const result = await dispatchPerSource(engine, queue, {
|
||||
repoPath,
|
||||
slot,
|
||||
@@ -880,6 +884,18 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
fanoutMax,
|
||||
jsonMode,
|
||||
});
|
||||
// #2194 fix #3 / #2227 bug #3: dispatch the single brain-wide
|
||||
// maintenance job (embed/orphans/purge/…) once per window — the per-
|
||||
// source cycles above no longer run global phases, so this is where
|
||||
// the brain-wide work happens (single-flight, no RSS blowout). Only on
|
||||
// the per-source path (legacy single-source still runs everything).
|
||||
if (!result.legacy_fallback) {
|
||||
try {
|
||||
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs, jsonMode });
|
||||
} catch (e) {
|
||||
if (jsonMode) process.stderr.write(JSON.stringify({ event: 'global_maintenance_dispatch_failed', error: e instanceof Error ? e.message : String(e) }) + '\n');
|
||||
}
|
||||
}
|
||||
if (result.dispatched.length > 0 || result.legacy_fallback) {
|
||||
lastFullCycleAt = Date.now();
|
||||
}
|
||||
@@ -889,6 +905,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
dispatched: result.dispatched,
|
||||
skipped_fresh: result.skipped_fresh,
|
||||
skipped_cap: result.skipped_cap,
|
||||
skipped_cooldown: result.skipped_cooldown,
|
||||
legacy_fallback: result.legacy_fallback,
|
||||
fanout_max: fanoutMax,
|
||||
score,
|
||||
@@ -896,7 +913,8 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
} else if (!result.legacy_fallback) {
|
||||
console.log(
|
||||
`[dispatch] fanout: ${result.dispatched.length} dispatched, ` +
|
||||
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped ` +
|
||||
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped, ` +
|
||||
`${result.skipped_cooldown.length} cooldown ` +
|
||||
`(score=${score}, max=${fanoutMax})`,
|
||||
);
|
||||
}
|
||||
|
||||
+66
-2
@@ -695,6 +695,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// issue #1801 — wedged_queue (cross-surface parity with buildChecks).
|
||||
checks.push(await computeWedgedQueueCheck(engine));
|
||||
|
||||
// #2194 fix #5 — warn when autopilot fan-out exceeds worker concurrency.
|
||||
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
|
||||
|
||||
// v0.41 Bug 2 / Eng D8 — subagent_health surfaces rate-lease pressure to the operator.
|
||||
checks.push(await checkSubagentHealth(engine));
|
||||
|
||||
@@ -1563,6 +1566,49 @@ export async function computeWedgedQueueCheck(engine: BrainEngine): Promise<Chec
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2194 fix #5: warn when autopilot's per-tick fan-out exceeds the worker's
|
||||
* effective concurrency. Fanning out more cycles than there are worker slots
|
||||
* guarantees waiters that race the stalled-sweeper — a silent misconfig today.
|
||||
* Advisory (started-event concurrency is fine here; the behavior-changing clamp
|
||||
* in resolveEffectiveFanoutMax is the one that gates on liveness). Surfaces only
|
||||
* when a supervisor has actually started (no noise on never-supervised brains).
|
||||
*/
|
||||
export async function computeAutopilotFanoutConcurrencyCheck(engine: BrainEngine): Promise<Check> {
|
||||
if (engine.kind !== 'postgres') {
|
||||
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'PGLite — single-writer, fan-out is 1' };
|
||||
}
|
||||
try {
|
||||
const { resolveFanoutMax, readSupervisorConcurrency } = await import('./autopilot-fanout.ts');
|
||||
const concurrency = await readSupervisorConcurrency('default');
|
||||
if (concurrency === null) {
|
||||
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'No supervisor observed — skipping fan-out/concurrency check' };
|
||||
}
|
||||
const fanoutMax = await resolveFanoutMax(engine);
|
||||
const effectiveSlots = Math.max(1, concurrency - 1);
|
||||
if (fanoutMax > effectiveSlots) {
|
||||
return {
|
||||
name: 'autopilot_fanout_concurrency',
|
||||
status: 'warn',
|
||||
message:
|
||||
`autopilot fan-out (${fanoutMax}/tick) exceeds worker concurrency (${concurrency}). ` +
|
||||
`Surplus cycles queue behind the worker and race the stalled-sweeper. ` +
|
||||
`Lower fan-out: \`gbrain config set autopilot.fanout_max_per_tick ${effectiveSlots}\`, ` +
|
||||
`or raise the supervisor's \`--concurrency\` to ${fanoutMax + 1}. ` +
|
||||
`(The clamp in autopilot does this automatically unless disabled.)`,
|
||||
details: { fanout_max: fanoutMax, concurrency, effective_slots: effectiveSlots },
|
||||
};
|
||||
}
|
||||
return {
|
||||
name: 'autopilot_fanout_concurrency',
|
||||
status: 'ok',
|
||||
message: `fan-out ${fanoutMax}/tick within worker concurrency ${concurrency}`,
|
||||
};
|
||||
} catch (e) {
|
||||
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: `Skipped (${e instanceof Error ? e.message : String(e)})` };
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkBatchRetryHealth(_engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
// Codex M-10: surface bad env config at doctor time.
|
||||
@@ -4348,7 +4394,22 @@ export async function buildChecks(
|
||||
|
||||
const pidStatus = readSupervisorPid(DEFAULT_PID_FILE);
|
||||
const supervisorPid = pidStatus.pid;
|
||||
const running = pidStatus.running;
|
||||
const pidfileRunning = pidStatus.running;
|
||||
|
||||
// issue #2227 fix #1/#3: DEFAULT_PID_FILE is HOME-derived, so a supervisor
|
||||
// started under a different $HOME reads as "not running" even when healthy.
|
||||
// Consult the queue-scoped DB singleton lock (#1849, HOME-independent) before
|
||||
// warning. PID-reuse-safe (isLockHolderLive keys on lock freshness).
|
||||
let detectedViaDbLock = false;
|
||||
if (!pidfileRunning && engine) {
|
||||
try {
|
||||
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
|
||||
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
|
||||
const snap = await inspectLock(engine, supervisorLockId('default'));
|
||||
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) detectedViaDbLock = true;
|
||||
} catch { /* pre-migration / transient: pidfile-only */ }
|
||||
}
|
||||
const running = pidfileRunning || detectedViaDbLock;
|
||||
|
||||
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
|
||||
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
|
||||
@@ -4396,7 +4457,7 @@ export async function buildChecks(
|
||||
checks.push({
|
||||
name: 'supervisor',
|
||||
status: 'ok',
|
||||
message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
|
||||
message: `running=true${detectedViaDbLock ? ' (detected via DB lock; pidfile not at the HOME-derived path)' : ` pid=${supervisorPid}`} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -7171,6 +7232,9 @@ export async function buildChecks(
|
||||
// waiting, zero live-lock active, stale completions) as a health error.
|
||||
progress.heartbeat('wedged_queue');
|
||||
checks.push(await computeWedgedQueueCheck(engine));
|
||||
// #2194 fix #5 — autopilot fan-out vs worker concurrency mismatch.
|
||||
progress.heartbeat('autopilot_fanout_concurrency');
|
||||
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
|
||||
// v0.40.4 graph_signals_coverage — global inbound-link density when
|
||||
// graph_signals is enabled in the active mode bundle.
|
||||
progress.heartbeat('graph_signals_coverage');
|
||||
|
||||
+119
-9
@@ -1020,10 +1020,38 @@ HANDLER TYPES (built in)
|
||||
|
||||
const pidStatus = readSupervisorPid(pidFile);
|
||||
const supervisorPid = pidStatus.pid;
|
||||
const running = pidStatus.running;
|
||||
const pidfileRunning = pidStatus.running;
|
||||
|
||||
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
|
||||
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
|
||||
|
||||
// issue #2227 fix #1/#3: the pidfile is HOME-derived, so a supervisor
|
||||
// started under a different $HOME (keeper=/root vs ops=/data) reads as
|
||||
// "not running" here even when it is healthy — the false signal that
|
||||
// makes an operator spawn a duplicate. Fall back to the queue-scoped DB
|
||||
// singleton lock (#1849), the HOME-independent authority. PID-reuse-safe:
|
||||
// isLockHolderLive keys on lock freshness, never process.kill.
|
||||
const supQueue = parseFlag(args, '--queue') ?? 'default';
|
||||
let detectedViaDbLock = false;
|
||||
let dbLockHolder: { holder_pid: number; holder_host: string } | null = null;
|
||||
if (!pidfileRunning) {
|
||||
try {
|
||||
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
|
||||
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
|
||||
const snap = await inspectLock(engine, supervisorLockId(supQueue));
|
||||
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) {
|
||||
detectedViaDbLock = true;
|
||||
dbLockHolder = { holder_pid: snap.holder_pid, holder_host: snap.holder_host };
|
||||
}
|
||||
} catch {
|
||||
// Pre-migration brains / transient DB errors: fall back to pidfile-only.
|
||||
}
|
||||
}
|
||||
const running = pidfileRunning || detectedViaDbLock;
|
||||
// Surface the supervisor's recorded config from the latest `started`
|
||||
// event (concurrency + effective --max-rss) so split-$HOME deployments
|
||||
// see what the live-but-pidfile-invisible supervisor is running.
|
||||
const startedEvt = events.filter(e => e.event === 'started').pop() ?? null;
|
||||
// Shared classifier — same code path runs in `gbrain doctor` so the
|
||||
// two surfaces cannot drift on what counts as a crash. Supersedes
|
||||
// v0.35.4.0's binary `classifyWorkerExit({code})` on this surface;
|
||||
@@ -1038,15 +1066,20 @@ HANDLER TYPES (built in)
|
||||
nice_requested: w.nice_requested,
|
||||
nice: w.nice_now,
|
||||
}));
|
||||
const supervisorNice = running && supervisorPid !== null
|
||||
const supervisorNice = pidfileRunning && supervisorPid !== null
|
||||
? getEffectiveNiceness(supervisorPid)
|
||||
: null;
|
||||
|
||||
const status = {
|
||||
running,
|
||||
supervisor_pid: supervisorPid,
|
||||
detected_via: detectedViaDbLock ? 'db_lock' : (pidfileRunning ? 'pidfile' : null),
|
||||
supervisor_pid: supervisorPid ?? dbLockHolder?.holder_pid ?? null,
|
||||
db_lock_holder: dbLockHolder,
|
||||
pid_file: pidFile,
|
||||
queue: supQueue,
|
||||
last_start: lastStart,
|
||||
concurrency: typeof startedEvt?.concurrency === 'number' ? startedEvt.concurrency : null,
|
||||
max_rss_mb: typeof startedEvt?.max_rss_mb === 'number' ? startedEvt.max_rss_mb : null,
|
||||
crashes_24h: summary.total,
|
||||
clean_exits_24h: summary.clean_exits,
|
||||
crashes_by_cause: summary.by_cause,
|
||||
@@ -1058,9 +1091,11 @@ HANDLER TYPES (built in)
|
||||
if (jsonMode) {
|
||||
console.log(JSON.stringify(status, null, 2));
|
||||
} else {
|
||||
console.log(`Supervisor: ${running ? 'running' : 'not running'}`);
|
||||
if (supervisorPid) console.log(` PID: ${supervisorPid}`);
|
||||
const via = detectedViaDbLock ? ' (detected via DB lock; pidfile not found at the configured path)' : '';
|
||||
console.log(`Supervisor: ${running ? 'running' : 'not running'}${via}`);
|
||||
if (status.supervisor_pid) console.log(` PID: ${status.supervisor_pid}${detectedViaDbLock ? ` @ ${dbLockHolder?.holder_host}` : ''}`);
|
||||
console.log(` PID file: ${pidFile}`);
|
||||
if (detectedViaDbLock && status.concurrency !== null) console.log(` Concurrency: ${status.concurrency}${status.max_rss_mb !== null ? ` (max-rss ${status.max_rss_mb}MB)` : ''}`);
|
||||
if (lastStart) console.log(` Last start: ${lastStart}`);
|
||||
console.log(` Crashes (24h): ${summary.total} (runtime=${summary.by_cause.runtime_error} oom=${summary.by_cause.oom_or_external_kill} unknown=${summary.by_cause.unknown} legacy=${summary.by_cause.legacy})`);
|
||||
console.log(` Clean exits (24h): ${summary.clean_exits}`);
|
||||
@@ -1607,6 +1642,13 @@ export async function registerBuiltinHandlers(
|
||||
// archived between fan-out and worker claim, skip cleanly.
|
||||
const rawSourceId = job.data.source_id;
|
||||
let sourceId: string | undefined;
|
||||
// issue #2227/#2194 (TODOS:634, codex #8): a per-source cycle must run its
|
||||
// FILESYSTEM phases (sync/lint/extract) against the SOURCE's own checkout,
|
||||
// not the global brain's. Pre-fix it inherited `repoPath` (the default
|
||||
// checkout) while writing DB freshness for `source_id` — mixed scope that
|
||||
// made cooldown/freshness attribute to the wrong source. We resolve the
|
||||
// source's `local_path` here and use it as the cycle's brainDir below.
|
||||
let sourceLocalPath: string | null = null;
|
||||
if (rawSourceId !== undefined && rawSourceId !== null) {
|
||||
if (typeof rawSourceId !== 'string') {
|
||||
throw new Error(`autopilot-cycle: invalid source_id (not a string): ${JSON.stringify(rawSourceId)}`);
|
||||
@@ -1620,9 +1662,10 @@ export async function registerBuiltinHandlers(
|
||||
}
|
||||
// Archive recheck (codex r1 P1-5): cheap pre-cycle lookup. Returns
|
||||
// immediately if source is gone or archived; runCycle never even
|
||||
// acquires a lock.
|
||||
const rows = await engine.executeRaw<{ archived: boolean | null }>(
|
||||
`SELECT archived FROM sources WHERE id = $1`,
|
||||
// acquires a lock. Also fetches local_path so FS phases bind to the
|
||||
// source's own checkout (the #2227/#2194 mixed-scope fix).
|
||||
const rows = await engine.executeRaw<{ archived: boolean | null; local_path: string | null }>(
|
||||
`SELECT archived, local_path FROM sources WHERE id = $1`,
|
||||
[rawSourceId],
|
||||
);
|
||||
if (rows.length === 0) {
|
||||
@@ -1640,8 +1683,17 @@ export async function registerBuiltinHandlers(
|
||||
};
|
||||
}
|
||||
sourceId = rawSourceId;
|
||||
sourceLocalPath = typeof rows[0].local_path === 'string' && rows[0].local_path.length > 0
|
||||
? rows[0].local_path
|
||||
: null;
|
||||
}
|
||||
|
||||
// Effective checkout for FS phases. For a per-source cycle, bind to the
|
||||
// SOURCE's local_path (or null → skip FS phases for a pure-DB source);
|
||||
// NEVER fall through to the global repoPath, which would run sync/lint
|
||||
// against the wrong tree. Legacy (no source_id) keeps the global repoPath.
|
||||
const effectiveBrainDir: string | null = sourceId ? sourceLocalPath : repoPath;
|
||||
|
||||
// Allow callers to select phases via job data (e.g. skip embed for
|
||||
// fast cycles). Validates against ALL_PHASES to prevent injection.
|
||||
const { ALL_PHASES } = await import('../core/cycle.ts');
|
||||
@@ -1653,8 +1705,23 @@ export async function registerBuiltinHandlers(
|
||||
// Pull default: legacy `true` for back-compat; explicit boolean wins.
|
||||
const pull = typeof job.data.pull === 'boolean' ? job.data.pull : true;
|
||||
|
||||
// #2194 fix #2 / codex #5 (D4): claim-time cooldown guard. A job already
|
||||
// queued or retrying (max_attempts:2) can reach the worker after the
|
||||
// dispatch gate decided to back this source off. Skip it here as a NO-OP
|
||||
// (status 'skipped', NOT a failure — a failure would re-arm the cooldown).
|
||||
if (sourceId) {
|
||||
const { isSourceInCooldown } = await import('./autopilot-fanout.ts');
|
||||
if (await isSourceInCooldown(engine, sourceId)) {
|
||||
return {
|
||||
partial: false,
|
||||
status: 'skipped',
|
||||
report: { reason: 'source_in_cooldown', source_id: sourceId },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const report = await runCycle(engine, {
|
||||
brainDir: repoPath,
|
||||
brainDir: effectiveBrainDir,
|
||||
pull,
|
||||
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
|
||||
...(sourceId ? { sourceId } : {}),
|
||||
@@ -1672,6 +1739,49 @@ export async function registerBuiltinHandlers(
|
||||
};
|
||||
});
|
||||
|
||||
// #2194 fix #3 / #2227 bug #3 — brain-wide maintenance. Runs the `global`
|
||||
// cycle phases (embed, orphans, purge, resolve_symbol_edges, grade_takes,
|
||||
// calibration_profile, synthesize_concepts, skillopt) ONCE per window instead
|
||||
// of N times concurrently across per-source cycles (the 4→10GB RSS blowout).
|
||||
// No source_id → uses the legacy global cycle lock; stamps autopilot.last_global_at
|
||||
// on success so the dispatch gate backs off.
|
||||
worker.register('autopilot-global-maintenance', async (job) => {
|
||||
const { runCycle, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY, ALL_PHASES } = await import('../core/cycle.ts');
|
||||
const repoPath: string | null = typeof job.data.repoPath === 'string'
|
||||
? job.data.repoPath
|
||||
: (await engine.getConfig('sync.repo_path')) ?? null;
|
||||
|
||||
const validPhases = new Set(ALL_PHASES);
|
||||
const requested = Array.isArray(job.data.phases)
|
||||
? (job.data.phases as string[]).filter((p) => validPhases.has(p as never))
|
||||
: GLOBAL_PHASES;
|
||||
const phases = (requested.length > 0 ? requested : GLOBAL_PHASES) as typeof GLOBAL_PHASES;
|
||||
|
||||
const report = await runCycle(engine, {
|
||||
brainDir: repoPath,
|
||||
pull: false, // brain-wide DB/maintenance work never git-pulls
|
||||
signal: job.signal,
|
||||
phases,
|
||||
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
|
||||
});
|
||||
|
||||
// Stamp last_global_at only on a non-failed run so a failed pass stays stale
|
||||
// and re-dispatches next tick (self-healing retry).
|
||||
if (report.status === 'ok' || report.status === 'clean' || report.status === 'partial') {
|
||||
try {
|
||||
await engine.setConfig(LAST_GLOBAL_AT_KEY, new Date().toISOString());
|
||||
} catch (e) {
|
||||
console.warn(`[autopilot-global-maintenance] failed to stamp last_global_at: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
partial: report.status === 'partial' || report.status === 'failed',
|
||||
status: report.status,
|
||||
report,
|
||||
};
|
||||
});
|
||||
|
||||
// Shell handler is always registered. Runtime env guard lives inside the
|
||||
// handler so claimed jobs emit a clear rejection log on workers missing
|
||||
// GBRAIN_ALLOW_SHELL_JOBS=1.
|
||||
|
||||
+32
-7
@@ -747,8 +747,26 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
// caught-up source reports lag 0 instead of growing wall-clock (v0.41.32.0).
|
||||
const metrics = await computeAllSourceMetrics(engine, sources, { probeContent: true });
|
||||
|
||||
// #1950: a source holding a live (non-TTL-expired) per-source sync lock is
|
||||
// actively syncing RIGHT NOW. Without this it printed "idle" while a sync
|
||||
// proc was live (the bug reported). Read the SAME live-lock signal `gbrain
|
||||
// doctor` uses, via the shared helper, so the two surfaces never disagree.
|
||||
const { liveSyncStatus } = await import('../core/db-lock.ts');
|
||||
const syncRunning = new Map<string, { holder_pid: number; holder_host: string }>();
|
||||
await Promise.all(
|
||||
metrics.map(async (m) => {
|
||||
const live = await liveSyncStatus(engine, m.source_id);
|
||||
if (live) syncRunning.set(m.source_id, live);
|
||||
}),
|
||||
);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ schema_version: 1, sources: metrics }, null, 2));
|
||||
const enriched = metrics.map((m) => ({
|
||||
...m,
|
||||
sync_running: syncRunning.has(m.source_id),
|
||||
sync_holder: syncRunning.get(m.source_id) ?? null,
|
||||
}));
|
||||
console.log(JSON.stringify({ schema_version: 1, sources: enriched }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -765,11 +783,15 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const embed = `${m.embed_coverage_pct.toFixed(0)}%`;
|
||||
// v0.41.31: embed-backfill state (active beats queued beats idle) so a
|
||||
// cron operator sees deferred embedding work after `sync --all`.
|
||||
const backfill = m.backfill_active > 0
|
||||
? `active(${m.backfill_active})`
|
||||
: m.backfill_queued > 0
|
||||
? `queued(${m.backfill_queued})`
|
||||
: 'idle';
|
||||
// #1950: a live sync lock wins the column — surfacing "actively syncing"
|
||||
// matters more than deferred embed-backfill state.
|
||||
const backfill = syncRunning.has(m.source_id)
|
||||
? 'running'
|
||||
: m.backfill_active > 0
|
||||
? `active(${m.backfill_active})`
|
||||
: m.backfill_queued > 0
|
||||
? `queued(${m.backfill_queued})`
|
||||
: 'idle';
|
||||
const fails = String(m.failed_jobs_24h);
|
||||
const queue = String(m.queue_depth);
|
||||
const pages = m.total_pages.toLocaleString();
|
||||
@@ -780,7 +802,10 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
for (const m of metrics) {
|
||||
const warns: string[] = [];
|
||||
if (!m.local_path) warns.push('no local_path');
|
||||
if (m.lag_seconds === null) warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``);
|
||||
// #1950: don't cry "never synced" while a sync lock is live — it's syncing now.
|
||||
if (m.lag_seconds === null && !syncRunning.has(m.source_id)) {
|
||||
warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``);
|
||||
}
|
||||
if (m.embed_coverage_pct < 95 && m.total_chunks > 100) {
|
||||
warns.push(`${(100 - m.embed_coverage_pct).toFixed(1)}% un-embedded — run \`gbrain embed --stale --source ${m.source_id}\``);
|
||||
}
|
||||
|
||||
+152
-23
@@ -39,6 +39,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { gbrainPath, loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
import {
|
||||
buildSyncStatusReport,
|
||||
type SyncStatusReport,
|
||||
@@ -104,6 +105,10 @@ export interface AutopilotStatus {
|
||||
|
||||
export interface StatusReport {
|
||||
schema_version: typeof SCHEMA_VERSION;
|
||||
/** #1984: the local gbrain CLI version, so a poller can pin behavior to a build. */
|
||||
version: string;
|
||||
/** #1984: the remote brain server's version (thin-client only; present when reported). */
|
||||
remote_version?: string;
|
||||
generated_at: string;
|
||||
mode: 'local' | 'thin-client';
|
||||
sync?: SyncStatusReport;
|
||||
@@ -113,6 +118,33 @@ export interface StatusReport {
|
||||
queue?: QueueCounts | { local_only_remote: true };
|
||||
autopilot?: AutopilotStatus | { local_only_remote: true };
|
||||
warnings?: string[];
|
||||
/** #1984: true when a --deadline-ms budget elided one or more sections. */
|
||||
partial?: boolean;
|
||||
/** #1984: names of the sections skipped/timed-out under the deadline. */
|
||||
stale_sections?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* #1984: race a section's work against the remaining deadline budget. Returns
|
||||
* the value, or undefined if the budget elapses first (the underlying query is
|
||||
* abandoned — the process is a one-shot, so a stranded read is fine). `ms`
|
||||
* undefined / <=0 means "no deadline" and the promise is awaited as-is.
|
||||
*/
|
||||
export async function withSectionDeadline<T>(
|
||||
p: Promise<T>,
|
||||
ms: number | undefined,
|
||||
onTimeout: () => void,
|
||||
): Promise<T | undefined> {
|
||||
if (ms === undefined || ms <= 0) return p;
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const timeout = new Promise<undefined>((resolve) => {
|
||||
timer = setTimeout(() => { onTimeout(); resolve(undefined); }, ms);
|
||||
});
|
||||
try {
|
||||
return await Promise.race([p, timeout]);
|
||||
} finally {
|
||||
if (timer) clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -305,6 +337,8 @@ function buildAutopilotStatus(): AutopilotStatus {
|
||||
|
||||
interface BuildOpts {
|
||||
sections?: Set<Section>;
|
||||
/** #1984: total wall-clock budget; sections share it, later ones get the remainder. */
|
||||
deadlineMs?: number;
|
||||
}
|
||||
|
||||
async function buildLocalReport(
|
||||
@@ -313,23 +347,44 @@ async function buildLocalReport(
|
||||
): Promise<StatusReport> {
|
||||
const want = (s: Section) => !opts.sections || opts.sections.has(s);
|
||||
const warnings: string[] = [];
|
||||
const staleSections: string[] = [];
|
||||
const report: StatusReport = {
|
||||
schema_version: SCHEMA_VERSION,
|
||||
version: VERSION,
|
||||
generated_at: new Date().toISOString(),
|
||||
mode: 'local',
|
||||
};
|
||||
|
||||
// #1984: a shared budget so `gbrain status --deadline-ms=N` never blocks a
|
||||
// poller past N. Each async section is raced against the REMAINING budget, so
|
||||
// one slow/hung section (cross-region DB, lock contention) can't strand the
|
||||
// whole snapshot — it's marked stale and the rest still return.
|
||||
const deadlineAt = opts.deadlineMs && opts.deadlineMs > 0 ? Date.now() + opts.deadlineMs : null;
|
||||
const remaining = (): number | undefined =>
|
||||
deadlineAt === null ? undefined : Math.max(1, deadlineAt - Date.now());
|
||||
const markStale = (name: Section) => {
|
||||
staleSections.push(name);
|
||||
report.partial = true;
|
||||
warnings.push(`${name} section exceeded the --deadline-ms budget (returned stale)`);
|
||||
};
|
||||
|
||||
if (want('sync')) {
|
||||
try {
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown> | null;
|
||||
}>(`SELECT id, name, local_path, config FROM sources ORDER BY id`);
|
||||
report.sync = await buildSyncStatusReport(
|
||||
engine,
|
||||
sources.map((s) => ({ id: s.id, name: s.name, local_path: s.local_path, config: s.config ?? {} })),
|
||||
report.sync = await withSectionDeadline(
|
||||
(async () => {
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown> | null;
|
||||
}>(`SELECT id, name, local_path, config FROM sources ORDER BY id`);
|
||||
return buildSyncStatusReport(
|
||||
engine,
|
||||
sources.map((s) => ({ id: s.id, name: s.name, local_path: s.local_path, config: s.config ?? {} })),
|
||||
);
|
||||
})(),
|
||||
remaining(),
|
||||
() => markStale('sync'),
|
||||
);
|
||||
} catch (err) {
|
||||
warnings.push(`sync section failed: ${(err as Error).message}`);
|
||||
@@ -337,23 +392,24 @@ async function buildLocalReport(
|
||||
}
|
||||
if (want('cycle')) {
|
||||
try {
|
||||
report.cycle = await buildCycleSnapshot(engine);
|
||||
report.cycle = await withSectionDeadline(buildCycleSnapshot(engine), remaining(), () => markStale('cycle'));
|
||||
} catch (err) {
|
||||
warnings.push(`cycle section failed: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
if (want('locks')) {
|
||||
report.locks = await buildLocks(engine);
|
||||
report.locks = await withSectionDeadline(buildLocks(engine), remaining(), () => markStale('locks'));
|
||||
}
|
||||
if (want('workers')) {
|
||||
report.workers = buildWorkerSummary();
|
||||
}
|
||||
if (want('queue')) {
|
||||
report.queue = await buildQueueCounts(engine);
|
||||
report.queue = await withSectionDeadline(buildQueueCounts(engine), remaining(), () => markStale('queue'));
|
||||
}
|
||||
if (want('autopilot')) {
|
||||
report.autopilot = buildAutopilotStatus();
|
||||
}
|
||||
if (staleSections.length > 0) report.stale_sections = staleSections;
|
||||
if (warnings.length > 0) report.warnings = warnings;
|
||||
return report;
|
||||
}
|
||||
@@ -366,20 +422,51 @@ async function buildThinClientReport(
|
||||
const warnings: string[] = [];
|
||||
const report: StatusReport = {
|
||||
schema_version: SCHEMA_VERSION,
|
||||
version: VERSION,
|
||||
generated_at: new Date().toISOString(),
|
||||
mode: 'thin-client',
|
||||
};
|
||||
|
||||
if (want('sync') || want('cycle')) {
|
||||
try {
|
||||
const raw = await callRemoteTool(cfg!, 'get_status_snapshot', {});
|
||||
const payload = unpackToolResult<{
|
||||
schema_version: number;
|
||||
sync: SyncStatusReport;
|
||||
cycle: CycleSnapshot;
|
||||
}>(raw);
|
||||
if (want('sync')) report.sync = payload.sync;
|
||||
if (want('cycle')) report.cycle = payload.cycle;
|
||||
const payload = await withSectionDeadline(
|
||||
(async () => {
|
||||
// #1984: pass the budget as the request timeout so the LOSING side of
|
||||
// the race actually cancels the in-flight MCP call instead of leaking
|
||||
// it (the section deadline only abandons the promise locally).
|
||||
const raw = await callRemoteTool(
|
||||
cfg!,
|
||||
'get_status_snapshot',
|
||||
{},
|
||||
opts.deadlineMs && opts.deadlineMs > 0 ? { timeoutMs: opts.deadlineMs } : {},
|
||||
);
|
||||
return unpackToolResult<{
|
||||
schema_version: number;
|
||||
version?: string;
|
||||
sync: SyncStatusReport;
|
||||
cycle: CycleSnapshot;
|
||||
}>(raw);
|
||||
})(),
|
||||
opts.deadlineMs && opts.deadlineMs > 0 ? opts.deadlineMs : undefined,
|
||||
() => {
|
||||
report.partial = true;
|
||||
// Only name the sections the caller actually requested; the remote
|
||||
// fetch backs both sync+cycle, but `--section sync` must not report
|
||||
// `cycle` (a section it excluded) as stale. Matches the local path.
|
||||
const elided: Section[] = [
|
||||
...(want('sync') ? (['sync'] as Section[]) : []),
|
||||
...(want('cycle') ? (['cycle'] as Section[]) : []),
|
||||
];
|
||||
report.stale_sections = [...(report.stale_sections ?? []), ...elided];
|
||||
warnings.push('remote snapshot exceeded the --deadline-ms budget (returned stale)');
|
||||
},
|
||||
);
|
||||
if (payload) {
|
||||
// #1984: surface the brain server's version for thin-client parity.
|
||||
if (payload.version) report.remote_version = payload.version;
|
||||
if (want('sync')) report.sync = payload.sync;
|
||||
if (want('cycle')) report.cycle = payload.cycle;
|
||||
}
|
||||
} catch (err) {
|
||||
warnings.push(`remote snapshot failed: ${(err as Error).message}`);
|
||||
}
|
||||
@@ -401,7 +488,13 @@ function renderHuman(report: StatusReport): string {
|
||||
lines.push('');
|
||||
lines.push('GBrain Status');
|
||||
lines.push('=============');
|
||||
lines.push(`Mode: ${report.mode} · ${report.generated_at}`);
|
||||
const ver = report.remote_version && report.remote_version !== report.version
|
||||
? `v${report.version} (remote v${report.remote_version})`
|
||||
: `v${report.version}`;
|
||||
lines.push(`Mode: ${report.mode} · ${ver} · ${report.generated_at}`);
|
||||
if (report.partial) {
|
||||
lines.push(`⚠ partial snapshot — stale sections: ${(report.stale_sections ?? []).join(', ')} (--deadline-ms budget hit)`);
|
||||
}
|
||||
lines.push('');
|
||||
|
||||
// Sync
|
||||
@@ -528,6 +621,36 @@ function renderHuman(report: StatusReport): string {
|
||||
* - Set<Section> → only these sections
|
||||
* - 'usage_error' → bad section name (caller exits 2)
|
||||
*/
|
||||
/** #1984: `--fast` preset budget (ms) when no explicit `--deadline-ms` is given. */
|
||||
export const FAST_DEADLINE_MS = 2000;
|
||||
|
||||
/**
|
||||
* #1984: parse the status deadline budget. `--deadline-ms=N` (or `--deadline-ms N`)
|
||||
* wins; bare `--fast` applies FAST_DEADLINE_MS. Returns undefined (no budget),
|
||||
* a positive ms value, or 'usage_error' on a non-positive / non-numeric value.
|
||||
*/
|
||||
export function parseDeadlineFlag(args: string[]): number | undefined | 'usage_error' {
|
||||
let raw: string | undefined;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--deadline-ms') {
|
||||
// #1984: a bare `--deadline-ms` with no following value is a usage error,
|
||||
// not a silent fall-through to no-budget / --fast (which would mask a
|
||||
// typo'd budget and let a poller hang it never meant to).
|
||||
if (i + 1 >= args.length) return 'usage_error';
|
||||
raw = args[i + 1];
|
||||
break;
|
||||
}
|
||||
if (a.startsWith('--deadline-ms=')) { raw = a.slice('--deadline-ms='.length); break; }
|
||||
}
|
||||
if (raw == null) {
|
||||
return args.includes('--fast') ? FAST_DEADLINE_MS : undefined;
|
||||
}
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n <= 0) return 'usage_error';
|
||||
return n;
|
||||
}
|
||||
|
||||
export function parseSectionFlag(args: string[]): Set<Section> | undefined | 'usage_error' {
|
||||
let raw: string | undefined;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
@@ -575,19 +698,25 @@ export async function runStatus(
|
||||
const sections = sectionFlag;
|
||||
const json = args.includes('--json');
|
||||
|
||||
const deadlineMs = parseDeadlineFlag(args);
|
||||
if (deadlineMs === 'usage_error') {
|
||||
stderr('gbrain status: --deadline-ms must be a positive number of milliseconds\n');
|
||||
return { exitCode: 2 };
|
||||
}
|
||||
|
||||
const cfg = loadConfig();
|
||||
const useThinClient = cfg ? isThinClient(cfg) : false;
|
||||
|
||||
let report: StatusReport;
|
||||
try {
|
||||
if (useThinClient) {
|
||||
report = await buildThinClientReport(cfg, { sections });
|
||||
report = await buildThinClientReport(cfg, { sections, deadlineMs });
|
||||
} else {
|
||||
if (!engine) {
|
||||
stderr('gbrain status: no engine connected (DB unreachable?). Run `gbrain doctor` to diagnose.\n');
|
||||
return { exitCode: 1 };
|
||||
}
|
||||
report = await buildLocalReport(engine, { sections });
|
||||
report = await buildLocalReport(engine, { sections, deadlineMs });
|
||||
}
|
||||
} catch (err) {
|
||||
stderr(`gbrain status: snapshot failed: ${(err as Error).message}\n`);
|
||||
|
||||
+74
-6
@@ -212,7 +212,7 @@ export interface SyncResult {
|
||||
* cron operators can disambiguate timeout vs pull-timeout in monitoring.
|
||||
*/
|
||||
filesImported?: number;
|
||||
reason?: 'timeout' | 'pull_timeout' | 'checkpoint_unavailable';
|
||||
reason?: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
|
||||
/**
|
||||
* v0.42.x (#1794): cumulative file paths durably banked to the checkpoint
|
||||
* across THIS run + prior resumed runs. Surfaced on every partial/blocked
|
||||
@@ -1440,7 +1440,7 @@ function buildPartialResult(opts: {
|
||||
modified: number;
|
||||
deleted: number;
|
||||
renamed: number;
|
||||
reason: 'timeout' | 'pull_timeout' | 'checkpoint_unavailable';
|
||||
reason: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
|
||||
bankedFiles?: number;
|
||||
}): SyncResult {
|
||||
return {
|
||||
@@ -2050,7 +2050,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// `bankedFiles` is surfaced so a killed run shows banked progress instead of
|
||||
// looking like total loss. toCommit reports the PINNED target; last_commit is
|
||||
// never advanced on a partial (the next run resumes from the checkpoint).
|
||||
const partial = async (reason: 'timeout' | 'pull_timeout'): Promise<SyncResult> => {
|
||||
const partial = async (reason: 'timeout' | 'pull_timeout' | 'stall_timeout'): Promise<SyncResult> => {
|
||||
deregisterCheckpointCleanup();
|
||||
if (!checkpointDead) {
|
||||
try { await flushCheckpoint(); } catch { /* best effort — we're aborting */ }
|
||||
@@ -2409,6 +2409,48 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
pacer = createNoopPacer();
|
||||
}
|
||||
|
||||
// #1950: progress-aware stall watchdog for the import drain. The incident
|
||||
// was a sync wedged ~29min while ALIVE — so the lock heartbeat kept
|
||||
// refreshing (it fires on its own timer) and the wall-clock deadline hadn't
|
||||
// hit yet, leaving only a manual `pkill`. This keys off FORWARD IMPORT
|
||||
// PROGRESS (progress.tick below bumps `progressAt`), not the heartbeat: if no
|
||||
// file completes for `resolveStallAbortSeconds()`, abort. The abort signal is
|
||||
// composed into `opts.signal`, so the existing per-iteration abort checks,
|
||||
// pacer.acquire/pace, and parallel-worker break-loops all observe it; the
|
||||
// drain returns partial() (last_commit unchanged, next run resumes from the
|
||||
// checkpoint) and withRefreshingLock's finally releases the lock. Limits
|
||||
// (TODOS: #1950 follow-up — thread a cancellation signal through importFile):
|
||||
// the abort is observed BETWEEN files (the per-iteration checks + the next
|
||||
// importOnePath's pre-acquire check), so a hang INSIDE a single importFile
|
||||
// call is not interrupted until that call returns — the watchdog fires and
|
||||
// logs, but the in-flight file finishes (or the wall-clock hard deadline is
|
||||
// the eventual backstop). This catches the documented #1950 incident shape
|
||||
// (a slow-but-progressing drain, many files) and a stalled between-file
|
||||
// drain; a single wedged file or a fully starved event loop is out of scope
|
||||
// here. stallAborted distinguishes this from a user --timeout/SIGINT so the
|
||||
// partial result reports `stall_timeout`, not `timeout`.
|
||||
const stallSeconds = resolveStallAbortSeconds();
|
||||
const progressAt = { last: Date.now() };
|
||||
let stallAborted = false;
|
||||
let stallTimer: ReturnType<typeof setInterval> | undefined;
|
||||
if (stallSeconds > 0) {
|
||||
const stallMs = stallSeconds * 1000;
|
||||
const stallController = new AbortController();
|
||||
stallTimer = setInterval(() => {
|
||||
if (Date.now() - progressAt.last >= stallMs) {
|
||||
serr(
|
||||
`[sync] no import progress for ${stallSeconds}s — aborting (stall watchdog). ` +
|
||||
`The per-source lock will release; the next 'gbrain sync' resumes from the checkpoint.`,
|
||||
);
|
||||
stallAborted = true;
|
||||
stallController.abort();
|
||||
}
|
||||
}, Math.min(5000, stallMs));
|
||||
// Don't keep the process alive on the watchdog alone.
|
||||
(stallTimer as unknown as { unref?: () => void }).unref?.();
|
||||
opts = { ...opts, signal: composeAbortSignals(opts.signal, stallController.signal) };
|
||||
}
|
||||
|
||||
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
|
||||
const filePath = join(syncRepoPath, path);
|
||||
if (!existsSync(filePath)) {
|
||||
@@ -2429,6 +2471,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// its row so it can't age doctor to a permanent FAIL. (This covers the
|
||||
// net-zero add-then-delete range where the path isn't in filtered.deleted.)
|
||||
succeededPaths.push(path);
|
||||
progressAt.last = Date.now(); // #1950: forward progress → reset stall watchdog
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
@@ -2484,6 +2527,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
} finally {
|
||||
permit.release();
|
||||
}
|
||||
progressAt.last = Date.now(); // #1950: forward progress → reset stall watchdog
|
||||
progress.tick(1, path);
|
||||
// v0.42.x (#1794): keep the lock-refresh heartbeat alive on big imports.
|
||||
await maybeYield();
|
||||
@@ -2510,7 +2554,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// serial fallback inside the parallel branch (database_url unset).
|
||||
if (opts.signal?.aborted) {
|
||||
progress.finish();
|
||||
return await partial('timeout');
|
||||
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
|
||||
}
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
@@ -2574,7 +2618,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// primary serial site.
|
||||
if (opts.signal?.aborted) {
|
||||
progress.finish();
|
||||
return await partial('timeout');
|
||||
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
|
||||
}
|
||||
await importOnePath(engine, path);
|
||||
}
|
||||
@@ -2583,6 +2627,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// paced-backfill: release any blocked acquirers + clear pacer state on
|
||||
// every exit path (including the early partial('timeout') returns above).
|
||||
pacer.dispose();
|
||||
// #1950: tear down the stall watchdog on every import-phase exit (normal,
|
||||
// partial('timeout'), or throw). The try wrapping the import loop guarantees
|
||||
// this runs before any post-import bookmark/anchor work.
|
||||
if (stallTimer) clearInterval(stallTimer);
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
@@ -2594,7 +2642,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// the bookmark write below. By returning partial here, we preserve
|
||||
// the D-V3-1 invariant that abort means "never advance last_commit."
|
||||
if (opts.signal?.aborted) {
|
||||
return await partial('timeout');
|
||||
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3144,6 +3192,26 @@ export interface HardDeadlineResolution {
|
||||
reason: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* #1950: default no-import-progress window before the in-band stall watchdog
|
||||
* aborts the drain. Generous on purpose — it must clear one legitimately large
|
||||
* file (cross-region, big page) without false-tripping; one file taking longer
|
||||
* than this trips it (documented limit). Distinct from the wall-clock hard
|
||||
* deadline (whole-run cap) and the lock heartbeat (refreshes regardless of
|
||||
* import progress). Env-tunable; <=0 disables.
|
||||
*/
|
||||
export const DEFAULT_SYNC_STALL_ABORT_SEC = 900;
|
||||
|
||||
export function resolveStallAbortSeconds(
|
||||
env: Record<string, string | undefined> = process.env,
|
||||
): number {
|
||||
const raw = env.GBRAIN_SYNC_STALL_ABORT_SECONDS;
|
||||
if (raw === undefined || raw === '') return DEFAULT_SYNC_STALL_ABORT_SEC;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return DEFAULT_SYNC_STALL_ABORT_SEC;
|
||||
return n; // n <= 0 disables the watchdog
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the out-of-band hard-deadline for a `gbrain sync` invocation (#1633).
|
||||
* Pure + argv/env-only so it runs BEFORE `connectEngine` (so a connect-phase hang
|
||||
|
||||
+30
-2
@@ -242,6 +242,25 @@ export const PHASE_SCOPE: Record<CyclePhase, PhaseScope> = {
|
||||
skillopt: 'global',
|
||||
};
|
||||
|
||||
/**
|
||||
* #2194 fix #3 / #2227 bug #3 — the cycle split.
|
||||
*
|
||||
* Per-source autopilot cycles run ONLY the source-scoped (and mixed) phases;
|
||||
* the brain-wide `global` phases (embed, orphans, purge, resolve_symbol_edges,
|
||||
* grade_takes, calibration_profile, synthesize_concepts, skillopt) run ONCE in
|
||||
* a separate `autopilot-global-maintenance` job instead of N times concurrently
|
||||
* across per-source cycles (the 4→10GB RSS blowout). Single-flight is
|
||||
* structural: one global job, not a skip-and-pretend-fresh hack (codex #1/#2).
|
||||
*
|
||||
* GLOBAL_PHASES ∪ NON_GLOBAL_PHASES == ALL_PHASES, with no overlap — pinned by
|
||||
* test/autopilot-global-maintenance.test.ts.
|
||||
*/
|
||||
export const GLOBAL_PHASES: CyclePhase[] = ALL_PHASES.filter((p) => PHASE_SCOPE[p] === 'global');
|
||||
export const NON_GLOBAL_PHASES: CyclePhase[] = ALL_PHASES.filter((p) => PHASE_SCOPE[p] !== 'global');
|
||||
|
||||
/** Config key holding the ISO timestamp of the last successful global-maintenance run. */
|
||||
export const LAST_GLOBAL_AT_KEY = 'autopilot.last_global_at';
|
||||
|
||||
/**
|
||||
* Phases that mutate state (filesystem or DB) and therefore should
|
||||
* coordinate via the cycle lock. Only orphans is truly read-only
|
||||
@@ -2305,12 +2324,21 @@ export async function runCycle(
|
||||
// the cost of missing a successful write (next cycle will redo work).
|
||||
if (opts.sourceId && engine && !dryRun && !aborted && (status === 'ok' || status === 'clean' || status === 'partial')) {
|
||||
try {
|
||||
const nowIso = new Date().toISOString();
|
||||
// #2194 fix #3 (the cycle split): `last_source_cycle_at` is the NEW gate
|
||||
// for per-source dispatch (source-scoped phases done). We ALSO keep
|
||||
// `last_full_cycle_at` current so doctor's cycle-freshness check and any
|
||||
// legacy reader stay valid — it's no longer a *gate* for the brain-wide
|
||||
// phases (those gate on autopilot.last_global_at), so writing it on a
|
||||
// source-only cycle does not re-introduce the freshness poisoning codex
|
||||
// flagged in the rejected skip-based design.
|
||||
await engine.updateSourceConfig(opts.sourceId, {
|
||||
last_full_cycle_at: new Date().toISOString(),
|
||||
last_source_cycle_at: nowIso,
|
||||
last_full_cycle_at: nowIso,
|
||||
});
|
||||
} catch (e) {
|
||||
// Best-effort; cycle already succeeded by the time we get here.
|
||||
console.warn(`[cycle] failed to write last_full_cycle_at for source ${opts.sourceId}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
console.warn(`[cycle] failed to write last_source_cycle_at for source ${opts.sourceId}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -132,6 +132,28 @@ export function isHolderDeadLocally(
|
||||
return classifyHolderLiveness(holderPid, holderHost, ageMs, opts) === 'dead_eligible';
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #2227: is a lock-row holder live enough to count as "running" for an
|
||||
* observability surface (`gbrain jobs supervisor status`, `gbrain doctor`)?
|
||||
*
|
||||
* PID-reuse-safe by design (`pid-liveness-alone-pid-reuse`): keys on the lock's
|
||||
* own freshness, NEVER `process.kill`. A live holder refreshes its TTL on a
|
||||
* timer; a dead one stops, so its `ttl_expires_at` lapses and (after the steal
|
||||
* grace) `last_refreshed_at` ages out. The primary signal is `!ttl_expired`;
|
||||
* the heartbeat grace covers a starved-but-alive holder whose TTL briefly
|
||||
* lapsed between refresh ticks (the #1794 thrash class), mirroring the
|
||||
* steal-grace semantics `tryAcquireDbLock` already uses. Cross-host holders are
|
||||
* still "running" for visibility (a supervisor exists, just elsewhere) — we
|
||||
* report freshness, not takeover-eligibility, so host is not consulted here.
|
||||
*/
|
||||
export function isLockHolderLive(snap: LockSnapshot, ttlMinutes: number = DEFAULT_TTL_MINUTES): boolean {
|
||||
if (!snap.ttl_expired) return true;
|
||||
if (snap.ms_since_last_refresh !== null) {
|
||||
return snap.ms_since_last_refresh < resolveStealGraceSeconds(ttlMinutes) * 1000;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Try to acquire a named DB lock.
|
||||
*
|
||||
@@ -707,6 +729,38 @@ export function syncLockId(sourceId: string): string {
|
||||
*/
|
||||
export const SYNC_LOCK_ID = syncLockId('default');
|
||||
|
||||
/**
|
||||
* #1950: is `sourceId` actively holding a live (non-TTL-expired) sync lock?
|
||||
*
|
||||
* Centralizes the live-sync signal that `gbrain doctor` already computes inline
|
||||
* so `gbrain sources status` (and future surfaces) read the SAME truth instead
|
||||
* of each re-deriving it. Returns the holder when a live lock is held, else
|
||||
* null (idle, or a stale/expired lock that's structurally available for the
|
||||
* next acquire). Inspect failures swallow to null — a status surface should
|
||||
* degrade to "no indicator", never crash.
|
||||
*
|
||||
* Honest scope: a live lock proves the holder process is heartbeating, NOT that
|
||||
* the import is making forward progress — `withRefreshingLock` refreshes
|
||||
* `last_refreshed_at` on its own timer regardless of import progress. So callers
|
||||
* report "running", NOT "healthy"; a wedged-but-alive holder still reads as
|
||||
* running here. Forward-progress stall detection lives in the sync drain loop
|
||||
* (#1950 stall-abort); stale-lock triage lives in `gbrain doctor`.
|
||||
*/
|
||||
export async function liveSyncStatus(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
): Promise<{ holder_pid: number; holder_host: string } | null> {
|
||||
try {
|
||||
const snap = await inspectLock(engine, syncLockId(sourceId));
|
||||
if (snap && !snap.ttl_expired) {
|
||||
return { holder_pid: snap.holder_pid, holder_host: snap.holder_host };
|
||||
}
|
||||
return null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.30.1 (T4 + A4): wrap long-running work in a refreshing TTL lock.
|
||||
*
|
||||
|
||||
@@ -127,6 +127,7 @@ export const SKILL_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
*/
|
||||
export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'alternative_providers',
|
||||
'autopilot_fanout_concurrency',
|
||||
'autopilot_lock_scope',
|
||||
'batch_retry_health',
|
||||
'brainstorm_health',
|
||||
|
||||
@@ -61,9 +61,12 @@ export type ChildSupervisorEvent =
|
||||
}
|
||||
| {
|
||||
kind: 'health_warn';
|
||||
reason: 'clean_restart_budget_exceeded' | 'rss_watchdog_loop';
|
||||
reason: 'clean_restart_budget_exceeded' | 'rss_watchdog_loop' | 'crash_budget_degraded';
|
||||
count: number;
|
||||
windowMs: number;
|
||||
/** Present for window-scoped warnings (budget/watchdog loops). */
|
||||
windowMs?: number;
|
||||
/** Present for crash_budget_degraded: the soft budget that was crossed. */
|
||||
max?: number;
|
||||
};
|
||||
|
||||
export interface ChildWorkerSupervisorOpts {
|
||||
@@ -73,8 +76,24 @@ export interface ChildWorkerSupervisorOpts {
|
||||
args: string[];
|
||||
/** Child env. Defaults to a clone of process.env. */
|
||||
env?: NodeJS.ProcessEnv;
|
||||
/** Give up after this many consecutive code != 0 exits. */
|
||||
/**
|
||||
* Soft crash budget. issue #1994 (#2227 tail): crossing this NO LONGER
|
||||
* permanently gives up. Instead the supervisor enters DEGRADED mode — it
|
||||
* keeps respawning with capped exponential backoff (60s cap) and emits a
|
||||
* loud `crash_budget_degraded` health_warn — so a transient DB-pooler outage
|
||||
* that trips the counter self-heals when the DB returns (the stable-run reset
|
||||
* clears crashCount once a respawn runs > stableRunResetMs) instead of
|
||||
* wedging the queue until a human restart.
|
||||
*/
|
||||
maxCrashes: number;
|
||||
/**
|
||||
* Hard ceiling: permanently give up (fire onMaxCrashesExceeded) ONLY after
|
||||
* this many consecutive crashes — the runaway backstop for a genuinely
|
||||
* unrecoverable hot crash-loop that the stable-run reset never escapes.
|
||||
* Default: maxCrashes * HARD_STOP_CRASH_MULTIPLIER. Set to 0 to disable
|
||||
* permanent give-up entirely (retry-forever-with-backoff).
|
||||
*/
|
||||
hardStopMaxCrashes?: number;
|
||||
/** Stable-run reset window: code != 0 after this duration resets crashCount to 1. Default 5 min. */
|
||||
stableRunResetMs?: number;
|
||||
|
||||
@@ -130,6 +149,12 @@ export interface ChildWorkerSupervisorOpts {
|
||||
_now?: () => number;
|
||||
}
|
||||
|
||||
/** issue #1994: how many multiples of the soft crash budget before the hard
|
||||
* permanent-give-up backstop fires. 10× the default soft budget of 10 = 100
|
||||
* consecutive crashes (each within the stable-run window) before we conclude
|
||||
* it's a genuine code bug and stop. A transient outage recovers long before. */
|
||||
export const HARD_STOP_CRASH_MULTIPLIER = 10;
|
||||
|
||||
const DEFAULTS = {
|
||||
stableRunResetMs: 5 * 60 * 1000,
|
||||
cleanRestartBudget: 10,
|
||||
@@ -287,19 +312,50 @@ export class ChildWorkerSupervisor {
|
||||
/**
|
||||
* Run the spawn-and-respawn loop. Resolves when:
|
||||
* 1. composer.isStopping() returns true, OR
|
||||
* 2. crashCount reaches maxCrashes (after firing onMaxCrashesExceeded).
|
||||
* 2. crashCount reaches the HARD ceiling (after firing onMaxCrashesExceeded).
|
||||
*
|
||||
* issue #1994 (#2227 tail): crossing the SOFT budget (`maxCrashes`) no longer
|
||||
* stops the supervisor. It enters degraded mode — keep respawning with capped
|
||||
* exponential backoff (60s cap, so it's a paced retry, not a hot loop) and
|
||||
* announce loudly — so a transient DB-pooler outage that trips the counter
|
||||
* recovers on its own (a respawn that runs > stableRunResetMs resets
|
||||
* crashCount to 1) instead of permanently wedging the queue. Permanent
|
||||
* give-up fires only at the much-higher hard ceiling: the runaway backstop
|
||||
* for a genuinely unrecoverable hot crash-loop.
|
||||
*/
|
||||
async run(): Promise<void> {
|
||||
while (!this.opts.isStopping() && this._crashCount < this.opts.maxCrashes) {
|
||||
const hardStop = this.opts.hardStopMaxCrashes ??
|
||||
this.opts.maxCrashes * HARD_STOP_CRASH_MULTIPLIER;
|
||||
let degradedAnnounced = false;
|
||||
while (!this.opts.isStopping()) {
|
||||
await this.spawnOnce();
|
||||
|
||||
if (this.opts.isStopping()) return;
|
||||
|
||||
if (this._crashCount >= this.opts.maxCrashes) {
|
||||
this.opts.onMaxCrashesExceeded(this._crashCount, this.opts.maxCrashes);
|
||||
// Hard ceiling: permanent give-up (the runaway backstop). hardStop <= 0
|
||||
// disables it entirely (retry-forever-with-backoff for deployments that
|
||||
// would rather never auto-stop a recoverable supervisor).
|
||||
if (hardStop > 0 && this._crashCount >= hardStop) {
|
||||
this.opts.onMaxCrashesExceeded(this._crashCount, hardStop);
|
||||
return;
|
||||
}
|
||||
|
||||
// Soft budget crossed → degraded mode. Announce once per degradation
|
||||
// episode; re-arm after a stable-run reset drops us back under budget.
|
||||
if (this._crashCount >= this.opts.maxCrashes) {
|
||||
if (!degradedAnnounced) {
|
||||
degradedAnnounced = true;
|
||||
this.opts.onEvent({
|
||||
kind: 'health_warn',
|
||||
reason: 'crash_budget_degraded',
|
||||
count: this._crashCount,
|
||||
max: this.opts.maxCrashes,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
degradedAnnounced = false;
|
||||
}
|
||||
|
||||
await this.applyBackoff();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -34,6 +34,9 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
|
||||
subagent_aggregator: THIRTY_MIN_MS,
|
||||
'embed-backfill': THIRTY_MIN_MS,
|
||||
'autopilot-cycle': THIRTY_MIN_MS,
|
||||
// #2194 fix #3: brain-wide maintenance (embed-all/orphans/purge/…) can run
|
||||
// longer than a single source cycle; give it the same 30-min budget.
|
||||
'autopilot-global-maintenance': THIRTY_MIN_MS,
|
||||
};
|
||||
|
||||
/**
|
||||
|
||||
@@ -647,9 +647,18 @@ export class MinionQueue {
|
||||
async handleTimeouts(): Promise<MinionJob[]> {
|
||||
return this.engine.transaction(async (tx) => {
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
// #1737: count the timed-out run as a spent attempt (terminal, no retry),
|
||||
// mirroring handleWallClockTimeouts + handleStalled. handleTimeouts is the
|
||||
// FIRST killer to fire for the long-lane handlers (timeout_ms stamped at
|
||||
// submit), so without this the job reads `attempts: 0/N (started: N)`.
|
||||
// Safe against double-count: the worker sweep runs handleStalled ->
|
||||
// handleTimeouts -> handleWallClockTimeouts sequentially and awaited, and
|
||||
// each guards on `status = 'active'`, so the first to set status='dead'
|
||||
// excludes the row from the later sweeps.
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
error_text = 'timeout exceeded',
|
||||
attempts_made = attempts_made + 1,
|
||||
lock_token = NULL,
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
|
||||
@@ -30,6 +30,7 @@ import { detectTini } from './spawn-helpers.ts';
|
||||
import { resolveDefaultMaxRssMb } from './rss-default.ts';
|
||||
import {
|
||||
ChildWorkerSupervisor,
|
||||
HARD_STOP_CRASH_MULTIPLIER,
|
||||
type ChildSupervisorEvent,
|
||||
} from './child-worker-supervisor.ts';
|
||||
import {
|
||||
@@ -191,6 +192,21 @@ export function buildWorkerArgs(
|
||||
* shutdown() drain window (issue #1801, D3). */
|
||||
const WEDGE_RESTART_GRACE_MS = 35_000;
|
||||
|
||||
/**
|
||||
* issue #1994: resolve the hard permanent-give-up ceiling. Default
|
||||
* maxCrashes × HARD_STOP_CRASH_MULTIPLIER; operators override (or disable with
|
||||
* 0 = never auto-stop) via GBRAIN_SUPERVISOR_HARD_STOP_CRASHES. A negative or
|
||||
* non-integer override is ignored (falls back to the default).
|
||||
*/
|
||||
export function resolveHardStopMaxCrashes(maxCrashes: number): number {
|
||||
const raw = process.env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES;
|
||||
if (raw !== undefined && raw !== '') {
|
||||
const n = Number(raw);
|
||||
if (Number.isInteger(n) && n >= 0) return n;
|
||||
}
|
||||
return maxCrashes * HARD_STOP_CRASH_MULTIPLIER;
|
||||
}
|
||||
|
||||
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
|
||||
export function calculateBackoffMs(crashCount: number): number {
|
||||
const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000);
|
||||
@@ -293,7 +309,11 @@ export const ExitCodes = {
|
||||
* pidfile path. TTL > refresh-interval × max-failures so we always exit
|
||||
* before our lock could lapse and let a second supervisor take over.
|
||||
*/
|
||||
const SUPERVISOR_LOCK_TTL_MIN = 5;
|
||||
// Exported (issue #2227) so observability surfaces (`gbrain jobs supervisor
|
||||
// status`, `gbrain doctor`) compute the lock-freshness steal grace with the
|
||||
// SAME TTL the supervisor refreshes against, when detecting a live supervisor
|
||||
// via the DB lock instead of the (possibly split-$HOME) pidfile.
|
||||
export const SUPERVISOR_LOCK_TTL_MIN = 5;
|
||||
const SUPERVISOR_LOCK_REFRESH_MS = 60_000;
|
||||
const SUPERVISOR_LOCK_REFRESH_MAX_FAILURES = 3; // 3 × 60s = 180s < 5min TTL
|
||||
|
||||
@@ -825,6 +845,10 @@ export class MinionSupervisor {
|
||||
args: workerArgs,
|
||||
env,
|
||||
maxCrashes: this.opts.maxCrashes,
|
||||
// issue #1994: hard permanent-give-up ceiling (the runaway backstop).
|
||||
// Operators can raise/lower or disable (0 = never auto-stop) via
|
||||
// GBRAIN_SUPERVISOR_HARD_STOP_CRASHES; default is maxCrashes × 10.
|
||||
hardStopMaxCrashes: resolveHardStopMaxCrashes(this.opts.maxCrashes),
|
||||
_backoffFloorMs: this.opts._backoffFloorMs,
|
||||
isStopping: () => this.stopping,
|
||||
onMaxCrashesExceeded: (count, max) => {
|
||||
@@ -907,6 +931,9 @@ export class MinionSupervisor {
|
||||
// ("raise --max-rss") is one glance away. Peak RSS stays in the
|
||||
// worker's own stderr line (the supervisor never sees it).
|
||||
...(event.reason === 'rss_watchdog_loop' ? { max_rss_mb: this.opts.maxRssMb } : {}),
|
||||
// issue #1994: degraded mode crossed the soft crash budget — surface
|
||||
// it so doctor/status show "retrying with backoff" instead of silence.
|
||||
...(event.max !== undefined ? { max_crashes: event.max } : {}),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -2417,7 +2417,10 @@ const get_status_snapshot: Operation = {
|
||||
}
|
||||
const sync = await buildSyncStatusReport(ctx.engine, sources);
|
||||
const cycle = await buildCycleSnapshot(ctx.engine);
|
||||
return { schema_version: 1 as const, sync, cycle };
|
||||
// #1984: report the brain server's version so a thin-client `gbrain status`
|
||||
// can surface remote_version alongside its own local CLI version.
|
||||
const { VERSION } = await import('../version.ts');
|
||||
return { schema_version: 1 as const, version: VERSION, sync, cycle };
|
||||
},
|
||||
scope: 'admin',
|
||||
localOnly: false,
|
||||
|
||||
+55
-3
@@ -43,7 +43,7 @@ describe('parseRunFlags', () => {
|
||||
expect(rest).toEqual(['hello', 'world']);
|
||||
});
|
||||
|
||||
test('flags before prompt are parsed, unknown token ends flag parsing', () => {
|
||||
test('leading flags parsed; first positional begins the prompt', () => {
|
||||
const { flags, rest } = agentTesting.parseRunFlags([
|
||||
'--model', 'claude-opus-4-7', '--max-turns', '30', 'summarize', 'everything',
|
||||
]);
|
||||
@@ -69,8 +69,60 @@ describe('parseRunFlags', () => {
|
||||
expect(rest).toEqual(['--not-a-flag']);
|
||||
});
|
||||
|
||||
test('unknown flag throws', () => {
|
||||
expect(() => agentTesting.parseRunFlags(['--what', 'x'])).toThrow(/unknown flag/);
|
||||
test('#1738: unknown --flag is prompt text, not an error', () => {
|
||||
const { rest } = agentTesting.parseRunFlags(['--what', 'x']);
|
||||
expect(rest).toEqual(['--what', 'x']);
|
||||
});
|
||||
|
||||
test('#1738: trailing --detach is hoisted out of the prompt', () => {
|
||||
const { flags, rest } = agentTesting.parseRunFlags(['do', 'the', 'thing', '--detach']);
|
||||
expect(flags.detach).toBe(true);
|
||||
expect(flags.follow).toBe(false);
|
||||
expect(rest).toEqual(['do', 'the', 'thing']);
|
||||
});
|
||||
|
||||
test('#1738: leading flags + trailing switch both apply', () => {
|
||||
const { flags, rest } = agentTesting.parseRunFlags(['--model', 'm', 'summarize', '--detach', '--no-follow']);
|
||||
expect(flags.model).toBe('m');
|
||||
expect(flags.detach).toBe(true);
|
||||
expect(flags.follow).toBe(false);
|
||||
expect(rest).toEqual(['summarize']);
|
||||
});
|
||||
|
||||
test('#1738: a --switch mid-prompt (not trailing) stays verbatim', () => {
|
||||
const { flags, rest } = agentTesting.parseRunFlags(['summarize', '--detach', 'the', 'doc']);
|
||||
expect(flags.detach).toBe(false);
|
||||
expect(rest).toEqual(['summarize', '--detach', 'the', 'doc']);
|
||||
});
|
||||
|
||||
test('#1738: a freeform prompt starting with --word is preserved', () => {
|
||||
const { rest } = agentTesting.parseRunFlags(['--note:', 'do', 'the', 'thing']);
|
||||
expect(rest).toEqual(['--note:', 'do', 'the', 'thing']);
|
||||
});
|
||||
|
||||
test('#1738: -- suppresses trailing-switch hoisting', () => {
|
||||
const { flags, rest } = agentTesting.parseRunFlags(['--', 'do', 'x', '--detach']);
|
||||
expect(flags.detach).toBe(false);
|
||||
expect(rest).toEqual(['do', 'x', '--detach']);
|
||||
});
|
||||
|
||||
test('#1738: -- AFTER a positional also suppresses hoisting (no silent detach flip)', () => {
|
||||
// The leading-flag loop breaks at the first positional, so the `escaped`
|
||||
// flag never fires for a `--` placed later. A literal `--` ANYWHERE must
|
||||
// still mean "hoist nothing" — otherwise `agent run note -- body --detach`
|
||||
// silently detaches and drops the `--` as junk.
|
||||
const { flags, rest } = agentTesting.parseRunFlags(['note', '--', 'body', '--detach']);
|
||||
expect(flags.detach).toBe(false);
|
||||
expect(rest).toEqual(['note', '--', 'body', '--detach']);
|
||||
});
|
||||
|
||||
test('#1738: value-flag missing its value throws a usage error', () => {
|
||||
expect(() => agentTesting.parseRunFlags(['--model'])).toThrow(/requires a value/);
|
||||
expect(() => agentTesting.parseRunFlags(['--model', '--detach', 'x'])).toThrow(/requires a value/);
|
||||
});
|
||||
|
||||
test('#1738: numeric value-flag rejects a non-number', () => {
|
||||
expect(() => agentTesting.parseRunFlags(['--max-turns', 'abc', 'x'])).toThrow(/expects a number/);
|
||||
});
|
||||
|
||||
test('--subagent-def + --timeout-ms parsed', () => {
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* issue #2227/#2194 (TODOS:634, codex #8) — a per-source `autopilot-cycle`
|
||||
* binds its filesystem phases to the SOURCE's own checkout (`local_path`),
|
||||
* never the global brain's `sync.repo_path`.
|
||||
*
|
||||
* Pre-fix the handler fed `repoPath` (the global checkout) into runCycle even
|
||||
* when `source_id` was set, so FS phases (sync/lint/extract) ran against the
|
||||
* wrong tree while DB freshness was stamped for `source_id` — mixed scope.
|
||||
* That made the failure-cooldown and freshness gates attribute work to the
|
||||
* wrong source, the prerequisite codex flagged before the storm-breaker could
|
||||
* be trusted.
|
||||
*
|
||||
* Drives the REAL handler captured from registerBuiltinHandlers (not a
|
||||
* source-grep) so a reintroduced repoPath fallthrough fails here. PGLite
|
||||
* in-memory. The report's `brain_dir` mirrors the cycle's effective brainDir
|
||||
* (cycle.ts:2324), so it's the observable proxy for "which checkout did FS
|
||||
* phases bind to".
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
async function captureHandlers(): Promise<Map<string, (job: any) => Promise<any>>> {
|
||||
const handlers = new Map<string, (job: any) => Promise<any>>();
|
||||
const fakeWorker = { register(name: string, fn: (job: any) => Promise<any>) { handlers.set(name, fn); } };
|
||||
await registerBuiltinHandlers(fakeWorker as never, engine);
|
||||
return handlers;
|
||||
}
|
||||
|
||||
describe('autopilot-cycle handler — per-source checkout binding (#2227/#2194)', () => {
|
||||
test('source_id with local_path → brainDir is the SOURCE checkout, not the global repo', async () => {
|
||||
const sourceDir = mkdtempSync(join(tmpdir(), 'gbrain-src-'));
|
||||
// A DIFFERENT global checkout must NOT win for a per-source job.
|
||||
await engine.setConfig('sync.repo_path', '/some/global/brain/checkout');
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
|
||||
VALUES ('repo-a', 'Repo A', $1, '{}'::jsonb, false, now())`,
|
||||
[sourceDir],
|
||||
);
|
||||
|
||||
const handlers = await captureHandlers();
|
||||
const handler = handlers.get('autopilot-cycle')!;
|
||||
// DB-only phase keeps the test cheap; brain_dir is stamped from opts.brainDir
|
||||
// regardless of which phases run, so it still proves the binding.
|
||||
const result = await handler({
|
||||
data: { source_id: 'repo-a', phases: ['resolve_symbol_edges'] },
|
||||
signal: undefined,
|
||||
});
|
||||
|
||||
expect(result.report.brain_dir).toBe(sourceDir);
|
||||
expect(result.report.brain_dir).not.toBe('/some/global/brain/checkout');
|
||||
});
|
||||
|
||||
test('source_id with NULL local_path → brainDir is null (FS phases skip), never the global repo', async () => {
|
||||
// The mixed-scope bug: a pure-DB source must NOT fall through to repoPath.
|
||||
await engine.setConfig('sync.repo_path', '/some/global/brain/checkout');
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
|
||||
VALUES ('db-only', 'DB Only', NULL, '{}'::jsonb, false, now())`,
|
||||
[],
|
||||
);
|
||||
|
||||
const handlers = await captureHandlers();
|
||||
const handler = handlers.get('autopilot-cycle')!;
|
||||
const result = await handler({
|
||||
data: { source_id: 'db-only', phases: ['resolve_symbol_edges'] },
|
||||
signal: undefined,
|
||||
});
|
||||
|
||||
expect(result.report.brain_dir).toBeNull();
|
||||
expect(result.report.brain_dir).not.toBe('/some/global/brain/checkout');
|
||||
});
|
||||
|
||||
test('legacy (no source_id) keeps the global repoPath — back-compat', async () => {
|
||||
const globalDir = mkdtempSync(join(tmpdir(), 'gbrain-global-'));
|
||||
await engine.setConfig('sync.repo_path', globalDir);
|
||||
|
||||
const handlers = await captureHandlers();
|
||||
const handler = handlers.get('autopilot-cycle')!;
|
||||
const result = await handler({
|
||||
data: { phases: ['resolve_symbol_edges'] },
|
||||
signal: undefined,
|
||||
});
|
||||
|
||||
expect(result.report.brain_dir).toBe(globalDir);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* #2194 fix #2 — failure cooldown (the storm-breaker).
|
||||
*
|
||||
* A source whose autopilot-cycle keeps failing re-dispatched every 5-min tick
|
||||
* (only SUCCESS gated dispatch), so the same handful of sources failed and
|
||||
* re-fanned-out forever — 200+ dead jobs/24h. The cooldown backs a failed
|
||||
* source off with bounded exponential delay, read at dispatch from minion_jobs
|
||||
* (dead/failed rows) AND re-checked at claim time (codex #5). A success clears
|
||||
* it (codex #7). These tests pin the pure math, the engine query, the
|
||||
* null-source guard (codex #6), and the dispatch/claim-time gates.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
cooldownMinForCount,
|
||||
isInFailureCooldown,
|
||||
readRecentSourceFailures,
|
||||
isSourceInCooldown,
|
||||
selectSourcesForDispatch,
|
||||
resolveFailureCooldownOpts,
|
||||
type SourceFailure,
|
||||
type CooldownOpts,
|
||||
} from '../src/commands/autopilot-fanout.ts';
|
||||
import type { SourceRow } from '../src/core/engine.ts';
|
||||
|
||||
const OPTS: CooldownOpts = { baseMin: 10, capMin: 120 };
|
||||
|
||||
describe('cooldownMinForCount — bounded exponential', () => {
|
||||
test('grows 10, 20, 40, 80 then caps at 120', () => {
|
||||
expect(cooldownMinForCount(1, OPTS)).toBe(10);
|
||||
expect(cooldownMinForCount(2, OPTS)).toBe(20);
|
||||
expect(cooldownMinForCount(3, OPTS)).toBe(40);
|
||||
expect(cooldownMinForCount(4, OPTS)).toBe(80);
|
||||
expect(cooldownMinForCount(5, OPTS)).toBe(120); // 160 capped to 120
|
||||
expect(cooldownMinForCount(99, OPTS)).toBe(120);
|
||||
});
|
||||
test('zero/negative count or disabled base → 0', () => {
|
||||
expect(cooldownMinForCount(0, OPTS)).toBe(0);
|
||||
expect(cooldownMinForCount(3, { baseMin: 0, capMin: 120 })).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isInFailureCooldown — pure decision', () => {
|
||||
const now = Date.UTC(2026, 5, 16, 12, 0, 0);
|
||||
const ago = (min: number) => new Date(now - min * 60_000);
|
||||
|
||||
test('no failure record → not in cooldown', () => {
|
||||
expect(isInFailureCooldown(undefined, null, now, OPTS)).toBe(false);
|
||||
});
|
||||
test('disabled (baseMin 0) → never in cooldown', () => {
|
||||
expect(isInFailureCooldown({ count: 5, lastFailedAt: ago(1) }, null, now, { baseMin: 0, capMin: 120 })).toBe(false);
|
||||
});
|
||||
test('failed 5min ago, count 1 (cooldown 10) → in cooldown', () => {
|
||||
expect(isInFailureCooldown({ count: 1, lastFailedAt: ago(5) }, null, now, OPTS)).toBe(true);
|
||||
});
|
||||
test('failed 15min ago, count 1 (cooldown 10) → recovered by time', () => {
|
||||
expect(isInFailureCooldown({ count: 1, lastFailedAt: ago(15) }, null, now, OPTS)).toBe(false);
|
||||
});
|
||||
test('success at/after the latest failure → cleared (codex #7)', () => {
|
||||
const failure: SourceFailure = { count: 3, lastFailedAt: ago(5) };
|
||||
expect(isInFailureCooldown(failure, ago(4), now, OPTS)).toBe(false); // success 4min ago > fail 5min ago
|
||||
});
|
||||
test('success BEFORE the latest failure, still in window → suppressed', () => {
|
||||
const failure: SourceFailure = { count: 3, lastFailedAt: ago(5) };
|
||||
expect(isInFailureCooldown(failure, ago(30), now, OPTS)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('selectSourcesForDispatch — cooldown bucket', () => {
|
||||
const src = (id: string): SourceRow => ({ id, name: id, config: {} } as SourceRow);
|
||||
test('a stale source in cooldown is held in skippedCooldown, not dispatched', () => {
|
||||
const sources = [src('a'), src('b')];
|
||||
const failures = new Map<string, SourceFailure>([
|
||||
['a', { count: 1, lastFailedAt: new Date(Date.now() - 60_000) }], // 1min ago, cooldown 10min
|
||||
]);
|
||||
const r = selectSourcesForDispatch(sources, 4, Date.now(), 60, failures, OPTS);
|
||||
expect(r.dispatch.map(s => s.id)).toEqual(['b']);
|
||||
expect(r.skippedCooldown.map(s => s.id)).toEqual(['a']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('readRecentSourceFailures + isSourceInCooldown (PGLite)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
afterAll(async () => { await engine.disconnect(); });
|
||||
beforeEach(async () => { await resetPgliteState(engine); });
|
||||
|
||||
async function addJob(status: string, sourceId: string | null, finishedMinAgo: number): Promise<void> {
|
||||
const finished = new Date(Date.now() - finishedMinAgo * 60_000).toISOString();
|
||||
const data = sourceId === null ? {} : { source_id: sourceId };
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data, finished_at) VALUES ('autopilot-cycle', $1, $2, $3)`,
|
||||
[status, data, finished],
|
||||
);
|
||||
}
|
||||
|
||||
test('groups dead/failed jobs by source with count + max(finished_at)', async () => {
|
||||
await addJob('dead', 'repo-a', 5);
|
||||
await addJob('failed', 'repo-a', 2);
|
||||
await addJob('dead', 'repo-b', 10);
|
||||
await addJob('completed', 'repo-a', 1); // not counted
|
||||
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
|
||||
expect(map.get('repo-a')?.count).toBe(2);
|
||||
expect(map.get('repo-b')?.count).toBe(1);
|
||||
expect(map.has('repo-a')).toBe(true);
|
||||
// last failed is the most recent of the two (2 min ago).
|
||||
const lastA = map.get('repo-a')!.lastFailedAt.getTime();
|
||||
expect(Date.now() - lastA).toBeLessThan(4 * 60_000);
|
||||
});
|
||||
|
||||
test('null source_id rows are excluded (codex #6)', async () => {
|
||||
await addJob('dead', null, 3);
|
||||
await addJob('dead', 'repo-c', 3);
|
||||
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
|
||||
expect(map.has('repo-c')).toBe(true);
|
||||
expect([...map.keys()].some(k => !k)).toBe(false);
|
||||
expect(map.size).toBe(1);
|
||||
});
|
||||
|
||||
test('failures older than the window are not counted', async () => {
|
||||
await addJob('dead', 'repo-old', 500); // way outside a 120min window
|
||||
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
|
||||
expect(map.has('repo-old')).toBe(false);
|
||||
});
|
||||
|
||||
test('isSourceInCooldown: recent failure → true; cleared after a success stamp', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, created_at) VALUES ('repo-cd', 'r', '{}'::jsonb, now())`, [],
|
||||
);
|
||||
await addJob('dead', 'repo-cd', 1); // 1 min ago, count 1 → 10min cooldown
|
||||
expect(await isSourceInCooldown(engine, 'repo-cd')).toBe(true);
|
||||
|
||||
// Operator repairs + a successful cycle stamps last_source_cycle_at NOW.
|
||||
await engine.updateSourceConfig('repo-cd', { last_source_cycle_at: new Date().toISOString() });
|
||||
expect(await isSourceInCooldown(engine, 'repo-cd')).toBe(false);
|
||||
});
|
||||
|
||||
test('isSourceInCooldown returns false when cooldown disabled (failure_cooldown_min=0)', async () => {
|
||||
await engine.setConfig('autopilot.failure_cooldown_min', '0');
|
||||
await addJob('dead', 'repo-dis', 1);
|
||||
expect(await isSourceInCooldown(engine, 'repo-dis')).toBe(false);
|
||||
const opts = await resolveFailureCooldownOpts(engine);
|
||||
expect(opts.baseMin).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* #2194 fix #1 (codex #9 / D5): resolveEffectiveFanoutMax clamps the per-tick
|
||||
* fan-out to the worker's effective concurrency (max(1, concurrency-1),
|
||||
* reserving ≥1 slot) — but ONLY when a LIVE supervisor holds the queue lock.
|
||||
* A stale `started` audit row must not shrink throughput for a supervisor that
|
||||
* isn't running that config, so with no live holder the clamp is skipped and
|
||||
* the unclamped base is used.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
|
||||
import { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } from '../src/core/minions/supervisor.ts';
|
||||
import { computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
|
||||
import { resolveEffectiveFanoutMax } from '../src/commands/autopilot-fanout.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let auditDir: string;
|
||||
const prevAuditDir = process.env.GBRAIN_AUDIT_DIR;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
auditDir = mkdtempSync(join(tmpdir(), 'gbrain-clamp-'));
|
||||
process.env.GBRAIN_AUDIT_DIR = auditDir;
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'gbrain-supervisor:%'`);
|
||||
// base fan-out: override to 8 so the clamp's effect is visible on PGLite
|
||||
// (whose natural default is 1). The clamp logic is engine-agnostic.
|
||||
await engine.setConfig('autopilot.fanout_max_per_tick', '8');
|
||||
await engine.setConfig('autopilot.fanout_clamp_to_concurrency', 'true');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (prevAuditDir === undefined) delete process.env.GBRAIN_AUDIT_DIR;
|
||||
else process.env.GBRAIN_AUDIT_DIR = prevAuditDir;
|
||||
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* noop */ }
|
||||
});
|
||||
|
||||
function writeStarted(concurrency: number): void {
|
||||
const file = join(auditDir, computeSupervisorAuditFilename());
|
||||
writeFileSync(file, JSON.stringify({
|
||||
event: 'started', ts: new Date().toISOString(), supervisor_pid: 4242,
|
||||
queue: 'default', concurrency,
|
||||
}) + '\n', 'utf8');
|
||||
}
|
||||
|
||||
describe('resolveEffectiveFanoutMax — clamp gated on live supervisor (#2194/codex #9)', () => {
|
||||
test('NO live holder → no clamp (stale audit row cannot shrink throughput)', async () => {
|
||||
writeStarted(3); // audit says concurrency 3, but no live lock holder
|
||||
const n = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
expect(n).toBe(8); // unclamped base
|
||||
});
|
||||
|
||||
test('live holder + concurrency 3 → clamp to max(1, 3-1) = 2', async () => {
|
||||
writeStarted(3);
|
||||
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
|
||||
expect(holder).not.toBeNull();
|
||||
try {
|
||||
const n = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
expect(n).toBe(2);
|
||||
} finally {
|
||||
await holder!.release();
|
||||
}
|
||||
});
|
||||
|
||||
test('live holder but clamp disabled → unclamped base', async () => {
|
||||
await engine.setConfig('autopilot.fanout_clamp_to_concurrency', 'false');
|
||||
writeStarted(3);
|
||||
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
|
||||
try {
|
||||
const n = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
expect(n).toBe(8);
|
||||
} finally {
|
||||
await holder!.release();
|
||||
}
|
||||
});
|
||||
|
||||
test('live holder + concurrency 1 → floor at 1 (never below 1)', async () => {
|
||||
writeStarted(1);
|
||||
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
|
||||
try {
|
||||
const n = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
expect(n).toBe(1);
|
||||
} finally {
|
||||
await holder!.release();
|
||||
}
|
||||
});
|
||||
|
||||
test('live holder but no started event (concurrency unknown) → no clamp', async () => {
|
||||
// lock row exists but audit has no concurrency → fall back to base.
|
||||
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
|
||||
try {
|
||||
const n = await resolveEffectiveFanoutMax(engine, 'default');
|
||||
expect(n).toBe(8);
|
||||
} finally {
|
||||
await holder!.release();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -28,8 +28,11 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('imports resolveFanoutMax (so PGLite gets fanoutMax=1 per codex P1-3)', () => {
|
||||
expect(AUTOPILOT_SRC).toMatch(/resolveFanoutMax/);
|
||||
test('imports resolveEffectiveFanoutMax (clamps to worker concurrency; PGLite base still 1)', () => {
|
||||
// #2194 fix #1: autopilot now resolves the CLAMPED fan-out (gated on a live
|
||||
// supervisor) instead of the raw resolveFanoutMax. The clamp wraps
|
||||
// resolveFanoutMax, so PGLite's base-1 still holds (codex P1-3).
|
||||
expect(AUTOPILOT_SRC).toMatch(/resolveEffectiveFanoutMax/);
|
||||
});
|
||||
|
||||
test('calls dispatchPerSource within the shouldFullCycle branch', () => {
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
/**
|
||||
* #2194 fix #3 / #2227 bug #3 — the cycle split.
|
||||
*
|
||||
* Per-source autopilot cycles run ONLY source-scoped (+ mixed) phases; the
|
||||
* brain-wide `global` phases run ONCE in a separate autopilot-global-maintenance
|
||||
* job. This replaces the rejected skip-and-stamp-fresh design (codex #1/#2): the
|
||||
* split makes single-flight structural (one global job, not N concurrent embeds)
|
||||
* and never marks a source "fresh" for global work it didn't do. These tests pin
|
||||
* the phase partition, the dispatch gate, the per-source phase set, and the
|
||||
* global handler stamping autopilot.last_global_at.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
|
||||
import {
|
||||
ALL_PHASES,
|
||||
GLOBAL_PHASES,
|
||||
NON_GLOBAL_PHASES,
|
||||
PHASE_SCOPE,
|
||||
LAST_GLOBAL_AT_KEY,
|
||||
} from '../src/core/cycle.ts';
|
||||
import {
|
||||
dispatchGlobalMaintenance,
|
||||
isGlobalMaintenanceStale,
|
||||
dispatchPerSource,
|
||||
} from '../src/commands/autopilot-fanout.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
describe('cycle phase partition (#2194 fix #3)', () => {
|
||||
test('GLOBAL ∪ NON_GLOBAL == ALL_PHASES, no overlap', () => {
|
||||
const union = new Set([...GLOBAL_PHASES, ...NON_GLOBAL_PHASES]);
|
||||
expect(union.size).toBe(ALL_PHASES.length);
|
||||
for (const p of ALL_PHASES) expect(union.has(p)).toBe(true);
|
||||
// No phase in both.
|
||||
const overlap = GLOBAL_PHASES.filter((p) => NON_GLOBAL_PHASES.includes(p));
|
||||
expect(overlap).toEqual([]);
|
||||
});
|
||||
|
||||
test('every GLOBAL phase is PHASE_SCOPE==="global"; embed is global, lint is not', () => {
|
||||
for (const p of GLOBAL_PHASES) expect(PHASE_SCOPE[p]).toBe('global');
|
||||
expect(GLOBAL_PHASES).toContain('embed');
|
||||
expect(GLOBAL_PHASES).toContain('orphans');
|
||||
expect(GLOBAL_PHASES).toContain('purge');
|
||||
expect(NON_GLOBAL_PHASES).toContain('lint');
|
||||
expect(NON_GLOBAL_PHASES).toContain('sync');
|
||||
expect(NON_GLOBAL_PHASES).not.toContain('embed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isGlobalMaintenanceStale', () => {
|
||||
const now = Date.UTC(2026, 5, 16, 12, 0, 0);
|
||||
test('null/unparseable → stale (must run)', () => {
|
||||
expect(isGlobalMaintenanceStale(null, now)).toBe(true);
|
||||
expect(isGlobalMaintenanceStale('not-a-date', now)).toBe(true);
|
||||
});
|
||||
test('older than floor → stale; within floor → fresh', () => {
|
||||
expect(isGlobalMaintenanceStale(new Date(now - 61 * 60_000).toISOString(), now, 60)).toBe(true);
|
||||
expect(isGlobalMaintenanceStale(new Date(now - 10 * 60_000).toISOString(), now, 60)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('dispatchGlobalMaintenance — single-flight gate', () => {
|
||||
function stubs(lastGlobalAt: string | null) {
|
||||
const added: Array<{ name: string; data: any; opts: any }> = [];
|
||||
const engine = {
|
||||
kind: 'postgres' as const,
|
||||
getConfig: async (k: string) => (k === LAST_GLOBAL_AT_KEY ? lastGlobalAt : null),
|
||||
} as unknown as BrainEngine;
|
||||
const queue = {
|
||||
add: async (name: string, data: unknown, opts: Record<string, unknown>) => {
|
||||
added.push({ name, data, opts }); return { id: 1 };
|
||||
},
|
||||
} as any;
|
||||
return { engine, queue, added };
|
||||
}
|
||||
|
||||
test('stale (never run) → dispatches one global job with single-flight opts', async () => {
|
||||
const { engine, queue, added } = stubs(null);
|
||||
const r = await dispatchGlobalMaintenance(engine, queue, { repoPath: '/tmp', slot: 's1', timeoutMs: 1, jsonMode: true, emit: () => {} });
|
||||
expect(r.dispatched).toBe(true);
|
||||
expect(added.length).toBe(1);
|
||||
expect(added[0].name).toBe('autopilot-global-maintenance');
|
||||
expect(added[0].opts.idempotency_key).toBe('autopilot-global:s1');
|
||||
expect(added[0].opts.maxWaiting).toBe(1); // structural single-flight
|
||||
expect(added[0].data.phases).toEqual(GLOBAL_PHASES);
|
||||
});
|
||||
|
||||
test('fresh → does NOT dispatch', async () => {
|
||||
const { engine, queue, added } = stubs(new Date().toISOString());
|
||||
const r = await dispatchGlobalMaintenance(engine, queue, { repoPath: '/tmp', slot: 's1', timeoutMs: 1, jsonMode: true, emit: () => {} });
|
||||
expect(r.dispatched).toBe(false);
|
||||
expect(added.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('dispatchPerSource — per-source jobs carry NON_GLOBAL phases (no embed)', () => {
|
||||
test('each per-source job sets phases = NON_GLOBAL_PHASES', async () => {
|
||||
const sources = [{ id: 'repo-a', name: 'a', config: {} }, { id: 'repo-b', name: 'b', config: {} }];
|
||||
const added: any[] = [];
|
||||
const engine = {
|
||||
kind: 'postgres' as const,
|
||||
listAllSources: async () => sources,
|
||||
getConfig: async () => null,
|
||||
executeRaw: async () => [],
|
||||
} as unknown as BrainEngine;
|
||||
const queue = { add: async (name: string, data: unknown, opts: unknown) => { added.push({ name, data, opts }); return { id: added.length }; } } as any;
|
||||
await dispatchPerSource(engine, queue, { repoPath: '/tmp', slot: 's', timeoutMs: 1, fanoutMax: 4, jsonMode: true, emit: () => {}, log: () => {} });
|
||||
expect(added.length).toBe(2);
|
||||
for (const j of added) {
|
||||
expect(j.data.phases).toEqual(NON_GLOBAL_PHASES);
|
||||
expect(j.data.phases).not.toContain('embed');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('autopilot-global-maintenance handler stamps last_global_at (PGLite)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 30000);
|
||||
afterAll(async () => { await engine.disconnect(); });
|
||||
beforeEach(async () => { await resetPgliteState(engine); });
|
||||
|
||||
async function captureHandlers() {
|
||||
const handlers = new Map<string, (job: any) => Promise<any>>();
|
||||
const fakeWorker = { register(name: string, fn: (job: any) => Promise<any>) { handlers.set(name, fn); } };
|
||||
await registerBuiltinHandlers(fakeWorker as never, engine);
|
||||
return handlers;
|
||||
}
|
||||
|
||||
test('runs global phases (no source_id) and stamps autopilot.last_global_at on success', async () => {
|
||||
expect(await engine.getConfig(LAST_GLOBAL_AT_KEY)).toBeNull();
|
||||
const handlers = await captureHandlers();
|
||||
const handler = handlers.get('autopilot-global-maintenance');
|
||||
expect(handler).toBeTruthy();
|
||||
|
||||
const result = await handler!({ data: { phases: ['orphans', 'embed'] }, signal: undefined });
|
||||
// The cycle ran the requested global phases (DB-only on an empty brain).
|
||||
expect(result.report.phases.some((p: any) => p.phase === 'orphans')).toBe(true);
|
||||
expect(['ok', 'clean', 'partial']).toContain(result.report.status);
|
||||
// Freshness stamped so the dispatch gate backs off.
|
||||
const stamped = await engine.getConfig(LAST_GLOBAL_AT_KEY);
|
||||
expect(stamped).not.toBeNull();
|
||||
expect(Number.isFinite(new Date(stamped!).getTime())).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -50,6 +50,7 @@ async function runUntilTerminal(
|
||||
h: Harness,
|
||||
overrides: Partial<{
|
||||
maxCrashes: number;
|
||||
hardStopMaxCrashes: number;
|
||||
_backoffFloorMs: number;
|
||||
cleanRestartBudget: number;
|
||||
cleanRestartWindowMs: number;
|
||||
@@ -71,6 +72,7 @@ async function runUntilTerminal(
|
||||
cliPath: h.workerScript,
|
||||
args: [],
|
||||
maxCrashes: overrides.maxCrashes ?? 3,
|
||||
hardStopMaxCrashes: overrides.hardStopMaxCrashes,
|
||||
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
|
||||
cleanRestartBudget: overrides.cleanRestartBudget,
|
||||
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
|
||||
@@ -159,12 +161,15 @@ if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
|
||||
try {
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
// issue #1994: the soft budget no longer gives up; pin the hard
|
||||
// ceiling to 3 so this counting test still fires give-up at 3.
|
||||
hardStopMaxCrashes: 3,
|
||||
_backoffFloorMs: 5,
|
||||
stopAfterEvents: 200,
|
||||
});
|
||||
|
||||
expect(res.maxCrashesFired).not.toBeNull();
|
||||
// 3 code!=0 exits → max_crashes=3
|
||||
// 3 code!=0 exits → hard ceiling=3
|
||||
expect(res.maxCrashesFired!.count).toBe(3);
|
||||
|
||||
const exits = res.events.filter((e) => e.kind === 'worker_exited');
|
||||
@@ -235,6 +240,7 @@ esac
|
||||
|
||||
const res = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
hardStopMaxCrashes: 3, // issue #1994: pin give-up to 3 for this counting test
|
||||
_backoffFloorMs: 5,
|
||||
_now: fakeNow,
|
||||
stopAfterEvents: 200,
|
||||
@@ -471,6 +477,66 @@ esac
|
||||
});
|
||||
});
|
||||
|
||||
// issue #1994 (#2227 tail): crossing the SOFT crash budget no longer
|
||||
// permanently gives up. The supervisor enters degraded mode (capped backoff
|
||||
// + loud warn) so a transient outage self-heals; permanent give-up fires only
|
||||
// at the much-higher hard ceiling.
|
||||
describe('degraded-mode crash backoff (issue #1994)', () => {
|
||||
it('crossing the soft budget does NOT give up; it warns and keeps retrying to the hard ceiling', async () => {
|
||||
const h = makeHarness('degraded-softbudget', 'exit 1');
|
||||
try {
|
||||
const { events, maxCrashesFired } = await runUntilTerminal(h, {
|
||||
maxCrashes: 3, // soft budget
|
||||
hardStopMaxCrashes: 6, // hard ceiling
|
||||
_backoffFloorMs: 1,
|
||||
stopAfterEvents: 200,
|
||||
});
|
||||
// Permanent give-up fired at the HARD ceiling (6), not the soft budget (3).
|
||||
expect(maxCrashesFired).not.toBeNull();
|
||||
expect(maxCrashesFired!.count).toBe(6);
|
||||
expect(maxCrashesFired!.max).toBe(6);
|
||||
|
||||
// The soft-budget crossing announced degraded mode (at least once).
|
||||
const degraded = events.filter(
|
||||
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
|
||||
e.kind === 'health_warn' && e.reason === 'crash_budget_degraded',
|
||||
);
|
||||
expect(degraded.length).toBeGreaterThanOrEqual(1);
|
||||
expect(degraded[0].max).toBe(3);
|
||||
expect(degraded[0].count).toBeGreaterThanOrEqual(3);
|
||||
|
||||
// It kept respawning past the soft budget (more than 3 crash exits).
|
||||
const crashes = events.filter(
|
||||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
|
||||
e.kind === 'worker_exited' && e.code === 1,
|
||||
);
|
||||
expect(crashes.length).toBe(6);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it('hardStopMaxCrashes=0 disables permanent give-up (retry-forever-with-backoff)', async () => {
|
||||
const h = makeHarness('degraded-noforever', 'exit 1');
|
||||
try {
|
||||
const { events, maxCrashesFired } = await runUntilTerminal(h, {
|
||||
maxCrashes: 3,
|
||||
hardStopMaxCrashes: 0, // never permanently stop
|
||||
_backoffFloorMs: 1,
|
||||
stopAfterEvents: 40, // the safety net stops the test, not a give-up
|
||||
});
|
||||
// Never gave up despite many crashes past the soft budget.
|
||||
expect(maxCrashesFired).toBeNull();
|
||||
const crashes = events.filter(
|
||||
(e) => e.kind === 'worker_exited' && (e as any).code === 1,
|
||||
);
|
||||
expect(crashes.length).toBeGreaterThan(3);
|
||||
} finally {
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #1801 — restartCurrentChild + killChild liveness fix', () => {
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
// ESRCH = no such process (dead). EPERM = process exists but we can't
|
||||
|
||||
@@ -21,6 +21,8 @@ import {
|
||||
inspectLock,
|
||||
listStaleLocks,
|
||||
deleteLockRow,
|
||||
liveSyncStatus,
|
||||
syncLockId,
|
||||
} from '../src/core/db-lock.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -183,3 +185,39 @@ describe('deleteLockRow', () => {
|
||||
await handle!.release();
|
||||
});
|
||||
});
|
||||
|
||||
describe('liveSyncStatus (#1950)', () => {
|
||||
test('returns null when the source holds no lock (idle)', async () => {
|
||||
const live = await liveSyncStatus(engine, 'never-synced');
|
||||
expect(live).toBeNull();
|
||||
});
|
||||
|
||||
test('returns the holder when a live (non-expired) sync lock is held', async () => {
|
||||
const handle = await tryAcquireDbLock(engine, syncLockId('busy-source'));
|
||||
expect(handle).not.toBeNull();
|
||||
const live = await liveSyncStatus(engine, 'busy-source');
|
||||
expect(live).not.toBeNull();
|
||||
expect(live!.holder_pid).toBe(process.pid);
|
||||
expect(typeof live!.holder_host).toBe('string');
|
||||
expect(live!.holder_host.length).toBeGreaterThan(0);
|
||||
await handle!.release();
|
||||
});
|
||||
|
||||
test('returns null for a stale (TTL-expired) lock — structurally available, not running', async () => {
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES ($1, $2, $3, NOW() - INTERVAL '2 hours', NOW() - INTERVAL '1 hour')`,
|
||||
[syncLockId('wedged-dead'), 31337, 'old-host'],
|
||||
);
|
||||
const live = await liveSyncStatus(engine, 'wedged-dead');
|
||||
expect(live).toBeNull();
|
||||
});
|
||||
|
||||
test('is scoped per source — one source live does not mark another running', async () => {
|
||||
const handle = await tryAcquireDbLock(engine, syncLockId('source-a'));
|
||||
expect(handle).not.toBeNull();
|
||||
expect(await liveSyncStatus(engine, 'source-a')).not.toBeNull();
|
||||
expect(await liveSyncStatus(engine, 'source-b')).toBeNull();
|
||||
await handle!.release();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* #2194 fix #5: `gbrain doctor` warns when autopilot's per-tick fan-out exceeds
|
||||
* the worker's effective concurrency. Fanning out more cycles than there are
|
||||
* worker slots guarantees waiters that race the stalled-sweeper — a silent
|
||||
* misconfig the operator never saw before this check.
|
||||
*
|
||||
* Drives computeAutopilotFanoutConcurrencyCheck directly with a fake engine so
|
||||
* the fan-out (config) and concurrency (audit) inputs are controllable without
|
||||
* spawning a supervisor. The audit read is stubbed via GBRAIN_AUDIT_DIR + a
|
||||
* hand-written started event.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { computeAutopilotFanoutConcurrencyCheck } from '../src/commands/doctor.ts';
|
||||
import { computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
|
||||
|
||||
// Minimal fake engine: postgres-kind + a config map for fanout override.
|
||||
function fakeEngine(config: Record<string, string> = {}) {
|
||||
return {
|
||||
kind: 'postgres' as const,
|
||||
getConfig: async (k: string) => config[k] ?? null,
|
||||
} as any;
|
||||
}
|
||||
|
||||
let auditDir: string;
|
||||
const prevAuditDir = process.env.GBRAIN_AUDIT_DIR;
|
||||
|
||||
beforeEach(() => {
|
||||
auditDir = mkdtempSync(join(tmpdir(), 'gbrain-fanout-doctor-'));
|
||||
process.env.GBRAIN_AUDIT_DIR = auditDir;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (prevAuditDir === undefined) delete process.env.GBRAIN_AUDIT_DIR;
|
||||
else process.env.GBRAIN_AUDIT_DIR = prevAuditDir;
|
||||
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* noop */ }
|
||||
});
|
||||
|
||||
/** Write a `started` audit event with the given concurrency for queue 'default'. */
|
||||
function writeStarted(concurrency: number): void {
|
||||
const file = join(auditDir, computeSupervisorAuditFilename());
|
||||
const line = JSON.stringify({
|
||||
event: 'started',
|
||||
ts: new Date().toISOString(),
|
||||
supervisor_pid: 4242,
|
||||
queue: 'default',
|
||||
concurrency,
|
||||
});
|
||||
writeFileSync(file, line + '\n', 'utf8');
|
||||
}
|
||||
|
||||
describe('computeAutopilotFanoutConcurrencyCheck (#2194 fix #5)', () => {
|
||||
test('warns when fan-out (4) exceeds effective slots (concurrency 2 → 1)', async () => {
|
||||
writeStarted(2);
|
||||
const check = await computeAutopilotFanoutConcurrencyCheck(fakeEngine());
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toContain('exceeds worker concurrency');
|
||||
expect(check.details).toMatchObject({ fanout_max: 4, concurrency: 2, effective_slots: 1 });
|
||||
});
|
||||
|
||||
test('ok when fan-out fits (override 1, concurrency 4)', async () => {
|
||||
writeStarted(4);
|
||||
const check = await computeAutopilotFanoutConcurrencyCheck(
|
||||
fakeEngine({ 'autopilot.fanout_max_per_tick': '1' }),
|
||||
);
|
||||
expect(check.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('ok/skip when no supervisor has ever started (no noise on unsupervised brains)', async () => {
|
||||
// No started event written.
|
||||
const check = await computeAutopilotFanoutConcurrencyCheck(fakeEngine());
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('No supervisor observed');
|
||||
});
|
||||
|
||||
test('PGLite short-circuits (single-writer, fan-out is 1)', async () => {
|
||||
const check = await computeAutopilotFanoutConcurrencyCheck({ kind: 'pglite', getConfig: async () => null } as any);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('PGLite');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* #2194 fix #2 — engine-parity for the failure-cooldown query.
|
||||
*
|
||||
* readRecentSourceFailures runs ONE SQL through engine.executeRaw (the
|
||||
* engine-agnostic path), so PGLite and Postgres must return identical
|
||||
* groupings. This pins that: it seeds the SAME dead/failed autopilot-cycle rows
|
||||
* into both engines and asserts the per-source counts + the null-source
|
||||
* exclusion (codex #6) match. PGLite always runs; Postgres runs only when
|
||||
* DATABASE_URL is set (mirrors engine-parity.test.ts's gating).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import { readRecentSourceFailures } from '../../src/commands/autopilot-fanout.ts';
|
||||
|
||||
const SKIP_PG = !hasDatabase();
|
||||
|
||||
async function seed(engine: BrainEngine): Promise<void> {
|
||||
const rows: Array<[string, Record<string, unknown>, number]> = [
|
||||
['dead', { source_id: 'repo-a' }, 5],
|
||||
['failed', { source_id: 'repo-a' }, 2],
|
||||
['dead', { source_id: 'repo-b' }, 10],
|
||||
['completed', { source_id: 'repo-a' }, 1], // excluded (not a failure)
|
||||
['dead', {}, 3], // excluded (null source_id, codex #6)
|
||||
];
|
||||
for (const [status, data, minAgo] of rows) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data, finished_at) VALUES ('autopilot-cycle', $1, $2, $3)`,
|
||||
[status, data, new Date(Date.now() - minAgo * 60_000).toISOString()],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function summarize(map: Map<string, { count: number }>): Record<string, number> {
|
||||
const out: Record<string, number> = {};
|
||||
for (const [k, v] of map) out[k] = v.count;
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('failure-cooldown query — PGLite', () => {
|
||||
let engine: PGLiteEngine;
|
||||
beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); await seed(engine); }, 30000);
|
||||
afterAll(async () => { await engine.disconnect(); });
|
||||
|
||||
test('groups failures by source, excludes completed + null-source', async () => {
|
||||
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
|
||||
expect(summarize(map)).toEqual({ 'repo-a': 2, 'repo-b': 1 });
|
||||
});
|
||||
});
|
||||
|
||||
(SKIP_PG ? describe.skip : describe)('failure-cooldown query — Postgres parity', () => {
|
||||
let engine: BrainEngine;
|
||||
beforeAll(async () => { await setupDB(); engine = await getEngine(); await seed(engine); }, 60000);
|
||||
afterAll(async () => { await teardownDB(); });
|
||||
|
||||
test('Postgres returns the SAME groupings as PGLite', async () => {
|
||||
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
|
||||
expect(summarize(map)).toEqual({ 'repo-a': 2, 'repo-b': 1 });
|
||||
});
|
||||
});
|
||||
@@ -138,6 +138,10 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
|
||||
|
||||
const final = await queue.getJob(job.id);
|
||||
expect(final?.error_text).toMatch(/timeout exceeded/i);
|
||||
// #1737 regression: the runaway run is counted as a spent attempt by
|
||||
// handleTimeouts (it's the first killer to fire), so accounting reads
|
||||
// honestly instead of `attempts: 0/1 (started: 1)`.
|
||||
expect(final?.attempts_made).toBe(1);
|
||||
} finally {
|
||||
await a.disconnect();
|
||||
await b.disconnect();
|
||||
|
||||
@@ -82,6 +82,11 @@ describe('gbrain status E2E (PGLite)', () => {
|
||||
expect(parsed).toHaveProperty('workers');
|
||||
expect(parsed).toHaveProperty('queue');
|
||||
expect(parsed).toHaveProperty('autopilot');
|
||||
// #1984: version field present (local CLI version) for poller pinning;
|
||||
// a normal (un-budgeted) run is never partial.
|
||||
expect(typeof parsed.version).toBe('string');
|
||||
expect(parsed.version.length).toBeGreaterThan(0);
|
||||
expect(parsed.partial).toBeUndefined();
|
||||
});
|
||||
|
||||
test('dual cycle rows: last_full + last_targeted surface independently', async () => {
|
||||
|
||||
@@ -64,6 +64,8 @@ describe('get_status_snapshot handler shape', () => {
|
||||
expect(result.schema_version).toBe(1);
|
||||
expect(result).toHaveProperty('sync');
|
||||
expect(result).toHaveProperty('cycle');
|
||||
// #1984: the op now also reports the brain server's version (thin-client parity).
|
||||
expect(typeof result.version).toBe('string');
|
||||
expect(result).not.toHaveProperty('locks');
|
||||
expect(result).not.toHaveProperty('workers');
|
||||
expect(result).not.toHaveProperty('queue');
|
||||
|
||||
@@ -1300,6 +1300,12 @@ describe('MinionQueue: handleTimeouts', () => {
|
||||
const dead = await queue.getJob(job.id);
|
||||
expect(dead!.status).toBe('dead');
|
||||
expect(dead!.error_text).toBe('timeout exceeded');
|
||||
// #1737 regression: the timed-out run counts as a spent attempt, mirroring
|
||||
// the wall-clock + stall dead-letter paths. Without this the job reads
|
||||
// `attempts: 0/N (started: N)`. Asserted on both the RETURNING row and the
|
||||
// persisted row so a future refactor can't silently drop the increment.
|
||||
expect(timedOut[0].attempts_made).toBe(1);
|
||||
expect(dead!.attempts_made).toBe(1);
|
||||
});
|
||||
|
||||
test('handleTimeouts ignores stalled jobs (lock_until > now guard)', async () => {
|
||||
|
||||
@@ -14,7 +14,65 @@
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { parseSectionFlag, runStatus } from '../src/commands/status.ts';
|
||||
import {
|
||||
parseSectionFlag,
|
||||
parseDeadlineFlag,
|
||||
withSectionDeadline,
|
||||
runStatus,
|
||||
FAST_DEADLINE_MS,
|
||||
} from '../src/commands/status.ts';
|
||||
|
||||
describe('parseDeadlineFlag (#1984)', () => {
|
||||
test('no flag → undefined (no budget)', () => {
|
||||
expect(parseDeadlineFlag([])).toBeUndefined();
|
||||
expect(parseDeadlineFlag(['--json'])).toBeUndefined();
|
||||
});
|
||||
|
||||
test('--fast applies the preset budget', () => {
|
||||
expect(parseDeadlineFlag(['--fast'])).toBe(FAST_DEADLINE_MS);
|
||||
});
|
||||
|
||||
test('--deadline-ms in both forms', () => {
|
||||
expect(parseDeadlineFlag(['--deadline-ms', '500'])).toBe(500);
|
||||
expect(parseDeadlineFlag(['--deadline-ms=750'])).toBe(750);
|
||||
});
|
||||
|
||||
test('explicit --deadline-ms wins over --fast', () => {
|
||||
expect(parseDeadlineFlag(['--fast', '--deadline-ms=100'])).toBe(100);
|
||||
});
|
||||
|
||||
test('non-positive / non-numeric → usage_error', () => {
|
||||
expect(parseDeadlineFlag(['--deadline-ms', '0'])).toBe('usage_error');
|
||||
expect(parseDeadlineFlag(['--deadline-ms', '-5'])).toBe('usage_error');
|
||||
expect(parseDeadlineFlag(['--deadline-ms', 'soon'])).toBe('usage_error');
|
||||
});
|
||||
|
||||
test('bare --deadline-ms with no value → usage_error (not a silent no-budget fallthrough)', () => {
|
||||
expect(parseDeadlineFlag(['--deadline-ms'])).toBe('usage_error');
|
||||
expect(parseDeadlineFlag(['--fast', '--deadline-ms'])).toBe('usage_error');
|
||||
});
|
||||
});
|
||||
|
||||
describe('withSectionDeadline (#1984)', () => {
|
||||
test('resolves the value when it beats the budget', async () => {
|
||||
let timedOut = false;
|
||||
const v = await withSectionDeadline(Promise.resolve(42), 1000, () => { timedOut = true; });
|
||||
expect(v).toBe(42);
|
||||
expect(timedOut).toBe(false);
|
||||
});
|
||||
|
||||
test('returns undefined + fires onTimeout when the budget elapses', async () => {
|
||||
let timedOut = false;
|
||||
const v = await withSectionDeadline(new Promise<number>(() => {}), 10, () => { timedOut = true; });
|
||||
expect(v).toBeUndefined();
|
||||
expect(timedOut).toBe(true);
|
||||
});
|
||||
|
||||
test('no budget (undefined/<=0) awaits the promise as-is', async () => {
|
||||
expect(await withSectionDeadline(Promise.resolve('x'), undefined, () => {})).toBe('x');
|
||||
expect(await withSectionDeadline(Promise.resolve('y'), 0, () => {})).toBe('y');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseSectionFlag', () => {
|
||||
test('no --section flag → undefined (all sections)', () => {
|
||||
@@ -73,4 +131,14 @@ describe('runStatus exit codes', () => {
|
||||
expect(r.exitCode).toBe(1);
|
||||
expect(captured).toMatch(/snapshot failed|no engine connected/);
|
||||
});
|
||||
|
||||
test('#1984: invalid --deadline-ms → exit 2 (usage error)', async () => {
|
||||
let captured = '';
|
||||
const r = await runStatus(null, ['--deadline-ms', '0'], {
|
||||
stdout: () => {},
|
||||
stderr: (s: string) => { captured += s; },
|
||||
});
|
||||
expect(r.exitCode).toBe(2);
|
||||
expect(captured).toContain('--deadline-ms');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -199,3 +199,45 @@ describe('summarizeCrashes — aggregation', () => {
|
||||
expect(summary.clean_exits).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// issue #2227 bug #1/#2: a duplicate supervisor (different $HOME / --pid-file)
|
||||
// passes the pidfile guard but loses the queue-scoped DB singleton lock
|
||||
// (#1849) and `process.exit(ExitCodes.LOCK_HELD)`s. That fence-exit happens in
|
||||
// supervisor.ts:start() BEFORE the worker is ever spawned and BEFORE the
|
||||
// `started` event is emitted (the DB-lock acquire at :528 precedes the
|
||||
// `started` emit at :555), so the loser contributes NO `worker_exited` events
|
||||
// to the audit trail. The crash ledger (summarizeCrashes) moves ONLY on
|
||||
// `worker_exited`, so a fence-collision can never burn the crash budget the
|
||||
// guard exists to protect. The field report claimed 20 "unknown" crashes were
|
||||
// fence exits; this pins that the fence path is structurally uncountable, so a
|
||||
// future refactor that (wrongly) logs a `worker_exited` on the LOCK_HELD path
|
||||
// would fail here instead of silently re-introducing the breaker-trip loop.
|
||||
describe('summarizeCrashes — LOCK_HELD fence-exit is never a crash (issue #2227)', () => {
|
||||
test('a losing duplicate supervisor (no worker_exited) contributes zero crashes', () => {
|
||||
// The loser exits at the DB-lock fence before emitting `started`, so its
|
||||
// audit contribution is empty. Even paired with the winner's healthy
|
||||
// stream, total crashes stay 0.
|
||||
const winnerHealthy: SupervisorEmission[] = [
|
||||
evt('started', { supervisor_pid: 4242, queue: 'default', concurrency: 3 }),
|
||||
evt('worker_spawned', { worker_pid: 4243 }),
|
||||
// ... no worker_exited yet (winner still running)
|
||||
];
|
||||
expect(summarizeCrashes(winnerHealthy).total).toBe(0);
|
||||
});
|
||||
|
||||
test('started + stopped with no worker_exited (clean fence path) is zero crashes', () => {
|
||||
// Defensive: even if a future change made the loser emit lifecycle events
|
||||
// (started/shutting_down/stopped) around the LOCK_HELD exit, none of those
|
||||
// are `worker_exited`, so the ledger must stay at 0. The only way this test
|
||||
// fails is if someone logs a `worker_exited` on the fence path — which is
|
||||
// exactly the regression #2227 fix #2 guards against.
|
||||
const fenceStream: SupervisorEmission[] = [
|
||||
evt('started', { supervisor_pid: 5252 }),
|
||||
evt('shutting_down', { reason: 'LOCK_HELD' }),
|
||||
evt('stopped', { exit_code: 2 }),
|
||||
];
|
||||
const summary = summarizeCrashes(fenceStream);
|
||||
expect(summary.total).toBe(0);
|
||||
expect(summary.clean_exits).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -14,9 +14,26 @@ import { existsSync, unlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
|
||||
import { MinionSupervisor, ExitCodes, supervisorLockId, classifySupervisorSingleton } from '../src/core/minions/supervisor.ts';
|
||||
import type { DbLockHandle } from '../src/core/db-lock.ts';
|
||||
import { tryAcquireDbLock, inspectLock, isLockHolderLive } from '../src/core/db-lock.ts';
|
||||
import { MinionSupervisor, ExitCodes, supervisorLockId, classifySupervisorSingleton, SUPERVISOR_LOCK_TTL_MIN } from '../src/core/minions/supervisor.ts';
|
||||
import type { DbLockHandle, LockSnapshot } from '../src/core/db-lock.ts';
|
||||
|
||||
// Build a LockSnapshot fixture for the isLockHolderLive matrix. Only ttl_expired
|
||||
// and ms_since_last_refresh are consulted; the rest are filled for shape.
|
||||
function snap(over: Partial<LockSnapshot>): LockSnapshot {
|
||||
return {
|
||||
id: 'gbrain-supervisor:default',
|
||||
holder_pid: 4242,
|
||||
holder_host: 'box',
|
||||
acquired_at: new Date(),
|
||||
ttl_expires_at: new Date(),
|
||||
age_ms: 1000,
|
||||
ttl_expired: false,
|
||||
last_refreshed_at: new Date(),
|
||||
ms_since_last_refresh: 0,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
@@ -147,6 +164,48 @@ describe('#1849 LOCK_HELD path does not strand the pidfile', () => {
|
||||
|
||||
});
|
||||
|
||||
describe('#2227 isLockHolderLive — PID-reuse-safe supervisor liveness', () => {
|
||||
test('fresh TTL → live (the normal running case)', () => {
|
||||
expect(isLockHolderLive(snap({ ttl_expired: false }), SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
|
||||
});
|
||||
|
||||
test('expired TTL but refreshed within the steal grace → live (starved-but-alive #1794)', () => {
|
||||
// ttl lapsed but the holder heartbeat is recent → it is alive, just starved.
|
||||
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: 5_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
|
||||
});
|
||||
|
||||
test('expired TTL and stale heartbeat → dead (a gone supervisor stops refreshing)', () => {
|
||||
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: 36_000_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
|
||||
});
|
||||
|
||||
test('expired TTL and no heartbeat column → dead', () => {
|
||||
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: null }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
|
||||
});
|
||||
|
||||
test('liveness NEVER consults process.kill (PID reuse cannot false-positive)', () => {
|
||||
// A row whose holder_pid happens to be a live, unrelated process (PID reuse)
|
||||
// but whose lock is stale must read as NOT live — proving freshness, not the
|
||||
// PID probe, is the signal. holder_pid=1 (init, always alive) + expired/stale.
|
||||
expect(isLockHolderLive(snap({ holder_pid: 1, ttl_expired: true, ms_since_last_refresh: 36_000_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2227 status detects a live supervisor via the DB lock (split-$HOME)', () => {
|
||||
test('a live queue lock with no local pidfile reads as running via inspectLock', async () => {
|
||||
// Simulate the keeper holding the queue lock under a different $HOME: there
|
||||
// is a live lock row but the local pidfile path is empty.
|
||||
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
|
||||
expect(holder).not.toBeNull();
|
||||
const live = await inspectLock(engine, supervisorLockId('default'));
|
||||
expect(live).not.toBeNull();
|
||||
expect(isLockHolderLive(live!, SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
|
||||
await holder!.release();
|
||||
// After release the row is gone → not running.
|
||||
const gone = await inspectLock(engine, supervisorLockId('default'));
|
||||
expect(gone).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1849 refresh-failure fails safe (F1A)', () => {
|
||||
test('exits LOCK_LOST after the failure threshold; tolerates a single blip', async () => {
|
||||
const sup = new MinionSupervisor(engine, { cliPath: '/bin/sh', healthInterval: 0, json: true });
|
||||
|
||||
+38
-1
@@ -4,7 +4,7 @@ import { spawn } from 'child_process';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { readSupervisorEvents, computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
|
||||
import { calculateBackoffMs } from '../src/core/minions/supervisor.ts';
|
||||
import { calculateBackoffMs, resolveHardStopMaxCrashes } from '../src/core/minions/supervisor.ts';
|
||||
|
||||
const TEST_PID_FILE = '/tmp/gbrain-supervisor-test.pid';
|
||||
|
||||
@@ -58,6 +58,16 @@ function spawnSupervisor(h: IntegrationHarness, overrides: Record<string, string
|
||||
SUP_HEALTH_INTERVAL_MS: '999999', // effectively off
|
||||
...overrides,
|
||||
};
|
||||
// issue #1994: the soft crash budget now DEGRADES (retry-with-backoff) rather
|
||||
// than permanently giving up; permanent give-up fires at a much-higher hard
|
||||
// ceiling (maxCrashes × 10). These integration tests assert the give-up
|
||||
// LIFECYCLE (audit events, exit code, pidfile cleanup), so pin the hard
|
||||
// ceiling to the soft budget by default — the degraded path is unit-tested in
|
||||
// child-worker-supervisor.test.ts. Tests that want true degraded behavior
|
||||
// pass GBRAIN_SUPERVISOR_HARD_STOP_CRASHES explicitly.
|
||||
if (env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES === undefined) {
|
||||
env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES = env.SUP_MAX_CRASHES;
|
||||
}
|
||||
|
||||
const child = spawn('bun', [join(import.meta.dir, 'fixtures/supervisor-runner.ts')], {
|
||||
env,
|
||||
@@ -104,6 +114,31 @@ async function waitFor(pred: () => boolean, timeoutMs: number, tickMs = 20): Pro
|
||||
}
|
||||
|
||||
describe('MinionSupervisor', () => {
|
||||
describe('resolveHardStopMaxCrashes (issue #1994)', () => {
|
||||
const KEY = 'GBRAIN_SUPERVISOR_HARD_STOP_CRASHES';
|
||||
afterEach(() => { delete process.env[KEY]; });
|
||||
|
||||
it('defaults to maxCrashes × 10 when no override', () => {
|
||||
delete process.env[KEY];
|
||||
expect(resolveHardStopMaxCrashes(10)).toBe(100);
|
||||
expect(resolveHardStopMaxCrashes(3)).toBe(30);
|
||||
});
|
||||
|
||||
it('honors a valid non-negative integer override', () => {
|
||||
process.env[KEY] = '0'; // 0 = disable permanent give-up
|
||||
expect(resolveHardStopMaxCrashes(10)).toBe(0);
|
||||
process.env[KEY] = '5';
|
||||
expect(resolveHardStopMaxCrashes(10)).toBe(5);
|
||||
});
|
||||
|
||||
it('ignores a negative or non-integer override (falls back to default)', () => {
|
||||
process.env[KEY] = '-1';
|
||||
expect(resolveHardStopMaxCrashes(10)).toBe(100);
|
||||
process.env[KEY] = 'abc';
|
||||
expect(resolveHardStopMaxCrashes(10)).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('calculateBackoffMs', () => {
|
||||
it('returns ~1s for first crash', () => {
|
||||
const backoff = calculateBackoffMs(0);
|
||||
@@ -197,6 +232,8 @@ describe('MinionSupervisor', () => {
|
||||
// hit max-crashes, then exit via shutdown() with code 1.
|
||||
const h = makeHarness('max-crashes', 'exit 1');
|
||||
try {
|
||||
// hard ceiling defaults to SUP_MAX_CRASHES in the harness (see
|
||||
// spawnSupervisor) so this give-up lifecycle still fires at 3 (#1994).
|
||||
const sup = spawnSupervisor(h, { SUP_MAX_CRASHES: '3' });
|
||||
const { code } = await sup.exited;
|
||||
|
||||
|
||||
@@ -6,11 +6,33 @@ import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
resolveSyncHardDeadline,
|
||||
composeAbortSignals,
|
||||
resolveStallAbortSeconds,
|
||||
DEFAULT_SYNC_STALL_ABORT_SEC,
|
||||
HARD_DEADLINE_GRACE_SEC,
|
||||
} from '../src/commands/sync.ts';
|
||||
|
||||
const GRACE_MS = HARD_DEADLINE_GRACE_SEC * 1000;
|
||||
|
||||
describe('resolveStallAbortSeconds (#1950)', () => {
|
||||
test('defaults to 900s when the env var is unset or empty', () => {
|
||||
expect(resolveStallAbortSeconds({})).toBe(DEFAULT_SYNC_STALL_ABORT_SEC);
|
||||
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '' })).toBe(900);
|
||||
});
|
||||
|
||||
test('honors a positive override', () => {
|
||||
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '120' })).toBe(120);
|
||||
});
|
||||
|
||||
test('<=0 disables the watchdog (returned verbatim)', () => {
|
||||
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '0' })).toBe(0);
|
||||
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '-1' })).toBe(-1);
|
||||
});
|
||||
|
||||
test('falls back to the default on a non-numeric value', () => {
|
||||
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: 'nope' })).toBe(900);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSyncHardDeadline', () => {
|
||||
test('--no-hard-deadline disables everything (even with --timeout)', () => {
|
||||
const r = resolveSyncHardDeadline(['--source', 'x', '--timeout', '60', '--no-hard-deadline'], { isTty: false });
|
||||
|
||||
Reference in New Issue
Block a user