Compare commits

..
2 Commits
Author SHA1 Message Date
Garry TanandClaude Fable 5 8bf23abf71 v0.46.6.0 fix(minions): verify-before-evict lock renewal + per-job leases (#4145) (#4170)
* minions: classify lock-renewal failure causes + starvation telemetry (#4145)

The incident's forensics cost ~8h because the eviction log lines could not
say WHY renewal failed. This commit makes every renewal fault self-explaining
without changing abort semantics:

- Named RenewalCallTimeoutError so cause classification (call-timeout vs
  refused vs fenced-lost) is name-based, never message-sniffing.
- Tick lateness (now - lastTickFiredAt - intervalMs) as the primary
  local-starvation signal — interval callbacks coalesce under a blocked
  loop, so a missed-tick counter cannot measure starvation; lateness can.
  overlap_skips counts tickInFlight re-entrancy skips only.
- Elapsed-time arithmetic now binds deps.now to performance.now()
  (monotonic): wall-clock jumps can no longer distort the deadline math.
  Date.now remains only in log/audit timestamps.
- loadSnapshot dep (raw loadavg[0] + cached core count), try/caught at
  every call site — telemetry must never re-open the unhandledRejection
  class this module exists to close.
- Worker-level event-loop-delay histogram (perf_hooks.monitorEventLoopDelay,
  fail-open when the runtime lacks it), RESET on every successful renewal
  so an eviction-time sample attributes to the exact window in which
  renewal was failing; sampled into the abort + grace-evict log lines.
- Per-launch abortMeta stash so the grace-evict line (which fires 30s
  after the abort) reports cause/lateness/load instead of just the Error
  string that made healthy evictions read like orphan leaks.
- Audit events gain additive optional fields (cause, lateness_ms,
  overlap_skips, load1, cores, via, deadline_deferred) behind a
  back-compatible optional trailing ctx param; the 4-outcome contract and
  pre-upgrade JSONL readback are unchanged (pinned by new tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* minions: inFlight generation-safety — lockToken-conditional deletes (#4145)

Force-evict and the handler's finally both deleted the inFlight entry by
bare job.id. When a force-evicted job is requeued and re-claimed by the
SAME worker while the old handler is still alive, the old execution's
late delete removed the NEW execution's entry — concurrency undercount
and lost tracking for the replacement run. Every claim mints a unique
lockToken and the entry already stores it, so the token is the
generation: both deletes now only remove the entry when it is still
their own.

Pre-existing bug surfaced by the #4145 outside-voice review (R2-1);
fixed in its own commit because the eviction path is exactly what this
wave modifies. Pinned by a deterministic same-worker reclaim test
(stale execution A's finally leaves execution B's entry intact).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* minions: stall-sweep reclaim grace (#4145)

handleStalled reclaimed on a bare lock_until < now(). When a CPU-starved
worker's event loop unblocks, its coalesced renewal tick and the stall
sweep fire in the same burst — if the sweep's UPDATE lands first it
steals the OWNER'S live job and discards its in-flight work. All three
sweep predicates now carry a reclaim grace (default 15s, env
GBRAIN_MINION_STALL_RECLAIM_GRACE_MS, 0 = exact legacy behavior).

The grace is a head-start for the owner's recovery renewal, not a
guarantee: it covers starvation bursts shorter than the grace; a healthy
second worker's sweep still wins beyond it. Cost: dead-worker recovery
becomes lock_until + grace + up to stalledInterval. This is the minion
analog of the cycle-lock steal grace, adapted because minion_jobs has no
last_refreshed_at column.

Existing stall tests move their synthetic lock_until offsets from 1s to
30s past (they pin stall mechanics on the production default path); new
tests pin within-grace hold, beyond-grace reclaim, grace=0 legacy, and
env resolution incl. warn-once fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* minions: verify-before-evict + hard eviction deadline (#4145)

The root-cause fix. The only abort path was the throw branch's local
arithmetic: one timed-out renewal on a starvation-delayed tick past
lockDuration - safetyMargin evicted a healthy job — under load the
renewal UPDATE may even have LANDED server-side while the local race
timeout won. 2,571 subagent jobs submitted / 24 done in 24h.

New contract (ports the cycle-lock fencing doctrine to Minion job locks):

- Fenced-false is the only CERTAIN eviction signal; a throw is not
  evidence of loss. When the NEXT tick would land past the soft deadline
  (cadence-aware: sinceLastSuccess + intervalMs >= deadline — the bare
  >= gate is unreachable under cadence quantization for long leases),
  the tick runs ONE bounded VERIFY renewal. renewLock fences on
  lock_token and deliberately ignores lock_until, so an
  expired-but-unstolen lease revives: fenced-true → starved-but-ours,
  keep working (the incident-saving path); fenced-false → certain loss,
  abort (stall detector requeues, no attempt burned); verify unreachable
  → defer + reconnect-once, aborting only past the hardEvictMs backstop
  (default 2×lease, env GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS, floored to
  the soft deadline) — a LOCAL decision under uncertainty that bounds
  blind external side effects during a total outage.
- The verify is a synchronous, cancelled()-guarded, callTimeoutMs-bounded
  call inside the tick's own flow — reconciled in-code with queue.ts's
  no-background-retry rationale (both UPDATEs are same-token idempotent
  lease extensions; a fenced row cannot gain two holders).
- Best-effort cancellation: the race timeout now aborts the in-flight
  renewLock via AbortSignal threaded to executeRawDirect; both engines'
  entry points gained an already-aborted preflight BEFORE dispatch/pool
  acquisition. The fence stays the correctness authority.
- Relational knob validation: margin < lease/2, callTimeout <= cadence,
  hardEvict >= soft deadline — clamped with warn-once; positive-integer
  parsing alone could silently re-break the deadline math.
- Doctrine comments rewritten (tick header + worker grace-evict) — the
  old abort-at-deadline prose actively misled.

Tests: incident replay (starved renewal times out, verify succeeds, job
survives — the exact #4145 shape), fenced-false via verify, deferral +
hard-backstop timelines, CDX-4 quantization pin (300s/60s verifies at
240s with lease left), first-tick verify at the 30s default,
mid-verify cancellation, reconnect-on-deferral, relational clamps, and
a DATABASE_URL-gated e2e pinning the DB-level foundations (expired-
but-unstolen revival, grace hold, post-reclaim fenced-false, and a
blocked-event-loop worker completing with zero stall bounces).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* minions: per-job lock_duration_ms end-to-end (#4145)

A single worker-global 30s lockDuration cannot serve both 2s shell jobs
and 173s-average LLM subagent jobs — the structural half of the #4145
incident. The lease is now a per-job column resolved through the same
three-layer model as timeout_ms (explicit submit → handler-type map →
worker default), claim-stamped so it survives worker restarts.

- Migration v129 + the 3 schema copies: nullable lock_duration_ms with
  the positive CHECK added via the idempotent drop-then-add pattern (v7
  precedent) — no fresh-vs-migrated asymmetry, no backfill (NULL = worker
  default = pre-#4145 behavior; claim COALESCE owns all defaulting).
- HANDLER_DEFAULT_LOCK_DURATION_MS beside the timeout map (long LLM
  handlers 300s, single-call LLM handlers 120s, shell deliberately absent
  for fast dead-worker reclaim) under one rewritten header explaining why
  lease and wall-clock budget are different quantities.
- Claim derives lock_until from COALESCE(row, map, worker default) and
  stamps the resolved lease; the map binds as a RAW object (jsonb
  double-encode rule). Wall-clock null-fallback becomes
  COALESCE(lock_duration_ms, worker default) so an explicit lease on an
  unmapped handler isn't killed at the old bound (NULL rows pinned to
  exact legacy behavior).
- Worker consumes the effective per-job lease at all three launch sites;
  renewal cadence clamps to min(lease/2, 60s) — a 300s lease renews 5x
  per window instead of every 150s; ≤120s leases keep legacy /2 exactly.
  Derived knob defaults cap at 15s call-timeout / 30s margin so long
  leases don't inherit wedge-inducing values.
- Shared clampLockDurationMs [5s, 1h] used by queue.add, the new
  gbrain jobs submit --lock-duration-ms flag, and the MCP submit_job
  param (handler-side clamp; ParamDef has no min/max — wrong types are
  rejected by the existing number validation). INSERT-only on idempotent
  re-submit, matching the max_stalled footgun rule.
- jobs get shows the lease line alongside the timeout line.

Tests: migration v129 structure/idempotency/CHECK/pre-shape re-run;
claim-stamps-from-map + lock_until horizon; explicit-wins + clamp bounds;
idempotent-resubmit immutability; wall-clock NULL-row regression pin +
leased-row survival; lifetime max_stalled accumulation pin (the known
coverage gap); per-lease knob derivation caps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(minions): lock-renewal knobs, eviction-forensics guide, verify-before-evict current-state (#4145)

- queue-operations-runbook.md gains the incident-reading section the
  #4145 forensics lacked: how to read a gave_up/eviction line (cause,
  lateness_ms, load1/cores, overlap_skips, deadline_deferred,
  event-loop-delay), the was-the-DB-down-or-the-worker-starved decision
  table, the full env-knob table (CALL_TIMEOUT / SAFETY_MARGIN /
  HARD_EVICT / MAX_FAILURES / STALL_RECLAIM_GRACE) with relational-clamp
  semantics and legacy escape hatches, and the honest zombie caveat
  (eviction is cooperative until the kill/reap follow-up lands).
- minions-deployment.md's 'what can still bite' section rewritten for
  verify-before-evict + per-type leases + reclaim grace; documents the
  mixed-version-fleet degradation (old workers = legacy behavior, no
  drain needed) and adds --lock-duration-ms to the per-job tuning list.
- KEY_FILES.md entries (lock-renewal-tick.ts ×2 duplicated entries,
  worker.ts, queue.ts, handler-timeouts.ts) updated to current state.
- TODOS.md: filed the kill/reap-evicted-handler follow-up (P2), the
  worker-level --lock-duration flag (P3), and the TODO-LR-2 note that
  its doctor-check inputs now exist in the audit events.
- llms bundles regenerated (unchanged — these guides aren't inlined).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: scrub leaked GBRAIN_HOME from doctor-minions-check subprocess env

The test seeds $HOME/.gbrain/migrations fixtures and spawns gbrain doctor
with HOME overridden — but doctor resolves its home via resolveGbrainHome,
which prefers GBRAIN_HOME over HOME. Sibling test files in the same bun
process (preferences, friction, bootstrap-*, and several .serial files'
beforeEach hooks) set process.env.GBRAIN_HOME; a value captured by the
{...process.env} spread makes the fixture invisible, doctor finds nothing,
and the expected FAIL exit code never happens. Scrub GBRAIN_HOME exactly
like the DATABASE_URL variables already scrubbed two lines up.

Surfaced while triaging a non-blessed whole-suite invocation during the
#4145 wave (reproduced identically on master); the blessed sharded runner
can also co-schedule a GBRAIN_HOME-mutating file into this shard, so the
scrub closes a real flake vector, not just a synthetic one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(bootstrap): verify probe cleanup hard-deletes instead of leaving soft-delete tombstones

verify's end-of-run probe cleanup invoked the delete_page OP, which since
v0.26.5 is a SOFT delete — every verify run left two probe tombstone rows
in the user's brain (visible to include_deleted readers) until the 72h
purge. Cleanup now hard-deletes via engine.deletePage(slug, {sourceId}),
the same primitive sweepProbeLeftovers already uses on both engines, with
the warning-capture semantics preserved. Pinned by the (previously
failing) probe-residue assertion in the Postgres bootstrap-verify e2e.

Surfaced by the e2e fix wave: CI runs only 6 of 185 test/e2e files
(.github/workflows/e2e.yml), so the developer-lane files rot silently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(search): deterministic relationalFanout path pick on equal-depth multi-seed ties

The fanout's representative-path pick ordered only by (depth, path
length); a node reachable at the same depth from multiple seeds had NO
tie-break, so the winner was plan/heap-order dependent — a fresh PGLite
and a lived-in Postgres heap could disagree, violating both engine parity
and the documented deterministic relational-retrieval contract. Appended
a lexicographic final tie-break to the array_agg ORDER BY in BOTH engines
(lockstep). The engine-parity e2e's multi-seed fanout case now passes on
real Postgres; its stale-page arm also moves off client wall-clock stamps
onto per-row updated_at_iso (the #1768 production semantics) so VM clock
drift can't skew the count.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): per-file outer-timeout override for LLM-bound Tier-2 files

run-e2e.sh's hard 180s-per-file gtimeout SIGKILLed skills.test.ts mid-run
(the ingest skill alone has been observed at ~131s of real provider
round-trips), producing a mystery failure with no assertion output. CI
runs the Tier-2 keyed files in their own job WITHOUT this wrapper, so the
cap only ever bit local runs. The cap is now GBRAIN_E2E_FILE_TIMEOUT
(default 180) with 4x for skills.test.ts + zeroentropy-live.test.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): un-rot the developer-lane files — 27 failures across 10 files, all green on both lanes

CI runs only 6 of the 185 test/e2e files; the rest run solely through the
developer-machine lane (bun run test:e2e / ci:local) and had rotted as src
moved deliberately underneath them. Every fix pins CURRENT intended
behavior with the causing commit cited in-file; no assertion was weakened
and several were strengthened. Root causes:

- sync.test.ts (13): the #2114 global-anchor ownership guard (636628fdb)
  refuses anchor writes when the default source's local_path names another
  repo; setupDB truncates config but not sources, so residue from earlier
  files vetoed every bookmark write. The test now resets the default
  source identity in beforeAll.
- v0_29-mcp-dispatch (2): the #4096 WP1/D7 locality backstop dispatches
  localOnly ops only on transport 'stdio'; tests now dispatch with the
  real stdio shape, plus a NEW fail-closed unknown_tool pin for unset
  transport markers across both trust values.
- extract-atoms-discovery-sql (4): PR #2615 widened discovery to the
  schema pack's extractable:true types ('note' included); tests re-pin
  with 'person' as the non-extractable control + wider seed cleanup.
- pglite-cli-exit (1) + bootstrap-harness-lifecycle (1): the wrapper's
  ambient DATABASE_URL leaked into subprocess/in-process env, silently
  retargeting PGLite tests onto shared Postgres (#801 env-override
  precedence); both files now scrub it at their boundaries.
- embedding-column-pglite (1): #3554 changed resetGateway() to restore
  the test-preload baseline; the #3461 fallback test now uses
  __unconfigureGatewayForTests() so the unconfigured path really fires.
- openclaw-plugin-load-real (1): the Retrieval Reflex import chain pulls
  PGLite WASM assets into the bundle — bun build --outfile cannot emit
  multi-output builds; switched to --outdir with entry naming + a
  version-robust runtime inspection helper.
- phantom-redirect (1): halfvec migration (v40) + the #2932 idempotent
  reconcile; the string-shape guard now accepts both legitimate vector
  types.
- serve-http-oauth (1): sql.array() on a fresh connection races the
  async typeArrayMap fetch and binds text instead of text[]; seed uses a
  plain-array bind (the same untyped approach production pgArray() uses).
- type-unification-full-flow (1): checkPackUpgradeAvailable reads the
  operator's real ~/.gbrain config; the test now isolates GBRAIN_HOME and
  pins both the warn and ok arms hermetically.

Verified: full e2e lane 186/186 files, 1279/1279 tests on a pristine
pgvector container; the 14 previously-failing files re-verified green on
the residue-carrying locally-configured database as well.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(minions): ship-review hardening — histogram lifecycle, cadence single-home, consumption re-clamp, DRY telemetry (#4145)

Fix-First batch from /ship's specialist + coverage reviews (21 findings,
all informational; the mechanical ones applied, the rest pinned by tests):

- The event-loop-delay histogram now disables in stop() and (re-)enables
  in start() — each cycled worker instance previously leaked a ~50Hz
  native sampling timer for process lifetime (embedding hosts and the
  test suite cycle workers constantly).
- renewalIntervalFor()/RENEWAL_INTERVAL_CAP_MS: ONE home for the
  min(lease/2, 60s) cadence formula, used by the worker's timer AND as
  resolveLockRenewalKnobs' default intervalMs — previously the knob
  default (lease/2 uncapped) diverged from production cadence for leases
  over 120s, silently weakening the CDX-10 relational validation for
  callers that omit intervalMs.
- Defense-in-depth re-clamp at consumption: launchJob clamps a row
  lease through clampLockDurationMs — the exposed submit surfaces clamp,
  but a writer bypassing add() could stamp a 1ms lease (renewal-storm
  interval) or a ~25-day one (weeks-long dead-worker pin).
- formatAbortMeta(): one formatter for the classified abort telemetry;
  the three log sites had already drifted (load1: vs load1_at_abort:).
- Knob docstrings updated to the capped defaults (min(lease/3, 15s) /
  min(lease/6, 30s)); KEY_FILES dedup + stale-count scrub; dead test
  binding removed; run-e2e.sh validates GBRAIN_E2E_FILE_TIMEOUT
  digits-only before arithmetic/interpolation.

New pins: worker renews with the per-job lease (launchJob wiring, GAP-1);
claim precedence row-beats-map at the claim UPDATE; MCP submit_job
clamp round-trip incl. the 0→default boundary; jobs get lease lines
(all three states); bootstrap-verify tombstone-proof pages count on
PGLite; relationalFanout lexicographic WINNER (not just parity);
grace-env warn-once dedupe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(minions): adversarial-review hardening — claim-time clamp, grace cap, range CHECK, honest dry-run echo (#4145)

Two independent adversarial passes (Codex + Claude) converged on the same
lease-bounds gaps; this batch closes them:

- Claim SQL clamps the resolved row/map lease to [5s,1h] before deriving
  lock_until, so a bypass-written out-of-range value can't produce a
  pathological lease at claim time. The worker-default path ($2) passes
  through untouched (tests pin a 1ms worker lease).
- Migration v130 + all 3 schema copies upgrade the CHECK to a full range
  constraint (>= 5000 AND <= 3600000) — the DB bound now matches the
  app-side clamp instead of only enforcing positivity.
- GBRAIN_MINION_STALL_RECLAIM_GRACE_MS is capped at 600s with a warn-once
  (an absurd env value silently disabled stall reclaim fleet-wide).
- Hard-evict comparison recomputes elapsed AFTER the bounded verify, so
  the backstop can't defer one extra cadence past its advertised bound;
  the should_abort payload carries the recomputed value.
- Verify-failure telemetry re-samples loadavg at its own failure instead
  of reusing a snapshot stale by the verify's duration; audit note that
  the attempt counter advances by 2 per at-deadline tick.
- racedRenewLock/attemptReconnectOnce clear the losing race timer on the
  win path (no stray late abort against a settled query).
- jobs submit --dry-run echoes the CLAMPED lease (annotated when it
  differs from the raw input) instead of echoing a value add() won't store.

Tests: migrations-v130 range-CHECK rejection loop + bypass-clamp claim
test; grace-cap pin; dry-run echo cases. 301 pass / 0 fail targeted;
typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.46.6.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: document-release sweep for v0.46.6.0 — lease-bound enforcement, grace cap, e2e file timeout

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): stop run-child-entry SIGTERM test from broadcasting to leaked process listeners

The SIGTERM-semantics test fired a bare process.emit('SIGTERM') in the
shared bun test process. When an earlier file in the same shard had
installed process-cleanup.ts's signal handlers (module-global, never
uninstalled), the broadcast reached its handler, whose cleanup pass ends
in process.exit(143) — killing the entire shard mid-suite. The runner
then misclassified rc=143 as an external kill ("sibling workspace pkill /
memory jetsam") and queued a rescue pass that died the same way when it
reached the same test. Whether it fired depended on file interleaving;
under heavy host load the schedule made it deterministic (three
consecutive suite runs died at the ~3-minute mark with zero test
failures).

The test now snapshots the SIGTERM listener set before invoking
runChildJobEntry and fires ONLY the listener(s) the entry registered —
same wiring under test, no global broadcast.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: changelog entry for the unit-suite shard self-kill fix

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): best-effort tmpdir cleanup in run-child-entry afterAll — EFAULT rmSync flake reds the CI shard

CI shard 3 failed with an "(unnamed)" test: bun treats a throwing afterAll
as a failed test, and the hook's recursive rmSync EFAULT'd (bun 1.3.13,
ubuntu-24.04) immediately after the PGLite WASM engine teardown. All 8
real tests passed. Cleanup is now try/retry/warn — a tmpdir the OS reaps
anyway must never red the suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 22:06:05 -07:00
Garry TanandClaude Fable 5 11ad23e346 v0.46.5.0 perf(test,ci,eval): CI in half — pooled serial lane, snapshot-in-CI, hermetic retrieval canary (#4154)
* perf(pglite): memoize snapshot loading — read the 42MB tar once per process, not per engine

tryLoadSnapshot re-read the snapshot tar and re-hashed all migration handler
sources on every engine construction (600+ per full suite, ~84MB transient
allocation each). The schema hash and the (versionLines, blob) pair are now
memoized per (path, process); missing/stale/torn paths memoize a terminal
null. The dims/model shape gate stays per-call so mid-process gateway
reconfiguration (the zembed/1280 class) still falls back to cold init —
pinned by new memo tests in snapshot-shape-guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* perf(test): pool the serial-test runner + wire the PGLite snapshot into every CI-facing runner

The serial lane ran 140 per-file bun processes strictly one-at-a-time (8.5
min in CI) — but the quarantine contract only requires per-PROCESS isolation.
run-serial-tests.sh now runs a pool (min(cpus,4), memory-adaptive, 120s
per-test budget, 300s SIGTERM-then-SIGKILL wall clock per file), keeps two
machine-global files on a sequential EXCLUSIVE lane (launchd/cron), treats a
missing exit sentinel as failure, and prints sorted per-file durations.
First full run: 140/140 in 145s at pool=4.

The snapshot fast-path (previously local-only) is now shared via
scripts/lib/test-env.sh (detect_cpus + mem detection + ensure_pglite_snapshot,
one implementation across the runner family) and wired into test-shard.sh
(+ --max-concurrency, mirroring the local runner), run-slow-tests.sh, and
run-e2e.sh's env-scrub keep-list. Serial lane also gains the #3485 ambient
DB-URL scrub its sibling lanes already had.

Guards: pool-behavior tests (fail → exit 1, full log; hang → timeout kill;
dry-run-list), EXCLUSIVE_FILES growth guard (≤3, justification comments),
missing-sentinel source pin, sandbox staging carries scripts/lib/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci(test): snapshot tar cache, one bun-cache saver, gitleaks tarball cache, shallow brainbench fetch

- PGLite snapshot (~42MB tar) cached across jobs keyed on schema inputs:
  serial-tests saves, the 10-shard matrix + slow jobs restore-only; the
  runner's own hash check stays authoritative (stale restores are rebuilt,
  never trusted). Slow jobs export the snapshot env via the shared lib.
- Bun install cache: verify becomes the single saver (admin/bun.lock joins
  its key); every other job restores with restore-keys so a bun.lock touch
  no longer cold-installs all six jobs. All installs are --frozen-lockfile.
- gitleaks: the release tarball is cached; its published checksum is
  fetched fresh and re-verified on every run, so a cache restore is never
  trusted.
- brainbench: fetch-depth 1 + a depth-1 fetch of the master ref replaces
  the full-history clone (the gate reads one file via git show).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci(e2e,heavy,release,osv): parallel e2e tiers behind a spend gate, content-hash skip, cache/timeout hygiene

e2e.yml: tier2 no longer waits ~2min behind tier1 (separate DBs — never
shared state); it now gates on jsonb-parity (~40s), keeping a fast
broken-build spend gate in front of the job that burns real provider
tokens. New e2e-cache-check/e2e-cache-write (e2e-pass-<hash> namespace)
skip the whole suite on doc-only pushes; scheduled nightly runs are
exempt so the live-provider drift check always fires. e2e-status is the
stable aggregate name. Bun caches + --frozen-lockfile on all tiers.

heavy-tests: bun cache restores on all 4 jobs. release: timeout-minutes
on all 4 jobs (was 360-min default), bun caches, frozen installs.
osv-scanner: concurrency group cancels superseded PR scans.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci(verify): bounded worker pool, longest-first ordering, chronicle gate, two revived guards

run-verify-parallel.sh fanned out 44 checks unbounded — two cp -R src +
bun build --compile builds, the admin vite build, tsc, and ~40 greps all
simultaneous on a 4-vCPU runner, feeding the 120s per-check timeout flake
class. The spawn loop is now a pool (default detect_cpus; escape hatch
GBRAIN_VERIFY_MAX_PARALLEL) with the heavy checks ordered first
(LPT-style makespan). 47 checks green in 38s locally.

New checks: check:eval-chronicle ($0 deterministic eval, exit-0-only-on-
perfect — first CLI-level CI gate for it), plus the two registered-but-
never-executed guards check:pagetype-exhaustive and check:pg-url-redaction
(the latter's marker now works inside block comments; its one legitimate
hit in the redactor's own docs carries the marker). A new registration⇒
execution coverage test closes the dead-guard class: every guards-manifest
row must be reachable from CHECKS or carry an explicit exemption reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): collect evals/ tests into the matrix behind a keyless allowlist

evals/functional-area-resolver/harness-runner.test.ts (47 keyless tests)
was collected by NO runner — real tests that never executed anywhere.
test-shard.sh now finds test/ AND evals/; a new allowlist guard asserts
every collected evals file is keyless-verified (this repo's eval harnesses
are key-requiring by default, so unlisted growth would silently spend
tokens in CI). The isolation (R1-R4) and real-names lints extend their
scan roots to evals/ so everything CI executes carries the same hygiene bar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* perf(test): shrink migrate dedup perf-gates to 200 rows; one engine for chunk-grain-fts

The two v8/v9 dedup gates inserted 1000 rows one-at-a-time each — ~15-25s
of row traffic per test that adds no discriminating power (the O(n²) shape
they guard is minutes-vs-sub-second at 200 rows; the full v7→current chain
replay they also pay is unchanged and still exercised).

chunk-grain-fts.test.ts booted three describe-scoped engines for 11 tests;
now one file-level engine + resetPgliteState per data-bearing describe
(the reset is required, not hygiene: the searchKeyword corpus would
pollute the searchKeywordChunks expectations).

The planned resetPgliteState pg_tables caching is deliberately NOT done:
the pre-agreed DDL check found 7 files that create/alter tables between
resets on a shared engine — a cached table list would skip truncating
mid-file tables and leak rows across tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(eval): hermetic CLI retrieval canary — deterministic embedder for eval gate, W1 mandate satisfied

gbrain eval gate gains an embedder option (deterministic) scoped to the
qrels correctness gate: query vectors come from a fixture-keyed basis
embedder (src/eval/deterministic-embed.ts, shared with the hermetic test)
through a new additive HybridSearchOpts.queryEmbedFn seam — the full RRF
pipeline (keyword/FTS, title, alias, relational arms + fusion) runs for
real with zero API keys. Bare hybridSearch is cache-free by construction
(lookup + writeback live only in hybridSearchCached), so deterministic
runs cannot poison query_cache. Behavior is unchanged when the seam is
absent. Flag registry regenerated.

scripts/run-eval-canary.ts seeds a throwaway PGLite brain from the qrels
corpus (expected pages visible to page-grain FTS via timeline — pages
FTS deliberately excludes compiled_truth) and spawns the real CLI:
check mode (CI, writes nothing) is wired as check:eval-canary in verify;
record mode appends the committed .gbrain-evals/eval-results.jsonl ledger.

Measured, deterministic across processes and keyless: recall@10 1.0000,
first_relevant 1.0000, expected_top1 0.8333 vs floors 0.70/0.60/0.50.
The FIX_WAVE_BASELINES W0 retrieval-canary mandate is now PASS (recorded
with honest scope: synthetic vectors gate the ranking pipeline; semantic
embedding regressions remain the keyed suites' job). Spike-first design
per the plan's OV2-1 respec.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): current-state TESTING.md for the pooled/pool-bounded runners; TODO ledger updates

TESTING.md: pooled serial lane (+EXCLUSIVE lane, knobs, timings), verify
worker pool + eval gates, snapshot-in-CI, evals/ collection, e2e cache-skip
+ e2e-status. ci-local.sh: stale "36 E2E files" comments (actual 181).
TODOS: deeper-speedup entry closed by this pass; test.concurrent P0
downgraded to P3 with stale-premise rationale; eval-gate baseline entry
narrowed to the sibling-repo regression half; 7 pass deferrals filed with
context (sleep-to-poll, e2e lanes, per-shape snapshots, persistent-engine
snapshot, engine-consolidation audit, verify double-spawn, image-decoders).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): re-mine shard weights post-snapshot; p75 fallback; balance test asserts what CI runs

Weights re-mined from the branch's green Test run (31893516029): 1200/1212
files covered (was 663/1204 — 45% of the corpus rode a 30ms median fallback
while really averaging seconds), total mined suite time 986s vs 3185s
pre-snapshot. Rebalanced 10-shard totals: ~99s each.

sharding.ts missing-file fallback median → p75 (the distribution is
right-skewed and unweighted files skew heavy — new integration tests land
unweighted more often than pure-unit ones).

The balance regression test previously recomputed shard totals with the
same weights map + same fallback the partitioner used (max/min ≈ 1.0 by
construction) and asserted 4/6-shard splits while CI runs 10. It now:
asserts the 10-shard split with the shard count cross-checked against
test.yml's matrix (js-yaml parse, not a format-brittle regex), gates
weights coverage ≥70% of matrix-eligible files (the anti-rot forcing
function the regen cadence never had), and fails on weight keys naming
untracked files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): external-kill rescue pass for the serial pool; brain-repo-durability goes exclusive

Two contention classes surfaced by the first pooled CI runs:

1. Stray SIGTERM/SIGKILL from outside the runner (sibling-workspace process
   cleanup, memory jetsam) killed 1-12s-old bun processes with exit 143 and
   truncated logs — the exact class run-unit-parallel.sh already rescues.
   The serial runner now queues exit 143/137 and missing-sentinel files for
   ONE sequential rescue re-run: phantoms stay green with a rescue note,
   real failures fail again and stay red. Pinned by a self-SIGTERM-once
   fixture test.

2. brain-repo-durability.serial.test.ts: hardenBrainRepo's scaffolding
   commit fires the just-installed post-commit hook (background push) which
   races the synchronous push-probe on the same bare remote — "cannot lock
   ref" lands in needs_attention. Near-deterministic on a contended 4-vCPU
   runner, never observed locally. Moved to the sequential EXCLUSIVE lane
   (third justified entry; growth guard capped at 3) until the probe learns
   to retry ref-lock contention.

Also fixes a duplicated word in the runner's timeout header line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): document the serial pool's external-kill rescue pass

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: pre-landing review fixes

From the ship-stage specialist review (testing/maintainability/security/
performance):
- serial runner: exclusive-lane files keep their no-kill contract on rescue
  re-runs; exit 137 at full duration is classified as our timeout's SIGKILL
  escalation (real failure), not an external kill — no ~315s re-hang in rescue
- snapshot memo: tar read deferred until the first shape-MATCHING caller —
  a process that only ever refuses (zembed/1280 class) now reads zero bytes
- e2e.yml: workflow_dispatch joins schedule in the cache-skip exemption (a
  manual dispatch is an explicit ask for a live run)
- test.yml: verify job restores the snapshot cache like its siblings;
  cache-key homes cross-referenced
- eval gate: the four embedder-flag validation exits are now asserted;
  legacy-qrels parsing deduped into deterministic-embed.ts
- canary test: git-status invariance scoped to touchable paths; outer spawn
  budget strictly above the inner CLI timeout
- doc/comment rot: TESTING.md exclusive-lane count, runner header, sharding
  test comments, ci-local.sh sentence, CI_SHARDS in a test name
- TODOs: snapshot-tar digest verification (P3) + eval-ledger redaction (P2)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): serial pool re-emits a bun-format pass aggregate; ledger gets merge=union

The pooled runner's compressed per-file lines starved run-unit-parallel.sh's
headline counter (awk wants " N pass") — bun run test's pass=N banner
silently dropped the entire serial suite. The runner now emits one
aggregate " N pass" line in bun's own summary format. Failing files' logs
stream raw, so fail counting was never affected and stays single-counted.

.gbrain-evals/eval-results.jsonl (append-only tracked ledger) takes
merge=union so concurrent workspaces recording runs don't conflict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.46.1.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.46.1.0

- eval-bench.md: document the hermetic --embedder deterministic correctness-gate
  mode and the check:eval-canary CI gate (scripts/run-eval-canary.ts, --record)
- KEY_FILES.md: current-state updates — eval-gate hermetic mode +
  deterministic-embed.ts + canary runner in the eval-loop entry, queryEmbedFn
  seam in the hybrid.ts entry, per-process snapshot memoization + the shared
  scripts/lib/test-env.sh helper in the snapshot entry, guard count 45→46
- TESTING.md: snapshot activation now via ensure_pglite_snapshot across five
  runners (was two callers), honest per-file speedup figures (~3.5x), serial
  runner output/knob semantics (GBRAIN_SERIAL_POOL=N width), CI shard
  --max-concurrency bound, p75 missing-weight fallback
- CONTRIBUTING.md: drop the orphaned duplicate guard-checks line
- CHANGELOG.md (voice only): "every CI runner" -> "the CI test runners";
  note the p75 fallback in the shard-rebalance clause

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: re-bump to v0.46.2.0 (0.46.1.0 claimed; user-pinned)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 21:51:52 -07:00
97 changed files with 5926 additions and 1293 deletions
+1
View File
@@ -0,0 +1 @@
{"schema_version":3,"run_id":"f2b40f7ef-retrieval-canary-na-0","ran_at":"2026-08-15T15:37:16.659Z","suite":"retrieval-canary","mode":"n/a","commit":"f2b40f7ef","seed":0,"params":{"qrels":"test/fixtures/eval-baselines/qrels-search.json","embedder":"deterministic","k":10,"metrics":{"mean_recall_at_k":1,"first_relevant_hit_rate":1,"expected_top1_hit_rate":0.8333333333333334,"expected_top1_denominator":12,"queries_run":12,"queries_total":12},"floors":{"recall_at_k":0.7,"first_relevant_hit":0.6,"expected_top1":0.5}},"status":"completed","duration_ms":2290}
+1
View File
@@ -27,3 +27,4 @@
# Markdown it does not own), but pinning this repo's own .md checkout to LF
# removes the whole class for anyone working here.
*.md text eol=lf
/.gbrain-evals/eval-results.jsonl merge=union
+126 -8
View File
@@ -20,6 +20,44 @@ concurrency:
cancel-in-progress: true
jobs:
# ──────────────────────────────────────────────────────────────────────
# e2e-cache-check: same content-hash skip as test.yml's cache-check, in
# its own key namespace (e2e-pass-<hash>). Doc-only pushes previously
# provisioned 3 pgvector services and spent real OpenAI/Anthropic/
# ZeroEntropy tokens in tier2; now they skip. SCHEDULED runs are exempt
# below — the nightly is a drift check against live providers and must
# run even when the tree is unchanged.
# ──────────────────────────────────────────────────────────────────────
e2e-cache-check:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
hit: ${{ steps.lookup.outputs.cache-hit }}
hash: ${{ steps.compute.outputs.hash }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Compute content hash
id: compute
run: |
HASH=$(bash scripts/ci-cache-hash.sh --verbose 2>/tmp/cache-diag)
cat /tmp/cache-diag
echo "Computed cache hash: $HASH"
echo "hash=$HASH" >> "$GITHUB_OUTPUT"
- name: Lookup actions/cache for e2e-pass-<hash>
id: lookup
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: e2e-pass-${{ steps.compute.outputs.hash }}
path: .e2e-cache-marker
lookup-only: true
- name: Cache status
run: |
if [ "${{ steps.lookup.outputs.cache-hit }}" = "true" ]; then
echo "✓ e2e cache HIT for hash ${{ steps.compute.outputs.hash }} — e2e jobs will skip (unless scheduled)"
else
echo "✗ e2e cache MISS for hash ${{ steps.compute.outputs.hash }} — e2e suite will run"
fi
jsonb-parity:
# Dedicated required guard for the JSONB double-encode bug-class (#2339).
# PGLite parses a double-encoded jsonb string silently, so this assertion can
@@ -28,6 +66,8 @@ jobs:
# Postgres and HARD-FAILS if DATABASE_URL is missing, so the guard can never
# silently skip.
name: JSONB parity (#2339 regression guard)
needs: e2e-cache-check
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 15
services:
@@ -49,7 +89,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- name: Require DATABASE_URL (no silent skip)
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
@@ -73,6 +118,8 @@ jobs:
tier1:
name: Tier 1 (Mechanical)
needs: e2e-cache-check
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 20
services:
@@ -94,7 +141,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- name: Run Tier 1 E2E tests
# job-isolation rides tier1 deliberately: e2e.yml runs only explicitly
# NAMED files (no glob) — an unwired e2e file is silent coverage loss.
@@ -106,12 +158,15 @@ jobs:
tier2:
name: Tier 2 (LLM Skills)
# Runs on every push/PR (promoted from schedule-only in v0.19.0), in
# PARALLEL with tier1 (own postgres service — the old `needs: tier1`
# serialized ~2min for no shared state). The jsonb-parity gate (~40s)
# stays in front as the broken-build SPEND gate: this job burns real
# OpenAI/Anthropic/ZeroEntropy tokens and must not fire when the build
# can't even pass the cheapest DB guard.
needs: [e2e-cache-check, jsonb-parity]
if: needs.e2e-cache-check.outputs.hit != 'true' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
# Runs on every push/PR now (promoted from schedule-only in v0.19.0).
# Tier 1 must pass first; Tier 2 uses OPENAI_API_KEY + ANTHROPIC_API_KEY
# from repo/org secrets. Nightly + manual triggers still supported via
# the workflow-level `on:` list.
needs: tier1
timeout-minutes: 30
services:
postgres:
@@ -132,7 +187,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- name: Install OpenClaw
# Bound + retry the install: a transient npm/registry stall here used to
# hang unbounded and (since the v0.42.50.0 job timeout) burn the entire
@@ -179,3 +239,61 @@ jobs:
# zeroEntropyCompatFetch response-rewriter + URL rewrite + flexible
# dim handling + gateway.rerank against the real provider.
ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}
# ──────────────────────────────────────────────────────────────────────
# e2e-cache-write: seals e2e-pass-<hash> only when every gated job
# succeeded (writing earlier would bless states the suite never proved).
# Scheduled runs may also write: a nightly green at an unchanged hash is
# the same proof a push green is.
# ──────────────────────────────────────────────────────────────────────
e2e-cache-write:
needs: [e2e-cache-check, jsonb-parity, tier1, tier2]
if: success() && needs.e2e-cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Create cache marker
run: |
mkdir -p .e2e-cache-marker
echo "${{ needs.e2e-cache-check.outputs.hash }}" > .e2e-cache-marker/hash
echo "$GITHUB_SHA" > .e2e-cache-marker/sha
echo "$GITHUB_REF" > .e2e-cache-marker/ref
- uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: e2e-pass-${{ needs.e2e-cache-check.outputs.hash }}
path: .e2e-cache-marker
# ──────────────────────────────────────────────────────────────────────
# e2e-status: the single stable "did E2E pass?" name (mirror of
# test.yml's test-status). Succeeds when the cache hit on a non-scheduled
# run, or when every gated job succeeded.
# ──────────────────────────────────────────────────────────────────────
e2e-status:
needs: [e2e-cache-check, jsonb-parity, tier1, tier2]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Aggregate result
run: |
HIT="${{ needs.e2e-cache-check.outputs.hit }}"
JSONB="${{ needs.jsonb-parity.result }}"
TIER1="${{ needs.tier1.result }}"
TIER2="${{ needs.tier2.result }}"
EVENT="${{ github.event_name }}"
echo "e2e-cache-check.hit=$HIT event=$EVENT"
echo "jsonb-parity=$JSONB tier1=$TIER1 tier2=$TIER2"
# schedule AND workflow_dispatch always run the real suite — a
# manual dispatch is an explicit ask for a live run, so a cache
# hit must not report green-without-running for either.
if [ "$HIT" = "true" ] && [ "$EVENT" != "schedule" ] && [ "$EVENT" != "workflow_dispatch" ]; then
echo "✓ e2e cache HIT for hash ${{ needs.e2e-cache-check.outputs.hash }} — E2E green"
exit 0
fi
for r in "$JSONB" "$TIER1" "$TIER2"; do
if [ "$r" != "success" ]; then
echo "✗ gated e2e job did not succeed (got $r) — E2E fail"
exit 1
fi
done
echo "✓ all e2e jobs succeeded — E2E green"
+24 -4
View File
@@ -64,7 +64,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- name: Run heavy tests
env:
@@ -148,7 +153,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
# Reference the door tests; run only the ones present (a door may land
# in a sibling PR). Missing binary/auth → the file self-skips, so a
@@ -209,7 +219,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
# `runner.temp` is not an allowed context in job-level env, so the
# evidence dir is derived here and exported for every later step (the
@@ -411,7 +426,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- name: Prepare evidence dir
run: |
+5
View File
@@ -19,6 +19,11 @@ on:
permissions:
contents: read
# Rapid pushes to the same PR previously queued duplicate scans.
concurrency:
group: osv-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
osv-scan:
permissions:
+17 -2
View File
@@ -35,6 +35,7 @@ concurrency:
jobs:
version:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
version: ${{ steps.v.outputs.version }}
exists: ${{ steps.v.outputs.exists }}
@@ -80,6 +81,7 @@ jobs:
target: bun-linux-x64
artifact: gbrain-linux-x64
runs-on: ${{ matrix.os }}
timeout-minutes: 30
permissions:
contents: read
id-token: write # for attest-build-provenance (Sigstore OIDC)
@@ -89,7 +91,12 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
# No test re-run here: the Test workflow already gated this exact SHA at
# merge (10 shards + E2E). Re-running the whole suite serially on the
# release runner is a flakier duplicate gate — it blocked the first
@@ -116,6 +123,7 @@ jobs:
needs: [version, build]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write # create the tag + release (scoped to this job only)
steps:
@@ -182,6 +190,7 @@ jobs:
needs: [version, release]
if: needs.version.outputs.exists == 'false'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
env:
@@ -207,7 +216,13 @@ jobs:
with:
bun-version: 1.3.13
- if: steps.gate.outputs.publish == 'true'
run: bun install
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
- if: steps.gate.outputs.publish == 'true'
run: bun install --frozen-lockfile
- if: steps.gate.outputs.publish == 'true'
name: Generate template tree and byte-diff against the vendored copy
run: |
+90 -21
View File
@@ -91,16 +91,24 @@ jobs:
# now enforces a paid GITLEAKS_LICENSE (fails the job with "missing
# gitleaks license" for accounts it can't validate). The CLI is free, uses
# the committed .gitleaks.toml allowlist, and scans the same commit range.
- name: Cache gitleaks tarball
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: /tmp/gitleaks-dl
key: gitleaks-8.30.1-linux-x64
- name: Install gitleaks (pinned + checksum-verified)
run: |
set -euo pipefail
VER=8.30.1
BASE="gitleaks_${VER}_linux_x64.tar.gz"
URL="https://github.com/gitleaks/gitleaks/releases/download/v${VER}"
curl -fsSL -o "/tmp/${BASE}" "${URL}/${BASE}"
mkdir -p /tmp/gitleaks-dl
[ -f "/tmp/gitleaks-dl/${BASE}" ] || curl -fsSL -o "/tmp/gitleaks-dl/${BASE}" "${URL}/${BASE}"
# Checksums fetched fresh EVERY run: a cache-restored tarball is
# re-verified against the published digest, never trusted.
curl -fsSL -o /tmp/gitleaks_checksums.txt "${URL}/gitleaks_${VER}_checksums.txt"
( cd /tmp && grep " ${BASE}\$" gitleaks_checksums.txt | sha256sum -c - )
tar -xzf "/tmp/${BASE}" -C /tmp gitleaks
( cd /tmp/gitleaks-dl && grep " ${BASE}\$" /tmp/gitleaks_checksums.txt | sha256sum -c - )
tar -xzf "/tmp/gitleaks-dl/${BASE}" -C /tmp gitleaks
install /tmp/gitleaks /usr/local/bin/gitleaks
gitleaks version
- name: Scan for secrets (gitleaks CLI, .gitleaks.toml)
@@ -134,11 +142,25 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
# This job is the ONE designated saver of the bun cache (the others
# restore-only, so 5 redundant post-job save attempts disappear).
# admin/bun.lock is in the key because verify's check:admin-build
# installs from it.
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock', 'admin/bun.lock') }}
restore-keys: bun-cache-${{ runner.os }}-
# verify's runner sources test-env.sh and builds the snapshot for its
# PGLite-booting eval checks — restore the cache so it's the ~40ms
# freshness check, not a cold build ahead of all ~47 checks.
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- run: bun run verify
# Guard: no bare `bun test` in workflows/scripts — bun ignores
# bunfig.toml's timeout, and hooks (beforeAll/afterAll) get the 5s
@@ -147,10 +169,11 @@ jobs:
- run: bash scripts/check-bun-test-timeout.sh
serial-tests:
# *.serial.test.ts at --max-concurrency=1. Lives in its own runner so
# the matrix shards aren't carrying the serial-pass tail (the old shape
# stuffed this into `test (1)` after the matrix work, which compounded
# shard 1's overload).
# *.serial.test.ts — one bun process per file (module-registry isolation),
# POOLED across files by scripts/run-serial-tests.sh (was strictly
# sequential: an 8.5-minute job whose serialization the quarantine
# contract never required). Lives in its own runner so the matrix shards
# aren't carrying the serial tail.
needs: cache-check
if: needs.cache-check.outputs.hit != 'true'
runs-on: ubuntu-latest
@@ -160,11 +183,27 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
restore-keys: bun-cache-${{ runner.os }}-
# PGLite schema snapshot (~42MB): the runner builds it when absent or
# stale (its runtime hash is authoritative — a stale restore is rebuilt,
# never trusted). Cached so the build is paid once per schema change,
# not once per job per run. This job SAVES; verify + matrix + slow jobs
# restore-only. The key is an approximation on purpose: it only has to
# be a superset-trigger of real schema changes.
# KEY HAS 5 HOMES in this file (this save + 4 restores: verify, matrix,
# slow-eval, slow-perf) — edit all together, or drift shows up only as
# silent rebuild cost.
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- run: bun run test:serial
slow-eval-longmemeval:
@@ -185,11 +224,20 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
restore-keys: bun-cache-${{ runner.os }}-
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- name: Ensure PGLite snapshot (build-or-validate, non-fatal)
run: bash -c '. scripts/lib/test-env.sh && ensure_pglite_snapshot slow-eval && echo "GBRAIN_PGLITE_SNAPSHOT=${GBRAIN_PGLITE_SNAPSHOT:-}" >> "$GITHUB_ENV"'
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
brainbench:
@@ -206,16 +254,19 @@ jobs:
timeout-minutes: 10 # ~15s hermetic run; matches the per-job-timeout hardening (#2254)
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
fetch-depth: 0 # the gate needs origin/master's baseline
- name: Fetch origin/master baseline ref (shallow)
# The gate reads ONE file via `git show origin/master:...` — a depth-1
# fetch of the master ref replaces the previous full 3700-commit clone.
run: git fetch --no-tags --depth=1 origin +refs/heads/master:refs/remotes/origin/master
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
restore-keys: bun-cache-${{ runner.os }}-
- run: bun install --frozen-lockfile
- run: bash scripts/ci-brainbench-gate.sh
env:
BRAINBENCH_OUT: ${{ runner.temp }}/brainbench-result.json
@@ -242,11 +293,20 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
restore-keys: bun-cache-${{ runner.os }}-
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- name: Ensure PGLite snapshot (build-or-validate, non-fatal)
run: bash -c '. scripts/lib/test-env.sh && ensure_pglite_snapshot slow-perf && echo "GBRAIN_PGLITE_SNAPSHOT=${GBRAIN_PGLITE_SNAPSHOT:-}" >> "$GITHUB_ENV"'
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
# MEMORY_VERBS v1 (Cathedral 1): the entity() p99 < 100ms contract gate
# (20K-page corpus + ratio guard) shares this runner — same perf-job
@@ -299,11 +359,20 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: ~/.bun/install/cache
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
- run: bun install
restore-keys: bun-cache-${{ runner.os }}-
# Restore-only: test-shard.sh validates the snapshot's runtime hash and
# rebuilds when stale (the serial-tests job is the designated saver).
- uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
path: |
test/fixtures/pglite-snapshot.tar
test/fixtures/pglite-snapshot.version
key: pglite-snapshot-${{ runner.os }}-${{ hashFiles('src/core/migrate.ts', 'src/core/pglite-schema.ts', 'test/helpers/legacy-embedding-config.ts', 'scripts/build-pglite-snapshot.ts') }}
- run: bun install --frozen-lockfile
- name: Run test shard ${{ matrix.shard }}/10
run: scripts/test-shard.sh ${{ matrix.shard }} 10
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- gbrain-runbook-stamp: 0.46.4.0 -->
<!-- gbrain-runbook-stamp: 0.46.6.0 -->
<!-- This stamp must equal the VERSION file at every release; CI enforces it
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
the installed binary and warns on skew. -->
+112
View File
@@ -2,6 +2,118 @@
All notable changes to GBrain will be documented in this file.
## [0.46.6.0] - 2026-08-15
**A busy machine can no longer make the job queue evict its own healthy
work.** ([#4145](https://github.com/garrytan/gbrain/issues/4145)) Under
sustained CPU load, a long-running background job (a subagent averaging
~3 minutes) could miss one lock-renewal window and get force-evicted
mid-inference — the queue would churn for hours while completions stayed
near zero, and the logs read like an orphan leak. Lock renewal is now
**verify-before-evict**: a slow or failed renewal is never treated as
loss; the worker asks the database the one authoritative question (a
fenced re-check) and keeps the job whenever the lease is still its own.
### Added
- **Per-job lock leases.** Long LLM handlers (subagent, autopilot-cycle,
embed-backfill, …) now hold a 300s lease by default instead of the
global 30s; single-call LLM handlers get 120s; short jobs keep 30s for
fast dead-worker recovery. Override per submission with
`gbrain jobs submit --lock-duration-ms N` (clamped to 5s1h; also an
MCP `submit_job` param). Stored on the job row (migration v130), so it
survives worker restarts, and renewed at a `min(lease/2, 60s)` cadence.
The bound is enforced end-to-end — at submit, again on the resolved
lease at claim, and by a database range constraint — and
`--dry-run` echoes the clamped value that will actually be stored.
- **Self-explaining eviction forensics.** Every renewal fault now logs and
audits WHY it failed (call-timeout vs refused vs fenced-lost), how late
the renewal timer fired vs its own cadence (the "was the worker starved
or was the database down?" discriminator), host load, and an
event-loop-delay sample scoped to the failing window. The ops runbook
gained a table for reading these plus the full env-knob reference
(`GBRAIN_LOCK_RENEWAL_*`, `GBRAIN_MINION_STALL_RECLAIM_GRACE_MS`).
- **Stall-sweep reclaim grace.** A lease that lapsed within the last 15s
is not reclaimed — a just-recovered worker's own renewal wins the race
against the sweep instead of having its live job stolen (env-tunable,
capped at 10 minutes with a warn-once clamp; `0` restores the previous
behavior).
### Changed
- **Eviction requires evidence.** The renewal state machine aborts a job
only on a fenced miss (the row was genuinely reclaimed — requeued with
no attempt burned) or after a hard backstop (default 2× the lease)
during a total database outage. Wall-clock jumps can no longer distort
the decision (elapsed-time math runs on a monotonic clock), and a
renewal timeout now also cancels the in-flight query so it stops
holding a pool slot.
- **`gbrain jobs get`** shows the job's lock lease alongside its
wall-clock budget, including the default that will stamp at claim.
### Fixed
- **The unit-suite's SIGTERM-semantics test no longer kills its own
shard.** A bare in-process signal broadcast could reach a leaked
cleanup handler from an earlier test file and exit the whole test
process mid-suite, misreading as an external kill; the test now fires
only the listeners it registered.
- **`gbrain verify` no longer leaves probe tombstones in your brain.**
The end-of-run probe cleanup previously soft-deleted its two probe
pages; every verify run left residue visible to `include_deleted`
readers until the 72h purge. Cleanup now hard-deletes.
- **Relational retrieval is deterministic on ties.** When a graph node is
reachable at the same depth from multiple seeds, the reported path was
plan/heap-order dependent (and could differ between engines); a
lexicographic tie-break restores the documented determinism.
- **A worker slot can no longer lose track of a re-claimed job.** After a
force-evict, the stale execution's cleanup could delete the tracking
entry of the SAME job re-claimed by the same worker; both cleanup sites
now verify generation (lock token) before deleting.
- **Developer e2e lane un-rotted.** 29 test failures across 13 files in
the developer-machine e2e lane (which CI does not run) were fixed:
ten rotted test files re-pinned to current intended behavior, plus the
wrapper's per-file timeout now accommodates the LLM-bound Tier-2 files
(`GBRAIN_E2E_FILE_TIMEOUT`).
To take advantage of v0.46.6.0: upgrade and restart your worker
(`gbrain jobs supervisor stop && gbrain jobs supervisor start --detach`).
Existing queues need no migration steps — the new lease column defaults
every existing row to its handler's lease at next claim. If you tuned
around the old eviction behavior (e.g. very high `--max-stalled`), you can
likely lower it now. Mixed fleets are safe: an old worker simply keeps the
old 30s behavior until restarted.
## [0.46.5.0] - 2026-08-15
**CI in half, evals actually gating.** The Test workflow ran 89.5 minutes on
every push; its long pole was a serial-test job that executed ~140 per-file bun
processes strictly one-at-a-time even though the quarantine only ever required
per-process isolation. This release pools that lane (8.5 min → ~4 min in CI,
~2.5 min locally), wires the PGLite schema-snapshot fast-path into the CI test
runners (it previously existed but only the local loop used it), and rebalances
the 10-shard matrix on freshly mined weights (new files without a mined weight
now fall back to the p75 file weight instead of the median) — a measured branch
run landed the whole workflow at 255s. E2E stops spending real provider tokens on doc-only
pushes (content-hash skip with nightly + manual-dispatch exemptions) and runs
its tiers in parallel behind a fast broken-build spend gate.
Retrieval quality now has a hermetic CLI canary: `gbrain eval gate` accepts a
deterministic embedder option that drives the full hybrid/RRF pipeline with
zero API keys, gated in CI on every run (`check:eval-canary`, alongside the new
`check:eval-chronicle` gate) with its run ledger committed to
`.gbrain-evals/eval-results.jsonl`. Two registered-but-never-executed guards
came alive, a registration⇒execution coverage test closes that class for good,
and 47 orphaned eval-harness tests joined the CI matrix behind a keyless
allowlist. Test reliability hardening rounds it out: externally-killed serial
files get a sequential rescue re-run (never a silent pass), machine-global
files live on a growth-guarded exclusive lane, and the shard-balance test now
asserts the matrix CI actually runs instead of recomputing its own inputs.
**To take advantage of v0.46.5.0:** nothing to configure — CI and the local
loops (`bun run test`, `bun run test:serial`, `bun run verify`) are just
faster. New knobs if you need them: `GBRAIN_SERIAL_POOL=1` restores the old
fully-sequential serial lane, `GBRAIN_VERIFY_MAX_PARALLEL` bounds verify's
worker pool, `GBRAIN_NO_SNAPSHOT=1` opts any runner out of the snapshot
fast-path. Run the retrieval canary yourself with
`bun run scripts/run-eval-canary.ts` (add `--record` to append the committed
ledger).
## [0.46.4.0] - 2026-08-15
**opencode joins the supported-client roster — at full parity from day one.**
-2
View File
@@ -127,8 +127,6 @@ the database name must carry "test" as a word segment (like `gbrain_test`
above) or destructive tests refuse to run — opt a differently-named database
in one-shot with `GBRAIN_E2E_ALLOW_DB=<name>`.
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
Use `bun run verify` before pushing. It runs 40+ guard checks in parallel
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
+111 -6
View File
@@ -1,5 +1,37 @@
# TODOS
## #4145 lock-renewal wave follow-ups (filed 2026-08-15)
- [ ] **P2 — Kill or reap the force-evicted handler process.** **What:** when the
grace-evict fires for a handler that ignores its AbortSignal, actually
terminate the handler's work (LLM loop cancellation vs shell child-tree
kill differ per handler class) or track it as a zombie instead of only
freeing the inFlight slot. **Why:** today the evicted handler keeps
burning CPU/spend on an already-saturated host while the worker claims
new work — the #4145 amplification loop — and the duplicate-external-
side-effect window during an asymmetric outage is bounded only by
handler cooperation, not by `hardEvictMs`. **Context:** deliberately
scoped out of the #4145 wave (grace-evict at
`src/core/minions/worker.ts` frees the slot; the alternative — retaining
the slot until handler exit — re-opens the wedged-slot class D8b closed).
Eviction frequency collapsed with verify-before-evict, so this is
hygiene, not the incident driver. Kill semantics need their own review.
**Effort:** M (human) / S (CC). **Priority:** P2.
- [ ] **P3 — Worker-level `--lock-duration` flag on `jobs work` + supervisor
passthrough.** **What:** a CLI flag for the worker-global default lease,
threaded through `buildWorkerArgs` (`src/core/minions/supervisor.ts`).
**Why:** convenience only — per-job/per-type leases
(`HANDLER_DEFAULT_LOCK_DURATION_MS`, `--lock-duration-ms`) plus the
`GBRAIN_LOCK_RENEWAL_*` env knobs already cover every incident-tuning
case shipped in the #4145 wave. **Context:** requested shape existed in
the issue; deferred because no production caller overrides
`lockDuration` and env wins for incident response. **Effort:** S.
**Priority:** P3.
- [ ] **Note for TODO-LR-2 (doctor `lock_renewal_health`, already filed
below):** the #4145 wave shipped exactly its inputs — audit events now
carry `cause`, `lateness_ms`, `overlap_skips`, `load1`/`cores`, `via`,
`deadline_deferred` — so the doctor check can classify starved-worker
vs DB-outage windows without new plumbing.
## v0.47 SEPTEMBER REMOVAL — ZeroEntropy (filed v0.46.3.0; TARGET: ship 2026-09-04..2026-09-08)
ZeroEntropy's hosted API dies 2026-09-04. v0.46.3.0 deprecated it (split-default:
@@ -177,14 +209,77 @@ fix-wave plan; the wave series (W0.5W9, 3.4, 3.6) tracks its own scope there.
- [ ] **Legacy Anthropic-SDK subagent loop deletion.** **Priority: P2.** One
release after W8 flips `agent.use_gateway_loop` default ON (flag stays as
the revert path for that release).
- [ ] **Deeper test-suite speedup** beyond the W0 snapshot default-on (which
already cut the full parallel suite ~4,900s → ~490s). **Priority: P3.**
Revisit with post-W0 timing data; diminishing returns until measured.
- [x] **Deeper test-suite speedup** beyond the W0 snapshot default-on
LANDED in the test/eval/CI speedup pass (serial pool 8.5min → ~2.5min,
snapshot in every CI runner + memoized loader, verify worker pool,
perf-gate row shrink, chunk-grain engine consolidation). Remaining
long-tail items are filed in "Test/eval/CI speedup pass deferrals" below.
- [ ] **PGLite schema build-time derivation** from SCHEMA_SQL via a named
transform list. **Priority: P3.** Only if W3's schema drift TEST proves
annoying in practice — the test alone kills the drift bug class (Codex
D4.8/D5.23: fresh-schema equivalence ≠ upgrade correctness; old-shape
bootstrap fixtures + replay coverage stay regardless).
## Test/eval/CI speedup pass deferrals (filed with the pass; plan: ~/.claude/plans/system-instruction-you-are-working-iterative-hopcroft.md)
Each was explicitly deferred in the pass's CEO/eng/outside-voice reviews.
- [ ] **Sleep-to-poll conversions.** **What:** replace ~49.5s of hard-coded
`setTimeout` waits with event/poll-based waits; no fake timers exist in the
suite. Worst offenders: test/minions.test.ts (12.2s across 43 sites),
test/process-cleanup.test.ts (5.0s), test/worker-lock-renewal-e2e.serial.test.ts
(4.0s), test/e2e/worker-abort-recovery.test.ts (3.6s), test/e2e/zombie-reaping.test.ts
(3.3s). **Why deferred:** careful per-site work against flake-hardened timings;
~50s ceiling. **Effort:** M. **Priority:** P3.
- [ ] **E2E: PGLite-only parallel lane + default SHARD.** **What:** run-e2e.sh runs
181 files sequentially (one bun cold start each); ~42 PGLite-only files need no
Postgres and no TRUNCATE-race protection — run them in a parallel lane; default
the existing SHARD support (only ci-local uses it). Fold into the Postgres
template-database entry below in this file (CREATE DATABASE … TEMPLATE, ~50ms).
**Why deferred:** e2e is off the CI critical path after the workflow restructure;
ci-local + nightly benefit only. **Effort:** M. **Priority:** P2.
- [ ] **Second PGLite snapshot keyed by dims/model.** **What:** ~34 test files
configure zembed/1280 and always cold-init (the snapshot's shape gate correctly
refuses the 1536 fixture). Bake a second snapshot per shape; the version-file
format already carries dims/model. **Why deferred:** moderate effort, small win,
and it interacts with the shape gate the memoized loader deliberately keeps hot.
**Effort:** M. **Priority:** P3.
- [ ] **Persistent-engine snapshot.** **What:** the snapshot fast-path only covers
in-memory engines (`!dataDir` gate at pglite-engine.ts). ~58 files pass
database_path and pay full cold init (~121s weighted). Needs tar-extract-into-
dataDir (or PGlite loadDataDir with a dataDir) design. **Effort:** M. **Priority:** P3.
- [ ] **Engine consolidation audit: doctor/bootstrap/migrations-v0_19_0.** **What:**
33 files construct 95 engines; chunk-grain-fts was consolidated in-pass, but
doctor.test.ts (9 engines), bootstrap.test.ts (9), migrations-v0_19_0.test.ts (7)
need a per-file audit — migration-from-old-schema tests structurally cannot share
a current-schema engine or use the snapshot. **Effort:** M. **Priority:** P3.
- [ ] **Verify per-check double-spawn removal.** **What:** each CHECKS entry costs a
`bun run <key>` startup before its bash script; invoking scripts directly from a
manifest would drop ~47 bun startups. **Why deferred:** micro-win; touches the
package.json-scripts-as-API convention. **Effort:** S. **Priority:** P3.
- [ ] **Snapshot-tar digest verification (defense-in-depth).** **What:** the CI
actions/cache for `test/fixtures/pglite-snapshot.tar` validates only the
schema-hash/dims lines in the sidecar `.version` — which travels in the SAME
cache entry, so both are forgeable together by anyone with cache write access.
Record a sha256 of the tar bytes in the version file at build time and have
`tryLoadSnapshot` verify it (mirror of the gitleaks fetch-fresh-digest
pattern). **Why deferred:** exploitability bounded by GitHub cache scoping
(fork caches isolated; poisoning needs push access) and impact is test-DB
contents only. **Effort:** S. **Priority:** P3.
- [ ] **Redact provider/DB strings in eval ledger writes.** **What:**
`EvalRunRecord.error` (free text) is persisted unredacted by
`persistRunRecord` (eval-run-all) and the canary's record mode into the now-
TRACKED `.gbrain-evals/eval-results.jsonl` — a failed keyed run whose error
embeds a connection string would ride a later commit into the public repo.
Route `record.error` + provider-derived params through
`redactConnectionInfo`/`redactPgUrl` before append; optionally add
`.gbrain-evals/` to the fixture-privacy scan surface. **Effort:** S.
**Priority:** P2.
- [ ] **check-image-decoders-embedded.sh into verify CHECKS.** **What:** the guard
runs its own `bun build --compile` (~60s) — too heavy per-verify. Revisit if the
binary-embed bug class recurs; guards-manifest.tsv carries the exemption note,
and the registration⇒execution coverage test allowlists it explicitly.
**Effort:** S. **Priority:** P3.
## Jobs fix-wave follow-ups (filed v0.45.15.0 — upstream issues #2/#3/#4)
- [ ] **P2 — `jobs submit --max-pending` public flag.** maxPending stays an
@@ -3011,8 +3106,13 @@ outside-voice triage on the reshaped plan.
- [ ] **v0.42+: ship the coordinated `gbrain-evals/baselines/v0.41-launch.baseline.ndjson`
+ `gbrain-evals/qrels/v0.41-launch.qrels.json` (hermetic-synthetic per D9).**
Generate locally via `gbrain bench publish --from <hermetic-test-corpus>` then
commit to the sibling gbrain-evals repo. Gives `gbrain eval gate` a canonical
baseline target so users don't have to bootstrap their own immediately.
commit to the sibling gbrain-evals repo. PARTIALLY SUPERSEDED by the test/eval/CI
speedup pass: an in-repo canonical qrels target now exists (`gbrain eval gate`
with the deterministic embedder option against `test/fixtures/eval-baselines/
qrels-search.json`; runner `scripts/run-eval-canary.ts`, CI-gated via
check:eval-canary, ledger `.gbrain-evals/eval-results.jsonl`). What remains
here is only the sibling-repo REGRESSION baseline (.baseline.ndjson for the
jaccard/top1 gate) — the correctness-gate half is done.
## v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+)
@@ -4088,7 +4188,12 @@ verify Voyage adapter integration in `src/core/ai/recipes/voyage.ts`).
## test infra (v0.26.4 follow-up — intra-file parallelism)
### Sweep cross-file shared-state contention; enable `bun test --concurrent` for another 2-3x speedup
**Priority:** P0
**Priority:** P3 (downgraded from P0 in the test/eval/CI speedup pass — premises stale:
the entry says "~58 PGLiteEngine instantiations", the suite now has 600+; the serial
quarantine grew from 4 files to ~140, and the pass's pooled serial runner + CI snapshot
+ verify pool delivered a comparable multiple for hours of work instead of the 1-2
weeks this sweep estimates. Re-scope against post-pass timing data before spending
anything here; `test.concurrent` adoption remains at zero.)
**Status:** v0.26.7 shipped foundation slice (helpers + lint + mock.module quarantine). v0.26.8 (env sweep) and v0.26.9 (PGLite sweep + codemod + measurement) carry the rest.
**What:** v0.26.4 shipped file-level parallel fan-out (8 shards) and got `bun run test` from 18 minutes to ~85s — a 12x speedup. The next layer is **intra-file** parallelism via Bun's `--concurrent` flag (or per-test `test.concurrent()` markers). This requires every test file to be safe under concurrent execution within the same `bun test` process.
+1 -1
View File
@@ -1 +1 @@
0.46.4.0
0.46.6.0
+17 -11
View File
@@ -11,11 +11,11 @@ Six test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Builds/refreshes the PGLite schema snapshot BEFORE the shard fan-out and exports `GBRAIN_PGLITE_SNAPSHOT` so PGLite-booting files restore a baked schema instead of replaying every migration (~10x wallclock on a full run; see "PGLite schema snapshot" below). Opt out: `GBRAIN_NO_SNAPSHOT=1`. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Builds/refreshes the PGLite schema snapshot BEFORE the shard fan-out and exports `GBRAIN_PGLITE_SNAPSHOT` so PGLite-booting files restore a baked schema instead of replaying every migration (~3.5x per booting file; see "PGLite schema snapshot" below). Opt out: `GBRAIN_NO_SNAPSHOT=1`. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out by `scripts/run-verify-parallel.sh` through a bounded worker pool (default `detect_cpus`; override `GBRAIN_VERIFY_MAX_PARALLEL`) with the heavy checks ordered first (typecheck, the two compile-embed checks, admin build, fuzz bundles, guard self-tests, the PGLite-booting eval checks, whole-tree greps). The battery includes the deterministic `check:eval-chronicle` and `check:eval-canary` eval gates. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~40s (pool-bounded; longest check dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation), run through a POOL of concurrent per-file processes — the isolation is per-process, not per-machine. Pool defaults to `min(detect_cpus, 4)` then memory-adapts (same doctrine as the parallel runner); a small growth-guarded set of files (machine-global state or contention-critical timing — see the justified `EXCLUSIVE_FILES` list in `scripts/run-serial-tests.sh`, capped at 3 by `test/scripts/serial-files.test.ts`) runs on a sequential EXCLUSIVE lane after the pool. Per-test timeout 120s (pooled contention headroom); each pooled file is wall-clock-killed at 300s (`timeout -k`, exit-hang containment). Externally-killed files (exit 143/137 or a missing exit sentinel — sibling-workspace cleanup, memory jetsam) get ONE sequential rescue re-run, mirroring the parallel runner's doctrine: phantoms stay green with a rescue note, real failures stay red. Prints per-file PASS lines plus a top-10 slowest-files list. Knobs: `GBRAIN_SERIAL_POOL=N` (explicit pool width — bypasses the memory clamp; `1` restores fully-sequential), `GBRAIN_SERIAL_FILE_TIMEOUT`. | ~2.5min for all ~140 files at pool=4 (was ~8.5min sequential) | Debugging quarantined files; CI's serial-tests job. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
There is no `check:all` script anymore — it was a second, hand-synced guard
@@ -32,11 +32,17 @@ self-test" below).
post-`initSchema()` PGLite data dir into `test/fixtures/pglite-snapshot.tar`
plus a version file; `PGLiteEngine.initSchema()` restores the tar instead of
replaying the embedded schema + all migrations when the env var
`GBRAIN_PGLITE_SNAPSHOT` points at it. Both `bun run test`
(`scripts/run-unit-parallel.sh`, before the shard fan-out) and
`scripts/ci-local.sh` call the builder unconditionally and export the env var.
Measured effect: a full parallel suite run drops ~10x (PGLite-booting files go
~1.63s → ~0.91s each). Properties:
`GBRAIN_PGLITE_SNAPSHOT` points at it. Runners activate it through the shared
`ensure_pglite_snapshot` helper in `scripts/lib/test-env.sh` (also home of
`detect_cpus` and `detect_available_mem_mb`), sourced by
`run-unit-parallel.sh`, `test-shard.sh`, `run-slow-tests.sh`,
`run-serial-tests.sh`, and `run-verify-parallel.sh`; `scripts/ci-local.sh`
calls the builder directly. The helper builds/refreshes the snapshot and
exports the env var, no-ops on `GBRAIN_NO_SNAPSHOT=1` or an already-inherited
path, and is non-fatal on build failure — tests fall back to cold init, with
a one-line "active" echo so a silent fallback stays visible in CI logs.
Measured effect: ~3.5x per PGLite-booting file (a cold boot replays every
migration, ~3.1s each on a CI shard). Properties:
- **Idempotent.** A hash short-circuit exits in ~40ms when the snapshot is
fresh, and REBUILDS a stale one. The hash covers `PGLITE_SCHEMA_SQL`, every
@@ -106,7 +112,7 @@ there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. CI is the ground truth for "did everything pass."
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`; files with no mined weight fall back to the p75 file weight so a new unweighted file can't silently unbalance a shard) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix) plus `evals/**/*.test.ts` (keyless-allowlist-gated — `test/scripts/evals-collection.test.ts`). Each shard's bun process is bounded by `--max-concurrency` (`GBRAIN_TEST_MAX_CONCURRENCY`, default 4). Every bun-test job — matrix shards, serial-tests, verify, the slow/eval jobs — activates the PGLite schema snapshot (built in-runner via `scripts/lib/test-env.sh`; the brainbench gate brings its own in-memory PGLite and skips it; the ~42MB tar is also cached across jobs via actions/cache, with the runner's own hash check staying authoritative). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in the pooled `serial-tests` job via `bun run test:serial` — one bun process per file preserves the `mock.module` quarantine; the pool runs those processes concurrently. `bun run verify` gets its own job too, as does the BrainBench memory-conformance gate (`brainbench` job → `scripts/ci-brainbench-gate.sh`, hermetic in-memory PGLite, ~15s), which compares HEAD's fresh run against master's committed baseline (`evals/brainbench/baselines/main.json`) — the `test-status` aggregate checks its result explicitly. E2E (`.github/workflows/e2e.yml`) mirrors the content-hash skip in its own `e2e-pass-<hash>` namespace (scheduled nightly runs are exempt and always run), runs tier1 and tier2 in parallel with the jsonb-parity job in front of tier2 as the token-spend gate, and aggregates through `e2e-status`. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include; `test/scripts/run-unit-parallel.test.ts` pins the wrapper's memory-adaptive concurrency and the OOM/external-kill serial rescue pass.
@@ -131,8 +137,8 @@ Triage rule: a `warn-pass` EXIT-HANG line in `.context/test-summary.txt` is NOT
- `*.test.ts` → fast loop (parallel up-to-4-shard fan-out, memory-adaptive).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset. One out-of-directory file rides this lane: `test/phantom-redirect-engine-parity.test.ts` (lives in `test/` for its PGLite arm, but its Postgres arm is only reachable through a DATABASE_URL-bearing lane — the unit wrappers strip the URL per #3485, so `run-e2e.sh`'s no-args list and CI's parity job carry it).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`), with those per-file processes POOLED (per-process isolation never required one-at-a-time execution). Files touching machine-global state (launchd/cron) live on the sequential `EXCLUSIVE_FILES` lane inside `scripts/run-serial-tests.sh` — growth-guarded to ≤3 entries with justification comments. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset. One out-of-directory file rides this lane: `test/phantom-redirect-engine-parity.test.ts` (lives in `test/` for its PGLite arm, but its Postgres arm is only reachable through a DATABASE_URL-bearing lane — the unit wrappers strip the URL per #3485, so `run-e2e.sh`'s no-args list and CI's parity job carry it). `run-e2e.sh` wraps each file in a hard outer timeout (default 180s; `GBRAIN_E2E_FILE_TIMEOUT=<seconds>` overrides) because a synchronously-blocking PGLite WASM call can outlive bun's timer-based `--timeout`; LLM-bound Tier-2 files (`skills.test.ts`, `zeroentropy-live.test.ts`) automatically get 4× the cap since real provider round-trips legitimately run past 180s.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
File diff suppressed because one or more lines are too long
+15
View File
@@ -64,6 +64,21 @@ CI. Production retrieval differs via the query cache, salience freshness,
expansion, etc. The gate measures retrieval quality with a fixed pipeline;
your users may see different results when the cache is warm.
For a fully hermetic run (CI canaries, keyless environments), add
`--embedder deterministic` to the correctness gate: query embeddings come
from the qrels fixture's basis-vector dims (`src/eval/deterministic-embed.ts`)
instead of the gateway, so the gate runs with no API keys and no network.
Correctness-gate-only — it is rejected together with `--baseline` (replay
re-embeds captured queries via the gateway) and requires `--qrels`. Bare
`hybridSearch` never reads or writes the semantic query cache, so a
deterministic run cannot poison cached production results. This is what CI's
`check:eval-canary` gate runs via `scripts/run-eval-canary.ts`: a throwaway
PGLite brain seeded from the qrels fixture, gating the hybrid ranking
pipeline (keyword/title/alias arms + RRF) with synthetic vectors. Honest
scope: semantic-embedding regressions remain the keyed eval suites' job.
Reproduce locally with `bun run scripts/run-eval-canary.ts` (`--record`
appends to the `.gbrain-evals/eval-results.jsonl` ledger).
### `.qrels.json` shape
Two equivalent representations per entry:
+8 -2
View File
@@ -51,8 +51,14 @@ Test infra: PGLite snapshot default-on for `bun run test`. Per-PGLite-file:
Full-suite wall-clock (post-snapshot): recorded in the W0 ship notes — see
the run banner of the W0 PR's `bun run test` evidence.
Retrieval canary: NOT RUN at W0 (production brain locked by live serve; W0
touches no search paths). REQUIRED before W1 lands.
Retrieval canary: PASS @ f2b40f7ef (hermetic deterministic-embedder CLI run;
recall@10=1.0000 first_relevant=1.0000 expected_top1=0.8333 vs floors
0.70/0.60/0.50; run `bun run scripts/run-eval-canary.ts` to reproduce, ledger:
.gbrain-evals/eval-results.jsonl). Honest scope: the canary gates the hybrid
ranking pipeline (keyword/title/alias arms + RRF against gold qrels) with
synthetic basis vectors — no API keys, no production brain, so the live-serve
lock is moot. Semantic-embedding regressions remain the keyed eval suites'
job. Wired into `bun run verify` as check:eval-canary.
Verified-bug status at W0 ship: cycle-lock refresh + fencing (TODO-OPS-2
closed), stall-death parent unblock, started_at ×4, modality carry, import
+24 -6
View File
@@ -361,16 +361,34 @@ claimable work waits. The escalation commands and thresholds live in the
[queue operations runbook](queue-operations-runbook.md) — that's the
canonical home for wedge recovery.
What can still bite: a *brief* blip during a long-running job can make
lock renewal miss, and the stall detector dead-letters the job after
`max_stalled` misses (schema column default 5; lock duration and stall
check interval are both 30 s).
What can still bite is now narrow. Lock renewal is verify-before-evict:
a thrown or timed-out renewal is never treated as loss — at the deadline
the worker asks the database the authoritative question (one fenced
re-check), so a starved-but-healthy job recovers its lease and keeps
working. Eviction happens only on a fenced miss (the row was genuinely
reclaimed — requeued with no attempt burned) or after a hard backstop
(default 2× the lease) during a total outage. Long LLM handlers also get
a 300 s lock lease by default (`HANDLER_DEFAULT_LOCK_DURATION_MS`)
instead of the worker-global 30 s, and the stall sweep grants a 15 s
reclaim grace so a just-recovered worker's renewal beats the sweep.
The remaining exposure: a genuinely dead worker's long-lease job waits
up to lease + grace + one sweep interval before requeue, and the stall
detector still dead-letters after `max_stalled` genuine misses (schema
column default 5).
Mixed-version fleets degrade gracefully: an old worker ignores the
`lock_duration_ms` column and runs the legacy 30 s behavior; new workers
honor old rows via the claim-time default. No drain or ordered restart
is required.
**Tune per-job.** `gbrain jobs submit` accepts `--max-stalled N`,
`--backoff-type fixed|exponential`, `--backoff-delay <ms>`,
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags.
`--timeout-ms N`, `--lock-duration-ms N` (lock lease, clamped to
[5 s, 1 h]), and `--backoff-jitter 0..1` as first-class flags.
These write onto the job row at submit time — which is what
`handleStalled()` reads — so per-job tuning is the real knob.
`handleStalled()` and the renewal timer read — so per-job tuning is the
real knob. The lock-renewal env knobs (incident escape hatches) are
documented in the [queue operations runbook](queue-operations-runbook.md).
### DO NOT pass `maxStalledCount` to `MinionWorker`
+51
View File
@@ -107,6 +107,57 @@ gbrain jobs smoke --wedge-rescue
`queue.add()` call. If you want a taller pile, raise the threshold via
`GBRAIN_QUEUE_WAITING_THRESHOLD=50 gbrain doctor`.
## Lock-renewal: reading an eviction, and the knobs
Since v0.46 lock renewal is **verify-before-evict**: a thrown or timed-out
renewal is never treated as loss. At the deadline the worker runs one fenced
re-check against the database — the only CERTAIN loss signal is that fenced
miss. Every renewal fault also writes a JSONL audit event
(`~/.gbrain/audit/lock-renewal-*.jsonl`) carrying the fields that answer the
first incident question — *was the database down, or was the worker starved?*
How to read a `gave_up` / eviction line:
| Field | Reading |
|---|---|
| `cause` | `call-timeout` = our own timer fired (starved loop, slow pool, or slow DB); `refused` = the driver threw (SQLSTATE in `error_code`); `fenced-lost` = certain reclaim, not an infrastructure fault. |
| `lateness_ms` | How late the renewal tick fired vs its own cadence. Tens of seconds = the WORKER was starved (the #4145 shape); ~0 with `refused` = the database was actually unreachable. |
| `load1` / `cores` | Raw loadavg at event time, with core count for normalization. |
| `overlap_skips` | Ticks skipped because a prior renewal call was still in flight. |
| `deadline_deferred` | The soft deadline passed but the fenced verify was unreachable — the job was KEPT and retried (the fence is the backstop). |
| `event_loop_delay …` (log line) | p99/max event-loop delay since the last successful renewal — the direct starvation measurement. |
A `Job N did not exit within 30s of abort` line after an infrastructure
abort is NOT an orphan leak: the handler is cooperatively cancelling. The
line carries the same cause/lateness/load fields. Caveat: eviction is
cooperative — an abort-IGNORING handler keeps running past every bound and
can duplicate external side effects until it exits; the worker only frees
the slot.
Env knobs (incident escape hatches; all validated, warn-once on bad values;
defaults derive from the per-job lease):
| Env var | Default | What it does |
|---|---|---|
| `GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS` | `min(lease/3, 15s)` | Per-call budget for each renewal attempt (raced + best-effort cancelled). |
| `GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS` | `min(lease/6, 30s)` | Headroom before lease expiry; the fenced verify fires when the NEXT tick would land past `lease - margin`. |
| `GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS` | `2 × lease` | Hard local backstop when even the verify is unreachable (total outage). Floored to the soft deadline. Setting it TO the soft deadline approximates the legacy abort-at-deadline behavior. |
| `GBRAIN_LOCK_RENEWAL_MAX_FAILURES` | 3 | Audit-event labeling only — never gates eviction. |
| `GBRAIN_MINION_STALL_RECLAIM_GRACE_MS` | 15000 | Stall-sweep reclaim grace: a lease that lapsed within this window is not reclaimed (starved-owner head start). `0` restores the legacy `lock_until < now()` predicate. Capped at 600000 (10 min, warn-once + clamp) — an oversized value would otherwise disable stalled-job recovery fleet-wide. |
Cross-knob invariants are enforced with warn-once clamps (margin < lease/2,
call timeout ≤ renewal cadence, hard evict ≥ soft deadline) — a
misconfigured knob can degrade cadence but cannot silently re-break the
deadline math.
Per-job lease: `gbrain jobs submit --lock-duration-ms N` (clamped
[5 s, 1 h] — enforced at submit, re-applied to the resolved lease at
claim, and backed by a database range constraint; `--dry-run` echoes the
clamped value that will actually be stored); long LLM handlers default to 300 s via
`HANDLER_DEFAULT_LOCK_DURATION_MS` in `src/core/minions/handler-timeouts.ts`.
Renewal cadence is `min(lease/2, 60 s)`. Trade-off: a genuinely dead
worker's long-lease job requeues after lease + grace + one sweep interval.
## Self-check: is a worker even running?
```bash
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "gbrain-context-engine",
"name": "gbrain",
"version": "0.46.4.0",
"version": "0.46.6.0",
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
"family": "bundle-plugin",
"configSchema": {
+5 -1
View File
@@ -94,6 +94,10 @@
"check:operations-filter-bypass": "bash scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "bash scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:eval-chronicle": "bun src/cli.ts eval chronicle",
"check:eval-canary": "bun run scripts/run-eval-canary.ts",
"check:pagetype-exhaustive": "bash scripts/check-pagetype-exhaustive.sh",
"check:pg-url-redaction": "bash scripts/check-pg-url-redaction.sh",
"check:source-scope-onboard": "bash scripts/check-source-scope-onboard.sh",
"postinstall": "bun run scripts/postinstall.ts",
"prepublish:clawhub": "bun run build:all",
@@ -160,7 +164,7 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.46.4.0",
"version": "0.46.6.0",
"overrides": {
"@hono/node-server": "^2.0.5",
"fast-uri": "^3.1.5",
+5 -2
View File
@@ -21,7 +21,10 @@ ROOT=$(cd "$(dirname "$0")/.." && pwd)
# - The redactor itself: src/core/url-redact.ts
# - Test fixtures that build redacted strings from full URLs
# - Documentation comments referring to the pattern
ALLOW_REGEX='url-redact\.ts|test/url-redact\.test\.ts|/\* allow-pg-url-literal \*/'
# The marker text is the exemption; its comment wrapper is not load-bearing
# (inside a /** block comment a literal `*/` would terminate the comment, so
# block-comment examples carry the bare marker).
ALLOW_REGEX='url-redact\.ts|test/url-redact\.test\.ts|allow-pg-url-literal'
# The pattern matches an unredacted Postgres URL appearing in a string
# literal, NOT preceded by `redactPgUrl(` or `***@`. We also match any
@@ -47,6 +50,6 @@ echo "ERROR: unredacted postgres:// URL found in source. Use redactPgUrl() befor
echo ""
echo "$FILTERED"
echo ""
echo "Allowed exemption: append \"/* allow-pg-url-literal */\" comment on the line"
echo "Allowed exemption: append an allow-pg-url-literal comment marker on the line"
echo "(only for fixtures and the redactor itself)."
exit 1
+9 -1
View File
@@ -41,6 +41,14 @@ ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
TARGET_DIR="${1:-test}"
# When scanning the default root, also lint evals/**/*.test.ts — those files
# are collected into the CI matrix (scripts/test-shard.sh) and must obey the
# same isolation rules as everything else CI executes. An explicit TARGET_DIR
# argument (guard self-test fixtures) scans only itself.
EXTRA_DIRS=""
if [ "$TARGET_DIR" = "test" ] && [ -d evals ]; then
EXTRA_DIRS="evals"
fi
ALLOWLIST_FILE="$ROOT/scripts/check-test-isolation.allowlist"
# Read allowlist (one filename per line, # comments allowed). Empty file
@@ -72,7 +80,7 @@ is_allowlisted() {
# Find non-serial unit test files (excluding test/e2e). Portable across
# bash 3.2 (macOS default) and bash 4+; no mapfile.
FILE_LIST="$(find "$TARGET_DIR" -name '*.test.ts' \
FILE_LIST="$(find "$TARGET_DIR" $EXTRA_DIRS -name '*.test.ts' \
-not -name '*.serial.test.ts' \
-not -path "*/e2e/*" \
-type f 2>/dev/null | sort)"
+4 -2
View File
@@ -101,10 +101,12 @@ done
IFS='|' eval 'PATTERN="${PATTERN_PARTS[*]}"'
# Find tool.
# evals/ joins the scan: its *.test.ts files are collected into the CI
# matrix (scripts/test-shard.sh) and carry the same privacy bar.
if command -v rg >/dev/null 2>&1; then
matches="$(rg -niH --no-heading -t ts "$PATTERN" test 2>/dev/null || true)"
matches="$(rg -niH --no-heading -t ts "$PATTERN" test evals 2>/dev/null || true)"
elif command -v grep >/dev/null 2>&1; then
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test 2>/dev/null || true)"
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test evals 2>/dev/null || true)"
else
echo "check-test-real-names: ERROR: neither rg nor grep available." >&2
exit 2
+2 -2
View File
@@ -67,13 +67,13 @@ if grep -Eq 'lockTimer[[:space:]]*=[[:space:]]*setInterval\([[:space:]]*async' "
echo " routes through src/core/minions/lock-renewal-tick.ts:"
echo
echo " setInterval(() => {"
echo " if (tickInFlight) return;"
echo " if (tickInFlight) { state.overlapSkips += 1; return; }"
echo " tickInFlight = true;"
echo " void runLockRenewalTick(deps, state)"
echo " .then(handleResult)"
echo " .catch(handlePostError)"
echo " .finally(() => { tickInFlight = false; });"
echo " }, lockDurationMs / 2);"
echo " }, renewalIntervalMs); // min(lease/2, 60s)"
exit 1
fi
+4 -4
View File
@@ -11,12 +11,12 @@
# bash scripts/ci-local.sh --clean # nuke named volumes for cold debug
# bash scripts/ci-local.sh --no-shard # debug: run E2E sequentially against postgres-1 only
#
# 4-way E2E sharding: 4 pgvector services on host ports 5434-5437. The 36 E2E
# files split N/4 per shard; shards run in parallel. Within a shard, files run
# 4-way E2E sharding: 4 pgvector services on host ports 5434-5437. The test/e2e/ file set splits
# roughly N/4 per shard; shards run in parallel. Within a shard, files run
# sequentially (TRUNCATE CASCADE no-race property documented in run-e2e.sh).
# Wall-time on a 16-core host: ~6 min sequential -> ~1.5-2 min sharded.
#
# Stronger than PR CI: PR CI runs only Tier 1's 2 files; this runs all 36.
# Stronger than PR CI: PR CI runs a handful of named files across its tiers; this runs every test/e2e file.
set -euo pipefail
@@ -231,7 +231,7 @@ else
echo "$SELECTED" | tr " " "\n" | grep -v "^$" > /tmp/e2e-selected.txt
fi'
else
# Empty file -> run-e2e.sh uses default glob (all 36 E2E files).
# Empty file -> run-e2e.sh uses default glob (every test/e2e file).
DIFF_E2E_PREP='> /tmp/e2e-selected.txt'
fi
RUN_PHASES_CMD="echo \"[runner] guards + typecheck (run once before sharding)\"
+6 -6
View File
@@ -17,15 +17,15 @@
# guard class selftest notes
check-no-double-retry.sh scanner yes regex hole fixed in W0 (could not match `() =>`); perl multi-line pass replaces never-installed pcregrep
check-jsonb-pattern.sh scanner yes nested-paren hole fixed in W0; safe ::text::jsonb spelling stays unflagged
check-jsonb-params.mjs scanner yes positional $N::jsonb AST-lite scanner; argv/env root override
check-jsonb-params.mjs scanner yes positional $N::jsonb AST-lite scanner; argv/env root override; not in verify CHECKS: exercised by its unit test + self-test fixtures
check-batch-audit-site.sh scanner todo
check-bun-test-timeout.sh scanner todo
check-bun-test-timeout.sh scanner todo not in verify CHECKS: runs directly as a test.yml verify-job step
check-fixture-privacy.sh scanner todo
check-no-legacy-getconnection.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
check-no-pii-in-agent-voice.sh scanner todo
check-operations-filter-bypass.sh scanner todo
check-pagetype-exhaustive.sh scanner todo
check-pg-url-redaction.sh scanner todo
check-pagetype-exhaustive.sh scanner todo wired into verify CHECKS (v0.45.x test/eval/CI pass; was registered-but-never-executed)
check-pg-url-redaction.sh scanner todo wired into verify CHECKS (v0.45.x test/eval/CI pass; was registered-but-never-executed)
check-privacy.sh scanner todo
check-progress-to-stdout.sh scanner todo
check-proposal-pii.sh scanner todo
@@ -47,12 +47,12 @@ check-exports-count.sh scanner todo was reachable from neither verify nor CI pre
check-trailing-newline.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
check-test-isolation.sh scanner todo allowlist data file: check-test-isolation.allowlist
check-admin-build.sh buildfresh exempt runs the admin build; the build is the test
check-admin-embedded.sh buildfresh exempt embed freshness diff
check-admin-embedded.sh buildfresh exempt embed freshness diff; not in verify CHECKS: duplicates check:admin-build's build
check-admin-scope-drift.sh buildfresh exempt regenerates + diffs
check-bootstrap-templates.sh buildfresh exempt regenerates template tree + diffs
check-eval-glossary-fresh.sh buildfresh exempt regenerates + diffs
check-fuzz-purity.sh buildfresh exempt executes fuzz corpus
check-image-decoders-embedded.sh buildfresh exempt binary embed check
check-image-decoders-embedded.sh buildfresh exempt binary embed check; not in verify CHECKS: own bun build --compile too heavy per-verify
check-pglite-embedded.sh buildfresh exempt binary embed check
check-skills-manifest-fresh.sh buildfresh exempt regenerates + diffs
check-tool-catalog-fresh.sh buildfresh exempt regenerates + diffs
1 # CI guard registry (W0 fix-wave, Tier-1 #11 / D5.14).
17 # guard
18 check-no-double-retry.sh
19 check-jsonb-pattern.sh
20 check-jsonb-params.mjs
21 check-batch-audit-site.sh
22 check-bun-test-timeout.sh
23 check-fixture-privacy.sh
24 check-no-legacy-getconnection.sh
25 check-no-pii-in-agent-voice.sh
26 check-operations-filter-bypass.sh
27 check-pagetype-exhaustive.sh
28 check-pg-url-redaction.sh
29 check-privacy.sh
30 check-progress-to-stdout.sh
31 check-proposal-pii.sh
47 check-trailing-newline.sh
48 check-test-isolation.sh
49 check-admin-build.sh
50 check-admin-embedded.sh
51 check-admin-scope-drift.sh
52 check-bootstrap-templates.sh
53 check-eval-glossary-fresh.sh
54 check-fuzz-purity.sh
55 check-image-decoders-embedded.sh
56 check-pglite-embedded.sh
57 check-skills-manifest-fresh.sh
58 check-tool-catalog-fresh.sh
+79
View File
@@ -0,0 +1,79 @@
# scripts/lib/test-env.sh — shared helpers for the test-runner family
# (test-shard.sh, run-serial-tests.sh, run-slow-tests.sh, run-unit-parallel.sh,
# run-verify-parallel.sh). Source AFTER cd'ing to the repo root:
#
# . scripts/lib/test-env.sh
#
# bash 3.2 compatible (macOS system bash): no mapfile, no wait -n, no ${var^^}.
# Every helper degrades gracefully inside the script-sandbox tests
# (test/scripts/run-unit-parallel.test.ts symlinks a minimal PATH with no
# sysctl/nproc/vm_stat/timeout and no package.json).
# ──────────────────────────────────────────────────────────────────────────
# CPU detection: Apple Silicon perf cores → Mac total physical → nproc → 4.
# Returns a single positive integer.
# ──────────────────────────────────────────────────────────────────────────
detect_cpus() {
local n=""
n=$(sysctl -n hw.perflevel0.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(sysctl -n hw.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(nproc 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
echo 4
}
# ──────────────────────────────────────────────────────────────────────────
# Available-memory detection (MB). macOS: vm_stat free + inactive +
# speculative + purgeable pages (inactive/purgeable are reclaimable on
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
# Unknown platform → 0, and the caller skips adaptation entirely.
# ──────────────────────────────────────────────────────────────────────────
detect_available_mem_mb() {
if command -v vm_stat >/dev/null 2>&1; then
vm_stat 2>/dev/null | awk '
/page size of/ { psize = $8 }
/Pages free/ { free = $NF }
/Pages inactive/ { inactive = $NF }
/Pages speculative/ { spec = $NF }
/Pages purgeable/ { purge = $NF }
END {
gsub(/\./, "", free); gsub(/\./, "", inactive)
gsub(/\./, "", spec); gsub(/\./, "", purge)
if (psize == 0) psize = 16384
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
}'
return
fi
if [ -r /proc/meminfo ]; then
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
return
fi
echo 0
}
# ──────────────────────────────────────────────────────────────────────────
# PGLite schema snapshot: build (idempotent, ~40ms when fresh; mkdir-lock
# concurrency-safe; hash folds handler-migration source) and export
# GBRAIN_PGLITE_SNAPSHOT for child bun processes. 500+ test files each
# cold-boot PGLite + replay every migration without it (~3.5x per booting
# file — see docs/TESTING.md).
#
# No-op when GBRAIN_NO_SNAPSHOT=1 or when a parent runner already exported
# the path (double-building is harmless but noisy). Non-fatal on build
# failure — tests fall back to cold init. The one-line "active" echo makes
# a silent fall-back-to-cold-init regression visible in CI logs.
# $1: label for log lines (defaults to test-env).
# ──────────────────────────────────────────────────────────────────────────
ensure_pglite_snapshot() {
local label="${1:-test-env}"
[ "${GBRAIN_NO_SNAPSHOT:-0}" = "1" ] && return 0
if [ -n "${GBRAIN_PGLITE_SNAPSHOT:-}" ]; then
echo "[$label] PGLite snapshot active (inherited): $GBRAIN_PGLITE_SNAPSHOT" >&2
return 0
fi
if bun run build:pglite-snapshot >/dev/null 2>&1; then
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
echo "[$label] PGLite snapshot active: $GBRAIN_PGLITE_SNAPSHOT" >&2
else
echo "[$label] snapshot build failed (non-fatal) — tests run with cold init" >&2
fi
}
+25 -8
View File
@@ -96,6 +96,7 @@ mkdir -p "$E2E_TMP_HOME/.gbrain"
for _e2e_var in $(env | grep -oE '^(CONDUCTOR_|MCP_|OPENCLAW_|HERMES_|GROK_|OPENCODE_|GBRAIN_)[A-Za-z0-9_]*' | sort -u); do
case "$_e2e_var" in
GBRAIN_HOME) ;; # required for HOME isolation (set above) — keep
GBRAIN_PGLITE_SNAPSHOT) ;; # snapshot fast-path fixture (exported by ci-local.sh / runners) — keep
GBRAIN_TEST_ALLOW_DATABASE_URL) ;; # #3485 preload opt-in (set above) — keep
GBRAIN_E2E_ALLOW_DB) ;; # #3485 name-floor opt-in — the guard's own error
# message tells operators to set it; stripping it
@@ -185,16 +186,32 @@ for f in "${files[@]}"; do
if [ -n "${DATABASE_URL:-}" ]; then
psql "$DATABASE_URL" -At -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid != pg_backend_pid() AND datname = current_database()" >/dev/null 2>&1 || true
fi
# Hard outer timeout (180s per file). bun's --timeout covers tests AND
# hooks (measured on 1.3.14), but it's timer-based: a PGLite WASM call
# that blocks the event loop synchronously never lets the timer fire and
# the file wedges indefinitely. gtimeout/timeout SIGKILLs the file so the
# suite advances. gtimeout (macOS via coreutils) preferred; timeout (Linux)
# fallback; bare bun (no outer cap) if neither is installed.
# Hard outer timeout (default 180s per file; GBRAIN_E2E_FILE_TIMEOUT
# overrides). bun's --timeout covers tests AND hooks (measured on 1.3.14),
# but it's timer-based: a PGLite WASM call that blocks the event loop
# synchronously never lets the timer fire and the file wedges indefinitely.
# gtimeout/timeout SIGKILLs the file so the suite advances. gtimeout (macOS
# via coreutils) preferred; timeout (Linux) fallback; bare bun (no outer
# cap) if neither is installed.
#
# LLM-bound Tier-2 files (real provider round-trips when .env.testing
# carries keys) legitimately run past 180s — the ingest skill alone has
# been observed at ~131s — and were being SIGKILLed mid-run with no
# assertion output, which reads like a mystery failure. CI runs those
# files in their own job WITHOUT this wrapper (see .github/workflows/
# e2e.yml tier2), so the cap only ever bit local runs: give them 4x.
file_timeout="${GBRAIN_E2E_FILE_TIMEOUT:-180}"
# Digits-only validation (same strict positive-int posture as the TS env
# knobs): a malformed value falls back to the default instead of
# word-splitting into extra gtimeout arguments or breaking the 4x math.
case "$file_timeout" in ''|*[!0-9]*) file_timeout=180 ;; esac
case "$f" in
*/skills.test.ts|*/zeroentropy-live.test.ts) file_timeout=$((file_timeout * 4)) ;;
esac
if command -v gtimeout >/dev/null 2>&1; then
TIMEOUT_CMD="gtimeout 180"
TIMEOUT_CMD="gtimeout $file_timeout"
elif command -v timeout >/dev/null 2>&1; then
TIMEOUT_CMD="timeout 180"
TIMEOUT_CMD="timeout $file_timeout"
else
TIMEOUT_CMD=""
fi
+320
View File
@@ -0,0 +1,320 @@
/**
* scripts/run-eval-canary.ts — hermetic CLI retrieval-quality canary.
*
* Boots a throwaway PGLite brain under a temp GBRAIN_HOME, seeds the qrels
* fixture corpus, then spawns the REAL gbrain CLI to run the qrels
* correctness gate with the deterministic embedder (basis-vector query
* embeddings). No API keys, no network, no writes to the personal brain,
* no writes to tracked files in check mode.
*
* Seeding is the "V2" shape (feasibility-spike finding, BINDING): the
* expected-top1 page carries its query text in the `timeline` column too,
* because page-grain FTS (`pages.search_vector`) indexes title(A) +
* timeline(C) ONLY — `compiled_truth` is deliberately unindexed. With
* V1-style seeding (query text in compiled_truth only) the title arm votes
* only for the sibling page and expected_top1 is structurally 0.0.
*
* Modes:
* default check mode (CI): assert exit 0 + floors, print a one-line
* summary, clean up. Writes nothing to tracked files.
* record mode (pass the record flag) everything above PLUS append one
* EvalRunRecord-shaped JSONL line to
* <repo>/.gbrain-evals/eval-results.jsonl (the eval ledger).
*
* Honest scope: this gates the hybrid ranking pipeline (keyword/title/alias
* arms + RRF against gold qrels) with synthetic vectors. Semantic-embedding
* regressions remain the keyed eval suites' job.
*
* Budget: ≤60s under a saturated pool (two PGLite boots). If it breaches
* ~100s under contention, move the verify entry into the serial-tests CI
* job instead (same fallback as the chronicle eval).
*/
import { appendFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { execSync, spawnSync } from 'node:child_process';
import type { BrainEngine } from '../src/core/engine.ts';
import type { ChunkInput } from '../src/core/types.ts';
import { basisEmbedding, parseLegacyQrels } from '../src/eval/deterministic-embed.ts';
import type { LegacyQrelsQuery } from '../src/eval/deterministic-embed.ts';
const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const QRELS_PATH = join(ROOT, 'test', 'fixtures', 'eval-baselines', 'qrels-search.json');
// The embedding space the throwaway brain is pinned to. The gateway must be
// configured with this BEFORE initSchema (the schema's vector(dims) columns
// derive from gateway config at initSchema time — config read only, no key
// needed), and the brain's GBRAIN_HOME config.json pins the same model+dims
// so the CLI subprocess resolves 1536 too.
const EMBEDDING_MODEL = 'openai:text-embedding-3-large';
const EMBEDDING_DIMENSIONS = 1536;
// Env the child must NOT inherit: engine reroutes (a stray DATABASE_URL
// flips the engine to postgres; a brain id reroutes to a mount), embedding
// overrides (would fight the pinned 1536 space), and provider keys (the
// canary must behave identically keyed and keyless — determinism by
// construction, not by the parent's shell profile).
const CHILD_ENV_STRIP = [
'DATABASE_URL',
'GBRAIN_DATABASE_URL',
'GBRAIN_BRAIN_ID',
'GBRAIN_SOURCE',
'GBRAIN_EMBEDDING_MODEL',
'GBRAIN_EMBEDDING_DIMENSIONS',
'OPENAI_API_KEY',
'ANTHROPIC_API_KEY',
'ZEROENTROPY_API_KEY',
'VOYAGE_API_KEY',
'OPENROUTER_API_KEY',
'DASHSCOPE_API_KEY',
'GOOGLE_GENERATIVE_AI_API_KEY',
'GEMINI_API_KEY',
];
// The legacy qrels parser lives with the embedder builder — one parser for
// the shape (re-exported here for the test that drives this runner).
export { parseLegacyQrels };
export type { LegacyQrelsQuery };
/**
* Seed the V2 canary corpus. For each query's relevant slugs:
* - putPage typed by prefix (person/company/note), title = slug tail;
* the expected-top1 page carries the primary text in BOTH
* compiled_truth and timeline (the V2 amendment — timeline is what
* page-grain FTS indexes); siblings carry "Mentioned in context of
* <query>" in timeline.
* - upsertChunks with the same fixture text, basisEmbedding at the
* query's dim, token_count 10, chunk_source compiled_truth/timeline.
*/
export async function seedCanaryCorpus(engine: BrainEngine, queries: LegacyQrelsQuery[]): Promise<void> {
const seenSlugs = new Set<string>();
for (const q of queries) {
for (const slug of q.relevant_slugs) {
if (seenSlugs.has(slug)) continue;
seenSlugs.add(slug);
const isExpected = slug === q.first_relevant_slug;
const primaryText = `Primary content about ${q.query}`;
const mentionText = `Mentioned in context of ${q.query}`;
const type = slug.startsWith('people/')
? 'person'
: slug.startsWith('companies/')
? 'company'
: 'note';
await engine.putPage(slug, {
type,
title: slug.split('/').pop() ?? slug,
compiled_truth: isExpected ? primaryText : '',
// V2: the expected page's query text goes in timeline too — that is
// the page-grain-FTS-indexed column (title A + timeline C).
timeline: isExpected ? primaryText : mentionText,
});
const chunk: ChunkInput = {
chunk_index: 0,
chunk_text: isExpected ? primaryText : mentionText,
chunk_source: isExpected ? 'compiled_truth' : 'timeline',
embedding: basisEmbedding(q.embedding_dim, EMBEDDING_DIMENSIONS),
token_count: 10,
};
await engine.upsertChunks(slug, [chunk]);
}
}
}
interface GateJson {
verdict: 'pass' | 'fail';
correctness_gate: {
ran: boolean;
summary?: {
k: number;
queries_total: number;
queries_run: number;
queries_errored: number;
mean_recall_at_k: number;
first_relevant_hit_rate: number;
expected_top1_hit_rate: number;
expected_top1_denominator: number;
};
thresholds?: {
recall_at_k: number;
first_relevant_hit: number;
expected_top1: number;
};
breaches?: Array<Record<string, unknown>>;
};
}
function shortSha(): string {
try {
return execSync('git rev-parse --short HEAD', { cwd: ROOT, encoding: 'utf-8' }).trim();
} catch {
return 'unknown';
}
}
async function main(): Promise<number> {
const recordMode = process.argv.includes('--record');
const startedAt = Date.now();
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-eval-canary-'));
// Keep the RUNNER's own gbrain home inside the sandbox too, so nothing in
// the seeding path can read or write the operator's real ~/.gbrain.
process.env.GBRAIN_HOME = tmpHome;
try {
const gbrainDir = join(tmpHome, '.gbrain');
mkdirSync(gbrainDir, { recursive: true });
const dbPath = join(gbrainDir, 'brain.pglite');
writeFileSync(
join(gbrainDir, 'config.json'),
JSON.stringify(
{
engine: 'pglite',
database_path: dbPath,
embedding_model: EMBEDDING_MODEL,
embedding_dimensions: EMBEDDING_DIMENSIONS,
},
null,
2,
) + '\n',
);
// Gateway config BEFORE initSchema (dims gotcha above). Empty env
// snapshot: no key is consulted, and none is needed for schema sizing.
const { configureGateway } = await import('../src/core/ai/gateway.ts');
configureGateway({
embedding_model: EMBEDDING_MODEL,
embedding_dimensions: EMBEDDING_DIMENSIONS,
env: {},
});
const { PGLiteEngine } = await import('../src/core/pglite-engine.ts');
const engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite', database_path: dbPath });
await engine.initSchema();
const qrelsRaw = readFileSync(QRELS_PATH, 'utf-8');
await seedCanaryCorpus(engine, parseLegacyQrels(qrelsRaw));
// PGLite is single-writer: release the brain before the CLI child opens it.
await engine.disconnect();
const childEnv: Record<string, string | undefined> = { ...process.env };
for (const k of CHILD_ENV_STRIP) delete childEnv[k];
childEnv.GBRAIN_HOME = tmpHome;
// Spawn the REAL CLI. cwd is the temp home (not the repo) so repo-local
// dotfiles and Bun-auto-loaded .env files can't reroute the brain.
const child = spawnSync(
process.execPath,
[join(ROOT, 'src', 'cli.ts'), 'eval', 'gate', '--qrels', QRELS_PATH, '--embedder', 'deterministic', '--json'],
{
cwd: tmpHome,
env: childEnv as NodeJS.ProcessEnv,
encoding: 'utf-8',
timeout: 110_000,
maxBuffer: 32 * 1024 * 1024,
},
);
if (child.error) {
process.stderr.write(`[eval-canary] FAIL: could not spawn the CLI: ${child.error.message}\n`);
return 1;
}
if (child.status !== 0) {
process.stderr.write(`[eval-canary] FAIL: gate exit=${child.status ?? 'null(timeout/signal)'}\n`);
process.stderr.write(`[eval-canary] gate stdout tail:\n${(child.stdout ?? '').slice(-2000)}\n`);
process.stderr.write(`[eval-canary] gate stderr tail:\n${(child.stderr ?? '').slice(-2000)}\n`);
return 1;
}
// Parse the gate's JSON envelope (stdout carries only the envelope; any
// engine warnings go to stderr).
const stdout = child.stdout ?? '';
const jsonStart = stdout.indexOf('{');
if (jsonStart < 0) {
process.stderr.write(`[eval-canary] FAIL: no JSON found on gate stdout:\n${stdout.slice(-2000)}\n`);
return 1;
}
const gate = JSON.parse(stdout.slice(jsonStart)) as GateJson;
const summary = gate.correctness_gate.summary;
const floors = gate.correctness_gate.thresholds;
if (gate.verdict !== 'pass' || !summary || !floors) {
process.stderr.write(`[eval-canary] FAIL: verdict=${gate.verdict} summary=${JSON.stringify(summary)}\n`);
return 1;
}
// Exit 0 already implies floors held; assert explicitly anyway so a
// future exit-code regression in the gate can't silently pass the canary.
const breaches: string[] = [];
if (summary.queries_errored > 0) breaches.push(`queries_errored=${summary.queries_errored}`);
if (summary.mean_recall_at_k < floors.recall_at_k) breaches.push(`recall ${summary.mean_recall_at_k} < ${floors.recall_at_k}`);
if (summary.first_relevant_hit_rate < floors.first_relevant_hit) breaches.push(`first_relevant ${summary.first_relevant_hit_rate} < ${floors.first_relevant_hit}`);
if (summary.expected_top1_denominator > 0 && summary.expected_top1_hit_rate < floors.expected_top1) {
breaches.push(`expected_top1 ${summary.expected_top1_hit_rate} < ${floors.expected_top1}`);
}
if (breaches.length > 0) {
process.stderr.write(`[eval-canary] FAIL: ${breaches.join('; ')}\n`);
return 1;
}
const commit = shortSha();
const durationMs = Date.now() - startedAt;
process.stdout.write(
`[eval-canary] PASS commit=${commit}` +
` mean_recall_at_k=${summary.mean_recall_at_k.toFixed(4)}` +
` first_relevant_hit_rate=${summary.first_relevant_hit_rate.toFixed(4)}` +
` expected_top1_hit_rate=${summary.expected_top1_hit_rate.toFixed(4)}` +
` floors=${floors.recall_at_k}/${floors.first_relevant_hit}/${floors.expected_top1}` +
` k=${summary.k} queries=${summary.queries_run}/${summary.queries_total}` +
` duration_ms=${durationMs}\n`,
);
if (recordMode) {
// EvalRunRecord-shaped ledger line (matches src/commands/eval-run-all.ts;
// suite widened to the canary's own name, mode 'n/a' per the
// search-mode-independent convention).
const record = {
schema_version: 3,
run_id: `${commit}-retrieval-canary-na-0`,
ran_at: new Date().toISOString(),
suite: 'retrieval-canary',
mode: 'n/a',
commit,
seed: 0,
params: {
qrels: 'test/fixtures/eval-baselines/qrels-search.json',
embedder: 'deterministic',
k: summary.k,
metrics: {
mean_recall_at_k: summary.mean_recall_at_k,
first_relevant_hit_rate: summary.first_relevant_hit_rate,
expected_top1_hit_rate: summary.expected_top1_hit_rate,
expected_top1_denominator: summary.expected_top1_denominator,
queries_run: summary.queries_run,
queries_total: summary.queries_total,
},
floors: {
recall_at_k: floors.recall_at_k,
first_relevant_hit: floors.first_relevant_hit,
expected_top1: floors.expected_top1,
},
},
status: 'completed',
duration_ms: durationMs,
};
const ledgerDir = join(ROOT, '.gbrain-evals');
mkdirSync(ledgerDir, { recursive: true });
const ledgerPath = join(ledgerDir, 'eval-results.jsonl');
appendFileSync(ledgerPath, JSON.stringify(record) + '\n', 'utf-8');
process.stdout.write(`[eval-canary] recorded → ${ledgerPath}\n`);
}
return 0;
} catch (err) {
process.stderr.write(`[eval-canary] FAIL: ${(err as Error).stack ?? (err as Error).message}\n`);
return 1;
} finally {
rmSync(tmpHome, { recursive: true, force: true });
}
}
if (import.meta.main) {
process.exit(await main());
}
+273 -16
View File
@@ -1,18 +1,81 @@
#!/usr/bin/env bash
# scripts/run-serial-tests.sh — run *.serial.test.ts files with --max-concurrency=1.
# scripts/run-serial-tests.sh — run *.serial.test.ts files, one bun process per
# file, POOLED across files.
#
# Serial files are tests that share file-wide state (top-level mock.module,
# module-level singletons that intentionally cross test cases) and would race
# under intra-file concurrency. Discovered via filename suffix; no annotation
# inside the file is needed.
#
# Each file gets its OWN bun process. `--max-concurrency=1` alone was not
# enough: files in the same process share the module registry, so a top-level
# `mock.module(...)` in one file leaks into the next file's imports. Per-file
# processes give true isolation — and that isolation is per-PROCESS, not
# per-machine, so separate processes run CONCURRENTLY through the pool below.
# (The previous runner executed the ~140 processes strictly one-at-a-time:
# an 8.5-minute CI job whose serialization was never required by the
# quarantine contract.)
#
# Excluded by run-unit-shard.sh and run-unit-parallel.sh's parallel pass.
# Invoked separately by run-unit-parallel.sh after the parallel pass succeeds.
#
# Knobs:
# GBRAIN_SERIAL_POOL=N pool width (default min(detect_cpus, 4),
# then memory-adapted; 1 restores the old
# fully-sequential behavior)
# GBRAIN_SERIAL_FILE_TIMEOUT=S wall-clock kill per file (default 300;
# needs timeout/gtimeout on PATH, else no wrap)
# GBRAIN_TEST_MEM_PER_FILE_MB per-process memory budget (default 1536)
# GBRAIN_TEST_NO_MEM_ADAPT=1 skip the memory clamp
set -euo pipefail
# #3485: serial tests need no database — strip ambient DB URLs at this
# wrapper boundary (same four-layer guard as run-slow-tests.sh / the
# parallel runner) so the bunfig preload guard passes and nothing can
# reach a real brain.
unset DATABASE_URL GBRAIN_DATABASE_URL
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
# ──────────────────────────────────────────────────────────────────────────
# EXCLUSIVE_FILES: files that must never run concurrently with anything else
# (machine-global state or contention-critical timing). They run sequentially
# AFTER the pool drains, without the wall-clock kill (a SIGKILL
# mid-registration could strand a real scheduled job). Growth guard:
# test/scripts/serial-files.test.ts fails when this list grows past 3
# entries — every addition needs a justification comment like the entries
# below.
# ──────────────────────────────────────────────────────────────────────────
EXCLUSIVE_FILES=(
# launchd/cron lifecycle arc: install → self-disable → reinstall →
# uninstall ordering against (PATH-shimmed) launchctl; the arc asserts
# machine-level sequencing and is the flake-class canary.
"test/autopilot-launchd-lifecycle.serial.test.ts"
# hardenBrainRepo({installCron:true}) executes REAL launchctl/crontab
# (src/core/brain-repo-durability.ts) — a concurrent or killed run could
# strand a real scheduled job on the machine.
"test/brain-durability-hook.serial.test.ts"
# hardenBrainRepo's own scaffolding commit fires the just-installed
# post-commit hook (background push) which races the synchronous
# push-probe on the same bare remote ("cannot lock ref" →
# needs_attention non-empty). The race is intra-call; pooled CPU
# contention widens the window past what the assertions tolerate
# (observed on a 4-vCPU CI runner, never locally). Sequential lane
# restores master-era timing until the probe learns to retry ref-lock
# contention.
"test/brain-repo-durability.serial.test.ts"
)
is_exclusive() {
local f="$1" e
for e in "${EXCLUSIVE_FILES[@]}"; do
[ "$f" = "$e" ] && return 0
done
return 1
}
# Use while-read for portability to macOS bash 3.2 (no mapfile).
files=()
while IFS= read -r f; do
@@ -24,35 +87,229 @@ if [ "${#files[@]}" -eq 0 ]; then
exit 0
fi
# --dry-run-list mirrors run-unit-shard.sh for inline checks/tests.
# --dry-run-list mirrors run-unit-shard.sh for inline checks/tests. Lists
# ALL discovered files, pooled and exclusive alike.
if [ "${1:-}" = "--dry-run-list" ]; then
printf '%s\n' "${files[@]}"
exit 0
fi
echo "[serial-tests] running ${#files[@]} file(s), one bun process per file"
ensure_pglite_snapshot "serial-tests"
# Each serial file gets its OWN bun process. `--max-concurrency=1` was not
# enough: files in the same process share the module registry, so a top-level
# `mock.module(...)` in one file leaks into the next file's imports
# (eval-takes-quality-runner mocks gateway.ts and the next file fails on
# `import { resetGateway }` because the mock factory didn't export it).
# Per-file processes give true isolation; cost is ~100ms startup × N files.
fail_count=0
failed_files=()
# Partition into pooled vs exclusive (exclusive entries missing from the
# discovered set are simply ignored — the list names repo files, and a
# sandbox copy of this script won't have them).
pool_files=()
exclusive_present=()
for f in "${files[@]}"; do
if ! bun test --max-concurrency=1 --timeout=60000 "$f"; then
fail_count=$((fail_count + 1))
failed_files+=("$f")
if is_exclusive "$f"; then
exclusive_present+=("$f")
else
pool_files+=("$f")
fi
done
# ──────────────────────────────────────────────────────────────────────────
# Pool sizing: min(detect_cpus, 4) — each pooled bun process can hold a
# PGLite WASM instance (~1.5GB) — then clamped by available memory (same
# layer-1 doctrine as run-unit-parallel.sh, 4GB OS reserve).
# ──────────────────────────────────────────────────────────────────────────
POOL="${GBRAIN_SERIAL_POOL:-}"
if [ -z "$POOL" ]; then
POOL=$(detect_cpus)
[ "$POOL" -gt 4 ] && POOL=4
if [ "${GBRAIN_TEST_NO_MEM_ADAPT:-0}" != "1" ]; then
MEM_PER_FILE_MB="${GBRAIN_TEST_MEM_PER_FILE_MB:-1536}"
AVAIL_MB=$(detect_available_mem_mb)
if [ "${AVAIL_MB:-0}" -gt 0 ] 2>/dev/null; then
BUDGET_MB=$((AVAIL_MB - 4096))
[ "$BUDGET_MB" -lt "$MEM_PER_FILE_MB" ] && BUDGET_MB="$MEM_PER_FILE_MB"
MAX_POOL=$((BUDGET_MB / MEM_PER_FILE_MB))
[ "$MAX_POOL" -lt 1 ] && MAX_POOL=1
[ "$POOL" -gt "$MAX_POOL" ] && POOL="$MAX_POOL"
fi
fi
fi
if ! printf '%s' "$POOL" | grep -qE '^[0-9]+$' || [ "$POOL" -lt 1 ]; then
echo "[serial-tests] ERROR: invalid pool size: $POOL" >&2
exit 2
fi
# Wall-clock kill per pooled file: contains the exit-hang class (a bun
# process that finishes its tests but never exits). SIGTERM first, SIGKILL
# after a grace period (`timeout -k`). macOS without coreutils has neither
# binary — run unwrapped there (CI is Linux and always wraps).
PER_FILE_TIMEOUT="${GBRAIN_SERIAL_FILE_TIMEOUT:-300}"
TIMEOUT_BIN=""
command -v timeout >/dev/null 2>&1 && TIMEOUT_BIN="timeout"
[ -z "$TIMEOUT_BIN" ] && command -v gtimeout >/dev/null 2>&1 && TIMEOUT_BIN="gtimeout"
LOG_DIR=$(mktemp -d "${TMPDIR:-/tmp}/gbrain-serial.XXXXXX")
trap 'rm -rf "$LOG_DIR"' EXIT
if [ -n "$TIMEOUT_BIN" ]; then
TIMEOUT_DESC="${PER_FILE_TIMEOUT}s via $TIMEOUT_BIN"
else
TIMEOUT_DESC="none (no timeout/gtimeout on PATH)"
fi
echo "[serial-tests] ${#files[@]} file(s): pool=$POOL (${#exclusive_present[@]} exclusive), per-file timeout=$TIMEOUT_DESC"
# Per-test timeout is 120s (not the fast-loop 60s): pooled contention can
# push a 30-50s file past 60s — the same flake class the slow lane hardened
# against in v0.40.10. The literal `bun test --max-concurrency=1` below is
# contract-pinned by test/scripts/serial-files.test.ts.
run_one_file() {
# $1 file, $2 log path, $3 exit-sentinel path, $4 wrap ("wrap"|"nowrap")
local f="$1" log="$2" exitf="$3" wrap="$4" rc=0
if [ "$wrap" = "wrap" ] && [ -n "$TIMEOUT_BIN" ]; then
"$TIMEOUT_BIN" -k 15 "$PER_FILE_TIMEOUT" \
bun test --max-concurrency=1 --timeout=120000 "$f" > "$log" 2>&1 || rc=$?
else
bun test --max-concurrency=1 --timeout=120000 "$f" > "$log" 2>&1 || rc=$?
fi
echo "$rc" > "$exitf"
}
start_epoch=$(date +%s)
idx=0
if [ "${#pool_files[@]}" -gt 0 ]; then
for f in "${pool_files[@]}"; do
while [ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$POOL" ]; do
sleep 0.2
done
(
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$idx.log" "$LOG_DIR/$idx.exit" "wrap"
e=$(date +%s)
echo "$((e - s))" > "$LOG_DIR/$idx.dur"
) &
idx=$((idx + 1))
done
wait
fi
# Exclusive lane: sequential, unwrapped (see EXCLUSIVE_FILES comment).
if [ "${#exclusive_present[@]}" -gt 0 ]; then
for f in "${exclusive_present[@]}"; do
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$idx.log" "$LOG_DIR/$idx.exit" "nowrap"
e=$(date +%s)
echo "$((e - s))" > "$LOG_DIR/$idx.dur"
idx=$((idx + 1))
done
fi
# ──────────────────────────────────────────────────────────────────────────
# Aggregate from the exit sentinels.
# exit 0 → pass
# exit 124 → killed by the per-file wall-clock timeout (real
# failure: the exit-hang class this cap exists for)
# exit 143 / 137, or a → EXTERNAL-KILL class: a stray SIGTERM/SIGKILL from
# missing sentinel outside this runner (sibling workspaces' process
# cleanup, memory jetsam — the same class
# run-unit-parallel.sh rescues). Queued for ONE
# sequential rescue re-run below; a rescue that
# fails again is a real failure. Never a silent pass.
# anything else → real failure
# ──────────────────────────────────────────────────────────────────────────
ordered_files=()
if [ "${#pool_files[@]}" -gt 0 ]; then ordered_files+=("${pool_files[@]}"); fi
if [ "${#exclusive_present[@]}" -gt 0 ]; then ordered_files+=("${exclusive_present[@]}"); fi
fail_count=0
failed_files=()
rescue_files=()
# Aggregate pass count across pooled files, re-emitted below in bun's own
# " N pass" summary format so run-unit-parallel.sh's headline counter
# (bun_summary_count) still sees the serial suite's tests. Failing files'
# logs are cat'ed raw (their " N pass/fail" lines land in the stream
# directly), so only PASSING files accumulate here — no double counting.
pass_total=0
i=0
for f in "${ordered_files[@]}"; do
dur="?"
[ -f "$LOG_DIR/$i.dur" ] && dur=$(cat "$LOG_DIR/$i.dur")
if [ ! -f "$LOG_DIR/$i.exit" ]; then
echo "[serial-tests] KILLED ${dur}s $f — missing exit sentinel (external kill/OOM) — queued for serial rescue" >&2
rescue_files+=("$f")
else
rc=$(cat "$LOG_DIR/$i.exit")
if [ "$rc" = "0" ]; then
summary=$(grep -E '^ *[0-9]+ pass' "$LOG_DIR/$i.log" | tail -1 | tr -d ' ' || true)
n=$(printf '%s' "$summary" | grep -oE '^[0-9]+' || echo 0)
pass_total=$((pass_total + n))
echo "[serial-tests] PASS ${dur}s $f ${summary:+($summary)}"
elif [ "$rc" = "137" ] && [ "$dur" != "?" ] && [ "$dur" -ge "$PER_FILE_TIMEOUT" ] 2>/dev/null; then
# 137 with full duration = OUR timeout's SIGKILL escalation (a hang
# that ignored SIGTERM), not an external kill — a real failure; a
# rescue re-run would just re-hang for another ~315s.
echo "[serial-tests] FAIL ${dur}s $f — exit 137 (hang survived SIGTERM; killed by ${PER_FILE_TIMEOUT}s per-file timeout)" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
elif [ "$rc" = "143" ] || [ "$rc" = "137" ]; then
echo "[serial-tests] KILLED ${dur}s $f — exit $rc (external SIGTERM/SIGKILL) — queued for serial rescue" >&2
rescue_files+=("$f")
else
note=""
[ "$rc" = "124" ] && note=" (killed by ${PER_FILE_TIMEOUT}s per-file timeout)"
echo "[serial-tests] FAIL ${dur}s $f — exit $rc$note" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
fi
fi
i=$((i + 1))
done
# Rescue pass: one sequential, unpooled re-run per externally-killed file.
# Phantoms pass here and the run stays green (with a rescue note); real
# failures fail again and go red. Mirrors run-unit-parallel.sh's doctrine.
if [ "${#rescue_files[@]}" -gt 0 ]; then
echo "[serial-tests] rescue pass: ${#rescue_files[@]} externally-killed file(s), re-running serially" >&2
for f in "${rescue_files[@]}"; do
# Exclusive-lane files keep their no-kill contract on rescue too (the
# lane exists because a SIGKILL mid-registration strands real state).
wrap_mode="wrap"
is_exclusive "$f" && wrap_mode="nowrap"
s=$(date +%s)
run_one_file "$f" "$LOG_DIR/$i.log" "$LOG_DIR/$i.exit" "$wrap_mode"
e=$(date +%s)
rc=$(cat "$LOG_DIR/$i.exit" 2>/dev/null || echo 1)
if [ "$rc" = "0" ]; then
summary=$(grep -E '^ *[0-9]+ pass' "$LOG_DIR/$i.log" | tail -1 | tr -d ' ' || true)
n=$(printf '%s' "$summary" | grep -oE '^[0-9]+' || echo 0)
pass_total=$((pass_total + n))
echo "[serial-tests] PASS $((e - s))s $f ${summary:+($summary)} (rescued: external-kill phantom)"
else
echo "[serial-tests] FAIL $((e - s))s $f — exit $rc on rescue re-run" >&2
cat "$LOG_DIR/$i.log" >&2
fail_count=$((fail_count + 1))
failed_files+=("$f")
fi
i=$((i + 1))
done
fi
# Slowest-file table: feeds flake triage + future weight mining.
echo "[serial-tests] slowest files:"
i=0
for f in "${ordered_files[@]}"; do
[ -f "$LOG_DIR/$i.dur" ] && echo "$(cat "$LOG_DIR/$i.dur") $f"
i=$((i + 1))
done | sort -rn | head -10 | sed 's/^/ /'
total_epoch=$(( $(date +%s) - start_epoch ))
if [ "$fail_count" -gt 0 ]; then
echo "" >&2
echo "[serial-tests] $fail_count file(s) failed:" >&2
echo "[serial-tests] $fail_count file(s) failed (${total_epoch}s total):" >&2
for f in "${failed_files[@]}"; do
echo " - $f" >&2
done
exit 1
fi
echo "[serial-tests] all ${#files[@]} file(s) passed"
# bun-summary-format aggregate: run-unit-parallel.sh's headline counter
# (bun_summary_count awk: $1 numeric, $2 == "pass") reads this line — without
# it the serial suite's tests vanish from `bun run test`'s pass=N banner.
echo " $pass_total pass"
echo "[serial-tests] all ${#ordered_files[@]} file(s) passed in ${total_epoch}s (pool=$POOL)"
+3
View File
@@ -11,6 +11,9 @@ set -euo pipefail
unset DATABASE_URL GBRAIN_DATABASE_URL
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
ensure_pglite_snapshot "run-slow-tests"
slow_files=()
while IFS= read -r f; do
slow_files+=("$f")
+5 -48
View File
@@ -62,54 +62,11 @@ cd "$(dirname "$0")/.."
# fan-out — shards inherit a finished fixture. Opt out: GBRAIN_NO_SNAPSHOT=1
# (the migration-replay canary tests clear the env themselves regardless).
# ──────────────────────────────────────────────────────────────────────────
if [ "${GBRAIN_NO_SNAPSHOT:-0}" != "1" ]; then
if bun run build:pglite-snapshot >/dev/null 2>&1; then
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
else
echo "[run-unit-parallel] snapshot build failed (non-fatal) — tests run with cold init" >&2
fi
fi
# ──────────────────────────────────────────────────────────────────────────
# CPU detection: Apple Silicon perf cores → Mac total physical → nproc → 4.
# Returns a single positive integer.
# ──────────────────────────────────────────────────────────────────────────
detect_cpus() {
local n=""
n=$(sysctl -n hw.perflevel0.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(sysctl -n hw.physicalcpu 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
n=$(nproc 2>/dev/null) && [ -n "$n" ] && [ "$n" -gt 0 ] && echo "$n" && return
echo 4
}
# ──────────────────────────────────────────────────────────────────────────
# Available-memory detection (MB). macOS: vm_stat free + inactive +
# speculative + purgeable pages (inactive/purgeable are reclaimable on
# pressure, which is exactly the scenario we size for). Linux: MemAvailable.
# Unknown platform → 0, and the caller skips adaptation entirely.
# ──────────────────────────────────────────────────────────────────────────
detect_available_mem_mb() {
if command -v vm_stat >/dev/null 2>&1; then
vm_stat 2>/dev/null | awk '
/page size of/ { psize = $8 }
/Pages free/ { free = $NF }
/Pages inactive/ { inactive = $NF }
/Pages speculative/ { spec = $NF }
/Pages purgeable/ { purge = $NF }
END {
gsub(/\./, "", free); gsub(/\./, "", inactive)
gsub(/\./, "", spec); gsub(/\./, "", purge)
if (psize == 0) psize = 16384
printf "%d\n", (free + inactive + spec + purge) * psize / 1048576
}'
return
fi
if [ -r /proc/meminfo ]; then
awk '/MemAvailable/ { printf "%d\n", $2 / 1024; found = 1 } END { if (!found) print 0 }' /proc/meminfo
return
fi
echo 0
}
# detect_cpus / detect_available_mem_mb / ensure_pglite_snapshot live in the
# shared lib (also sourced by test-shard.sh, run-serial-tests.sh,
# run-slow-tests.sh) — one implementation, no copy drift.
. scripts/lib/test-env.sh
ensure_pglite_snapshot "run-unit-parallel"
# ──────────────────────────────────────────────────────────────────────────
# Argument parsing. --shards N override wins over $SHARDS; both are clamped.
+65 -21
View File
@@ -25,28 +25,61 @@ set -uo pipefail
cd "$(dirname "$0")/.."
# detect_cpus + ensure_pglite_snapshot (the PGLite-booting eval checks use
# the snapshot fast-path when the shape matches).
. scripts/lib/test-env.sh
# ──────────────────────────────────────────────────────────────────────────
# Checks to run. Order is irrelevant (parallel), but keep stable for log
# determinism + grep-ability. Each entry is a bun-script name (the
# `package.json` "scripts" key), invoked as `bun run <name>`.
# Checks to run. Each entry is a bun-script name (the `package.json`
# "scripts" key), invoked as `bun run <name>`.
#
# To add a check: append to this array. To skip in CI temporarily, comment
# the line — the parallel runner doesn't care about count.
# ORDER MATTERS for wallclock: the spawn loop below is capped at
# GBRAIN_VERIFY_MAX_PARALLEL workers, so the heaviest checks go FIRST
# (LPT-style — makespan ≈ max(longest check, total/POOL)). The heavy block:
# typecheck (tsc), two `cp -R src` + `bun build --compile` binary builds,
# the admin vite+tsc build, the fuzz bundles, guard self-tests, the
# PGLite-booting eval checks, and the whole-tree greps. Everything after is
# sub-second; that tail keeps its historical order for grep-ability.
#
# To add a check: append to the right block. To skip in CI temporarily,
# comment the line — the runner doesn't care about count.
# ──────────────────────────────────────────────────────────────────────────
CHECKS=(
# ── heavy (longest-first) ──
"typecheck"
"check:admin-build"
"check:wasm"
"check:pglite-embedded"
"check:fuzz-purity"
# W0 fix-wave (Tier-1 #11): guard self-tests — every scanner guard proves it
# can fail (bad fixture → exit 1) before it counts as coverage. Registry:
# scripts/guards-manifest.tsv (package.json's stale `check:all` copy deleted).
"check:guard-self-test"
# Chronicle eval: $0, deterministic, exit-0-only-on-perfect (6 gold tasks).
# Boots its own PGLite — budget ≤60s under a saturated pool; if it breaches
# ~100s under contention, move it into the serial-tests CI job instead.
"check:eval-chronicle"
# Retrieval canary: $0, hermetic, deterministic-embedder CLI run of the
# qrels correctness gate. Boots two PGLite processes (seed + real CLI) —
# budget ≤60s under a saturated pool; if it breaches ~100s under
# contention, move it into the serial-tests CI job instead (same fallback
# as eval-chronicle above).
"check:eval-canary"
"check:bootstrap-templates"
"check:skill-brain-first"
"check:conversation-parser"
"check:resolver"
"check:privacy"
"check:proposal-pii"
"check:test-names"
"check:test-isolation"
# ── light tail (sub-second greps; historical order) ──
"check:proposal-pii"
"check:jsonb"
"check:search-path"
"check:source-id-projection"
"check:source-config-leak"
"check:progress"
"check:no-tracked-symlinks"
"check:test-isolation"
"check:wasm"
"check:pglite-embedded"
"check:admin-build"
"check:admin-scope-drift"
"check:cli-exec"
"check:system-of-record"
@@ -55,15 +88,11 @@ CHECKS=(
"check:skills-manifest"
"check:no-pii-agent-voice"
"check:synthetic-corpus-privacy"
"check:skill-brain-first"
"check:fuzz-purity"
"check:operations-filter-bypass"
"check:gateway-routed"
"check:worker-pool-atomicity"
"check:doc-history"
"check:fixture-privacy"
"check:conversation-parser"
"check:resolver"
"check:source-scope-onboard"
"check:no-double-retry"
"check:batch-audit-site"
@@ -73,17 +102,14 @@ CHECKS=(
"check:pin-doc-privacy"
"check:worker-lock-renewal-shape"
"check:bootstrap-tag"
"check:bootstrap-templates"
"check:skill-refs"
# W0 fix-wave (Tier-1 #11): guard self-tests — every scanner guard proves it
# can fail (bad fixture → exit 1) before it counts as coverage. Registry:
# scripts/guards-manifest.tsv (package.json's stale `check:all` copy deleted).
"check:guard-self-test"
# Previously reachable ONLY from the deleted check:all (i.e. never run):
"check:newlines"
"check:exports-count"
"check:no-legacy-getconnection"
"typecheck"
# Revived registered-but-never-executed guards (this pass):
"check:pagetype-exhaustive"
"check:pg-url-redaction"
)
if [ "${#CHECKS[@]}" -eq 0 ]; then
@@ -124,8 +150,22 @@ if command -v gtimeout >/dev/null 2>&1; then TIMEOUT_BIN="gtimeout"
elif command -v timeout >/dev/null 2>&1; then TIMEOUT_BIN="timeout"
fi
# Bounded worker pool. Unbounded fan-out ran two `cp -R src` +
# `bun build --compile` builds, the admin vite build, tsc, and ~40 greps
# simultaneously on a 4-vCPU CI runner — pushing slow checks into the
# 120s per-check timeout (the documented flake class on slower hosts).
# Default = detect_cpus so a many-core dev machine keeps its wide fan-out;
# escape hatch: GBRAIN_VERIFY_MAX_PARALLEL=999.
MAX_PAR="${GBRAIN_VERIFY_MAX_PARALLEL:-$(detect_cpus)}"
if ! printf '%s' "$MAX_PAR" | grep -qE '^[0-9]+$' || [ "$MAX_PAR" -lt 1 ]; then
echo "ERROR: invalid GBRAIN_VERIFY_MAX_PARALLEL: $MAX_PAR" >&2
exit 2
fi
ensure_pglite_snapshot "verify-parallel"
START_TS=$(date +%s)
echo "[verify-parallel] running ${#CHECKS[@]} checks in parallel (timeout=${TIMEOUT}s, logs=$LOG_DIR)" >&2
echo "[verify-parallel] running ${#CHECKS[@]} checks (pool=$MAX_PAR, timeout=${TIMEOUT}s, logs=$LOG_DIR)" >&2
# ──────────────────────────────────────────────────────────────────────────
# Spawn one background process per check. Each child captures its own exit
@@ -138,6 +178,10 @@ echo "[verify-parallel] running ${#CHECKS[@]} checks in parallel (timeout=${TIME
PIDS=()
SAFE_NAMES=()
for c in "${CHECKS[@]}"; do
# Throttle to the worker pool (bash 3.2 — no wait -n; jobs -rp reaps).
while [ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$MAX_PAR" ]; do
sleep 0.1
done
safe="${c//:/_}"
SAFE_NAMES+=("$safe")
LOG_FILE="$LOG_DIR/$safe.log"
+21 -3
View File
@@ -96,6 +96,17 @@ export function computeMedian(values: number[]): number {
: sorted[mid]!;
}
/**
* Quantile (nearest-rank) of a list of numbers. Empty input returns 0.
* q in [0, 1]; q=0.75 is the missing-file fallback weight (see partition).
*/
export function computeQuantile(values: number[], q: number): number {
if (values.length === 0) return 0;
const sorted = [...values].sort((a, b) => a - b);
const idx = Math.min(sorted.length - 1, Math.max(0, Math.ceil(q * sorted.length) - 1));
return sorted[idx]!;
}
export interface PartitionOpts {
/**
* Weight to assign files that are absent from the weights map. Defaults
@@ -133,8 +144,15 @@ export function partition(
const shards: string[][] = Array.from({ length: n }, () => []);
if (files.length === 0) return shards;
// Compute fallback weight from the median of present weights, unless
// the caller supplied an explicit override.
// Compute fallback weight from the p75 of present weights, unless the
// caller supplied an explicit override. p75, not median: the weight
// distribution is extremely right-skewed (median ~27ms, mean ~800ms —
// most files are trivial greps, the tail boots PGLite), and files
// MISSING from the map skew heavy (new integration tests land unweighted
// more often than new pure-unit tests). A median fallback modeled 45% of
// the corpus at ~30ms and let one shard silently carry the unweighted
// heavies; p75 over-weights small new files slightly (harmless — LPT
// self-corrects on the next mine) instead of under-weighting big ones.
let fallback: number;
if (opts.fallbackWeight !== undefined) {
if (!Number.isFinite(opts.fallbackWeight) || opts.fallbackWeight < 0) {
@@ -144,7 +162,7 @@ export function partition(
}
fallback = opts.fallbackWeight;
} else {
fallback = computeMedian(Array.from(weights.values()));
fallback = computeQuantile(Array.from(weights.values()), 0.75);
}
// Cold-start guard: if the weights map is empty AND no explicit
// fallback was supplied, every effective weight would be 0 and LPT
+19 -2
View File
@@ -56,6 +56,8 @@ fi
cd "$(dirname "$0")/.."
. scripts/lib/test-env.sh
# Collect non-E2E, non-serial unit test files. Slow files INCLUDED — see
# header comment. Local run-unit-shard.sh excludes slow files (different
# policy by design).
@@ -72,7 +74,13 @@ cd "$(dirname "$0")/.."
# total bounded. With 10 matrix shards the per-shard total drops to ~272s.
# Dedicated jobs run in parallel so total CI wallclock = max(matrix ~4.5min,
# slow-eval ~3.3min, slow-entity-resolve-perf ~2.6min) ≈ 4.5min.
ALL_FILES=$(find test -name '*.test.ts' \
# evals/ is included: its *.test.ts files (eval-harness unit tests) were
# previously collected by NO runner — 45+ real tests never executed anywhere.
# Every collected evals file must be KEYLESS (no API keys, no network) —
# enforced by the allowlist guard in test/scripts/evals-collection.test.ts.
# The local fast loop (run-unit-shard.sh) stays test-only by design (see
# docs/TESTING.md "CI vs local: intentionally divergent file sets").
ALL_FILES=$(find test evals -name '*.test.ts' \
-not -name '*.serial.test.ts' \
-not -name 'eval-longmemeval-e2e.slow.test.ts' \
-not -name 'entity-resolve-perf.slow.test.ts' \
@@ -94,6 +102,12 @@ if [ "$DRY_RUN_LIST" = "1" ]; then
exit 0
fi
# Snapshot fast-path (after the dry-run exit so list-only calls stay
# instant): ~370 PGLite-booting matrix files pay ~3.1s cold init each
# without it. The echo inside makes silent cold-init regressions visible
# in CI logs.
ensure_pglite_snapshot "test-shard"
ALL_COUNT=$(printf '%s\n' "$ALL_FILES" | grep -c '^' || true)
SHARD_COUNT=$(printf '%s\n' "$SHARD_FILES" | grep -c '^' || true)
# grep -c on empty input returns 0 even with trailing newline edge cases
@@ -108,4 +122,7 @@ fi
# Convert newline-separated file list to argv. xargs handles the
# whitespace correctly without word-splitting on spaces in paths.
printf '%s\n' "$SHARD_FILES" | xargs bun test --timeout=60000
# --max-concurrency mirrors the local runner: unbounded intra-process
# concurrency under parallel PGLite boots produced real shard deaths (the
# 22-minute matrix timeout in test.yml records 13 of them).
printf '%s\n' "$SHARD_FILES" | xargs bun test --timeout=60000 --max-concurrency="${GBRAIN_TEST_MAX_CONCURRENCY:-4}"
+1201 -717
View File
File diff suppressed because it is too large Load Diff
+74 -3
View File
@@ -40,7 +40,7 @@ import {
parseQrelsFile,
type QrelsFile,
} from '../core/bench/qrels-file.ts';
import { runCorrectnessGate, type CorrectnessResult } from '../core/bench/correctness-gate.ts';
import { runCorrectnessGate, type CorrectnessGateOpts, type CorrectnessResult } from '../core/bench/correctness-gate.ts';
import { replayCore, type ReplaySummary } from './eval-replay.ts';
interface GateOpts {
@@ -55,6 +55,18 @@ interface GateOpts {
thresholdRecallAtK?: number;
thresholdFirstRelevantHit?: number;
thresholdExpectedTop1?: number;
/**
* Hermetic embedder selector. The only accepted value is 'deterministic':
* query embeddings come from the qrels fixture's basis-vector dims
* (src/eval/deterministic-embed.ts) instead of the gateway, so the
* correctness gate runs with no API keys. Correctness-gate-only; rejected
* when combined with the baseline regression gate (replay re-embeds
* captured queries via the gateway). Cache safety: this path drives bare
* `hybridSearch`, which never reads or writes the semantic query cache
* (both live in `hybridSearchCached`), so deterministic runs cannot
* poison cached production results by construction.
*/
embedder?: string;
}
interface Breach {
@@ -139,6 +151,10 @@ function parseArgs(args: string[]): GateOpts {
opts.thresholdExpectedTop1 = Number(next);
i++;
break;
case '--embedder':
opts.embedder = next;
i++;
break;
default:
break;
}
@@ -169,6 +185,13 @@ Thresholds (override baseline metadata; CLI > embedded > defaults):
--threshold-expected-top1 FLOAT Correctness: expected_top1-hit-rate floor (default ${DEFAULT_QRELS_THRESHOLDS.expected_top1})
-k, --k N Top-K for recall@K (default ${DEFAULT_QRELS_THRESHOLDS.k})
Hermetic mode (correctness gate only):
--embedder deterministic Embed queries as the qrels fixture's basis
vectors instead of calling the gateway
no API keys, fully reproducible (eval
canaries/CI). Rejected together with the
baseline regression gate.
Output:
--json Print JSON envelope to stdout
-h, --help Show this help
@@ -286,6 +309,7 @@ function runCorrectnessGateDispatch(
qrelsPath: string,
k: number,
cliOverrides: Pick<GateOpts, 'thresholdRecallAtK' | 'thresholdFirstRelevantHit' | 'thresholdExpectedTop1'>,
searchFn?: CorrectnessGateOpts['searchFn'],
): Promise<GateResult['correctness_gate']> {
return (async () => {
let qrelsFile: QrelsFile;
@@ -312,7 +336,7 @@ function runCorrectnessGateDispatch(
let result: CorrectnessResult;
try {
result = await runCorrectnessGate(engine, qrelsFile, { k });
result = await runCorrectnessGate(engine, qrelsFile, { k, ...(searchFn ? { searchFn } : {}) });
} catch (err) {
return {
ran: true,
@@ -448,6 +472,29 @@ export async function runEvalGate(engine: BrainEngine, args: string[]): Promise<
process.exit(2);
}
// Hermetic embedder validation. Only 'deterministic' is supported; the
// regression gate is out of scope (replay re-embeds captured queries via
// the gateway, which needs a provider key — defeating the hermetic point).
if (opts.embedder !== undefined) {
if (opts.embedder !== 'deterministic') {
console.error(
`Error: unsupported embedder "${opts.embedder}" — the only supported value is "deterministic".`,
);
process.exit(2);
}
if (opts.baseline) {
console.error(
'Error: the deterministic embedder cannot be combined with the baseline regression gate ' +
'(replay re-embeds captured queries via the gateway). Use it with the qrels correctness gate only.',
);
process.exit(2);
}
if (!opts.qrels) {
console.error('Error: the deterministic embedder requires a qrels file.');
process.exit(2);
}
}
const result: GateResult = {
schema_version: 1,
verdict: 'pass',
@@ -468,11 +515,35 @@ export async function runEvalGate(engine: BrainEngine, args: string[]): Promise<
if (opts.qrels) {
const k = opts.k ?? DEFAULT_QRELS_THRESHOLDS.k;
// Deterministic embedder: build a searchFn that threads basis-vector
// query embeddings (derived from the qrels fixture itself) into bare
// hybridSearch via the queryEmbedFn seam. The rest of the pipeline
// (keyword/title/alias arms, RRF, boosts) runs exactly as production.
let deterministicSearchFn: CorrectnessGateOpts['searchFn'] | undefined;
if (opts.embedder === 'deterministic') {
let queryEmbedFn: (text: string) => Float32Array;
try {
const { buildQrelsQueryEmbedFn } = await import('../eval/deterministic-embed.ts');
queryEmbedFn = buildQrelsQueryEmbedFn(readFileSync(opts.qrels, 'utf-8'));
} catch (err) {
console.error(
`Error: could not build the deterministic embedder from ${opts.qrels}: ${(err as Error).message}`,
);
process.exit(2);
}
const { hybridSearch } = await import('../core/search/hybrid.ts');
deterministicSearchFn = async (e, q, o) => {
const results = await hybridSearch(e, q, { limit: o.limit, queryEmbedFn });
return results.map(r => ({ source_id: r.source_id, slug: r.slug }));
};
}
result.correctness_gate = await runCorrectnessGateDispatch(engine, opts.qrels, k, {
thresholdRecallAtK: opts.thresholdRecallAtK,
thresholdFirstRelevantHit: opts.thresholdFirstRelevantHit,
thresholdExpectedTop1: opts.thresholdExpectedTop1,
});
}, deterministicSearchFn);
if (result.correctness_gate.breaches && result.correctness_gate.breaches.length > 0) {
result.verdict = 'fail';
}
+34 -2
View File
@@ -17,7 +17,7 @@ import type { PaceKeyOverrides } from '../core/pace-mode.ts';
import { loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
import { parseNiceValue, applyNiceness, getEffectiveNiceness, formatNice } from '../core/minions/niceness.ts';
import { defaultTimeoutMsFor } from '../core/minions/handler-timeouts.ts';
import { defaultTimeoutMsFor, defaultLockDurationMsFor, clampLockDurationMs } from '../core/minions/handler-timeouts.ts';
function parseFlag(args: string[], flag: string): string | undefined {
const idx = args.indexOf(flag);
@@ -242,7 +242,17 @@ function formatTimeoutLines(job: MinionJob): string[] {
if (d != null) {
lines.push(` Timeout: (unset) — handler default ${d}ms stamps at claim`);
} else {
lines.push(` Timeout: (unset) — null-default wall-clock sweep applies (2 x lock-duration x max_stalled, ~5m at defaults)`);
lines.push(` Timeout: (unset) — null-default wall-clock sweep applies (2 x lock lease x max_stalled, ~5m at 30s-lease defaults)`);
}
}
// #4145: the lock lease line mirrors the timeout line — row value when
// stamped, otherwise the handler-map default that WILL stamp at claim.
if (job.lock_duration_ms != null) {
lines.push(` Lock lease: ${job.lock_duration_ms}ms (renewed at min(lease/2, 60s) cadence)`);
} else {
const lease = defaultLockDurationMsFor(job.name);
if (lease != null) {
lines.push(` Lock lease: (unset) — handler default ${lease}ms stamps at claim`);
}
}
return lines;
@@ -286,6 +296,7 @@ USAGE
[--max-waiting N]
[--backoff-type fixed|exponential] [--backoff-delay Nms]
[--backoff-jitter 0..1] [--timeout-ms Nms]
[--lock-duration-ms Nms]
[--idempotency-key K] [--queue Q] [--dry-run]
[--redact-secrets] (shell only; scrubs inherit
values from stdout/stderr)
@@ -455,6 +466,7 @@ USAGE
[--max-waiting N]
[--backoff-type fixed|exponential] [--backoff-delay Nms]
[--backoff-jitter 0..1] [--timeout-ms Nms]
[--lock-duration-ms Nms]
[--idempotency-key K] [--queue Q] [--dry-run]
[--redact-secrets]
@@ -471,6 +483,10 @@ OPTIONS
source before coalescing new submissions ([1,100])
--timeout-ms Nms Per-job wall-clock budget. Long-lane handlers get a
default from HANDLER_DEFAULT_TIMEOUT_MS when omitted.
--lock-duration-ms N Per-job lock lease (#4145). Clamped to [5s, 1h].
Long-lane handlers default to 300s via
HANDLER_DEFAULT_LOCK_DURATION_MS; others use the
worker default (30s).
--idempotency-key K At-most-one row per key (dead/cancelled free the key)
--queue Q Target queue (default: default)
--dry-run Print what would be submitted, submit nothing
@@ -580,6 +596,15 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
console.error('Error: --timeout-ms must be a positive integer (milliseconds)');
process.exit(1);
}
// #4145: per-job lock lease. Clamped to [5s,1h] in queue.add via
// clampLockDurationMs (shared with the MCP op); NULL falls to the
// handler map, then the worker default.
const lockDurationMsRaw = parseFlag(args, '--lock-duration-ms');
const lockDurationMs = lockDurationMsRaw !== undefined ? parseInt(lockDurationMsRaw, 10) : undefined;
if (lockDurationMsRaw !== undefined && (isNaN(lockDurationMs!) || lockDurationMs! <= 0)) {
console.error('Error: --lock-duration-ms must be a positive integer (milliseconds)');
process.exit(1);
}
const idempotencyKey = parseFlag(args, '--idempotency-key');
const queueName = parseFlag(args, '--queue') ?? 'default';
const dryRun = hasFlag(args, '--dry-run');
@@ -603,6 +628,12 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
if (timeoutMs !== undefined) console.log(` Timeout: ${timeoutMs}ms`);
if (lockDurationMs !== undefined) {
// Echo what will actually be STORED (queue.add clamps to [5s,1h]);
// a dry-run that prints the raw out-of-range input lies.
const stored = clampLockDurationMs(lockDurationMs);
console.log(` Lock lease: ${stored}ms${stored !== lockDurationMs ? ` (clamped from ${lockDurationMs}ms)` : ''}`);
}
if (idempotencyKey) console.log(` Idempotency key: ${idempotencyKey}`);
if (delay > 0) console.log(` Delay: ${delay}ms`);
console.log(` Data: ${JSON.stringify(data)}`);
@@ -648,6 +679,7 @@ export async function runJobs(engineOrNull: BrainEngine | null, args: string[]):
backoff_delay: backoffDelay,
backoff_jitter: backoffJitter,
timeout_ms: timeoutMs,
lock_duration_ms: lockDurationMs,
idempotency_key: idempotencyKey,
queue: queueName,
}, trusted);
+46 -6
View File
@@ -47,6 +47,7 @@ import * as fs from 'fs';
import * as path from 'path';
import { createAuditWriter, resolveAuditDir, computeIsoWeekFilename } from './audit-writer.ts';
import { redactConnectionInfo } from './redact-connection-info.ts';
import type { LockRenewalTelemetryCtx } from '../minions/lock-renewal-tick.ts';
export type LockRenewalOutcome =
| 'failure'
@@ -77,6 +78,23 @@ export interface LockRenewalAuditEvent {
error_message_summary?: string;
/** Postgres SQLSTATE if present (e.g. '08006' for connection failure). */
error_code?: string;
// -- v0.46 additive telemetry (issue #4145 request 3). All optional: --
// -- pre-upgrade JSONL lines parse unchanged; the 4-outcome contract --
// -- is untouched. --
/** Failure-cause classification: call-timeout | refused | fenced-lost. */
cause?: string;
/** How late the renewal tick fired vs its own cadence (ms) — the local-starvation signal. */
lateness_ms?: number;
/** Interval callbacks skipped by the tickInFlight re-entrancy guard (overlap, NOT missed intervals). */
overlap_skips?: number;
/** os.loadavg()[0] at event time (raw, not normalized; 0 on Windows). */
load1?: number;
/** Core count paired with load1 so operators can normalize. */
cores?: number;
/** For success_after_failure: which path recovered — plain renewal or the at-deadline verify. */
via?: string;
/** True when the deadline was reached but eviction was deferred (verify threw pre-hard-deadline). */
deadline_deferred?: boolean;
}
const FEATURE_NAME = 'lock-renewal';
@@ -93,14 +111,33 @@ const writer = createAuditWriter<LockRenewalAuditEvent>({
* without writing to disk.
*/
export interface LockRenewalAuditSink {
logFailure(jobId: number, jobName: string, attempt: number, err: unknown): void;
logSuccessAfterFailure(jobId: number, jobName: string, recoveredAfterAttempts: number): void;
logGaveUp(jobId: number, jobName: string, totalFailures: number, err: unknown): void;
// Trailing ctx params are optional + additive (ENG-E2): the tick's
// structural SinkLike and pre-existing test fakes stay assignable.
logFailure(jobId: number, jobName: string, attempt: number, err: unknown, ctx?: LockRenewalTelemetryCtx): void;
logSuccessAfterFailure(jobId: number, jobName: string, recoveredAfterAttempts: number, ctx?: LockRenewalTelemetryCtx): void;
logGaveUp(jobId: number, jobName: string, totalFailures: number, err: unknown, ctx?: LockRenewalTelemetryCtx): void;
logExecuteJobRejected(jobId: number, jobName: string, err: unknown): void;
}
/**
* Copy only DEFINED ctx fields onto the event so absent telemetry stays
* absent from the JSONL (no `"load1":undefined` noise, stable byte size).
*/
function compactCtx(ctx?: LockRenewalTelemetryCtx): Partial<LockRenewalAuditEvent> {
if (!ctx) return {};
const out: Partial<LockRenewalAuditEvent> = {};
if (ctx.cause !== undefined) out.cause = ctx.cause;
if (ctx.lateness_ms !== undefined) out.lateness_ms = ctx.lateness_ms;
if (ctx.overlap_skips !== undefined) out.overlap_skips = ctx.overlap_skips;
if (ctx.load1 !== undefined) out.load1 = ctx.load1;
if (ctx.cores !== undefined) out.cores = ctx.cores;
if (ctx.via !== undefined) out.via = ctx.via;
if (ctx.deadline_deferred !== undefined) out.deadline_deferred = ctx.deadline_deferred;
return out;
}
export const lockRenewalAudit: LockRenewalAuditSink = {
logFailure(jobId, jobName, attempt, err) {
logFailure(jobId, jobName, attempt, err, ctx) {
writer.log({
job_id: jobId,
job_name: jobName,
@@ -108,18 +145,20 @@ export const lockRenewalAudit: LockRenewalAuditSink = {
outcome: 'failure',
error_message_summary: summarizeError(err),
error_code: extractErrorCode(err),
...compactCtx(ctx),
});
},
logSuccessAfterFailure(jobId, jobName, recoveredAfterAttempts) {
logSuccessAfterFailure(jobId, jobName, recoveredAfterAttempts, ctx) {
writer.log({
job_id: jobId,
job_name: jobName,
attempt: recoveredAfterAttempts,
outcome: 'success_after_failure',
// No error_message_summary or error_code: recovery has no error.
...compactCtx(ctx),
});
},
logGaveUp(jobId, jobName, totalFailures, err) {
logGaveUp(jobId, jobName, totalFailures, err, ctx) {
writer.log({
job_id: jobId,
job_name: jobName,
@@ -127,6 +166,7 @@ export const lockRenewalAudit: LockRenewalAuditSink = {
outcome: 'gave_up',
error_message_summary: summarizeError(err),
error_code: extractErrorCode(err),
...compactCtx(ctx),
});
},
logExecuteJobRejected(jobId, jobName, err) {
+2 -2
View File
@@ -8,7 +8,7 @@
* connection string into the error message:
* - `connection to server at "db.example.supabase.com" (1.2.3.4), port 5432 failed: ...`
* - `FATAL: password authentication failed for user "postgres"`
* - `could not connect to server: postgresql://user:pass@host:5432/db`
* - `could not connect to server: postgresql://user:pass@host:5432/db` (allow-pg-url-literal)
*
* If an operator pastes a JSONL audit dump into a GitHub issue or Slack,
* those errors leak credentials. The project's audit-as-debug-tool
@@ -34,7 +34,7 @@ interface RedactPattern {
* occurrences in a single string get redacted.
*/
const PATTERNS: ReadonlyArray<RedactPattern> = [
// postgres:// and postgresql:// URLs. Includes user:pass@host:port/db
// postgres:// and postgresql:// URLs. Includes user:pass@host:port/db /* allow-pg-url-literal */
// shapes plus query-string variants. Terminator is whitespace or
// common JSON/markdown delimiters.
{ kind: 'pg_url', re: /postgres(?:ql)?:\/\/[^\s"'>)]+/gi },
+7 -2
View File
@@ -570,7 +570,6 @@ async function runRoundtrip(
const putPage = findOp('put_page');
const getPage = findOp('get_page');
const queryOp = findOp('query');
const deletePage = findOp('delete_page');
await sweepProbeLeftovers(engine, ws, sourceId);
@@ -675,10 +674,16 @@ async function runRoundtrip(
}
// 6. Delete the probes [G13] — failure is a WARNING, never a verify fail.
// HARD delete via the engine primitive (same as sweepProbeLeftovers): the
// probe is not user content and verify is a trusted local caller. The
// delete_page OP is a v0.26.5 SOFT delete (sets deleted_at, row stays in
// pages until the 72h purge) — using it here left two probe tombstones in
// the user's brain after every verify run, visible to include_deleted
// readers and pinned as residue by the Postgres e2e cleanup assertion.
const deleteWarnings: string[] = [];
for (const slug of [VERIFY_PROBE_SLUG, VERIFY_PROBE_ENTITY_SLUG]) {
try {
await deletePage.handler(ctx, { slug });
await engine.deletePage(slug, { sourceId });
} catch (e) {
deleteWarnings.push(`${slug}: ${(e as Error).message}`);
}
+2 -2
View File
@@ -45,7 +45,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'edges-backfill': ['--aliases', '--all', '--all-sources', '--brain', '--concurrency', '--federated', '--help', '--include-null-signature', '--json', '--max-age', '--max-chunks', '--max-cost-usd', '--no-extract', '--no-federated', '--older-than', '--path', '--pattern', '--pending', '--repo', '--reset', '--resolve', '--restore-only', '--source', '--stale', '--supersessions', '--thin', '--timeout', '--workers'],
'embed': ['--aliases', '--all', '--background', '--batch-size', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--catch-up', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--model', '--multimodal', '--name', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--pace', '--pace-max-concurrency', '--parallel', '--path', '--pattern', '--pending', '--pglite', '--prefix', '--priority', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--serial', '--slugs', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--symbol-kind', '--thin', '--timeout', '--to', '--token-ttl', '--version'],
'enrich': ['--aliases', '--all', '--all-sources', '--allow-empty', '--apply', '--asof', '--auto', '--background', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--budget-usd-per-day', '--by-mention', '--clone-dir', '--code', '--concurrency', '--confirm-destructive', '--content', '--date', '--days', '--detail', '--dry-run', '--embedding-dimensions', '--embedding-model', '--entities', '--explain', '--fast', '--federated', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--from-pages', '--help', '--http', '--image', '--include-null-signature', '--json', '--judge-model', '--kind', '--lang', '--limit', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--max-usd', '--min-context', '--mode', '--model', '--multimodal', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--offset', '--older-than', '--order', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--reenrich-after', '--remediate', '--reset', '--resolve', '--restore-only', '--resume', '--save', '--session', '--session-id', '--since', '--slug', '--slugs', '--source', '--source-id', '--stale', '--stats', '--supersessions', '--surface', '--symbol-kind', '--thin', '--thin-threshold', '--timeout', '--to', '--token-ttl', '--trusted-extraction', '--types', '--url', '--url-managed', '--version', '--with-db', '--workers', '--yes'],
'eval': ['--ab-relational', '--against', '--aliases', '--all', '--allow-regression', '--background', '--baseline', '--batch', '--brain', '--brain-wide-max-cost-usd', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--committed-baseline', '--compare', '--compare-limit', '--concurrent', '--config-a', '--config-b', '--corpus', '--cycles', '--days', '--dedup-cosine', '--dedup-max-per-page', '--dedup-type-ratio', '--dimensions', '--distance-min', '--embedding-dimensions', '--embedding-model', '--expand', '--explain', '--fast', '--fixtures', '--follow', '--force', '--from-capture', '--from-db', '--from-pages', '--gold', '--grounding-min', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--json', '--judge', '--justification', '--k', '--limit', '--llm', '--max-pair-chars', '--max-tokens', '--max-usd', '--md', '--metric', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--no', '--no-cache', '--no-embed', '--no-embedding', '--no-expand', '--no-extract', '--no-llm', '--older-than', '--out', '--output', '--output-dir', '--parallel', '--pattern', '--pending', '--progress-interval', '--progress-json', '--qrels', '--queries-file', '--query', '--questions', '--quiet', '--receipt-dir', '--refresh-cache', '--remediate', '--reset', '--resolve', '--rrf-k', '--rubric-version', '--runs', '--sampling', '--save', '--seed', '--severity', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--stale', '--strategy', '--strict', '--suite', '--suites', '--supersessions', '--surface', '--task', '--thin', '--threshold', '--threshold-expected-top1', '--threshold-first-relevant-hit', '--threshold-jaccard', '--threshold-latency-multiplier', '--threshold-recall-at-k', '--threshold-top1', '--timeout', '--to', '--token-ttl', '--tool', '--top-k', '--top-regressions', '--until', '--update-baseline', '--usefulness-min', '--verbose', '--version', '--with-code-intel', '--yes'],
'eval': ['--ab-relational', '--against', '--aliases', '--all', '--allow-regression', '--background', '--baseline', '--batch', '--brain', '--brain-wide-max-cost-usd', '--budget-usd', '--budget-usd-answer', '--budget-usd-retrieval', '--committed-baseline', '--compare', '--compare-limit', '--concurrent', '--config-a', '--config-b', '--corpus', '--cycles', '--days', '--dedup-cosine', '--dedup-max-per-page', '--dedup-type-ratio', '--dimensions', '--distance-min', '--embedder', '--embedding-dimensions', '--embedding-model', '--expand', '--explain', '--fast', '--fixtures', '--follow', '--force', '--from-capture', '--from-db', '--from-pages', '--gold', '--grounding-min', '--harness', '--help', '--http', '--include-holdout', '--include-null-signature', '--input', '--json', '--judge', '--justification', '--k', '--limit', '--llm', '--max-pair-chars', '--max-tokens', '--max-usd', '--md', '--metric', '--min-recall', '--mode', '--model', '--models', '--modes', '--multimodal', '--name', '--no', '--no-cache', '--no-embed', '--no-embedding', '--no-expand', '--no-extract', '--no-llm', '--older-than', '--out', '--output', '--output-dir', '--parallel', '--pattern', '--pending', '--progress-interval', '--progress-json', '--qrels', '--queries-file', '--query', '--questions', '--quiet', '--receipt-dir', '--refresh-cache', '--remediate', '--reset', '--resolve', '--rrf-k', '--rubric-version', '--runs', '--sampling', '--save', '--seed', '--severity', '--short', '--show-toplevel', '--since', '--skip-replay', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slug-prefix', '--source', '--stale', '--strategy', '--strict', '--suite', '--suites', '--supersessions', '--surface', '--task', '--thin', '--threshold', '--threshold-expected-top1', '--threshold-first-relevant-hit', '--threshold-jaccard', '--threshold-latency-multiplier', '--threshold-recall-at-k', '--threshold-top1', '--timeout', '--to', '--token-ttl', '--tool', '--top-k', '--top-regressions', '--until', '--update-baseline', '--usefulness-min', '--verbose', '--version', '--with-code-intel', '--yes'],
'export': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dir', '--explain', '--federated', '--fix', '--follow', '--help', '--include-null-signature', '--json', '--lang', '--markdown', '--multimodal', '--near-symbol', '--no-extract', '--no-federated', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--repo', '--reset', '--resolve', '--restore-only', '--slug-prefix', '--source', '--stale', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--type'],
'extract': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--by-mention', '--catch-up', '--code', '--concurrency', '--dir', '--dry-run', '--explain', '--federated', '--follow', '--from-meetings', '--help', '--include-frontmatter', '--include-null-signature', '--infer-dates', '--json', '--kind', '--lang', '--markdown', '--max-age', '--max-cost-usd', '--multimodal', '--name-status', '--near-symbol', '--ner', '--no-extract', '--no-federated', '--older-than', '--pack', '--path', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--repo', '--reset', '--resolve', '--restore-only', '--run-id', '--since', '--slug', '--source', '--source-id', '--stale', '--strategy', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--type', '--verbose', '--workers', '--yes'],
'extract-conversation-facts': ['--aliases', '--all', '--all-sources', '--background', '--brain', '--brain-wide-max-cost-usd', '--break-lock', '--by-mention', '--clone-dir', '--code', '--concurrency', '--confirm-destructive', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fix', '--follow', '--force', '--force-break-lock', '--help', '--include-null-signature', '--json', '--judge-model', '--lang', '--limit', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-runtime', '--model', '--multimodal', '--near-symbol', '--no', '--no-embed', '--no-embedding', '--no-extract', '--older-than', '--override-disabled', '--path', '--pattern', '--pending', '--pglite', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--restore-only', '--segment-limit', '--session', '--since', '--sleep', '--slug', '--source', '--source-id', '--stale', '--supabase', '--supersessions', '--symbol-kind', '--thin', '--timeout', '--to', '--types', '--url', '--url-managed', '--version', '--workers', '--yes'],
@@ -61,7 +61,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'init': ['--all', '--brain', '--chat-model', '--check', '--ctx-size', '--embedding-dimensions', '--embedding-model', '--embeddings', '--entity', '--expansion-model', '--fast', '--flag', '--force', '--from-pages', '--grant-types', '--help', '--http', '--issuer-url', '--json', '--judge-model', '--key', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--multimodal', '--no', '--no-embed', '--no-embedding', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--path', '--pglite', '--provenance', '--reranking', '--schema-pack', '--scopes', '--skip-embed-check', '--source', '--stale', '--supabase', '--surface', '--to', '--token-ttl', '--touchpoint', '--url', '--version'],
'integrations': ['--auto', '--brain', '--dry-run', '--embeddings', '--fast', '--force', '--from-pages', '--help', '--http', '--json', '--no-embedding', '--overwrite', '--refresh', '--reranking', '--source', '--surface', '--target', '--token-ttl'],
'integrity': ['--aliases', '--all', '--auto', '--backend', '--background', '--brain', '--brain-wide-max-cost-usd', '--check', '--confidence', '--cost', '--dry-run', '--explain', '--fast', '--follow', '--force', '--fresh', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--limit', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--review-lower', '--skip-bare-tweet', '--skip-urls', '--source', '--stale', '--supabase', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl', '--type', '--url'],
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--job-id', '--job-isolation', '--json', '--kind', '--lang', '--limit', '--lock', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
'jobs': ['--abbrev-ref', '--aliases', '--all', '--allow-empty', '--allow-protected', '--allow-shell-jobs', '--apply', '--asof', '--auto', '--auto-fix', '--auto-with-prompt', '--background', '--backoff-delay', '--backoff-jitter', '--backoff-type', '--batch', '--batch-size', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--break-lock', '--budget-usd', '--budget-usd-per-day', '--by-mention', '--by-type', '--cached', '--catch-up', '--check', '--cli-path', '--cluster', '--cluster-errors', '--code', '--concurrency', '--confidence', '--confirm-destructive', '--content', '--date', '--days', '--delay', '--detach', '--diff-filter', '--dim', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--entities', '--exclude', '--exclude-standard', '--explain', '--fast', '--federated', '--federated-read', '--ff-only', '--file', '--fix', '--follow', '--force', '--force-break-lock', '--force-retry', '--format', '--fresh', '--from-meetings', '--from-pages', '--full', '--hard-deadline', '--health-interval', '--held-out', '--help', '--http', '--idempotency-key', '--image', '--include-frontmatter', '--include-gitignored', '--include-null-signature', '--infer-dates', '--inject-bootstrap', '--inline', '--input', '--install', '--interval', '--is-ancestor', '--job-id', '--job-isolation', '--json', '--kind', '--lang', '--limit', '--lock', '--lock-duration-ms', '--markdown', '--max-age', '--max-attempts', '--max-cost-usd', '--max-crashes', '--max-rss', '--max-runtime-min', '--max-sources', '--max-stalled', '--max-usd', '--max-waiting', '--mcp-only', '--migrate-only', '--min-context', '--missing-path', '--mode', '--model', '--multimodal', '--name-only', '--name-status', '--near-symbol', '--ner', '--nice', '--no', '--no-auto-embed', '--no-embed', '--no-embedding', '--no-extract', '--no-federate', '--no-federated', '--no-gpg-sign', '--no-hard-deadline', '--no-inject', '--no-mutate', '--no-pull', '--no-renames', '--no-schema-pack', '--no-verify', '--no-worker', '--non-interactive', '--now', '--offset', '--older-than', '--once', '--order', '--orphan', '--others', '--output', '--override-disabled', '--pace', '--pace-max-concurrency', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--phase', '--pid-file', '--priority', '--progress-interval', '--progress-json', '--queue', '--quiet', '--redact-secrets', '--reenrich-after', '--refresh-cache', '--refresh-ms', '--remediate', '--remediation-plan', '--repo', '--reset', '--resolve', '--respect-gitignore', '--restore-only', '--resume', '--retry-failed', '--review-lower', '--run-id', '--save', '--segment-limit', '--serial', '--session', '--session-id', '--short', '--show-toplevel', '--sigkill-rescue', '--since', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--sleep', '--slug', '--slugs', '--source', '--source-id', '--src-subpath', '--stale', '--stats', '--status', '--strategy', '--supersessions', '--surface', '--swap-only', '--symbol-kind', '--target', '--target-score', '--thin', '--thin-threshold', '--timeout', '--timeout-ms', '--to', '--token-ttl', '--trusted-extraction', '--type', '--types', '--uninstall', '--unsafe-bypass-dream-guard', '--url', '--user', '--verbose', '--verify', '--version', '--watch', '--wedge-rescue', '--with-db', '--workers', '--yes'],
'lint': ['--aliases', '--all', '--background', '--brain', '--brain-wide-max-cost-usd', '--dry-run', '--exclude', '--explain', '--fast', '--fix', '--follow', '--force', '--from-pages', '--help', '--http', '--include-null-signature', '--json', '--no-embedding', '--no-extract', '--pattern', '--pending', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--reset', '--resolve', '--source', '--stale', '--supersessions', '--surface', '--thin', '--timeout', '--token-ttl'],
'lsd': ['--brain', '--force-resume', '--help', '--json', '--judge-model', '--limit', '--list-runs', '--max-cost', '--max-far-set', '--max-ideas-per-judge-call', '--no-save', '--resume', '--retry-judge', '--save', '--source', '--strict-budget', '--yes'],
'maintain': ['--aliases', '--all', '--background', '--brain', '--break-lock', '--by-mention', '--catch-up', '--column', '--concurrency', '--content-audit', '--count', '--detach', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--explain', '--fast', '--fix', '--force', '--force-retry', '--force-schema', '--from-meetings', '--full', '--help', '--include-flagged', '--include-frontmatter', '--include-null-signature', '--index-audit', '--infer-dates', '--input', '--json', '--kind', '--lang', '--locks', '--markdown', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-usd', '--migrate-only', '--multimodal', '--near-symbol', '--ner', '--nice', '--no-extract', '--no-mutate', '--older-than', '--once', '--pack', '--parallel', '--params', '--path', '--pattern', '--pending', '--pglite', '--phase', '--pid-file', '--porcelain', '--probe-pglite', '--progress-json', '--query', '--queue', '--quiet', '--rebuild-rollup', '--regenerate', '--remediate', '--remediation-plan', '--reset', '--resolve', '--restore-only', '--resume', '--run-id', '--safe', '--scope', '--since', '--skills-dir', '--skip-failed', '--slugs', '--source', '--source-id', '--stale', '--status', '--supabase', '--supersessions', '--symbol-kind', '--target', '--target-score', '--thin', '--to', '--top-k', '--type', '--unsafe-bypass-dream-guard', '--url', '--verbose', '--window', '--workers', '--yes'],
+24
View File
@@ -5829,6 +5829,30 @@ export const MIGRATIONS: Migration[] = [
ALTER TABLE dream_verdicts ADD COLUMN IF NOT EXISTS triage_version INT;
`,
},
{
version: 130,
name: 'minion_jobs_lock_duration_ms',
// #4145: per-job lock lease. A single worker-global 30s lockDuration
// cannot serve both 2s shell jobs and 173s-average LLM subagent jobs —
// under host CPU saturation the renewal window was missed and healthy
// long-running handlers were force-evicted. The column carries an
// optional per-job lease; NULL means "worker default" (the pre-#4145
// behavior), so NO backfill is needed — the claim-time COALESCE against
// HANDLER_DEFAULT_LOCK_DURATION_MS (handler-timeouts.ts) owns all
// defaulting from here on. CHECK added via the idempotent drop-then-add
// pattern (v7 precedent) so migrated brains carry the same DB bound as
// fresh installs; the operative [5s,1h] range clamp lives app-side in
// clampLockDurationMs. No index: only per-row reads on already-indexed
// access paths (bootstrap-coverage: column-only, no probe needed).
// (Authored as v129; renumbered to v130 when #4152's
// dream_verdicts_triage_v1_columns landed the number first.)
idempotent: true,
sql: `
ALTER TABLE minion_jobs ADD COLUMN IF NOT EXISTS lock_duration_ms INTEGER;
ALTER TABLE minion_jobs DROP CONSTRAINT IF EXISTS chk_lock_duration_positive;
ALTER TABLE minion_jobs ADD CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000));
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+75 -20
View File
@@ -1,28 +1,44 @@
/**
* handler-timeouts.ts per-handler default wall-clock budgets (#1737).
* handler-timeouts.ts per-handler-type defaults for the TWO time knobs a
* Minion job carries: the wall-clock budget (`timeout_ms`, #1737) and the
* lock lease (`lock_duration_ms`, #4145). They are DIFFERENT quantities
* the budget bounds total runtime (subagent: 30min), the lease bounds how
* long a dead worker's claim survives before the stall sweep reclaims it
* (subagent: 300s) and deriving one from the other would couple two
* unrelated tunables behind an implicit ratio policy. Both maps live here,
* under one header, precisely so they can't drift apart unseen.
*
* Short jobs (shell, lint, backlinks) want the tight default wall-clock
* (`2 * lockDuration * max_stalled`, computed in `handleWallClockTimeouts`
* when `timeout_ms IS NULL`). Long jobs do not: a 30-min LLM loop or a
* 10-15 min embed backfill submitted WITHOUT an explicit `timeout_ms` would
* inherit that short null-default and get wall-clock-killed mid-progress
* one half of #1737's thrash.
* WALL-CLOCK BUDGETS: short jobs (shell, lint, backlinks) want the tight
* default wall-clock (`2 * lock * max_stalled`, computed in
* `handleWallClockTimeouts` when `timeout_ms IS NULL`). Long jobs do not: a
* 30-min LLM loop submitted WITHOUT an explicit `timeout_ms` would inherit
* that short null-default and get wall-clock-killed mid-progress.
*
* Three layers apply the default (an explicit `opts.timeout_ms` always wins):
* LOCK LEASES: the worker-global default lease is 30s right for 2s shell
* jobs (fast dead-worker reclaim), structurally wrong for 173s-average LLM
* jobs, which had to renew ~12 consecutive times per run; under host CPU
* saturation a missed window force-evicted healthy handlers (the #4145
* incident). Long handlers get a 300s lease (5 renewal chances at the
* clamped 60s cadence); `shell` deliberately stays NULL worker default
* (child processes don't starve the loop, and verify-before-evict protects
* them anyway). Trade: dead-worker reclaim for mapped types slows to
* lease + reclaim grace + sweep cadence.
*
* Both defaults apply through the same layers (an explicit opts value
* always wins):
*
* 1. SUBMIT `MinionQueue.add` stamps the default onto the row. The value
* lives in `minion_jobs.timeout_ms`, not worker memory, so wall-clock
* behavior is stable across worker restart.
* 2. CLAIM `MinionQueue.claim` COALESCEs a NULL `timeout_ms` from this
* map (and derives `timeout_at` from the coalesced value). This is the
* durable invariant: it covers rows inserted before layer 1 existed and
* any writer that bypasses add(). Persisted by the claim UPDATE, so the
* restart-stability property holds here too.
* 3. ONE-SHOT migration v128 backfilled `timeout_ms` for non-terminal
* rows that predate both layers (they would otherwise never re-claim
* or die at the short null-default first). v128's values are a
* deliberate authoring-time SNAPSHOT of this map do NOT sync v128
* when editing the map below; layer 2 owns all future drift.
* lives on minion_jobs, not worker memory, so behavior is stable
* across worker restart.
* 2. CLAIM `MinionQueue.claim` COALESCEs a NULL column from the map
* (and derives `timeout_at` / `lock_until` from the coalesced value).
* This is the durable invariant: it covers rows inserted before
* layer 1 existed and any writer that bypasses add().
* 3. ONE-SHOT migration v128 backfilled `timeout_ms` for pre-#1737
* rows. v128's values are a deliberate authoring-time SNAPSHOT do
* NOT sync v128 when editing the maps; layer 2 owns all future drift.
* (`lock_duration_ms` needs no backfill: NULL simply means "worker
* default", which is the pre-#4145 behavior.)
*
* The 30-min anchor matches the explicit value cycle/patterns.ts already
* passes for subagent jobs, so this generalizes an existing convention
@@ -67,3 +83,42 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
export function defaultTimeoutMsFor(jobName: string): number | null {
return HANDLER_DEFAULT_TIMEOUT_MS[jobName] ?? null;
}
const FIVE_MIN_MS = 5 * 60 * 1000;
const TWO_MIN_MS = 2 * 60 * 1000;
/**
* Default lock lease (ms) for long-running handler types (#4145). A handler
* not in this map returns `null` the worker-global `lockDuration` default
* (30s) applies see the header for why `shell` is deliberately absent.
* 300s = the issue's proposed lease for 173s-average LLM jobs (5 renewal
* chances at the 60s clamped cadence); 120s covers the single-LLM-call
* handlers whose p99 comfortably exceeds a 30s lease.
*/
export const HANDLER_DEFAULT_LOCK_DURATION_MS: Readonly<Record<string, number>> = {
subagent: FIVE_MIN_MS,
subagent_aggregator: FIVE_MIN_MS,
'embed-backfill': FIVE_MIN_MS,
'autopilot-cycle': FIVE_MIN_MS,
'autopilot-global-maintenance': FIVE_MIN_MS,
contextual_reindex_per_chunk: FIVE_MIN_MS,
chronicle_extract: TWO_MIN_MS,
'facts-absorb': TWO_MIN_MS,
};
export function defaultLockDurationMsFor(jobName: string): number | null {
return HANDLER_DEFAULT_LOCK_DURATION_MS[jobName] ?? null;
}
/** Clamp bounds for explicit per-job lease input [5s, 1h]. The floor
* kills the 1ms-lock foot-gun (instant stall-reclaim thrash); the ceiling
* kills the immortal-lock foot-gun (a crashed worker's claim surviving an
* hour). Shared by queue.add, the CLI flag, and the MCP op (ENG-E4) so
* three hand-copied bounds can't drift. The DB CHECK only enforces > 0;
* this clamp is the operative bound (precedent: max_stalled [1,100]). */
export const LOCK_DURATION_MS_MIN = 5_000;
export const LOCK_DURATION_MS_MAX = 3_600_000;
export function clampLockDurationMs(raw: number): number {
return Math.max(LOCK_DURATION_MS_MIN, Math.min(LOCK_DURATION_MS_MAX, Math.floor(raw)));
}
+429 -76
View File
@@ -18,14 +18,25 @@
* `Promise.race(call, timeoutPromise)` with `callTimeoutMs` so the
* call cannot pend longer than the configured budget.
*
* - **Threshold math**: with `lockDuration=30s` and `interval=15s`,
* a 3-strike count-based abort fires at t=45s but the lock has
* been reclaimable since t=30s a 15s window where another
* worker can claim the same job. `runLockRenewalTick` aborts based
* on `Date.now() - lastSuccessfulRenewalAt >= lockDuration -
* safetyMargin` (time-based), so the worker voluntarily releases
* BEFORE the stall detector can reclaim. The failure counter is
* kept for audit-event labeling only.
* - **Verify-before-evict (#4145, replaces the v0.41.22.2 abort-at-
* deadline doctrine)**: a thrown/timed-out renewal is NOT evidence of
* loss under event-loop starvation the UPDATE may even have landed
* server-side while the local race timeout won. When the NEXT tick
* would land past the soft deadline (`lockDuration - safetyMargin`,
* cadence-aware per CDX-4), the tick runs ONE bounded VERIFY renewal:
* fenced-true starved-but-ours, lease re-extended, keep working;
* fenced-false CERTAIN loss, abort (stall detector requeues, no
* attempt burned); verify unreachable defer and retry next tick,
* aborting only past the `hardEvictMs` backstop (a LOCAL decision
* under uncertainty that bounds blind external side effects during a
* total outage). The local clock only schedules WHEN to verify
* the DB fence decides WHETHER to evict; production binds `deps.now`
* to a monotonic source so wall-clock jumps can't distort the math.
* Accepted timing behavior (ENG-E1): at the 30s default a failed
* renewal (10s) + verify (10s) can span 20s > the 15s cadence, so
* tickInFlight skips one tick benign (a verify-success re-extends
* the lease and resets the baseline); do not "fix" it into an
* overlap. The failure counter is kept for audit-event labeling only.
*
* - **Cancelled-tick race**: if the job ends while a renewLock call
* is mid-flight, the IIFE in worker.ts must bail without writing
@@ -47,6 +58,43 @@
* below; tests + workers both consume it.
*/
/**
* Named error for the per-call timeout race so cause classification is
* name-based (`call-timeout` vs `refused`), never message-sniffing
* (issue #4145 request 3).
*/
export class RenewalCallTimeoutError extends Error {
constructor(what: string, ms: number) {
super(`${what} timed out after ${ms}ms`);
this.name = 'RenewalCallTimeoutError';
}
}
/**
* Why a renewal attempt failed, as far as the tick can tell:
* - `call-timeout` our own race timer fired (starved loop, slow pool,
* or slow DB the tick can't distinguish; lateness
* + load telemetry does).
* - `refused` the driver threw (SQLSTATE rides in the audit event).
* - `fenced-lost` the fenced UPDATE matched 0 rows: CERTAIN loss.
*/
export type RenewalFailureCause = 'call-timeout' | 'refused' | 'fenced-lost';
/**
* Optional telemetry threaded to the audit sink alongside each event.
* All fields additive the 4-outcome audit contract is unchanged and
* pre-upgrade JSONL lines parse fine without them.
*/
export interface LockRenewalTelemetryCtx {
cause?: RenewalFailureCause;
lateness_ms?: number;
overlap_skips?: number;
load1?: number;
cores?: number;
via?: 'renewal' | 'verify';
deadline_deferred?: boolean;
}
export interface LockRenewalKnobs {
/**
* Failure counter cap used ONLY for audit-event labeling.
@@ -56,17 +104,36 @@ export interface LockRenewalKnobs {
maxFailuresForAudit: number;
/**
* Per-renewLock-call timeout enforced via `Promise.race`.
* Env: `GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`. Default: `lockDuration / 3`.
* Env: `GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`. Default: `min(lockDuration / 3, 15s)`
* (the cap keeps a long per-job lease from inheriting a call budget that
* wedges tickInFlight across cadence windows).
* Bounds the "hung renewLock wedges the re-entrancy guard forever" vector.
*/
callTimeoutMs: number;
/**
* Time-based abort fires when `now - lastSuccessfulRenewalAt >=
* lockDuration - safetyMarginMs`. Default safety margin gives ~5s
* of headroom before another worker could reclaim the lock.
* Env: `GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`. Default: `lockDuration / 6`.
* The soft deadline is `lockDuration - safetyMarginMs`: once the NEXT
* tick would land past it, the tick runs the at-deadline VERIFY renewal
* (fenced re-check) instead of retrying blindly. The margin is the
* headroom the verify has to complete before the lease actually lapses.
* Env: `GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`. Default: `min(lockDuration / 6, 30s)`.
*/
safetyMarginMs: number;
/**
* The hard local backstop (#4145): when even the VERIFY renewal is
* unreachable (throws/times out) for this long since the last success,
* abort anyway. This is an uncertainty BOUND, not a certainty claim
* fenced-false is the only CERTAIN loss signal; the hard evict merely
* caps how long a handler with non-fenced EXTERNAL side effects may run
* blind during a total DB outage. APPROXIMATE by design: checked after
* a primary call + verify (overshoot up to ~2×callTimeoutMs + timer
* lateness), and eviction stays cooperative (an abort-ignoring handler
* outlives it the kill/reap follow-up TODO is the true bound).
* Setting this to the soft deadline approximates the legacy
* abort-at-deadline behavior (the verify still runs once).
* Env: `GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS`. Default: `2 × lockDuration`,
* floored to the soft deadline (warn-once on floor).
*/
hardEvictMs: number;
}
/**
@@ -90,15 +157,36 @@ export function _resetKnobWarningsForTests(): void {
* operator who sets `GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS=abc` gets a
* loud-but-not-fatal nudge AND a working worker.
*/
/**
* The renewal cadence cap: a long lease renews every 60s (multiple chances
* per window, matching the cycle refresher's philosophy) instead of the
* bare lease/2. Leases 120s keep the legacy /2 exactly. ONE home for the
* formula the worker's timer and resolveLockRenewalKnobs' default both
* derive from it, so the relational validation always runs against the
* cadence production actually uses.
*/
export const RENEWAL_INTERVAL_CAP_MS = 60_000;
export function renewalIntervalFor(lockDurationMs: number): number {
return Math.max(1, Math.min(Math.floor(lockDurationMs / 2), RENEWAL_INTERVAL_CAP_MS));
}
export function resolveLockRenewalKnobs(
env: Record<string, string | undefined>,
lockDurationMs: number,
intervalMs: number = renewalIntervalFor(lockDurationMs),
): LockRenewalKnobs {
const defaultMaxFailures = 3;
const defaultCallTimeout = Math.max(1, Math.floor(lockDurationMs / 3));
const defaultSafetyMargin = Math.max(1, Math.floor(lockDurationMs / 6));
// #4145: the derived defaults CAP at 15s/30s so a long per-job lease
// (300s) doesn't inherit a 100s call timeout (which would wedge
// tickInFlight across cadence windows) or a 50s margin. The 30s worker
// default keeps today's 10s/5s exactly. Env overrides are still applied
// first and then pass the relational validation below.
const defaultCallTimeout = Math.min(Math.max(1, Math.floor(lockDurationMs / 3)), 15_000);
const defaultSafetyMargin = Math.min(Math.max(1, Math.floor(lockDurationMs / 6)), 30_000);
const defaultHardEvict = lockDurationMs * 2;
return {
const knobs: LockRenewalKnobs = {
maxFailuresForAudit: parsePositiveInt(
env.GBRAIN_LOCK_RENEWAL_MAX_FAILURES,
defaultMaxFailures,
@@ -114,7 +202,54 @@ export function resolveLockRenewalKnobs(
defaultSafetyMargin,
'GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS',
),
hardEvictMs: parsePositiveInt(
env.GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS,
defaultHardEvict,
'GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS',
),
};
// [CDX-10] Relational validation: positive-integer parsing alone lets a
// margin exceed the lease or a call timeout exceed the cadence, which
// silently re-breaks the deadline math. Clamp the offending knob (to a
// relationally-valid derivation) and warn once per process per knob.
// Runs per job launch against the EFFECTIVE per-job lease.
if (knobs.safetyMarginMs >= lockDurationMs / 2) {
warnRelationalClamp(
'GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS',
`safetyMarginMs (${knobs.safetyMarginMs}) must be < lockDuration/2 (${lockDurationMs / 2})`,
defaultSafetyMargin,
);
knobs.safetyMarginMs = defaultSafetyMargin;
}
if (knobs.callTimeoutMs > intervalMs) {
warnRelationalClamp(
'GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS',
`callTimeoutMs (${knobs.callTimeoutMs}) must be <= the renewal cadence (${intervalMs}) or a slow call wedges tickInFlight across intervals`,
intervalMs,
);
knobs.callTimeoutMs = intervalMs;
}
const softDeadline = lockDurationMs - knobs.safetyMarginMs;
if (knobs.hardEvictMs < softDeadline) {
warnRelationalClamp(
'GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS',
`hardEvictMs (${knobs.hardEvictMs}) must be >= the soft deadline (${softDeadline})`,
softDeadline,
);
knobs.hardEvictMs = softDeadline;
}
return knobs;
}
function warnRelationalClamp(name: string, violation: string, clampedTo: number): void {
const key = `relational:${name}`;
if (!_warnedKnobs.has(key)) {
_warnedKnobs.add(key);
process.stderr.write(
`[lock-renewal] ${violation}; clamping to ${clampedTo}\n`,
);
}
}
function parsePositiveInt(raw: string | undefined, fallback: number, name: string): number {
@@ -148,11 +283,15 @@ function warnAndFallback(name: string, raw: string, fallback: number): number {
*/
export interface LockRenewalDeps {
/**
* The optional `opts.signal` is aborted when this call loses the tick's
* timeout race, so the underlying UPDATE is CANCELLED (postgres.js
* `.cancel()` via executeRawDirect) instead of orphaned on a checked-out
* pool slot for its full server-side duration the #6 starvation class.
* Optional-param widening keeps the legacy 3-arg test mocks compiling.
* The fenced renewal UPDATE. The optional `opts.signal` is aborted when
* this call loses the tick's timeout race (both attempts: primary AND the
* #4145 at-deadline verify), so the underlying UPDATE is CANCELLED
* (postgres.js `.cancel()` via executeRawDirect) instead of orphaned on a
* checked-out pool slot for its full server-side duration the #6
* starvation class. Cancellation is BEST-EFFORT (pool acquisition and PG
* protocol cancel are async; PGLite ignores the signal); the token FENCE
* is the correctness authority. Optional-param widening keeps the legacy
* 3-arg test mocks compiling.
*/
renewLock: (
jobId: number,
@@ -166,8 +305,9 @@ export interface LockRenewalDeps {
/**
* Injectable for hermetic Promise.race tests. Production:
* `globalThis.setTimeout`. The function must return a value that
* `clearTimeout` accepts, but this seam doesn't expose clearTimeout
* because the timeout race fires-and-forgets. The losing renewLock is no
* `clearTimeout` accepts: on the win path the race clears the losing
* timer (via global clearTimeout) so it doesn't later fire a stray
* no-op abort against a settled query. The losing renewLock is no
* longer merely abandoned: the timeout callback also aborts the per-call
* signal so the query releases its pool slot.
*/
@@ -190,6 +330,21 @@ export interface LockRenewalDeps {
* (optional-param) signature stays back-compatible with no-arg test mocks.
*/
reconnect?: (ctx?: { error?: unknown }) => Promise<void>;
/**
* OPTIONAL host-load probe for eviction telemetry (issue #4145 req 3).
* The worker binds `os.loadavg()[0]` + a cached core count. Every call
* is wrapped in try/catch here (CEO-F2) a throwing telemetry hook
* must never re-open the unhandledRejection class this module exists
* to close; on throw the event simply logs without load fields.
*/
loadSnapshot?: () => { load1: number; cores: number };
/**
* OPTIONAL success hook. The worker resets its event-loop-delay
* histogram here (R2-9) so an eviction-time sample attributes to the
* window since the LAST SUCCESSFUL renewal, not process lifetime.
* Best-effort: called inside try/catch.
*/
onRenewalSuccess?: () => void;
}
/**
@@ -198,9 +353,11 @@ export interface LockRenewalDeps {
* import the full audit module and inflate the test surface.
*/
export interface LockRenewalAuditSinkLike {
logFailure(jobId: number, jobName: string, attempt: number, err: unknown): void;
logSuccessAfterFailure(jobId: number, jobName: string, recoveredAfterAttempts: number): void;
logGaveUp(jobId: number, jobName: string, totalFailures: number, err: unknown): void;
// The trailing `ctx` is optional and additive (ENG-E2): pre-existing
// test fakes with the old arity stay structurally assignable.
logFailure(jobId: number, jobName: string, attempt: number, err: unknown, ctx?: LockRenewalTelemetryCtx): void;
logSuccessAfterFailure(jobId: number, jobName: string, recoveredAfterAttempts: number, ctx?: LockRenewalTelemetryCtx): void;
logGaveUp(jobId: number, jobName: string, totalFailures: number, err: unknown, ctx?: LockRenewalTelemetryCtx): void;
}
export interface LockRenewalState {
@@ -221,13 +378,45 @@ export interface LockRenewalState {
* cancellation event AND the post-await branch decisions.
*/
cancelled: () => boolean;
/**
* The renewal timer's cadence (ms) the SAME value the worker gave
* setInterval. Used for tick-lateness telemetry (below) and the
* cadence-aware verify trigger.
*/
intervalMs: number;
/**
* Timestamp of the previous tick's entry, on the SAME clock as
* `deps.now` (production binds a monotonic source; R2-4). Seeded to
* launch time. Lateness = max(0, now - lastTickFiredAt - intervalMs)
* is the PRIMARY local-starvation signal: interval callbacks COALESCE
* under a blocked event loop (one late callback, not N), so a
* missed-tick counter cannot measure starvation lateness can.
*/
lastTickFiredAt: number;
/**
* Count of interval callbacks skipped by the worker's tickInFlight
* re-entrancy guard. These are OVERLAP skips (a prior tick still in
* flight), NOT missed intervals see lastTickFiredAt. Incremented by
* the worker's interval closure; read here for telemetry.
*/
overlapSkips: number;
}
export type TickResult =
| { kind: 'ok' }
| { kind: 'cancelled' }
| { kind: 'lock_lost' }
| { kind: 'should_abort'; reason: 'lock-renewal-failed' };
| { kind: 'lock_lost'; cause: 'fenced-lost'; via: 'renewal' | 'verify' }
| {
kind: 'should_abort';
reason: 'lock-renewal-failed';
/** What the FINAL failed attempt looked like (R2-7 telemetry). */
cause: RenewalFailureCause;
latenessMs: number;
sinceLastSuccessMs: number;
overlapSkips: number;
load1?: number;
cores?: number;
};
/**
* Execute one renewal tick. Returns a tagged result the worker switches
@@ -238,69 +427,145 @@ export async function runLockRenewalTick(
deps: LockRenewalDeps,
state: LockRenewalState,
): Promise<TickResult> {
// Tick-lateness telemetry (CDX-13): how late did this callback fire vs
// its own cadence? Computed FIRST — even a cancelled tick advances the
// baseline so the next measurement stays honest.
const tickEnteredAt = deps.now();
const latenessMs = Math.max(0, tickEnteredAt - state.lastTickFiredAt - state.intervalMs);
state.lastTickFiredAt = tickEnteredAt;
if (state.cancelled()) return { kind: 'cancelled' };
let renewed: boolean;
// Per-call cancellation: when the timeout wins the race, abort the signal
// so the losing UPDATE releases its pool slot instead of holding it until
// the server finishes (issue #6 — an abandoned racer under a saturated
// pooler pinned a checked-out connection for minutes). On the win path the
// late-firing timer aborts an already-settled query, which runUnsafe
// ignores (abort listener removed in its .finally).
const callAbort = new AbortController();
try {
renewed = await Promise.race([
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs, { signal: callAbort.signal }),
new Promise<never>((_, reject) => {
deps.setTimeout(() => {
callAbort.abort();
reject(new Error(`renewLock timed out after ${state.knobs.callTimeoutMs}ms`));
}, state.knobs.callTimeoutMs);
}),
]);
renewed = await racedRenewLock(deps, state);
} catch (err) {
if (state.cancelled()) return { kind: 'cancelled' };
state.consecutiveFailures += 1;
const cause = classifyFailure(err);
const load = safeLoadSnapshot(deps);
const telemetry: LockRenewalTelemetryCtx = {
cause,
lateness_ms: latenessMs,
overlap_skips: state.overlapSkips,
...load,
};
// Defense-in-depth (codex C4): audit must never escape this catch.
try {
deps.audit.logFailure(state.jobId, state.jobName, state.consecutiveFailures, err);
deps.audit.logFailure(state.jobId, state.jobName, state.consecutiveFailures, err, telemetry);
} catch { /* audit best-effort */ }
const sinceLastSuccess = deps.now() - state.lastSuccessfulRenewalAt;
const deadline = state.lockDurationMs - state.knobs.safetyMarginMs;
if (sinceLastSuccess >= deadline) {
try {
deps.audit.logGaveUp(state.jobId, state.jobName, state.consecutiveFailures, err);
} catch { /* audit best-effort */ }
return { kind: 'should_abort', reason: 'lock-renewal-failed' };
}
// issue #1678 (Codex #2): not yet at the deadline, so we'll retry on the
// next tick. If the engine can rebuild its pool, do it ONCE now (bounded
// by callTimeoutMs) so the next renewLock sees a live connection instead
// of throwing the same reaped-socket error until the deadline. Best-effort:
// a reconnect throw/timeout is swallowed (next tick retries) and must NEVER
// escape this catch — that would re-introduce the unhandledRejection class
// this module was built to close.
if (deps.reconnect) {
const reconnect = deps.reconnect;
try {
await Promise.race([
// Thread the triggering renewLock error (CODEX impl review #2) so the
// engine can classify a CONNECTION_ENDED pooler reap as `reap_detected`.
reconnect({ error: err }),
new Promise<never>((_, reject) => {
deps.setTimeout(
() => reject(new Error(`reconnect timed out after ${state.knobs.callTimeoutMs}ms`)),
state.knobs.callTimeoutMs,
);
}),
]);
} catch { /* reconnect best-effort; next tick retries against a fresh attempt */ }
// [CDX-4] Cadence-aware trigger: run the at-deadline VERIFY when the
// NEXT tick would land past the soft deadline — `>= deadline` alone is
// unreachable under cadence quantization (a 300s lease with a 60s
// cadence has no tick between 240s and expiry; the first eligible tick
// would already be past the lease).
if (sinceLastSuccess + state.intervalMs < deadline) {
// Not near the deadline: retry on the next tick.
// issue #1678 (Codex #2): if the engine can rebuild its pool, do it
// ONCE now (bounded) so the next renewLock sees a live connection.
await attemptReconnectOnce(deps, state, err);
if (state.cancelled()) return { kind: 'cancelled' };
return { kind: 'ok' }; // counter incremented; not yet at deadline
}
return { kind: 'ok' }; // counter incremented; not yet at deadline
// ── At the deadline: VERIFY before evicting (#4145) ──────────────────
// The throw above is NOT evidence of loss (db-lock.ts doctrine:
// fenced-false = certain, throw = transient). Under event-loop
// starvation the renewal may even have LANDED server-side while our
// local race timeout won. renewLock is fenced on lock_token and does
// NOT check lock_until, so an expired-but-unstolen lease renews fine —
// ask the DB the authoritative question.
//
// Reconciling with queue.ts's "renewLock is deliberately not
// retry-wrapped" rationale (two-holders risk): that comment forbids
// BACKGROUND retries that outlive this tick's own timeout race. This
// verify is a synchronous, cancelled()-guarded, callTimeoutMs-bounded
// call INSIDE the tick's flow, and both UPDATEs are same-token
// idempotent lease extensions — a fenced row cannot gain two holders.
let verified: boolean | null = null; // null = the verify itself threw
let verifyErr: unknown = null;
try {
verified = await racedRenewLock(deps, state);
} catch (vErr) {
verifyErr = vErr;
}
if (state.cancelled()) return { kind: 'cancelled' };
if (verified === true) {
// Starved-but-ours: the lease is re-extended; the incident-saving path.
try {
deps.audit.logSuccessAfterFailure(
state.jobId, state.jobName, state.consecutiveFailures, { via: 'verify' },
);
} catch { /* audit best-effort */ }
state.consecutiveFailures = 0;
state.lastSuccessfulRenewalAt = deps.now();
try { deps.onRenewalSuccess?.(); } catch { /* telemetry best-effort */ }
return { kind: 'ok' };
}
if (verified === false) {
// Fenced miss: CERTAIN loss (stall sweep reclaimed / pauseJob).
return { kind: 'lock_lost', cause: 'fenced-lost', via: 'verify' };
}
// The verify was unreachable too — that is a second failed attempt.
// NOTE for audit readers: at the deadline `attempt` advances by 2 per
// tick (primary + verify) — the counter counts ATTEMPTS, not ticks.
state.consecutiveFailures += 1;
const verifyCause = classifyFailure(verifyErr);
// Re-sample load: the bounded verify can consume up to callTimeoutMs,
// and the verify-failure event should carry the load at ITS failure,
// not a snapshot stale by the verify's whole duration.
const verifyLoad = safeLoadSnapshot(deps);
const verifyTelemetry: LockRenewalTelemetryCtx = {
cause: verifyCause,
lateness_ms: latenessMs,
overlap_skips: state.overlapSkips,
...verifyLoad,
};
// Recompute elapsed AFTER the verify: the bounded verify itself can
// consume up to callTimeoutMs, and comparing the stale pre-verify value
// would defer one extra cadence past the advertised backstop.
const sinceLastSuccessAfterVerify = deps.now() - state.lastSuccessfulRenewalAt;
if (sinceLastSuccessAfterVerify >= state.knobs.hardEvictMs) {
// Hard backstop: a LOCAL decision under uncertainty, by design —
// bounds how long non-fenced external side effects run blind during
// a total outage. DB writes stay split-brain-safe via the fence.
try {
deps.audit.logGaveUp(
state.jobId, state.jobName, state.consecutiveFailures, verifyErr ?? err, verifyTelemetry,
);
} catch { /* audit best-effort */ }
return {
kind: 'should_abort',
reason: 'lock-renewal-failed',
cause: verifyCause,
latenessMs,
sinceLastSuccessMs: sinceLastSuccessAfterVerify,
overlapSkips: state.overlapSkips,
...verifyLoad,
};
}
// Deferred past the soft deadline: keep the job and retry next tick —
// the fence is the correctness backstop (a reclaim surfaces as a
// fenced-false on the next attempt; no attempt is burned). CEO-F3:
// give the next tick a live pool.
try {
deps.audit.logFailure(
state.jobId, state.jobName, state.consecutiveFailures, verifyErr ?? err,
{ ...verifyTelemetry, deadline_deferred: true },
);
} catch { /* audit best-effort */ }
await attemptReconnectOnce(deps, state, verifyErr ?? err);
if (state.cancelled()) return { kind: 'cancelled' };
return { kind: 'ok' };
}
if (state.cancelled()) return { kind: 'cancelled' };
@@ -308,8 +573,9 @@ export async function runLockRenewalTick(
// Token-fence failure: another worker reclaimed the row, or pauseJob
// cleared the token. NOT an infrastructure fault — no audit event
// (audit channel is for infrastructure faults only). The worker
// observes `lock_lost` and stderr-warns + aborts.
return { kind: 'lock_lost' };
// observes `lock_lost` and stderr-warns + aborts. This is the ONLY
// CERTAIN loss signal (exactly 0 rows matched the fence).
return { kind: 'lock_lost', cause: 'fenced-lost', via: 'renewal' };
}
if (state.consecutiveFailures > 0) {
@@ -318,10 +584,97 @@ export async function runLockRenewalTick(
state.jobId,
state.jobName,
state.consecutiveFailures,
{ via: 'renewal' },
);
} catch { /* audit best-effort */ }
state.consecutiveFailures = 0;
}
state.lastSuccessfulRenewalAt = deps.now();
// R2-9: let the worker reset its event-loop-delay histogram so the next
// eviction-time sample attributes to the window since THIS success.
try { deps.onRenewalSuccess?.(); } catch { /* telemetry best-effort */ }
return { kind: 'ok' };
}
/**
* CEO-F2: telemetry must never throw into the tick's control flow a
* throwing loadavg probe re-opening the unhandledRejection class would
* be a bitter irony. On throw, the event logs without load fields.
*/
function safeLoadSnapshot(deps: LockRenewalDeps): { load1?: number; cores?: number } {
try {
const s = deps.loadSnapshot?.();
return s ? { load1: s.load1, cores: s.cores } : {};
} catch {
return {};
}
}
/**
* One bounded renewal attempt: renewLock raced against callTimeoutMs.
* When the timeout wins the race it also ABORTS the per-call signal so the
* losing UPDATE releases its pool slot instead of holding it until the
* server finishes (issue #6 an abandoned racer under a saturated pooler
* pinned a checked-out connection for minutes). On the win path the
* late-firing timer aborts an already-settled query, which runUnsafe
* ignores (abort listener removed in its .finally). Cancellation is
* BEST-EFFORT (CDX-2/R2-2 the fence is the correctness authority; PGLite
* ignores the signal and resolves via the race alone). The race attaches
* handlers to both contenders, so a late loser settlement is absorbed,
* never an unhandledRejection.
*/
function racedRenewLock(deps: LockRenewalDeps, state: LockRenewalState): Promise<boolean> {
const controller = new AbortController();
let timer: unknown;
return Promise.race([
deps.renewLock(state.jobId, state.lockToken, state.lockDurationMs, { signal: controller.signal }),
new Promise<never>((_, reject) => {
timer = deps.setTimeout(() => {
try { controller.abort(); } catch { /* best-effort */ }
reject(new RenewalCallTimeoutError('renewLock', state.knobs.callTimeoutMs));
}, state.knobs.callTimeoutMs);
}),
]).finally(() => {
// Win-path hygiene: clear the losing timer so it can't fire a stray
// late abort at a settled query (absorbed by runUnsafe, but noisy).
// Test fakes may return null from the seam; clearTimeout(null) is a
// harmless no-op.
try { clearTimeout(timer as ReturnType<typeof setTimeout>); } catch { /* best-effort */ }
}) as Promise<boolean>;
}
function classifyFailure(err: unknown): RenewalFailureCause {
return err instanceof Error && err.name === 'RenewalCallTimeoutError' ? 'call-timeout' : 'refused';
}
/**
* issue #1678 (Codex #2): bounded, best-effort, ONCE-per-tick pool rebuild
* so the NEXT renewal attempt sees a live connection instead of throwing
* the same reaped-socket error. Threads the triggering error so the engine
* can classify a CONNECTION_ENDED pooler reap as `reap_detected`. A
* reconnect throw/timeout is swallowed it must NEVER escape into the
* tick's catch (the unhandledRejection class this module exists to close).
*/
async function attemptReconnectOnce(
deps: LockRenewalDeps,
state: LockRenewalState,
err: unknown,
): Promise<void> {
if (!deps.reconnect) return;
const reconnect = deps.reconnect;
let timer: unknown;
try {
await Promise.race([
reconnect({ error: err }),
new Promise<never>((_, reject) => {
timer = deps.setTimeout(
() => reject(new Error(`reconnect timed out after ${state.knobs.callTimeoutMs}ms`)),
state.knobs.callTimeoutMs,
);
}),
]);
} catch { /* reconnect best-effort; next tick retries against a fresh attempt */
} finally {
try { clearTimeout(timer as ReturnType<typeof setTimeout>); } catch { /* best-effort */ }
}
}
+111 -14
View File
@@ -16,7 +16,10 @@ import type {
import { rowToMinionJob, rowToInboxMessage, rowToAttachment } from './types.ts';
import { validateAttachment } from './attachments.ts';
import { isProtectedJobName } from './protected-names.ts';
import { defaultTimeoutMsFor, HANDLER_DEFAULT_TIMEOUT_MS } from './handler-timeouts.ts';
import {
defaultTimeoutMsFor, HANDLER_DEFAULT_TIMEOUT_MS,
defaultLockDurationMsFor, HANDLER_DEFAULT_LOCK_DURATION_MS, clampLockDurationMs,
} from './handler-timeouts.ts';
import {
withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay,
isRetryableConnError,
@@ -38,6 +41,62 @@ export interface TrustedSubmitOpts {
const MIGRATION_VERSION = 7;
const DEFAULT_MAX_SPAWN_DEPTH = 5;
/**
* Stall-sweep reclaim grace (#4145, CDX-7): don't reclaim a row whose
* `lock_until` lapsed within the last N ms. When a CPU-starved worker's
* event loop unblocks, its coalesced renewal tick and the stall sweep
* fire in the same burst if the sweep's UPDATE lands first it steals
* the OWNER'S live job. The grace is a HEAD-START for the owner's
* recovery renewal, not a guarantee: it only covers starvation bursts
* shorter than the grace, and a healthy second worker's sweep still
* wins beyond it. Minion analog of `GBRAIN_LOCK_STEAL_GRACE_SECONDS`
* (db-lock.ts), adapted because minion_jobs has no last_refreshed_at.
*
* Cost: dead-worker recovery becomes lock_until + grace + up to
* stalledInterval. Env `GBRAIN_MINION_STALL_RECLAIM_GRACE_MS` (0 allowed
* restores the exact legacy reclaim predicate).
*/
export const DEFAULT_STALL_RECLAIM_GRACE_MS = 15_000;
const _warnedGraceEnv = new Set<string>();
export function _resetStallGraceWarningsForTests(): void {
_warnedGraceEnv.clear();
}
export function resolveStallReclaimGraceMs(
env: Record<string, string | undefined> = process.env,
): number {
const raw = env.GBRAIN_MINION_STALL_RECLAIM_GRACE_MS;
if (raw === undefined || raw.trim() === '') return DEFAULT_STALL_RECLAIM_GRACE_MS;
// Unlike the lock-renewal knobs, 0 is a VALID value here (legacy reclaim).
if (!/^\d+$/.test(raw.trim())) {
if (!_warnedGraceEnv.has(raw)) {
_warnedGraceEnv.add(raw);
process.stderr.write(
`[minions] env GBRAIN_MINION_STALL_RECLAIM_GRACE_MS=${JSON.stringify(raw)} is not a non-negative integer; ` +
`falling back to default ${DEFAULT_STALL_RECLAIM_GRACE_MS}\n`,
);
}
return DEFAULT_STALL_RECLAIM_GRACE_MS;
}
const n = Number(raw.trim());
// Cap at 10 minutes: an absurd digit string (Number → huge/Infinity)
// would otherwise push the sweep cutoff to -infinity and silently
// disable stalled-job recovery altogether.
const MAX_GRACE_MS = 600_000;
if (n > MAX_GRACE_MS) {
if (!_warnedGraceEnv.has(raw)) {
_warnedGraceEnv.add(raw);
process.stderr.write(
`[minions] env GBRAIN_MINION_STALL_RECLAIM_GRACE_MS=${JSON.stringify(raw)} exceeds the ${MAX_GRACE_MS}ms cap; clamping\n`,
);
}
return MAX_GRACE_MS;
}
return n;
}
const DEFAULT_MAX_ATTACHMENT_BYTES = 5 * 1024 * 1024; // 5 MiB
const TERMINAL_STATUSES = ['completed', 'failed', 'dead', 'cancelled'] as const;
@@ -354,11 +413,11 @@ export class MinionQueue {
const baseCols = `name, queue, status, priority, data, max_attempts, backoff_type,
backoff_delay, backoff_jitter, delay_until, parent_job_id, on_child_fail,
depth, max_children, timeout_ms, remove_on_complete, remove_on_fail, idempotency_key,
depth, max_children, timeout_ms, lock_duration_ms, remove_on_complete, remove_on_fail, idempotency_key,
quiet_hours, stagger_key`;
const baseVals = `$1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19::jsonb, $20`;
const baseVals = `$1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20::jsonb, $21`;
const cols = hasMaxStalled ? `${baseCols}, max_stalled` : baseCols;
const vals = hasMaxStalled ? `${baseVals}, $21` : baseVals;
const vals = hasMaxStalled ? `${baseVals}, $22` : baseVals;
const insertSql = opts?.idempotency_key
? `INSERT INTO minion_jobs (${cols})
@@ -388,6 +447,14 @@ export class MinionQueue {
// sane long wall-clock default stamped at submit when the caller didn't
// pass one, so they aren't killed mid-progress by the short null-default.
opts?.timeout_ms ?? defaultTimeoutMsFor(jobName),
// #4145: same three-layer pattern for the lock lease. Explicit input
// is clamped to [5s,1h]; absent → handler map default; NULL row =
// worker-global lockDuration at claim. INSERT-only (see the
// max_stalled footgun note above): an idempotency-key re-submit
// never mutates the first submitter's lease.
opts?.lock_duration_ms != null
? clampLockDurationMs(opts.lock_duration_ms)
: defaultLockDurationMsFor(jobName),
opts?.remove_on_complete ?? false,
opts?.remove_on_fail ?? false,
opts?.idempotency_key ?? null,
@@ -772,11 +839,26 @@ export class MinionQueue {
// Direct (session-mode) pool: claim opens the lock that renewLock then
// heartbeats. Both must live on a connection the transaction-mode pooler
// won't recycle mid-hold, or the lock orphans and the worker wedges.
//
// #4145: lock_duration_ms resolves row → handler map ($6, RAW object —
// same double-encode rule as $5) → worker default ($2), is STAMPED onto
// the row (durable, like timeout_ms), and lock_until derives from the
// same COALESCE (OLD-row semantics: repeat the expression, don't
// reference the assigned column). Both the stamp and lock_until are
// CASE-clamped to the [5s,1h] bound IN SQL (row/map resolution only —
// the worker-default fallback $2 is operator-configured, not row data,
// and tests/short-lived workers legitimately use sub-5s leases): the exposed
// submit surfaces clamp already, but a bypass-written row (direct SQL
// repair, foreign tooling) must not grant a ~24-day lease to a worker
// that crashes before its first renewal (or a 1ms one that thrashes).
const rows = await this.engine.executeRawDirect<Record<string, unknown>>(
`UPDATE minion_jobs SET
status = 'active',
lock_token = $1,
lock_until = now() + ($2::double precision * interval '1 millisecond'),
lock_until = now() + ((CASE WHEN COALESCE(lock_duration_ms, ($6::jsonb ->> name)::int) IS NULL THEN $2
ELSE LEAST(GREATEST(COALESCE(lock_duration_ms, ($6::jsonb ->> name)::int), 5000), 3600000) END)::double precision * interval '1 millisecond'),
lock_duration_ms = CASE WHEN COALESCE(lock_duration_ms, ($6::jsonb ->> name)::int) IS NULL THEN NULL
ELSE LEAST(GREATEST(COALESCE(lock_duration_ms, ($6::jsonb ->> name)::int), 5000), 3600000) END,
timeout_ms = COALESCE(timeout_ms, ($5::jsonb ->> name)::int),
timeout_at = CASE WHEN COALESCE(timeout_ms, ($5::jsonb ->> name)::int) IS NOT NULL
THEN now() + (COALESCE(timeout_ms, ($5::jsonb ->> name)::int)::double precision * interval '1 millisecond')
@@ -792,7 +874,7 @@ export class MinionQueue {
LIMIT 1
)
RETURNING *`,
[lockToken, lockDurationMs, queue, registeredNames, HANDLER_DEFAULT_TIMEOUT_MS]
[lockToken, lockDurationMs, queue, registeredNames, HANDLER_DEFAULT_TIMEOUT_MS, HANDLER_DEFAULT_LOCK_DURATION_MS]
);
return rows.length > 0 ? rowToMinionJob(rows[0]) : null;
}
@@ -959,7 +1041,7 @@ export class MinionQueue {
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
CASE
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
ELSE COALESCE(lock_duration_ms, $1)::double precision * 2 * GREATEST(max_stalled, 1)
END`,
[lockDurationMs]
);
@@ -982,7 +1064,7 @@ export class MinionQueue {
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
CASE
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
ELSE COALESCE(lock_duration_ms, $1)::double precision * 2 * GREATEST(max_stalled, 1)
END
FOR UPDATE SKIP LOCKED
)
@@ -1297,9 +1379,15 @@ export class MinionQueue {
/**
* Renew lock (token-fenced). Returns false if token mismatch (job was reclaimed).
*
* `opts.signal` cancels the in-flight UPDATE (postgres.js `.cancel()`) when the
* caller's timeout race gives up on it otherwise the abandoned query holds a
* checked-out pool slot for its full server-side duration (issue #6).
* Cancellation is BEST-EFFORT (#4145 CDX-2/R2-2): pool acquisition and PG
* protocol cancel are asynchronous, and PGLite ignores the signal so
* correctness never rests on it. A late-landing renewal UPDATE is fenced on
* OUR token, meaning it can only extend a lock nobody else has claimed;
* worst case is a stall-requeue delayed by one lease.
*/
async renewLock(
id: number,
@@ -1370,18 +1458,25 @@ export class MinionQueue {
}
/** Detect and handle stalled jobs. Single CTE, no off-by-one. Returns affected jobs. */
async handleStalled(): Promise<{ requeued: MinionJob[]; dead: MinionJob[] }> {
async handleStalled(graceMsOverride?: number): Promise<{ requeued: MinionJob[]; dead: MinionJob[] }> {
// W0 fix-wave (Tier-1 #4): the dead-letter branch previously emitted NO
// child_done and never unblocked aggregator parents — a child that died
// via max-stall stranded its parent in 'waiting-children' forever (the
// exact hang the v0.15 comment says was fixed for timeouts; there was no
// compensating sweep anywhere). Restructured into the parents-first
// discover/lock/kill shape (D5.12) with the shared killJobs() tail.
//
// #4145 (CDX-7): the reclaim predicate carries a grace — see
// resolveStallReclaimGraceMs. Callers (tests) may pass an explicit
// override; the worker sweep resolves from env/default.
const graceMs = graceMsOverride ?? resolveStallReclaimGraceMs();
return this.engine.transaction(async (tx) => {
const candidates = await tx.executeRaw<{ id: number; parent_job_id: number | null; stalled_counter: number; max_stalled: number }>(
`SELECT id, parent_job_id, stalled_counter, max_stalled
FROM minion_jobs
WHERE status = 'active' AND lock_until < now()`
WHERE status = 'active'
AND lock_until < now() - ($1::double precision * interval '1 millisecond')`,
[graceMs]
);
if (candidates.length === 0) return { requeued: [], dead: [] };
const ids = candidates.map(c => c.id);
@@ -1399,12 +1494,13 @@ export class MinionQueue {
WHERE id IN (
SELECT id FROM minion_jobs
WHERE id = ANY($1::bigint[])
AND status = 'active' AND lock_until < now()
AND status = 'active'
AND lock_until < now() - ($2::double precision * interval '1 millisecond')
AND stalled_counter + 1 < max_stalled
FOR UPDATE SKIP LOCKED
)
RETURNING *`,
[ids]
[ids, graceMs]
);
const deadRows = await tx.executeRaw<Record<string, unknown>>(
`UPDATE minion_jobs SET
@@ -1415,12 +1511,13 @@ export class MinionQueue {
WHERE id IN (
SELECT id FROM minion_jobs
WHERE id = ANY($1::bigint[])
AND status = 'active' AND lock_until < now()
AND status = 'active'
AND lock_until < now() - ($2::double precision * interval '1 millisecond')
AND stalled_counter + 1 >= max_stalled
FOR UPDATE SKIP LOCKED
)
RETURNING *`,
[ids]
[ids, graceMs]
);
// THE FIX: stall-death now notifies + unblocks parents like every
// other terminal kill. Outcome 'dead' (not 'timeout') so consumers can
+5
View File
@@ -71,6 +71,8 @@ export interface MinionJob {
max_children: number | null;
timeout_ms: number | null;
timeout_at: Date | null;
/** Per-job lock lease (ms, #4145). NULL = worker-global lockDuration default. */
lock_duration_ms: number | null;
remove_on_complete: boolean;
remove_on_fail: boolean;
idempotency_key: string | null;
@@ -128,6 +130,8 @@ export interface MinionJobInput {
max_children?: number;
/** Wall-clock per-job deadline in ms. Set on claim → timeout_at. Terminal on expire (no retry). */
timeout_ms?: number;
/** Per-job lock lease in ms (#4145). Clamped to [5s,1h]; NULL/undefined → handler map, then worker default. INSERT-only: an idempotency-key re-submit never mutates the first submitter's lease. */
lock_duration_ms?: number;
/** DELETE row on successful completion (after token rollup + child_done insert). */
remove_on_complete?: boolean;
/** DELETE row on terminal failure (after parent failure hook). */
@@ -435,6 +439,7 @@ export function rowToMinionJob(row: Record<string, unknown>): MinionJob {
depth: (row.depth as number) ?? 0,
max_children: (row.max_children as number) ?? null,
timeout_ms: (row.timeout_ms as number) ?? null,
lock_duration_ms: (row.lock_duration_ms as number) ?? null,
timeout_at: row.timeout_at ? new Date(row.timeout_at as string) : null,
remove_on_complete: row.remove_on_complete === true,
remove_on_fail: row.remove_on_fail === true,
+164 -16
View File
@@ -30,9 +30,12 @@ import { logLeasePressure } from './lease-pressure-audit.ts';
import {
runLockRenewalTick,
resolveLockRenewalKnobs,
renewalIntervalFor,
type LockRenewalDeps,
type LockRenewalState,
type TickResult,
} from './lock-renewal-tick.ts';
import { clampLockDurationMs } from './handler-timeouts.ts';
import {
runDbProbe,
getConnectionRouting,
@@ -48,6 +51,8 @@ import {
ChildNotClaimedError,
} from './child-job-runner.ts';
import { lockRenewalAudit } from '../audit/lock-renewal-audit.ts';
import { loadavg, cpus } from 'os';
import { monitorEventLoopDelay } from 'perf_hooks';
import { isRetryableConnError } from '../retry-matcher.ts';
import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from './reconnect.ts';
@@ -230,6 +235,23 @@ export class MinionWorker extends EventEmitter {
private opts: Required<MinionWorkerOpts>;
/**
* Event-loop-delay histogram (CDX-1/R2-9, issue #4145): the DIRECT
* measurement of local starvation, sampled at eviction time and RESET
* on every successful lock renewal so a sample attributes to the
* window since the last success not process lifetime. The histogram
* is deliberately WORKER-global, not per-job: the event loop is one
* shared resource, and ANY job's successful renewal proves the loop was
* healthy enough to process a round-trip at that moment a legitimate
* truncation of the starvation window even for a sibling job that
* evicts moments later (its per-job discriminator is tick lateness,
* which IS per-job). Null when the runtime doesn't ship
* `monitorEventLoopDelay` (fail-open: eviction logs omit eld fields).
*/
private eldHistogram: ReturnType<typeof monitorEventLoopDelay> | null = null;
/** Core count cached once — pairs with raw loadavg in eviction telemetry. */
private readonly cpuCores: number;
constructor(
private engine: BrainEngine,
opts?: MinionWorkerOpts & MinionQueueOpts,
@@ -239,6 +261,17 @@ export class MinionWorker extends EventEmitter {
maxSpawnDepth: opts?.maxSpawnDepth,
maxAttachmentBytes: opts?.maxAttachmentBytes,
});
let cores = 0;
try { cores = cpus().length; } catch { /* telemetry best-effort */ }
this.cpuCores = cores;
try {
if (typeof monitorEventLoopDelay === 'function') {
// Created DISABLED: start() enables and stop() disables, so a
// constructed-but-never-started worker (setup failures, probe
// instances) never holds a native sampling timer.
this.eldHistogram = monitorEventLoopDelay({ resolution: 20 });
}
} catch { this.eldHistogram = null; /* fail-open */ }
this.opts = {
queue: opts?.queue ?? 'default',
concurrency: opts?.concurrency ?? 1,
@@ -346,6 +379,10 @@ export class MinionWorker extends EventEmitter {
await this.queue.ensureSchema();
this.running = true;
// R2-9 lifecycle: (re-)enable the event-loop-delay histogram for this
// run; stop() disables it so embedding hosts / test suites that cycle
// start()/stop() don't leak a ~50Hz native sampling timer per instance.
try { this.eldHistogram?.enable(); } catch { /* fail-open */ }
// Graceful shutdown. Fires shutdownAbort so handlers subscribed to
// `ctx.shutdownSignal` (currently: shell handler) can run their own cleanup
@@ -805,6 +842,7 @@ export class MinionWorker extends EventEmitter {
/** Stop the worker gracefully. */
stop(): void {
this.running = false;
try { this.eldHistogram?.disable(); } catch { /* fail-open */ }
}
/**
@@ -923,6 +961,40 @@ export class MinionWorker extends EventEmitter {
* `failJob` throwing during the same DB outage) can't propagate to
* the process-level handler and crash the daemon.
*/
/**
* One formatter for the classified abort telemetry (R2-7) so the
* should_abort warn and the 30s-later grace-evict line can never drift
* apart (they briefly did: `load1:` vs `load1_at_abort:`).
*/
private formatAbortMeta(meta: Extract<TickResult, { kind: 'lock_lost' | 'should_abort' }>): string {
if (meta.kind === 'lock_lost') {
return `cause: ${meta.cause}, via: ${meta.via}`;
}
return `cause: ${meta.cause}, since_last_success_ms: ${Math.round(meta.sinceLastSuccessMs)}, ` +
`tick_lateness_ms: ${Math.round(meta.latenessMs)}, overlap_skips: ${meta.overlapSkips}` +
`${meta.load1 !== undefined ? `, load1: ${meta.load1.toFixed(2)}/${meta.cores} cores` : ''}`;
}
/**
* Event-loop-delay sample for eviction log lines (CDX-1). The histogram
* resets on every successful renewal (R2-9 via deps.onRenewalSuccess),
* so these numbers attribute to the window since the last success
* i.e. exactly the window in which renewal was failing. nsms. Empty
* string when the runtime lacks the histogram or sampling throws
* (fail-open: never let telemetry break the eviction path).
*/
private formatEvictionTelemetry(): string {
const h = this.eldHistogram;
if (h === null) return '';
try {
const p99Ms = Math.round(h.percentile(99) / 1e6);
const maxMs = Math.round(h.max / 1e6);
return ` [event_loop_delay since last renewal: p99 ${p99Ms}ms, max ${maxMs}ms]`;
} catch {
return '';
}
}
private launchJob(job: MinionJob, lockToken: string): void {
const abort = new AbortController();
@@ -930,18 +1002,46 @@ export class MinionWorker extends EventEmitter {
let cancelled = false;
// --- re-entrancy guard for overlapping ticks during PgBouncer stalls ---
let tickInFlight = false;
// --- R2-7: the tick's final result, stashed at abort time so the ---
// --- grace-evict log (which fires 30s LATER) can report cause/ ---
// --- lateness/load instead of just the Error string. ---
let abortMeta: Extract<TickResult, { kind: 'lock_lost' | 'should_abort' }> | null = null;
// R2-4: ALL elapsed-time arithmetic in the renewal state machine runs
// on a monotonic clock — a wall-clock jump (NTP step, DST bug) must
// never evict or indefinitely defer. Date.now stays only in log/audit
// timestamps (the audit writer stamps its own `ts`).
const monotonicNow = () => performance.now();
// --- D3: pure-function lock renewal ---
const knobs = resolveLockRenewalKnobs(process.env, this.opts.lockDuration);
// #4145: the EFFECTIVE lease is per-job (claim stamped it from the
// handler map / explicit submit; NULL = worker default). The cadence
// clamps to 60s so a 300s lease renews 5x per window (matching the
// cycle refresher's multiple-chances philosophy) instead of the bare
// lease/2 = every 150s; leases ≤120s keep the legacy /2 exactly.
// Defense-in-depth: re-clamp the row value at consumption. The exposed
// submit surfaces already clamp, but the claim COALESCE trusts the row
// and the DB CHECK only enforces > 0 — a writer that bypasses add()
// (direct SQL repair, foreign tooling) could otherwise stamp a 1ms
// lease (renewal-storm setInterval) or a ~25-day one (weeks-long
// dead-worker pin).
const effectiveLockMs = job.lock_duration_ms != null
? clampLockDurationMs(job.lock_duration_ms)
: this.opts.lockDuration;
const renewalIntervalMs = renewalIntervalFor(effectiveLockMs);
const knobs = resolveLockRenewalKnobs(process.env, effectiveLockMs, renewalIntervalMs);
const renewalState: LockRenewalState = {
jobId: job.id,
jobName: job.name,
lockToken,
lockDurationMs: this.opts.lockDuration,
lockDurationMs: effectiveLockMs,
knobs,
lastSuccessfulRenewalAt: Date.now(),
lastSuccessfulRenewalAt: monotonicNow(),
consecutiveFailures: 0,
cancelled: () => cancelled,
intervalMs: renewalIntervalMs,
lastTickFiredAt: monotonicNow(),
overlapSkips: 0,
};
// issue #1678 (Codex #2): hand the tick a bounded reconnect-once hook when
// the engine owns a pool that a transaction-mode pooler can reap. Postgres
@@ -951,15 +1051,30 @@ export class MinionWorker extends EventEmitter {
const renewalDeps: LockRenewalDeps = {
renewLock: (id, tok, dur, opts) => this.queue.renewLock(id, tok, dur, opts),
audit: lockRenewalAudit,
now: Date.now,
// R2-4: monotonic — see monotonicNow above.
now: monotonicNow,
setTimeout: (cb, ms) => globalThis.setTimeout(cb, ms),
// Forward the tick's classified error (CODEX impl review #2) so a pooler
// reap during lock renewal is audited as reap_detected, not reconnect_other.
...(engineReconnect ? { reconnect: (ctx?: { error?: unknown }) => engineReconnect.call(this.engine, ctx) } : {}),
// Issue #4145 telemetry: raw loadavg + cached cores (CEO-F2: the tick
// try/catches every call), and the R2-9 histogram reset-on-success.
loadSnapshot: () => ({ load1: loadavg()[0], cores: this.cpuCores }),
onRenewalSuccess: () => { try { this.eldHistogram?.reset(); } catch { /* fail-open */ } },
};
const lockTimer = setInterval(() => {
if (tickInFlight) return;
if (tickInFlight) {
// Overlap skip (CDX-13): a prior tick is still awaiting its
// renewal call. Count it — this is NOT a missed interval (those
// coalesce and show up as tick LATENESS instead) — and ADVANCE the
// lateness baseline: this callback fired on schedule, so the next
// executed tick must not book the skipped window as event-loop
// lateness (that would misclassify a slow DB call as starvation).
renewalState.overlapSkips += 1;
renewalState.lastTickFiredAt = monotonicNow();
return;
}
tickInFlight = true;
void runLockRenewalTick(renewalDeps, renewalState)
.then((result) => {
@@ -970,13 +1085,24 @@ export class MinionWorker extends EventEmitter {
return;
case 'lock_lost':
if (!abort.signal.aborted) {
console.warn(`Lock lost for job ${job.id}, aborting execution`);
abortMeta = result;
console.warn(
`Lock lost for job ${job.id}, aborting execution ` +
`(${this.formatAbortMeta(result)}${this.formatEvictionTelemetry()})`,
);
clearInterval(lockTimer);
abort.abort(new Error('lock-lost'));
}
return;
case 'should_abort':
if (!abort.signal.aborted) {
abortMeta = result;
// Issue #4145 request 3: the one line that saves the 8h of
// forensics — WHY renewal failed + was the loop starved.
console.warn(
`Lock renewal failed for job ${job.id} (${job.name}); aborting ` +
`(${this.formatAbortMeta(result)}${this.formatEvictionTelemetry()})`,
);
clearInterval(lockTimer);
abort.abort(new Error(result.reason));
}
@@ -996,7 +1122,7 @@ export class MinionWorker extends EventEmitter {
.finally(() => {
tickInFlight = false;
});
}, this.opts.lockDuration / 2);
}, renewalIntervalMs);
// --- D8b: universal grace-eviction timer ---
// Fires for ANY abort reason (not just job.timeout_ms). Without
@@ -1012,18 +1138,35 @@ export class MinionWorker extends EventEmitter {
const reason = abort.signal.reason instanceof Error
? abort.signal.reason.message
: String(abort.signal.reason);
// R2-7: abortMeta carries the tick's classified cause + starvation
// telemetry captured AT abort time — the Error string alone would
// make this line read like an orphan leak (the #4145 forensics trap).
const meta = abortMeta === null ? '' : ` (${this.formatAbortMeta(abortMeta)})`;
console.warn(
`Job ${job.id} (${job.name}) did not exit within 30s of abort (reason: ${reason}). ` +
`Job ${job.id} (${job.name}) did not exit within 30s of abort (reason: ${reason}).${meta} ` +
`Force-evicting from inFlight to unblock worker. ` +
`The handler is still running but the worker will claim new jobs.`
`The handler is still running but the worker will claim new jobs.` +
this.formatEvictionTelemetry()
);
clearInterval(lockTimer);
this.inFlight.delete(job.id);
// D8a: don't failJob on infrastructure aborts (stall detector
// reclaims after lock expiry). Isolation mode: also skip — the
// group SIGKILL already fired and executeJob's own recording
// follows; a competing evict failJob('dead') could dead-letter a
// job with attempts remaining (adversarial-review P3).
// R2-1 generation-safety: delete ONLY our own execution's entry.
// After a force-evict, this job id can be requeued and re-claimed
// by THIS worker while the old handler is still alive — an
// unconditional delete-by-id would then remove the NEW
// execution's entry (concurrency undercount, lost tracking).
// The lockToken is minted per claim, so it is the generation.
if (this.inFlight.get(job.id)?.lockToken === lockToken) {
this.inFlight.delete(job.id);
}
// D8a: don't failJob if the abort was infrastructure. The stall
// detector will reclaim the row cleanly: a lock-renewal abort
// now fires only after the at-deadline VERIFY either returned
// fenced-false (row already reclaimed) or stayed unreachable
// past hardEvictMs (lease long expired) — see #4145
// verify-before-evict in lock-renewal-tick.ts. Isolation mode:
// also skip — the group SIGKILL already fired and executeJob's
// own recording follows; a competing evict failJob('dead') could
// dead-letter a job with attempts remaining (adversarial-review P3).
if (!INFRASTRUCTURE_ABORT_REASONS.has(reason) && this.opts.jobIsolation !== 'process') {
this.queue.failJob(
job.id,
@@ -1066,7 +1209,12 @@ export class MinionWorker extends EventEmitter {
clearInterval(lockTimer);
if (timeoutTimer) clearTimeout(timeoutTimer);
if (graceTimer) clearTimeout(graceTimer);
this.inFlight.delete(job.id);
// R2-1 generation-safety: a force-evicted execution's finally can
// fire long after the same job id was re-claimed by this worker.
// Only delete the entry if it is still OURS (token = generation).
if (this.inFlight.get(job.id)?.lockToken === lockToken) {
this.inFlight.delete(job.id);
}
this.jobsCompleted += 1;
this.checkMemoryLimit('post-job');
})
+5
View File
@@ -3588,6 +3588,7 @@ const submit_job: Operation = {
max_attempts: { type: 'number', description: 'Max retry attempts (default: 3)' },
delay: { type: 'number', description: 'Delay in ms before eligible' },
timeout_ms: { type: 'number', description: 'Per-job wall-clock timeout in ms; aborted job goes to dead' },
lock_duration_ms: { type: 'number', description: 'Per-job lock lease in ms (#4145). Out-of-range values are clamped to [5000, 3600000] — remote writers cannot pin an immortal lock. Omit to use the handler-type default (300s for long LLM handlers) or the worker default (30s).' },
},
mutating: true,
scope: 'admin',
@@ -3635,6 +3636,10 @@ const submit_job: Operation = {
max_attempts: (p.max_attempts as number) || 3,
delay: (p.delay as number) || undefined,
timeout_ms: (p.timeout_ms as number) || undefined,
// #4145 [CEO-F7/R2-6]: range enforcement lives in queue.add's
// clampLockDurationMs (ParamDef has no min/max support; wrong TYPE is
// rejected by the shared number validation upstream of this handler).
lock_duration_ms: (p.lock_duration_ms as number) || undefined,
}, trusted);
// v0.35.8.0: submit_job audit-log parity with the CLI path (codex F-CDX-4).
+97 -34
View File
@@ -117,35 +117,85 @@ const PGLITE_EDGE_BATCH_MAX_BIND_PARAMS = 30_000;
// silently fall through to a normal initSchema (snapshot is just an
// optimization, never authoritative).
let _snapshotWarnLogged = false;
// Per-process memo. MIGRATIONS + PGLITE_SCHEMA_SQL are static for the life of
// the process, so the schema hash is too; the version file and the ~42MB tar
// are read once per (path, process) instead of once per engine construction
// (a full suite constructs 600+ engines — the un-memoized loader re-read the
// tar and re-hashed 131 migration handler sources every time, ~84MB of
// transient allocation per call). A null entry means the path is terminally
// unusable this process (missing/stale/torn) — no retry per construction.
// The dims/model shape gate is deliberately NOT memoized: tests reconfigure
// the gateway mid-process (zembed/1280) and a mismatched engine must fall
// back to cold init even when an earlier engine loaded this same snapshot.
// Accepted limitation: a snapshot file rewritten mid-process is not observed;
// the only writer (build-pglite-snapshot.ts) runs before test fan-out.
let _snapshotSchemaHashMemo: string | null = null;
// blob stays null until the FIRST caller whose shape gate passes — a process
// whose gateway shape never matches the snapshot (the zembed/1280 test
// files) never pays the 42MB tar read at all.
const _snapshotFileMemo = new Map<string, { versionLines: string[]; blob: Blob | null } | null>();
let _snapshotTarReads = 0;
export function __snapshotMemoStatsForTests(): { tarReads: number; memoEntries: number } {
return { tarReads: _snapshotTarReads, memoEntries: _snapshotFileMemo.size };
}
export function __resetSnapshotMemoForTests(): void {
_snapshotSchemaHashMemo = null;
_snapshotFileMemo.clear();
_snapshotTarReads = 0;
_snapshotWarnLogged = false;
}
export function tryLoadSnapshot(snapshotPath: string): Blob | null {
try {
// Lazy require so production builds without these imports don't crash.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const crypto = require('node:crypto') as typeof import('node:crypto'); // engine-dynamic-import-ok
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts'); // engine-dynamic-import-ok
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts'); // engine-dynamic-import-ok
let entry = _snapshotFileMemo.get(snapshotPath);
if (entry === null) return null; // terminally unusable this process
if (entry === undefined) {
// First touch of this path in this process — do the file work once.
// Lazy require so production builds without these imports don't crash.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const crypto = require('node:crypto') as typeof import('node:crypto'); // engine-dynamic-import-ok
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts'); // engine-dynamic-import-ok
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts'); // engine-dynamic-import-ok
if (!fs.existsSync(snapshotPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] GBRAIN_PGLITE_SNAPSHOT set but file missing: ${snapshotPath} — using normal init.`);
_snapshotWarnLogged = true;
if (!fs.existsSync(snapshotPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] GBRAIN_PGLITE_SNAPSHOT set but file missing: ${snapshotPath} — using normal init.`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
return null;
}
const versionPath = snapshotPath.replace(/\.tar(?:\.gz)?$/, '.version');
if (!fs.existsSync(versionPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot version file missing: ${versionPath} — using normal init.`);
_snapshotWarnLogged = true;
const versionPath = snapshotPath.replace(/\.tar(?:\.gz)?$/, '.version');
if (!fs.existsSync(versionPath)) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot version file missing: ${versionPath} — using normal init.`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
return null;
if (_snapshotSchemaHashMemo === null) {
_snapshotSchemaHashMemo = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
}
const versionLines = fs.readFileSync(versionPath, 'utf8').trim().split('\n');
if (_snapshotSchemaHashMemo !== (versionLines[0] ?? '')) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot stale (schema hash mismatch) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
entry = { versionLines, blob: null };
_snapshotFileMemo.set(snapshotPath, entry);
}
const expectedHash = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
const versionLines = fs.readFileSync(versionPath, 'utf8').trim().split('\n');
const actualHash = versionLines[0] ?? '';
// W0 fix-wave: the version file's dims=/model= lines record the embedding
// shape the snapshot was BAKED with. A snapshot whose vector(dims) columns
@@ -154,6 +204,8 @@ export function tryLoadSnapshot(snapshotPath: string): Blob | null {
// fixture went default-on). Resolve our would-be shape through the same
// gateway-with-default fallback initSchema uses and refuse a mismatch.
// Version files without the shape lines (pre-W0) are treated as stale.
// Re-evaluated on EVERY call against the CURRENT gateway config — never
// memoized (see memo comment above).
let wantDims: number | string = DEFAULT_EMBEDDING_DIMENSIONS;
let wantModel: string = DEFAULT_EMBEDDING_MODEL;
try {
@@ -161,25 +213,30 @@ export function tryLoadSnapshot(snapshotPath: string): Blob | null {
wantDims = gw.getEmbeddingDimensions();
wantModel = gw.getEmbeddingModel();
} catch { /* gateway not configured — defaults, same as initSchema */ }
const shapeOk = versionLines[1] === `dims=${wantDims}` && versionLines[2] === `model=${wantModel}`;
const shapeOk = entry.versionLines[1] === `dims=${wantDims}` && entry.versionLines[2] === `model=${wantModel}`;
if (!shapeOk) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot embedding shape mismatch (want dims=${wantDims} model=${wantModel}, have ${versionLines[1] ?? 'none'} ${versionLines[2] ?? ''}) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
console.warn(`[pglite] snapshot embedding shape mismatch (want dims=${wantDims} model=${wantModel}, have ${entry.versionLines[1] ?? 'none'} ${entry.versionLines[2] ?? ''}) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
}
return null;
}
if (expectedHash !== actualHash) {
if (!_snapshotWarnLogged) {
// eslint-disable-next-line no-console
console.warn(`[pglite] snapshot stale (schema hash mismatch) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
_snapshotWarnLogged = true;
if (entry.blob === null) {
// Tar read deferred until the first shape-matching caller (see memo
// comment above). A torn/unreadable tar is terminal for the process.
try {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
const buf = fs.readFileSync(snapshotPath);
_snapshotTarReads += 1;
entry.blob = new Blob([new Uint8Array(buf.buffer as ArrayBuffer, buf.byteOffset, buf.byteLength)]);
} catch {
_snapshotFileMemo.set(snapshotPath, null);
return null;
}
return null;
}
const buf = fs.readFileSync(snapshotPath);
return new Blob([buf]);
return entry.blob;
} catch {
// Any failure -> fall through to normal init. Never block tests.
return null;
@@ -3856,8 +3913,14 @@ export class PGLiteEngine implements BrainEngine {
COUNT(DISTINCT n.last_link_type) AS edge_count,
array_agg(DISTINCT n.last_link_type)
FILTER (WHERE n.last_link_type IS NOT NULL) AS via_link_types,
-- Final path tie-break (lexicographic) makes the pick deterministic
-- when a node is reachable at the same depth from multiple seeds;
-- without it the winner is plan/heap-order dependent and the two
-- engines (or two runs) can disagree. Relational retrieval is
-- documented deterministic; keep in lockstep with postgres-engine.ts.
(array_agg(array_to_string(n.path, chr(9))
ORDER BY n.depth ASC, array_length(n.path, 1) ASC))[1] AS path_str,
ORDER BY n.depth ASC, array_length(n.path, 1) ASC,
array_to_string(n.path, chr(9)) ASC))[1] AS path_str,
(SELECT cc.id FROM content_chunks cc
WHERE cc.page_id = n.id ORDER BY cc.chunk_index ASC LIMIT 1) AS canonical_chunk_id
FROM walk n
+3 -1
View File
@@ -462,6 +462,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
depth INTEGER NOT NULL DEFAULT 0,
max_children INTEGER,
timeout_ms INTEGER,
lock_duration_ms INTEGER,
timeout_at TIMESTAMPTZ,
remove_on_complete BOOLEAN NOT NULL DEFAULT FALSE,
remove_on_fail BOOLEAN NOT NULL DEFAULT FALSE,
@@ -482,7 +483,8 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
CONSTRAINT chk_nonnegative CHECK (attempts_made >= 0 AND attempts_started >= 0 AND stalled_counter >= 0 AND max_attempts >= 1 AND max_stalled >= 0),
CONSTRAINT chk_depth_nonnegative CHECK (depth >= 0),
CONSTRAINT chk_max_children_positive CHECK (max_children IS NULL OR max_children > 0),
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0)
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0),
CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000))
);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_claim ON minion_jobs (queue, priority ASC, created_at ASC) WHERE status = 'waiting';
+23 -11
View File
@@ -3785,8 +3785,14 @@ export class PostgresEngine implements BrainEngine {
COUNT(DISTINCT n.last_link_type) AS edge_count,
array_agg(DISTINCT n.last_link_type)
FILTER (WHERE n.last_link_type IS NOT NULL) AS via_link_types,
-- Final path tie-break (lexicographic) makes the pick deterministic
-- when a node is reachable at the same depth from multiple seeds;
-- without it the winner is plan/heap-order dependent and the two
-- engines (or two runs) can disagree. Relational retrieval is
-- documented deterministic; keep in lockstep with pglite-engine.ts.
(array_agg(array_to_string(n.path, chr(9))
ORDER BY n.depth ASC, array_length(n.path, 1) ASC))[1] AS path_str,
ORDER BY n.depth ASC, array_length(n.path, 1) ASC,
array_to_string(n.path, chr(9)) ASC))[1] AS path_str,
(SELECT cc.id FROM content_chunks cc
WHERE cc.page_id = n.id ORDER BY cc.chunk_index ASC LIMIT 1) AS canonical_chunk_id
FROM walk n
@@ -6242,18 +6248,17 @@ export class PostgresEngine implements BrainEngine {
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
// #4145 R2-2 preflight: an ALREADY-aborted signal must short-circuit
// BEFORE the query is dispatched — the previous order created the
// pending query first and cancelled it after, which still burned a
// round-trip (and on a saturated pool, a slot). Cancellation remains
// BEST-EFFORT overall (PG protocol cancel is async); callers that need
// correctness must rely on their own fencing, not this signal.
if (opts?.signal?.aborted) {
throw new DOMException('aborted', 'AbortError');
}
const pending = conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]);
if (opts?.signal) {
if (opts.signal.aborted) {
// .cancel() is fire-and-forget; the awaited query rejects with the
// postgres "query was cancelled" error which the caller catches.
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
} catch {
// best-effort
}
throw new DOMException('aborted', 'AbortError');
}
const onAbort = () => {
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
@@ -6312,6 +6317,13 @@ export class PostgresEngine implements BrainEngine {
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
// #4145 R2-2: observe the signal BEFORE (potentially slow) direct-pool
// acquisition — a caller whose timeout already fired must not queue for
// a pool slot just to be cancelled afterwards. runUnsafe re-checks after
// acquisition.
if (opts?.signal?.aborted) {
throw new DOMException('aborted', 'AbortError');
}
// Inside an open transaction, _sql is the reserved tx connection (set via
// defineProperty in transaction()); never reroute off it.
const inTransaction = this._sql !== null && this.connectionManager?.peekReadPool() !== this._sql;
+3 -1
View File
@@ -926,6 +926,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
depth INTEGER NOT NULL DEFAULT 0,
max_children INTEGER,
timeout_ms INTEGER,
lock_duration_ms INTEGER,
timeout_at TIMESTAMPTZ,
remove_on_complete BOOLEAN NOT NULL DEFAULT FALSE,
remove_on_fail BOOLEAN NOT NULL DEFAULT FALSE,
@@ -942,7 +943,8 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
CONSTRAINT chk_nonnegative CHECK (attempts_made >= 0 AND attempts_started >= 0 AND stalled_counter >= 0 AND max_attempts >= 1 AND max_stalled >= 0),
CONSTRAINT chk_depth_nonnegative CHECK (depth >= 0),
CONSTRAINT chk_max_children_positive CHECK (max_children IS NULL OR max_children > 0),
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0)
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0),
CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000))
);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_claim ON minion_jobs (queue, priority ASC, created_at ASC) WHERE status = 'waiting';
+29 -3
View File
@@ -812,6 +812,21 @@ export interface HybridSearchOpts extends SearchOpts {
*/
_queryEmbedDeadline?: QueryEmbedDeadline;
/**
* Hermetic eval canaries/CI non-semantic embeddings. When set, the query
* embedding for the TEXT vector arm comes from this function (e.g. qrels
* basis vectors) INSTEAD of the gateway's query-embed path, and the
* no-embedding-provider keyword-only short-circuit is bypassed so the
* vector arm runs with no provider key configured at all. Never set on
* production paths; when absent, behavior is byte-for-byte unchanged.
*
* Cache note: bare `hybridSearch` neither reads nor writes the semantic
* query cache by construction both the lookup and the store live only in
* `hybridSearchCached` so a deterministic-embedding eval run through this
* seam cannot poison `query_cache` for production queries.
*/
queryEmbedFn?: (text: string) => Float32Array | Promise<Float32Array>;
/**
* INTERNAL cache-consult outcome threaded from `hybridSearchCached` into
* the inner `hybridSearch` so the ONE telemetry record per search (emitted
@@ -1264,7 +1279,10 @@ export async function hybridSearch(
earlyModality === 'both' ||
mayEscalateToMultimodal) &&
isAvailable('embedding', multimodalProviderProbe);
if (!isAvailable('embedding', providerProbe) && !willTryMultimodal) {
// Hermetic eval canaries/CI: a caller-supplied queryEmbedFn produces the
// vector-arm query embedding without the gateway, so provider
// availability is irrelevant — skip the keyword-only short-circuit.
if (!opts?.queryEmbedFn && !isAvailable('embedding', providerProbe) && !willTryMultimodal) {
// v0.43 — fuse the relational arm with keyword so typed-edge answers
// survive on the no-embedding-provider path (the relational win is most
// valuable exactly when vector is unavailable). The title arm fuses here
@@ -1502,12 +1520,20 @@ export async function hybridSearch(
// share one ~6s budget); direct callers get a fresh deadline. On timeout
// the embed rejects → salvage below (or keyword-only when all reject).
const embedDl = opts?._queryEmbedDeadline ?? makeQueryEmbedDeadline();
// Hermetic eval canaries/CI: queryEmbedFn (non-semantic deterministic
// embeddings) replaces the gateway query-embed for the text vector arm.
// No deadline needed — it's a synchronous-ish local computation with no
// network. Absent queryEmbedFn, the bounded gateway path is unchanged.
const embedOneQuery = (q: string): Promise<Float32Array> =>
opts?.queryEmbedFn
? Promise.resolve(opts.queryEmbedFn(q))
: embedQueryBounded(q, embedOpts, embedDl);
if (!searchSalvageEnabled()) {
// ENG-7 kill switch (GBRAIN_SEARCH_SALVAGE=off): pre-wave
// all-or-nothing fan-outs — one variant's failure abandons every
// embedding and falls back to keyword-only.
try {
const embeddings = await Promise.all(queries.map(q => embedQueryBounded(q, embedOpts, embedDl)));
const embeddings = await Promise.all(queries.map(q => embedOneQuery(q)));
queryEmbedding = embeddings[0];
const textLists = await Promise.all(
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
@@ -1537,7 +1563,7 @@ export async function hybridSearch(
// WP2/T3 (ENG-15) salvage fan-outs: allSettled on BOTH the embed
// fan-out and the searchVector fan-out so one variant's failure no
// longer abandons the survivors (the query-vs-search asymmetry fix).
const settled = await Promise.allSettled(queries.map(q => embedQueryBounded(q, embedOpts, embedDl)));
const settled = await Promise.allSettled(queries.map(q => embedOneQuery(q)));
const okEmbeds: Float32Array[] = [];
const embedFailures: unknown[] = [];
for (const s of settled) {
+83
View File
@@ -0,0 +1,83 @@
/**
* Deterministic basis-vector embeddings for hermetic eval canaries/CI.
*
* NON-SEMANTIC embeddings: each query embeds as a unit basis vector at a
* fixed dimension, so retrieval through the full hybrid pipeline (vector +
* keyword/title/alias arms + RRF) is exactly reproducible with no API keys,
* no network, and no provider drift. Used by the qrels correctness gate's
* deterministic embedder path and the retrieval canary runner
* (scripts/run-eval-canary.ts). Mirrors the basis-vector convention in
* test/eval-replay-gate.test.ts and test/fixtures/eval-baselines/
* qrels-search.json (each fixture query carries an `embedding_dim`).
*/
/** Unit basis vector with 1.0 at `idx % dim` and 0.0 elsewhere. */
export function basisEmbedding(idx: number, dim = 1536): Float32Array {
const emb = new Float32Array(dim);
emb[idx % dim] = 1.0;
return emb;
}
/**
* 32-bit FNV-1a hash. Deterministic, dependency-free; used only to derive a
* stable fallback basis dimension for query texts not present in the qrels
* fixture.
*/
export function fnv1a(text: string): number {
let h = 0x811c9dc5;
for (let i = 0; i < text.length; i++) {
h ^= text.charCodeAt(i);
h = Math.imul(h, 0x01000193);
}
return h | 0;
}
export interface LegacyQrelsQuery {
query_id: string;
query: string;
embedding_dim: number;
relevant_slugs: string[];
first_relevant_slug: string;
}
/**
* Parse the legacy qrels fixture shape
* ({queries: [{query, embedding_dim, relevant_slugs, first_relevant_slug}]}).
* THE single parser for this shape the canary runner and the embedder
* builder both consume it. Throws on malformed JSON or a missing `queries`
* array (callers surface a usage error; the gate's own qrels parser reports
* shape problems in detail).
*/
export function parseLegacyQrels(raw: string): LegacyQrelsQuery[] {
const parsed = JSON.parse(raw) as { queries?: unknown };
if (!Array.isArray(parsed.queries)) {
throw new Error('qrels fixture missing "queries" array');
}
return parsed.queries as LegacyQrelsQuery[];
}
/**
* Build a query-embed function from a raw qrels fixture (the legacy shape:
* `{queries: [{query, embedding_dim, ...}]}`). Known query texts map to
* `basisEmbedding(embedding_dim)`; unknown texts fall back to a
* deterministic FNV-1a-derived basis dimension in [100, 1099] outside the
* fixture's low dims, so an unknown query can never accidentally vote for a
* fixture page's basis direction (fixture dims are small integers).
*
* Throws on malformed JSON or a missing `queries` array (caller surfaces a
* usage error; the gate's own qrels parser reports shape problems in detail).
*/
export function buildQrelsQueryEmbedFn(qrelsRaw: string): (text: string) => Float32Array {
const queries = parseLegacyQrels(qrelsRaw);
const dimByQuery = new Map<string, number>();
for (const q of queries) {
if (typeof (q as { query?: unknown })?.query === 'string' && typeof (q as { embedding_dim?: unknown })?.embedding_dim === 'number') {
dimByQuery.set(q.query, q.embedding_dim);
}
}
return (text: string): Float32Array => {
const dim = dimByQuery.get(text);
if (dim !== undefined) return basisEmbedding(dim);
return basisEmbedding(100 + (Math.abs(fnv1a(text)) % 1000));
};
}
+3 -1
View File
@@ -922,6 +922,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
depth INTEGER NOT NULL DEFAULT 0,
max_children INTEGER,
timeout_ms INTEGER,
lock_duration_ms INTEGER,
timeout_at TIMESTAMPTZ,
remove_on_complete BOOLEAN NOT NULL DEFAULT FALSE,
remove_on_fail BOOLEAN NOT NULL DEFAULT FALSE,
@@ -938,7 +939,8 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
CONSTRAINT chk_nonnegative CHECK (attempts_made >= 0 AND attempts_started >= 0 AND stalled_counter >= 0 AND max_attempts >= 1 AND max_stalled >= 0),
CONSTRAINT chk_depth_nonnegative CHECK (depth >= 0),
CONSTRAINT chk_max_children_positive CHECK (max_children IS NULL OR max_children > 0),
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0)
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0),
CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000))
);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_claim ON minion_jobs (queue, priority ASC, created_at ASC) WHERE status = 'waiting';
+1 -1
View File
@@ -1,6 +1,6 @@
# gbrain agent workspace — template
<!-- gbrain-template-stamp: 0.46.4.0 -->
<!-- gbrain-template-stamp: 0.46.6.0 -->
This repository is the **"Use this template"** distribution artifact for a
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
+63
View File
@@ -85,6 +85,69 @@ describe('lockRenewalAudit: 4-outcome contract', () => {
});
});
describe('lockRenewalAudit: v0.46 (#4145) additive telemetry fields', () => {
test('case 9a — ctx fields round-trip through the JSONL (cause/lateness/overlap/load/via)', async () => {
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
lockRenewalAudit.logFailure(7, 'subagent', 2, new Error('x'), {
cause: 'call-timeout',
lateness_ms: 40_000,
overlap_skips: 1,
load1: 28.12,
cores: 32,
deadline_deferred: true,
});
lockRenewalAudit.logSuccessAfterFailure(7, 'subagent', 2, { via: 'verify' });
const result = readRecentLockRenewalEvents(24);
expect(result.events).toHaveLength(2);
expect(result.events[0]).toMatchObject({
outcome: 'failure',
cause: 'call-timeout',
lateness_ms: 40_000,
overlap_skips: 1,
load1: 28.12,
cores: 32,
deadline_deferred: true,
});
expect(result.events[1]).toMatchObject({ outcome: 'success_after_failure', via: 'verify' });
});
});
test('case 9b — omitted ctx leaves the new keys OUT of the JSONL entirely (no undefined noise)', async () => {
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
lockRenewalAudit.logGaveUp(8, 'sync', 3, new Error('y'));
const result = readRecentLockRenewalEvents(24);
expect(result.events).toHaveLength(1);
const raw = result.events[0] as unknown as Record<string, unknown>;
for (const key of ['cause', 'lateness_ms', 'overlap_skips', 'load1', 'cores', 'via', 'deadline_deferred']) {
expect(key in raw).toBe(false);
}
});
});
test('case 9c — pre-upgrade JSONL lines (no telemetry fields) still parse in readback', async () => {
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
// A line exactly as a pre-v0.46 build would have written it.
const legacy = JSON.stringify({
ts: new Date().toISOString(),
job_id: 5,
job_name: 'embed',
attempt: 1,
outcome: 'failure',
error_message_summary: 'Connection terminated',
error_code: '08006',
});
const file = path.join(tmpDir, computeIsoWeekFilename(LOCK_RENEWAL_FEATURE_NAME, new Date()));
fs.mkdirSync(path.dirname(file), { recursive: true });
fs.writeFileSync(file, `${legacy}\n`);
const result = readRecentLockRenewalEvents(24);
expect(result.corrupted_lines).toBe(0);
expect(result.events).toHaveLength(1);
expect(result.events[0].outcome).toBe('failure');
expect(result.events[0].cause).toBeUndefined();
});
});
});
describe('lockRenewalAudit: privacy via redactor (D9)', () => {
test('case 5a — logFailure with PG connection-failure error: no DSN/IP in JSONL', async () => {
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
+7
View File
@@ -170,6 +170,13 @@ describe('verifyWorkspace — keyless pass', () => {
// Probe cleanup [G13]: pages, files, and the reconciled fact are gone.
expect(existsSync(join(ws, 'brain', `${VERIFY_PROBE_SLUG}.md`))).toBe(false);
expect(existsSync(join(ws, 'brain', `${VERIFY_PROBE_ENTITY_SLUG}.md`))).toBe(false);
// Tombstone-proof: the cleanup HARD-deletes via engine.deletePage — a
// soft delete (deleted_at tombstone) would leave these rows countable.
const probeRows = await engine.executeRaw<{ n: string }>(
`SELECT count(*)::text AS n FROM pages WHERE slug = ANY($1::text[])`,
[[VERIFY_PROBE_SLUG, VERIFY_PROBE_ENTITY_SLUG]],
);
expect(probeRows[0].n).toBe('0');
const facts = await engine.executeRaw<{ fact: string }>(
`SELECT fact FROM facts WHERE source_id = $1 AND fact LIKE $2`,
['workspace', `%${VERIFY_MAGIC_TOKEN}%`],
+21 -27
View File
@@ -20,6 +20,24 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MIGRATIONS } from '../src/core/migrate.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
// ONE engine for the whole file (was three describe-scoped engines = three
// full PGLite boots for 11 tests). Each data-bearing describe resets state
// in its own beforeAll and re-seeds — required, not just hygiene: the
// 'refactor' corpus of the searchKeyword describe would otherwise pollute
// the searchKeywordChunks describe's expectations.
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
describe('Cathedral II v28 migration — search_vector backfill', () => {
test('v28 migration exists in registry', () => {
@@ -44,12 +62,8 @@ describe('Cathedral II v28 migration — search_vector backfill', () => {
});
describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
await resetPgliteState(engine);
// Two pages, each with multiple chunks that match "refactor" so we can
// verify the dedup pass returns one chunk per page. upsertChunks fires
@@ -86,10 +100,6 @@ describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
]);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('returns one row per matched page (dedup to best chunk per page)', async () => {
const results = await engine.searchKeyword('refactor');
const slugs = results.map(r => r.slug).sort();
@@ -121,12 +131,8 @@ describe('Cathedral II Layer 3 — searchKeyword external contract', () => {
});
describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
await resetPgliteState(engine);
// Page with multiple matching chunks so chunk-grain results can
// return two chunks from the same page (no dedup).
@@ -143,10 +149,6 @@ describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', ()
]);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('does not dedup: can return multiple chunks from the same page', async () => {
const results = await engine.searchKeywordChunks('refactor', { limit: 20 });
const slugs = results.map(r => r.slug);
@@ -174,12 +176,8 @@ describe('Cathedral II Layer 3 — searchKeywordChunks (internal primitive)', ()
});
describe('Cathedral II Layer 3 — doc-comment weight precedence (A4 foundation)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
await resetPgliteState(engine);
// Two pages, each with one chunk. Alpha's chunk has the target term
// 'hexagon' in its doc_comment (weight A); Beta's chunk has it in
@@ -217,10 +215,6 @@ describe('Cathedral II Layer 3 — doc-comment weight precedence (A4 foundation)
);
});
afterAll(async () => {
await engine.disconnect();
}, 30_000);
test('doc-comment match outranks body-text match on the same term', async () => {
const results = await engine.searchKeyword('hexagon');
expect(results.length).toBeGreaterThan(0);
+7
View File
@@ -30,6 +30,13 @@ function run(args: string[]): { exitCode: number; stdout: string; stderr: string
const env = { ...process.env, HOME: tmp } as Record<string, string | undefined>;
delete env.DATABASE_URL;
delete env.GBRAIN_DATABASE_URL;
// Cross-file poisoning guard: sibling test files in the same bun process
// set process.env.GBRAIN_HOME (preferences, friction, bootstrap-* et al),
// and doctor resolves ~/.gbrain via resolveGbrainHome — which prefers
// GBRAIN_HOME over HOME. A leaked value makes the seeded
// $HOME/.gbrain/migrations fixture invisible and doctor exits 0 where the
// test expects the FAIL exit. Scrub it like the DB URLs above.
delete env.GBRAIN_HOME;
try {
const stdout = execFileSync('bun', ['run', CLI, ...args], {
env: env as Record<string, string>,
+1 -1
View File
@@ -217,7 +217,7 @@ describeIfDB('autopilot fan-out — Postgres E2E', () => {
// expired-lock active must NOT suppress (wedge detectors stay fed):
// the next slot INSERTS a fresh waiting row.
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '1 second'
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds'
WHERE id = $1`, [claimed!.id],
);
const r2 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-2'));
@@ -70,11 +70,22 @@ describe('bootstrap harness lifecycle E2E (PGLite + real serve --http)', () => {
// remapped HOME (it reads the password database), so HOME alone does NOT
// sandbox user-scope writes — that leak is exactly why claudeUserSettingsPath
// honors CLAUDE_CONFIG_DIR/HOME explicitly now.
//
// DATABASE_URL/GBRAIN_DATABASE_URL must be scrubbed for the IN-PROCESS
// runBootstrap lane too (the beforeAll already scrubs them for its
// subprocesses): since v0.31.3 (9c60b3a06, #801) an env DATABASE_URL
// deliberately overrides the file-backed PGLite engine in loadConfig().
// Under the DATABASE_URL-bearing e2e wrapper, a leaked URL retargets the
// mint at the shared Postgres test DB — no PGLite single-writer lock, so
// the mint-refusal contract under a live serve never fires and the
// fresh-minted token fails the bearer smoke against the PGLite serve.
const envFor = () => ({
GBRAIN_HOME: parent,
HOME: sandboxHome,
CLAUDE_CONFIG_DIR: join(sandboxHome, '.claude'),
CODEX_HOME: codexHome,
DATABASE_URL: undefined,
GBRAIN_DATABASE_URL: undefined,
});
beforeAll(async () => {
@@ -286,10 +286,18 @@ describe.skipIf(!DATABASE_URL)('Postgres bootstrap verify (real Postgres)', () =
assertSafeE2eDatabaseUrl(DATABASE_URL!);
await engine.connect({ database_url: DATABASE_URL! });
await engine.initSchema();
// This file runs against the shared e2e DB WITHOUT setupDB's TRUNCATE, so
// a prior standalone run's `workspace` source row survives and addSource
// (whose `force` only bypasses git validation, not the id-collision check)
// would throw source_id_taken. Sweep it first; the FK cascade removes any
// leftover pages/facts under it.
await engine.executeRaw(`DELETE FROM sources WHERE id = 'workspace'`, []);
await addSource(engine, { id: 'workspace', localPath: join(ws, 'brain'), force: true });
}, 60_000);
afterAll(async () => {
// Leave the shared DB clean for the next file / next standalone run.
try { await engine.executeRaw(`DELETE FROM sources WHERE id = 'workspace'`, []); } catch { /* noop */ }
try { await engine.disconnect(); } catch { /* noop */ }
if (prevHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = prevHome;
+8 -2
View File
@@ -27,6 +27,7 @@ import {
import {
configureGateway,
resetGateway,
__unconfigureGatewayForTests,
__setEmbedTransportForTests,
} from '../../src/core/ai/gateway.ts';
import type { ResolvedColumn } from '../../src/core/types.ts';
@@ -261,8 +262,13 @@ describe('upsertChunks — model provenance uses gateway-resolved model, not com
test('#3461: unconfigured gateway falls back to the brain config model, never the compiled default', async () => {
await engine.setConfig('embedding_model', 'voyage:voyage-3-large');
// The preload's beforeEach re-configures the gateway before every test,
// so the reset must happen INSIDE the test body.
resetGateway();
// so the unconfigure must happen INSIDE the test body. Since commit
// 3aa064bcc (#3554), `resetGateway()` RESTORES the preload's OpenAI/1536
// baseline instead of unconfiguring — this test needs genuine no-gateway
// behavior (getEmbeddingModel() must THROW), which is exactly what
// `__unconfigureGatewayForTests()` was added for. The preload's
// beforeEach restores the baseline before the next test.
__unconfigureGatewayForTests();
await engine.putPage('docs/provenance-throw-path', {
type: 'concept',
+14 -3
View File
@@ -740,9 +740,20 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
expect(pgRow.updated_at).toBeInstanceOf(Date);
// markPagesExtractedBatch: stamp one → count drops to 2 on both.
const stampAt = new Date().toISOString();
await pgEngine.markPagesExtractedBatch([{ slug: 'sp/1', source_id: SRC }], stampAt);
await pgliteEngine.markPagesExtractedBatch([{ slug: 'sp/1', source_id: SRC }], stampAt);
// Stamp with the row's OWN updated_at_iso (per-ref extractedAt — the
// #1768/D4 production semantics used by extractStaleFromDB), NOT client
// `new Date()`: the test client's clock and the DB server's clock are
// different clocks (docker VM drift under load), so a client-now stamp can
// land before the row's server-side `updated_at`, leaving sp/1 flagged
// `updated_at > links_extracted_at` and the count stuck at 3.
for (const eng of [pgEngine, pgliteEngine]) {
const sp1 = (await eng.listStalePagesForExtraction({ batchSize: 10, sourceId: SRC }))
.find((r) => r.slug === 'sp/1')!;
await eng.markPagesExtractedBatch(
[{ slug: 'sp/1', source_id: SRC, extractedAt: sp1.updated_at_iso }],
sp1.updated_at_iso,
);
}
expect(await pgEngine.countStalePagesForExtraction({ sourceId: SRC })).toBe(2);
expect(await pgliteEngine.countStalePagesForExtraction({ sourceId: SRC })).toBe(2);
+19 -5
View File
@@ -12,6 +12,12 @@
* 3. sourceId scoping isolates between two seeded sources (no leak)
* 4. ANY($2::text[]) binding actually filters by type set
*
* Type allowlist updated for e1e1f3bac (PR #2615): discovery now honors the
* active schema pack's `extractable: true` flags in addition to the legacy
* 6-type floor. Under the default `gbrain-base` pack that makes `note`
* extractable; the non-extractable control type here is `person`
* (pack-declared `extractable: false`).
*
* ~4 structural assertions; ~3-5s wallclock budget.
* Skips gracefully when DATABASE_URL is unset.
*/
@@ -37,8 +43,10 @@ afterAll(async () => {
beforeEach(async () => {
if (skip) return;
// Clean test-source rows + atoms + meeting pages between tests
await engine.executeRaw(`DELETE FROM pages WHERE source_id IN ('default', 'dept-x') AND (type = 'atom' OR type IN ('meeting', 'source', 'article', 'video', 'book', 'original'))`);
// Clean test-source rows + atoms + seeded pages between tests. Includes
// 'note' (pack-extractable since e1e1f3bac / PR #2615) and 'person' (the
// non-extractable control) so seeds can't leak into later tests.
await engine.executeRaw(`DELETE FROM pages WHERE source_id IN ('default', 'dept-x') AND (type = 'atom' OR type IN ('meeting', 'source', 'article', 'video', 'book', 'original', 'note', 'person'))`);
await engine.executeRaw(`DELETE FROM sources WHERE id = 'dept-x'`);
});
@@ -74,13 +82,17 @@ describeIfDB('v0.41.2.1 D10 — discoverExtractablePages on real Postgres', () =
test('returns extractable rows when seeded', async () => {
await seedPage({ slug: 'meeting/a', type: 'meeting', content_hash: 'hash-A-1234567890abc' });
await seedPage({ slug: 'source/b', type: 'source', content_hash: 'hash-B-1234567890abc' });
await seedPage({ slug: 'notes/skip', type: 'note', content_hash: 'hash-N-1234567890abc' });
// e1e1f3bac (PR #2615): `note` is pack-extractable under gbrain-base, so
// it IS discovered now. `person` (extractable: false) is the skip control.
await seedPage({ slug: 'notes/kept', type: 'note', content_hash: 'hash-N-1234567890abc' });
await seedPage({ slug: 'people/skip', type: 'person', content_hash: 'hash-P-1234567890abc' });
const discovered = await discoverExtractablePages(engine, 'default');
const slugs = discovered.map((d) => d.slug).sort();
expect(slugs).toContain('meeting/a');
expect(slugs).toContain('source/b');
expect(slugs).not.toContain('notes/skip');
expect(slugs).toContain('notes/kept');
expect(slugs).not.toContain('people/skip');
});
test('ANY($::text[]) bind works through postgres.unsafe (PGLite parity proof)', async () => {
@@ -90,7 +102,9 @@ describeIfDB('v0.41.2.1 D10 — discoverExtractablePages on real Postgres', () =
for (const type of ['meeting', 'source', 'article', 'video', 'book', 'original']) {
await seedPage({ slug: `${type}/x`, type, content_hash: `hash-${type}-1234567890ab` });
}
await seedPage({ slug: 'note/skip', type: 'note', content_hash: 'hash-note-1234567890' });
// e1e1f3bac (PR #2615): `note` is now pack-extractable, so the
// non-extractable control is `person` (extractable: false in gbrain-base).
await seedPage({ slug: 'person/skip', type: 'person', content_hash: 'hash-pers-1234567890' });
const discovered = await discoverExtractablePages(engine, 'default');
const slugs = discovered.map((d) => d.slug).sort();
+1 -1
View File
@@ -173,7 +173,7 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
VALUES
('rescue-me', 'default', 'active', 0, '{}'::jsonb, 3, 1, 1,
'exponential', 1000, 0.2, 0, 3,
'crashed-worker:123', now() - interval '10 seconds', 'fail_parent', 0, false, false,
'crashed-worker:123', now() - interval '30 seconds', 'fail_parent', 0, false, false,
now() - interval '1 minute')
RETURNING id
`);
+41 -10
View File
@@ -18,10 +18,10 @@
* ship to ClawHub).
* 2. `openclaw plugins install --link` against an isolated `--profile`
* directory.
* 3. `openclaw plugins inspect <id> --runtime --json` imports the plugin and
* reads our default-export shape
* back from the runtime registry (`status: 'loaded'`, `imported: true`,
* id/name/description match).
* 3. `openclaw plugins inspect <id> --json` (with `--runtime` on older
* CLIs see inspectRuntime) imports the plugin and reads our
* default-export shape back from the runtime registry
* (`status: 'loaded'`, `imported: true`, id/name/description match).
* 4. `openclaw config set plugins.slots.contextEngine gbrain-context`
* `openclaw config validate` confirms the slot binding is accepted.
* 5. `openclaw plugins doctor` surfaces zero error-level diagnostics for
@@ -82,6 +82,25 @@ function runOpenclaw(args: string[], opts: { timeoutMs?: number } = {}): {
};
}
/**
* Version-robust runtime inspect. The test was pinned to
* `plugins inspect <id> --runtime --json` (#3742), but OpenClaw has since
* folded the runtime import into the default inspect and REMOVED the
* `--runtime` flag (observed on OpenClaw 2026.4.10: `error: unknown option
* '--runtime'`, while plain `inspect --json` reports status/imported/
* activated + diagnostics). Try the flagged form first for older CLIs and
* fall back to the plain form when the flag is unknown, so the assertions
* pin the plugin-load contract rather than one CLI version's flag surface.
*/
function inspectRuntime(id: string): { exitCode: number; stdout: string; stderr: string } {
const flagged = runOpenclaw(['plugins', 'inspect', id, '--runtime', '--json'], { timeoutMs: 30_000 });
if (flagged.exitCode === 0) return flagged;
if (/unknown option '--runtime'/.test(`${flagged.stdout}\n${flagged.stderr}`)) {
return runOpenclaw(['plugins', 'inspect', id, '--json'], { timeoutMs: 30_000 });
}
return flagged;
}
function cleanup() {
if (!OPENCLAW) return;
// Best-effort: uninstall the plugin and rm the profile dir. Both may
@@ -121,17 +140,29 @@ describe('openclaw-plugin-load-real (Tier 2 e2e)', () => {
readFileSync(join(fixtureTemplate, 'openclaw.plugin.json.template'), 'utf8'),
);
// Build our real entry to a single JS bundle. This is the same source
// Build our real entry into the fixture dir. This is the same source
// (`src/openclaw-context-engine.ts`) that the release ships; only the
// packaging layer (test fixture's package.json) is test-specific.
//
// `--outdir` (not `--outfile`): since the Retrieval Reflex ladder
// (v0.42.39.0, 8f45624e5 #2019) the entry's import chain reaches
// engine-factory → pglite-engine → @electric-sql/pglite, whose WASM/
// data assets become sibling build outputs — and `bun build` refuses
// `--outfile` when a build produces multiple output files. The fixed
// `--entry-naming` keeps the bundle at `entry.js`, matching the
// fixture package.json's `openclaw.extensions` entry; the assets land
// alongside it in the same directory, where the bundle's relative
// references resolve.
const buildResult = spawnSync(
'bun',
[
'build',
join(repoRoot, 'src', 'openclaw-context-engine.ts'),
'--target=bun',
'--outfile',
join(fixtureDir, 'entry.js'),
'--outdir',
fixtureDir,
'--entry-naming',
'[dir]/entry.[ext]',
],
{ encoding: 'utf8', timeout: 60_000 },
);
@@ -168,7 +199,7 @@ describe('openclaw-plugin-load-real (Tier 2 e2e)', () => {
it.skipIf(SKIP)(
'openclaw imports the entry file and reports status=loaded',
() => {
const r = runOpenclaw(['plugins', 'inspect', PLUGIN_ID, '--runtime', '--json'], { timeoutMs: 30_000 });
const r = inspectRuntime(PLUGIN_ID);
expect(r.exitCode).toBe(0);
const inspect = JSON.parse(r.stdout);
@@ -184,7 +215,7 @@ describe('openclaw-plugin-load-real (Tier 2 e2e)', () => {
it.skipIf(SKIP)(
'default export carries the expected id / name / description metadata',
() => {
const r = runOpenclaw(['plugins', 'inspect', PLUGIN_ID, '--runtime', '--json'], { timeoutMs: 30_000 });
const r = inspectRuntime(PLUGIN_ID);
expect(r.exitCode).toBe(0);
const inspect = JSON.parse(r.stdout);
@@ -199,7 +230,7 @@ describe('openclaw-plugin-load-real (Tier 2 e2e)', () => {
it.skipIf(SKIP)(
'register(api) ran without producing error-level diagnostics',
() => {
const r = runOpenclaw(['plugins', 'inspect', PLUGIN_ID, '--runtime', '--json'], { timeoutMs: 30_000 });
const r = inspectRuntime(PLUGIN_ID);
expect(r.exitCode).toBe(0);
const inspect = JSON.parse(r.stdout);
+8
View File
@@ -96,6 +96,14 @@ beforeAll(() => {
delete runEnv.OPENAI_API_KEY;
delete runEnv.ANTHROPIC_API_KEY;
delete runEnv.GOOGLE_API_KEY;
// Strip DB-URL env vars: since v0.31.3 (9c60b3a06, #801) an env
// DATABASE_URL deliberately overrides a file-backed PGLite engine
// selection in loadConfig(). This whole file is a hermetic-PGLite
// suite; when run under the DATABASE_URL-bearing e2e wrapper, an
// inherited URL would silently retarget every subprocess (including
// the torn-WAL fixture below) at the shared Postgres test DB.
delete runEnv.DATABASE_URL;
delete runEnv.GBRAIN_DATABASE_URL;
// NOTE: init grew strict flag validation (#2201); `--repo`/`--yes` were
// never real init flags (previously silently ignored). The repo is wired
+27 -16
View File
@@ -235,27 +235,38 @@ describeMaybe('phantom-redirect E2E (Postgres)', () => {
FROM facts WHERE source_id='default'
ORDER BY id`,
);
// After migrateFactsToCanonical, the row is now under canonical.
// The migrate step preserves embedding. Then the main reconcile
// visits canonical (added via touched_canonicals) and does
// wipe-then-insert from fence — which DROPS embedding because
// the fence doesn't carry it. So embedding ends up NULL.
// After migrateFactsToCanonical, the row is now under canonical with
// every other column — including embedding — preserved (the UPDATE
// rewrites only the slug columns). The main reconcile then visits
// canonical (added via touched_canonicals); since 54a807064 (#2932,
// idempotent extract_facts) an in-sync page is a NO-OP instead of the
// old wipe-then-reinsert, so the migrated row and its embedding
// SURVIVE the pass. (Pre-#2932 the wipe dropped the embedding to
// NULL, which this test used to document.)
//
// This test EXISTS to document this baseline behavior: the migrate
// step itself does NOT corrupt the embedding column on Postgres
// (the round-12 concern), but the subsequent fence reconcile
// intentionally re-derives from fence and embedding is regenerated
// by the embed phase.
//
// The pinning assertion: at NO point is the embedding column
// populated with a STRING (postgres-js's text shape leak — that
// bug class would produce a non-null text-shaped value here, not
// a clean NULL).
// The round-12 pinning assertions:
// 1. The embedding survived the migrate + reconcile round-trip
// non-NULL at its original dimensionality — postgres-js did not
// mangle it through its text representation.
// 2. At NO point is the embedding column populated with a
// non-vector-typed value. Since v0.31.0 (89ae72095, migration
// v40) the facts.embedding column is HALFVEC(1536) on pgvector
// >= 0.7 (full-precision VECTOR fallback below that), so BOTH
// real vector types are legitimate here.
const survived = await engine.executeRaw<{ ct: string; dims: number | null }>(
`SELECT COUNT(*)::text AS ct, MIN(vector_dims(embedding::vector)) AS dims
FROM facts
WHERE source_id='default'
AND source_markdown_slug='people/alice-example'
AND embedding IS NOT NULL`,
);
expect(parseInt(survived[0].ct, 10)).toBe(1);
expect(Number(survived[0].dims)).toBe(1536);
const stringShaped = await engine.executeRaw<{ ct: string }>(
`SELECT COUNT(*)::text AS ct FROM facts
WHERE source_id='default'
AND embedding IS NOT NULL
AND pg_typeof(embedding)::text != 'vector'`,
AND pg_typeof(embedding)::text NOT IN ('vector', 'halfvec')`,
);
expect(parseInt(stringShaped[0].ct, 10)).toBe(0);
expect(rows.length).toBeGreaterThan(0);
+15 -1
View File
@@ -580,9 +580,23 @@ describeE2E('serve-http OAuth 2.1 E2E (v0.26.1 + v0.26.2 + v0.26.3)', () => {
const postgres = (await import('postgres')).default;
const sql = postgres(process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL || '', { prepare: false });
try {
// Plain-array bind, NOT `sql.array([...])`: sql.array resolves its
// array OID (and serializer) through postgres.js's typeArrayMap, which
// is fetched asynchronously on connection startup. This INSERT is the
// FIRST query on this fresh connection, so the map is still empty and
// sql.array falls back to the element OID (25 = text) with scalar
// serialization — real Postgres rejects it with 42804 ("column scopes
// is of type text[] but expression is of type text"; an explicit
// ::text[] cast just shifts the failure to 22P02 "malformed array
// literal" because the value still serializes as a bare scalar). A
// plain JS array always serializes to the `{...}` literal and binds
// with an unspecified OID, so Postgres coerces it from column context
// deterministically — same untyped-bind approach as pgArray() in
// src/core/oauth-provider.ts. Latent since d61808d80 (v0.42.64.0):
// CI's e2e.yml never runs this file.
await sql`
INSERT INTO oauth_tokens (token_hash, token_type, client_id, scopes, expires_at)
VALUES (${tokenHash}, ${'access'}, ${publicClientId!}, ${sql.array(['read'])}, ${Math.floor(Date.now() / 1000) + 3600})
VALUES (${tokenHash}, ${'access'}, ${publicClientId!}, ${['read']}, ${Math.floor(Date.now() / 1000) + 3600})
`;
} finally {
await sql.end();
+23
View File
@@ -70,11 +70,31 @@ function gitCommit(repoPath: string, message: string) {
execSync(`git add -A && git commit -m "${message}"`, { cwd: repoPath, stdio: 'pipe' });
}
/**
* #2114 (commit 636628fdb): global sync.* anchors only move for the brain repo
* they describe. `writeSyncAnchor` proves ownership via `ownsGlobalSyncAnchor`,
* which falls back to the default source row's `local_path` when
* `config.sync.repo_path` is unset and `setupDB()` truncates `config` but
* NOT `sources`, so a stale `local_path` from a previously-run e2e file (e.g.
* sync-credential-preflight.test.ts, which sorts before this file in the full
* lane) survives in the shared DB and silently refuses every anchor write
* here. That wedges the bookmark: every sync re-runs as `first_sync` and
* `sync.last_commit` never persists. Reset the default-source identity so this
* file's mkdtemp repo is a legitimate fresh-brain bootstrap.
*/
async function resetBrainRepoIdentity() {
const engine = getEngine();
await engine.executeRaw(
`UPDATE sources SET local_path = NULL, last_commit = NULL WHERE id = 'default'`,
);
}
describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
let repoPath: string;
beforeAll(async () => {
await setupDB();
await resetBrainRepoIdentity();
repoPath = createTestRepo();
}, 30_000);
@@ -417,6 +437,9 @@ describeE2E('E2E: sync --skip-failed structured summary loop (v0.22.12, issue #5
beforeAll(async () => {
await setupDB();
// #2114: see resetBrainRepoIdentity above — setupDB truncates config but
// not sources, and this block syncs a brand-new mkdtemp repo.
await resetBrainRepoIdentity();
// Save+clear the real ~/.gbrain/sync-failures.jsonl so the test starts from
// a known-empty state. Restored in afterAll. This file is per-machine, NOT
@@ -13,6 +13,9 @@
// - re-running is idempotent (total_applied: 0)
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test';
import { mkdtempSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
import { runUnifyTypes } from '../../src/core/schema-pack/unify-types-handler.ts';
@@ -21,7 +24,25 @@ import { _resetPackCacheForTests } from '../../src/core/schema-pack/registry.ts'
let engine: PGLiteEngine;
// GBRAIN_HOME isolation (in-process; configDir() reads process.env at call
// time). Two leaks without it, both through the file-plane config:
// 1. checkPackUpgradeAvailable() reads loadConfigFileOnly() — an ambient
// ~/.gbrain/config.json carrying `schema_pack: gbrain-base-v2` (any
// machine whose brain already unified) makes the pre-unify check return
// 'ok' instead of 'warn' and the first test fails.
// 2. runUnifyTypes({apply: true}) WRITES the flip via saveConfig() — under
// bare `bun test` that lands in the operator's REAL config (the exact
// breach scripts/run-e2e.sh's md5 check exists to catch), and under the
// full e2e lane it lands in the wrapper's shared tmp HOME, poisoning
// every later loadActivePack caller in the same invocation.
// Same in-process pattern as test/preferences.test.ts.
let tmpHome: string;
let origGbrainHome: string | undefined;
beforeAll(async () => {
origGbrainHome = process.env.GBRAIN_HOME;
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-type-uni-e2e-'));
process.env.GBRAIN_HOME = tmpHome;
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
@@ -29,11 +50,17 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
if (origGbrainHome === undefined) delete process.env.GBRAIN_HOME;
else process.env.GBRAIN_HOME = origGbrainHome;
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* best-effort */ }
});
beforeEach(async () => {
await resetPgliteState(engine);
_resetPackCacheForTests();
// The first test's apply flips schema_pack in the isolated file-plane
// config; wipe it so every test starts pre-unify (order-independence).
try { rmSync(join(tmpHome, '.gbrain'), { recursive: true, force: true }); } catch { /* best-effort */ }
});
function ctxOf() {
+34 -9
View File
@@ -10,7 +10,13 @@
* 3. handler invocation + JSON serialization (ToolResult shape)
* 4. Error path: OperationError isError + JSON envelope
* 5. Trust gate: ctx.remote === true on get_recent_transcripts must
* reach the handler and produce a permission_denied error.
* reach the handler and produce a permission_denied error. Since
* commit 6a905a1e5 (v0.45.13.0, #4096 WP1/D7) localOnly ops only
* DISPATCH on the stdio local pipe (transport: 'stdio'); any other
* or unset transport is denied fail-closed with the unknown_tool
* envelope BEFORE the handler (pinned in
* test/dispatch-localonly.test.ts). The trust axis (`remote`) is a
* separate, in-handler check.
*
* Runs against PGLite in-memory. No DATABASE_URL, no API keys.
*/
@@ -111,11 +117,13 @@ describe('v0.29 E2E — dispatchToolCall for the three new ops', () => {
test('get_recent_transcripts rejects with permission_denied when ctx.remote === true', async () => {
// Defense-in-depth: even though serve-http filters localOnly: true ops
// out of the MCP tool list, the in-handler ctx.remote check is the
// last line. dispatchToolCall defaults remote=true, which is what
// every MCP transport sets, so the reject must fire here.
// last line. This dispatch shape (remote: true, transport: 'stdio') is
// exactly what the real stdio MCP server sets (src/mcp/server.ts) —
// stdio passes the #4096 WP1/D7 locality backstop but stays
// remote/untrusted, so the in-handler trust gate must fire.
const result = await dispatchToolCall(engine, 'get_recent_transcripts', {
days: 7,
}, { remote: true, sourceId: 'default' });
}, { remote: true, transport: 'stdio', sourceId: 'default' });
expect(result.isError).toBe(true);
const err = JSON.parse(result.content[0].text);
@@ -124,13 +132,30 @@ describe('v0.29 E2E — dispatchToolCall for the three new ops', () => {
expect(err.message.toLowerCase()).toContain('local-only');
});
test('get_recent_transcripts succeeds when ctx.remote === false (CLI path)', async () => {
// The local-CLI path explicitly sets remote: false. Op should run
// (returning [] is fine — no corpus dir is configured in this test
// fixture; the test just asserts the trust gate didn't reject).
test('get_recent_transcripts is denied fail-closed (unknown_tool) when the transport marker is unset', async () => {
// #4096 WP1/D7 backstop: localOnly ops dispatch only on the stdio local
// pipe. An unset transport marker — even from a trusted (remote: false)
// caller — gets the same envelope as a nonexistent op, so the catalog
// never leaks which localOnly names exist.
for (const remote of [true, false]) {
const result = await dispatchToolCall(engine, 'get_recent_transcripts', {
days: 7,
}, { remote, sourceId: 'default' });
expect(result.isError).toBe(true);
expect(JSON.parse(result.content[0].text).error).toBe('unknown_tool');
}
});
test('get_recent_transcripts succeeds when ctx.remote === false (trusted local dispatch)', async () => {
// Trusted local callers set remote: false; the dispatch must arrive on
// the stdio local pipe to pass the #4096 locality backstop (the real
// CLI path in src/cli.ts invokes handlers directly with remote: false
// and never crosses the backstop). Op should run (returning [] is fine
// — no corpus dir is configured in this test fixture; the test just
// asserts the trust gate didn't reject).
const result = await dispatchToolCall(engine, 'get_recent_transcripts', {
days: 7,
}, { remote: false });
}, { remote: false, transport: 'stdio' });
expect(result.isError).toBeFalsy();
const rows = JSON.parse(result.content[0].text);
@@ -0,0 +1,164 @@
/**
* E2E: verify-before-evict under real Postgres (#4145, CDX-12/R2-5).
*
* The hermetic suite (test/worker-lock-renewal.test.ts) proves the tick's
* state machine, including the fake-time incident replay. These tests prove
* the DB-level invariants the verify RELIES on, against real Postgres:
*
* 1. renewLock revives an expired-but-UNSTOLEN lease (it fences on
* lock_token and deliberately does not check lock_until) the
* foundation of the starved-but-ours recovery path.
* 2. The stall-sweep reclaim grace holds the sweep off a freshly-lapsed
* lease (the same-burst self-steal race), while grace=0 restores the
* legacy predicate.
* 3. After a genuine reclaim, the owner's renewal is fenced-false the
* verify's CERTAIN-loss signal and the requeue burned no attempt.
* 4. Worker end-to-end: a synchronously-blocked event loop past lease
* expiry does NOT evict a healthy job it completes with zero
* stall bounces (the #4145 incident's user-visible contract).
*
* A faithful cross-process starvation replay (blocked worker + concurrent
* foreign sweeper) needs a second OS process and lives out of scope; the
* hermetic incident replay carries that contract deterministically.
*
* Run: DATABASE_URL=... bun test test/e2e/worker-lock-renewal-starvation.test.ts
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { hasDatabase, setupDB, teardownDB, getConn, getEngine } from './helpers.ts';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
import { MinionQueue } from '../../src/core/minions/queue.ts';
import { MinionWorker } from '../../src/core/minions/worker.ts';
import { runMigrations } from '../../src/core/migrate.ts';
const skip = !hasDatabase();
const describeE2E = skip ? describe.skip : describe;
if (skip) {
console.log('Skipping E2E lock-renewal starvation tests (DATABASE_URL not set)');
}
async function makeEngine(): Promise<PostgresEngine> {
const engine = new PostgresEngine();
await engine.connect({ engine: 'postgres', database_url: process.env.DATABASE_URL!, poolSize: 4 });
return engine;
}
/** Synchronously block the event loop — the incident's starvation shape. */
function blockEventLoop(ms: number): void {
const until = Date.now() + ms;
while (Date.now() < until) { /* spin */ }
}
describeE2E('E2E: lock-renewal verify-before-evict foundations (#4145)', () => {
beforeAll(async () => {
await setupDB();
await runMigrations(getEngine());
}, 30_000);
afterAll(async () => {
await teardownDB();
});
beforeEach(async () => {
const conn = getConn();
await conn.unsafe(`TRUNCATE minion_attachments, minion_inbox, minion_jobs RESTART IDENTITY CASCADE`);
});
test('renewLock revives an expired-but-unstolen lease; grace holds the sweep; reclaim yields fenced-false', async () => {
const engine = await makeEngine();
try {
const queue = new MinionQueue(engine);
const job = await queue.add('starve-target', {});
const claimed = await queue.claim('tok-owner', 60_000, 'default', ['starve-target']);
expect(claimed?.id).toBe(job.id);
// Lapse the lease as a starved owner would (2s past — inside the 15s grace).
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '2 seconds' WHERE id = $1`,
[job.id],
);
// (2) The default-grace sweep does NOT steal the freshly-lapsed lease.
const held = await queue.handleStalled();
expect(held.requeued).toHaveLength(0);
expect(held.dead).toHaveLength(0);
// (1) The owner's (verify) renewal revives the expired-but-unstolen lease.
const revived = await queue.renewLock(job.id, 'tok-owner', 60_000);
expect(revived).toBe(true);
const afterRevive = await queue.getJob(job.id);
expect(afterRevive!.status).toBe('active');
expect(afterRevive!.lock_until!.getTime()).toBeGreaterThan(Date.now() + 30_000);
// Lapse again; grace=0 restores the legacy predicate and reclaims.
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '2 seconds' WHERE id = $1`,
[job.id],
);
const swept = await queue.handleStalled(0);
expect(swept.requeued).toHaveLength(1);
// (3) The owner's next renewal is fenced-false — CERTAIN loss — and
// the infrastructure requeue burned no attempt.
const fenced = await queue.renewLock(job.id, 'tok-owner', 60_000);
expect(fenced).toBe(false);
const requeued = await queue.getJob(job.id);
expect(requeued!.status).toBe('waiting');
expect(requeued!.attempts_made).toBe(0);
expect(requeued!.stalled_counter).toBe(1);
} finally {
await engine.disconnect();
}
}, 30_000);
test('worker survives a blocked event loop past lease expiry: job completes, zero stall bounces', async () => {
const engine = await makeEngine();
try {
const queue = new MinionQueue(engine);
const job = await queue.add('starved-but-healthy', {});
const worker = new MinionWorker(engine, {
concurrency: 1,
pollInterval: 50,
lockDuration: 1_000, // lease expires DURING the synchronous block
stalledInterval: 200,
});
let blocked = false;
worker.register('starved-but-healthy', async () => {
// The incident shape: the process is healthy but the loop is
// saturated past the entire lease window. No renewal (and no
// foreign steal — single worker), so the lease lapses; the
// post-block renewal must recover it rather than evict.
blockEventLoop(2_500);
blocked = true;
return { survived: true };
});
const p = worker.start();
const started = Date.now();
let completed = false;
while (Date.now() - started < 10_000) {
const j = await queue.getJob(job.id);
if (j?.status === 'completed') { completed = true; break; }
await new Promise(r => setTimeout(r, 50));
}
worker.stop();
await p;
expect(blocked).toBe(true);
expect(completed).toBe(true);
const final = await queue.getJob(job.id);
expect(final!.status).toBe('completed');
expect(final!.result).toEqual({ survived: true });
// The user-visible #4145 contract: a starved-but-healthy job is not
// bounced through the stall detector, and no attempt is burned
// (attempts_made only bumps on failure; one clean claim).
expect(final!.stalled_counter).toBe(0);
expect(final!.attempts_made).toBe(0);
expect(final!.attempts_started).toBe(1);
} finally {
await engine.disconnect();
}
}, 30_000);
});
+186
View File
@@ -0,0 +1,186 @@
/**
* test/eval-canary.test.ts hermetic retrieval canary (deterministic
* embedder + runner script).
*
* Pins:
* 1. basisEmbedding determinism + dims (src/eval/deterministic-embed.ts).
* 2. buildQrelsQueryEmbedFn maps fixture queries to their basis dims and
* falls back deterministically (FNV-1a-derived dim) for unknown texts.
* 3. scripts/run-eval-canary.ts end-to-end in check mode: real CLI
* subprocess, exit 0, metrics at/above the qrels default floors.
* 4. Determinism: two in-process runs of the correctness gate through the
* queryEmbedFn seam produce identical metrics.
* 5. Check mode writes nothing to tracked files (git status unchanged).
*
* Fully hermetic: no API keys, no network, no DATABASE_URL.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { readFileSync } from 'node:fs';
import { join, resolve } from 'node:path';
import { execSync, spawnSync } from 'node:child_process';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { basisEmbedding, buildQrelsQueryEmbedFn, fnv1a } from '../src/eval/deterministic-embed.ts';
import { parseLegacyQrels, seedCanaryCorpus } from '../scripts/run-eval-canary.ts';
import { runCorrectnessGate, type CorrectnessGateOpts } from '../src/core/bench/correctness-gate.ts';
import { parseQrelsFile, DEFAULT_QRELS_THRESHOLDS } from '../src/core/bench/qrels-file.ts';
import { hybridSearch } from '../src/core/search/hybrid.ts';
const ROOT = resolve(import.meta.dir, '..');
const QRELS_PATH = join(ROOT, 'test', 'fixtures', 'eval-baselines', 'qrels-search.json');
// ---------------------------------------------------------------------------
// Canonical PGLite block (CLAUDE.md R3+R4)
// ---------------------------------------------------------------------------
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
// ---------------------------------------------------------------------------
// 1. basisEmbedding
// ---------------------------------------------------------------------------
describe('basisEmbedding', () => {
test('default dim 1536, 1.0 at idx, 0.0 elsewhere', () => {
const e = basisEmbedding(5);
expect(e.length).toBe(1536);
expect(e[5]).toBe(1.0);
expect(e[0]).toBe(0.0);
expect(e.reduce((s, v) => s + v, 0)).toBe(1.0);
});
test('custom dim + idx wraparound', () => {
const e = basisEmbedding(103, 100);
expect(e.length).toBe(100);
expect(e[3]).toBe(1.0); // 103 % 100
expect(e.reduce((s, v) => s + v, 0)).toBe(1.0);
});
test('deterministic across calls', () => {
expect([...basisEmbedding(42)]).toEqual([...basisEmbedding(42)]);
});
});
// ---------------------------------------------------------------------------
// 2. buildQrelsQueryEmbedFn
// ---------------------------------------------------------------------------
describe('buildQrelsQueryEmbedFn', () => {
const raw = readFileSync(QRELS_PATH, 'utf-8');
test('maps every fixture query to its embedding_dim basis vector', () => {
const fn = buildQrelsQueryEmbedFn(raw);
const fixture = parseLegacyQrels(raw);
expect(fixture.length).toBeGreaterThanOrEqual(10);
for (const q of fixture) {
expect([...fn(q.query)]).toEqual([...basisEmbedding(q.embedding_dim)]);
}
});
test('unknown text falls back to a deterministic FNV-1a-derived dim in [100, 1099]', () => {
const fn = buildQrelsQueryEmbedFn(raw);
const unknown = 'a query text that is definitely not in the fixture';
const a = fn(unknown);
const b = fn(unknown);
expect([...a]).toEqual([...b]);
const idx = a.findIndex(v => v === 1.0);
expect(idx).toBe(100 + (Math.abs(fnv1a(unknown)) % 1000));
expect(idx).toBeGreaterThanOrEqual(100);
expect(idx).toBeLessThanOrEqual(1099);
});
test('throws on malformed input', () => {
expect(() => buildQrelsQueryEmbedFn('not json')).toThrow();
expect(() => buildQrelsQueryEmbedFn('{"no_queries": true}')).toThrow(/queries/);
});
});
// ---------------------------------------------------------------------------
// 4. In-process determinism through the queryEmbedFn seam
// ---------------------------------------------------------------------------
describe('correctness gate through the queryEmbedFn seam', () => {
test('two in-process runs produce identical metrics, at/above the default floors', async () => {
const raw = readFileSync(QRELS_PATH, 'utf-8');
await seedCanaryCorpus(engine, parseLegacyQrels(raw));
const queryEmbedFn = buildQrelsQueryEmbedFn(raw);
const qrels = parseQrelsFile(raw);
const searchFn: NonNullable<CorrectnessGateOpts['searchFn']> = async (e, q, o) => {
const results = await hybridSearch(e, q, { limit: o.limit, queryEmbedFn });
return results.map(r => ({ source_id: r.source_id, slug: r.slug }));
};
const a = await runCorrectnessGate(engine, qrels, { searchFn });
const b = await runCorrectnessGate(engine, qrels, { searchFn });
expect(a.summary).toEqual(b.summary);
expect(a.per_query).toEqual(b.per_query);
expect(a.summary.queries_errored).toBe(0);
expect(a.summary.mean_recall_at_k).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.recall_at_k);
expect(a.summary.first_relevant_hit_rate).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.first_relevant_hit);
expect(a.summary.expected_top1_denominator).toBeGreaterThan(0);
expect(a.summary.expected_top1_hit_rate).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.expected_top1);
}, 60_000);
});
// ---------------------------------------------------------------------------
// 3 + 5. Runner end-to-end (check mode) + tracked-file invariance
// ---------------------------------------------------------------------------
describe('run-eval-canary.ts (check mode)', () => {
// Captured by the end-to-end test; asserted separately below so a
// tracked-file write shows up as its own named failure.
let statusBefore: string | null = null;
let statusAfter: string | null = null;
let runExit: number | null = null;
test('spawns the real CLI gate hermetically and passes the floors', () => {
// Status scoped to the paths check mode could plausibly touch — a
// whole-tree porcelain diff flakes when a concurrent shard sibling (or a
// developer save) creates an unrelated file during the ~30s window.
const STATUS_SCOPE = 'git status --porcelain -- .gbrain-evals test/fixtures docs/eval';
statusBefore = execSync(STATUS_SCOPE, { cwd: ROOT, encoding: 'utf-8' });
const child = spawnSync(
process.execPath,
[join(ROOT, 'scripts', 'run-eval-canary.ts')],
// Outer budget strictly above the runner's inner CLI-child timeout so
// the runner's own diagnostics always win the race.
{ cwd: ROOT, encoding: 'utf-8', timeout: 118_000 },
);
statusAfter = execSync(STATUS_SCOPE, { cwd: ROOT, encoding: 'utf-8' });
runExit = child.status;
const combined = (child.stdout ?? '') + (child.stderr ?? '');
expect(child.status).toBe(0);
const m = combined.match(
/mean_recall_at_k=([\d.]+) first_relevant_hit_rate=([\d.]+) expected_top1_hit_rate=([\d.]+)/,
);
expect(m).not.toBeNull();
expect(Number(m![1])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.recall_at_k);
expect(Number(m![2])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.first_relevant_hit);
expect(Number(m![3])).toBeGreaterThanOrEqual(DEFAULT_QRELS_THRESHOLDS.expected_top1);
}, 120_000);
test('check mode writes nothing to tracked files (git status unchanged)', () => {
// Guard: the end-to-end test above must have actually run + passed spawn.
expect(runExit).toBe(0);
expect(statusBefore).not.toBeNull();
expect(statusAfter).toBe(statusBefore!);
});
});
+46
View File
@@ -141,6 +141,52 @@ describe('eval gate: usage errors', () => {
});
});
describe('eval gate: embedder flag validation', () => {
// The hermetic-canary embedder option accepts exactly one value and only
// composes with the correctness (qrels) gate. A regression that silently
// accepts a bad value would fall through to the keyed gateway path and
// defeat the hermetic guarantee.
const REAL_QRELS = 'test/fixtures/eval-baselines/qrels-search.json';
test('unsupported embedder value → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'semantic', '--qrels', REAL_QRELS]),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder combined with the baseline gate → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, [
'--embedder', 'deterministic',
'--baseline', '/tmp/does-not-exist-12345.ndjson',
'--qrels', REAL_QRELS,
]),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder without a qrels file → exit 2', async () => {
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'deterministic']),
);
expect(out.exitCode).toBe(2);
});
test('deterministic embedder with a malformed qrels file → exit 2', async () => {
const { mkdtempSync, writeFileSync } = await import('node:fs');
const { tmpdir } = await import('node:os');
const { join } = await import('node:path');
const dir = mkdtempSync(join(tmpdir(), 'gate-embedder-'));
const bad = join(dir, 'malformed.json');
writeFileSync(bad, '{"not_queries": []}');
const out = await withExitCapture(() =>
runEvalGate(engine, ['--embedder', 'deterministic', '--qrels', bad]),
);
expect(out.exitCode).toBe(2);
});
});
describe('eval gate: regression-only path', () => {
test('malformed baseline → surfaces as breach (verdict fail, exit 1)', async () => {
const dir = mkdtempSync(join(tmpdir(), 'eval-gate-test-'));
+4 -6
View File
@@ -26,6 +26,7 @@ import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { withEnv } from './helpers/with-env.ts';
import { basisEmbedding } from '../src/eval/deterministic-embed.ts';
import type { ChunkInput } from '../src/core/types.ts';
// ---------------------------------------------------------------------------
@@ -72,12 +73,9 @@ function loadFixture(): QrelFixture {
return fix;
}
/** Basis vector with 1.0 at `idx` and 0.0 elsewhere. Mirrors search-quality.test.ts. */
function basisEmbedding(idx: number, dim = 1536): Float32Array {
const emb = new Float32Array(dim);
emb[idx % dim] = 1.0;
return emb;
}
// basisEmbedding (1.0 at `idx`, 0.0 elsewhere) is imported from
// src/eval/deterministic-embed.ts — the shared home for hermetic
// basis-vector embeddings (also used by the retrieval canary).
/**
* Seed each relevant slug with a chunk whose embedding aligns with the
+17
View File
@@ -82,3 +82,20 @@ describe('formatJobDetail timeout/deadline lines', () => {
expect(out).toContain('null-default wall-clock sweep applies');
});
});
describe('lock lease rendering (#4145)', () => {
test('stamped row value renders with the cadence note', () => {
const out = formatJobDetail(job({ lock_duration_ms: 120000 }));
expect(out).toContain('Lock lease: 120000ms (renewed at min(lease/2, 60s) cadence)');
});
test('unset lease on a mapped handler renders the claim-time default', () => {
const out = formatJobDetail(job({ name: 'subagent', lock_duration_ms: null }));
expect(out).toContain('Lock lease: (unset) — handler default 300000ms stamps at claim');
});
test('unset lease on an unmapped handler renders no lease line (worker default applies)', () => {
const out = formatJobDetail(job({ name: 'sync', lock_duration_ms: null }));
expect(out).not.toContain('Lock lease:');
});
});
+19 -11
View File
@@ -569,7 +569,7 @@ describe('migration v35 — auto_rls_event_trigger structural guards', () => {
// 1. Structural — assert the migration SQL literally contains the helper
// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression
// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)).
// 2. Behavioral — populate 1000 duplicates and assert the migration completes
// 2. Behavioral — populate 200 duplicates and assert the migration completes
// under the wall-clock cap. Sanity check at small scale; the structural
// assertion is the real guard.
@@ -962,7 +962,7 @@ describe('migrate runner v67 — typed-claim columns materialized on PGLite', ()
});
});
describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => {
describe('migrate: v8 (links_dedup) regression — must be fast on 200 duplicate rows', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
@@ -975,7 +975,7 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate
await engine.disconnect();
});
test('1000 duplicate links dedup completes in <90s and leaves table deduped', async () => {
test('200 duplicate links dedup completes in <90s and leaves table deduped', async () => {
// Set up: drop BOTH the old (v8) and new (v11) unique constraints so
// duplicates can be inserted, then reset version so v8 + v11 re-run.
// v11 replaces the v8 constraint name; we drop whichever is present.
@@ -989,15 +989,19 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate
const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id;
const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id;
// Insert 1000 duplicates of the same (from, to, type) row
for (let i = 0; i < 1000; i++) {
// Insert 200 duplicates of the same (from, to, type) row
// 200 rows, not 1000: the O(n²) shape this gate guards is still
// unmistakable at 200 (minutes vs sub-second dedup) and the insert loop
// stops burning ~15-25s of suite budget per test on row traffic that
// adds no discriminating power.
for (let i = 0; i < 200; i++) {
await db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`,
[fromId, toId, 'mention', `dup-${i}`]
);
}
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
expect(beforeCount).toBe(1000);
expect(beforeCount).toBe(200);
// Reset version to 7 so v8 + v9 + v10 + v11 re-run
await engine.setConfig('version', '7');
@@ -1039,7 +1043,7 @@ describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate
});
});
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => {
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 200 duplicate rows', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
@@ -1052,22 +1056,26 @@ describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K d
await engine.disconnect();
});
test('1000 duplicate timeline entries dedup completes in <90s and leaves table deduped', async () => {
test('200 duplicate timeline entries dedup completes in <90s and leaves table deduped', async () => {
const db = (engine as any).db;
await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`);
await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' });
const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id;
// Insert 1000 duplicates of the same (page_id, date, summary) row
for (let i = 0; i < 1000; i++) {
// Insert 200 duplicates of the same (page_id, date, summary) row
// 200 rows, not 1000: the O(n²) shape this gate guards is still
// unmistakable at 200 (minutes vs sub-second dedup) and the insert loop
// stops burning ~15-25s of suite budget per test on row traffic that
// adds no discriminating power.
for (let i = 0; i < 200; i++) {
await db.query(
`INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`,
[pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`]
);
}
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
expect(beforeCount).toBe(1000);
expect(beforeCount).toBe(200);
await engine.setConfig('version', '7');
+121
View File
@@ -0,0 +1,121 @@
/**
* #4145 migration v130 (minion_jobs_lock_duration_ms).
*
* Pinned contracts:
* 1. v130 exists in MIGRATIONS with the canonical name, idempotent flag, and
* one engine-agnostic sql block; adds the column AND the CHECK via the
* idempotent drop-then-add pattern (v7 precedent) so migrated brains
* carry the same DB bound as fresh installs.
* 2. NO backfill: NULL lock_duration_ms means "worker default" (pre-#4145
* behavior); the claim-time COALESCE owns all defaulting.
* 3. SQL-level idempotency: re-executing the v130 statements directly on an
* already-migrated schema changes nothing and throws nothing.
* 4. The CHECK holds: 0 / negative direct writes are rejected; NULL and
* positive values pass.
* 5. A pre-v130-shaped table (column dropped) gains the column on re-run.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionQueue } from '../src/core/minions/queue.ts';
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
let engine: PGLiteEngine;
let queue: MinionQueue;
const V130_SQL = MIGRATIONS.find(m => m.version === 130)?.sql ?? '';
const V130_STATEMENTS = V130_SQL.split(';').map(s => s.trim()).filter(Boolean);
async function execV130Directly(): Promise<void> {
for (const stmt of V130_STATEMENTS) {
await engine.executeRaw(stmt);
}
}
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' }); // in-memory
await engine.initSchema();
queue = new MinionQueue(engine);
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw('DELETE FROM minion_jobs');
});
describe('migration v130 — structure', () => {
test('exists with canonical name, idempotent flag, engine-agnostic sql', () => {
const v130 = MIGRATIONS.find(m => m.version === 130);
expect(v130).toBeDefined();
expect(v130?.name).toBe('minion_jobs_lock_duration_ms');
expect(v130?.idempotent).toBe(true);
expect(v130?.sqlFor).toBeUndefined();
expect(LATEST_VERSION).toBeGreaterThanOrEqual(130);
});
test('adds the column IF NOT EXISTS and the CHECK via drop-then-add (v7 precedent); NO backfill', () => {
expect(V130_SQL).toContain('ADD COLUMN IF NOT EXISTS lock_duration_ms INTEGER');
expect(V130_SQL).toContain('DROP CONSTRAINT IF EXISTS chk_lock_duration_positive');
expect(V130_SQL).toContain('ADD CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000))');
// NULL = worker default = pre-#4145 behavior; a backfill would change
// legacy rows' semantics for no benefit (claim COALESCE owns defaulting).
expect(V130_SQL).not.toContain('UPDATE minion_jobs');
});
});
describe('migration v130 — semantics (PGLite)', () => {
test('SQL idempotency: direct re-run on a migrated schema is a clean no-op', async () => {
await execV130Directly();
await execV130Directly(); // twice — the drop-then-add pair must converge
const rows = await engine.executeRaw<{ count: string }>(
`SELECT count(*)::text AS count FROM information_schema.columns
WHERE table_name = 'minion_jobs' AND column_name = 'lock_duration_ms'`,
);
expect(rows[0].count).toBe('1');
});
test('CHECK holds: out-of-range rejected, NULL + in-range pass', async () => {
const job = await queue.add('lease-check', {});
await engine.executeRaw(`UPDATE minion_jobs SET lock_duration_ms = 300000 WHERE id = $1`, [job.id]);
await engine.executeRaw(`UPDATE minion_jobs SET lock_duration_ms = NULL WHERE id = $1`, [job.id]);
// The CHECK enforces the full advertised [5s,1h] range so a bypass
// writer can't stamp a thrash-lease or a weeks-long one (codex P2).
for (const bad of [0, -5, 1, 4999, 3_600_001]) {
await expect(
engine.executeRaw(`UPDATE minion_jobs SET lock_duration_ms = ${bad} WHERE id = $1`, [job.id]),
).rejects.toThrow(/chk_lock_duration_positive/);
}
});
test('pre-v130 shape (column dropped) gains column + CHECK on re-run', async () => {
await engine.executeRaw(`ALTER TABLE minion_jobs DROP CONSTRAINT IF EXISTS chk_lock_duration_positive`);
await engine.executeRaw(`ALTER TABLE minion_jobs DROP COLUMN IF EXISTS lock_duration_ms`);
await execV130Directly();
const job = await queue.add('lease-regain', {});
const row = await queue.getJob(job.id);
expect(row!.lock_duration_ms).toBeNull(); // unmapped name → no default stamped
await expect(
engine.executeRaw(`UPDATE minion_jobs SET lock_duration_ms = 0 WHERE id = $1`, [job.id]),
).rejects.toThrow(/chk_lock_duration_positive/);
});
test('claim SQL clamps a bypass-written lease before deriving lock_until (codex P2)', async () => {
// Simulate foreign tooling stamping a lease outside the advertised
// range directly (the CHECK blocks this on current schemas, so drop it
// for the simulation — the claim clamp is the layer under test).
await engine.executeRaw(`ALTER TABLE minion_jobs DROP CONSTRAINT IF EXISTS chk_lock_duration_positive`);
const job = await queue.add('bypass-lease', {});
await engine.executeRaw(`UPDATE minion_jobs SET lock_duration_ms = 2147483647 WHERE id = $1`, [job.id]);
const before = Date.now();
const claimed = await queue.claim('tok-bypass', 30_000, 'default', ['bypass-lease']);
expect(claimed!.lock_duration_ms).toBe(3_600_000); // stamped clamped
const horizon = claimed!.lock_until!.getTime() - before;
expect(horizon).toBeLessThan(3_700_000); // lock_until bounded to ~1h, not ~24.8 days
// Restore the range CHECK for subsequent tests.
await engine.executeRaw(`ALTER TABLE minion_jobs ADD CONSTRAINT chk_lock_duration_positive CHECK (lock_duration_ms IS NULL OR (lock_duration_ms >= 5000 AND lock_duration_ms <= 3600000))`);
});
});
+314 -12
View File
@@ -210,7 +210,7 @@ describe('MinionQueue: Stall Detection', () => {
await queue.claim('tok1', 30000, 'default', ['sync']);
// Force lock_until to the past
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const { requeued, dead } = await queue.handleStalled();
@@ -227,7 +227,7 @@ describe('MinionQueue: Stall Detection', () => {
// First stall: counter 0+1=1 < 3, requeued
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const r1 = await queue.handleStalled();
@@ -237,7 +237,7 @@ describe('MinionQueue: Stall Detection', () => {
// Second stall: counter 1+1=2 < 3, requeued
await queue.claim('tok2', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const r2 = await queue.handleStalled();
@@ -246,7 +246,7 @@ describe('MinionQueue: Stall Detection', () => {
// Third stall: counter 2+1=3 >= 3, dead-lettered
await queue.claim('tok3', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const r3 = await queue.handleStalled();
@@ -260,7 +260,7 @@ describe('MinionQueue: Stall Detection', () => {
await engine.executeRaw('UPDATE minion_jobs SET max_stalled = 0 WHERE id = $1', [job.id]);
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const { requeued, dead } = await queue.handleStalled();
@@ -317,7 +317,7 @@ describe('MinionQueue: #1737 attempt accounting on dead-letter', () => {
// First stall: requeued, attempts_made stays 0 (lease-loss recovery, not an app attempt).
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const r1 = await queue.handleStalled();
@@ -327,7 +327,7 @@ describe('MinionQueue: #1737 attempt accounting on dead-letter', () => {
// Second stall: dead-lettered, attempts_made now increments.
await queue.claim('tok2', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const r2 = await queue.handleStalled();
@@ -444,7 +444,7 @@ describe('MinionQueue: v0.13.1 max_stalled schema default (#219)', () => {
for (let i = 0; i < 4; i++) {
await queue.claim(`tok-${i}`, 30000, 'default', ['noop']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const { requeued, dead } = await queue.handleStalled();
@@ -456,7 +456,7 @@ describe('MinionQueue: v0.13.1 max_stalled schema default (#219)', () => {
// With stalled_counter now at 4, next stall: 4+1=5 >= 5 = dead.
await queue.claim('tok-final', 30000, 'default', ['noop']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id]
);
const { dead } = await queue.handleStalled();
@@ -885,13 +885,13 @@ describe('MinionQueue: Cancel & Retry', () => {
// one requeue stall, then one dead-lettering stall.
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id],
);
await queue.handleStalled();
await queue.claim('tok2', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id],
);
const r2 = await queue.handleStalled();
@@ -906,7 +906,7 @@ describe('MinionQueue: Cancel & Retry', () => {
expect(retried!.stalled_counter).toBe(0);
await queue.claim('tok3', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
"UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1",
[job.id],
);
const r3 = await queue.handleStalled();
@@ -3203,3 +3203,305 @@ describe('MinionWorker: self-health-check behavior (v0.22.14)', () => {
expect(() => new MinionWorker(engine, {})).not.toThrow();
});
});
// --- v0.46 (#4145 R2-1): inFlight generation-safety ---
describe('MinionWorker: inFlight generation-safety (#4145 R2-1)', () => {
test('a stale execution\'s finally does not delete the reclaimed execution\'s entry', async () => {
const worker = new MinionWorker(engine, { concurrency: 2, pollInterval: 50, lockDuration: 60000 });
// Two gated executions of the SAME job id: A (stale, will be superseded)
// and B (the same-worker re-claim). Each handler invocation blocks on
// its own gate so the test controls the interleave deterministically.
const gates: Array<() => void> = [];
const gatePromises = [
new Promise<void>(r => gates.push(r)),
new Promise<void>(r => gates.push(r)),
];
let invocation = 0;
worker.register('gensafe', async () => {
const idx = invocation++;
await gatePromises[idx];
return { ok: true };
});
const row = await queue.add('gensafe', {});
// Execution A claims + launches.
const claimedA = await queue.claim('tok-A', 60000, 'default', ['gensafe']);
expect(claimedA?.id).toBe(row.id);
(worker as unknown as { launchJob(j: MinionJob, t: string): void }).launchJob(claimedA!, 'tok-A');
// Simulate the post-force-evict requeue (what handleStalled does) and a
// SAME-WORKER re-claim as execution B while A's handler is still alive.
await engine.executeRaw(
`UPDATE minion_jobs SET status='waiting', lock_token=NULL, lock_until=NULL, started_at=NULL WHERE id = $1`,
[row.id]
);
const claimedB = await queue.claim('tok-B', 60000, 'default', ['gensafe']);
expect(claimedB?.id).toBe(row.id);
(worker as unknown as { launchJob(j: MinionJob, t: string): void }).launchJob(claimedB!, 'tok-B');
const inFlight = (worker as unknown as { inFlight: Map<number, { lockToken: string }> }).inFlight;
expect(inFlight.get(row.id)?.lockToken).toBe('tok-B');
// Release stale execution A: its completeJob is fenced-false ("completion
// dropped") and its finally fires. Pre-fix, that finally deleted B's
// entry by bare job.id — the R2-1 generation race.
gates[0]();
await new Promise(r => setTimeout(r, 150));
expect(inFlight.get(row.id)?.lockToken).toBe('tok-B'); // B survived A's finally
// Release B: its OWN finally removes its own entry and completes the row.
gates[1]();
await new Promise(r => setTimeout(r, 150));
expect(inFlight.has(row.id)).toBe(false);
const final = await queue.getJob(row.id);
expect(final!.status).toBe('completed');
});
});
// --- v0.46 (#4145 CDX-7): stall-sweep reclaim grace ---
describe('MinionQueue: stall-sweep reclaim grace (#4145)', () => {
async function activeJobWithLockLapsedMs(msAgo: number): Promise<MinionJob> {
await queue.add('grace-test', {});
const claimed = await queue.claim('tok-grace', 60000, 'default', ['grace-test']);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - ($1::double precision * interval '1 millisecond') WHERE id = $2`,
[msAgo, claimed!.id]
);
return claimed!;
}
test('a lock that lapsed WITHIN the grace window is NOT reclaimed (starved-owner head start)', async () => {
const job = await activeJobWithLockLapsedMs(5_000); // grace default = 15_000
const { requeued, dead } = await queue.handleStalled();
expect(requeued).toHaveLength(0);
expect(dead).toHaveLength(0);
const still = await queue.getJob(job.id);
expect(still!.status).toBe('active');
expect(still!.lock_token).toBe('tok-grace');
});
test('a lock that lapsed BEYOND the grace window IS reclaimed', async () => {
const job = await activeJobWithLockLapsedMs(20_000);
const { requeued, dead } = await queue.handleStalled();
expect(requeued).toHaveLength(1);
expect(dead).toHaveLength(0);
const requeuedJob = await queue.getJob(job.id);
expect(requeuedJob!.status).toBe('waiting');
expect(requeuedJob!.lock_token).toBeNull();
});
test('grace=0 restores the exact legacy predicate (lock_until < now())', async () => {
const job = await activeJobWithLockLapsedMs(100);
const { requeued } = await queue.handleStalled(0);
expect(requeued).toHaveLength(1);
const requeuedJob = await queue.getJob(job.id);
expect(requeuedJob!.status).toBe('waiting');
});
test('env knob GBRAIN_MINION_STALL_RECLAIM_GRACE_MS resolves; bad value warns + falls back', async () => {
const { resolveStallReclaimGraceMs, DEFAULT_STALL_RECLAIM_GRACE_MS, _resetStallGraceWarningsForTests } =
await import('../src/core/minions/queue.ts');
_resetStallGraceWarningsForTests();
expect(resolveStallReclaimGraceMs({})).toBe(DEFAULT_STALL_RECLAIM_GRACE_MS);
expect(resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: '0' })).toBe(0);
expect(resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: '25000' })).toBe(25_000);
expect(resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: '-5' })).toBe(DEFAULT_STALL_RECLAIM_GRACE_MS);
expect(resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: 'abc' })).toBe(DEFAULT_STALL_RECLAIM_GRACE_MS);
// Cap: an absurd digit string must not become Infinity and push the
// sweep cutoff to -infinity (which would disable stalled-job recovery).
expect(resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: '9'.repeat(40) })).toBe(600_000);
});
test('grace env warn fires once per bad value, not per call (warn-once dedupe)', async () => {
const { resolveStallReclaimGraceMs, _resetStallGraceWarningsForTests } =
await import('../src/core/minions/queue.ts');
_resetStallGraceWarningsForTests();
const captured: string[] = [];
const origWrite = process.stderr.write.bind(process.stderr);
(process.stderr as { write: (chunk: string | Uint8Array) => boolean }).write = (chunk) => {
captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8'));
return true;
};
try {
resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: 'bogus' });
resolveStallReclaimGraceMs({ GBRAIN_MINION_STALL_RECLAIM_GRACE_MS: 'bogus' });
} finally {
process.stderr.write = origWrite;
}
expect(captured.filter(c => c.includes('GBRAIN_MINION_STALL_RECLAIM_GRACE_MS')).length).toBe(1);
});
});
// --- v0.46 (#4145): per-job lock_duration_ms — three-layer lease resolution ---
describe('MinionQueue: per-job lock lease (#4145)', () => {
test('claim stamps the handler-map default (300s) for subagent and derives lock_until from it', async () => {
await queue.add('subagent', {}, undefined, { allowProtectedSubmit: true });
const before = Date.now();
const claimed = await queue.claim('tok-lease', 30_000, 'default', ['subagent']);
expect(claimed).not.toBeNull();
expect(claimed!.lock_duration_ms).toBe(300_000); // stamped from the map
const horizon = claimed!.lock_until!.getTime() - before;
// lock_until derives from the 300s lease, NOT the worker's 30s default.
expect(horizon).toBeGreaterThan(250_000);
expect(horizon).toBeLessThan(360_000);
});
test('an unmapped handler keeps NULL lease → worker default horizon (legacy behavior)', async () => {
await queue.add('shortling', {});
const before = Date.now();
const claimed = await queue.claim('tok-short', 30_000, 'default', ['shortling']);
expect(claimed!.lock_duration_ms).toBeNull();
const horizon = claimed!.lock_until!.getTime() - before;
expect(horizon).toBeGreaterThan(20_000);
expect(horizon).toBeLessThan(40_000);
});
test('explicit submit value wins over the map and is clamped to [5s, 1h]', async () => {
const explicit = await queue.add('subagent', {}, { lock_duration_ms: 120_000 }, { allowProtectedSubmit: true });
expect(explicit.lock_duration_ms).toBe(120_000);
const floored = await queue.add('floorling', {}, { lock_duration_ms: 1 });
expect(floored.lock_duration_ms).toBe(5_000);
const ceiled = await queue.add('ceiling', {}, { lock_duration_ms: 99_999_999 });
expect(ceiled.lock_duration_ms).toBe(3_600_000);
});
test('claim precedence: an explicit row lease beats the handler map at the claim UPDATE', async () => {
// The claim COALESCE is (row, map, worker default) in that order — if it
// were ever flipped (map before row) the explicit lease would be silently
// overwritten at claim while the add()-time tests stayed green.
await queue.add('subagent', {}, { lock_duration_ms: 120_000 }, { allowProtectedSubmit: true });
const before = Date.now();
const claimed = await queue.claim('tok-precedence', 30_000, 'default', ['subagent']);
expect(claimed!.lock_duration_ms).toBe(120_000); // row wins, not the 300s map
const horizon = claimed!.lock_until!.getTime() - before;
expect(horizon).toBeGreaterThan(90_000);
expect(horizon).toBeLessThan(150_000);
});
test('worker renews with the per-job lease, not the worker default (launchJob wiring)', async () => {
// GAP-1 pin: effectiveLockMs = row lease drives BOTH the renewal call's
// duration arg and the cadence. A 5s lease (clamp floor) under a worker
// configured at 60s yields a 2.5s cadence — observable within test time.
const worker = new MinionWorker(engine, { concurrency: 1, pollInterval: 50, lockDuration: 60_000 });
let release: () => void = () => {};
const gate = new Promise<void>(r => { release = r; });
worker.register('leased-renewal', async () => { await gate; return { ok: true }; });
await queue.add('leased-renewal', {}, { lock_duration_ms: 5_000 });
const claimed = await queue.claim('tok-lease-renew', 5_000, 'default', ['leased-renewal']);
expect(claimed!.lock_duration_ms).toBe(5_000);
const durs: number[] = [];
const q = (worker as unknown as { queue: MinionQueue }).queue;
const orig = q.renewLock.bind(q);
q.renewLock = ((id: number, tok: string, dur: number, opts?: { signal?: AbortSignal }) => {
durs.push(dur);
return orig(id, tok, dur, opts);
}) as typeof q.renewLock;
(worker as unknown as { launchJob(j: MinionJob, t: string): void }).launchJob(claimed!, 'tok-lease-renew');
// Cadence = min(5000/2, 60000) = 2500ms; wait for at least one tick.
const started = Date.now();
while (durs.length === 0 && Date.now() - started < 8_000) {
await new Promise(r => setTimeout(r, 100));
}
release();
await new Promise(r => setTimeout(r, 150));
expect(durs.length).toBeGreaterThanOrEqual(1);
// Every renewal used the per-job 5s lease — NOT the worker's 60s default.
expect(new Set(durs)).toEqual(new Set([5_000]));
});
test('MCP submit_job threads lock_duration_ms through the shared clamp (round-trip)', async () => {
const { operationsByName } = await import('../src/core/operations.ts');
const op = operationsByName['submit_job'];
const ctx = { engine, remote: false, dryRun: false } as never;
const clamped = await op.handler(ctx, { name: 'lease-op-test', lock_duration_ms: 99_999_999 }) as { id: number };
expect((await queue.getJob(clamped.id))!.lock_duration_ms).toBe(3_600_000); // ceiling
const exact = await op.handler(ctx, { name: 'lease-op-test', lock_duration_ms: 60_000 }) as { id: number };
expect((await queue.getJob(exact.id))!.lock_duration_ms).toBe(60_000);
// Boundary pin: 0 is falsy through the op's `|| undefined` coercion and
// falls to the handler-map/worker default (NULL for an unmapped name) —
// the documented remote semantics, distinct from the CLI's exit-1 reject.
const zero = await op.handler(ctx, { name: 'lease-op-test', lock_duration_ms: 0 }) as { id: number };
expect((await queue.getJob(zero.id))!.lock_duration_ms).toBeNull();
});
test('idempotent re-submit never mutates the first submitter\'s lease (INSERT-only)', async () => {
const first = await queue.add('lease-idem', {}, { lock_duration_ms: 60_000, idempotency_key: 'lease-key-1' });
expect(first.lock_duration_ms).toBe(60_000);
const second = await queue.add('lease-idem', {}, { lock_duration_ms: 600_000, idempotency_key: 'lease-key-1' });
expect(second.id).toBe(first.id);
expect(second.lock_duration_ms).toBe(60_000); // unchanged
});
test('REGRESSION pin: NULL-lease rows keep the exact legacy wall-clock null-fallback', async () => {
// handleWallClockTimeouts' null-timeout branch is COALESCE(lock_duration_ms, $1)
// — rows WITHOUT a lease must behave exactly as before (worker default drives
// the 2x * max_stalled bound), and rows WITH a lease use their own.
const legacy = await queue.add('wallclock-legacy', {}, { max_stalled: 1 });
await queue.claim('tok-wc-legacy', 1_000, 'default', ['wallclock-legacy']);
// started 10s ago; NULL lease → threshold = 2 * 1000ms (worker default $1) * 1 = 2s → dead.
await engine.executeRaw(
`UPDATE minion_jobs SET started_at = now() - interval '10 seconds', timeout_ms = NULL, timeout_at = NULL WHERE id = $1`,
[legacy.id]
);
const killedLegacy = await queue.handleWallClockTimeouts(1_000);
expect(killedLegacy.map(j => j.id)).toContain(legacy.id);
// Same shape WITH a 60s lease: threshold = 2 * 60000 * 1 = 120s → survives 10s.
const leased = await queue.add('wallclock-leased', {}, { max_stalled: 1, lock_duration_ms: 60_000 });
await queue.claim('tok-wc-leased', 1_000, 'default', ['wallclock-leased']);
await engine.executeRaw(
`UPDATE minion_jobs SET started_at = now() - interval '10 seconds', timeout_ms = NULL, timeout_at = NULL WHERE id = $1`,
[leased.id]
);
const killedLeased = await queue.handleWallClockTimeouts(1_000);
expect(killedLeased.map(j => j.id)).not.toContain(leased.id);
const survivor = await queue.getJob(leased.id);
expect(survivor!.status).toBe('active');
});
test('repeated infrastructure evictions eventually dead-letter via max_stalled (lifetime accumulation pin)', async () => {
// The "no attempt burned" contract survives only max_stalled - 1
// requeues: stalled_counter accumulates across the job's lifetime and
// the final stall burns one attempt and dead-letters. Known coverage
// gap flagged in the #4145 review — pinned here.
const job = await queue.add('evict-accumulate', {}, { max_stalled: 3 });
for (let round = 1; round <= 2; round++) {
const claimed = await queue.claim(`tok-ev-${round}`, 30_000, 'default', ['evict-accumulate']);
expect(claimed!.id).toBe(job.id);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`,
[job.id]
);
const { requeued, dead } = await queue.handleStalled();
expect(requeued.map(j => j.id)).toContain(job.id);
expect(dead).toHaveLength(0);
const after = await queue.getJob(job.id);
expect(after!.stalled_counter).toBe(round);
expect(after!.attempts_made).toBe(0); // no attempt burned on requeue
}
// Third stall: stalled_counter + 1 >= max_stalled → dead + attempt burned.
await queue.claim('tok-ev-3', 30_000, 'default', ['evict-accumulate']);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`,
[job.id]
);
const finalSweep = await queue.handleStalled();
expect(finalSweep.dead.map(j => j.id)).toContain(job.id);
const final = await queue.getJob(job.id);
expect(final!.status).toBe('dead');
expect(final!.attempts_made).toBe(1);
expect(final!.error_text).toBe('max stalled count exceeded');
});
});
+2 -2
View File
@@ -52,7 +52,7 @@ test('stall-exhausted child → child_done(dead) in parent inbox + parent unbloc
const claimed = await queue.claim('tok-stall', 30_000, 'default', ['child', 'aggregator']);
expect(claimed?.id).toBe(child.id);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`, [child.id],
);
const { requeued, dead } = await queue.handleStalled();
@@ -85,7 +85,7 @@ test('stall-requeued child (budget left) does NOT touch the parent', async () =>
const claimed = await queue.claim('tok-stall2', 30_000, 'default', ['child', 'aggregator']);
expect(claimed?.id).toBe(child.id);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`, [child.id],
);
const { requeued, dead } = await queue.handleStalled();
+2 -2
View File
@@ -73,7 +73,7 @@ test("failJob's delayed branch clears started_at (terminal branches keep it)", a
test("handleStalled's requeue branch clears started_at", async () => {
const { id } = await addAndClaim();
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [id],
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`, [id],
);
const { requeued } = await queue.handleStalled();
expect(requeued.map(j => j.id)).toContain(id);
@@ -146,7 +146,7 @@ test('red-team 5th path: a re-claimed aggregator parent survives the wall-clock
// Child stalls to death → killJobs unblocks the parent.
await queue.claim('tok-wc', 30_000, 'default', ['child-wc', 'agg-wallclock']);
await engine.executeRaw(
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
`UPDATE minion_jobs SET lock_until = now() - interval '30 seconds' WHERE id = $1`, [child.id],
);
await queue.handleStalled();
expect(await jobStatus(parent.id)).toBe('waiting');
+12
View File
@@ -103,6 +103,18 @@ describe('relationalFanout', () => {
expect(a!.path[a!.path.length - 1]).toBe('people/investor-a');
});
test('equal-depth multi-seed tie picks the lexicographically-smallest path (deterministic winner)', async () => {
// people/investor-a is reachable at depth 1 from BOTH seeds; parity alone
// would pass if both engines agreed on a wrong-but-deterministic pick
// (e.g. an ORDER BY direction flip). Pin the WINNER: the final
// lexicographic tie-break must choose the smallest path string, whose
// first hop is 'companies/other-co' (< 'companies/widget-co').
const rows = await eng.relationalFanout(['companies/widget-co', 'companies/other-co'], { direction: 'both' });
const a = rows.find(r => r.slug === 'people/investor-a');
expect(a).toBeDefined();
expect(a!.path[0]).toBe('companies/other-co');
});
test('empty seeds → []', async () => {
expect(await eng.relationalFanout([])).toEqual([]);
});
+31 -4
View File
@@ -41,7 +41,23 @@ beforeAll(async () => {
afterAll(async () => {
await engine.disconnect();
rmSync(dir, { recursive: true, force: true });
// Best-effort tmpdir cleanup with one retry. Bun's recursive rmSync has
// EFAULT'd here on CI (bun 1.3.13, ubuntu-24.04) immediately after the
// WASM engine teardown — a runtime flake, not a test failure. bun treats
// a throwing afterAll as an "(unnamed)" failed test and reds the shard;
// the OS reaps tmpdir anyway, so cleanup must never fail the suite.
try {
rmSync(dir, { recursive: true, force: true });
} catch {
await new Promise((r) => setTimeout(r, 50));
try {
rmSync(dir, { recursive: true, force: true });
} catch (e) {
console.warn(
`[run-child-entry.test] tmpdir cleanup failed (non-fatal): ${e instanceof Error ? e.message : String(e)}`,
);
}
}
});
beforeEach(async () => {
@@ -168,6 +184,7 @@ describe('runChildJobEntry', () => {
const resultPath = join(dir, `sigterm-${job.id}.json`);
let ctxSignalAbortedAtShutdown: boolean | null = null;
const listenersBefore = new Set(process.listeners('SIGTERM'));
const entry = runChildJobEntry(
engine,
{ jobId: job.id, lockToken: 'parent-tok-1', resultPath, parentPid: 0 },
@@ -184,9 +201,19 @@ describe('runChildJobEntry', () => {
}),
);
await new Promise((r) => setTimeout(r, 100));
// Trigger the entry's process.on('SIGTERM') handler in-process without
// sending a real signal to the test runner.
(process as unknown as { emit: (event: string) => boolean }).emit('SIGTERM');
// Trigger ONLY the SIGTERM listener(s) the entry registered. A bare
// process.emit('SIGTERM') broadcasts to EVERY listener in the shared
// bun test process — including process-cleanup.ts's leaked handler,
// whose runCleanupPass().finally(process.exit(143)) kills the whole
// shard mid-suite when an earlier file installed it. That exit code
// reads as rc=143 and gets misclassified as an "external kill" by
// run-unit-parallel.sh (bit three consecutive suite runs under host
// load before being traced here).
const added = process
.listeners('SIGTERM')
.filter((l) => !listenersBefore.has(l));
expect(added.length).toBeGreaterThan(0);
for (const l of added) (l as (...args: unknown[]) => void)('SIGTERM');
const code = await entry;
expect(code).toBe(0);
+58
View File
@@ -0,0 +1,58 @@
/**
* Guard for the evals/ CI-matrix collection (scripts/test-shard.sh).
*
* evals/**\/*.test.ts files run in the keyless 10-shard matrix. This repo's
* eval HARNESSES are key-requiring by default (Anthropic/OpenAI), so a new
* test file dropped under evals/ could silently start spending tokens in CI
* or fail keyless. Growth is therefore allowlist-gated: every collected
* evals file must be named here, and adding one asserts you checked it runs
* with NO API keys and NO network (mirror of the serial runner's
* EXCLUSIVE_FILES guard).
*/
import { describe, expect, it } from "bun:test";
import { execFileSync } from "child_process";
import { resolve } from "path";
const REPO_ROOT = resolve(import.meta.dir, "..", "..");
// Keyless-verified evals test files. Verify before adding:
// env -u ANTHROPIC_API_KEY -u OPENAI_API_KEY bun test <file>
const KEYLESS_ALLOWLIST = new Set([
// pure-function scoring/CI-parsing helpers; imports no gateway (verified)
"evals/functional-area-resolver/harness-runner.test.ts",
]);
describe("evals/ collection into the CI matrix", () => {
const collected = (): string[] => {
const lists: string[] = [];
// Union across all shards = the full collected set.
for (let i = 1; i <= 10; i++) {
const out = execFileSync(
"bash",
["scripts/test-shard.sh", "--dry-run-list", String(i), "10"],
{ cwd: REPO_ROOT, encoding: "utf-8" },
);
lists.push(out);
}
return lists
.join("\n")
.split("\n")
.map((s) => s.trim())
.filter((f) => f.startsWith("evals/"));
};
it("every collected evals file is on the keyless allowlist", () => {
const files = collected();
expect(files.length).toBeGreaterThan(0); // the collection itself works
const unlisted = files.filter((f) => !KEYLESS_ALLOWLIST.has(f));
expect(unlisted).toEqual([]);
});
it("every allowlisted file is actually collected (no dead entries)", () => {
const files = new Set(collected());
for (const f of KEYLESS_ALLOWLIST) {
expect(files.has(f)).toBe(true);
}
});
});
+175
View File
@@ -0,0 +1,175 @@
/**
* Behavioral tests for scripts/run-serial-tests.sh's POOLED execution:
*
* 1. All-pass: pooled files run concurrently, one-line PASS summaries,
* exit 0.
* 2. One failing file: exit 1, full log echoed, failed-files summary.
* 3. Hung file: killed by the per-file wall-clock timeout (exit 124/137
* surfaced with a timeout note) the exit-hang class containment.
* Skipped when no timeout/gtimeout binary exists on the host.
*
* The missing-sentinel(=failure) and EXCLUSIVE_FILES growth guards are
* source-pinned in test/scripts/serial-files.test.ts; these tests exercise
* the live pool in a minimal-PATH sandbox (same pattern as
* run-unit-parallel.test.ts).
*/
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import { execFileSync } from 'child_process';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, copyFileSync, chmodSync, symlinkSync } from 'fs';
import { tmpdir } from 'os';
import { dirname, join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
let ROOT: string;
let ENV: Record<string, string>;
let hasTimeoutBin = false;
function stageSandbox(): string {
const root = mkdtempSync(join(tmpdir(), 'gbrain-serial-pool-'));
mkdirSync(join(root, 'scripts', 'lib'), { recursive: true });
mkdirSync(join(root, 'test'), { recursive: true });
for (const s of ['run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(root, 'scripts', s)), { recursive: true });
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(root, 'scripts', s));
}
chmodSync(join(root, 'scripts', 'run-serial-tests.sh'), 0o755);
const bin = join(root, 'bin');
mkdirSync(bin);
const tools = [
'bash', 'sh', 'env', 'dirname', 'basename', 'mktemp', 'date', 'sleep',
'cat', 'tail', 'head', 'rm', 'mkdir', 'grep', 'sed', 'awk', 'wc', 'tr',
'find', 'sort', 'bun', 'timeout', 'gtimeout',
];
for (const tool of tools) {
const p = Bun.which(tool);
if (p) {
symlinkSync(p, join(bin, tool));
if (tool === 'timeout' || tool === 'gtimeout') hasTimeoutBin = true;
}
}
return root;
}
function runScript(extraEnv: Record<string, string> = {}): { code: number; out: string } {
try {
const out = execFileSync('bash', [join(ROOT, 'scripts', 'run-serial-tests.sh')], {
cwd: ROOT,
encoding: 'utf-8',
env: { ...ENV, ...extraEnv },
stdio: ['ignore', 'pipe', 'pipe'],
});
return { code: 0, out };
} catch (err) {
const e = err as { status?: number; stdout?: string; stderr?: string };
return { code: e.status ?? -1, out: `${e.stdout ?? ''}${e.stderr ?? ''}` };
}
}
const PASSING = `import { describe, it, expect } from 'bun:test';
describe('passing', () => { it('works', () => { expect(1 + 1).toBe(2); }); });`;
const FAILING = `import { describe, it, expect } from 'bun:test';
describe('failing', () => { it('POOL_SENTINEL_ASSERTION breaks', () => { expect(1).toBe(2); }); });`;
const HANGING = `import { it } from 'bun:test';
it('hangs forever', async () => { await new Promise(() => {}); });`;
beforeAll(() => {
ROOT = stageSandbox();
ENV = {
PATH: join(ROOT, 'bin'),
HOME: process.env.HOME ?? ROOT,
TMPDIR: process.env.TMPDIR ?? '/tmp',
// Sandbox has no package.json — skip the snapshot build path entirely.
GBRAIN_NO_SNAPSHOT: '1',
GBRAIN_SERIAL_POOL: '2',
};
});
afterAll(() => {
rmSync(ROOT, { recursive: true, force: true });
});
describe('pooled serial runner', () => {
it('runs pooled files and passes with one-line summaries', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
writeFileSync(join(ROOT, 'test', 'b-ok.serial.test.ts'), PASSING);
const r = runScript();
expect(r.code).toBe(0);
expect(r.out).toContain('PASS');
expect(r.out).toContain('test/a-ok.serial.test.ts');
expect(r.out).toContain('test/b-ok.serial.test.ts');
expect(r.out).toContain('all 2 file(s) passed');
expect(r.out).toContain('pool=2');
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
rmSync(join(ROOT, 'test', 'b-ok.serial.test.ts'));
});
it('a failing file fails the run with its full log and a failed-files summary', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
writeFileSync(join(ROOT, 'test', 'z-bad.serial.test.ts'), FAILING);
const r = runScript();
expect(r.code).toBe(1);
// Full bun log of the failing file is echoed (its assertion name shows).
expect(r.out).toContain('POOL_SENTINEL_ASSERTION');
expect(r.out).toContain('1 file(s) failed');
expect(r.out).toContain('test/z-bad.serial.test.ts');
// The passing sibling still reports PASS (pool completes, no fail-fast).
expect(r.out).toContain('PASS');
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
rmSync(join(ROOT, 'test', 'z-bad.serial.test.ts'));
});
it('--dry-run-list lists every serial file without running anything', () => {
writeFileSync(join(ROOT, 'test', 'a-ok.serial.test.ts'), PASSING);
const out = execFileSync(
'bash',
[join(ROOT, 'scripts', 'run-serial-tests.sh'), '--dry-run-list'],
{ cwd: ROOT, encoding: 'utf-8', env: ENV },
);
expect(out.trim().split('\n')).toEqual(['test/a-ok.serial.test.ts']);
rmSync(join(ROOT, 'test', 'a-ok.serial.test.ts'));
});
it('an externally-SIGTERMed file is rescued by a sequential re-run (phantom stays green)', () => {
// Self-kills with SIGTERM on first run (exit 143 — the external-kill
// class: sibling-workspace cleanup, memory jetsam), passes on the
// rescue re-run. Mirrors run-unit-parallel's oom-once fixture.
const sentinel = join(ROOT, 'test', 'killed-once.sentinel');
const KILLED_ONCE = `import { it, expect } from 'bun:test';
import { existsSync, writeFileSync } from 'fs';
it('passes after one external SIGTERM', () => {
const sentinel = ${JSON.stringify(sentinel)};
if (!existsSync(sentinel)) {
writeFileSync(sentinel, '1');
process.kill(process.pid, 'SIGTERM');
}
expect(1).toBe(1);
});`;
writeFileSync(join(ROOT, 'test', 'k-killed.serial.test.ts'), KILLED_ONCE);
try {
const r = runScript();
// (The "queued for serial rescue" line goes to stderr, which the
// success path of runScript doesn't capture — the stdout rescue
// marker + exit 0 are the contract.)
expect(r.out).toContain('rescued: external-kill phantom');
expect(r.code).toBe(0);
} finally {
rmSync(join(ROOT, 'test', 'k-killed.serial.test.ts'), { force: true });
rmSync(sentinel, { force: true });
}
}, 60000);
it('a hung file is killed by the per-file wall-clock timeout', () => {
if (!hasTimeoutBin) return; // macOS without coreutils: no wrapper, documented
writeFileSync(join(ROOT, 'test', 'h-hang.serial.test.ts'), HANGING);
const r = runScript({ GBRAIN_SERIAL_FILE_TIMEOUT: '3' });
expect(r.code).toBe(1);
expect(r.out).toContain('per-file timeout');
expect(r.out).toContain('test/h-hang.serial.test.ts');
rmSync(join(ROOT, 'test', 'h-hang.serial.test.ts'));
}, 60000);
});
+10 -4
View File
@@ -23,12 +23,15 @@ import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
import { execFileSync, spawnSync } from 'child_process';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync, copyFileSync, chmodSync, symlinkSync } from 'fs';
import { tmpdir } from 'os';
import { join, resolve } from 'path';
import { dirname, join, resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
const PARALLEL_SH_SRC = resolve(REPO_ROOT, 'scripts/run-unit-parallel.sh');
const SHARD_SH_SRC = resolve(REPO_ROOT, 'scripts/run-unit-shard.sh');
const SERIAL_SH_SRC = resolve(REPO_ROOT, 'scripts/run-serial-tests.sh');
// The runners `source scripts/lib/test-env.sh` — every sandbox copy of a
// runner must stage the lib too or the source line fails at startup.
const TESTENV_SH_SRC = resolve(REPO_ROOT, 'scripts/lib/test-env.sh');
let TMPROOT: string;
@@ -37,8 +40,9 @@ beforeAll(() => {
// and 4 fixture test files (3 pass, 1 fail). The wrapper's `find test`
// expression will pick them up via cwd.
TMPROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-test-'));
mkdirSync(join(TMPROOT, 'scripts'), { recursive: true });
mkdirSync(join(TMPROOT, 'scripts', 'lib'), { recursive: true });
mkdirSync(join(TMPROOT, 'test'), { recursive: true });
copyFileSync(TESTENV_SH_SRC, join(TMPROOT, 'scripts', 'lib', 'test-env.sh'));
copyFileSync(PARALLEL_SH_SRC, join(TMPROOT, 'scripts', 'run-unit-parallel.sh'));
copyFileSync(SHARD_SH_SRC, join(TMPROOT, 'scripts', 'run-unit-shard.sh'));
@@ -190,7 +194,8 @@ describe('run-unit-parallel.sh no-timeout-binary fallback (rc from shard wait, n
FROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-fallback-'));
mkdirSync(join(FROOT, 'scripts'), { recursive: true });
mkdirSync(join(FROOT, 'test'), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh']) {
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(FROOT, 'scripts', s)), { recursive: true });
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(FROOT, 'scripts', s));
chmodSync(join(FROOT, 'scripts', s), 0o755);
}
@@ -275,7 +280,8 @@ describe('run-unit-parallel.sh OOM rescue lane', () => {
OROOT = mkdtempSync(join(tmpdir(), 'gbrain-parallel-oom-'));
mkdirSync(join(OROOT, 'scripts'), { recursive: true });
mkdirSync(join(OROOT, 'test'), { recursive: true });
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh']) {
for (const s of ['run-unit-parallel.sh', 'run-unit-shard.sh', 'run-serial-tests.sh', 'lib/test-env.sh']) {
mkdirSync(dirname(join(OROOT, 'scripts', s)), { recursive: true });
copyFileSync(resolve(REPO_ROOT, 'scripts', s), join(OROOT, 'scripts', s));
chmodSync(join(OROOT, 'scripts', s), 0o755);
}
+50 -2
View File
@@ -61,6 +61,52 @@ describe("run-verify-parallel.sh — CLI contract", () => {
});
});
describe("guard registration ⇒ execution coverage", () => {
// guard-self-test.sh enforces that every scripts/check-* guard is
// REGISTERED in guards-manifest.tsv, but nothing enforced that a
// registered guard actually EXECUTES anywhere — five guards sat
// registered-but-dead until the v0.45.x test/eval/CI pass. This closes
// the loop: every manifest guard must be reachable from verify's CHECKS
// (via a package.json script that invokes its file), or be explicitly
// exempted HERE with the reason it runs elsewhere / deliberately not.
const EXECUTION_EXEMPT: Record<string, string> = {
"check-bun-test-timeout.sh":
"runs directly as a test.yml verify-job step (not via CHECKS — avoids a package.json edit)",
"check-jsonb-params.mjs":
"exercised by test/check-jsonb-params.test.ts + guard self-test fixtures",
"check-admin-embedded.sh":
"duplicates check:admin-build's vite+tsc build; embed freshness covered there",
"check-image-decoders-embedded.sh":
"runs its own bun build --compile — too heavy for per-verify cadence",
};
it("every manifest guard is executed by verify or explicitly exempt", () => {
const manifestLines = readFileSync("scripts/guards-manifest.tsv", "utf8")
.split("\n")
.filter((l) => l.trim() && !l.startsWith("#"));
const guards = manifestLines.map((l) => l.split("\t")[0]!).filter(Boolean);
expect(guards.length).toBeGreaterThan(30);
const pkg = JSON.parse(readFileSync("package.json", "utf8")) as {
scripts: Record<string, string>;
};
const dry = spawnSync("bash", [SCRIPT, "--dry-list"], { encoding: "utf8" });
expect(dry.status).toBe(0);
const executed = new Set(dry.stdout.trim().split("\n"));
const missing: string[] = [];
for (const g of guards) {
if (EXECUTION_EXEMPT[g]) continue;
const invokingKeys = Object.entries(pkg.scripts)
.filter(([, cmd]) => cmd.includes(`scripts/${g}`))
.map(([key]) => key);
const covered = invokingKeys.some((k) => executed.has(k));
if (!covered) missing.push(g);
}
expect(missing).toEqual([]);
});
});
describe("run-verify-parallel.sh — failure surfacing (synthetic dispatcher)", () => {
// We can't inject a fake check into the real script without touching the
// CHECKS array. Instead, we write a SMALLER synthetic dispatcher that
@@ -203,13 +249,15 @@ describe("run-verify-parallel.sh — no-timeout-binary fallback rc capture (regr
function makeFallbackHarness(): { root: string; env: Record<string, string> } {
const root = mkdtempSync(join(tmpdir(), "verify-fallback-"));
mkdirSync(join(root, "scripts"), { recursive: true });
mkdirSync(join(root, "scripts", "lib"), { recursive: true });
copyFileSync(SCRIPT, join(root, "scripts", "run-verify-parallel.sh"));
// The dispatcher sources the shared runner lib — stage it too (E3).
copyFileSync("scripts/lib/test-env.sh", join(root, "scripts", "lib", "test-env.sh"));
const bin = join(root, "bin");
mkdirSync(bin);
// Everything the dispatcher and its subshells invoke, minus timeout bins.
for (const tool of ["bash", "sh", "env", "dirname", "mktemp", "date", "sleep", "cat", "tail", "head", "rm", "mkdir", "pkill", "grep", "sed", "awk"]) {
for (const tool of ["bash", "sh", "env", "dirname", "mktemp", "date", "sleep", "cat", "tail", "head", "rm", "mkdir", "pkill", "grep", "sed", "awk", "wc", "tr"]) {
const p = Bun.which(tool);
if (p) symlinkSync(p, join(bin, tool));
}
+29 -1
View File
@@ -18,7 +18,7 @@
import { describe, it, expect } from 'bun:test';
import { execFileSync } from 'child_process';
import { readFileSync } from 'fs';
import { existsSync, readFileSync } from 'fs';
import { resolve } from 'path';
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
@@ -68,3 +68,31 @@ describe('run-serial-tests.sh contract', () => {
expect(overlap).toEqual([]);
});
});
describe('EXCLUSIVE_FILES (pooled-runner opt-out) guards', () => {
const src = () => readFileSync(SERIAL_SH, 'utf-8');
const entries = () =>
[...src().matchAll(/^\s*"(test\/[^"]+\.serial\.test\.ts)"\s*$/gm)].map(m => m[1]);
it('every exclusive entry exists on disk and is discovered by the runner', () => {
const listed = dryRunList(SERIAL_SH);
const e = entries();
expect(e.length).toBeGreaterThan(0);
for (const f of e) {
expect(existsSync(resolve(REPO_ROOT, f))).toBe(true);
expect(listed).toContain(f);
}
});
it('the exclusive list does not silently grow (quarantine-growth guard)', () => {
// Exclusivity re-serializes the runner one file at a time — the exact
// 8.5-minute disease the pool removed. Each entry must carry a
// justification comment; past 3 entries, stop and rethink the design
// (per-file PATH shims are usually the right fix) instead of quarantining.
expect(entries().length).toBeLessThanOrEqual(3);
});
it('a missing exit sentinel is a failure, never a silent pass', () => {
expect(src()).toMatch(/missing exit sentinel/);
});
});
+20 -7
View File
@@ -7,6 +7,7 @@
import { describe, expect, it } from "bun:test";
import {
computeMedian,
computeQuantile,
imbalanceRatio,
loadWeights,
partition,
@@ -83,20 +84,32 @@ describe("partition — happy path", () => {
});
describe("partition — fallback semantics", () => {
it("missing weights default to corpus median", () => {
// weights = {a:100, b:50}, median = 75. Files c + d are unknown → 75 each.
it("missing weights default to corpus p75 (missing files skew heavy)", () => {
// weights = {a:100, b:50}, p75 (nearest-rank of [50,100] at 0.75) = 100.
// Files c + d are unknown → 100 each.
const weights: WeightMap = new Map([
["a", 100],
["b", 50],
]);
const out = partition(["a", "b", "c", "d"], weights, 2);
// Effective weights: a=100, b=50, c=75, d=75. LPT: 100→s0, 75→s1 (c),
// 75→s1 (d, ties broken alpha)... actually: 100→s0=100, 75→s1=75,
// 75→s1 vs s0 → s1=150, 50→s0=150. Balanced 150/150.
// Effective weights: a=100, b=50, c=100, d=100. Exact LPT placement of
// ties is implementation-defined — assert the invariant instead: totals
// differ by ≤ the fallback (the LPT bound), and every file landed once.
const totalsEffective = out.map((s) =>
s.reduce((acc, f) => acc + (weights.get(f) ?? 75), 0),
s.reduce((acc, f) => acc + (weights.get(f) ?? 100), 0),
);
expect(totalsEffective[0]).toBe(totalsEffective[1]);
expect(Math.abs(totalsEffective[0]! - totalsEffective[1]!)).toBeLessThanOrEqual(100);
const flat = out.flat().sort();
expect(flat).toEqual(["a", "b", "c", "d"]);
});
it("computeQuantile: nearest-rank p75 on a skewed corpus", () => {
// Right-skewed like the real weights file: p75 lands in the tail's
// foothills, far above the median.
expect(computeQuantile([1, 1, 1, 1000], 0.75)).toBe(1);
expect(computeQuantile([1, 2, 3, 4], 0.75)).toBe(3);
expect(computeQuantile([50, 100], 0.75)).toBe(100);
expect(computeQuantile([], 0.75)).toBe(0);
});
it("explicit fallback override beats median", () => {
+49 -14
View File
@@ -130,33 +130,68 @@ describe('test-shard.sh — LPT balance contract', () => {
});
function totalsFor(shards: string[][]): number[] {
// Use 30ms as the cold-start fallback (matches mine-shard-weights
// median observation). When weights are loaded, missing files get
// the corpus median anyway via sharding.ts.
// p75 fallback mirroring sharding.ts's computeQuantile choice (the
// weight distribution is right-skewed and missing files skew heavy).
const sorted = Array.from(weightsMap.values()).sort((a, b) => a - b);
const fallback = weightsLoaded
? Array.from(weightsMap.values()).sort((a, b) => a - b)[
Math.floor(weightsMap.size / 2)
] ?? 30
? sorted[Math.min(sorted.length - 1, Math.ceil(0.75 * sorted.length) - 1)] ?? 30
: 1;
return shards.map((s) =>
s.reduce((acc, f) => acc + (weightsMap.get(f) ?? fallback), 0),
);
}
it('4-shard wallclock imbalance ratio ≤ 1.5', () => {
const shards = [1, 2, 3, 4].map(s => dryRunList(s, 4));
for (const s of shards) expect(s.length).toBeGreaterThan(0);
const totals = totalsFor(shards);
const ratio = Math.max(...totals) / Math.min(...totals);
expect(ratio).toBeLessThanOrEqual(1.5);
// CI runs THIS many shards (test.yml matrix). The old version of this
// test asserted 4- and 6-shard splits — configurations nothing runs.
const CI_SHARDS = 10;
it('CI_SHARDS matches the test.yml matrix (parsed, not regexed)', () => {
const fs = require('fs');
const yaml = require('js-yaml');
const wf = yaml.load(
fs.readFileSync(resolve(REPO_ROOT, '.github/workflows/test.yml'), 'utf8'),
) as { jobs: { test: { strategy: { matrix: { shard: unknown[] } } } } };
const matrix = wf.jobs.test.strategy.matrix.shard;
expect(Array.isArray(matrix)).toBe(true);
expect(matrix.length).toBe(CI_SHARDS);
});
it('6-shard wallclock imbalance ratio ≤ 1.5', () => {
const shards = [1, 2, 3, 4, 5, 6].map(s => dryRunList(s, 6));
it(`${CI_SHARDS}-shard wallclock imbalance ratio ≤ 1.5 (the configuration CI actually runs)`, () => {
const shards = Array.from({ length: CI_SHARDS }, (_, i) => dryRunList(i + 1, CI_SHARDS));
for (const s of shards) expect(s.length).toBeGreaterThan(0);
const totals = totalsFor(shards);
const ratio = Math.max(...totals) / Math.min(...totals);
expect(ratio).toBeLessThanOrEqual(1.5);
}, 60_000);
// The two guards below are what make the ratio assertion above MEAN
// something: recomputing totals with the same weights the partitioner
// used is near-tautological — unless the weights actually cover the
// corpus and refer to real files. Weight rot (files added without a
// re-mine, or renamed away from their entries) used to be invisible:
// 45% of the corpus once rode a 30ms median fallback while really
// averaging ~4s, and the "balanced" partition was balanced on fiction.
it('weights cover ≥70% of matrix-eligible files (anti-rot gate)', () => {
const all = Array.from(
new Set(Array.from({ length: CI_SHARDS }, (_, i) => dryRunList(i + 1, CI_SHARDS)).flat()),
);
expect(all.length).toBeGreaterThan(0);
const covered = all.filter((f) => weightsMap.has(f)).length;
const coverage = covered / all.length;
// Regenerate from the latest green Test run:
// bun run scripts/mine-shard-weights.ts --run <run id>
expect(coverage).toBeGreaterThanOrEqual(0.7);
}, 60_000);
it('no stale weight keys — every entry names a tracked file', () => {
const tracked = new Set(
execFileSync('git', ['ls-files', 'test', 'evals'], { cwd: REPO_ROOT, encoding: 'utf-8' })
.split('\n')
.map((s) => s.trim())
.filter(Boolean),
);
const stale = [...weightsMap.keys()].filter((k) => !tracked.has(k));
expect(stale).toEqual([]);
});
it('6-shard partition is deterministic across runs', () => {
+42 -1
View File
@@ -12,7 +12,12 @@ import { mkdtempSync, rmSync, writeFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import * as crypto from 'node:crypto';
import { tryLoadSnapshot, computeSnapshotSchemaHash } from '../src/core/pglite-engine.ts';
import {
tryLoadSnapshot,
computeSnapshotSchemaHash,
__snapshotMemoStatsForTests,
__resetSnapshotMemoForTests,
} from '../src/core/pglite-engine.ts';
import { MIGRATIONS } from '../src/core/migrate.ts';
import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts';
import { getEmbeddingDimensions, getEmbeddingModel } from '../src/core/ai/gateway.ts';
@@ -21,6 +26,7 @@ let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'gbrain-snap-guard-'));
__resetSnapshotMemoForTests();
});
afterEach(() => {
@@ -63,6 +69,41 @@ test('matching hash + shape loads the blob', () => {
expect(blob!.size).toBeGreaterThan(0);
});
test('memo: same path is read once per process, blob identical across calls', () => {
const tar = writeFixture(`${currentHash()}\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
const b1 = tryLoadSnapshot(tar);
const afterFirst = __snapshotMemoStatsForTests().tarReads;
const b2 = tryLoadSnapshot(tar);
const afterSecond = __snapshotMemoStatsForTests().tarReads;
expect(b1).not.toBeNull();
expect(b2).toBe(b1); // same Blob instance — the tar was not re-read
expect(afterFirst).toBe(1);
expect(afterSecond).toBe(1);
});
test('memo: shape refusal is per-call, never cached as terminal — and costs zero tar reads', () => {
// Hash matches but dims mismatch: the version entry is memoized yet every
// call re-runs the shape gate against the CURRENT gateway config — an
// engine with a matching config later in the same process could still
// load this snapshot (the zembed/1280 poisoning guard staying hot behind
// the memo). The 42MB tar read is deferred until a shape-MATCHING caller,
// so a process that only ever refuses never reads it at all.
const tar = writeFixture(`${currentHash()}\ndims=99999\nmodel=${getEmbeddingModel()}\n`);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
expect(__snapshotMemoStatsForTests().memoEntries).toBe(1); // entry exists — not terminal
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
});
test('memo: stale hash is terminal — tar never read, repeat calls short-circuit', () => {
const tar = writeFixture(`deadbeef\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
expect(tryLoadSnapshot(tar)).toBeNull();
expect(__snapshotMemoStatsForTests().tarReads).toBe(0);
});
test('D5.13: a migration handler edit changes the hash (sql-only hashing missed 19 handler migrations)', () => {
const base = [
{ version: 1, name: 'a', sql: 'CREATE TABLE t(x int)' },
+10 -5
View File
@@ -144,9 +144,11 @@ describe('G: tickInFlight re-entrancy guard', () => {
test('the setInterval callback checks tickInFlight and bails on re-entry', () => {
const launchJobBody = extractFunctionBody(workerSource, 'private launchJob(');
// The pattern is `if (tickInFlight) return;` — minor whitespace
// variation tolerated.
expect(launchJobBody).toMatch(/if\s*\(\s*tickInFlight\s*\)\s*return/);
// v0.46 (#4145): the guard grew a block body — it counts the overlap
// skip (CDX-13 telemetry) before bailing. The load-bearing shape is
// unchanged: check tickInFlight, return WITHOUT scheduling a tick.
expect(launchJobBody).toMatch(/if\s*\(\s*tickInFlight\s*\)\s*\{[^}]*return/);
expect(launchJobBody).toMatch(/overlapSkips\s*\+=\s*1/);
});
test('the setInterval callback sets tickInFlight=true before scheduling work', () => {
@@ -221,8 +223,11 @@ describe('E: universal grace-evict listener (D8b)', () => {
// Find the addEventListener and look in its function body.
const listenerIdx = launchJobBody.indexOf("abort.signal.addEventListener('abort'");
expect(listenerIdx).toBeGreaterThan(-1);
// Grab ~1500 chars after the listener to capture its body.
const listenerWindow = launchJobBody.slice(listenerIdx, listenerIdx + 1500);
// Grab ~4500 chars after the listener to capture its body (v0.46:
// the grace-evict log gained the #4145 abortMeta/telemetry block AND
// the #4151 isolation-mode failJob guard commentary, both of which sit
// between the listener head and the 30_000 literal).
const listenerWindow = launchJobBody.slice(listenerIdx, listenerIdx + 4500);
expect(listenerWindow).toMatch(/30_000|30000/);
});
});
+350 -45
View File
@@ -38,9 +38,9 @@ import { withEnv } from './helpers/with-env.ts';
// --- fakes ----------------------------------------------------------------
interface AuditLog {
failures: Array<{ jobId: number; jobName: string; attempt: number; err: unknown }>;
recoveries: Array<{ jobId: number; jobName: string; recoveredAfterAttempts: number }>;
gaveUps: Array<{ jobId: number; jobName: string; totalFailures: number; err: unknown }>;
failures: Array<{ jobId: number; jobName: string; attempt: number; err: unknown; ctx?: unknown }>;
recoveries: Array<{ jobId: number; jobName: string; recoveredAfterAttempts: number; ctx?: unknown }>;
gaveUps: Array<{ jobId: number; jobName: string; totalFailures: number; err: unknown; ctx?: unknown }>;
}
function freshAudit(): { sink: LockRenewalAuditSinkLike; log: AuditLog } {
@@ -48,12 +48,12 @@ function freshAudit(): { sink: LockRenewalAuditSinkLike; log: AuditLog } {
return {
log,
sink: {
logFailure: (jobId, jobName, attempt, err) =>
log.failures.push({ jobId, jobName, attempt, err }),
logSuccessAfterFailure: (jobId, jobName, recoveredAfterAttempts) =>
log.recoveries.push({ jobId, jobName, recoveredAfterAttempts }),
logGaveUp: (jobId, jobName, totalFailures, err) =>
log.gaveUps.push({ jobId, jobName, totalFailures, err }),
logFailure: (jobId, jobName, attempt, err, ctx) =>
log.failures.push({ jobId, jobName, attempt, err, ctx }),
logSuccessAfterFailure: (jobId, jobName, recoveredAfterAttempts, ctx) =>
log.recoveries.push({ jobId, jobName, recoveredAfterAttempts, ctx }),
logGaveUp: (jobId, jobName, totalFailures, err, ctx) =>
log.gaveUps.push({ jobId, jobName, totalFailures, err, ctx }),
},
};
}
@@ -88,6 +88,7 @@ const DEFAULT_KNOBS: LockRenewalKnobs = {
maxFailuresForAudit: 3,
callTimeoutMs: 10_000,
safetyMarginMs: 5_000,
hardEvictMs: 60_000, // 2 × lockDuration (the #4145 backstop)
};
function makeState(overrides?: Partial<LockRenewalState>): LockRenewalState {
@@ -100,6 +101,11 @@ function makeState(overrides?: Partial<LockRenewalState>): LockRenewalState {
lastSuccessfulRenewalAt: 0,
consecutiveFailures: 0,
cancelled: () => false,
// v0.46 (#4145) telemetry fields: cadence for lateness math, seeded
// previous-tick timestamp, and the worker-maintained overlap counter.
intervalMs: DEFAULT_LOCK_MS / 2,
lastTickFiredAt: 0,
overlapSkips: 0,
...overrides,
};
}
@@ -178,10 +184,12 @@ describe('runLockRenewalTick: failure counter + audit', () => {
});
});
describe('runLockRenewalTick: time-based abort', () => {
test('case 4 — sustained throws past deadline returns should_abort, gave_up logged', async () => {
describe('runLockRenewalTick: verify-before-evict + hard backstop (#4145)', () => {
test('case 4 — sustained throws past the SOFT deadline are DEFERRED (verify attempted), abort only past hardEvictMs', async () => {
const audit = freshAudit();
// deadline = 30000 - 5000 = 25000; we've been failing for 26s.
// Soft deadline = 30000 - 5000 = 25000; hardEvict = 60000.
// At 26s of failure the pre-#4145 code aborted — the incident. Now the
// tick attempts a verify (which also throws here) and DEFERS.
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('persistent outage'); },
audit: audit.sink,
@@ -189,46 +197,177 @@ describe('runLockRenewalTick: time-based abort', () => {
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0, consecutiveFailures: 2 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'should_abort', reason: 'lock-renewal-failed' });
expect(audit.log.failures).toHaveLength(1);
const deferred = await runLockRenewalTick(deps, state);
expect(deferred).toEqual({ kind: 'ok' });
// Primary attempt (3) + failed verify (4), the latter deadline_deferred.
expect(state.consecutiveFailures).toBe(4);
expect(audit.log.failures).toHaveLength(2);
expect(audit.log.failures[1].ctx).toMatchObject({ deadline_deferred: true, cause: 'refused' });
expect(audit.log.gaveUps).toHaveLength(0);
// Past the hard backstop the tick gives up: primary (5) + verify (6),
// gave_up carries the classified cause + starvation telemetry.
// lateness = max(0, 61000 - 26000 - 15000) = 20000.
const hardDeps: LockRenewalDeps = { ...deps, now: () => 61_000 };
const result = await runLockRenewalTick(hardDeps, state);
expect(result).toMatchObject({
kind: 'should_abort',
reason: 'lock-renewal-failed',
cause: 'refused',
sinceLastSuccessMs: 61_000,
latenessMs: 20_000,
overlapSkips: 0,
});
expect(state.consecutiveFailures).toBe(6);
expect(audit.log.gaveUps).toHaveLength(1);
expect(audit.log.gaveUps[0].totalFailures).toBe(3);
expect(audit.log.gaveUps[0].totalFailures).toBe(6);
});
test('case 10 — time-based abort fires BEFORE count-based threshold', async () => {
// Critical regression: deadline at 25s, 5 failures over 30s.
// count-based (3-strike) would have aborted at failure #3 — but
// failure #3 happens at t=15s, well inside the 25s deadline.
// Time-based correctly waits until the deadline crosses.
test('case 10 — INCIDENT REPLAY: renewal times out under starvation, verify succeeds → job survives', async () => {
// The #4145 incident shape: the event loop starves, the renewal's
// 10s race timeout wins even though the DB is healthy (external
// SELECT 1 probes ran 32ms). Pre-fix: abort at the 25s deadline,
// discarding ~173s of LLM work. Post-fix: ONE fenced verify — the
// DB is the authority — recovers the lease and the job survives.
const audit = freshAudit();
let calls = 0;
const timer = makeFakeTimer();
const deps: LockRenewalDeps = {
renewLock: async () => {
calls++;
if (calls === 1) return new Promise<boolean>(() => { /* starved: never settles */ });
return true; // the verify — DB was healthy all along
},
audit: audit.sink,
now: () => 26_000, // past the soft deadline of 25000
setTimeout: timer.setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const pending = runLockRenewalTick(deps, state);
timer.runAll(); // fire the primary race timeout (the starvation symptom)
const result = await pending;
expect(result).toEqual({ kind: 'ok' }); // NO abort — the incident fix
expect(state.consecutiveFailures).toBe(0); // verify-success resets
expect(state.lastSuccessfulRenewalAt).toBe(26_000);
expect(audit.log.failures).toHaveLength(1); // the primary call-timeout
expect(audit.log.failures[0].ctx).toMatchObject({ cause: 'call-timeout' });
expect(audit.log.recoveries).toHaveLength(1);
expect(audit.log.recoveries[0].ctx).toMatchObject({ via: 'verify' });
expect(audit.log.gaveUps).toHaveLength(0);
});
test('case 10b — verify returns fenced-false: CERTAIN loss → lock_lost via verify', async () => {
const audit = freshAudit();
let calls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => {
calls++;
if (calls === 1) throw new Error('outage');
return false; // the stall sweep reclaimed the row during the blip
},
audit: audit.sink,
now: () => 26_000,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'lock_lost', cause: 'fenced-lost', via: 'verify' });
expect(audit.log.gaveUps).toHaveLength(0); // certain loss ≠ infrastructure gave-up
});
test('case 10c — CDX-4 cadence quantization: 300s lease / 60s cadence verifies at 240s with lease remaining', async () => {
// With the bare `>= deadline` gate there is NO tick between 240s and
// lease expiry (300s): deadline 270s is unreachable and the first
// eligible tick is already past expiry. The next-tick-too-late form
// (sinceLastSuccess + intervalMs >= deadline) verifies at 240s.
const audit = freshAudit();
const knobs: LockRenewalKnobs = {
maxFailuresForAudit: 3,
callTimeoutMs: 10_000,
safetyMarginMs: 5_000,
callTimeoutMs: 15_000,
safetyMarginMs: 30_000, // deadline = 270_000
hardEvictMs: 600_000,
};
let nowMs = 0;
let calls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('outage'); },
renewLock: async () => {
calls++;
if (calls === 1) throw new Error('blip at the 240s tick');
return true; // verify recovers with 60s of lease left
},
audit: audit.sink,
now: () => nowMs,
now: () => 240_000,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ knobs, lastSuccessfulRenewalAt: 0 });
const state = makeState({
lockDurationMs: 300_000,
knobs,
intervalMs: 60_000,
lastSuccessfulRenewalAt: 0,
lastTickFiredAt: 180_000,
});
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'ok' });
expect(calls).toBe(2); // primary + verify — the verify DID fire at 240s
expect(audit.log.recoveries[0].ctx).toMatchObject({ via: 'verify' });
// Five sequential failures at t=5, 10, 15, 20, 26.
for (const t of [5_000, 10_000, 15_000, 20_000]) {
nowMs = t;
const r = await runLockRenewalTick(deps, state);
expect(r.kind).toBe('ok'); // within deadline despite counter > maxFailuresForAudit
}
expect(state.consecutiveFailures).toBe(4);
expect(audit.log.gaveUps).toHaveLength(0);
// Control: at the 180s tick (180 + 60 = 240 < 270) no verify fires.
calls = 0;
const controlState = makeState({
lockDurationMs: 300_000,
knobs,
intervalMs: 60_000,
lastSuccessfulRenewalAt: 0,
lastTickFiredAt: 120_000,
});
const controlDeps: LockRenewalDeps = {
...deps,
renewLock: async () => { calls++; throw new Error('blip'); },
now: () => 180_000,
};
expect(await runLockRenewalTick(controlDeps, controlState)).toEqual({ kind: 'ok' });
expect(calls).toBe(1); // primary only — deferred to the next tick
});
nowMs = 26_000; // crosses deadline of 25000
const final = await runLockRenewalTick(deps, state);
expect(final).toEqual({ kind: 'should_abort', reason: 'lock-renewal-failed' });
expect(audit.log.gaveUps).toHaveLength(1);
test('case 10d — ENG-E1 pin: 30s default verifies on the FIRST failed tick (15s), strictly safer than abort-at-30s', async () => {
// interval 15s, deadline 25s: at the t=15s tick, 15000 + 15000 >= 25000
// → the verify fires immediately on the first failure. This is the
// accepted timing behavior — a renewal(≤10s) + verify(≤10s) pair can
// span 20s and skip one tick via tickInFlight; benign because a
// verify-success re-extends the lease.
let calls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => {
calls++;
if (calls === 1) throw new Error('first-tick blip');
return true;
},
audit: freshAudit().sink,
now: () => 15_000,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
expect(await runLockRenewalTick(deps, state)).toEqual({ kind: 'ok' });
expect(calls).toBe(2);
expect(state.lastSuccessfulRenewalAt).toBe(15_000);
});
test('case 10e — cancelled() flips mid-verify: returns cancelled, no abort', async () => {
let cancelled = false;
let calls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => {
calls++;
if (calls === 1) throw new Error('outage');
cancelled = true; // the job ends while the verify is in flight
return true;
},
audit: freshAudit().sink,
now: () => 26_000,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0, cancelled: () => cancelled });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'cancelled' });
});
});
@@ -243,7 +382,9 @@ describe('runLockRenewalTick: lock_lost (token mismatch)', () => {
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'lock_lost' });
// v0.46 (#4145): lock_lost names its cause — the fenced UPDATE matched
// 0 rows, the only CERTAIN loss signal.
expect(result).toEqual({ kind: 'lock_lost', cause: 'fenced-lost', via: 'renewal' });
expect(audit.log.failures).toHaveLength(0);
expect(audit.log.gaveUps).toHaveLength(0);
expect(audit.log.recoveries).toHaveLength(0);
@@ -374,12 +515,13 @@ describe('runLockRenewalTick: audit defense-in-depth (codex C4)', () => {
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('outage'); },
audit,
now: () => 26_000,
// Past hardEvictMs (60000) so the verify-throws path reaches gave_up.
now: () => 61_000,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'should_abort', reason: 'lock-renewal-failed' });
expect(result).toMatchObject({ kind: 'should_abort', reason: 'lock-renewal-failed' });
});
test('case 11c — audit.logSuccessAfterFailure throws: tick still returns ok, counter resets', async () => {
@@ -482,6 +624,42 @@ describe('resolveLockRenewalKnobs', () => {
// a deliberate two-line edit (default + this test).
expect(resolveLockRenewalKnobs({}, 30_000).maxFailuresForAudit).toBe(3);
});
test('case 15a — hardEvictMs defaults to 2×lockDuration; env override + floor to soft deadline (#4145)', () => {
_resetKnobWarningsForTests();
expect(resolveLockRenewalKnobs({}, 30_000).hardEvictMs).toBe(60_000);
expect(resolveLockRenewalKnobs({ GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS: '120000' }, 30_000).hardEvictMs).toBe(120_000);
// Below the soft deadline (25000) → warn-and-floor to the deadline.
const floored = resolveLockRenewalKnobs({ GBRAIN_LOCK_RENEWAL_HARD_EVICT_MS: '1000' }, 30_000);
expect(floored.hardEvictMs).toBe(25_000);
});
test('case 15b — CDX-10 relational validation: margin >= lease/2 and callTimeout > cadence are clamped', () => {
_resetKnobWarningsForTests();
// Margin 20000 >= 30000/2 → clamped back to the default (5000).
const m = resolveLockRenewalKnobs({ GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS: '20000' }, 30_000);
expect(m.safetyMarginMs).toBe(5_000);
// Call timeout 20000 > cadence 15000 → clamped to the cadence.
const c = resolveLockRenewalKnobs({ GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS: '20000' }, 30_000, 15_000);
expect(c.callTimeoutMs).toBe(15_000);
// A valid env override still wins untouched (env-beats-default, passes validation).
const ok = resolveLockRenewalKnobs({ GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS: '12000' }, 30_000, 15_000);
expect(ok.callTimeoutMs).toBe(12_000);
});
test('case 15c — per-job lease knob derivation: 300s lease gets capped 15s/30s defaults + 600s hardEvict', () => {
_resetKnobWarningsForTests();
const knobs = resolveLockRenewalKnobs({}, 300_000, 60_000);
expect(knobs.hardEvictMs).toBe(600_000);
// The lock/3 and lock/6 derivations CAP at 15s/30s for long leases —
// a 100s call timeout would wedge tickInFlight across cadence windows.
expect(knobs.callTimeoutMs).toBe(15_000);
expect(knobs.safetyMarginMs).toBe(30_000);
// 30s worker default keeps today's numbers exactly.
const legacy = resolveLockRenewalKnobs({}, 30_000, 15_000);
expect(legacy.callTimeoutMs).toBe(10_000);
expect(legacy.safetyMarginMs).toBe(5_000);
});
});
// issue #1678 (Codex #2): the bounded reconnect-once hook. NOT a withRetry on
@@ -553,23 +731,150 @@ describe('runLockRenewalTick: reconnect-once dep (issue #1678)', () => {
expect(reconnectCalls).toBe(0);
});
test('reconnect is NOT called when the tick aborts at the deadline', async () => {
test('reconnect IS called on a deadline DEFERRAL (CEO-F3: next tick needs a live pool)', async () => {
let reconnectCalls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('write CONNECTION_ENDED'); },
audit: freshAudit().sink,
// sinceLastSuccess = 30000 - 0 = 30000 >= deadline (30000-5000=25000) → abort
// Past the soft deadline (25000) but inside hardEvict (60000):
// primary throw → verify throw → DEFER, reconnect-once for next tick.
now: () => 30_000,
setTimeout: makeFakeTimer().setTimeout,
reconnect: async () => { reconnectCalls++; },
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'should_abort', reason: 'lock-renewal-failed' });
expect(result).toEqual({ kind: 'ok' }); // deferred, not aborted
expect(reconnectCalls).toBe(1);
});
test('reconnect is NOT called when the tick gives up past hardEvictMs', async () => {
let reconnectCalls = 0;
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('write CONNECTION_ENDED'); },
audit: freshAudit().sink,
now: () => 61_000, // past hardEvict (60000) → gave_up
setTimeout: makeFakeTimer().setTimeout,
reconnect: async () => { reconnectCalls++; },
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
expect(result).toMatchObject({ kind: 'should_abort', reason: 'lock-renewal-failed' });
expect(reconnectCalls).toBe(0); // pointless to reconnect when we're giving up the lock
});
});
// =============================================================================
// v0.46 (#4145) — failure-cause classification + starvation telemetry
// =============================================================================
describe('runLockRenewalTick: telemetry (issue #4145)', () => {
test('case 13a — race timeout classifies as call-timeout (named error, not message-sniff)', async () => {
const audit = freshAudit();
const timer = makeFakeTimer();
const deps: LockRenewalDeps = {
renewLock: () => new Promise<boolean>(() => { /* hangs forever */ }),
audit: audit.sink,
now: () => 61_000, // past hardEvict so the verify-throws path aborts
setTimeout: timer.setTimeout,
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const pending = runLockRenewalTick(deps, state);
timer.runAll(); // fire the PRIMARY race timeout
await new Promise(r => globalThis.setTimeout(r, 0)); // let the catch schedule the verify
timer.runAll(); // fire the VERIFY race timeout
const result = await pending;
expect(result).toMatchObject({ kind: 'should_abort', cause: 'call-timeout' });
});
test('case 13b — audit failure events carry cause + lateness + overlap_skips ctx (primary + deferred verify)', async () => {
const ctxLog: unknown[] = [];
const audit: LockRenewalAuditSinkLike = {
logFailure: (_id, _name, _attempt, _err, ctx) => { ctxLog.push(ctx); },
logSuccessAfterFailure: () => { /* noop */ },
logGaveUp: () => { /* noop */ },
};
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('outage'); },
audit,
// lateness = max(0, 20000 - 0 - 15000) = 5000. 20000 + 15000 >= 25000
// → the verify fires (and also throws) → a second, deferred failure.
now: () => 20_000,
setTimeout: makeFakeTimer().setTimeout,
loadSnapshot: () => ({ load1: 28.5, cores: 32 }),
};
const state = makeState({ lastSuccessfulRenewalAt: 0, overlapSkips: 3 });
const result = await runLockRenewalTick(deps, state);
expect(result).toEqual({ kind: 'ok' });
expect(ctxLog).toHaveLength(2);
expect(ctxLog[0]).toMatchObject({
cause: 'refused',
lateness_ms: 5_000,
overlap_skips: 3,
load1: 28.5,
cores: 32,
});
expect(ctxLog[1]).toMatchObject({ cause: 'refused', deadline_deferred: true });
expect(state.consecutiveFailures).toBe(2); // primary + verify each count
});
test('case 13c — a THROWING loadSnapshot never breaks the tick (CEO-F2)', async () => {
const audit = freshAudit();
const deps: LockRenewalDeps = {
renewLock: async () => { throw new Error('outage'); },
audit: audit.sink,
now: () => 61_000, // past hardEvict → the verify-throws path aborts
setTimeout: makeFakeTimer().setTimeout,
loadSnapshot: () => { throw new Error('os.loadavg exploded'); },
};
const state = makeState({ lastSuccessfulRenewalAt: 0 });
const result = await runLockRenewalTick(deps, state);
// Still aborts cleanly; load fields simply absent.
expect(result).toMatchObject({ kind: 'should_abort', cause: 'refused' });
expect('load1' in result).toBe(false);
});
test('case 13d — onRenewalSuccess fires on success (R2-9 histogram reset hook) and its throw is swallowed', async () => {
let resets = 0;
const okDeps: LockRenewalDeps = {
renewLock: async () => true,
audit: freshAudit().sink,
now: () => 1000,
setTimeout: makeFakeTimer().setTimeout,
onRenewalSuccess: () => { resets++; },
};
expect(await runLockRenewalTick(okDeps, makeState())).toEqual({ kind: 'ok' });
expect(resets).toBe(1);
const throwingDeps: LockRenewalDeps = {
...okDeps,
onRenewalSuccess: () => { throw new Error('histogram on fire'); },
};
expect(await runLockRenewalTick(throwingDeps, makeState())).toEqual({ kind: 'ok' });
});
test('case 13e — lateness baseline advances every tick (coalesced-timer semantics)', async () => {
let nowMs = 15_000; // exactly one interval after the seeded baseline (0)
const deps: LockRenewalDeps = {
renewLock: async () => true,
audit: freshAudit().sink,
now: () => nowMs,
setTimeout: makeFakeTimer().setTimeout,
};
const state = makeState();
await runLockRenewalTick(deps, state);
expect(state.lastTickFiredAt).toBe(15_000); // baseline advanced
// Next tick fires 61s later (starved past hardEvict — verify throws too):
// lateness = 76000 - 15000 - 15000 = 46000 attributes the miss to LOCAL
// starvation; sinceLastSuccess = 76000 - 15000 = 61000 >= 60000 → abort.
nowMs = 76_000;
const failDeps: LockRenewalDeps = { ...deps, renewLock: async () => { throw new Error('x'); } };
const result = await runLockRenewalTick(failDeps, state);
expect(result).toMatchObject({ kind: 'should_abort', latenessMs: 46_000, sinceLastSuccessMs: 61_000 });
});
});
describe('runLockRenewalTick: per-call cancellation signal (issue #6)', () => {
test('renewLock receives a live AbortSignal; not aborted on the happy path', async () => {
const audit = freshAudit();