mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-15 09:22:18 +00:00
Compare commits
5
Commits
v0.45.15.0
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4deee227be | ||
|
|
52140808fd | ||
|
|
83a4a94c38 | ||
|
|
0f03a0f929 | ||
|
|
2b8c200b6e |
@@ -1,4 +1,4 @@
|
||||
<!-- gbrain-runbook-stamp: 0.45.15.0 -->
|
||||
<!-- gbrain-runbook-stamp: 0.45.18.0 -->
|
||||
<!-- This stamp must equal the VERSION file at every release; CI enforces it
|
||||
(scripts/check-bootstrap-tag.sh). `gbrain bootstrap status` compares it to
|
||||
the installed binary and warns on skew. -->
|
||||
|
||||
+50
-2
@@ -2,7 +2,17 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.45.15.0] - 2026-08-14
|
||||
## [0.45.18.0] - 2026-08-15
|
||||
|
||||
**Today's agent spend now reads correctly at every hour, in every timezone.** The admin spend endpoint computed "today" against a naive timestamp that each database session reinterpreted in its own timezone — on any non-UTC session (a PGLite brain following the host clock, a timezone-configured Postgres role), the day boundary shifted by the offset and every evening's spend silently underreported as 0. The boundary is now a UTC instant, independent of session timezone, pinned by a regression test that exercises sessions 12 hours either side of UTC at any wall-clock hour.
|
||||
|
||||
The same class also made the new test-suite snapshot fixture time-of-day flaky: the snapshot bakes the build machine's timezone into the restored cluster, so snapshot-restored engines ran sessions in the builder's zone while cold-init engines followed the running process. Restored engines now re-pin their session to the runtime zone (existing tarballs heal without a rebuild), the snapshot builder pins UTC so tarballs are deterministic across hosts, and a parity test asserts cold and snapshot engines agree on their UTC offset.
|
||||
|
||||
### Fixed
|
||||
- `/admin/api/agents/spend`: `spent_cents_today` no longer underreports on non-UTC sessions (UTC-instant day boundary).
|
||||
- Snapshot-restored PGLite engines behave identically to cold-init engines regardless of the machine that built the tarball.
|
||||
|
||||
## [0.45.17.0] - 2026-08-15
|
||||
|
||||
**A test run can no longer silently touch a real brain.** `gbrain init` writes your
|
||||
database URL into `~/.gbrain/.env`; anyone who had that sourced and ran a bare
|
||||
@@ -55,7 +65,7 @@ each one fails loudly instead of silently skipping:
|
||||
- The phantom-redirect engine-parity test's Postgres arm is now carried by
|
||||
the e2e lane and CI's parity job — previously no lane could reach it.
|
||||
|
||||
### To take advantage of v0.45.15.0
|
||||
### To take advantage of v0.45.17.0
|
||||
|
||||
Nothing to configure. If a bare `bun test` now refuses to start, the message
|
||||
tells you exactly why and what to do — usually just unset the database URL
|
||||
@@ -65,6 +75,44 @@ with `GBRAIN_E2E_ALLOW_DB=<name>` rather than exporting it in your shell
|
||||
profile — a permanent export would disarm the guard for exactly the database
|
||||
it protects.
|
||||
|
||||
## [0.45.16.0] - 2026-08-14
|
||||
|
||||
**Fix wave W0: the verified-bug hotfix pass of the code-smell series.** A 10-auditor sweep of the codebase produced 122 findings; the top claims were adversarially verified, and this release fixes every verified live bug — the ones that survived the skeptic pass. Long-running brains get the biggest wins: background cycles can no longer silently run twice, dead background jobs no longer strand their parents, and image search no longer silently degrades after re-embedding. Developers get a test suite that runs 10x faster.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The background-cycle lock is now actually refreshed while a cycle runs.** Long phases (synthesis, pattern extraction, consolidation — up to 35-minute waits) previously outlived the 5-minute lock TTL with no heartbeat, so a second cycle could start against the same brain and both would write concurrently — duplicated LLM spend and racy writes on Postgres/Supabase brains. A dedicated refresher now heartbeats the lock, every refresh and release is fenced to the exact acquisition (a recycled PID or a superseded run can never touch a successor's lock — including the PGLite file lock, which is no longer rewritten after a detected steal), and a run that loses its lock stops at the next phase boundary with a structured `lock_stolen` report instead of compounding. The job supervisor treats a fenced miss as certain loss and exits for a clean restart.
|
||||
- **Background jobs that die from repeated stalls now notify and unblock their waiting parents.** Previously an aggregator parent whose child was dead-lettered by the stall sweep waited forever; a self-healing sweep also releases parents stranded before the upgrade.
|
||||
- **Retried jobs no longer burn their wall-clock budget while waiting in backoff.** Every automatic re-run path — and every parent-unblock path — resets the per-attempt clock, so exponential-backoff retries and long-waiting aggregators aren't dead-lettered before executing a line.
|
||||
- **Re-embedding no longer flips image chunks to text.** `gbrain embed --stale` (including the autopilot path) preserved every chunk field except `modality`, silently zeroing image retrieval until the next full import. One shared carry list now serves every re-embed path.
|
||||
- **A failed first sync no longer kills the MCP server.** Import preflight failures (missing embedding credentials, unreadable target) now surface as normal tool errors instead of terminating the serving process mid-call.
|
||||
- **`gbrain lint --fix` reports the true fix count** (it previously scanned everything twice and reported "0 auto-fixed" after fixing issues) and walks the tree once.
|
||||
- **The PGLite repair and re-init confirmation prompts can no longer hang forever** on closed or piped stdin: EOF declines safely, and prompts write to stderr so `--json` output stays clean.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`bun run test` is ~10x faster** (measured: a full parallel suite run dropped from ~82 to ~8 minutes). The PGLite schema snapshot is now default-on for the everyday test loop, rebuilt automatically when migrations or the pinned embedding shape change, concurrency-safe across parallel shards and workspaces, and refused on any shape mismatch so a wrong fixture can never poison the suite.
|
||||
- **CI guards now prove they can fail.** A guard registry classifies all 45 check scripts; self-tested scanner guards run against known-bad fixtures on every verify (the registry tracks fixture coverage for the rest), so a guard whose pattern rots into a permanently-green no-op fails the build instead of masquerading as coverage. Two such rotted patterns were found and fixed in the process, along with three guards that were wired into a registry nobody ran.
|
||||
|
||||
To take advantage of v0.45.16.0: upgrade and restart any long-running `gbrain serve`, autopilot, or jobs supervisor/worker daemon so the fenced lock refresh and job-reaper fixes take effect. If you run image search, run `gbrain backfill modality` once after upgrading to restore any image chunks a prior re-embed flipped to text (`gbrain doctor` surfaces the affected count and the exact command). No schema migration and no config changes are required.
|
||||
## [0.45.15.0] - 2026-08-14
|
||||
|
||||
**The queue that drains itself: three background-jobs fixes reported from a downstream agent deployment (upstream issues #2, #3, #4).** A brain whose autopilot cycle stalled mid-run could accumulate byte-identical queued cycles forever while every long job queued before an upgrade died minutes in — and the operator diagnosing it couldn't even find the worker entry point, because `gbrain jobs --help` printed a one-line stub. All three failure modes are closed, and the queue now tells you when it's holding work back.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Stalled cycles no longer breed duplicates.** Autopilot dispatch now uses a single-flight guard (`maxPending`) that counts waiting jobs AND actively-running jobs with a live lock, scoped per source — so a cycle stuck in `active` suppresses re-dispatch instead of minting a new duplicate every tick. A job whose worker died stops counting the moment its lock lapses, so a dead worker can never silently freeze dispatch: the fresh waiting row keeps feeding the existing wedge detectors. Applies to per-source cycles, the legacy single-source path, and brain-wide maintenance.
|
||||
- **Long jobs queued before an upgrade get their real budget.** Handler wall-clock budgets now also resolve at claim time (not just at submit), so rows inserted with no budget — including anything queued on an older version — run with their documented allowance instead of being dead-lettered by the minutes-scale default. Migration v128 backfills budgets for everything still in flight and cancels the duplicate cycle backlog (newest per source survives; manually submitted cycles without a ticker-style idempotency key are never touched; cancelled rows are kept for audit).
|
||||
- **`gbrain jobs --help` prints the real surface.** The full subcommand list plus dedicated help for `work`, `supervisor`, `submit`, `watch`, and `prune` — engine-free, and a help flag after a subcommand can no longer fall through and start a real worker daemon.
|
||||
|
||||
### Added
|
||||
|
||||
- **`gbrain jobs stats` shows suppressed dispatch.** A `Backpressure (24h)` line reports submissions coalesced onto in-flight jobs, plus a hint naming the specific in-flight job holding a queue-empty name back — the visibility that was missing when "nothing queued, nothing completing" was the only symptom.
|
||||
- **`gbrain jobs get <id>` shows the effective wall-clock budget** — the stamped timeout and deadline, or which default applies and when it kicks in.
|
||||
- Autopilot cycle dispatch tells the truth: a submission that coalesced onto an existing job reports `dispatch_coalesced` (and `coalesced: true` in `jobs submit`'s JSON output) instead of claiming a dispatch that never inserted a row.
|
||||
|
||||
To take advantage of v0.45.15.0: upgrade and run any gbrain command — migration v128 applies automatically, backfilling budgets for queued long jobs and clearing any duplicate cycle backlog. If a queue looked wedged before, `gbrain jobs stats` now names the in-flight job to inspect and `gbrain jobs work --help` documents the worker daemon flags end to end.
|
||||
|
||||
## [0.45.14.0] - 2026-08-14
|
||||
|
||||
**The box that already has a brain: framework-spawned coding agents get brain access by default.** The bootstrap door built in v0.45.0.0 was for a human at a laptop. A growing share of Claude Code and Codex sessions are spawned by an agent framework — your OpenClaw, or anything that shells out to headless sessions — on a machine that already hosts a brain and a running `gbrain serve --http`. Until now those sessions got nothing unless someone hand-replicated settings writers across every project directory. One command fixes that:
|
||||
|
||||
@@ -70,7 +70,11 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
|
||||
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
|
||||
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
|
||||
imports use static top-level imports. The only current dynamic-`import()` exceptions
|
||||
imports use static top-level imports. Besides the snapshot loader's lazy
|
||||
`require()` cluster in `pglite-engine.ts:tryLoadSnapshot` (fs/crypto/
|
||||
migrate/pglite-schema + one gateway shape lookup — lazy so production
|
||||
builds without the test-fixture path don't eager-load; the guard now
|
||||
matches `require()` calls too), the only dynamic-`import()` exceptions
|
||||
are the four `ai/gateway.ts` lookups in both engines'
|
||||
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
|
||||
local `try/catch` because the gateway has a large provider/config closure and,
|
||||
|
||||
+13
-5
@@ -92,11 +92,11 @@ The canonical reference for test tiers, isolation rules, timing, and the E2E
|
||||
lifecycle is [`docs/TESTING.md`](docs/TESTING.md). The short version:
|
||||
|
||||
```bash
|
||||
# Inner edit loop (~85s on a Mac dev box)
|
||||
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass
|
||||
# Inner edit loop (~8min full suite on a Mac dev box; single files in seconds)
|
||||
bun run test # parallel 4-shard fan-out (memory-adaptive) + serial post-pass; PGLite snapshot default-on
|
||||
bun test test/markdown.test.ts # specific unit test
|
||||
|
||||
# Pre-push gate (19+ parallel checks + typecheck)
|
||||
# Pre-push gate (40+ parallel checks + typecheck)
|
||||
bun run verify
|
||||
|
||||
# Pre-merge sanity (everything CI runs)
|
||||
@@ -128,6 +128,8 @@ above) or destructive tests refuse to run — opt a differently-named database
|
||||
in one-shot with `GBRAIN_E2E_ALLOW_DB=<name>`.
|
||||
|
||||
Use `bun run verify` before pushing. It runs 19+ guard checks in parallel
|
||||
|
||||
Use `bun run verify` before pushing. It runs 40+ guard checks in parallel
|
||||
(`scripts/run-verify-parallel.sh`), including: banned fork-name leaks
|
||||
(`scripts/check-privacy.sh`), `JSON.stringify(x)::jsonb` interpolation
|
||||
patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
|
||||
@@ -136,8 +138,14 @@ patterns (`scripts/check-jsonb-pattern.sh`), `\r` progress bleed to stdout
|
||||
loop" below), silent fallback to recursive chunking in the compiled binary
|
||||
(`scripts/check-wasm-embedded.sh`), stale admin-dashboard build artifacts
|
||||
(`scripts/check-admin-build.sh`), resolver drift on bundled skills
|
||||
(`bun run check:resolver`), and typecheck. `bun run check:all` runs the full
|
||||
historical sweep including the trailing-newline and exports-count checks.
|
||||
(`bun run check:resolver`), and typecheck. The guard REGISTRY is
|
||||
`scripts/guards-manifest.tsv`, and `scripts/guard-self-test.sh` (also in
|
||||
`verify`) proves each self-tested scanner guard (`selftest=yes` in the
|
||||
manifest; coverage ratchets up from the `todo` rows) can actually fail by
|
||||
running it against known-bad fixtures — a new `scripts/check-*` guard must be
|
||||
registered in the manifest or the build fails. There is no `check:all` script; the
|
||||
trailing-newline, exports-count, and no-legacy-getconnection checks run in
|
||||
`verify` with everything else.
|
||||
|
||||
### Writing tests that survive the parallel loop
|
||||
|
||||
|
||||
@@ -12,6 +12,68 @@
|
||||
campaign's Codex review (CX-11); the campaign deliberately shipped only the
|
||||
toggle + reporter acknowledgment. Start from the responsible-disclosure rules
|
||||
already in CLAUDE.md and docs/RELEASING.md. **Effort:** M. **Priority:** P2.
|
||||
## Code-smell fix-wave deferrals (filed at W0; plan: ~/.claude/plans/system-instruction-you-are-working-encapsulated-eclipse.md)
|
||||
|
||||
Each was individually decided as a deferral in the CEO/eng reviews of the
|
||||
fix-wave plan; the wave series (W0.5–W9, 3.4, 3.6) tracks its own scope there.
|
||||
|
||||
- [ ] **Full engine staged merge** (~10 domains onto shared query modules +
|
||||
Dialect record). **Priority: P2.** Gated on the W9 two-slice pilot criteria
|
||||
(structure+params+results parity on chronicle AND the searchKeyword/CJK
|
||||
hard seam; ≥40% domain LOC cut; Dialect ≤~6 fields; query-builder extension
|
||||
≤~150 lines). The terminal fix for the engine-divergence/JSONB class —
|
||||
blast radius is the production hot path, hence pilot-gated. Blocked by: W9.
|
||||
- [ ] **gateway.ts file split** behind a re-export facade (~121 import sites
|
||||
unmoved). **Priority: P3.** After W8's behavior changes so the split is
|
||||
pure motion; needs the CLAUDE.md engine-dynamic-import exemption-path
|
||||
chasers + check-engine-dynamic-import.sh + build:llms.
|
||||
- [ ] **BrainEngine 149-method interface → domain repos** (65 methods have
|
||||
0-1 callers; 3 already deleted in W3). **Priority: P3.** Shape informed by
|
||||
the W9 pilot's query-module seam.
|
||||
- [ ] **Legacy Anthropic-SDK subagent loop deletion.** **Priority: P2.** One
|
||||
release after W8 flips `agent.use_gateway_loop` default ON (flag stays as
|
||||
the revert path for that release).
|
||||
- [ ] **Deeper test-suite speedup** beyond the W0 snapshot default-on (which
|
||||
already cut the full parallel suite ~4,900s → ~490s). **Priority: P3.**
|
||||
Revisit with post-W0 timing data; diminishing returns until measured.
|
||||
- [ ] **PGLite schema build-time derivation** from SCHEMA_SQL via a named
|
||||
transform list. **Priority: P3.** Only if W3's schema drift TEST proves
|
||||
annoying in practice — the test alone kills the drift bug class (Codex
|
||||
D4.8/D5.23: fresh-schema equivalence ≠ upgrade correctness; old-shape
|
||||
bootstrap fixtures + replay coverage stay regardless).
|
||||
## Jobs fix-wave follow-ups (filed v0.45.15.0 — upstream issues #2/#3/#4)
|
||||
|
||||
- [ ] **P2 — `jobs submit --max-pending` public flag.** maxPending stays an
|
||||
internal submit option this wave (Codex C4): its semantics exclude
|
||||
delayed/paused/waiting-children rows, and identity is (name, queue, source)
|
||||
so distinct payloads collapse. Decide the public contract (include delayed?
|
||||
explicit scope key?) after the primitive soaks in autopilot, then mirror
|
||||
parseMaxWaitingFlag (clamp [1,100]) + help + flag-registry regen + optional
|
||||
submit_job MCP param. Where: src/commands/jobs.ts, src/core/operations.ts.
|
||||
- [ ] **P2 — maxPending at the other single-flight dispatch sites.** The
|
||||
freshness sync submit (src/commands/autopilot.ts freshness loop) and the
|
||||
targeted remediation steps (autopilot.ts targeted-submit loop) still use
|
||||
maxWaiting: 1; widening to maxPending changes behavior of those lanes
|
||||
(suppression while a long run is active) and needs its own review. Where:
|
||||
src/commands/autopilot.ts.
|
||||
- [ ] **P2 — Help-stub sweep for the other CLI_ONLY commands.** The `jobs`
|
||||
defect class exists elsewhere: `gbrain search modes --help` connects an
|
||||
engine before help routing, and the search subcommands have no help guards
|
||||
(jobs/bootstrap/skillpack now carry the guard pattern to copy). Audit every
|
||||
CLI_ONLY member missing from CLI_ONLY_SELF_HELP; the top-level help promises
|
||||
per-command help for all of them. Where: src/cli.ts, src/commands/search.ts.
|
||||
- [ ] **P3 — jobs stats: fuller backpressure/audit surfacing.** The 24h
|
||||
Backpressure line + suppressed-by hint shipped; per-decision breakdowns,
|
||||
longer windows, and doctor integration remain (the audit file header's B4
|
||||
follow-up). Where: src/commands/jobs.ts, src/core/minions/backpressure-audit.ts.
|
||||
- [ ] **P3 — jobs watch: timeout/deadline column.** `jobs get` shows the
|
||||
effective budget; the live dashboard doesn't. Where: src/commands/jobs-watch.ts.
|
||||
- [ ] **P3 — jobs help + operator docs: handler catalog and dispatch-event
|
||||
schema.** `gbrain jobs --help`'s HANDLER TYPES section lists 8 of the ~40
|
||||
registered handlers, and the autopilot dispatch JSON events (`dispatched`,
|
||||
`dispatch_coalesced`, `fanout_summary` with its `coalesced` array) have no
|
||||
schema documentation outside the CHANGELOG. Where: src/commands/jobs.ts
|
||||
(JOBS_HELP), docs/guides/queue-operations-runbook.md.
|
||||
|
||||
## Truthful-surface wave follow-ups (filed with T14, amendment 35 + D14.5)
|
||||
|
||||
@@ -707,6 +769,16 @@ job) and sync. See CLAUDE.md "Pace Mode".
|
||||
supervisor-detection downgrade. Today these inherit config/env pacing only when
|
||||
they call `runEmbedCore`.
|
||||
- [ ] **P1-companion — Supervisor concurrency 3→2 + job-kind slot fairness (E7).**
|
||||
**v0.45.15.0 annotation (jobs fix wave):** make the whole wedge-detector FAMILY
|
||||
suppression-aware while here — the supervisor watchdog (supervisor.ts wedge
|
||||
predicate) and doctor's `wedged_queue` check both require waiting > 0, and
|
||||
`maxPending` single-flight keeps waiting at 0 while a job is in flight.
|
||||
Mitigations already shipped: maxPending counts only LIVE-LOCK actives (a
|
||||
dead/blocked worker's expired-lock row never suppresses, so fresh waiting rows
|
||||
re-feed the detectors) and `jobs stats` prints a Backpressure line + a
|
||||
suppressed-by hint. Remaining: teach watchdog/doctor to treat
|
||||
recent-coalesces + stale live active as wedge signal; also note the worker
|
||||
in-flight stall-check hole (worker.ts stall check skips when inFlight > 0).
|
||||
The daemon-side root cause the external wrapper's probe was blind to:
|
||||
`embed-backfill`/`autopilot-cycle` jobs can occupy all supervisor slots
|
||||
(`:215` below). Pacing makes backfills safe; this fixes the residual death rate.
|
||||
@@ -1260,11 +1332,14 @@ but were deliberately scoped OUT — neither is a #1784 regression.
|
||||
deserves its own deliberate change. Fix: mirror the extracted
|
||||
`buildCostRefusal({json, ...})` helper (`reindex-code.ts`). The guardrail
|
||||
(exit 2, no spend) stays; only the FORMAT splits on `--json`.
|
||||
- [ ] **P3 — `gbrain jobs --help` has no subcommand list.** jobs.ts dispatches
|
||||
- [x] **P3 — `gbrain jobs --help` has no subcommand list.** jobs.ts dispatches
|
||||
on a bare subcommand string with no HELP const, so `watch` (and every other
|
||||
jobs subcommand) is undocumented in `--help`. The new `watch` `--json` /
|
||||
`--follow` flags are documented only in the file JSDoc. Add a HELP table to the
|
||||
`jobs` command listing every subcommand + its flags.
|
||||
**Completed:** v0.45.15.0 (2026-08-14) — JOBS_HELP + JOBS_SUBCOMMAND_HELP with a
|
||||
guard above the thin-client refusal; `jobs`/`jobs work` etc. `--help` print real
|
||||
usage engine-free and can never start a daemon.
|
||||
|
||||
## v0.42.12.0 self-upgrade follow-ups (v0.43+)
|
||||
|
||||
@@ -1436,7 +1511,11 @@ and tested; these are documented tradeoffs and stronger-but-bigger versions.
|
||||
Deferred from the v0.41.38.0 wave (code-callers/callees pin + dream-on-postgres).
|
||||
Documented tradeoffs, not blockers — the shipped bug fixes are complete and tested.
|
||||
|
||||
- [ ] **P1 — Per-source autopilot fan-out passes the global repoPath.**
|
||||
- [x] **P1 — Per-source autopilot fan-out passes the global repoPath.**
|
||||
**Completed (verified already fixed):** v0.45.15.0 audit (2026-08-14) — the
|
||||
handler binds FS phases to the source's `local_path` and never falls through
|
||||
to the global repoPath (`effectiveBrainDir = sourceId ? sourceLocalPath :
|
||||
repoPath` in src/commands/jobs.ts, with per-source null → skip FS phases).
|
||||
`src/commands/autopilot-fanout.ts:~206` submits every per-source `autopilot-cycle`
|
||||
job with `repoPath: opts.repoPath` (the global checkout), not `src.local_path`.
|
||||
With v0.41.38.0's `cycleSourceId = opts.sourceId ?? resolveSourceForDir(...)`,
|
||||
@@ -1802,18 +1881,14 @@ single canonical `src/core/model-pricing.ts` with `canonicalLookup`.
|
||||
operator pipes directly into `crontab -e` instead of copy-paste-massage.
|
||||
~80 LOC. Mirrors `gbrain sync --break-lock` argv shape.
|
||||
|
||||
- **TODO-OPS-2 (P2)**: Lock-loss detection — extend `DbLockHandle.refresh()`
|
||||
to throw `LockLostError` on 0 rows affected. Codex caught during the
|
||||
v0.41.19.0 plan review: `refresh()` runs `UPDATE ... WHERE holder_pid = pid`
|
||||
with no rows-affected check (`db-lock.ts:108-114`, `:151-156`). If the
|
||||
TTL expired and another worker took over, the original keeps writing
|
||||
silently. v0.41.19.0 ships TTL=5min + active in-phase refresh via
|
||||
`buildYieldDuringPhase` which makes the race window much narrower, but
|
||||
an `await chat()` call that exceeds the 5min wallclock window can still
|
||||
hit it. Fix: `RETURNING id` on the UPDATE + check `rows.length === 0` →
|
||||
throw tagged `LockLostError`. Phases catch + abort cleanly (write partial
|
||||
progress, return `status: 'fail'` with reason `'lock_lost'`). Behavioral
|
||||
contract change with phase-abort fallout; needs its own design pass.
|
||||
- [x] **TODO-OPS-2 (P2)**: Lock-loss detection — CLOSED by the W0 fix-wave
|
||||
(code-smell series). `refresh()` now runs a FENCED update (id + holder_pid +
|
||||
epoch-rendered `acquired_at`) with `RETURNING id`, returns `false` on 0
|
||||
rows, and runCycle's steal controller aborts the run at the next boundary
|
||||
with a structured `reason: 'lock_stolen'` partial report (LockStolenError;
|
||||
raced awaits cover the 5 long phases). The supervisor exits LOCK_LOST
|
||||
immediately on a fenced miss. Pinned by `test/db-lock-fencing.test.ts` +
|
||||
`test/cycle-lock-steal.serial.test.ts`.
|
||||
|
||||
## v0.41.20.0 status + doctor-categories wave follow-ups (v0.42+)
|
||||
|
||||
|
||||
+72
-4
@@ -7,17 +7,76 @@ only.
|
||||
|
||||
### Test command tiers
|
||||
|
||||
Seven test command tiers, each with a clear scope:
|
||||
Six test command tiers, each with a clear scope:
|
||||
|
||||
| Command | What it runs | Wallclock | When to use |
|
||||
|---|---|---|---|
|
||||
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
|
||||
| `bun run test` | Parallel unit-test fast loop. Sharded fan-out via `scripts/run-unit-parallel.sh` (default 4 shards — CPU-detected, clamped to a max of 8; 4 matches CI's fan-out and avoids PGLite WASM-init contention), then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. Builds/refreshes the PGLite schema snapshot BEFORE the shard fan-out and exports `GBRAIN_PGLITE_SNAPSHOT` so PGLite-booting files restore a baked schema instead of replaying every migration (~10x wallclock on a full run; see "PGLite schema snapshot" below). Opt out: `GBRAIN_NO_SNAPSHOT=1`. Memory-safe by default: total concurrency (shards × intra-shard files) is capped to available memory at `GBRAIN_TEST_MEM_PER_FILE_MB` (default 1536 — a PGLite WASM instance) per concurrent file, and two phantom-failure classes are automatically re-run serially (the rescue pass): failures carrying the WASM out-of-memory signature, and shards killed externally (SIGTERM/SIGKILL well before the shard timeout — sibling workspaces' process cleanup, memory jetsam). Phantoms pass serially and the run goes green with an `oom_rescued` note; real failures fail again serially and stay red. Knobs: `GBRAIN_TEST_NO_MEM_ADAPT=1`, `GBRAIN_TEST_NO_OOM_FALLBACK=1`, `GBRAIN_TEST_MAX_CONCURRENCY` (intra-shard, default 4), `GBRAIN_TEST_SHARD_TIMEOUT` / `GBRAIN_TEST_SHARD_KILL_AFTER`, plus `--shards N` / `--max-concurrency N` / `--dry-run` script args. | a few minutes on a Mac dev box | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | The historical pre-check scripts (chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
There is no `check:all` script anymore — it was a second, hand-synced guard
|
||||
registry that drifted from `verify` (three checks were reachable ONLY from it,
|
||||
i.e. never ran anywhere). The `CHECKS` array in `scripts/run-verify-parallel.sh`
|
||||
is the single execution list, and it now includes the former `check:all`-only
|
||||
extras (`check:newlines`, `check:exports-count`, `check:no-legacy-getconnection`).
|
||||
The guard REGISTRY is `scripts/guards-manifest.tsv` (see "Guard registry and
|
||||
self-test" below).
|
||||
|
||||
### PGLite schema snapshot (default-on)
|
||||
|
||||
`scripts/build-pglite-snapshot.ts` (`bun run build:pglite-snapshot`) bakes a
|
||||
post-`initSchema()` PGLite data dir into `test/fixtures/pglite-snapshot.tar`
|
||||
plus a version file; `PGLiteEngine.initSchema()` restores the tar instead of
|
||||
replaying the embedded schema + all migrations when the env var
|
||||
`GBRAIN_PGLITE_SNAPSHOT` points at it. Both `bun run test`
|
||||
(`scripts/run-unit-parallel.sh`, before the shard fan-out) and
|
||||
`scripts/ci-local.sh` call the builder unconditionally and export the env var.
|
||||
Measured effect: a full parallel suite run drops ~10x (PGLite-booting files go
|
||||
~1.63s → ~0.91s each). Properties:
|
||||
|
||||
- **Idempotent.** A hash short-circuit exits in ~40ms when the snapshot is
|
||||
fresh, and REBUILDS a stale one. The hash covers `PGLITE_SCHEMA_SQL`, every
|
||||
migration's `sql` + `sqlFor.pglite`, AND each migration `handler`'s function
|
||||
source (`Function.prototype.toString`) — 19+ migrations carry executable
|
||||
handler code with empty `sql` that a sql-only hash cannot see.
|
||||
- **Concurrency-safe.** Parallel shard runners / sibling workspaces serialize
|
||||
on an atomic `mkdir` lock (`test/fixtures/.pglite-snapshot.lock`) with
|
||||
staleness-verified takeover of a crashed builder; the tar is written first
|
||||
and the version file last, so a crash can never leave a fresh-looking torn
|
||||
fixture. `GBRAIN_SNAPSHOT_LOCK_TIMEOUT_MS` (default 120000) bounds the
|
||||
waiter; an exhausted waiter facing a still-live lock proceeds unlocked as a
|
||||
last resort (the loader gate below validates the version file, not the tar
|
||||
bytes).
|
||||
- **Never authoritative.** The loader (`tryLoadSnapshot` in
|
||||
`src/core/pglite-engine.ts`) verifies the schema hash AND the embedding
|
||||
shape the snapshot was baked with (`dims=` / `model=` lines in the version
|
||||
file) against what this process would create; any mismatch — including a
|
||||
version file without shape lines — warns once and falls through to normal
|
||||
cold init. A wrong fixture can never poison the suite.
|
||||
- **Opt out.** `GBRAIN_NO_SNAPSHOT=1` skips the build + env export for a run;
|
||||
the migration-replay canary tests clear the env themselves regardless.
|
||||
|
||||
Pinned by `test/snapshot-shape-guard.test.ts` (hash + shape refusal matrix,
|
||||
handler-source hash sensitivity).
|
||||
|
||||
### Guard registry and self-test
|
||||
|
||||
`scripts/guards-manifest.tsv` is THE single registry of `scripts/check-*`
|
||||
guards (currently 45), each classified `scanner` (greps/parses repo sources —
|
||||
must eventually carry fixtures), `buildfresh`, or `repostate` (build/freshness
|
||||
guards are exempt-with-reason, not fixture-tested).
|
||||
`scripts/guard-self-test.sh` (`bun run check:guard-self-test`, wired into
|
||||
`bun run verify`) proves every `selftest=yes` scanner CAN fail: it runs each
|
||||
one against known-bad (must exit non-zero) and known-good (must pass) fixture
|
||||
trees under `test/fixtures/guards/<guard>/{bad,good}/` via the
|
||||
`GBRAIN_GUARD_ROOT` env seam, and enforces manifest completeness — a new
|
||||
`scripts/check-*` script that isn't registered in the manifest fails the
|
||||
build. A guard whose pattern rots into a permanently-green no-op now fails CI
|
||||
instead of masquerading as coverage.
|
||||
|
||||
### Shell dispatch and Windows
|
||||
|
||||
@@ -90,7 +149,7 @@ Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-
|
||||
|
||||
**This section is the canonical home of the test-isolation discipline** — CONTRIBUTING.md and other docs link here rather than restating the rules.
|
||||
|
||||
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
|
||||
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
|
||||
|
||||
| Rule | What it bans | Fix |
|
||||
|---|---|---|
|
||||
@@ -293,6 +352,15 @@ Unit tests and what they cover:
|
||||
- `test/longmemeval-sanitize.test.ts` — sanitization parity pinning that `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` is the single source of truth (adding a pattern there must cover both `<take>` framing and `<chat_session>` framing, no per-surface regex drift).
|
||||
- `test/openai-compat-multimodal.test.ts` — gateway's openai-compatible multimodal path: happy-path single + multi-input embedding, unauthenticated proxy mode, dimension-mismatch guard (throws `AIConfigError` with model id + observed + expected pre-storage), default-dim fallback when recipe declares `default_dims`, HTTP 401 / 400 / malformed-JSON / non-array error paths, regression that the existing Voyage `/multimodalembeddings` recipe still routes through its dedicated path. Hermetic via the `__setEmbedTransportForTests` seam.
|
||||
- `test/serve-stdio-lifecycle.test.ts` — `MCP_STDIO=1` env guard: stdin EOF does NOT trigger shutdown when the env is set, SIGTERM still does (guard scope is correct), unset env preserves the CLI lifecycle. Exercises the `ServeOptions.mcpStdio?: boolean` test seam directly so tests don't mutate `process.env`.
|
||||
- `test/db-lock-fencing.test.ts` — fenced lock identity: a `DbLockHandle` carries its acquisition fence, `refresh()` returns true while owned and false after a steal (0-row fenced UPDATE), a stolen-from handle's `release()` is a fenced no-op that leaves the successor's row intact, and `startCycleLockRefresher` aborts its controller with `LockStolenError` on a fenced miss while serializing ticks (a slow refresh never overlaps the next).
|
||||
- `test/cycle-lock-steal.serial.test.ts` — runCycle steal-abort arc end-to-end: a mid-run steal produces a structured partial report (`reason: 'lock_stolen'`), runs no further phases, and never touches the successor's lock row; a steal-free cycle completes and releases normally.
|
||||
- `test/cycle-any-abort-signal.test.ts` — `anyAbortSignal` combining: pre-aborted inputs, late aborts propagating their reason, duck-typed signal stubs (no `addEventListener`) observed via poll, and `dispose()` detaching the caller-signal listener + clearing the poll timer (the daemon leak class).
|
||||
- `test/queue-stall-parent-unblock.test.ts` — the shared `killJobs` tail: a stall-exhausted child lands `child_done(dead)` in its parent's inbox and unblocks the parent, a requeued child doesn't touch the parent, all three reapers route through the tail with their own outcome, and the idempotent stranded-parent sweep self-heals parents whose children were already dead (without unblocking parents that still have a live child).
|
||||
- `test/queue-started-at-retry.test.ts` — every automatic re-run path clears `started_at` (failJob delayed branch, stall requeue, lease release, promoteDelayed, parent re-claim) so a retried job's wall-clock budget measures execution, not backoff wait; end-to-end survival of the wall-clock sweep on a fresh attempt.
|
||||
- `test/embed-modality-preserved.test.ts` — `carryChunkMetadata` carries modality + all code-metadata fields through re-embed merges (an image chunk stays image), plus the write-side contract that omitting modality resets it to text (why the shared list is load-bearing).
|
||||
- `test/import-abort-error.test.ts` — `runImport` preflight/argv failures throw typed `ImportAbortError` instead of exiting the process; the calling process survives the abort.
|
||||
- `test/lint-fix-single-pass.test.ts` — `gbrain lint --fix` walks the tree once and `total_fixed` reports the fixes THIS run applied.
|
||||
- `test/snapshot-shape-guard.test.ts` — PGLite snapshot loader refusal matrix: shape-less version files, dims/model mismatches, and stale schema hashes are all refused; matching hash + shape loads; a migration-handler edit changes the hash.
|
||||
|
||||
### E2E test inventory
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,61 @@
|
||||
# Fix-wave series baselines (W0 → W9)
|
||||
|
||||
Recorded per wave so the series' "10x better for 2x effort" claim is measured,
|
||||
not vibed (fix-wave plan D4.13). Update this file in each wave's PR; keep the
|
||||
prior rows — the deltas ARE the receipt.
|
||||
|
||||
## How to refresh
|
||||
|
||||
```bash
|
||||
wc -l src/commands/doctor.ts src/core/pglite-engine.ts src/core/postgres-engine.ts \
|
||||
src/core/operations.ts src/core/migrate.ts src/commands/sync.ts \
|
||||
src/core/ai/gateway.ts src/cli.ts src/core/engine.ts \
|
||||
src/core/search/hybrid.ts src/core/search/mode.ts src/core/cycle.ts
|
||||
ls scripts/check-* | wc -l # guard count
|
||||
bash scripts/guard-self-test.sh # self-tested count + harness runtime
|
||||
bun run test > /tmp/suite.txt 2>&1; echo $? # wall-clock from the run banner
|
||||
```
|
||||
|
||||
Retrieval-quality canary (MANDATORY before W1, and after W1/W3/W9): run
|
||||
`gbrain eval gate` against a NON-PRODUCTION brain (the production PGLite brain
|
||||
is single-writer and usually held by a live `gbrain serve`; eval runs never
|
||||
touch `~/.gbrain` per the eval discipline — results land in
|
||||
`<repo>/.gbrain-evals/eval-results.jsonl`). Record the gate verdict + headline
|
||||
metrics here per run.
|
||||
|
||||
## W0 (2026-08-14, branch garrytan/code-smell-fix-wave @ post-hotfix)
|
||||
|
||||
God-file line counts (the audit's structural targets, BEFORE the registry waves):
|
||||
|
||||
| File | Lines |
|
||||
|---|---|
|
||||
| src/commands/doctor.ts | 10,057 |
|
||||
| src/core/operations.ts | 7,459 |
|
||||
| src/core/pglite-engine.ts | 6,874 |
|
||||
| src/core/postgres-engine.ts | 6,847 |
|
||||
| src/core/migrate.ts | 6,201 |
|
||||
| src/commands/sync.ts | 5,991 |
|
||||
| src/core/ai/gateway.ts | 4,049 |
|
||||
| src/cli.ts | 3,301 |
|
||||
| src/core/cycle.ts | 2,933 |
|
||||
| src/core/search/hybrid.ts | 2,453 |
|
||||
| src/core/engine.ts | 2,320 |
|
||||
| src/core/search/mode.ts | 1,232 |
|
||||
|
||||
Guards: 47 scripts/check-* files; 3 self-tested (harness <1s, budget 30s);
|
||||
single registry established (guards-manifest.tsv; `check:all` deleted; 3
|
||||
previously-unreachable guards wired into verify).
|
||||
|
||||
Test infra: PGLite snapshot default-on for `bun run test`. Per-PGLite-file:
|
||||
1.63s cold → 0.91s snapshotted (measured on test/db-lock-fencing.test.ts).
|
||||
Full-suite wall-clock (post-snapshot): recorded in the W0 ship notes — see
|
||||
the run banner of the W0 PR's `bun run test` evidence.
|
||||
|
||||
Retrieval canary: NOT RUN at W0 (production brain locked by live serve; W0
|
||||
touches no search paths). REQUIRED before W1 lands.
|
||||
|
||||
Verified-bug status at W0 ship: cycle-lock refresh + fencing (TODO-OPS-2
|
||||
closed), stall-death parent unblock, started_at ×4, modality carry, import
|
||||
typed aborts, lint single-pass, prompt EOF safety, guard self-test harness,
|
||||
snapshot default-on. W0a superseded by master's WP1/D7 (port-ledger in the
|
||||
plan file).
|
||||
@@ -140,7 +140,7 @@ Three-command pattern an agent can drive without shell archaeology:
|
||||
```bash
|
||||
# Start (returns PIDs + pid_file on stdout as JSON, then detaches)
|
||||
gbrain jobs supervisor start --detach --json
|
||||
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
|
||||
# → {"event":"started","supervisor_pid":1234,"pid_file":"/Users/you/.gbrain/supervisor-<brain-id>.pid","detached":true}
|
||||
|
||||
# Check health (machine-parseable JSON, no log scraping)
|
||||
gbrain jobs supervisor status --json
|
||||
|
||||
@@ -85,8 +85,20 @@ gbrain jobs smoke --wedge-rescue
|
||||
|
||||
- **stalled-forever** — A worker claimed a job, started executing, and has
|
||||
held the row for over an hour. The wall-clock sweep evicts jobs past
|
||||
2× `timeout_ms`; if one's still active, either no `timeout_ms` was set
|
||||
or the sweep is newly deployed and this job predates it. Cancel it.
|
||||
2× `timeout_ms`. Long-lane handlers (subagent, autopilot-cycle,
|
||||
embed-backfill, …) always have a budget now: it stamps at submit, is
|
||||
COALESCEd from `HANDLER_DEFAULT_TIMEOUT_MS` at claim for legacy NULL rows,
|
||||
and migration v128 backfilled rows that predate both. `gbrain jobs get <id>`
|
||||
prints the effective budget and which kill path applies. If a short-lane
|
||||
job is still active with no budget, the null-default sweep
|
||||
(2 × lock-duration × max_stalled) evicts it within minutes. Cancel it if
|
||||
you can't wait.
|
||||
- **duplicate cycles** — Historic brains could accumulate byte-identical
|
||||
waiting `autopilot-cycle` rows when a job stalled in `active`. v128
|
||||
cancelled that backlog (newest ticker-keyed row per source survives), and
|
||||
the `maxPending` dispatch guard prevents new accumulation. Suppressed
|
||||
dispatches are visible in `jobs stats` (Backpressure line) and the
|
||||
backpressure audit JSONL.
|
||||
- **waiting-depth** — Submitters are piling up jobs faster than workers
|
||||
drain them. Set `--max-waiting N` on the submission or on the programmatic
|
||||
`queue.add()` call. If you want a taller pile, raise the threshold via
|
||||
|
||||
+5
-1
@@ -225,7 +225,11 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
- **Engine-live paths avoid runtime dynamic `import()` for helper dependencies.** In
|
||||
`src/core/pglite-engine.ts`, `src/core/postgres-engine.ts`, and
|
||||
`src/core/migrate.ts`, dependencies previously reached through runtime dynamic
|
||||
imports use static top-level imports. The only current dynamic-`import()` exceptions
|
||||
imports use static top-level imports. Besides the snapshot loader's lazy
|
||||
`require()` cluster in `pglite-engine.ts:tryLoadSnapshot` (fs/crypto/
|
||||
migrate/pglite-schema + one gateway shape lookup — lazy so production
|
||||
builds without the test-fixture path don't eager-load; the guard now
|
||||
matches `require()` calls too), the only dynamic-`import()` exceptions
|
||||
are the four `ai/gateway.ts` lookups in both engines'
|
||||
`initSchema()` and `_upsertChunksOnce()` methods; each remains lazy inside a
|
||||
local `try/catch` because the gateway has a large provider/config closure and,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "gbrain-context-engine",
|
||||
"name": "gbrain",
|
||||
"version": "0.45.15.0",
|
||||
"version": "0.45.18.0",
|
||||
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
|
||||
"family": "bundle-plugin",
|
||||
"configSchema": {
|
||||
|
||||
+4
-3
@@ -50,7 +50,6 @@
|
||||
"check:admin-scope-drift": "bash scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "bash scripts/check-cli-executable.sh",
|
||||
"check:engine-dynamic-import": "bash scripts/check-engine-dynamic-import.sh",
|
||||
"check:all": "bash scripts/check-privacy.sh && bash scripts/check-proposal-pii.sh && bash scripts/check-test-real-names.sh && bash scripts/check-jsonb-pattern.sh && bash scripts/check-source-id-projection.sh && bash scripts/check-source-config-leak.sh && bash scripts/check-progress-to-stdout.sh && bash scripts/check-no-tracked-symlinks.sh && bash scripts/check-no-legacy-getconnection.sh && bash scripts/check-test-isolation.sh && bash scripts/check-trailing-newline.sh && bash scripts/check-wasm-embedded.sh && bash scripts/check-pglite-embedded.sh && bash scripts/check-exports-count.sh && bash scripts/check-admin-build.sh && bash scripts/check-admin-scope-drift.sh && bash scripts/check-cli-executable.sh && bash scripts/check-skill-brain-first.sh && bash scripts/check-operations-filter-bypass.sh && bash scripts/check-gateway-routed-no-direct-anthropic.sh && bash scripts/check-worker-pool-atomicity.sh && bash scripts/check-key-files-current-state.sh && bash scripts/check-no-double-retry.sh && bash scripts/check-batch-audit-site.sh && bash scripts/check-engine-dynamic-import.sh",
|
||||
"check:gateway-routed": "bash scripts/check-gateway-routed-no-direct-anthropic.sh",
|
||||
"check:worker-pool-atomicity": "bash scripts/check-worker-pool-atomicity.sh",
|
||||
"check:doc-history": "bash scripts/check-key-files-current-state.sh",
|
||||
@@ -97,7 +96,9 @@
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin",
|
||||
"check:skill-refs": "bun scripts/check-skill-refs.mjs",
|
||||
"gate:skills": "bash scripts/skills-commit-gate.sh"
|
||||
"gate:skills": "bash scripts/skills-commit-gate.sh",
|
||||
"check:guard-self-test": "bash scripts/guard-self-test.sh",
|
||||
"check:no-legacy-getconnection": "bash scripts/check-no-legacy-getconnection.sh"
|
||||
},
|
||||
"openclaw": {
|
||||
"compat": {
|
||||
@@ -155,7 +156,7 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.45.15.0",
|
||||
"version": "0.45.18.0",
|
||||
"overrides": {
|
||||
"@hono/node-server": "^2.0.5",
|
||||
"fast-uri": "^3.1.5",
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
#!/usr/bin/env bun
|
||||
// scripts/build-pglite-snapshot.ts
|
||||
//
|
||||
// TZ pinned to UTC BEFORE any PGLite work: dumpDataDir bakes this process's
|
||||
// TimeZone into the tar's cluster defaults. Building under the host zone made
|
||||
// restored engines run sessions in the build machine's zone (the engine also
|
||||
// re-pins at restore — this is the belt to that suspender, and it keeps any
|
||||
// OTHER zone-derived state baked into the tar deterministic across hosts).
|
||||
process.env.TZ = 'UTC';
|
||||
//
|
||||
// Tier 3 fast-restore: boot a fresh PGLite, run the full initSchema (forward
|
||||
// bootstrap + PGLITE_SCHEMA_SQL + every migration), dump the post-init state
|
||||
// to a tar fixture. Test files that read GBRAIN_PGLITE_SNAPSHOT can skip the
|
||||
@@ -18,10 +25,14 @@
|
||||
//
|
||||
// Re-run whenever you touch src/core/migrate.ts or src/schema.sql.
|
||||
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import { writeFileSync, mkdirSync, existsSync, readFileSync, rmdirSync, rmSync, mkdtempSync, statSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import * as crypto from "node:crypto";
|
||||
|
||||
import { configureGateway, getEmbeddingDimensions, getEmbeddingModel } from "../src/core/ai/gateway.ts";
|
||||
import { LEGACY_EMBEDDING_CONFIG } from "../test/helpers/legacy-embedding-config.ts";
|
||||
|
||||
import { PGLiteEngine, computeSnapshotSchemaHash } from "../src/core/pglite-engine.ts";
|
||||
import { MIGRATIONS } from "../src/core/migrate.ts";
|
||||
import { PGLITE_SCHEMA_SQL } from "../src/core/pglite-schema.ts";
|
||||
@@ -33,9 +44,84 @@ function computeSchemaHash(): string {
|
||||
async function main() {
|
||||
const fixturePath = "test/fixtures/pglite-snapshot.tar";
|
||||
const versionPath = "test/fixtures/pglite-snapshot.version";
|
||||
const lockPath = "test/fixtures/.pglite-snapshot.lock";
|
||||
mkdirSync(dirname(fixturePath), { recursive: true });
|
||||
|
||||
// W0 fix-wave: build under the EXACT embedding shape the test suite pins.
|
||||
// bunfig.toml preloads test/helpers/legacy-embedding-preload.ts, which
|
||||
// configures the gateway to the shared LEGACY_EMBEDDING_CONFIG (OpenAI
|
||||
// 1536-d) for every `bun test` file — so the snapshot's baked vector(dims)
|
||||
// columns MUST match that shape, not the builder machine's ambient config
|
||||
// (nor the shipped 1280-d default an unconfigured gateway falls back to).
|
||||
// Set in main(), not module scope: ESM hoists imports, so module-scope
|
||||
// placement implied an ordering it never had — config reads are lazy.
|
||||
configureGateway({ ...LEGACY_EMBEDDING_CONFIG, env: { ...process.env } });
|
||||
|
||||
const schemaHash = computeSchemaHash();
|
||||
|
||||
// W0 fix-wave (Tier-1 #16): idempotent short-circuit. Runners now call this
|
||||
// script UNCONDITIONALLY (build-if-missing left stale-but-present snapshots
|
||||
// permanently on the warn+slow path); a fresh snapshot exits in ~ms.
|
||||
const isFresh = () => {
|
||||
if (!existsSync(fixturePath) || !existsSync(versionPath)) return false;
|
||||
const lines = readFileSync(versionPath, "utf-8").trim().split("\n");
|
||||
return lines[0] === schemaHash
|
||||
&& lines[1] === `dims=${getEmbeddingDimensions()}`
|
||||
&& lines[2] === `model=${getEmbeddingModel()}`;
|
||||
};
|
||||
if (isFresh()) {
|
||||
console.log(`[build-pglite-snapshot] up to date (hash ${schemaHash.slice(0, 16)}...) — nothing to do`);
|
||||
return;
|
||||
}
|
||||
|
||||
// GBRAIN_HOME isolation is only needed once we actually BUILD (the engine
|
||||
// boot reads config). Red-team catch: creating it before the isFresh()
|
||||
// short-circuit leaked one temp dir per invocation on the COMMON path
|
||||
// (this script runs on every `bun run test`).
|
||||
const hermeticHome = mkdtempSync(join(tmpdir(), "gbrain-snapshot-hermetic-"));
|
||||
process.env.GBRAIN_HOME = hermeticHome;
|
||||
|
||||
// W0 fix-wave (D5.8): concurrency lock. Parallel shard runners / concurrent
|
||||
// Conductor workspaces invoking this simultaneously must not tear the tar.
|
||||
// mkdir is atomic; the loser polls until the winner finishes, then
|
||||
// re-checks freshness and exits.
|
||||
let ownLock = false;
|
||||
try {
|
||||
mkdirSync(lockPath);
|
||||
ownLock = true;
|
||||
} catch {
|
||||
console.log(`[build-pglite-snapshot] another builder holds ${lockPath}; waiting...`);
|
||||
const timeoutMs = Number(process.env.GBRAIN_SNAPSHOT_LOCK_TIMEOUT_MS) || 120_000;
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (existsSync(lockPath) && Date.now() < deadline) {
|
||||
await new Promise(r => setTimeout(r, 250));
|
||||
}
|
||||
if (isFresh()) {
|
||||
console.log(`[build-pglite-snapshot] concurrent builder finished; snapshot fresh`);
|
||||
return;
|
||||
}
|
||||
// Stale lock (crashed builder) or still-stale snapshot: TAKE OVER.
|
||||
// W0 ship-review catch: mkdirSync on a still-existing dir always throws
|
||||
// EEXIST — the original retry could never acquire, so a single crashed
|
||||
// builder left every future rebuild waiting the full deadline and then
|
||||
// proceeding UNLOCKED forever (the stale dir was never removed).
|
||||
// Red-team refinement: verify STALENESS (lock dir mtime older than the
|
||||
// full wait window) before the rmdir — two exhausted waiters would
|
||||
// otherwise each rmdir+mkdir and the second would steal the first's
|
||||
// just-created LIVE lock, re-opening the torn-tar window.
|
||||
try {
|
||||
if (existsSync(lockPath)) {
|
||||
const ageMs = Date.now() - statSync(lockPath).mtimeMs;
|
||||
if (ageMs > timeoutMs) {
|
||||
console.log(`[build-pglite-snapshot] stale lock (age ${Math.round(ageMs / 1000)}s > ${Math.round(timeoutMs / 1000)}s) — taking over`);
|
||||
rmdirSync(lockPath);
|
||||
}
|
||||
}
|
||||
mkdirSync(lockPath);
|
||||
ownLock = true;
|
||||
} catch { /* lock is LIVE (fresh mtime) or takeover raced; proceed unlocked as last resort */ }
|
||||
}
|
||||
try {
|
||||
console.log(`[build-pglite-snapshot] schema hash: ${schemaHash.slice(0, 16)}...`);
|
||||
console.log(`[build-pglite-snapshot] booting PGLite (in-memory)...`);
|
||||
const engine = new PGLiteEngine();
|
||||
@@ -53,12 +139,21 @@ async function main() {
|
||||
const dump = await engine.db.dumpDataDir("none");
|
||||
const buffer = Buffer.from(await dump.arrayBuffer());
|
||||
|
||||
// Write tar first, version LAST — the version file is the commit point, so
|
||||
// a crash between the writes leaves a stale-hash (ignored) snapshot, never
|
||||
// a fresh-looking torn one. Lines 2-3 record the embedding shape the
|
||||
// snapshot was baked with; the loader refuses a shape-mismatched snapshot
|
||||
// (the W0 1280-vs-1536 incident class).
|
||||
writeFileSync(fixturePath, buffer);
|
||||
writeFileSync(versionPath, schemaHash + "\n");
|
||||
writeFileSync(versionPath, `${schemaHash}\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
|
||||
await engine.disconnect();
|
||||
|
||||
console.log(`[build-pglite-snapshot] wrote ${fixturePath} (${buffer.length} bytes)`);
|
||||
console.log(`[build-pglite-snapshot] wrote ${versionPath}`);
|
||||
} finally {
|
||||
if (ownLock) { try { rmdirSync(lockPath); } catch { /* best effort */ } }
|
||||
try { rmSync(hermeticHome, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
await main();
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# scopes to admin/src/ so we can't import the source list directly; instead
|
||||
# this script extracts both lists and diffs them.
|
||||
#
|
||||
# Wired into `bun run verify` and `bun run check:all`.
|
||||
# Wired into `bun run verify` (single guard registry: scripts/guards-manifest.tsv).
|
||||
#
|
||||
# Exits 0 on match, 1 on drift, 2 on internal error (file missing, parse fail).
|
||||
#
|
||||
|
||||
@@ -49,7 +49,16 @@ for (const file of files) {
|
||||
}
|
||||
|
||||
function visit(node: ts.Node): void {
|
||||
if (ts.isCallExpression(node) && node.expression.kind === ts.SyntaxKind.ImportKeyword) {
|
||||
// W0 ship-review catch: match BOTH lazy-loading forms. The guard
|
||||
// previously matched only `import(...)` call expressions, so a
|
||||
// `require(...)` on an engine-live path passed silently and its
|
||||
// engine-dynamic-import-ok marker was decorative.
|
||||
const isDynamicImport = ts.isCallExpression(node)
|
||||
&& node.expression.kind === ts.SyntaxKind.ImportKeyword;
|
||||
const isRequireCall = ts.isCallExpression(node)
|
||||
&& ts.isIdentifier(node.expression)
|
||||
&& node.expression.text === 'require';
|
||||
if (isDynamicImport || isRequireCall) {
|
||||
const { line } = sourceFile.getLineAndCharacterOfPosition(node.expression.getStart(sourceFile));
|
||||
const sourceLine = lines[line] ?? '';
|
||||
if (!markerLines.has(line)) {
|
||||
|
||||
@@ -17,26 +17,43 @@ set -euo pipefail
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Match the interpolated form: ${JSON.stringify(...)}::jsonb
|
||||
# Using grep -P for Perl-compatible regex (lookahead-free pattern is enough here).
|
||||
PATTERN='\$\{JSON\.stringify\([^)]*\)\}::jsonb'
|
||||
# W0 fix-wave (Tier-1 #11): self-test seam — the guard harness points this at
|
||||
# a known-bad fixture tree and asserts exit 1.
|
||||
SCAN_ROOT="${GBRAIN_GUARD_ROOT:-src/}"
|
||||
|
||||
if grep -rEn "$PATTERN" src/ 2>/dev/null; then
|
||||
# Match the interpolated form: ${JSON.stringify(...)}::jsonb
|
||||
#
|
||||
# W0 fix-wave (Tier-1 #11): the previous `\([^)]*\)` argument matcher could
|
||||
# not cross a nested `)` — `${JSON.stringify(obj.get())}::jsonb` was
|
||||
# invisible (the same regex-hole class that made check-no-double-retry a
|
||||
# permanently-green no-op). `[^}]*` spans nested parens but CANNOT cross the
|
||||
# interpolation's closing `}`, so a safe `${JSON.stringify(x)}::text::jsonb`
|
||||
# followed by a separate `${expr()}::jsonb` on the same line is not spanned
|
||||
# into a false positive (ship-review catch — the greedy `.*` variant was).
|
||||
PATTERN='\$\{JSON\.stringify\([^}]*\)\}::jsonb'
|
||||
|
||||
if grep -rEn "$PATTERN" "$SCAN_ROOT" 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Found JSON.stringify(...)::jsonb pattern in src/."
|
||||
echo "ERROR: Found JSON.stringify(...)::jsonb pattern in $SCAN_ROOT."
|
||||
echo " postgres.js v3 stringifies again, producing JSONB string literals."
|
||||
echo " Use sql.json(x) instead. See feedback_postgres_jsonb_double_encode.md."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no JSON.stringify(x)::jsonb interpolation pattern in src/"
|
||||
echo "OK: no JSON.stringify(x)::jsonb interpolation pattern in $SCAN_ROOT"
|
||||
|
||||
# v0.13.1 #219: guard against max_stalled DEFAULT 1 regressing in any schema
|
||||
# source file. DEFAULT 1 dead-lettered any SIGKILL'd job on first stall, making
|
||||
# the "10/10 rescued" claim false for out-of-the-box users. Default is 5 now.
|
||||
MAX_STALLED_PATTERN='max_stalled\s+INTEGER\s+NOT\s+NULL\s+DEFAULT\s+1\b'
|
||||
|
||||
if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/pglite-schema.ts src/core/schema-embedded.ts 2>/dev/null; then
|
||||
# Schema files are fixed paths; under a fixture root (self-test) they don't
|
||||
# exist — skip rather than fail on the missing-file grep.
|
||||
SCHEMA_FILES=()
|
||||
for f in src/schema.sql src/core/migrate.ts src/core/pglite-schema.ts src/core/schema-embedded.ts; do
|
||||
[ -f "$f" ] && SCHEMA_FILES+=("$f")
|
||||
done
|
||||
if [ "${#SCHEMA_FILES[@]}" -gt 0 ] && grep -rEn "$MAX_STALLED_PATTERN" "${SCHEMA_FILES[@]}" 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: max_stalled DEFAULT 1 reintroduced in schema."
|
||||
echo " Must be DEFAULT 5 to preserve SIGKILL-rescue guarantee. See #219."
|
||||
@@ -51,10 +68,11 @@ echo "OK: max_stalled defaults are 5 in all schema sources"
|
||||
# [JSON.stringify(x)]) — which is the exact shape that double-encoded the
|
||||
# op_checkpoints pin and aborted every sync in #2339. The AST-lite scanner below
|
||||
# catches it. `set -e` propagates its non-zero exit.
|
||||
# Under a fixture root, scan that root; the AST-lite scanner takes roots as argv.
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
node scripts/check-jsonb-params.mjs
|
||||
node scripts/check-jsonb-params.mjs ${GBRAIN_GUARD_ROOT:+"$GBRAIN_GUARD_ROOT"}
|
||||
elif command -v bun >/dev/null 2>&1; then
|
||||
bun scripts/check-jsonb-params.mjs
|
||||
bun scripts/check-jsonb-params.mjs ${GBRAIN_GUARD_ROOT:+"$GBRAIN_GUARD_ROOT"}
|
||||
else
|
||||
echo "WARN: neither node nor bun on PATH; skipping check-jsonb-params.mjs" >&2
|
||||
fi
|
||||
|
||||
@@ -19,16 +19,24 @@ set -euo pipefail
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# W0 fix-wave (Tier-1 #11): self-test seam. The guard harness points this at
|
||||
# a known-bad fixture tree and asserts exit 1 — the guard can no longer rot
|
||||
# into a permanently-green no-op unnoticed.
|
||||
SCAN_ROOT="${GBRAIN_GUARD_ROOT:-src/}"
|
||||
|
||||
# Match: withRetry(...) wrapping any of the 3 engine batch methods.
|
||||
# The greedy `.*` between `withRetry(` and `engine.` covers both the
|
||||
# arrow-fn form and any direct invocation. (gbrain-allow-direct-insert: doc comment)
|
||||
# Multi-line wraps are caught by `grep -E` per file (line-wise) for the
|
||||
# common single-line case; multi-line wraps still get caught by a separate
|
||||
# multi-line pass below.
|
||||
PATTERN='withRetry\([^)]*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)'
|
||||
#
|
||||
# W0 fix-wave (Tier-1 #11): the previous pattern used `[^)]*` between
|
||||
# `withRetry(` and `engine.`, which can never cross the `)` in `() =>` — so
|
||||
# the CANONICAL banned shape (an arrow function wrapping the engine batch
|
||||
# call) was invisible and the guard had been permanently green since it
|
||||
# shipped. `.*` (line-bounded by grep) covers the arrow form, async arrows,
|
||||
# and any argument shape. (Spelled without the literal call token here —
|
||||
# check-system-of-record scans scripts/ comments too: the prose-bleed class.)
|
||||
PATTERN='withRetry\(.*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)'
|
||||
|
||||
# Single-line scan (covers ~95% of real cases).
|
||||
if grep -rEn "$PATTERN" src/ --include='*.ts' 2>/dev/null; then
|
||||
if grep -rEn "$PATTERN" "$SCAN_ROOT" --include='*.ts' 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Found withRetry(...engine.{addLinksBatch|addTimelineEntriesBatch|upsertChunks})"
|
||||
echo " pattern in src/."
|
||||
@@ -47,17 +55,26 @@ if grep -rEn "$PATTERN" src/ --include='*.ts' 2>/dev/null; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Multi-line scan: a withRetry( on one line and the engine call on the next
|
||||
# few. Bounded to 3-line window so we don't flag distant unrelated calls.
|
||||
# Uses pcregrep if available, else falls back to a simple awk window.
|
||||
if command -v pcregrep >/dev/null 2>&1; then
|
||||
if pcregrep -r -M -n --include='\.ts$' \
|
||||
'withRetry\([^)]*\n\s*\(?[^)]*=>\s*engine\.(addLinksBatch|addTimelineEntriesBatch|upsertChunks)' \
|
||||
src/ 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Multi-line withRetry(...engine.batch...) wrap found in src/. See above."
|
||||
exit 1
|
||||
fi
|
||||
# Multi-line scan: a withRetry( on one line and the engine call within the
|
||||
# next 3 lines. W0 fix-wave (Tier-1 #11): the previous pass was gated on
|
||||
# pcregrep, which is not installed on dev machines OR CI — it never ran.
|
||||
# perl is always available; same 3-line window, always on.
|
||||
#
|
||||
# Ship-review catch: perl must ALWAYS exit 0 and let OUTPUT PRESENCE decide.
|
||||
# An exit-1-from-clean-batches design breaks under `set -o pipefail` the
|
||||
# moment src/ outgrows one xargs batch (xargs exits 123, overriding grep's
|
||||
# verdict) — a silently missed violation, the same permanently-green class
|
||||
# this guard was just cured of.
|
||||
MULTILINE_MATCHES=$(find "$SCAN_ROOT" -name '*.ts' -type f -print0 2>/dev/null | xargs -0 perl -0777 -ne '
|
||||
if (/withRetry\([^\n]*\n(?:[^\n]*\n){0,2}?[^\n]*engine\.(?:addLinksBatch|addTimelineEntriesBatch|upsertChunks)/) {
|
||||
print "$ARGV: multi-line withRetry wrap around an engine batch call\n";
|
||||
}
|
||||
' 2>/dev/null || true)
|
||||
if [ -n "$MULTILINE_MATCHES" ]; then
|
||||
echo "$MULTILINE_MATCHES"
|
||||
echo
|
||||
echo "ERROR: Multi-line withRetry(...engine.batch...) wrap found in $SCAN_ROOT. See above."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no withRetry(...engine.batch...) double-retry patterns in src/"
|
||||
echo "OK: no withRetry(...engine.batch...) double-retry patterns in $SCAN_ROOT"
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
# - new logging APIs that may show up later (the regex matches the URL,
|
||||
# not the consumer; any leak will trip)
|
||||
#
|
||||
# Wired into bun run check:all and bun run verify.
|
||||
# Wired into bun run verify (single guard registry: scripts/guards-manifest.tsv).
|
||||
#
|
||||
# Exit codes: 0 = clean, 1 = found at least one suspect line.
|
||||
set -euo pipefail
|
||||
|
||||
+6
-6
@@ -236,12 +236,12 @@ bash scripts/check-progress-to-stdout.sh
|
||||
bash scripts/check-trailing-newline.sh
|
||||
bash scripts/check-wasm-embedded.sh
|
||||
bun run typecheck
|
||||
echo \"[runner] Tier 3: building PGLite snapshot fixture (cached across reruns)\"
|
||||
if [ ! -f test/fixtures/pglite-snapshot.tar ] || [ ! -f test/fixtures/pglite-snapshot.version ]; then
|
||||
bun run build:pglite-snapshot
|
||||
else
|
||||
echo \"[runner] snapshot fixture exists; engine will validate hash at load time\"
|
||||
fi
|
||||
echo \"[runner] Tier 3: PGLite snapshot fixture (idempotent; rebuilds on hash drift)\"
|
||||
# W0 fix-wave (Tier-1 #16): unconditional call — the build script self-
|
||||
# short-circuits on a fresh hash and rebuilds STALE snapshots (the old
|
||||
# if-missing guard left a stale-but-present snapshot permanently on the
|
||||
# warn+slow path). Concurrency-safe via the script's mkdir lock (D5.8).
|
||||
bun run build:pglite-snapshot
|
||||
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
|
||||
echo \"[runner] resolving E2E file selection (--diff aware)\"
|
||||
${DIFF_E2E_PREP}
|
||||
|
||||
Executable
+94
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# W0 fix-wave (Tier-1 #11 / D5.14): guard self-test harness.
|
||||
#
|
||||
# The audit found scripts/check-no-double-retry.sh had been PERMANENTLY GREEN
|
||||
# since it shipped: its regex could not match the canonical banned shape, and
|
||||
# its multi-line fallback was gated on pcregrep, which is installed nowhere.
|
||||
# A guard that cannot fail is worse than no guard — it reads as coverage.
|
||||
#
|
||||
# This harness makes that class structurally impossible for scanner guards:
|
||||
# every guard marked `selftest yes` in scripts/guards-manifest.tsv is run
|
||||
# against test/fixtures/guards/<guard>/bad (MUST exit non-zero) and
|
||||
# .../good (MUST exit 0), via the GBRAIN_GUARD_ROOT override each guard
|
||||
# honors. Adding a self-test to a `todo` scanner = flip the manifest flag +
|
||||
# drop two fixture files.
|
||||
#
|
||||
# Also prints total harness wall-clock (guard-runtime budget line, D4.5):
|
||||
# fails if the self-test pass exceeds the budget, so guard sprawl shows up
|
||||
# here before it shows up as slow `bun run verify`.
|
||||
#
|
||||
# Usage: scripts/guard-self-test.sh
|
||||
# Exit: 0 = every self-tested guard fails on bad + passes on good.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
MANIFEST="scripts/guards-manifest.tsv"
|
||||
FIXTURES="test/fixtures/guards"
|
||||
BUDGET_SECONDS=30
|
||||
START=$(date +%s)
|
||||
failures=0
|
||||
tested=0
|
||||
|
||||
if [ ! -f "$MANIFEST" ]; then
|
||||
echo "ERROR: $MANIFEST missing — the guard registry is load-bearing."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run_guard() {
|
||||
local guard="$1" fixture_root="$2"
|
||||
case "$guard" in
|
||||
*.mjs) GBRAIN_GUARD_ROOT="$fixture_root" node "scripts/$guard" "$fixture_root" >/dev/null 2>&1 ;;
|
||||
*) GBRAIN_GUARD_ROOT="$fixture_root" bash "scripts/$guard" >/dev/null 2>&1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
while IFS=$'\t' read -r guard klass selftest _notes; do
|
||||
case "$guard" in ''|'#'*) continue ;; esac
|
||||
[ "$selftest" = "yes" ] || continue
|
||||
tested=$((tested + 1))
|
||||
|
||||
bad="$FIXTURES/$guard/bad"
|
||||
good="$FIXTURES/$guard/good"
|
||||
if [ ! -d "$bad" ] || [ ! -d "$good" ]; then
|
||||
echo "FAIL $guard: manifest says selftest=yes but fixtures missing under $FIXTURES/$guard/{bad,good}"
|
||||
failures=$((failures + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_guard "$guard" "$bad"; then
|
||||
echo "FAIL $guard: did NOT flag the known-bad fixture — the guard is a no-op (the check-no-double-retry class)"
|
||||
failures=$((failures + 1))
|
||||
elif ! run_guard "$guard" "$good"; then
|
||||
echo "FAIL $guard: flagged the known-good fixture — false positive"
|
||||
failures=$((failures + 1))
|
||||
else
|
||||
echo "ok $guard (bad→fail, good→pass)"
|
||||
fi
|
||||
done < "$MANIFEST"
|
||||
|
||||
# Manifest completeness: every scripts/check-* guard must have a manifest row
|
||||
# (new guards can't silently skip classification).
|
||||
for f in scripts/check-*.sh scripts/check-*.mjs; do
|
||||
base="$(basename "$f")"
|
||||
# The .ts companion of check-engine-dynamic-import is an implementation file.
|
||||
if ! grep -q "^${base} " "$MANIFEST"; then
|
||||
echo "FAIL $base: no row in $MANIFEST — classify it (scanner|buildfresh|repostate)"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
ELAPSED=$(( $(date +%s) - START ))
|
||||
echo "guard self-test: $tested guard(s) self-tested, ${ELAPSED}s (budget ${BUDGET_SECONDS}s)"
|
||||
if [ "$ELAPSED" -gt "$BUDGET_SECONDS" ]; then
|
||||
echo "FAIL guard self-test exceeded the ${BUDGET_SECONDS}s runtime budget — trim fixtures or parallelize before adding more"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
|
||||
if [ "$failures" -gt 0 ]; then
|
||||
echo "ERROR: $failures guard self-test failure(s)."
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: all self-tested guards catch their bad fixtures and pass their good ones"
|
||||
@@ -0,0 +1,62 @@
|
||||
# CI guard registry (W0 fix-wave, Tier-1 #11 / D5.14).
|
||||
# THE single registry of scripts/check-* guards. package.json's `check:all`
|
||||
# (a second, stale, hand-synced copy) was deleted; scripts/run-verify-parallel.sh
|
||||
# executes guards, and scripts/guard-self-test.sh consumes THIS file to
|
||||
# self-test every scanner guard against known-bad/known-good fixtures under
|
||||
# test/fixtures/guards/<guard>/{bad,good}/ (env: GBRAIN_GUARD_ROOT).
|
||||
#
|
||||
# class: scanner = greps/parses repo sources; MUST eventually carry fixtures
|
||||
# (selftest yes|todo). A scanner guard with selftest=todo is
|
||||
# tracked debt — the class that produced two permanently-
|
||||
# green guards (check-no-double-retry, pcregrep-gated pass).
|
||||
# buildfresh = runs builds/regenerators and diffs outputs; self-tests
|
||||
# don't apply (the build IS the test). exempt.
|
||||
# repostate = checks repo/file state (modes, symlinks, VERSION stamps);
|
||||
# exempt with reason.
|
||||
#
|
||||
# guard class selftest notes
|
||||
check-no-double-retry.sh scanner yes regex hole fixed in W0 (could not match `() =>`); perl multi-line pass replaces never-installed pcregrep
|
||||
check-jsonb-pattern.sh scanner yes nested-paren hole fixed in W0; safe ::text::jsonb spelling stays unflagged
|
||||
check-jsonb-params.mjs scanner yes positional $N::jsonb AST-lite scanner; argv/env root override
|
||||
check-batch-audit-site.sh scanner todo
|
||||
check-bun-test-timeout.sh scanner todo
|
||||
check-fixture-privacy.sh scanner todo
|
||||
check-no-legacy-getconnection.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
|
||||
check-no-pii-in-agent-voice.sh scanner todo
|
||||
check-operations-filter-bypass.sh scanner todo
|
||||
check-pagetype-exhaustive.sh scanner todo
|
||||
check-pg-url-redaction.sh scanner todo
|
||||
check-privacy.sh scanner todo
|
||||
check-progress-to-stdout.sh scanner todo
|
||||
check-proposal-pii.sh scanner todo
|
||||
check-search-path.sh scanner todo
|
||||
check-skill-brain-first.sh scanner todo
|
||||
check-skill-refs.mjs scanner todo
|
||||
check-source-config-leak.sh scanner todo
|
||||
check-source-id-projection.sh scanner todo
|
||||
check-source-scope-onboard.sh scanner todo
|
||||
check-synthetic-corpus-privacy.sh scanner todo
|
||||
check-system-of-record.sh scanner todo
|
||||
check-test-real-names.sh scanner todo
|
||||
check-worker-lock-renewal-shape.sh scanner todo
|
||||
check-worker-pool-atomicity.sh scanner todo
|
||||
check-gateway-routed-no-direct-anthropic.sh scanner todo
|
||||
check-engine-dynamic-import.sh scanner todo .ts companion is its implementation, not a separate guard
|
||||
check-key-files-current-state.sh scanner todo
|
||||
check-exports-count.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
|
||||
check-trailing-newline.sh scanner todo was reachable from neither verify nor CI pre-W0 (check:all only)
|
||||
check-test-isolation.sh scanner todo allowlist data file: check-test-isolation.allowlist
|
||||
check-admin-build.sh buildfresh exempt runs the admin build; the build is the test
|
||||
check-admin-embedded.sh buildfresh exempt embed freshness diff
|
||||
check-admin-scope-drift.sh buildfresh exempt regenerates + diffs
|
||||
check-bootstrap-templates.sh buildfresh exempt regenerates template tree + diffs
|
||||
check-eval-glossary-fresh.sh buildfresh exempt regenerates + diffs
|
||||
check-fuzz-purity.sh buildfresh exempt executes fuzz corpus
|
||||
check-image-decoders-embedded.sh buildfresh exempt binary embed check
|
||||
check-pglite-embedded.sh buildfresh exempt binary embed check
|
||||
check-skills-manifest-fresh.sh buildfresh exempt regenerates + diffs
|
||||
check-tool-catalog-fresh.sh buildfresh exempt regenerates + diffs
|
||||
check-wasm-embedded.sh buildfresh exempt binary embed check
|
||||
check-bootstrap-tag.sh repostate exempt VERSION stamp drift check
|
||||
check-cli-executable.sh repostate exempt file-mode check
|
||||
check-no-tracked-symlinks.sh repostate exempt git index state check
|
||||
|
@@ -51,6 +51,25 @@ unset DATABASE_URL GBRAIN_DATABASE_URL
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# W0 fix-wave (Tier-1 #16): PGLite schema snapshot, DEFAULT-ON for the plain
|
||||
# `bun run test` loop. 500+ test files each cold-boot PGLite + replay 126
|
||||
# migrations without it; the fixture was previously enabled ONLY inside
|
||||
# scripts/ci-local.sh, so the everyday loop paid the full cost. The build
|
||||
# script is idempotent (hash short-circuit) and concurrency-safe (mkdir
|
||||
# lock, D5.8), and its hash folds handler-migration source (D5.13), so an
|
||||
# unconditional call here is cheap and always current. Runs BEFORE the shard
|
||||
# fan-out — shards inherit a finished fixture. Opt out: GBRAIN_NO_SNAPSHOT=1
|
||||
# (the migration-replay canary tests clear the env themselves regardless).
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
if [ "${GBRAIN_NO_SNAPSHOT:-0}" != "1" ]; then
|
||||
if bun run build:pglite-snapshot >/dev/null 2>&1; then
|
||||
export GBRAIN_PGLITE_SNAPSHOT=test/fixtures/pglite-snapshot.tar
|
||||
else
|
||||
echo "[run-unit-parallel] snapshot build failed (non-fatal) — tests run with cold init" >&2
|
||||
fi
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# CPU detection: Apple Silicon perf cores → Mac total physical → nproc → 4.
|
||||
# Returns a single positive integer.
|
||||
|
||||
@@ -72,6 +72,14 @@ CHECKS=(
|
||||
"check:bootstrap-tag"
|
||||
"check:bootstrap-templates"
|
||||
"check:skill-refs"
|
||||
# W0 fix-wave (Tier-1 #11): guard self-tests — every scanner guard proves it
|
||||
# can fail (bad fixture → exit 1) before it counts as coverage. Registry:
|
||||
# scripts/guards-manifest.tsv (package.json's stale `check:all` copy deleted).
|
||||
"check:guard-self-test"
|
||||
# Previously reachable ONLY from the deleted check:all (i.e. never run):
|
||||
"check:newlines"
|
||||
"check:exports-count"
|
||||
"check:no-legacy-getconnection"
|
||||
"typecheck"
|
||||
)
|
||||
|
||||
|
||||
+24
-4
@@ -154,6 +154,11 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
// would leave that help dead code behind the generic stub (the init.ts:117
|
||||
// trap ENG-2 names).
|
||||
'bootstrap', 'hook', 'sweep',
|
||||
// jobs ships JOBS_HELP + a per-subcommand record (JOBS_SUBCOMMAND_HELP) in
|
||||
// jobs.ts, guarded BEFORE the thin-client refusal and the subcommand switch
|
||||
// so `jobs work --help` prints help instead of starting a worker daemon.
|
||||
// Without this entry the generic stub hid the worker entry point entirely.
|
||||
'jobs',
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -172,6 +177,9 @@ const SELF_HELP_WITHOUT_ENGINE: Record<string, () => Promise<(engine: never, arg
|
||||
maintain: async () => (await import('./commands/maintain.ts')).runMaintain as never,
|
||||
'extract-conversation-facts': async () =>
|
||||
(await import('./commands/extract-conversation-facts.ts')).runExtractConversationFacts as never,
|
||||
// runJobs accepts BrainEngine | null and its help guard returns before any
|
||||
// engine (or subcommand body) is touched.
|
||||
jobs: async () => (await import('./commands/jobs.ts')).runJobs as never,
|
||||
};
|
||||
|
||||
/** Returns true when the command's own help was printed. */
|
||||
@@ -2344,16 +2352,26 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
try {
|
||||
switch (command) {
|
||||
case 'import': {
|
||||
const { runImport } = await import('./commands/import.ts');
|
||||
const { runImport, ImportAbortError } = await import('./commands/import.ts');
|
||||
// v0.41 (Codex r2 #3 fix): honor errors counter for exit code.
|
||||
// runImport's per-file catch already records failures, but the
|
||||
// CLI was discarding the result so the process exited 0 even
|
||||
// when files failed (e.g. content-sanity hard-block throws,
|
||||
// size-cap throws, parse errors). Surface non-zero on errors > 0
|
||||
// so wrappers (sync, CI scripts, `&& gbrain doctor`) propagate.
|
||||
const importResult = await runImport(engine, args);
|
||||
if (importResult.errors > 0) {
|
||||
setCliExitVerdict(1);
|
||||
try {
|
||||
const importResult = await runImport(engine, args);
|
||||
if (importResult.errors > 0) {
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
} catch (e) {
|
||||
// W0 (Tier-1 #5): runImport throws typed aborts instead of
|
||||
// process.exit(1) so in-process callers (sync_brain MCP op,
|
||||
// autopilot, minion handler) survive a preflight failure. The CLI
|
||||
// keeps the exact pre-fix behavior: message already printed at the
|
||||
// throw site, exit non-zero here.
|
||||
if (e instanceof ImportAbortError) process.exit(e.exitCode);
|
||||
throw e;
|
||||
}
|
||||
break;
|
||||
}
|
||||
@@ -3229,7 +3247,9 @@ JOBS (Minions)
|
||||
jobs retry <id> Re-queue failed/dead job
|
||||
jobs prune [--older-than 30d] Clean old jobs
|
||||
jobs stats Job health dashboard
|
||||
jobs watch [--follow] Live queue dashboard
|
||||
jobs work [--queue Q] Start worker daemon (Postgres only)
|
||||
jobs supervisor [start|status|stop] Auto-restarting worker wrapper
|
||||
|
||||
ADMIN
|
||||
stats Brain statistics
|
||||
|
||||
@@ -69,8 +69,13 @@ export interface FanoutOpts {
|
||||
}
|
||||
|
||||
export interface FanoutResult {
|
||||
/** Source ids dispatched this tick. */
|
||||
/** Source ids whose submission INSERTED a fresh job this tick. */
|
||||
dispatched: string[];
|
||||
/** Source ids whose submission coalesced onto an existing pending job
|
||||
* (maxPending single-flight or same-slot idempotency) — work is in
|
||||
* flight, but no new row was created. Kept separate so no surface
|
||||
* claims a dispatch that didn't insert. */
|
||||
coalesced: string[];
|
||||
/** Source ids skipped because their last_full_cycle_at is still fresh. */
|
||||
skipped_fresh: string[];
|
||||
/** Source ids beyond the fanoutMax cap (will retry next tick). */
|
||||
@@ -400,19 +405,32 @@ export async function dispatchPerSource(
|
||||
{ repoPath: opts.repoPath },
|
||||
{
|
||||
queue: 'default',
|
||||
// Slot key dedups repeats within one slot; maxPending: 1 is the
|
||||
// cross-slot guard — an in-flight (waiting or live-lock active)
|
||||
// cycle suppresses re-dispatch even after the slot rotates. This
|
||||
// closes the unbounded-duplicate loop: slot rotation used to mint
|
||||
// a fresh key every baseInterval while maxWaiting ignored the
|
||||
// active row, growing the queue forever when a cycle stalled.
|
||||
idempotency_key: `autopilot-cycle:${opts.slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: opts.timeoutMs,
|
||||
maxWaiting: 1,
|
||||
maxPending: 1,
|
||||
},
|
||||
);
|
||||
if (opts.jsonMode) {
|
||||
if (job.coalesced) {
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({ event: 'dispatch_coalesced', job_id: job.id, mode: 'legacy', slot: opts.slot }));
|
||||
} else {
|
||||
log(`[dispatch] coalesced onto job #${job.id} autopilot-cycle (legacy single-source; already in flight)`);
|
||||
}
|
||||
} else if (opts.jsonMode) {
|
||||
emit(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'legacy', slot: opts.slot }));
|
||||
} else {
|
||||
log(`[dispatch] job #${job.id} autopilot-cycle (legacy single-source)`);
|
||||
}
|
||||
return {
|
||||
dispatched: [],
|
||||
coalesced: [],
|
||||
skipped_fresh: [],
|
||||
skipped_cap: [],
|
||||
skipped_cooldown: [],
|
||||
@@ -448,6 +466,7 @@ export async function dispatchPerSource(
|
||||
);
|
||||
|
||||
const dispatched: string[] = [];
|
||||
const coalesced: string[] = [];
|
||||
for (const src of dispatch) {
|
||||
try {
|
||||
const shouldPull = sourceConfigHasRemoteUrl(src.config);
|
||||
@@ -470,26 +489,43 @@ export async function dispatchPerSource(
|
||||
idempotency_key: `autopilot-cycle:${src.id}:${opts.slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: opts.timeoutMs,
|
||||
// DELIBERATELY no maxWaiting: 1 here. maxWaiting is per
|
||||
// (name, queue), so it would coalesce all N per-source jobs
|
||||
// sharing name='autopilot-cycle' down to ONE waiting job —
|
||||
// killing the fan-out. The per-source idempotency_key
|
||||
// already provides the right dedup granularity (one job per
|
||||
// source per slot, regardless of how many ticks try).
|
||||
// Still DELIBERATELY no maxWaiting here (its NULL-as-wildcard
|
||||
// source scope would coalesce N per-source jobs down to one).
|
||||
// maxPending is safe: its scope is EXACT on
|
||||
// COALESCE(data.sourceId, data.source_id), so each source keeps
|
||||
// an independent single-flight cap — and unlike the slot key, it
|
||||
// suppresses cross-slot re-dispatch while THIS source's cycle is
|
||||
// still in flight (waiting or live-lock active).
|
||||
maxPending: 1,
|
||||
},
|
||||
);
|
||||
dispatched.push(src.id);
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({
|
||||
event: 'dispatched',
|
||||
job_id: job.id,
|
||||
mode: 'per_source',
|
||||
source_id: src.id,
|
||||
pull: shouldPull,
|
||||
slot: opts.slot,
|
||||
}));
|
||||
if (job.coalesced) {
|
||||
coalesced.push(src.id);
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({
|
||||
event: 'dispatch_coalesced',
|
||||
job_id: job.id,
|
||||
mode: 'per_source',
|
||||
source_id: src.id,
|
||||
slot: opts.slot,
|
||||
}));
|
||||
} else {
|
||||
log(`[dispatch] coalesced onto job #${job.id} autopilot-cycle source=${src.id} (already in flight)`);
|
||||
}
|
||||
} else {
|
||||
log(`[dispatch] job #${job.id} autopilot-cycle source=${src.id}${shouldPull ? ' pull=yes' : ''}`);
|
||||
dispatched.push(src.id);
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({
|
||||
event: 'dispatched',
|
||||
job_id: job.id,
|
||||
mode: 'per_source',
|
||||
source_id: src.id,
|
||||
pull: shouldPull,
|
||||
slot: opts.slot,
|
||||
}));
|
||||
} else {
|
||||
log(`[dispatch] job #${job.id} autopilot-cycle source=${src.id}${shouldPull ? ' pull=yes' : ''}`);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
// Per-source submit failure does NOT abort the tick (codex E1 F1
|
||||
@@ -524,6 +560,7 @@ export async function dispatchPerSource(
|
||||
|
||||
return {
|
||||
dispatched,
|
||||
coalesced,
|
||||
skipped_fresh: skippedFresh.map(s => s.id),
|
||||
skipped_cap: skippedCap.map(s => s.id),
|
||||
skipped_cooldown: skippedCooldown.map(s => s.id),
|
||||
@@ -546,16 +583,18 @@ export function isGlobalMaintenanceStale(lastGlobalAtIso: string | null, now = D
|
||||
* #2194 fix #3 / #2227 bug #3 — dispatch the single brain-wide maintenance job
|
||||
* that runs the `global` cycle phases (embed, orphans, purge, …) ONCE per
|
||||
* window, instead of N per-source cycles each running them concurrently (the
|
||||
* RSS blowout). Single-flight is structural: one `idempotency_key` +
|
||||
* `maxWaiting:1`, so a slow run never stacks. Gated on `autopilot.last_global_at`
|
||||
* (stamped by the handler on success). Postgres-only fan-out concern; on PGLite
|
||||
* the file lock already serializes, but the job is still correct there.
|
||||
* RSS blowout). Single-flight is structural: one `idempotency_key` per slot +
|
||||
* `maxPending:1` (an in-flight waiting/live-lock-active run suppresses
|
||||
* re-dispatch even across slot rotation), so a slow run never stacks. Gated on
|
||||
* `autopilot.last_global_at` (stamped by the handler on success). Postgres-only
|
||||
* fan-out concern; on PGLite the file lock already serializes, but the job is
|
||||
* still correct there.
|
||||
*/
|
||||
export async function dispatchGlobalMaintenance(
|
||||
engine: BrainEngine,
|
||||
queue: MinionQueue,
|
||||
opts: { repoPath: string; slot: string; timeoutMs: number; jsonMode: boolean; emit?: (l: string) => void; log?: (l: string) => void },
|
||||
): Promise<{ dispatched: boolean; reason: 'stale' | 'fresh' }> {
|
||||
): Promise<{ dispatched: boolean; coalesced?: boolean; reason: 'stale' | 'fresh' }> {
|
||||
const emit = opts.emit ?? ((line) => process.stderr.write(line + '\n'));
|
||||
const log = opts.log ?? ((line) => console.log(line));
|
||||
|
||||
@@ -575,14 +614,26 @@ export async function dispatchGlobalMaintenance(
|
||||
{ repoPath: opts.repoPath, phases: GLOBAL_PHASES },
|
||||
{
|
||||
queue: 'default',
|
||||
// Structural single-flight: one global job per slot; maxWaiting:1 coalesces
|
||||
// any surplus so a slow brain-wide pass never stacks duplicates.
|
||||
// Structural single-flight: one global job per slot; maxPending:1
|
||||
// coalesces any surplus — including across slot rotation while a slow
|
||||
// brain-wide pass is still in flight — so duplicates never stack.
|
||||
idempotency_key: `autopilot-global:${opts.slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: opts.timeoutMs,
|
||||
maxWaiting: 1,
|
||||
maxPending: 1,
|
||||
},
|
||||
);
|
||||
if (job.coalesced) {
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({ event: 'dispatch_coalesced', job_id: job.id, mode: 'global_maintenance', slot: opts.slot }));
|
||||
} else {
|
||||
log(`[dispatch] coalesced onto job #${job.id} autopilot-global-maintenance (already in flight)`);
|
||||
}
|
||||
// dispatched: false — no row was inserted (same honest-dispatch contract
|
||||
// as dispatchPerSource, where coalesced sources are excluded from
|
||||
// `dispatched`). The coalesced flag says work is already in flight.
|
||||
return { dispatched: false, coalesced: true, reason: 'stale' };
|
||||
}
|
||||
if (opts.jsonMode) {
|
||||
emit(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'global_maintenance', slot: opts.slot }));
|
||||
} else {
|
||||
|
||||
@@ -1209,13 +1209,19 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// source timestamps say every source is fresh, advance the local
|
||||
// clock too; otherwise a non-empty targeted plan would be skipped
|
||||
// on every tick until the persisted 60-minute window elapsed.
|
||||
if (result.dispatched.length > 0 || result.legacy_fallback || result.all_sources_fresh) {
|
||||
// Coalesced counts as work-in-flight: before dispatched/coalesced
|
||||
// split, a coalesced submission advanced this clock via dispatched —
|
||||
// keep that behavior, or an all-coalesced tick (single-flight
|
||||
// suppression) would retake the full-cycle branch every tick and
|
||||
// starve the targeted-plan path for the whole in-flight window.
|
||||
if (result.dispatched.length > 0 || result.coalesced.length > 0 || result.legacy_fallback || result.all_sources_fresh) {
|
||||
lastFullCycleAt = Date.now();
|
||||
}
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({
|
||||
event: 'fanout_summary',
|
||||
dispatched: result.dispatched,
|
||||
coalesced: result.coalesced,
|
||||
skipped_fresh: result.skipped_fresh,
|
||||
skipped_cap: result.skipped_cap,
|
||||
skipped_cooldown: result.skipped_cooldown,
|
||||
@@ -1225,7 +1231,8 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
}) + '\n');
|
||||
} else if (!result.legacy_fallback) {
|
||||
console.log(
|
||||
`[dispatch] fanout: ${result.dispatched.length} dispatched, ` +
|
||||
`[dispatch] fanout: ${result.dispatched.length} dispatched` +
|
||||
`${result.coalesced.length > 0 ? ` (${result.coalesced.length} coalesced onto in-flight)` : ''}, ` +
|
||||
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped, ` +
|
||||
`${result.skipped_cooldown.length} cooldown ` +
|
||||
`(score=${score}, max=${fanoutMax})`,
|
||||
@@ -1254,7 +1261,16 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
submitOpts,
|
||||
isProtected ? { allowProtectedSubmit: true } : undefined,
|
||||
);
|
||||
if (jsonMode) {
|
||||
// Honest-dispatch contract (same as the fanout paths): a
|
||||
// coalesced submission never claims a dispatch that didn't
|
||||
// insert a row.
|
||||
if (job.coalesced) {
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'dispatch_coalesced', job_id: job.id, mode: 'targeted', step: step.id, score, plan_size: plan.length }) + '\n');
|
||||
} else {
|
||||
console.log(`[dispatch] coalesced onto job #${job.id} ${step.job} (targeted: ${step.id}; already in flight)`);
|
||||
}
|
||||
} else if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'targeted', step: step.id, score, plan_size: plan.length }) + '\n');
|
||||
} else {
|
||||
console.log(`[dispatch] job #${job.id} ${step.job} (targeted: ${step.id}; score=${score})`);
|
||||
|
||||
@@ -1350,7 +1350,9 @@ export async function runBootstrap(args: string[], opts: RunBootstrapOpts = {}):
|
||||
// interview) never falls through into the real operation, regardless of
|
||||
// what other flags/values precede it in `rest`. No install-log entry
|
||||
// either — this isn't a phase run.
|
||||
if (SUBCOMMAND_HELP[sub] && hasHelpToken(rest, sub !== 'interview')) {
|
||||
if (Object.hasOwn(SUBCOMMAND_HELP, sub) && hasHelpToken(rest, sub !== 'interview')) {
|
||||
// Object.hasOwn: a plain-object lookup resolves inherited keys, so
|
||||
// `bootstrap constructor --help` would print Object.prototype.constructor.
|
||||
console.log(SUBCOMMAND_HELP[sub]);
|
||||
return 0;
|
||||
}
|
||||
|
||||
+13
-14
@@ -1,6 +1,7 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { carryChunkMetadata } from '../core/embed-stale.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { createProgress, type ProgressReporter } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
@@ -784,9 +785,11 @@ async function embedPage(
|
||||
}
|
||||
|
||||
/**
|
||||
* Carry code-chunk metadata (language, symbol_name, symbol_type, line range,
|
||||
* parent scope, doc comment, qualified name) from a loaded Chunk back into a
|
||||
* ChunkInput destined for upsertChunks.
|
||||
* Carry per-chunk metadata — modality (the W0 fix: its omission flipped
|
||||
* image chunks to text) plus the code fields (language, symbol_name,
|
||||
* symbol_type, line range, parent scope, doc comment, qualified name) — from
|
||||
* a loaded Chunk back into a ChunkInput destined for upsertChunks. The
|
||||
* shared carryChunkMetadata list (core/embed-stale.ts) is authoritative.
|
||||
*
|
||||
* Issue #769: every re-embed used to strip these fields, and upsertChunks
|
||||
* overwrites (does not COALESCE) the metadata columns from EXCLUDED, so
|
||||
@@ -795,17 +798,13 @@ async function embedPage(
|
||||
* (embedPage, embedAll non-stale, embedAllStale) in lock-step.
|
||||
*/
|
||||
function preserveCodeMetadata(loaded: any, base: ChunkInput): ChunkInput {
|
||||
return {
|
||||
...base,
|
||||
language: loaded.language ?? undefined,
|
||||
symbol_name: loaded.symbol_name ?? undefined,
|
||||
symbol_type: loaded.symbol_type ?? undefined,
|
||||
start_line: loaded.start_line ?? undefined,
|
||||
end_line: loaded.end_line ?? undefined,
|
||||
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
|
||||
doc_comment: loaded.doc_comment ?? undefined,
|
||||
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
|
||||
};
|
||||
// W0 fix-wave (Tier-1 #3, CONFIRMED): delegate to the single shared carry
|
||||
// list. This local copy was missing `modality`, so every CLI re-embed path
|
||||
// (embedPage, embedAll, embedAllStale) flipped image chunks to
|
||||
// modality='text' — upsertChunks overwrites from EXCLUDED — silently
|
||||
// zeroing image retrieval until the next full import. The minion twin in
|
||||
// core/embed-stale.ts carried it correctly; one list now serves both.
|
||||
return carryChunkMetadata(loaded, base);
|
||||
}
|
||||
|
||||
async function embedAll(
|
||||
|
||||
+26
-5
@@ -64,6 +64,27 @@ function defaultWorkers(): number {
|
||||
return Math.min(byPool, byCpu, byMem);
|
||||
}
|
||||
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #5): typed abort for runImport's preflight/argv
|
||||
* failures. Pre-fix these five sites called process.exit(1) directly —
|
||||
* correct for the CLI, but runImport is ALSO invoked in-process by the
|
||||
* sync_brain MCP op (via performFullSync), the autopilot daemon, and the
|
||||
* minion sync handler, so a first sync with unconfigured embedding
|
||||
* credentials TERMINATED the MCP server / daemon / worker mid-call. The
|
||||
* user-facing messages are printed BEFORE the throw (byte-identical CLI
|
||||
* output); the CLI dispatch site maps this error back to exit(exitCode).
|
||||
*/
|
||||
export class ImportAbortError extends Error {
|
||||
readonly exitCode: number;
|
||||
/** True: the user-facing message was already printed at the throw site. */
|
||||
readonly alreadyReported = true;
|
||||
constructor(reason: string, exitCode = 1) {
|
||||
super(`import aborted: ${reason}`);
|
||||
this.name = 'ImportAbortError';
|
||||
this.exitCode = exitCode;
|
||||
}
|
||||
}
|
||||
|
||||
/** Bug 9 — surface per-file failures so callers (performFullSync) can gate state advances. */
|
||||
export interface RunImportResult {
|
||||
imported: number;
|
||||
@@ -129,7 +150,7 @@ export async function runImport(
|
||||
} catch (e) {
|
||||
console.error(`\n${e instanceof Error ? e.message : e}`);
|
||||
console.error('Tip: run `gbrain import <dir> --no-embed` to import without embedding now.');
|
||||
process.exit(1);
|
||||
throw new ImportAbortError('embedding disabled (deferred-setup sentinel)');
|
||||
}
|
||||
|
||||
// v0.41.6.0 D1: preflight embedding credentials. Closes the bug class
|
||||
@@ -147,7 +168,7 @@ export async function runImport(
|
||||
console.error(e.userMessage);
|
||||
console.error('');
|
||||
}
|
||||
process.exit(1);
|
||||
throw new ImportAbortError('embedding credentials missing');
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
@@ -221,7 +242,7 @@ export async function runImport(
|
||||
workerCount = parseWorkers(workersArg ?? undefined) ?? 1;
|
||||
} catch (e) {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
process.exit(1);
|
||||
throw new ImportAbortError('invalid --workers value');
|
||||
}
|
||||
// Find dir: first non-flag arg that isn't a value for --workers
|
||||
const flagValues = new Set<number>();
|
||||
@@ -231,7 +252,7 @@ export async function runImport(
|
||||
|
||||
if (!dirArg) {
|
||||
console.error('Usage: gbrain import <dir> [--no-embed] [--workers N] [--fresh] [--source-id <id>] [--include-gitignored] [--json]');
|
||||
process.exit(1);
|
||||
throw new ImportAbortError('no import directory given');
|
||||
}
|
||||
// #1728: capture the import target ONCE as an absolute real path. Every
|
||||
// downstream consumer of `dir` (collection, checkpoint load/save, resume
|
||||
@@ -244,7 +265,7 @@ export async function runImport(
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`Import target is not readable: ${dirArg} (${msg})`);
|
||||
process.exit(1);
|
||||
throw new ImportAbortError(`import target not readable: ${dirArg}`);
|
||||
}
|
||||
|
||||
// v0.31.2: collect under the right strategy. Pre-fix this called
|
||||
|
||||
+272
-29
@@ -12,6 +12,7 @@ import type { PaceKeyOverrides } from '../core/pace-mode.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
import { parseNiceValue, applyNiceness, getEffectiveNiceness, formatNice } from '../core/minions/niceness.ts';
|
||||
import { defaultTimeoutMsFor } from '../core/minions/handler-timeouts.ts';
|
||||
|
||||
function parseFlag(args: string[], flag: string): string | undefined {
|
||||
const idx = args.indexOf(flag);
|
||||
@@ -163,6 +164,7 @@ export function resolveWorkerConcurrency(args: string[], env: NodeJS.ProcessEnv
|
||||
*/
|
||||
const JOB_DATE_FIELDS = [
|
||||
'created_at', 'updated_at', 'started_at', 'finished_at', 'lock_until', 'delay_until',
|
||||
'timeout_at',
|
||||
] as const;
|
||||
|
||||
export function rehydrateJobDates<T>(job: T): T {
|
||||
@@ -187,12 +189,41 @@ function formatJob(job: MinionJob): string {
|
||||
return ` ${String(job.id).padEnd(6)} ${job.name.padEnd(14)} ${(job.status + stalled).padEnd(20)} ${job.queue.padEnd(10)} ${dur.padEnd(8)} ${job.created_at.toISOString().slice(0, 19)}`;
|
||||
}
|
||||
|
||||
function formatJobDetail(job: MinionJob): string {
|
||||
/** Render a timestamp that is a Date locally but may arrive as an ISO string
|
||||
* on the thin-client path against an OLDER server (rehydrateJobDates only
|
||||
* converts fields it knows about; a field the peer predates stays a string).
|
||||
* Never call .toISOString() unguarded on wire-shaped job fields. */
|
||||
function formatWhen(v: Date | string | null | undefined): string {
|
||||
if (v instanceof Date) return v.toISOString();
|
||||
return String(v ?? '');
|
||||
}
|
||||
|
||||
/** The effective wall-clock budget line for `jobs get`. Wording matters: the
|
||||
* 1x deadline (handleTimeouts, stamped at claim) is the NORMAL kill; the 2x
|
||||
* wall-clock sweep is the lock-state-agnostic backstop. */
|
||||
function formatTimeoutLines(job: MinionJob): string[] {
|
||||
const lines: string[] = [];
|
||||
if (job.timeout_ms != null) {
|
||||
lines.push(` Timeout: ${job.timeout_ms}ms (deadline kill at 1x when claimed; wall-clock backstop at 2x)`);
|
||||
if (job.timeout_at) lines.push(` Deadline: ${formatWhen(job.timeout_at)}`);
|
||||
} else {
|
||||
const d = defaultTimeoutMsFor(job.name);
|
||||
if (d != null) {
|
||||
lines.push(` Timeout: (unset) — handler default ${d}ms stamps at claim`);
|
||||
} else {
|
||||
lines.push(` Timeout: (unset) — null-default wall-clock sweep applies (2 x lock-duration x max_stalled, ~5m at defaults)`);
|
||||
}
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
export function formatJobDetail(job: MinionJob): string {
|
||||
const lines = [
|
||||
`Job #${job.id}: ${job.name} (${job.status.toUpperCase()}${job.status === 'dead' ? ` after ${job.attempts_made} attempts` : ''})`,
|
||||
` Queue: ${job.queue} | Priority: ${job.priority}`,
|
||||
` Attempts: ${job.attempts_made}/${job.max_attempts} (started: ${job.attempts_started}, stalled: ${job.stalled_counter}/${job.max_stalled})`,
|
||||
` Backoff: ${job.backoff_type} ${job.backoff_delay}ms (jitter: ${job.backoff_jitter})`,
|
||||
...formatTimeoutLines(job),
|
||||
];
|
||||
if (job.started_at) lines.push(` Started: ${job.started_at.toISOString()}`);
|
||||
if (job.finished_at) lines.push(` Finished: ${job.finished_at.toISOString()}`);
|
||||
@@ -210,25 +241,13 @@ function formatJobDetail(job: MinionJob): string {
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
|
||||
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
|
||||
// with remote MCP routing (`list`, `get`) so no scratch local engine is
|
||||
// ever built. Any other subcommand arriving with a null engine is a
|
||||
// routing bug upstream of this function — refuse instead of crashing
|
||||
// inside MinionQueue.
|
||||
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
|
||||
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
|
||||
process.exit(1);
|
||||
}
|
||||
// Null only ever reaches the MCP-routed `list`/`get` branches, which
|
||||
// never touch the engine — narrowed once here so the host-only cases
|
||||
// below typecheck unchanged.
|
||||
const engine = engineOrNull as BrainEngine;
|
||||
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(`gbrain jobs — Minions job queue
|
||||
/**
|
||||
* The full jobs help block. Hoisted to a constant so `gbrain jobs --help`
|
||||
* (routed engine-free via cli.ts SELF_HELP_WITHOUT_ENGINE) and bare
|
||||
* `gbrain jobs` print the same text. Issue: jobs --help used to print the
|
||||
* generic CLI stub because 'jobs' was missing from CLI_ONLY_SELF_HELP.
|
||||
*/
|
||||
const JOBS_HELP = `gbrain jobs — Minions job queue
|
||||
|
||||
USAGE
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
|
||||
@@ -245,8 +264,9 @@ USAGE
|
||||
gbrain jobs retry <id>
|
||||
gbrain jobs prune [--older-than 30d] [--dry-run]
|
||||
gbrain jobs delete <id>
|
||||
gbrain jobs stats
|
||||
gbrain jobs smoke
|
||||
gbrain jobs stats [--queue Q] [--cluster-errors]
|
||||
gbrain jobs smoke [--sigkill-rescue] [--wedge-rescue]
|
||||
gbrain jobs watch [--json] [--follow] [--refresh-ms=N]
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
[--health-interval MS] [--nice N]
|
||||
gbrain jobs supervisor [start] [--detach] [--json]
|
||||
@@ -266,8 +286,9 @@ USAGE
|
||||
|
||||
Auto-restarting wrapper around 'gbrain jobs work'. Spawns the worker
|
||||
as a child process and restarts on crash with exponential backoff
|
||||
(1s -> 60s cap). Writes a PID file to ~/.gbrain/supervisor.pid by
|
||||
default (override via --pid-file or GBRAIN_SUPERVISOR_PID_FILE env).
|
||||
(1s -> 60s cap). Writes a brain-scoped PID file to
|
||||
~/.gbrain/supervisor-<brain-id>.pid by default (override via
|
||||
--pid-file or GBRAIN_SUPERVISOR_PID_FILE env).
|
||||
Lifecycle events are appended to
|
||||
\${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl
|
||||
|
||||
@@ -305,9 +326,169 @@ HANDLER TYPES (built in)
|
||||
shell Run a command or argv. Requires GBRAIN_ALLOW_SHELL_JOBS=1
|
||||
on the worker. Params: {cmd?, argv?, cwd, env?}.
|
||||
See: docs/guides/minions-shell-jobs.md
|
||||
`);
|
||||
|
||||
Detailed help: gbrain jobs {work|supervisor|submit|watch|prune} --help
|
||||
Other subcommands are fully described above.
|
||||
`;
|
||||
|
||||
/**
|
||||
* Per-subcommand help for the flag-heavy / side-effectful subcommands.
|
||||
* Pattern from bootstrap.ts SUBCOMMAND_HELP: the guard below prints these
|
||||
* BEFORE the switch, so \`jobs work --help\` can never start a worker
|
||||
* daemon (the defect class this record exists to prevent). Subcommands
|
||||
* without an entry fall back to JOBS_HELP, which documents them fully.
|
||||
*/
|
||||
const JOBS_SUBCOMMAND_HELP: Record<string, string> = {
|
||||
work: `gbrain jobs work — start a worker daemon (Postgres only)
|
||||
|
||||
USAGE
|
||||
gbrain jobs work [--queue Q] [--concurrency N] [--max-rss MB]
|
||||
[--health-interval MS] [--nice N]
|
||||
|
||||
OPTIONS
|
||||
--queue Q Queue to claim from (default: default)
|
||||
--concurrency N Max jobs in flight. Resolution: flag, then
|
||||
GBRAIN_WORKER_CONCURRENCY env, then 1. Values < 1
|
||||
are clamped to 1 with a loud stderr note.
|
||||
--max-rss MB RSS watchdog. Absent: auto-sized to 50% of
|
||||
min(cgroup limit, host RAM), capped at 16384 MB,
|
||||
raised to a 4096 MB floor when the basis allows.
|
||||
0 disables the watchdog. Values 1-255 are rejected
|
||||
(megabytes, not gigabytes — unit-confusion guard).
|
||||
--health-interval MS Health probe cadence (default 60000). 0 disables.
|
||||
Values 1-999 are rejected as unit confusion.
|
||||
Under GBRAIN_SUPERVISED=1 stall detection is off;
|
||||
the DB probe stays.
|
||||
--nice N OS scheduling priority, -20 (highest) to 19
|
||||
(nicest). Env fallback: GBRAIN_NICE; flag wins.
|
||||
Negative values need root.
|
||||
|
||||
NOTES
|
||||
Requires the Postgres engine — PGLite's exclusive file lock cannot host
|
||||
a long-lived daemon. For crash-resilient operation prefer:
|
||||
gbrain jobs supervisor start --detach --json
|
||||
`,
|
||||
supervisor: `gbrain jobs supervisor — auto-restarting wrapper around 'gbrain jobs work'
|
||||
|
||||
USAGE
|
||||
gbrain jobs supervisor [start] [--detach] [--json]
|
||||
[--concurrency N] [--queue Q] [--pid-file PATH]
|
||||
[--max-crashes N] [--health-interval N]
|
||||
[--allow-shell-jobs] [--cli-path PATH]
|
||||
[--max-rss MB] [--nice N]
|
||||
gbrain jobs supervisor status [--json] [--pid-file PATH]
|
||||
gbrain jobs supervisor stop [--json] [--pid-file PATH]
|
||||
|
||||
OPTIONS (start)
|
||||
--detach Fork and print {event, supervisor_pid, pid_file} JSON
|
||||
--json JSONL lifecycle events on stdout
|
||||
--concurrency N Worker concurrency (default 2)
|
||||
--queue Q Queue to claim from (default: default)
|
||||
--pid-file PATH PID file (default: brain-scoped
|
||||
~/.gbrain/supervisor-<brain-id>.pid;
|
||||
env GBRAIN_SUPERVISOR_PID_FILE)
|
||||
--max-crashes N Soft crash threshold (default 10): past N crashes in
|
||||
24h the supervisor reports degraded and keeps backing
|
||||
off. It only STOPS permanently at the hard ceiling —
|
||||
default 10 x N; override or disable (0 = never) via
|
||||
GBRAIN_SUPERVISOR_HARD_STOP_CRASHES.
|
||||
--health-interval N Worker health probe cadence in ms
|
||||
--allow-shell-jobs Enable the shell handler on the spawned worker
|
||||
--cli-path PATH Explicit gbrain binary for the worker child
|
||||
--max-rss MB RSS watchdog for the worker (same rules as jobs work)
|
||||
--nice N OS priority for supervisor + worker children
|
||||
|
||||
EXIT CODES (start)
|
||||
0 clean shutdown 1 max crashes exceeded
|
||||
2 another supervisor holds the PID lock 3 PID file unwritable
|
||||
4 DB queue lock lost (repeated refresh failures; restart re-acquires)
|
||||
`,
|
||||
submit: `gbrain jobs submit — enqueue a background job
|
||||
|
||||
USAGE
|
||||
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
|
||||
[--delay Nms] [--max-attempts N] [--max-stalled N]
|
||||
[--max-waiting N]
|
||||
[--backoff-type fixed|exponential] [--backoff-delay Nms]
|
||||
[--backoff-jitter 0..1] [--timeout-ms Nms]
|
||||
[--idempotency-key K] [--queue Q] [--dry-run]
|
||||
[--redact-secrets]
|
||||
|
||||
OPTIONS
|
||||
--params JSON Job payload (handler-specific; see HANDLER TYPES in
|
||||
'gbrain jobs --help')
|
||||
--follow Run inline and stream progress (constructs a real
|
||||
worker; works on both engines)
|
||||
--priority N Lower runs first (default 0)
|
||||
--delay Nms Delay before the job becomes claimable (default 0)
|
||||
--max-attempts N Retry budget (default 3)
|
||||
--max-stalled N Stall-requeue budget before dead-letter (default 5)
|
||||
--max-waiting N Backpressure: cap waiting jobs with this name/queue/
|
||||
source before coalescing new submissions ([1,100])
|
||||
--timeout-ms Nms Per-job wall-clock budget. Long-lane handlers get a
|
||||
default from HANDLER_DEFAULT_TIMEOUT_MS when omitted.
|
||||
--idempotency-key K At-most-one row per key (dead/cancelled free the key)
|
||||
--queue Q Target queue (default: default)
|
||||
--dry-run Print what would be submitted, submit nothing
|
||||
--redact-secrets (shell jobs) scrub inherited env values from output
|
||||
`,
|
||||
watch: `gbrain jobs watch — live queue dashboard
|
||||
|
||||
USAGE
|
||||
gbrain jobs watch [--json] [--follow] [--refresh-ms=N]
|
||||
|
||||
OPTIONS
|
||||
--json JSON snapshots instead of the human dashboard
|
||||
--follow Keep refreshing (default: on for TTY, off otherwise)
|
||||
--refresh-ms=N Refresh cadence in ms (default 1000). Equals form only —
|
||||
'watch' does not accept a space-separated value.
|
||||
`,
|
||||
prune: `gbrain jobs prune — delete old terminal jobs
|
||||
|
||||
USAGE
|
||||
gbrain jobs prune [--older-than 30d] [--dry-run]
|
||||
|
||||
OPTIONS
|
||||
--older-than AGE Delete completed/failed/dead/cancelled jobs older than
|
||||
AGE in days (default 30d; bare N or Nd — hour forms
|
||||
are not supported)
|
||||
--dry-run Report what would be deleted without deleting
|
||||
`,
|
||||
};
|
||||
|
||||
export async function runJobs(engineOrNull: BrainEngine | null, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
|
||||
// Help guards run BEFORE the thin-client refusal below: cli.ts routes
|
||||
// `jobs … --help` here engine-free (SELF_HELP_WITHOUT_ENGINE), and help
|
||||
// must never require an engine — or worse, fall through to a subcommand
|
||||
// body and start a real daemon. Only --help/-h are recognized; the bare
|
||||
// word 'help' is NOT (e.g. `jobs submit help` is a legitimate job name).
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(JOBS_HELP);
|
||||
return;
|
||||
}
|
||||
if (args.slice(1).includes('--help') || args.slice(1).includes('-h')) {
|
||||
// Object.hasOwn: a plain-object lookup resolves inherited keys, so
|
||||
// `jobs constructor --help` (toString/valueOf/…) would print the
|
||||
// Object.prototype function instead of falling back to the full help.
|
||||
console.log(Object.hasOwn(JOBS_SUBCOMMAND_HELP, sub) ? JOBS_SUBCOMMAND_HELP[sub] : JOBS_HELP);
|
||||
return;
|
||||
}
|
||||
|
||||
// Thin-client dispatch (cli.ts) passes engine=null for the subcommands
|
||||
// with remote MCP routing (`list`, `get`) so no scratch local engine is
|
||||
// ever built. Any other subcommand arriving with a null engine is a
|
||||
// routing bug upstream of this function — refuse instead of crashing
|
||||
// inside MinionQueue.
|
||||
if (!engineOrNull && sub !== 'list' && sub !== 'get') {
|
||||
console.error(`\`gbrain jobs ${sub ?? ''}\` needs a local engine and cannot run on a thin client.`);
|
||||
process.exit(1);
|
||||
}
|
||||
// Null only ever reaches the MCP-routed `list`/`get` branches, which
|
||||
// never touch the engine — narrowed once here so the host-only cases
|
||||
// below typecheck unchanged.
|
||||
const engine = engineOrNull as BrainEngine;
|
||||
|
||||
// The constructor just stores the reference; on the null (thin-client
|
||||
// list/get) paths no queue method is ever reached.
|
||||
@@ -720,6 +901,67 @@ HANDLER TYPES (built in)
|
||||
` gbrain jobs retry <id> # for dead-lettered jobs`,
|
||||
);
|
||||
}
|
||||
|
||||
// Backpressure visibility: maxPending suppression keeps `waiting` at 0
|
||||
// while a job is in flight, which silences the waiting>0 wedge line
|
||||
// above — the exact operator-confusion cost of the duplicate-cycle
|
||||
// incident. Surface the last 24h of coalesce events (per name, this
|
||||
// queue) from the backpressure audit JSONL, plus a hint naming the
|
||||
// in-flight job when a name shows suppression with zero waiting rows
|
||||
// and a stale live-lock active. Best-effort: unreadable audit files
|
||||
// simply omit the line.
|
||||
try {
|
||||
const { readRecentCoalesceCounts } = await import('../core/minions/backpressure-audit.ts');
|
||||
const coalesceCounts = readRecentCoalesceCounts({ queue: statsQueue, windowMs: 24 * 3600_000 });
|
||||
if (coalesceCounts.size > 0) {
|
||||
// Sort once, reuse for the summary AND the hint slice — slicing
|
||||
// insertion order would let low-volume early-in-file names crowd
|
||||
// out the highest-volume (most likely wedged) ones the summary
|
||||
// line just highlighted.
|
||||
const sortedCoalesces = [...coalesceCounts.entries()]
|
||||
.sort((a, b) => b[1].count - a[1].count);
|
||||
const parts = sortedCoalesces.map(([name, s]) => `${name}: ${s.count}`);
|
||||
console.log(`\n Backpressure (24h): submissions coalesced onto in-flight jobs — ${parts.join(', ')}`);
|
||||
// Hint loop is bounded: names come from the 24h audit window
|
||||
// (normally a handful), capped defensively — this is an
|
||||
// operator-invoked diagnostic, not a hot path. Each hint is
|
||||
// driven by the LATEST coalesce target for the name (the audit's
|
||||
// returned_job_id), scoped to that job's source — a name-wide
|
||||
// aggregate would let source A's waiting row mask source B's
|
||||
// wedge, or name A's job for B's coalesce (multi-source brains).
|
||||
const hints = sortedCoalesces.slice(0, 10);
|
||||
for (const [name, summary] of hints) {
|
||||
if (summary.last_returned_job_id == null) continue;
|
||||
// The target CTE re-checks name+queue: the audit dir is shared
|
||||
// across brains in one GBRAIN_HOME, so an id from another
|
||||
// brain's audit trail must fail the match here rather than
|
||||
// name an unrelated job as the suppressor.
|
||||
const rows = await engine.executeRaw<{ waiting: string; live_id: string | null; age_min: string | null }>(
|
||||
`WITH target AS (
|
||||
SELECT id, started_at, status, lock_until,
|
||||
COALESCE(data->>'sourceId', data->>'source_id') AS scope
|
||||
FROM minion_jobs WHERE id = $3 AND name = $1 AND queue = $2
|
||||
)
|
||||
SELECT (SELECT count(*)::text FROM minion_jobs m, target t
|
||||
WHERE m.name = $1 AND m.queue = $2 AND m.status = 'waiting'
|
||||
AND COALESCE(m.data->>'sourceId', m.data->>'source_id') IS NOT DISTINCT FROM t.scope) AS waiting,
|
||||
(SELECT id::text FROM target WHERE status = 'active' AND lock_until > now()) AS live_id,
|
||||
(SELECT floor(EXTRACT(EPOCH FROM (now() - started_at)) / 60)::text FROM target
|
||||
WHERE status = 'active' AND lock_until > now()) AS age_min`,
|
||||
[name, statsQueue, summary.last_returned_job_id],
|
||||
);
|
||||
const r = rows[0];
|
||||
const ageMin = r?.age_min != null ? parseInt(r.age_min, 10) : null;
|
||||
if (r && parseInt(r.waiting ?? '0', 10) === 0 && r.live_id != null && ageMin != null && ageMin > wedgeMins) {
|
||||
console.log(
|
||||
` ${name}: dispatch suppressed by in-flight job #${r.live_id} (age ${ageMin}m) — check \`gbrain jobs get ${r.live_id}\``,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Audit read is advisory; never break stats.
|
||||
}
|
||||
}
|
||||
|
||||
// v0.41 Bug 2 / Eng D8 — surface lease pressure to the operator.
|
||||
@@ -1695,10 +1937,11 @@ export async function registerBuiltinHandlers(
|
||||
});
|
||||
|
||||
worker.register('import', async (job) => {
|
||||
// import.ts Core extraction deferred to v0.12.0 (import has parallel
|
||||
// workers + checkpointing). Keep the CLI wrapper call but note the
|
||||
// worker-kill risk is bounded: import's only process.exit fires on
|
||||
// a missing dir arg, which this handler always passes.
|
||||
// import.ts Core extraction deferred (import has parallel workers +
|
||||
// checkpointing; the typed-API split lands in W7 of the fix-wave).
|
||||
// W0 (Tier-1 #5): runImport no longer contains ANY process.exit — all
|
||||
// five preflight sites throw typed ImportAbortError, which this
|
||||
// handler's catch converts to a normal failJob. No worker-kill risk.
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs: string[] = [];
|
||||
if (job.data.dir) importArgs.push(String(job.data.dir));
|
||||
|
||||
+52
-45
@@ -441,6 +441,24 @@ export interface LintOpts {
|
||||
* single-file targets.
|
||||
*/
|
||||
exclude?: string[];
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #14): per-page hook fired for every page WITH
|
||||
* issues, after this run's fix attempt for that page — fixedCount is the
|
||||
* number of fixes just applied (0 when --fix is off or nothing was
|
||||
* fixable). The CLI passes a printer so human detail and the aggregate
|
||||
* counts come from ONE scan — pre-fix, runLint ran its own full
|
||||
* read+lint+fix loop and THEN called runLintCore for the summary, linting
|
||||
* every page twice and reporting "0 auto-fixed" because the second pass
|
||||
* saw already-fixed files.
|
||||
*/
|
||||
onPageIssues?: (relPath: string, issues: LintIssue[], fixedCount: number) => void;
|
||||
/** Companion to onPageIssues: per-page progress tick (CLI progress bar). */
|
||||
onPageScanned?: () => void;
|
||||
/**
|
||||
* Fired once with the collected page count before scanning starts, so the
|
||||
* CLI can size its progress bar without walking the tree a second time.
|
||||
*/
|
||||
onPagesCollected?: (count: number) => void;
|
||||
}
|
||||
|
||||
export interface LintResult {
|
||||
@@ -468,6 +486,7 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
|
||||
|
||||
const isSingleFile = statSync(opts.target).isFile();
|
||||
const pages = isSingleFile ? [opts.target] : collectPages(opts.target, opts.exclude ?? []);
|
||||
opts.onPagesCollected?.(pages.length);
|
||||
|
||||
// Resolve content-sanity config once for this lint run (D1: lift DB
|
||||
// config when reachable). Caller can pre-pass via opts.contentSanity
|
||||
@@ -490,21 +509,25 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
}
|
||||
const content = readFileSync(page, 'utf-8');
|
||||
const issues = lintContent(content, isSingleFile ? page : relative(opts.target, page), lintOpts);
|
||||
const relPath = isSingleFile ? page : relative(opts.target, page);
|
||||
const issues = lintContent(content, relPath, lintOpts);
|
||||
opts.onPageScanned?.();
|
||||
if (issues.length === 0) continue;
|
||||
pagesWithIssues++;
|
||||
totalIssues += issues.length;
|
||||
|
||||
let fixCount = 0;
|
||||
if (opts.fix && issues.some(i => i.fixable)) {
|
||||
const fixed = fixContent(content);
|
||||
if (fixed !== content) {
|
||||
const fixCount = issues.filter(i => i.fixable).length;
|
||||
fixCount = issues.filter(i => i.fixable).length;
|
||||
totalFixed += fixCount;
|
||||
if (!opts.dryRun) {
|
||||
writeFileSync(page, fixed);
|
||||
}
|
||||
}
|
||||
}
|
||||
opts.onPageIssues?.(relPath, issues, fixCount);
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -547,57 +570,41 @@ export async function runLint(args: string[]) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Single file or directory — print human detail as we go, then rely on
|
||||
// Core for the aggregate numbers at the end.
|
||||
const isSingleFile = statSync(target).isFile();
|
||||
const pages = isSingleFile ? [target] : collectPages(target, extraExcludes);
|
||||
|
||||
// W0 fix-wave (Tier-1 #14): ONE scan. Pre-fix this function ran its own
|
||||
// full read+lint+fix loop for human output and THEN called runLintCore for
|
||||
// the summary — every page linted twice, and with --fix the second pass
|
||||
// saw already-fixed files so the summary reported "0 auto-fixed" after
|
||||
// fixing N. Human detail now streams from runLintCore's per-page hooks
|
||||
// and the counts come from the same single pass.
|
||||
// Progress on stderr. Stdout keeps the per-issue human output it always had.
|
||||
// Ship-review perf catch: the tree is walked ONCE — runLintCore reports the
|
||||
// collected count via onPagesCollected (pre-fix the CLI ran its own
|
||||
// collectPages just to size the progress bar, a second full readdir/stat
|
||||
// walk on every directory lint).
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('lint.pages', pages.length);
|
||||
|
||||
// v0.41 (D1): resolve content-sanity config once for this lint run.
|
||||
// Mirrors runLintCore. The two paths must agree because runLint
|
||||
// prints human details inline; runLintCore at end computes the
|
||||
// aggregate. Sharing the resolved opts keeps both surfaces seeing
|
||||
// the same rule firings.
|
||||
const contentSanity = await resolveLintContentSanity();
|
||||
const lintContentOpts: LintContentOpts = { contentSanity };
|
||||
|
||||
for (const page of pages) {
|
||||
const content = readFileSync(page, 'utf-8');
|
||||
const relPath = isSingleFile ? page : relative(target, page);
|
||||
const issues = lintContent(content, relPath, lintContentOpts);
|
||||
progress.tick(1);
|
||||
if (issues.length === 0) continue;
|
||||
|
||||
console.log(`\n${relPath}:`);
|
||||
for (const issue of issues) {
|
||||
const fixLabel = issue.fixable ? ' [fixable]' : '';
|
||||
console.log(` L${issue.line} ${issue.rule}: ${issue.message}${fixLabel}`);
|
||||
}
|
||||
|
||||
if (doFix && issues.some(i => i.fixable)) {
|
||||
const fixed = fixContent(content);
|
||||
if (fixed !== content) {
|
||||
const fixCount = issues.filter(i => i.fixable).length;
|
||||
if (!dryRun) {
|
||||
writeFileSync(page, fixed);
|
||||
}
|
||||
console.log(` ${dryRun ? '(dry run) ' : ''}Fixed ${fixCount} issue(s)`);
|
||||
const result = await runLintCore({
|
||||
target,
|
||||
fix: doFix,
|
||||
dryRun,
|
||||
exclude: extraExcludes,
|
||||
onPagesCollected: (count) => progress.start('lint.pages', count),
|
||||
onPageScanned: () => progress.tick(1),
|
||||
onPageIssues: (relPath, issues, fixedCount) => {
|
||||
console.log(`\n${relPath}:`);
|
||||
for (const issue of issues) {
|
||||
const fixLabel = issue.fixable ? ' [fixable]' : '';
|
||||
console.log(` L${issue.line} ${issue.rule}: ${issue.message}${fixLabel}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (fixedCount > 0) {
|
||||
console.log(` ${dryRun ? '(dry run) ' : ''}Fixed ${fixedCount} issue(s)`);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
progress.finish();
|
||||
|
||||
// Re-run core for the aggregate counts (cheap; re-parses contents but
|
||||
// produces canonical numbers for the summary line).
|
||||
// Pass contentSanity through so runLintCore skips its own resolve
|
||||
// (we already resolved once for the human-detail loop above).
|
||||
const result = await runLintCore({ target, fix: doFix, dryRun, contentSanity, exclude: extraExcludes });
|
||||
console.log(`\n${result.pages_scanned} pages scanned. ${result.total_issues} issue(s) in ${result.pages_with_issues} page(s).`);
|
||||
if (doFix) {
|
||||
console.log(`${dryRun ? '(dry run) ' : ''}${result.total_fixed} auto-fixed.`);
|
||||
|
||||
@@ -101,9 +101,15 @@ function emitError(jsonOutput: boolean, code: string, message: string): void {
|
||||
}
|
||||
|
||||
async function promptYesNo(question: string): Promise<boolean> {
|
||||
// W0 fix-wave (Tier-1 #15): non-interactive stdin (CI, pipes, spawned
|
||||
// agents) must resolve to the safe default instead of hanging forever —
|
||||
// this prompt had no TTY guard and no close/EOF handler, so a piped or
|
||||
// closed stdin parked the process permanently.
|
||||
if (!process.stdin.isTTY) return false;
|
||||
// Prompt on stderr: stdout stays clean for --json payloads.
|
||||
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
||||
return new Promise((resolve) => {
|
||||
rl.on('close', () => resolve(false)); // EOF (^D) = decline, never hang
|
||||
rl.question(`${question} [y/N] `, (answer) => {
|
||||
rl.close();
|
||||
resolve(/^y(es)?$/i.test(answer.trim()));
|
||||
|
||||
@@ -331,18 +331,29 @@ function fail(jsonOutput: boolean, reason: string, message: string): never {
|
||||
}
|
||||
|
||||
async function promptYesNo(question: string): Promise<boolean> {
|
||||
// Minimal TTY prompt — no external deps. Bun's process.stdin reads
|
||||
// a single line synchronously via the async iterator.
|
||||
process.stdout.write(`${question} (y/N): `);
|
||||
// W0 fix-wave (Tier-1 #15): non-interactive stdin (CI, pipes, spawned
|
||||
// agents) resolves to the safe default instead of hanging — this prompt
|
||||
// had no TTY guard and no end/EOF path, so a closed stdin parked the
|
||||
// process permanently. This command wipes the store; decline-by-default
|
||||
// is the only safe non-interactive answer (--yes stays the escape hatch).
|
||||
if (!process.stdin.isTTY) return false;
|
||||
// Prompt on stderr so stdout stays clean for --json payloads.
|
||||
process.stderr.write(`${question} (y/N): `);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const stdin = process.stdin as any;
|
||||
stdin.setEncoding?.('utf8');
|
||||
return new Promise<boolean>((resolve) => {
|
||||
const cleanup = () => {
|
||||
stdin.off?.('data', onData);
|
||||
stdin.off?.('end', onEnd);
|
||||
};
|
||||
const onEnd = () => { cleanup(); resolve(false); }; // EOF = decline
|
||||
const onData = (chunk: string) => {
|
||||
const answer = chunk.trim().toLowerCase();
|
||||
stdin.off?.('data', onData);
|
||||
cleanup();
|
||||
resolve(answer === 'y' || answer === 'yes');
|
||||
};
|
||||
stdin.on?.('data', onData);
|
||||
stdin.on?.('end', onEnd);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -551,7 +551,13 @@ export async function queryAgentClientSpend(engine: BrainEngine): Promise<AgentC
|
||||
SELECT SUM(spend_cents)::text
|
||||
FROM mcp_spend_log
|
||||
WHERE client_id = c.client_id
|
||||
AND created_at >= date_trunc('day', now() AT TIME ZONE 'UTC')
|
||||
-- Double AT TIME ZONE: the inner one yields NAIVE UTC-midnight;
|
||||
-- the outer one converts it back to a timestamptz INSTANT. Without
|
||||
-- it, the naive value is reinterpreted in the SESSION timezone, so
|
||||
-- any non-UTC session (host-tz PGLite, a tz-configured Postgres
|
||||
-- role) shifts the day boundary by the offset and today's spend
|
||||
-- underreports every evening.
|
||||
AND created_at >= date_trunc('day', now() AT TIME ZONE 'UTC') AT TIME ZONE 'UTC'
|
||||
), '0') AS spent_cents_today,
|
||||
COALESCE((
|
||||
SELECT SUM(estimated_cents)::text
|
||||
|
||||
+264
-51
@@ -49,7 +49,7 @@ import { gbrainPath } from './config.ts';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { createProgress, type ProgressReporter } from './progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
|
||||
import { tryAcquireDbLock, reapDeadHolderLocks, type DbLockHandle } from './db-lock.ts';
|
||||
import { tryAcquireDbLock, reapDeadHolderLocks, LockStolenError, type DbLockHandle } from './db-lock.ts';
|
||||
import { assertValidSourceId } from './source-id.ts';
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────
|
||||
@@ -572,7 +572,14 @@ const getLockFilePathDefault = () => gbrainPath('cycle.lock');
|
||||
|
||||
export interface LockHandle {
|
||||
release: () => Promise<void>;
|
||||
refresh: () => Promise<void>;
|
||||
/**
|
||||
* W0 fix-wave: returns true while this holder still owns the lock. The
|
||||
* DB-backed handle runs a fenced UPDATE (db-lock.ts, D5.10) and returns
|
||||
* false after a steal; the file-lock handle rewrites its file and always
|
||||
* returns true (single-host, pid-checked at acquire). Callers that ignore
|
||||
* the boolean keep their old behavior.
|
||||
*/
|
||||
refresh: () => Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -690,6 +697,7 @@ function acquireFileLock(lockPath = getLockFilePathDefault()): LockHandle | null
|
||||
} catch {
|
||||
/* non-fatal — a next-run stale check will notice */
|
||||
}
|
||||
return true;
|
||||
},
|
||||
release: async () => {
|
||||
try {
|
||||
@@ -726,21 +734,72 @@ function acquireFileLock(lockPath = getLockFilePathDefault()): LockHandle | null
|
||||
* Returns `undefined` when there's no lock AND no outer hook so phases
|
||||
* short-circuit via their `if (!opts.yieldDuringPhase) return;` guard.
|
||||
*/
|
||||
/**
|
||||
* W0 fix-wave: combine abort signals (external caller signal + the internal
|
||||
* lock-steal controller) into one REAL AbortSignal.
|
||||
*
|
||||
* Deliberately NOT AbortSignal.any: CycleOpts.signal has always been duck-
|
||||
* typed in practice (test stubs pass `{ aborted: false }` and flip the flag;
|
||||
* pre-W0 the raw object flowed straight into checkAborted, which only reads
|
||||
* `.aborted`/`.reason`). AbortSignal.any throws ERR_INVALID_ARG_TYPE on
|
||||
* those. Manual fan-in: real signals propagate via listener; listener-less
|
||||
* stubs are polled at 50ms — semantically the flip is seen within a tick,
|
||||
* and the RETURNED signal is a genuine AbortSignal so phases can hand it to
|
||||
* fetch/timers safely.
|
||||
*
|
||||
* ALWAYS call dispose() when the consuming scope ends (runCycle's finally):
|
||||
* the forward listeners live on the CALLER's signals, and long-lived callers
|
||||
* (the autopilot daemon passes its daemon-lifetime shutdown signal into
|
||||
* every cycle tick) would otherwise accumulate one listener + captured
|
||||
* controller per invocation forever (ship-review perf catch).
|
||||
*/
|
||||
export function anyAbortSignal(signals: AbortSignal[]): { signal: AbortSignal; dispose: () => void } {
|
||||
const c = new AbortController();
|
||||
const cleanups: Array<() => void> = [];
|
||||
const forward = (s: AbortSignal) => { if (!c.signal.aborted) c.abort(s.reason); };
|
||||
for (const s of signals) {
|
||||
if (!s) continue;
|
||||
if (s.aborted) { forward(s); break; }
|
||||
if (typeof (s as Partial<AbortSignal>).addEventListener === 'function') {
|
||||
const listener = () => forward(s);
|
||||
s.addEventListener('abort', listener, { once: true });
|
||||
cleanups.push(() => s.removeEventListener('abort', listener));
|
||||
} else {
|
||||
const t = setInterval(() => { if (s.aborted) { forward(s); clearInterval(t); } }, 50);
|
||||
(t as unknown as { unref?: () => void }).unref?.();
|
||||
c.signal.addEventListener('abort', () => clearInterval(t), { once: true });
|
||||
cleanups.push(() => clearInterval(t));
|
||||
}
|
||||
}
|
||||
return {
|
||||
signal: c.signal,
|
||||
dispose: () => { for (const fn of cleanups) { try { fn(); } catch { /* best effort */ } } },
|
||||
};
|
||||
}
|
||||
|
||||
export function buildYieldDuringPhase(
|
||||
lock: LockHandle | null,
|
||||
outer?: () => Promise<void>,
|
||||
onStolen?: (err: LockStolenError) => void,
|
||||
): (() => Promise<void>) | undefined {
|
||||
if (!lock && !outer) return undefined;
|
||||
return async () => {
|
||||
if (lock) {
|
||||
try {
|
||||
await lock.refresh();
|
||||
const stillOwned = await lock.refresh();
|
||||
if (stillOwned === false) {
|
||||
// W0 (D5.10/D5.11): the fenced refresh proved the lock is gone.
|
||||
// Don't throw mid-LLM-call — signal the cycle's steal controller
|
||||
// so the run stops at the next boundary / raced await instead of
|
||||
// compounding writes against a concurrent successor.
|
||||
console.error('[cycle] lock refresh matched 0 rows — lock stolen; signaling cycle abort');
|
||||
onStolen?.(new LockStolenError('cycle-lock'));
|
||||
}
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
// Non-fatal: a refresh error doesn't crash the phase. If the
|
||||
// lock truly expired and was stolen, the next acquire by another
|
||||
// worker has already happened — let this run wind down rather
|
||||
// than throw mid-phase.
|
||||
// Non-fatal: a TRANSIENT refresh error doesn't crash the phase (it
|
||||
// is not evidence of a steal; the TTL is the backstop and the next
|
||||
// tick retries).
|
||||
console.error(`[cycle] lock refresh failed (non-fatal): ${msg}`);
|
||||
}
|
||||
}
|
||||
@@ -750,6 +809,68 @@ export function buildYieldDuringPhase(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #1 + D5.11): runCycle-owned serialized lock refresher.
|
||||
*
|
||||
* Production callers never set `CycleOpts.yieldDuringPhase`, so before this
|
||||
* timer the cycle lock was refreshed only by phases that happened to receive
|
||||
* a wrapped hook — with a 5-min TTL against 35-min subagent waits, the lock
|
||||
* was effectively NEVER refreshed in production (verified in the 2026-08-14
|
||||
* audit). This interval owns the CYCLE lock only; Minion job-lock renewal
|
||||
* stays on the yieldDuringPhase/yieldBetweenPhases hooks (the cycle.ts:618
|
||||
* decision — a background timer must not replace the phase-boundary hook).
|
||||
*
|
||||
* Serialized: at most one refresh in flight (a slow refresh never overlaps
|
||||
* the next tick). On a fenced refresh returning false, aborts `controller`
|
||||
* with a LockStolenError; a thrown (transient) refresh error is logged and
|
||||
* retried next tick — the TTL is the backstop.
|
||||
*/
|
||||
export function startCycleLockRefresher(
|
||||
lock: LockHandle,
|
||||
controller: AbortController,
|
||||
lockId: string,
|
||||
intervalMs: number = resolveCycleLockRefreshMs(),
|
||||
): () => void {
|
||||
let inFlight = false;
|
||||
const timer = setInterval(() => {
|
||||
if (inFlight || controller.signal.aborted) return;
|
||||
inFlight = true;
|
||||
void (async () => {
|
||||
try {
|
||||
const stillOwned = await lock.refresh();
|
||||
if (stillOwned === false && !controller.signal.aborted) {
|
||||
controller.abort(new LockStolenError(lockId));
|
||||
}
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.error(`[cycle] background lock refresh failed (non-fatal, retrying next tick): ${msg}`);
|
||||
} finally {
|
||||
inFlight = false;
|
||||
}
|
||||
})();
|
||||
}, intervalMs);
|
||||
// Don't let the refresher pin the event loop open past real work.
|
||||
(timer as unknown as { unref?: () => void }).unref?.();
|
||||
return () => clearInterval(timer);
|
||||
}
|
||||
|
||||
/** Refresh 6x per TTL window (~50s at the 5-min TTL), matching withRefreshingLock's cadence. */
|
||||
const CYCLE_LOCK_REFRESH_INTERVAL_MS = Math.max(15_000, LOCK_TTL_MS / 6);
|
||||
|
||||
/**
|
||||
* GBRAIN_CYCLE_LOCK_REFRESH_MS: env-only escape hatch (incident tuning +
|
||||
* deterministic tests), same posture as the GBRAIN_SYNC_* knobs. Floor of
|
||||
* 10ms guards against a zero/NaN wedging the event loop.
|
||||
*/
|
||||
function resolveCycleLockRefreshMs(): number {
|
||||
const raw = process.env.GBRAIN_CYCLE_LOCK_REFRESH_MS;
|
||||
if (raw) {
|
||||
const n = Number(raw);
|
||||
if (Number.isFinite(n) && n >= 10) return n;
|
||||
}
|
||||
return CYCLE_LOCK_REFRESH_INTERVAL_MS;
|
||||
}
|
||||
|
||||
// ─── Helpers ───────────────────────────────────────────────────────
|
||||
|
||||
function makeErrorFromException(e: unknown, fallbackClass = 'InternalError'): PhaseError {
|
||||
@@ -1712,8 +1833,18 @@ export async function runCycle(
|
||||
lock = pgliteFileLock
|
||||
? {
|
||||
refresh: async () => {
|
||||
await dbLock!.refresh();
|
||||
await pgliteFileLock!.refresh();
|
||||
// W0 (D5.10): the DB row is the authoritative multi-writer
|
||||
// identity; the file refresh is best-effort freshness. Propagate
|
||||
// the fenced result so steal detection reaches the refresher.
|
||||
// Red-team catch: NEVER rewrite the file half after the fence
|
||||
// reports loss — the unconditional rewrite let the losing
|
||||
// holder clobber the successor's file lock (our pid back in the
|
||||
// file) on the very tick it detected the steal, after which our
|
||||
// pid-checked file release would DELETE the successor's only
|
||||
// host-local protection mid-run.
|
||||
const stillOwned = await dbLock!.refresh();
|
||||
if (stillOwned) await pgliteFileLock!.refresh();
|
||||
return stillOwned;
|
||||
},
|
||||
release: async () => {
|
||||
try {
|
||||
@@ -1745,6 +1876,51 @@ export async function runCycle(
|
||||
}
|
||||
}
|
||||
|
||||
// W0 fix-wave (Tier-1 #1 + D5.11): lock-steal detection and propagation.
|
||||
//
|
||||
// refresher (50s tick) ──fenced UPDATE──▶ 0 rows? ──▶ stolen.abort(LockStolenError)
|
||||
// │ │
|
||||
// └── yieldDuringPhase hooks also report steals ────────┤
|
||||
// ▼
|
||||
// cycleSignal = any(opts.signal, stolen.signal) → checkAborted() at every
|
||||
// phase boundary; raceStolen() additionally races the 5 long-phase awaits
|
||||
// (their opts can't carry a signal yet — full threading lands in W6).
|
||||
//
|
||||
// External aborts (opts.signal) keep today's throw-out semantics; ONLY a
|
||||
// steal is caught below and returned as a structured partial report.
|
||||
const externalSignal = opts.signal;
|
||||
const stolen: AbortController | null = lock ? new AbortController() : null;
|
||||
const combinedSignal = stolen && externalSignal
|
||||
? anyAbortSignal([externalSignal, stolen.signal])
|
||||
: null;
|
||||
const cycleSignal: AbortSignal | undefined = combinedSignal
|
||||
? combinedSignal.signal
|
||||
: (stolen?.signal ?? externalSignal);
|
||||
const stopRefresher: (() => void) | undefined = lock && stolen
|
||||
? startCycleLockRefresher(lock, stolen, cycleLockIdFor(opts.sourceId))
|
||||
: undefined;
|
||||
const onStolen = stolen ? (e: LockStolenError) => { if (!stolen.signal.aborted) stolen.abort(e); } : undefined;
|
||||
const raceStolen = !stolen
|
||||
? <T,>(p: Promise<T>): Promise<T> => p
|
||||
: <T,>(p: Promise<T>): Promise<T> => {
|
||||
if (stolen.signal.aborted) return Promise.reject(stolen.signal.reason);
|
||||
let onAbort!: () => void;
|
||||
const abortP = new Promise<never>((_, rej) => {
|
||||
onAbort = () => rej(stolen.signal.reason);
|
||||
stolen.signal.addEventListener('abort', onAbort, { once: true });
|
||||
});
|
||||
return Promise.race([p, abortP]).finally(() => {
|
||||
stolen.signal.removeEventListener('abort', onAbort);
|
||||
}) as Promise<T>;
|
||||
};
|
||||
let lockStolenAbort = false;
|
||||
// Raced variant for the 5 long phases (synthesize / extract_atoms / patterns
|
||||
// / synthesize_concepts / consolidate): their opts can't carry a signal yet
|
||||
// (W6), so a steal must be able to stop the WAIT even though the phase's
|
||||
// in-flight work runs to its own bounded timeout. Steal-free cycles behave
|
||||
// byte-identically to timePhase.
|
||||
const racedTimePhase = <T,>(fn: () => Promise<T>) => raceStolen(timePhase(fn));
|
||||
|
||||
// #1972: reap dead-holder sync/cycle locks at cycle start — before the sync
|
||||
// phase needs them — so a crashed sync's stranded lock self-heals THIS tick
|
||||
// instead of waiting out its TTL. Best-effort, namespace-scoped + host-scoped;
|
||||
@@ -1766,12 +1942,12 @@ export async function runCycle(
|
||||
try {
|
||||
// ── Phase 1: lint ────────────────────────────────────────────
|
||||
if (phases.includes('lint')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (brainDir === null) {
|
||||
phaseResults.push(skipNoBrainDir('lint'));
|
||||
} else {
|
||||
progress.start('cycle.lint');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseLint(brainDir, dryRun, engine, opts.signal));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseLint(brainDir, dryRun, engine, cycleSignal));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
@@ -1781,7 +1957,7 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 2: backlinks ──────────────────────────────────────
|
||||
if (phases.includes('backlinks')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (brainDir === null) {
|
||||
phaseResults.push(skipNoBrainDir('backlinks'));
|
||||
} else {
|
||||
@@ -1806,7 +1982,7 @@ export async function runCycle(
|
||||
let syncAttempted = false;
|
||||
let synthesizeWrittenSlugs: string[] | undefined;
|
||||
if (phases.includes('sync')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'sync',
|
||||
@@ -1820,7 +1996,13 @@ export async function runCycle(
|
||||
} else {
|
||||
progress.start('cycle.sync');
|
||||
syncAttempted = true; // sync ran its work; undefined pagesAffected now means failure
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, brainDir, dryRun, pull, phases.includes('extract')));
|
||||
// Red-team catch: sync is production's LONGEST phase (resumable
|
||||
// imports can run hours) and was the one long await outside steal
|
||||
// coverage. Raced like the other five: an abandoned wait is safe —
|
||||
// sync checkpoints its progress, holds its own per-source lock (the
|
||||
// successor's sync phase skips with lock-busy), and its stall
|
||||
// watchdog bounds the dangling import. Signal threading lands in W6.
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhaseSync(engine, brainDir, dryRun, pull, phases.includes('extract')));
|
||||
result.duration_ms = duration_ms;
|
||||
// Capture changed slugs for incremental extract.
|
||||
syncPagesAffected = (result as SyncPhaseResult).pagesAffected;
|
||||
@@ -1845,10 +2027,13 @@ export async function runCycle(
|
||||
} else {
|
||||
progress.start('cycle.synthesize');
|
||||
const { runPhaseSynthesize } = await import('./cycle/synthesize.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSynthesize(engine, {
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhaseSynthesize(engine, {
|
||||
brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
// W0 (Tier-1 #1): wrap the caller hook so this phase ALSO refreshes
|
||||
// the cycle lock (pre-fix these sites passed the raw — in production
|
||||
// always-undefined — hook, so long phases never refreshed).
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase, onStolen),
|
||||
inputFile: opts.synthInputFile,
|
||||
date: opts.synthDate,
|
||||
from: opts.synthFrom,
|
||||
@@ -1873,7 +2058,7 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 5: extract (now picks up synthesize output) ───────
|
||||
if (phases.includes('extract')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'extract',
|
||||
@@ -1889,7 +2074,7 @@ export async function runCycle(
|
||||
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
|
||||
// is undefined → extract falls back to full walk (safe default).
|
||||
progress.start('cycle.extract');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, opts.signal, cycleSourceId));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, cycleSignal, cycleSourceId));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
@@ -1905,7 +2090,7 @@ export async function runCycle(
|
||||
// refuses to run while v0.31 legacy facts are pending the
|
||||
// v0_32_2 backfill (Codex R2-#7).
|
||||
if (phases.includes('extract_facts')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'extract_facts',
|
||||
@@ -1938,7 +2123,7 @@ export async function runCycle(
|
||||
const syncRanButFailed = syncAttempted && syncPagesAffected === undefined;
|
||||
const xfSlugs = syncRanButFailed ? [] : syncPagesAffected;
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, xfSlugs, opts.signal));
|
||||
runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, xfSlugs, cycleSignal));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
@@ -1959,7 +2144,7 @@ export async function runCycle(
|
||||
// resolved active pack's `phases:` list ONLY; not the extends chain
|
||||
// or borrow_from targets.
|
||||
if (phases.includes('extract_atoms')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'extract_atoms',
|
||||
@@ -1995,13 +2180,13 @@ export async function runCycle(
|
||||
...(synthesizeWrittenSlugs ?? []),
|
||||
]
|
||||
: undefined;
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseExtractAtoms(engine, {
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhaseExtractAtoms(engine, {
|
||||
brainDir: brainDir ?? undefined,
|
||||
sourceId: xaSourceId,
|
||||
dryRun,
|
||||
affectedSlugs: xaAffectedSlugs,
|
||||
// v0.41.19.0 (T3): closure refreshes cycle lock + fires outer hook.
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase),
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase, onStolen),
|
||||
// v0.41.19.0 (T4): pass same reporter (not a child — cycle.ts
|
||||
// owns start/finish; phase only ticks).
|
||||
progress,
|
||||
@@ -2019,7 +2204,7 @@ export async function runCycle(
|
||||
// BATCH_SIZE * 10 chunks per invocation so a 60s watchdog tick stays
|
||||
// responsive even on a 100K-chunk brain.
|
||||
if (phases.includes('resolve_symbol_edges')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'resolve_symbol_edges',
|
||||
@@ -2057,10 +2242,13 @@ export async function runCycle(
|
||||
} else {
|
||||
progress.start('cycle.patterns');
|
||||
const { runPhasePatterns } = await import('./cycle/patterns.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhasePatterns(engine, {
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhasePatterns(engine, {
|
||||
brainDir,
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
// W0 (Tier-1 #1): wrap the caller hook so this phase ALSO refreshes
|
||||
// the cycle lock (pre-fix these sites passed the raw — in production
|
||||
// always-undefined — hook, so long phases never refreshed).
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase, onStolen),
|
||||
once: opts.onceForPhase === 'patterns',
|
||||
deadlineAtMs: opts.deadlineAtMs ?? null,
|
||||
// #1586: scope pattern writes to the cycle's resolved source, same as
|
||||
@@ -2083,7 +2271,7 @@ export async function runCycle(
|
||||
// resolved active pack manifest; no-op when this phase isn't
|
||||
// declared. Real body in T6 — synthesize-concepts.ts is a stub today.
|
||||
if (phases.includes('synthesize_concepts')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'synthesize_concepts',
|
||||
@@ -2107,11 +2295,11 @@ export async function runCycle(
|
||||
} else {
|
||||
progress.start('cycle.synthesize_concepts');
|
||||
const { runPhaseSynthesizeConcepts } = await import('./cycle/synthesize-concepts.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSynthesizeConcepts(engine, {
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhaseSynthesizeConcepts(engine, {
|
||||
brainDir: brainDir ?? undefined,
|
||||
dryRun,
|
||||
// v0.41.19.0 (T3): closure refreshes cycle lock + fires outer hook.
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase),
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase, onStolen),
|
||||
// v0.41.19.0 (T4): pass same reporter (not a child).
|
||||
progress,
|
||||
}));
|
||||
@@ -2127,7 +2315,7 @@ export async function runCycle(
|
||||
// every page touched in this cycle. Incremental mode uses union(sync,
|
||||
// synthesize); full mode walks every page in the brain.
|
||||
if (phases.includes('recompute_emotional_weight')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'recompute_emotional_weight',
|
||||
@@ -2167,7 +2355,7 @@ export async function runCycle(
|
||||
// per cluster, INSERT into takes(kind='fact'), mark facts as
|
||||
// consolidated_into. Never DELETE — facts are the audit trail.
|
||||
if (phases.includes('consolidate')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'consolidate',
|
||||
@@ -2179,10 +2367,13 @@ export async function runCycle(
|
||||
} else {
|
||||
progress.start('cycle.consolidate');
|
||||
const { runPhaseConsolidate } = await import('./cycle/phases/consolidate.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseConsolidate(engine, {
|
||||
const { result, duration_ms } = await racedTimePhase(() => runPhaseConsolidate(engine, {
|
||||
dryRun,
|
||||
yieldDuringPhase: opts.yieldDuringPhase,
|
||||
signal: opts.signal,
|
||||
// W0 (Tier-1 #1): wrap the caller hook so this phase ALSO refreshes
|
||||
// the cycle lock (pre-fix these sites passed the raw — in production
|
||||
// always-undefined — hook, so long phases never refreshed).
|
||||
yieldDuringPhase: buildYieldDuringPhase(lock, opts.yieldDuringPhase, onStolen),
|
||||
signal: cycleSignal,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
@@ -2218,7 +2409,7 @@ export async function runCycle(
|
||||
} as never;
|
||||
|
||||
if (phases.includes('propose_takes')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
progress.start('cycle.propose_takes');
|
||||
const { runPhaseProposeTakes } = await import('./cycle/propose-takes.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseProposeTakes(calibrationCtx, { repoPath: brainDir ?? undefined }) as Promise<PhaseResult>);
|
||||
@@ -2229,7 +2420,7 @@ export async function runCycle(
|
||||
}
|
||||
|
||||
if (phases.includes('grade_takes')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
progress.start('cycle.grade_takes');
|
||||
const { runPhaseGradeTakes } = await import('./cycle/grade-takes.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseGradeTakes(calibrationCtx, {}) as Promise<PhaseResult>);
|
||||
@@ -2240,7 +2431,7 @@ export async function runCycle(
|
||||
}
|
||||
|
||||
if (phases.includes('calibration_profile')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
progress.start('cycle.calibration_profile');
|
||||
const { runPhaseCalibrationProfile } = await import('./cycle/calibration-profile.ts');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseCalibrationProfile(calibrationCtx, {}) as Promise<PhaseResult>);
|
||||
@@ -2270,7 +2461,7 @@ export async function runCycle(
|
||||
// reports/drift-<date> page, mutates no takes regardless of
|
||||
// dream.drift.auto_update.
|
||||
if (phases.includes('drift')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'drift',
|
||||
@@ -2315,7 +2506,7 @@ export async function runCycle(
|
||||
// tracker passed in from the phase wrapper (NOT nested-wrapped in
|
||||
// core — would REPLACE not stack).
|
||||
if (phases.includes('conversation_facts_backfill')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'conversation_facts_backfill',
|
||||
@@ -2330,7 +2521,7 @@ export async function runCycle(
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseConversationFactsBackfill(engine, {
|
||||
dryRun,
|
||||
signal: opts.signal,
|
||||
signal: cycleSignal,
|
||||
once: opts.onceForPhase === 'conversation_facts_backfill',
|
||||
}),
|
||||
);
|
||||
@@ -2347,7 +2538,7 @@ export async function runCycle(
|
||||
// cost AND walltime caps; budget tracker created in the phase wrapper and
|
||||
// passed into the core (NOT nested-wrapped — would REPLACE not stack).
|
||||
if (phases.includes('enrich_thin')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'enrich_thin',
|
||||
@@ -2362,7 +2553,7 @@ export async function runCycle(
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseEnrichThin(engine, {
|
||||
dryRun,
|
||||
signal: opts.signal,
|
||||
signal: cycleSignal,
|
||||
once: opts.onceForPhase === 'enrich_thin',
|
||||
}),
|
||||
);
|
||||
@@ -2379,7 +2570,7 @@ export async function runCycle(
|
||||
// safety (D16): the phase ALWAYS runs in --no-mutate mode — proposed
|
||||
// bests land at skills/<name>/skillopt/best.md for review.
|
||||
if (phases.includes('skillopt')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'skillopt' as never,
|
||||
@@ -2396,7 +2587,7 @@ export async function runCycle(
|
||||
engine,
|
||||
dryRun,
|
||||
once: opts.onceForPhase === 'skillopt',
|
||||
...(opts.signal ? { signal: opts.signal } : {}),
|
||||
...(cycleSignal ? { signal: cycleSignal } : {}),
|
||||
}),
|
||||
);
|
||||
result.duration_ms = duration_ms;
|
||||
@@ -2408,7 +2599,7 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 8: embed ──────────────────────────────────────────
|
||||
if (phases.includes('embed')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'embed',
|
||||
@@ -2419,7 +2610,7 @@ export async function runCycle(
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.embed');
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun, opts.signal));
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun, cycleSignal));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
@@ -2429,7 +2620,7 @@ export async function runCycle(
|
||||
|
||||
// ── Phase 9: orphans ────────────────────────────────────────
|
||||
if (phases.includes('orphans')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'orphans',
|
||||
@@ -2455,7 +2646,7 @@ export async function runCycle(
|
||||
// (T15) and the disk-derived candidate set surfaced by `gbrain schema
|
||||
// review-candidates`.
|
||||
if (phases.includes('schema-suggest')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'schema-suggest',
|
||||
@@ -2499,7 +2690,7 @@ export async function runCycle(
|
||||
// 72h recovery window. Runs last so the rest of the cycle sees the
|
||||
// recoverable set; the purge then drops what's truly expired.
|
||||
if (phases.includes('purge')) {
|
||||
checkAborted(opts.signal);
|
||||
checkAborted(cycleSignal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'purge',
|
||||
@@ -2517,9 +2708,31 @@ export async function runCycle(
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
} catch (e) {
|
||||
// W0 (Tier-1 #1): a lock steal aborts the run at the next boundary/raced
|
||||
// await. Completed phases' DB writes are durable (persisted-and-resumable
|
||||
// per D5.6); report a structured partial instead of throwing so daemon
|
||||
// callers (jobs.ts / autopilot) don't have to classify an exception.
|
||||
// External aborts (cycleSignal) keep the existing throw-out contract.
|
||||
const stolenFired = stolen?.signal.aborted === true
|
||||
&& stolen.signal.reason instanceof LockStolenError
|
||||
&& externalSignal?.aborted !== true;
|
||||
if (stolenFired) {
|
||||
lockStolenAbort = true;
|
||||
console.error(`[cycle] aborting: ${stolen!.signal.reason.message} — ${phaseResults.length} phase(s) completed before the steal; their writes are durable`);
|
||||
} else {
|
||||
throw e;
|
||||
}
|
||||
} finally {
|
||||
stopRefresher?.();
|
||||
// Detach the combined-signal forwarders from the CALLER's signal — the
|
||||
// autopilot daemon reuses one shutdown signal across every tick, and
|
||||
// undisposed listeners accumulate for the daemon's lifetime.
|
||||
combinedSignal?.dispose();
|
||||
if (lock) {
|
||||
try {
|
||||
// Safe after a steal: release() is fenced on (id, pid, acquired_at),
|
||||
// so it can never delete the successor's row (deletes 0 rows).
|
||||
await lock.release();
|
||||
} catch (e) {
|
||||
// #1470: best-effort, but never silent — a swallowed release failure
|
||||
@@ -2541,7 +2754,7 @@ export async function runCycle(
|
||||
// a cancelled run as a completed full cycle, which makes the next tick skip
|
||||
// work it never actually did. Treat an aborted signal as a non-success run:
|
||||
// skip the freshness stamp and report status 'partial' with reason 'aborted'.
|
||||
const aborted = opts.signal?.aborted === true;
|
||||
const aborted = cycleSignal?.aborted === true;
|
||||
|
||||
// #1972 (Decision 7A gating): attribute force-evicts. The minion worker
|
||||
// force-evicts a job 30s after abort and logs "handler ignored abort signal";
|
||||
@@ -2614,7 +2827,7 @@ export async function runCycle(
|
||||
timestamp,
|
||||
duration_ms,
|
||||
status: effectiveStatus,
|
||||
...(aborted ? { reason: 'aborted' } : stampWriteFailed ? { reason: 'stamp_write_failed' } : {}),
|
||||
...(lockStolenAbort ? { reason: 'lock_stolen' } : aborted ? { reason: 'aborted' } : stampWriteFailed ? { reason: 'stamp_write_failed' } : {}),
|
||||
...(stampWriteFailed ? { stamp_write_failed: stampWriteFailed } : {}),
|
||||
...(reapedLocks ? { reaped_dead_holder_locks: reapedLocks } : {}),
|
||||
brain_dir: opts.brainDir,
|
||||
|
||||
+77
-17
@@ -26,8 +26,38 @@ import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export interface DbLockHandle {
|
||||
id: string;
|
||||
/**
|
||||
* Per-acquisition fencing identity (W0 fix-wave, D5.10): the row's
|
||||
* acquired_at rendered as epoch-seconds text (GUC-independent — timestamptz::text
|
||||
* would vary with per-session TimeZone/DateStyle across pools), captured at
|
||||
* acquire time. refresh()
|
||||
* and release() require an exact match, so a PID-reuse impostor (or this
|
||||
* handle after a steal) can never refresh or delete a successor's row.
|
||||
* `(id, holder_pid)` alone is NOT identity — PIDs recycle.
|
||||
*/
|
||||
acquiredAt: string;
|
||||
release: () => Promise<void>;
|
||||
refresh: () => Promise<void>;
|
||||
/**
|
||||
* Bump ttl_expires_at + last_refreshed_at. Returns true when this handle
|
||||
* still owns the row (exactly one row matched the fenced predicate);
|
||||
* false means the lock was stolen or released — the caller must stop
|
||||
* relying on mutual exclusion. Transient DB errors still THROW (they are
|
||||
* not evidence of a steal; the TTL is the backstop).
|
||||
*/
|
||||
refresh: () => Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
* W0 fix-wave: thrown (or used as an AbortSignal reason) when a fenced
|
||||
* refresh discovers the lock row no longer belongs to this acquisition.
|
||||
*/
|
||||
export class LockStolenError extends Error {
|
||||
readonly lockId: string;
|
||||
constructor(lockId: string) {
|
||||
super(`lock '${lockId}' was stolen or released out from under this holder (fenced refresh matched 0 rows)`);
|
||||
this.name = 'LockStolenError';
|
||||
this.lockId = lockId;
|
||||
}
|
||||
}
|
||||
|
||||
/** Default TTL: 30 minutes, same as cycle lock. */
|
||||
@@ -205,7 +235,7 @@ export async function tryAcquireDbLock(
|
||||
// `gbrain sync --break-lock --max-age <s>` uses last_refreshed_at (not
|
||||
// acquired_at) to identify wedged-but-alive holders without stealing
|
||||
// healthy long-running holders that are actively refreshing.
|
||||
const rows: Array<{ id: string }> = await sql`
|
||||
const rows: Array<{ id: string; fence: string }> = await sql`
|
||||
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
||||
VALUES (${lockId}, ${pid}, ${host}, NOW(), NOW() + ${ttl}::interval, NOW())
|
||||
ON CONFLICT (id) DO UPDATE
|
||||
@@ -217,35 +247,47 @@ export async function tryAcquireDbLock(
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
AND (gbrain_cycle_locks.last_refreshed_at IS NULL
|
||||
OR gbrain_cycle_locks.last_refreshed_at < NOW() - ${stealGraceSeconds} * INTERVAL '1 second')
|
||||
RETURNING id
|
||||
RETURNING id, extract(epoch from acquired_at)::text AS fence
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
// Fencing identity (D5.10): acquired_at is written fresh on INSERT and on
|
||||
// every steal, so it uniquely names THIS acquisition. Rendered as
|
||||
// extract(epoch ...)::text — GUC-INDEPENDENT (ship-review catch, 3
|
||||
// specialists): plain timestamptz::text varies with per-session
|
||||
// TimeZone/DateStyle, and the fence is captured on the acquire pool but
|
||||
// compared on the direct pool; a GUC divergence would turn every fenced
|
||||
// refresh into a false steal. Epoch text is stable across sessions and
|
||||
// keeps microsecond precision.
|
||||
const fence = rows[0].fence;
|
||||
const deregister = registerCleanup(`db-lock:${lockId}`, async () => {
|
||||
await sql`
|
||||
DELETE FROM gbrain_cycle_locks
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid} AND extract(epoch from acquired_at)::text = ${fence}
|
||||
`;
|
||||
});
|
||||
return {
|
||||
id: lockId,
|
||||
acquiredAt: fence,
|
||||
refresh: async () => {
|
||||
// v0.41.13.0: bump BOTH ttl_expires_at AND last_refreshed_at.
|
||||
// v0.42.x (#1794): route through the DIRECT session pool, not the
|
||||
// transaction pool, so a Supavisor pooler exhaustion (EMAXCONNSESSION)
|
||||
// can't kill the heartbeat and let the live lock get stolen.
|
||||
await engine.executeRawDirect(
|
||||
const updated = await engine.executeRawDirect<{ id: string }>(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + ($1)::interval,
|
||||
last_refreshed_at = NOW()
|
||||
WHERE id = $2 AND holder_pid = $3`,
|
||||
[ttl, lockId, pid],
|
||||
WHERE id = $2 AND holder_pid = $3 AND extract(epoch from acquired_at)::text = $4
|
||||
RETURNING id`,
|
||||
[ttl, lockId, pid, fence],
|
||||
);
|
||||
return updated.length > 0;
|
||||
},
|
||||
release: async () => {
|
||||
deregister();
|
||||
await sql`
|
||||
DELETE FROM gbrain_cycle_locks
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid}
|
||||
WHERE id = ${lockId} AND holder_pid = ${pid} AND extract(epoch from acquired_at)::text = ${fence}
|
||||
`;
|
||||
},
|
||||
};
|
||||
@@ -266,32 +308,37 @@ export async function tryAcquireDbLock(
|
||||
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
|
||||
AND (gbrain_cycle_locks.last_refreshed_at IS NULL
|
||||
OR gbrain_cycle_locks.last_refreshed_at < NOW() - $5 * INTERVAL '1 second')
|
||||
RETURNING id`,
|
||||
RETURNING id, extract(epoch from acquired_at)::text AS fence`,
|
||||
[lockId, pid, host, ttl, stealGraceSeconds],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
// Fencing identity (D5.10) — see the postgres branch for rationale.
|
||||
const fence = String((rows[0] as { fence: string }).fence);
|
||||
const deregister = registerCleanup(`db-lock:${lockId}`, async () => {
|
||||
await db.query(
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
|
||||
[lockId, pid],
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2 AND extract(epoch from acquired_at)::text = $3`,
|
||||
[lockId, pid, fence],
|
||||
);
|
||||
});
|
||||
return {
|
||||
id: lockId,
|
||||
acquiredAt: fence,
|
||||
refresh: async () => {
|
||||
await db.query(
|
||||
const res = await db.query(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET ttl_expires_at = NOW() + $1::interval,
|
||||
last_refreshed_at = NOW()
|
||||
WHERE id = $2 AND holder_pid = $3`,
|
||||
[ttl, lockId, pid],
|
||||
WHERE id = $2 AND holder_pid = $3 AND extract(epoch from acquired_at)::text = $4
|
||||
RETURNING id`,
|
||||
[ttl, lockId, pid, fence],
|
||||
);
|
||||
return res.rows.length > 0;
|
||||
},
|
||||
release: async () => {
|
||||
deregister();
|
||||
await db.query(
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
|
||||
[lockId, pid],
|
||||
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2 AND extract(epoch from acquired_at)::text = $3`,
|
||||
[lockId, pid, fence],
|
||||
);
|
||||
},
|
||||
};
|
||||
@@ -852,7 +899,20 @@ export async function withRefreshingLock<T>(
|
||||
const timeout = new Promise<never>((_, reject) =>
|
||||
setTimeout(() => reject(new Error('refresh_timeout')), heartbeatTimeoutMs)
|
||||
);
|
||||
await Promise.race([handle.refresh(), timeout]);
|
||||
const stillOwned = await Promise.race([handle.refresh(), timeout]);
|
||||
if (stillOwned === false) {
|
||||
// W0 (D5.10): the fenced refresh matched 0 rows — the lock was
|
||||
// stolen or force-cleared. Further refreshes are pointless (and a
|
||||
// fenced refresh can never re-take the successor's row). Stop the
|
||||
// heartbeat and shout; the work itself keeps running (this wrapper
|
||||
// has no cancellation seam — W7 threads one through its consumers),
|
||||
// but mutual exclusion is GONE and the degraded-heartbeat exit
|
||||
// message names it.
|
||||
clearInterval(interval);
|
||||
healthOk = false;
|
||||
process.stderr.write(`[lock-refresh] ${lockId}: ${new LockStolenError(lockId).message} — heartbeat stopped, mutual exclusion lost\n`);
|
||||
return;
|
||||
}
|
||||
healthOk = true;
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
|
||||
+34
-18
@@ -18,12 +18,44 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { ChunkInput } from './types.ts';
|
||||
import type { Chunk, ChunkInput } from './types.ts';
|
||||
import { embedBatchWithBackoff, restampIfDemotedToTitleTier } from '../commands/embed.ts';
|
||||
import { wrapChunkTextsForStoredMode } from './embedding-context.ts';
|
||||
import { type DbPacer, createNoopPacer, observed } from './db-pacer.ts';
|
||||
import { AbortError } from './abort-check.ts';
|
||||
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #3, CONFIRMED): the ONE carry-through field list for
|
||||
* re-embed upserts. upsertChunks writes these columns as EXCLUDED.<col>
|
||||
* (overwrite, not COALESCE), so any re-embed path that omits a field resets
|
||||
* it — omitting `modality` flipped every image chunk to modality='text',
|
||||
* silently zeroing the image search arm (filter `cc.modality = 'image'`).
|
||||
* Pre-fix this list existed twice: here (correct, with modality) and in
|
||||
* commands/embed.ts preserveCodeMetadata (missing modality — the bug). Both
|
||||
* consumers now share THIS list; embedding_image is deliberately NOT carried
|
||||
* (the upsert COALESCEs it, and getChunks returns the pgvector as a string
|
||||
* which upsertChunks would mis-serialize).
|
||||
*/
|
||||
export function carryChunkMetadata(
|
||||
loaded: Pick<Partial<Chunk>,
|
||||
'modality' | 'language' | 'symbol_name' | 'symbol_type' | 'start_line'
|
||||
| 'end_line' | 'parent_symbol_path' | 'doc_comment' | 'symbol_name_qualified'>,
|
||||
base: ChunkInput,
|
||||
): ChunkInput {
|
||||
return {
|
||||
...base,
|
||||
modality: loaded.modality ?? undefined,
|
||||
language: loaded.language ?? undefined,
|
||||
symbol_name: loaded.symbol_name ?? undefined,
|
||||
symbol_type: loaded.symbol_type ?? undefined,
|
||||
start_line: loaded.start_line ?? undefined,
|
||||
end_line: loaded.end_line ?? undefined,
|
||||
parent_symbol_path: loaded.parent_symbol_path ?? undefined,
|
||||
doc_comment: loaded.doc_comment ?? undefined,
|
||||
symbol_name_qualified: loaded.symbol_name_qualified ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
/** Last visited (page_id, chunk_index) for keyset-resume across runs. */
|
||||
export interface StaleCursor {
|
||||
afterPageId: number;
|
||||
@@ -208,28 +240,12 @@ export async function embedStaleForSource(
|
||||
for (let j = 0; j < stale.length; j++) {
|
||||
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
|
||||
}
|
||||
const merged: ChunkInput[] = existing.map((c) => ({
|
||||
const merged: ChunkInput[] = existing.map((c) => carryChunkMetadata(c, {
|
||||
chunk_index: c.chunk_index,
|
||||
chunk_text: c.chunk_text,
|
||||
chunk_source: c.chunk_source,
|
||||
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
// Carry through per-chunk metadata. upsertChunks writes these as
|
||||
// EXCLUDED.<col> (not COALESCE), so omitting them here resets image
|
||||
// rows to modality='text' (breaking the image search arm's
|
||||
// modality='image' filter) and wipes code-chunk symbol metadata on
|
||||
// every embed-stale pass. embedding_image is deliberately NOT
|
||||
// carried: the upsert COALESCEs it, and getChunks returns the
|
||||
// pgvector as a string which upsertChunks would mis-serialize.
|
||||
modality: c.modality ?? undefined,
|
||||
language: c.language ?? undefined,
|
||||
symbol_name: c.symbol_name ?? undefined,
|
||||
symbol_type: c.symbol_type ?? undefined,
|
||||
start_line: c.start_line ?? undefined,
|
||||
end_line: c.end_line ?? undefined,
|
||||
parent_symbol_path: c.parent_symbol_path ?? undefined,
|
||||
doc_comment: c.doc_comment ?? undefined,
|
||||
symbol_name_qualified: c.symbol_name_qualified ?? undefined,
|
||||
}));
|
||||
await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId }));
|
||||
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
|
||||
|
||||
@@ -5710,6 +5710,90 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON minion_jobs (queue, status, updated_at);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 128,
|
||||
name: 'minion_jobs_timeout_backfill_and_duplicate_cycle_cleanup',
|
||||
// Jobs fix wave (upstream issues #2/#3) — two one-shot repairs for rows
|
||||
// that predate the fixes shipping alongside this migration:
|
||||
//
|
||||
// 1. HANDLER_DEFAULT_TIMEOUT_MS backfill. Submit-time stamping (#1737)
|
||||
// never touched already-queued rows, so their timeout_ms = NULL fell
|
||||
// to the minutes-scale null-default wall-clock sweep and long handlers
|
||||
// were dead-lettered mid-progress. Values are SNAPSHOTTED at authoring
|
||||
// time — deliberately NOT generated from the live map, so every brain
|
||||
// applies identical SQL under this version number; the claim-time
|
||||
// COALESCE in MinionQueue.claim owns all future drift. Do NOT sync
|
||||
// this list when handler-timeouts.ts changes. No timeout_at stamp for
|
||||
// already-active rows: that would arm the tighter 1x handleTimeouts
|
||||
// kill mid-flight; the 2x wall-clock bound (via non-NULL timeout_ms)
|
||||
// is the gentler, sufficient repair, and every future claim stamps
|
||||
// timeout_at correctly.
|
||||
//
|
||||
// 2. Duplicate autopilot-cycle backlog cleanup. The slot-scoped dispatch
|
||||
// guards accumulated byte-identical waiting cycles when a job stalled
|
||||
// in 'active' (unbounded — one observed brain held ~111). Cancel all
|
||||
// but the newest waiting row per (name, queue, source scope),
|
||||
// restricted to ticker-keyed rows (idempotency-key prefix heuristic:
|
||||
// manually submitted cycles carry no key unless the operator passes
|
||||
// one; mimicking the ticker prefix opts into ticker dedup semantics)
|
||||
// AND to rows without a camelCase data.sourceId — the ticker only
|
||||
// ever writes snake_case source_id, so a sourceId row is by
|
||||
// definition not ticker-provenance and must never be swept
|
||||
// (adversarial-review tightening: prevents distinct sourceId scopes
|
||||
// collapsing into the empty source_id group).
|
||||
// Rows are preserved as 'cancelled' for audit; their idempotency keys
|
||||
// free naturally via the dead/cancelled key-NULLing rule in add().
|
||||
// Leaves <=1 waiting (+ possibly 1 active) per scope — converges to
|
||||
// single-flight within one wall-clock window; the maxPending guard
|
||||
// (shipped with this wave) prevents new accumulation.
|
||||
//
|
||||
// Non-terminal status set + row-lock race-safety per the v15 precedent
|
||||
// (serializes against claim()'s FOR UPDATE SKIP LOCKED). Idempotent:
|
||||
// statement 1 re-runs match zero rows (timeout_ms IS NULL guard);
|
||||
// statement 2 re-runs keep only survivors.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
UPDATE minion_jobs
|
||||
SET timeout_ms = CASE name
|
||||
WHEN 'subagent' THEN 1800000
|
||||
WHEN 'subagent_aggregator' THEN 1800000
|
||||
WHEN 'embed-backfill' THEN 1800000
|
||||
WHEN 'autopilot-cycle' THEN 1800000
|
||||
WHEN 'autopilot-global-maintenance' THEN 1800000
|
||||
WHEN 'chronicle_extract' THEN 600000
|
||||
WHEN 'facts-absorb' THEN 600000
|
||||
WHEN 'contextual_reindex_per_chunk' THEN 3600000
|
||||
END,
|
||||
updated_at = now()
|
||||
WHERE timeout_ms IS NULL
|
||||
AND status IN ('waiting','active','delayed','waiting-children','paused')
|
||||
AND name IN ('subagent','subagent_aggregator','embed-backfill',
|
||||
'autopilot-cycle','autopilot-global-maintenance',
|
||||
'chronicle_extract','facts-absorb',
|
||||
'contextual_reindex_per_chunk');
|
||||
|
||||
UPDATE minion_jobs
|
||||
SET status = 'cancelled',
|
||||
finished_at = now(),
|
||||
updated_at = now(),
|
||||
error_text = 'v128: superseded duplicate autopilot cycle'
|
||||
WHERE status = 'waiting' AND parent_job_id IS NULL
|
||||
AND name IN ('autopilot-cycle','autopilot-global-maintenance')
|
||||
AND (idempotency_key LIKE 'autopilot-cycle:%' OR idempotency_key LIKE 'autopilot-global:%')
|
||||
AND data->>'sourceId' IS NULL
|
||||
AND id NOT IN (
|
||||
SELECT id FROM (
|
||||
SELECT DISTINCT ON (name, queue, COALESCE(data->>'source_id','')) id
|
||||
FROM minion_jobs
|
||||
WHERE status = 'waiting' AND parent_job_id IS NULL
|
||||
AND name IN ('autopilot-cycle','autopilot-global-maintenance')
|
||||
AND (idempotency_key LIKE 'autopilot-cycle:%' OR idempotency_key LIKE 'autopilot-global:%')
|
||||
AND data->>'sourceId' IS NULL
|
||||
ORDER BY name, queue, COALESCE(data->>'source_id',''), created_at DESC, id DESC
|
||||
) keep
|
||||
);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
/**
|
||||
* Backpressure audit log — operational trace for `maxWaiting` coalesce events.
|
||||
* Backpressure audit log — operational trace for `maxWaiting` AND `maxPending`
|
||||
* coalesce events.
|
||||
*
|
||||
* Mirrors the shell-audit.ts pattern (ISO-week-rotated JSONL, best-effort writes,
|
||||
* failures go to stderr but never block submission). The incident that motivated
|
||||
@@ -8,12 +9,16 @@
|
||||
* trail answers "why is queue depth steady at 2 for this name?" without any
|
||||
* doctor scan.
|
||||
*
|
||||
* A maxWaiting event carries waiting_count/max_waiting; a maxPending event
|
||||
* carries pending_count/max_pending. Both use decision:'coalesced'.
|
||||
*
|
||||
* File: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl` (override dir via
|
||||
* `GBRAIN_AUDIT_DIR` for container/sandbox deployments where `$HOME` is read-only).
|
||||
*
|
||||
* `gbrain jobs stats` will surface coalesce counts from this file in a v0.19.2+
|
||||
* follow-up (B4). The audit trail is for operators debugging live queues, not
|
||||
* for compliance — a disk-full attacker can silently disable it.
|
||||
* `gbrain jobs stats` surfaces a 24h coalesce summary from this file (the
|
||||
* Backpressure line); deeper per-decision history stays a follow-up. The audit
|
||||
* trail is for operators debugging live queues, not for compliance — a
|
||||
* disk-full attacker can silently disable it.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
@@ -24,8 +29,12 @@ export interface BackpressureAuditEvent {
|
||||
ts: string;
|
||||
queue: string;
|
||||
name: string;
|
||||
waiting_count: number;
|
||||
max_waiting: number;
|
||||
/** Present on maxWaiting coalesce events. */
|
||||
waiting_count?: number;
|
||||
max_waiting?: number;
|
||||
/** Present on maxPending (single-flight) coalesce events. */
|
||||
pending_count?: number;
|
||||
max_pending?: number;
|
||||
decision: 'coalesced';
|
||||
returned_job_id: number;
|
||||
}
|
||||
@@ -57,6 +66,89 @@ export function resolveAuditDir(): string {
|
||||
return gbrainPath('audit');
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-name coalesce counts within the window, for `jobs stats`.
|
||||
*
|
||||
* Reads the CURRENT and PREVIOUS ISO-week files — a 24h window crosses the
|
||||
* week boundary for the first day of each ISO week, and a one-file read
|
||||
* would silently under-count exactly then. Filters to the queue the stats
|
||||
* command is scoped to. Best-effort: missing/unreadable files and malformed
|
||||
* lines are skipped; callers omit the display line when the map is empty.
|
||||
*/
|
||||
export interface CoalesceSummary {
|
||||
count: number;
|
||||
/** returned_job_id of the LATEST coalesce event in the window — the
|
||||
* specific in-flight job submissions coalesced onto. Lets consumers
|
||||
* (jobs stats) scope diagnostics to the actual target instead of
|
||||
* aggregating across every source sharing the job name. */
|
||||
last_returned_job_id: number | null;
|
||||
}
|
||||
|
||||
export function readRecentCoalesceCounts(opts: {
|
||||
queue: string;
|
||||
windowMs: number;
|
||||
now?: Date;
|
||||
}): Map<string, CoalesceSummary> {
|
||||
const now = opts.now ?? new Date();
|
||||
const dir = resolveAuditDir();
|
||||
const cutoff = now.getTime() - opts.windowMs;
|
||||
const files = new Set([
|
||||
computeAuditFilename(now),
|
||||
computeAuditFilename(new Date(now.getTime() - 7 * 86400000)),
|
||||
]);
|
||||
const counts = new Map<string, CoalesceSummary>();
|
||||
const latestTs = new Map<string, number>();
|
||||
// Cap per-file reads: coalesce volume is caller-influenced (webhook-driven
|
||||
// submitters can grow the weekly file), and the newest events — the ones a
|
||||
// 24h window wants — are at the END. Reading an uncapped file into memory
|
||||
// would let the audit trail OOM the diagnostic that reads it.
|
||||
const MAX_READ_BYTES = 4 * 1024 * 1024;
|
||||
for (const filename of files) {
|
||||
let raw: string;
|
||||
try {
|
||||
const fullPath = path.join(dir, filename);
|
||||
const size = fs.statSync(fullPath).size;
|
||||
if (size > MAX_READ_BYTES) {
|
||||
const fd = fs.openSync(fullPath, 'r');
|
||||
try {
|
||||
const buf = Buffer.alloc(MAX_READ_BYTES);
|
||||
fs.readSync(fd, buf, 0, MAX_READ_BYTES, size - MAX_READ_BYTES);
|
||||
raw = buf.toString('utf8');
|
||||
// Drop the first (likely partial) line of the tail slice.
|
||||
raw = raw.slice(raw.indexOf('\n') + 1);
|
||||
} finally {
|
||||
fs.closeSync(fd);
|
||||
}
|
||||
} else {
|
||||
raw = fs.readFileSync(fullPath, 'utf8');
|
||||
}
|
||||
} catch {
|
||||
continue; // missing file for that week — fine
|
||||
}
|
||||
for (const line of raw.split('\n')) {
|
||||
if (!line.trim()) continue;
|
||||
try {
|
||||
const ev = JSON.parse(line) as BackpressureAuditEvent;
|
||||
if (ev.decision !== 'coalesced') continue;
|
||||
if (ev.queue !== opts.queue) continue;
|
||||
const t = Date.parse(ev.ts);
|
||||
if (!Number.isFinite(t) || t < cutoff || t > now.getTime()) continue;
|
||||
const prev = counts.get(ev.name);
|
||||
const entry: CoalesceSummary = prev ?? { count: 0, last_returned_job_id: null };
|
||||
entry.count += 1;
|
||||
if ((latestTs.get(ev.name) ?? -Infinity) <= t && typeof ev.returned_job_id === 'number') {
|
||||
entry.last_returned_job_id = ev.returned_job_id;
|
||||
latestTs.set(ev.name, t);
|
||||
}
|
||||
counts.set(ev.name, entry);
|
||||
} catch {
|
||||
// malformed line — skip
|
||||
}
|
||||
}
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
export function logBackpressureCoalesce(event: Omit<BackpressureAuditEvent, 'ts' | 'decision'>): void {
|
||||
const dir = resolveAuditDir();
|
||||
const filename = computeAuditFilename();
|
||||
|
||||
@@ -8,15 +8,21 @@
|
||||
* inherit that short null-default and get wall-clock-killed mid-progress —
|
||||
* one half of #1737's thrash.
|
||||
*
|
||||
* Fix: known long handlers get a sane long default STAMPED ONTO THE JOB ROW
|
||||
* at submit time (see `MinionQueue.add`). Stamping at submit (not mutating at
|
||||
* claim) keeps the wall-clock behavior stable across worker restart — the
|
||||
* value lives in `minion_jobs.timeout_ms`, not in worker memory.
|
||||
* Three layers apply the default (an explicit `opts.timeout_ms` always wins):
|
||||
*
|
||||
* Existing already-queued jobs are NOT backfilled: they keep whatever
|
||||
* `timeout_ms` they were inserted with (usually NULL → the old behavior).
|
||||
* Only NEW submissions pick up the default. An explicit `opts.timeout_ms`
|
||||
* always wins.
|
||||
* 1. SUBMIT — `MinionQueue.add` stamps the default onto the row. The value
|
||||
* lives in `minion_jobs.timeout_ms`, not worker memory, so wall-clock
|
||||
* behavior is stable across worker restart.
|
||||
* 2. CLAIM — `MinionQueue.claim` COALESCEs a NULL `timeout_ms` from this
|
||||
* map (and derives `timeout_at` from the coalesced value). This is the
|
||||
* durable invariant: it covers rows inserted before layer 1 existed and
|
||||
* any writer that bypasses add(). Persisted by the claim UPDATE, so the
|
||||
* restart-stability property holds here too.
|
||||
* 3. ONE-SHOT — migration v128 backfilled `timeout_ms` for non-terminal
|
||||
* rows that predate both layers (they would otherwise never re-claim
|
||||
* or die at the short null-default first). v128's values are a
|
||||
* deliberate authoring-time SNAPSHOT of this map — do NOT sync v128
|
||||
* when editing the map below; layer 2 owns all future drift.
|
||||
*
|
||||
* The 30-min anchor matches the explicit value cycle/patterns.ts already
|
||||
* passes for subagent jobs, so this generalizes an existing convention
|
||||
|
||||
+377
-189
@@ -11,12 +11,12 @@
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type {
|
||||
MinionJob, MinionJobInput, MinionJobStatus, InboxMessage, TokenUpdate,
|
||||
MinionQueueOpts, ChildDoneMessage, Attachment, AttachmentInput,
|
||||
MinionQueueOpts, ChildDoneMessage, ChildOutcome, Attachment, AttachmentInput,
|
||||
} from './types.ts';
|
||||
import { rowToMinionJob, rowToInboxMessage, rowToAttachment } from './types.ts';
|
||||
import { validateAttachment } from './attachments.ts';
|
||||
import { isProtectedJobName } from './protected-names.ts';
|
||||
import { defaultTimeoutMsFor } from './handler-timeouts.ts';
|
||||
import { defaultTimeoutMsFor, HANDLER_DEFAULT_TIMEOUT_MS } from './handler-timeouts.ts';
|
||||
import {
|
||||
withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay,
|
||||
isRetryableConnError,
|
||||
@@ -42,6 +42,33 @@ const DEFAULT_MAX_ATTACHMENT_BYTES = 5 * 1024 * 1024; // 5 MiB
|
||||
|
||||
const TERMINAL_STATUSES = ['completed', 'failed', 'dead', 'cancelled'] as const;
|
||||
|
||||
/** Audit payload deferred from inside the submission transaction. */
|
||||
type CoalesceAuditEvent = {
|
||||
queue: string; name: string; returned_job_id: number;
|
||||
waiting_count?: number; max_waiting?: number;
|
||||
pending_count?: number; max_pending?: number;
|
||||
};
|
||||
|
||||
/** Shared cap-hit coalesce return for the backpressure guards: hydrate the
|
||||
* existing row, stamp the non-persisted `coalesced` marker, hand the audit
|
||||
* payload to the caller's sink, return. Both maxWaiting and maxPending route
|
||||
* through here so the coalesce contract cannot drift between them.
|
||||
*
|
||||
* The sink DEFERS the audit write to after the transaction commits: the
|
||||
* audit append is filesystem I/O, and doing it while holding the advisory
|
||||
* lock + a pool connection would let a hung audit volume serialize every
|
||||
* submission for the scope (adversarial-review finding). */
|
||||
function coalesceReturn(
|
||||
row: Record<string, unknown>,
|
||||
audit: Omit<CoalesceAuditEvent, 'returned_job_id'>,
|
||||
sink: (ev: CoalesceAuditEvent) => void,
|
||||
): MinionJob {
|
||||
const coalesced = rowToMinionJob(row);
|
||||
coalesced.coalesced = true;
|
||||
sink({ ...audit, returned_job_id: coalesced.id });
|
||||
return coalesced;
|
||||
}
|
||||
|
||||
export class MinionQueue {
|
||||
readonly maxSpawnDepth: number;
|
||||
readonly maxAttachmentBytes: number;
|
||||
@@ -129,7 +156,11 @@ export class MinionQueue {
|
||||
const delayUntil = opts?.delay ? new Date(Date.now() + opts.delay) : null;
|
||||
const maxSpawnDepth = opts?.max_spawn_depth ?? this.maxSpawnDepth;
|
||||
|
||||
return this.engine.transaction(async (tx) => {
|
||||
// Set inside the transaction by a cap-hit coalesce; flushed AFTER commit
|
||||
// so audit filesystem I/O never runs while holding the advisory lock.
|
||||
let coalesceAudit: CoalesceAuditEvent | null = null;
|
||||
|
||||
const result = await this.engine.transaction(async (tx) => {
|
||||
// 1. Idempotency fast path — if a row already exists for this key, return it
|
||||
// without doing any other work. The unique partial index guarantees
|
||||
// no second row can be inserted with the same non-null key.
|
||||
@@ -151,77 +182,120 @@ export class MinionQueue {
|
||||
[existingJob.id]
|
||||
);
|
||||
} else {
|
||||
existingJob.coalesced = true;
|
||||
return existingJob;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 1b. Submission-time backpressure for high-frequency named jobs.
|
||||
// If waiting jobs for this (name, queue) already hit maxWaiting, return
|
||||
// the most-recent waiting row instead of inserting another slot.
|
||||
// Two guards share the advisory-lock machinery but differ in what they
|
||||
// count and how they scope:
|
||||
// - maxWaiting (rate cap): counts status='waiting' only. Source scope
|
||||
// is NULL-as-wildcard — a submission with no source key counts ALL
|
||||
// rows for (name, queue). Intentional; existing callers rely on it.
|
||||
// - maxPending (single-flight): counts waiting rows PLUS live-lock
|
||||
// active rows (lock_until > now()). An expired-lock active belongs
|
||||
// to a dead/blocked worker and must NOT suppress dispatch — the
|
||||
// fresh waiting row keeps feeding the waitingClaimable>0 wedge
|
||||
// detectors (supervisor watchdog, jobs stats) that a suppressed
|
||||
// queue would otherwise starve. Source scope is EXACT (NULL matches
|
||||
// only NULL-source rows), so a legacy no-source dispatch can never
|
||||
// coalesce into an arbitrary per-source row.
|
||||
//
|
||||
// Correctness: two concurrent submitters could both see waitingCount <
|
||||
// maxWaiting and both insert, violating the cap. `pg_advisory_xact_lock`
|
||||
// keyed on (name, queue) serializes concurrent count+insert decisions
|
||||
// for the SAME key while leaving different keys fully parallel. The
|
||||
// lock releases on txn commit/rollback automatically — no cleanup path
|
||||
// to leak. Cost: one no-op SELECT on the hot path per coalesce-guarded
|
||||
// submission; trivial compared to the protection.
|
||||
// Correctness: two concurrent submitters could both see count < cap and
|
||||
// both insert, violating the cap. `pg_advisory_xact_lock` keyed on
|
||||
// (name, queue, source) serializes concurrent count+insert decisions
|
||||
// for the SAME scope while leaving other scopes fully parallel; both
|
||||
// guards share the key namespace so maxWaiting and maxPending
|
||||
// submitters for one scope serialize against each other. The lock
|
||||
// releases on txn commit/rollback automatically — no cleanup path to
|
||||
// leak.
|
||||
//
|
||||
// Queue scope: the filter includes `queue=$2` so a waiting
|
||||
// Queue scope: the filters include `queue=$2` so a waiting
|
||||
// 'autopilot-cycle' in queue 'default' does NOT suppress submissions
|
||||
// to queue 'shell' with the same name. Pre-D2 code filtered on `name`
|
||||
// alone — a real cross-queue bleed that sequential tests missed.
|
||||
// to queue 'shell' with the same name (pre-D2 cross-queue bleed).
|
||||
//
|
||||
// Engine compatibility: PGLite (WASM Postgres 17) supports
|
||||
// pg_advisory_xact_lock, so this works on both engines without branching.
|
||||
if (opts?.maxWaiting !== undefined) {
|
||||
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
|
||||
if (opts?.maxWaiting !== undefined || opts?.maxPending !== undefined) {
|
||||
const backpressureQueue = opts?.queue ?? 'default';
|
||||
// Multi-source scope: jobs of the same (name, queue) but different
|
||||
// data.sourceId are independent workstreams (per-source sync/cycle).
|
||||
// Counting them together made a waiting default-source sync swallow
|
||||
// every other source's freshness sync — a secondary source sat 29h stale
|
||||
// while dispatch logs showed its syncs "dispatched" (coalesced into
|
||||
// the default row). Key the lock and the count on sourceId when the
|
||||
// submission carries one; NULL keeps legacy single-scope behavior.
|
||||
const bpSourceId = typeof (data as Record<string, unknown> | undefined)?.sourceId === 'string'
|
||||
? (data as Record<string, unknown>).sourceId as string
|
||||
// source are independent workstreams (per-source sync/cycle). Counting
|
||||
// them together made a waiting default-source sync swallow every other
|
||||
// source's freshness sync. Both payload spellings are read: sync/
|
||||
// webhook payloads carry camelCase sourceId; per-source autopilot
|
||||
// payloads carry snake_case source_id.
|
||||
const d = data as Record<string, unknown> | undefined;
|
||||
const bpSourceId = typeof d?.sourceId === 'string' ? d.sourceId as string
|
||||
: typeof d?.source_id === 'string' ? d.source_id as string
|
||||
: null;
|
||||
await tx.executeRaw(
|
||||
`SELECT pg_advisory_xact_lock(hashtext('minion_maxwaiting:' || $1 || ':' || $2 || ':' || coalesce($3, '')))`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
const waitingCountRows = await tx.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count
|
||||
FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
|
||||
if (waitingCount >= maxWaiting) {
|
||||
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
|
||||
`SELECT * FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ($3::text IS NULL OR data->>'sourceId' IS NOT DISTINCT FROM $3)
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1`,
|
||||
const scopeExact = `COALESCE(data->>'sourceId', data->>'source_id') IS NOT DISTINCT FROM $3`;
|
||||
const scopeWildcard = `($3::text IS NULL OR ${scopeExact})`;
|
||||
|
||||
// maxPending first: the stricter, in-flight-aware guard.
|
||||
if (opts?.maxPending !== undefined) {
|
||||
const maxPending = Math.max(1, Math.floor(opts.maxPending));
|
||||
const pendingCond = `(status = 'waiting' OR (status = 'active' AND lock_until > now()))`;
|
||||
const pendingCountRows = await tx.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count
|
||||
FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND ${pendingCond}
|
||||
AND ${scopeExact}`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
if (existingWaiting.length > 0) {
|
||||
const coalesced = rowToMinionJob(existingWaiting[0]);
|
||||
try {
|
||||
const { logBackpressureCoalesce } = await import('./backpressure-audit.ts');
|
||||
logBackpressureCoalesce({
|
||||
const pendingCount = parseInt(pendingCountRows[0]?.count ?? '0', 10);
|
||||
if (pendingCount >= maxPending) {
|
||||
const existingPending = await tx.executeRaw<Record<string, unknown>>(
|
||||
`SELECT * FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND ${pendingCond}
|
||||
AND ${scopeExact}
|
||||
ORDER BY CASE WHEN status = 'waiting' THEN 0 ELSE 1 END, created_at DESC, id DESC
|
||||
LIMIT 1`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
if (existingPending.length > 0) {
|
||||
return coalesceReturn(existingPending[0], {
|
||||
queue: backpressureQueue,
|
||||
name: jobName,
|
||||
pending_count: pendingCount,
|
||||
max_pending: maxPending,
|
||||
}, ev => { coalesceAudit = ev; });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (opts?.maxWaiting !== undefined) {
|
||||
const maxWaiting = Math.max(1, Math.floor(opts.maxWaiting));
|
||||
const waitingCountRows = await tx.executeRaw<{ count: string }>(
|
||||
`SELECT count(*)::text AS count
|
||||
FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ${scopeWildcard}`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
const waitingCount = parseInt(waitingCountRows[0]?.count ?? '0', 10);
|
||||
if (waitingCount >= maxWaiting) {
|
||||
const existingWaiting = await tx.executeRaw<Record<string, unknown>>(
|
||||
`SELECT * FROM minion_jobs
|
||||
WHERE name = $1 AND queue = $2 AND status = 'waiting'
|
||||
AND ${scopeWildcard}
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1`,
|
||||
[jobName, backpressureQueue, bpSourceId]
|
||||
);
|
||||
if (existingWaiting.length > 0) {
|
||||
return coalesceReturn(existingWaiting[0], {
|
||||
queue: backpressureQueue,
|
||||
name: jobName,
|
||||
waiting_count: waitingCount,
|
||||
max_waiting: maxWaiting,
|
||||
returned_job_id: coalesced.id,
|
||||
});
|
||||
} catch { /* audit failures never block submission */ }
|
||||
return coalesced;
|
||||
}, ev => { coalesceAudit = ev; });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -330,7 +404,9 @@ export class MinionQueue {
|
||||
if (existing.length === 0) {
|
||||
throw new Error(`idempotency_key ${opts.idempotency_key} insert returned no row and no existing row found`);
|
||||
}
|
||||
return rowToMinionJob(existing[0]);
|
||||
const raced = rowToMinionJob(existing[0]);
|
||||
raced.coalesced = true; // third coalesce path: lost the insert race
|
||||
return raced;
|
||||
}
|
||||
|
||||
const child = rowToMinionJob(inserted[0]);
|
||||
@@ -347,6 +423,18 @@ export class MinionQueue {
|
||||
|
||||
return child;
|
||||
});
|
||||
|
||||
// Deferred audit flush — after commit, advisory lock released, connection
|
||||
// returned to the pool. A hung/slow audit volume degrades only this one
|
||||
// submission's latency, never the queue.
|
||||
if (coalesceAudit) {
|
||||
try {
|
||||
const { logBackpressureCoalesce } = await import('./backpressure-audit.ts');
|
||||
logBackpressureCoalesce(coalesceAudit);
|
||||
} catch { /* audit failures never block submission */ }
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Get a job by ID. Returns null if not found. */
|
||||
@@ -482,7 +570,7 @@ export class MinionQueue {
|
||||
// waiting-children whose last open child we just cancelled.
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
@@ -661,6 +749,18 @@ export class MinionQueue {
|
||||
*
|
||||
* Sets timeout_at = now() + timeout_ms when the job has a per-job deadline,
|
||||
* so handleTimeouts() can dead-letter expired jobs without rereading timeout_ms.
|
||||
*
|
||||
* Claim-time budget fallback: rows inserted before the submit-time stamping
|
||||
* (or by any writer that bypasses add()) carry timeout_ms = NULL and used to
|
||||
* fall through to the minutes-scale null-default wall-clock sweep — a 30-min
|
||||
* handler died at ~5 min purely because of WHEN its row was inserted. The
|
||||
* COALESCE below resolves HANDLER_DEFAULT_TIMEOUT_MS at claim as the durable
|
||||
* invariant (the v128 migration is the one-shot repair for rows already in
|
||||
* flight). Names outside the map stay NULL — exactly today's behavior.
|
||||
* Postgres evaluates SET expressions against the OLD row, so the timeout_at
|
||||
* CASE must repeat the COALESCE rather than reference the assigned column.
|
||||
* The map binds as a RAW object (never JSON.stringify into ::jsonb — the
|
||||
* postgres.js double-encode trap; PGLite hides it, real PG does not).
|
||||
*/
|
||||
async claim(lockToken: string, lockDurationMs: number, queue: string, registeredNames: string[]): Promise<MinionJob | null> {
|
||||
if (registeredNames.length === 0) return null;
|
||||
@@ -673,8 +773,9 @@ export class MinionQueue {
|
||||
status = 'active',
|
||||
lock_token = $1,
|
||||
lock_until = now() + ($2::double precision * interval '1 millisecond'),
|
||||
timeout_at = CASE WHEN timeout_ms IS NOT NULL
|
||||
THEN now() + (timeout_ms::double precision * interval '1 millisecond')
|
||||
timeout_ms = COALESCE(timeout_ms, ($5::jsonb ->> name)::int),
|
||||
timeout_at = CASE WHEN COALESCE(timeout_ms, ($5::jsonb ->> name)::int) IS NOT NULL
|
||||
THEN now() + (COALESCE(timeout_ms, ($5::jsonb ->> name)::int)::double precision * interval '1 millisecond')
|
||||
ELSE NULL END,
|
||||
attempts_started = attempts_started + 1,
|
||||
started_at = COALESCE(started_at, now()),
|
||||
@@ -687,7 +788,7 @@ export class MinionQueue {
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING *`,
|
||||
[lockToken, lockDurationMs, queue, registeredNames]
|
||||
[lockToken, lockDurationMs, queue, registeredNames, HANDLER_DEFAULT_TIMEOUT_MS]
|
||||
);
|
||||
return rows.length > 0 ? rowToMinionJob(rows[0]) : null;
|
||||
}
|
||||
@@ -706,15 +807,26 @@ export class MinionQueue {
|
||||
*/
|
||||
async handleTimeouts(): Promise<MinionJob[]> {
|
||||
return this.engine.transaction(async (tx) => {
|
||||
// #1737: count the timed-out run as a spent attempt (terminal, no retry).
|
||||
// Safe against double-count: the worker sweep runs handleStalled ->
|
||||
// handleTimeouts -> handleWallClockTimeouts sequentially and awaited, and
|
||||
// each guards on `status = 'active'`, so the first to set status='dead'
|
||||
// excludes the row from the later sweeps.
|
||||
//
|
||||
// W0 (D5.12): candidates are discovered with a plain read, PARENTS are
|
||||
// locked first in sorted order (matching failJob's parent-before-child
|
||||
// order), and the child UPDATE re-checks every predicate under a
|
||||
// SKIP LOCKED subselect — see killJobs() for the shared tail.
|
||||
const candidates = await tx.executeRaw<{ id: number; parent_job_id: number | null }>(
|
||||
`SELECT id, parent_job_id FROM minion_jobs
|
||||
WHERE status = 'active'
|
||||
AND timeout_at IS NOT NULL
|
||||
AND timeout_at < now()
|
||||
AND lock_until > now()`
|
||||
);
|
||||
if (candidates.length === 0) return [];
|
||||
await this.lockParentsOrdered(tx, candidates);
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
// #1737: count the timed-out run as a spent attempt (terminal, no retry),
|
||||
// mirroring handleWallClockTimeouts + handleStalled. handleTimeouts is the
|
||||
// FIRST killer to fire for the long-lane handlers (timeout_ms stamped at
|
||||
// submit), so without this the job reads `attempts: 0/N (started: N)`.
|
||||
// Safe against double-count: the worker sweep runs handleStalled ->
|
||||
// handleTimeouts -> handleWallClockTimeouts sequentially and awaited, and
|
||||
// each guards on `status = 'active'`, so the first to set status='dead'
|
||||
// excludes the row from the later sweeps.
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
error_text = 'timeout exceeded',
|
||||
@@ -723,60 +835,106 @@ export class MinionQueue {
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND timeout_at IS NOT NULL
|
||||
AND timeout_at < now()
|
||||
AND lock_until > now()
|
||||
RETURNING *`
|
||||
WHERE id IN (
|
||||
SELECT id FROM minion_jobs
|
||||
WHERE id = ANY($1::bigint[])
|
||||
AND status = 'active'
|
||||
AND timeout_at IS NOT NULL
|
||||
AND timeout_at < now()
|
||||
AND lock_until > now()
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *`,
|
||||
[candidates.map(c => c.id)]
|
||||
);
|
||||
|
||||
// v0.15: emit child_done(outcome='timeout') for every timed-out job that
|
||||
// had a parent. Without this, an aggregator waiting for N child_done
|
||||
// messages hangs forever when a child times out (codex iteration 3).
|
||||
// Outcome 'timeout' is distinct from 'dead' so consumers can distinguish
|
||||
// "timed out during run" from "died via max-stall".
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: r.id as number,
|
||||
job_name: r.name as string,
|
||||
result: null,
|
||||
outcome: 'timeout',
|
||||
error: 'timeout exceeded',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
// Unblock any aggregator parents whose last open child we just killed.
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
|
||||
await this.killJobs(tx, rows, 'timeout', 'timeout exceeded');
|
||||
return rows.map(rowToMinionJob);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #4, D5.12): the ONE parent-notification tail shared
|
||||
* by every reaper that terminally kills active jobs. Pre-fix this ~45-line
|
||||
* block was hand-copied between handleTimeouts and handleWallClockTimeouts
|
||||
* (differing only in the error string), and handleStalled's dead-letter
|
||||
* branch had NO copy at all — a child that died via max-stall left its
|
||||
* aggregator parent in 'waiting-children' forever (the exact hang the v0.15
|
||||
* comment says was fixed for timeouts).
|
||||
*
|
||||
* Runs inside the caller's transaction, AFTER the child transitions.
|
||||
* Callers must have locked the parents first via lockParentsOrdered() —
|
||||
* parents-before-children is the queue-wide lock order (failJob locks the
|
||||
* parent before touching the child), so the reapers can never deadlock
|
||||
* against a concurrent failJob/completeJob.
|
||||
*
|
||||
* Emits child_done(outcome) to each non-terminal parent's inbox, then flips
|
||||
* any 'waiting-children' parent whose last open child we just killed back
|
||||
* to 'waiting'.
|
||||
*/
|
||||
private async killJobs(
|
||||
tx: Pick<BrainEngine, 'executeRaw'>,
|
||||
rows: Array<Record<string, unknown>>,
|
||||
outcome: ChildOutcome,
|
||||
errorText: string,
|
||||
): Promise<void> {
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: r.id as number,
|
||||
job_name: r.name as string,
|
||||
result: null,
|
||||
outcome,
|
||||
error: errorText,
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
// Unblock any aggregator parents whose last open child we just killed.
|
||||
for (const parentId of [...parentIds].sort((a, b) => a - b)) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* W0 (D5.12): take parent row locks in ASCENDING id order before any child
|
||||
* transition. Matches failJob's parent-first order so the three reapers and
|
||||
* failJob can never deadlock each other on parent/child lock acquisition.
|
||||
*/
|
||||
private async lockParentsOrdered(
|
||||
tx: Pick<BrainEngine, 'executeRaw'>,
|
||||
candidates: Array<{ parent_job_id: number | null }>,
|
||||
): Promise<void> {
|
||||
const parentIds = [...new Set(
|
||||
candidates.map(c => c.parent_job_id).filter((p): p is number => p != null),
|
||||
)].sort((a, b) => a - b);
|
||||
if (parentIds.length === 0) return;
|
||||
await tx.executeRaw(
|
||||
`SELECT id FROM minion_jobs WHERE id = ANY($1::bigint[]) ORDER BY id FOR UPDATE`,
|
||||
[parentIds]
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dead-letter active jobs that exceed a wall-clock runtime threshold,
|
||||
* regardless of lock state. This catches jobs stuck while still holding
|
||||
@@ -788,6 +946,21 @@ export class MinionQueue {
|
||||
*/
|
||||
async handleWallClockTimeouts(lockDurationMs: number): Promise<MinionJob[]> {
|
||||
return this.engine.transaction(async (tx) => {
|
||||
// W0 (D5.12): same parents-first discover/lock/kill shape as
|
||||
// handleTimeouts; shared tail in killJobs().
|
||||
const candidates = await tx.executeRaw<{ id: number; parent_job_id: number | null }>(
|
||||
`SELECT id, parent_job_id FROM minion_jobs
|
||||
WHERE status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
|
||||
CASE
|
||||
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
|
||||
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
|
||||
END`,
|
||||
[lockDurationMs]
|
||||
);
|
||||
if (candidates.length === 0) return [];
|
||||
await this.lockParentsOrdered(tx, candidates);
|
||||
const rows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead',
|
||||
@@ -797,54 +970,22 @@ export class MinionQueue {
|
||||
lock_until = NULL,
|
||||
finished_at = now(),
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
|
||||
CASE
|
||||
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
|
||||
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
|
||||
END
|
||||
WHERE id IN (
|
||||
SELECT id FROM minion_jobs
|
||||
WHERE id = ANY($2::bigint[])
|
||||
AND status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND EXTRACT(EPOCH FROM (now() - started_at)) * 1000 >
|
||||
CASE
|
||||
WHEN timeout_ms IS NOT NULL THEN timeout_ms * 2
|
||||
ELSE $1::double precision * 2 * GREATEST(max_stalled, 1)
|
||||
END
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *`,
|
||||
[lockDurationMs]
|
||||
[lockDurationMs, candidates.map(c => c.id)]
|
||||
);
|
||||
|
||||
const parentIds = new Set<number>();
|
||||
for (const r of rows) {
|
||||
const parentJobId = r.parent_job_id as number | null;
|
||||
if (parentJobId == null) continue;
|
||||
parentIds.add(parentJobId);
|
||||
const childDone: ChildDoneMessage = {
|
||||
type: 'child_done',
|
||||
child_id: r.id as number,
|
||||
job_name: r.name as string,
|
||||
result: null,
|
||||
outcome: 'timeout',
|
||||
error: 'wall-clock timeout exceeded',
|
||||
};
|
||||
await tx.executeRaw(
|
||||
`INSERT INTO minion_inbox (job_id, sender, payload)
|
||||
SELECT $1, 'minions', $2::jsonb
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE id = $1 AND status NOT IN ('completed','failed','dead','cancelled')
|
||||
)`,
|
||||
[parentJobId, childDone]
|
||||
);
|
||||
}
|
||||
|
||||
for (const parentId of parentIds) {
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
WHERE parent_job_id = $1
|
||||
AND status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`,
|
||||
[parentId]
|
||||
);
|
||||
}
|
||||
|
||||
await this.killJobs(tx, rows, 'timeout', 'wall-clock timeout exceeded');
|
||||
return rows.map(rowToMinionJob);
|
||||
});
|
||||
}
|
||||
@@ -933,7 +1074,7 @@ export class MinionQueue {
|
||||
// child with on_child_fail='continue'/'ignore' doesn't strand the
|
||||
// parent in waiting-children forever (v0.15 aggregator fix).
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
@@ -1000,6 +1141,7 @@ export class MinionQueue {
|
||||
stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($3::text),
|
||||
delay_until = CASE WHEN $1 = 'delayed' THEN now() + ($4::double precision * interval '1 millisecond') ELSE NULL END,
|
||||
finished_at = CASE WHEN $1 IN ('failed', 'dead') THEN now() ELSE NULL END,
|
||||
started_at = CASE WHEN $1 = 'delayed' THEN NULL ELSE started_at END,
|
||||
lock_token = NULL, lock_until = NULL, updated_at = now()
|
||||
WHERE id = $5 AND status = 'active' AND lock_token = $6
|
||||
RETURNING *`,
|
||||
@@ -1050,7 +1192,7 @@ export class MinionQueue {
|
||||
// After dropping the dep, try to resolve the parent if all OTHER
|
||||
// kids are terminal. Terminal set includes 'failed' (v0.15).
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
@@ -1068,7 +1210,7 @@ export class MinionQueue {
|
||||
// remain. Run the resolve check here so the last child transitioning
|
||||
// via THIS code path still unblocks the parent.
|
||||
await tx.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
@@ -1128,6 +1270,7 @@ export class MinionQueue {
|
||||
error_text = $1,
|
||||
stacktrace = COALESCE(stacktrace, '[]'::jsonb) || to_jsonb($1::text),
|
||||
delay_until = now() + ($2::double precision * interval '1 millisecond'),
|
||||
started_at = NULL,
|
||||
lock_token = NULL, lock_until = NULL, updated_at = now()
|
||||
WHERE id = $3 AND status = 'active' AND lock_token = $4
|
||||
RETURNING *`,
|
||||
@@ -1203,6 +1346,7 @@ export class MinionQueue {
|
||||
async promoteDelayed(): Promise<MinionJob[]> {
|
||||
const rows = await this.lockRetry(() => this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET status = 'waiting', delay_until = NULL,
|
||||
started_at = NULL,
|
||||
lock_token = NULL, lock_until = NULL, updated_at = now()
|
||||
WHERE status = 'delayed' AND delay_until <= now()
|
||||
RETURNING *`
|
||||
@@ -1212,40 +1356,84 @@ export class MinionQueue {
|
||||
|
||||
/** Detect and handle stalled jobs. Single CTE, no off-by-one. Returns affected jobs. */
|
||||
async handleStalled(): Promise<{ requeued: MinionJob[]; dead: MinionJob[] }> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown> & { action: string }>(
|
||||
`WITH stalled AS (
|
||||
SELECT id, stalled_counter, max_stalled
|
||||
FROM minion_jobs
|
||||
WHERE status = 'active' AND lock_until < now()
|
||||
FOR UPDATE SKIP LOCKED
|
||||
),
|
||||
requeued AS (
|
||||
UPDATE minion_jobs SET
|
||||
// W0 fix-wave (Tier-1 #4): the dead-letter branch previously emitted NO
|
||||
// child_done and never unblocked aggregator parents — a child that died
|
||||
// via max-stall stranded its parent in 'waiting-children' forever (the
|
||||
// exact hang the v0.15 comment says was fixed for timeouts; there was no
|
||||
// compensating sweep anywhere). Restructured into the parents-first
|
||||
// discover/lock/kill shape (D5.12) with the shared killJobs() tail.
|
||||
return this.engine.transaction(async (tx) => {
|
||||
const candidates = await tx.executeRaw<{ id: number; parent_job_id: number | null; stalled_counter: number; max_stalled: number }>(
|
||||
`SELECT id, parent_job_id, stalled_counter, max_stalled
|
||||
FROM minion_jobs
|
||||
WHERE status = 'active' AND lock_until < now()`
|
||||
);
|
||||
if (candidates.length === 0) return { requeued: [], dead: [] };
|
||||
const ids = candidates.map(c => c.id);
|
||||
// Only the dead-letter branch touches parents; lock just those, sorted.
|
||||
await this.lockParentsOrdered(
|
||||
tx,
|
||||
candidates.filter(c => Number(c.stalled_counter) + 1 >= Number(c.max_stalled)),
|
||||
);
|
||||
|
||||
const requeuedRows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'waiting', stalled_counter = stalled_counter + 1,
|
||||
started_at = NULL,
|
||||
lock_token = NULL, lock_until = NULL, updated_at = now()
|
||||
WHERE id IN (SELECT id FROM stalled WHERE stalled_counter + 1 < max_stalled)
|
||||
RETURNING *, 'requeued' as action
|
||||
),
|
||||
dead_lettered AS (
|
||||
UPDATE minion_jobs SET
|
||||
WHERE id IN (
|
||||
SELECT id FROM minion_jobs
|
||||
WHERE id = ANY($1::bigint[])
|
||||
AND status = 'active' AND lock_until < now()
|
||||
AND stalled_counter + 1 < max_stalled
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *`,
|
||||
[ids]
|
||||
);
|
||||
const deadRows = await tx.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET
|
||||
status = 'dead', stalled_counter = stalled_counter + 1,
|
||||
attempts_made = attempts_made + 1,
|
||||
error_text = 'max stalled count exceeded',
|
||||
lock_token = NULL, lock_until = NULL, finished_at = now(), updated_at = now()
|
||||
WHERE id IN (SELECT id FROM stalled WHERE stalled_counter + 1 >= max_stalled)
|
||||
RETURNING *, 'dead' as action
|
||||
)
|
||||
SELECT * FROM requeued UNION ALL SELECT * FROM dead_lettered`
|
||||
);
|
||||
|
||||
const requeued: MinionJob[] = [];
|
||||
const dead: MinionJob[] = [];
|
||||
for (const r of rows) {
|
||||
const job = rowToMinionJob(r);
|
||||
if (r.action === 'requeued') requeued.push(job);
|
||||
else dead.push(job);
|
||||
}
|
||||
return { requeued, dead };
|
||||
WHERE id IN (
|
||||
SELECT id FROM minion_jobs
|
||||
WHERE id = ANY($1::bigint[])
|
||||
AND status = 'active' AND lock_until < now()
|
||||
AND stalled_counter + 1 >= max_stalled
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING *`,
|
||||
[ids]
|
||||
);
|
||||
// THE FIX: stall-death now notifies + unblocks parents like every
|
||||
// other terminal kill. Outcome 'dead' (not 'timeout') so consumers can
|
||||
// distinguish "died via max-stall" from "timed out during run".
|
||||
await this.killJobs(tx, deadRows, 'dead', 'max stalled count exceeded');
|
||||
return { requeued: requeuedRows.map(rowToMinionJob), dead: deadRows.map(rowToMinionJob) };
|
||||
}).then(async (result) => {
|
||||
// W0 ship-review (data-migration): the per-kill unblock above is
|
||||
// forward-only — parents stranded in 'waiting-children' by PRE-upgrade
|
||||
// stall-deaths (children already status='dead') are never revisited by
|
||||
// any per-event unblock site. This idempotent sweep self-heals ALL
|
||||
// stranding classes, retroactive included, once per stall tick: any
|
||||
// waiting-children parent with zero non-terminal children flips back to
|
||||
// 'waiting'. Cheap (single UPDATE, NOT EXISTS on an indexed FK) at the
|
||||
// 30s sweep cadence.
|
||||
try {
|
||||
await this.engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs c
|
||||
WHERE c.parent_job_id = minion_jobs.id
|
||||
AND c.status NOT IN ('completed', 'failed', 'dead', 'cancelled')
|
||||
)`
|
||||
);
|
||||
} catch { /* best-effort backstop; the per-kill unblock is the primary path */ }
|
||||
return result;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1257,7 +1445,7 @@ export class MinionQueue {
|
||||
*/
|
||||
async resolveParent(parentId: number): Promise<MinionJob | null> {
|
||||
const rows = await this.engine.executeRaw<Record<string, unknown>>(
|
||||
`UPDATE minion_jobs SET status = 'waiting', updated_at = now()
|
||||
`UPDATE minion_jobs SET status = 'waiting', started_at = NULL, updated_at = now()
|
||||
WHERE id = $1 AND status = 'waiting-children'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM minion_jobs
|
||||
|
||||
@@ -839,7 +839,22 @@ export class MinionSupervisor {
|
||||
private async refreshDbLock(): Promise<void> {
|
||||
if (!this.dbLock || this.stopping) return;
|
||||
try {
|
||||
await this.dbLock.refresh();
|
||||
const stillOwned = await this.dbLock.refresh();
|
||||
if (stillOwned === false) {
|
||||
// W0 fix-wave (D5.10): the fenced refresh matched 0 rows — the lock
|
||||
// was stolen or force-cleared. That is CERTAIN loss, not a blip:
|
||||
// counting it toward the failure threshold (or worse, resetting the
|
||||
// counter as a "success") would let two supervisors drain the same
|
||||
// queue for up to two more refresh windows. Exit immediately; the
|
||||
// process manager restarts a single clean supervisor.
|
||||
this.emit('health_error', {
|
||||
reason: 'supervisor_lock_lost',
|
||||
detail: 'fenced refresh matched 0 rows (stolen or force-cleared)',
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
await this.shutdown('supervisor_lock_lost', ExitCodes.LOCK_LOST);
|
||||
return;
|
||||
}
|
||||
this.lockRefreshFailures = 0;
|
||||
} catch (e) {
|
||||
this.lockRefreshFailures++;
|
||||
|
||||
@@ -90,6 +90,14 @@ export interface MinionJob {
|
||||
started_at: Date | null;
|
||||
finished_at: Date | null;
|
||||
updated_at: Date;
|
||||
|
||||
/** Submission metadata, NOT a DB column: set only by MinionQueue.add()
|
||||
* when it returned an EXISTING row instead of inserting (idempotency
|
||||
* fast-path, backpressure cap-hit, or the ON CONFLICT zero-row fallback).
|
||||
* rowToMinionJob never sets it; absent on fresh inserts and on rows read
|
||||
* back later. Surfaces in `jobs submit`'s JSON output — intended and
|
||||
* additive, so scripts can tell a real dispatch from a coalesce. */
|
||||
coalesced?: boolean;
|
||||
}
|
||||
|
||||
// --- Input Types ---
|
||||
@@ -128,8 +136,29 @@ export interface MinionJobInput {
|
||||
max_spawn_depth?: number;
|
||||
/** Global dedup key. Same key returns the existing job, no second row created. */
|
||||
idempotency_key?: string;
|
||||
/** Submission backpressure: cap waiting jobs with this name before inserting a new row. */
|
||||
/** Submission backpressure: cap waiting jobs with this name before inserting
|
||||
* a new row. Scope is (name, queue, source), where source reads
|
||||
* data.sourceId ?? data.source_id; a submission with NO source key counts
|
||||
* ALL rows for (name, queue) — the NULL-as-wildcard arm is intentional and
|
||||
* relied on by existing rate-cap callers. For single-flight semantics see
|
||||
* maxPending (exact scoping, counts in-flight work too). */
|
||||
maxWaiting?: number;
|
||||
/** Submission single-flight: cap PENDING jobs — waiting rows plus LIVE-LOCK
|
||||
* active rows (status='active' AND lock_until > now()) — for this
|
||||
* (name, queue, source) scope before inserting a new row. Expired-lock
|
||||
* actives belong to a dead/blocked worker and never count, so a wedged
|
||||
* worker cannot suppress dispatch (new waiting rows keep feeding the
|
||||
* waitingClaimable>0 wedge detectors); dead/cancelled/completed never
|
||||
* count either. Cap-hit returns the most-recent waiting row, else the
|
||||
* most-recent live-lock active row, stamped `coalesced: true`. Scope is
|
||||
* EXACT (unlike maxWaiting): COALESCE(data.sourceId, data.source_id)
|
||||
* compared with IS NOT DISTINCT FROM — a NULL-source submission matches
|
||||
* only NULL-source rows, never a wildcard. If both maxPending and
|
||||
* maxWaiting are supplied, both guards apply; maxPending is checked
|
||||
* first. Internal option (autopilot dispatch single-flight); not exposed
|
||||
* as a public submit flag yet — semantics exclude delayed/paused/
|
||||
* waiting-children rows deliberately. */
|
||||
maxPending?: number;
|
||||
|
||||
// v12: scheduler polish
|
||||
/**
|
||||
|
||||
@@ -117,14 +117,14 @@ const PGLITE_EDGE_BATCH_MAX_BIND_PARAMS = 30_000;
|
||||
// silently fall through to a normal initSchema (snapshot is just an
|
||||
// optimization, never authoritative).
|
||||
let _snapshotWarnLogged = false;
|
||||
function tryLoadSnapshot(snapshotPath: string): Blob | null {
|
||||
export function tryLoadSnapshot(snapshotPath: string): Blob | null {
|
||||
try {
|
||||
// Lazy require so production builds without these imports don't crash.
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
const fs = require('node:fs') as typeof import('node:fs');
|
||||
const crypto = require('node:crypto') as typeof import('node:crypto');
|
||||
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts');
|
||||
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts');
|
||||
const fs = require('node:fs') as typeof import('node:fs'); // engine-dynamic-import-ok
|
||||
const crypto = require('node:crypto') as typeof import('node:crypto'); // engine-dynamic-import-ok
|
||||
const { MIGRATIONS } = require('./migrate.ts') as typeof import('./migrate.ts'); // engine-dynamic-import-ok
|
||||
const { PGLITE_SCHEMA_SQL } = require('./pglite-schema.ts') as typeof import('./pglite-schema.ts'); // engine-dynamic-import-ok
|
||||
|
||||
if (!fs.existsSync(snapshotPath)) {
|
||||
if (!_snapshotWarnLogged) {
|
||||
@@ -144,7 +144,32 @@ function tryLoadSnapshot(snapshotPath: string): Blob | null {
|
||||
return null;
|
||||
}
|
||||
const expectedHash = computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
|
||||
const actualHash = fs.readFileSync(versionPath, 'utf8').trim();
|
||||
const versionLines = fs.readFileSync(versionPath, 'utf8').trim().split('\n');
|
||||
const actualHash = versionLines[0] ?? '';
|
||||
|
||||
// W0 fix-wave: the version file's dims=/model= lines record the embedding
|
||||
// shape the snapshot was BAKED with. A snapshot whose vector(dims) columns
|
||||
// differ from what THIS process would create poisons every embedding
|
||||
// write ("expected 1280 dimensions, not 1536" — the W0 incident when the
|
||||
// fixture went default-on). Resolve our would-be shape through the same
|
||||
// gateway-with-default fallback initSchema uses and refuse a mismatch.
|
||||
// Version files without the shape lines (pre-W0) are treated as stale.
|
||||
let wantDims: number | string = DEFAULT_EMBEDDING_DIMENSIONS;
|
||||
let wantModel: string = DEFAULT_EMBEDDING_MODEL;
|
||||
try {
|
||||
const gw = require('./ai/gateway.ts') as typeof import('./ai/gateway.ts'); // engine-dynamic-import-ok
|
||||
wantDims = gw.getEmbeddingDimensions();
|
||||
wantModel = gw.getEmbeddingModel();
|
||||
} catch { /* gateway not configured — defaults, same as initSchema */ }
|
||||
const shapeOk = versionLines[1] === `dims=${wantDims}` && versionLines[2] === `model=${wantModel}`;
|
||||
if (!shapeOk) {
|
||||
if (!_snapshotWarnLogged) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(`[pglite] snapshot embedding shape mismatch (want dims=${wantDims} model=${wantModel}, have ${versionLines[1] ?? 'none'} ${versionLines[2] ?? ''}) — using normal init. Rebuild with: bun run build:pglite-snapshot`);
|
||||
_snapshotWarnLogged = true;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
if (expectedHash !== actualHash) {
|
||||
if (!_snapshotWarnLogged) {
|
||||
// eslint-disable-next-line no-console
|
||||
@@ -162,7 +187,7 @@ function tryLoadSnapshot(snapshotPath: string): Blob | null {
|
||||
}
|
||||
|
||||
export function computeSnapshotSchemaHash(
|
||||
migrations: Array<{ version: number; name: string; sql?: string; sqlFor?: { pglite?: string } }>,
|
||||
migrations: Array<{ version: number; name: string; sql?: string; sqlFor?: { pglite?: string }; handler?: unknown }>,
|
||||
schemaSQL: string,
|
||||
crypto: typeof import('node:crypto'),
|
||||
): string {
|
||||
@@ -178,6 +203,13 @@ export function computeSnapshotSchemaHash(
|
||||
hash.update(m.sql ?? '');
|
||||
hash.update('\t');
|
||||
hash.update(m.sqlFor?.pglite ?? '');
|
||||
hash.update('\t');
|
||||
// W0 fix-wave (D5.13, Codex #4): 19+ migrations carry executable
|
||||
// `handler` code with empty/absent sql — invisible to the sql-only hash,
|
||||
// so editing a handler reused a stale snapshot. Function.prototype
|
||||
// .toString folds the handler SOURCE into the hash (deterministic within
|
||||
// a checkout; this is a dev/test fixture, not a shipped artifact).
|
||||
hash.update(typeof m.handler === 'function' ? String(m.handler) : '');
|
||||
hash.update('\n');
|
||||
}
|
||||
return hash.digest('hex');
|
||||
@@ -584,6 +616,18 @@ export class PGLiteEngine implements BrainEngine {
|
||||
...embedded,
|
||||
}),
|
||||
);
|
||||
// Snapshot-timezone parity: dumpDataDir bakes the BUILD process's
|
||||
// TimeZone into the restored cluster's defaults, so a snapshot-loaded
|
||||
// engine would run sessions in the build machine's zone while a
|
||||
// cold-init engine follows this process (bun test pins TZ=UTC; bun run
|
||||
// follows the host). That divergence shifted every naive-timestamp
|
||||
// day-boundary comparison by the offset — date-dependent tests failed
|
||||
// only in the evening, only under the snapshot. Pin the session to the
|
||||
// RUNTIME zone so restored engines behave exactly like cold ones.
|
||||
if (this._snapshotLoaded && this._db) {
|
||||
const runtimeZone = Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC';
|
||||
await this._db.query(`SELECT set_config('TimeZone', $1, false)`, [runtimeZone]);
|
||||
}
|
||||
// Healthy open: close any repair episode left open by a prior failed
|
||||
// attempt (red-team: episodes otherwise stayed open forever — doctor
|
||||
// kept reporting corruption-likely and a weeks-stale episode backup
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# gbrain agent workspace — template
|
||||
|
||||
<!-- gbrain-template-stamp: 0.45.15.0 -->
|
||||
<!-- gbrain-template-stamp: 0.45.18.0 -->
|
||||
|
||||
This repository is the **"Use this template"** distribution artifact for a
|
||||
[gbrain](https://github.com/garrytan/gbrain) personal-agent workspace — the same
|
||||
|
||||
@@ -150,6 +150,41 @@ describe('queryAgentClientSpend (v0.38 Slice 4 — /admin/api/agents/spend SQL)'
|
||||
expect(rows[0].spent_cents_today).toBe(50);
|
||||
});
|
||||
|
||||
it('day boundary is a UTC INSTANT, independent of the session timezone', async () => {
|
||||
// Regression pin for the snapshot-timezone incident: the old predicate
|
||||
// compared created_at against a NAIVE date_trunc result, which the
|
||||
// session timezone reinterpreted — a non-UTC session (host-tz PGLite,
|
||||
// tz-configured Postgres role, snapshot-restored engine pre-parity-fix)
|
||||
// shifted the day boundary by its offset and underreported evening spend.
|
||||
//
|
||||
// Deterministic at ANY wall-clock hour: rows exactly AT UTC midnight and
|
||||
// 1s BEFORE it must classify identically under sessions ±12h from UTC.
|
||||
// Under the old predicate, Etc/GMT+12 excluded the midnight row and
|
||||
// Etc/GMT-12 included the pre-midnight row — one of the two always broke.
|
||||
await seedClient({ id: 'tz-edge', scope: 'read agent' });
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO mcp_spend_log (client_id, operation, spend_cents, created_at)
|
||||
VALUES
|
||||
('tz-edge', 'subagent_loop', 7,
|
||||
date_trunc('day', now() AT TIME ZONE 'UTC') AT TIME ZONE 'UTC'),
|
||||
('tz-edge', 'subagent_loop', 999,
|
||||
(date_trunc('day', now() AT TIME ZONE 'UTC') AT TIME ZONE 'UTC') - interval '1 second')`,
|
||||
);
|
||||
const original = (await engine.executeRaw<{ TimeZone: string }>(`SHOW timezone`))[0].TimeZone;
|
||||
for (const zone of ['Etc/GMT+12', 'Etc/GMT-12', 'UTC']) {
|
||||
await engine.executeRaw(`SELECT set_config('TimeZone', '${zone}', false)`);
|
||||
try {
|
||||
const rows = await queryAgentClientSpend(engine);
|
||||
const edge = rows.find(r => r.client_id === 'tz-edge')!;
|
||||
// Only the exactly-at-midnight row (7¢) counts as today — never the
|
||||
// 1s-before row (999¢) — regardless of session zone.
|
||||
expect(`${zone}:${edge.spent_cents_today}`).toBe(`${zone}:7`);
|
||||
} finally {
|
||||
await engine.executeRaw(`SELECT set_config('TimeZone', $1, false)`, [original]);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('isolates spend by client_id (no cross-client leakage)', async () => {
|
||||
await seedClient({ id: 'alice', scope: 'read agent' });
|
||||
await seedClient({ id: 'bob', scope: 'read agent' });
|
||||
|
||||
@@ -73,6 +73,28 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('lastFullCycleAt advance-gate counts coalesced ticks (parity with pre-split dispatched)', () => {
|
||||
// Before the dispatched/coalesced split, a coalesced submission advanced
|
||||
// the local full-cycle clock via result.dispatched. Without this arm, an
|
||||
// all-coalesced tick (maxPending single-flight suppression) retakes the
|
||||
// full-cycle branch every tick and starves the targeted-plan path for the
|
||||
// whole in-flight window.
|
||||
expect(AUTOPILOT_SRC).toMatch(
|
||||
/result\.dispatched\.length > 0 \|\| result\.coalesced\.length > 0 \|\| result\.legacy_fallback \|\| result\.all_sources_fresh/,
|
||||
);
|
||||
});
|
||||
|
||||
test('fanout_summary reports coalesced separately from dispatched (honest surfaces)', () => {
|
||||
expect(AUTOPILOT_SRC).toMatch(/event: 'fanout_summary',[\s\S]{0,200}coalesced: result\.coalesced/);
|
||||
});
|
||||
|
||||
test('targeted-plan dispatch honors the honest-dispatch contract (red-team finding)', () => {
|
||||
// The targeted remediation loop submits with maxWaiting:1 and expects
|
||||
// coalesces when a handler outlives one interval — its events must split
|
||||
// on job.coalesced like every other dispatch surface in this file.
|
||||
expect(AUTOPILOT_SRC).toMatch(/event: 'dispatch_coalesced',[\s\S]{0,80}mode: 'targeted'/);
|
||||
});
|
||||
|
||||
test('freshness sync dispatch uses the parsed source config for pull policy', () => {
|
||||
const freshnessIdx = AUTOPILOT_SRC.indexOf('idempotency_key: `autopilot-sync:');
|
||||
expect(freshnessIdx).toBeGreaterThan(-1);
|
||||
@@ -111,8 +133,10 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
|
||||
test('updates lastFullCycleAt after dispatch or an all-fresh restart check', () => {
|
||||
// After the dispatchPerSource call, the lastFullCycleAt module var
|
||||
// must update so the next tick doesn't immediately re-fan-out.
|
||||
// Coalesced counts as work-in-flight (see the dedicated advance-gate
|
||||
// test below for the full condition).
|
||||
expect(AUTOPILOT_SRC).toMatch(
|
||||
/result\.dispatched\.length > 0 \|\| result\.legacy_fallback \|\| result\.all_sources_fresh/,
|
||||
/result\.dispatched\.length > 0 \|\| result\.coalesced\.length > 0 \|\| result\.legacy_fallback \|\| result\.all_sources_fresh/,
|
||||
);
|
||||
expect(AUTOPILOT_SRC).toMatch(/lastFullCycleAt\s*=\s*Date\.now\(\)/);
|
||||
});
|
||||
|
||||
@@ -354,22 +354,63 @@ describe('dispatchPerSource — integration with stubbed engine + queue', () =>
|
||||
expect(parsed.pending.length).toBe(2);
|
||||
});
|
||||
|
||||
test('per-source submit MUST NOT pass maxWaiting (regression — coalesces all sources to one job)', async () => {
|
||||
// Direct unit-stub queues can't enforce maxWaiting semantics (the
|
||||
test('per-source submit MUST NOT pass maxWaiting, MUST pass maxPending: 1 (fan-out preserved + single-flight)', async () => {
|
||||
// Direct unit-stub queues can't enforce backpressure semantics (the
|
||||
// production MinionQueue implementation does), so this catches the
|
||||
// regression by inspecting the submit opts at the dispatch boundary.
|
||||
// If a future refactor re-adds maxWaiting:1 to the per-source path,
|
||||
// the production fan-out would silently coalesce N sources to ONE
|
||||
// waiting job per tick — killing the entire feature. The e2e test
|
||||
// also catches this against a real queue, but this guard fires in
|
||||
// unit tests too so the bug surfaces 100x faster.
|
||||
// maxWaiting's NULL-as-wildcard source scope would coalesce N per-source
|
||||
// jobs sharing name='autopilot-cycle' down to ONE waiting job — killing
|
||||
// the fan-out. maxPending is required instead: its EXACT source scope
|
||||
// keeps N independent per-source caps while suppressing cross-slot
|
||||
// re-dispatch when a source's cycle is still in flight (upstream
|
||||
// issue #2). The e2e test also pins both against a real queue.
|
||||
const { engine, queue, added, fanoutOpts } = makeStubs([src('a'), src('b'), src('c')]);
|
||||
await dispatchPerSource(engine, queue, fanoutOpts);
|
||||
expect(added.length).toBe(3);
|
||||
for (const job of added) {
|
||||
expect(job.opts.maxWaiting).toBeUndefined();
|
||||
expect(job.opts.maxPending).toBe(1);
|
||||
}
|
||||
});
|
||||
|
||||
test('legacy fallback submit passes maxPending: 1 (cross-slot single-flight) and no maxWaiting', async () => {
|
||||
const { engine, queue, added, fanoutOpts } = makeStubs([]);
|
||||
await dispatchPerSource(engine, queue, fanoutOpts);
|
||||
expect(added.length).toBe(1);
|
||||
expect(added[0].opts.maxPending).toBe(1);
|
||||
expect(added[0].opts.maxWaiting).toBeUndefined();
|
||||
});
|
||||
|
||||
test('coalesced submissions are reported separately and emit dispatch_coalesced', async () => {
|
||||
// Stub queue marks the second source's job as coalesced (already in
|
||||
// flight) — the fanout must not claim it as a dispatch.
|
||||
const added: Array<{ name: string; data: Record<string, unknown>; opts: Record<string, unknown> }> = [];
|
||||
const events: string[] = [];
|
||||
let nextId = 200;
|
||||
const engine = {
|
||||
kind: 'postgres' as const,
|
||||
listAllSources: async () => [src('a'), src('b')],
|
||||
getConfig: async () => null,
|
||||
executeRaw: async () => [],
|
||||
} as unknown as BrainEngine;
|
||||
const queue = {
|
||||
add: async (name: string, data: Record<string, unknown>, addOpts: Record<string, unknown>) => {
|
||||
added.push({ name, data, opts: addOpts });
|
||||
const coalesce = data.source_id === 'b';
|
||||
return { id: nextId++, ...(coalesce ? { coalesced: true } : {}) };
|
||||
},
|
||||
} as unknown as Parameters<typeof dispatchPerSource>[1];
|
||||
const result = await dispatchPerSource(engine, queue, {
|
||||
repoPath: '/tmp/brain', slot: 's', timeoutMs: 1, fanoutMax: 4, jsonMode: true,
|
||||
emit: (l: string) => events.push(l), log: () => {},
|
||||
});
|
||||
expect(result.dispatched).toEqual(['a']);
|
||||
expect(result.coalesced).toEqual(['b']);
|
||||
const kinds = events.map(e => JSON.parse(e).event);
|
||||
expect(kinds).toContain('dispatched');
|
||||
expect(kinds).toContain('dispatch_coalesced');
|
||||
});
|
||||
|
||||
test('all-fresh tick dispatches nothing (no jobs added)', async () => {
|
||||
const NOW = Date.now();
|
||||
const recent = (id: string) =>
|
||||
|
||||
@@ -86,7 +86,11 @@ describe('dispatchGlobalMaintenance — single-flight gate', () => {
|
||||
expect(added.length).toBe(1);
|
||||
expect(added[0].name).toBe('autopilot-global-maintenance');
|
||||
expect(added[0].opts.idempotency_key).toBe('autopilot-global:s1');
|
||||
expect(added[0].opts.maxWaiting).toBe(1); // structural single-flight
|
||||
// Structural single-flight: maxPending (waiting + live-lock active),
|
||||
// NOT maxWaiting — an in-flight active run must suppress re-dispatch
|
||||
// across slot rotation (upstream issue #2).
|
||||
expect(added[0].opts.maxPending).toBe(1);
|
||||
expect(added[0].opts.maxWaiting).toBeUndefined();
|
||||
expect(added[0].data.phases).toEqual(GLOBAL_PHASES);
|
||||
});
|
||||
|
||||
@@ -96,6 +100,27 @@ describe('dispatchGlobalMaintenance — single-flight gate', () => {
|
||||
expect(r.dispatched).toBe(false);
|
||||
expect(added.length).toBe(0);
|
||||
});
|
||||
|
||||
test('coalesced submission → coalesced-aware return + dispatch_coalesced event (never claims a dispatch that did not insert)', async () => {
|
||||
const events: string[] = [];
|
||||
const engine = {
|
||||
kind: 'postgres' as const,
|
||||
getConfig: async (k: string) => (k === LAST_GLOBAL_AT_KEY ? null : null),
|
||||
} as unknown as BrainEngine;
|
||||
const queue = {
|
||||
add: async () => ({ id: 7, coalesced: true }),
|
||||
} as any;
|
||||
const r = await dispatchGlobalMaintenance(engine, queue, {
|
||||
repoPath: '/tmp', slot: 's1', timeoutMs: 1, jsonMode: true, emit: (l: string) => events.push(l),
|
||||
});
|
||||
// Honest-dispatch contract: nothing was inserted, so dispatched is false;
|
||||
// coalesced says the work is already in flight.
|
||||
expect(r.dispatched).toBe(false);
|
||||
expect(r.coalesced).toBe(true);
|
||||
const kinds = events.map(e => JSON.parse(e).event);
|
||||
expect(kinds).toContain('dispatch_coalesced');
|
||||
expect(kinds).not.toContain('dispatched');
|
||||
});
|
||||
});
|
||||
|
||||
describe('dispatchPerSource — per-source jobs carry NON_GLOBAL phases (no embed)', () => {
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
/**
|
||||
* readRecentCoalesceCounts — the `jobs stats` Backpressure line's data source.
|
||||
*
|
||||
* Pins the three spec points from the review (Codex F5):
|
||||
* 1. The 24h window reads the CURRENT and PREVIOUS ISO-week files — on the
|
||||
* first day of an ISO week a one-file read silently under-counts.
|
||||
* 2. Counts are filtered to the requested queue.
|
||||
* 3. Best-effort: missing files and malformed lines are skipped, never thrown.
|
||||
*
|
||||
* Env discipline: GBRAIN_AUDIT_DIR is applied via withEnv() per test (rule
|
||||
* R1 — no bare process.env mutation in parallel test files).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { computeAuditFilename, readRecentCoalesceCounts } from '../src/core/minions/backpressure-audit.ts';
|
||||
|
||||
let dir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), 'gbrain-bp-audit-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeEvents(filename: string, events: Array<Record<string, unknown>>): void {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(join(dir, filename), events.map(e => JSON.stringify(e)).join('\n') + '\n', 'utf8');
|
||||
}
|
||||
|
||||
describe('readRecentCoalesceCounts', () => {
|
||||
test('counts per name within the window, filtered by queue; both event field shapes', async () => {
|
||||
const now = new Date('2026-08-14T12:00:00.000Z');
|
||||
const recent = (offsetMin: number) => new Date(now.getTime() - offsetMin * 60_000).toISOString();
|
||||
writeEvents(computeAuditFilename(now), [
|
||||
{ ts: recent(10), queue: 'default', name: 'autopilot-cycle', pending_count: 1, max_pending: 1, decision: 'coalesced', returned_job_id: 1 },
|
||||
{ ts: recent(20), queue: 'default', name: 'autopilot-cycle', pending_count: 1, max_pending: 1, decision: 'coalesced', returned_job_id: 1 },
|
||||
{ ts: recent(30), queue: 'default', name: 'sync', waiting_count: 1, max_waiting: 1, decision: 'coalesced', returned_job_id: 2 },
|
||||
{ ts: recent(40), queue: 'shell', name: 'autopilot-cycle', pending_count: 1, max_pending: 1, decision: 'coalesced', returned_job_id: 3 },
|
||||
{ ts: recent(25 * 60), queue: 'default', name: 'autopilot-cycle', pending_count: 1, max_pending: 1, decision: 'coalesced', returned_job_id: 4 },
|
||||
]);
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const counts = readRecentCoalesceCounts({ queue: 'default', windowMs: 24 * 3600_000, now });
|
||||
expect(counts.get('autopilot-cycle')?.count).toBe(2); // other-queue + >24h excluded
|
||||
expect(counts.get('sync')?.count).toBe(1);
|
||||
expect(counts.size).toBe(2);
|
||||
// Latest event wins the hint target: the 10-min-ago event (job 1) is
|
||||
// newer than the 20-min-ago one, and other-queue/out-of-window events
|
||||
// never contribute a target.
|
||||
expect(counts.get('autopilot-cycle')?.last_returned_job_id).toBe(1);
|
||||
expect(counts.get('sync')?.last_returned_job_id).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
test('24h window crossing an ISO-week boundary reads BOTH week files', async () => {
|
||||
// 2026-08-10 is a Monday: 06:00 Monday minus 24h lands in the previous
|
||||
// ISO week. Events split across the two files must both count.
|
||||
const now = new Date('2026-08-10T06:00:00.000Z');
|
||||
const currentFile = computeAuditFilename(now);
|
||||
const prevFile = computeAuditFilename(new Date(now.getTime() - 7 * 86400000));
|
||||
expect(prevFile).not.toBe(currentFile); // sanity: genuinely two files
|
||||
writeEvents(currentFile, [
|
||||
{ ts: new Date(now.getTime() - 3600_000).toISOString(), queue: 'default', name: 'autopilot-cycle', decision: 'coalesced', returned_job_id: 1 },
|
||||
]);
|
||||
writeEvents(prevFile, [
|
||||
// 20h ago = Sunday of the previous ISO week — inside the 24h window.
|
||||
{ ts: new Date(now.getTime() - 20 * 3600_000).toISOString(), queue: 'default', name: 'autopilot-cycle', decision: 'coalesced', returned_job_id: 2 },
|
||||
// 30h ago — outside the window, same file.
|
||||
{ ts: new Date(now.getTime() - 30 * 3600_000).toISOString(), queue: 'default', name: 'autopilot-cycle', decision: 'coalesced', returned_job_id: 3 },
|
||||
]);
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const counts = readRecentCoalesceCounts({ queue: 'default', windowMs: 24 * 3600_000, now });
|
||||
expect(counts.get('autopilot-cycle')?.count).toBe(2);
|
||||
// Latest in-window event (1h ago, current week, job 1) wins the target
|
||||
// over the older previous-week event (20h ago, job 2).
|
||||
expect(counts.get('autopilot-cycle')?.last_returned_job_id).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('missing files → empty map; malformed lines skipped', async () => {
|
||||
const now = new Date('2026-08-14T12:00:00.000Z');
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
expect(readRecentCoalesceCounts({ queue: 'default', windowMs: 24 * 3600_000, now }).size).toBe(0);
|
||||
writeFileSync(join(dir, computeAuditFilename(now)),
|
||||
'not-json\n' +
|
||||
JSON.stringify({ ts: now.toISOString(), queue: 'default', name: 'ok', decision: 'coalesced', returned_job_id: 1 }) + '\n' +
|
||||
'{"truncated":\n', 'utf8');
|
||||
const counts = readRecentCoalesceCounts({ queue: 'default', windowMs: 24 * 3600_000, now });
|
||||
expect(counts.get('ok')?.count).toBe(1);
|
||||
expect(counts.size).toBe(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -25,6 +25,7 @@ const HELP_WITHOUT_BRAIN = [
|
||||
'skillopt',
|
||||
'maintain',
|
||||
'extract-conversation-facts',
|
||||
'jobs',
|
||||
];
|
||||
|
||||
/**
|
||||
|
||||
@@ -142,14 +142,19 @@ describe('autopilot-cycle handler contract (v0.20.5)', () => {
|
||||
'utf8',
|
||||
);
|
||||
|
||||
// Count checkAborted calls in the runCycle function body
|
||||
// Count checkAborted calls in the runCycle function body. W0 fix-wave:
|
||||
// boundaries check `cycleSignal` — the combined external-caller +
|
||||
// lock-steal signal — so a stolen lock aborts at the same seams an
|
||||
// external abort always did.
|
||||
const runCycleBody = cycleSource.slice(
|
||||
cycleSource.indexOf('export async function runCycle'),
|
||||
);
|
||||
const checkCalls = (runCycleBody.match(/checkAborted\(opts\.signal\)/g) || []).length;
|
||||
const checkCalls = (runCycleBody.match(/checkAborted\(cycleSignal\)/g) || []).length;
|
||||
|
||||
// Should have at least 6 (one per phase)
|
||||
expect(checkCalls).toBeGreaterThanOrEqual(6);
|
||||
// And the combined signal must actually fold BOTH sources.
|
||||
expect(runCycleBody).toContain('anyAbortSignal([externalSignal, stolen.signal])');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -174,11 +179,12 @@ describe('#1972 — complete cooperative-abort coverage', () => {
|
||||
const fs = await import('fs');
|
||||
const src = fs.readFileSync(new URL('../src/core/cycle.ts', import.meta.url), 'utf8');
|
||||
const body = src.slice(src.indexOf('export async function runCycle'));
|
||||
// Each long phase receives the signal.
|
||||
expect(body).toContain('runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, opts.signal, cycleSourceId)');
|
||||
expect(body).toMatch(/runPhaseExtractFacts\([^)]*opts\.signal\)/);
|
||||
expect(body).toContain('signal: opts.signal'); // consolidate opts
|
||||
expect(body).toContain('runPhaseLint(brainDir, dryRun, engine, opts.signal)');
|
||||
// Each long phase receives the signal (W0: the combined cycleSignal, so
|
||||
// phases also stop on lock-steal, not just external aborts).
|
||||
expect(body).toContain('runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, cycleSignal, cycleSourceId)');
|
||||
expect(body).toMatch(/runPhaseExtractFacts\([^)]*cycleSignal\)/);
|
||||
expect(body).toContain('signal: cycleSignal'); // consolidate opts
|
||||
expect(body).toContain('runPhaseLint(brainDir, dryRun, engine, cycleSignal)');
|
||||
// Reaper runs at cycle start.
|
||||
expect(body).toContain('reapDeadHolderLocks(engine)');
|
||||
// Terminal guard: the success stamp is gated on !aborted, and the report
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* W0 ship-review coverage — anyAbortSignal behavioral tests (the combined
|
||||
* external-caller + lock-steal signal). Previously pinned only by a
|
||||
* source-string contract assertion.
|
||||
*/
|
||||
|
||||
import { test, expect } from 'bun:test';
|
||||
import { anyAbortSignal } from '../src/core/cycle.ts';
|
||||
|
||||
test('pre-aborted input aborts the combined signal immediately with the same reason', () => {
|
||||
const c = new AbortController();
|
||||
const why = new Error('already done');
|
||||
c.abort(why);
|
||||
const combined = anyAbortSignal([c.signal, new AbortController().signal]);
|
||||
try {
|
||||
expect(combined.signal.aborted).toBe(true);
|
||||
expect(combined.signal.reason).toBe(why);
|
||||
} finally {
|
||||
combined.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('a real signal aborting later propagates reason to the combined signal', async () => {
|
||||
const a = new AbortController();
|
||||
const b = new AbortController();
|
||||
const combined = anyAbortSignal([a.signal, b.signal]);
|
||||
try {
|
||||
expect(combined.signal.aborted).toBe(false);
|
||||
const why = new Error('steal');
|
||||
b.abort(why);
|
||||
expect(combined.signal.aborted).toBe(true);
|
||||
expect(combined.signal.reason).toBe(why);
|
||||
} finally {
|
||||
combined.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('duck-typed stub (no addEventListener) is observed via the 50ms poll', async () => {
|
||||
const stub = { aborted: false } as unknown as AbortSignal;
|
||||
const real = new AbortController();
|
||||
const combined = anyAbortSignal([stub, real.signal]);
|
||||
try {
|
||||
expect(combined.signal.aborted).toBe(false);
|
||||
(stub as { aborted: boolean }).aborted = true;
|
||||
const deadline = Date.now() + 2_000;
|
||||
while (!combined.signal.aborted && Date.now() < deadline) {
|
||||
await new Promise(r => setTimeout(r, 25));
|
||||
}
|
||||
expect(combined.signal.aborted).toBe(true);
|
||||
} finally {
|
||||
combined.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('dispose() detaches the forward listener from the CALLER signal (the daemon leak class)', () => {
|
||||
const daemonSignal = new AbortController();
|
||||
// Simulate many cycle ticks against one daemon-lifetime signal.
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const combined = anyAbortSignal([daemonSignal.signal, new AbortController().signal]);
|
||||
combined.dispose();
|
||||
}
|
||||
// After dispose, aborting the daemon signal must not touch the (disposed)
|
||||
// combined controllers — and no listener buildup means no
|
||||
// MaxListenersExceededWarning. Behavioral proxy: a fresh combine still
|
||||
// works and the abort propagates exactly once.
|
||||
const last = anyAbortSignal([daemonSignal.signal]);
|
||||
try {
|
||||
daemonSignal.abort(new Error('shutdown'));
|
||||
expect(last.signal.aborted).toBe(true);
|
||||
} finally {
|
||||
last.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
test('dispose() clears the stub poll interval (no immortal timers)', async () => {
|
||||
const stub = { aborted: false } as unknown as AbortSignal;
|
||||
const combined = anyAbortSignal([stub]);
|
||||
combined.dispose();
|
||||
// Flip after dispose: the poll must be dead — combined never aborts.
|
||||
(stub as { aborted: boolean }).aborted = true;
|
||||
await new Promise(r => setTimeout(r, 150));
|
||||
expect(combined.signal.aborted).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #1, D5.6 abort-determinism) — end-to-end lock-steal
|
||||
* abort through runCycle.
|
||||
*
|
||||
* A cycle whose lock row is stolen mid-run must:
|
||||
* 1. detect the steal via the background refresher (fenced refresh → false),
|
||||
* 2. stop at the next phase boundary (no further phases run),
|
||||
* 3. return a STRUCTURED partial report (reason 'lock_stolen') instead of
|
||||
* throwing — completed phases' results are present (their DB writes are
|
||||
* durable, persisted-and-resumable),
|
||||
* 4. leave the successor's lock row intact (release is fenced).
|
||||
*
|
||||
* Serial: mutates GBRAIN_HOME + GBRAIN_CYCLE_LOCK_REFRESH_MS.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runCycle } from '../src/core/cycle.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let gbrainHome: string;
|
||||
let brainDir: string;
|
||||
const PRIOR_HOME = process.env.GBRAIN_HOME;
|
||||
const PRIOR_REFRESH = process.env.GBRAIN_CYCLE_LOCK_REFRESH_MS;
|
||||
|
||||
beforeAll(async () => {
|
||||
// GBRAIN_HOME isolation so the PGLite file lock doesn't touch the real
|
||||
// ~/.gbrain/cycle.lock (same pattern as cycle-pglite-lock-ordering).
|
||||
gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-steal-home-'));
|
||||
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-steal-brain-'));
|
||||
process.env.GBRAIN_HOME = gbrainHome;
|
||||
// Deterministic fast refresher: 20ms tick against the 200ms in-test sleep.
|
||||
process.env.GBRAIN_CYCLE_LOCK_REFRESH_MS = '20';
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
if (PRIOR_HOME === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = PRIOR_HOME;
|
||||
if (PRIOR_REFRESH === undefined) delete process.env.GBRAIN_CYCLE_LOCK_REFRESH_MS; else process.env.GBRAIN_CYCLE_LOCK_REFRESH_MS = PRIOR_REFRESH;
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('mid-run steal → structured partial report, no further phases, successor row intact', async () => {
|
||||
const lockId = 'gbrain-cycle:stealtest';
|
||||
let stole = false;
|
||||
|
||||
const report = await runCycle(engine, {
|
||||
brainDir,
|
||||
sourceId: 'stealtest',
|
||||
phases: ['lint', 'backlinks'],
|
||||
yieldBetweenPhases: async () => {
|
||||
if (stole) return;
|
||||
stole = true;
|
||||
// Simulate a successor taking the row: new acquisition identity.
|
||||
await engine.executeRaw(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET acquired_at = acquired_at + INTERVAL '1 millisecond',
|
||||
ttl_expires_at = NOW() + INTERVAL '5 minutes',
|
||||
last_refreshed_at = NOW()
|
||||
WHERE id = $1`,
|
||||
[lockId],
|
||||
);
|
||||
// Give the 20ms refresher a 1.5s window (75 nominal ticks) to observe
|
||||
// the steal — sized for shard-load timer starvation, the flake class
|
||||
// db-lock-fencing.test.ts documents (a 200ms window ≈ 10 ticks proved
|
||||
// too tight under a loaded suite).
|
||||
await new Promise(r => setTimeout(r, 1_500));
|
||||
},
|
||||
});
|
||||
|
||||
expect(report.status).toBe('partial');
|
||||
expect(report.reason).toBe('lock_stolen');
|
||||
// lint completed before the steal; backlinks must never have started.
|
||||
expect(report.phases.map(p => p.phase)).toEqual(['lint']);
|
||||
|
||||
// The fenced release must NOT have deleted the successor's row.
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM gbrain_cycle_locks WHERE id = $1`,
|
||||
[lockId],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
// Cleanup for other suites.
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id = $1`, [lockId]);
|
||||
});
|
||||
|
||||
test('steal-free cycle still completes and releases normally (regression guard)', async () => {
|
||||
const report = await runCycle(engine, {
|
||||
brainDir,
|
||||
sourceId: 'stealtest2',
|
||||
phases: ['lint', 'backlinks'],
|
||||
});
|
||||
expect(report.status === 'ok' || report.status === 'clean' || report.status === 'partial').toBe(true);
|
||||
expect(report.reason).not.toBe('lock_stolen');
|
||||
expect(report.phases.map(p => p.phase)).toEqual(['lint', 'backlinks']);
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM gbrain_cycle_locks WHERE id = $1`,
|
||||
['gbrain-cycle:stealtest2'],
|
||||
);
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
@@ -15,7 +15,7 @@ import type { LockHandle } from '../../src/core/cycle.ts';
|
||||
function makeMockLock(): { lock: LockHandle; refreshCount: number; releaseCount: number } {
|
||||
const state = { refreshCount: 0, releaseCount: 0 };
|
||||
const lock: LockHandle = {
|
||||
refresh: async () => { state.refreshCount++; },
|
||||
refresh: async () => { state.refreshCount++; return true; },
|
||||
release: async () => { state.releaseCount++; },
|
||||
};
|
||||
return {
|
||||
@@ -60,7 +60,7 @@ describe('buildYieldDuringPhase (T3 codex fix)', () => {
|
||||
const outer = async () => { callOrder.push('outer'); };
|
||||
const fn = buildYieldDuringPhase({
|
||||
...tracker.lock,
|
||||
refresh: async () => { callOrder.push('refresh'); tracker.lock.refresh(); },
|
||||
refresh: async () => { callOrder.push('refresh'); return tracker.lock.refresh(); },
|
||||
}, outer);
|
||||
await fn!();
|
||||
expect(callOrder).toEqual(['refresh', 'outer']);
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #1, D5.10) — per-acquisition lock fencing.
|
||||
*
|
||||
* The refresh/release predicates require (id, holder_pid, acquired_at::text),
|
||||
* so a handle from a PREVIOUS acquisition — a PID-reuse impostor, or this
|
||||
* process after its row was stolen — can never refresh or delete a
|
||||
* successor's row. Pre-fix the predicates were (id, holder_pid) only, and an
|
||||
* expired holder could refresh a successor's lock while reporting success
|
||||
* (Codex eng-review #1 on the fix-wave plan).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { tryAcquireDbLock, LockStolenError } from '../src/core/db-lock.ts';
|
||||
import { startCycleLockRefresher, buildYieldDuringPhase, type LockHandle } from '../src/core/cycle.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'test-fence-%'`);
|
||||
});
|
||||
|
||||
/** Simulate a successor stealing the row: rewrite acquisition identity in place. */
|
||||
async function stealRow(lockId: string): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`UPDATE gbrain_cycle_locks
|
||||
SET holder_pid = holder_pid,
|
||||
acquired_at = acquired_at + INTERVAL '1 millisecond',
|
||||
ttl_expires_at = NOW() + INTERVAL '5 minutes',
|
||||
last_refreshed_at = NOW()
|
||||
WHERE id = $1`,
|
||||
[lockId],
|
||||
);
|
||||
}
|
||||
|
||||
describe('fenced refresh/release (D5.10)', () => {
|
||||
test('handle carries its acquisition fence and refresh() returns true while owned', async () => {
|
||||
const handle = await tryAcquireDbLock(engine, 'test-fence-own', 5);
|
||||
expect(handle).not.toBeNull();
|
||||
expect(typeof handle!.acquiredAt).toBe('string');
|
||||
expect(handle!.acquiredAt.length).toBeGreaterThan(0);
|
||||
expect(await handle!.refresh()).toBe(true);
|
||||
// Refresh must not change the acquisition identity.
|
||||
expect(await handle!.refresh()).toBe(true);
|
||||
await handle!.release();
|
||||
});
|
||||
|
||||
test('after a steal, the old handle refresh() returns false and cannot re-take the row', async () => {
|
||||
const handle = await tryAcquireDbLock(engine, 'test-fence-steal', 5);
|
||||
expect(handle).not.toBeNull();
|
||||
await stealRow('test-fence-steal');
|
||||
|
||||
expect(await handle!.refresh()).toBe(false);
|
||||
// The failed refresh must not have bumped the successor's TTL under the
|
||||
// OLD identity — i.e. the row still exists and refresh stays false.
|
||||
expect(await handle!.refresh()).toBe(false);
|
||||
});
|
||||
|
||||
test("release() after a steal is a fenced no-op — the successor's row survives", async () => {
|
||||
const handle = await tryAcquireDbLock(engine, 'test-fence-release', 5);
|
||||
expect(handle).not.toBeNull();
|
||||
await stealRow('test-fence-release');
|
||||
|
||||
await handle!.release(); // must NOT delete the successor's row
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM gbrain_cycle_locks WHERE id = $1`,
|
||||
['test-fence-release'],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
});
|
||||
|
||||
test('release() while still owned deletes the row (normal path unchanged)', async () => {
|
||||
const handle = await tryAcquireDbLock(engine, 'test-fence-normal', 5);
|
||||
expect(handle).not.toBeNull();
|
||||
await handle!.release();
|
||||
const rows = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM gbrain_cycle_locks WHERE id = $1`,
|
||||
['test-fence-normal'],
|
||||
);
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('startCycleLockRefresher (Tier-1 #1 + D5.11)', () => {
|
||||
const fakeLock = (impl: () => Promise<boolean>): LockHandle => ({
|
||||
refresh: impl,
|
||||
release: async () => {},
|
||||
});
|
||||
|
||||
test('aborts the controller with LockStolenError when a fenced refresh returns false', async () => {
|
||||
const controller = new AbortController();
|
||||
const stop = startCycleLockRefresher(fakeLock(async () => false), controller, 'test-lock', 15);
|
||||
try {
|
||||
// Poll instead of a fixed sleep: under full-suite shard load, timer
|
||||
// ticks can be starved well past the nominal interval.
|
||||
const deadline = Date.now() + 5_000;
|
||||
while (!controller.signal.aborted && Date.now() < deadline) {
|
||||
await new Promise(r => setTimeout(r, 25));
|
||||
}
|
||||
expect(controller.signal.aborted).toBe(true);
|
||||
expect(controller.signal.reason).toBeInstanceOf(LockStolenError);
|
||||
} finally {
|
||||
stop();
|
||||
}
|
||||
});
|
||||
|
||||
test('serializes refreshes — a slow refresh never overlaps the next tick', async () => {
|
||||
let inFlight = 0;
|
||||
let maxInFlight = 0;
|
||||
const controller = new AbortController();
|
||||
const stop = startCycleLockRefresher(fakeLock(async () => {
|
||||
inFlight++;
|
||||
maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise(r => setTimeout(r, 60));
|
||||
inFlight--;
|
||||
return true;
|
||||
}), controller, 'test-lock', 15);
|
||||
try {
|
||||
await new Promise(r => setTimeout(r, 250));
|
||||
expect(maxInFlight).toBe(1);
|
||||
expect(controller.signal.aborted).toBe(false);
|
||||
} finally {
|
||||
stop();
|
||||
}
|
||||
});
|
||||
|
||||
test('a THROWN refresh error is transient — logged, retried, never treated as a steal', async () => {
|
||||
let calls = 0;
|
||||
const controller = new AbortController();
|
||||
const stop = startCycleLockRefresher(fakeLock(async () => {
|
||||
calls++;
|
||||
throw new Error('pooler blip');
|
||||
}), controller, 'test-lock', 15);
|
||||
try {
|
||||
await new Promise(r => setTimeout(r, 120));
|
||||
expect(calls).toBeGreaterThan(1); // kept retrying
|
||||
expect(controller.signal.aborted).toBe(false);
|
||||
} finally {
|
||||
stop();
|
||||
}
|
||||
});
|
||||
|
||||
test('stop() halts ticking', async () => {
|
||||
let calls = 0;
|
||||
const controller = new AbortController();
|
||||
const stop = startCycleLockRefresher(fakeLock(async () => { calls++; return true; }), controller, 'test-lock', 15);
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
stop();
|
||||
const after = calls;
|
||||
await new Promise(r => setTimeout(r, 60));
|
||||
expect(calls).toBe(after);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildYieldDuringPhase steal reporting', () => {
|
||||
test('invokes onStolen when refresh() returns false, still fires the outer hook', async () => {
|
||||
let stolenErr: LockStolenError | undefined;
|
||||
let outerRan = false;
|
||||
const fn = buildYieldDuringPhase(
|
||||
{ refresh: async () => false, release: async () => {} },
|
||||
async () => { outerRan = true; },
|
||||
(e) => { stolenErr = e; },
|
||||
);
|
||||
await fn!();
|
||||
expect(stolenErr).toBeInstanceOf(LockStolenError);
|
||||
expect(outerRan).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT invoke onStolen on a thrown (transient) refresh error', async () => {
|
||||
let stolen = false;
|
||||
const fn = buildYieldDuringPhase(
|
||||
{ refresh: async () => { throw new Error('blip'); }, release: async () => {} },
|
||||
undefined,
|
||||
() => { stolen = true; },
|
||||
);
|
||||
await fn!();
|
||||
expect(stolen).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -83,8 +83,12 @@ describeIfDB('autopilot fan-out — Postgres E2E', () => {
|
||||
|
||||
expect(result.legacy_fallback).toBe(false);
|
||||
expect(result.dispatched.sort()).toEqual(['alpha', 'beta', 'gamma']);
|
||||
expect(result.coalesced).toEqual([]);
|
||||
|
||||
// Verify the 3 jobs land in minion_jobs with distinct idempotency keys
|
||||
// REGRESSION (fan-out preservation): the per-source path now submits with
|
||||
// maxPending: 1 — its EXACT source scope must keep N independent caps.
|
||||
// If the scope ever regressed to maxWaiting's NULL-as-wildcard shape,
|
||||
// sources beta/gamma would coalesce onto alpha's row and this would be 1.
|
||||
const jobs = await engine.executeRaw<{ name: string; data: any; idempotency_key: string }>(
|
||||
`SELECT name, data, idempotency_key FROM minion_jobs
|
||||
WHERE name = 'autopilot-cycle' ORDER BY id`,
|
||||
@@ -120,7 +124,11 @@ describeIfDB('autopilot fan-out — Postgres E2E', () => {
|
||||
const r1 = await dispatchPerSource(engine, queue, opts);
|
||||
const r2 = await dispatchPerSource(engine, queue, opts);
|
||||
expect(r1.dispatched).toEqual(['alpha']);
|
||||
expect(r2.dispatched).toEqual(['alpha']);
|
||||
// Honest dispatch surfaces: the second tick coalesced onto the existing
|
||||
// row (idempotency fast-path) — it is reported as coalesced, NOT as a
|
||||
// dispatch that didn't insert.
|
||||
expect(r2.dispatched).toEqual([]);
|
||||
expect(r2.coalesced).toEqual(['alpha']);
|
||||
// Only ONE row in minion_jobs (idempotency-key coalesce)
|
||||
const jobs = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM minion_jobs WHERE name = 'autopilot-cycle'`,
|
||||
@@ -128,6 +136,145 @@ describeIfDB('autopilot fan-out — Postgres E2E', () => {
|
||||
expect(jobs.length).toBe(1);
|
||||
});
|
||||
|
||||
test('issue-#2 regression: stalled ACTIVE cycle suppresses cross-slot re-dispatch; dead frees the cap', async () => {
|
||||
await seedSource('stuck');
|
||||
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
|
||||
// The #2194 failure cooldown is an INDEPENDENT gate: a freshly dead
|
||||
// autopilot-cycle row puts the source into backoff, which would mask the
|
||||
// property under test (the maxPending CAP freeing on dead). Disable it.
|
||||
await engine.setConfig('autopilot.failure_cooldown_min', '0');
|
||||
const queue = new MinionQueue(engine);
|
||||
const mkOpts = (slot: string) => ({
|
||||
repoPath: '/tmp', slot, timeoutMs: 60_000, fanoutMax: 10, jsonMode: true,
|
||||
emit: () => {}, log: () => {},
|
||||
});
|
||||
|
||||
// Tick 1 dispatches; the job is then claimed and stalls in 'active' with
|
||||
// a LIVE lock (worker renewing) — the incident shape.
|
||||
const r1 = await dispatchPerSource(engine, queue, mkOpts('slot-A'));
|
||||
expect(r1.dispatched).toEqual(['stuck']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'active', lock_token = 'stuck-worker',
|
||||
lock_until = now() + interval '5 minutes', started_at = now()
|
||||
WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
|
||||
// Tick 2 in a DIFFERENT slot: the rotated idempotency key would have
|
||||
// minted a fresh duplicate forever (the ~111-row incident); maxPending
|
||||
// now coalesces onto the in-flight active row. Row count stays 1.
|
||||
const r2 = await dispatchPerSource(engine, queue, mkOpts('slot-B'));
|
||||
expect(r2.dispatched).toEqual([]);
|
||||
expect(r2.coalesced).toEqual(['stuck']);
|
||||
let rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(1);
|
||||
|
||||
// Dead-letter frees the cap: tick 3 dispatches a fresh row.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'dead', finished_at = now() WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
const r3 = await dispatchPerSource(engine, queue, mkOpts('slot-C'));
|
||||
expect(r3.dispatched).toEqual(['stuck']);
|
||||
rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle' AND status = 'waiting'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(1);
|
||||
});
|
||||
|
||||
test('recovery loop: legacy NULL-budget row → claim stamps budget → live lock suppresses → expiry frees → stall requeue re-suppresses', async () => {
|
||||
// The mechanism test (Codex C9/F3): claim() is driven directly with a
|
||||
// token and NO renewer — a live worker would renew the manually expired
|
||||
// lock and mask the recovery path.
|
||||
await engine.executeRaw(`UPDATE sources SET local_path = NULL WHERE id = 'default'`);
|
||||
const queue = new MinionQueue(engine);
|
||||
const mkOpts = (slot: string) => ({
|
||||
repoPath: '/tmp/legacy', slot, timeoutMs: 60_000, fanoutMax: 10, jsonMode: true,
|
||||
emit: () => {}, log: () => {},
|
||||
});
|
||||
|
||||
// Phase 0: a legacy row queued before budget stamping existed.
|
||||
const r0 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-0'));
|
||||
expect(r0.legacy_fallback).toBe(true);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET timeout_ms = NULL, timeout_at = NULL WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
|
||||
// Phase 1: a REAL claim stamps the handler budget (claim-time fallback)
|
||||
// and holds a live lock — dispatch in a new slot coalesces, no insert.
|
||||
const claimed = await queue.claim('rl-token', 60_000, 'default', ['autopilot-cycle']);
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.timeout_ms).toBe(30 * 60 * 1000);
|
||||
expect(claimed!.timeout_at).not.toBeNull();
|
||||
const r1 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-1'));
|
||||
expect(r1.dispatched).toEqual([]);
|
||||
let rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(1);
|
||||
|
||||
// Phase 2: the worker dies — lock expires without renewal. An
|
||||
// expired-lock active must NOT suppress (wedge detectors stay fed):
|
||||
// the next slot INSERTS a fresh waiting row.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET lock_until = now() - interval '1 second'
|
||||
WHERE id = $1`, [claimed!.id],
|
||||
);
|
||||
const r2 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-2'));
|
||||
expect(r2.legacy_fallback).toBe(true);
|
||||
rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(2);
|
||||
|
||||
// Phase 3: the real sweep requeues the stalled original (max_stalled
|
||||
// default 5 → requeue, not dead-letter). Two waiting rows in scope →
|
||||
// the next dispatch coalesces onto the newest; no third row.
|
||||
const stalled = await queue.handleStalled();
|
||||
expect(stalled.requeued.map(j => j.id)).toContain(claimed!.id);
|
||||
const requeued = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status FROM minion_jobs WHERE id = $1`, [claimed!.id],
|
||||
);
|
||||
expect(requeued[0].status).toBe('waiting');
|
||||
const r3 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-3'));
|
||||
expect(r3.dispatched).toEqual([]);
|
||||
rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(2);
|
||||
|
||||
// Phase 4: force the requeued original terminal → single-flight converges
|
||||
// (one waiting row remains and still suppresses new dispatch).
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'dead', finished_at = now() WHERE id = $1`, [claimed!.id],
|
||||
);
|
||||
const r4 = await dispatchPerSource(engine, queue, mkOpts('rl-slot-4'));
|
||||
expect(r4.dispatched).toEqual([]);
|
||||
rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'autopilot-cycle' AND status = 'waiting'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(1);
|
||||
});
|
||||
|
||||
test('concurrent same-scope submissions hold maxPending: 1 on real Postgres (advisory-lock guarantee)', async () => {
|
||||
// PGLite is single-writer, so its Promise.all race is only a smoke test.
|
||||
// The postgres.js pool gives genuine concurrent connections — this is
|
||||
// the test that actually validates the pg_advisory_xact_lock serialization.
|
||||
const queue = new MinionQueue(engine);
|
||||
const results = await Promise.all(
|
||||
Array.from({ length: 8 }, () =>
|
||||
queue.add('race-single-flight', { source_id: 'race-src' }, { maxPending: 1 })),
|
||||
);
|
||||
const ids = new Set(results.map(r => r.id));
|
||||
expect(ids.size).toBe(1);
|
||||
const rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT count(*)::text AS n FROM minion_jobs WHERE name = 'race-single-flight'`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(1);
|
||||
// Exactly one submission inserted; the other seven carry coalesce metadata.
|
||||
expect(results.filter(r => r.coalesced).length).toBe(7);
|
||||
});
|
||||
|
||||
test('source with last_full_cycle_at < 60min ago is skipped by gate', async () => {
|
||||
const recent = new Date(Date.now() - 30 * 60 * 1000).toISOString();
|
||||
await seedSource('fresh');
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #3, CONFIRMED) — re-embed upserts must carry
|
||||
* `modality`, or image chunks flip to modality='text' and the image search
|
||||
* arm (filter cc.modality='image') silently returns nothing.
|
||||
*
|
||||
* Pins two layers:
|
||||
* 1. carryChunkMetadata (the now-single shared field list) carries modality
|
||||
* + all 8 code-metadata fields.
|
||||
* 2. The write-side contract that makes the carry load-bearing: upsertChunks
|
||||
* OVERWRITES modality from EXCLUDED, so a merge built via
|
||||
* carryChunkMetadata round-trips an image chunk intact, while a merge
|
||||
* that omits the field (the pre-fix CLI behavior) demonstrably resets it.
|
||||
*/
|
||||
|
||||
import { test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { carryChunkMetadata } from '../src/core/embed-stale.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('carryChunkMetadata carries modality and all code-metadata fields', () => {
|
||||
const loaded = {
|
||||
modality: 'image' as const,
|
||||
language: 'ts',
|
||||
symbol_name: 'fn',
|
||||
symbol_type: 'function',
|
||||
start_line: 1,
|
||||
end_line: 10,
|
||||
parent_symbol_path: ['Mod'],
|
||||
doc_comment: 'doc',
|
||||
symbol_name_qualified: 'Mod.fn',
|
||||
};
|
||||
const base: ChunkInput = { chunk_index: 0, chunk_text: 'x', chunk_source: 'compiled_truth' };
|
||||
const out = carryChunkMetadata(loaded, base);
|
||||
expect(out.modality).toBe('image');
|
||||
expect(out.language).toBe('ts');
|
||||
expect(out.symbol_name_qualified).toBe('Mod.fn');
|
||||
// Absent fields stay undefined, never null-through (DB rows can carry null
|
||||
// at runtime; the ?? undefined in the carry normalizes them).
|
||||
const sparse = carryChunkMetadata({}, base);
|
||||
expect(sparse.modality).toBeUndefined();
|
||||
});
|
||||
|
||||
test('re-embed merge via carryChunkMetadata keeps an image chunk image', async () => {
|
||||
await engine.putPage('img-page', { type: 'note', title: 'img', compiled_truth: 'ocr text here', frontmatter: {} });
|
||||
await engine.upsertChunks('img-page', [
|
||||
{ chunk_index: 0, chunk_text: 'ocr text here', chunk_source: 'compiled_truth', modality: 'image' },
|
||||
]);
|
||||
|
||||
// Simulate the embed --stale merge: load chunks, rebuild ChunkInputs, upsert.
|
||||
const loaded = await engine.getChunks('img-page');
|
||||
expect(loaded[0]!.modality).toBe('image');
|
||||
const merged = loaded.map(c => carryChunkMetadata(c, {
|
||||
chunk_index: c.chunk_index,
|
||||
chunk_text: c.chunk_text,
|
||||
chunk_source: c.chunk_source,
|
||||
token_count: c.token_count || 1,
|
||||
}));
|
||||
await engine.upsertChunks('img-page', merged);
|
||||
|
||||
const after = await engine.getChunks('img-page');
|
||||
expect(after[0]!.modality).toBe('image');
|
||||
});
|
||||
|
||||
test('write-side contract: omitting modality (pre-fix behavior) resets it to text', async () => {
|
||||
await engine.putPage('img-page-2', { type: 'note', title: 'img2', compiled_truth: 'more ocr', frontmatter: {} });
|
||||
await engine.upsertChunks('img-page-2', [
|
||||
{ chunk_index: 0, chunk_text: 'more ocr', chunk_source: 'compiled_truth', modality: 'image' },
|
||||
]);
|
||||
// The pre-fix CLI merge: field list without modality.
|
||||
await engine.upsertChunks('img-page-2', [
|
||||
{ chunk_index: 0, chunk_text: 'more ocr', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
const after = await engine.getChunks('img-page-2');
|
||||
// This assertion documents WHY the carry is load-bearing: the upsert
|
||||
// overwrites, so the omission class corrupts. If this ever flips to
|
||||
// COALESCE semantics, the carry (and this test) can be revisited.
|
||||
expect(after[0]!.modality).toBe('text');
|
||||
});
|
||||
@@ -12,6 +12,7 @@
|
||||
|
||||
import { test, expect, describe, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import {
|
||||
readContentChunksEmbeddingDim,
|
||||
embeddingMismatchMessage,
|
||||
@@ -73,8 +74,12 @@ describe('readContentChunksEmbeddingDim', () => {
|
||||
test('returns { exists: false, dims: null } on a fresh brain (no initSchema)', async () => {
|
||||
// One-off engine for the fresh-brain case. Never call initSchema so
|
||||
// content_chunks doesn't exist yet. Cleaned up at end of test.
|
||||
// W0: the default-on snapshot loads a fully-migrated schema at connect,
|
||||
// which breaks this test's truly-empty-DB premise — opt out for this boot.
|
||||
const fresh = new PGLiteEngine();
|
||||
await fresh.connect({});
|
||||
await withEnv({ GBRAIN_PGLITE_SNAPSHOT: undefined }, async () => {
|
||||
await fresh.connect({});
|
||||
});
|
||||
try {
|
||||
const result = await readContentChunksEmbeddingDim(fresh);
|
||||
expect(result.exists).toBe(false);
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
// Guard self-test fixture (known-BAD): positional $N::jsonb + JSON.stringify
|
||||
// in the same call span — the #2339 shape that aborted every sync.
|
||||
declare const engine: { executeRaw: (sql: string, params?: unknown[]) => Promise<unknown[]> };
|
||||
declare const x: Record<string, unknown>;
|
||||
export async function bad(): Promise<void> {
|
||||
await engine.executeRaw(`UPDATE op_checkpoints SET pin = $1::jsonb WHERE id = $2`, [JSON.stringify(x), 1]);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
// Guard self-test fixture (known-GOOD): the text-hop positional spelling.
|
||||
declare const engine: { executeRaw: (sql: string, params?: unknown[]) => Promise<unknown[]> };
|
||||
declare const x: Record<string, unknown>;
|
||||
export async function good(): Promise<void> {
|
||||
await engine.executeRaw(`UPDATE op_checkpoints SET pin = $1::text::jsonb WHERE id = $2`, [JSON.stringify(x), 1]);
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
// Guard self-test fixture (known-BAD): template-interpolated stringify into a
|
||||
// direct ::jsonb cast, WITH nested parens (the pre-W0 regex hole).
|
||||
declare const sql: (strings: TemplateStringsArray, ...vals: unknown[]) => Promise<unknown>;
|
||||
declare const obj: { get: () => unknown };
|
||||
export async function bad(): Promise<void> {
|
||||
await sql`UPDATE pages SET frontmatter = ${JSON.stringify(obj.get())}::jsonb WHERE id = 1`;
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
// Guard self-test fixture (known-GOOD): the safe text-hop spelling — binds as
|
||||
// text, the cast parses it (no direct )}::jsonb adjacency).
|
||||
declare const sql: (strings: TemplateStringsArray, ...vals: unknown[]) => Promise<unknown>;
|
||||
declare const obj: { get: () => unknown };
|
||||
export async function good(): Promise<void> {
|
||||
await sql`UPDATE pages SET frontmatter = ${JSON.stringify(obj.get())}::text::jsonb WHERE id = 1`;
|
||||
}
|
||||
|
||||
// Multi-interpolation line: a SAFE ::text::jsonb stringify followed by a
|
||||
// separate paren-bearing non-stringify ::jsonb interpolation on the SAME
|
||||
// line. A greedy argument matcher spans from stringify( to the second
|
||||
// interpolation's `)}::jsonb` and false-positives; the bracket-bounded
|
||||
// pattern must not (ship-review edge case, proven by repro).
|
||||
declare function getRaw(): unknown;
|
||||
export async function goodMultiInterpolation(): Promise<void> {
|
||||
await sql`UPDATE pages SET a = ${JSON.stringify(obj.get())}::text::jsonb, b = ${getRaw()}::jsonb WHERE id = 2`;
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
// Guard self-test fixture (known-BAD, multi-line): withRetry( on one line and
|
||||
// the engine batch call two lines later — exercises the perl window pass
|
||||
// (the single-line grep cannot see this shape).
|
||||
declare const engine: { upsertChunks: (slug: string, chunks: unknown[]) => Promise<void> };
|
||||
declare function withRetry<T>(fn: () => Promise<T>, opts?: unknown): Promise<T>;
|
||||
declare const chunks: unknown[];
|
||||
export async function badMultiline(): Promise<void> {
|
||||
await withRetry(
|
||||
async () =>
|
||||
engine.upsertChunks('slug', chunks),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
// Guard self-test fixture (known-BAD): the CANONICAL banned shape — the exact
|
||||
// arrow-fn form the pre-W0 regex could not match ([^)]* stopped at `() =>`).
|
||||
declare const engine: { addLinksBatch: (rows: unknown[]) => Promise<void> };
|
||||
declare function withRetry<T>(fn: () => Promise<T>, opts?: unknown): Promise<T>;
|
||||
declare const rows: unknown[];
|
||||
declare const BULK_RETRY_OPTS: unknown;
|
||||
export async function bad(): Promise<void> {
|
||||
await withRetry(() => engine.addLinksBatch(rows), BULK_RETRY_OPTS);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
// Guard self-test fixture (known-GOOD): direct engine call, no outer retry.
|
||||
declare const engine: { addLinksBatch: (rows: unknown[], opts?: unknown) => Promise<void> };
|
||||
declare const rows: unknown[];
|
||||
export async function good(): Promise<void> {
|
||||
await engine.addLinksBatch(rows, { auditSite: 'fixture.good' });
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* The pinned test-suite embedding shape (OpenAI legacy 1536-d).
|
||||
*
|
||||
* ONE definition, two consumers (W0 fix-wave — no hand-copied twins):
|
||||
* - test/helpers/legacy-embedding-preload.ts (bunfig preload) pins the
|
||||
* gateway to this shape for every `bun test` file.
|
||||
* - scripts/build-pglite-snapshot.ts builds the snapshot fixture under the
|
||||
* SAME shape, so the baked vector(dims) columns match what the pinned
|
||||
* tests write. (The W0 incident: the build script ran with an
|
||||
* unconfigured gateway → shipped-default 1280-d columns → every
|
||||
* 1536-d-writing test failed with "expected 1280 dimensions, not 1536"
|
||||
* the moment the snapshot became default-on.)
|
||||
*/
|
||||
export const LEGACY_EMBEDDING_CONFIG = {
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
} as const;
|
||||
@@ -25,10 +25,9 @@ import {
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
import { beforeEach } from 'bun:test';
|
||||
|
||||
const LEGACY_CONFIG = {
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
} as const;
|
||||
// W0 fix-wave: shared with scripts/build-pglite-snapshot.ts so the snapshot
|
||||
// fixture is baked under the exact shape this preload pins.
|
||||
import { LEGACY_EMBEDDING_CONFIG as LEGACY_CONFIG } from './legacy-embedding-config.ts';
|
||||
|
||||
function legacyGatewayConfig() {
|
||||
return {
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #5) — runImport must THROW a typed ImportAbortError on
|
||||
* preflight/argv failures, never process.exit(1).
|
||||
*
|
||||
* runImport is invoked in-process by the sync_brain MCP op (performFullSync),
|
||||
* the autopilot daemon, and the minion sync handler. Pre-fix, a first sync
|
||||
* against a brain with unusable embedding credentials TERMINATED the calling
|
||||
* process — the stdio MCP server just vanished mid-tool-call. The CLI dispatch
|
||||
* site maps the typed error back to exit(1), keeping CLI behavior identical.
|
||||
*/
|
||||
|
||||
import { test, expect } from 'bun:test';
|
||||
import { runImport, ImportAbortError } from '../src/commands/import.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// The abort sites fire before any WRITE; the preamble runs a couple of
|
||||
// read-only lookups (sources, config), so the stub answers those with
|
||||
// empty sets.
|
||||
const engineStub = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async () => [],
|
||||
getConfig: async () => null,
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
async function expectAbort(args: string[], reasonFragment: string): Promise<void> {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await runImport(engineStub, args);
|
||||
} catch (e) {
|
||||
thrown = e;
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(ImportAbortError);
|
||||
const err = thrown as ImportAbortError;
|
||||
expect(err.exitCode).toBe(1);
|
||||
expect(err.alreadyReported).toBe(true);
|
||||
expect(err.message).toContain(reasonFragment);
|
||||
}
|
||||
|
||||
test('missing dir arg → typed abort, not process death', async () => {
|
||||
await expectAbort(['--no-embed'], 'no import directory');
|
||||
});
|
||||
|
||||
test('invalid --workers → typed abort', async () => {
|
||||
await expectAbort(['--no-embed', '--workers', '0', '/tmp'], 'invalid --workers');
|
||||
});
|
||||
|
||||
test('unreadable import target → typed abort', async () => {
|
||||
await expectAbort(['--no-embed', '/definitely/not/a/real/dir-w0-test'], 'not readable');
|
||||
});
|
||||
|
||||
test('the calling process survives the abort (the actual Tier-1 bug)', async () => {
|
||||
// Trivially true if we got here after three aborts above, but assert it
|
||||
// explicitly: the process is alive and can keep dispatching.
|
||||
expect(process.pid).toBeGreaterThan(0);
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* `jobs get` timeout/deadline rendering (jobs fix wave, upstream issue #3).
|
||||
*
|
||||
* Pins:
|
||||
* 1. All three effective-budget states render, with the corrected wording:
|
||||
* the 1x deadline (handleTimeouts) is the NORMAL kill and the 2x
|
||||
* wall-clock sweep is the backstop — not the other way around.
|
||||
* 2. Defensive timeout_at rendering: a thin client pointed at an OLDER
|
||||
* server receives timeout_at as an ISO string (the peer predates the
|
||||
* rehydration list entry) — the formatter must print it, never crash on
|
||||
* an unguarded .toISOString().
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { formatJobDetail } from '../src/commands/jobs.ts';
|
||||
import type { MinionJob } from '../src/core/minions/types.ts';
|
||||
|
||||
function job(overrides: Partial<MinionJob> & Record<string, unknown>): MinionJob {
|
||||
return {
|
||||
id: 42,
|
||||
name: 'sync',
|
||||
queue: 'default',
|
||||
status: 'waiting',
|
||||
priority: 0,
|
||||
data: {},
|
||||
attempts_made: 0,
|
||||
attempts_started: 0,
|
||||
max_attempts: 3,
|
||||
stalled_counter: 0,
|
||||
max_stalled: 5,
|
||||
backoff_type: 'exponential',
|
||||
backoff_delay: 1000,
|
||||
backoff_jitter: 0,
|
||||
created_at: new Date('2026-08-14T00:00:00.000Z'),
|
||||
updated_at: new Date('2026-08-14T00:00:00.000Z'),
|
||||
started_at: null,
|
||||
finished_at: null,
|
||||
lock_token: null,
|
||||
lock_until: null,
|
||||
delay_until: null,
|
||||
timeout_ms: null,
|
||||
timeout_at: null,
|
||||
parent_job_id: null,
|
||||
on_child_fail: 'fail',
|
||||
error_text: null,
|
||||
stacktrace: [],
|
||||
progress: null,
|
||||
result: null,
|
||||
...overrides,
|
||||
} as unknown as MinionJob;
|
||||
}
|
||||
|
||||
describe('formatJobDetail timeout/deadline lines', () => {
|
||||
test('explicit budget: 1x-deadline wording + Deadline line from a Date', () => {
|
||||
const out = formatJobDetail(job({
|
||||
name: 'autopilot-cycle',
|
||||
timeout_ms: 1_800_000,
|
||||
timeout_at: new Date('2026-08-14T01:00:00.000Z'),
|
||||
}));
|
||||
expect(out).toContain('Timeout: 1800000ms (deadline kill at 1x when claimed; wall-clock backstop at 2x)');
|
||||
expect(out).toContain('Deadline: 2026-08-14T01:00:00.000Z');
|
||||
});
|
||||
|
||||
test('defensive render: timeout_at arriving as an ISO STRING (older server) does not crash', () => {
|
||||
const out = formatJobDetail(job({
|
||||
name: 'subagent',
|
||||
timeout_ms: 1_800_000,
|
||||
// Deliberately a string — rehydrateJobDates on an older peer never
|
||||
// converted this field. An unguarded .toISOString() would throw here.
|
||||
timeout_at: '2026-08-14T01:00:00.000Z' as unknown as Date,
|
||||
}));
|
||||
expect(out).toContain('Deadline: 2026-08-14T01:00:00.000Z');
|
||||
});
|
||||
|
||||
test('unset budget + mapped handler: names the claim-time default', () => {
|
||||
const out = formatJobDetail(job({ name: 'autopilot-cycle', timeout_ms: null }));
|
||||
expect(out).toContain('Timeout: (unset) — handler default 1800000ms stamps at claim');
|
||||
});
|
||||
|
||||
test('unset budget + unmapped handler: names the null-default sweep', () => {
|
||||
const out = formatJobDetail(job({ name: 'sync', timeout_ms: null }));
|
||||
expect(out).toContain('null-default wall-clock sweep applies');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* `jobs stats` Backpressure (24h) line + suppression hint — WIRING test.
|
||||
*
|
||||
* readRecentCoalesceCounts (the data source) is pinned in
|
||||
* test/backpressure-audit-read.test.ts; this file covers the stats-case
|
||||
* composition the coverage audit flagged as the remaining gap:
|
||||
* 1. the "Backpressure (24h)" line renders per-name counts for the queue
|
||||
* 2. the suppression hint fires when waiting=0 AND a live-lock active row
|
||||
* is older than the shared wedge threshold, naming the job id
|
||||
* 3. no audit events → no Backpressure line (omission path)
|
||||
*
|
||||
* Serial: mutates GBRAIN_AUDIT_DIR + GBRAIN_WEDGED_QUEUE_WARN_MINUTES via
|
||||
* withEnv around the runJobs call and captures console.log.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import { computeAuditFilename } from '../src/core/minions/backpressure-audit.ts';
|
||||
import { runJobs } from '../src/commands/jobs.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function captureStats(auditDir: string): Promise<string> {
|
||||
const origLog = console.log;
|
||||
let out = '';
|
||||
console.log = (...args: unknown[]) => { out += args.map(String).join(' ') + '\n'; };
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditDir, GBRAIN_WEDGED_QUEUE_WARN_MINUTES: '15' }, async () => {
|
||||
await runJobs(engine, ['stats']);
|
||||
});
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('jobs stats — Backpressure line + suppression hint wiring', () => {
|
||||
test('renders per-name counts and the suppressed-by hint for a stale live-lock active with zero waiting', async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
// One ACTIVE autopilot-cycle with a LIVE lock, 30 minutes in (> 15m
|
||||
// threshold), zero waiting rows — the exact post-maxPending suppression
|
||||
// shape that used to be invisible to the waiting>0 wedge detectors.
|
||||
const job = await queue.add('autopilot-cycle', {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'active', lock_token = 'live-worker',
|
||||
lock_until = now() + interval '5 minutes',
|
||||
started_at = now() - interval '30 minutes'
|
||||
WHERE id = $1`,
|
||||
[job.id],
|
||||
);
|
||||
const auditDir = mkdtempSync(join(tmpdir(), 'gbrain-stats-bp-'));
|
||||
try {
|
||||
mkdirSync(auditDir, { recursive: true });
|
||||
writeFileSync(join(auditDir, computeAuditFilename()),
|
||||
JSON.stringify({
|
||||
ts: new Date().toISOString(), queue: 'default', name: 'autopilot-cycle',
|
||||
pending_count: 1, max_pending: 1, decision: 'coalesced', returned_job_id: job.id,
|
||||
}) + '\n', 'utf8');
|
||||
|
||||
const out = await captureStats(auditDir);
|
||||
|
||||
expect(out).toContain('Backpressure (24h)');
|
||||
expect(out).toContain('autopilot-cycle: 1');
|
||||
expect(out).toContain(`dispatch suppressed by in-flight job #${job.id}`);
|
||||
expect(out).toContain(`gbrain jobs get ${job.id}`);
|
||||
} finally {
|
||||
rmSync(auditDir, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
test('no audit events → Backpressure line omitted entirely', async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
const emptyDir = mkdtempSync(join(tmpdir(), 'gbrain-stats-bp-empty-'));
|
||||
try {
|
||||
const out = await captureStats(emptyDir);
|
||||
expect(out).not.toContain('Backpressure (24h)');
|
||||
expect(out).not.toContain('dispatch suppressed');
|
||||
} finally {
|
||||
rmSync(emptyDir, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
/**
|
||||
* `gbrain jobs [--help]` + `gbrain jobs <subcommand> --help` guard.
|
||||
*
|
||||
* Regression test for two defects:
|
||||
* 1. `jobs` was in CLI_ONLY but not CLI_ONLY_SELF_HELP, so `jobs --help`
|
||||
* printed the generic one-line stub — the real help block in jobs.ts was
|
||||
* unreachable, and the worker entry point (`jobs work`) was undiscoverable
|
||||
* from the CLI during an incident.
|
||||
* 2. With the stub gone, a help token AFTER the subcommand name would have
|
||||
* fallen through into the subcommand body — `jobs work --help` would have
|
||||
* started a REAL worker daemon (same defect class the bootstrap
|
||||
* subcommand-help guard fixed). The guard in runJobs intercepts before
|
||||
* the thin-client refusal and the switch.
|
||||
*
|
||||
* Each case spawns the actual CLI with an empty GBRAIN_HOME and no database
|
||||
* env (the cli-help-without-brain env hygiene), so a pass proves the whole
|
||||
* dispatch path answers engine-free AND exits — a started daemon would hang
|
||||
* the spawn until the test timeout.
|
||||
*
|
||||
* Serial: spawns the CLI; keeps its wall clock out of the parallel shards.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { mkdtempSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
const REPO = new URL('..', import.meta.url).pathname;
|
||||
const STUB_MARKER = 'run gbrain --help for the full command list';
|
||||
|
||||
async function runJobsHelp(args: string[]): Promise<{ code: number; out: string }> {
|
||||
const home = mkdtempSync(join(tmpdir(), 'gbrain-jobshelp-'));
|
||||
// Empty GBRAIN_HOME is not enough: loadConfig also honours
|
||||
// GBRAIN_DATABASE_URL / DATABASE_URL, and bun auto-loads .env from cwd —
|
||||
// both would let the CLI connect an engine and mask a broken guard.
|
||||
const env: Record<string, string | undefined> = { ...process.env, GBRAIN_HOME: home };
|
||||
delete env.GBRAIN_DATABASE_URL;
|
||||
delete env.DATABASE_URL;
|
||||
const proc = Bun.spawn(['bun', '--no-env-file', 'run', 'src/cli.ts', 'jobs', ...args], {
|
||||
cwd: REPO,
|
||||
env,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const [stdout, stderr] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
]);
|
||||
const code = await proc.exited;
|
||||
return { code, out: stdout + stderr };
|
||||
}
|
||||
|
||||
describe('jobs --help and jobs <subcommand> --help print real help, never the stub, never a daemon', () => {
|
||||
test('jobs --help: full help block, not the stub', async () => {
|
||||
const { code, out } = await runJobsHelp(['--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('Minions job queue');
|
||||
expect(out).toContain('jobs work');
|
||||
expect(out).toContain('jobs supervisor');
|
||||
expect(out).toContain('jobs watch');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
expect(out).not.toContain('No brain configured');
|
||||
}, 30_000);
|
||||
|
||||
test('jobs -h: short-flag parity', async () => {
|
||||
const { code, out } = await runJobsHelp(['-h']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('Minions job queue');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs work --help: worker flags documented; NO worker daemon starts (fast exit proves it)', async () => {
|
||||
const { code, out } = await runJobsHelp(['work', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('--max-rss');
|
||||
expect(out).toContain('--health-interval');
|
||||
expect(out).toContain('GBRAIN_WORKER_CONCURRENCY');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
// A real `jobs work` on this fixture would refuse (thin-client/engine
|
||||
// path) or start a daemon; either output would differ from the help.
|
||||
expect(out).not.toContain('needs a local engine');
|
||||
}, 30_000);
|
||||
|
||||
test('jobs supervisor --help: supervisor flags + exit codes', async () => {
|
||||
const { code, out } = await runJobsHelp(['supervisor', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('--max-crashes');
|
||||
expect(out).toContain('--pid-file');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs submit -h: submit flags documented', async () => {
|
||||
const { code, out } = await runJobsHelp(['submit', '-h']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('--idempotency-key');
|
||||
expect(out).toContain('--max-waiting');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs watch --help: documents the equals-only --refresh-ms form', async () => {
|
||||
const { code, out } = await runJobsHelp(['watch', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('--refresh-ms=N');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs prune --help: prune flags documented, nothing pruned', async () => {
|
||||
const { code, out } = await runJobsHelp(['prune', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('--older-than');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs stats --help: no dedicated entry → falls back to the full help block', async () => {
|
||||
const { code, out } = await runJobsHelp(['stats', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('Minions job queue');
|
||||
expect(out).toContain('--cluster-errors');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
|
||||
test('jobs constructor --help: prototype key falls back to the full help, never Object.prototype garbage', async () => {
|
||||
// Red-team finding: a plain-object lookup resolves inherited keys, so
|
||||
// without Object.hasOwn this printed `function Object() { ... }`.
|
||||
const { code, out } = await runJobsHelp(['constructor', '--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(out).toContain('Minions job queue');
|
||||
expect(out).not.toContain('native code');
|
||||
expect(out).not.toContain('function Object');
|
||||
expect(out).not.toContain(STUB_MARKER);
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -27,6 +27,7 @@ describe('rehydrateJobDates', () => {
|
||||
started_at: '2026-07-21T04:02:12.001Z',
|
||||
finished_at: '2026-07-21T04:02:14.900Z',
|
||||
lock_until: '2026-07-21T04:03:12.001Z',
|
||||
timeout_at: '2026-07-21T04:32:12.001Z',
|
||||
delay_until: null,
|
||||
};
|
||||
const job = rehydrateJobDates(wire);
|
||||
@@ -35,6 +36,7 @@ describe('rehydrateJobDates', () => {
|
||||
expect(job.started_at).toBeInstanceOf(Date);
|
||||
expect(job.finished_at).toBeInstanceOf(Date);
|
||||
expect(job.lock_until).toBeInstanceOf(Date);
|
||||
expect(job.timeout_at).toBeInstanceOf(Date);
|
||||
expect((job.started_at as unknown as Date).toISOString()).toBe('2026-07-21T04:02:12.001Z');
|
||||
// Date math used by formatJob's duration column works post-rehydration.
|
||||
expect((job.finished_at as unknown as Date).getTime() - (job.started_at as unknown as Date).getTime())
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #14) — `gbrain lint --fix` scans once and reports the
|
||||
* TRUE fixed count.
|
||||
*
|
||||
* Pre-fix, runLint ran its own read+lint+fix loop for human detail, then
|
||||
* called runLintCore a second time for the summary — every page linted
|
||||
* twice, and because the first pass had already written the fixes, the
|
||||
* summary's total_fixed counted fixes against already-fixed content: the
|
||||
* CLI printed "0 auto-fixed" after fixing N issues.
|
||||
*/
|
||||
|
||||
import { test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { runLintCore } from '../src/commands/lint.ts';
|
||||
|
||||
let dir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), 'gbrain-lint-w0-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('single pass: total_fixed reports the fixes THIS run applied', async () => {
|
||||
// A whole-file ```markdown wrapper is a fixable rule (code-fence-wrap).
|
||||
writeFileSync(join(dir, 'wrapped.md'), '```markdown\n# Hello\n\nBody text.\n```\n');
|
||||
writeFileSync(join(dir, 'clean.md'), '# Clean\n\nNothing to fix here.\n');
|
||||
|
||||
const scanned: string[] = [];
|
||||
const reported: Array<{ relPath: string; fixedCount: number }> = [];
|
||||
const result = await runLintCore({
|
||||
target: dir,
|
||||
fix: true,
|
||||
contentSanity: { disabled: true } as never,
|
||||
onPageScanned: () => scanned.push('tick'),
|
||||
onPageIssues: (relPath, _issues, fixedCount) => reported.push({ relPath, fixedCount }),
|
||||
});
|
||||
|
||||
// The bug: this was 0 after fixing. It must equal the fixes applied NOW.
|
||||
expect(result.total_fixed).toBeGreaterThan(0);
|
||||
// Each page scanned exactly once (2 pages, 2 ticks — not 4).
|
||||
expect(scanned.length).toBe(2);
|
||||
// The per-page hook carried the real fixed count for the fixed page
|
||||
// (clean.md may still surface non-fixable advisory issues — irrelevant here).
|
||||
const wrapped = reported.find(r => r.relPath === 'wrapped.md');
|
||||
expect(wrapped).toBeDefined();
|
||||
expect(wrapped!.fixedCount).toBeGreaterThan(0);
|
||||
expect(result.total_fixed).toBe(wrapped!.fixedCount);
|
||||
// And the fix actually landed on disk.
|
||||
expect(readFileSync(join(dir, 'wrapped.md'), 'utf-8').startsWith('```')).toBe(false);
|
||||
|
||||
// Second run: nothing left to fix — counts stay honest.
|
||||
const again = await runLintCore({ target: dir, fix: true, contentSanity: { disabled: true } as never });
|
||||
expect(again.total_fixed).toBe(0);
|
||||
});
|
||||
@@ -50,8 +50,13 @@ describe('hasPendingMigrations', () => {
|
||||
}, 30000);
|
||||
|
||||
test('returns true when version config is missing entirely (defensive default)', async () => {
|
||||
// W0: opt out of the default-on snapshot — this test's premise is an
|
||||
// empty PGlite with no config table at all.
|
||||
const priorSnapshot = process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
const engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
if (priorSnapshot !== undefined) process.env.GBRAIN_PGLITE_SNAPSHOT = priorSnapshot;
|
||||
try {
|
||||
// Don't call initSchema. Probe against an empty PGlite — getConfig should
|
||||
// either return null (treated as version=1) or throw on missing config
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
/**
|
||||
* Jobs fix wave (upstream issues #2/#3) — migration v128
|
||||
* (minion_jobs_timeout_backfill_and_duplicate_cycle_cleanup).
|
||||
*
|
||||
* Pinned contracts:
|
||||
* 1. v128 exists in MIGRATIONS with the canonical name, idempotent flag, and
|
||||
* one engine-agnostic sql block (no sqlFor split).
|
||||
* 2. Statement 1 backfills NULL timeout_ms for the 8 long-lane handler names
|
||||
* across ALL five non-terminal statuses — and ONLY those rows: terminal
|
||||
* rows, unmapped names, and explicit budgets are untouched. Active rows
|
||||
* get timeout_ms only (never a timeout_at stamp — that would arm the
|
||||
* tighter 1x handleTimeouts kill mid-flight; the 2x wall-clock bound is
|
||||
* the intended repair).
|
||||
* 3. Statement 2 cancels all-but-newest ticker-keyed waiting autopilot
|
||||
* cycles per (name, queue, source scope). Manual submissions (no ticker
|
||||
* idempotency-key prefix) and parented rows are never touched; distinct
|
||||
* sources each keep their newest row; cancelled rows carry error_text.
|
||||
* 4. Ledger round-trip: rewind to 127 → runMigrations applies v128; re-run
|
||||
* is 0 applied. AND (Codex C9) SQL-level idempotency: re-executing the
|
||||
* v128 SQL directly changes zero rows — the ledger check alone only
|
||||
* proves version bookkeeping.
|
||||
* 5. Empty table → 0-row no-op.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import { MIGRATIONS, LATEST_VERSION, runMigrations } from '../src/core/migrate.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
const V128_SQL = MIGRATIONS.find(m => m.version === 128)?.sql ?? '';
|
||||
|
||||
/** PGLite's prepared-statement path rejects multi-command SQL; the migration
|
||||
* runner uses a different execution path. For the direct SQL-rerun check,
|
||||
* split on statement terminators (the blob's only two `;` are terminators). */
|
||||
const V128_STATEMENTS = V128_SQL.split(';').map(s => s.trim()).filter(Boolean);
|
||||
|
||||
async function execV128Directly(): Promise<void> {
|
||||
for (const stmt of V128_STATEMENTS) {
|
||||
await engine.executeRaw(stmt);
|
||||
}
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' }); // in-memory
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
/** Force a row into a specific historical shape (add() stamps budgets and
|
||||
* every row starts 'waiting', so legacy states are seeded by direct UPDATE —
|
||||
* the same trick the wall-clock tests use). */
|
||||
async function forceRow(id: number, sets: string, params: unknown[] = []): Promise<void> {
|
||||
await engine.executeRaw(`UPDATE minion_jobs SET ${sets} WHERE id = $${params.length + 1}`, [...params, id]);
|
||||
}
|
||||
|
||||
async function snapshot(): Promise<Array<Record<string, unknown>>> {
|
||||
return engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT id, name, status, timeout_ms::text AS timeout_ms,
|
||||
timeout_at IS NULL AS timeout_at_null, error_text
|
||||
FROM minion_jobs ORDER BY id`,
|
||||
);
|
||||
}
|
||||
|
||||
describe('migration v128 — structure', () => {
|
||||
test('exists with canonical name, idempotent flag, engine-agnostic sql', () => {
|
||||
const v128 = MIGRATIONS.find(m => m.version === 128);
|
||||
expect(v128).toBeDefined();
|
||||
expect(v128?.name).toBe('minion_jobs_timeout_backfill_and_duplicate_cycle_cleanup');
|
||||
expect(v128?.idempotent).toBe(true);
|
||||
expect(v128?.sqlFor).toBeUndefined();
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(128);
|
||||
});
|
||||
|
||||
test('statement 1 covers all five non-terminal statuses and never stamps timeout_at', () => {
|
||||
expect(V128_SQL).toContain(`status IN ('waiting','active','delayed','waiting-children','paused')`);
|
||||
expect(V128_SQL).not.toContain('SET timeout_at');
|
||||
});
|
||||
|
||||
test('statement 2 is restricted to ticker idempotency-key prefixes and top-level rows', () => {
|
||||
expect(V128_SQL).toContain(`idempotency_key LIKE 'autopilot-cycle:%'`);
|
||||
expect(V128_SQL).toContain(`idempotency_key LIKE 'autopilot-global:%'`);
|
||||
expect(V128_SQL).toContain('parent_job_id IS NULL');
|
||||
});
|
||||
});
|
||||
|
||||
describe('migration v128 — backfill + cleanup semantics (PGLite)', () => {
|
||||
test('full matrix: backfill hits legacy non-terminal mapped rows only; cleanup keeps newest ticker row per scope', async () => {
|
||||
// --- Backfill fixtures (statement 1) ---
|
||||
// Legacy waiting subagent: NULL budget → 30min.
|
||||
const legacyWaiting = await queue.add('subagent', {}, undefined, { allowProtectedSubmit: true });
|
||||
await forceRow(legacyWaiting.id, `timeout_ms = NULL, timeout_at = NULL`);
|
||||
// Legacy ACTIVE autopilot-cycle mid-flight: NULL budget → 30min, but
|
||||
// timeout_at must STAY NULL (Codex C7 — no 1x kill armed mid-flight).
|
||||
const legacyActive = await queue.add('autopilot-cycle', { source_id: 'src-active' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-active:slot-0',
|
||||
});
|
||||
await forceRow(legacyActive.id,
|
||||
`timeout_ms = NULL, timeout_at = NULL, status = 'active',
|
||||
lock_token = 'legacy-worker', lock_until = now() + interval '30 seconds',
|
||||
started_at = now() - interval '10 minutes'`);
|
||||
// Terminal row: NULL budget stays NULL (completed is not rescued).
|
||||
const completed = await queue.add('subagent', { done: true }, undefined, { allowProtectedSubmit: true });
|
||||
await forceRow(completed.id, `timeout_ms = NULL, status = 'completed', finished_at = now()`);
|
||||
// Unmapped name: stays NULL.
|
||||
const unmapped = await queue.add('sync', {});
|
||||
expect(unmapped.timeout_ms).toBeNull();
|
||||
// Explicit budget: untouched.
|
||||
const explicit = await queue.add('embed-backfill', { sourceId: 'x' }, { timeout_ms: 5000 });
|
||||
|
||||
// --- Cleanup fixtures (statement 2) ---
|
||||
// Three ticker-keyed waiting cycles for ONE source, staggered ages —
|
||||
// newest must survive, older two cancelled.
|
||||
const dupOld = await queue.add('autopilot-cycle', { source_id: 'src-dup' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-dup:slot-1',
|
||||
});
|
||||
await forceRow(dupOld.id, `timeout_ms = NULL, created_at = now() - interval '3 hours'`);
|
||||
const dupMid = await queue.add('autopilot-cycle', { source_id: 'src-dup' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-dup:slot-2',
|
||||
});
|
||||
await forceRow(dupMid.id, `timeout_ms = NULL, created_at = now() - interval '2 hours'`);
|
||||
const dupNew = await queue.add('autopilot-cycle', { source_id: 'src-dup' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-dup:slot-3',
|
||||
});
|
||||
await forceRow(dupNew.id, `timeout_ms = NULL, created_at = now() - interval '1 hour'`);
|
||||
// A DIFFERENT source keeps its own newest row (scope isolation).
|
||||
const otherSrc = await queue.add('autopilot-cycle', { source_id: 'src-other' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-other:slot-1',
|
||||
});
|
||||
await forceRow(otherSrc.id, `created_at = now() - interval '4 hours'`);
|
||||
// Global maintenance duplicates (NULL source scope) dedupe independently.
|
||||
const globalOld = await queue.add('autopilot-global-maintenance', {}, {
|
||||
idempotency_key: 'autopilot-global:slot-1',
|
||||
});
|
||||
await forceRow(globalOld.id, `created_at = now() - interval '2 hours'`);
|
||||
const globalNew = await queue.add('autopilot-global-maintenance', {}, {
|
||||
idempotency_key: 'autopilot-global:slot-2',
|
||||
});
|
||||
// Manual submission: same name + source as the dup scope, custom phases,
|
||||
// NO ticker key — must never be touched (Codex C5).
|
||||
const manual = await queue.add('autopilot-cycle', { source_id: 'src-dup', phases: ['sync'] });
|
||||
await forceRow(manual.id, `created_at = now() - interval '5 hours'`);
|
||||
// Ticker-mimicking key but camelCase sourceId payload: the ticker never
|
||||
// writes sourceId, so this row is not ticker-provenance — the cleanup's
|
||||
// sourceId-IS-NULL guard must leave it alone even though its key matches
|
||||
// the prefix (adversarial-review tightening: without the guard, distinct
|
||||
// sourceId scopes would collapse into the empty source_id group).
|
||||
const camelMimic = await queue.add('autopilot-cycle', { sourceId: 'camel-src' }, {
|
||||
idempotency_key: 'autopilot-cycle:camel-mimic:slot-1',
|
||||
});
|
||||
await forceRow(camelMimic.id, `created_at = now() - interval '7 hours'`);
|
||||
// Parented row with a ticker-looking key: guarded by parent_job_id IS NULL.
|
||||
// Parent is seeded by direct UPDATE — add()'s parent_job_id opt would flip
|
||||
// the parent row to 'waiting-children' and pollute the dup-scope fixtures.
|
||||
const parented = await queue.add('autopilot-cycle', { source_id: 'src-dup' }, {
|
||||
idempotency_key: 'autopilot-cycle:src-dup:slot-child',
|
||||
});
|
||||
await forceRow(parented.id,
|
||||
`created_at = now() - interval '6 hours', parent_job_id = $1`, [dupNew.id]);
|
||||
|
||||
// --- Apply v128 via the real migration runner (ledger rewind). ---
|
||||
await engine.setConfig('version', '127');
|
||||
const res = await runMigrations(engine);
|
||||
expect(res.applied).toBeGreaterThanOrEqual(1);
|
||||
expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION));
|
||||
|
||||
const rows = await engine.executeRaw<{
|
||||
id: number; status: string; timeout_ms: string | null;
|
||||
timeout_at: string | null; error_text: string | null;
|
||||
}>(`SELECT id, status, timeout_ms::text AS timeout_ms, timeout_at::text AS timeout_at, error_text
|
||||
FROM minion_jobs`);
|
||||
const byId = new Map(rows.map(r => [Number(r.id), r]));
|
||||
|
||||
// Statement 1 assertions.
|
||||
expect(Number(byId.get(legacyWaiting.id)!.timeout_ms)).toBe(1_800_000);
|
||||
expect(Number(byId.get(legacyActive.id)!.timeout_ms)).toBe(1_800_000);
|
||||
expect(byId.get(legacyActive.id)!.timeout_at).toBeNull(); // C7: no 1x kill armed
|
||||
expect(byId.get(legacyActive.id)!.status).toBe('active');
|
||||
expect(byId.get(completed.id)!.timeout_ms).toBeNull(); // terminal untouched
|
||||
expect(byId.get(unmapped.id)!.timeout_ms).toBeNull(); // not in map
|
||||
expect(Number(byId.get(explicit.id)!.timeout_ms)).toBe(5000); // explicit wins
|
||||
|
||||
// Statement 2 assertions.
|
||||
expect(byId.get(dupNew.id)!.status).toBe('waiting'); // newest survives
|
||||
expect(byId.get(dupOld.id)!.status).toBe('cancelled');
|
||||
expect(byId.get(dupMid.id)!.status).toBe('cancelled');
|
||||
expect(byId.get(dupOld.id)!.error_text).toContain('superseded duplicate autopilot cycle');
|
||||
expect(byId.get(otherSrc.id)!.status).toBe('waiting'); // other source keeps its one row
|
||||
expect(byId.get(globalNew.id)!.status).toBe('waiting');
|
||||
expect(byId.get(globalOld.id)!.status).toBe('cancelled');
|
||||
expect(byId.get(manual.id)!.status).toBe('waiting'); // manual never touched (C5)
|
||||
expect(byId.get(manual.id)!.error_text).toBeNull();
|
||||
expect(byId.get(parented.id)!.status).toBe('waiting'); // parented never touched
|
||||
expect(byId.get(camelMimic.id)!.status).toBe('waiting'); // camelCase sourceId never swept
|
||||
|
||||
// Ledger idempotency: re-run applies nothing.
|
||||
const rerun = await runMigrations(engine);
|
||||
expect(rerun.applied).toBe(0);
|
||||
|
||||
// SQL-level idempotency (Codex C9): re-execute the v128 SQL directly and
|
||||
// prove zero rows change — version bookkeeping alone can't show this.
|
||||
const before = await snapshot();
|
||||
await execV128Directly();
|
||||
const after = await snapshot();
|
||||
expect(after).toEqual(before);
|
||||
|
||||
// Sanity on the split used above: exactly the two statements shipped.
|
||||
expect(V128_STATEMENTS.length).toBe(2);
|
||||
}, 30_000);
|
||||
|
||||
test('empty table: v128 SQL is a 0-row no-op', async () => {
|
||||
const before = await snapshot();
|
||||
expect(before).toEqual([]);
|
||||
await execV128Directly();
|
||||
expect(await snapshot()).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -381,6 +381,55 @@ describe('MinionQueue: #1737 per-handler default timeout', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// --- Claim-time budget fallback (jobs fix wave, upstream issue #3) ---
|
||||
//
|
||||
// Rows inserted before submit-time stamping existed (or by writers that
|
||||
// bypass add()) carry timeout_ms = NULL and used to fall to the minutes-scale
|
||||
// null-default wall-clock sweep. claim() now COALESCEs the budget from
|
||||
// HANDLER_DEFAULT_TIMEOUT_MS and derives timeout_at from the coalesced value.
|
||||
// Seeding NULL requires a direct UPDATE because add() stamps at submit.
|
||||
|
||||
describe('MinionQueue: claim-time timeout fallback', () => {
|
||||
test('legacy NULL-timeout long-handler row gets the map budget stamped at claim', async () => {
|
||||
const job = await queue.add('subagent', {}, undefined, { allowProtectedSubmit: true });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET timeout_ms = NULL, timeout_at = NULL WHERE id = $1`,
|
||||
[job.id],
|
||||
);
|
||||
const before = Date.now();
|
||||
const claimed = await queue.claim('tok-fallback', 30_000, 'default', ['subagent']);
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.id).toBe(job.id);
|
||||
expect(claimed!.timeout_ms).toBe(30 * 60 * 1000);
|
||||
expect(claimed!.timeout_at).toBeInstanceOf(Date);
|
||||
const deadline = claimed!.timeout_at!.getTime();
|
||||
// timeout_at ≈ claim time + 30min (generous 60s slop for slow CI).
|
||||
expect(deadline).toBeGreaterThan(before + 30 * 60 * 1000 - 60_000);
|
||||
expect(deadline).toBeLessThan(before + 30 * 60 * 1000 + 60_000);
|
||||
// Persisted, not just returned — a restarted worker sees the same budget.
|
||||
const rows = await engine.executeRaw<{ timeout_ms: number | null }>(
|
||||
`SELECT timeout_ms FROM minion_jobs WHERE id = $1`, [job.id],
|
||||
);
|
||||
expect(Number(rows[0].timeout_ms)).toBe(30 * 60 * 1000);
|
||||
});
|
||||
|
||||
test('name outside the map keeps NULL budget at claim (fail-open, todays behavior)', async () => {
|
||||
const job = await queue.add('noop', {});
|
||||
expect(job.timeout_ms).toBeNull();
|
||||
const claimed = await queue.claim('tok-nomap', 30_000, 'default', ['noop']);
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.timeout_ms).toBeNull();
|
||||
expect(claimed!.timeout_at).toBeNull();
|
||||
});
|
||||
|
||||
test('explicit timeout_ms is never overridden at claim', async () => {
|
||||
await queue.add('embed-backfill', { sourceId: 'x' }, { timeout_ms: 5000 });
|
||||
const claimed = await queue.claim('tok-explicit', 30_000, 'default', ['embed-backfill']);
|
||||
expect(claimed).not.toBeNull();
|
||||
expect(claimed!.timeout_ms).toBe(5000);
|
||||
});
|
||||
});
|
||||
|
||||
// --- v0.13.1 #219 — max_stalled default + input surface ---
|
||||
|
||||
describe('MinionQueue: v0.13.1 max_stalled schema default (#219)', () => {
|
||||
@@ -2090,6 +2139,167 @@ describe('MinionQueue: v0.19.1 maxWaiting — cap correctness + race (D2/H2)', (
|
||||
const c = await queue.add('uncapped', {});
|
||||
expect(new Set([a.id, b.id, c.id]).size).toBe(3);
|
||||
});
|
||||
|
||||
// REGRESSION (jobs fix wave): the backpressure scope now reads BOTH payload
|
||||
// spellings. A snake_case source_id submission previously fell into the
|
||||
// NULL-wildcard arm (counted ALL rows for name+queue); it now scopes
|
||||
// exactly like camelCase sourceId. Pin the new arm for maxWaiting too —
|
||||
// the maxPending tests below cover it for the new option only.
|
||||
test('maxWaiting + snake_case source_id: scoped per source, not wildcard', async () => {
|
||||
const a1 = await queue.add('srcsync2', { source_id: 'src-a' }, { maxWaiting: 1 });
|
||||
// Different source: must NOT be swallowed by src-a's waiting row.
|
||||
const b1 = await queue.add('srcsync2', { source_id: 'src-b' }, { maxWaiting: 1 });
|
||||
expect(b1.id).not.toBe(a1.id);
|
||||
// Same source coalesces.
|
||||
const a2 = await queue.add('srcsync2', { source_id: 'src-a' }, { maxWaiting: 1 });
|
||||
expect(a2.id).toBe(a1.id);
|
||||
expect(a2.coalesced).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// --- maxPending — single-flight counting waiting + LIVE-LOCK active rows ---
|
||||
//
|
||||
// Jobs fix wave (upstream issue #2): the autopilot dispatch guards failed once
|
||||
// a job sat in 'active' — maxWaiting counts only waiting rows and the slot
|
||||
// idempotency key rotates every baseInterval, so a stalled cycle accumulated
|
||||
// unbounded byte-identical duplicates. maxPending counts waiting rows PLUS
|
||||
// live-lock actives (lock_until > now()); an expired-lock active belongs to a
|
||||
// dead/blocked worker and must NOT suppress dispatch — the fresh waiting row
|
||||
// keeps feeding the waitingClaimable>0 wedge detectors. Scope is EXACT on
|
||||
// COALESCE(data.sourceId, data.source_id): NULL matches only NULL.
|
||||
//
|
||||
// NOTE: the Promise.all race here runs on single-writer PGLite — a smoke
|
||||
// check. The advisory-lock guarantee under real concurrency is pinned by the
|
||||
// DATABASE_URL-gated e2e (concurrent same-scope submissions on Postgres).
|
||||
|
||||
describe('MinionQueue: maxPending — single-flight (waiting + live-lock active)', () => {
|
||||
async function forceActive(id: number, lockUntilSql: string): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'active', lock_token = 'tok-mp',
|
||||
lock_until = ${lockUntilSql}, started_at = now() - interval '5 minutes'
|
||||
WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
}
|
||||
|
||||
test('cap 1: second submission coalesces onto the waiting row (coalesced metadata set)', async () => {
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(a.coalesced).toBeUndefined(); // fresh insert carries no metadata
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(b.id).toBe(a.id);
|
||||
expect(b.coalesced).toBe(true);
|
||||
});
|
||||
|
||||
test('LIVE-LOCK active row suppresses dispatch (the issue-#2 fix)', async () => {
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
await forceActive(a.id, `now() + interval '5 minutes'`);
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(b.id).toBe(a.id); // coalesced onto the in-flight ACTIVE row
|
||||
expect(b.coalesced).toBe(true);
|
||||
expect(b.status).toBe('active');
|
||||
});
|
||||
|
||||
test('EXPIRED-lock active row does NOT suppress — fresh insert (wedge detectors stay fed)', async () => {
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
await forceActive(a.id, `now() - interval '1 second'`);
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(b.id).not.toBe(a.id);
|
||||
expect(b.status).toBe('waiting');
|
||||
expect(b.coalesced).toBeUndefined();
|
||||
});
|
||||
|
||||
test('waiting row preferred over live active on cap-hit', async () => {
|
||||
const active = await queue.add('single-flight', {}, { maxPending: 2 });
|
||||
await forceActive(active.id, `now() + interval '5 minutes'`);
|
||||
const waiting = await queue.add('single-flight', {}, { maxPending: 2 });
|
||||
expect(waiting.status).toBe('waiting');
|
||||
const c = await queue.add('single-flight', {}, { maxPending: 2 });
|
||||
expect(c.id).toBe(waiting.id); // most-recent WAITING wins the coalesce
|
||||
expect(c.coalesced).toBe(true);
|
||||
});
|
||||
|
||||
test('dead row frees the cap', async () => {
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'dead', finished_at = now() WHERE id = $1`, [a.id],
|
||||
);
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(b.id).not.toBe(a.id);
|
||||
expect(b.status).toBe('waiting');
|
||||
});
|
||||
|
||||
test('EXACT source scope: NULL-source submission never coalesces onto a per-source row (and vice versa)', async () => {
|
||||
const perSource = await queue.add('single-flight', { source_id: 'src-a' }, { maxPending: 1 });
|
||||
// NULL-source submission: per-source row must not count for it.
|
||||
const legacy = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(legacy.id).not.toBe(perSource.id);
|
||||
// And a second NULL-source submission coalesces onto the legacy row only.
|
||||
const legacy2 = await queue.add('single-flight', {}, { maxPending: 1 });
|
||||
expect(legacy2.id).toBe(legacy.id);
|
||||
// A second per-source submission coalesces onto the per-source row only.
|
||||
const perSource2 = await queue.add('single-flight', { source_id: 'src-a' }, { maxPending: 1 });
|
||||
expect(perSource2.id).toBe(perSource.id);
|
||||
});
|
||||
|
||||
test('snake_case source_id scoping: two sources keep independent caps; same source coalesces', async () => {
|
||||
const a1 = await queue.add('single-flight', { source_id: 'src-a' }, { maxPending: 1 });
|
||||
const b1 = await queue.add('single-flight', { source_id: 'src-b' }, { maxPending: 1 });
|
||||
expect(b1.id).not.toBe(a1.id);
|
||||
const a2 = await queue.add('single-flight', { source_id: 'src-a' }, { maxPending: 1 });
|
||||
expect(a2.id).toBe(a1.id);
|
||||
});
|
||||
|
||||
test('camelCase sourceId scoping parity', async () => {
|
||||
const a1 = await queue.add('single-flight', { sourceId: 'src-a' }, { maxPending: 1 });
|
||||
const b1 = await queue.add('single-flight', { sourceId: 'src-b' }, { maxPending: 1 });
|
||||
expect(b1.id).not.toBe(a1.id);
|
||||
const a2 = await queue.add('single-flight', { sourceId: 'src-a' }, { maxPending: 1 });
|
||||
expect(a2.id).toBe(a1.id);
|
||||
});
|
||||
|
||||
test('clamp: maxPending 0 → 1; floor: 1.7 → 1', async () => {
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 0 });
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 0 });
|
||||
expect(b.id).toBe(a.id); // 0 clamps to 1 → coalesce
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
const c = await queue.add('single-flight', {}, { maxPending: 1.7 });
|
||||
const d = await queue.add('single-flight', {}, { maxPending: 1.7 });
|
||||
expect(d.id).toBe(c.id); // floor(1.7)=1 → coalesce
|
||||
});
|
||||
|
||||
test('race: concurrent submissions hold the cap (PGLite smoke; PG e2e pins the real guarantee)', async () => {
|
||||
const results = await Promise.all(
|
||||
Array.from({ length: 4 }, () => queue.add('single-flight', {}, { maxPending: 1 })),
|
||||
);
|
||||
const ids = new Set(results.map(r => r.id));
|
||||
expect(ids.size).toBe(1);
|
||||
});
|
||||
|
||||
test('both guards supplied: maxPending checked first, both enforced', async () => {
|
||||
// One waiting row. maxPending: 2 passes (1 pending < 2) but
|
||||
// maxWaiting: 1 must still coalesce — both guards apply.
|
||||
const a = await queue.add('single-flight', {}, { maxPending: 2, maxWaiting: 1 });
|
||||
const b = await queue.add('single-flight', {}, { maxPending: 2, maxWaiting: 1 });
|
||||
expect(b.id).toBe(a.id);
|
||||
expect(b.coalesced).toBe(true);
|
||||
// Now force it ACTIVE (live lock): maxWaiting alone would let a new row
|
||||
// in (waiting=0), but maxPending: 1 fires FIRST and coalesces onto the
|
||||
// in-flight row.
|
||||
await forceActive(a.id, `now() + interval '5 minutes'`);
|
||||
const c = await queue.add('single-flight', {}, { maxPending: 1, maxWaiting: 1 });
|
||||
expect(c.id).toBe(a.id);
|
||||
expect(c.coalesced).toBe(true);
|
||||
});
|
||||
|
||||
test('ON CONFLICT idempotency race fallback also carries coalesced metadata', async () => {
|
||||
// Same idempotency_key twice: the fast-path SELECT returns the existing
|
||||
// row with coalesced: true (first coalesce path).
|
||||
const a = await queue.add('single-flight', {}, { idempotency_key: 'sf-key-1' });
|
||||
expect(a.coalesced).toBeUndefined();
|
||||
const b = await queue.add('single-flight', {}, { idempotency_key: 'sf-key-1' });
|
||||
expect(b.id).toBe(a.id);
|
||||
expect(b.coalesced).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveWorkerConcurrency (v0.19.1 H3): clamp + validation', () => {
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* Snapshot-timezone parity pin.
|
||||
*
|
||||
* dumpDataDir bakes the BUILD process's TimeZone into the snapshot tar's
|
||||
* cluster defaults. Un-pinned, a snapshot-restored engine ran sessions in the
|
||||
* build machine's zone while cold-init engines follow the runtime process
|
||||
* (bun test pins TZ=UTC) — so every naive-timestamp day-boundary comparison
|
||||
* shifted by the offset, and date-dependent tests failed only in the evening,
|
||||
* only under GBRAIN_PGLITE_SNAPSHOT. Two fixes hold the line: the build
|
||||
* script pins TZ=UTC before dumping, and the engine re-pins the session to
|
||||
* the runtime zone on snapshot restore.
|
||||
*
|
||||
* This test is the deterministic pin: at ANY wall-clock hour, a cold engine
|
||||
* and a snapshot engine created by the same process must report the same
|
||||
* session TimeZone. Serial file: it mutates GBRAIN_PGLITE_SNAPSHOT (R1).
|
||||
*/
|
||||
import { describe, test, expect, afterAll } from 'bun:test';
|
||||
import { existsSync } from 'fs';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
const SNAPSHOT = 'test/fixtures/pglite-snapshot.tar';
|
||||
|
||||
async function sessionUtcOffsetSeconds(env: Record<string, string | undefined>): Promise<number> {
|
||||
const prev = process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
if (env.GBRAIN_PGLITE_SNAPSHOT === undefined) delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
else process.env.GBRAIN_PGLITE_SNAPSHOT = env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
const engine = new PGLiteEngine();
|
||||
try {
|
||||
await engine.connect({} as never);
|
||||
await engine.initSchema();
|
||||
// Compare the effective UTC OFFSET, not the zone label: cold init spells
|
||||
// the runtime zone one way (Etc/GMT0), the restore re-pin another (UTC) —
|
||||
// the invariant is identical instant arithmetic, not identical strings.
|
||||
const rows = await engine.executeRaw<{ off: string }>(
|
||||
`SELECT extract(timezone FROM now())::text AS off`,
|
||||
);
|
||||
return Number(rows[0].off);
|
||||
} finally {
|
||||
await engine.disconnect?.();
|
||||
if (prev === undefined) delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
||||
else process.env.GBRAIN_PGLITE_SNAPSHOT = prev;
|
||||
}
|
||||
}
|
||||
|
||||
describe('PGLite snapshot timezone parity', () => {
|
||||
// Build the fixture in-test (idempotent hash short-circuit makes re-runs
|
||||
// cheap) so this pin cannot silently skip in lanes without a prebuilt tar.
|
||||
test('snapshot fixture builds', async () => {
|
||||
const proc = Bun.spawnSync(['bun', 'run', 'build:pglite-snapshot'], {
|
||||
cwd: `${import.meta.dir}/..`,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
timeout: 300_000,
|
||||
killSignal: 'SIGKILL',
|
||||
});
|
||||
expect(proc.exitCode).toBe(0);
|
||||
expect(existsSync(`${import.meta.dir}/../${SNAPSHOT}`)).toBe(true);
|
||||
}, 320_000);
|
||||
|
||||
test('snapshot-restored session UTC offset equals cold-init session UTC offset', async () => {
|
||||
const cold = await sessionUtcOffsetSeconds({ GBRAIN_PGLITE_SNAPSHOT: undefined });
|
||||
const snap = await sessionUtcOffsetSeconds({ GBRAIN_PGLITE_SNAPSHOT: SNAPSHOT });
|
||||
expect(snap).toBe(cold);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #4) — a child dead-lettered via max-stall must notify
|
||||
* and unblock its aggregator parent, exactly like timeout/failure kills do.
|
||||
*
|
||||
* Pre-fix, handleStalled's dead-letter branch emitted no child_done and never
|
||||
* flipped the parent out of 'waiting-children': the parent hung forever
|
||||
* unless it happened to carry its own timeout_ms (self-heal-proof — verified
|
||||
* in adversarial review: resolveParent has no periodic caller and the worker
|
||||
* only logged counts).
|
||||
*/
|
||||
|
||||
import { test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
import type { ChildDoneMessage } from '../src/core/minions/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_inbox');
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
async function jobStatus(id: number): Promise<string> {
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status FROM minion_jobs WHERE id = $1`, [id],
|
||||
);
|
||||
return rows[0]!.status;
|
||||
}
|
||||
|
||||
test('stall-exhausted child → child_done(dead) in parent inbox + parent unblocked', async () => {
|
||||
const parent = await queue.add('aggregator', {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [parent.id],
|
||||
);
|
||||
const child = await queue.add('child', {}, { parent_job_id: parent.id, max_stalled: 1 });
|
||||
|
||||
// Claim the child, then expire its lock so the stall sweep sees it; with
|
||||
// max_stalled=1 the first stall dead-letters it.
|
||||
const claimed = await queue.claim('tok-stall', 30_000, 'default', ['child', 'aggregator']);
|
||||
expect(claimed?.id).toBe(child.id);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
|
||||
);
|
||||
|
||||
const { requeued, dead } = await queue.handleStalled();
|
||||
expect(requeued).toHaveLength(0);
|
||||
expect(dead.map(j => j.id)).toEqual([child.id]);
|
||||
|
||||
// The fix, part 1: parent got child_done with outcome 'dead'.
|
||||
const inbox = await engine.executeRaw<{ payload: unknown }>(
|
||||
`SELECT payload FROM minion_inbox WHERE job_id = $1`, [parent.id],
|
||||
);
|
||||
expect(inbox).toHaveLength(1);
|
||||
const msg = (typeof inbox[0]!.payload === 'string'
|
||||
? JSON.parse(inbox[0]!.payload as string)
|
||||
: inbox[0]!.payload) as ChildDoneMessage;
|
||||
expect(msg.type).toBe('child_done');
|
||||
expect(msg.child_id).toBe(child.id);
|
||||
expect(msg.outcome).toBe('dead');
|
||||
expect(msg.error).toBe('max stalled count exceeded');
|
||||
|
||||
// The fix, part 2: parent flipped out of waiting-children.
|
||||
expect(await jobStatus(parent.id)).toBe('waiting');
|
||||
});
|
||||
|
||||
test('stall-requeued child (budget left) does NOT touch the parent', async () => {
|
||||
const parent = await queue.add('aggregator', {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [parent.id],
|
||||
);
|
||||
const child = await queue.add('child', {}, { parent_job_id: parent.id, max_stalled: 5 });
|
||||
const claimed = await queue.claim('tok-stall2', 30_000, 'default', ['child', 'aggregator']);
|
||||
expect(claimed?.id).toBe(child.id);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
|
||||
);
|
||||
|
||||
const { requeued, dead } = await queue.handleStalled();
|
||||
expect(dead).toHaveLength(0);
|
||||
expect(requeued.map(j => j.id)).toEqual([child.id]);
|
||||
expect(await jobStatus(child.id)).toBe('waiting');
|
||||
|
||||
const inbox = await engine.executeRaw<{ payload: unknown }>(
|
||||
`SELECT payload FROM minion_inbox WHERE job_id = $1`, [parent.id],
|
||||
);
|
||||
expect(inbox).toHaveLength(0);
|
||||
expect(await jobStatus(parent.id)).toBe('waiting-children');
|
||||
});
|
||||
|
||||
test('all three reapers route through the shared kill tail with their own outcome/error (D5.5)', async () => {
|
||||
// handleTimeouts → outcome 'timeout', error 'timeout exceeded'
|
||||
const p1 = await queue.add('agg1', {});
|
||||
await engine.executeRaw(`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [p1.id]);
|
||||
const c1 = await queue.add('c1', {}, { parent_job_id: p1.id });
|
||||
await queue.claim('t1', 30_000, 'default', ['c1', 'agg1']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET timeout_at = now() - interval '1 second', lock_until = now() + interval '1 minute' WHERE id = $1`, [c1.id],
|
||||
);
|
||||
await queue.handleTimeouts();
|
||||
|
||||
// handleWallClockTimeouts → outcome 'timeout', error 'wall-clock timeout exceeded'
|
||||
const p2 = await queue.add('agg2', {});
|
||||
await engine.executeRaw(`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [p2.id]);
|
||||
const c2 = await queue.add('c2', {}, { parent_job_id: p2.id });
|
||||
await queue.claim('t2', 30_000, 'default', ['c2', 'agg2']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET started_at = now() - interval '2 hours' WHERE id = $1`, [c2.id],
|
||||
);
|
||||
await queue.handleWallClockTimeouts(30_000);
|
||||
|
||||
const expectMsg = async (parentId: number, childId: number, outcome: string, error: string) => {
|
||||
const inbox = await engine.executeRaw<{ payload: unknown }>(
|
||||
`SELECT payload FROM minion_inbox WHERE job_id = $1`, [parentId],
|
||||
);
|
||||
expect(inbox).toHaveLength(1);
|
||||
const msg = (typeof inbox[0]!.payload === 'string'
|
||||
? JSON.parse(inbox[0]!.payload as string)
|
||||
: inbox[0]!.payload) as ChildDoneMessage;
|
||||
expect(msg.child_id).toBe(childId);
|
||||
expect(msg.outcome).toBe(outcome as ChildDoneMessage['outcome']);
|
||||
expect(msg.error).toBe(error);
|
||||
expect(await jobStatus(parentId)).toBe('waiting');
|
||||
};
|
||||
await expectMsg(p1.id, c1.id, 'timeout', 'timeout exceeded');
|
||||
await expectMsg(p2.id, c2.id, 'timeout', 'wall-clock timeout exceeded');
|
||||
});
|
||||
|
||||
test('retroactive sweep: a parent stranded BEFORE the fix (children already dead) self-heals on the next stall tick', async () => {
|
||||
// Simulate the pre-upgrade world: child is already terminal ('dead') with
|
||||
// NO child_done ever emitted, parent parked in waiting-children. No
|
||||
// per-kill unblock will ever revisit this pair — only the sweep can.
|
||||
const parent = await queue.add('agg-stranded', {});
|
||||
const child = await queue.add('child-stranded', {}, { parent_job_id: parent.id });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [parent.id],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'dead', error_text = 'max stalled count exceeded (pre-upgrade)' WHERE id = $1`, [child.id],
|
||||
);
|
||||
|
||||
const { requeued, dead } = await queue.handleStalled();
|
||||
expect(requeued).toHaveLength(0);
|
||||
expect(dead).toHaveLength(0);
|
||||
// The sweep healed the stranded parent even though THIS tick killed nothing.
|
||||
expect(await jobStatus(parent.id)).toBe('waiting');
|
||||
});
|
||||
|
||||
test('retroactive sweep does NOT unblock a parent with a live child', async () => {
|
||||
const parent = await queue.add('agg-live', {});
|
||||
await queue.add('child-live', {}, { parent_job_id: parent.id });
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting-children' WHERE id = $1`, [parent.id],
|
||||
);
|
||||
await queue.handleStalled();
|
||||
expect(await jobStatus(parent.id)).toBe('waiting-children');
|
||||
});
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* W0 fix-wave (Tier-1 #7) — started_at is reset on every AUTOMATIC re-run
|
||||
* path, so handleWallClockTimeouts (anchored on now() - started_at) measures
|
||||
* per-attempt runtime, never time a job spent parked in backoff.
|
||||
*
|
||||
* Pre-fix, only the manual `jobs retry` path cleared started_at (its
|
||||
* docstring documented the bug); the four automatic paths — failJob's
|
||||
* delayed branch, handleStalled's requeue, promoteDelayed, and
|
||||
* releaseLeaseFullJob (the 4th site, found in adversarial verification) —
|
||||
* all preserved the FIRST claim's timestamp, so an exponential-backoff job
|
||||
* could be dead-lettered by the wall-clock sweep before executing a single
|
||||
* line of its retry attempt.
|
||||
*/
|
||||
|
||||
import { test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
async function jobStatus(id: number): Promise<string> {
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status FROM minion_jobs WHERE id = $1`, [id],
|
||||
);
|
||||
return rows[0]!.status;
|
||||
}
|
||||
|
||||
async function startedAtOf(id: number): Promise<string | null> {
|
||||
const rows = await engine.executeRaw<{ started_at: string | null }>(
|
||||
`SELECT started_at::text AS started_at FROM minion_jobs WHERE id = $1`, [id],
|
||||
);
|
||||
return rows[0]?.started_at ?? null;
|
||||
}
|
||||
|
||||
async function addAndClaim(name = 'noop'): Promise<{ id: number; token: string }> {
|
||||
const job = await queue.add(name, {});
|
||||
const token = `tok-${Math.random().toString(36).slice(2)}`;
|
||||
const claimed = await queue.claim(token, 30_000, 'default', [name]);
|
||||
expect(claimed?.id).toBe(job.id);
|
||||
expect(await startedAtOf(job.id)).not.toBeNull();
|
||||
return { id: job.id, token };
|
||||
}
|
||||
|
||||
test("failJob's delayed branch clears started_at (terminal branches keep it)", async () => {
|
||||
const { id, token } = await addAndClaim();
|
||||
const delayed = await queue.failJob(id, token, 'boom', 'delayed', 50);
|
||||
expect(delayed?.status).toBe('delayed');
|
||||
expect(await startedAtOf(id)).toBeNull();
|
||||
|
||||
// Terminal failure keeps started_at (it feeds duration accounting).
|
||||
const { id: id2, token: token2 } = await addAndClaim();
|
||||
const dead = await queue.failJob(id2, token2, 'boom', 'dead');
|
||||
expect(dead?.status).toBe('dead');
|
||||
expect(await startedAtOf(id2)).not.toBeNull();
|
||||
});
|
||||
|
||||
test("handleStalled's requeue branch clears started_at", async () => {
|
||||
const { id } = await addAndClaim();
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [id],
|
||||
);
|
||||
const { requeued } = await queue.handleStalled();
|
||||
expect(requeued.map(j => j.id)).toContain(id);
|
||||
expect(await startedAtOf(id)).toBeNull();
|
||||
});
|
||||
|
||||
test('releaseLeaseFullJob clears started_at (lease bounce burns no wall-clock)', async () => {
|
||||
const { id, token } = await addAndClaim();
|
||||
const released = await queue.releaseLeaseFullJob(id, token, 'rate lease full (8/8)', 50);
|
||||
expect(released?.status).toBe('delayed');
|
||||
expect(await startedAtOf(id)).toBeNull();
|
||||
});
|
||||
|
||||
test('promoteDelayed defensively clears any stale started_at', async () => {
|
||||
const job = await queue.add('noop', {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs
|
||||
SET status = 'delayed', delay_until = now() - interval '1 second',
|
||||
started_at = now() - interval '1 hour'
|
||||
WHERE id = $1`, [job.id],
|
||||
);
|
||||
const promoted = await queue.promoteDelayed();
|
||||
expect(promoted.map(j => j.id)).toContain(job.id);
|
||||
expect(await startedAtOf(job.id)).toBeNull();
|
||||
});
|
||||
|
||||
test('end-to-end: a retried job survives the wall-clock sweep on its fresh attempt', async () => {
|
||||
const { id, token } = await addAndClaim();
|
||||
// Simulate a long first attempt: backdate started_at far past the sweep
|
||||
// threshold (no timeout_ms → threshold = lockDurationMs * 2 * max_stalled).
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET started_at = now() - interval '1 hour' WHERE id = $1`, [id],
|
||||
);
|
||||
await queue.failJob(id, token, 'first attempt failed', 'delayed', 0);
|
||||
const promoted = await queue.promoteDelayed();
|
||||
expect(promoted.map(j => j.id)).toContain(id);
|
||||
|
||||
const token2 = 'tok-second-attempt';
|
||||
const reclaimed = await queue.claim(token2, 30_000, 'default', ['noop']);
|
||||
expect(reclaimed?.id).toBe(id);
|
||||
|
||||
// Pre-fix: started_at still said now()-1h → 3.6M ms > 30_000*2*max_stalled
|
||||
// and the very next sweep dead-lettered the fresh attempt. Post-fix the
|
||||
// re-claim stamped a fresh started_at, so the sweep spares it.
|
||||
const killed = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed.map(j => j.id)).not.toContain(id);
|
||||
|
||||
// Negative control — the sweep itself still works: backdate the ACTIVE
|
||||
// attempt past the threshold and it dies.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET started_at = now() - interval '1 hour' WHERE id = $1`, [id],
|
||||
);
|
||||
const killed2 = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed2.map(j => j.id)).toContain(id);
|
||||
});
|
||||
|
||||
test('red-team 5th path: a re-claimed aggregator parent survives the wall-clock sweep', async () => {
|
||||
// Parent parked in waiting-children for LONGER than the wall-clock
|
||||
// threshold while its child runs; every unblock path (killJobs,
|
||||
// completeJob resolve, failJob branches, cancelJob, retroactive sweep,
|
||||
// resolveParent) must clear started_at, or claim()'s COALESCE keeps the
|
||||
// attempt-1 anchor and the sweep dead-letters the aggregation attempt
|
||||
// before it executes a line.
|
||||
const parent = await queue.add('agg-wallclock', {});
|
||||
const child = await queue.add('child-wc', {}, { parent_job_id: parent.id, max_stalled: 1 });
|
||||
// Parent claimed long ago (attempt-1 anchor), then parked waiting-children.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status = 'waiting-children', started_at = now() - interval '1 hour' WHERE id = $1`, [parent.id],
|
||||
);
|
||||
// Child stalls to death → killJobs unblocks the parent.
|
||||
await queue.claim('tok-wc', 30_000, 'default', ['child-wc', 'agg-wallclock']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1`, [child.id],
|
||||
);
|
||||
await queue.handleStalled();
|
||||
expect(await jobStatus(parent.id)).toBe('waiting');
|
||||
// The fix: unblock cleared the stale anchor.
|
||||
expect(await startedAtOf(parent.id)).toBeNull();
|
||||
|
||||
// Parent re-claims for its aggregation attempt → fresh anchor → survives.
|
||||
const reclaimed = await queue.claim('tok-wc2', 30_000, 'default', ['agg-wallclock', 'child-wc']);
|
||||
expect(reclaimed?.id).toBe(parent.id);
|
||||
const killed = await queue.handleWallClockTimeouts(30_000);
|
||||
expect(killed.map(j => j.id)).not.toContain(parent.id);
|
||||
});
|
||||
@@ -304,16 +304,21 @@ describe('check-engine-dynamic-import.sh', () => {
|
||||
});
|
||||
|
||||
describe('engine dynamic-import guard wiring', () => {
|
||||
it('is invoked through bash by check:all', () => {
|
||||
it('is wired into the verify registry (W0: check:all deleted; CHECKS array is THE registry)', () => {
|
||||
const pkg = JSON.parse(readFileSync(PACKAGE_JSON, 'utf8')) as {
|
||||
scripts: Record<string, string>;
|
||||
};
|
||||
expect(pkg.scripts['check:engine-dynamic-import']).toBe(
|
||||
'bash scripts/check-engine-dynamic-import.sh',
|
||||
);
|
||||
expect(pkg.scripts['check:all']).toContain(
|
||||
'bash scripts/check-engine-dynamic-import.sh',
|
||||
// The stale duplicate registry is gone for good…
|
||||
expect(pkg.scripts['check:all']).toBeUndefined();
|
||||
// …and the single registry carries this guard.
|
||||
const checks = readFileSync(
|
||||
new URL('../../scripts/run-verify-parallel.sh', import.meta.url),
|
||||
'utf8',
|
||||
);
|
||||
expect(checks).toContain('"check:engine-dynamic-import"');
|
||||
});
|
||||
|
||||
it('is listed by the authoritative verify dispatcher', () => {
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
/**
|
||||
* W0 ship-review coverage (GAP-2) — the snapshot loader's shape + hash guards.
|
||||
*
|
||||
* The fixture is default-on for every `bun run test`, so a wrong snapshot
|
||||
* poisons the whole suite (the 1280-vs-1536 incident: 115 failures from one
|
||||
* root cause). These tests pin the three refusal paths and the
|
||||
* handler-aware hash (D5.13).
|
||||
*/
|
||||
|
||||
import { test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import * as crypto from 'node:crypto';
|
||||
import { tryLoadSnapshot, computeSnapshotSchemaHash } from '../src/core/pglite-engine.ts';
|
||||
import { MIGRATIONS } from '../src/core/migrate.ts';
|
||||
import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts';
|
||||
import { getEmbeddingDimensions, getEmbeddingModel } from '../src/core/ai/gateway.ts';
|
||||
|
||||
let dir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), 'gbrain-snap-guard-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeFixture(versionContent: string): string {
|
||||
const tarPath = join(dir, 'snap.tar');
|
||||
writeFileSync(tarPath, 'not-a-real-tar-but-existence-is-what-matters');
|
||||
writeFileSync(join(dir, 'snap.version'), versionContent);
|
||||
return tarPath;
|
||||
}
|
||||
|
||||
const currentHash = () => computeSnapshotSchemaHash(MIGRATIONS, PGLITE_SCHEMA_SQL, crypto);
|
||||
|
||||
test('pre-W0 hash-only version file (no shape lines) is refused', () => {
|
||||
const tar = writeFixture(`${currentHash()}\n`);
|
||||
expect(tryLoadSnapshot(tar)).toBeNull();
|
||||
});
|
||||
|
||||
test('dims mismatch is refused even with a matching hash', () => {
|
||||
const tar = writeFixture(`${currentHash()}\ndims=99999\nmodel=${getEmbeddingModel()}\n`);
|
||||
expect(tryLoadSnapshot(tar)).toBeNull();
|
||||
});
|
||||
|
||||
test('model mismatch is refused even with a matching hash', () => {
|
||||
const tar = writeFixture(`${currentHash()}\ndims=${getEmbeddingDimensions()}\nmodel=other:model\n`);
|
||||
expect(tryLoadSnapshot(tar)).toBeNull();
|
||||
});
|
||||
|
||||
test('stale schema hash is refused even with a matching shape', () => {
|
||||
const tar = writeFixture(`deadbeef\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
|
||||
expect(tryLoadSnapshot(tar)).toBeNull();
|
||||
});
|
||||
|
||||
test('matching hash + shape loads the blob', () => {
|
||||
const tar = writeFixture(`${currentHash()}\ndims=${getEmbeddingDimensions()}\nmodel=${getEmbeddingModel()}\n`);
|
||||
const blob = tryLoadSnapshot(tar);
|
||||
expect(blob).not.toBeNull();
|
||||
expect(blob!.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('D5.13: a migration handler edit changes the hash (sql-only hashing missed 19 handler migrations)', () => {
|
||||
const base = [
|
||||
{ version: 1, name: 'a', sql: 'CREATE TABLE t(x int)' },
|
||||
{ version: 2, name: 'b', sql: '', handler: async () => 'original' },
|
||||
];
|
||||
const edited = [
|
||||
{ version: 1, name: 'a', sql: 'CREATE TABLE t(x int)' },
|
||||
{ version: 2, name: 'b', sql: '', handler: async () => 'EDITED BODY' },
|
||||
];
|
||||
const h1 = computeSnapshotSchemaHash(base, 'schema', crypto);
|
||||
const h2 = computeSnapshotSchemaHash(edited, 'schema', crypto);
|
||||
expect(h1).not.toBe(h2);
|
||||
// And identical handlers hash identically (determinism).
|
||||
const h3 = computeSnapshotSchemaHash(base, 'schema', crypto);
|
||||
expect(h1).toBe(h3);
|
||||
});
|
||||
@@ -216,6 +216,7 @@ describe('#1849 refresh-failure fails safe (F1A)', () => {
|
||||
let refreshCalls = 0;
|
||||
const failingLock: DbLockHandle = {
|
||||
id: 'x',
|
||||
acquiredAt: 'test-fence',
|
||||
refresh: async () => { refreshCalls++; throw new Error('pooler down'); },
|
||||
release: async () => {},
|
||||
};
|
||||
@@ -244,7 +245,8 @@ describe('#1849 refresh-failure fails safe (F1A)', () => {
|
||||
let mode: 'fail' | 'ok' = 'fail';
|
||||
const flakyLock: DbLockHandle = {
|
||||
id: 'x',
|
||||
refresh: async () => { if (mode === 'fail') throw new Error('blip'); },
|
||||
acquiredAt: 'test-fence',
|
||||
refresh: async () => { if (mode === 'fail') throw new Error('blip'); return true; },
|
||||
release: async () => {},
|
||||
};
|
||||
sup._setDbLockForTests(flakyLock);
|
||||
@@ -264,3 +266,31 @@ describe('#1849 refresh-failure fails safe (F1A)', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test('W0 (D5.10): fenced refresh returning false is CERTAIN loss — immediate LOCK_LOST, no threshold', async () => {
|
||||
const sup = new MinionSupervisor(engine, { cliPath: '/bin/sh', healthInterval: 0, json: true });
|
||||
const exitSpy = spyOn(process, 'exit').mockImplementation(((_code?: number) => {
|
||||
throw new Error(`exit:${_code}`);
|
||||
}) as never);
|
||||
|
||||
let refreshCalls = 0;
|
||||
const stolenLock: DbLockHandle = {
|
||||
id: 'x',
|
||||
acquiredAt: 'test-fence',
|
||||
refresh: async () => { refreshCalls++; return false; },
|
||||
release: async () => {},
|
||||
};
|
||||
sup._setDbLockForTests(stolenLock);
|
||||
|
||||
try {
|
||||
// FIRST tick exits — a fenced miss is proof of loss, not a blip to
|
||||
// count toward SUPERVISOR_LOCK_REFRESH_MAX_FAILURES (pre-fix it was
|
||||
// treated as SUCCESS and reset the failure counter while a second
|
||||
// supervisor drained the same queue).
|
||||
try { await sup._refreshDbLockForTests(); } catch { /* exit stub throws */ }
|
||||
expect(exitSpy).toHaveBeenCalledWith(ExitCodes.LOCK_LOST);
|
||||
expect(refreshCalls).toBe(1);
|
||||
} finally {
|
||||
exitSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user