Files
gbrain/test/fix-wave-structural.test.ts
Garry TanandClaude Fable 5 f15480b9d0 v0.42.75.0 fix(pglite): in-place WAL auto-repair for the macOS Aborted() startup crash (#2575, #223, #1670) (#3901)
* fix(pglite): in-place WAL auto-repair for the Aborted() startup crash (#223, #1670, #2575)

The 'macOS 26.x WASM bug' was a misdiagnosis: an unclean shutdown (typically
the OS-upgrade reboot) tears the data dir's WAL, and every subsequent open
fails WAL replay inside WASM with an opaque RuntimeError: Aborted(). This
ports the pg_resetwal recovery upstream rejected (electric-sql/pglite#994,
by @yestheboxer) and wires it into connect() as bounded auto-repair:

- src/core/pglite-resetwal.ts: pg_resetwal for PG17 NodeFS dirs, fail-closed
  layout validation, atomic+durable writes (tmp+fsync+rename), idempotent.
- src/core/pglite-repair.ts: whole-pg_wal-dir rename backup (zero transient
  disk), overwrite-order restore with mtime guard, cooldown sidecar +
  episode-scoped backup retention (newest 3 episodes), and a never-throws
  engine seam. Kill-switch: GBRAIN_PGLITE_WAL_REPAIR=off.
- pglite-engine.ts: verdict rename macos-26-3 -> wasm-abort, classifier now
  matches the real production message (it previously fell to 'unknown'),
  corrupt-beats-wasm precedence preserved, honest per-outcome error copy
  incl. the failed-not-restored arm, and repair only under a cleanly-acquired
  lock (new LockHandle.reaped provenance; never after reaping a holder).
- gbrain pglite-repair: manual dry-run/repair command (validate-before-lock,
  serve/reaped refusals, no --force by design).
- doctor: pglite_data_dir fs-check with recurrence escalation and backup
  inventory when a PGLite brain fails to connect.
- reinit-pglite: embedding flags default from file-only config so the
  recovery ladder's rebuild rung works bare mid-outage.
- stringifyPgliteInitError: message-less Emscripten ErrnoError objects no
  longer surface as [object Object].

Regression-tested against real brains: corrupt every WAL segment (truncate
and garbage variants), reopen, auto-repair fires, original rows readable,
process.exitCode stays contained (#2084).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): replace the macOS-26.x misdiagnosis with the corrupt-WAL recovery ladder

README + INSTALL.md shipped (via #1671) the claim that PGLite is incompatible
with macOS 26.x and that a Bun/WASM fix would restore it. The real cause is
torn WAL state from the upgrade reboot, now auto-repaired in place. Rewrites
those sections around the recovery ladder (auto-repair -> gbrain pglite-repair
-> reinit-pglite -> engine switch; native-Postgres recipe kept, credit
@roysaurav), adds the ENGINES.md troubleshooting section, updates the
KEY_FILES.md entries to current truth, files the two follow-up TODOs
(SIGTERM engine-close extension; pglite upgrade blocker), and regenerates
the llms bundles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(pglite): harden WAL auto-repair (pre-landing + adversarial review)

Review-army (security/testing/maintainability/perf) + Claude & Codex
adversarial passes on the WAL-repair wave. Correctness + safety hardening,
no behavior change to the happy path:

- Live-writer safety: repair refuses any reaped lock acquisition, a corrupt
  (unknowable-liveness) reap writes a cross-process quarantine marker that
  gates auto-repair AND the manual command for 10 min, isProcessAlive treats
  only ESRCH as dead (EPERM/malformed-pid read as alive), and a live
  postmaster.pid (native Postgres) is refused. Lock heartbeat + initial write
  are atomic (tmp+rename) so a torn read can't misclassify a healthy holder;
  an in-flight acquisition is no longer mistaken for corrupt.
- resetWal verifies the stored pg_control CRC before trusting/re-signing it —
  a damaged control file routes to rebuild instead of laundering corrupt
  checkpoint counters under a fresh CRC. Atomic 'wx' writes (no symlink
  follow), whole-pg_wal-dir rename backup, 64MB seg-size cap.
- Honest failure reporting: repairPgliteWal threads the real restore result
  out via WalRepairError so the 'failed-restored' vs 'failed-not-restored'
  message never lies; the not-restored copy names the correct restore paths.
- Episode lifecycle: episodes close on the next healthy connect (not just on
  a verified repair), a gutted (restored) backup loses its pin, stale (>24h)
  episode backups aren't reused, and the cooldown also caps repaired-only
  crash loops. Empty backup dirs are pruned on refusal.
- Command: rejects unknown flags and valueless --path (a destructive command
  must not silently mis-parse), confirm prompt goes to stderr (stdout stays
  clean for --json), embedding-flag defaults come from the config file only.
- Symlink confinement extended to global/; sidecar reuse path validated
  (prefix + no '..' + must still hold pg_wal); sidecar writes atomic.
- doctor recurrence escalation counts all attempts; data dir absolutized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(pglite): current-state KEY_FILES + WAL-repair follow-up TODOs

KEY_FILES.md pglite entries updated to the hardened truth (reap marker +
quarantine, atomic writes, CRC gate, global-symlink refusal, WalRepairError,
episode lifecycle). TODOS.md files the deferred judgment-call follow-ups
(unclean-shutdown gate on auto-repair; non-gbrain pglite consumer boundary;
mixed-version torn-lock double-read).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.75.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 17:01:20 -07:00

304 lines
16 KiB
TypeScript

/**
* Structural assertions for fix-wave fixes whose behavior is best verified
* at source-shape level rather than via a runtime harness:
*
* - #1125 query drain cache writes — assert cli.ts awaits the drain after
* the query op completes.
* - #1090 admin embed — assert the two-tier resolution (cwd path + embedded
* manifest fallback) is in serve-http.ts and consumes ADMIN_ASSETS.
* - #1077 admin register-client PKCE — assert the admin endpoint honors
* grantTypes / redirectUris / tokenEndpointAuthMethod from the body.
* - #1100 PGLite phaseASchema — assert the v0.11.0 orchestrator routes
* in-process when the engine is pglite (not via execSync subprocess).
* - #1124 query no-expand — assert the parseOpArgs negation logic exists.
*
* Source-grep regression tests are the right tool when the rule is "this
* specific line shape must stay present"; a behavioral test would either
* duplicate what an E2E covers or require heavy mocking that hides the
* regression behind a test seam.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
describe('v0.42.20.0 — search-cache drained via the background-work registry', () => {
// Supersedes the v0.36.1.x #1125 query-only drain: search-cache now registers
// a registry drainer (drained for BOTH search and query, bounded), and cli.ts
// drains the whole registry rather than calling awaitPendingSearchCacheWrites
// directly for the 'query' op only.
test('hybrid.ts registers a bounded search-cache drainer', () => {
const src = readFileSync('src/core/search/hybrid.ts', 'utf8');
expect(src).toMatch(/export async function awaitPendingSearchCacheWrites/);
expect(src).toMatch(/pendingCacheWrites\.add\(promise\)/);
expect(src).toMatch(/trackCacheWrite\(/);
// Now bounded (was an unbounded Promise.allSettled) + registered.
expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'search-cache'/);
expect(src).toMatch(/Promise\.race/);
});
});
describe('v0.36.1.x #1090 — admin embed two-tier resolution', () => {
test('serve-http.ts uses ADMIN_ASSETS manifest when admin/dist is not next to cwd', () => {
const src = readFileSync('src/commands/serve-http.ts', 'utf8');
expect(src).toMatch(/import\(['"]\.\.\/admin-embedded/);
expect(src).toMatch(/ADMIN_ASSETS/);
expect(src).toMatch(/ADMIN_INDEX_HTML/);
// Two-tier: dev path (cwd-relative admin/dist) AND embedded manifest fallback
expect(src).toMatch(/useDevPath/);
});
test('src/admin-embedded.ts is auto-generated with file: imports', () => {
const src = readFileSync('src/admin-embedded.ts', 'utf8');
expect(src).toMatch(/AUTO-GENERATED/);
expect(src).toMatch(/with \{ type: 'file' \}/);
expect(src).toMatch(/export const ADMIN_ASSETS/);
expect(src).toMatch(/export const ADMIN_INDEX_HTML/);
});
test('build script + CI guard exist', () => {
const buildSrc = readFileSync('scripts/build-admin-embedded.ts', 'utf8');
expect(buildSrc).toMatch(/walk\(DIST/);
expect(buildSrc).toMatch(/with \{ type: 'file' \}/);
const guard = readFileSync('scripts/check-admin-embedded.sh', 'utf8');
expect(guard).toMatch(/git diff --exit-code -- src\/admin-embedded\.ts/);
});
});
describe('v0.36.1.x #1077 — admin register-client supports PKCE public clients', () => {
test('admin endpoint reads grantTypes / redirectUris / tokenEndpointAuthMethod from request body', () => {
const src = readFileSync('src/commands/serve-http.ts', 'utf8');
// The destructure must surface name / tokenTtl / grantTypes /
// redirectUris / tokenEndpointAuthMethod from req.body. v0.39.3.0
// WARN-9 (PR #1308) moved `scopes` to a separate read line that
// accepts BOTH `scopes` (admin SPA) AND `scope` (OAuth wire singular)
// via `?? `, so this regex no longer requires `scopes` in the inline
// destructure — it's separately covered by the scope-source check
// below.
expect(src).toMatch(/const\s+\{\s*name,\s*(?:[^}]*?,\s*)?tokenTtl,\s*grantTypes,\s*redirectUris,\s*tokenEndpointAuthMethod\s*\}\s*=\s*req\.body/);
// v0.39.3.0 WARN-9: the route must still read a `scope`/`scopes` field
// (under either name) from req.body. Pin the fallback pattern so the
// PKCE-fix regression contract stays load-bearing.
expect(src).toMatch(/req\.body[^;]*scopes\s*\?\?\s*[^;]*scope\b/);
// v0.41.3 (T4 atomicity fix, codex F4): admin endpoint now validates
// tokenEndpointAuthMethod via the shared validator and passes it to
// registerClientManual as a positional arg. Pre-v0.41.3 the route did
// INSERT (confidential) → UPDATE (NULL out secret_hash) for the 'none'
// case, which left a confidential row stranded if the UPDATE failed.
// Atomic now: one INSERT writes the correct shape; no post-insert
// UPDATE block (the regex deliberately asserts the post-insert UPDATE
// is GONE).
expect(src).toMatch(/validateTokenEndpointAuthMethod\(tokenEndpointAuthMethod\)/);
expect(src).toMatch(/registerClientManual\([^)]*validatedAuthMethod[^)]*\)/);
// Regression guard: post-insert UPDATE flipping client_secret_hash to
// NULL based on a runtime check is exactly the non-atomic pattern T4
// killed. Re-introducing it brings back codex F4.
expect(src).not.toMatch(/UPDATE oauth_clients SET client_secret_hash = NULL, token_endpoint_auth_method = 'none'/);
});
});
describe('v0.41.37.0 #1605 — v0.11.0 phaseASchema routes in-process for ALL engines', () => {
test('phaseASchema calls runMigrateOnlyCore (in-process) + is awaited', () => {
// Supersedes #1100's PGLite-only in-process branch. v0.41.37.0 #1605 routes
// EVERY engine through runMigrateOnlyCore (no execSync subprocess at all),
// which is strictly stronger: PGLite still never subprocesses, AND the
// Windows+Postgres getaddrinfo-ENOTFOUND spawn bug is closed too.
// The eng.initSchema() call moved into src/commands/migrations/in-process.ts.
const src = readFileSync('src/commands/migrations/v0_11_0.ts', 'utf8');
expect(src).toContain('runMigrateOnlyCore()');
expect(src).not.toContain("execSync('gbrain init --migrate-only'");
expect(src).toMatch(/await\s+phaseASchema/);
});
test('apply-migrations skips pre-flight schema-version probe on PGLite', () => {
const src = readFileSync('src/commands/apply-migrations.ts', 'utf8');
expect(src).toMatch(/skipPreflight\s*=\s*cfg\.engine\s*===\s*'pglite'/);
});
});
describe('v0.36.1.x #1124 — query --no-expand actually negates expand', () => {
test("cli.ts parseOpArgs handles --no-<key> as boolean negation", () => {
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).toMatch(/arg\.startsWith\(['"]--no-['"]\)/);
expect(src).toMatch(/positiveDef\?\.type\s*===\s*'boolean'/);
expect(src).toMatch(/params\[positiveKey\]\s*=\s*false/);
});
});
describe('v0.42.20.0 — background-work registry drains every sink before disconnect', () => {
// Supersedes the v0.41.8.0 #1247/#1269/#1290 per-call last-retrieved drain:
// last-retrieved is one of four registry sinks. #2084 moved the registry
// drain out of cli.ts's inline finallys into finishCliTeardown
// (cli-force-exit.ts), which every cli.ts teardown site routes through —
// the drain-before-disconnect invariant is pinned there (and behaviorally
// by test/cli-finish-teardown.test.ts).
test('cli-force-exit.ts imports + drains the registry inside finishCliTeardown', () => {
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit[\s\S]*?\}\s*from\s+['"]\.\/background-work\.ts['"]/);
expect(src).toMatch(/export async function finishCliTeardown/);
});
test('last-retrieved.ts still exports the bounded drain + registers a drainer', () => {
const src = readFileSync('src/core/last-retrieved.ts', 'utf8');
expect(src).toMatch(/export async function awaitPendingLastRetrievedWrites/);
expect(src).toMatch(/pendingLastRetrievedWrites\s*=\s*new\s+Set/);
expect(src).toMatch(/Promise\.race/);
expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'last-retrieved'/);
});
test('all four sinks register a drainer', () => {
expect(readFileSync('src/core/facts/queue.ts', 'utf8'))
.toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'facts'[\s\S]*?abort:/);
expect(readFileSync('src/core/search/hybrid.ts', 'utf8'))
.toMatch(/name:\s*'search-cache'/);
expect(readFileSync('src/core/last-retrieved.ts', 'utf8'))
.toMatch(/name:\s*'last-retrieved'/);
expect(readFileSync('src/core/eval-capture.ts', 'utf8'))
.toMatch(/name:\s*'eval-capture'/);
});
test('finishCliTeardown positioning: registry drain appears BEFORE engine disconnect', () => {
// #2084: the invariant moved from cli.ts's inline finallys into the shared
// helper. The drain must run against a live engine (facts abort-path
// logIngest, #1762) before disconnect tears the pools down.
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
const drainCallRe = /await\s+drain\s*\(\s*\{\s*timeoutMs:\s*drainTimeoutMs\s*\}\s*\)/;
const disconnectCallRe = /await\s+opts\.engine\.disconnect\s*\(/;
expect(src).toMatch(drainCallRe);
expect(src).toMatch(disconnectCallRe);
const drainIdx = src.indexOf(src.match(drainCallRe)![0]);
const disconnectIdx = src.indexOf(src.match(disconnectCallRe)![0]);
expect(drainIdx).toBeLessThan(disconnectIdx);
});
test('background-work.ts: Map registry, ordered drain, awaited abort, test seam', () => {
const src = readFileSync('src/core/background-work.ts', 'utf8');
expect(src).toMatch(/new\s+Map<string,\s*BackgroundWorkDrainer>/);
expect(src).toMatch(/sort\(\s*\(a,\s*b\)\s*=>\s*a\.order\s*-\s*b\.order/);
expect(src).toMatch(/if\s*\(unfinished\s*>\s*0\s*&&\s*d\.abort\)\s*\{[\s\S]*?await\s+d\.abort\(\)/);
expect(src).toMatch(/export function __registerDrainerForTest/);
});
test('cli-force-exit.ts daemon guard excludes "serve"', () => {
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
expect(src).toMatch(/export function shouldForceExitAfterMain/);
expect(src).toMatch(/DAEMON_COMMANDS[\s\S]*serve/);
});
});
describe('#2084 — cli.ts owns process-exit teardown via finishCliTeardown', () => {
test('no bare awaited engine disconnects remain in cli.ts', () => {
// The awaited forms are the call-site contract (comments never use the
// awaited literal, so this is comment-proof — eng-review D13.2). A bare
// disconnect skips the bounded drain + computed-deadline backstop and
// reopens the lingering-socket hang class.
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).not.toContain('await engine.disconnect()');
expect(src).not.toContain('await eng.disconnect()');
});
test('the pre-handler hard-deadline timer is gone (handler time is not teardown budget)', () => {
// Pre-#2084 the op-dispatch timer armed BEFORE the op handler, so any op
// slower than 10s was force-killed mid-run with exit 0 and truncated
// output. The deadline now arms inside finishCliTeardown, at teardown
// start only.
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
});
test('all nine swept sites route through finishCliTeardown; one exit seam', () => {
const src = readFileSync('src/cli.ts', 'utf8');
const calls = src.match(/await finishCliTeardown\(/g) ?? [];
expect(calls.length).toBeGreaterThanOrEqual(9);
// The single process-exit seam: flushThenExit in the import.meta.main
// block, fed by currentExitCode().
expect(src).toMatch(/import\.meta\.main/);
expect(src).toMatch(/flushThenExit\(currentExitCode\(\)\)/);
});
test('pglite-engine contains the Emscripten process.exitCode hijack', () => {
// PGLite's WASM runtime writes its own status into process.exitCode (99
// alive / exit status on close) and ignores `undefined` assignment. The
// create call runs inside preservingProcessExitCode to keep the global
// tidy; close is deliberately unwrapped (see below) — the CLI's verdict
// is immune either way via the owned channel.
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/preservingProcessExitCode\(\(\)\s*=>\s*\n?\s*PGlite\.create/);
// close stays UNWRAPPED by design: its status write is baseline behavior
// test runners depend on; the CLI's verdict is immune because it lives in
// the gbrain-owned channel, never read back from process.exitCode.
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
expect(helper).toMatch(/let cliVerdict: number \| null = null/);
expect(helper).toMatch(/return cliVerdict \?\? 0/);
// The op-dispatch catch must set the verdict through the owned channel.
const cli = readFileSync('src/cli.ts', 'utf8');
expect(cli).toMatch(/setCliExitVerdict\(1\);/);
});
});
describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
test('pglite-engine.ts exports classifyPgliteInitError + buildPgliteInitErrorMessage', () => {
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/export function classifyPgliteInitError/);
expect(src).toMatch(/export function buildPgliteInitErrorMessage/);
// Per Codex finding #9: regex tightened to $$bunfs OR ENOENT+pglite.data
expect(src).toMatch(/\$\$bunfs/);
expect(src).toMatch(/ENOENT/);
});
test('pglite-engine.ts connect catch block routes through the classifier', () => {
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/classifyPgliteInitError\(original\)/);
// WAL-repair wave: the call gained platform + repair-context args, so pin
// only the (verdict, original, …) prefix — the routing seam, not the arity.
expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original/);
});
});
describe('WAL-repair wave structural pins (#223/#2575)', () => {
test('connect() catch wires the WAL auto-repair seam', () => {
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/attemptWalRepairAndRetry\(/);
});
test('the repair-retry lambda stays inside the #2084 exitCode guard', () => {
// The retry re-runs PGlite.create; unguarded, Emscripten would hijack
// process.exitCode on the retry path exactly as it did on the first
// attempt (the #2084 class). Pin the wrap at the seam call-site.
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/attemptWalRepairAndRetry\([\s\S]{0,300}preservingProcessExitCode/);
});
test('no bare PGlite.create outside the wrapped engine sites', () => {
// The repair/resetwal modules take the retry as a callback — if either
// grew its own PGlite.create call it would bypass BOTH the exitCode
// guard and the single-writer lock.
const repair = readFileSync('src/core/pglite-repair.ts', 'utf8');
const resetwal = readFileSync('src/core/pglite-resetwal.ts', 'utf8');
expect(repair).not.toMatch(/PGlite\.create/);
expect(resetwal).not.toMatch(/PGlite\.create/);
});
test('pglite-resetwal.ts carries the upstream attribution', () => {
// The reset-WAL sequence mirrors upstream electric-sql/pglite PR #994;
// the pointer is the audit trail for future divergence.
const resetwal = readFileSync('src/core/pglite-resetwal.ts', 'utf8');
expect(resetwal).toContain('electric-sql/pglite/pull/994');
});
});
describe('v0.42.43.0 #2095 — volunteer-events sink + cycle purge wiring (structural pins)', () => {
test('volunteer-events registers a background-work drainer (order 4)', () => {
// Deleting this registration would silently drop volunteer events on
// every CLI exit with no behavioral test failing — same pin class as the
// other four sinks above.
const src = readFileSync('src/core/context/volunteer-events.ts', 'utf8');
expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'volunteer-events'/);
expect(src).toMatch(/order:\s*4/);
});
test("the dream cycle's purge phase invokes purgeStaleVolunteerEvents and reports the count", () => {
const src = readFileSync('src/core/cycle.ts', 'utf8');
expect(src).toMatch(/purgeStaleVolunteerEvents\(engine\)/);
expect(src).toMatch(/purged_volunteer_events_count/);
});
});