From f15480b9d04b342d8d261fb4e8a6784bd9478be3 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 8 Aug 2026 17:01:20 -0700 Subject: [PATCH] v0.42.75.0 fix(pglite): in-place WAL auto-repair for the macOS Aborted() startup crash (#2575, #223, #1670) (#3901) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(pglite): in-place WAL auto-repair for the Aborted() startup crash (#223, #1670, #2575) The 'macOS 26.x WASM bug' was a misdiagnosis: an unclean shutdown (typically the OS-upgrade reboot) tears the data dir's WAL, and every subsequent open fails WAL replay inside WASM with an opaque RuntimeError: Aborted(). This ports the pg_resetwal recovery upstream rejected (electric-sql/pglite#994, by @yestheboxer) and wires it into connect() as bounded auto-repair: - src/core/pglite-resetwal.ts: pg_resetwal for PG17 NodeFS dirs, fail-closed layout validation, atomic+durable writes (tmp+fsync+rename), idempotent. - src/core/pglite-repair.ts: whole-pg_wal-dir rename backup (zero transient disk), overwrite-order restore with mtime guard, cooldown sidecar + episode-scoped backup retention (newest 3 episodes), and a never-throws engine seam. Kill-switch: GBRAIN_PGLITE_WAL_REPAIR=off. - pglite-engine.ts: verdict rename macos-26-3 -> wasm-abort, classifier now matches the real production message (it previously fell to 'unknown'), corrupt-beats-wasm precedence preserved, honest per-outcome error copy incl. the failed-not-restored arm, and repair only under a cleanly-acquired lock (new LockHandle.reaped provenance; never after reaping a holder). - gbrain pglite-repair: manual dry-run/repair command (validate-before-lock, serve/reaped refusals, no --force by design). - doctor: pglite_data_dir fs-check with recurrence escalation and backup inventory when a PGLite brain fails to connect. - reinit-pglite: embedding flags default from file-only config so the recovery ladder's rebuild rung works bare mid-outage. - stringifyPgliteInitError: message-less Emscripten ErrnoError objects no longer surface as [object Object]. Regression-tested against real brains: corrupt every WAL segment (truncate and garbage variants), reopen, auto-repair fires, original rows readable, process.exitCode stays contained (#2084). Co-Authored-By: Claude Fable 5 * docs(pglite): replace the macOS-26.x misdiagnosis with the corrupt-WAL recovery ladder README + INSTALL.md shipped (via #1671) the claim that PGLite is incompatible with macOS 26.x and that a Bun/WASM fix would restore it. The real cause is torn WAL state from the upgrade reboot, now auto-repaired in place. Rewrites those sections around the recovery ladder (auto-repair -> gbrain pglite-repair -> reinit-pglite -> engine switch; native-Postgres recipe kept, credit @roysaurav), adds the ENGINES.md troubleshooting section, updates the KEY_FILES.md entries to current truth, files the two follow-up TODOs (SIGTERM engine-close extension; pglite upgrade blocker), and regenerates the llms bundles. Co-Authored-By: Claude Fable 5 * fix(pglite): harden WAL auto-repair (pre-landing + adversarial review) Review-army (security/testing/maintainability/perf) + Claude & Codex adversarial passes on the WAL-repair wave. Correctness + safety hardening, no behavior change to the happy path: - Live-writer safety: repair refuses any reaped lock acquisition, a corrupt (unknowable-liveness) reap writes a cross-process quarantine marker that gates auto-repair AND the manual command for 10 min, isProcessAlive treats only ESRCH as dead (EPERM/malformed-pid read as alive), and a live postmaster.pid (native Postgres) is refused. Lock heartbeat + initial write are atomic (tmp+rename) so a torn read can't misclassify a healthy holder; an in-flight acquisition is no longer mistaken for corrupt. - resetWal verifies the stored pg_control CRC before trusting/re-signing it — a damaged control file routes to rebuild instead of laundering corrupt checkpoint counters under a fresh CRC. Atomic 'wx' writes (no symlink follow), whole-pg_wal-dir rename backup, 64MB seg-size cap. - Honest failure reporting: repairPgliteWal threads the real restore result out via WalRepairError so the 'failed-restored' vs 'failed-not-restored' message never lies; the not-restored copy names the correct restore paths. - Episode lifecycle: episodes close on the next healthy connect (not just on a verified repair), a gutted (restored) backup loses its pin, stale (>24h) episode backups aren't reused, and the cooldown also caps repaired-only crash loops. Empty backup dirs are pruned on refusal. - Command: rejects unknown flags and valueless --path (a destructive command must not silently mis-parse), confirm prompt goes to stderr (stdout stays clean for --json), embedding-flag defaults come from the config file only. - Symlink confinement extended to global/; sidecar reuse path validated (prefix + no '..' + must still hold pg_wal); sidecar writes atomic. - doctor recurrence escalation counts all attempts; data dir absolutized. Co-Authored-By: Claude Fable 5 * docs(pglite): current-state KEY_FILES + WAL-repair follow-up TODOs KEY_FILES.md pglite entries updated to the hardened truth (reap marker + quarantine, atomic writes, CRC gate, global-symlink refusal, WalRepairError, episode lifecycle). TODOS.md files the deferred judgment-call follow-ups (unclean-shutdown gate on auto-repair; non-gbrain pglite consumer boundary; mixed-version torn-lock double-read). Co-Authored-By: Claude Fable 5 * chore: bump version and changelog (v0.42.75.0) Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 31 + README.md | 2 +- TODOS.md | 49 ++ VERSION | 2 +- docs/ENGINES.md | 60 ++ docs/INSTALL.md | 19 +- docs/architecture/KEY_FILES.md | 8 +- llms-full.txt | 62 +- package.json | 2 +- src/cli.ts | 12 +- src/commands/doctor.ts | 111 ++++ src/commands/pglite-repair.ts | 340 ++++++++++ src/commands/reinit-pglite.ts | 71 +- src/core/doctor-categories.ts | 1 + src/core/pglite-engine.ts | 220 +++++- src/core/pglite-lock.ts | 101 ++- src/core/pglite-repair.ts | 771 ++++++++++++++++++++++ src/core/pglite-resetwal.ts | 354 ++++++++++ test/doctor-pglite-datadir.test.ts | 240 +++++++ test/e2e/pglite-cli-exit.serial.test.ts | 49 +- test/fix-wave-structural.test.ts | 36 +- test/pglite-init-classifier.test.ts | 126 +++- test/pglite-lock.test.ts | 97 ++- test/pglite-repair-command.serial.test.ts | 418 ++++++++++++ test/pglite-repair.test.ts | 548 +++++++++++++++ test/pglite-resetwal.test.ts | 190 ++++++ test/pglite-wal-repair.serial.test.ts | 200 ++++++ test/v0_37_gap_fill.serial.test.ts | 135 +++- 28 files changed, 4192 insertions(+), 63 deletions(-) create mode 100644 src/commands/pglite-repair.ts create mode 100644 src/core/pglite-repair.ts create mode 100644 src/core/pglite-resetwal.ts create mode 100644 test/doctor-pglite-datadir.test.ts create mode 100644 test/pglite-repair-command.serial.test.ts create mode 100644 test/pglite-repair.test.ts create mode 100644 test/pglite-resetwal.test.ts create mode 100644 test/pglite-wal-repair.serial.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index af63efcf4..0680908b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,37 @@ All notable changes to GBrain will be documented in this file. +## [0.42.75.0] - 2026-08-08 + +**The "PGLite crashes on macOS 26" era is over: gbrain now repairs a torn brain in place, automatically, with your data preserved.** + +The dreaded `RuntimeError: Aborted()` at startup — the one that made zero-config brains unusable after a macOS upgrade and pushed people onto Homebrew Postgres — was never a macOS or WASM bug. An unclean shutdown (typically the upgrade reboot) tears the write-ahead log inside the data dir, and every open after that dies replaying it. gbrain now detects that failure on any command, backs up the WAL state to a sibling directory, resets it in place (the pg_resetwal recovery Postgres has shipped for decades, ported to run against PGLite data dirs), and reopens your brain — pages, embeddings, and history intact. Transactions that never reached a checkpoint may be lost; that is the standard trade for a database that would otherwise not open at all. + +### Added +- **Automatic WAL repair on startup.** A torn-WAL abort self-heals on the next gbrain command: backup → in-place reset → retry, with a loud notice naming the backup and recommending `gbrain doctor`. Disable with `GBRAIN_PGLITE_WAL_REPAIR=off`. +- **`gbrain pglite-repair`** — the deliberate version: `--dry-run` gives a read-only diagnosis of the data dir; `--yes` runs the same in-place repair manually. Refuses to operate while any live process holds the brain, and never force-removes another process's lock. +- **`gbrain doctor` diagnoses unopenable PGLite brains.** A new `pglite_data_dir` check reads the data dir from disk when connect fails, names the right recovery rung (repair vs rebuild), inventories repair backups, and escalates when repairs keep recurring — the signal that something is still killing gbrain mid-write. +- **Recovery guardrails throughout:** repair runs only under a cleanly-acquired lock (never after taking over another process's lock, with a quarantine window when a lock's holder couldn't be verified); a live database — including a native Postgres one — is refused by a `postmaster.pid` liveness check; repeated attempts inside one corruption episode reuse one backup instead of stacking copies (newest three episodes retained); a cooldown stops repair loops from silently eating data on machines where crashes keep recurring; and every restore path reports honestly whether your original files are back in place or waiting in the backup. + +### Changed +- **`gbrain reinit-pglite` works bare.** The embedding model and dimensions now default from your config file, so the rebuild rung of the recovery ladder is one command mid-outage (explicit flags still win; environment overrides are deliberately ignored so a stale shell export can't change the rebuild target). +- **Honest error messages.** The startup-abort hint now names the real cause (torn WAL after an unclean shutdown), states exactly what auto-repair did or why it stood down, and lays out the full ladder: repair → rebuild → engine switch. The docs that claimed PGLite is "incompatible with macOS 26.x" have been rewritten (README, INSTALL, ENGINES) — thanks @roysaurav for the original native-Postgres walkthrough, which remains the engine-switch rung. +- Message-less WASM error objects no longer surface as `[object Object]`. + +### Fixed +- The classifier that routes startup failures now matches the abort message PGLite actually produces (it previously fell through to a generic hint), while catalog corruption keeps routing to rebuild — WAL repair is never suggested for damage it cannot fix. +- Lock-file reads can no longer misclassify a healthy live holder as corrupt (writes are atomic now), a holder owned by another user is treated as alive, and an in-flight acquisition is no longer mistaken for a corrupt lock. + +Credit where due: @yang1996202-cpu (#2575), @AndreLYL (#223), and @roysaurav (#1670) for reports and diagnosis, the #223 thread contributors whose recoveries proved the root cause, and @yestheboxer, whose rejected upstream recovery PR (electric-sql/pglite#994) this port builds on. + +### To take advantage of v0.42.75.0 + +```bash +gbrain upgrade +``` + +If your brain currently won't open, that's it — the next command repairs it. If you'd rather look first: `gbrain pglite-repair --dry-run`. + ## [0.42.74.0] - 2026-08-07 **Two fixes for agents that reach a brain over the network: takes-holder visibility now works the way you set it, and the voice recipe is safe by default.** diff --git a/README.md b/README.md index 000b4bd72..b11c2e36d 100644 --- a/README.md +++ b/README.md @@ -317,7 +317,7 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h ## Troubleshooting -**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe). +**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe). **`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model :` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing. diff --git a/TODOS.md b/TODOS.md index af31e9b0a..34ff1362b 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,54 @@ # TODOS +## WAL-repair wave follow-ups (#223/#1670/#2575) + +- [ ] **P2 — gate auto-repair on an unclean-shutdown marker (adversarial F7).** The classifier + deliberately over-matches (`RuntimeError`/`unreachable` → `wasm-abort`). If an unclean + shutdown leaves a REPLAYABLE WAL tail (normal crash recovery would restore those committed + txns) and the reopen then fails on a transient WASM error (OOM), auto-repair fires, layout + validation can't tell torn from replayable, and resetWal discards the tail while the notice + says "data preserved." Bounded today (backup always taken + restore + honest failure + repeated + attempts capped), but a false-positive-with-successful-retry silently drops committed data. + Fix direction (probe-verified): PGLite removes `postmaster.pid` on clean close, so gate AUTO + repair (not the manual command) on `postmaster.pid` presence — a clean dir that aborts is not + torn-WAL. Requires making the serial regression test stamp a `postmaster.pid` before corrupting + (it currently clean-disconnects then corrupts, which the red-team flagged as unfaithful anyway). + Needs a recall/precision call before landing. +- [ ] **P3 — live non-gbrain PGLite consumer not caught by the postmaster.pid liveness guard + (adversarial F8).** PGLite writes a sentinel `postmaster.pid` of `-42`; the liveness refusal in + `validateWalRepairTarget` requires `pid > 0`, so it protects native Postgres dirs but not a + non-gbrain pglite app that has the dir open (such an app writes no `.gbrain-lock`). Deliberate + misuse of `pglite-repair --path ` required. Option: refuse when + postmaster.pid holds pid ≤ 0 with a very recent mtime, or document the boundary. +- [ ] **P3 — mixed-version torn-lock read (adversarial F10 residual).** The heartbeat + initial + lock writes are atomic (tmp+rename) now, but an OLD gbrain binary writing heartbeats IN PLACE + while a NEW binary poll-reads can still catch a torn read → corrupt-lock verdict → a live + holder's lock reaped → two writers (the #2348 class, version-skew-triggered). The reap marker + quarantines repair, not the concurrent open. Cheap hardening: double-read the lock file (~50ms + apart) before declaring it corrupt. + + +- [ ] **P2 — graceful PGLite close on SIGTERM for the remaining long-running paths.** + The torn-WAL genesis this wave repairs is an unclean shutdown: `src/core/process-cleanup.ts` + releases locks on SIGTERM but never closes the PGlite handle, so `serve` / `jobs work` / + `sync` killed mid-write (macOS-upgrade reboot, `systemctl stop`) leave the WAL torn. + Autopilot already ships the pattern (d2fd1f29, #3178/#1872: `registerCleanup('autopilot-engine-close', ...)` + — abort in-flight work → ≤2s bounded wait inside the 3s cleanup deadline → + `engine.disconnect()`, double-call safe; rationale comment at autopilot.ts:438-452). + Extend that exact pattern to the remaining long-running PGLite paths (register in + connect()/command scope; dedupe so autopilot doesn't double-close), pinned by a serial + lifecycle test. Interacts with #2084 exitCode containment + #1337 close ordering — read + those comments in pglite-engine.ts first. Auto-repair makes recurrence self-healing + meanwhile, so this is prevention, not recovery. +- [ ] **P3 — pglite upgrade blocker tracker.** Two couplings make a "routine" pglite bump a + breaking change: (a) pglite ≥0.5 removes the `@electric-sql/pglite/vector` export that + `pglite-engine.ts` imports (verified against npm); (b) the pg_resetwal port + (`src/core/pglite-resetwal.ts`) is coupled to the PG17 pg_control layout + (`PG_CONTROL_VERSION` 1700 — guarded at runtime by `WalResetUnsupportedError`, so a + mismatched bump makes the repair tool refuse every dir rather than corrupt, but it still + means the repair feature silently dies). Any future pglite upgrade wave must revisit BOTH + together and re-derive the ControlFileData offset table for the new PG major. + ## serve --http takes-holders + agent-voice hardening follow-ups (filed v0.42.74.0) Deferred from the #2529/#2477 security-fix wave (plan-eng-review + codex outside diff --git a/VERSION b/VERSION index f2d3e9ef8..b1b737ca8 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.74.0 +0.42.75.0 diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 257a6e7cd..6636ea9b2 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -221,6 +221,66 @@ live in `test/postgres-engine-rls-scope.test.ts`. **Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless. +### Troubleshooting: startup abort (`RuntimeError: Aborted()`) + +**Symptom:** every PGLite-touching command dies at startup with +`PGLite failed to initialize its WASM runtime … Aborted(). Build with +-sASSERTIONS for more info.` — commonly first seen right after a macOS +upgrade. + +**Real root cause:** corrupt WAL/checkpoint state in the data dir after an +unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the +write-ahead log; every subsequent open fails WAL replay inside WASM and +Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM +incompatibility — the same signature reproduces across macOS versions and on +Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun +version bump changes it. + +**Recovery ladder** (top rung first): + +1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort, + backs up `pg_wal/` + `pg_control` into a sibling + `.wal-repair-backup-/` dir, resets the WAL in place + (pg_resetwal semantics — data files preserved; transactions not + checkpointed before the corruption may be lost), and retries once. On + success it prints a loud stderr notice naming the backup and recommending + `gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired + data-dir lock (never after reaping another process's lock), skips for a + cooldown window after a failed attempt + (`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one + backup per corruption episode (newest 3 episodes retained), and restores + the original files if the retry still fails. Kill-switch: + `GBRAIN_PGLITE_WAL_REPAIR=off`. +2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir + (read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset + deliberately. Refuses when another gbrain process holds the brain (a live + `gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`. +3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default + from your config) wipes and re-creates the brain from your brain repo, or + manually: back up `~/.gbrain`, move `brain.pglite` aside, + `gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`. + Required for *catalog* corruption (58P01 / pgvector load failure) — WAL + repair cannot fix that class. +4. **Switch engines.** `gbrain init --supabase`, or native Postgres + + pgvector (recipe below, contributed by @roysaurav): + + ```bash + brew install postgresql@17 + brew services start postgresql@17 + createdb gbrain + cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git + cd pgvector && make && make install + psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;" + # ~/.gbrain/config.json: { "engine": "postgres", + # "database_url": "postgresql://localhost:5432/gbrain" } + gbrain apply-migrations --yes && gbrain doctor + ``` + +`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails +to connect: it diagnoses the dir from disk, names the repair command, reports +retained repair backups, and escalates when repairs keep recurring (that +means the unclean-shutdown genesis is still active — see the ladder's rung 4). + ## JSONB writes: never double-encode (the #2339 trap) Writing a JS value into a `jsonb` column has exactly two correct forms. Get this diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 2033280af..b70e1611f 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -117,7 +117,20 @@ If anything's yellow, `gbrain doctor` names the fix command in the message. Most ### PGLite crashes on macOS 26.x (Tahoe) -PGLite's embedded WASM engine is incompatible with macOS 26.x (Tahoe) on Apple Silicon. If `gbrain init --pglite` crashes during engine initialization, switch to native Homebrew PostgreSQL: +This crash (`RuntimeError: Aborted()` at engine startup, typically first seen +after a macOS upgrade) is **not** a macOS/WASM incompatibility. The upgrade +reboot kills gbrain mid-write and tears the data dir's write-ahead log; every +subsequent open then fails WAL replay. Recovery ladder: + +1. **Auto-repair (default):** just run any gbrain command — gbrain detects the + abort, resets the WAL in place (data preserved; a backup of the pre-repair + state is kept next to the data dir), and continues. Then run `gbrain doctor`. +2. **Manual repair:** `gbrain pglite-repair --dry-run` to diagnose, + `gbrain pglite-repair --yes` to repair in place. +3. **Rebuild:** `gbrain reinit-pglite` (wipes and re-creates the brain from + your brain repo; embedding settings default from your config). +4. **Switch engines** — if you prefer a server database anyway, native + Homebrew PostgreSQL works great and supports multiple concurrent agents: ```bash # Install PostgreSQL + pgvector @@ -144,6 +157,4 @@ gbrain apply-migrations --yes gbrain doctor ``` -All 102 migrations run on first try. Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend. - -> **Note:** This workaround is temporary. When the upstream WASM runtime fix ships (likely via a Bun update), `--pglite` will work on Tahoe again. +Once `gbrain doctor` shows green, the brain works identically to PGLite — same commands, same skills, same data model. The only difference is the storage backend (plus multi-connection support: several agents can share one Postgres brain, which PGLite's single-process lock doesn't allow). diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index db9896f42..e2681379f 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -32,8 +32,12 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/doctor.ts` extension — silent-failure batch (#2250/#2784/#2788): `content_hash_duplicates` (single GROUP BY over `(source_id, content_hash)` with FILTER aggregates — never N² — flagging hash groups that hold BOTH a bare and a path-prefixed slug, the wrong-import-root pattern; warn carries sample pairs + the `pages delete` → `purge-deleted --older-than 0` remediation); `undeclared_db_only_pages` (per source with a local repo: markdown pages with no backing file outside every declared + derive-phase-default db_only prefix — the one check deliberately allowed to stat the repo); `db_only_collector_collision` (configured recipe `output_paths` inside a declared db_only dir — auto-gitignore means sync AND import silently skip the collector's files; same warning fires in sync's `manageGitignore` at config-write time). All warn-level, engine-parity pinned by `test/e2e/doctor-silent-death-parity.test.ts`; units in `test/doctor-silent-death-checks.test.ts`. - `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`). -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error (macOS 26.3 WASM bug #223, points at `gbrain doctor`); the lock is released on failure so the next process can retry cleanly. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `searchTakes`/`searchTakesVector` take full `SearchOpts` and apply the standard source-scope predicates (federated `sourceIds[]` wins over scalar `sourceId`, via the joined page's `source_id`) alongside the holder allow-list — parity SQL in postgres-engine.ts; pinned by `test/e2e/think-source-isolation-pglite.test.ts`. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`, `timeline_entries.event_page_id` — column-only, migration v121 stays the source of truth for its FK + indexes) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. `PGlite.create()` runs inside `preservingProcessExitCode` (#2084): PGLite's Emscripten runtime writes its own status into `process.exitCode` (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close — and assigning `undefined` cannot clear a prior value), which would otherwise silently clobber an errored op's exit 1 back to 0. The wrapper keeps the global tidy for external readers; `db.close()` stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI's exit verdict is immune either way — it lives in the gbrain-owned channel in `cli-force-exit.ts` and never reads `process.exitCode` back. Exports `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `macos-26-3` keeps the #223 link; `unknown` falls through); Pinned by `test/pglite-init-classifier.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers). Engine-path helper dependencies (`retry`, ontology, recency decay) avoid dynamic `import()`; the only lazy dynamic imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass. -- `src/core/pglite-lock.ts` — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at, command, subcommand}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer; informational). A waiting acquirer reaps a holder ONLY when its PID is dead — a LIVE holder is NEVER stolen, regardless of how stale its heartbeat is (#2348). A live `gbrain serve` holder is identified from the parsed `subcommand` and reported immediately with separate CLI-retry and MCP-tool choices; other live holders keep the bounded wait. The heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/CHECKPOINTs, so a working `dream`/`embed` holder can look stale while alive; the old steal-on-stale-heartbeat grace let a second OS process open the same data dir and corrupt the catalog + pgvector extension (58P01 / `internal_load_library` / `type "vector" does not exist`), recoverable only by wipe+restore. A wedged-but-alive or PID-reused holder is never stolen: serve-tagged holders report immediately, while other holders time out with a message naming the PID. Each holder carries an ownership token (`:`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it. In-memory engines take no lock. There is deliberately NO same-process reentrancy or same-PID special case: a second `acquireLock` from the process that already holds the lock waits out the timeout like any other live holder (#1963 was this shape — a command double-connecting a second engine on the same data dir; the fix is to reuse the connected engine at the dispatch layer, never to soften the lock). Pinned by `test/pglite-lock.test.ts`. A corrupted store surfaces a `reinit-pglite` recovery hint via `classifyPgliteInitError`'s `corrupt` verdict in `pglite-engine.ts`. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that classifies the failure and, for the `wasm-abort` verdict on a persistent data dir (torn WAL/checkpoint state after an unclean shutdown — the #223/#1670/#2575 class, historically misdiagnosed as a macOS WASM bug), runs in-place auto-repair via `attemptWalRepairAndRetry` (static import from `pglite-repair.ts`, #3596 engine-live rule; the retry create is `preservingProcessExitCode`-wrapped; success sets the public `walRepairReceipt` field + prints `buildWalRepairNotice` to stderr and returns with the lock held). The seam never throws, so every non-repaired path funnels through the single lock-release-then-throw site; repair refuses when the lock was acquired by reaping (`LockHandle.reaped` → `'possibly-live-writer'`), when disabled (`GBRAIN_PGLITE_WAL_REPAIR=off`), on layout-validation failure, or inside the post-failure cooldown. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `searchTakes`/`searchTakesVector` take full `SearchOpts` and apply the standard source-scope predicates (federated `sourceIds[]` wins over scalar `sourceId`, via the joined page's `source_id`) alongside the holder allow-list — parity SQL in postgres-engine.ts; pinned by `test/e2e/think-source-isolation-pglite.test.ts`. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`, `timeline_entries.event_page_id` — column-only, migration v121 stays the source of truth for its FK + indexes) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. `PGlite.create()` runs inside `preservingProcessExitCode` (#2084): PGLite's Emscripten runtime writes its own status into `process.exitCode` (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close — and assigning `undefined` cannot clear a prior value), which would otherwise silently clobber an errored op's exit 1 back to 0. The wrapper keeps the global tidy for external readers; `db.close()` stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI's exit verdict is immune either way — it lives in the gbrain-owned channel in `cli-force-exit.ts` and never reads `process.exitCode` back. Exports `classifyPgliteInitError(message): 'bunfs' | 'wasm-abort' | 'corrupt' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original, platform?, ctx?)` + `stringifyPgliteInitError(err)` + `buildWalRepairNotice(receipt)` + the `PgliteInitRepairContext` type, routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `corrupt` — 58P01/`internal_load_library`/missing vector type, catalog corruption WAL repair can't fix — stays matched BEFORE the wasm arm and routes to `reinit-pglite`; `wasm-abort` matches the real production shapes `Aborted()`/`RuntimeError`/`unreachable` plus legacy signatures, names the corrupt-WAL root cause + the recovery ladder (`pglite-repair` → rebuild → engine switch) + what auto-repair did per `ctx` incl. the honesty-critical `failed-not-restored` arm, and keeps the #223 link; `unknown` is platform-gated per #2674). `stringifyPgliteInitError` also surfaces message-less Emscripten objects (`ErrnoError (errno N)`) instead of `[object Object]`. Pinned by `test/pglite-init-classifier.test.ts` + `test/pglite-wal-repair.serial.test.ts` + `test/fix-wave-structural.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers). Engine-path helper dependencies (`retry`, ontology, recency decay) avoid dynamic `import()`; the only lazy dynamic imports are `ai/gateway.ts` in `initSchema` and `_upsertChunksOnce`, line-marked because their local catches preserve compiled-default and stored-config fallbacks that eager module evaluation would bypass. +- `src/core/pglite-lock.ts` — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at, command, subcommand}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer; informational). A waiting acquirer reaps a holder ONLY when its PID is dead — a LIVE holder is NEVER stolen, regardless of how stale its heartbeat is (#2348). A live `gbrain serve` holder is identified from the parsed `subcommand` and reported immediately with separate CLI-retry and MCP-tool choices; other live holders keep the bounded wait. The heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/CHECKPOINTs, so a working `dream`/`embed` holder can look stale while alive; the old steal-on-stale-heartbeat grace let a second OS process open the same data dir and corrupt the catalog + pgvector extension (58P01 / `internal_load_library` / `type "vector" does not exist`), recoverable only by wipe+restore. A wedged-but-alive or PID-reused holder is never stolen: serve-tagged holders report immediately, while other holders time out with a message naming the PID. Each holder carries an ownership token (`:`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it. In-memory engines take no lock. There is deliberately NO same-process reentrancy or same-PID special case: a second `acquireLock` from the process that already holds the lock waits out the timeout like any other live holder (#1963 was this shape — a command double-connecting a second engine on the same data dir; the fix is to reuse the connected engine at the dispatch layer, never to soften the lock). `LockHandle.reaped` marks an acquisition that reaped a prior holder's lock (dead-PID reap or corrupt-lock-file removal — the only reaps that exist post-#2348); the WAL auto-repair gate refuses to run surgery on a reaped acquisition since a corrupt lock file cannot prove its holder is dead. A corrupt-lock reap ALSO writes a persisted marker (`.lock-reap.json`, read via exported `msSinceLastReap`) so the NEXT process's clean acquisition is still repair-quarantined for 10 minutes — the in-process flag alone let the reaper's successor run surgery under a possibly-live writer; dead-PID reaps (affirmative ESRCH verdict; EPERM reads as ALIVE) deliberately skip the marker so dead-holder recovery stays one-failed-command-plus-one-re-run. Heartbeat refreshes write via tmp+rename (a torn in-place write could be read mid-flight by a polling acquirer and misclassify a HEALTHY live holder as a corrupt lock). Pinned by `test/pglite-lock.test.ts`. A corrupted store surfaces a `reinit-pglite` recovery hint via `classifyPgliteInitError`'s `corrupt` verdict in `pglite-engine.ts`. +- `src/core/pglite-resetwal.ts` — pg_resetwal for PGLite NodeFS data dirs, in TypeScript (ported from electric-sql/pglite PR #994 by @yestheboxer, Apache-2.0, rejected upstream as "should be a separate tool" — gbrain is that tool). Validates the PG17 pg_control layout fail-closed (`WalResetUnsupportedError` on any unsupported shape — PG_VERSION ≠ 17, control ≠ 8192 bytes, control version ≠ 1700, bad seg/block size), removes stale postmaster.pid + old WAL segments + archive_status/summaries entries, writes a replacement shutdown-checkpoint WAL segment + CRC32C'd pg_control. Both file writes are atomic + durable (tmp cleared then opened `'wx'` so a pre-planted symlink at the predictable tmp name can never redirect the write, + fsync(tmp) + rename + fsync(parent dir)); write order is segment-first/control-last so a mid-write kill leaves a state that still fails startup and the next attempt re-runs (idempotent — a torn pair can never claim success). WAL segment size is capped at 64MB (pglite ships 16MB; the Postgres-general 1GB bound would let a corrupt-but-plausible control field drive a 1GB allocation on the repair path). Exports the shared PG17 layout literals (`PG_CONTROL_FILE_SIZE`, `isWalSegmentName`) consumed by pglite-repair.ts. LAYOUT COUPLING: any pglite bump past PG17 must revisit this file together with the `./vector` export blocker (TODOS.md "pglite upgrade blocker" entry). Pinned by `test/pglite-resetwal.test.ts`. +- `src/core/pglite-repair.ts` — WAL-repair orchestrator wrapping the resetWal port with the safety layers that make it runnable automatically from `connect()`: `validateWalRepairTarget` (read-only, fail-closed; refuses symlinked dataDir/pg_wal/`global`/pg_control — lstat follows INTERMEDIATE symlinks, so `global/` itself must be checked or surgery would write pg_control through it into a foreign dir; tolerates the in-dir `.gbrain-lock`), rename-based backup (the ENTIRE `pg_wal/` dir + postmaster.pid renamed into a sibling `.wal-repair-backup-/`, only the 8KB pg_control copied — zero transient disk cost), `restoreWalBackup` (overwrite order: control first via atomic tmp+rename, then a pg_wal dir swap with the reset dir set ASIDE inside the backup — nothing is ever deleted during restore; mtime guard refuses when a foreign segment is newer than the backup; a missing/empty backup NEVER reports `restored:true`), `WalRepairError` (thrown when resetWal fails AFTER the backup — carries the receipt + the best-effort restore's REAL result so the seam's `restored` flag and the `failed-restored`/`failed-not-restored` message arms never lie), a cooldown sidecar `.wal-repair-attempt.json` (skip `'recently-failed'` inside `GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600 — bounds the autopilot/supervisor reconnect loops) with episode-scoped backups (attempts within one corruption episode REUSE the episode's first backup — the pre-damage forensic state, honored only when the sidecar's path is a real non-symlink `.wal-repair-backup-*` sibling since the sidecar is user-writable JSON; retention keeps the newest 3 episodes, never pruning the open episode's), and `attemptWalRepairAndRetry` — the engine seam that NEVER throws (gates: kill-switch → reaped-lock `'possibly-live-writer'` → 10-minute reap-marker quarantine (`msSinceLastReap`, cross-process) → validation → cooldown; then repair → retry create once → restore-and-record on failure; prints a repair-start stderr line so a timeout-killed attempt is self-explaining). `inspectPgliteDataDir` is the read-only diagnosis for `gbrain doctor` + `pglite-repair --dry-run`. Imports runtime values only from `pglite-lock.ts`/`pglite-resetwal.ts`/node:fs — never from `pglite-engine.ts` (no cycle; the engine statically imports THIS file per the #3596 engine-live rule). Pinned by `test/pglite-repair.test.ts` + `test/pglite-wal-repair.serial.test.ts`. +- `src/commands/pglite-repair.ts` — `gbrain pglite-repair`: the manual surface for WAL repair (`--dry-run | --yes | --json | --path `; CLI_ONLY + SELF_HELP; returns an exit code via `setCliExitVerdict`, never `process.exit`). Never connects an engine — works when the DB won't open and when auto-repair is disabled. `--dry-run` is strictly read-only. The real run validates BEFORE locking (`acquireLock` mkdirs the data dir — a typo'd `--path` must not create directories), refuses a live lock holder (pre-lock diagnosis names the PID; a live `gbrain serve` is called out), refuses a reaped acquisition (`refused_reaped_lock` — no `--force` by design: force-removing `.gbrain-lock` would reopen the #2348 concurrent-writer hole), re-validates under the lock, repairs with episode-backup reuse, and records the attempt in the sidecar. Pinned by `test/pglite-repair-command.serial.test.ts`. +- `src/commands/doctor.ts` `pglite_data_dir` check — fs-only check that runs when a PGLite brain FAILS to connect (`!fastMode && !engine && config.engine === 'pglite'`, placed after `orphan_clones`, before the DB-checks gate): `computePgliteDataDirCheck(dataDir, diagnosis)` (exported pure fn, `computeWorkerOomLoopCheck` convention) maps the `inspectPgliteDataDir` verdict to a Check — corruption-likely/looks-healthy-but-unopenable/unsupported-layout → `fail` naming `gbrain pglite-repair --dry-run`/`--yes` or the rebuild path, live-lock/missing-dir → `warn`; all `remediation_status: 'human_only'` (Minion remediation needs the DB that is down). Escalates when ≥2 repair attempts failed inside 7 days (unclean-shutdown genesis still active → engine-switch pointer) and reports retained backup-dir inventory (orphan_clones disk-visibility class). Registered in `doctor-categories.ts` OPS_CHECK_NAMES. Pinned by `test/doctor-pglite-datadir.test.ts`. - `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`, `timeline_entries.event_page_id`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite. - `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger. diff --git a/llms-full.txt b/llms-full.txt index b89b90a30..13a0383ed 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1831,7 +1831,7 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h ## Troubleshooting -**`gbrain init --pglite` crashes on macOS 26.x (Tahoe)?** PGLite's embedded WASM engine is incompatible with macOS 26.x on Apple Silicon. The fix is to use native Homebrew PostgreSQL + pgvector instead. Full step-by-step setup in [`docs/INSTALL.md` — Troubleshooting: PGLite crashes on macOS 26.x](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe). +**PGLite crashes at startup with `RuntimeError: Aborted()` (often right after a macOS upgrade)?** Not a macOS incompatibility — the OS-upgrade reboot killed gbrain mid-write and tore the data dir's WAL. gbrain now repairs this automatically on the next command (data preserved, backup kept); if auto-repair is disabled or skipped, run `gbrain pglite-repair --dry-run` to diagnose and `gbrain pglite-repair --yes` to repair in place. Full recovery ladder (repair → rebuild → engine switch) in [`docs/ENGINES.md` — Troubleshooting: startup abort](docs/ENGINES.md#troubleshooting-startup-abort-runtimeerror-aborted) and [`docs/INSTALL.md`](docs/INSTALL.md#pglite-crashes-on-macos-26x-tahoe). **`gbrain import` fails with `expected N dimensions, not M`?** Run `gbrain doctor`. It will print the exact `gbrain config set ...` or `gbrain retrieval-upgrade` command to repair the mismatch. You should not need to delete `~/.gbrain`. Fresh `gbrain init --pglite` auto-detects your embedding provider from API keys in your environment: set `OPENAI_API_KEY` (or `ZEROENTROPY_API_KEY` / `VOYAGE_API_KEY`) before running init, or pass `--embedding-model :` explicitly. With multiple keys set, init fires an interactive picker. In non-TTY contexts (CI, Docker) with no keys, init exits 1 with a paste-ready setup hint; pass `--no-embedding` to defer setup until runtime. See [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for the full provider matrix and [`docs/operations/headless-install.md`](docs/operations/headless-install.md) for Docker/CI sequencing. @@ -2216,6 +2216,66 @@ live in `test/postgres-engine-rls-scope.test.ts`. **Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless. +### Troubleshooting: startup abort (`RuntimeError: Aborted()`) + +**Symptom:** every PGLite-touching command dies at startup with +`PGLite failed to initialize its WASM runtime … Aborted(). Build with +-sASSERTIONS for more info.` — commonly first seen right after a macOS +upgrade. + +**Real root cause:** corrupt WAL/checkpoint state in the data dir after an +unclean shutdown (the OS-upgrade reboot kills gbrain mid-write and tears the +write-ahead log; every subsequent open fails WAL replay inside WASM and +Emscripten surfaces only the opaque abort). It is **not** a macOS/WASM +incompatibility — the same signature reproduces across macOS versions and on +Linux, and rebuilding the data dir on the same OS fixes it. No pglite or Bun +version bump changes it. + +**Recovery ladder** (top rung first): + +1. **Auto-repair (default).** `PGLiteEngine.connect()` detects the abort, + backs up `pg_wal/` + `pg_control` into a sibling + `.wal-repair-backup-/` dir, resets the WAL in place + (pg_resetwal semantics — data files preserved; transactions not + checkpointed before the corruption may be lost), and retries once. On + success it prints a loud stderr notice naming the backup and recommending + `gbrain doctor`. Safety bounds: repair only runs under a cleanly-acquired + data-dir lock (never after reaping another process's lock), skips for a + cooldown window after a failed attempt + (`GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS`, default 3600), reuses one + backup per corruption episode (newest 3 episodes retained), and restores + the original files if the retry still fails. Kill-switch: + `GBRAIN_PGLITE_WAL_REPAIR=off`. +2. **Manual repair.** `gbrain pglite-repair --dry-run` diagnoses the data dir + (read-only); `gbrain pglite-repair --yes` runs the same in-place WAL reset + deliberately. Refuses when another gbrain process holds the brain (a live + `gbrain serve` is named explicitly) and never force-removes `.gbrain-lock`. +3. **Rebuild.** `gbrain reinit-pglite` (embedding model/dimensions default + from your config) wipes and re-creates the brain from your brain repo, or + manually: back up `~/.gbrain`, move `brain.pglite` aside, + `gbrain init --pglite`, re-add sources, `gbrain sync`, `gbrain embed`. + Required for *catalog* corruption (58P01 / pgvector load failure) — WAL + repair cannot fix that class. +4. **Switch engines.** `gbrain init --supabase`, or native Postgres + + pgvector (recipe below, contributed by @roysaurav): + + ```bash + brew install postgresql@17 + brew services start postgresql@17 + createdb gbrain + cd /tmp && git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git + cd pgvector && make && make install + psql gbrain -c "CREATE EXTENSION IF NOT EXISTS vector;" + # ~/.gbrain/config.json: { "engine": "postgres", + # "database_url": "postgresql://localhost:5432/gbrain" } + gbrain apply-migrations --yes && gbrain doctor + ``` + +`gbrain doctor` runs a `pglite_data_dir` check whenever a PGLite brain fails +to connect: it diagnoses the dir from disk, names the repair command, reports +retained repair backups, and escalates when repairs keep recurring (that +means the unclean-shutdown genesis is still active — see the ladder's rung 4). + ## JSONB writes: never double-encode (the #2339 trap) Writing a JS value into a `jsonb` column has exactly two correct forms. Get this diff --git a/package.json b/package.json index 70a55c655..b034bd26a 100644 --- a/package.json +++ b/package.json @@ -148,7 +148,7 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.74.0", + "version": "0.42.75.0", "overrides": { "@hono/node-server": "^2.0.5", "fast-uri": "^3.1.5", diff --git a/src/cli.ts b/src/cli.ts index 990a661d1..ec60567c9 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -55,7 +55,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown { } // CLI-only commands that bypass the operation layer -export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']); +export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'pglite-repair', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector', 'pages', 'bench', 'backfill']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -101,6 +101,9 @@ const CLI_ONLY_SELF_HELP = new Set([ // the generic short-circuit so the destructive-action warning text // reaches the user. 'reinit-pglite', + // WAL-repair wave: pglite-repair ships its own --help with the + // dry-run/repair semantics + the un-checkpointed-tail caveat. + 'pglite-repair', // v0.40.6.0 Schema Cathedral v3 — `gbrain schema --help` should hit // schema.ts printHelp() with the full 22+ verb taxonomy, not the // generic short-circuit's one-line stub. @@ -1315,6 +1318,13 @@ async function handleCliOnly(command: string, args: string[]) { await runReinitPglite(args); return; } + // WAL-repair wave (#223/#1670/#2575): in-place torn-WAL recovery. Never + // connects an engine — the whole point is that the DB won't open. + if (command === 'pglite-repair') { + const { runPgliteRepair } = await import('./commands/pglite-repair.ts'); + setCliExitVerdict(await runPgliteRepair(args)); + return; + } if (command === 'auth') { const { runAuth } = await import('./commands/auth.ts'); await runAuth(args); diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index e7c446579..4cfe2ad43 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -4651,6 +4651,96 @@ export async function checkCycleFreshness( * - `progress` reporter writes to stderr (heartbeats per check) * - `engine.executeRaw` / handler-leaf calls (the actual probe work) */ +// ≥2 failed repair attempts inside 7 days = the corruption keeps regenerating. +const REPAIR_RECURRENCE_WINDOW_MS = 7 * 24 * 3600 * 1000; +const REPAIR_RECURRENCE_THRESHOLD = 2; + +/** + * WAL-repair wave (#223/#1670/#2575): when the DB failed to connect on a + * PGLite brain, diagnose the data dir from the FILESYSTEM (the connect error + * itself was swallowed by doctor's fs-only fallback — this check re-derives + * the state from disk). Pure: interprets an `inspectPgliteDataDir` diagnosis + * into a Check; exported so `test/doctor-pglite-datadir.test.ts` drives it + * directly (same convention as computeWorkerOomLoopCheck). Returns a Check + * always — the call site only runs it when connect already failed, so even a + * healthy-looking dir warrants a pointer at the repair tooling. + * + * Recurrence escalation (eng-review 2A): repeated failed repair attempts on + * record mean the corruption keeps regenerating (unclean-shutdown genesis) — + * escalate to the engine-switch ladder instead of letting the brain silently + * lose a WAL tail per cycle. Backup-dir inventory rides along (same + * disk-visibility class as orphan_clones). + */ +export function computePgliteDataDirCheck( + dataDir: string, + diagnosis: import('../core/pglite-repair.ts').PgliteDirDiagnosis, +): Check { + const backupNote = diagnosis.backupDirs.length > 0 + ? ` ${diagnosis.backupDirs.length} repair backup dir(s) on disk (newest: ${diagnosis.backupDirs[0]}) — delete old ones to reclaim space once the brain is healthy.` + : ''; + // Count BOTH outcomes (adversarial review F12): a >1h-period crash loop where + // each repair "succeeds" discards a WAL tail per cycle with zero FAILED + // attempts on record — escalation must still fire. + const recentAttempts = diagnosis.recentAttempts.filter( + (a) => Date.now() - a.ts < REPAIR_RECURRENCE_WINDOW_MS, + ).length; + const recurrence = recentAttempts >= REPAIR_RECURRENCE_THRESHOLD + ? ` Auto-repair has run ${recentAttempts}x this week — the corruption keeps regenerating (likely an unclean-shutdown loop). Consider switching engines (docs/ENGINES.md: \`gbrain init --supabase\` or native Postgres).` + : ''; + + switch (diagnosis.verdict) { + case 'locked': + return { + name: 'pglite_data_dir', + status: 'warn', + message: + `Could not connect, and the PGLite data-dir lock is held by live PID ${diagnosis.lockHolderPid} — ` + + `another gbrain process (often \`gbrain serve\`) has the brain open. Stop it and re-run.${backupNote}`, + remediation_status: 'human_only', + }; + case 'missing': + return { + name: 'pglite_data_dir', + status: 'warn', + message: `No PGLite data dir at ${dataDir}. Run \`gbrain init --pglite\` to create one.`, + remediation_status: 'human_only', + }; + case 'unsupported-layout': + return { + name: 'pglite_data_dir', + status: 'fail', + message: + `PGLite data dir at ${dataDir} is not repairable in place (${diagnosis.detail}). ` + + `Rebuild from your brain repo: \`gbrain reinit-pglite\` (or back up ~/.gbrain, move the dir aside, ` + + `\`gbrain init --pglite\`, re-add sources + sync + embed).${backupNote}${recurrence}`, + remediation_status: 'human_only', + }; + case 'wal-corruption-likely': + return { + name: 'pglite_data_dir', + status: 'fail', + message: + `PGLite failed to open and the data dir shows unclean-shutdown state (${diagnosis.detail}). ` + + `This is the torn-WAL class behind issue #223 — repairable in place, data preserved: ` + + `\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair.${backupNote}${recurrence}`, + remediation_status: 'human_only', + }; + case 'looks-healthy': + default: + return { + name: 'pglite_data_dir', + status: 'fail', + message: + `PGLite failed to open but the data dir layout validates (${diagnosis.detail}). ` + + `IF the connect error mentions \`Aborted()\` this is likely torn WAL state — ` + + `\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair in place ` + + `(repair discards the un-checkpointed WAL tail — don't run it for lock-contention or ` + + `catalog-corruption errors; 58P01/pgvector load failures need \`gbrain reinit-pglite\` instead).${backupNote}${recurrence}`, + remediation_status: 'human_only', + }; + } +} + /** * issue #1685 (GAP A) — the single authoritative "worker is OOM-looping" signal. * @@ -5947,6 +6037,27 @@ export async function buildChecks( // Filesystem read failure is non-fatal. } + // 3d. PGLite data-dir diagnosis (WAL-repair wave). Only meaningful when the + // connect already FAILED on a PGLite brain (engine === null): the connect + // error was swallowed by the fs-only fallback, so this check re-derives the + // dir state from disk and names the repair ladder. Skipped under --fast + // (connect wasn't attempted, so "engine === null" proves nothing there). + if (!fastMode && !engine) { + try { + const cfg = loadConfig(); + if (cfg?.engine === 'pglite') { + const { inspectPgliteDataDir } = await import('../core/pglite-repair.ts'); + const { resolve } = await import('node:path'); + // Absolutize: a RELATIVE database_path would make the sidecar/backup + // lookups resolve against doctor's cwd instead of the engine's. + const pgliteDataDir = resolve(cfg.database_path || gbrainPath('brain.pglite')); + checks.push(computePgliteDataDirCheck(pgliteDataDir, inspectPgliteDataDir(pgliteDataDir))); + } + } catch { + // Best-effort: an unreadable config or fs failure must not stop doctor. + } + } + // --- DB checks (skip if --fast or no engine) --- if (fastMode || !engine) { diff --git a/src/commands/pglite-repair.ts b/src/commands/pglite-repair.ts new file mode 100644 index 000000000..7a7d4a954 --- /dev/null +++ b/src/commands/pglite-repair.ts @@ -0,0 +1,340 @@ +/** + * `gbrain pglite-repair` — diagnose and repair a torn-WAL PGLite data dir + * in place (#223 / #1670 / #2575 recovery, the manual surface for the + * auto-repair that `PGLiteEngine.connect()` runs on `wasm-abort` failures). + * + * Never connects an engine (the whole point is that the DB won't open), so it + * works when auto-repair is disabled (GBRAIN_PGLITE_WAL_REPAIR=off) or was + * skipped. `--dry-run` is strictly read-only. The real run: + * + * validate (BEFORE locking — `acquireLock` mkdirs the data dir, and a + * typo'd --path must not create directories) → TTY confirm unless --yes → + * acquireLock (refuses a reaped acquisition: a corrupt-lock reap cannot + * prove the holder is dead, and WAL surgery under a possibly-live writer is + * never correct; a live `gbrain serve` holder fast-fails via + * LiveServeLockError) → re-validate under the lock → repair → receipt. + * + * There is deliberately NO --force: force-removing `.gbrain-lock` while the + * holder is alive would reopen exactly the concurrent-writer corruption hole + * #2348 closed. For catalog corruption (the `corrupt` classifier verdict) WAL + * repair does not help — `gbrain reinit-pglite` is the rebuild path. + */ + +import { createInterface } from 'readline'; +import { loadConfig, gbrainPath } from '../core/config.ts'; +import { acquireLock, releaseLock, LiveServeLockError, msSinceLastReap } from '../core/pglite-lock.ts'; +import { + inspectPgliteDataDir, + listRepairBackups, + readRepairSidecar, + recordRepairAttempt, + repairPgliteWal, + validateWalRepairTarget, + WalRepairError, +} from '../core/pglite-repair.ts'; + +interface RepairCmdOpts { + dryRun: boolean; + yes: boolean; + jsonOutput: boolean; + customPath: string | null; + help: boolean; +} + +class UnknownFlagError extends Error {} + +function parseArgs(args: string[]): RepairCmdOpts { + const opts: RepairCmdOpts = { dryRun: false, yes: false, jsonOutput: false, customPath: null, help: false }; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--dry-run') opts.dryRun = true; + else if (a === '--yes' || a === '-y') opts.yes = true; + else if (a === '--json') opts.jsonOutput = true; + else if (a === '--path') { + const val = args[++i]; + // F1: a valueless --path (typo / shell mangling) must NOT fall through + // to the configured default brain and run surgery on the wrong dir. + if (val === undefined || val.startsWith('-')) throw new UnknownFlagError('--path requires a directory argument'); + opts.customPath = val; + } + else if (a === '--help' || a === '-h') opts.help = true; + // Reject unknown args on a DESTRUCTIVE command (codex): silently ignoring + // a typo like `--dry-rnu` would run a real WAL reset instead of a dry run. + else throw new UnknownFlagError(`unknown argument: ${a}`); + } + return opts; +} + +function printHelp(): void { + console.log(`gbrain pglite-repair — repair a torn-WAL PGLite data dir in place + +The default gbrain engine (PGLite) can fail to open after an unclean shutdown +(commonly a macOS-upgrade reboot) with "RuntimeError: Aborted()". The cause is +torn WAL/checkpoint state on disk, not a macOS WASM bug. This command resets +the WAL in place (pg_resetwal semantics): data files are preserved; +transactions not checkpointed before the corruption may be lost. The +pre-repair pg_wal + pg_control are kept in a sibling backup directory. + +Usage: + gbrain pglite-repair --dry-run [--json] [--path ] diagnose only + gbrain pglite-repair [--yes] [--json] [--path ] repair (confirm on TTY) + +Flags: + --dry-run Read-only diagnosis of the data dir. Mutates nothing. + --yes, -y Skip the confirmation prompt (required in non-TTY runs). + --json Machine-readable output on stdout. + --path Repair a specific data dir (default: the configured brain). + +Notes: + Auto-repair runs on ordinary commands by default; disable it with + GBRAIN_PGLITE_WAL_REPAIR=off and use this command deliberately. + Catalog corruption (58P01 / pgvector load failure) is NOT repairable in + place — use \`gbrain reinit-pglite\` for that class.`); +} + +function emitError(jsonOutput: boolean, code: string, message: string): void { + if (jsonOutput) { + console.log(JSON.stringify({ status: 'error', code, message })); + } else { + console.error(`Error (${code}): ${message}`); + } +} + +async function promptYesNo(question: string): Promise { + // Prompt on stderr: stdout stays clean for --json payloads. + const rl = createInterface({ input: process.stdin, output: process.stderr }); + return new Promise((resolve) => { + rl.question(`${question} [y/N] `, (answer) => { + rl.close(); + resolve(/^y(es)?$/i.test(answer.trim())); + }); + }); +} + +export async function runPgliteRepair(args: string[]): Promise { + let opts: RepairCmdOpts; + try { + opts = parseArgs(args); + } catch (err) { + if (err instanceof UnknownFlagError) { + const jsonOut = args.includes('--json'); + emitError(jsonOut, 'unknown_flag', `${err.message}. Run \`gbrain pglite-repair --help\`.`); + return 2; + } + throw err; + } + if (opts.help) { + printHelp(); + return 0; + } + + // Resolve dir: --path > config > default brain path. With an explicit + // --path we skip the engine check (repairing an arbitrary dir is the point). + let dataDir = opts.customPath; + if (!dataDir) { + const cfg = loadConfig(); + if (cfg?.engine !== 'pglite') { + emitError( + opts.jsonOutput, + 'not_pglite', + `gbrain pglite-repair is for PGLite brains (current engine: ${cfg?.engine || 'none'}). ` + + 'Pass --path to repair a specific data dir.', + ); + return 1; + } + dataDir = cfg.database_path || gbrainPath('brain.pglite'); + } + + // Read-only diagnosis first — BEFORE any lock (acquireLock mkdirs the data + // dir; a typo'd --path must produce a clean refusal with zero side effects). + const validation = validateWalRepairTarget(dataDir); + const diagnosis = inspectPgliteDataDir(dataDir); + + if (opts.dryRun) { + if (opts.jsonOutput) { + console.log(JSON.stringify({ + status: 'ok', + action: 'dry-run', + data_dir: dataDir, + validation, + diagnosis, + })); + } else { + console.log(`PGLite data dir: ${dataDir}`); + console.log(` Verdict: ${diagnosis.verdict} — ${diagnosis.detail}`); + console.log(` PG_VERSION: ${diagnosis.pgVersion ?? '(unreadable)'} pg_control: ${diagnosis.pgControlOk ? 'ok (8192 bytes)' : 'BAD'}`); + console.log(` WAL segments: ${diagnosis.walSegments.length} stale postmaster.pid: ${diagnosis.postmasterPid ? 'YES' : 'no'}`); + console.log(` Lock: ${diagnosis.lockHeld ? `HELD by live PID ${diagnosis.lockHolderPid}` : 'free'}`); + if (diagnosis.recentAttempts.length > 0) { + console.log(` Repair attempts on record: ${diagnosis.recentAttempts.map((a) => `${a.outcome}@${new Date(a.ts).toISOString()}`).join(', ')}`); + } + if (diagnosis.backupDirs.length > 0) { + console.log(` Repair backups on disk: ${diagnosis.backupDirs.join(', ')}`); + } + if (!validation.ok) { + console.log(` Repairable: NO — ${validation.detail}`); + } else if (diagnosis.verdict === 'looks-healthy') { + console.log(' Repairable: yes — but no unclean-shutdown markers found; repair is likely'); + console.log(' unnecessary. Run `gbrain pglite-repair --yes` ONLY if PGLite fails to open'); + console.log(' with `RuntimeError: Aborted()` (repair discards the un-checkpointed WAL tail).'); + } else { + console.log(' Repairable: yes — run `gbrain pglite-repair --yes` to reset the WAL in place.'); + } + } + return 0; + } + + if (!validation.ok) { + emitError(opts.jsonOutput, `refused_${validation.reason}`, validation.detail); + return 1; + } + if (diagnosis.lockHeld) { + emitError( + opts.jsonOutput, + 'refused_locked', + `another gbrain process (PID ${diagnosis.lockHolderPid}) is using this brain — stop it, then re-run.`, + ); + return 1; + } + const sinceReap = msSinceLastReap(dataDir); + const REAP_QUARANTINE_MS = 10 * 60 * 1000; + if (sinceReap !== null && sinceReap >= 0 && sinceReap < REAP_QUARANTINE_MS) { + emitError( + opts.jsonOutput, + 'refused_reap_quarantine', + `a lock on this brain was reaped ${Math.round(sinceReap / 1000)}s ago from a holder whose ` + + 'liveness could not be verified — that process may still be writing. Confirm no gbrain ' + + `process is running (\`pgrep -af gbrain\`), wait ${Math.ceil((REAP_QUARANTINE_MS - sinceReap) / 60000)} more minute(s), then re-run.`, + ); + return 1; + } + + if (!opts.yes) { + if (!process.stdin.isTTY) { + emitError(opts.jsonOutput, 'no_tty_no_yes', 'Non-TTY environment requires --yes to confirm the WAL reset.'); + return 1; + } + console.error(`About to reset the WAL of ${dataDir} in place.`); + console.error('Data files are preserved; un-checkpointed transactions may be lost.'); + console.error('The current pg_wal + pg_control are kept in a sibling backup directory.'); + const confirmed = await promptYesNo('Repair now?'); + if (!confirmed) { + if (opts.jsonOutput) { + console.log(JSON.stringify({ status: 'aborted', reason: 'user_declined' })); + } else { + console.log('Aborted. Data dir untouched.'); + } + return 0; + } + } + + // Short timeout: the diagnosis said the lock is free; if we still can't get + // it quickly, someone raced us — refuse rather than queue behind them. + let lock; + try { + lock = await acquireLock(dataDir, { timeoutMs: 5_000 }); + } catch (err) { + if (err instanceof LiveServeLockError) { + emitError( + opts.jsonOutput, + 'refused_live_serve', + `a live \`gbrain serve\` (MCP) process holds this brain — stop \`gbrain serve\` first, then re-run. ${String((err as Error).message)}`, + ); + return 1; + } + emitError(opts.jsonOutput, 'refused_lock_timeout', String((err as Error)?.message ?? err)); + return 1; + } + try { + if (!lock.acquired) { + emitError(opts.jsonOutput, 'refused_locked', 'could not acquire the PGLite data-dir lock — another gbrain process is using this brain.'); + return 1; + } + if (lock.reaped) { + // A reaped acquisition (dead-PID or corrupt-lock-file reap) cannot prove + // the prior holder is gone. WAL surgery under a possibly-live writer is + // never correct — no --force by design. + emitError( + opts.jsonOutput, + 'refused_reaped_lock', + 'the data-dir lock was acquired by reaping a prior holder’s lock — ' + + 'another gbrain process may still be using this brain. Confirm no gbrain ' + + 'process is running, then re-run (a cleanly-acquired lock enables repair).', + ); + return 1; + } + + // Cheap re-validate under the lock (TOCTOU window between diagnosis and + // lock acquisition). + const revalidation = validateWalRepairTarget(dataDir); + if (!revalidation.ok) { + emitError(opts.jsonOutput, `refused_${revalidation.reason}`, revalidation.detail); + return 1; + } + + process.stderr.write(`Repairing WAL of ${dataDir} in place…\n`); + const sidecar = readRepairSidecar(dataDir); + // F4: only reuse a FRESH (<24h) episode backup — a stale pin may predate + // real data (same bound as the auto seam's episodeFresh). + const episodeFresh = + sidecar.episodeStartedAt !== null && + Date.now() - sidecar.episodeStartedAt >= 0 && + Date.now() - sidecar.episodeStartedAt < 24 * 3600 * 1000; + let receipt; + try { + receipt = await repairPgliteWal(dataDir, { + reuseBackupPath: episodeFresh ? sidecar.episodeBackupPath ?? undefined : undefined, + }); + } catch (err) { + if (err instanceof WalRepairError) { + // Reset failed after the backup was taken — report the restore's REAL + // outcome so the user knows whether the dir is back or in reset state. + recordRepairAttempt(dataDir, 'failed', err.receipt.backupPath); + emitError( + opts.jsonOutput, + 'repair_failed', + err.message + (err.restore.restored + ? ` (data dir restored to its pre-repair state; backup kept at ${err.receipt.backupPath})` + : ` (RESTORE ALSO FAILED — the dir is in a reset state; your pre-repair files are intact at ${err.receipt.backupPath}: ${err.restore.detail ?? ''})`), + ); + return 1; + } + recordRepairAttempt(dataDir, 'failed', sidecar.episodeBackupPath); + emitError(opts.jsonOutput, 'repair_failed', String((err as Error)?.message ?? err)); + return 1; + } + // "repaired" here means the reset completed; the next connect PROVES it. + // Record a FAILED attempt (not repaired-with-closeEpisode:false, which + // leaves the episode null when none was open — codex): this opens/keeps an + // episode pinned to this backup so a later healthy connect closes it and + // prunes, and repeated manual runs during one incident reuse the pinned + // backup instead of deleting the pre-damage forensic copy. + recordRepairAttempt(dataDir, 'failed', receipt.backupPath); + + if (opts.jsonOutput) { + console.log(JSON.stringify({ + status: 'ok', + action: 'repaired', + data_dir: receipt.dataDir, + backup_path: receipt.backupPath, + backed_up: receipt.backedUpFiles, + reused_episode_backup: receipt.reusedEpisodeBackup, + reset_segment: receipt.resetSegment, + timeline_id: receipt.timelineId, + wal_seg_size: receipt.walSegSize, + repaired_at: receipt.repairedAt, + backups_on_disk: listRepairBackups(dataDir), + })); + } else { + console.log('WAL reset complete.'); + console.log(` Data dir: ${receipt.dataDir}`); + console.log(` Backup: ${receipt.backupPath}${receipt.reusedEpisodeBackup ? ' (reused this episode’s existing backup)' : ''}`); + console.log(` Reset segment: ${receipt.resetSegment} (timeline ${receipt.timelineId}, ${receipt.walSegSize / (1024 * 1024)}MB segments)`); + console.log(' Data files were preserved; un-checkpointed transactions may be lost.'); + console.log(' Next: run any gbrain command to reopen the brain, then `gbrain doctor`.'); + } + return 0; + } finally { + await releaseLock(lock); + } +} diff --git a/src/commands/reinit-pglite.ts b/src/commands/reinit-pglite.ts index 8727dc6f7..ec214d8a5 100644 --- a/src/commands/reinit-pglite.ts +++ b/src/commands/reinit-pglite.ts @@ -18,7 +18,7 @@ * output via `--json` for scripted callers. */ -import { existsSync, renameSync, statSync } from 'fs'; +import { existsSync, renameSync, statSync, rmSync } from 'fs'; import { dirname } from 'path'; import { loadConfig, loadConfigFileOnly, gbrainPath } from '../core/config.ts'; @@ -122,6 +122,13 @@ export async function runReinitPglite(args: string[]): Promise { try { renameSync(dbPath, bakPath); + // WAL-repair state travels with the OLD brain (red-team: a fresh brain at + // the same path must not inherit the old brain's open repair episode, + // cooldown, or reap quarantine — a stale episodeBackupPath would be reused + // over the NEW brain's WAL). + for (const sibling of [`${dbPath}.wal-repair-attempt.json`, `${dbPath}.lock-reap.json`]) { + try { rmSync(sibling, { force: true }); } catch { /* best-effort */ } + } } catch (e: unknown) { fail( opts.jsonOutput, @@ -210,21 +217,59 @@ function parseArgs(args: string[]): ReinitOpts { const dimsIdx = args.indexOf('--embedding-dimensions'); const pathIdx = args.indexOf('--path'); - if (modelIdx < 0 || modelIdx === args.length - 1) { - fail(jsonOutput, 'missing_model', '--embedding-model is required.'); + // Default omitted flags from the config FILE. Deliberately + // `loadConfigFileOnly()`, NOT `loadConfig()`: loadConfig merges the + // GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS env overrides, + // and a transient outage-shell export must not silently change the + // rebuild target. Precedence: explicit flag > config-file value > + // hard-fail (the original missing_model/missing_dims errors). + const fileCfg = (modelIdx < 0 || dimsIdx < 0) ? loadConfigFileOnly() : null; + + let embeddingModel: string; + if (modelIdx >= 0) { + if (modelIdx === args.length - 1) { + fail(jsonOutput, 'missing_model', '--embedding-model is required.'); + } + embeddingModel = args[modelIdx + 1]; + } else if (fileCfg?.embedding_model) { + embeddingModel = fileCfg.embedding_model; + console.error(`--embedding-model defaulted from config: ${embeddingModel}`); + } else { + fail( + jsonOutput, + 'missing_model', + '--embedding-model is required (no embedding_model in the config file to default from).', + ); } - if (dimsIdx < 0 || dimsIdx === args.length - 1) { - fail(jsonOutput, 'missing_dims', '--embedding-dimensions is required.'); + + let dimsStr: string; + let dimsFromConfig = false; + if (dimsIdx >= 0) { + if (dimsIdx === args.length - 1) { + fail(jsonOutput, 'missing_dims', '--embedding-dimensions is required.'); + } + dimsStr = args[dimsIdx + 1]; + } else if (fileCfg?.embedding_dimensions !== undefined && fileCfg?.embedding_dimensions !== null) { + dimsStr = String(fileCfg.embedding_dimensions); + dimsFromConfig = true; + } else { + fail( + jsonOutput, + 'missing_dims', + '--embedding-dimensions is required (no embedding_dimensions in the config file to default from).', + ); } - const dimsStr = args[dimsIdx + 1]; const dims = parseInt(dimsStr, 10); if (!Number.isInteger(dims) || dims <= 0) { fail(jsonOutput, 'invalid_dims', `--embedding-dimensions must be a positive integer (got: ${dimsStr}).`); } + if (dimsFromConfig) { + console.error(`--embedding-dimensions defaulted from config: ${dims}`); + } return { - embeddingModel: args[modelIdx + 1], + embeddingModel, embeddingDimensions: dims, yes, jsonOutput, @@ -240,9 +285,16 @@ Wipe the PGLite brain and re-init with new embedding model/dimensions. This is the canonical path for switching embedding providers on PGLite because pgvector (WASM) cannot ALTER vector column types in place. -Required: +Embedding target (each defaults from the config file when omitted): --embedding-model New embedding model (e.g. openai:text-embedding-3-large). + Defaults to embedding_model in ~/.gbrain/config.json. --embedding-dimensions New dimension count (e.g. 1280, 1536, 2048). + Defaults to embedding_dimensions in ~/.gbrain/config.json. + +Defaults read the config FILE only; GBRAIN_EMBEDDING_MODEL / +GBRAIN_EMBEDDING_DIMENSIONS env overrides are deliberately ignored so a +transient shell export cannot change the rebuild target. If neither the +flag nor the config file provides a value, the command fails. Optional: --path Active brain path (default: ~/.gbrain/brain.pglite). @@ -258,6 +310,9 @@ Examples: gbrain reinit-pglite --embedding-model openai:text-embedding-3-large \\ --embedding-dimensions 1536 --no-sync + # Rebuild with the model/dimensions already in the config file: + gbrain reinit-pglite --yes + The old brain is preserved as \`.bak\`. To roll back, mv it back. See also: diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts index e041b1780..fb34aef0d 100644 --- a/src/core/doctor-categories.ts +++ b/src/core/doctor-categories.ts @@ -153,6 +153,7 @@ export const OPS_CHECK_NAMES: ReadonlySet = new Set([ 'oauth_confidential_client_health', 'orphan_clones', 'pgbouncer_prepare', + 'pglite_data_dir', 'pgvector', 'pool_budget', 'progressive_batch_audit_health', diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index db7d2df8a..0727689cc 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -46,6 +46,9 @@ import { DELETE_BATCH_SIZE } from './engine-constants.ts'; import { SOURCE_CONFIG_OBJECT_SQL } from './source-config-sql.ts'; import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts'; import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts'; +// Engine-live path (#3596): static import, never a lazy `import()` in the +// connect() catch. No cycle: pglite-repair.ts imports nothing from this file. +import { attemptWalRepairAndRetry, closeRepairEpisodeIfOpen, type WalRepairReceipt } from './pglite-repair.ts'; import { getFtsLanguage } from './fts-language.ts'; import type { Page, PageInput, PageFilters, PageType, @@ -173,23 +176,49 @@ export function computeSnapshotSchemaHash( * payload. Fix: `bun upgrade` (newer Bun versions mount the vfs * writable) or run via Node. * - * `macos-26-3` — the pre-existing #223 hint signature (early macOS - * 26.3 builds shipped a broken WASM runtime). + * `corrupt` — catalog/pgvector corruption (#2348): 58P01 / + * internal_load_library / missing vector type or core relation. + * WAL reset cannot fix this class; routes to `gbrain reinit-pglite`. + * MUST stay matched BEFORE the wasm arm — a `58P01 … Aborted()` + * message is catalog corruption, not a WAL tear. + * + * `wasm-abort` — the Emscripten runtime abort (`Aborted(). Build with + * -sASSERTIONS…`, `RuntimeError: unreachable`, and the legacy #223 + * signatures). Root cause is almost always corrupt WAL/checkpoint + * state after an unclean shutdown (historically misdiagnosed as a + * "macOS 26.3 WASM bug" — see #223); this verdict is the trigger + * for the in-place WAL auto-repair (`pglite-repair.ts`). * * `unknown` — falls through to a generic hint that names the doctor - * command; the macOS 26.3 link is offered only on darwin (#2674). + * command; the #223 pointer is offered only on darwin (#2674). * * Regex tightened per Codex eng-review finding #9: don't match * generic `pglite.data` substring (could fire on unrelated PGLite * errors). Match the literal `$$bunfs` marker OR ENOENT+pglite.data * co-occurrence. */ -export type PgliteInitFailure = 'bunfs' | 'macos-26-3' | 'corrupt' | 'unknown'; +export type PgliteInitFailure = 'bunfs' | 'wasm-abort' | 'corrupt' | 'unknown'; // #2674: non-Error rejections (Emscripten aborts can throw plain objects) // used to stringify as "[object Object]" — prefer .message when present. +// WAL-repair wave: Emscripten's FS layer also throws message-LESS objects +// (e.g. `ErrnoError { name: 'ErrnoError', errno: 20 }` when the data dir is a +// symlink NODEFS refuses to mount) — surface name+errno / JSON instead of the +// useless "[object Object]". export function stringifyPgliteInitError(err: unknown): string { - return String((err as { message?: unknown })?.message ?? err); + const message = (err as { message?: unknown })?.message; + if (message != null) return String(message); + if (typeof err === 'object' && err !== null) { + const name = (err as { name?: unknown }).name; + const errno = (err as { errno?: unknown }).errno; + if (typeof name === 'string' && errno != null) return `${name} (errno ${errno})`; + try { + const json = JSON.stringify(err); + if (json && json !== '{}') return typeof name === 'string' ? `${name}: ${json}` : json; + } catch { /* circular — fall through */ } + if (typeof name === 'string') return name; + } + return String(err); } export function classifyPgliteInitError(message: string): PgliteInitFailure { @@ -202,18 +231,79 @@ export function classifyPgliteInitError(message: string): PgliteInitFailure { if (/58P01|internal_load_library|type "?vector"? does not exist|relation "?content_chunks"? does not exist/i.test(message)) { return 'corrupt'; } - if (/abort.*runtime|macos.*26\.3|wasm.*runtime/i.test(message)) { - return 'macos-26-3'; + // Broadened (v0.42.x WAL-repair wave): the REAL production message is + // `Aborted(). Build with -sASSERTIONS for more info.` — no "runtime" in it, + // so the legacy arms alone let the primary crash fall through to 'unknown'. + // Deliberately over-matches (RuntimeError/unreachable are generic WASM + // traps); the repair path downstream is bounded by layout validation, the + // reaped-lock gate, and restore-on-failure. + if (/aborted\s*\(\)|RuntimeError|unreachable|abort.*runtime|macos.*26\.3|wasm.*runtime/i.test(message)) { + return 'wasm-abort'; } return 'unknown'; } +/** + * What the auto-repair path did (or why it didn't run) for a `wasm-abort` + * failure — folded into the user-facing error so the message never lies about + * the state of the data dir. `'failed-not-restored'` is the arm that matters + * most: repair ran, PGLite still failed, AND the automatic restore failed — + * the dir is in a reset state and the user must restore from the backup. + */ +export interface PgliteInitRepairContext { + repair: + | 'not-attempted' + | 'in-memory' + | 'disabled' + | 'skipped-validation' + | 'skipped-live-writer' + | 'skipped-cooldown' + | 'failed-restored' + | 'failed-not-restored'; + backupPath?: string; + detail?: string; +} + +function repairContextLine(ctx: PgliteInitRepairContext): string { + switch (ctx.repair) { + case 'in-memory': + return ' This engine is in-memory (no data dir), so there is no stored state to\n' + + ' repair — this is an environment/runtime failure, not data corruption.'; + case 'disabled': + return ' Auto-repair is disabled (GBRAIN_PGLITE_WAL_REPAIR=off). Run\n' + + ' `gbrain pglite-repair` to repair manually.'; + case 'skipped-validation': + return ` Auto-repair skipped: ${ctx.detail ?? 'the data dir did not validate as a PG17 pglite layout'}.`; + case 'skipped-live-writer': + return ` Auto-repair skipped: ${ctx.detail ?? 'the data-dir lock was acquired by reaping a prior holder'}`; + case 'skipped-cooldown': + return ` Auto-repair skipped: ${ctx.detail ?? 'a recent attempt failed (cooldown active)'}`; + case 'failed-restored': + return ' Auto-repair ran but PGLite still failed to start. The data dir was\n' + + ` RESTORED to its pre-repair state (backup kept at ${ctx.backupPath ?? '.wal-repair-backup-*'}).` + + (ctx.detail ? `\n Detail: ${ctx.detail}` : ''); + case 'failed-not-restored': + return ' Auto-repair ran, PGLite still failed to start, AND the automatic restore\n' + + ' itself failed — the data dir is currently in a RESET state. Your\n' + + ` pre-repair files are intact in the backup at ${ctx.backupPath ?? '.wal-repair-backup-*'};\n` + + ' restore manually: move the backup\'s `pg_wal` dir back to `/pg_wal`\n' + + ' and its `pg_control` file back to `/global/pg_control`.' + + (ctx.detail ? `\n Detail: ${ctx.detail}` : ''); + case 'not-attempted': + default: + return ' Auto-repair was not attempted.'; + } +} + export function buildPgliteInitErrorMessage( verdict: PgliteInitFailure, original: string, // #2674: threaded (defaulted) so tests can exercise both branches without // monkey-patching process.platform. platform: NodeJS.Platform = process.platform, + // WAL-repair wave: what auto-repair did for a wasm-abort, so the hint tells + // the truth about the current state of the data dir. + ctx?: PgliteInitRepairContext, ): string { const header = 'PGLite failed to initialize its WASM runtime.'; let hint: string; @@ -226,17 +316,31 @@ export function buildPgliteInitErrorMessage( ' does not help, run via Node: `node src/cli.ts` or install gbrain\n' + ' using the Node-based path. See #1340 for details.'; break; - case 'macos-26-3': + case 'wasm-abort': hint = - ' This is most commonly the macOS 26.3 WASM bug:\n' + - ' https://github.com/garrytan/gbrain/issues/223'; + ' Most common cause: corrupt WAL/checkpoint state after an unclean\n' + + ' shutdown (often a macOS-upgrade reboot killing gbrain mid-write) —\n' + + ' NOT a macOS WASM bug, despite the historical diagnosis in\n' + + ' https://github.com/garrytan/gbrain/issues/223.\n' + + repairContextLine(ctx ?? { repair: 'not-attempted' }) + '\n' + + ' Recovery ladder:\n' + + ' 1. gbrain pglite-repair --dry-run (diagnose, mutates nothing)\n' + + ' gbrain pglite-repair --yes (in-place WAL repair, data preserved)\n' + + ' 2. Rebuild from your brain repo: `gbrain reinit-pglite` (or manually:\n' + + ' back up ~/.gbrain, move brain.pglite aside, `gbrain init --pglite`,\n' + + ' re-add sources + `gbrain sync` + `gbrain embed`).\n' + + ' 3. Switch engines (docs/ENGINES.md): `gbrain init --supabase` or\n' + + ' native Postgres.\n' + + ' Run `gbrain doctor` for a full diagnosis.'; break; case 'corrupt': hint = ' Your PGLite store looks corrupted (the catalog or the pgvector\n' + ' extension cannot load). This happens when two processes opened the\n' + ' same brain at once — now prevented (#2348), but an already-damaged\n' + - ' store cannot be repaired in place. Recover:\n' + + ' store cannot be repaired in place (WAL repair does not fix catalog\n' + + ' corruption; `gbrain pglite-repair --dry-run` can still report the\n' + + ' state of the data dir). Recover:\n' + ' 1. Restore a backup of the brain.pglite directory if you have one, OR\n' + ' 2. Rebuild from your brain repo:\n' + ' gbrain reinit-pglite --embedding-model --embedding-dimensions \n' + @@ -245,21 +349,42 @@ export function buildPgliteInitErrorMessage( break; case 'unknown': default: - // #2674: only blame the macOS 26.3 WASM bug on macOS. On other - // platforms, point at the causes that are actually plausible there. + // #2674: name the plausible causes per platform. The darwin branch keeps + // the #223 pointer (readers arrive from that issue), reframed to the + // real root cause behind those reports: torn WAL from unclean shutdown. hint = platform === 'darwin' - ? ' Possible cause: the macOS 26.3 WASM bug\n' + - ' (https://github.com/garrytan/gbrain/issues/223).\n' + - ' Run `gbrain doctor` for a full diagnosis.' + ? ' Possible cause: corrupt WAL/checkpoint state after an unclean\n' + + ' shutdown — the failure class behind\n' + + ' https://github.com/garrytan/gbrain/issues/223.\n' + + ' Try `gbrain pglite-repair --dry-run` to diagnose the data dir, and\n' + + ' run `gbrain doctor` for a full diagnosis.' : ' Possible causes: another gbrain process holding the database\n' + ' (lock contention), or a damaged PGLite data directory.\n' + - ' Run `gbrain doctor` for a full diagnosis; if the data dir is\n' + + ' Try `gbrain pglite-repair --dry-run` to diagnose the data dir, and\n' + + ' run `gbrain doctor` for a full diagnosis; if the data dir is\n' + ' damaged, `gbrain reinit-pglite` rebuilds it from your brain repo.'; break; } return `${header}\n${hint}\n Original error: ${original}`; } +/** + * The loud stderr notice printed when connect() auto-repaired the data dir in + * place. Exported for the serial regression test. + */ +export function buildWalRepairNotice(receipt: WalRepairReceipt): string { + return [ + '⚠️ gbrain repaired this brain\'s PGLite WAL in place.', + ` Data dir: ${receipt.dataDir}`, + ` Cause: torn WAL/checkpoint state from an unclean shutdown (issue #223 class).`, + ` Data files were preserved; transactions not checkpointed before the`, + ` corruption may be lost (the standard pg_resetwal caveat).`, + ` Pre-repair backup: ${receipt.backupPath}`, + ` Recommended: run \`gbrain doctor\` to verify brain integrity.`, + ` Disable auto-repair with GBRAIN_PGLITE_WAL_REPAIR=off.`, + ].join('\n'); +} + /** * #2084 — PGLite's Emscripten runtime hijacks `process.exitCode` as ITS status * channel: instantiation REPLACES the property with an accessor whose getter @@ -297,6 +422,12 @@ export class PGLiteEngine implements BrainEngine { // PGlite.create(loadDataDir), initSchema is a no-op (schema is already // present + migrations already applied). Saves ~1-3s per fresh test PGLite. private _snapshotLoaded = false; + /** + * Set when connect() auto-repaired the data dir's WAL in place (mirrors + * upstream PR #994's `repairedDataDir`). Null on every non-repaired connect. + * Test seam + programmatic callers can surface the receipt. + */ + walRepairReceipt: WalRepairReceipt | null = null; get db(): PGLiteDB { if (!this._db) throw new Error('PGLite not connected. Call connect() first.'); @@ -306,6 +437,7 @@ export class PGLiteEngine implements BrainEngine { // Lifecycle async connect(config: EngineConfig): Promise { this._savedConfig = config; // #2034: remember for reconnect() + this.walRepairReceipt = null; // per-connect: stale receipts must not survive reconnect() const dataDir = config.database_path || undefined; // undefined = in-memory // Acquire file lock to prevent concurrent PGLite access (crashes with Aborted()) @@ -343,6 +475,11 @@ export class PGLiteEngine implements BrainEngine { extensions: { vector, pg_trgm }, }), ); + // Healthy open: close any repair episode left open by a prior failed + // attempt (red-team: episodes otherwise stayed open forever — doctor + // kept reporting corruption-likely and a weeks-stale episode backup + // could be reused over much newer data). Cheap no-op without a sidecar. + if (dataDir) closeRepairEpisodeIfOpen(dataDir); } catch (err) { // v0.13.1: any PGLite.create() failure becomes actionable. v0.41.8.0 // (#1340): the previous error hint hardcoded the macOS 26.3 link, but @@ -352,7 +489,54 @@ export class PGLiteEngine implements BrainEngine { // users get the right next step. const original = stringifyPgliteInitError(err); // #2674 const verdict = classifyPgliteInitError(original); - const wrapped = new Error(buildPgliteInitErrorMessage(verdict, original)); + let ctx: PgliteInitRepairContext = { repair: 'not-attempted' }; + + // WAL-repair wave (#223/#1670/#2575): a wasm-abort on a PERSISTENT data + // dir is almost always torn WAL/checkpoint state from an unclean + // shutdown — repairable in place. The seam NEVER throws (its failure + // modes fold into `ctx`), so every non-repaired path still funnels + // through the single lock-release-then-throw site below. + if (verdict === 'wasm-abort') { + if (!dataDir) { + ctx = { repair: 'in-memory' }; + } else { + const attempt = await attemptWalRepairAndRetry( + dataDir, + () => preservingProcessExitCode(() => + // No loadDataDir on the retry: the snapshot path is + // in-memory-only (see above), and dataDir is persistent here. + PGlite.create({ + dataDir, + extensions: { vector, pg_trgm }, + }), + ), + { reaped: this._lock?.reaped }, + ); + if (attempt.status === 'repaired') { + this._db = attempt.db; + this.walRepairReceipt = attempt.receipt; + console.warn(buildWalRepairNotice(attempt.receipt)); + return; // success: lock stays held, normal connect contract + } + if (attempt.status === 'skipped') { + const reasonToCtx = { + 'disabled': 'disabled', + 'validation-failed': 'skipped-validation', + 'possibly-live-writer': 'skipped-live-writer', + 'recently-failed': 'skipped-cooldown', + } as const; + ctx = { repair: reasonToCtx[attempt.reason], detail: attempt.detail }; + } else { + ctx = { + repair: attempt.restored ? 'failed-restored' : 'failed-not-restored', + backupPath: attempt.receipt?.backupPath, + detail: attempt.repairError, + }; + } + } + } + + const wrapped = new Error(buildPgliteInitErrorMessage(verdict, original, process.platform, ctx)); // Release the lock so a fresh process can try again; leaking the lock // here turns a recoverable init error into a stuck-brain state. if (this._lock?.acquired) { diff --git a/src/core/pglite-lock.ts b/src/core/pglite-lock.ts index 188e5c666..1de86bc4e 100644 --- a/src/core/pglite-lock.ts +++ b/src/core/pglite-lock.ts @@ -14,7 +14,7 @@ * try { ... } finally { await releaseLock(lock); } */ -import { mkdirSync, existsSync, readFileSync, writeFileSync, rmSync, statSync } from 'fs'; +import { mkdirSync, existsSync, readFileSync, writeFileSync, rmSync, statSync, renameSync } from 'fs'; import { join } from 'path'; import { parseGlobalFlags } from './cli-options.ts'; @@ -25,7 +25,7 @@ const LOCK_FILE = 'lock'; // LIVE holder (embed jobs run for many minutes) is never mistaken for stale. const HEARTBEAT_INTERVAL_MS = 30_000; -class LiveServeLockError extends Error {} +export class LiveServeLockError extends Error {} function isServeCommand(lockData: { subcommand?: unknown; command?: unknown }): boolean { // New lock files store the command after the same global-flag parsing used @@ -71,6 +71,15 @@ export interface LockHandle { * the NEW owner's live lock and re-open the concurrent-writer hole). */ ownerToken?: string; + /** + * WAL-repair gate (#223 auto-repair): true when this acquisition reaped a + * prior holder's lock — dead-PID reap or corrupt-lock-file removal. A + * corrupt lock file cannot prove its holder is dead, and even a dead-PID + * verdict can be wrong under PID reuse, so auto WAL surgery refuses to run + * on a reaped acquisition (`'possibly-live-writer'`) and asks for a clean + * re-run instead. Never set for in-memory engines. + */ + reaped?: boolean; } /** The on-disk lock identity, used to detect "we were reaped and replaced". */ @@ -97,13 +106,51 @@ function startHeartbeat(lockPath: string, ownerToken: string): ReturnType void }).unref?.(); return timer; } +/** + * Persisted reap marker (security review): written ONLY for corrupt-lock-file + * reaps, where the holder's liveness is UNKNOWABLE (the PID can't be read). + * The in-process `reaped` flag dies with the acquisition — so the reaper + * destroys a possibly-live holder's lock, exits, and the NEXT process + * acquires "cleanly" and would run WAL surgery under a live writer. The + * marker makes that reap visible across processes: `attemptWalRepairAndRetry` + * refuses auto-repair while a recent unknowable-liveness reap is on record. + * Dead-PID reaps (affirmative ESRCH verdict) deliberately do NOT write it — + * the dead-holder recovery cost stays at one failed command + one re-run. + */ +function reapMarkerPath(dataDir: string): string { + return `${dataDir}.lock-reap.json`; +} + +function recordReap(dataDir: string): void { + try { + writeFileSync(reapMarkerPath(dataDir), JSON.stringify({ ts: Date.now(), by: process.pid }), { mode: 0o644 }); + } catch { /* best-effort — a marker write failure must not block acquisition */ } +} + +/** Milliseconds since the last recorded reap on this data dir, or null. */ +export function msSinceLastReap(dataDir: string | undefined): number | null { + if (!dataDir) return null; + try { + const raw = JSON.parse(readFileSync(reapMarkerPath(dataDir), 'utf-8')) as { ts?: unknown }; + return typeof raw.ts === 'number' && Number.isFinite(raw.ts) ? Date.now() - raw.ts : null; + } catch { + return null; + } +} + function getLockDir(dataDir: string | undefined): string { // Use the parent of the data dir for the lock, or a temp location for in-memory if (!dataDir) { @@ -114,13 +161,17 @@ function getLockDir(dataDir: string | undefined): string { return join(dataDir, LOCK_DIR_NAME); } -function isProcessAlive(pid: number): boolean { +export function isProcessAlive(pid: number): boolean { + // Only ESRCH (no such process) is affirmative proof of death. EPERM means + // the process EXISTS under another user; ERR_INVALID_ARG_TYPE / a malformed + // or non-finite pid means we can't tell — all of which must read as ALIVE, + // because a false "dead" reaps a live holder's lock (security/codex review). + if (!Number.isInteger(pid) || pid <= 0) return true; try { - // Sending signal 0 checks existence without actually sending a signal - process.kill(pid, 0); + process.kill(pid, 0); // signal 0 = existence check, no signal delivered return true; - } catch { - return false; + } catch (err) { + return (err as NodeJS.ErrnoException)?.code !== 'ESRCH'; } } @@ -172,6 +223,7 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM const timeoutMs = opts?.timeoutMs ?? 30_000; // 30 second default timeout const startTime = Date.now(); + let reaped = false; // see LockHandle.reaped while (Date.now() - startTime < timeoutMs) { // Check for stale lock first @@ -187,7 +239,12 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM // heartbeat" is NOT evidence of death — only a dead PID is. const alive = isProcessAlive(lockPid); if (!alive) { - // Holder process is gone — reap and try to acquire. + // Holder process is gone — reap and try to acquire. This verdict is + // affirmative (kill-0 threw ESRCH; EPERM reads as alive), so no + // cross-process quarantine marker: the same-acquisition `reaped` + // flag alone gates repair, keeping the dead-holder recovery cost at + // one failed command + one re-run. + reaped = true; try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ } } else { if (isServeCommand(lockData)) { @@ -209,7 +266,22 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM // A live MCP server is not a stale or corrupt lock. Surface the useful // explanation without touching the lock it still owns. if (err instanceof LiveServeLockError) throw err; - // Corrupt lock file — remove it + // ENOENT = acquisition in flight (a concurrent acquirer did mkdir but + // hasn't written the lock file yet) — reaping HERE would destroy a + // LIVE acquirer's lock and put two writers on one dir (red-team). + // Give the writer a grace window keyed on the lock dir's age. + if ((err as NodeJS.ErrnoException)?.code === 'ENOENT') { + let lockDirAgeMs = Infinity; + try { lockDirAgeMs = Date.now() - statSync(lockDir).mtimeMs; } catch { /* dir gone — retry loop handles */ } + if (lockDirAgeMs < 10_000) { + await new Promise(r => setTimeout(r, 200)); + continue; + } + } + // Corrupt lock file — remove it. The holder's liveness is UNKNOWABLE + // here (unreadable PID), so this counts as a reap for the repair gate. + reaped = true; + recordReap(dataDir as string); try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ } } } @@ -221,16 +293,21 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM // the heartbeat so this holder reads as alive-and-working to others. const lockPath = join(lockDir, LOCK_FILE); const now = Date.now(); - writeFileSync(lockPath, JSON.stringify({ + // Atomic tmp+rename, same torn-read protection as the heartbeat: a + // concurrent poll-reader must see the file complete or absent, never + // mid-write (a torn read classifies a LIVE holder as corrupt). + const initTmp = `${lockPath}.tmp-${process.pid}`; + writeFileSync(initTmp, JSON.stringify({ pid: process.pid, acquired_at: now, refreshed_at: now, command: process.argv.slice(1).join(' '), subcommand: parseGlobalFlags(process.argv.slice(2)).rest[0] ?? null, }), { mode: 0o644 }); + renameSync(initTmp, lockPath); const ownerToken = tokenOf({ pid: process.pid, acquired_at: now }); - return { lockDir, acquired: true, lockPath, ownerToken, heartbeat: startHeartbeat(lockPath, ownerToken) }; + return { lockDir, acquired: true, lockPath, ownerToken, reaped, heartbeat: startHeartbeat(lockPath, ownerToken) }; } catch (e: unknown) { // mkdir failed — someone else grabbed it between our check and mkdir // This is fine, we'll retry diff --git a/src/core/pglite-repair.ts b/src/core/pglite-repair.ts new file mode 100644 index 000000000..b629bb908 --- /dev/null +++ b/src/core/pglite-repair.ts @@ -0,0 +1,771 @@ +/** + * PGLite WAL-repair orchestrator (#223 / #1670 / #2575). + * + * Wraps the pg_resetwal port (`pglite-resetwal.ts`) with everything that makes + * it safe to run automatically from `PGLiteEngine.connect()`: + * + * validate (read-only, fail-closed) → back up (rename, not copy) → + * resetWal → retry create() once → restore on failure. + * + * Safety posture (eng-review 1A/2A/3A/4A + codex round): + * - WAL surgery only runs under a CLEANLY-acquired data-dir lock. A reaped + * acquisition (dead-PID or corrupt-lock-file reap — the only reaps that + * exist post-#2348) refuses with `'possibly-live-writer'`; this module + * never force-removes `.gbrain-lock`. + * - Backup is a whole-`pg_wal/`-directory rename into a sibling dir (O(1), + * zero extra disk — a real brain's pg_wal is ~144MB and a copy would + * transiently double it, ENOSPC-ing exactly on disk-pressure machines); + * only the 8KB `global/pg_control` is copied (it is mutated in place). + * - Restore never deletes anything and never leaves the dir without a valid + * pg_control: control is restored first (atomic tmp+rename), then the reset + * pg_wal is renamed ASIDE into the backup dir and the original renamed back. + * - A cooldown sidecar + episode-scoped backups bound the reconnect loops + * (autopilot ~10s tick under launchd KeepAlive; minion supervisor): repeated + * attempts inside one corruption episode reuse the episode's first backup + * (the pre-damage forensic state) instead of stacking new ones, and a + * recently-failed attempt skips repair entirely for the cooldown window. + * + * `attemptWalRepairAndRetry` NEVER throws — `connect()`'s catch consumes the + * discriminated union, so no new code path can bypass the engine's single + * lock-release-then-throw site. + * + * Env knobs (incident escape hatches, env-only by design): + * GBRAIN_PGLITE_WAL_REPAIR=off disable auto-repair + * GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS default 3600 + */ +import { + existsSync, lstatSync, readdirSync, readFileSync, statSync, writeFileSync, + mkdirSync, rmSync, renameSync, +} from 'node:fs'; +import { readFile, rename } from 'node:fs/promises'; +import { basename, dirname, join } from 'node:path'; +import { resetWal, writeFileAtomicSynced, WalResetUnsupportedError, PG_CONTROL_FILE_SIZE, isWalSegmentName } from './pglite-resetwal.ts'; +import { msSinceLastReap, isProcessAlive } from './pglite-lock.ts'; + +// A recent reap on this data dir — by ANY process — means a holder that may +// still be alive lost its lock; auto WAL surgery stays off until the window +// clears (security review: the in-process `reaped` flag alone let the NEXT +// acquirer look clean while the reaped holder was still writing). +const REAP_QUARANTINE_MS = 10 * 60 * 1000; + +const BACKUP_DIR_MARKER = '.wal-repair-backup-'; +const SIDECAR_SUFFIX = '.wal-repair-attempt.json'; +const MAX_SIDECAR_ATTEMPTS = 10; +const KEEP_EPISODES = 3; +const DEFAULT_COOLDOWN_SECONDS = 3600; + +export interface WalRepairReceipt { + dataDir: string; + /** Sibling dir holding the pre-repair state: `.wal-repair-backup-/` */ + backupPath: string; + /** Relative paths preserved in the backup (e.g. 'pg_wal/', 'postmaster.pid', 'global/pg_control'). */ + backedUpFiles: string[]; + /** True when this attempt reused an open episode's existing backup. */ + reusedEpisodeBackup: boolean; + resetSegment: string; + timelineId: number; + walSegSize: number; + repairedAt: string; // ISO +} + +export type WalRepairValidation = + | { ok: true } + | { + ok: false; + reason: 'missing-dir' | 'not-pglite-layout' | 'unsupported-pg-version' | 'bad-pg-control'; + detail: string; + }; + +export interface RestoreResult { + restored: boolean; + steps: string[]; + detail?: string; +} + +/** + * Thrown by `repairPgliteWal` when the reset failed AFTER the backup was + * taken. Carries the receipt and the result of the best-effort restore so the + * seam can report `restored` HONESTLY instead of assuming the restore worked + * (the `failed-not-restored` message arm depends on this being truthful). + */ +export class WalRepairError extends Error { + constructor( + message: string, + readonly receipt: WalRepairReceipt, + readonly restore: RestoreResult, + ) { + super(message); + this.name = 'WalRepairError'; + } +} + +export type WalRepairAttempt = + | { status: 'repaired'; db: T; receipt: WalRepairReceipt } + | { + status: 'skipped'; + reason: 'disabled' | 'validation-failed' | 'possibly-live-writer' | 'recently-failed'; + detail: string; + } + | { status: 'failed'; receipt: WalRepairReceipt | null; restored: boolean; repairError: string }; + +export interface PgliteDirDiagnosis { + exists: boolean; + postmasterPid: boolean; + pgControlOk: boolean; + pgVersion: string | null; + walSegments: string[]; + lockHeld: boolean; + lockHolderPid: number | null; + /** Sibling `*.wal-repair-backup-*` dirs, newest first. */ + backupDirs: string[]; + /** Recent repair attempts from the sidecar, newest last. */ + recentAttempts: Array<{ ts: number; outcome: 'repaired' | 'failed' }>; + verdict: 'looks-healthy' | 'wal-corruption-likely' | 'locked' | 'missing' | 'unsupported-layout'; + detail: string; +} + +interface RepairSidecar { + /** ts of the first failed attempt of the open episode; null = no open episode. */ + episodeStartedAt: number | null; + /** The open episode's (first) backup dir — the pre-damage forensic state. */ + episodeBackupPath: string | null; + attempts: Array<{ ts: number; outcome: 'repaired' | 'failed'; backupPath: string | null }>; +} + +function sidecarPath(dataDir: string): string { + return `${dataDir}${SIDECAR_SUFFIX}`; +} + +export function readRepairSidecar(dataDir: string): RepairSidecar { + try { + const raw = JSON.parse(readFileSync(sidecarPath(dataDir), 'utf-8')) as Partial; + return { + episodeStartedAt: typeof raw.episodeStartedAt === 'number' ? raw.episodeStartedAt : null, + episodeBackupPath: typeof raw.episodeBackupPath === 'string' ? raw.episodeBackupPath : null, + attempts: Array.isArray(raw.attempts) + ? raw.attempts.filter( + (a): a is RepairSidecar['attempts'][number] => + !!a && typeof a.ts === 'number' && (a.outcome === 'repaired' || a.outcome === 'failed'), + ) + : [], + }; + } catch { + return { episodeStartedAt: null, episodeBackupPath: null, attempts: [] }; + } +} + +function writeRepairSidecar(dataDir: string, sidecar: RepairSidecar): void { + try { + // Atomic tmp+rename: a kill/power-loss mid-write must not truncate the + // sidecar to invalid JSON (readRepairSidecar would then silently reset the + // episode/cooldown state — codex review). rename is atomic; a torn tmp is + // discarded on the next write. + const tmp = `${sidecarPath(dataDir)}.tmp-${process.pid}`; + writeFileSync(tmp, JSON.stringify(sidecar), { mode: 0o644 }); + renameSync(tmp, sidecarPath(dataDir)); + } catch { /* best-effort — a sidecar write failure must never block recovery */ } +} + +/** + * Record a real repair attempt (repaired|failed) and manage episode state: + * a `failed` attempt opens an episode (if none is open) pinning its backup as + * the episode backup; a VERIFIED `repaired` attempt closes the episode and + * prunes retained backups to the newest KEEP_EPISODES. The manual command + * passes `closeEpisode: false` — its "repaired" is unverified (the next + * connect proves it), and closing+pruning on unverified success let repeated + * manual runs delete the pre-damage forensic backup (red-team finding); the + * episode instead closes on the next successful connect + * (`closeRepairEpisodeIfOpen`). + * + * Re-pin rule (red-team finding): a restore MOVES pg_wal back out of the + * backup, gutting it — if a later failed attempt took a FRESH backup while an + * episode pinned a gutted dir, the pin moves to the fresh backup so the + * episode's protected copy is always one that still holds pg_wal. + */ +export function recordRepairAttempt( + dataDir: string, + outcome: 'repaired' | 'failed', + backupPath: string | null, + opts?: { closeEpisode?: boolean }, +): void { + const sidecar = readRepairSidecar(dataDir); + sidecar.attempts.push({ ts: Date.now(), outcome, backupPath }); + if (sidecar.attempts.length > MAX_SIDECAR_ATTEMPTS) { + sidecar.attempts = sidecar.attempts.slice(-MAX_SIDECAR_ATTEMPTS); + } + if (outcome === 'failed') { + if (sidecar.episodeStartedAt === null) { + sidecar.episodeStartedAt = Date.now(); + sidecar.episodeBackupPath = backupPath; + } else if ( + backupPath && + backupPath !== sidecar.episodeBackupPath && + (!sidecar.episodeBackupPath || !existsSync(join(sidecar.episodeBackupPath, 'pg_wal'))) + ) { + sidecar.episodeBackupPath = backupPath; + } + } else if (opts?.closeEpisode !== false) { + sidecar.episodeStartedAt = null; + sidecar.episodeBackupPath = null; + } + writeRepairSidecar(dataDir, sidecar); + if (outcome === 'repaired' && opts?.closeEpisode !== false) { + pruneRepairBackups(dataDir); + } +} + +/** + * Close any open repair episode after a HEALTHY connect (red-team finding: a + * plain successful open never touched the sidecar, so an episode stayed open + * forever — doctor kept reporting corruption-likely, and a weeks-stale + * episode backup could be reused over much newer data). Cheap no-op when no + * sidecar exists. Called by PGLiteEngine.connect() on every non-repaired + * success and by the seam's 'repaired' arm via recordRepairAttempt. + */ +export function closeRepairEpisodeIfOpen(dataDir: string): void { + try { + if (!existsSync(sidecarPath(dataDir))) return; + const sidecar = readRepairSidecar(dataDir); + if (sidecar.episodeStartedAt === null) return; + sidecar.episodeStartedAt = null; + sidecar.episodeBackupPath = null; + writeRepairSidecar(dataDir, sidecar); + pruneRepairBackups(dataDir); + } catch { /* best-effort */ } +} + +/** + * Cooldown: true when the last FAILED attempt is inside the cooldown window. + * DELIBERATE: a later successful repair does NOT clear the cooldown — repeated + * corruption right after a "success" usually means the unclean-shutdown + * genesis is still active, and looping surgery would silently eat a WAL tail + * per cycle. The manual `gbrain pglite-repair` command bypasses the cooldown. + */ +export function repairCooldownActive(dataDir: string): { active: boolean; detail: string } { + const seconds = Number(process.env.GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS ?? DEFAULT_COOLDOWN_SECONDS); + const windowMs = (Number.isFinite(seconds) && seconds >= 0 ? seconds : DEFAULT_COOLDOWN_SECONDS) * 1000; + if (windowMs === 0) return { active: false, detail: 'cooldown disabled (0s)' }; + const sidecar = readRepairSidecar(dataDir); + // Repaired-loop guard (red-team finding): a crash loop where every reopen + // aborts but repair "succeeds" each time would silently discard a WAL tail + // per cycle with no failed attempt ever recorded. Two successful repairs + // inside one window = the corruption genesis is active — stop auto-repair + // and let doctor escalate. + const repairedInWindow = sidecar.attempts.filter( + (a) => a.outcome === 'repaired' && Date.now() - a.ts >= 0 && Date.now() - a.ts < windowMs, + ).length; + if (repairedInWindow >= 2) { + return { + active: true, + detail: + `auto-repair already ran ${repairedInWindow}x in the last ${windowMs / 1000}s — repeated ` + + 'corruption means the unclean-shutdown genesis is still active; refusing to silently ' + + 'discard another WAL tail. Run `gbrain doctor`, or `gbrain pglite-repair` manually.', + }; + } + const lastFailed = [...sidecar.attempts].reverse().find((a) => a.outcome === 'failed'); + if (!lastFailed) return { active: false, detail: 'no prior failed attempt' }; + const ageMs = Date.now() - lastFailed.ts; + // Clock skew (unclean-reboot recovery is exactly when clocks step): a + // negative age means the recorded ts is in the future — treat as expired + // rather than suppressing auto-repair until wall-clock catches up. + if (ageMs >= 0 && ageMs < windowMs) { + return { + active: true, + detail: + `last auto-repair attempt failed ${Math.round(ageMs / 1000)}s ago ` + + `(cooldown ${windowMs / 1000}s — set GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS=0 to bypass, ` + + `or run \`gbrain pglite-repair\` manually)`, + }; + } + return { active: false, detail: 'cooldown expired' }; +} + +/** Sibling `*.wal-repair-backup-*` dirs for this data dir, newest first. */ +export function listRepairBackups(dataDir: string): string[] { + try { + const parent = dirname(dataDir); + const prefix = `${basename(dataDir)}${BACKUP_DIR_MARKER}`; + return readdirSync(parent) + .filter((name) => name.startsWith(prefix)) + .sort() + .reverse() + .map((name) => join(parent, name)); + } catch { + return []; + } +} + +/** + * Keep the newest KEEP_EPISODES backups; never prune the open episode's backup. + * Runs only after a successful repair (episode close) — never mid-incident. + */ +export function pruneRepairBackups(dataDir: string): void { + const sidecar = readRepairSidecar(dataDir); + const backups = listRepairBackups(dataDir); // newest first + const keep = new Set(backups.slice(0, KEEP_EPISODES)); + if (sidecar.episodeBackupPath) keep.add(sidecar.episodeBackupPath); + for (const dir of backups) { + if (!keep.has(dir)) { + try { rmSync(dir, { recursive: true, force: true }); } catch { /* best-effort */ } + } + } +} + +export function walRepairEnabled(): boolean { + return process.env.GBRAIN_PGLITE_WAL_REPAIR !== 'off'; +} + +function isSymlink(path: string): boolean { + try { + return lstatSync(path).isSymbolicLink(); + } catch { + return false; + } +} + +/** + * Read-only, fail-closed: does this look like a PG17 pglite data dir we know + * how to repair? Tolerates the `.gbrain-lock` entry (the lock lives INSIDE the + * data dir). Refuses symlinked components (codex 14.8 — a symlinked `pg_wal` + * or `pg_control` could redirect the backup/restore renames at unrelated + * files; same confinement discipline as the v0.42.55.0 security wave). + */ +export function validateWalRepairTarget(dataDir: string): WalRepairValidation { + if (!dataDir) return { ok: false, reason: 'missing-dir', detail: 'no data dir configured (in-memory engine)' }; + if (!existsSync(dataDir)) return { ok: false, reason: 'missing-dir', detail: `${dataDir} does not exist` }; + if ( + isSymlink(dataDir) || + isSymlink(join(dataDir, 'pg_wal')) || + // `global/` itself must be checked too: lstat on global/pg_control follows + // the INTERMEDIATE symlink, so a symlinked global/ would pass and surgery + // would write a forged pg_control through it into a foreign directory + // (security review finding). + isSymlink(join(dataDir, 'global')) || + isSymlink(join(dataDir, 'global', 'pg_control')) + ) { + return { ok: false, reason: 'not-pglite-layout', detail: 'data dir, pg_wal, global, or pg_control is a symlink — refusing to run rename-based repair through symlinks' }; + } + let pgVersion: string; + try { + pgVersion = readFileSync(join(dataDir, 'PG_VERSION'), 'utf-8').trim(); + } catch { + return { ok: false, reason: 'not-pglite-layout', detail: `no readable PG_VERSION in ${dataDir}` }; + } + if (pgVersion !== '17') { + return { ok: false, reason: 'unsupported-pg-version', detail: `PG_VERSION is ${pgVersion}, this repair understands 17 only` }; + } + if (!existsSync(join(dataDir, 'base'))) { + return { ok: false, reason: 'not-pglite-layout', detail: `no base/ directory in ${dataDir}` }; + } + // Live-postmaster refusal (red-team finding): real pg_resetwal refuses when + // postmaster.pid exists. A LIVE native Postgres 17 data dir passes every + // layout check here — without this guard, `gbrain pglite-repair --path` at + // such a dir would rename pg_wal out from under a running postmaster the + // gbrain lock cannot see. A stale pid file (dead process) stays repairable. + try { + const pidRaw = readFileSync(join(dataDir, 'postmaster.pid'), 'utf-8').split('\n')[0]?.trim(); + const pid = Number(pidRaw); + if (Number.isInteger(pid) && pid > 0 && isProcessAlive(pid)) { + return { + ok: false, + reason: 'not-pglite-layout', + detail: `postmaster.pid names a LIVE process (PID ${pid}) — refusing WAL surgery on a possibly-running database`, + }; + } + } catch { /* no postmaster.pid or unreadable — fine */ } + const controlPath = join(dataDir, 'global', 'pg_control'); + try { + const size = statSync(controlPath).size; + if (size !== PG_CONTROL_FILE_SIZE) { + return { ok: false, reason: 'bad-pg-control', detail: `pg_control is ${size} bytes, expected ${PG_CONTROL_FILE_SIZE}` }; + } + } catch { + return { ok: false, reason: 'bad-pg-control', detail: `no readable ${controlPath}` }; + } + return { ok: true }; +} + +/** Read-only diagnosis for `gbrain doctor` and `pglite-repair --dry-run`. */ +export function inspectPgliteDataDir(dataDir: string): PgliteDirDiagnosis { + const sidecar = readRepairSidecar(dataDir); + const base: Omit = { + exists: !!dataDir && existsSync(dataDir), + postmasterPid: !!dataDir && existsSync(join(dataDir, 'postmaster.pid')), + pgControlOk: false, + pgVersion: null, + walSegments: [], + lockHeld: false, + lockHolderPid: null, + backupDirs: listRepairBackups(dataDir), + recentAttempts: sidecar.attempts.map(({ ts, outcome }) => ({ ts, outcome })), + }; + if (!base.exists) { + return { ...base, verdict: 'missing', detail: `${dataDir || '(in-memory)'} does not exist` }; + } + try { + base.pgVersion = readFileSync(join(dataDir, 'PG_VERSION'), 'utf-8').trim(); + } catch { /* leave null */ } + try { + base.pgControlOk = statSync(join(dataDir, 'global', 'pg_control')).size === PG_CONTROL_FILE_SIZE; + } catch { /* leave false */ } + try { + base.walSegments = readdirSync(join(dataDir, 'pg_wal')).filter(isWalSegmentName).sort(); + } catch { /* leave empty */ } + try { + const lockData = JSON.parse(readFileSync(join(dataDir, '.gbrain-lock', 'lock'), 'utf-8')) as { pid?: number }; + if (typeof lockData.pid === 'number' && isProcessAlive(lockData.pid)) { + base.lockHeld = true; + base.lockHolderPid = lockData.pid; + } + } catch { /* no lock / unreadable — not held */ } + + if (base.lockHeld) { + return { ...base, verdict: 'locked', detail: `data dir lock held by live PID ${base.lockHolderPid}` }; + } + const validation = validateWalRepairTarget(dataDir); + if (!validation.ok) { + return { ...base, verdict: 'unsupported-layout', detail: validation.detail }; + } + if (base.postmasterPid || sidecar.episodeStartedAt !== null) { + return { + ...base, + verdict: 'wal-corruption-likely', + detail: base.postmasterPid + ? 'stale postmaster.pid present — an unclean shutdown left WAL/checkpoint state torn' + : 'an unresolved repair episode is open (a prior auto-repair attempt failed)', + }; + } + return { ...base, verdict: 'looks-healthy', detail: 'layout validates; no unclean-shutdown markers on disk' }; +} + +/** + * Mechanical repair: (backup unless reusing an episode backup) → resetWal. + * Backup = rename the ENTIRE pg_wal/ dir + postmaster.pid into the backup dir + * (covers archive_status/summaries too — restore is truly byte-identical) and + * COPY the 8KB pg_control. If resetWal throws after the backup was taken, a + * best-effort restore runs before the error propagates — this function never + * leaves the dir backed-up-but-unrepaired without attempting to put it back. + */ +export async function repairPgliteWal( + dataDir: string, + opts?: { reuseBackupPath?: string }, +): Promise { + const validation = validateWalRepairTarget(dataDir); + if (!validation.ok) { + throw new WalResetUnsupportedError(`refusing repair: ${validation.detail}`); + } + + // Defense-in-depth (security + red-team reviews): the reuse path comes from + // the user-writable sidecar JSON — only honor it when it is a real, + // non-symlink, traversal-free sibling backup dir of THIS data dir that + // STILL CONTAINS pg_wal (a restore MOVES pg_wal back out, gutting the + // backup; reusing a gutted backup would let resetWal unlink the only + // surviving WAL copy in place). Anything else gets a fresh backup. + const safeReusePath = + opts?.reuseBackupPath && + opts.reuseBackupPath.startsWith(`${dataDir}${BACKUP_DIR_MARKER}`) && + !opts.reuseBackupPath.includes('..') && + existsSync(opts.reuseBackupPath) && + !isSymlink(opts.reuseBackupPath) && + existsSync(join(opts.reuseBackupPath, 'pg_wal')) && + !isSymlink(join(opts.reuseBackupPath, 'pg_wal')) + ? opts.reuseBackupPath + : undefined; + const backedUpFiles: string[] = []; + const reusedEpisodeBackup = !!safeReusePath; + + let backupPath: string; + if (reusedEpisodeBackup) { + // Episode reuse: the open episode's backup still holds pg_wal (enforced by + // safeReusePath — a restore MOVES pg_wal back out and guts the backup; a + // gutted backup must never be reused or resetWal would unlink the only + // surviving WAL copy in place). resetWal's own deletion loops clear the + // current segments. + backupPath = safeReusePath!; + } else { + // Fresh backup dir: mkdir with recursive:false and fail-closed on + // collision (red-team: a predictable pre-existing dir or symlink-to-dir + // would silently receive the renames, and restore would later read + // pg_control bytes back OUT of it). Retry with a suffix, then verify we + // created a real directory. + backupPath = `${dataDir}${BACKUP_DIR_MARKER}${Date.now()}`; + for (let attempt = 0; ; attempt++) { + try { + mkdirSync(backupPath, { recursive: false }); + break; + } catch (err) { + if ((err as NodeJS.ErrnoException)?.code === 'EEXIST' && attempt < 5) { + backupPath = `${dataDir}${BACKUP_DIR_MARKER}${Date.now()}-${attempt + 1}`; + continue; + } + throw err; + } + } + if (isSymlink(backupPath) || !statSync(backupPath).isDirectory()) { + throw new WalResetUnsupportedError(`backup path ${backupPath} is not a real directory`); + } + } + // Track whether the backup dir received anything, so refusal paths can prune + // an empty leftover (adversarial review F11: doctor would otherwise inventory + // an empty `.wal-repair-backup-*` as a real backup). + const pruneEmptyBackup = () => { + if (reusedEpisodeBackup) return; + try { if (readdirSync(backupPath).length === 0) rmSync(backupPath, { recursive: true, force: true }); } catch { /* best-effort */ } + }; + + const receipt: WalRepairReceipt = { + dataDir, + backupPath, + backedUpFiles, + reusedEpisodeBackup, + resetSegment: '', + timelineId: 0, + walSegSize: 0, + repairedAt: new Date().toISOString(), + }; + + if (!reusedEpisodeBackup) { + // Backup phase. Once the FIRST rename lands, any failure here must run a + // restore and surface via WalRepairError — a generic throw would read as + // "dir never touched" while pg_wal is actually sitting in the backup dir + // (red-team finding). + let backupStarted = false; + try { + const walDir = join(dataDir, 'pg_wal'); + if (existsSync(walDir)) { + await rename(walDir, join(backupPath, 'pg_wal')); + backupStarted = true; + backedUpFiles.push('pg_wal/'); + } + const pidFile = join(dataDir, 'postmaster.pid'); + if (existsSync(pidFile)) { + await rename(pidFile, join(backupPath, 'postmaster.pid')); + backupStarted = true; + backedUpFiles.push('postmaster.pid'); + } + const control = await readFile(join(dataDir, 'global', 'pg_control')); + await writeFileAtomicSynced(backupPath, 'pg_control', Buffer.from(control)); + backedUpFiles.push('global/pg_control'); + } catch (err) { + if (!backupStarted) { pruneEmptyBackup(); throw err; } // dir genuinely untouched + const restore = await restoreWalBackup(receipt); + throw new WalRepairError(String((err as Error)?.message ?? err), receipt, restore); + } + } + + try { + const result = await resetWal(dataDir); + receipt.resetSegment = result.resetSegment; + receipt.timelineId = result.timelineId; + receipt.walSegSize = result.walSegSize; + } catch (err) { + // Best-effort restore — never leave backed-up-but-unrepaired. The result + // is THREADED OUT via WalRepairError so callers can report `restored` + // honestly (review finding: discarding it let 'failed-restored' lie). + const restore = await restoreWalBackup(receipt); + throw new WalRepairError(String((err as Error)?.message ?? err), receipt, restore); + } + return receipt; +} + +/** + * Put the data dir back to the backed-up state. Overwrite order (eng-review + * 3A): pg_control FIRST (atomic tmp+rename — no instant leaves the dir without + * a valid control file), then the pg_wal dir swap (reset dir renamed ASIDE + * into the backup dir — nothing is ever deleted during restore), postmaster.pid + * deliberately NOT restored (it was stale by definition). Mtime guard + * (eng-review 1A): refuses when the current pg_wal contains segments newer + * than the backup that this repair did not write — a live writer advanced the + * dir; renaming it away would destroy real WAL. + * Never throws — reports `{restored:false, detail}` instead. + */ +export async function restoreWalBackup(receipt: WalRepairReceipt): Promise { + const steps: string[] = []; + try { + const { dataDir, backupPath } = receipt; + const walDir = join(dataDir, 'pg_wal'); + const backupWal = join(backupPath, 'pg_wal'); + const backupControl = join(backupPath, 'pg_control'); + + // Symlinked backup CHILDREN would let restore read attacker-chosen + // pg_control bytes or rename a foreign pg_wal into the data dir + // (red-team) — the top-level checks don't cover them. + if (isSymlink(backupWal) || isSymlink(backupControl)) { + return { restored: false, steps, detail: `backup at ${backupPath} contains symlinked components — refusing restore` }; + } + + // Mtime guard: any foreign WAL segment newer than this repair's start? + const backupTs = Date.parse(receipt.repairedAt); + if (existsSync(walDir)) { + for (const f of readdirSync(walDir)) { + if (!isWalSegmentName(f) || f === receipt.resetSegment) continue; + try { + if (statSync(join(walDir, f)).mtimeMs > backupTs) { + return { + restored: false, + steps, + detail: `mtime-guard: ${f} in pg_wal is newer than the backup — a live writer may have advanced this dir; refusing to swap WAL back`, + }; + } + } catch { /* statable race — ignore */ } + } + } + + if (existsSync(backupControl)) { + const control = await readFile(backupControl); + await writeFileAtomicSynced(join(dataDir, 'global'), 'pg_control', Buffer.from(control)); + steps.push('pg_control restored'); + } + + if (existsSync(backupWal)) { + if (existsSync(walDir)) { + const aside = join(backupPath, `pg_wal.reset-aside-${Date.now()}`); + await rename(walDir, aside); + steps.push(`reset pg_wal set aside at ${aside}`); + } + await rename(backupWal, walDir); + steps.push('pg_wal restored'); + } + if (steps.length === 0) { + // A missing/empty backup means nothing was put back — never claim + // restoration that did not happen (the 'failed-not-restored' honesty arm). + return { restored: false, steps, detail: `nothing to restore from backup at ${backupPath} (missing or empty)` }; + } + return { restored: true, steps }; + } catch (err) { + return { + restored: false, + steps, + detail: `restore failed after [${steps.join(', ') || 'nothing'}]: ${String((err as Error)?.message ?? err)}`, + }; + } +} + +/** + * The engine seam. NEVER throws. Gates (in order): kill-switch → live-writer + * (reaped lock) → layout validation → cooldown. Then: repair (reusing the open + * episode's backup when present) → retry create() ONCE → on failure, restore + * and record. Prints a repair-start stderr line the instant surgery begins so + * a timeout-killed attempt is self-explaining (eng-review 4A). + */ +export async function attemptWalRepairAndRetry( + dataDir: string, + retryCreate: () => Promise, + opts?: { reaped?: boolean }, +): Promise> { + try { + if (!walRepairEnabled()) { + return { status: 'skipped', reason: 'disabled', detail: 'GBRAIN_PGLITE_WAL_REPAIR=off' }; + } + if (opts?.reaped) { + return { + status: 'skipped', + reason: 'possibly-live-writer', + detail: + 'this process acquired the data-dir lock by reaping a prior holder — ' + + 'another gbrain process may still be using this brain. Stop it (or confirm ' + + 'none is running), then re-run; a cleanly-acquired lock enables auto-repair.', + }; + } + const sinceReap = msSinceLastReap(dataDir); + // `>= 0` guard (adversarial review F5): a future-dated marker (clock step + // during the unclean-reboot recovery this feature exists for) yields a + // negative age; treat it as expired rather than quarantining forever — + // same policy as repairCooldownActive. + if (sinceReap !== null && sinceReap >= 0 && sinceReap < REAP_QUARANTINE_MS) { + return { + status: 'skipped', + reason: 'possibly-live-writer', + detail: + `a lock on this brain was reaped ${Math.round(sinceReap / 1000)}s ago (possibly from a ` + + 'still-live process) — auto-repair stays off for ' + + `${REAP_QUARANTINE_MS / 60000} minutes after any reap. Confirm no gbrain process is ` + + 'running, then re-run or use `gbrain pglite-repair`.', + }; + } + const validation = validateWalRepairTarget(dataDir); + if (!validation.ok) { + return { status: 'skipped', reason: 'validation-failed', detail: validation.detail }; + } + const cooldown = repairCooldownActive(dataDir); + if (cooldown.active) { + return { status: 'skipped', reason: 'recently-failed', detail: cooldown.detail }; + } + + try { + process.stderr.write( + `gbrain: PGLite failed to open ${dataDir} — attempting automatic WAL repair ` + + `(backup at ${dataDir}${BACKUP_DIR_MARKER}*). If this command times out, run ` + + `\`gbrain pglite-repair\` to finish. Disable auto-repair with GBRAIN_PGLITE_WAL_REPAIR=off.\n`, + ); + } catch { /* EPIPE under a closed-pipe daemon parent must not read as surgery failure */ } + + const sidecar = readRepairSidecar(dataDir); + // Stale-episode bound (red-team): an episode left open for a long time + // means the pinned backup may predate real data — take a fresh backup. + const episodeFresh = + sidecar.episodeStartedAt !== null && + Date.now() - sidecar.episodeStartedAt >= 0 && + Date.now() - sidecar.episodeStartedAt < 24 * 3600 * 1000; + let receipt: WalRepairReceipt; + try { + receipt = await repairPgliteWal(dataDir, { + reuseBackupPath: episodeFresh ? sidecar.episodeBackupPath ?? undefined : undefined, + }); + } catch (err) { + if (err instanceof WalRepairError) { + // Reset failed AFTER the backup was taken; report the best-effort + // restore's REAL outcome (hardcoding restored:true here made the + // 'failed-restored' message lie when the restore itself failed). + recordRepairAttempt(dataDir, 'failed', err.receipt.backupPath); + return { + status: 'failed', + receipt: err.receipt, + restored: err.restore.restored, + repairError: err.message + + (err.restore.restored ? '' : ` [restore: ${err.restore.detail}]`), + }; + } + // Pre-backup refusal (validation) — the dir was never touched, so there + // is nothing to restore and `restored: true` reads as "dir intact". + recordRepairAttempt(dataDir, 'failed', sidecar.episodeBackupPath); + return { + status: 'failed', + receipt: null, + restored: true, + repairError: String((err as Error)?.message ?? err), + }; + } + + try { + const db = await retryCreate(); + recordRepairAttempt(dataDir, 'repaired', receipt.backupPath); + return { status: 'repaired', db, receipt }; + } catch (retryErr) { + const restore = await restoreWalBackup(receipt); + recordRepairAttempt(dataDir, 'failed', receipt.backupPath); + return { + status: 'failed', + receipt, + restored: restore.restored, + repairError: String((retryErr as Error)?.message ?? retryErr) + + (restore.restored ? '' : ` [restore: ${restore.detail}]`), + }; + } + } catch (err) { + // The seam's never-throw contract is load-bearing (single lock-release site + // in connect()'s catch) — any unexpected error degrades to 'failed'. + // `restored: true` here is honest: repairPgliteWal/restoreWalBackup handle + // their own mutation failures via WalRepairError above, so a throw landing + // HERE happened outside surgery and the dir is untouched (red-team: the + // old restored:false told users to manually restore a nonexistent backup). + try { recordRepairAttempt(dataDir, 'failed', null); } catch { /* best-effort — cooldown still engages when possible */ } + return { + status: 'failed', + receipt: null, + restored: true, + repairError: `unexpected repair-path error: ${String((err as Error)?.message ?? err)}`, + }; + } +} diff --git a/src/core/pglite-resetwal.ts b/src/core/pglite-resetwal.ts new file mode 100644 index 000000000..777fb2321 --- /dev/null +++ b/src/core/pglite-resetwal.ts @@ -0,0 +1,354 @@ +/** + * pg_resetwal for PGLite NodeFS data dirs, in TypeScript. + * + * Ported from electric-sql/pglite PR #994 by @yestheboxer (Apache-2.0, + * https://github.com/electric-sql/pglite/pull/994 — closed upstream as + * "should be a separate tool"; gbrain is that tool). The byte surgery is + * regression-tested upstream (create DB → insert → corrupt WAL → reopen → + * row readable) and deliberately NOT "improved" here. + * + * What it does: for a data dir whose WAL/checkpoint state is torn (unclean + * shutdown — the #223/#1670/#2575 class), rewrite `global/pg_control` with a + * fresh shutdown checkpoint and emit one replacement WAL segment containing + * that checkpoint record, so Postgres-in-WASM can start without replaying the + * torn WAL. Data files are preserved; transactions not checkpointed before + * the corruption are lost (the standard pg_resetwal caveat). + * + * LAYOUT COUPLING: the `OFF` table below is the PostgreSQL 17 ControlFileData + * layout (`PG_CONTROL_VERSION` 1700) that @electric-sql/pglite 0.4.x ships. + * Any pglite bump PAST PG17 must revisit this file together with the + * `./vector` export blocker — see the TODOS.md "pglite upgrade blocker" entry. + * Every unsupported shape throws `WalResetUnsupportedError` (fail-closed). + * + * gbrain adaptation on top of the upstream port: both file writes (the new + * WAL segment AND pg_control) go through tmp-file + fsync(tmp) + rename + + * fsync(parent dir) instead of upstream's in-place `writeFileSynced` — a + * mid-write kill (SIGKILL, `process.exit(124)` from a read-only command + * timeout) leaves old-or-new per file, never a torn file. Write ORDER stays + * upstream's (segment first, control last): the pair is not atomic across + * files, but a kill between them leaves pg_control pointing at old state, so + * startup still fails and the next repair attempt re-runs this (idempotent). + * A torn pair can never claim success. + */ +import { existsSync } from 'node:fs'; +import { mkdir, open, readdir, readFile, rename, unlink } from 'node:fs/promises'; +import { join } from 'node:path'; + +// Exported: pglite-repair.ts validates against the same layout literals — the +// PG17 coupling the TODOS "pglite upgrade blocker" entry says moves together. +export const PG_CONTROL_FILE_SIZE = 8192; +const WAL_SEGMENT_RE = /^[0-9A-F]{24}(?:\.partial)?$/; +/** Is this filename a WAL segment (incl. `.partial`)? Shared layout predicate. */ +export function isWalSegmentName(name: string): boolean { + return WAL_SEGMENT_RE.test(name); +} +const PG_CONTROL_VERSION = 1700; +const DB_SHUTDOWNED = 1; +const XLOG_BLCKSZ = 8192; +const MIN_WAL_SEG_SIZE = 1024 * 1024; +// Postgres-general max is 1GB, but this port targets pglite (ships 16MB +// segments). A corrupt-but-plausible control field must not be able to drive +// a 1GB zero-fill allocation + write on the repair path (perf review) — cap +// at 64MB and fail closed above it. +const MAX_WAL_SEG_SIZE = 64 * 1024 * 1024; +const SIZE_OF_XLOG_LONG_PHD = 40; +const SIZE_OF_XLOG_RECORD = 24; +const SIZE_OF_CHECKPOINT = 88; +const XLOG_PAGE_MAGIC = 0xd116; +const XLP_LONG_HEADER = 0x0002; +const XLOG_CHECKPOINT_SHUTDOWN = 0x00; +const XLR_BLOCK_ID_DATA_SHORT = 255; +const RM_XLOG_ID = 0; + +// PostgreSQL 17 ControlFileData offsets (pglite 0.4.x layout). +const OFF = { + systemIdentifier: 0, + pgControlVersion: 8, + state: 16, + time: 24, + checkPoint: 32, + checkPointCopy: 40, + checkPointCopyRedo: 40, + checkPointCopyThisTimeLineID: 48, + checkPointCopyTime: 104, + minRecoveryPoint: 136, + minRecoveryPointTLI: 144, + backupStartPoint: 152, + backupEndPoint: 160, + backupEndRequired: 168, + walLevel: 172, + walLogHints: 176, + maxConnections: 180, + maxWorkerProcesses: 184, + maxWalSenders: 188, + maxPreparedXacts: 192, + maxLocksPerXact: 196, + trackCommitTimestamp: 200, + xlogBlcksz: 224, + xlogSegSize: 228, + crc: 288, +} as const; + +/** The dir does not look like a PG17 pglite layout — refuse to touch it. */ +export class WalResetUnsupportedError extends Error { + constructor(message: string) { + super(message); + this.name = 'WalResetUnsupportedError'; + } +} + +export interface WalResetResult { + /** Filename of the replacement WAL segment written (24 hex chars). */ + resetSegment: string; + timelineId: number; + walSegSize: number; +} + +const crcTable = new Uint32Array(256); +for (let i = 0; i < 256; i++) { + let crc = i; + for (let j = 0; j < 8; j++) { + crc = crc & 1 ? (crc >>> 1) ^ 0x82f63b78 : crc >>> 1; + } + crcTable[i] = crc >>> 0; +} + +export function crc32c(chunks: Uint8Array[]): number { + let crc = 0xffffffff; + for (const chunk of chunks) { + for (const byte of chunk) { + crc = (crc >>> 8) ^ crcTable[(crc ^ byte) & 0xff]!; + } + } + return (crc ^ 0xffffffff) >>> 0; +} + +function readUInt64LE(buf: Buffer, offset: number): bigint { + return buf.readBigUInt64LE(offset); +} + +function writeUInt64LE(buf: Buffer, value: bigint, offset: number): void { + buf.writeBigUInt64LE(value, offset); +} + +export function parseWalSegNo(fileName: string, walSegSize: number): bigint | null { + if (!/^[0-9A-F]{24}$/.test(fileName)) return null; + const log = BigInt(`0x${fileName.slice(8, 16)}`); + const seg = BigInt(`0x${fileName.slice(16, 24)}`); + return log * (0x100000000n / BigInt(walSegSize)) + seg; +} + +export function xlogFileName(tli: number, segNo: bigint, walSegSize: number): string { + const segmentsPerXlogId = 0x100000000n / BigInt(walSegSize); + const log = segNo / segmentsPerXlogId; + const seg = segNo % segmentsPerXlogId; + return [ + tli.toString(16).toUpperCase().padStart(8, '0'), + log.toString(16).toUpperCase().padStart(8, '0'), + seg.toString(16).toUpperCase().padStart(8, '0'), + ].join(''); +} + +async function unlinkIfExists(path: string): Promise { + await unlink(path).catch((error: NodeJS.ErrnoException) => { + if (error.code !== 'ENOENT') throw error; + }); +} + +/** + * Atomic + durable single-file write: tmp in the same dir → fsync(tmp) → + * rename over the target → fsync(parent dir). A kill at any instant leaves + * either the old file or the new file, never a torn one; the dir fsync makes + * the rename itself survive power loss. (Directory fsync is best-effort — + * some filesystems refuse it; the rename is still atomic without it.) + */ +export async function writeFileAtomicSynced(dir: string, name: string, data: Buffer): Promise { + const tmpName = `.${name}.tmp-${process.pid}`; + const tmpPath = join(dir, tmpName); + // 'wx' (exclusive create, never follows an existing symlink) after clearing + // any stale tmp: a pre-planted symlink at the predictable tmp path must not + // redirect the write (security review). + await unlinkIfExists(tmpPath); + const file = await open(tmpPath, 'wx'); + try { + await file.writeFile(data); + await file.sync(); + } finally { + await file.close(); + } + await rename(tmpPath, join(dir, name)); + try { + const dirHandle = await open(dir, 'r'); + try { + await dirHandle.sync(); + } finally { + await dirHandle.close(); + } + } catch { + // Best-effort: some platforms/filesystems reject fsync on a directory fd. + } +} + +/** + * Reset WAL in place for a PG17 pglite NodeFS data dir. Preserves data files; + * discards the (torn) WAL tail. Throws `WalResetUnsupportedError` when the + * layout does not match what this port understands; throws raw fs errors on + * I/O failure. Idempotent: re-running after a partial attempt converges. + */ +export async function resetWal(rootDir: string): Promise { + await unlinkIfExists(join(rootDir, 'postmaster.pid')); + + let pgVersion: string; + try { + pgVersion = (await readFile(join(rootDir, 'PG_VERSION'), 'utf8')).trim(); + } catch { + throw new WalResetUnsupportedError(`No readable PG_VERSION in ${rootDir}`); + } + if (pgVersion !== '17') { + throw new WalResetUnsupportedError(`Cannot reset WAL for unsupported PG_VERSION ${pgVersion}`); + } + + const controlPath = join(rootDir, 'global', 'pg_control'); + let control: Buffer; + try { + control = Buffer.from(await readFile(controlPath)); + } catch { + throw new WalResetUnsupportedError(`No readable global/pg_control in ${rootDir}`); + } + if (control.length !== PG_CONTROL_FILE_SIZE) { + throw new WalResetUnsupportedError(`Unexpected pg_control size ${control.length}`); + } + if (control.readUInt32LE(OFF.pgControlVersion) !== PG_CONTROL_VERSION) { + throw new WalResetUnsupportedError('Unsupported pg_control version'); + } + // Verify the STORED CRC before trusting (and re-signing) the checkpoint copy + // (adversarial review F6): a torn pg_control with an intact version field + // but garbage checkpoint counters (nextXid/nextOid/...) would otherwise be + // preserved verbatim and laundered under a fresh valid CRC — Postgres then + // starts and corrupts silently (xid-wraparound class). Real pg_resetwal + // refuses on CRC mismatch; so do we → the caller falls to the rebuild rung. + const storedCrc = control.readUInt32LE(OFF.crc); + if (crc32c([control.subarray(0, OFF.crc)]) !== storedCrc) { + throw new WalResetUnsupportedError( + 'pg_control CRC mismatch — the control file itself is damaged; WAL reset ' + + 'would launder corrupt checkpoint counters. Rebuild the brain instead ' + + '(`gbrain reinit-pglite`).', + ); + } + + const walSegSize = control.readUInt32LE(OFF.xlogSegSize); + const xlogBlcksz = control.readUInt32LE(OFF.xlogBlcksz); + if ( + walSegSize < MIN_WAL_SEG_SIZE || + walSegSize > MAX_WAL_SEG_SIZE || + (walSegSize & (walSegSize - 1)) !== 0 || + 0x100000000 % walSegSize !== 0 + ) { + throw new WalResetUnsupportedError(`Unsupported WAL segment size ${walSegSize}`); + } + if (xlogBlcksz !== XLOG_BLCKSZ) { + throw new WalResetUnsupportedError(`Unsupported WAL block size ${xlogBlcksz}`); + } + + const tli = control.readUInt32LE(OFF.checkPointCopyThisTimeLineID); + let newSegNo = readUInt64LE(control, OFF.checkPointCopyRedo) / BigInt(walSegSize); + const walDir = join(rootDir, 'pg_wal'); + // Recreates pg_wal/archive_status when the whole pg_wal dir was renamed + // away into the repair backup (pglite-repair.ts) — resetWal then starts + // from an empty WAL dir and numbers the fresh segment off pg_control alone. + await mkdir(join(walDir, 'archive_status'), { recursive: true }); + for (const file of await readdir(walDir)) { + const segNo = parseWalSegNo(file, walSegSize); + if (segNo !== null && segNo > newSegNo) { + newSegNo = segNo; + } + } + newSegNo += 1n; + + const redo = newSegNo * BigInt(walSegSize) + BigInt(SIZE_OF_XLOG_LONG_PHD); + const now = BigInt(Math.floor(Date.now() / 1000)); + + writeUInt64LE(control, redo, OFF.checkPointCopyRedo); + writeUInt64LE(control, now, OFF.checkPointCopyTime); + control.writeInt32LE(DB_SHUTDOWNED, OFF.state); + writeUInt64LE(control, now, OFF.time); + writeUInt64LE(control, redo, OFF.checkPoint); + writeUInt64LE(control, 0n, OFF.minRecoveryPoint); + control.writeUInt32LE(0, OFF.minRecoveryPointTLI); + writeUInt64LE(control, 0n, OFF.backupStartPoint); + writeUInt64LE(control, 0n, OFF.backupEndPoint); + control.writeUInt8(0, OFF.backupEndRequired); + control.writeInt32LE(0, OFF.walLevel); + control.writeUInt8(0, OFF.walLogHints); + control.writeInt32LE(100, OFF.maxConnections); + control.writeInt32LE(8, OFF.maxWorkerProcesses); + control.writeInt32LE(10, OFF.maxWalSenders); + control.writeInt32LE(0, OFF.maxPreparedXacts); + control.writeInt32LE(64, OFF.maxLocksPerXact); + control.writeUInt8(0, OFF.trackCommitTimestamp); + control.writeUInt32LE(crc32c([control.subarray(0, OFF.crc)]), OFF.crc); + + for (const file of await readdir(walDir)) { + if (isWalSegmentName(file)) { + await unlink(join(walDir, file)); + } + } + + const archiveStatusDir = join(walDir, 'archive_status'); + if (existsSync(archiveStatusDir)) { + for (const file of await readdir(archiveStatusDir)) { + if (/^[0-9A-F]{24}(?:\.partial)?\.(?:ready|done)$/.test(file)) { + await unlink(join(archiveStatusDir, file)); + } + } + } + const walSummaryDir = join(walDir, 'summaries'); + if (existsSync(walSummaryDir)) { + for (const file of await readdir(walSummaryDir)) { + if (/^[0-9A-F]{40}\.summary$/.test(file)) { + await unlink(join(walSummaryDir, file)); + } + } + } + + const wal = Buffer.alloc(walSegSize); + wal.writeUInt16LE(XLOG_PAGE_MAGIC, 0); + wal.writeUInt16LE(XLP_LONG_HEADER, 2); + wal.writeUInt32LE(tli, 4); + writeUInt64LE(wal, redo - BigInt(SIZE_OF_XLOG_LONG_PHD), 8); + wal.writeUInt32LE(0, 16); + writeUInt64LE(wal, readUInt64LE(control, OFF.systemIdentifier), 24); + wal.writeUInt32LE(walSegSize, 32); + wal.writeUInt32LE(XLOG_BLCKSZ, 36); + + const recordOffset = SIZE_OF_XLOG_LONG_PHD; + const recordTotalLength = SIZE_OF_XLOG_RECORD + 2 + SIZE_OF_CHECKPOINT; + wal.writeUInt32LE(recordTotalLength, recordOffset); + wal.writeUInt32LE(0, recordOffset + 4); + writeUInt64LE(wal, 0n, recordOffset + 8); + wal.writeUInt8(XLOG_CHECKPOINT_SHUTDOWN, recordOffset + 16); + wal.writeUInt8(RM_XLOG_ID, recordOffset + 17); + wal.writeUInt16LE(0, recordOffset + 18); + wal.writeUInt8(XLR_BLOCK_ID_DATA_SHORT, recordOffset + SIZE_OF_XLOG_RECORD); + wal.writeUInt8(SIZE_OF_CHECKPOINT, recordOffset + SIZE_OF_XLOG_RECORD + 1); + control.copy( + wal, + recordOffset + SIZE_OF_XLOG_RECORD + 2, + OFF.checkPointCopy, + OFF.checkPointCopy + SIZE_OF_CHECKPOINT, + ); + + const record = wal.subarray(recordOffset, recordOffset + recordTotalLength); + const recordCrc = crc32c([ + record.subarray(SIZE_OF_XLOG_RECORD), + record.subarray(0, 20), + ]); + wal.writeUInt32LE(recordCrc, recordOffset + 20); + + const resetSegment = xlogFileName(tli, newSegNo, walSegSize); + // Segment FIRST, control LAST (upstream order — see header comment). + await writeFileAtomicSynced(walDir, resetSegment, wal); + await writeFileAtomicSynced(join(rootDir, 'global'), 'pg_control', control); + + return { resetSegment, timelineId: tli, walSegSize }; +} diff --git a/test/doctor-pglite-datadir.test.ts b/test/doctor-pglite-datadir.test.ts new file mode 100644 index 000000000..e43adca8c --- /dev/null +++ b/test/doctor-pglite-datadir.test.ts @@ -0,0 +1,240 @@ +/** + * `doctor` pglite_data_dir check (#223 WAL-repair wave) — parallel-safe unit + * coverage of: + * + * - `computePgliteDataDirCheck` — the PURE verdict → Check mapping in + * src/commands/doctor.ts (synthetic PgliteDirDiagnosis inputs, no fs). + * - `OPS_CHECK_NAMES` carrying 'pglite_data_dir' (category routing). + * - `inspectPgliteDataDir` — the read-only diagnoser in + * src/core/pglite-repair.ts, exercised against synthetic on-disk layouts + * in hermetic mkdtemp dirs. + * + * No process.env writes, no PGLite cold starts — safe for the parallel unit + * shards. The command-level (real repair) coverage lives in + * test/pglite-repair-command.serial.test.ts. + */ +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, symlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { computePgliteDataDirCheck } from '../src/commands/doctor.ts'; +import { OPS_CHECK_NAMES } from '../src/core/doctor-categories.ts'; +import { inspectPgliteDataDir } from '../src/core/pglite-repair.ts'; +import type { PgliteDirDiagnosis } from '../src/core/pglite-repair.ts'; + +function tmp(prefix: string): string { + return mkdtempSync(join(tmpdir(), prefix)); +} + +/** Synthetic diagnosis with healthy-looking defaults; override per case. */ +function makeDiagnosis( + overrides: Partial & { verdict: PgliteDirDiagnosis['verdict'] }, +): PgliteDirDiagnosis { + return { + exists: true, + postmasterPid: false, + pgControlOk: true, + pgVersion: '17', + walSegments: [], + lockHeld: false, + lockHolderPid: null, + backupDirs: [], + recentAttempts: [], + detail: 'synthetic diagnosis', + ...overrides, + }; +} + +/** + * A minimal fake PG17 pglite layout that passes `validateWalRepairTarget`: + * PG_VERSION '17', base/ dir, 8192-byte global/pg_control, empty pg_wal/. + */ +function makeFakeLayout(dir: string): void { + mkdirSync(join(dir, 'base'), { recursive: true }); + mkdirSync(join(dir, 'global'), { recursive: true }); + mkdirSync(join(dir, 'pg_wal'), { recursive: true }); + writeFileSync(join(dir, 'PG_VERSION'), '17\n'); + writeFileSync(join(dir, 'global', 'pg_control'), Buffer.alloc(8192)); +} + +describe('computePgliteDataDirCheck — verdict → Check mapping', () => { + const DIR = '/synthetic/brain.pglite'; + + test('every verdict maps to a check named pglite_data_dir with human_only remediation', () => { + const verdicts: PgliteDirDiagnosis['verdict'][] = [ + 'looks-healthy', + 'wal-corruption-likely', + 'locked', + 'missing', + 'unsupported-layout', + ]; + for (const verdict of verdicts) { + const check = computePgliteDataDirCheck(DIR, makeDiagnosis({ verdict })); + expect(check.name).toBe('pglite_data_dir'); + expect(check.remediation_status).toBe('human_only'); + } + }); + + test('wal-corruption-likely → fail, names pglite-repair --dry-run and #223', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ + verdict: 'wal-corruption-likely', + postmasterPid: true, + detail: 'stale postmaster.pid present', + }), + ); + expect(check.status).toBe('fail'); + expect(check.message).toContain('pglite-repair --dry-run'); + expect(check.message).toContain('#223'); + expect(check.message).toContain('gbrain pglite-repair'); + }); + + test('looks-healthy (but connect failed) → fail, points at gbrain pglite-repair', () => { + const check = computePgliteDataDirCheck(DIR, makeDiagnosis({ verdict: 'looks-healthy' })); + expect(check.status).toBe('fail'); + expect(check.message).toContain('gbrain pglite-repair'); + }); + + test('unsupported-layout → fail', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ verdict: 'unsupported-layout', pgControlOk: false, pgVersion: null }), + ); + expect(check.status).toBe('fail'); + expect(check.remediation_status).toBe('human_only'); + }); + + test('locked → warn, names the live holder PID', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ verdict: 'locked', lockHeld: true, lockHolderPid: 12345 }), + ); + expect(check.status).toBe('warn'); + expect(check.message).toContain('12345'); + }); + + test('missing → warn', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ verdict: 'missing', exists: false, pgControlOk: false, pgVersion: null }), + ); + expect(check.status).toBe('warn'); + expect(check.message).toContain(DIR); + }); + + test('recurrence: >=2 failed attempts within 7 days → message points at docs/ENGINES.md', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ + verdict: 'wal-corruption-likely', + postmasterPid: true, + recentAttempts: [ + { ts: Date.now() - 1000, outcome: 'failed' }, + { ts: Date.now() - 2000, outcome: 'failed' }, + ], + }), + ); + expect(check.status).toBe('fail'); + expect(check.message).toContain('docs/ENGINES.md'); + }); + + test('single recent failed attempt does NOT trigger the engine-switch recurrence note', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ + verdict: 'wal-corruption-likely', + postmasterPid: true, + recentAttempts: [{ ts: Date.now() - 1000, outcome: 'failed' }], + }), + ); + expect(check.message).not.toContain('docs/ENGINES.md'); + }); + + test('non-empty backupDirs → message mentions the backup(s)', () => { + const check = computePgliteDataDirCheck( + DIR, + makeDiagnosis({ + verdict: 'wal-corruption-likely', + postmasterPid: true, + backupDirs: [`${DIR}.wal-repair-backup-1700000000000`], + }), + ); + expect(check.message).toContain('backup'); + expect(check.message).toContain(`${DIR}.wal-repair-backup-1700000000000`); + }); +}); + +describe('doctor-categories', () => { + test('OPS_CHECK_NAMES contains pglite_data_dir', () => { + expect(OPS_CHECK_NAMES.has('pglite_data_dir')).toBe(true); + }); +}); + +describe('inspectPgliteDataDir — synthetic on-disk layouts', () => { + test('(a) nonexistent path → missing', () => { + const parent = tmp('gbrain-inspect-a-'); + const diagnosis = inspectPgliteDataDir(join(parent, 'does-not-exist.pglite')); + expect(diagnosis.verdict).toBe('missing'); + expect(diagnosis.exists).toBe(false); + }); + + test('(b) dir without PG_VERSION → unsupported-layout', () => { + const dir = tmp('gbrain-inspect-b-'); + const diagnosis = inspectPgliteDataDir(dir); + expect(diagnosis.verdict).toBe('unsupported-layout'); + expect(diagnosis.pgVersion).toBeNull(); + }); + + test('(c) full fake layout + stale postmaster.pid → wal-corruption-likely', () => { + const dir = join(tmp('gbrain-inspect-c-'), 'brain.pglite'); + makeFakeLayout(dir); + writeFileSync(join(dir, 'postmaster.pid'), '99999\n'); + const diagnosis = inspectPgliteDataDir(dir); + expect(diagnosis.verdict).toBe('wal-corruption-likely'); + expect(diagnosis.postmasterPid).toBe(true); + expect(diagnosis.pgVersion).toBe('17'); + expect(diagnosis.pgControlOk).toBe(true); + }); + + test('(d) full fake layout, no postmaster.pid → looks-healthy', () => { + const dir = join(tmp('gbrain-inspect-d-'), 'brain.pglite'); + makeFakeLayout(dir); + const diagnosis = inspectPgliteDataDir(dir); + expect(diagnosis.verdict).toBe('looks-healthy'); + expect(diagnosis.postmasterPid).toBe(false); + expect(diagnosis.lockHeld).toBe(false); + }); + + test('(e) fake layout + live .gbrain-lock holder → locked with the holder PID', () => { + const dir = join(tmp('gbrain-inspect-e-'), 'brain.pglite'); + makeFakeLayout(dir); + mkdirSync(join(dir, '.gbrain-lock'), { recursive: true }); + writeFileSync( + join(dir, '.gbrain-lock', 'lock'), + JSON.stringify({ + pid: process.pid, // this test process — provably alive + acquired_at: Date.now(), + refreshed_at: Date.now(), + command: 'gbrain embed', + subcommand: 'embed', + }), + ); + const diagnosis = inspectPgliteDataDir(dir); + expect(diagnosis.verdict).toBe('locked'); + expect(diagnosis.lockHeld).toBe(true); + expect(diagnosis.lockHolderPid).toBe(process.pid); + }); + + test('(f) symlinked data dir → unsupported-layout (rename-based repair refuses symlinks)', () => { + const parent = tmp('gbrain-inspect-f-'); + const real = join(parent, 'real.pglite'); + makeFakeLayout(real); + const link = join(parent, 'link.pglite'); + symlinkSync(real, link); + const diagnosis = inspectPgliteDataDir(link); + expect(diagnosis.verdict).toBe('unsupported-layout'); + expect(diagnosis.detail).toContain('symlink'); + }); +}); diff --git a/test/e2e/pglite-cli-exit.serial.test.ts b/test/e2e/pglite-cli-exit.serial.test.ts index 9291aee37..c90ece9b7 100644 --- a/test/e2e/pglite-cli-exit.serial.test.ts +++ b/test/e2e/pglite-cli-exit.serial.test.ts @@ -33,6 +33,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; import { spawn, spawnSync } from 'child_process'; +import { randomBytes } from 'crypto'; import { cpSync, mkdirSync, @@ -96,9 +97,12 @@ beforeAll(() => { delete runEnv.ANTHROPIC_API_KEY; delete runEnv.GOOGLE_API_KEY; + // NOTE: init grew strict flag validation (#2201); `--repo`/`--yes` were + // never real init flags (previously silently ignored). The repo is wired + // through `sync --repo` below, matching the other e2e suites. const initResult = spawnSync( SHIM_PATH, - ['init', '--pglite', '--repo', repoSourceDir, '--no-embedding', '--yes'], + ['init', '--pglite', '--no-embedding', '--non-interactive'], { cwd: REPO_ROOT, env: runEnv, @@ -409,6 +413,49 @@ describe('#2084 — explicit-exit teardown: every swept site exits clean, exit c }, 30_000); }); +describe('WAL-repair wave — corrupt persistent brain, auto-repair off: owned exit 1 (#2084 class)', () => { + test('gbrain status on a torn-WAL brain with GBRAIN_PGLITE_WAL_REPAIR=off exits 1 (not 0, not 99)', async () => { + // Fixture: a fake-but-layout-valid PG17 pglite data dir whose control + + // WAL state is garbage, so PGlite.create aborts. With auto-repair + // disabled the CLI must fail LOUDLY through the owned verdict channel: + // real process exit 1 — never 0 (silent success over a broken brain), + // never 99 (Emscripten's hijacked process.exitCode, the #2084 class). + const corruptHome = mkdtempSync(join(tmpdir(), 'gbrain-pglite-corrupt-')); + try { + const dataDir = join(corruptHome, 'brain.pglite'); + // GBRAIN_HOME is a parent dir: config lives at /.gbrain/config.json. + mkdirSync(join(corruptHome, '.gbrain'), { recursive: true }); + writeFileSync( + join(corruptHome, '.gbrain', 'config.json'), + JSON.stringify({ engine: 'pglite', database_path: dataDir }, null, 2) + '\n', + 'utf-8', + ); + mkdirSync(join(dataDir, 'base'), { recursive: true }); + mkdirSync(join(dataDir, 'global'), { recursive: true }); + mkdirSync(join(dataDir, 'pg_wal'), { recursive: true }); + writeFileSync(join(dataDir, 'PG_VERSION'), '17\n', 'utf-8'); + writeFileSync(join(dataDir, 'global', 'pg_control'), randomBytes(8192)); + writeFileSync(join(dataDir, 'pg_wal', '000000010000000000000001'), randomBytes(1024)); + + const { code, stdout, stderr, durationMs } = await runWithTimeout( + ['status'], + 30_000, + { GBRAIN_HOME: corruptHome, GBRAIN_PGLITE_WAL_REPAIR: 'off' }, + ); + if (code !== 1) { + throw new Error( + `expected exit 1, got ${code}; duration=${durationMs}ms\n` + + `STDOUT:\n${stdout}\nSTDERR:\n${stderr}`, + ); + } + expect(code).toBe(1); + expect(stdout + stderr).toContain('PGLite failed to initialize'); + } finally { + rmSync(corruptHome, { recursive: true, force: true }); + } + }, 60_000); +}); + describe('v0.41.8.0 — daemon survival (regression guard for narrow force-exit)', () => { test('gbrain serve --http stays alive past the timeout window', async () => { // Pick a likely-free ephemeral port. We're testing "still alive diff --git a/test/fix-wave-structural.test.ts b/test/fix-wave-structural.test.ts index 1b734617c..3207f7802 100644 --- a/test/fix-wave-structural.test.ts +++ b/test/fix-wave-structural.test.ts @@ -247,7 +247,41 @@ describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => { test('pglite-engine.ts connect catch block routes through the classifier', () => { const src = readFileSync('src/core/pglite-engine.ts', 'utf8'); expect(src).toMatch(/classifyPgliteInitError\(original\)/); - expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/); + // WAL-repair wave: the call gained platform + repair-context args, so pin + // only the (verdict, original, …) prefix — the routing seam, not the arity. + expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original/); + }); +}); + +describe('WAL-repair wave structural pins (#223/#2575)', () => { + test('connect() catch wires the WAL auto-repair seam', () => { + const src = readFileSync('src/core/pglite-engine.ts', 'utf8'); + expect(src).toMatch(/attemptWalRepairAndRetry\(/); + }); + + test('the repair-retry lambda stays inside the #2084 exitCode guard', () => { + // The retry re-runs PGlite.create; unguarded, Emscripten would hijack + // process.exitCode on the retry path exactly as it did on the first + // attempt (the #2084 class). Pin the wrap at the seam call-site. + const src = readFileSync('src/core/pglite-engine.ts', 'utf8'); + expect(src).toMatch(/attemptWalRepairAndRetry\([\s\S]{0,300}preservingProcessExitCode/); + }); + + test('no bare PGlite.create outside the wrapped engine sites', () => { + // The repair/resetwal modules take the retry as a callback — if either + // grew its own PGlite.create call it would bypass BOTH the exitCode + // guard and the single-writer lock. + const repair = readFileSync('src/core/pglite-repair.ts', 'utf8'); + const resetwal = readFileSync('src/core/pglite-resetwal.ts', 'utf8'); + expect(repair).not.toMatch(/PGlite\.create/); + expect(resetwal).not.toMatch(/PGlite\.create/); + }); + + test('pglite-resetwal.ts carries the upstream attribution', () => { + // The reset-WAL sequence mirrors upstream electric-sql/pglite PR #994; + // the pointer is the audit trail for future divergence. + const resetwal = readFileSync('src/core/pglite-resetwal.ts', 'utf8'); + expect(resetwal).toContain('electric-sql/pglite/pull/994'); }); }); diff --git a/test/pglite-init-classifier.test.ts b/test/pglite-init-classifier.test.ts index d3259af95..d26d5b31e 100644 --- a/test/pglite-init-classifier.test.ts +++ b/test/pglite-init-classifier.test.ts @@ -31,9 +31,33 @@ describe('classifyPgliteInitError', () => { expect(classifyPgliteInitError(msg)).toBe('bunfs'); }); - test('macos-26-3 verdict for the existing #223 signature', () => { + test('wasm-abort verdict for the existing #223 signature', () => { const msg = 'abort() called from wasm runtime on macOS 26.3 build'; - expect(classifyPgliteInitError(msg)).toBe('macos-26-3'); + expect(classifyPgliteInitError(msg)).toBe('wasm-abort'); + }); + + // WAL-repair wave: THE real production message from a torn-WAL Emscripten + // abort — no "runtime"/"wasm" in it, so the legacy arms let it fall + // through to 'unknown' (which is exactly how #223 got misdiagnosed). + test('wasm-abort verdict for the bare Emscripten Aborted() message', () => { + expect( + classifyPgliteInitError('Aborted(). Build with -sASSERTIONS for more info.'), + ).toBe('wasm-abort'); + }); + + test('wasm-abort verdict for RuntimeError-prefixed Aborted()', () => { + expect( + classifyPgliteInitError('RuntimeError: Aborted(). Build with -sASSERTIONS for more info.'), + ).toBe('wasm-abort'); + }); + + test('wasm-abort verdict for the generic RuntimeError: unreachable trap', () => { + expect(classifyPgliteInitError('RuntimeError: unreachable')).toBe('wasm-abort'); + }); + + test('bunfs still wins when a wasm-abort marker co-occurs (bunfs arm is first)', () => { + const msg = "RuntimeError: Aborted(). ENOENT open '/$$bunfs/root/pglite.data'"; + expect(classifyPgliteInitError(msg)).toBe('bunfs'); }); test('unknown verdict for generic / unrecognized errors', () => { @@ -64,8 +88,9 @@ describe('classifyPgliteInitError', () => { }); test('corrupt verdict beats the wasm-runtime match (58P01 wins over "wasm runtime")', () => { - // A message mentioning both must classify as corrupt, not macos-26-3 — - // recovery guidance, not the wrong macOS-WASM hint. + // A message mentioning both must classify as corrupt, not wasm-abort — + // recovery guidance, not the WAL-repair hint (WAL repair cannot fix + // catalog corruption). expect(classifyPgliteInitError('wasm runtime: 58P01 internal_load_library')).toBe('corrupt'); }); }); @@ -82,14 +107,67 @@ describe('buildPgliteInitErrorMessage — hint routing', () => { expect(msg).not.toContain('issues/223'); }); - test('macos-26-3 verdict surfaces the #223 link AND original error', () => { - const msg = buildPgliteInitErrorMessage('macos-26-3', original); + test('wasm-abort verdict names torn WAL as the cause, keeps the #223 link, AND original error', () => { + const msg = buildPgliteInitErrorMessage('wasm-abort', original); + // The re-diagnosis is the load-bearing copy: corrupt WAL after an unclean + // shutdown, explicitly NOT the historical macOS-WASM attribution. + expect(msg).toContain('NOT a macOS WASM bug'); expect(msg).toContain('https://github.com/garrytan/gbrain/issues/223'); - expect(msg).toContain('macOS 26.3'); - expect(msg).toContain(original); + // Full recovery ladder: in-place repair → rebuild → switch engines. + expect(msg).toContain('gbrain pglite-repair --dry-run'); + expect(msg).toContain('reinit-pglite'); + expect(msg).toContain('docs/ENGINES.md'); + expect(msg).toContain('gbrain doctor'); + expect(msg).toContain(`Original error: ${original}`); expect(msg).not.toContain('Bun vfs'); }); + // WAL-repair wave: the 4th param folds what auto-repair did (or why it + // didn't run) into the hint so the message never lies about the state of + // the data dir. + test('wasm-abort + {repair: disabled} names the off switch', () => { + const msg = buildPgliteInitErrorMessage('wasm-abort', original, 'darwin', { repair: 'disabled' }); + expect(msg).toContain('GBRAIN_PGLITE_WAL_REPAIR=off'); + expect(msg).toContain(original); + }); + + test('wasm-abort + {repair: failed-restored} says RESTORED and names the backup path', () => { + const msg = buildPgliteInitErrorMessage('wasm-abort', original, 'darwin', { + repair: 'failed-restored', + backupPath: '/x/b', + }); + expect(msg).toContain('RESTORED'); + expect(msg).toContain('/x/b'); + expect(msg).toContain(original); + }); + + test('wasm-abort + {repair: failed-not-restored} says RESET state, backup path, restore manually', () => { + const msg = buildPgliteInitErrorMessage('wasm-abort', original, 'darwin', { + repair: 'failed-not-restored', + backupPath: '/x/b', + }); + expect(msg).toContain('RESET state'); + expect(msg).toContain('/x/b'); + expect(msg.toLowerCase()).toContain('restore manually'); + expect(msg).toContain(original); + }); + + test('wasm-abort + {repair: in-memory} says there is no stored state to repair', () => { + const msg = buildPgliteInitErrorMessage('wasm-abort', original, 'darwin', { repair: 'in-memory' }); + expect(msg).toContain('in-memory'); + expect(msg).toContain(original); + }); + + test('wasm-abort + {repair: skipped-live-writer} surfaces the skip detail verbatim', () => { + const detail = 'the data-dir lock was reaped from pid 4242 (SENTINEL-LIVE-WRITER)'; + const msg = buildPgliteInitErrorMessage('wasm-abort', original, 'darwin', { + repair: 'skipped-live-writer', + detail, + }); + expect(msg).toContain(detail); + expect(msg).toContain(original); + }); + // #2674: the unknown-verdict hint is platform-gated. The macOS 26.3 // attribution (#223) only appears on darwin; elsewhere the hint names // the causes that are actually plausible off-macOS. @@ -97,6 +175,12 @@ describe('buildPgliteInitErrorMessage — hint routing', () => { const msg = buildPgliteInitErrorMessage('unknown', original, 'darwin'); expect(msg).toContain('gbrain doctor'); expect(msg).toContain('issues/223'); + // WAL-repair wave: the darwin branch is reframed to the real root cause + // behind the #223 reports (torn WAL from unclean shutdown) and offers the + // mutation-free diagnosis command. + expect(msg).toContain('corrupt WAL/checkpoint state'); + expect(msg).toContain('unclean'); + expect(msg).toContain('gbrain pglite-repair --dry-run'); expect(msg).toContain(original); }); @@ -115,12 +199,15 @@ describe('buildPgliteInitErrorMessage — hint routing', () => { const msg = buildPgliteInitErrorMessage('corrupt', original); expect(msg).toContain('gbrain reinit-pglite'); expect(msg).toContain('corrupted'); + // WAL-repair wave: the dry-run diagnosis is offered (report-only — WAL + // repair cannot fix catalog corruption, and the copy says so). + expect(msg).toContain('gbrain pglite-repair --dry-run'); expect(msg).toContain(original); expect(msg).not.toContain('issues/223'); }); test('all verdicts produce the canonical header line', () => { - for (const v of ['bunfs', 'macos-26-3', 'corrupt', 'unknown'] as const) { + for (const v of ['bunfs', 'wasm-abort', 'corrupt', 'unknown'] as const) { const msg = buildPgliteInitErrorMessage(v, original); expect(msg.startsWith('PGLite failed to initialize its WASM runtime.')).toBe(true); } @@ -145,6 +232,27 @@ describe('stringifyPgliteInitError — non-Error rejections (#2674)', () => { expect(stringifyPgliteInitError(null)).toBe('null'); expect(stringifyPgliteInitError(undefined)).toBe('undefined'); }); + + // WAL-repair wave: Emscripten's FS layer throws message-LESS objects (e.g. + // `ErrnoError { name: 'ErrnoError', errno: 20 }` when the data dir is a + // symlink NODEFS refuses to mount) — never "[object Object]". + test('message-less ErrnoError-shaped object yields name + errno', () => { + expect(stringifyPgliteInitError({ name: 'ErrnoError', errno: 20 })).toBe('ErrnoError (errno 20)'); + }); + + test('message-less nameless object with other props yields its JSON', () => { + expect(stringifyPgliteInitError({ code: 'ENOENT' })).toBe('{"code":"ENOENT"}'); + }); + + test('message-less object with a name and serializable props yields name-prefixed JSON', () => { + expect(stringifyPgliteInitError({ name: 'Weird' })).toBe('Weird: {"name":"Weird"}'); + }); + + test('circular object with a name falls back to the bare name (JSON.stringify throws)', () => { + const c: Record = { name: 'Circ' }; + c.self = c; + expect(stringifyPgliteInitError(c)).toBe('Circ'); + }); }); describe('#1340 reproducer — exact reporter error string maps to bunfs', () => { diff --git a/test/pglite-lock.test.ts b/test/pglite-lock.test.ts index 0f8c545a1..6d2782003 100644 --- a/test/pglite-lock.test.ts +++ b/test/pglite-lock.test.ts @@ -1,5 +1,5 @@ import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; -import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs'; +import { mkdirSync, mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs'; import { join } from 'path'; import { tmpdir } from 'os'; import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock'; @@ -271,3 +271,98 @@ describe('pglite-lock #2058 heartbeat + steal-grace', () => { expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(false); }); }); + +describe('pglite-lock reap classification (WAL-repair wave)', () => { + // Unique per-test tmpdirs: the reap marker lands at `${dataDir}.lock-reap.json` + // — a SIBLING of the data dir — so each test gets its own parent to rm. + function freshDataDir(): { parent: string; dataDir: string } { + const parent = mkdtempSync(join(tmpdir(), 'gbrain-lock-reap-')); + return { parent, dataDir: join(parent, 'data') }; + } + + /** + * A PID that provably belongs to no live process: spawn a short-lived child, + * wait for it (spawnSync reaps it), then verify kill(pid, 0) throws. Retries + * to dodge instant PID reuse. + */ + function deadPid(): number { + for (let attempt = 0; attempt < 5; attempt++) { + const proc = Bun.spawnSync(['bash', '-c', 'exit 0']); + const pid = proc.pid; + try { + process.kill(pid, 0); // still alive/visible → PID reused, try again + } catch { + return pid; + } + } + throw new Error('could not obtain a provably-dead PID after 5 spawns'); + } + + test('corrupt lock file: reaped acquisition + persisted .lock-reap.json marker', async () => { + const { parent, dataDir } = freshDataDir(); + try { + const lockDir = join(dataDir, '.gbrain-lock'); + mkdirSync(lockDir, { recursive: true }); + writeFileSync(join(lockDir, 'lock'), 'not json {{{'); // holder liveness UNKNOWABLE + + const lock = await acquireLock(dataDir, { timeoutMs: 5000 }); + try { + expect(lock.acquired).toBe(true); + expect(lock.reaped).toBe(true); + // Unknowable-liveness reap is persisted cross-process for the repair gate. + expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(true); + const marker = JSON.parse(readFileSync(`${dataDir}.lock-reap.json`, 'utf-8')); + expect(typeof marker.ts).toBe('number'); + expect(marker.by).toBe(process.pid); + } finally { + await releaseLock(lock); + } + } finally { + rmSync(parent, { recursive: true, force: true }); + } + }); + + test('clean acquisition: reaped falsy, no .lock-reap.json marker', async () => { + const { parent, dataDir } = freshDataDir(); + try { + const lock = await acquireLock(dataDir, { timeoutMs: 5000 }); + try { + expect(lock.acquired).toBe(true); + expect(lock.reaped).toBeFalsy(); + expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(false); + } finally { + await releaseLock(lock); + } + } finally { + rmSync(parent, { recursive: true, force: true }); + } + }); + + test('dead-PID lock: reaped acquisition but NO marker (affirmative ESRCH verdict)', async () => { + const { parent, dataDir } = freshDataDir(); + try { + const lockDir = join(dataDir, '.gbrain-lock'); + mkdirSync(lockDir, { recursive: true }); + const now = Date.now(); + writeFileSync(join(lockDir, 'lock'), JSON.stringify({ + pid: deadPid(), + acquired_at: now - 60_000, + refreshed_at: now - 60_000, + command: 'gbrain embed', + subcommand: 'embed', + })); + + const lock = await acquireLock(dataDir, { timeoutMs: 5000 }); + try { + expect(lock.acquired).toBe(true); + expect(lock.reaped).toBe(true); + // Dead-PID reaps deliberately do NOT quarantine the next acquirer. + expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(false); + } finally { + await releaseLock(lock); + } + } finally { + rmSync(parent, { recursive: true, force: true }); + } + }, 30_000); +}); diff --git a/test/pglite-repair-command.serial.test.ts b/test/pglite-repair-command.serial.test.ts new file mode 100644 index 000000000..4ad0e850b --- /dev/null +++ b/test/pglite-repair-command.serial.test.ts @@ -0,0 +1,418 @@ +/** + * `gbrain pglite-repair` command surface (#223 WAL-repair wave) — SERIAL: + * the happy path does a real persistent-PGLite cold start (create → corrupt + * pg_wal → repair in place → reconnect), which is too heavy + lock-contended + * for the parallel unit shards. + * + * `runPgliteRepair` is imported directly (no process spawns); stdout/stderr + * are captured by spying console.log/console.error per test and restored in + * finally. Every case uses `--path ` so the user's real + * brain and config are never touched, and `--json` so assertions parse a + * machine receipt instead of prose. + * + * Refusal-order note (cases 5-7): the fake layout deliberately PASSES + * `validateWalRepairTarget` (PG_VERSION 17 + base/ + 8192-byte pg_control) so + * the command reaches its lock gates; an actual repair on the garbage + * pg_control would fail in resetWal, but all three cases must refuse BEFORE + * repair — asserting the refused_* codes proves the ordering. + */ +import { describe, test, expect } from 'bun:test'; +import { + existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, join } from 'node:path'; + +import { runPgliteRepair } from '../src/commands/pglite-repair.ts'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { withEnv } from './helpers/with-env.ts'; + +function tmp(prefix: string): string { + return mkdtempSync(join(tmpdir(), prefix)); +} + +/** Fake PG17 layout that passes validateWalRepairTarget (see file header). */ +function makeFakeLayout(dir: string): void { + mkdirSync(join(dir, 'base'), { recursive: true }); + mkdirSync(join(dir, 'global'), { recursive: true }); + mkdirSync(join(dir, 'pg_wal'), { recursive: true }); + writeFileSync(join(dir, 'PG_VERSION'), '17\n'); + writeFileSync(join(dir, 'global', 'pg_control'), Buffer.alloc(8192)); +} + +function writeLockFile(dir: string, lock: Record): string { + const lockDir = join(dir, '.gbrain-lock'); + mkdirSync(lockDir, { recursive: true }); + writeFileSync(join(lockDir, 'lock'), JSON.stringify(lock), { mode: 0o644 }); + return lockDir; +} + +interface Captured { + logs: string[]; + errors: string[]; + restore: () => void; +} + +/** Spy console.log/console.error; caller MUST call restore() in finally. */ +function captureConsole(): Captured { + const logs: string[] = []; + const errors: string[] = []; + const origLog = console.log; + const origErr = console.error; + console.log = (...args: unknown[]) => { logs.push(args.map(String).join(' ')); }; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + return { + logs, + errors, + restore: () => { + console.log = origLog; + console.error = origErr; + }, + }; +} + +/** Last parseable JSON object line from captured console.log output. */ +function parseJsonLine(logs: string[]): Record { + for (let i = logs.length - 1; i >= 0; i--) { + const line = logs[i].trim(); + if (!line.startsWith('{')) continue; + try { + return JSON.parse(line) as Record; + } catch { /* not this line — keep looking */ } + } + throw new Error(`no JSON line in captured output: ${JSON.stringify(logs)}`); +} + +/** + * A PID that provably belongs to no live process: spawn a short-lived child, + * wait for it (spawnSync reaps it), then verify kill(pid, 0) throws. Retries + * to dodge instant PID reuse. + */ +function deadPid(): number { + for (let attempt = 0; attempt < 5; attempt++) { + const proc = Bun.spawnSync(['bash', '-c', 'exit 0']); + const pid = proc.pid; + try { + process.kill(pid, 0); // still alive/visible → PID reused, try again + } catch { + return pid; + } + } + throw new Error('could not obtain a provably-dead PID after 5 spawns'); +} + +const backupDirsBeside = (dir: string): string[] => + readdirSync(join(dir, '..')).filter((n) => n.startsWith(`${basename(dir)}.wal-repair-backup-`)); + +describe('gbrain pglite-repair — dry-run is strictly read-only', () => { + test('1. dry-run on a corrupt-ish layout: exit 0, JSON diagnosis, zero mutation', async () => { + const parent = tmp('gbrain-repair-dry-'); + const dir = join(parent, 'brain.pglite'); + makeFakeLayout(dir); + writeFileSync(join(dir, 'postmaster.pid'), '99999\n'); // unclean-shutdown marker + const parentBefore = readdirSync(parent).sort(); + const dirBefore = readdirSync(dir).sort(); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--dry-run', '--path', dir, '--json']); + } finally { + cap.restore(); + } + + expect(rc).toBe(0); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('ok'); + expect(out.action).toBe('dry-run'); + expect(out.data_dir).toBe(dir); + expect(out.validation.ok).toBe(true); + expect(out.diagnosis.verdict).toBeDefined(); + expect(out.diagnosis.verdict).toBe('wal-corruption-likely'); + + // Read-only: no backup dirs, no sidecar, nothing added or removed. + expect(backupDirsBeside(dir)).toEqual([]); + expect(existsSync(`${dir}.wal-repair-attempt.json`)).toBe(false); + expect(readdirSync(parent).sort()).toEqual(parentBefore); + expect(readdirSync(dir).sort()).toEqual(dirBefore); + expect(existsSync(join(dir, 'postmaster.pid'))).toBe(true); + }); + + test('2. dry-run on a missing path: exit 0, validation.ok false, path NOT created', async () => { + const missing = join(tmp('gbrain-repair-dry-missing-'), 'never-created.pglite'); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--dry-run', '--path', missing, '--json']); + } finally { + cap.restore(); + } + + expect(rc).toBe(0); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('ok'); + expect(out.action).toBe('dry-run'); + expect(out.validation.ok).toBe(false); + expect(out.diagnosis.verdict).toBe('missing'); + expect(existsSync(missing)).toBe(false); + }); +}); + +describe('gbrain pglite-repair — refusals (validate before lock, never mkdir a typo)', () => { + test('3. non-dry-run on a missing path: exit 1, refused_missing-dir, path NOT created', async () => { + const missing = join(tmp('gbrain-repair-missing-'), 'typo.pglite'); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', missing, '--yes', '--json']); + } finally { + cap.restore(); + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('refused_missing-dir'); + // validate-before-lock: acquireLock would have mkdir'd the dir. + expect(existsSync(missing)).toBe(false); + }); + + test('5. live lock holder: exit 1, refused_locked, no repair attempted', async () => { + const dir = join(tmp('gbrain-repair-locked-'), 'brain.pglite'); + makeFakeLayout(dir); + const lockDir = writeLockFile(dir, { + pid: process.pid, // this test process — provably alive + acquired_at: Date.now(), + refreshed_at: Date.now(), + command: 'gbrain embed', + subcommand: 'embed', + }); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--yes', '--json']); + } finally { + cap.restore(); + rmSync(lockDir, { recursive: true, force: true }); + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('refused_locked'); + // Refused BEFORE repair: no backup dir, no sidecar. + expect(backupDirsBeside(dir)).toEqual([]); + expect(existsSync(`${dir}.wal-repair-attempt.json`)).toBe(false); + }); + + test('6. reaped (dead-PID) lock: exit 1, refused_reaped_lock', async () => { + const dir = join(tmp('gbrain-repair-reaped-'), 'brain.pglite'); + makeFakeLayout(dir); + writeLockFile(dir, { + pid: deadPid(), // provably dead — acquireLock reaps it, then refuses + acquired_at: Date.now() - 60_000, + refreshed_at: Date.now() - 60_000, + command: 'gbrain embed', + subcommand: 'embed', + }); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--yes', '--json']); + } finally { + cap.restore(); + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('refused_reaped_lock'); + // Refused BEFORE repair: no backup dir, no sidecar. + expect(backupDirsBeside(dir)).toEqual([]); + expect(existsSync(`${dir}.wal-repair-attempt.json`)).toBe(false); + }, 30_000); + + test('7. live serve holder: exit 1, refused_locked names the PID', async () => { + const dir = join(tmp('gbrain-repair-serve-'), 'brain.pglite'); + makeFakeLayout(dir); + const lockDir = writeLockFile(dir, { + pid: process.pid, // alive — the pre-lock diagnosis catches it as 'locked' + acquired_at: Date.now(), + refreshed_at: Date.now(), + command: 'gbrain serve', + subcommand: 'serve', + }); + + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--yes', '--json']); + } finally { + cap.restore(); + rmSync(lockDir, { recursive: true, force: true }); + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('refused_locked'); + expect(out.message).toContain(String(process.pid)); + expect(backupDirsBeside(dir)).toEqual([]); + }); +}); + +describe('gbrain pglite-repair — TTY + config gates', () => { + test('8. non-TTY without --yes refuses: exit 1, no_tty_no_yes, zero mutation', async () => { + const dir = join(tmp('gbrain-repair-notty-'), 'brain.pglite'); + makeFakeLayout(dir); + + // Pin stdin to non-TTY: under `bun test` in a terminal stdin can still be + // a TTY, which would route into the interactive confirm instead. + const origTty = Object.getOwnPropertyDescriptor(process.stdin, 'isTTY'); + Object.defineProperty(process.stdin, 'isTTY', { value: false, configurable: true }); + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--json']); // no --yes + } finally { + cap.restore(); + if (origTty) Object.defineProperty(process.stdin, 'isTTY', origTty); + else delete (process.stdin as unknown as Record).isTTY; + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('no_tty_no_yes'); + // Refused BEFORE any surgery: no backup dir, no sidecar. + expect(backupDirsBeside(dir)).toEqual([]); + expect(existsSync(`${dir}.wal-repair-attempt.json`)).toBe(false); + }); + + test('9. no --path with a non-pglite configured engine: exit 1, not_pglite', async () => { + // Hermetic GBRAIN_HOME (same convention as apply-migrations-pglite-spawn): + // configDir() appends '.gbrain', so the config lands at /.gbrain/. + const home = tmp('gbrain-repair-home-'); + mkdirSync(join(home, '.gbrain'), { recursive: true }); + writeFileSync( + join(home, '.gbrain', 'config.json'), + JSON.stringify({ engine: 'postgres', database_url: 'postgresql://localhost:5432/x' }) + '\n', + ); + + const cap = captureConsole(); + let rc: number; + try { + rc = await withEnv({ GBRAIN_HOME: home }, () => runPgliteRepair(['--yes', '--json'])); + } finally { + cap.restore(); + } + + expect(rc).toBe(1); + const out = parseJsonLine(cap.logs); + expect(out.status).toBe('error'); + expect(out.code).toBe('not_pglite'); + expect(out.message).toContain('--path'); + }); +}); + +describe('gbrain pglite-repair — the happy path (real PGLite)', () => { + test('4. corrupt pg_wal → repair in place → data survives, no auto-repair on reconnect', async () => { + const dir = join(tmp('gbrain-repair-happy-'), 'brain.pglite'); + + // 1) Real brain with a probe row, closed cleanly. + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dir }); + try { + await engine.executeRaw('CREATE TABLE repair_probe(id int)'); + await engine.executeRaw('INSERT INTO repair_probe VALUES (42)'); + } finally { + await engine.disconnect(); + } + + // 2) Corrupt every WAL segment with garbage. + const walDir = join(dir, 'pg_wal'); + const segments = readdirSync(walDir).filter((f) => /^[0-9A-F]{24}$/.test(f)); + expect(segments.length).toBeGreaterThan(0); // sanity: there IS WAL to corrupt + for (const seg of segments) { + writeFileSync(join(walDir, seg), Buffer.alloc(1024, 0xff)); + } + + // 3) Repair in place. + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--yes', '--json']); + } finally { + cap.restore(); + } + expect(rc).toBe(0); + const receipt = parseJsonLine(cap.logs); + expect(receipt.status).toBe('ok'); + expect(receipt.action).toBe('repaired'); + expect(receipt.data_dir).toBe(dir); + expect(receipt.reset_segment).toMatch(/^[0-9A-F]{24}$/); + expect(existsSync(receipt.backup_path)).toBe(true); + expect(existsSync(join(receipt.backup_path, 'pg_wal'))).toBe(true); + + // 4) The repaired dir opens WITHOUT auto-repair firing, data intact. + const engine2 = new PGLiteEngine(); + await engine2.connect({ database_path: dir }); + try { + expect(engine2.walRepairReceipt).toBeNull(); + const rows = await engine2.executeRaw<{ id: number }>('SELECT id FROM repair_probe'); + expect(rows.length).toBe(1); + expect(rows[0].id).toBe(42); + } finally { + await engine2.disconnect(); + } + }, 180_000); +}); + +describe('gbrain pglite-repair — argument + quarantine hardening (adversarial fixes)', () => { + test('unknown flag is rejected (exit 2), not silently ignored on a destructive command', async () => { + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--dry-rnu', '--yes', '--json']); + } finally { + cap.restore(); + } + expect(rc).toBe(2); + expect(parseJsonLine(cap.logs).code).toBe('unknown_flag'); + }); + + test('--path with no value is rejected (does NOT retarget the default brain)', async () => { + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--yes', '--json', '--path']); + } finally { + cap.restore(); + } + expect(rc).toBe(2); + expect(parseJsonLine(cap.logs).code).toBe('unknown_flag'); + }); + + test('the command honors the cross-process reap quarantine (F3: a second --yes cannot bypass it)', async () => { + const dir = join(mkdtempSync(join(tmpdir(), 'pgrepaircmd-')), 'brain.pglite'); + makeFakeLayout(dir); + // A fresh corrupt-lock reap marker from a prior run — the possibly-live + // writer it protects must not be repaired under. + writeFileSync(`${dir}.lock-reap.json`, JSON.stringify({ ts: Date.now(), by: 999999 }), { mode: 0o644 }); + const cap = captureConsole(); + let rc: number; + try { + rc = await runPgliteRepair(['--path', dir, '--yes', '--json']); + } finally { + cap.restore(); + } + expect(rc).toBe(1); + expect(parseJsonLine(cap.logs).code).toBe('refused_reap_quarantine'); + // No surgery: no backup dir created. + expect(readdirSync(join(dir, '..')).some((f) => f.includes('.wal-repair-backup-'))).toBe(false); + }); +}); diff --git a/test/pglite-repair.test.ts b/test/pglite-repair.test.ts new file mode 100644 index 000000000..7faafda4c --- /dev/null +++ b/test/pglite-repair.test.ts @@ -0,0 +1,548 @@ +/** + * Unit tests for the WAL-repair orchestrator (src/core/pglite-repair.ts): + * validation, rename-based backup, overwrite-order restore + mtime guard, + * cooldown sidecar, episode-scoped retention, and the never-throws engine + * seam (attemptWalRepairAndRetry) with injected retryCreate — no real PGLite. + * + * The real-engine regression (corrupt a real brain → connect() auto-repairs → + * row readable) lives in test/pglite-wal-repair.serial.test.ts. + */ +import { describe, test, expect } from 'bun:test'; +import { + mkdtempSync, mkdirSync, writeFileSync, existsSync, readFileSync, readdirSync, + symlinkSync, rmSync, utimesSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { withEnv } from './helpers/with-env.ts'; +import { xlogFileName, crc32c } from '../src/core/pglite-resetwal.ts'; +import { + validateWalRepairTarget, + inspectPgliteDataDir, + repairPgliteWal, + restoreWalBackup, + attemptWalRepairAndRetry, + readRepairSidecar, + recordRepairAttempt, + repairCooldownActive, + listRepairBackups, + pruneRepairBackups, + WalRepairError, + closeRepairEpisodeIfOpen, +} from '../src/core/pglite-repair.ts'; + +const SEG_SIZE = 1024 * 1024; + +function makeControl(): Buffer { + const control = Buffer.alloc(8192); + control.writeBigUInt64LE(0x1122334455667788n, 0); // systemIdentifier + control.writeUInt32LE(1700, 8); // pg_control version + control.writeUInt32LE(1, 48); // timeline + control.writeUInt32LE(8192, 224); // xlogBlcksz + control.writeUInt32LE(SEG_SIZE, 228); // xlogSegSize + control.writeBigUInt64LE(3n * BigInt(SEG_SIZE) + 40n, 40); // redo → seg 3 + control.writeUInt32LE(crc32c([control.subarray(0, 288)]), 288); // valid CRC + return control; +} + +/** A synthetic PG17 layout that resetWal fully accepts. */ +function makeLayout(opts?: { segments?: string[]; postmasterPid?: boolean }): string { + const parent = mkdtempSync(join(tmpdir(), 'pgrepair-')); + const dir = join(parent, 'brain.pglite'); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'PG_VERSION'), '17\n'); + mkdirSync(join(dir, 'global'), { recursive: true }); + writeFileSync(join(dir, 'global', 'pg_control'), makeControl()); + mkdirSync(join(dir, 'base'), { recursive: true }); + mkdirSync(join(dir, 'pg_wal', 'archive_status'), { recursive: true }); + for (const seg of opts?.segments ?? [xlogFileName(1, 3n, SEG_SIZE)]) { + writeFileSync(join(dir, 'pg_wal', seg), Buffer.alloc(2048, 0xaa)); + } + if (opts?.postmasterPid) writeFileSync(join(dir, 'postmaster.pid'), '12345\n'); + return dir; +} + +/** + * Overwrite pg_control with an 8192-byte buffer carrying a WRONG control + * version: it PASSES validateWalRepairTarget (size-only check) but FAILS + * resetWal's version check — the fixture for the reset-fails-AFTER-backup + * (WalRepairError) path. + */ +function poisonControlVersion(dir: string): void { + const control = makeControl(); + control.writeUInt32LE(1600, 8); + writeFileSync(join(dir, 'global', 'pg_control'), control); +} + +describe('validateWalRepairTarget', () => { + test('accepts a full PG17 layout, tolerating .gbrain-lock inside it', () => { + const dir = makeLayout(); + mkdirSync(join(dir, '.gbrain-lock'), { recursive: true }); + writeFileSync(join(dir, '.gbrain-lock', 'lock'), '{}'); + expect(validateWalRepairTarget(dir)).toEqual({ ok: true }); + }); + + test('refusal matrix: missing dir / no PG_VERSION / wrong version / no base / bad control', () => { + expect(validateWalRepairTarget('')).toMatchObject({ ok: false, reason: 'missing-dir' }); + expect(validateWalRepairTarget('/nope/never/exists')).toMatchObject({ ok: false, reason: 'missing-dir' }); + + const noVersion = mkdtempSync(join(tmpdir(), 'pgrepair-')); + expect(validateWalRepairTarget(noVersion)).toMatchObject({ ok: false, reason: 'not-pglite-layout' }); + + const v16 = makeLayout(); + writeFileSync(join(v16, 'PG_VERSION'), '16\n'); + expect(validateWalRepairTarget(v16)).toMatchObject({ ok: false, reason: 'unsupported-pg-version' }); + + const noBase = makeLayout(); + rmSync(join(noBase, 'base'), { recursive: true }); + expect(validateWalRepairTarget(noBase)).toMatchObject({ ok: false, reason: 'not-pglite-layout' }); + + const badControl = makeLayout(); + writeFileSync(join(badControl, 'global', 'pg_control'), Buffer.alloc(100)); + expect(validateWalRepairTarget(badControl)).toMatchObject({ ok: false, reason: 'bad-pg-control' }); + }); + + test('refuses symlinked dataDir and symlinked pg_wal (codex 14.8)', () => { + const real = makeLayout(); + const link = join(mkdtempSync(join(tmpdir(), 'pgrepair-')), 'link.pglite'); + symlinkSync(real, link); + expect(validateWalRepairTarget(link)).toMatchObject({ ok: false, reason: 'not-pglite-layout' }); + + const dir = makeLayout(); + const walBackup = join(dir, 'pg_wal_real'); + rmSync(join(dir, 'pg_wal'), { recursive: true }); + mkdirSync(walBackup); + symlinkSync(walBackup, join(dir, 'pg_wal')); + expect(validateWalRepairTarget(dir)).toMatchObject({ ok: false, reason: 'not-pglite-layout' }); + }); + + test('refuses a symlinked global/ dir (security review — lstat on pg_control follows the intermediate link)', () => { + const dir = makeLayout(); + // A foreign dir holding a perfectly valid 8192-byte pg_control: without the + // global/ lstat check, surgery would write a forged control THROUGH the + // link into this directory. + const foreign = mkdtempSync(join(tmpdir(), 'pgrepair-foreign-')); + writeFileSync(join(foreign, 'pg_control'), makeControl()); + rmSync(join(dir, 'global'), { recursive: true }); + symlinkSync(foreign, join(dir, 'global')); + const result = validateWalRepairTarget(dir); + expect(result).toMatchObject({ ok: false, reason: 'not-pglite-layout' }); + if (!result.ok) expect(result.detail).toContain('symlink'); + }); +}); + +describe('repairPgliteWal — rename-based backup', () => { + test('backs up the WHOLE pg_wal dir + postmaster.pid (rename) and copies pg_control', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg], postmasterPid: true }); + writeFileSync(join(dir, 'pg_wal', 'archive_status', `${seg}.ready`), ''); + const originalControl = readFileSync(join(dir, 'global', 'pg_control')); + + const receipt = await repairPgliteWal(dir); + + expect(receipt.backupPath.includes('.wal-repair-backup-')).toBe(true); + expect(receipt.backedUpFiles).toEqual(['pg_wal/', 'postmaster.pid', 'global/pg_control']); + // Backup holds the ORIGINAL bytes, archive_status entries included. + expect(readFileSync(join(receipt.backupPath, 'pg_wal', seg), 'utf-8')).toBe(Buffer.alloc(2048, 0xaa).toString()); + expect(existsSync(join(receipt.backupPath, 'pg_wal', 'archive_status', `${seg}.ready`))).toBe(true); + expect(existsSync(join(receipt.backupPath, 'postmaster.pid'))).toBe(true); + expect(readFileSync(join(receipt.backupPath, 'pg_control')).equals(originalControl)).toBe(true); + // Data dir: fresh pg_wal with exactly the reset segment; pid gone. + expect(existsSync(join(dir, 'postmaster.pid'))).toBe(false); + const segs = readdirSync(join(dir, 'pg_wal')).filter((f) => /^[0-9A-F]{24}$/.test(f)); + expect(segs).toEqual([receipt.resetSegment]); + }); + + test('refuses (typed) on an invalid layout without touching anything', async () => { + const dir = mkdtempSync(join(tmpdir(), 'pgrepair-')); + await expect(repairPgliteWal(dir)).rejects.toThrow(/refusing repair/); + expect(listRepairBackups(dir)).toEqual([]); + }); + + test('throws WalRepairError after backup and restores the dir when resetWal fails', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + poisonControlVersion(dir); // passes validation, fails resetWal + + let caught: unknown; + try { + await repairPgliteWal(dir); + } catch (e) { + caught = e; + } + expect(caught).toBeInstanceOf(WalRepairError); + const err = caught as WalRepairError; + // The best-effort restore ran and is reported HONESTLY on the error. + expect(err.restore.restored).toBe(true); + expect(existsSync(join(dir, 'pg_wal'))).toBe(true); + expect(existsSync(join(dir, 'pg_wal', seg))).toBe(true); // original segment back + expect(existsSync(err.receipt.backupPath)).toBe(true); // forensic backup kept + }); +}); + +describe('restoreWalBackup — overwrite order + guards', () => { + test('byte-identical restore: control first, dir swap, nothing deleted', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + const originalControl = readFileSync(join(dir, 'global', 'pg_control')); + + const receipt = await repairPgliteWal(dir); + const result = await restoreWalBackup(receipt); + + expect(result.restored).toBe(true); + // Original WAL + control are back, byte-identical. + expect(readFileSync(join(dir, 'pg_wal', seg), 'utf-8')).toBe(Buffer.alloc(2048, 0xaa).toString()); + expect(readFileSync(join(dir, 'global', 'pg_control')).equals(originalControl)).toBe(true); + // The reset-state pg_wal was set ASIDE inside the backup dir, not deleted. + const asides = readdirSync(receipt.backupPath).filter((f) => f.startsWith('pg_wal.reset-aside-')); + expect(asides.length).toBe(1); + expect(existsSync(join(receipt.backupPath, asides[0]!, receipt.resetSegment))).toBe(true); + // The dir still has a valid 8192-byte pg_control at every observable point. + expect(readFileSync(join(dir, 'global', 'pg_control')).length).toBe(8192); + }); + + test('mtime guard: refuses when a foreign WAL segment is newer than the backup', async () => { + const dir = makeLayout(); + const receipt = await repairPgliteWal(dir); + // A "live writer" drops a fresh segment into the (reset) pg_wal. + const foreign = xlogFileName(1, 99n, SEG_SIZE); + writeFileSync(join(dir, 'pg_wal', foreign), 'live-writer-bytes'); + const future = new Date(Date.now() + 60_000); + utimesSync(join(dir, 'pg_wal', foreign), future, future); + + const result = await restoreWalBackup(receipt); + expect(result.restored).toBe(false); + expect(result.detail).toContain('mtime-guard'); + // Nothing was swapped or deleted; backup remains intact. + expect(existsSync(join(receipt.backupPath, 'pg_wal'))).toBe(true); + }); + + test('missing/empty backup never claims restoration (8A honesty)', async () => { + const dir = makeLayout(); + const receipt = await repairPgliteWal(dir); + rmSync(receipt.backupPath, { recursive: true, force: true }); + const result = await restoreWalBackup(receipt); + expect(result.restored).toBe(false); + expect(result.detail).toContain('nothing to restore'); + }); +}); + +describe('cooldown sidecar + episode retention', () => { + test('recordRepairAttempt opens an episode on failure, closes on success, caps history', () => { + const dir = makeLayout(); + // Real backup dirs: the re-pin rule inspects them for pg_wal (a gutted + // pinned backup — restore moved its pg_wal back — must lose the pin). + const backupOne = `${dir}.wal-repair-backup-1001`; + const backupTwo = `${dir}.wal-repair-backup-1002`; + const backupThree = `${dir}.wal-repair-backup-1003`; + mkdirSync(join(backupOne, 'pg_wal'), { recursive: true }); + mkdirSync(join(backupTwo, 'pg_wal'), { recursive: true }); + mkdirSync(join(backupThree, 'pg_wal'), { recursive: true }); + + recordRepairAttempt(dir, 'failed', backupOne); + let sidecar = readRepairSidecar(dir); + expect(sidecar.episodeStartedAt).not.toBeNull(); + expect(sidecar.episodeBackupPath).toBe(backupOne); + // Second failure does NOT re-pin while the pinned backup still holds pg_wal. + recordRepairAttempt(dir, 'failed', backupTwo); + sidecar = readRepairSidecar(dir); + expect(sidecar.episodeBackupPath).toBe(backupOne); + // …but a GUTTED pinned backup loses the pin to the fresh one (red-team: + // the episode's protected copy must always be one that still has pg_wal). + rmSync(join(backupOne, 'pg_wal'), { recursive: true }); + recordRepairAttempt(dir, 'failed', backupThree); + sidecar = readRepairSidecar(dir); + expect(sidecar.episodeBackupPath).toBe(backupThree); + + recordRepairAttempt(dir, 'repaired', backupThree); + sidecar = readRepairSidecar(dir); + expect(sidecar.episodeStartedAt).toBeNull(); + expect(sidecar.episodeBackupPath).toBeNull(); + for (let i = 0; i < 15; i++) recordRepairAttempt(dir, 'repaired', null); + expect(readRepairSidecar(dir).attempts.length).toBeLessThanOrEqual(10); + }); + + test('unverified success (closeEpisode:false) keeps the episode open; closeRepairEpisodeIfOpen closes it', () => { + const dir = makeLayout(); + const backup = `${dir}.wal-repair-backup-2001`; + mkdirSync(join(backup, 'pg_wal'), { recursive: true }); + recordRepairAttempt(dir, 'failed', backup); + // The manual command's unverified "repaired" must NOT close/prune. + recordRepairAttempt(dir, 'repaired', backup, { closeEpisode: false }); + let sidecar = readRepairSidecar(dir); + expect(sidecar.episodeStartedAt).not.toBeNull(); + expect(existsSync(backup)).toBe(true); + // A healthy connect closes it. + closeRepairEpisodeIfOpen(dir); + sidecar = readRepairSidecar(dir); + expect(sidecar.episodeStartedAt).toBeNull(); + expect(sidecar.episodeBackupPath).toBeNull(); + }); + + test('repairCooldownActive: active after a recent failure, respects the env knob', async () => { + // Pin a known baseline (default cooldown, repair enabled): an ambient + // GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS=0 would flip the assertions. + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => { + const dir = makeLayout(); + expect(repairCooldownActive(dir).active).toBe(false); + recordRepairAttempt(dir, 'failed', null); + expect(repairCooldownActive(dir).active).toBe(true); + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => { + expect(repairCooldownActive(dir).active).toBe(false); + }); + // A success clears nothing retroactively, but cooldown keys on the LAST + // failed attempt — still inside the window here. + recordRepairAttempt(dir, 'repaired', null); + expect(repairCooldownActive(dir).active).toBe(true); + }); + }); + + test('pruneRepairBackups keeps the newest 3 and never the open episode backup', () => { + const dir = makeLayout(); + const parentBackups: string[] = []; + for (let i = 1; i <= 5; i++) { + const b = `${dir}.wal-repair-backup-${1000 + i}`; + mkdirSync(b, { recursive: true }); + parentBackups.push(b); + } + // Pin the OLDEST as the open episode's backup. + recordRepairAttempt(dir, 'failed', parentBackups[0]!); + pruneRepairBackups(dir); + const kept = listRepairBackups(dir); + // Newest 3 + the protected episode backup. + expect(kept).toContain(parentBackups[0]!); + expect(kept).toContain(parentBackups[4]!); + expect(kept).toContain(parentBackups[3]!); + expect(kept).toContain(parentBackups[2]!); + expect(kept).not.toContain(parentBackups[1]!); + }); +}); + +describe('attemptWalRepairAndRetry — the never-throws engine seam', () => { + test('repaired: retryCreate succeeds → db returned, episode closed, notice printed', async () => { + const dir = makeLayout(); + const stderrChunks: string[] = []; + const origWrite = process.stderr.write.bind(process.stderr); + process.stderr.write = ((chunk: string | Uint8Array) => { + stderrChunks.push(String(chunk)); + return true; + }) as typeof process.stderr.write; + try { + const attempt = await attemptWalRepairAndRetry(dir, async () => 'the-db-handle'); + expect(attempt.status).toBe('repaired'); + if (attempt.status === 'repaired') { + expect(attempt.db).toBe('the-db-handle'); + expect(attempt.receipt.resetSegment).toMatch(/^[0-9A-F]{24}$/); + } + } finally { + process.stderr.write = origWrite; + } + expect(stderrChunks.join('')).toContain('gbrain pglite-repair'); + expect(readRepairSidecar(dir).episodeStartedAt).toBeNull(); + expect(readRepairSidecar(dir).attempts.at(-1)?.outcome).toBe('repaired'); + }); + + test('failed + restored: retryCreate keeps throwing → dir restored, episode opened', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + const attempt = await attemptWalRepairAndRetry(dir, async () => { + throw new Error('Aborted(). still broken'); + }); + expect(attempt.status).toBe('failed'); + if (attempt.status === 'failed') { + expect(attempt.restored).toBe(true); + expect(attempt.receipt).not.toBeNull(); + expect(attempt.repairError).toContain('still broken'); + } + // Original segment is back in place. + expect(existsSync(join(dir, 'pg_wal', seg))).toBe(true); + const sidecar = readRepairSidecar(dir); + expect(sidecar.episodeStartedAt).not.toBeNull(); + expect(sidecar.attempts.at(-1)?.outcome).toBe('failed'); + }); + + test('failed + restored:false (8A): restore blocked by the mtime guard is reported honestly', async () => { + const dir = makeLayout(); + const attempt = await attemptWalRepairAndRetry(dir, async () => { + // Simulate a live writer advancing pg_wal between repair and restore. + const foreign = xlogFileName(1, 99n, SEG_SIZE); + writeFileSync(join(dir, 'pg_wal', foreign), 'live-writer-bytes'); + const future = new Date(Date.now() + 60_000); + utimesSync(join(dir, 'pg_wal', foreign), future, future); + throw new Error('Aborted(). still broken'); + }); + expect(attempt.status).toBe('failed'); + if (attempt.status === 'failed') { + expect(attempt.restored).toBe(false); + expect(attempt.repairError).toContain('mtime-guard'); + } + }); + + test('guards: disabled / reaped lock / validation-failed — no backup dir is ever created', async () => { + const before = process.env.GBRAIN_PGLITE_WAL_REPAIR; + const dir = makeLayout(); + // Pin a known baseline (repair enabled, default cooldown) so an ambient + // GBRAIN_PGLITE_WAL_REPAIR=off can't turn every arm into 'disabled'. + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => { + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: 'off' }, async () => { + const attempt = await attemptWalRepairAndRetry(dir, async () => 'x'); + expect(attempt).toMatchObject({ status: 'skipped', reason: 'disabled' }); + }); + const reaped = await attemptWalRepairAndRetry(dir, async () => 'x', { reaped: true }); + expect(reaped).toMatchObject({ status: 'skipped', reason: 'possibly-live-writer' }); + const invalid = await attemptWalRepairAndRetry('/nope/never', async () => 'x'); + expect(invalid).toMatchObject({ status: 'skipped', reason: 'validation-failed' }); + expect(listRepairBackups(dir)).toEqual([]); + }); + // withEnv restored whatever the ambient value was (including "unset"). + expect(process.env.GBRAIN_PGLITE_WAL_REPAIR).toBe(before); + }); + + test('cooldown skip + episode backup reuse across attempts', async () => { + // Pin a known baseline: an ambient COOLDOWN_SECONDS=0 would break the + // 'recently-failed' gate assertion; an ambient WAL_REPAIR=off breaks all. + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + // Attempt 1 fails → episode opens with backup #1. + const first = await attemptWalRepairAndRetry(dir, async () => { throw new Error('Aborted()'); }); + expect(first.status).toBe('failed'); + const backupsAfterFirst = listRepairBackups(dir); + expect(backupsAfterFirst.length).toBe(1); + + // Immediate retry is cooldown-gated… + const gated = await attemptWalRepairAndRetry(dir, async () => 'x'); + expect(gated).toMatchObject({ status: 'skipped', reason: 'recently-failed' }); + + // …and with the cooldown off, the retry takes a FRESH backup: attempt 1's + // restore MOVED pg_wal back out of its backup, so reusing that gutted dir + // would let resetWal destroy the only surviving WAL copy (red-team). + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => { + const second = await attemptWalRepairAndRetry(dir, async () => 'db'); + expect(second.status).toBe('repaired'); + if (second.status === 'repaired') { + expect(second.receipt.reusedEpisodeBackup).toBe(false); + expect(second.receipt.backupPath).not.toBe(backupsAfterFirst[0]!); + } + }); + expect(listRepairBackups(dir).length).toBe(2); + expect(readRepairSidecar(dir).episodeStartedAt).toBeNull(); // episode closed + }); + }); + + test('episode backup IS reused when it still holds pg_wal (restore was blocked)', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => { + // Attempt 1: retry fails AND restore is blocked by the mtime guard + // (a foreign future-dated segment appears mid-attempt) — the backup + // KEEPS pg_wal. + const first = await attemptWalRepairAndRetry(dir, async () => { + const foreign = xlogFileName(1, 99n, SEG_SIZE); + writeFileSync(join(dir, 'pg_wal', foreign), 'live-writer-bytes'); + const future = new Date(Date.now() + 60_000); + utimesSync(join(dir, 'pg_wal', foreign), future, future); + throw new Error('Aborted(). still broken'); + }); + expect(first.status).toBe('failed'); + if (first.status === 'failed') expect(first.restored).toBe(false); + const episodeBackup = readRepairSidecar(dir).episodeBackupPath!; + expect(existsSync(join(episodeBackup, 'pg_wal'))).toBe(true); + // Clear the foreign segment so attempt 2's surgery isn't re-blocked. + rmSync(join(dir, 'pg_wal'), { recursive: true, force: true }); + mkdirSync(join(dir, 'pg_wal', 'archive_status'), { recursive: true }); + const second = await attemptWalRepairAndRetry(dir, async () => 'db'); + expect(second.status).toBe('repaired'); + if (second.status === 'repaired') { + expect(second.receipt.reusedEpisodeBackup).toBe(true); + expect(second.receipt.backupPath).toBe(episodeBackup); + } + }); + }); + + test('seam reports honest restored from WalRepairError (reset fails after backup)', async () => { + const seg = xlogFileName(1, 3n, SEG_SIZE); + const dir = makeLayout({ segments: [seg] }); + poisonControlVersion(dir); // backup succeeds, resetWal throws → WalRepairError + const attempt = await attemptWalRepairAndRetry(dir, async () => 'x'); + expect(attempt.status).toBe('failed'); + if (attempt.status === 'failed') { + // `restored` is threaded from WalRepairError.restore — not hardcoded. + expect(attempt.restored).toBe(true); + expect(attempt.receipt).not.toBeNull(); + expect(attempt.repairError).toContain('pg_control version'); + } + // Restore actually happened: original segment is back. + expect(existsSync(join(dir, 'pg_wal', seg))).toBe(true); + }); + + test('poisoned sidecar episodeBackupPath outside the backup prefix is IGNORED — fresh backup taken', async () => { + const dir = makeLayout(); + // An existing dir that fails the `${dataDir}.wal-repair-backup-` prefix + // check: the user-writable sidecar must not be able to point repair's + // renames at an arbitrary target. + const evil = mkdtempSync(join(tmpdir(), 'pgrepair-evil-')); + writeFileSync(`${dir}.wal-repair-attempt.json`, JSON.stringify({ + episodeStartedAt: Date.now(), + episodeBackupPath: evil, + attempts: [], + })); + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => { + const attempt = await attemptWalRepairAndRetry(dir, async () => 'db'); + expect(attempt.status).toBe('repaired'); + if (attempt.status === 'repaired') { + expect(attempt.receipt.reusedEpisodeBackup).toBe(false); + expect(attempt.receipt.backupPath.startsWith(`${dir}.wal-repair-backup-`)).toBe(true); + } + }); + // The poisoned target was never renamed into or written through. + expect(existsSync(evil)).toBe(true); + expect(readdirSync(evil)).toEqual([]); + }); + + test('reap quarantine gates the seam; a marker older than the window does not', async () => { + const dir = makeLayout(); + const marker = `${dir}.lock-reap.json`; + writeFileSync(marker, JSON.stringify({ ts: Date.now(), by: 1 })); + + const gated = await attemptWalRepairAndRetry(dir, async () => 'x'); + expect(gated).toMatchObject({ status: 'skipped', reason: 'possibly-live-writer' }); + if (gated.status === 'skipped') expect(gated.detail).toContain('reaped'); + // Gated BEFORE any surgery: no backup dir was created. + expect(listRepairBackups(dir)).toEqual([]); + + // Marker older than the 10-minute quarantine → the seam proceeds. + writeFileSync(marker, JSON.stringify({ ts: Date.now() - 11 * 60 * 1000, by: 1 })); + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => { + const attempt = await attemptWalRepairAndRetry(dir, async () => 'db'); + expect(attempt.status).toBe('repaired'); + }); + }); +}); + +describe('inspectPgliteDataDir', () => { + test('verdicts: missing / unsupported / wal-corruption-likely / looks-healthy', () => { + expect(inspectPgliteDataDir('/nope/never').verdict).toBe('missing'); + expect(inspectPgliteDataDir(mkdtempSync(join(tmpdir(), 'pgrepair-'))).verdict).toBe('unsupported-layout'); + const withPid = makeLayout({ postmasterPid: true }); + expect(inspectPgliteDataDir(withPid).verdict).toBe('wal-corruption-likely'); + const clean = makeLayout(); + expect(inspectPgliteDataDir(clean).verdict).toBe('looks-healthy'); + }); + + test('locked verdict for a live-PID lock; open episode reads as corruption-likely', () => { + const dir = makeLayout(); + mkdirSync(join(dir, '.gbrain-lock'), { recursive: true }); + writeFileSync( + join(dir, '.gbrain-lock', 'lock'), + JSON.stringify({ pid: process.pid, acquired_at: Date.now(), refreshed_at: Date.now(), command: 'gbrain embed', subcommand: 'embed' }), + ); + const diag = inspectPgliteDataDir(dir); + expect(diag.verdict).toBe('locked'); + expect(diag.lockHolderPid).toBe(process.pid); + rmSync(join(dir, '.gbrain-lock'), { recursive: true }); + + recordRepairAttempt(dir, 'failed', null); // opens an episode + expect(inspectPgliteDataDir(dir).verdict).toBe('wal-corruption-likely'); + }); +}); diff --git a/test/pglite-resetwal.test.ts b/test/pglite-resetwal.test.ts new file mode 100644 index 000000000..7ef795910 --- /dev/null +++ b/test/pglite-resetwal.test.ts @@ -0,0 +1,190 @@ +/** + * Unit tests for the pg_resetwal port (src/core/pglite-resetwal.ts). + * + * Everything here runs on SYNTHETIC PG17 layouts (hand-built pg_control + * buffers) — fast, parallel-safe, no PGLite. The real-engine proof (corrupt a + * real brain's WAL → reopen → row readable) lives in + * test/pglite-wal-repair.serial.test.ts. + */ +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + resetWal, + crc32c, + parseWalSegNo, + xlogFileName, + WalResetUnsupportedError, +} from '../src/core/pglite-resetwal.ts'; + +const SEG_SIZE = 1024 * 1024; // 1MB — valid (power of two, divides 2^32), fast to write + +const OFF = { + systemIdentifier: 0, + pgControlVersion: 8, + state: 16, + checkPoint: 32, + checkPointCopyRedo: 40, + checkPointCopyThisTimeLineID: 48, + xlogBlcksz: 224, + xlogSegSize: 228, + crc: 288, +} as const; + +function makeControl(opts?: { version?: number; segSize?: number; blcksz?: number; tli?: number; redoSegNo?: bigint }): Buffer { + const control = Buffer.alloc(8192); + control.writeBigUInt64LE(0x1122334455667788n, OFF.systemIdentifier); + control.writeUInt32LE(opts?.version ?? 1700, OFF.pgControlVersion); + control.writeUInt32LE(opts?.tli ?? 1, OFF.checkPointCopyThisTimeLineID); + control.writeUInt32LE(opts?.blcksz ?? 8192, OFF.xlogBlcksz); + control.writeUInt32LE(opts?.segSize ?? SEG_SIZE, OFF.xlogSegSize); + const redoSegNo = opts?.redoSegNo ?? 3n; + control.writeBigUInt64LE(redoSegNo * BigInt(opts?.segSize ?? SEG_SIZE) + 40n, OFF.checkPointCopyRedo); + control.writeUInt32LE(crc32c([control.subarray(0, OFF.crc)]), OFF.crc); // valid CRC + return control; +} + +function makeLayout(opts?: Parameters[0] & { pgVersion?: string; segments?: string[] }): string { + const dir = mkdtempSync(join(tmpdir(), 'resetwal-')); + writeFileSync(join(dir, 'PG_VERSION'), `${opts?.pgVersion ?? '17'}\n`); + mkdirSync(join(dir, 'global'), { recursive: true }); + writeFileSync(join(dir, 'global', 'pg_control'), makeControl(opts)); + mkdirSync(join(dir, 'base'), { recursive: true }); + mkdirSync(join(dir, 'pg_wal', 'archive_status'), { recursive: true }); + for (const seg of opts?.segments ?? []) { + writeFileSync(join(dir, 'pg_wal', seg), Buffer.alloc(1024, 0xaa)); + } + return dir; +} + +describe('resetWal — validation refusals (fail-closed)', () => { + test('refuses PG_VERSION 16', async () => { + const dir = makeLayout({ pgVersion: '16' }); + await expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError); + }); + + test('refuses missing PG_VERSION', async () => { + const dir = mkdtempSync(join(tmpdir(), 'resetwal-')); + await expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError); + }); + + test('refuses wrong pg_control size', async () => { + const dir = makeLayout(); + writeFileSync(join(dir, 'global', 'pg_control'), Buffer.alloc(100)); + await expect(resetWal(dir)).rejects.toThrow(/pg_control size/); + }); + + test('refuses wrong pg_control version', async () => { + const dir = makeLayout({ version: 1600 }); + await expect(resetWal(dir)).rejects.toThrow(/pg_control version/); + }); + + test('refuses non-power-of-two WAL segment size', async () => { + const dir = makeLayout({ segSize: 3 * 1024 * 1024 }); + await expect(resetWal(dir)).rejects.toThrow(/segment size/); + }); + + test('refuses unsupported WAL block size', async () => { + const dir = makeLayout({ blcksz: 4096 }); + await expect(resetWal(dir)).rejects.toThrow(/block size/); + }); + + test('refuses a pg_control whose stored CRC does not verify (F6: no laundering corrupt counters)', async () => { + const dir = makeLayout(); + // A structurally-valid control (right size/version/seg/block) but with a + // damaged checkpoint copy and a STALE crc — real pg_resetwal refuses this. + const control = readFileSync(join(dir, 'global', 'pg_control')); + control.writeBigUInt64LE(0xdeadbeefn, 56); // trash a checkpointCopy field + // leave the old CRC in place → mismatch + writeFileSync(join(dir, 'global', 'pg_control'), control); + await expect(resetWal(dir)).rejects.toThrow(/CRC mismatch/); + }); +}); + +describe('resetWal — byte surgery on a synthetic PG17 layout', () => { + test('resets WAL: pid removed, old segments gone, fresh checkpoint segment + CRC-valid control', async () => { + const oldSegs = [xlogFileName(1, 3n, SEG_SIZE), xlogFileName(1, 4n, SEG_SIZE)]; + const dir = makeLayout({ redoSegNo: 3n, segments: oldSegs }); + writeFileSync(join(dir, 'postmaster.pid'), '12345\n'); + writeFileSync(join(dir, 'pg_wal', 'archive_status', `${oldSegs[0]}.ready`), ''); + mkdirSync(join(dir, 'pg_wal', 'summaries'), { recursive: true }); + writeFileSync(join(dir, 'pg_wal', 'summaries', `${'0'.repeat(40)}.summary`), ''); + + const result = await resetWal(dir); + + // Stale run state + old WAL removed. + expect(existsSync(join(dir, 'postmaster.pid'))).toBe(false); + for (const seg of oldSegs) { + expect(existsSync(join(dir, 'pg_wal', seg))).toBe(false); + } + expect(readdirSync(join(dir, 'pg_wal', 'archive_status'))).toEqual([]); + expect(readdirSync(join(dir, 'pg_wal', 'summaries'))).toEqual([]); + + // newSegNo = max(redo=3, existing max=4) + 1 = 5. + expect(result.resetSegment).toBe(xlogFileName(1, 5n, SEG_SIZE)); + expect(result.timelineId).toBe(1); + expect(result.walSegSize).toBe(SEG_SIZE); + + const segPath = join(dir, 'pg_wal', result.resetSegment); + expect(existsSync(segPath)).toBe(true); + expect(statSync(segPath).size).toBe(SEG_SIZE); + const wal = readFileSync(segPath); + expect(wal.readUInt16LE(0)).toBe(0xd116); // XLOG_PAGE_MAGIC + expect(wal.readUInt16LE(2) & 0x0002).toBe(0x0002); // XLP_LONG_HEADER + + // Control: shutdown state + self-consistent CRC32C over bytes 0..288. + const control = readFileSync(join(dir, 'global', 'pg_control')); + expect(control.length).toBe(8192); + expect(control.readInt32LE(OFF.state)).toBe(1); // DB_SHUTDOWNED + expect(control.readUInt32LE(OFF.crc)).toBe(crc32c([control.subarray(0, OFF.crc)])); + // checkPoint points into the new segment. + const checkPoint = control.readBigUInt64LE(OFF.checkPoint); + expect(checkPoint / BigInt(SEG_SIZE)).toBe(5n); + + // No torn tmp files left behind (atomic-write hygiene). + expect(readdirSync(join(dir, 'pg_wal')).filter((f) => f.includes('.tmp-'))).toEqual([]); + expect(readdirSync(join(dir, 'global')).filter((f) => f.includes('.tmp-'))).toEqual([]); + }); + + test('works on an EMPTY pg_wal (the whole-dir-rename backup path) and numbers off pg_control alone', async () => { + const dir = makeLayout({ redoSegNo: 7n, segments: [] }); + const result = await resetWal(dir); + // No existing segments — newSegNo = redo(7) + 1. + expect(result.resetSegment).toBe(xlogFileName(1, 8n, SEG_SIZE)); + expect(existsSync(join(dir, 'pg_wal', 'archive_status'))).toBe(true); + }); + + test('is idempotent: a second run converges (numbering keeps moving forward)', async () => { + const dir = makeLayout({ redoSegNo: 3n }); + const first = await resetWal(dir); + const second = await resetWal(dir); + const firstNo = parseWalSegNo(first.resetSegment, SEG_SIZE)!; + const secondNo = parseWalSegNo(second.resetSegment, SEG_SIZE)!; + expect(secondNo).toBeGreaterThan(firstNo); + // Exactly one segment remains after each run. + const segs = readdirSync(join(dir, 'pg_wal')).filter((f) => /^[0-9A-F]{24}$/.test(f)); + expect(segs).toEqual([second.resetSegment]); + }); +}); + +describe('WAL segment name helpers', () => { + test('parseWalSegNo / xlogFileName round-trip', () => { + for (const segNo of [0n, 1n, 255n, 4096n, 0x1_0000_0000n / BigInt(SEG_SIZE) + 7n]) { + const name = xlogFileName(1, segNo, SEG_SIZE); + expect(name).toMatch(/^[0-9A-F]{24}$/); + expect(parseWalSegNo(name, SEG_SIZE)).toBe(segNo); + } + }); + + test('parseWalSegNo rejects non-segment names', () => { + expect(parseWalSegNo('archive_status', SEG_SIZE)).toBeNull(); + expect(parseWalSegNo('000000010000000000000001.partial', SEG_SIZE)).toBeNull(); + expect(parseWalSegNo('lowercase0000000000000001', SEG_SIZE)).toBeNull(); + }); + + test('crc32c matches a known vector', () => { + // CRC-32C of ASCII "123456789" is 0xE3069283 (Castagnoli test vector). + expect(crc32c([Buffer.from('123456789')])).toBe(0xe3069283); + }); +}); diff --git a/test/pglite-wal-repair.serial.test.ts b/test/pglite-wal-repair.serial.test.ts new file mode 100644 index 000000000..0a552b498 --- /dev/null +++ b/test/pglite-wal-repair.serial.test.ts @@ -0,0 +1,200 @@ +/** + * The ported upstream regression (electric-sql/pglite PR #994) against a REAL + * PGLite brain: create → insert → clean shutdown → corrupt the WAL → reopen + * through PGLiteEngine.connect() → auto-repair fires → the original row is + * still readable. Plus the kill-switch, gate-level negatives, and the #2084 + * exitCode pin. + * + * .serial: real PGLite cold starts + process.env writes (docs/TESTING.md R1). + */ +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, existsSync, readdirSync, readFileSync, truncateSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, dirname, basename } from 'node:path'; +import { withEnv } from './helpers/with-env.ts'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import type { EngineConfig } from '../src/core/types.ts'; + +const COLD_START_TIMEOUT = 120_000; + +function engineConfig(dir: string): EngineConfig { + return { engine: 'pglite', database_path: dir } as EngineConfig; +} + +/** Build a real brain with one probe row, cleanly shut down. */ +async function buildRealBrain(): Promise { + const dir = join(mkdtempSync(join(tmpdir(), 'walrepair-')), 'brain.pglite'); + const engine = new PGLiteEngine(); + await engine.connect(engineConfig(dir)); + await engine.db.exec('CREATE TABLE repair_probe (id int); INSERT INTO repair_probe VALUES (42);'); + await engine.disconnect(); + return dir; +} + +function walSegments(dir: string): string[] { + return readdirSync(join(dir, 'pg_wal')).filter((f) => /^[0-9A-F]{24}$/.test(f)); +} + +function backupDirs(dir: string): string[] { + return readdirSync(dirname(dir)).filter((f) => f.startsWith(`${basename(dir)}.wal-repair-backup-`)); +} + +function corruptAllSegments(dir: string, mode: 'truncate' | 'garbage'): void { + const segs = walSegments(dir); + expect(segs.length).toBeGreaterThan(0); + for (const seg of segs) { + const p = join(dir, 'pg_wal', seg); + if (mode === 'truncate') { + truncateSync(p, 1024); + } else { + // Overwrite the whole segment with garbage, keeping its size. + const size = readFileSync(p).length; + writeFileSync(p, Buffer.alloc(size, 0xff)); + } + } +} + +async function connectExpectingRepair(dir: string): Promise { + const engine = new PGLiteEngine(); + const warns: string[] = []; + const origWarn = console.warn; + console.warn = (...args: unknown[]) => { warns.push(args.join(' ')); }; + try { + await engine.connect(engineConfig(dir)); + } finally { + console.warn = origWarn; + } + // #2084 pin: the retry create's exit-status scribble is contained. + expect(Number(process.exitCode ?? 0)).toBe(0); + expect(engine.walRepairReceipt).not.toBeNull(); + expect(warns.join('\n')).toContain('repaired'); + return engine; +} + +describe('WAL auto-repair — real-brain regression (#223/#1670/#2575)', () => { + test('case A (truncated WAL): connect() auto-repairs and the row survives', async () => { + const dir = await buildRealBrain(); + corruptAllSegments(dir, 'truncate'); + + const engine = await connectExpectingRepair(dir); + try { + const receipt = engine.walRepairReceipt!; + expect(existsSync(receipt.backupPath)).toBe(true); + expect(existsSync(join(receipt.backupPath, 'pg_wal'))).toBe(true); + const rows = await engine.db.query('SELECT id FROM repair_probe'); + expect((rows.rows[0] as { id: number }).id).toBe(42); + } finally { + await engine.disconnect(); + } + }, COLD_START_TIMEOUT); + + test('case B (garbage-overwritten WAL): connect() auto-repairs and the row survives', async () => { + const dir = await buildRealBrain(); + corruptAllSegments(dir, 'garbage'); + + const engine = await connectExpectingRepair(dir); + try { + const rows = await engine.db.query('SELECT id FROM repair_probe'); + expect((rows.rows[0] as { id: number }).id).toBe(42); + // A healthy reconnect afterwards does NOT re-fire repair. + await engine.disconnect(); + const engine2 = new PGLiteEngine(); + await engine2.connect(engineConfig(dir)); + expect(engine2.walRepairReceipt).toBeNull(); + await engine2.disconnect(); + } finally { + try { await engine.disconnect(); } catch { /* already disconnected */ } + } + }, COLD_START_TIMEOUT); + + test('kill-switch: GBRAIN_PGLITE_WAL_REPAIR=off → honest error, no backup, lock released', async () => { + const dir = await buildRealBrain(); + corruptAllSegments(dir, 'garbage'); + const backupsBefore = backupDirs(dir).length; + + await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: 'off' }, async () => { + const engine = new PGLiteEngine(); + let message = ''; + try { + await engine.connect(engineConfig(dir)); + throw new Error('connect unexpectedly succeeded'); + } catch (err) { + message = String((err as Error).message); + } + expect(message).toContain('PGLite failed to initialize'); + expect(message).toContain('GBRAIN_PGLITE_WAL_REPAIR=off'); + expect(message).toContain('gbrain pglite-repair'); + expect(message).toContain('Original error:'); + }); + // No surgery happened… + expect(backupDirs(dir).length).toBe(backupsBefore); + // …and the lock was released: repair works on the next (enabled) connect. + const engine = await connectExpectingRepair(dir); + const rows = await engine.db.query('SELECT id FROM repair_probe'); + expect((rows.rows[0] as { id: number }).id).toBe(42); + await engine.disconnect(); + }, COLD_START_TIMEOUT); + + test('gate negative: a REAL wasm-abort with the cooldown gate active refuses repair — no new backup, honest skip reason', async () => { + // A genuinely corrupt brain (create() aborts with the production + // `RuntimeError: Aborted()` signature) whose sidecar records a fresh + // failed attempt — the classifier says wasm-abort, but the cooldown gate + // must refuse BEFORE any surgery. Pins the skip path end-to-end: verdict + // fired, gate refused, zero backup dirs created, honest message. + const { recordRepairAttempt } = await import('../src/core/pglite-repair.ts'); + const dir = await buildRealBrain(); + corruptAllSegments(dir, 'garbage'); + recordRepairAttempt(dir, 'failed', null); + + const engine = new PGLiteEngine(); + let message = ''; + try { + await engine.connect(engineConfig(dir)); + throw new Error('connect unexpectedly succeeded'); + } catch (err) { + message = String((err as Error).message); + } + expect(message).toContain('PGLite failed to initialize'); + expect(message).toContain('Auto-repair skipped'); + expect(message).toContain('gbrain pglite-repair'); + expect(backupDirs(dir).length).toBe(0); + }, COLD_START_TIMEOUT); + + test('gate negative: symlinked data dir — Emscripten refuses the mount with a NAMED error, repair never fires', async () => { + // PGLite's NODEFS cannot mount through a symlinked data dir: it throws a + // message-less `ErrnoError { errno: 20 }`. Two pins: (a) the error + // stringifier surfaces name+errno instead of "[object Object]" (#2674 + // class), (b) no repair surgery runs on either path. + const { symlinkSync } = await import('node:fs'); + const real = await buildRealBrain(); + corruptAllSegments(real, 'garbage'); + const link = join(mkdtempSync(join(tmpdir(), 'walrepair-')), 'link.pglite'); + symlinkSync(real, link); + + const engine = new PGLiteEngine(); + let message = ''; + try { + await engine.connect(engineConfig(link)); + throw new Error('connect unexpectedly succeeded'); + } catch (err) { + message = String((err as Error).message); + } + expect(message).toContain('PGLite failed to initialize'); + expect(message).not.toContain('[object Object]'); + expect(message).toContain('ErrnoError (errno 20)'); + expect(backupDirs(real).length).toBe(0); + expect(backupDirs(link).length).toBe(0); + }, COLD_START_TIMEOUT); + + test('gate shape: the seam only runs for wasm-abort + persistent dataDir (structural pin)', () => { + const src = readFileSync('src/core/pglite-engine.ts', 'utf-8'); + expect(src).toMatch(/if \(verdict === 'wasm-abort'\)/); + expect(src).toMatch(/if \(!dataDir\) \{\s*\n\s*ctx = \{ repair: 'in-memory' \}/); + // The seam call sits INSIDE the wasm-abort branch (no call site outside it). + const firstSeamCall = src.indexOf('await attemptWalRepairAndRetry('); + const gate = src.indexOf("if (verdict === 'wasm-abort')"); + expect(gate).toBeGreaterThan(-1); + expect(firstSeamCall).toBeGreaterThan(gate); + expect(src.indexOf('await attemptWalRepairAndRetry(', firstSeamCall + 1)).toBe(-1); + }); +}); diff --git a/test/v0_37_gap_fill.serial.test.ts b/test/v0_37_gap_fill.serial.test.ts index 022256d21..7f5c279a1 100644 --- a/test/v0_37_gap_fill.serial.test.ts +++ b/test/v0_37_gap_fill.serial.test.ts @@ -424,20 +424,151 @@ describe('reinit-pglite — backup + reinit', () => { expect(exits).toContain(1); }); - test('refuses when missing required --embedding-model / --embedding-dimensions', async () => { + // ── Flag defaulting from the config FILE (eng-review 6A + codex 14.10) ── + // Omitted --embedding-model / --embedding-dimensions default from + // loadConfigFileOnly() (NOT loadConfig(): a transient outage-shell + // GBRAIN_EMBEDDING_* export must not silently change the rebuild target). + // Precedence: explicit flag > config-file value > missing_model/missing_dims. + + /** + * Run runReinitPglite with process.exit stubbed (same throw-on-exit + * pattern as the tests above) and console.log/console.error captured, + * so the plan output + defaulting notes are assertable. + */ + async function captureRun(args: string[]): Promise<{ exits: number[]; logs: string[]; errs: string[] }> { const { runReinitPglite } = await import('../src/commands/reinit-pglite.ts'); const origExit = process.exit; + const origLog = console.log; + const origErr = console.error; const exits: number[] = []; + const logs: string[] = []; + const errs: string[] = []; // eslint-disable-next-line @typescript-eslint/no-explicit-any (process as any).exit = ((code?: number) => { exits.push(code ?? 0); throw new Error('exit:' + (code ?? 0)); }); + console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); }; + console.error = (...a: unknown[]) => { errs.push(a.map(String).join(' ')); }; try { - await runReinitPglite(['--json']); + await runReinitPglite(args); } catch (e) { expect((e as Error).message).toMatch(/^exit:/); } finally { // eslint-disable-next-line @typescript-eslint/no-explicit-any (process as any).exit = origExit; + console.log = origLog; + console.error = origErr; } + return { exits, logs, errs }; + } + + test('no flags: defaults BOTH from the config file', async () => { + // Sentinel: pre-create the .bak so the run halts at bak_exists AFTER + // parseArgs + the plan print — proving the defaulting resolved from + // the file without invoking the real (destructive) init+sync path. + writeFileSync(join(tmpHome, '.gbrain', 'brain.pglite.bak'), 'sentinel'); + + const { exits, logs, errs } = await captureRun(['--yes']); + + // Halted at the sentinel — parseArgs did NOT fail missing_model/missing_dims. expect(exits).toContain(1); + const err = errs.join('\n'); + expect(err).toContain('Backup already exists'); + // The plan shows the config-file values. + const out = logs.join('\n'); + expect(out).toContain('New embedding model: openai:text-embedding-3-large'); + expect(out).toMatch(/New dimensions:\s+1536/); + // One stderr note per defaulted flag. + expect(err).toContain('--embedding-model defaulted from config: openai:text-embedding-3-large'); + expect(err).toContain('--embedding-dimensions defaulted from config: 1536'); + }); + + test('no flags + config missing the values: still fails missing_model / missing_dims', async () => { + const cfgPath = join(tmpHome, '.gbrain', 'config.json'); + + // Neither value in the file → missing_model (checked first). + writeFileSync(cfgPath, JSON.stringify({ + engine: 'pglite', + database_path: join(tmpHome, '.gbrain', 'brain.pglite'), + })); + const noModel = await captureRun(['--json']); + expect(noModel.exits).toContain(1); + const noModelPayload = JSON.parse(noModel.logs[noModel.logs.length - 1]); + expect(noModelPayload.status).toBe('error'); + expect(noModelPayload.reason).toBe('missing_model'); + + // Model present but no dimensions → missing_dims. + writeFileSync(cfgPath, JSON.stringify({ + engine: 'pglite', + database_path: join(tmpHome, '.gbrain', 'brain.pglite'), + embedding_model: 'openai:text-embedding-3-large', + })); + const noDims = await captureRun(['--json']); + expect(noDims.exits).toContain(1); + const noDimsPayload = JSON.parse(noDims.logs[noDims.logs.length - 1]); + expect(noDimsPayload.reason).toBe('missing_dims'); + }); + + test('flag present but valueless still fails missing_model (no silent config fallback)', async () => { + // A malformed explicit flag is a typo, not an omission — it must not + // silently rebuild against whatever the config file happens to hold. + const { exits, logs } = await captureRun(['--json', '--embedding-model']); + expect(exits).toContain(1); + const payload = JSON.parse(logs[logs.length - 1]); + expect(payload.reason).toBe('missing_model'); + }); + + test('explicit flags win over config-file values', async () => { + writeFileSync(join(tmpHome, '.gbrain', 'brain.pglite.bak'), 'sentinel'); + + const { exits, logs, errs } = await captureRun([ + '--embedding-model', 'zeroentropyai:zembed-1', + '--embedding-dimensions', '1280', + '--yes', + ]); + + expect(exits).toContain(1); // bak_exists sentinel + const out = logs.join('\n'); + expect(out).toContain('New embedding model: zeroentropyai:zembed-1'); + expect(out).toMatch(/New dimensions:\s+1280/); + expect(out).not.toContain('openai:text-embedding-3-large'); + // No defaulting note when both values came from flags. + expect(errs.join('\n')).not.toContain('defaulted from config'); + }); + + test('env poisoning: GBRAIN_EMBEDDING_* env is ignored — config FILE values win', async () => { + writeFileSync(join(tmpHome, '.gbrain', 'brain.pglite.bak'), 'sentinel'); + + await withEnv({ + GBRAIN_EMBEDDING_MODEL: 'voyage:poisoned-model', + GBRAIN_EMBEDDING_DIMENSIONS: '9999', + }, async () => { + const { exits, logs, errs } = await captureRun(['--yes']); + + expect(exits).toContain(1); // bak_exists sentinel + const out = logs.join('\n'); + expect(out).toContain('New embedding model: openai:text-embedding-3-large'); + expect(out).toMatch(/New dimensions:\s+1536/); + expect(out).not.toContain('voyage:poisoned-model'); + // Scoped to the plan line — the tmpdir's random suffix in the path + // lines could otherwise collide with a bare '9999' substring check. + expect(out).not.toMatch(/New dimensions:\s+9999/); + const err = errs.join('\n'); + expect(err).toContain('--embedding-model defaulted from config: openai:text-embedding-3-large'); + expect(err).toContain('--embedding-dimensions defaulted from config: 1536'); + }); + }); + + test('invalid_dims validation applies to the config-sourced value too', async () => { + const cfgPath = join(tmpHome, '.gbrain', 'config.json'); + writeFileSync(cfgPath, JSON.stringify({ + engine: 'pglite', + database_path: join(tmpHome, '.gbrain', 'brain.pglite'), + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: -5, + })); + + const { exits, logs } = await captureRun(['--json']); + expect(exits).toContain(1); + const payload = JSON.parse(logs[logs.length - 1]); + expect(payload.reason).toBe('invalid_dims'); }); });