admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-08-14 08:53:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(pglite): harden WAL auto-repair (pre-landing + adversarial review)

Browse Source 
Review-army (security/testing/maintainability/perf) + Claude & Codex
adversarial passes on the WAL-repair wave. Correctness + safety hardening,
no behavior change to the happy path:

- Live-writer safety: repair refuses any reaped lock acquisition, a corrupt
  (unknowable-liveness) reap writes a cross-process quarantine marker that
  gates auto-repair AND the manual command for 10 min, isProcessAlive treats
  only ESRCH as dead (EPERM/malformed-pid read as alive), and a live
  postmaster.pid (native Postgres) is refused. Lock heartbeat + initial write
  are atomic (tmp+rename) so a torn read can't misclassify a healthy holder;
  an in-flight acquisition is no longer mistaken for corrupt.
- resetWal verifies the stored pg_control CRC before trusting/re-signing it —
  a damaged control file routes to rebuild instead of laundering corrupt
  checkpoint counters under a fresh CRC. Atomic 'wx' writes (no symlink
  follow), whole-pg_wal-dir rename backup, 64MB seg-size cap.
- Honest failure reporting: repairPgliteWal threads the real restore result
  out via WalRepairError so the 'failed-restored' vs 'failed-not-restored'
  message never lies; the not-restored copy names the correct restore paths.
- Episode lifecycle: episodes close on the next healthy connect (not just on
  a verified repair), a gutted (restored) backup loses its pin, stale (>24h)
  episode backups aren't reused, and the cooldown also caps repaired-only
  crash loops. Empty backup dirs are pruned on refusal.
- Command: rejects unknown flags and valueless --path (a destructive command
  must not silently mis-parse), confirm prompt goes to stderr (stdout stays
  clean for --json), embedding-flag defaults come from the config file only.
- Symlink confinement extended to global/; sidecar reuse path validated
  (prefix + no '..' + must still hold pg_wal); sidecar writes atomic.
- doctor recurrence escalation counts all attempts; data dir absolutized.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-08 10:35:10 -07:00
co-authored by Claude Fable 5
parent f2b8619b37
commit 4758be5217
12 changed files with 983 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/commands/doctor.ts
+41 -29
View File
@@ -4651,28 +4651,10 @@ export async function checkCycleFreshness(
* - `progress` reporter writes to stderr (heartbeats per check)
* - `engine.executeRaw` / handler-leaf calls (the actual probe work)
*/
/**
* issue #1685 (GAP A) — the single authoritative "worker is OOM-looping" signal.
*
* One `gbrain doctor` line replaces the hours of log archaeology the #1678
* incident required: `cap=8192MB, N watchdog kills/24h → raise --max-rss`.
*
* UNIONS two sources so it's authoritative for BOTH worker modes (CODEX #5):
* - SUPERVISED workers: supervisor audit `worker_exited likely_cause=rss_watchdog`,
* read cross-week (CODEX #7) so a Mon read doesn't lose a Sun loop.
* - BARE `gbrain jobs work`: NO supervisor event is written; the only trace is
* `minion_jobs.error_text = 'aborted: watchdog'` (the same source queue_health
* subcheck 3 reads). Reading supervisor-only would miss bare workers entirely
* and the queue_health cross-reference would point at an unemitted check.
*
* Cap (CODEX #6): the breaker alert stamps `max_rss_mb`, but a fail from
* oomKills>=5 spread over 24h may have no breaker event → no stamped cap. Fall
* back to `resolveDefaultMaxRssMb()` so the message always renders a number.
*
* Returns null when the worker never OOM'd (don't warn installs that never hit
* it). Pure-ish: filesystem audit read + one minion_jobs count; no process.exit.
* Exported so `test/doctor-worker-oom-loop.test.ts` drives it directly.
*/
// ≥2 failed repair attempts inside 7 days = the corruption keeps regenerating.
const REPAIR_RECURRENCE_WINDOW_MS = 7 * 24 * 3600 * 1000;
const REPAIR_RECURRENCE_THRESHOLD = 2;
/**
* WAL-repair wave (#223/#1670/#2575): when the DB failed to connect on a
* PGLite brain, diagnose the data dir from the FILESYSTEM (the connect error
@@ -4696,11 +4678,14 @@ export function computePgliteDataDirCheck(
const backupNote = diagnosis.backupDirs.length > 0
? ` ${diagnosis.backupDirs.length} repair backup dir(s) on disk (newest: ${diagnosis.backupDirs[0]}) — delete old ones to reclaim space once the brain is healthy.`
: '';
const recentFailed = diagnosis.recentAttempts.filter(
(a) => a.outcome === 'failed' && Date.now() - a.ts < 7 * 24 * 3600 * 1000,
// Count BOTH outcomes (adversarial review F12): a >1h-period crash loop where
// each repair "succeeds" discards a WAL tail per cycle with zero FAILED
// attempts on record — escalation must still fire.
const recentAttempts = diagnosis.recentAttempts.filter(
(a) => Date.now() - a.ts < REPAIR_RECURRENCE_WINDOW_MS,
).length;
const recurrence = recentFailed >= 2
? ` Auto-repair has failed ${recentFailed}x this week — the corruption keeps regenerating (likely an unclean-shutdown loop). Consider switching engines (docs/ENGINES.md: \`gbrain init --supabase\` or native Postgres).`
const recurrence = recentAttempts >= REPAIR_RECURRENCE_THRESHOLD
? ` Auto-repair has run ${recentAttempts}x this week — the corruption keeps regenerating (likely an unclean-shutdown loop). Consider switching engines (docs/ENGINES.md: \`gbrain init --supabase\` or native Postgres).`
: '';
switch (diagnosis.verdict) {
@@ -4747,13 +4732,37 @@ export function computePgliteDataDirCheck(
status: 'fail',
message:
`PGLite failed to open but the data dir layout validates (${diagnosis.detail}). ` +
`Most likely torn WAL/checkpoint state without on-disk markers: ` +
`\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair in place.${backupNote}${recurrence}`,
`IF the connect error mentions \`Aborted()\` this is likely torn WAL state — ` +
`\`gbrain pglite-repair --dry-run\` to diagnose, \`gbrain pglite-repair --yes\` to repair in place ` +
`(repair discards the un-checkpointed WAL tail — don't run it for lock-contention or ` +
`catalog-corruption errors; 58P01/pgvector load failures need \`gbrain reinit-pglite\` instead).${backupNote}${recurrence}`,
remediation_status: 'human_only',
};
}
}
/**
* issue #1685 (GAP A) — the single authoritative "worker is OOM-looping" signal.
*
* One `gbrain doctor` line replaces the hours of log archaeology the #1678
* incident required: `cap=8192MB, N watchdog kills/24h → raise --max-rss`.
*
* UNIONS two sources so it's authoritative for BOTH worker modes (CODEX #5):
* - SUPERVISED workers: supervisor audit `worker_exited likely_cause=rss_watchdog`,
* read cross-week (CODEX #7) so a Mon read doesn't lose a Sun loop.
* - BARE `gbrain jobs work`: NO supervisor event is written; the only trace is
* `minion_jobs.error_text = 'aborted: watchdog'` (the same source queue_health
* subcheck 3 reads). Reading supervisor-only would miss bare workers entirely
* and the queue_health cross-reference would point at an unemitted check.
*
* Cap (CODEX #6): the breaker alert stamps `max_rss_mb`, but a fail from
* oomKills>=5 spread over 24h may have no breaker event → no stamped cap. Fall
* back to `resolveDefaultMaxRssMb()` so the message always renders a number.
*
* Returns null when the worker never OOM'd (don't warn installs that never hit
* it). Pure-ish: filesystem audit read + one minion_jobs count; no process.exit.
* Exported so `test/doctor-worker-oom-loop.test.ts` drives it directly.
*/
export async function computeWorkerOomLoopCheck(
engine: BrainEngine | null,
): Promise<Check | null> {
@@ -6038,7 +6047,10 @@ export async function buildChecks(
const cfg = loadConfig();
if (cfg?.engine === 'pglite') {
const { inspectPgliteDataDir } = await import('../core/pglite-repair.ts');
const pgliteDataDir = cfg.database_path || gbrainPath('brain.pglite');
const { resolve } = await import('node:path');
// Absolutize: a RELATIVE database_path would make the sidecar/backup
// lookups resolve against doctor's cwd instead of the engine's.
const pgliteDataDir = resolve(cfg.database_path || gbrainPath('brain.pglite'));
checks.push(computePgliteDataDirCheck(pgliteDataDir, inspectPgliteDataDir(pgliteDataDir)));
}
} catch {
src/commands/pglite-repair.ts
+78 -14
View File
@@ -22,7 +22,7 @@
import { createInterface } from 'readline';
import { loadConfig, gbrainPath } from '../core/config.ts';
import { acquireLock, releaseLock, LiveServeLockError } from '../core/pglite-lock.ts';
import { acquireLock, releaseLock, LiveServeLockError, msSinceLastReap } from '../core/pglite-lock.ts';
import {
inspectPgliteDataDir,
listRepairBackups,
@@ -30,6 +30,7 @@ import {
recordRepairAttempt,
repairPgliteWal,
validateWalRepairTarget,
WalRepairError,
} from '../core/pglite-repair.ts';
interface RepairCmdOpts {
@@ -40,6 +41,8 @@ interface RepairCmdOpts {
help: boolean;
}
class UnknownFlagError extends Error {}
function parseArgs(args: string[]): RepairCmdOpts {
const opts: RepairCmdOpts = { dryRun: false, yes: false, jsonOutput: false, customPath: null, help: false };
for (let i = 0; i < args.length; i++) {
@@ -47,8 +50,17 @@ function parseArgs(args: string[]): RepairCmdOpts {
if (a === '--dry-run') opts.dryRun = true;
else if (a === '--yes' || a === '-y') opts.yes = true;
else if (a === '--json') opts.jsonOutput = true;
else if (a === '--path') opts.customPath = args[++i] ?? null;
else if (a === '--path') {
const val = args[++i];
// F1: a valueless --path (typo / shell mangling) must NOT fall through
// to the configured default brain and run surgery on the wrong dir.
if (val === undefined || val.startsWith('-')) throw new UnknownFlagError('--path requires a directory argument');
opts.customPath = val;
}
else if (a === '--help' || a === '-h') opts.help = true;
// Reject unknown args on a DESTRUCTIVE command (codex): silently ignoring
// a typo like `--dry-rnu` would run a real WAL reset instead of a dry run.
else throw new UnknownFlagError(`unknown argument: ${a}`);
}
return opts;
}
@@ -89,7 +101,8 @@ function emitError(jsonOutput: boolean, code: string, message: string): void {
}
async function promptYesNo(question: string): Promise<boolean> {
const rl = createInterface({ input: process.stdin, output: process.stdout });
// Prompt on stderr: stdout stays clean for --json payloads.
const rl = createInterface({ input: process.stdin, output: process.stderr });
return new Promise((resolve) => {
rl.question(`${question} [y/N] `, (answer) => {
rl.close();
@@ -99,7 +112,17 @@ async function promptYesNo(question: string): Promise<boolean> {
}
export async function runPgliteRepair(args: string[]): Promise<number> {
const opts = parseArgs(args);
let opts: RepairCmdOpts;
try {
opts = parseArgs(args);
} catch (err) {
if (err instanceof UnknownFlagError) {
const jsonOut = args.includes('--json');
emitError(jsonOut, 'unknown_flag', `${err.message}. Run \`gbrain pglite-repair --help\`.`);
return 2;
}
throw err;
}
if (opts.help) {
printHelp();
return 0;
@@ -148,9 +171,15 @@ export async function runPgliteRepair(args: string[]): Promise<number> {
if (diagnosis.backupDirs.length > 0) {
console.log(` Repair backups on disk: ${diagnosis.backupDirs.join(', ')}`);
}
console.log(validation.ok
? ' Repairable: yes — run `gbrain pglite-repair --yes` to reset the WAL in place.'
: ` Repairable: NO — ${validation.detail}`);
if (!validation.ok) {
console.log(` Repairable: NO — ${validation.detail}`);
} else if (diagnosis.verdict === 'looks-healthy') {
console.log(' Repairable: yes — but no unclean-shutdown markers found; repair is likely');
console.log(' unnecessary. Run `gbrain pglite-repair --yes` ONLY if PGLite fails to open');
console.log(' with `RuntimeError: Aborted()` (repair discards the un-checkpointed WAL tail).');
} else {
console.log(' Repairable: yes — run `gbrain pglite-repair --yes` to reset the WAL in place.');
}
}
return 0;
}
@@ -167,15 +196,27 @@ export async function runPgliteRepair(args: string[]): Promise<number> {
);
return 1;
}
const sinceReap = msSinceLastReap(dataDir);
const REAP_QUARANTINE_MS = 10 * 60 * 1000;
if (sinceReap !== null && sinceReap >= 0 && sinceReap < REAP_QUARANTINE_MS) {
emitError(
opts.jsonOutput,
'refused_reap_quarantine',
`a lock on this brain was reaped ${Math.round(sinceReap / 1000)}s ago from a holder whose ` +
'liveness could not be verified — that process may still be writing. Confirm no gbrain ' +
`process is running (\`pgrep -af gbrain\`), wait ${Math.ceil((REAP_QUARANTINE_MS - sinceReap) / 60000)} more minute(s), then re-run.`,
);
return 1;
}
if (!opts.yes) {
if (!process.stdin.isTTY) {
emitError(opts.jsonOutput, 'no_tty_no_yes', 'Non-TTY environment requires --yes to confirm the WAL reset.');
return 1;
}
console.log(`About to reset the WAL of ${dataDir} in place.`);
console.log('Data files are preserved; un-checkpointed transactions may be lost.');
console.log('The current pg_wal + pg_control are kept in a sibling backup directory.');
console.error(`About to reset the WAL of ${dataDir} in place.`);
console.error('Data files are preserved; un-checkpointed transactions may be lost.');
console.error('The current pg_wal + pg_control are kept in a sibling backup directory.');
const confirmed = await promptYesNo('Repair now?');
if (!confirmed) {
if (opts.jsonOutput) {
@@ -233,19 +274,42 @@ export async function runPgliteRepair(args: string[]): Promise<number> {
process.stderr.write(`Repairing WAL of ${dataDir} in place…\n`);
const sidecar = readRepairSidecar(dataDir);
// F4: only reuse a FRESH (<24h) episode backup — a stale pin may predate
// real data (same bound as the auto seam's episodeFresh).
const episodeFresh =
sidecar.episodeStartedAt !== null &&
Date.now() - sidecar.episodeStartedAt >= 0 &&
Date.now() - sidecar.episodeStartedAt < 24 * 3600 * 1000;
let receipt;
try {
receipt = await repairPgliteWal(dataDir, {
reuseBackupPath: sidecar.episodeBackupPath ?? undefined,
reuseBackupPath: episodeFresh ? sidecar.episodeBackupPath ?? undefined : undefined,
});
} catch (err) {
if (err instanceof WalRepairError) {
// Reset failed after the backup was taken — report the restore's REAL
// outcome so the user knows whether the dir is back or in reset state.
recordRepairAttempt(dataDir, 'failed', err.receipt.backupPath);
emitError(
opts.jsonOutput,
'repair_failed',
err.message + (err.restore.restored
? ` (data dir restored to its pre-repair state; backup kept at ${err.receipt.backupPath})`
: ` (RESTORE ALSO FAILED — the dir is in a reset state; your pre-repair files are intact at ${err.receipt.backupPath}: ${err.restore.detail ?? ''})`),
);
return 1;
}
recordRepairAttempt(dataDir, 'failed', sidecar.episodeBackupPath);
emitError(opts.jsonOutput, 'repair_failed', String((err as Error)?.message ?? err));
return 1;
}
// "repaired" here means the reset completed; the next connect proves it.
// If it still fails, that connect opens a fresh episode.
recordRepairAttempt(dataDir, 'repaired', receipt.backupPath);
// "repaired" here means the reset completed; the next connect PROVES it.
// Record a FAILED attempt (not repaired-with-closeEpisode:false, which
// leaves the episode null when none was open — codex): this opens/keeps an
// episode pinned to this backup so a later healthy connect closes it and
// prunes, and repeated manual runs during one incident reuse the pinned
// backup instead of deleting the pre-damage forensic copy.
recordRepairAttempt(dataDir, 'failed', receipt.backupPath);
if (opts.jsonOutput) {
console.log(JSON.stringify({
src/commands/reinit-pglite.ts
+8 -1
View File
@@ -18,7 +18,7 @@
* output via `--json` for scripted callers.
*/
import { existsSync, renameSync, statSync } from 'fs';
import { existsSync, renameSync, statSync, rmSync } from 'fs';
import { dirname } from 'path';
import { loadConfig, loadConfigFileOnly, gbrainPath } from '../core/config.ts';
@@ -122,6 +122,13 @@ export async function runReinitPglite(args: string[]): Promise<void> {
try {
renameSync(dbPath, bakPath);
// WAL-repair state travels with the OLD brain (red-team: a fresh brain at
// the same path must not inherit the old brain's open repair episode,
// cooldown, or reap quarantine — a stale episodeBackupPath would be reused
// over the NEW brain's WAL).
for (const sibling of [`${dbPath}.wal-repair-attempt.json`, `${dbPath}.lock-reap.json`]) {
try { rmSync(sibling, { force: true }); } catch { /* best-effort */ }
}
} catch (e: unknown) {
fail(
opts.jsonOutput,
src/core/pglite-engine.ts
+8 -3
View File
@@ -48,7 +48,7 @@ import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
// Engine-live path (#3596): static import, never a lazy `import()` in the
// connect() catch. No cycle: pglite-repair.ts imports nothing from this file.
import { attemptWalRepairAndRetry, type WalRepairReceipt } from './pglite-repair.ts';
import { attemptWalRepairAndRetry, closeRepairEpisodeIfOpen, type WalRepairReceipt } from './pglite-repair.ts';
import { getFtsLanguage } from './fts-language.ts';
import type {
Page, PageInput, PageFilters, PageType,
@@ -286,8 +286,8 @@ function repairContextLine(ctx: PgliteInitRepairContext): string {
return ' Auto-repair ran, PGLite still failed to start, AND the automatic restore\n' +
' itself failed — the data dir is currently in a RESET state. Your\n' +
` pre-repair files are intact in the backup at ${ctx.backupPath ?? '<dataDir>.wal-repair-backup-*'};\n` +
' restore manually by moving its `pg_wal` dir and `pg_control` back into\n' +
' the data dir.' +
' restore manually: move the backup\'s `pg_wal` dir back to `<dataDir>/pg_wal`\n' +
' and its `pg_control` file back to `<dataDir>/global/pg_control`.' +
(ctx.detail ? `\n Detail: ${ctx.detail}` : '');
case 'not-attempted':
default:
@@ -475,6 +475,11 @@ export class PGLiteEngine implements BrainEngine {
extensions: { vector, pg_trgm },
}),
);
// Healthy open: close any repair episode left open by a prior failed
// attempt (red-team: episodes otherwise stayed open forever — doctor
// kept reporting corruption-likely and a weeks-stale episode backup
// could be reused over much newer data). Cheap no-op without a sidecar.
if (dataDir) closeRepairEpisodeIfOpen(dataDir);
} catch (err) {
// v0.13.1: any PGLite.create() failure becomes actionable. v0.41.8.0
// (#1340): the previous error hint hardcoded the macOS 26.3 link, but
src/core/pglite-lock.ts
+73 -9
View File
@@ -14,7 +14,7 @@
* try { ... } finally { await releaseLock(lock); }
*/
import { mkdirSync, existsSync, readFileSync, writeFileSync, rmSync, statSync } from 'fs';
import { mkdirSync, existsSync, readFileSync, writeFileSync, rmSync, statSync, renameSync } from 'fs';
import { join } from 'path';
import { parseGlobalFlags } from './cli-options.ts';
@@ -106,13 +106,51 @@ function startHeartbeat(lockPath: string, ownerToken: string): ReturnType<typeof
return;
}
raw.refreshed_at = Date.now();
writeFileSync(lockPath, JSON.stringify(raw), { mode: 0o644 });
// Atomic tmp+rename (security review): waiting acquirers poll-read this
// file every second — an in-place write can be caught mid-flight and a
// torn read misclassifies a HEALTHY live holder as a corrupt lock,
// getting it reaped. rename makes every read see old-or-new, never torn.
const tmpPath = `${lockPath}.tmp-${process.pid}`;
writeFileSync(tmpPath, JSON.stringify(raw), { mode: 0o644 });
renameSync(tmpPath, lockPath);
} catch { /* best-effort — file removed or transient FS error */ }
}, HEARTBEAT_INTERVAL_MS);
(timer as { unref?: () => void }).unref?.();
return timer;
}
/**
* Persisted reap marker (security review): written ONLY for corrupt-lock-file
* reaps, where the holder's liveness is UNKNOWABLE (the PID can't be read).
* The in-process `reaped` flag dies with the acquisition — so the reaper
* destroys a possibly-live holder's lock, exits, and the NEXT process
* acquires "cleanly" and would run WAL surgery under a live writer. The
* marker makes that reap visible across processes: `attemptWalRepairAndRetry`
* refuses auto-repair while a recent unknowable-liveness reap is on record.
* Dead-PID reaps (affirmative ESRCH verdict) deliberately do NOT write it —
* the dead-holder recovery cost stays at one failed command + one re-run.
*/
function reapMarkerPath(dataDir: string): string {
return `${dataDir}.lock-reap.json`;
}
function recordReap(dataDir: string): void {
try {
writeFileSync(reapMarkerPath(dataDir), JSON.stringify({ ts: Date.now(), by: process.pid }), { mode: 0o644 });
} catch { /* best-effort — a marker write failure must not block acquisition */ }
}
/** Milliseconds since the last recorded reap on this data dir, or null. */
export function msSinceLastReap(dataDir: string | undefined): number | null {
if (!dataDir) return null;
try {
const raw = JSON.parse(readFileSync(reapMarkerPath(dataDir), 'utf-8')) as { ts?: unknown };
return typeof raw.ts === 'number' && Number.isFinite(raw.ts) ? Date.now() - raw.ts : null;
} catch {
return null;
}
}
function getLockDir(dataDir: string | undefined): string {
// Use the parent of the data dir for the lock, or a temp location for in-memory
if (!dataDir) {
@@ -123,13 +161,17 @@ function getLockDir(dataDir: string | undefined): string {
return join(dataDir, LOCK_DIR_NAME);
}
function isProcessAlive(pid: number): boolean {
export function isProcessAlive(pid: number): boolean {
// Only ESRCH (no such process) is affirmative proof of death. EPERM means
// the process EXISTS under another user; ERR_INVALID_ARG_TYPE / a malformed
// or non-finite pid means we can't tell — all of which must read as ALIVE,
// because a false "dead" reaps a live holder's lock (security/codex review).
if (!Number.isInteger(pid) || pid <= 0) return true;
try {
// Sending signal 0 checks existence without actually sending a signal
process.kill(pid, 0);
process.kill(pid, 0); // signal 0 = existence check, no signal delivered
return true;
} catch {
return false;
} catch (err) {
return (err as NodeJS.ErrnoException)?.code !== 'ESRCH';
}
}
@@ -197,7 +239,11 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
// heartbeat" is NOT evidence of death — only a dead PID is.
const alive = isProcessAlive(lockPid);
if (!alive) {
// Holder process is gone — reap and try to acquire.
// Holder process is gone — reap and try to acquire. This verdict is
// affirmative (kill-0 threw ESRCH; EPERM reads as alive), so no
// cross-process quarantine marker: the same-acquisition `reaped`
// flag alone gates repair, keeping the dead-holder recovery cost at
// one failed command + one re-run.
reaped = true;
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ }
} else {
@@ -220,9 +266,22 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
// A live MCP server is not a stale or corrupt lock. Surface the useful
// explanation without touching the lock it still owns.
if (err instanceof LiveServeLockError) throw err;
// ENOENT = acquisition in flight (a concurrent acquirer did mkdir but
// hasn't written the lock file yet) — reaping HERE would destroy a
// LIVE acquirer's lock and put two writers on one dir (red-team).
// Give the writer a grace window keyed on the lock dir's age.
if ((err as NodeJS.ErrnoException)?.code === 'ENOENT') {
let lockDirAgeMs = Infinity;
try { lockDirAgeMs = Date.now() - statSync(lockDir).mtimeMs; } catch { /* dir gone — retry loop handles */ }
if (lockDirAgeMs < 10_000) {
await new Promise(r => setTimeout(r, 200));
continue;
}
}
// Corrupt lock file — remove it. The holder's liveness is UNKNOWABLE
// here (unreadable PID), so this counts as a reap for the repair gate.
reaped = true;
recordReap(dataDir as string);
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ }
}
}
@@ -234,13 +293,18 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
// the heartbeat so this holder reads as alive-and-working to others.
const lockPath = join(lockDir, LOCK_FILE);
const now = Date.now();
writeFileSync(lockPath, JSON.stringify({
// Atomic tmp+rename, same torn-read protection as the heartbeat: a
// concurrent poll-reader must see the file complete or absent, never
// mid-write (a torn read classifies a LIVE holder as corrupt).
const initTmp = `${lockPath}.tmp-${process.pid}`;
writeFileSync(initTmp, JSON.stringify({
pid: process.pid,
acquired_at: now,
refreshed_at: now,
command: process.argv.slice(1).join(' '),
subcommand: parseGlobalFlags(process.argv.slice(2)).rest[0] ?? null,
}), { mode: 0o644 });
renameSync(initTmp, lockPath);
const ownerToken = tokenOf({ pid: process.pid, acquired_at: now });
return { lockDir, acquired: true, lockPath, ownerToken, reaped, heartbeat: startHeartbeat(lockPath, ownerToken) };
src/core/pglite-repair.ts
+278 -51
View File
@@ -35,11 +35,18 @@
*/
import {
existsSync, lstatSync, readdirSync, readFileSync, statSync, writeFileSync,
mkdirSync, rmSync,
mkdirSync, rmSync, renameSync,
} from 'node:fs';
import { readFile, rename } from 'node:fs/promises';
import { basename, dirname, join } from 'node:path';
import { resetWal, writeFileAtomicSynced, WalResetUnsupportedError } from './pglite-resetwal.ts';
import { resetWal, writeFileAtomicSynced, WalResetUnsupportedError, PG_CONTROL_FILE_SIZE, isWalSegmentName } from './pglite-resetwal.ts';
import { msSinceLastReap, isProcessAlive } from './pglite-lock.ts';
// A recent reap on this data dir — by ANY process — means a holder that may
// still be alive lost its lock; auto WAL surgery stays off until the window
// clears (security review: the in-process `reaped` flag alone let the NEXT
// acquirer look clean while the reaped holder was still writing).
const REAP_QUARANTINE_MS = 10 * 60 * 1000;
const BACKUP_DIR_MARKER = '.wal-repair-backup-';
const SIDECAR_SUFFIX = '.wal-repair-attempt.json';
@@ -75,6 +82,23 @@ export interface RestoreResult {
detail?: string;
}
/**
* Thrown by `repairPgliteWal` when the reset failed AFTER the backup was
* taken. Carries the receipt and the result of the best-effort restore so the
* seam can report `restored` HONESTLY instead of assuming the restore worked
* (the `failed-not-restored` message arm depends on this being truthful).
*/
export class WalRepairError extends Error {
constructor(
message: string,
readonly receipt: WalRepairReceipt,
readonly restore: RestoreResult,
) {
super(message);
this.name = 'WalRepairError';
}
}
export type WalRepairAttempt<T> =
| { status: 'repaired'; db: T; receipt: WalRepairReceipt }
| {
@@ -132,20 +156,37 @@ export function readRepairSidecar(dataDir: string): RepairSidecar {
function writeRepairSidecar(dataDir: string, sidecar: RepairSidecar): void {
try {
writeFileSync(sidecarPath(dataDir), JSON.stringify(sidecar), { mode: 0o644 });
// Atomic tmp+rename: a kill/power-loss mid-write must not truncate the
// sidecar to invalid JSON (readRepairSidecar would then silently reset the
// episode/cooldown state — codex review). rename is atomic; a torn tmp is
// discarded on the next write.
const tmp = `${sidecarPath(dataDir)}.tmp-${process.pid}`;
writeFileSync(tmp, JSON.stringify(sidecar), { mode: 0o644 });
renameSync(tmp, sidecarPath(dataDir));
} catch { /* best-effort — a sidecar write failure must never block recovery */ }
}
/**
* Record a real repair attempt (repaired|failed) and manage episode state:
* a `failed` attempt opens an episode (if none is open) pinning its backup as
* the episode backup; a `repaired` attempt closes the episode. Prunes retained
* backups down to the newest KEEP_EPISODES after a successful close.
* the episode backup; a VERIFIED `repaired` attempt closes the episode and
* prunes retained backups to the newest KEEP_EPISODES. The manual command
* passes `closeEpisode: false` — its "repaired" is unverified (the next
* connect proves it), and closing+pruning on unverified success let repeated
* manual runs delete the pre-damage forensic backup (red-team finding); the
* episode instead closes on the next successful connect
* (`closeRepairEpisodeIfOpen`).
*
* Re-pin rule (red-team finding): a restore MOVES pg_wal back out of the
* backup, gutting it — if a later failed attempt took a FRESH backup while an
* episode pinned a gutted dir, the pin moves to the fresh backup so the
* episode's protected copy is always one that still holds pg_wal.
*/
export function recordRepairAttempt(
dataDir: string,
outcome: 'repaired' | 'failed',
backupPath: string | null,
opts?: { closeEpisode?: boolean },
): void {
const sidecar = readRepairSidecar(dataDir);
sidecar.attempts.push({ ts: Date.now(), outcome, backupPath });
@@ -156,27 +197,79 @@ export function recordRepairAttempt(
if (sidecar.episodeStartedAt === null) {
sidecar.episodeStartedAt = Date.now();
sidecar.episodeBackupPath = backupPath;
} else if (
backupPath &&
backupPath !== sidecar.episodeBackupPath &&
(!sidecar.episodeBackupPath || !existsSync(join(sidecar.episodeBackupPath, 'pg_wal')))
) {
sidecar.episodeBackupPath = backupPath;
}
} else {
} else if (opts?.closeEpisode !== false) {
sidecar.episodeStartedAt = null;
sidecar.episodeBackupPath = null;
}
writeRepairSidecar(dataDir, sidecar);
if (outcome === 'repaired') {
if (outcome === 'repaired' && opts?.closeEpisode !== false) {
pruneRepairBackups(dataDir);
}
}
/** Cooldown: true when the last FAILED attempt is inside the cooldown window. */
/**
* Close any open repair episode after a HEALTHY connect (red-team finding: a
* plain successful open never touched the sidecar, so an episode stayed open
* forever — doctor kept reporting corruption-likely, and a weeks-stale
* episode backup could be reused over much newer data). Cheap no-op when no
* sidecar exists. Called by PGLiteEngine.connect() on every non-repaired
* success and by the seam's 'repaired' arm via recordRepairAttempt.
*/
export function closeRepairEpisodeIfOpen(dataDir: string): void {
try {
if (!existsSync(sidecarPath(dataDir))) return;
const sidecar = readRepairSidecar(dataDir);
if (sidecar.episodeStartedAt === null) return;
sidecar.episodeStartedAt = null;
sidecar.episodeBackupPath = null;
writeRepairSidecar(dataDir, sidecar);
pruneRepairBackups(dataDir);
} catch { /* best-effort */ }
}
/**
* Cooldown: true when the last FAILED attempt is inside the cooldown window.
* DELIBERATE: a later successful repair does NOT clear the cooldown — repeated
* corruption right after a "success" usually means the unclean-shutdown
* genesis is still active, and looping surgery would silently eat a WAL tail
* per cycle. The manual `gbrain pglite-repair` command bypasses the cooldown.
*/
export function repairCooldownActive(dataDir: string): { active: boolean; detail: string } {
const seconds = Number(process.env.GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS ?? DEFAULT_COOLDOWN_SECONDS);
const windowMs = (Number.isFinite(seconds) && seconds >= 0 ? seconds : DEFAULT_COOLDOWN_SECONDS) * 1000;
if (windowMs === 0) return { active: false, detail: 'cooldown disabled (0s)' };
const sidecar = readRepairSidecar(dataDir);
// Repaired-loop guard (red-team finding): a crash loop where every reopen
// aborts but repair "succeeds" each time would silently discard a WAL tail
// per cycle with no failed attempt ever recorded. Two successful repairs
// inside one window = the corruption genesis is active — stop auto-repair
// and let doctor escalate.
const repairedInWindow = sidecar.attempts.filter(
(a) => a.outcome === 'repaired' && Date.now() - a.ts >= 0 && Date.now() - a.ts < windowMs,
).length;
if (repairedInWindow >= 2) {
return {
active: true,
detail:
`auto-repair already ran ${repairedInWindow}x in the last ${windowMs / 1000}s — repeated ` +
'corruption means the unclean-shutdown genesis is still active; refusing to silently ' +
'discard another WAL tail. Run `gbrain doctor`, or `gbrain pglite-repair` manually.',
};
}
const lastFailed = [...sidecar.attempts].reverse().find((a) => a.outcome === 'failed');
if (!lastFailed) return { active: false, detail: 'no prior failed attempt' };
const ageMs = Date.now() - lastFailed.ts;
if (ageMs < windowMs) {
// Clock skew (unclean-reboot recovery is exactly when clocks step): a
// negative age means the recorded ts is in the future — treat as expired
// rather than suppressing auto-repair until wall-clock catches up.
if (ageMs >= 0 && ageMs < windowMs) {
return {
active: true,
detail:
@@ -241,8 +334,17 @@ function isSymlink(path: string): boolean {
export function validateWalRepairTarget(dataDir: string): WalRepairValidation {
if (!dataDir) return { ok: false, reason: 'missing-dir', detail: 'no data dir configured (in-memory engine)' };
if (!existsSync(dataDir)) return { ok: false, reason: 'missing-dir', detail: `${dataDir} does not exist` };
if (isSymlink(dataDir) || isSymlink(join(dataDir, 'pg_wal')) || isSymlink(join(dataDir, 'global', 'pg_control'))) {
return { ok: false, reason: 'not-pglite-layout', detail: 'data dir, pg_wal, or pg_control is a symlink — refusing to run rename-based repair through symlinks' };
if (
isSymlink(dataDir) ||
isSymlink(join(dataDir, 'pg_wal')) ||
// `global/` itself must be checked too: lstat on global/pg_control follows
// the INTERMEDIATE symlink, so a symlinked global/ would pass and surgery
// would write a forged pg_control through it into a foreign directory
// (security review finding).
isSymlink(join(dataDir, 'global')) ||
isSymlink(join(dataDir, 'global', 'pg_control'))
) {
return { ok: false, reason: 'not-pglite-layout', detail: 'data dir, pg_wal, global, or pg_control is a symlink — refusing to run rename-based repair through symlinks' };
}
let pgVersion: string;
try {
@@ -256,11 +358,27 @@ export function validateWalRepairTarget(dataDir: string): WalRepairValidation {
if (!existsSync(join(dataDir, 'base'))) {
return { ok: false, reason: 'not-pglite-layout', detail: `no base/ directory in ${dataDir}` };
}
// Live-postmaster refusal (red-team finding): real pg_resetwal refuses when
// postmaster.pid exists. A LIVE native Postgres 17 data dir passes every
// layout check here — without this guard, `gbrain pglite-repair --path` at
// such a dir would rename pg_wal out from under a running postmaster the
// gbrain lock cannot see. A stale pid file (dead process) stays repairable.
try {
const pidRaw = readFileSync(join(dataDir, 'postmaster.pid'), 'utf-8').split('\n')[0]?.trim();
const pid = Number(pidRaw);
if (Number.isInteger(pid) && pid > 0 && isProcessAlive(pid)) {
return {
ok: false,
reason: 'not-pglite-layout',
detail: `postmaster.pid names a LIVE process (PID ${pid}) — refusing WAL surgery on a possibly-running database`,
};
}
} catch { /* no postmaster.pid or unreadable — fine */ }
const controlPath = join(dataDir, 'global', 'pg_control');
try {
const size = statSync(controlPath).size;
if (size !== 8192) {
return { ok: false, reason: 'bad-pg-control', detail: `pg_control is ${size} bytes, expected 8192` };
if (size !== PG_CONTROL_FILE_SIZE) {
return { ok: false, reason: 'bad-pg-control', detail: `pg_control is ${size} bytes, expected ${PG_CONTROL_FILE_SIZE}` };
}
} catch {
return { ok: false, reason: 'bad-pg-control', detail: `no readable ${controlPath}` };
@@ -289,19 +407,16 @@ export function inspectPgliteDataDir(dataDir: string): PgliteDirDiagnosis {
base.pgVersion = readFileSync(join(dataDir, 'PG_VERSION'), 'utf-8').trim();
} catch { /* leave null */ }
try {
base.pgControlOk = statSync(join(dataDir, 'global', 'pg_control')).size === 8192;
base.pgControlOk = statSync(join(dataDir, 'global', 'pg_control')).size === PG_CONTROL_FILE_SIZE;
} catch { /* leave false */ }
try {
base.walSegments = readdirSync(join(dataDir, 'pg_wal')).filter((f) => /^[0-9A-F]{24}(?:\.partial)?$/.test(f)).sort();
base.walSegments = readdirSync(join(dataDir, 'pg_wal')).filter(isWalSegmentName).sort();
} catch { /* leave empty */ }
try {
const lockData = JSON.parse(readFileSync(join(dataDir, '.gbrain-lock', 'lock'), 'utf-8')) as { pid?: number };
if (typeof lockData.pid === 'number') {
try {
process.kill(lockData.pid, 0);
base.lockHeld = true;
base.lockHolderPid = lockData.pid;
} catch { /* holder dead — not held */ }
if (typeof lockData.pid === 'number' && isProcessAlive(lockData.pid)) {
base.lockHeld = true;
base.lockHolderPid = lockData.pid;
}
} catch { /* no lock / unreadable — not held */ }
@@ -341,33 +456,63 @@ export async function repairPgliteWal(
throw new WalResetUnsupportedError(`refusing repair: ${validation.detail}`);
}
let backupPath: string;
// Defense-in-depth (security + red-team reviews): the reuse path comes from
// the user-writable sidecar JSON — only honor it when it is a real,
// non-symlink, traversal-free sibling backup dir of THIS data dir that
// STILL CONTAINS pg_wal (a restore MOVES pg_wal back out, gutting the
// backup; reusing a gutted backup would let resetWal unlink the only
// surviving WAL copy in place). Anything else gets a fresh backup.
const safeReusePath =
opts?.reuseBackupPath &&
opts.reuseBackupPath.startsWith(`${dataDir}${BACKUP_DIR_MARKER}`) &&
!opts.reuseBackupPath.includes('..') &&
existsSync(opts.reuseBackupPath) &&
!isSymlink(opts.reuseBackupPath) &&
existsSync(join(opts.reuseBackupPath, 'pg_wal')) &&
!isSymlink(join(opts.reuseBackupPath, 'pg_wal'))
? opts.reuseBackupPath
: undefined;
const backedUpFiles: string[] = [];
const reusedEpisodeBackup = !!opts?.reuseBackupPath && existsSync(opts.reuseBackupPath);
const reusedEpisodeBackup = !!safeReusePath;
let backupPath: string;
if (reusedEpisodeBackup) {
// Episode reuse: the open episode's backup already holds the pre-damage
// state (restore-on-failure returned the dir to exactly that state, or a
// restored:false dir is in reset-state whose re-backup would be useless).
// resetWal's own deletion loops clear the current segments in place.
backupPath = opts!.reuseBackupPath!;
// Episode reuse: the open episode's backup still holds pg_wal (enforced by
// safeReusePath — a restore MOVES pg_wal back out and guts the backup; a
// gutted backup must never be reused or resetWal would unlink the only
// surviving WAL copy in place). resetWal's own deletion loops clear the
// current segments.
backupPath = safeReusePath!;
} else {
// Fresh backup dir: mkdir with recursive:false and fail-closed on
// collision (red-team: a predictable pre-existing dir or symlink-to-dir
// would silently receive the renames, and restore would later read
// pg_control bytes back OUT of it). Retry with a suffix, then verify we
// created a real directory.
backupPath = `${dataDir}${BACKUP_DIR_MARKER}${Date.now()}`;
mkdirSync(backupPath, { recursive: true });
const walDir = join(dataDir, 'pg_wal');
if (existsSync(walDir)) {
await rename(walDir, join(backupPath, 'pg_wal'));
backedUpFiles.push('pg_wal/');
for (let attempt = 0; ; attempt++) {
try {
mkdirSync(backupPath, { recursive: false });
break;
} catch (err) {
if ((err as NodeJS.ErrnoException)?.code === 'EEXIST' && attempt < 5) {
backupPath = `${dataDir}${BACKUP_DIR_MARKER}${Date.now()}-${attempt + 1}`;
continue;
}
throw err;
}
}
const pidFile = join(dataDir, 'postmaster.pid');
if (existsSync(pidFile)) {
await rename(pidFile, join(backupPath, 'postmaster.pid'));
backedUpFiles.push('postmaster.pid');
if (isSymlink(backupPath) || !statSync(backupPath).isDirectory()) {
throw new WalResetUnsupportedError(`backup path ${backupPath} is not a real directory`);
}
const control = await readFile(join(dataDir, 'global', 'pg_control'));
await writeFileAtomicSynced(backupPath, 'pg_control', Buffer.from(control));
backedUpFiles.push('global/pg_control');
}
// Track whether the backup dir received anything, so refusal paths can prune
// an empty leftover (adversarial review F11: doctor would otherwise inventory
// an empty `<dataDir>.wal-repair-backup-*` as a real backup).
const pruneEmptyBackup = () => {
if (reusedEpisodeBackup) return;
try { if (readdirSync(backupPath).length === 0) rmSync(backupPath, { recursive: true, force: true }); } catch { /* best-effort */ }
};
const receipt: WalRepairReceipt = {
dataDir,
@@ -380,14 +525,46 @@ export async function repairPgliteWal(
repairedAt: new Date().toISOString(),
};
if (!reusedEpisodeBackup) {
// Backup phase. Once the FIRST rename lands, any failure here must run a
// restore and surface via WalRepairError — a generic throw would read as
// "dir never touched" while pg_wal is actually sitting in the backup dir
// (red-team finding).
let backupStarted = false;
try {
const walDir = join(dataDir, 'pg_wal');
if (existsSync(walDir)) {
await rename(walDir, join(backupPath, 'pg_wal'));
backupStarted = true;
backedUpFiles.push('pg_wal/');
}
const pidFile = join(dataDir, 'postmaster.pid');
if (existsSync(pidFile)) {
await rename(pidFile, join(backupPath, 'postmaster.pid'));
backupStarted = true;
backedUpFiles.push('postmaster.pid');
}
const control = await readFile(join(dataDir, 'global', 'pg_control'));
await writeFileAtomicSynced(backupPath, 'pg_control', Buffer.from(control));
backedUpFiles.push('global/pg_control');
} catch (err) {
if (!backupStarted) { pruneEmptyBackup(); throw err; } // dir genuinely untouched
const restore = await restoreWalBackup(receipt);
throw new WalRepairError(String((err as Error)?.message ?? err), receipt, restore);
}
}
try {
const result = await resetWal(dataDir);
receipt.resetSegment = result.resetSegment;
receipt.timelineId = result.timelineId;
receipt.walSegSize = result.walSegSize;
} catch (err) {
await restoreWalBackup(receipt); // best-effort — never leave backed-up-but-unrepaired
throw err;
// Best-effort restore — never leave backed-up-but-unrepaired. The result
// is THREADED OUT via WalRepairError so callers can report `restored`
// honestly (review finding: discarding it let 'failed-restored' lie).
const restore = await restoreWalBackup(receipt);
throw new WalRepairError(String((err as Error)?.message ?? err), receipt, restore);
}
return receipt;
}
@@ -411,11 +588,18 @@ export async function restoreWalBackup(receipt: WalRepairReceipt): Promise<Resto
const backupWal = join(backupPath, 'pg_wal');
const backupControl = join(backupPath, 'pg_control');
// Symlinked backup CHILDREN would let restore read attacker-chosen
// pg_control bytes or rename a foreign pg_wal into the data dir
// (red-team) — the top-level checks don't cover them.
if (isSymlink(backupWal) || isSymlink(backupControl)) {
return { restored: false, steps, detail: `backup at ${backupPath} contains symlinked components — refusing restore` };
}
// Mtime guard: any foreign WAL segment newer than this repair's start?
const backupTs = Date.parse(receipt.repairedAt);
if (existsSync(walDir)) {
for (const f of readdirSync(walDir)) {
if (!/^[0-9A-F]{24}(?:\.partial)?$/.test(f) || f === receipt.resetSegment) continue;
if (!isWalSegmentName(f) || f === receipt.resetSegment) continue;
try {
if (statSync(join(walDir, f)).mtimeMs > backupTs) {
return {
@@ -484,6 +668,22 @@ export async function attemptWalRepairAndRetry<T>(
'none is running), then re-run; a cleanly-acquired lock enables auto-repair.',
};
}
const sinceReap = msSinceLastReap(dataDir);
// `>= 0` guard (adversarial review F5): a future-dated marker (clock step
// during the unclean-reboot recovery this feature exists for) yields a
// negative age; treat it as expired rather than quarantining forever —
// same policy as repairCooldownActive.
if (sinceReap !== null && sinceReap >= 0 && sinceReap < REAP_QUARANTINE_MS) {
return {
status: 'skipped',
reason: 'possibly-live-writer',
detail:
`a lock on this brain was reaped ${Math.round(sinceReap / 1000)}s ago (possibly from a ` +
'still-live process) — auto-repair stays off for ' +
`${REAP_QUARANTINE_MS / 60000} minutes after any reap. Confirm no gbrain process is ` +
'running, then re-run or use `gbrain pglite-repair`.',
};
}
const validation = validateWalRepairTarget(dataDir);
if (!validation.ok) {
return { status: 'skipped', reason: 'validation-failed', detail: validation.detail };
@@ -493,20 +693,42 @@ export async function attemptWalRepairAndRetry<T>(
return { status: 'skipped', reason: 'recently-failed', detail: cooldown.detail };
}
process.stderr.write(
`gbrain: PGLite failed to open ${dataDir} — attempting automatic WAL repair ` +
`(backup at ${dataDir}${BACKUP_DIR_MARKER}*). If this command times out, run ` +
`\`gbrain pglite-repair\` to finish. Disable auto-repair with GBRAIN_PGLITE_WAL_REPAIR=off.\n`,
);
try {
process.stderr.write(
`gbrain: PGLite failed to open ${dataDir} — attempting automatic WAL repair ` +
`(backup at ${dataDir}${BACKUP_DIR_MARKER}*). If this command times out, run ` +
`\`gbrain pglite-repair\` to finish. Disable auto-repair with GBRAIN_PGLITE_WAL_REPAIR=off.\n`,
);
} catch { /* EPIPE under a closed-pipe daemon parent must not read as surgery failure */ }
const sidecar = readRepairSidecar(dataDir);
// Stale-episode bound (red-team): an episode left open for a long time
// means the pinned backup may predate real data — take a fresh backup.
const episodeFresh =
sidecar.episodeStartedAt !== null &&
Date.now() - sidecar.episodeStartedAt >= 0 &&
Date.now() - sidecar.episodeStartedAt < 24 * 3600 * 1000;
let receipt: WalRepairReceipt;
try {
receipt = await repairPgliteWal(dataDir, {
reuseBackupPath: sidecar.episodeBackupPath ?? undefined,
reuseBackupPath: episodeFresh ? sidecar.episodeBackupPath ?? undefined : undefined,
});
} catch (err) {
// repairPgliteWal already best-effort-restored; no receipt to restore from.
if (err instanceof WalRepairError) {
// Reset failed AFTER the backup was taken; report the best-effort
// restore's REAL outcome (hardcoding restored:true here made the
// 'failed-restored' message lie when the restore itself failed).
recordRepairAttempt(dataDir, 'failed', err.receipt.backupPath);
return {
status: 'failed',
receipt: err.receipt,
restored: err.restore.restored,
repairError: err.message +
(err.restore.restored ? '' : ` [restore: ${err.restore.detail}]`),
};
}
// Pre-backup refusal (validation) — the dir was never touched, so there
// is nothing to restore and `restored: true` reads as "dir intact".
recordRepairAttempt(dataDir, 'failed', sidecar.episodeBackupPath);
return {
status: 'failed',
@@ -534,10 +756,15 @@ export async function attemptWalRepairAndRetry<T>(
} catch (err) {
// The seam's never-throw contract is load-bearing (single lock-release site
// in connect()'s catch) — any unexpected error degrades to 'failed'.
// `restored: true` here is honest: repairPgliteWal/restoreWalBackup handle
// their own mutation failures via WalRepairError above, so a throw landing
// HERE happened outside surgery and the dir is untouched (red-team: the
// old restored:false told users to manually restore a nonexistent backup).
try { recordRepairAttempt(dataDir, 'failed', null); } catch { /* best-effort — cooldown still engages when possible */ }
return {
status: 'failed',
receipt: null,
restored: false,
restored: true,
repairError: `unexpected repair-path error: ${String((err as Error)?.message ?? err)}`,
};
}
src/core/pglite-resetwal.ts
+33 -4
View File
@@ -34,12 +34,23 @@ import { existsSync } from 'node:fs';
import { mkdir, open, readdir, readFile, rename, unlink } from 'node:fs/promises';
import { join } from 'node:path';
const PG_CONTROL_FILE_SIZE = 8192;
// Exported: pglite-repair.ts validates against the same layout literals — the
// PG17 coupling the TODOS "pglite upgrade blocker" entry says moves together.
export const PG_CONTROL_FILE_SIZE = 8192;
const WAL_SEGMENT_RE = /^[0-9A-F]{24}(?:\.partial)?$/;
/** Is this filename a WAL segment (incl. `.partial`)? Shared layout predicate. */
export function isWalSegmentName(name: string): boolean {
return WAL_SEGMENT_RE.test(name);
}
const PG_CONTROL_VERSION = 1700;
const DB_SHUTDOWNED = 1;
const XLOG_BLCKSZ = 8192;
const MIN_WAL_SEG_SIZE = 1024 * 1024;
const MAX_WAL_SEG_SIZE = 1024 * 1024 * 1024;
// Postgres-general max is 1GB, but this port targets pglite (ships 16MB
// segments). A corrupt-but-plausible control field must not be able to drive
// a 1GB zero-fill allocation + write on the repair path (perf review) — cap
// at 64MB and fail closed above it.
const MAX_WAL_SEG_SIZE = 64 * 1024 * 1024;
const SIZE_OF_XLOG_LONG_PHD = 40;
const SIZE_OF_XLOG_RECORD = 24;
const SIZE_OF_CHECKPOINT = 88;
@@ -154,7 +165,11 @@ async function unlinkIfExists(path: string): Promise<void> {
export async function writeFileAtomicSynced(dir: string, name: string, data: Buffer): Promise<void> {
const tmpName = `.${name}.tmp-${process.pid}`;
const tmpPath = join(dir, tmpName);
const file = await open(tmpPath, 'w');
// 'wx' (exclusive create, never follows an existing symlink) after clearing
// any stale tmp: a pre-planted symlink at the predictable tmp path must not
// redirect the write (security review).
await unlinkIfExists(tmpPath);
const file = await open(tmpPath, 'wx');
try {
await file.writeFile(data);
await file.sync();
@@ -206,6 +221,20 @@ export async function resetWal(rootDir: string): Promise<WalResetResult> {
if (control.readUInt32LE(OFF.pgControlVersion) !== PG_CONTROL_VERSION) {
throw new WalResetUnsupportedError('Unsupported pg_control version');
}
// Verify the STORED CRC before trusting (and re-signing) the checkpoint copy
// (adversarial review F6): a torn pg_control with an intact version field
// but garbage checkpoint counters (nextXid/nextOid/...) would otherwise be
// preserved verbatim and laundered under a fresh valid CRC — Postgres then
// starts and corrupts silently (xid-wraparound class). Real pg_resetwal
// refuses on CRC mismatch; so do we → the caller falls to the rebuild rung.
const storedCrc = control.readUInt32LE(OFF.crc);
if (crc32c([control.subarray(0, OFF.crc)]) !== storedCrc) {
throw new WalResetUnsupportedError(
'pg_control CRC mismatch — the control file itself is damaged; WAL reset ' +
'would launder corrupt checkpoint counters. Rebuild the brain instead ' +
'(`gbrain reinit-pglite`).',
);
}
const walSegSize = control.readUInt32LE(OFF.xlogSegSize);
const xlogBlcksz = control.readUInt32LE(OFF.xlogBlcksz);
@@ -260,7 +289,7 @@ export async function resetWal(rootDir: string): Promise<WalResetResult> {
control.writeUInt32LE(crc32c([control.subarray(0, OFF.crc)]), OFF.crc);
for (const file of await readdir(walDir)) {
if (/^[0-9A-F]{24}(?:\.partial)?$/.test(file)) {
if (isWalSegmentName(file)) {
await unlink(join(walDir, file));
}
}
test/pglite-init-classifier.test.ts
+21
View File
@@ -232,6 +232,27 @@ describe('stringifyPgliteInitError — non-Error rejections (#2674)', () => {
expect(stringifyPgliteInitError(null)).toBe('null');
expect(stringifyPgliteInitError(undefined)).toBe('undefined');
});
// WAL-repair wave: Emscripten's FS layer throws message-LESS objects (e.g.
// `ErrnoError { name: 'ErrnoError', errno: 20 }` when the data dir is a
// symlink NODEFS refuses to mount) — never "[object Object]".
test('message-less ErrnoError-shaped object yields name + errno', () => {
expect(stringifyPgliteInitError({ name: 'ErrnoError', errno: 20 })).toBe('ErrnoError (errno 20)');
});
test('message-less nameless object with other props yields its JSON', () => {
expect(stringifyPgliteInitError({ code: 'ENOENT' })).toBe('{"code":"ENOENT"}');
});
test('message-less object with a name and serializable props yields name-prefixed JSON', () => {
expect(stringifyPgliteInitError({ name: 'Weird' })).toBe('Weird: {"name":"Weird"}');
});
test('circular object with a name falls back to the bare name (JSON.stringify throws)', () => {
const c: Record<string, unknown> = { name: 'Circ' };
c.self = c;
expect(stringifyPgliteInitError(c)).toBe('Circ');
});
});
describe('#1340 reproducer — exact reporter error string maps to bunfs', () => {
test/pglite-lock.test.ts
+96 -1
View File
@@ -1,5 +1,5 @@
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs';
import { mkdirSync, mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock';
@@ -271,3 +271,98 @@ describe('pglite-lock #2058 heartbeat + steal-grace', () => {
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(false);
});
});
describe('pglite-lock reap classification (WAL-repair wave)', () => {
// Unique per-test tmpdirs: the reap marker lands at `${dataDir}.lock-reap.json`
// — a SIBLING of the data dir — so each test gets its own parent to rm.
function freshDataDir(): { parent: string; dataDir: string } {
const parent = mkdtempSync(join(tmpdir(), 'gbrain-lock-reap-'));
return { parent, dataDir: join(parent, 'data') };
}
/**
* A PID that provably belongs to no live process: spawn a short-lived child,
* wait for it (spawnSync reaps it), then verify kill(pid, 0) throws. Retries
* to dodge instant PID reuse.
*/
function deadPid(): number {
for (let attempt = 0; attempt < 5; attempt++) {
const proc = Bun.spawnSync(['bash', '-c', 'exit 0']);
const pid = proc.pid;
try {
process.kill(pid, 0); // still alive/visible → PID reused, try again
} catch {
return pid;
}
}
throw new Error('could not obtain a provably-dead PID after 5 spawns');
}
test('corrupt lock file: reaped acquisition + persisted .lock-reap.json marker', async () => {
const { parent, dataDir } = freshDataDir();
try {
const lockDir = join(dataDir, '.gbrain-lock');
mkdirSync(lockDir, { recursive: true });
writeFileSync(join(lockDir, 'lock'), 'not json {{{'); // holder liveness UNKNOWABLE
const lock = await acquireLock(dataDir, { timeoutMs: 5000 });
try {
expect(lock.acquired).toBe(true);
expect(lock.reaped).toBe(true);
// Unknowable-liveness reap is persisted cross-process for the repair gate.
expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(true);
const marker = JSON.parse(readFileSync(`${dataDir}.lock-reap.json`, 'utf-8'));
expect(typeof marker.ts).toBe('number');
expect(marker.by).toBe(process.pid);
} finally {
await releaseLock(lock);
}
} finally {
rmSync(parent, { recursive: true, force: true });
}
});
test('clean acquisition: reaped falsy, no .lock-reap.json marker', async () => {
const { parent, dataDir } = freshDataDir();
try {
const lock = await acquireLock(dataDir, { timeoutMs: 5000 });
try {
expect(lock.acquired).toBe(true);
expect(lock.reaped).toBeFalsy();
expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(false);
} finally {
await releaseLock(lock);
}
} finally {
rmSync(parent, { recursive: true, force: true });
}
});
test('dead-PID lock: reaped acquisition but NO marker (affirmative ESRCH verdict)', async () => {
const { parent, dataDir } = freshDataDir();
try {
const lockDir = join(dataDir, '.gbrain-lock');
mkdirSync(lockDir, { recursive: true });
const now = Date.now();
writeFileSync(join(lockDir, 'lock'), JSON.stringify({
pid: deadPid(),
acquired_at: now - 60_000,
refreshed_at: now - 60_000,
command: 'gbrain embed',
subcommand: 'embed',
}));
const lock = await acquireLock(dataDir, { timeoutMs: 5000 });
try {
expect(lock.acquired).toBe(true);
expect(lock.reaped).toBe(true);
// Dead-PID reaps deliberately do NOT quarantine the next acquirer.
expect(existsSync(`${dataDir}.lock-reap.json`)).toBe(false);
} finally {
await releaseLock(lock);
}
} finally {
rmSync(parent, { recursive: true, force: true });
}
}, 30_000);
});
test/pglite-repair-command.serial.test.ts
+100
View File
@@ -25,6 +25,7 @@ import { basename, join } from 'node:path';
import { runPgliteRepair } from '../src/commands/pglite-repair.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { withEnv } from './helpers/with-env.ts';
function tmp(prefix: string): string {
return mkdtempSync(join(tmpdir(), prefix));
@@ -264,6 +265,60 @@ describe('gbrain pglite-repair — refusals (validate before lock, never mkdir a
});
});
describe('gbrain pglite-repair — TTY + config gates', () => {
test('8. non-TTY without --yes refuses: exit 1, no_tty_no_yes, zero mutation', async () => {
const dir = join(tmp('gbrain-repair-notty-'), 'brain.pglite');
makeFakeLayout(dir);
// Pin stdin to non-TTY: under `bun test` in a terminal stdin can still be
// a TTY, which would route into the interactive confirm instead.
const origTty = Object.getOwnPropertyDescriptor(process.stdin, 'isTTY');
Object.defineProperty(process.stdin, 'isTTY', { value: false, configurable: true });
const cap = captureConsole();
let rc: number;
try {
rc = await runPgliteRepair(['--path', dir, '--json']); // no --yes
} finally {
cap.restore();
if (origTty) Object.defineProperty(process.stdin, 'isTTY', origTty);
else delete (process.stdin as unknown as Record<string, unknown>).isTTY;
}
expect(rc).toBe(1);
const out = parseJsonLine(cap.logs);
expect(out.status).toBe('error');
expect(out.code).toBe('no_tty_no_yes');
// Refused BEFORE any surgery: no backup dir, no sidecar.
expect(backupDirsBeside(dir)).toEqual([]);
expect(existsSync(`${dir}.wal-repair-attempt.json`)).toBe(false);
});
test('9. no --path with a non-pglite configured engine: exit 1, not_pglite', async () => {
// Hermetic GBRAIN_HOME (same convention as apply-migrations-pglite-spawn):
// configDir() appends '.gbrain', so the config lands at <home>/.gbrain/.
const home = tmp('gbrain-repair-home-');
mkdirSync(join(home, '.gbrain'), { recursive: true });
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({ engine: 'postgres', database_url: 'postgresql://localhost:5432/x' }) + '\n',
);
const cap = captureConsole();
let rc: number;
try {
rc = await withEnv({ GBRAIN_HOME: home }, () => runPgliteRepair(['--yes', '--json']));
} finally {
cap.restore();
}
expect(rc).toBe(1);
const out = parseJsonLine(cap.logs);
expect(out.status).toBe('error');
expect(out.code).toBe('not_pglite');
expect(out.message).toContain('--path');
});
});
describe('gbrain pglite-repair — the happy path (real PGLite)', () => {
test('4. corrupt pg_wal → repair in place → data survives, no auto-repair on reconnect', async () => {
const dir = join(tmp('gbrain-repair-happy-'), 'brain.pglite');
@@ -316,3 +371,48 @@ describe('gbrain pglite-repair — the happy path (real PGLite)', () => {
}
}, 180_000);
});
describe('gbrain pglite-repair — argument + quarantine hardening (adversarial fixes)', () => {
test('unknown flag is rejected (exit 2), not silently ignored on a destructive command', async () => {
const cap = captureConsole();
let rc: number;
try {
rc = await runPgliteRepair(['--dry-rnu', '--yes', '--json']);
} finally {
cap.restore();
}
expect(rc).toBe(2);
expect(parseJsonLine(cap.logs).code).toBe('unknown_flag');
});
test('--path with no value is rejected (does NOT retarget the default brain)', async () => {
const cap = captureConsole();
let rc: number;
try {
rc = await runPgliteRepair(['--yes', '--json', '--path']);
} finally {
cap.restore();
}
expect(rc).toBe(2);
expect(parseJsonLine(cap.logs).code).toBe('unknown_flag');
});
test('the command honors the cross-process reap quarantine (F3: a second --yes cannot bypass it)', async () => {
const dir = join(mkdtempSync(join(tmpdir(), 'pgrepaircmd-')), 'brain.pglite');
makeFakeLayout(dir);
// A fresh corrupt-lock reap marker from a prior run — the possibly-live
// writer it protects must not be repaired under.
writeFileSync(`${dir}.lock-reap.json`, JSON.stringify({ ts: Date.now(), by: 999999 }), { mode: 0o644 });
const cap = captureConsole();
let rc: number;
try {
rc = await runPgliteRepair(['--path', dir, '--yes', '--json']);
} finally {
cap.restore();
}
expect(rc).toBe(1);
expect(parseJsonLine(cap.logs).code).toBe('refused_reap_quarantine');
// No surgery: no backup dir created.
expect(readdirSync(join(dir, '..')).some((f) => f.includes('.wal-repair-backup-'))).toBe(false);
});
});
test/pglite-repair.test.ts
+229 -43
View File
@@ -15,7 +15,7 @@ import {
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { withEnv } from './helpers/with-env.ts';
import { xlogFileName } from '../src/core/pglite-resetwal.ts';
import { xlogFileName, crc32c } from '../src/core/pglite-resetwal.ts';
import {
validateWalRepairTarget,
inspectPgliteDataDir,
@@ -27,7 +27,8 @@ import {
repairCooldownActive,
listRepairBackups,
pruneRepairBackups,
walRepairEnabled,
WalRepairError,
closeRepairEpisodeIfOpen,
} from '../src/core/pglite-repair.ts';
const SEG_SIZE = 1024 * 1024;
@@ -40,6 +41,7 @@ function makeControl(): Buffer {
control.writeUInt32LE(8192, 224); // xlogBlcksz
control.writeUInt32LE(SEG_SIZE, 228); // xlogSegSize
control.writeBigUInt64LE(3n * BigInt(SEG_SIZE) + 40n, 40); // redo → seg 3
control.writeUInt32LE(crc32c([control.subarray(0, 288)]), 288); // valid CRC
return control;
}
@@ -60,6 +62,18 @@ function makeLayout(opts?: { segments?: string[]; postmasterPid?: boolean }): st
return dir;
}
/**
* Overwrite pg_control with an 8192-byte buffer carrying a WRONG control
* version: it PASSES validateWalRepairTarget (size-only check) but FAILS
* resetWal's version check — the fixture for the reset-fails-AFTER-backup
* (WalRepairError) path.
*/
function poisonControlVersion(dir: string): void {
const control = makeControl();
control.writeUInt32LE(1600, 8);
writeFileSync(join(dir, 'global', 'pg_control'), control);
}
describe('validateWalRepairTarget', () => {
test('accepts a full PG17 layout, tolerating .gbrain-lock inside it', () => {
const dir = makeLayout();
@@ -101,6 +115,20 @@ describe('validateWalRepairTarget', () => {
symlinkSync(walBackup, join(dir, 'pg_wal'));
expect(validateWalRepairTarget(dir)).toMatchObject({ ok: false, reason: 'not-pglite-layout' });
});
test('refuses a symlinked global/ dir (security review — lstat on pg_control follows the intermediate link)', () => {
const dir = makeLayout();
// A foreign dir holding a perfectly valid 8192-byte pg_control: without the
// global/ lstat check, surgery would write a forged control THROUGH the
// link into this directory.
const foreign = mkdtempSync(join(tmpdir(), 'pgrepair-foreign-'));
writeFileSync(join(foreign, 'pg_control'), makeControl());
rmSync(join(dir, 'global'), { recursive: true });
symlinkSync(foreign, join(dir, 'global'));
const result = validateWalRepairTarget(dir);
expect(result).toMatchObject({ ok: false, reason: 'not-pglite-layout' });
if (!result.ok) expect(result.detail).toContain('symlink');
});
});
describe('repairPgliteWal — rename-based backup', () => {
@@ -127,9 +155,29 @@ describe('repairPgliteWal — rename-based backup', () => {
test('refuses (typed) on an invalid layout without touching anything', async () => {
const dir = mkdtempSync(join(tmpdir(), 'pgrepair-'));
expect(repairPgliteWal(dir)).rejects.toThrow(/refusing repair/);
await expect(repairPgliteWal(dir)).rejects.toThrow(/refusing repair/);
expect(listRepairBackups(dir)).toEqual([]);
});
test('throws WalRepairError after backup and restores the dir when resetWal fails', async () => {
const seg = xlogFileName(1, 3n, SEG_SIZE);
const dir = makeLayout({ segments: [seg] });
poisonControlVersion(dir); // passes validation, fails resetWal
let caught: unknown;
try {
await repairPgliteWal(dir);
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(WalRepairError);
const err = caught as WalRepairError;
// The best-effort restore ran and is reported HONESTLY on the error.
expect(err.restore.restored).toBe(true);
expect(existsSync(join(dir, 'pg_wal'))).toBe(true);
expect(existsSync(join(dir, 'pg_wal', seg))).toBe(true); // original segment back
expect(existsSync(err.receipt.backupPath)).toBe(true); // forensic backup kept
});
});
describe('restoreWalBackup — overwrite order + guards', () => {
@@ -182,15 +230,31 @@ describe('restoreWalBackup — overwrite order + guards', () => {
describe('cooldown sidecar + episode retention', () => {
test('recordRepairAttempt opens an episode on failure, closes on success, caps history', () => {
const dir = makeLayout();
recordRepairAttempt(dir, 'failed', '/b/one');
// Real backup dirs: the re-pin rule inspects them for pg_wal (a gutted
// pinned backup — restore moved its pg_wal back — must lose the pin).
const backupOne = `${dir}.wal-repair-backup-1001`;
const backupTwo = `${dir}.wal-repair-backup-1002`;
const backupThree = `${dir}.wal-repair-backup-1003`;
mkdirSync(join(backupOne, 'pg_wal'), { recursive: true });
mkdirSync(join(backupTwo, 'pg_wal'), { recursive: true });
mkdirSync(join(backupThree, 'pg_wal'), { recursive: true });
recordRepairAttempt(dir, 'failed', backupOne);
let sidecar = readRepairSidecar(dir);
expect(sidecar.episodeStartedAt).not.toBeNull();
expect(sidecar.episodeBackupPath).toBe('/b/one');
// Second failure does NOT re-pin the episode backup.
recordRepairAttempt(dir, 'failed', '/b/two');
expect(sidecar.episodeBackupPath).toBe(backupOne);
// Second failure does NOT re-pin while the pinned backup still holds pg_wal.
recordRepairAttempt(dir, 'failed', backupTwo);
sidecar = readRepairSidecar(dir);
expect(sidecar.episodeBackupPath).toBe('/b/one');
recordRepairAttempt(dir, 'repaired', '/b/one');
expect(sidecar.episodeBackupPath).toBe(backupOne);
// …but a GUTTED pinned backup loses the pin to the fresh one (red-team:
// the episode's protected copy must always be one that still has pg_wal).
rmSync(join(backupOne, 'pg_wal'), { recursive: true });
recordRepairAttempt(dir, 'failed', backupThree);
sidecar = readRepairSidecar(dir);
expect(sidecar.episodeBackupPath).toBe(backupThree);
recordRepairAttempt(dir, 'repaired', backupThree);
sidecar = readRepairSidecar(dir);
expect(sidecar.episodeStartedAt).toBeNull();
expect(sidecar.episodeBackupPath).toBeNull();
@@ -198,18 +262,39 @@ describe('cooldown sidecar + episode retention', () => {
expect(readRepairSidecar(dir).attempts.length).toBeLessThanOrEqual(10);
});
test('repairCooldownActive: active after a recent failure, respects the env knob', async () => {
test('unverified success (closeEpisode:false) keeps the episode open; closeRepairEpisodeIfOpen closes it', () => {
const dir = makeLayout();
expect(repairCooldownActive(dir).active).toBe(false);
recordRepairAttempt(dir, 'failed', null);
expect(repairCooldownActive(dir).active).toBe(true);
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
const backup = `${dir}.wal-repair-backup-2001`;
mkdirSync(join(backup, 'pg_wal'), { recursive: true });
recordRepairAttempt(dir, 'failed', backup);
// The manual command's unverified "repaired" must NOT close/prune.
recordRepairAttempt(dir, 'repaired', backup, { closeEpisode: false });
let sidecar = readRepairSidecar(dir);
expect(sidecar.episodeStartedAt).not.toBeNull();
expect(existsSync(backup)).toBe(true);
// A healthy connect closes it.
closeRepairEpisodeIfOpen(dir);
sidecar = readRepairSidecar(dir);
expect(sidecar.episodeStartedAt).toBeNull();
expect(sidecar.episodeBackupPath).toBeNull();
});
test('repairCooldownActive: active after a recent failure, respects the env knob', async () => {
// Pin a known baseline (default cooldown, repair enabled): an ambient
// GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS=0 would flip the assertions.
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => {
const dir = makeLayout();
expect(repairCooldownActive(dir).active).toBe(false);
recordRepairAttempt(dir, 'failed', null);
expect(repairCooldownActive(dir).active).toBe(true);
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
expect(repairCooldownActive(dir).active).toBe(false);
});
// A success clears nothing retroactively, but cooldown keys on the LAST
// failed attempt — still inside the window here.
recordRepairAttempt(dir, 'repaired', null);
expect(repairCooldownActive(dir).active).toBe(true);
});
// A success clears nothing retroactively, but cooldown keys on the LAST
// failed attempt — still inside the window here.
recordRepairAttempt(dir, 'repaired', null);
expect(repairCooldownActive(dir).active).toBe(true);
});
test('pruneRepairBackups keeps the newest 3 and never the open episode backup', () => {
@@ -294,43 +379,144 @@ describe('attemptWalRepairAndRetry — the never-throws engine seam', () => {
});
test('guards: disabled / reaped lock / validation-failed — no backup dir is ever created', async () => {
const before = process.env.GBRAIN_PGLITE_WAL_REPAIR;
const dir = makeLayout();
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: 'off' }, async () => {
const attempt = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(attempt).toMatchObject({ status: 'skipped', reason: 'disabled' });
// Pin a known baseline (repair enabled, default cooldown) so an ambient
// GBRAIN_PGLITE_WAL_REPAIR=off can't turn every arm into 'disabled'.
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => {
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: 'off' }, async () => {
const attempt = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(attempt).toMatchObject({ status: 'skipped', reason: 'disabled' });
});
const reaped = await attemptWalRepairAndRetry(dir, async () => 'x', { reaped: true });
expect(reaped).toMatchObject({ status: 'skipped', reason: 'possibly-live-writer' });
const invalid = await attemptWalRepairAndRetry('/nope/never', async () => 'x');
expect(invalid).toMatchObject({ status: 'skipped', reason: 'validation-failed' });
expect(listRepairBackups(dir)).toEqual([]);
});
const reaped = await attemptWalRepairAndRetry(dir, async () => 'x', { reaped: true });
expect(reaped).toMatchObject({ status: 'skipped', reason: 'possibly-live-writer' });
const invalid = await attemptWalRepairAndRetry('/nope/never', async () => 'x');
expect(invalid).toMatchObject({ status: 'skipped', reason: 'validation-failed' });
expect(listRepairBackups(dir)).toEqual([]);
expect(walRepairEnabled()).toBe(true); // env restored by withEnv
// withEnv restored whatever the ambient value was (including "unset").
expect(process.env.GBRAIN_PGLITE_WAL_REPAIR).toBe(before);
});
test('cooldown skip + episode backup reuse across attempts', async () => {
// Pin a known baseline: an ambient COOLDOWN_SECONDS=0 would break the
// 'recently-failed' gate assertion; an ambient WAL_REPAIR=off breaks all.
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: undefined }, async () => {
const seg = xlogFileName(1, 3n, SEG_SIZE);
const dir = makeLayout({ segments: [seg] });
// Attempt 1 fails → episode opens with backup #1.
const first = await attemptWalRepairAndRetry(dir, async () => { throw new Error('Aborted()'); });
expect(first.status).toBe('failed');
const backupsAfterFirst = listRepairBackups(dir);
expect(backupsAfterFirst.length).toBe(1);
// Immediate retry is cooldown-gated…
const gated = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(gated).toMatchObject({ status: 'skipped', reason: 'recently-failed' });
// …and with the cooldown off, the retry takes a FRESH backup: attempt 1's
// restore MOVED pg_wal back out of its backup, so reusing that gutted dir
// would let resetWal destroy the only surviving WAL copy (red-team).
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
const second = await attemptWalRepairAndRetry(dir, async () => 'db');
expect(second.status).toBe('repaired');
if (second.status === 'repaired') {
expect(second.receipt.reusedEpisodeBackup).toBe(false);
expect(second.receipt.backupPath).not.toBe(backupsAfterFirst[0]!);
}
});
expect(listRepairBackups(dir).length).toBe(2);
expect(readRepairSidecar(dir).episodeStartedAt).toBeNull(); // episode closed
});
});
test('episode backup IS reused when it still holds pg_wal (restore was blocked)', async () => {
const seg = xlogFileName(1, 3n, SEG_SIZE);
const dir = makeLayout({ segments: [seg] });
// Attempt 1 fails → episode opens with backup #1.
const first = await attemptWalRepairAndRetry(dir, async () => { throw new Error('Aborted()'); });
expect(first.status).toBe('failed');
const backupsAfterFirst = listRepairBackups(dir);
expect(backupsAfterFirst.length).toBe(1);
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR: undefined, GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
// Attempt 1: retry fails AND restore is blocked by the mtime guard
// (a foreign future-dated segment appears mid-attempt) — the backup
// KEEPS pg_wal.
const first = await attemptWalRepairAndRetry(dir, async () => {
const foreign = xlogFileName(1, 99n, SEG_SIZE);
writeFileSync(join(dir, 'pg_wal', foreign), 'live-writer-bytes');
const future = new Date(Date.now() + 60_000);
utimesSync(join(dir, 'pg_wal', foreign), future, future);
throw new Error('Aborted(). still broken');
});
expect(first.status).toBe('failed');
if (first.status === 'failed') expect(first.restored).toBe(false);
const episodeBackup = readRepairSidecar(dir).episodeBackupPath!;
expect(existsSync(join(episodeBackup, 'pg_wal'))).toBe(true);
// Clear the foreign segment so attempt 2's surgery isn't re-blocked.
rmSync(join(dir, 'pg_wal'), { recursive: true, force: true });
mkdirSync(join(dir, 'pg_wal', 'archive_status'), { recursive: true });
const second = await attemptWalRepairAndRetry(dir, async () => 'db');
expect(second.status).toBe('repaired');
if (second.status === 'repaired') {
expect(second.receipt.reusedEpisodeBackup).toBe(true);
expect(second.receipt.backupPath).toBe(episodeBackup);
}
});
});
// Immediate retry is cooldown-gated…
const gated = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(gated).toMatchObject({ status: 'skipped', reason: 'recently-failed' });
test('seam reports honest restored from WalRepairError (reset fails after backup)', async () => {
const seg = xlogFileName(1, 3n, SEG_SIZE);
const dir = makeLayout({ segments: [seg] });
poisonControlVersion(dir); // backup succeeds, resetWal throws → WalRepairError
const attempt = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(attempt.status).toBe('failed');
if (attempt.status === 'failed') {
// `restored` is threaded from WalRepairError.restore — not hardcoded.
expect(attempt.restored).toBe(true);
expect(attempt.receipt).not.toBeNull();
expect(attempt.repairError).toContain('pg_control version');
}
// Restore actually happened: original segment is back.
expect(existsSync(join(dir, 'pg_wal', seg))).toBe(true);
});
// …and with the cooldown off, the retry REUSES the episode backup: no new dir.
test('poisoned sidecar episodeBackupPath outside the backup prefix is IGNORED — fresh backup taken', async () => {
const dir = makeLayout();
// An existing dir that fails the `${dataDir}.wal-repair-backup-` prefix
// check: the user-writable sidecar must not be able to point repair's
// renames at an arbitrary target.
const evil = mkdtempSync(join(tmpdir(), 'pgrepair-evil-'));
writeFileSync(`${dir}.wal-repair-attempt.json`, JSON.stringify({
episodeStartedAt: Date.now(),
episodeBackupPath: evil,
attempts: [],
}));
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
const second = await attemptWalRepairAndRetry(dir, async () => 'db');
expect(second.status).toBe('repaired');
if (second.status === 'repaired') {
expect(second.receipt.reusedEpisodeBackup).toBe(true);
expect(second.receipt.backupPath).toBe(backupsAfterFirst[0]!);
const attempt = await attemptWalRepairAndRetry(dir, async () => 'db');
expect(attempt.status).toBe('repaired');
if (attempt.status === 'repaired') {
expect(attempt.receipt.reusedEpisodeBackup).toBe(false);
expect(attempt.receipt.backupPath.startsWith(`${dir}.wal-repair-backup-`)).toBe(true);
}
});
expect(listRepairBackups(dir).length).toBe(1);
expect(readRepairSidecar(dir).episodeStartedAt).toBeNull(); // episode closed
// The poisoned target was never renamed into or written through.
expect(existsSync(evil)).toBe(true);
expect(readdirSync(evil)).toEqual([]);
});
test('reap quarantine gates the seam; a marker older than the window does not', async () => {
const dir = makeLayout();
const marker = `${dir}.lock-reap.json`;
writeFileSync(marker, JSON.stringify({ ts: Date.now(), by: 1 }));
const gated = await attemptWalRepairAndRetry(dir, async () => 'x');
expect(gated).toMatchObject({ status: 'skipped', reason: 'possibly-live-writer' });
if (gated.status === 'skipped') expect(gated.detail).toContain('reaped');
// Gated BEFORE any surgery: no backup dir was created.
expect(listRepairBackups(dir)).toEqual([]);
// Marker older than the 10-minute quarantine → the seam proceeds.
writeFileSync(marker, JSON.stringify({ ts: Date.now() - 11 * 60 * 1000, by: 1 }));
await withEnv({ GBRAIN_PGLITE_WAL_REPAIR_COOLDOWN_SECONDS: '0' }, async () => {
const attempt = await attemptWalRepairAndRetry(dir, async () => 'db');
expect(attempt.status).toBe('repaired');
});
});
});
test/pglite-resetwal.test.ts
+18 -6
View File
@@ -41,6 +41,7 @@ function makeControl(opts?: { version?: number; segSize?: number; blcksz?: numbe
control.writeUInt32LE(opts?.segSize ?? SEG_SIZE, OFF.xlogSegSize);
const redoSegNo = opts?.redoSegNo ?? 3n;
control.writeBigUInt64LE(redoSegNo * BigInt(opts?.segSize ?? SEG_SIZE) + 40n, OFF.checkPointCopyRedo);
control.writeUInt32LE(crc32c([control.subarray(0, OFF.crc)]), OFF.crc); // valid CRC
return control;
}
@@ -60,33 +61,44 @@ function makeLayout(opts?: Parameters<typeof makeControl>[0] & { pgVersion?: str
describe('resetWal — validation refusals (fail-closed)', () => {
test('refuses PG_VERSION 16', async () => {
const dir = makeLayout({ pgVersion: '16' });
expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError);
await expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError);
});
test('refuses missing PG_VERSION', async () => {
const dir = mkdtempSync(join(tmpdir(), 'resetwal-'));
expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError);
await expect(resetWal(dir)).rejects.toThrow(WalResetUnsupportedError);
});
test('refuses wrong pg_control size', async () => {
const dir = makeLayout();
writeFileSync(join(dir, 'global', 'pg_control'), Buffer.alloc(100));
expect(resetWal(dir)).rejects.toThrow(/pg_control size/);
await expect(resetWal(dir)).rejects.toThrow(/pg_control size/);
});
test('refuses wrong pg_control version', async () => {
const dir = makeLayout({ version: 1600 });
expect(resetWal(dir)).rejects.toThrow(/pg_control version/);
await expect(resetWal(dir)).rejects.toThrow(/pg_control version/);
});
test('refuses non-power-of-two WAL segment size', async () => {
const dir = makeLayout({ segSize: 3 * 1024 * 1024 });
expect(resetWal(dir)).rejects.toThrow(/segment size/);
await expect(resetWal(dir)).rejects.toThrow(/segment size/);
});
test('refuses unsupported WAL block size', async () => {
const dir = makeLayout({ blcksz: 4096 });
expect(resetWal(dir)).rejects.toThrow(/block size/);
await expect(resetWal(dir)).rejects.toThrow(/block size/);
});
test('refuses a pg_control whose stored CRC does not verify (F6: no laundering corrupt counters)', async () => {
const dir = makeLayout();
// A structurally-valid control (right size/version/seg/block) but with a
// damaged checkpoint copy and a STALE crc — real pg_resetwal refuses this.
const control = readFileSync(join(dir, 'global', 'pg_control'));
control.writeBigUInt64LE(0xdeadbeefn, 56); // trash a checkpointCopy field
// leave the old CRC in place → mismatch
writeFileSync(join(dir, 'global', 'pg_control'), control);
await expect(resetWal(dir)).rejects.toThrow(/CRC mismatch/);
});
});