Compare commits

...
Author SHA1 Message Date
Garry Tan 8ab8178a17 fix(pglite): stop blaming macOS for a bare Aborted(); name WAL damage (#2674)
Diagnosed against a real broken brain, not from the issue text.

The reported loop: a damaged store aborts init, gbrain prints "Most common
cause: the macOS 26.3 WASM bug (#223)", and the user chases a phantom OS bug.
#2891 platform-gated that banner for non-darwin but left it in place on macOS,
which is where every reporter is.

What I actually measured. On macOS 26.4 / Bun 1.3.14 / PGLite 0.4.3:

  - A FRESH scratch store creates, writes, closes and reopens fine. So the
    runtime is not broken on this OS.
  - A COPY of the damaged store fails identically, ruling out lock contention
    and concurrent access.
  - With PGLite debug on, the real cause appears:
        LOG:   database system was interrupted; last known up at <ts>
        LOG:   invalid resource manager ID in checkpoint record
        PANIC: could not locate a valid checkpoint record at 0/37ADDF8
        Aborted()

So it is WAL/checkpoint damage: Postgres started, then PANICked during
recovery. Nothing to do with the OS, and `postmaster.pid` removal and
re-running migrations both no-op (verified).

The load-bearing detail for the fix: PGLite reports only `Aborted()` to JS and
prints the PANIC to its OWN stderr. The classifier never sees the PANIC, so no
regex on checkpoint text can help on the default path — and a damaged store is
genuinely indistinguishable from a broken runtime from that string alone.
Guessing "macOS" there is what caused the misdiagnosis.

Three changes:

1. `abort-ambiguous` for a bare `Aborted()`. Instead of guessing, the hint
   teaches the 5-second discriminator the issue asked for:
   `GBRAIN_HOME=$(mktemp -d) gbrain init --pglite --no-embedding` — succeeds
   means your store is damaged, fails too means the runtime. Identical text on
   darwin and elsewhere, because the string carries no platform signal.

2. `wal-corrupt` for the recovery-PANIC signatures, so any caller that DOES
   surface stderr gets a precise verdict. Its hint rules out the OS bug and
   locks, says plainly that markdown is unaffected because the DB holds derived
   data, points at `reinit-pglite`, and names the two things that cannot work.

3. The header is verdict-dependent. "failed to initialize its WASM runtime" was
   asserted for every verdict including a damaged store, where the runtime
   started fine — the header was itself part of the misdiagnosis. Store-fault
   verdicts now say "PGLite could not open your brain."

Tests discriminate: 50 pass here, 6 fail on master. The pre-existing
all-verdicts-share-a-header test is updated rather than deleted, since its
intent (every verdict has a header and carries the original error) is preserved
as two assertions.

Does not close #223 — that umbrella also covers a genuinely distinct
compiled-binary `$bunfs` mode. This fixes the diagnosis, not a crash.
2026-07-28 12:44:36 -07:00
2 changed files with 168 additions and 7 deletions
+75 -3
View File
@@ -164,7 +164,17 @@ export function computeSnapshotSchemaHash(
* errors). Match the literal `$$bunfs` marker OR ENOENT+pglite.data
* co-occurrence.
*/
export type PgliteInitFailure = 'bunfs' | 'macos-26-3' | 'corrupt' | 'unknown';
export type PgliteInitFailure =
| 'bunfs'
| 'macos-26-3'
| 'corrupt'
// #2674: Postgres PANICked during WAL recovery — a damaged store, not a
// broken runtime. Recoverable diagnosis; distinct from `corrupt` (catalog).
| 'wal-corrupt'
// #2674: a bare `Aborted()` with no further signal. Deliberately NOT guessed
// at — the hint teaches the 5-second isolation test instead.
| 'abort-ambiguous'
| 'unknown';
// #2674: non-Error rejections (Emscripten aborts can throw plain objects)
// used to stringify as "[object Object]" — prefer .message when present.
@@ -182,9 +192,33 @@ export function classifyPgliteInitError(message: string): PgliteInitFailure {
if (/58P01|internal_load_library|type "?vector"? does not exist|relation "?content_chunks"? does not exist/i.test(message)) {
return 'corrupt';
}
if (/abort.*runtime|macos.*26\.3|wasm.*runtime/i.test(message)) {
// #2674: WAL / checkpoint damage is a DIFFERENT corruption shape from #2348's
// catalog damage, and it is what the reporters in #223 actually hit. Postgres
// PANICs during recovery rather than failing a catalog lookup:
//
// LOG: database system was interrupted; last known up at <ts>
// LOG: invalid resource manager ID in checkpoint record
// PANIC: could not locate a valid checkpoint record at 0/37ADDF8
//
// Matched here so that any caller which DOES surface PGLite's stderr gets a
// precise verdict. Note the default path does not: PGLite reports only
// `Aborted()` to JS and prints the PANIC to its own stderr, which is exactly
// why `abort-ambiguous` below exists.
if (
/could not locate a valid checkpoint record|invalid resource manager ID|database system was interrupted|could not access status of transaction|dead heap-only tuple|invalid checkpoint record/i.test(
message,
)
) {
return 'wal-corrupt';
}
if (/macos.*26\.3|wasm.*runtime/i.test(message)) {
return 'macos-26-3';
}
// A bare Emscripten abort with nothing else to go on. Do NOT guess a cause:
// measured on macOS 26.4 / Bun 1.3.14 / PGLite 0.4.3, a store with WAL damage
// and a healthy runtime produce the SAME string here. Blaming the OS sent
// #223, #1954, #1955 and #2674 chasing a phantom platform bug for weeks.
if (/^(RuntimeError:\s*)?Aborted\(\)/i.test(message.trim())) return 'abort-ambiguous';
return 'unknown';
}
@@ -195,7 +229,14 @@ export function buildPgliteInitErrorMessage(
// monkey-patching process.platform.
platform: NodeJS.Platform = process.platform,
): string {
const header = 'PGLite failed to initialize its WASM runtime.';
// #2674: only assert "WASM runtime" when we actually believe the runtime is
// at fault. For a damaged store the runtime started fine and then PANICked
// during recovery, so the old unconditional header was itself part of the
// misdiagnosis.
const header =
verdict === 'wal-corrupt' || verdict === 'corrupt' || verdict === 'abort-ambiguous'
? 'PGLite could not open your brain.'
: 'PGLite failed to initialize its WASM runtime.';
let hint: string;
switch (verdict) {
case 'bunfs':
@@ -223,6 +264,37 @@ export function buildPgliteInitErrorMessage(
' (wipes + re-inits + re-syncs; DB-only state is re-derived).\n' +
' Deleting .gbrain-lock/ or postmaster.pid does NOT fix this.';
break;
case 'wal-corrupt':
hint =
' Postgres started but PANICked replaying the write-ahead log, so the\n' +
' store is damaged — this is NOT the macOS WASM bug and NOT a lock.\n' +
' The "last known up at" timestamp above is when the brain was last\n' +
' healthy; anything written after it is not recoverable in place.\n' +
' Your markdown is unaffected — the DB holds derived data (chunks,\n' +
' embeddings, links, facts), all of which a re-sync rebuilds. Recover:\n' +
' 1. Restore a backup of the brain.pglite directory if you have one, OR\n' +
' 2. gbrain reinit-pglite --embedding-model <id> --embedding-dimensions <N>\n' +
' (wipes + re-inits + re-syncs from your sources).\n' +
' Deleting postmaster.pid does NOT fix this; neither does re-running\n' +
' migrations. pg_resetwal would, but PGLite does not ship it.';
break;
case 'abort-ambiguous':
hint =
' PGLite aborted without saying why. Two causes produce this exact\n' +
' message, so run the 5-second test that tells them apart:\n' +
'\n' +
' GBRAIN_HOME=$(mktemp -d) gbrain init --pglite --no-embedding\n' +
'\n' +
' - That SUCCEEDS -> the runtime is fine and YOUR STORE is damaged.\n' +
' Recover with `gbrain reinit-pglite` (your markdown is untouched;\n' +
' the DB holds derived data a re-sync rebuilds), or restore a backup.\n' +
' - That FAILS TOO -> the runtime itself cannot start here. Report it\n' +
' with your OS + Bun version; see #223 for prior reports.\n' +
'\n' +
' `gbrain doctor` runs the same check. Note a damaged store and a\n' +
' broken runtime are indistinguishable from this string alone, so\n' +
' do not assume the macOS 26.3 bug without running the test above.';
break;
case 'unknown':
default:
// #2674: only blame the macOS 26.3 WASM bug on macOS. On other
+93 -4
View File
@@ -119,11 +119,100 @@ describe('buildPgliteInitErrorMessage — hint routing', () => {
expect(msg).not.toContain('issues/223');
});
test('all verdicts produce the canonical header line', () => {
for (const v of ['bunfs', 'macos-26-3', 'corrupt', 'unknown'] as const) {
const msg = buildPgliteInitErrorMessage(v, original);
expect(msg.startsWith('PGLite failed to initialize its WASM runtime.')).toBe(true);
test('runtime-fault verdicts claim the WASM runtime; store-fault verdicts do not', () => {
// #2674: the header used to assert "failed to initialize its WASM runtime"
// for EVERY verdict — including a damaged store, where the runtime started
// fine and then PANICked during recovery. That header was itself part of
// the misdiagnosis, so it is now verdict-dependent.
for (const v of ['bunfs', 'macos-26-3', 'unknown'] as const) {
expect(buildPgliteInitErrorMessage(v, original).startsWith(
'PGLite failed to initialize its WASM runtime.',
)).toBe(true);
}
for (const v of ['corrupt', 'wal-corrupt', 'abort-ambiguous'] as const) {
const msg = buildPgliteInitErrorMessage(v, original);
expect(msg.startsWith('PGLite could not open your brain.')).toBe(true);
expect(msg).not.toContain('failed to initialize its WASM runtime');
}
});
test('every verdict still carries a hint and the original error', () => {
for (const v of [
'bunfs', 'macos-26-3', 'corrupt', 'wal-corrupt', 'abort-ambiguous', 'unknown',
] as const) {
const msg = buildPgliteInitErrorMessage(v, original);
expect(msg).toContain(`Original error: ${original}`);
expect(msg.split('\n').length).toBeGreaterThan(2);
}
});
});
describe('#2674: a bare Aborted() must not be blamed on macOS', () => {
// Measured on macOS 26.4 / Bun 1.3.14 / PGLite 0.4.3: a store with WAL
// damage and a healthy runtime yield this identical string, so the OS
// cannot be inferred from it. #223/#1954/#1955/#2674 all lost time to that.
const BARE = 'Aborted(). Build with -sASSERTIONS for more info.';
test('classifies as abort-ambiguous, not macos-26-3 and not unknown', () => {
expect(classifyPgliteInitError(BARE)).toBe('abort-ambiguous');
expect(classifyPgliteInitError('RuntimeError: Aborted()')).toBe('abort-ambiguous');
});
test('the hint leads with the isolation test and does not assert the OS bug', () => {
const msg = buildPgliteInitErrorMessage('abort-ambiguous', BARE, 'darwin');
expect(msg).toContain('GBRAIN_HOME=$(mktemp -d) gbrain init --pglite');
// It may *mention* not assuming the macOS bug, but must not present it as
// the cause the way the old darwin `unknown` branch did.
expect(msg).not.toContain('Possible cause: the macOS 26.3 WASM bug');
expect(msg).toContain('YOUR STORE is damaged');
});
test('the same message is given on darwin and non-darwin', () => {
// The old code branched on platform here. The string carries no platform
// signal, so branching on it was guessing.
expect(buildPgliteInitErrorMessage('abort-ambiguous', BARE, 'darwin')).toBe(
buildPgliteInitErrorMessage('abort-ambiguous', BARE, 'linux'),
);
});
test('an explicit macOS 26.3 mention still routes to macos-26-3', () => {
expect(classifyPgliteInitError('known macOS 26.3 issue')).toBe('macos-26-3');
expect(classifyPgliteInitError('wasm runtime could not start')).toBe('macos-26-3');
});
});
describe('#2674: WAL / checkpoint damage is its own verdict', () => {
// Reproduced end to end: garbling pg_wal on a healthy PGLite 0.4.3 store and
// reopening produces exactly these log lines before the abort. PGLite prints
// them to its own stderr, so the default path sees only `Aborted()` — these
// matches exist for any caller that does surface stderr.
const CASES = [
'PANIC: could not locate a valid checkpoint record at 0/37ADDF8',
'LOG: invalid resource manager ID in checkpoint record',
'LOG: database system was interrupted; last known up at 2026-07-11 00:12:20',
'could not access status of transaction 0',
'dead heap-only tuple (0, 136) is not linked to from any HOT',
];
test('each recovery-PANIC signature classifies as wal-corrupt', () => {
for (const c of CASES) {
expect(classifyPgliteInitError(c), c).toBe('wal-corrupt');
}
});
test('the hint rules out the OS bug and locks, and points at reinit', () => {
const msg = buildPgliteInitErrorMessage('wal-corrupt', CASES[0], 'darwin');
expect(msg).toContain('NOT the macOS WASM bug');
expect(msg).toContain('gbrain reinit-pglite');
// The single most useful fact for a panicking user.
expect(msg).toContain('Your markdown is unaffected');
// Steer away from the two things people try that cannot work.
expect(msg).toContain('postmaster.pid does NOT fix this');
});
test('catalog corruption (#2348) still routes to corrupt, not wal-corrupt', () => {
expect(classifyPgliteInitError('58P01 internal_load_library failed')).toBe('corrupt');
expect(classifyPgliteInitError('type "vector" does not exist')).toBe('corrupt');
});
});