Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 a42352081c test(extract): pin the version-arm stamp floor with an explicit regression test
The floor in extractStaleFromDB was only covered incidentally — the #1768
test's updated_at (2026-06-02) fell below the bumped VERSION_TS, but its
comment still says the date was chosen to sit ABOVE the watermark. A future
date 'repair' there would silently drop floor coverage. This test pins it
directly: pre-watermark page clears after --stale and stays cleared.
Verified fail-without-fix against master's extract.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:59:09 -07:00
Garry TanandClaude Fable 5 070f4c5678 fix(extract): floor --stale stamp at LINK_EXTRACTOR_VERSION_TS so version bumps don't create permanently-stale pages
The version-ts bump in this PR exposed a latent conflict: extractStaleFromDB
stamps links_extracted_at = the page's read updated_at (#1768 µs fix), but a
page last edited BEFORE the new LINK_EXTRACTOR_VERSION_TS then lands below the
version watermark and the 'links_extracted_at < versionTs' arm re-flags it
stale on every run — an infinite re-extraction loop. Stamp
max(updated_at, versionTs); versionTs is always a past release date, so the
D4 concurrent-edit race guard still holds.

Fixes the test/extract-stale.test.ts CI failure on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 10:56:38 -07:00
Garry TanandClaude Fable 5 4b2df89935 fix(extract): resolve path-style wikilinks outside DIR_PATTERN + add ops/ to whitelist (#2576 bug 2)
- Add ops to DIR_PATTERN so [[ops/services/...]] wikilinks and bare
  ops/... slug refs are recognized as qualified entity references.
- queryBasenameIndex (the shared basename matcher behind
  resolveBasenameMatches, the FS resolver, and the doctor check) now
  falls back to the path tail when a path-style ref misses — before,
  normalizeBasename stripped slashes into a garbage key that could
  never hit the tail-keyed index.
- Bump LINK_EXTRACTOR_VERSION_TS so extract --stale re-sweeps
  previously-stamped pages with the new extraction logic.

Bugs 1+3 of #2576 are covered by PR #2717 (--stale nullResolver).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:33:45 -07:00
9 changed files with 70 additions and 138 deletions
+5 -8
View File
@@ -21,17 +21,14 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
### The Primitives
+5 -8
View File
@@ -2720,17 +2720,14 @@ GBrain is tuned for the Supabase **Transaction pooler** (port 6543): it
auto-disables prepared statements there and routes `engine.transaction()`
(migrations, DDL, sync imports) to a derived **direct** connection
(`db.<ref>.supabase.co:5432`). That direct host is IPv6-only, so on an
IPv4-only host it is unreachable. When that happens gbrain now falls back to
the pooler automatically (one stderr warning, then single-pool mode for the
rest of the process) — but the pooler's ~2-min statement timeout can truncate
very long migrations or bulk imports.
IPv4-only host, reads work but sync **silently skips most pages**. This is the
number one cause of "sync ran but nothing happened."
Fix: make the direct connection reachable over IPv4. Either set
`GBRAIN_DIRECT_DATABASE_URL` to the **Session pooler** string (port 5432 on the
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on.
`GBRAIN_DISABLE_DIRECT_POOL=1` skips the direct pool (and the fallback warning)
entirely. Verify by running `gbrain sync` and checking that the page count in
`gbrain stats` matches the syncable file count in the repo.
`pooler.supabase.com` host, IPv4), or enable Supabase's IPv4 add-on. Verify by
running `gbrain sync` and checking that the page count in `gbrain stats` matches
the syncable file count in the repo.
### The Primitives
+9 -1
View File
@@ -1743,7 +1743,15 @@ async function extractStaleFromDB(
// `page.updated_at.toISOString()` — the JS Date is ms-truncated, so the
// µs-precision DB updated_at stayed strictly greater and the page never
// cleared on Postgres. Stamping the exact value makes them equal.
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at_iso });
//
// Version-arm floor: a page last edited BEFORE LINK_EXTRACTOR_VERSION_TS
// would otherwise be stamped below the version watermark and stay
// permanently stale (`links_extracted_at < versionTs` re-fires every run).
// Stamp max(updated_at, versionTs) — versionTs is always a past release
// date, so a concurrent edit's now() still exceeds the stamp and D4 holds.
// Tie at ms precision picks updated_at_iso (its µs ≥ versionTs's .000000).
const stampTs = new Date(page.updated_at_iso) >= new Date(versionTs) ? page.updated_at_iso : versionTs;
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: stampTs });
}
// Flush NON-swallowing (CDX-4): a throw here propagates out of the sweep so
-6
View File
@@ -1078,9 +1078,6 @@ async function initPostgres(opts: {
console.warn(' Direct connections are IPv6 only and fail in many environments.');
console.warn(' Use the Transaction pooler connection string instead (port 6543):');
console.warn(' Supabase Dashboard > Connect (top bar) > Connection String > Transaction pooler');
console.warn(' (With a pooler URL, gbrain derives a direct connection for DDL and falls back');
console.warn(' to the pooler automatically if that host is unreachable. Power users:');
console.warn(' GBRAIN_DIRECT_DATABASE_URL overrides the derived URL; GBRAIN_DISABLE_DIRECT_POOL=1 disables it.)');
console.warn('');
}
@@ -1094,9 +1091,6 @@ async function initPostgres(opts: {
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Transaction pooler connection string instead (port 6543).');
console.error('(gbrain derives its own direct connection from pooler URLs for DDL; if that host is');
console.error('unreachable it falls back to the pooler. GBRAIN_DIRECT_DATABASE_URL overrides the');
console.error('derived URL; GBRAIN_DISABLE_DIRECT_POOL=1 disables the direct pool entirely.)');
}
throw e;
}
+2 -48
View File
@@ -167,25 +167,6 @@ export function deriveDirectUrl(url: string): string | null {
}
}
/**
* Error codes that mean "the direct host is unreachable from this network"
* (#1641). The auto-derived db.<ref>.supabase.co host is IPv6-only without
* the paid IPv4 add-on, so ENOTFOUND/ECONNREFUSED here is expected on
* IPv4-only networks — we fall back to the pooler instead of failing init.
*/
const NETWORK_UNREACHABLE_CODES = [
'ENOTFOUND', 'ECONNREFUSED', 'ENETUNREACH', 'EHOSTUNREACH',
'ETIMEDOUT', 'CONNECT_TIMEOUT',
];
/** True when err looks like a network-unreachable failure (not auth/SQL). */
export function isNetworkUnreachableError(err: unknown): boolean {
const code = (err as { code?: unknown } | null)?.code;
if (typeof code === 'string' && NETWORK_UNREACHABLE_CODES.includes(code)) return true;
const msg = err instanceof Error ? err.message : String(err);
return NETWORK_UNREACHABLE_CODES.some(c => msg.includes(c));
}
/**
* Read kill-switch state from env. Subordinate to parent manager's state
* when present (A2 inheritance).
@@ -338,30 +319,7 @@ export class ConnectionManager {
throw err;
});
}
let pool: Sql | null;
try {
pool = await this._directInit;
} catch (err) {
// #1641: the derived direct host (db.<ref>.supabase.co) is IPv6-only
// without Supabase's IPv4 add-on. On IPv4-only networks the direct
// pool can never connect — permanently fall back to the read pool
// (self-activating kill-switch) instead of failing init/migrations.
// Non-network errors (auth, SQL) still throw: they mean misconfig,
// not unreachability.
if (isNetworkUnreachableError(err)) {
const alreadyWarned = this._killSwitch;
this._killSwitch = true;
const msg = err instanceof Error ? err.message : String(err);
if (!alreadyWarned) console.error(
`gbrain: direct connection to ${this._directUrl ? this.hostOnly(this._directUrl) : 'unknown host'} unreachable (${msg}); ` +
'falling back to the pooler for DDL/bulk (long migrations may hit the pooler statement timeout). ' +
'Set GBRAIN_DIRECT_DATABASE_URL to a reachable direct URL (e.g. the Session pooler, port 5432) or enable the Supabase IPv4 add-on; ' +
'GBRAIN_DISABLE_DIRECT_POOL=1 silences this.',
);
return this.getReadPool();
}
throw err;
}
const pool = await this._directInit;
if (!pool) {
// Defensive — initDirectPool should have thrown.
throw new Error('connection-manager: direct pool init returned null');
@@ -392,9 +350,8 @@ export class ConnectionManager {
},
};
const t0 = Date.now();
let pool: Sql | null = null;
try {
pool = postgres(this._directUrl, opts);
const pool = postgres(this._directUrl, opts);
// Probe to validate connectivity early.
await pool`SELECT 1`;
logConnectionEvent({
@@ -405,9 +362,6 @@ export class ConnectionManager {
});
return pool;
} catch (err) {
// Don't leak the failed pool's sockets/timers (#1641 fallback keeps
// the process running afterward).
if (pool) await endPoolBounded(pool);
logConnectionEvent({
pool: 'ddl',
op: 'error',
+13 -4
View File
@@ -28,7 +28,7 @@ import { ensureWellFormed } from './text-safe.ts';
* OR updated_at > links_extracted_at`. It is an ISO-8601 string (NOT a number) —
* the column is TIMESTAMPTZ and the predicate binds it as `::timestamptz`.
*/
export const LINK_EXTRACTOR_VERSION_TS = '2026-05-31T00:00:00Z';
export const LINK_EXTRACTOR_VERSION_TS = '2026-07-21T00:00:00Z';
// ─── Entity references ──────────────────────────────────────────
@@ -80,10 +80,10 @@ export type LinkResolutionType = 'qualified' | 'unqualified';
* Directory prefix whitelist. These are the top-level slug dirs the extractor
* recognizes as entity references. Upstream canonical + our extensions:
* - Gbrain canonical: people, companies, meetings, concepts, deal, civic, project, source, media, yc, projects
* - Our domain extensions: tech, finance, personal, openclaw (domain-organized wikis)
* - Our domain extensions: tech, finance, personal, openclaw, ops (domain-organized wikis)
* - Our entity prefix: entities (we kept some legacy entities/projects/ pages)
*/
const DIR_PATTERN = '(?:people|companies|meetings|concepts|deal|civic|project|projects|source|media|yc|tech|finance|personal|openclaw|entities)';
const DIR_PATTERN = '(?:people|companies|meetings|concepts|deal|civic|project|projects|source|media|yc|tech|finance|personal|openclaw|entities|ops)';
/**
* Match `[Name](path)` markdown links pointing to entity directories.
@@ -865,7 +865,16 @@ export function queryBasenameIndex(idx: Map<string, string[]>, name: string): st
if (!name || typeof name !== 'string') return [];
const trimmed = name.trim();
if (!trimmed) return [];
const hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
let hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
// Issue #2576 bug 2: path-style refs (`runbooks/2026-05-01-x`) from dirs
// outside DIR_PATTERN reach here, but normalizeBasename strips slashes
// into a garbage key (`runbooks2026-05-01-x`) that can never hit the
// tail-keyed index. Fall back to the path tail so qualified refs resolve
// by basename like everything else.
if (!hit && trimmed.includes('/')) {
const tail = trimmed.slice(trimmed.lastIndexOf('/') + 1).trim();
if (tail) hit = idx.get(tail) ?? idx.get(tail.toLowerCase()) ?? idx.get(normalizeBasename(tail));
}
return hit ? [...hit].sort(basenameSort) : [];
}
-63
View File
@@ -3,7 +3,6 @@ import {
isSupabasePoolerUrl,
deriveDirectUrl,
readKillSwitchEnv,
isNetworkUnreachableError,
resolveDirectPoolSize,
ConnectionManager,
DEFAULT_DIRECT_POOL_SIZE,
@@ -239,65 +238,3 @@ describe('ConnectionManager — parent inheritance (A2)', () => {
}
});
});
describe('isNetworkUnreachableError (#1641)', () => {
test('classifies network codes as unreachable', () => {
for (const code of ['ENOTFOUND', 'ECONNREFUSED', 'ENETUNREACH', 'EHOSTUNREACH', 'ETIMEDOUT', 'CONNECT_TIMEOUT']) {
const err = Object.assign(new Error('connect failed'), { code });
expect(isNetworkUnreachableError(err)).toBe(true);
}
});
test('classifies by message when code absent', () => {
expect(isNetworkUnreachableError(new Error('getaddrinfo ENOTFOUND db.abc.supabase.co'))).toBe(true);
});
test('auth/SQL errors are NOT unreachable', () => {
expect(isNetworkUnreachableError(new Error('password authentication failed for user "postgres"'))).toBe(false);
expect(isNetworkUnreachableError(new Error('syntax error at or near "SELEC"'))).toBe(false);
expect(isNetworkUnreachableError(null)).toBe(false);
});
});
describe('ConnectionManager — direct-pool fallback on unreachable host (#1641)', () => {
let originalKillSwitch: string | undefined;
let originalError: typeof console.error;
let errLines: string[];
beforeEach(() => {
originalKillSwitch = process.env.GBRAIN_DISABLE_DIRECT_POOL;
delete process.env.GBRAIN_DISABLE_DIRECT_POOL;
originalError = console.error;
errLines = [];
console.error = (...args: unknown[]) => { errLines.push(args.join(' ')); };
});
afterEach(() => {
console.error = originalError;
if (originalKillSwitch === undefined) delete process.env.GBRAIN_DISABLE_DIRECT_POOL;
else process.env.GBRAIN_DISABLE_DIRECT_POOL = originalKillSwitch;
});
test('ddl() falls back to the read pool when the direct host is unreachable', async () => {
const cm = new ConnectionManager({
url: 'postgresql://postgres.abc:p@aws.pooler.supabase.com:6543/db',
// 127.0.0.1:9 (discard) → instant ECONNREFUSED, the IPv4-only-network shape.
directUrl: 'postgresql://postgres:p@127.0.0.1:9/db',
});
const fakeReadPool = {} as ReturnType<typeof ConnectionManager.prototype.read>;
cm.setReadPool(fakeReadPool);
expect(cm.isDualPoolActive()).toBe(true);
const pool = await cm.ddl(); // without the fix this throws ECONNREFUSED
expect(pool).toBe(fakeReadPool);
// Self-activating kill-switch: subsequent calls skip the direct pool.
expect(cm.isKillSwitchActive()).toBe(true);
expect(cm.isDualPoolActive()).toBe(false);
expect(cm.describeMode().mode).toBe('single (kill-switch)');
// One stderr line mentioning the power-user override.
const warning = errLines.filter(l => l.includes('GBRAIN_DIRECT_DATABASE_URL'));
expect(warning.length).toBe(1);
const again = await cm.ddl();
expect(again).toBe(fakeReadPool);
expect(errLines.filter(l => l.includes('GBRAIN_DIRECT_DATABASE_URL')).length).toBe(1);
}, 20000);
});
+18
View File
@@ -209,6 +209,24 @@ describe('gbrain extract --stale', () => {
expect(usRows[0]?.eq).toBe(true);
});
test('version-arm floor: page edited BEFORE LINK_EXTRACTOR_VERSION_TS clears after --stale (issue #2576 bug 3)', async () => {
// A page whose updated_at predates the version watermark used to be
// stamped at its updated_at (< versionTs), so the version arm re-fired
// every run — permanently stale. The sweep now floors the stamp at
// versionTs. (The #1768 test above also covers this since the v0.42.x
// VERSION_TS bump moved its date below the watermark, but this pins the
// behavior explicitly so a date "repair" there can't drop coverage.)
await engine.putPage('people/alice', personPage('Alice'));
await engine.executeRaw(`UPDATE pages SET updated_at = '2000-01-01T00:00:00Z' WHERE slug = 'people/alice'`);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(1);
await runExtract(engine, ['--stale']);
// Pre-floor this stayed 1 forever (stamp < versionTs → version arm re-fires).
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
await runExtract(engine, ['--stale']);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
});
test('CDX-4 (D2): a link-flush throw aborts the sweep and leaves pages UNSTAMPED', async () => {
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) founded [Acme](companies/acme).'));
+18
View File
@@ -140,6 +140,15 @@ describe('extractEntityRefs', () => {
expect(wikiRefs[0].needsResolution).toBe(true);
});
test('recognizes ops/ qualified wikilinks (issue #2576 bug 2)', () => {
// `ops` was missing from DIR_PATTERN, so [[ops/...]] fell through to
// the generic 2c pass (needsResolution) instead of being a real ref.
const refs = extractEntityRefs('Deployed via [[ops/services/pointer-agent]].');
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('ops/services/pointer-agent');
expect(refs[0].needsResolution).toBeUndefined();
});
test('skips qualified-syntax tokens (those belong to 2a)', () => {
// [[wiki:topics/ai]] looks like 2a's qualified shape — even though
// it wouldn't satisfy DIR_PATTERN, 2c must not claim it either
@@ -1069,6 +1078,15 @@ describe('makeResolver — fallback chain', () => {
]);
});
test('resolveBasenameMatches: path-style ref falls back to the tail (issue #2576 bug 2)', async () => {
// normalizeBasename strips slashes, so `runbooks/2026-05-01-pointer-agent`
// used to normalize to a garbage key that never hit the tail-keyed index.
const engine = makeFakeEngineWithSlugs(['ops/changes/2026-05-01-pointer-agent']);
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('runbooks/2026-05-01-pointer-agent'))
.toEqual(['ops/changes/2026-05-01-pointer-agent']);
});
test('resolveBasenameMatches: case-insensitive fallback', async () => {
const engine = makeFakeEngineWithSlugs(['companies/fast-weigh']);
const r = makeResolver(engine);