Compare commits

..
Author SHA1 Message Date
Garry TanandClaude Fable 5 a42352081c test(extract): pin the version-arm stamp floor with an explicit regression test
The floor in extractStaleFromDB was only covered incidentally — the #1768
test's updated_at (2026-06-02) fell below the bumped VERSION_TS, but its
comment still says the date was chosen to sit ABOVE the watermark. A future
date 'repair' there would silently drop floor coverage. This test pins it
directly: pre-watermark page clears after --stale and stays cleared.
Verified fail-without-fix against master's extract.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:59:09 -07:00
Garry TanandClaude Fable 5 070f4c5678 fix(extract): floor --stale stamp at LINK_EXTRACTOR_VERSION_TS so version bumps don't create permanently-stale pages
The version-ts bump in this PR exposed a latent conflict: extractStaleFromDB
stamps links_extracted_at = the page's read updated_at (#1768 µs fix), but a
page last edited BEFORE the new LINK_EXTRACTOR_VERSION_TS then lands below the
version watermark and the 'links_extracted_at < versionTs' arm re-flags it
stale on every run — an infinite re-extraction loop. Stamp
max(updated_at, versionTs); versionTs is always a past release date, so the
D4 concurrent-edit race guard still holds.

Fixes the test/extract-stale.test.ts CI failure on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 10:56:38 -07:00
Garry TanandClaude Fable 5 4b2df89935 fix(extract): resolve path-style wikilinks outside DIR_PATTERN + add ops/ to whitelist (#2576 bug 2)
- Add ops to DIR_PATTERN so [[ops/services/...]] wikilinks and bare
  ops/... slug refs are recognized as qualified entity references.
- queryBasenameIndex (the shared basename matcher behind
  resolveBasenameMatches, the FS resolver, and the doctor check) now
  falls back to the path tail when a path-style ref misses — before,
  normalizeBasename stripped slashes into a garbage key that could
  never hit the tail-keyed index.
- Bump LINK_EXTRACTOR_VERSION_TS so extract --stale re-sweeps
  previously-stamped pages with the new extraction logic.

Bugs 1+3 of #2576 are covered by PR #2717 (--stale nullResolver).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 14:33:45 -07:00
6 changed files with 79 additions and 64 deletions
+9 -1
View File
@@ -1743,7 +1743,15 @@ async function extractStaleFromDB(
// `page.updated_at.toISOString()` — the JS Date is ms-truncated, so the
// µs-precision DB updated_at stayed strictly greater and the page never
// cleared on Postgres. Stamping the exact value makes them equal.
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at_iso });
//
// Version-arm floor: a page last edited BEFORE LINK_EXTRACTOR_VERSION_TS
// would otherwise be stamped below the version watermark and stay
// permanently stale (`links_extracted_at < versionTs` re-fires every run).
// Stamp max(updated_at, versionTs) — versionTs is always a past release
// date, so a concurrent edit's now() still exceeds the stamp and D4 holds.
// Tie at ms precision picks updated_at_iso (its µs ≥ versionTs's .000000).
const stampTs = new Date(page.updated_at_iso) >= new Date(versionTs) ? page.updated_at_iso : versionTs;
processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: stampTs });
}
// Flush NON-swallowing (CDX-4): a throw here propagates out of the sweep so
+3 -11
View File
@@ -35,11 +35,10 @@
*
* The doctor renders both side by side.
*
* Drift contract: every check name that ships through doctor MUST appear in
* Drift contract: every check name that ships in doctor.ts MUST appear in
* exactly one set below. The drift-guard test in
* `test/doctor-categories.test.ts` enforces this by reading doctor check
* emitter sources via a tagged-string scan and asserting set membership
* exactly.
* `test/doctor-categories.test.ts` enforces this by reading doctor.ts source
* via a tagged-string scan and asserting set membership exactly.
*
* If you add a new doctor check, you MUST add its name to the appropriate
* set here. The categorize step in `src/commands/doctor.ts` falls through
@@ -68,15 +67,12 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'conversation_parser_probe_health',
'cross_modal_modality_backfill',
'cycle_freshness',
'dangling_aliases',
'effective_date_health',
'embed_staleness',
'embedding_column_registry',
'embedding_env_override',
'embedding_provider',
'embedding_width_consistency',
'embeddings',
'entity_link_coverage',
'eval_drift',
'extract_atoms_backlog',
'extract_health',
@@ -106,9 +102,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'stub_guard_24h',
'sync_failures',
'sync_freshness',
'takes_count',
'takes_weight_grid',
'timeline_coverage',
'unified_multimodal_coverage',
'voice_gate_health',
]);
@@ -176,14 +170,12 @@ export const META_CHECK_NAMES: ReadonlySet<string> = new Set([
'eval_capture',
'minions_migration',
'multi_source_drift',
'pack_upgrade_available',
'schema_pack_active',
'schema_pack_consistency',
'schema_pack_source_drift',
'schema_version',
'slug_fallback_audit',
'timeline_dedup_index',
'type_proliferation',
'upgrade_errors',
]);
+13 -4
View File
@@ -28,7 +28,7 @@ import { ensureWellFormed } from './text-safe.ts';
* OR updated_at > links_extracted_at`. It is an ISO-8601 string (NOT a number) —
* the column is TIMESTAMPTZ and the predicate binds it as `::timestamptz`.
*/
export const LINK_EXTRACTOR_VERSION_TS = '2026-05-31T00:00:00Z';
export const LINK_EXTRACTOR_VERSION_TS = '2026-07-21T00:00:00Z';
// ─── Entity references ──────────────────────────────────────────
@@ -80,10 +80,10 @@ export type LinkResolutionType = 'qualified' | 'unqualified';
* Directory prefix whitelist. These are the top-level slug dirs the extractor
* recognizes as entity references. Upstream canonical + our extensions:
* - Gbrain canonical: people, companies, meetings, concepts, deal, civic, project, source, media, yc, projects
* - Our domain extensions: tech, finance, personal, openclaw (domain-organized wikis)
* - Our domain extensions: tech, finance, personal, openclaw, ops (domain-organized wikis)
* - Our entity prefix: entities (we kept some legacy entities/projects/ pages)
*/
const DIR_PATTERN = '(?:people|companies|meetings|concepts|deal|civic|project|projects|source|media|yc|tech|finance|personal|openclaw|entities)';
const DIR_PATTERN = '(?:people|companies|meetings|concepts|deal|civic|project|projects|source|media|yc|tech|finance|personal|openclaw|entities|ops)';
/**
* Match `[Name](path)` markdown links pointing to entity directories.
@@ -865,7 +865,16 @@ export function queryBasenameIndex(idx: Map<string, string[]>, name: string): st
if (!name || typeof name !== 'string') return [];
const trimmed = name.trim();
if (!trimmed) return [];
const hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
let hit = idx.get(trimmed) ?? idx.get(trimmed.toLowerCase()) ?? idx.get(normalizeBasename(trimmed));
// Issue #2576 bug 2: path-style refs (`runbooks/2026-05-01-x`) from dirs
// outside DIR_PATTERN reach here, but normalizeBasename strips slashes
// into a garbage key (`runbooks2026-05-01-x`) that can never hit the
// tail-keyed index. Fall back to the path tail so qualified refs resolve
// by basename like everything else.
if (!hit && trimmed.includes('/')) {
const tail = trimmed.slice(trimmed.lastIndexOf('/') + 1).trim();
if (tail) hit = idx.get(tail) ?? idx.get(tail.toLowerCase()) ?? idx.get(normalizeBasename(tail));
}
return hit ? [...hit].sort(basenameSort) : [];
}
+18 -48
View File
@@ -1,10 +1,10 @@
/**
* Drift guard for src/core/doctor-categories.ts.
*
* Reads doctor check emitter source via a literal-string scan, enumerates every
* `name: '<...>'` Check name, and asserts each appears in exactly ONE category
* set. The union of the four sets must equal the discovered names exactly —
* no orphans, no extras.
* Reads src/commands/doctor.ts source via a literal-string scan, enumerates
* every `name: '<...>'` Check name, and asserts each appears in exactly ONE
* category set. The union of the four sets must equal the discovered names
* exactly — no orphans, no extras.
*
* This is the structural failure the v0.41.19.0 plan-eng-review caught:
* doctor.ts grows new checks regularly; without this guard, the
@@ -25,30 +25,26 @@ import {
} from '../src/core/doctor-categories.ts';
const DOCTOR_TS_PATH = join(import.meta.dir, '..', 'src', 'commands', 'doctor.ts');
const ONBOARD_CHECKS_TS_PATH = join(import.meta.dir, '..', 'src', 'core', 'onboard', 'checks.ts');
const CHECK_SOURCE_PATHS = [DOCTOR_TS_PATH, ONBOARD_CHECKS_TS_PATH];
function enumerateCheckNames(): Set<string> {
const source = readFileSync(DOCTOR_TS_PATH, 'utf-8');
const names = new Set<string>();
for (const path of CHECK_SOURCE_PATHS) {
const source = readFileSync(path, 'utf-8');
// 1) Inline object-literal form: `{ name: 'foo', ... }`.
for (const m of source.matchAll(/name:\s*['"]([a-z][a-z0-9_]+)['"]/g)) {
names.add(m[1]);
}
// 2) Helper-function form: `const name = 'foo';` inside a check helper.
// Catches checks like `nightly_quality_probe_health` and
// `conversation_facts_backlog` that build the Check from a captured
// name constant.
for (const m of source.matchAll(/const\s+name\s*=\s*['"]([a-z][a-z0-9_]+)['"]/g)) {
names.add(m[1]);
}
// 1) Inline object-literal form: `{ name: 'foo', ... }`.
for (const m of source.matchAll(/name:\s*['"]([a-z][a-z0-9_]+)['"]/g)) {
names.add(m[1]);
}
// 2) Helper-function form: `const name = 'foo';` inside a check helper.
// Catches checks like `nightly_quality_probe_health` and
// `conversation_facts_backlog` that build the Check from a captured
// name constant.
for (const m of source.matchAll(/const\s+name\s*=\s*['"]([a-z][a-z0-9_]+)['"]/g)) {
names.add(m[1]);
}
return names;
}
describe('doctor-categories drift guard', () => {
test('every doctor-emitted check name belongs to exactly one category set', () => {
test('every check name in doctor.ts source belongs to exactly one category set', () => {
const discovered = enumerateCheckNames();
const allCategorized = new Set<string>([
...BRAIN_CHECK_NAMES,
@@ -63,7 +59,7 @@ describe('doctor-categories drift guard', () => {
}
if (missing.length > 0) {
throw new Error(
`These check names appear in doctor check emitters but are not categorized in ` +
`These check names appear in doctor.ts but are not categorized in ` +
`src/core/doctor-categories.ts: ${missing.sort().join(', ')}. ` +
`Add each to BRAIN/SKILL/OPS/META_CHECK_NAMES.`,
);
@@ -90,7 +86,7 @@ describe('doctor-categories drift guard', () => {
expect(dupes).toEqual([]);
});
test('every categorized name is currently used in doctor check emitters (no stale entries)', () => {
test('every categorized name is currently used in doctor.ts source (no stale entries)', () => {
const discovered = enumerateCheckNames();
const allCategorized = new Set<string>([
...BRAIN_CHECK_NAMES,
@@ -128,14 +124,6 @@ describe('categorizeCheck', () => {
expect(categorizeCheck('sync_freshness')).toBe('brain');
});
test('returns the right category for onboard data-quality check names', () => {
expect(categorizeCheck('embed_staleness')).toBe('brain');
expect(categorizeCheck('entity_link_coverage')).toBe('brain');
expect(categorizeCheck('timeline_coverage')).toBe('brain');
expect(categorizeCheck('takes_count')).toBe('brain');
expect(categorizeCheck('dangling_aliases')).toBe('brain');
});
test('returns the right category for a known skill name', () => {
expect(categorizeCheck('resolver_health')).toBe('skill');
expect(categorizeCheck('skill_conformance')).toBe('skill');
@@ -152,24 +140,6 @@ describe('categorizeCheck', () => {
expect(categorizeCheck('upgrade_errors')).toBe('meta');
});
test('returns the right category for onboard schema-pack check names without warning', () => {
const originalWrite = process.stderr.write.bind(process.stderr);
const captured: string[] = [];
(process.stderr as { write: typeof process.stderr.write }).write = ((
chunk: string | Uint8Array,
) => {
captured.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString());
return true;
}) as typeof process.stderr.write;
try {
expect(categorizeCheck('pack_upgrade_available')).toBe('meta');
expect(categorizeCheck('type_proliferation')).toBe('meta');
expect(captured.filter((c) => c.includes('[doctor-categories]'))).toEqual([]);
} finally {
(process.stderr as { write: typeof process.stderr.write }).write = originalWrite;
}
});
test('unknown check name falls through to meta with a stderr warn (once per process)', () => {
const originalWrite = process.stderr.write.bind(process.stderr);
const captured: string[] = [];
+18
View File
@@ -209,6 +209,24 @@ describe('gbrain extract --stale', () => {
expect(usRows[0]?.eq).toBe(true);
});
test('version-arm floor: page edited BEFORE LINK_EXTRACTOR_VERSION_TS clears after --stale (issue #2576 bug 3)', async () => {
// A page whose updated_at predates the version watermark used to be
// stamped at its updated_at (< versionTs), so the version arm re-fired
// every run — permanently stale. The sweep now floors the stamp at
// versionTs. (The #1768 test above also covers this since the v0.42.x
// VERSION_TS bump moved its date below the watermark, but this pins the
// behavior explicitly so a date "repair" there can't drop coverage.)
await engine.putPage('people/alice', personPage('Alice'));
await engine.executeRaw(`UPDATE pages SET updated_at = '2000-01-01T00:00:00Z' WHERE slug = 'people/alice'`);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(1);
await runExtract(engine, ['--stale']);
// Pre-floor this stayed 1 forever (stamp < versionTs → version arm re-fires).
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
await runExtract(engine, ['--stale']);
expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
});
test('CDX-4 (D2): a link-flush throw aborts the sweep and leaves pages UNSTAMPED', async () => {
await engine.putPage('people/alice', personPage('Alice'));
await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) founded [Acme](companies/acme).'));
+18
View File
@@ -140,6 +140,15 @@ describe('extractEntityRefs', () => {
expect(wikiRefs[0].needsResolution).toBe(true);
});
test('recognizes ops/ qualified wikilinks (issue #2576 bug 2)', () => {
// `ops` was missing from DIR_PATTERN, so [[ops/...]] fell through to
// the generic 2c pass (needsResolution) instead of being a real ref.
const refs = extractEntityRefs('Deployed via [[ops/services/pointer-agent]].');
expect(refs.length).toBe(1);
expect(refs[0].slug).toBe('ops/services/pointer-agent');
expect(refs[0].needsResolution).toBeUndefined();
});
test('skips qualified-syntax tokens (those belong to 2a)', () => {
// [[wiki:topics/ai]] looks like 2a's qualified shape — even though
// it wouldn't satisfy DIR_PATTERN, 2c must not claim it either
@@ -1069,6 +1078,15 @@ describe('makeResolver — fallback chain', () => {
]);
});
test('resolveBasenameMatches: path-style ref falls back to the tail (issue #2576 bug 2)', async () => {
// normalizeBasename strips slashes, so `runbooks/2026-05-01-pointer-agent`
// used to normalize to a garbage key that never hit the tail-keyed index.
const engine = makeFakeEngineWithSlugs(['ops/changes/2026-05-01-pointer-agent']);
const r = makeResolver(engine);
expect(await r.resolveBasenameMatches!('runbooks/2026-05-01-pointer-agent'))
.toEqual(['ops/changes/2026-05-01-pointer-agent']);
});
test('resolveBasenameMatches: case-insensitive fallback', async () => {
const engine = makeFakeEngineWithSlugs(['companies/fast-weigh']);
const r = makeResolver(engine);