mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
32
Commits
@@ -0,0 +1,482 @@
|
||||
# MERGE_PHANTOMS — design retrospective + future implementation guide
|
||||
|
||||
**Status:** Deferred. The `gbrain merge-phantoms` command was built and then
|
||||
stripped from PR #1010 before merge. This doc captures the full context for a
|
||||
future agent to pick up the work with a cleaner abstraction.
|
||||
|
||||
**History pointers:**
|
||||
- PR #1010 (`fix/entity-resolution-prefix-expansion`) — the resolver + stub-guard
|
||||
+ backstop-audit work that DID land.
|
||||
- Plan: `~/.claude/plans/mossy-popping-crown.md` (decisions D1–D9).
|
||||
- Codex iteration commits: 32 commits on `fix/entity-resolution-prefix-expansion`,
|
||||
rounds 1–30. The stripped `src/commands/merge-phantoms.ts` and
|
||||
`test/merge-phantoms.test.ts` are recoverable from any commit between
|
||||
`c861b43a` (initial scaffold) and `7f7a080f` (round-30 final).
|
||||
|
||||
---
|
||||
|
||||
## Problem statement
|
||||
|
||||
Pre-v0.34.5, the entity resolver had this failure mode:
|
||||
|
||||
1. User says "I just talked to Alice" in a session.
|
||||
2. `extract_facts` calls `resolveEntitySlug(engine, source_id, "Alice")`.
|
||||
3. The resolver tries exact-match, then fuzzy via pg_trgm. Short bare names
|
||||
like "Alice" score below the 0.4 similarity threshold, so fuzzy fails.
|
||||
4. Resolver falls through to `slugify("Alice")` → `"alice"`.
|
||||
5. `writeFactsToFence(..., { slug: "alice" })` doesn't find `alice.md` on disk,
|
||||
so it calls `stubEntityPage("alice")` which produces a minimal stub:
|
||||
```
|
||||
---
|
||||
type: concept
|
||||
title: alice
|
||||
slug: alice
|
||||
---
|
||||
|
||||
# alice
|
||||
```
|
||||
6. A phantom page is born at the brain's root. The fact lands in its `## Facts`
|
||||
fence. The user's REAL `people/alice-example.md` doesn't see the fact.
|
||||
7. Repeat for every bare-name reference. Over months, the brain accumulates a
|
||||
pile of phantom unprefixed entity pages (`alice.md`, `jared.md`, `diana.md`)
|
||||
splitting facts away from their canonical prefixed pages.
|
||||
|
||||
The phantom population on a real production brain (the original PR was filed
|
||||
against an OpenClaw deployment) was in the hundreds.
|
||||
|
||||
---
|
||||
|
||||
## What the PR was supposed to do — the original three-layer fix
|
||||
|
||||
PR #1010 landed three independent layers, all of which are still in master:
|
||||
|
||||
1. **Resolver prefix expansion** (`src/core/entities/resolve.ts`). When a bare
|
||||
name's exact + fuzzy match fails, the resolver walks each configured entity
|
||||
directory (`entities.prefix_expansion_dirs`, default `['people', 'companies',
|
||||
'deals', 'topics', 'concepts']`) and queries `slug = '<dir>/<token>'` OR `slug
|
||||
LIKE '<dir>/<token>-%'`. Picks the highest-connection match. `Alice` now
|
||||
finds `people/alice-example` BEFORE falling through to slugify.
|
||||
|
||||
2. **Stub-creation guard** (`src/core/facts/fence-write.ts`). When
|
||||
`writeFactsToFence` would stub-create a new entity page whose slug has no
|
||||
directory prefix, it refuses and returns `stubGuardBlocked: true`. The
|
||||
guard also fires on existing stub-shaped files (post-round-24) and consults
|
||||
the DB body to avoid blocking legitimate DB-only pages (post-round-26).
|
||||
|
||||
3. **Backstop dropped-fact audit** (`src/core/facts/backstop.ts` +
|
||||
`src/core/facts/dropped-audit.ts`). When the stub-guard fires, the backstop
|
||||
no longer inserts a legacy-shape DB row (that path tripped the v0.32.2
|
||||
`extract_facts` reconciliation guard). Instead it appends a structured entry
|
||||
to `~/.gbrain/facts.dropped.jsonl` for operator recovery. The fact text is
|
||||
preserved verbatim so a future `gbrain replay-dropped` tool can re-process
|
||||
the entries once the canonical entity pages exist.
|
||||
|
||||
These three layers stop NEW phantom creation. They do not address the existing
|
||||
pile.
|
||||
|
||||
---
|
||||
|
||||
## What `gbrain merge-phantoms` was supposed to do
|
||||
|
||||
D7 of the plan-eng-review (the user upgraded my "add to TODOs" recommendation
|
||||
to "build it now in this PR") was a destructive operator command:
|
||||
|
||||
```
|
||||
gbrain merge-phantoms [--dry-run] [--source SOURCE_ID] [--json]
|
||||
```
|
||||
|
||||
For each unprefixed entity page in the brain:
|
||||
1. Find the canonical target via prefix expansion (e.g. `alice` → `people/alice-example`).
|
||||
2. Re-fence the phantom's active facts into the canonical's `## Facts` fence.
|
||||
3. Soft-delete the phantom page (v0.26.5 destructive-guard machinery).
|
||||
4. Hard-purge after 72h via the autopilot cycle's purge phase.
|
||||
|
||||
Stated simply: "find phantoms, merge them into their canonical, delete the
|
||||
phantom." This sounded straightforward.
|
||||
|
||||
It was not.
|
||||
|
||||
---
|
||||
|
||||
## How it became a bug farm
|
||||
|
||||
The command was built in PR #1010 and reviewed by `codex review --base master`
|
||||
30 times. Each round found 1-2 real bugs. By round 30 the file had grown to
|
||||
~600 lines with 8 skip reasons, bi-directional drift detection, rollback
|
||||
machinery on import failure, on-disk stub detection, DB stale-detection,
|
||||
materialize-from-DB-before-fence, and tuple-set comparison for content drift.
|
||||
|
||||
Below is the round-by-round account. Each round found a real bug — codex
|
||||
wasn't inventing problems. The patches WERE necessary given the chosen
|
||||
abstraction. But the abstraction itself was wrong, which is why the work
|
||||
never converged.
|
||||
|
||||
### The cascade table
|
||||
|
||||
| Round | Severity | Finding | Fix |
|
||||
|-------|----------|---------|-----|
|
||||
| 1 | P1 | Real names in test fixtures break `check:privacy` | Scrub to placeholders |
|
||||
| 1 | P1 | Private OpenClaw-fork names in stripped proposal doc | Strip doc |
|
||||
| 1 | P2 | Full table GROUP-BYs in connection-count query | Correlated subqueries |
|
||||
| 2 | P1 | Backstop legacy-DB insert trips v0.32.2 extract_facts guard | Drop fact + JSONL audit |
|
||||
| 2 | P2 | Merge UPDATE only moves entity_slug, leaves source_markdown_slug stale | Re-fence into canonical |
|
||||
| 3 | P2 | Resolver misses `<dir>/<token>` (no hyphen suffix) | Match both shapes |
|
||||
| 3 | P2 | Default dir list missing `concepts/` | Add to default |
|
||||
| 3 | P2 | merge-phantoms missing from thin-client refusal | Add |
|
||||
| 4 | P1 | Cross-type collision (acme-company merged into people/acme-*) | Type-constrained search |
|
||||
| 4 | P2 | Dry-run reports merge that real run would skip | Move feasibility check before dry-run |
|
||||
| 5 | P2 | Real top-level pages (rag.md) classified as phantoms | Body-size threshold |
|
||||
| 5 | P2 | Tests acquire page-locks under user's real ~/.gbrain | GBRAIN_HOME isolation |
|
||||
| 6 | P2 | Pre-fix phantoms have `type: concept` default — type filter breaks them | Search all dirs |
|
||||
| 6 | P2 | Fact-bearing phantoms exceed body threshold | Strip fence from body count |
|
||||
| 7 | P2 | Live resolver still routes new facts to existing phantoms | Prefix-first ordering |
|
||||
| 7 | P2 | Timeline content stripped from stub detection | Preserve Timeline |
|
||||
| 8 | P2 | Stub-strip matches `## Facts` heading without machine markers | Match only fence markers |
|
||||
| 8 | P2 | Prefix-first overrides real top-level pages | Stub-shape gate |
|
||||
| 9 | **P1** | `stubBodyChars` regex used fictional fence markers (`<!-- facts -->`) — never matched real fences (`<!--- gbrain:facts:begin -->`) | Real markers |
|
||||
| 9 | **P1** | `listFactsByEntity({ limit: 10_000 })` clamps at MAX_SEARCH_LIMIT=100 — overflow lost | Raw SQL |
|
||||
| 10 | P2 | `valid_until` dropped during fact migration | Thread through FenceInputFact |
|
||||
| 11 | P2 | 50-char threshold misclassifies terse real pages | Lower to 0 |
|
||||
| 12 | P2 | postgres-js returns embeddings as text strings, not Float32Array | tryParseEmbedding |
|
||||
| 13 | P2 | Hardcoded 'default' ignores GBRAIN_SOURCE / .gbrain-source chain | Use resolveSourceId |
|
||||
| 14 | **P1** | writeFactsToFence doesn't refresh `pages.compiled_truth` — next extract_facts wipes migrated rows | Re-import canonical |
|
||||
| 15 | P2 | importFromFile non-throw failure (skipped/error) still proceeds to delete phantom | Check ImportResult status |
|
||||
| 16 | P2 | Timeline-only pages misclassified as stubs | Include pages.timeline column |
|
||||
| 16 | P2 | Retry idempotency comment was wrong (real rerun fails on UNIQUE) | Rollback logic |
|
||||
| 17 | P2 | DB-only canonical (put_page MCP, no .md) gets stub-overwritten | Materialize from DB |
|
||||
| 18 | **P1** | writeFactsToFence stub-guard drops facts for legitimate DB-only bare pages | Materialize-then-append |
|
||||
| 18 | P2 | tryExactSlugBody misses timeline column | Include in concat |
|
||||
| 19 | P2 | Soft-deleted phantom .md file lingers — next sync resurrects | Unlink on soft-delete |
|
||||
| 20 | P2 | Factless-phantom early continue bypasses unlink | Apply unlink in factless branch |
|
||||
| 20 | P2 | Rerun NOT idempotent — engine.insertFacts uses plain INSERTs | Rollback canonical rows on import failure |
|
||||
| 21 | P2 | DB stale relative to disk — unsynced .md edits get unlinked | Disk-side stub check |
|
||||
| 22 | P2 | Prefix expansion overrides legitimate fuzzy title matches (Liz/Elizabeth) | Stub-only override |
|
||||
| 23 | — | **CLEAN ROUND** (1 of 30) | — |
|
||||
| 24 | P2 | Existing stub-shaped .md files slip past guard | Guard on existing files |
|
||||
| 25 | P2 | Capitalized bare name `Alice` bounces past real bare slug | Return token immediately |
|
||||
| 26 | P2 | Disk-stub gate drops facts when DB has real content | Check DB before dropping |
|
||||
| 27 | **P1** | Factless DB but populated disk fence — unlink loses unreconciled facts | Parse fence, skip with fence_drift |
|
||||
| 28 | P2 | Dry-run reports merge for fence-drift case real run would skip | Move drift check before dry-run |
|
||||
| 29 | **P1** | Drift check only ran when facts_moved=0; mixed-state phantoms still lost disk-only facts | Run drift check for all file-backed phantoms |
|
||||
| 30 | P2 | One-directional drift detection — user's strikethroughs get resurrected from stale DB | Bi-directional drift + tuple-set comparison |
|
||||
|
||||
### Why the cascade kept producing real bugs
|
||||
|
||||
Read down that table. There's a pattern. Each fix changed the shape of the
|
||||
state machine. Each new shape exposed a new axis the previous shape didn't
|
||||
consider:
|
||||
|
||||
- Body-size threshold (5) → fence inflates body length (6)
|
||||
- Strip Facts fence (6) → also stripped Timeline (7)
|
||||
- Prefix-first ordering (7) → overrides real pages (8) → 50-char threshold (8) →
|
||||
misclassifies terse pages (11) → strict-zero threshold (11)
|
||||
- Type-constrained search (4) → pre-fix concept default (6) → search all dirs +
|
||||
ambiguity skip (6) → ambiguous candidates field (6)
|
||||
- Stale compiled_truth (14) → re-import canonical → import can fail without
|
||||
throwing (15) → rollback on failure (15) → rollback is incomplete (16)
|
||||
- Fence-drift check (27) → only ran for factless (29) → only one-directional (30)
|
||||
|
||||
This is a textbook "the design grows its own bugs" pattern. The defensive
|
||||
checks were all real — codex caught them with concrete reproductions — but
|
||||
they were defending the wrong shape.
|
||||
|
||||
---
|
||||
|
||||
## The meta-insight — why this is the wrong shape
|
||||
|
||||
The fence is the system of record per the v0.32.2 contract (`src/core/cycle/extract-facts.ts:1-32`).
|
||||
The DB index is downstream. Reconciliation between the two is the
|
||||
`extract_facts` cycle phase's job.
|
||||
|
||||
`merge-phantoms` was trying to do BOTH:
|
||||
1. Route facts from phantom slug → canonical slug (an entity-resolution concern).
|
||||
2. Reconcile the markdown fence ↔ DB index across the move (a reconciliation concern).
|
||||
|
||||
By doing #2 in a parallel command instead of letting the existing reconciliation
|
||||
infrastructure handle it, the command had to duplicate every drift-handling
|
||||
case that `extract_facts` already handles. That's the bug farm:
|
||||
**every drift case `extract_facts` knows about had to be re-discovered by
|
||||
codex review on `merge-phantoms`.**
|
||||
|
||||
Examples of duplication:
|
||||
- `extract_facts` has the v0.32.2 reconciliation guard (`row_num IS NULL AND
|
||||
entity_slug IS NOT NULL` → refuse to reconcile). `merge-phantoms`'s round-2
|
||||
P1 was tripping that guard.
|
||||
- `extract_facts` deletes facts by `source_markdown_slug = slug` and re-inserts
|
||||
from the fence. `merge-phantoms`'s round-14 P1 was forgetting to refresh
|
||||
`compiled_truth` so the next `extract_facts` would do exactly this delete-and-
|
||||
re-insert against stale state.
|
||||
- `extract_facts` already handles fence drift (strikethrough, forgotten,
|
||||
superseded). `merge-phantoms`'s rounds 27–30 were re-implementing the same
|
||||
drift detection.
|
||||
|
||||
---
|
||||
|
||||
## Speculation — the platonic-ideal implementations
|
||||
|
||||
Three plausible shapes for a future agent. Listed in order of "amount of
|
||||
existing infrastructure reused." Pick based on operator UX preferences.
|
||||
|
||||
### Option Alpha — report-only command, manual remediation
|
||||
|
||||
```
|
||||
gbrain merge-phantoms [--source SOURCE_ID] [--json]
|
||||
```
|
||||
|
||||
Read-only. Lists phantoms + suggested canonical targets. User runs
|
||||
existing primitives to remediate:
|
||||
|
||||
```
|
||||
$ gbrain merge-phantoms
|
||||
3 phantom unprefixed entity pages found in source=default:
|
||||
|
||||
alice.md → people/alice-example (4 facts on phantom, 0 on canonical)
|
||||
jared.md → people/jared-friedman (12 facts on phantom, 3 on canonical)
|
||||
acme.md → companies/acme-example (1 fact on phantom)
|
||||
|
||||
To merge:
|
||||
1. Run `gbrain dream --phase extract_facts` to reconcile fence ↔ DB.
|
||||
2. For each phantom:
|
||||
- Edit alice.md's facts fence: move row to people/alice-example.md.
|
||||
- `rm alice.md`
|
||||
- `gbrain sync`
|
||||
|
||||
To verify no facts are lost, compare counts before/after via `gbrain recall
|
||||
--entity people/alice-example | wc -l`.
|
||||
```
|
||||
|
||||
- **Code size:** ~80 lines
|
||||
- **Risk:** zero (no destructive paths)
|
||||
- **Operator burden:** high
|
||||
- **Best when:** the brain has fewer than ~20 phantoms and the operator wants
|
||||
full control.
|
||||
|
||||
### Option Beta — phantom redirect in the extract_facts cycle phase
|
||||
|
||||
Don't build a separate command. Add phantom-redirect logic to the existing
|
||||
`runExtractFacts` function in `src/core/cycle/extract-facts.ts`.
|
||||
|
||||
When `extract_facts` walks pages:
|
||||
1. If the page has an unprefixed slug AND is type=person/company/deal/topic/concept:
|
||||
2. Compute the canonical target via `tryPrefixExpansion`.
|
||||
3. If canonical exists and is unambiguous:
|
||||
- For each fact row keyed on the phantom, move it to the canonical
|
||||
(update `entity_slug` + `source_markdown_slug`).
|
||||
- Append the migrated fence rows to the canonical's markdown body.
|
||||
- Soft-delete the phantom page + unlink the .md file.
|
||||
4. If canonical is ambiguous or missing, leave the phantom alone (continue to
|
||||
reconcile in place; the operator can resolve manually later).
|
||||
|
||||
This piggybacks on:
|
||||
- The existing `extract_facts` empty-fence guard (the v0.32.2 reconciliation
|
||||
contract).
|
||||
- The existing fence parser / strikethrough / forget semantics.
|
||||
- The existing `deleteFactsForPage` + `insertFacts` reconcile pattern.
|
||||
- The existing autopilot purge phase (72h soft-delete TTL).
|
||||
|
||||
Reconciliation drift is the EXISTING handler's problem, not a parallel
|
||||
implementation. The phantom-redirect concern is small: "compute the canonical
|
||||
target, treat the migrated fence as the new source-of-record for the canonical
|
||||
page."
|
||||
|
||||
- **Code size:** ~150 lines added to extract-facts.ts + minimal new tests
|
||||
- **Risk:** low (reuses battle-tested code paths)
|
||||
- **Operator burden:** zero (automatic on next autopilot cycle)
|
||||
- **Best when:** the brain is actively running autopilot. This is the right
|
||||
default.
|
||||
|
||||
**Open design question for Beta:** does the operator want SEE the migration
|
||||
happen, or should it be invisible? If invisible, the operator might be
|
||||
surprised when `alice.md` disappears from their brain repo. Suggest: emit a
|
||||
progress event (`cycle.extract_facts.phantom_redirected`) and tally counts
|
||||
in the cycle report so `gbrain doctor` can surface them.
|
||||
|
||||
### Option Gamma — phantoms as a first-class schema concept
|
||||
|
||||
The most invasive option. Add `pages.canonical_of TEXT REFERENCES pages.slug`
|
||||
to the schema:
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ADD COLUMN canonical_of TEXT;
|
||||
-- canonical_of points at the page this row is a phantom of, NULL when
|
||||
-- the row is itself canonical.
|
||||
CREATE INDEX idx_pages_canonical_of ON pages(canonical_of) WHERE canonical_of IS NOT NULL;
|
||||
```
|
||||
|
||||
Then:
|
||||
- `resolveEntitySlug` follows `canonical_of` transparently: if it lands on a
|
||||
phantom page, return its canonical.
|
||||
- `writeFactsToFence` follows `canonical_of` BEFORE picking a target path.
|
||||
- Search (`hybridSearch`) hides phantom pages from results (already in the
|
||||
visibility chain via `deleted_at`).
|
||||
- Migration becomes a SQL UPDATE: `UPDATE pages SET canonical_of = $canonical
|
||||
WHERE slug = $phantom_slug AND source_id = $source_id`.
|
||||
- The markdown file stays on disk as a tombstone with frontmatter
|
||||
`canonical_of: people/alice-example` until the operator deletes it manually
|
||||
(no destructive command needed).
|
||||
|
||||
- **Code size:** schema migration + ~20 lines per affected callsite (resolver,
|
||||
fence-write, search). Maybe 300 lines total.
|
||||
- **Risk:** medium (touches schema + multiple callsites)
|
||||
- **Operator burden:** zero
|
||||
- **Best when:** the brain has thousands of phantoms or wants phantom-as-
|
||||
first-class concept for other reasons (e.g. alias support).
|
||||
|
||||
**This option also unlocks:** entity aliases (`canonical_of` becomes "alias
|
||||
of"). User can have `alice.md` with `canonical_of: people/alice-example` as a
|
||||
deliberate redirect for legacy URLs. The phantom-fix becomes a special case
|
||||
of a general alias system.
|
||||
|
||||
### Recommendation among the three
|
||||
|
||||
Build **Beta** first. It's the smallest change that solves the actual problem,
|
||||
reuses existing infrastructure, and runs automatically. The bug farm went away
|
||||
the moment the reconciliation concern moved into the existing reconcile path.
|
||||
|
||||
Iterate to **Gamma** if/when alias support is needed for other reasons. The
|
||||
schema column is small enough that adding it later is fine — `canonical_of`
|
||||
defaults NULL and only the phantom-redirect callsite needs to set it.
|
||||
|
||||
Skip **Alpha** unless the operator explicitly wants manual control.
|
||||
|
||||
---
|
||||
|
||||
## What's recoverable from PR #1010's iteration
|
||||
|
||||
Even though the implementation is being scrapped, the codex iteration found
|
||||
real bugs that a future implementation MUST handle. Treat the round-by-round
|
||||
commit messages as a regression checklist:
|
||||
|
||||
- **Round 9 (markers):** the fence markers are `<!--- gbrain:facts:begin -->`
|
||||
and `<!--- gbrain:facts:end -->`, NOT `<!-- facts -->`. They live as
|
||||
exported constants in `src/core/facts-fence.ts:53-54`. Use them.
|
||||
- **Round 9 (clamp):** `listFactsByEntity` clamps `limit` at MAX_SEARCH_LIMIT
|
||||
(100). For unbounded reads, go through raw SQL.
|
||||
- **Round 10 (valid_until):** `FenceInputFact.validUntil` is now part of the
|
||||
fence-write contract. Carry it through migrations.
|
||||
- **Round 12 (embeddings):** postgres-js returns pgvector embeddings as text
|
||||
strings; PGLite returns Float32Array directly. Normalize via
|
||||
`tryParseEmbedding` from `src/core/utils.ts`.
|
||||
- **Round 13 (source resolution):** any operator command must honor the
|
||||
4-tier resolveSourceId chain.
|
||||
- **Round 14 (stale compiled_truth):** writeFactsToFence does NOT refresh
|
||||
`pages.compiled_truth`. The next extract_facts cycle will reconcile from
|
||||
the markdown, but if anything reads compiled_truth between writeFactsToFence
|
||||
and the next cycle, it sees stale state.
|
||||
- **Round 17 (DB-only canonical):** canonical pages can exist in the DB via
|
||||
MCP `put_page` without ever having a .md file. Any code that calls
|
||||
writeFactsToFence on them must materialize the body from DB first.
|
||||
- **Round 18 (timeline column):** `pages.timeline` is a separate column. Stub
|
||||
detection must read both compiled_truth + timeline.
|
||||
- **Round 22 (fuzzy precedence):** prefix expansion should NOT short-circuit
|
||||
fuzzy when no bare slug exists. The "Liz/Elizabeth" case.
|
||||
- **Round 27 + 29 + 30 (fence drift):** the fence is the system of record. Any
|
||||
operation that mutates DB rows MUST verify fence/DB consistency first, in
|
||||
both directions, including tuple-content comparison when counts match.
|
||||
|
||||
The stripped `merge-phantoms.ts` (last good version is commit `7f7a080f`)
|
||||
is a worked example of EVERY one of these gotchas. Read it before building
|
||||
Option Beta — not to copy, but as a regression checklist.
|
||||
|
||||
---
|
||||
|
||||
## Code pointers — what was stripped, what stayed
|
||||
|
||||
**Stripped from PR #1010 (will not land):**
|
||||
- `src/commands/merge-phantoms.ts` — the entire 600-line command
|
||||
- `test/merge-phantoms.test.ts` — 31 tests
|
||||
- `src/cli.ts` entries: `CLI_ONLY`, `CLI_ONLY_SELF_HELP`,
|
||||
`THIN_CLIENT_REFUSED_COMMANDS`, `THIN_CLIENT_REFUSE_HINTS`, the dispatch
|
||||
case, and the help text line
|
||||
|
||||
**Kept (lands with PR #1010):**
|
||||
- `src/core/entities/resolve.ts` — full resolver with prefix expansion, stub
|
||||
detection, real-page preservation. ALL of this is independently valuable.
|
||||
- `src/core/facts/fence-write.ts` — stub-guard (rounds 1, 24, 26) and the
|
||||
DB-materialize path (round 18).
|
||||
- `src/core/facts/backstop.ts` — dropped-fact audit (round 2 P1).
|
||||
- `src/core/facts/dropped-audit.ts` — JSONL audit log infrastructure.
|
||||
- `test/entity-resolve.test.ts` — 33 tests for resolver behavior.
|
||||
- `entities.prefix_expansion_dirs` config key.
|
||||
|
||||
**Useful primitives (kept, intentionally exported for future Option Beta):**
|
||||
- `resolve.ts:tryPrefixExpansion(engine, source_id, token, opts?)` — search
|
||||
configured directories for prefix-match candidates.
|
||||
- `resolve.ts:stubBodyChars(compiled_truth)` — detect v0.34.5 stub shape.
|
||||
- `resolve.ts:isStubBody(compiled_truth)` — boolean wrapper.
|
||||
- `resolve.ts:PHANTOM_STUB_MAX_BODY_CHARS` — threshold constant (0).
|
||||
- `resolve.ts:getPrefixExpansionDirs()` — config-driven resolver dir list.
|
||||
|
||||
A future Option Beta implementation will likely use all five.
|
||||
|
||||
---
|
||||
|
||||
## Open questions for the future implementer
|
||||
|
||||
1. **Should phantom redirect happen during extract_facts (autopilot-time) or
|
||||
eagerly during resolveEntitySlug (write-time)?** Beta proposes the former
|
||||
so the heavy lifting happens in batch and gets cycle-level reporting.
|
||||
Write-time would route facts AROUND the phantom without ever migrating
|
||||
the page — different semantics.
|
||||
|
||||
2. **What's the right UX for ambiguous canonical?** When `alice` matches
|
||||
both `people/alice-example` AND `people/alice-other`, what happens?
|
||||
merge-phantoms skipped with `ambiguous`. The plan-eng-review suggested
|
||||
surfacing this for operator resolution. A redirect-during-cycle approach
|
||||
could log to `~/.gbrain/audit/phantom-ambiguous.jsonl` and continue.
|
||||
|
||||
3. **What about phantoms in non-default sources?** The current PR has source
|
||||
isolation (the `resolveSourceId` chain), but a multi-source brain might
|
||||
have the phantom in source A and the canonical in source B (mounted brain).
|
||||
Cross-source redirect is out of scope for v0.34.5 but worth thinking about.
|
||||
|
||||
4. **Should Option Beta also handle phantom links?** The `links` table has
|
||||
`from_page_id` / `to_page_id` referencing the phantom row. After redirect,
|
||||
those need to point at the canonical. Easy SQL but it needs to happen.
|
||||
|
||||
5. **What about `find_orphans` / `gbrain doctor` reporting?** A redirect-aware
|
||||
doctor check could surface "N phantom pages pending redirect" so operators
|
||||
know what's coming.
|
||||
|
||||
---
|
||||
|
||||
## How to pick this up
|
||||
|
||||
A future agent doing this work should:
|
||||
|
||||
1. Read this doc top to bottom.
|
||||
2. Read the round-by-round commit messages on PR #1010's commits between
|
||||
`c861b43a` and `7f7a080f` — they're a regression checklist.
|
||||
3. Decide Alpha / Beta / Gamma after a real `/plan-eng-review` on the
|
||||
abstraction question. **Do not** start from the stripped
|
||||
`merge-phantoms.ts` and try to clean it up. That code is the wrong
|
||||
shape; rewriting it as Option Beta is faster than refactoring it.
|
||||
4. If choosing Option Beta, the test surface should pin every regression
|
||||
in the cascade table above. The cascade table is the test backlog.
|
||||
5. Land the rewrite as its own PR, not bolted onto a resolver fix.
|
||||
|
||||
---
|
||||
|
||||
## Lessons learned (about the iteration, not the bug)
|
||||
|
||||
This isn't an indictment of any particular decision. The cascade was a
|
||||
predictable outcome of three things compounding:
|
||||
|
||||
1. **The plan-eng-review user-upgraded D7 from "follow-up" to "build it now."**
|
||||
That decision turned a small PR into a large one. Future plans should
|
||||
resist this — destructive operator commands should ALWAYS be follow-ups,
|
||||
not riders on the fix that motivated them.
|
||||
|
||||
2. **The chosen abstraction duplicated existing infrastructure.** The fence
|
||||
is the system of record. Any code that mutates fence + DB independently
|
||||
has to re-implement reconciliation. The cascade was the cost of that
|
||||
duplication.
|
||||
|
||||
3. **Codex review is brutally thorough.** Each round caught a real issue.
|
||||
The bugs WERE in the new code. But codex can't tell you "this whole
|
||||
abstraction is wrong" — it can only point at specific failure modes.
|
||||
The meta-insight required stepping out of the loop.
|
||||
|
||||
For the next destructive cleanup command in this codebase: do the design
|
||||
work BEFORE the implementation. Make the reviewer answer "is this the right
|
||||
abstraction?" before they're asked to review "does this code work?"
|
||||
@@ -106,6 +106,23 @@ export interface GBrainConfig {
|
||||
oauth_client_id: string;
|
||||
oauth_client_secret?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* v0.34.5 — entity-resolver tuning. Controls which slug directories
|
||||
* the prefix-expansion step (`src/core/entities/resolve.ts`) walks
|
||||
* when canonicalizing a bare first-name reference like "Alice" into
|
||||
* `people/alice-example`. The first matching directory wins (after the
|
||||
* connection-count tiebreak inside that directory).
|
||||
*
|
||||
* Default (when unset): `['people', 'companies', 'deals', 'topics']` —
|
||||
* matches the stub-guard's recognized prefix set in
|
||||
* `src/core/facts/fence-write.ts`. Override to support custom entity
|
||||
* schemas (`['funds', 'advisors']` etc.) once the rest of the system
|
||||
* grows to recognize them. Order matters: list higher-priority dirs first.
|
||||
*/
|
||||
entities?: {
|
||||
prefix_expansion_dirs?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+314
-11
@@ -9,9 +9,26 @@
|
||||
* Pure helper; the engine layer is the data dependency injected by callers.
|
||||
* Lives under `src/core/entities/` so signal-detector can reuse it for the
|
||||
* Sonnet pass too without circular import through facts/.
|
||||
*
|
||||
* v0.34.5 — added a prefix-expansion step between fuzzy match and
|
||||
* slugify fallback. Bare first names like "Alice" scored too low on
|
||||
* pg_trgm (short strings have terrible trigram overlap), so they fell
|
||||
* through to slugify("Alice") → "alice", which then spawned a phantom
|
||||
* `people/alice.md` stub instead of resolving to an existing
|
||||
* `people/alice-example` page. The fix queries `slug LIKE 'people/X-%'`
|
||||
* (then `companies/X-%`, etc., per the configured dir list) when fuzzy
|
||||
* fails on a single-word bare name, and uses connection count
|
||||
* (links + chunks) as the tiebreaker when multiple candidates match.
|
||||
*
|
||||
* The dir list is config-driven via `entities.prefix_expansion_dirs`
|
||||
* (see `src/core/config.ts`). Default covers the four directories the
|
||||
* stub-guard recognizes; custom brains override to support funds/,
|
||||
* advisors/, etc. See plan `mossy-popping-crown.md` D2.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { isUndefinedColumnError } from '../utils.ts';
|
||||
import { loadConfig } from '../config.ts';
|
||||
|
||||
/**
|
||||
* Canonicalize a free-form entity reference to a page slug.
|
||||
@@ -21,7 +38,10 @@ import type { BrainEngine } from '../engine.ts';
|
||||
* exact pages.slug row in this source), return it untouched.
|
||||
* 2. Try fuzzy match against pages.slug + pages.title within the source
|
||||
* (case-insensitive). Pick the highest-trgm-score match if any.
|
||||
* 3. Fall back to a deterministic slugify: lowercase-no-spaces with
|
||||
* 3. Prefix-expansion match for bare single-token names: walk each
|
||||
* configured entity directory (`entities.prefix_expansion_dirs`) and
|
||||
* query `<dir>/<token>-%`. Highest-connection wins.
|
||||
* 4. Fall back to a deterministic slugify: lowercase-no-spaces with
|
||||
* hyphen-collapse. NOT prefixed with a directory — caller decides
|
||||
* whether to prefix `people/`, `companies/`, etc.
|
||||
*
|
||||
@@ -37,22 +57,255 @@ export async function resolveEntitySlug(
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) return null;
|
||||
|
||||
// 1. Exact match on slug. If raw already looks like a slug (or matches
|
||||
// a row exactly), use it.
|
||||
// 1. Bare-name STUB override (codex rounds 7 + 8 + 22): when the
|
||||
// bare slug EXISTS as a stub-shaped page (a v0.34.5 phantom),
|
||||
// redirect to the canonical via prefix expansion. This is the
|
||||
// ONLY case where prefix expansion runs before exact/fuzzy —
|
||||
// we want to stop fact-writes from accumulating on the phantom.
|
||||
//
|
||||
// Round-22 calibration: when the bare slug is MISSING (no page
|
||||
// at all), prefix expansion runs LATER (step 4) so fuzzy gets
|
||||
// a chance first. Otherwise a bare reference like "Liz" would
|
||||
// redirect to `people/liz-smith` even when `people/elizabeth-
|
||||
// example` has title 'Liz' (a much better fuzzy match).
|
||||
if (isBareName(trimmed)) {
|
||||
const token = slugify(trimmed);
|
||||
const bareBody = await tryExactSlugBody(engine, source_id, token);
|
||||
if (bareBody !== 'missing') {
|
||||
if (isStubBody(bareBody)) {
|
||||
// Phantom-shaped bare slug exists — redirect to canonical.
|
||||
const expanded = await tryPrefixExpansion(engine, source_id, token);
|
||||
if (expanded) return expanded;
|
||||
// Prefix expansion found nothing; fall through to fuzzy/etc.
|
||||
} else {
|
||||
// Real bare page (intentional top-level entity). Return the
|
||||
// token NOW so a capitalized input like `Alice` doesn't bounce
|
||||
// through fuzzy/prefix expansion and get misrouted to a
|
||||
// sibling `people/alice-*` page. Codex round-25 P2.
|
||||
return token;
|
||||
}
|
||||
}
|
||||
// bareBody === 'missing': no page yet. Fall through to exact,
|
||||
// fuzzy, and catch-all prefix expansion (in that order).
|
||||
}
|
||||
|
||||
// 2. Exact match on slug. Catches prefixed slugs (`people/alice-example`)
|
||||
// that the caller passed verbatim, and intentional bare slugs
|
||||
// that step 1 declined to override.
|
||||
if (looksLikeSlug(trimmed)) {
|
||||
const exact = await tryExactSlug(engine, source_id, trimmed);
|
||||
if (exact) return exact;
|
||||
}
|
||||
|
||||
// 2. Fuzzy match against existing pages within the source. Match either
|
||||
// on slug fragment or on title.
|
||||
// 3. Fuzzy match against existing pages within the source. Match either
|
||||
// on slug fragment or on title. Title matches are how "Liz" finds
|
||||
// `people/elizabeth-example` when that page's title is 'Liz'.
|
||||
const fuzzy = await tryFuzzyMatch(engine, source_id, trimmed);
|
||||
if (fuzzy) return fuzzy;
|
||||
|
||||
// 3. Fallback: deterministic slugify.
|
||||
// 4. Bare-name catch-all prefix expansion: the bare slug doesn't
|
||||
// exist at all (step 1 saw 'missing'), exact didn't match, and
|
||||
// fuzzy didn't find a strong title hit. Try prefix expansion as
|
||||
// a last resort so first-name references like "Jared" (no page,
|
||||
// no fuzzy hit, low pg_trgm score) still land on
|
||||
// `people/jared-friedman` instead of slugifying to a phantom.
|
||||
if (isBareName(trimmed)) {
|
||||
const expanded = await tryPrefixExpansion(engine, source_id, slugify(trimmed));
|
||||
if (expanded) return expanded;
|
||||
}
|
||||
|
||||
// 5. Fallback: deterministic slugify. NOT prefixed — caller decides.
|
||||
return slugify(trimmed);
|
||||
}
|
||||
|
||||
/**
|
||||
* "Bare name" detector — true when the input is a single word with no
|
||||
* slash, no embedded prefix marker, and slugifies to a non-empty token.
|
||||
* Multi-word inputs (e.g. "Alice Example") are handled by fuzzy match;
|
||||
* this gate only fires for short first-name-shaped tokens.
|
||||
*/
|
||||
function isBareName(raw: string): boolean {
|
||||
if (raw.includes('/')) return false;
|
||||
// One-token input. Whitespace-tokenize: "Alice" → 1, "Alice Example" → 2.
|
||||
const tokens = raw.trim().split(/\s+/).filter(Boolean);
|
||||
if (tokens.length !== 1) return false;
|
||||
const slug = slugify(raw);
|
||||
if (!slug) return false;
|
||||
// Reject hyphenated multi-token slugs like "alice-example" — those
|
||||
// should hit the exact-slug or fuzzy path, not prefix expansion.
|
||||
if (slug.includes('-')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Default prefix-expansion directories. Covers the four entity types
|
||||
* the stub-guard recognizes in `src/core/facts/fence-write.ts` plus
|
||||
* `concepts/` which is the canonical home for `type: concept` pages
|
||||
* across gbrain docs and example schemas (e.g. `concepts/rag`,
|
||||
* `concepts/agentic-workflows`). Bare-name references like "RAG" or
|
||||
* "Bitcoin" therefore resolve out of the box. Override per-brain via
|
||||
* the `entities.prefix_expansion_dirs` config key.
|
||||
*/
|
||||
export const DEFAULT_PREFIX_EXPANSION_DIRS = ['people', 'companies', 'deals', 'topics', 'concepts'] as const;
|
||||
|
||||
/**
|
||||
* Body-content threshold — anything above this is treated as a real
|
||||
* user page rather than a v0.34.5-era stub.
|
||||
*
|
||||
* Threshold is 0 (codex round-11 P2): stubs from `stubEntityPage` in
|
||||
* fence-write.ts produce exactly `# Title\n` after the frontmatter,
|
||||
* which strips to "" via the regex chain below. Any non-empty
|
||||
* remainder means the user wrote something — even one sentence — and
|
||||
* the page should be preserved.
|
||||
*
|
||||
* A previous 50-char threshold misclassified terse hand-written pages
|
||||
* (e.g. `# RAG` + a single sentence). The strict-zero version flips
|
||||
* the contract: "stub iff nothing-but-stub-shape remains."
|
||||
*/
|
||||
export const PHANTOM_STUB_MAX_BODY_CHARS = 0;
|
||||
|
||||
/**
|
||||
* Strip frontmatter, H1 title, and the v0.32.2 facts-fence MARKERS
|
||||
* (not the heading text alone) from a `compiled_truth` body. Returns
|
||||
* the trimmed remainder.
|
||||
*
|
||||
* Codex rounds 6 + 7 + 8 calibration:
|
||||
* - Round 6: facts fence inflates body length on fact-bearing
|
||||
* phantoms; must be excluded so the stub detector catches them.
|
||||
* - Round 7: timeline content is NOT migrated by merge-phantoms;
|
||||
* must be preserved in the body count so pages with a populated
|
||||
* Timeline trip not_a_stub.
|
||||
* - Round 8: strip ONLY the canonical `<!-- facts -->` ...
|
||||
* `<!-- /facts -->` fence pair (and an immediately-preceding
|
||||
* `## Facts` heading if and only if it's paired with the fence
|
||||
* markers). NEVER strip arbitrary content under a `## Facts`
|
||||
* heading without machine markers — that's user-authored prose.
|
||||
*
|
||||
* A v0.34.5 stub returns ~0 chars; a fact-bearing stub does too
|
||||
* (the fence between machine markers strips out); a real page with
|
||||
* user-authored content under any heading returns hundreds.
|
||||
*/
|
||||
export function stubBodyChars(compiledTruth: string | null | undefined): number {
|
||||
if (!compiledTruth) return 0;
|
||||
const stripped = compiledTruth
|
||||
.replace(/^---\n[\s\S]*?\n---\n?/, '')
|
||||
.replace(/^#\s+.+\r?\n?/m, '')
|
||||
// Strip the canonical machine-generated facts fence. Markers MUST
|
||||
// match the constants in src/core/facts-fence.ts exactly —
|
||||
// `<!--- gbrain:facts:begin -->` / `<!--- gbrain:facts:end -->`.
|
||||
// Codex round-9 P1 #1 caught a marker mismatch: an earlier
|
||||
// version used the fictional `<!-- facts -->` shape which never
|
||||
// matched real fences, so fact-bearing phantoms slipped past
|
||||
// not_a_stub and merge-phantoms skipped them. The optional
|
||||
// preceding `## Facts` heading is consumed too so the auto-
|
||||
// generated section disappears cleanly. A user-authored
|
||||
// `## Facts` heading WITHOUT the machine markers is preserved.
|
||||
.replace(/(?:^##\s*Facts\s*\n+)?<!---\s*gbrain:facts:begin\s*-->[\s\S]*?<!---\s*gbrain:facts:end\s*-->\n?/m, '')
|
||||
.trim();
|
||||
return stripped.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Predicate wrapper around `stubBodyChars` for callers that don't
|
||||
* need the raw char count. True when the body looks like a v0.34.5-era
|
||||
* stub (or an empty page).
|
||||
*/
|
||||
export function isStubBody(compiledTruth: string | null | undefined): boolean {
|
||||
return stubBodyChars(compiledTruth) <= PHANTOM_STUB_MAX_BODY_CHARS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the active prefix-expansion dir list. Config-first, default-fallback.
|
||||
* Exported for tests and for callers that want to inspect the configured set.
|
||||
*/
|
||||
export function getPrefixExpansionDirs(): readonly string[] {
|
||||
const cfg = loadConfig();
|
||||
const fromConfig = cfg?.entities?.prefix_expansion_dirs;
|
||||
if (Array.isArray(fromConfig) && fromConfig.length > 0) {
|
||||
// Filter to non-empty strings; anything else silently drops so a bad
|
||||
// config entry can't 500 the resolver. Matches the v0.31.12
|
||||
// model-tier resolver's "fall back to defaults on bad input" posture.
|
||||
const cleaned = fromConfig.filter((s): s is string => typeof s === 'string' && s.length > 0);
|
||||
if (cleaned.length > 0) return cleaned;
|
||||
}
|
||||
return DEFAULT_PREFIX_EXPANSION_DIRS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up pages whose slug is `<dir>/<token>` OR starts with
|
||||
* `<dir>/<token>-` for each configured entity directory. When multiple
|
||||
* candidates match within a directory, pick the one with the highest
|
||||
* connection count (links_in + links_out + chunk count) — the most-
|
||||
* mentioned entity is the most likely canonical target for a bare-name
|
||||
* reference. When no candidates match in any directory, returns null
|
||||
* and the caller falls through to slugify.
|
||||
*
|
||||
* Also exported so `gbrain merge-phantoms` can run the prefix step in
|
||||
* isolation: resolveEntitySlug's exact-slug short-circuit would match
|
||||
* a phantom against itself before reaching prefix expansion. `token`
|
||||
* should be a pre-slugified single word (e.g. `'alice'`, not `'Alice'`).
|
||||
*
|
||||
* The connection-count subqueries are correlated (per page row) rather
|
||||
* than full table aggregates so the cost scales with the number of
|
||||
* slug-matching candidates (typically 1-3), not with brain size.
|
||||
* Indexes used: `idx_links_to`, `idx_links_from`, `idx_chunks_page`.
|
||||
*/
|
||||
export async function tryPrefixExpansion(
|
||||
engine: BrainEngine,
|
||||
source_id: string,
|
||||
token: string,
|
||||
opts: { dirs?: readonly string[] } = {},
|
||||
): Promise<string | null> {
|
||||
// Callers can constrain the search to a specific subset of directories
|
||||
// (e.g. merge-phantoms passes a single-element list matching the
|
||||
// phantom's entity type to prevent cross-type mismerges). Defaults to
|
||||
// the full configured set via getPrefixExpansionDirs.
|
||||
const searchDirs = opts.dirs ?? getPrefixExpansionDirs();
|
||||
for (const dir of searchDirs) {
|
||||
// Match BOTH `<dir>/<token>` exactly AND `<dir>/<token>-%` (hyphenated
|
||||
// variants). Without the exact-match leg, a brain whose canonical slug
|
||||
// is `companies/acme` (no hyphen-suffix) would never resolve bare
|
||||
// "Acme" — codex caught this in the third /codex review pass. The
|
||||
// ORDER BY tiebreak still applies across both candidate shapes.
|
||||
const exact = `${dir}/${token}`;
|
||||
const hyphen = `${dir}/${token}-%`;
|
||||
try {
|
||||
const rows = await engine.executeRaw<{
|
||||
slug: string;
|
||||
connection_count: number;
|
||||
}>(
|
||||
// Correlated subqueries: each per p.id, hitting the existing
|
||||
// (to_page_id), (from_page_id), (page_id) indexes. The outer
|
||||
// WHERE clause uses `slug = $2 OR slug LIKE $3` so both the
|
||||
// exact and the prefix candidates feed the same tiebreak.
|
||||
`SELECT p.slug,
|
||||
((SELECT COUNT(*) FROM links WHERE to_page_id = p.id) +
|
||||
(SELECT COUNT(*) FROM links WHERE from_page_id = p.id) +
|
||||
(SELECT COUNT(*) FROM content_chunks WHERE page_id = p.id))
|
||||
AS connection_count
|
||||
FROM pages p
|
||||
WHERE p.source_id = $1
|
||||
AND p.deleted_at IS NULL
|
||||
AND (p.slug = $2 OR p.slug LIKE $3)
|
||||
ORDER BY connection_count DESC, p.slug ASC
|
||||
LIMIT 1`,
|
||||
[source_id, exact, hyphen],
|
||||
);
|
||||
if (rows.length === 0) continue;
|
||||
return rows[0].slug;
|
||||
} catch (err) {
|
||||
// Narrow probe: column-missing on legacy brains (most likely
|
||||
// `deleted_at` pre-v0.26.5) falls through to the next directory.
|
||||
// Genuine failures (pool exhaustion, lock timeout, network blip)
|
||||
// propagate so they're visible instead of silently masquerading
|
||||
// as "no prefix match." See v0.26.9 D14 in CLAUDE.md.
|
||||
if (isUndefinedColumnError(err, 'deleted_at')) continue;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function looksLikeSlug(s: string): boolean {
|
||||
// Slug shape: lowercase letters/digits with at least one slash OR matches
|
||||
// [a-z0-9-]+ exactly. Anything with whitespace or capital letters fails.
|
||||
@@ -61,6 +314,48 @@ function looksLikeSlug(s: string): boolean {
|
||||
return /^[a-z0-9/_-]+$/.test(s);
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe the body of a bare-slug exact match without returning the
|
||||
* slug. Returns:
|
||||
* - 'missing' when no page exists with that exact slug
|
||||
* - the body string when a page exists (may be empty or a stub)
|
||||
*
|
||||
* Used by step 1 of resolveEntitySlug to distinguish phantom-shaped
|
||||
* bare slugs (where the canonical should win) from real top-level
|
||||
* pages (where the bare slug should win). The 3-value return shape
|
||||
* makes the caller's intent explicit (missing vs. stub vs. real).
|
||||
*/
|
||||
async function tryExactSlugBody(
|
||||
engine: BrainEngine,
|
||||
source_id: string,
|
||||
candidate: string,
|
||||
): Promise<string | 'missing'> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ compiled_truth: string | null; timeline: string | null }>(
|
||||
`SELECT compiled_truth, timeline FROM pages
|
||||
WHERE source_id = $1 AND slug = $2 AND deleted_at IS NULL
|
||||
LIMIT 1`,
|
||||
[source_id, candidate],
|
||||
);
|
||||
if (rows.length === 0) return 'missing';
|
||||
// Codex round-18 P2: include the timeline column so a real
|
||||
// top-level page whose substantive content lives in
|
||||
// pages.timeline isn't classified as stub-shaped by the
|
||||
// resolver's bare-name-override gate. The downstream
|
||||
// `isStubBody(body)` check will see a non-empty body whenever
|
||||
// EITHER compiled_truth has real content OR the timeline column
|
||||
// does. Concatenated via a separator so a single non-empty
|
||||
// column is enough to defeat the stub heuristic.
|
||||
const compiled = rows[0].compiled_truth ?? '';
|
||||
const timeline = (rows[0].timeline ?? '').trim();
|
||||
if (timeline.length > 0) return `${compiled}\n\n## Timeline\n\n${timeline}`;
|
||||
return compiled;
|
||||
} catch (err) {
|
||||
if (isUndefinedColumnError(err, 'deleted_at')) return 'missing';
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async function tryExactSlug(
|
||||
engine: BrainEngine,
|
||||
source_id: string,
|
||||
@@ -72,8 +367,11 @@ async function tryExactSlug(
|
||||
[source_id, candidate],
|
||||
);
|
||||
if (rows.length > 0) return rows[0].slug;
|
||||
} catch {
|
||||
// Defensive: fail open. Caller still gets a slug from the fallback.
|
||||
} catch (err) {
|
||||
// Legacy brain without `deleted_at` column — fall through to slugify
|
||||
// fallback. Other failures propagate.
|
||||
if (isUndefinedColumnError(err, 'deleted_at')) return null;
|
||||
throw err;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -107,9 +405,14 @@ async function tryFuzzyMatch(
|
||||
[source_id, lc, fragment],
|
||||
);
|
||||
if (rows.length > 0 && rows[0].score >= 0.4) return rows[0].slug;
|
||||
} catch {
|
||||
// pg_trgm functions might not be available on every engine config;
|
||||
// fall through to slugify.
|
||||
} catch (err) {
|
||||
// pg_trgm functions (`similarity`, `%`) might not be available on
|
||||
// every engine config; same for `deleted_at` on legacy brains.
|
||||
// Either fall through to the prefix-expansion + slugify path.
|
||||
if (isUndefinedColumnError(err, 'deleted_at')) return null;
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (/similarity|operator does not exist|function similarity/i.test(msg)) return null;
|
||||
throw err;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -40,6 +40,7 @@
|
||||
import type { BrainEngine, FactInsertStatus, NewFact } from '../engine.ts';
|
||||
import { isFactsBackstopEligible } from './eligibility.ts';
|
||||
import type { PageType } from '../types.ts';
|
||||
import { appendDroppedFactAudit } from './dropped-audit.ts';
|
||||
|
||||
export interface FactsBackstopCtx {
|
||||
engine: BrainEngine;
|
||||
@@ -455,6 +456,41 @@ async function runPipelineWithBody(
|
||||
// would write rows to a DB index whose fence is broken.
|
||||
continue;
|
||||
}
|
||||
if (result.stubGuardBlocked) {
|
||||
// v0.34.5 (codex P1 follow-up): writeFactsToFence refused to spawn a
|
||||
// phantom unprefixed entity page (e.g. `zoolander.md` at brain root).
|
||||
// The pre-codex-review version of this branch inserted these facts
|
||||
// via `engine.insertFact` to avoid silent data loss, but that
|
||||
// produced rows shaped like pre-v0.32.2 legacy migration rows
|
||||
// (`row_num IS NULL AND entity_slug IS NOT NULL`), and the v0.32.2
|
||||
// extract_facts cycle phase guard at `src/core/cycle/extract-facts.ts:83`
|
||||
// refuses to run reconciliation while any such rows exist. One
|
||||
// unknown bare entity reference silently broke autopilot extract_facts
|
||||
// forever.
|
||||
//
|
||||
// Fix: log loudly + append a structured entry to
|
||||
// `~/.gbrain/facts.dropped.jsonl` for operator recovery. The fact
|
||||
// text is preserved verbatim so a future `gbrain replay-dropped` tool
|
||||
// can re-process once the canonical entity page exists.
|
||||
for (const { f } of group) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts] dropping fact (no canonical for unprefixed slug=${slug}): "${f.fact.slice(0, 80)}${f.fact.length > 80 ? '…' : ''}". Logged to ~/.gbrain/facts.dropped.jsonl for recovery.`,
|
||||
);
|
||||
appendDroppedFactAudit({
|
||||
source_id: ctx.sourceId,
|
||||
phantom_slug: slug,
|
||||
reason: 'stub_guard_blocked',
|
||||
fact: f.fact,
|
||||
kind: f.kind ?? null,
|
||||
notability: f.notability ?? null,
|
||||
visibility,
|
||||
source: f.source,
|
||||
source_session: f.source_session ?? null,
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (result.legacyFallback) {
|
||||
// Defensive: writeFactsToFence sees localPath as null. We
|
||||
// checked above so this shouldn't fire — log loud + skip.
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* v0.34.5 — facts.dropped.jsonl operator-recoverable audit trail.
|
||||
*
|
||||
* When the stub-guard backstop fallback fires (an unprefixed bare-name
|
||||
* entity reference like "Zoolander" that couldn't resolve to any
|
||||
* canonical people/companies/deals/topics page), we used to insert a
|
||||
* DB-only fact row via `engine.insertFact` to avoid silent data loss.
|
||||
* That created rows shaped like pre-v0.32.2 legacy migration rows
|
||||
* (`row_num IS NULL AND entity_slug IS NOT NULL`), which the v0.32.2
|
||||
* extract_facts cycle phase guard at `src/core/cycle/extract-facts.ts`
|
||||
* REFUSES to reconcile around. Net: one unknown bare entity reference
|
||||
* silently broke the autopilot reconciliation pass forever (codex P1
|
||||
* caught this in /codex review post-implementation).
|
||||
*
|
||||
* Fix: don't insert. Log to stderr (operator visibility) AND append a
|
||||
* structured entry to `~/.gbrain/facts.dropped.jsonl` so a follow-up
|
||||
* tool can re-process the dropped facts once the user creates the
|
||||
* canonical entity pages. The fact text is preserved verbatim — no
|
||||
* lossy summarization — so recovery is lossless.
|
||||
*
|
||||
* Schema (per line, JSON object):
|
||||
* - ts: ISO 8601 timestamp
|
||||
* - source_id: brain source
|
||||
* - phantom_slug: the unresolved bare slug (e.g. 'zoolander')
|
||||
* - reason: 'stub_guard_blocked' (only reason today; field is open
|
||||
* for future operator-visibility surfaces)
|
||||
* - fact: the verbatim fact text
|
||||
* - kind: NewFact kind (fact / event / preference / commitment / belief)
|
||||
* - notability: high / medium / low
|
||||
* - visibility: private / world
|
||||
* - source: provenance string (e.g. 'mcp:put_page', 'sync:import')
|
||||
* - source_session: session id if present
|
||||
*
|
||||
* Best-effort: every write is wrapped in try/catch and never throws
|
||||
* back into the caller. The hot-path facts pipeline must not break
|
||||
* because the audit dir is read-only / disk full / etc.
|
||||
*/
|
||||
|
||||
import { appendFileSync, mkdirSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
import { gbrainPath } from '../config.ts';
|
||||
|
||||
export interface DroppedFactEntry {
|
||||
source_id: string;
|
||||
phantom_slug: string;
|
||||
reason: 'stub_guard_blocked';
|
||||
fact: string;
|
||||
kind: string | null;
|
||||
notability: 'high' | 'medium' | 'low' | null;
|
||||
visibility: 'private' | 'world';
|
||||
source: string;
|
||||
source_session: string | null;
|
||||
}
|
||||
|
||||
const AUDIT_PATH = (): string => gbrainPath('facts.dropped.jsonl');
|
||||
|
||||
export function appendDroppedFactAudit(entry: DroppedFactEntry): void {
|
||||
try {
|
||||
const path = AUDIT_PATH();
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
const line = JSON.stringify({
|
||||
ts: new Date().toISOString(),
|
||||
...entry,
|
||||
});
|
||||
appendFileSync(path, `${line}\n`, 'utf-8');
|
||||
} catch (err) {
|
||||
// Best-effort: log to stderr but never throw back into the facts
|
||||
// pipeline. The fact is already lost; the audit log is a recovery
|
||||
// aid, not a critical path.
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts.dropped] couldn't append audit entry: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -63,6 +63,15 @@ export interface FenceInputFact {
|
||||
/** Defaults to 1.0 when undefined (matches engine.insertFact behavior). */
|
||||
confidence?: number;
|
||||
validFrom?: Date;
|
||||
/**
|
||||
* Optional time-bound expiration for the fact (e.g. "valid through
|
||||
* 2026-08-31"). When set, the rendered fence row carries the
|
||||
* date so re-fence operations preserve the validity window
|
||||
* (codex round-10 P2 fix). Defaults to undefined for normal
|
||||
* extraction paths where the LLM never produces an explicit upper
|
||||
* bound.
|
||||
*/
|
||||
validUntil?: Date;
|
||||
embedding: Float32Array | null;
|
||||
sessionId: string | null;
|
||||
}
|
||||
@@ -76,6 +85,16 @@ export interface FenceWriteResult {
|
||||
legacyFallback?: true;
|
||||
/** True when fence parse-validate failed; rows were NOT inserted, .tmp quarantined. */
|
||||
fenceWriteFailed?: true;
|
||||
/**
|
||||
* True when the stub-creation guard refused to spawn a phantom entity
|
||||
* page for an unprefixed bare slug (e.g. `jared` with no `people/`
|
||||
* directory). Rows were NOT inserted; the caller is expected to route
|
||||
* the facts to the legacy DB-only path so they aren't silently dropped.
|
||||
*
|
||||
* This is the v0.34.5 fix for the entity-resolution bug where `"Jared"`
|
||||
* fell through resolution and produced a top-level `jared.md` stub.
|
||||
*/
|
||||
stubGuardBlocked?: true;
|
||||
}
|
||||
|
||||
const FAILURE_LOG_PATH = (): string => gbrainPath('facts.write_failures.jsonl');
|
||||
@@ -161,14 +180,101 @@ export async function writeFactsToFence(
|
||||
return withPageLock(
|
||||
target.slug,
|
||||
async () => {
|
||||
// 1. Read existing body or stub-create.
|
||||
// 1. Read existing body, materialize from DB, or stub-create.
|
||||
let body: string;
|
||||
if (existsSync(filePath)) {
|
||||
body = readFileSync(filePath, 'utf-8');
|
||||
// Codex round-24 + round-26 stub-guard: an existing phantom
|
||||
// .md file on disk should block fact appends so future facts
|
||||
// don't keep accumulating on a v0.34.5-era phantom. But the
|
||||
// check must also consult the DB row — if disk is stale-stub
|
||||
// while pages.compiled_truth has real content (e.g. a
|
||||
// subsequent put_page / MCP update never re-synced to disk),
|
||||
// the page is intentional and the fact must land.
|
||||
if (!target.slug.includes('/')) {
|
||||
const { isStubBody } = await import('../entities/resolve.ts');
|
||||
if (isStubBody(body)) {
|
||||
// Disk says stub. Check the DB body too before dropping.
|
||||
const dbPage = await engine.getPage(target.slug, {
|
||||
sourceId: target.sourceId,
|
||||
});
|
||||
const dbStub =
|
||||
dbPage === null ||
|
||||
(isStubBody(dbPage.compiled_truth ?? '') &&
|
||||
(dbPage.timeline ?? '').trim().length === 0);
|
||||
if (dbStub) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts] refusing to append to existing stub-shaped phantom slug=${target.slug} (file present, body stub-shape, DB also empty/stub). Routing to dropped-audit.`,
|
||||
);
|
||||
return { inserted: 0, ids: [], stubGuardBlocked: true };
|
||||
}
|
||||
// DB has real content; disk is stale. Materialize the DB
|
||||
// body over the stale stub so the fence write appends to
|
||||
// real content rather than a stub.
|
||||
const { serializeMarkdown } = await import('../markdown.ts');
|
||||
const tags = Array.isArray(dbPage.frontmatter?.tags)
|
||||
? (dbPage.frontmatter!.tags as string[])
|
||||
: [];
|
||||
body = serializeMarkdown(
|
||||
(dbPage.frontmatter as Record<string, unknown>) ?? {},
|
||||
dbPage.compiled_truth ?? '',
|
||||
dbPage.timeline ?? '',
|
||||
{ type: dbPage.type, title: dbPage.title, tags },
|
||||
);
|
||||
}
|
||||
// Body is non-stub (intentional bare page) — append below.
|
||||
}
|
||||
} else {
|
||||
// Stub-create the parent directory if it doesn't exist.
|
||||
// The stub-creation guard prevents v0.34.5-era phantom entity
|
||||
// pages (`jared.md`, `alice.md` at brain root) from being
|
||||
// spawned. But there's a benign case: a legitimate top-level
|
||||
// DB page (created via MCP put_page or importFromContent on a
|
||||
// source with local_path but never synced to disk). For that
|
||||
// case we materialize the DB body to disk BEFORE the fence
|
||||
// append, so the user's content survives intact.
|
||||
//
|
||||
// Codex round-17 P2 + round-18 P1 — distinguish:
|
||||
// - Real DB page with non-stub body → materialize + append.
|
||||
// - DB page with stub body OR no DB row at all → fire guard.
|
||||
const existingPage = await engine.getPage(target.slug, {
|
||||
sourceId: target.sourceId,
|
||||
});
|
||||
const { isStubBody } = await import('../entities/resolve.ts');
|
||||
const hasRealContent =
|
||||
existingPage !== null &&
|
||||
(!isStubBody(existingPage.compiled_truth ?? '') ||
|
||||
(existingPage.timeline ?? '').trim().length > 0);
|
||||
|
||||
if (!hasRealContent && !target.slug.includes('/')) {
|
||||
// No real DB body AND unprefixed slug — this would spawn a
|
||||
// phantom. Caller routes to the dropped-audit log.
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts] refusing to stub-create unprefixed entity page slug=${target.slug} — routing to legacy DB-only path. Provide a directory prefix (people/, companies/, etc.) to opt into fence writes.`,
|
||||
);
|
||||
return { inserted: 0, ids: [], stubGuardBlocked: true };
|
||||
}
|
||||
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
body = stubEntityPage(target.slug);
|
||||
if (existingPage && hasRealContent) {
|
||||
// Materialize the existing DB body to disk so the fence
|
||||
// write appends to the real content rather than a stub.
|
||||
const { serializeMarkdown } = await import('../markdown.ts');
|
||||
const tags = Array.isArray(existingPage.frontmatter?.tags)
|
||||
? (existingPage.frontmatter!.tags as string[])
|
||||
: [];
|
||||
body = serializeMarkdown(
|
||||
(existingPage.frontmatter as Record<string, unknown>) ?? {},
|
||||
existingPage.compiled_truth ?? '',
|
||||
existingPage.timeline ?? '',
|
||||
{ type: existingPage.type, title: existingPage.title, tags },
|
||||
);
|
||||
} else {
|
||||
// Prefixed slug with no DB row → legitimately stub-create
|
||||
// a new entity page (the path that always worked).
|
||||
body = stubEntityPage(target.slug);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Upsert each fact onto the fence in input order. row_num
|
||||
@@ -176,6 +282,9 @@ export async function writeFactsToFence(
|
||||
const assignedRowNums: number[] = [];
|
||||
for (const f of facts) {
|
||||
const validFromStr = (f.validFrom ?? new Date()).toISOString().slice(0, 10);
|
||||
const validUntilStr = f.validUntil
|
||||
? f.validUntil.toISOString().slice(0, 10)
|
||||
: undefined;
|
||||
const { body: updated, rowNum } = upsertFactRow(body, {
|
||||
claim: f.fact,
|
||||
kind: (f.kind ?? 'fact') as 'fact' | 'event' | 'preference' | 'commitment' | 'belief',
|
||||
@@ -183,7 +292,7 @@ export async function writeFactsToFence(
|
||||
visibility: f.visibility,
|
||||
notability: f.notability ?? 'medium',
|
||||
validFrom: validFromStr,
|
||||
validUntil: undefined,
|
||||
validUntil: validUntilStr,
|
||||
source: f.source,
|
||||
context: f.context ?? undefined,
|
||||
});
|
||||
|
||||
@@ -0,0 +1,899 @@
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import {
|
||||
resolveEntitySlug,
|
||||
slugify,
|
||||
getPrefixExpansionDirs,
|
||||
DEFAULT_PREFIX_EXPANSION_DIRS,
|
||||
} from '../src/core/entities/resolve.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { writeFactsToFence } from '../src/core/facts/fence-write.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
/**
|
||||
* v0.34.5 — entity resolution prefix expansion tests.
|
||||
*
|
||||
* Validates that bare first names resolve to existing pages via prefix
|
||||
* expansion, preventing phantom stub creation.
|
||||
*
|
||||
* Privacy: all seed data uses canonical placeholders per CLAUDE.md
|
||||
* "Privacy rule: scrub real names from public docs" — alice-example,
|
||||
* bob-example, charlie-example, dave-example, acme-example. The
|
||||
* bug being fixed is name-agnostic; slugify('Alice') exercises the
|
||||
* same path that slugify('Jared') did pre-fix.
|
||||
*
|
||||
* Coverage matrix per plan mossy-popping-crown.md D5:
|
||||
* - prefix expansion happy path (single + multi candidate)
|
||||
* - tiebreak via connection_count DESC, slug ASC
|
||||
* - companies/ prefix
|
||||
* - links contribution to connection_count (not just chunks)
|
||||
* - identical connection_count → slug ASC fallback
|
||||
* - source-id isolation
|
||||
* - config-driven dirs (entities.prefix_expansion_dirs)
|
||||
* - integration regression: stub-guard → backstop → DB row, no markdown file
|
||||
*/
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
|
||||
// Seed test pages. Two variants per first name so the tiebreak
|
||||
// path has real candidates to choose between.
|
||||
const pages = [
|
||||
{ slug: 'people/alice-example', title: 'Alice Example', type: 'person' },
|
||||
{ slug: 'people/charlie-example', title: 'Charlie Example', type: 'person' },
|
||||
{ slug: 'people/charlie-ross-example', title: 'Charlie Ross Example', type: 'person' },
|
||||
{ slug: 'people/bob-example', title: 'Bob Example', type: 'person' },
|
||||
{ slug: 'people/bob-bankman-example', title: 'Bob Bankman Example', type: 'person' },
|
||||
{ slug: 'people/dave-example', title: 'Dave Example', type: 'person' },
|
||||
{ slug: 'companies/acme-example', title: 'Acme Example', type: 'company' },
|
||||
{ slug: 'companies/acme-atlas-example', title: 'Acme Atlas Example', type: 'company' },
|
||||
// Links-only candidate (no chunks) — exercises the links contribution
|
||||
// to connection_count.
|
||||
{ slug: 'people/eve-example', title: 'Eve Example', type: 'person' },
|
||||
{ slug: 'people/eve-friend-example', title: 'Eve Friend Example', type: 'person' },
|
||||
// Tiebreak candidates with identical connection counts — exercises
|
||||
// the deterministic slug ASC secondary sort.
|
||||
{ slug: 'people/frank-aaa-example', title: 'Frank Aaa Example', type: 'person' },
|
||||
{ slug: 'people/frank-zzz-example', title: 'Frank Zzz Example', type: 'person' },
|
||||
// Custom-dir candidate for config-driven test (`funds`).
|
||||
{ slug: 'funds/founders-x-example', title: 'Founders X Example', type: 'concept' },
|
||||
// Exact-prefix-match coverage (no hyphen suffix): `companies/glob` and
|
||||
// `concepts/rag` are the canonical shape codex flagged the resolver
|
||||
// would miss in its post-D9 review pass.
|
||||
{ slug: 'companies/glob', title: 'Glob', type: 'company' },
|
||||
{ slug: 'concepts/rag', title: 'RAG', type: 'concept' },
|
||||
];
|
||||
|
||||
for (const p of pages) {
|
||||
await engine.putPage(p.slug, {
|
||||
type: p.type as any,
|
||||
title: p.title,
|
||||
compiled_truth: `# ${p.title}`,
|
||||
frontmatter: { type: p.type, title: p.title, slug: p.slug },
|
||||
}, { sourceId: 'default' });
|
||||
}
|
||||
|
||||
// Give alice-example 10 chunks — exercises the single-best-match path.
|
||||
await seedChunks(engine, 'people/alice-example', 10);
|
||||
// bob-example > bob-bankman-example — tiebreak via chunk count.
|
||||
await seedChunks(engine, 'people/bob-example', 20);
|
||||
await seedChunks(engine, 'people/bob-bankman-example', 3);
|
||||
// charlie-example > charlie-ross-example.
|
||||
await seedChunks(engine, 'people/charlie-example', 15);
|
||||
await seedChunks(engine, 'people/charlie-ross-example', 2);
|
||||
// acme-example > acme-atlas-example.
|
||||
await seedChunks(engine, 'companies/acme-example', 8);
|
||||
await seedChunks(engine, 'companies/acme-atlas-example', 1);
|
||||
// dave-example: single companion match, low chunks.
|
||||
await seedChunks(engine, 'people/dave-example', 4);
|
||||
|
||||
// Links-only contribution: eve-example wins via inbound links rather
|
||||
// than chunks. The UNIQUE constraint on (from_page_id, to_page_id,
|
||||
// link_type, link_source, origin_page_id) means each (from, to, type)
|
||||
// pair allows at most 4 distinct rows (link_source: markdown/frontmatter/
|
||||
// manual/NULL). Seed multiple from-pages so eve accumulates enough
|
||||
// inbound links to beat eve-friend-example deterministically.
|
||||
const eve = await pageId(engine, 'people/eve-example');
|
||||
const alice = await pageId(engine, 'people/alice-example');
|
||||
const charlie = await pageId(engine, 'people/charlie-example');
|
||||
const dave = await pageId(engine, 'people/dave-example');
|
||||
const eveFriend = await pageId(engine, 'people/eve-friend-example');
|
||||
if (eve && alice && charlie && dave) {
|
||||
const sources: Array<string | null> = ['markdown', 'frontmatter', 'manual', null];
|
||||
for (const from of [alice, charlie, dave]) {
|
||||
for (const ls of sources) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
|
||||
VALUES ($1, $2, 'mentions', $3)
|
||||
ON CONFLICT DO NOTHING`,
|
||||
[from, eve, ls],
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Eve-friend gets exactly one inbound link so it's a clear loser.
|
||||
if (eveFriend && alice) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
|
||||
VALUES ($1, $2, 'mentions', 'markdown')
|
||||
ON CONFLICT DO NOTHING`,
|
||||
[alice, eveFriend],
|
||||
);
|
||||
}
|
||||
|
||||
// Identical-count tiebreak: frank-aaa-example and frank-zzz-example
|
||||
// each get 5 chunks. The resolver should deterministically pick
|
||||
// frank-aaa-example (ASC).
|
||||
await seedChunks(engine, 'people/frank-aaa-example', 5);
|
||||
await seedChunks(engine, 'people/frank-zzz-example', 5);
|
||||
|
||||
// funds candidate gets 3 chunks so the config-driven test has data to match.
|
||||
await seedChunks(engine, 'funds/founders-x-example', 3);
|
||||
|
||||
// Source-isolation setup: create a sibling source and put `people/alice-example`
|
||||
// there too with WAY more chunks. The resolver in source 'default' must
|
||||
// still return `people/alice-example` (the default's row), NOT leak to
|
||||
// the high-chunk row in the other source.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other-src', 'other-src', '{}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
[],
|
||||
);
|
||||
await engine.putPage('people/alice-example', {
|
||||
type: 'person' as any,
|
||||
title: 'Alice (other source)',
|
||||
compiled_truth: `# Alice in other source`,
|
||||
frontmatter: { type: 'person', title: 'Alice (other source)', slug: 'people/alice-example' },
|
||||
}, { sourceId: 'other-src' });
|
||||
await seedChunks(engine, 'people/alice-example', 50, 'other-src');
|
||||
// Sanity: put a name only in other-src so it must NOT resolve in default.
|
||||
await engine.putPage('people/grace-other-example', {
|
||||
type: 'person' as any,
|
||||
title: 'Grace (other source only)',
|
||||
compiled_truth: `# Grace only in other source`,
|
||||
frontmatter: { type: 'person', title: 'Grace (other source only)', slug: 'people/grace-other-example' },
|
||||
}, { sourceId: 'other-src' });
|
||||
await seedChunks(engine, 'people/grace-other-example', 20, 'other-src');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function seedChunks(eng: PGLiteEngine, slug: string, count: number, sourceId = 'default'): Promise<void> {
|
||||
const rows = await eng.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM pages WHERE slug = $1 AND source_id = $2`,
|
||||
[slug, sourceId],
|
||||
);
|
||||
if (rows.length === 0) return;
|
||||
const pid = rows[0].id;
|
||||
for (let i = 0; i < count; i++) {
|
||||
await eng.executeRaw(
|
||||
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (page_id, chunk_index) DO NOTHING`,
|
||||
[pid, i, `Chunk ${i} about ${slug}`],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function pageId(eng: PGLiteEngine, slug: string, sourceId = 'default'): Promise<string | null> {
|
||||
const rows = await eng.executeRaw<{ id: string }>(
|
||||
`SELECT id FROM pages WHERE slug = $1 AND source_id = $2 LIMIT 1`,
|
||||
[slug, sourceId],
|
||||
);
|
||||
return rows[0]?.id ?? null;
|
||||
}
|
||||
|
||||
describe('resolveEntitySlug — prefix expansion (v0.34.5)', () => {
|
||||
it('fuzzy title match wins over prefix expansion when no bare slug exists (round-22 P2)', async () => {
|
||||
// Codex round-22 P2: when a bare token is the TITLE of one
|
||||
// entity (e.g. "Liz" on people/elizabeth-example) AND also a
|
||||
// slug prefix of another (people/liz-smith), the resolver
|
||||
// previously routed via prefix expansion → people/liz-smith
|
||||
// (wrong; user meant Liz=Elizabeth). Fix: bare-name prefix
|
||||
// expansion only short-circuits exact/fuzzy when there's a
|
||||
// STUB-shaped bare slug to override. With no bare slug at all,
|
||||
// fuzzy runs first.
|
||||
await engine.putPage(
|
||||
'people/elizabeth-fuzzy-example',
|
||||
{
|
||||
type: 'person' as any,
|
||||
title: 'Liz',
|
||||
compiled_truth: '# Liz (Elizabeth)',
|
||||
frontmatter: { type: 'person', title: 'Liz', slug: 'people/elizabeth-fuzzy-example' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await seedChunks(engine, 'people/elizabeth-fuzzy-example', 50);
|
||||
await engine.putPage(
|
||||
'people/liz-smith-example',
|
||||
{
|
||||
type: 'person' as any,
|
||||
title: 'Liz Smith',
|
||||
compiled_truth: '# Liz Smith',
|
||||
frontmatter: { type: 'person', title: 'Liz Smith', slug: 'people/liz-smith-example' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await seedChunks(engine, 'people/liz-smith-example', 5);
|
||||
|
||||
// "Liz" should fuzzy-match the title and return elizabeth, NOT
|
||||
// prefix-expand to liz-smith.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Liz');
|
||||
expect(result).toBe('people/elizabeth-fuzzy-example');
|
||||
});
|
||||
|
||||
it('prefers prefix expansion over an existing STUB-shaped unprefixed phantom (round-7 P2)', async () => {
|
||||
// Codex round-7 P2 #1: when both the canonical `people/alice-example`
|
||||
// AND an unprefixed STUB phantom `alice` exist, resolveEntitySlug
|
||||
// must return the canonical, NOT exact-match the phantom and keep
|
||||
// splitting facts onto it.
|
||||
await engine.putPage(
|
||||
'alice-phantom-test',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'alice-phantom-test',
|
||||
compiled_truth: '# alice-phantom-test',
|
||||
frontmatter: { type: 'concept', title: 'alice-phantom-test', slug: 'alice-phantom-test' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
// The seed `people/alice-example` already exists from beforeAll.
|
||||
// resolveEntitySlug('alice') must return the canonical, not the
|
||||
// bare slug.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'alice');
|
||||
expect(result).toBe('people/alice-example');
|
||||
});
|
||||
|
||||
it('returns a real bare page for capitalized input without fuzzy/prefix detours (round-25 P2)', async () => {
|
||||
// Codex round-25 P2: when pg_trgm is unavailable and the input
|
||||
// is a capitalized real-bare-page name (`Alice` with `alice` real
|
||||
// page), the old code fell through to fuzzy (null on no pg_trgm)
|
||||
// then catch-all prefix expansion → people/alice-* (wrong; user
|
||||
// meant the bare `alice` page). The fix returns the token NOW
|
||||
// when bareBody is real (non-stub).
|
||||
const realBody = '# Realbare\n\nIntentional top-level page with prose.';
|
||||
await engine.putPage(
|
||||
'realbare',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'Realbare',
|
||||
compiled_truth: realBody,
|
||||
frontmatter: { type: 'concept', title: 'Realbare', slug: 'realbare' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await engine.putPage(
|
||||
'people/realbare-other-example',
|
||||
{
|
||||
type: 'person' as any,
|
||||
title: 'Realbare Other Example',
|
||||
compiled_truth: '# Realbare Other',
|
||||
frontmatter: { type: 'person', title: 'Realbare Other Example', slug: 'people/realbare-other-example' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await seedChunks(engine, 'people/realbare-other-example', 50);
|
||||
|
||||
// Capitalized single-word bare name `Realbare`. isBareName→true,
|
||||
// slugify('Realbare')→'realbare'. The bare `realbare` page exists
|
||||
// and is real (non-stub). Must return 'realbare' from step 1
|
||||
// directly — without bouncing through fuzzy/catch-all prefix
|
||||
// expansion that could mis-route to people/realbare-*.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Realbare');
|
||||
expect(result).toBe('realbare');
|
||||
});
|
||||
|
||||
it('preserves a bare page whose content lives in the timeline column (round-18 P2)', async () => {
|
||||
// Codex round-18 P2: tryExactSlugBody must look at BOTH
|
||||
// compiled_truth AND timeline when deciding whether a bare slug
|
||||
// is real or stub-shaped. A real page with stubby
|
||||
// compiled_truth + populated pages.timeline column would
|
||||
// previously look stub-shaped and get overridden by prefix
|
||||
// expansion.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES ('default', 'tl-only-page', 'concept', 'markdown', 'TL Only Page', '# TL Only Page',
|
||||
'- 2026-04-01: First milestone\n- 2026-05-01: Second milestone',
|
||||
$1::jsonb, 'h', now())
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET timeline = EXCLUDED.timeline`,
|
||||
[JSON.stringify({ type: 'concept', title: 'TL Only Page', slug: 'tl-only-page' })],
|
||||
);
|
||||
await engine.putPage(
|
||||
'concepts/tl-only-page-canonical',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'TL Only Page Canonical',
|
||||
compiled_truth: '# canonical',
|
||||
frontmatter: { type: 'concept', title: 'TL Only Page Canonical', slug: 'concepts/tl-only-page-canonical' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
// The bare slug exists with a timeline-column body, so the
|
||||
// resolver must NOT override to the canonical.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'tl-only-page');
|
||||
expect(result).toBe('tl-only-page');
|
||||
});
|
||||
|
||||
it('preserves a TERSE real top-level page (one sentence) (round-11 P2)', async () => {
|
||||
// Codex round-11 P2: a 50-char threshold misclassifies terse but
|
||||
// intentional pages (e.g. `# RAG` + a one-sentence note). The
|
||||
// fix is threshold = 0 — only the literal stub shape (frontmatter
|
||||
// + H1 + maybe an empty fence) is a stub. Any user content beyond
|
||||
// that, however short, makes the page real.
|
||||
await engine.putPage(
|
||||
'rag-terse',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'RAG Terse',
|
||||
compiled_truth: '# RAG Terse\n\nLook it up.',
|
||||
frontmatter: { type: 'concept', title: 'RAG Terse', slug: 'rag-terse' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await engine.putPage(
|
||||
'concepts/rag-terse-example',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'RAG Terse Example',
|
||||
compiled_truth: '# RAG Terse Example',
|
||||
frontmatter: { type: 'concept', title: 'RAG Terse Example', slug: 'concepts/rag-terse-example' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
// The 10-char "Look it up." should be enough to keep the bare
|
||||
// page intact even with the canonical candidate present.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'rag-terse');
|
||||
expect(result).toBe('rag-terse');
|
||||
});
|
||||
|
||||
it('preserves exact bare-slug match for a REAL top-level page (round-8 P2 #2)', async () => {
|
||||
// Codex round-8 P2 #2: when a user has a legitimate top-level
|
||||
// `rag-real` page with real body content AND a `concepts/rag-real`
|
||||
// prefixed page, the bare `rag-real` must NOT be overridden by
|
||||
// prefix expansion. The bare page is intentional, not a phantom.
|
||||
const realBody = '# RAG Real\n\nThis is a real top-level page with intentional content. '.repeat(20);
|
||||
await engine.putPage(
|
||||
'rag-real',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'RAG Real',
|
||||
compiled_truth: realBody,
|
||||
frontmatter: { type: 'concept', title: 'RAG Real', slug: 'rag-real' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
await engine.putPage(
|
||||
'concepts/rag-real-example',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'RAG Real Example',
|
||||
compiled_truth: '# RAG Real Example',
|
||||
frontmatter: { type: 'concept', title: 'RAG Real Example', slug: 'concepts/rag-real-example' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
// resolveEntitySlug('rag-real') must NOT redirect to concepts/...
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'rag-real');
|
||||
expect(result).toBe('rag-real');
|
||||
});
|
||||
|
||||
it('resolves "Alice" to people/alice-example', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Alice');
|
||||
expect(result).toBe('people/alice-example');
|
||||
});
|
||||
|
||||
it('resolves "alice" (lowercase) to people/alice-example', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'alice');
|
||||
expect(result).toBe('people/alice-example');
|
||||
});
|
||||
|
||||
it('resolves "Charlie" to people/charlie-example (more connections)', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Charlie');
|
||||
expect(result).toBe('people/charlie-example');
|
||||
});
|
||||
|
||||
it('resolves "Bob" to people/bob-example (more connections)', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Bob');
|
||||
expect(result).toBe('people/bob-example');
|
||||
});
|
||||
|
||||
it('resolves "Dave" to people/dave-example (single match)', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Dave');
|
||||
expect(result).toBe('people/dave-example');
|
||||
});
|
||||
|
||||
it('falls through to slugify for unknown names', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Zyxwvut');
|
||||
expect(result).toBe('zyxwvut');
|
||||
});
|
||||
|
||||
it('exact match still works for fully-qualified slugs', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'people/alice-example');
|
||||
expect(result).toBe('people/alice-example');
|
||||
});
|
||||
|
||||
it('multi-word input does NOT trigger prefix expansion', async () => {
|
||||
// "Alice Example" should go through fuzzy match, not prefix expansion
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Alice Example');
|
||||
// Should resolve via fuzzy match to the same page
|
||||
expect(result).toContain('alice-example');
|
||||
});
|
||||
|
||||
it('hyphenated input does NOT trigger prefix expansion', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'alice-example');
|
||||
expect(result).toBe('people/alice-example');
|
||||
});
|
||||
|
||||
it('returns null for empty input', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', '');
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveEntitySlug — additional coverage (D5)', () => {
|
||||
it('expands bare "Acme" to companies/acme-example via the companies/ directory', async () => {
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Acme');
|
||||
expect(result).toBe('companies/acme-example');
|
||||
});
|
||||
|
||||
it('uses inbound + outbound link counts (not just chunks) for connection_count', async () => {
|
||||
// eve-example has many inbound links + zero chunks; eve-friend-example
|
||||
// has one inbound link + zero chunks. The winner is whichever the
|
||||
// connection_count expression scores higher, and that scoring HAS
|
||||
// to consider links or eve-example would tie eve-friend-example.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Eve');
|
||||
expect(result).toBe('people/eve-example');
|
||||
});
|
||||
|
||||
it('breaks identical-count ties deterministically via slug ASC', async () => {
|
||||
// frank-aaa-example and frank-zzz-example both have exactly 5
|
||||
// chunks and 0 links. Tiebreak is slug ASC → "aaa" wins.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Frank');
|
||||
expect(result).toBe('people/frank-aaa-example');
|
||||
});
|
||||
|
||||
it('matches exact prefix slug (no hyphen suffix) — companies/glob', async () => {
|
||||
// Codex review post-D9: prefix expansion previously only matched
|
||||
// `<dir>/<token>-%` and missed canonical slugs like `companies/glob`
|
||||
// or `people/alice`. The fix also matches `<dir>/<token>` exactly.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Glob');
|
||||
expect(result).toBe('companies/glob');
|
||||
});
|
||||
|
||||
it('resolves bare concept names via the default concepts/ directory', async () => {
|
||||
// Codex review post-D9: `concepts/` is documented everywhere and
|
||||
// the default `type: concept` home, but the original default dir
|
||||
// list omitted it.
|
||||
const result = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'RAG');
|
||||
expect(result).toBe('concepts/rag');
|
||||
});
|
||||
|
||||
it('scopes prefix expansion to the requested source_id', async () => {
|
||||
// people/alice-example exists in BOTH 'default' (10 chunks) and
|
||||
// 'other-src' (50 chunks). Resolving "Alice" in 'default' must
|
||||
// return default's row, not leak to other-src's higher-chunk row.
|
||||
// If the SQL ever drops `WHERE p.source_id = $1`, the high-chunk
|
||||
// other-src row would win the tiebreak — this test pins that.
|
||||
const defaultResult = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Alice');
|
||||
expect(defaultResult).toBe('people/alice-example');
|
||||
|
||||
// "Grace" only exists in other-src. Resolving in 'default' must
|
||||
// fall through to slugify, NOT find the other-src row.
|
||||
const defaultGrace = await resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Grace');
|
||||
expect(defaultGrace).toBe('grace');
|
||||
|
||||
// And in other-src, Grace resolves cleanly.
|
||||
const otherGrace = await resolveEntitySlug(engine as unknown as BrainEngine, 'other-src', 'Grace');
|
||||
expect(otherGrace).toBe('people/grace-other-example');
|
||||
});
|
||||
});
|
||||
|
||||
describe('getPrefixExpansionDirs — config-driven (D2)', () => {
|
||||
it('returns DEFAULT_PREFIX_EXPANSION_DIRS when no config exists', async () => {
|
||||
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-prefix-default-'));
|
||||
try {
|
||||
const dirs = await withEnv({ GBRAIN_HOME: tmpHome }, () => getPrefixExpansionDirs());
|
||||
expect(dirs).toEqual([...DEFAULT_PREFIX_EXPANSION_DIRS]);
|
||||
} finally {
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('default dir list includes concepts/ for concept entity resolution', async () => {
|
||||
expect(DEFAULT_PREFIX_EXPANSION_DIRS).toContain('concepts');
|
||||
expect([...DEFAULT_PREFIX_EXPANSION_DIRS]).toEqual([
|
||||
'people',
|
||||
'companies',
|
||||
'deals',
|
||||
'topics',
|
||||
'concepts',
|
||||
]);
|
||||
});
|
||||
|
||||
it('honors entities.prefix_expansion_dirs config override', async () => {
|
||||
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-prefix-custom-'));
|
||||
try {
|
||||
// GBRAIN_HOME=<tmp> → config lives at <tmp>/.gbrain/config.json.
|
||||
const cfgDir = join(tmpHome, '.gbrain');
|
||||
const { mkdirSync } = await import('node:fs');
|
||||
mkdirSync(cfgDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(cfgDir, 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
entities: { prefix_expansion_dirs: ['funds', 'people'] },
|
||||
}),
|
||||
'utf-8',
|
||||
);
|
||||
const dirs = await withEnv({ GBRAIN_HOME: tmpHome }, () => getPrefixExpansionDirs());
|
||||
expect(dirs).toEqual(['funds', 'people']);
|
||||
} finally {
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('falls back to defaults when config override is empty or malformed', async () => {
|
||||
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-prefix-bad-'));
|
||||
try {
|
||||
const cfgDir = join(tmpHome, '.gbrain');
|
||||
const { mkdirSync } = await import('node:fs');
|
||||
mkdirSync(cfgDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(cfgDir, 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
entities: { prefix_expansion_dirs: [123, '', null] }, // all bad
|
||||
}),
|
||||
'utf-8',
|
||||
);
|
||||
const dirs = await withEnv({ GBRAIN_HOME: tmpHome }, () => getPrefixExpansionDirs());
|
||||
expect(dirs).toEqual([...DEFAULT_PREFIX_EXPANSION_DIRS]);
|
||||
} finally {
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('resolves through a custom funds/ directory when configured', async () => {
|
||||
const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-prefix-funds-'));
|
||||
try {
|
||||
const cfgDir = join(tmpHome, '.gbrain');
|
||||
const { mkdirSync } = await import('node:fs');
|
||||
mkdirSync(cfgDir, { recursive: true });
|
||||
writeFileSync(
|
||||
join(cfgDir, 'config.json'),
|
||||
JSON.stringify({
|
||||
engine: 'pglite',
|
||||
entities: { prefix_expansion_dirs: ['funds'] },
|
||||
}),
|
||||
'utf-8',
|
||||
);
|
||||
const result = await withEnv({ GBRAIN_HOME: tmpHome }, () =>
|
||||
resolveEntitySlug(engine as unknown as BrainEngine, 'default', 'Founders'),
|
||||
);
|
||||
expect(result).toBe('funds/founders-x-example');
|
||||
} finally {
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('stub-guard + backstop integration (D5 regression — IRON RULE)', () => {
|
||||
it('writeFactsToFence refuses to stub-create an unprefixed slug; backstop drops fact + audits to JSONL', async () => {
|
||||
// Regression for the literal bug class this PR fixes. When
|
||||
// resolveEntitySlug falls through to slugify("Zander") → "zander"
|
||||
// (no matching page, no prefix), writeFactsToFence must NOT create
|
||||
// a phantom `zander.md`. Instead it returns stubGuardBlocked: true
|
||||
// so the caller (backstop) logs + audits + skips.
|
||||
//
|
||||
// The pre-codex-review version of this PR inserted these facts into
|
||||
// the DB to avoid silent data loss, but that produced
|
||||
// legacy-shape rows (row_num NULL + entity_slug NOT NULL) that
|
||||
// trip the v0.32.2 extract_facts reconciliation guard. The fix is
|
||||
// to NOT insert and instead write a structured audit entry to
|
||||
// ~/.gbrain/facts.dropped.jsonl for operator recovery.
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-stub-guard-'));
|
||||
const gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-home-stub-'));
|
||||
try {
|
||||
// Codex round-5 P2: writeFactsToFence acquires page-locks under
|
||||
// `gbrainPath('page-locks')`. Without GBRAIN_HOME isolation the
|
||||
// test writes lockfiles into the developer's real ~/.gbrain and
|
||||
// fails in hermetic / read-only-home runners with EPERM.
|
||||
const result = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
|
||||
writeFactsToFence(
|
||||
engine as unknown as BrainEngine,
|
||||
{ sourceId: 'default', localPath: brainDir, slug: 'zander' },
|
||||
[
|
||||
{
|
||||
fact: 'Zander likes integration tests.',
|
||||
kind: 'fact' as const,
|
||||
notability: 'medium' as const,
|
||||
source: 'test:regression',
|
||||
visibility: 'private' as const,
|
||||
embedding: null,
|
||||
sessionId: null,
|
||||
},
|
||||
],
|
||||
),
|
||||
);
|
||||
|
||||
// The guard fired.
|
||||
expect(result.stubGuardBlocked).toBe(true);
|
||||
expect(result.inserted).toBe(0);
|
||||
expect(result.ids).toEqual([]);
|
||||
|
||||
// No phantom markdown file was created at the brain root.
|
||||
expect(existsSync(join(brainDir, 'zander.md'))).toBe(false);
|
||||
|
||||
// Simulate the backstop's fallback under a controlled GBRAIN_HOME
|
||||
// so the JSONL lands in the tempdir and not the real brain.
|
||||
const { appendDroppedFactAudit } = await import('../src/core/facts/dropped-audit.ts');
|
||||
await withEnv({ GBRAIN_HOME: gbrainHome }, async () => {
|
||||
appendDroppedFactAudit({
|
||||
source_id: 'default',
|
||||
phantom_slug: 'zander',
|
||||
reason: 'stub_guard_blocked',
|
||||
fact: 'Zander likes integration tests.',
|
||||
kind: 'fact',
|
||||
notability: 'medium',
|
||||
visibility: 'private',
|
||||
source: 'test:regression',
|
||||
source_session: null,
|
||||
});
|
||||
});
|
||||
|
||||
// The fact is NOT in the DB — this is the v0.32.2 reconciliation
|
||||
// guard fix. Previous regression test asserted the OPPOSITE; this
|
||||
// updated assertion pins the codex P1 fix.
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM facts WHERE source_id = 'default' AND fact = 'Zander likes integration tests.'`,
|
||||
[],
|
||||
);
|
||||
expect(rows.length).toBe(0);
|
||||
|
||||
// No legacy-shape rows produced by this code path.
|
||||
const legacyRows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM facts WHERE row_num IS NULL AND entity_slug = 'zander'`,
|
||||
[],
|
||||
);
|
||||
expect(legacyRows[0].n).toBe(0);
|
||||
|
||||
// The audit log captured the fact for operator recovery.
|
||||
const auditPath = join(gbrainHome, '.gbrain', 'facts.dropped.jsonl');
|
||||
expect(existsSync(auditPath)).toBe(true);
|
||||
const auditLines = readFileSync(auditPath, 'utf-8').trim().split('\n');
|
||||
expect(auditLines.length).toBe(1);
|
||||
const entry = JSON.parse(auditLines[0]);
|
||||
expect(entry.phantom_slug).toBe('zander');
|
||||
expect(entry.reason).toBe('stub_guard_blocked');
|
||||
expect(entry.fact).toBe('Zander likes integration tests.');
|
||||
expect(entry.kind).toBe('fact');
|
||||
expect(entry.notability).toBe('medium');
|
||||
|
||||
// And no markdown file.
|
||||
expect(existsSync(join(brainDir, 'zander.md'))).toBe(false);
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('writeFactsToFence appends to a real DB page even when the .md file is stale-stub (round-26 P2)', async () => {
|
||||
// Codex round-26 P2: when the .md file on disk is a stale stub
|
||||
// but pages.compiled_truth has real content (e.g. subsequent
|
||||
// put_page / MCP write never re-synced to disk), facts must
|
||||
// land on the real page rather than dropping to the audit log.
|
||||
const realBody = '# Real Stale-Disk Page\n\nIntentional real DB content that the disk file lost.';
|
||||
await engine.putPage(
|
||||
'stale-disk-real',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'Real Stale-Disk Page',
|
||||
compiled_truth: realBody,
|
||||
frontmatter: { type: 'concept', title: 'Real Stale-Disk Page', slug: 'stale-disk-real' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-stale-disk-'));
|
||||
const gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-stale-disk-home-'));
|
||||
try {
|
||||
// Stub-shaped .md on disk (out of sync with DB).
|
||||
writeFileSync(
|
||||
join(brainDir, 'stale-disk-real.md'),
|
||||
'---\ntype: concept\ntitle: Stale\nslug: stale-disk-real\n---\n\n# Stale\n',
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
const result = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
|
||||
writeFactsToFence(
|
||||
engine as unknown as BrainEngine,
|
||||
{ sourceId: 'default', localPath: brainDir, slug: 'stale-disk-real' },
|
||||
[
|
||||
{
|
||||
fact: 'A new fact for the stale-disk real page.',
|
||||
kind: 'fact' as const,
|
||||
notability: 'medium' as const,
|
||||
source: 'test:regression',
|
||||
visibility: 'private' as const,
|
||||
embedding: null,
|
||||
sessionId: null,
|
||||
},
|
||||
],
|
||||
),
|
||||
);
|
||||
|
||||
// No stub-guard fire; fact landed.
|
||||
expect(result.stubGuardBlocked).toBeUndefined();
|
||||
expect(result.inserted).toBe(1);
|
||||
|
||||
// Markdown reconciled to DB body + appended fact.
|
||||
const onDisk = readFileSync(join(brainDir, 'stale-disk-real.md'), 'utf-8');
|
||||
expect(onDisk).toContain('Intentional real DB content');
|
||||
expect(onDisk).toContain('A new fact for the stale-disk real page.');
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('writeFactsToFence blocks appends to an existing stub-shaped phantom file (round-24 P2)', async () => {
|
||||
// Codex round-24 P2: a pre-v0.34.5 phantom file on disk would
|
||||
// previously slip past the stub-guard because the guard only
|
||||
// fired on missing files. New facts kept appending to the
|
||||
// phantom, growing the split.
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-existing-phantom-'));
|
||||
const gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-existing-phantom-home-'));
|
||||
try {
|
||||
// Seed a stub-shaped phantom file on disk (no machine fence yet).
|
||||
const phantomFile = join(brainDir, 'zoltan-phantom.md');
|
||||
writeFileSync(
|
||||
phantomFile,
|
||||
'---\ntype: person\ntitle: zoltan-phantom\nslug: zoltan-phantom\n---\n\n# zoltan-phantom\n',
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
const result = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
|
||||
writeFactsToFence(
|
||||
engine as unknown as BrainEngine,
|
||||
{ sourceId: 'default', localPath: brainDir, slug: 'zoltan-phantom' },
|
||||
[
|
||||
{
|
||||
fact: 'A fact about Zoltan.',
|
||||
kind: 'fact' as const,
|
||||
notability: 'medium' as const,
|
||||
source: 'test:regression',
|
||||
visibility: 'private' as const,
|
||||
embedding: null,
|
||||
sessionId: null,
|
||||
},
|
||||
],
|
||||
),
|
||||
);
|
||||
|
||||
// Stub-guard fired, no insert.
|
||||
expect(result.stubGuardBlocked).toBe(true);
|
||||
expect(result.inserted).toBe(0);
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('writeFactsToFence materializes a real DB-only unprefixed page instead of firing stub-guard (round-18 P1)', async () => {
|
||||
// Codex round-18 P1: when an unprefixed slug has a real DB body
|
||||
// (put_page MCP created it on a source with local_path but never
|
||||
// synced to disk), writeFactsToFence must NOT fire the stub-
|
||||
// guard and drop the fact. Instead it should materialize the DB
|
||||
// body to disk, then append the fence — preserving both the
|
||||
// existing page content AND the new fact.
|
||||
const realBody = '# Real Bare Page\n\nThis is real content that must be preserved.';
|
||||
await engine.putPage(
|
||||
'real-bare-page',
|
||||
{
|
||||
type: 'concept' as any,
|
||||
title: 'Real Bare Page',
|
||||
compiled_truth: realBody,
|
||||
frontmatter: { type: 'concept', title: 'Real Bare Page', slug: 'real-bare-page' },
|
||||
},
|
||||
{ sourceId: 'default' },
|
||||
);
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-stub-guard-real-'));
|
||||
const gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-stub-guard-real-home-'));
|
||||
try {
|
||||
// The bare-slug .md file does NOT exist on disk pre-call.
|
||||
expect(existsSync(join(brainDir, 'real-bare-page.md'))).toBe(false);
|
||||
|
||||
const result = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
|
||||
writeFactsToFence(
|
||||
engine as unknown as BrainEngine,
|
||||
{ sourceId: 'default', localPath: brainDir, slug: 'real-bare-page' },
|
||||
[
|
||||
{
|
||||
fact: 'A fact about the real bare page.',
|
||||
kind: 'fact' as const,
|
||||
notability: 'medium' as const,
|
||||
source: 'test:regression',
|
||||
visibility: 'private' as const,
|
||||
embedding: null,
|
||||
sessionId: null,
|
||||
},
|
||||
],
|
||||
),
|
||||
);
|
||||
|
||||
// No stub-guard fire; fact inserted.
|
||||
expect(result.stubGuardBlocked).toBeUndefined();
|
||||
expect(result.inserted).toBe(1);
|
||||
|
||||
// Markdown file now exists at the bare path WITH the real body + the fence.
|
||||
const filePath = join(brainDir, 'real-bare-page.md');
|
||||
expect(existsSync(filePath)).toBe(true);
|
||||
const onDisk = readFileSync(filePath, 'utf-8');
|
||||
expect(onDisk).toContain('This is real content that must be preserved.');
|
||||
expect(onDisk).toContain('A fact about the real bare page.');
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('writeFactsToFence DOES create the page when the slug has a directory prefix', async () => {
|
||||
// Inverse coverage: confirm the guard ONLY fires on unprefixed slugs.
|
||||
// A `people/yvonne-example` write should land normally.
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-stub-guard-ok-'));
|
||||
const gbrainHome = mkdtempSync(join(tmpdir(), 'gbrain-stub-guard-ok-home-'));
|
||||
try {
|
||||
// GBRAIN_HOME isolation per codex round-5 P2 (page-locks).
|
||||
const result = await withEnv({ GBRAIN_HOME: gbrainHome }, () =>
|
||||
writeFactsToFence(
|
||||
engine as unknown as BrainEngine,
|
||||
{ sourceId: 'default', localPath: brainDir, slug: 'people/yvonne-example' },
|
||||
[
|
||||
{
|
||||
fact: 'Yvonne likes prefixed slugs.',
|
||||
kind: 'fact' as const,
|
||||
notability: 'medium' as const,
|
||||
source: 'test:regression',
|
||||
visibility: 'private' as const,
|
||||
embedding: null,
|
||||
sessionId: null,
|
||||
},
|
||||
],
|
||||
),
|
||||
);
|
||||
|
||||
// No guard fire; row inserted.
|
||||
expect(result.stubGuardBlocked).toBeUndefined();
|
||||
expect(result.inserted).toBe(1);
|
||||
expect(result.ids.length).toBe(1);
|
||||
|
||||
// Markdown file exists at the prefixed path.
|
||||
expect(existsSync(join(brainDir, 'people/yvonne-example.md'))).toBe(true);
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
rmSync(gbrainHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('slugify', () => {
|
||||
it('lowercases and hyphenates', () => {
|
||||
expect(slugify('Alice Example')).toBe('alice-example');
|
||||
});
|
||||
|
||||
it('handles single word', () => {
|
||||
expect(slugify('Alice')).toBe('alice');
|
||||
});
|
||||
|
||||
it('strips accents', () => {
|
||||
expect(slugify('José García')).toBe('jose-garcia');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user