Compare commits

...
Author SHA1 Message Date
Garry TanandClaude Fable 5 3cfd0c4b42 fix(search): keep semantic cache alive for the query op (expandFn folded into knobs hash, not unsafe)
isSemanticCacheRequestSafe listed expandFn as cache-unsafe, but
operations.ts passes expandFn on every default `query` op call
(expand = p.expand !== false), so the semantic cache was silently
disabled for the flagship op (permanent 'disabled' status, 0% hit
rate).

expandFn's result-shaping effect is exactly the expansion arm, so fold
EFFECTIVE expansion into the cache-key resolver instead:
hybridSearchCached now threads `expandFn ? expansion : false` into
resolvedForCache.perCall (the inner search can only expand when
expansionAllowed AND expandFn is wired), and expandFn leaves the
unsafe list. No-expandFn callers under an expansion-on bundle key as
expansion=false, matching what actually ran — the cross-serve gap the
unsafe listing was defending against stays closed.

Pinned by a query-op-shaped miss→hit roundtrip in
test/search/hybrid-reranker-integration.serial.test.ts (verified
failing with 'disabled' before this commit) and a unit case in
test/cache-scope-key.test.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 11:32:50 -07:00
Garry TanandClaude Fable 5 1f10561eb1 test(search): update buildVisibilityClause verbatim expectation for hardened quarantine fragment
The PR hardened quarantineFilterFragment with a jsonb_typeof object guard
but missed the hardcoded clause string in sql-ranking.test.ts (and the
matching doc comment example). CI shard 4 caught it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 10:47:18 -07:00
3381dd7658 fix(search): preserve email citation metadata across result paths (takeover of #2875)
Salvage of PR #2875 (which subsumes the base projection from #2873),
rebased onto current master with the release bookkeeping (VERSION /
package.json / CHANGELOG) dropped, plus one hot-path hardening fix.

Salvaged (verified on this head):
- project trusted message_id / thread_id / Message-ID-gated source_subject
  through keyword, chunk-keyword, CJK, and vector paths in BOTH engines
- preserve the citation DTO through alias injection, relational
  recall/fanout, two-pass hydration, vector fusion/reranking, and
  semantic-cache hits
- raw source_subject is never trusted; only allowlisted `subject` may
  supply it, and only when a nonblank Message-ID proves the page is an
  email; malformed/non-object frontmatter fails closed (no double-decode)
- source visibility / quarantine / deletion rechecked across indirect
  retrieval paths (alias hop, relational hydrate, two-pass expansion,
  graph walk, cache-hit gate)
- typed JSON cache scope keys (scalar/set/all) — injective encoding, no
  forged-key collisions; store-side write gate skips writeback when the
  page-generation clock advanced during the producing search
- KNOBS_HASH_VERSION 12 -> 13 so pre-projection cached DTOs miss instead
  of replaying the old shape

Fixed on top of the original head (the flagged hot-path defect):
- cacheScopeKey's forged-id rejection was evaluated inline at the cache
  lookup/store call sites inside hybridSearchCached, outside any catch —
  an invalid scope id broke the whole search instead of skipping the
  cache. The key is now computed once, fail-open: invalid scope =>
  cache skipped, search unaffected. Pinned by
  test/search/hybrid-cache-scope-failopen.serial.test.ts (fails on the
  original head, passes here).

Verified on this exact head: typecheck clean; 379 touched unit tests,
3 serial files (one process each), pglite cache-gate/source-isolation
e2e, and real-PostgreSQL engine-parity (26 pass) + source-routing —
all green. jsonb-pattern/params, key-files-current-state,
test-isolation, progress-to-stdout guards clean.

Closes #2962

Co-authored-by: amtagrwl <amtagrwl@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 15:07:56 -07:00
36 changed files with 1675 additions and 157 deletions
+1 -1
View File
@@ -42,7 +42,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/reindex-search-vector.ts``gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`.
- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path.
- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`.
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToSearchResult` projects email `message_id` / `thread_id` metadata and exposes `source_subject` only when a non-empty Message-ID proves the page is an email, so generated page titles never become authoritative email subjects. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
- `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise<boolean>``true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` (#1972) — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the #2084 contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack.
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`). Copies the complete source catalog FIRST (`copyMigrationSources` — every `sources` row incl. archived rows and sync/routing metadata, `ON CONFLICT (id) DO UPDATE`, `default` ordered first) so every page write has a valid `pages.source_id` FK parent and the target preserves per-source behavior; pages copy afterward, tracked in the resume manifest by composite `(source_id, slug)` key. The resume manifest is target-aware: `migrationTargetId(config)` hashes `(engine, locator)` (`database_url` for Postgres, resolved `database_path` for PGLite) and `manifestMatchesTarget` requires `schema_version === 2` plus a matching `target_id` — a legacy engine-only manifest, or one from a DIFFERENT target of the same engine kind, starts fresh instead of skipping "completed" pages the new target never received. Pinned by `test/migrate-engine-resume.test.ts` (manifest identity) + `test/e2e/migrate-engine-sources-postgres.test.ts` (source catalog lands before overlapping-slug pages, PGLite → real Postgres).
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`.
+37 -8
View File
@@ -1636,6 +1636,10 @@ export class PGLiteEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
@@ -1880,6 +1884,10 @@ export class PGLiteEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
${scoreExpr} AS score,
CASE WHEN p.updated_at < (
@@ -1905,6 +1913,10 @@ export class PGLiteEngine implements BrainEngine {
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
${scoreExpr} AS score,
CASE WHEN p.updated_at < (
@@ -2001,6 +2013,10 @@ export class PGLiteEngine implements BrainEngine {
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
@@ -2113,6 +2129,10 @@ export class PGLiteEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
1 - (cc.${col} <=> ${castSql}) AS raw_score
FROM content_chunks cc
@@ -2135,6 +2155,7 @@ export class PGLiteEngine implements BrainEngine {
SELECT
bpp.slug, bpp.page_id, bpp.title, bpp.type, bpp.source_id,
bpp.effective_date, bpp.effective_date_source,
bpp.message_id, bpp.thread_id, bpp.source_subject,
bpp.chunk_id, bpp.chunk_index, bpp.chunk_text, bpp.chunk_source,
bpp.score,
CASE WHEN bpp.updated_at < (
@@ -3186,14 +3207,17 @@ export class PGLiteEngine implements BrainEngine {
typeFilter = `AND l.link_type = ANY($${params.length}::text[])`;
}
const mentionsFilter = opts?.includeMentions ? '' : `AND l.link_source IS DISTINCT FROM 'mentions'`;
const seedVisibility = buildVisibilityClause('p', 's');
const stepVisibility = buildVisibilityClause('p2', 's2');
const recurStep =
direction === 'out'
? `JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id`
? `JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id JOIN sources s2 ON s2.id = p2.source_id`
: direction === 'in'
? `JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id`
? `JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id JOIN sources s2 ON s2.id = p2.source_id`
: `JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`;
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
JOIN sources s2 ON s2.id = p2.source_id`;
const sql = `
WITH RECURSIVE walk AS (
@@ -3201,7 +3225,8 @@ export class PGLiteEngine implements BrainEngine {
ARRAY[p.id] AS visited, ARRAY[p.slug] AS path,
p.source_id AS seed_source, NULL::text AS last_link_type
FROM pages p
WHERE p.slug = ANY($1::text[]) ${seedScope} AND p.deleted_at IS NULL
JOIN sources s ON s.id = p.source_id
WHERE p.slug = ANY($1::text[]) ${seedScope} ${seedVisibility}
UNION ALL
SELECT p2.id, p2.slug, p2.source_id, w.depth + 1,
w.visited || p2.id, w.path || p2.slug,
@@ -3211,7 +3236,7 @@ export class PGLiteEngine implements BrainEngine {
WHERE w.depth < $2
AND NOT (p2.id = ANY(w.visited))
AND p2.source_id = w.seed_source
AND p2.deleted_at IS NULL
${stepVisibility}
${mentionsFilter}
${typeFilter}
)
@@ -5385,13 +5410,17 @@ export class PGLiteEngine implements BrainEngine {
: opts?.sourceId
? [opts.sourceId]
: null;
let q = `SELECT alias_norm, slug, source_id FROM page_aliases WHERE alias_norm = ANY($1::text[])`;
let q = `SELECT pa.alias_norm, pa.slug, pa.source_id
FROM page_aliases pa
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
JOIN sources s ON s.id = p.source_id
WHERE pa.alias_norm = ANY($1::text[]) ${buildVisibilityClause('p', 's')}`;
const params: unknown[] = [aliasNorms];
if (sources) {
params.push(sources);
q += ` AND source_id = ANY($2::text[])`;
q += ` AND pa.source_id = ANY($2::text[])`;
}
q += ` ORDER BY alias_norm, source_id, slug`;
q += ` ORDER BY pa.alias_norm, pa.source_id, pa.slug`;
const { rows } = await this.db.query(q, params);
for (const r of rows as Array<{ alias_norm: string; slug: string; source_id: string }>) {
const list = out.get(r.alias_norm) ?? [];
+39 -14
View File
@@ -1767,6 +1767,10 @@ export class PostgresEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score
FROM content_chunks cc
@@ -1794,6 +1798,7 @@ export class PostgresEngine implements BrainEngine {
${buildBestPerPagePoolCte('ranked_chunks')}
SELECT slug, page_id, title, type, source_id,
effective_date, effective_date_source,
message_id, thread_id, source_subject,
chunk_id, chunk_index, chunk_text, chunk_source, score,
false AS stale
FROM best_per_page
@@ -2065,6 +2070,10 @@ export class PostgresEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
false AS stale
@@ -2217,6 +2226,10 @@ export class PostgresEngine implements BrainEngine {
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
1 - (cc.${col} <=> ${castSql}) AS raw_score
FROM content_chunks cc
@@ -2251,6 +2264,7 @@ export class PostgresEngine implements BrainEngine {
SELECT
slug, page_id, title, type, source_id,
effective_date, effective_date_source,
message_id, thread_id, source_subject,
chunk_id, chunk_index, chunk_text, chunk_source,
score,
false AS stale
@@ -3357,15 +3371,18 @@ export class PostgresEngine implements BrainEngine {
const mentionsFilter = opts?.includeMentions
? sql``
: sql`AND l.link_source IS DISTINCT FROM 'mentions'`;
const seedVisibility = sql.unsafe(buildVisibilityClause('p', 's'));
const stepVisibility = sql.unsafe(buildVisibilityClause('p2', 's2'));
// Recursive step join differs by direction; everything else is shared.
const recurStep =
direction === 'out'
? sql`JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id`
? sql`JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id JOIN sources s2 ON s2.id = p2.source_id`
: direction === 'in'
? sql`JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id`
? sql`JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id JOIN sources s2 ON s2.id = p2.source_id`
: sql`JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`;
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
JOIN sources s2 ON s2.id = p2.source_id`;
const rows = await sql`
WITH RECURSIVE walk AS (
@@ -3373,7 +3390,8 @@ export class PostgresEngine implements BrainEngine {
ARRAY[p.id] AS visited, ARRAY[p.slug] AS path,
p.source_id AS seed_source, NULL::text AS last_link_type
FROM pages p
WHERE p.slug = ANY(${seeds}::text[]) ${seedScope} AND p.deleted_at IS NULL
JOIN sources s ON s.id = p.source_id
WHERE p.slug = ANY(${seeds}::text[]) ${seedScope} ${seedVisibility}
UNION ALL
SELECT p2.id, p2.slug, p2.source_id, w.depth + 1,
w.visited || p2.id, w.path || p2.slug,
@@ -3383,7 +3401,7 @@ export class PostgresEngine implements BrainEngine {
WHERE w.depth < ${depth}
AND NOT (p2.id = ANY(w.visited))
AND p2.source_id = w.seed_source
AND p2.deleted_at IS NULL
${stepVisibility}
${mentionsFilter}
${typeFilter}
)
@@ -5492,18 +5510,25 @@ export class PostgresEngine implements BrainEngine {
: opts?.sourceId
? [opts.sourceId]
: null;
const visibility = sql.unsafe(buildVisibilityClause('p', 's'));
const rows = sources
? await sql`
SELECT alias_norm, slug, source_id
FROM page_aliases
WHERE alias_norm = ANY(${aliasNorms}::text[])
AND source_id = ANY(${sources}::text[])
ORDER BY alias_norm, source_id, slug`
SELECT pa.alias_norm, pa.slug, pa.source_id
FROM page_aliases pa
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
JOIN sources s ON s.id = p.source_id
WHERE pa.alias_norm = ANY(${aliasNorms}::text[])
AND pa.source_id = ANY(${sources}::text[])
${visibility}
ORDER BY pa.alias_norm, pa.source_id, pa.slug`
: await sql`
SELECT alias_norm, slug, source_id
FROM page_aliases
WHERE alias_norm = ANY(${aliasNorms}::text[])
ORDER BY alias_norm, source_id, slug`;
SELECT pa.alias_norm, pa.slug, pa.source_id
FROM page_aliases pa
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
JOIN sources s ON s.id = p.source_id
WHERE pa.alias_norm = ANY(${aliasNorms}::text[])
${visibility}
ORDER BY pa.alias_norm, pa.source_id, pa.slug`;
for (const r of rows) {
const a = r.alias_norm as string;
const list = out.get(a) ?? [];
+4 -1
View File
@@ -45,7 +45,10 @@ export const QUARANTINE_KEY = 'quarantine';
* JSONB `?` existence, negated so we KEEP rows WITHOUT the marker.
* `pageAlias` is engine-supplied (never user input), so no escaping needed. */
export function quarantineFilterFragment(pageAlias: string): string {
return `NOT (COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ? '${QUARANTINE_KEY}')`;
const frontmatter = `COALESCE(${pageAlias}.frontmatter, '{}'::jsonb)`;
// Fail closed on legacy/double-encoded JSONB. A top-level string can hide a
// quarantine marker from the JSONB `?` operator and must not be searchable.
return `(jsonb_typeof(${frontmatter}) = 'object' AND NOT (${frontmatter} ? '${QUARANTINE_KEY}'))`;
}
/** The `p`-aliased instance — the common case (all 6 search call sites alias
+86 -11
View File
@@ -14,6 +14,8 @@ import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts';
import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
import { embed, embedQuery } from '../embedding.ts';
import { registerBackgroundWorkDrainer } from '../background-work.ts';
import { projectEmailCitationFrontmatter } from '../utils.ts';
import { assertValidSourceId } from '../source-id.ts';
import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
import { resolveHardExcludes } from './source-boost.ts';
import {
@@ -45,6 +47,7 @@ import {
SemanticQueryCache,
loadCacheConfig,
} from './query-cache.ts';
import { readPageGenerationClock } from './query-cache-gate.ts';
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
@@ -696,6 +699,7 @@ export async function applyAliasHop(
score: injectScore,
base_score: injectScore,
alias_hit: true,
...projectEmailCitationFrontmatter(page.frontmatter),
} as SearchResult);
}
out.sort((a, b) => b.score - a.score);
@@ -1469,6 +1473,7 @@ export async function hybridSearch(
walkDepth,
nearSymbol: opts?.nearSymbol,
sourceId: opts?.sourceId,
sourceIds: opts?.sourceIds,
});
// Resolve new chunk IDs (not already in fused) into full rows.
const existingIds = new Set(fused.map(r => r.chunk_id));
@@ -1476,7 +1481,10 @@ export async function hybridSearch(
.filter(e => !existingIds.has(e.chunk_id))
.map(e => e.chunk_id);
if (newIds.length > 0) {
const hydrated = await hydrateChunks(engine, newIds);
const hydrated = await hydrateChunks(engine, newIds, {
sourceId: opts?.sourceId,
sourceIds: opts?.sourceIds,
});
const scoreById = new Map(expanded.map(e => [e.chunk_id, e.score]));
for (const r of hydrated) {
r.score = scoreById.get(r.chunk_id) ?? 0.01;
@@ -1651,7 +1659,18 @@ export async function hybridSearchCached(
perCall: {
cache_enabled: opts?.useCache,
tokenBudget: opts?.tokenBudget,
expansion: opts?.expansion,
// Fold EFFECTIVE expansion into the cache key: the inner hybridSearch
// only expands when the resolved knob is on AND an expandFn is wired
// in (`expansionAllowed && opts?.expandFn`). Without an expandFn the
// knob can never fire, so force `false` here — otherwise a
// no-expandFn caller under a tokenmax bundle would share a cache row
// with the expanded results of an expandFn caller (and vice versa).
// This is what makes `expandFn` cache-SAFE (it is deliberately NOT in
// isSemanticCacheRequestSafe's unsafe list): its result-shaping effect
// is fully expressed by the expansion bit of the knobs hash. The
// production `query` op always passes expandFn, so listing it unsafe
// would silently disable the semantic cache for the flagship op.
expansion: opts?.expandFn ? opts?.expansion : false,
intentWeighting: opts?.intentWeighting,
searchLimit: opts?.limit,
// v0.35.6.0 — floor-ratio threaded through cache resolver too so
@@ -1728,11 +1747,23 @@ export async function hybridSearchCached(
opts?.adaptiveReturn,
cfgCached as unknown as Record<string, unknown> | null,
);
// Compute the scope key ONCE, fail-open. `cacheScopeKey` throws on a
// malformed source id (forged encodings, invalid charset) — that rejection
// is correct for direct callers, but inside the search hot path an invalid
// scope must degrade to "skip the cache", never break the search itself.
let cacheScope: string | null = null;
try {
cacheScope = cacheScopeKey(opts);
} catch {
// invalid scope id — cache lookup/store skipped below
}
const skipCache =
!cache.isEnabled() ||
(opts?.walkDepth ?? 0) > 0 ||
Boolean(opts?.nearSymbol) ||
isNonDefaultColumn ||
!isSemanticCacheRequestSafe(opts) ||
cacheScope === null ||
adaptiveReturnOn;
let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss';
@@ -1777,7 +1808,7 @@ export async function hybridSearchCached(
}
if (!skipCache && queryEmbedding && cacheStatus !== 'disabled') {
const hit = await cache.lookup(queryEmbedding, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash });
const hit = await cache.lookup(queryEmbedding, { sourceId: cacheScope!, knobsHash: cacheKnobsHash });
if (hit.hit && hit.results) {
cacheStatus = 'hit';
cacheSimilarity = hit.similarity;
@@ -1851,6 +1882,10 @@ export async function hybridSearchCached(
// we use a single-element box to keep the type stable.
const innerMetaBox: { current: HybridSearchMeta | null } = { current: null };
const userOnMeta = opts?.onMeta;
const maxGenerationAtSearchStart =
!skipCache && cacheStatus === 'miss' && queryEmbedding
? await readPageGenerationClock(engine)
: null;
const results = await hybridSearch(engine, query, {
...opts,
// v0.42.20.0 (Fix 3) — share the query-embed deadline so the inner embed
@@ -1909,11 +1944,16 @@ export async function hybridSearchCached(
cacheStatus === 'miss' &&
queryEmbedding &&
results.length > 0 &&
(innerMeta?.vector_enabled ?? false)
(innerMeta?.vector_enabled ?? false) &&
maxGenerationAtSearchStart !== null
) {
trackCacheWrite(
cache
.store(query, queryEmbedding, results, finalMeta, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash })
.store(query, queryEmbedding, results, finalMeta, {
sourceId: cacheScope!,
knobsHash: cacheKnobsHash,
maxGenerationAtSearchStart,
})
.catch(() => { /* swallow */ }),
);
}
@@ -1945,16 +1985,51 @@ function rrfKey(r: SearchResult): string {
* cache only saw scalar `sourceId`; a federated query fell through to
* `'default'` and could cross-serve an unrelated scope.
*
* - federated (sourceIds set) → `__set__:` + sorted, comma-joined ids
* (order-independent; two different source-sets get distinct keys)
* - scalar sourceId → the id itself (single-source unchanged)
* - unscoped → `'default'` (single-source brains unchanged)
* Keys are typed JSON tuples so arbitrary scalar text cannot collide with a
* federated encoding. Every source id is validated before it reaches cache
* lookup/store; this keeps internal/trusted callers on the same boundary as
* registered sources.
*/
export function cacheScopeKey(opts?: { sourceId?: string; sourceIds?: string[] }): string {
if (opts?.sourceIds && opts.sourceIds.length > 0) {
return '__set__:' + [...opts.sourceIds].sort().join(',');
const ids = [...new Set(opts.sourceIds)].sort();
for (const id of ids) assertValidSourceId(id);
return JSON.stringify(['set', ...ids]);
}
return opts?.sourceId ?? 'default';
if (opts?.sourceId !== undefined) {
assertValidSourceId(opts.sourceId);
return JSON.stringify(['scalar', opts.sourceId]);
}
return JSON.stringify(['all']);
}
/**
* Semantic-cache replay is safe only when every result-shaping input is either
* represented by the existing knobs/scope key or absent. Unsupported shapes
* bypass lookup and writeback rather than risk cross-serving a different view.
*/
export function isSemanticCacheRequestSafe(opts?: HybridSearchOpts): boolean {
if (!opts) return true;
return !(
(opts.offset !== undefined && opts.offset !== 0) ||
opts.type !== undefined ||
opts.types !== undefined ||
opts.exclude_slugs !== undefined ||
opts.detail !== undefined ||
opts.language !== undefined ||
opts.symbolKind !== undefined ||
opts.afterDate !== undefined ||
opts.beforeDate !== undefined ||
opts.recencyBoost !== undefined ||
opts.salience !== undefined ||
opts.recency !== undefined ||
opts.since !== undefined ||
opts.until !== undefined ||
opts.crossModal !== undefined ||
opts.reranker !== undefined ||
opts.rrfK !== undefined ||
opts.dedupOpts !== undefined
);
}
/**
+7 -1
View File
@@ -756,7 +756,13 @@ export function attributeKnob<K extends keyof ModeBundle>(
// slugs written by a process without it, and vice versa. Same one-time
// global cold-miss pattern as the bumps above; refills within
// cache.ttl_seconds (3600s default).
export const KNOBS_HASH_VERSION = 12;
//
// bump 12→13: SearchResult now projects allowlisted email citation metadata
// (`message_id`, `thread_id`, and Message-ID-gated `source_subject`). Cached
// rows store the complete result DTO, so pre-projection rows must miss rather
// than keep returning the old shape after deployment. Same one-time global
// cold-miss pattern; refills within cache.ttl_seconds.
export const KNOBS_HASH_VERSION = 13;
/**
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
+29 -4
View File
@@ -51,6 +51,7 @@
*/
import type { BrainEngine } from '../engine.ts';
import { buildVisibilityClause } from './sql-ranking.ts';
/**
* Snapshot of (pageId, generation) pairs plus the corpus-state MAX
@@ -71,6 +72,18 @@ export interface PageGenerationsSnapshot {
max_generation_at_store: number;
}
/** Read the global page-generation clock. Null means the safety substrate is unavailable. */
export async function readPageGenerationClock(engine: BrainEngine): Promise<number | null> {
try {
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
return Number(rows[0]?.v ?? 0);
} catch {
return null;
}
}
/**
* Build the page-generations snapshot for a set of page_ids in one SQL
* round trip. Used by query-cache.ts:store() at cache-write time.
@@ -102,10 +115,7 @@ export async function buildPageGenerationsSnapshot(
// Empty-result query: only need the Layer 1 bookmark (clock value).
// Per D20, empty-result cache rows trust Layer 1 exclusively;
// bumping the clock on subsequent writes correctly invalidates them.
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0);
snapshot.max_generation_at_store = (await readPageGenerationClock(engine)) ?? 0;
return snapshot;
}
@@ -187,6 +197,21 @@ export const CACHE_GATE_WHERE_CLAUSE = `
)
)
)
AND NOT EXISTS (
-- Visibility is independent of freshness. Archiving a source does not
-- mutate its pages or advance the page-generation clock, so this check
-- MUST sit outside the Layer 1 / Layer 2 OR. Otherwise Layer 1 can serve
-- a fresh-but-now-hidden cached result until TTL expiry.
SELECT 1
FROM jsonb_array_elements(qc.results) AS visible(result)
LEFT JOIN pages p_visible ON p_visible.id = (visible.result->>'page_id')::int
LEFT JOIN sources s_visible ON s_visible.id = p_visible.source_id
WHERE NOT (
p_visible.id IS NOT NULL
AND s_visible.id IS NOT NULL
${buildVisibilityClause('p_visible', 's_visible')}
)
)
`;
/**
+48 -4
View File
@@ -32,6 +32,7 @@
import { createHash } from 'node:crypto';
import type { BrainEngine } from '../engine.ts';
import type { SearchResult, HybridSearchMeta } from '../types.ts';
import { assertValidSourceId } from '../source-id.ts';
import { buildPageGenerationsSnapshot, CACHE_GATE_WHERE_CLAUSE } from './query-cache-gate.ts';
/** Default cosine similarity threshold for cache hits. */
@@ -205,9 +206,20 @@ export class SemanticQueryCache {
queryEmbedding: Float32Array | null,
results: SearchResult[],
meta: HybridSearchMeta,
opts: { sourceId?: string; ttlSeconds?: number; knobsHash?: string } = {},
opts: {
sourceId?: string;
ttlSeconds?: number;
knobsHash?: string;
/** Global generation captured immediately before the producing search. */
maxGenerationAtSearchStart?: number;
} = {},
): Promise<void> {
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) return;
if (
!this.enabled ||
!queryEmbedding ||
queryEmbedding.length === 0 ||
opts.maxGenerationAtSearchStart === undefined
) return;
const sourceId = opts.sourceId ?? 'default';
const knobsHash = opts.knobsHash ?? '';
const ttl = clampTtl(opts.ttlSeconds ?? this.ttlSeconds);
@@ -222,6 +234,13 @@ export class SemanticQueryCache {
.map((r) => r.page_id)
.filter((id): id is number => typeof id === 'number' && Number.isFinite(id));
const snapshot = await buildPageGenerationsSnapshot(this.engine, pageIds);
// Production search captures the clock before materializing results. If
// any page write landed during that window, the serialized DTO may already
// be stale; skip writeback rather than certify it with the newer snapshot.
if (
opts.maxGenerationAtSearchStart !== undefined &&
snapshot.max_generation_at_store !== opts.maxGenerationAtSearchStart
) return;
try {
// v0.32.3 [CDX-4]: knobs_hash threaded into the row so concurrent
@@ -269,10 +288,35 @@ export class SemanticQueryCache {
async clear(opts: { sourceId?: string } = {}): Promise<number> {
try {
if (opts.sourceId) {
assertValidSourceId(opts.sourceId);
const typedScalar = JSON.stringify(['scalar', opts.sourceId]);
const typedSetMemberPattern = `%${JSON.stringify(opts.sourceId)}%`;
const legacySetOnly = `__set__:${opts.sourceId}`;
const legacySetFirst = `__set__:${opts.sourceId},%`;
const legacySetMiddle = `__set__:%,${opts.sourceId},%`;
const legacySetLast = `__set__:%,${opts.sourceId}`;
const rows = await this.engine.executeRaw<{ n: number }>(
`WITH deleted AS (DELETE FROM query_cache WHERE source_id = $1 RETURNING 1)
`WITH deleted AS (
DELETE FROM query_cache
WHERE source_id = $1
OR source_id = $2
OR (source_id LIKE '["set",%' AND source_id LIKE $3)
OR source_id = $4
OR source_id LIKE $5
OR source_id LIKE $6
OR source_id LIKE $7
RETURNING 1
)
SELECT COUNT(*)::int AS n FROM deleted`,
[opts.sourceId],
[
opts.sourceId,
typedScalar,
typedSetMemberPattern,
legacySetOnly,
legacySetFirst,
legacySetMiddle,
legacySetLast,
],
);
return rows[0]?.n ?? 0;
}
+16 -2
View File
@@ -29,6 +29,8 @@ import type { SearchResult, PageType, RelationalFanoutRow } from '../types.ts';
import { createAuditWriter } from '../audit/audit-writer.ts';
import { resolveEntitySlugWithSource } from '../entities/resolve.ts';
import { parseRelationalQuery, type RelationalQuery, type RelationVocab } from './relational-intent.ts';
import { projectEmailCitationMetadata } from '../utils.ts';
import { buildVisibilityClause } from './sql-ranking.ts';
export interface RelationalArmOpts {
sourceId?: string;
@@ -108,11 +110,22 @@ async function hydrate(
const slugs = Array.from(new Set(rows.map(r => r.slug)));
const pageRows = await engine.executeRaw<{
page_id: number; slug: string; source_id: string; title: string; type: string; synopsis: string | null;
message_id: string | null; thread_id: string | null; source_subject: string | null;
}>(
`SELECT p.id AS page_id, p.slug, p.source_id, p.title, p.type,
LEFT(p.compiled_truth, 240) AS synopsis
LEFT(p.compiled_truth, 240) AS synopsis,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id,
CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string'
THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string'
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject
FROM pages p
WHERE p.slug = ANY($1::text[]) AND p.deleted_at IS NULL`,
JOIN sources s ON s.id = p.source_id
WHERE p.slug = ANY($1::text[]) ${buildVisibilityClause('p', 's')}`,
[slugs],
);
const byKey = new Map<string, typeof pageRows[number]>();
@@ -141,6 +154,7 @@ async function hydrate(
relational_seed: seedSlug,
relational_hop: r.hop,
relational_path: r.path,
...projectEmailCitationMetadata(pr),
});
}
return out;
+1 -1
View File
@@ -146,7 +146,7 @@ export function buildHardExcludeClause(slugColumn: string, prefixes: string[]):
* responsible for joining `sources` so this alias resolves.
*
* @returns raw SQL fragment, e.g.
* `AND p.deleted_at IS NULL AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine')`
* `AND p.deleted_at IS NULL AND NOT s.archived AND (jsonb_typeof(COALESCE(p.frontmatter, '{}'::jsonb)) = 'object' AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine'))`
*/
export function buildVisibilityClause(pageAlias: string, sourceAlias: string): string {
// Single source of truth for the quarantine SQL lives in quarantine.ts so
+91 -31
View File
@@ -23,6 +23,8 @@
import type { BrainEngine } from '../engine.ts';
import type { SearchResult } from '../types.ts';
import { projectEmailCitationMetadata } from '../utils.ts';
import { buildVisibilityClause } from './sql-ranking.ts';
const MAX_WALK_DEPTH = 2;
const NEIGHBOR_CAP_PER_HOP = 50;
@@ -34,6 +36,8 @@ export interface TwoPassOpts {
nearSymbol?: string;
/** Filter expansion to one source. When unset, crosses sources. */
sourceId?: string;
/** Federated source grant. Non-empty arrays take precedence over sourceId. */
sourceIds?: string[];
}
interface ChunkWithScore {
@@ -54,6 +58,16 @@ export async function expandAnchors(
opts: TwoPassOpts = {},
): Promise<ChunkWithScore[]> {
const depth = Math.min(Math.max(opts.walkDepth ?? 0, 0), MAX_WALK_DEPTH);
const federatedSourceIds = opts.sourceIds && opts.sourceIds.length > 0
? opts.sourceIds
: undefined;
const sourceScopeSql = federatedSourceIds
? ' AND p.source_id = ANY($2::text[])'
: opts.sourceId
? ' AND p.source_id = $2'
: '';
const sourceScopeValue = federatedSourceIds ?? opts.sourceId;
const visibilityClause = buildVisibilityClause('p', 's');
if (depth === 0 && !opts.nearSymbol) {
return anchors.map(a => ({
chunk_id: a.chunk_id,
@@ -83,18 +97,14 @@ export async function expandAnchors(
// filter; undefined → cross-source (matches the documented contract).
if (opts.nearSymbol) {
try {
const rows = opts.sourceId
? await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified = $1 AND p.source_id = $2
LIMIT 50`,
[opts.nearSymbol, opts.sourceId],
)
: await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = $1 LIMIT 50`,
[opts.nearSymbol],
);
const rows = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.symbol_name_qualified = $1${sourceScopeSql} ${visibilityClause}
LIMIT 50`,
sourceScopeValue === undefined ? [opts.nearSymbol] : [opts.nearSymbol, sourceScopeValue],
);
const baseScore = anchors.length > 0 ? anchors[0]!.score : 1.0;
for (const r of rows) {
if (!seen.has(r.id)) {
@@ -135,8 +145,19 @@ export async function expandAnchors(
const directChunkIds: number[] = [];
const unresolvedTargets: string[] = [];
for (const e of edges) {
if (e.to_chunk_id != null) directChunkIds.push(e.to_chunk_id);
else if (e.to_symbol_qualified) unresolvedTargets.push(e.to_symbol_qualified);
if (e.to_chunk_id != null) {
// getEdgesByChunk(direction:'both') returns incoming and outgoing
// edges. Follow the endpoint opposite the current chunk; always
// taking to_chunk_id makes incoming A→B edges reselect B.
const neighborId = e.from_chunk_id === chunkId
? e.to_chunk_id
: e.to_chunk_id === chunkId
? e.from_chunk_id
: null;
if (neighborId != null) directChunkIds.push(neighborId);
} else if (e.to_symbol_qualified) {
unresolvedTargets.push(e.to_symbol_qualified);
}
}
// Resolve unresolved edges by looking up chunks whose
// symbol_name_qualified matches. One batch query per frontier node.
@@ -146,26 +167,40 @@ export async function expandAnchors(
// boundaries silently in multi-source brains.
if (unresolvedTargets.length > 0) {
try {
const resolved = opts.sourceId
? await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified = ANY($1::text[])
AND p.source_id = $2
LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
[unresolvedTargets, opts.sourceId],
)
: await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = ANY($1::text[]) LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
[unresolvedTargets],
);
const resolved = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.symbol_name_qualified = ANY($1::text[])${sourceScopeSql} ${visibilityClause}
LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
sourceScopeValue === undefined ? [unresolvedTargets] : [unresolvedTargets, sourceScopeValue],
);
for (const r of resolved) directChunkIds.push(r.id);
} catch {
// best-effort
}
}
for (const tid of directChunkIds) {
// Resolved chunk edges can cross sources even when the originating edge
// carries an allowed source. Filter the destination page itself. A scope
// query failure must fail closed rather than exposing an unverified row.
let scopedChunkIds: number[] = [];
if (directChunkIds.length > 0) {
try {
const scoped = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE cc.id = ANY($1::int[])${sourceScopeSql} ${visibilityClause}`,
sourceScopeValue === undefined ? [directChunkIds] : [directChunkIds, sourceScopeValue],
);
scopedChunkIds = scoped.map((row) => row.id);
} catch {
scopedChunkIds = [];
}
}
for (const tid of scopedChunkIds) {
if (seen.has(tid)) continue;
const nbScore = current.score * decay;
seen.set(tid, { chunk_id: tid, score: nbScore, hop, source: 'neighbor' });
@@ -188,18 +223,42 @@ export async function expandAnchors(
export async function hydrateChunks(
engine: BrainEngine,
chunkIds: number[],
opts: Pick<TwoPassOpts, 'sourceId' | 'sourceIds'> = {},
): Promise<SearchResult[]> {
if (chunkIds.length === 0) return [];
const federatedSourceIds = opts.sourceIds && opts.sourceIds.length > 0
? opts.sourceIds
: undefined;
const scopeSql = federatedSourceIds
? ' AND p.source_id = ANY($2::text[])'
: opts.sourceId
? ' AND p.source_id = $2'
: '';
const params: unknown[] = [chunkIds];
if (federatedSourceIds) params.push(federatedSourceIds);
else if (opts.sourceId) params.push(opts.sourceId);
const visibilityClause = buildVisibilityClause('p', 's');
const rows = await engine.executeRaw<{
slug: string; page_id: number; title: string; type: string; source_id: string;
chunk_id: number; chunk_index: number; chunk_text: string; chunk_source: string;
message_id: string | null; thread_id: string | null; source_subject: string | null;
}>(
`SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
THEN p.frontmatter->>'message_id' END AS message_id,
CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string'
THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
AND jsonb_typeof(p.frontmatter->'subject') = 'string'
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
JOIN sources s ON s.id = p.source_id
WHERE cc.id = ANY($1::int[])${scopeSql} ${visibilityClause}`,
params,
);
return rows.map((r) => ({
slug: r.slug,
@@ -213,5 +272,6 @@ export async function hydrateChunks(
score: 0, // two-pass caller assigns scores.
stale: false,
source_id: r.source_id,
...projectEmailCitationMetadata(r),
} as SearchResult));
}
+6
View File
@@ -723,6 +723,12 @@ export interface SearchResult {
*/
effective_date?: string | null;
effective_date_source?: string | null;
/** RFC 5322 Message-ID projected from allowlisted email frontmatter. */
message_id?: string;
/** Gmail thread id projected from allowlisted email frontmatter. */
thread_id?: string;
/** Exact email subject, projected only when the page has a Message-ID. */
source_subject?: string;
/**
* v0.40.4 graph signals — populated by applyGraphSignals when the
* graph_signals mode-bundle knob is on. Surfaced in JSON envelope
+47 -2
View File
@@ -117,7 +117,16 @@ export function rowToPage(row: Record<string, unknown>): Page {
title: row.title as string,
compiled_truth: row.compiled_truth as string,
timeline: row.timeline as string,
frontmatter: (typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) : row.frontmatter) as Record<string, unknown>,
// Postgres and PGLite both decode JSONB objects. A string here means the
// JSONB top level itself is a string (legacy double-encoding), not a driver
// transport shape. Never decode it into trusted frontmatter.
frontmatter: (
row.frontmatter !== null &&
typeof row.frontmatter === 'object' &&
!Array.isArray(row.frontmatter)
? row.frontmatter
: {}
) as Record<string, unknown>,
content_hash: row.content_hash as string | undefined,
// v0.29 (column added in migration v40). Old brains pre-migration return undefined.
emotional_weight: row.emotional_weight == null ? undefined : Number(row.emotional_weight),
@@ -162,7 +171,9 @@ export function rowToStalePage(row: Record<string, unknown>): StalePageRow {
title: (row.title as string | null) ?? '',
compiled_truth: (row.compiled_truth as string | null) ?? '',
timeline: (row.timeline as string | null) ?? '',
frontmatter: (fm == null ? {} : (typeof fm === 'string' ? JSON.parse(fm) : fm)) as Record<string, unknown>,
frontmatter: (
fm !== null && typeof fm === 'object' && !Array.isArray(fm) ? fm : {}
) as Record<string, unknown>,
updated_at: new Date(row.updated_at as string),
// #1768: full-µs UTC string projected by the SELECT (`updated_at_iso`).
// Fallback derives an ISO string from the Date — NEVER String(Date), which
@@ -333,6 +344,39 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
};
}
function projectCitationMetadata(
record: Record<string, unknown> | null | undefined,
subjectField: 'subject' | 'source_subject',
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
if (!record) return {};
const metadata: Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> = {};
if (typeof record.message_id === 'string' && record.message_id.trim().length > 0) {
metadata.message_id = record.message_id;
}
if (typeof record.thread_id === 'string' && record.thread_id.length > 0) {
metadata.thread_id = record.thread_id;
}
const subject = record[subjectField];
if (metadata.message_id && typeof subject === 'string' && subject.length > 0) {
metadata.source_subject = subject;
}
return metadata;
}
/** Project normalized SQL result columns. `source_subject` is an internal alias. */
export function projectEmailCitationMetadata(
record: Record<string, unknown> | null | undefined,
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
return projectCitationMetadata(record, 'source_subject');
}
/** Project raw page frontmatter. Internal SQL aliases are never trusted here. */
export function projectEmailCitationFrontmatter(
record: Record<string, unknown> | null | undefined,
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
return projectCitationMetadata(record, 'subject');
}
export function rowToSearchResult(row: Record<string, unknown>): SearchResult {
const result: SearchResult = {
slug: row.slug as string,
@@ -381,6 +425,7 @@ export function rowToSearchResult(row: Record<string, unknown>): SearchResult {
result.effective_date_source = raw;
}
}
Object.assign(result, projectEmailCitationMetadata(row));
return result;
}
+47 -10
View File
@@ -3,29 +3,30 @@
*
* A federated search reads a different graph than a single-source one, so
* the semantic cache must key them apart. `cacheScopeKey` produces an
* order-independent key for federated scopes and leaves single-source
* brains on their existing key (scalar id or 'default'), so single-source
* cache hit-rate is unchanged.
* order-independent key for federated scopes and keeps unscoped all-source
* reads distinct from every scalar source key.
*/
import { describe, test, expect } from 'bun:test';
import { cacheScopeKey } from '../src/core/search/hybrid.ts';
import { cacheScopeKey, isSemanticCacheRequestSafe } from '../src/core/search/hybrid.ts';
import type { HybridSearchOpts } from '../src/core/search/hybrid.ts';
describe('cacheScopeKey', () => {
test('unscoped → default (single-source unchanged)', () => {
expect(cacheScopeKey(undefined)).toBe('default');
expect(cacheScopeKey({})).toBe('default');
test('unscoped uses a typed key and never collides with scalar default', () => {
expect(cacheScopeKey(undefined)).toBe('["all"]');
expect(cacheScopeKey({})).toBe('["all"]');
expect(cacheScopeKey({})).not.toBe(cacheScopeKey({ sourceId: 'default' }));
});
test('scalar sourceId → itself (single-source unchanged)', () => {
expect(cacheScopeKey({ sourceId: 'host' })).toBe('host');
test('scalar sourceId uses a typed key', () => {
expect(cacheScopeKey({ sourceId: 'host' })).toBe('["scalar","host"]');
});
test('federated sourceIds → order-independent set key', () => {
const k1 = cacheScopeKey({ sourceIds: ['team-b', 'team-a', 'host'] });
const k2 = cacheScopeKey({ sourceIds: ['host', 'team-a', 'team-b'] });
expect(k1).toBe(k2); // order does not matter
expect(k1).toBe('__set__:host,team-a,team-b');
expect(k1).toBe('["set","host","team-a","team-b"]');
});
test('different source-sets do NOT share a key', () => {
@@ -39,4 +40,40 @@ describe('cacheScopeKey', () => {
const scalar = cacheScopeKey({ sourceId: 'host' });
expect(set).not.toBe(scalar); // a 1-element set still cannot serve a scalar read
});
test('rejects forged scalar/set encodings and invalid federated ids', () => {
expect(() => cacheScopeKey({ sourceId: '__set__:a,b' })).toThrow('Invalid source_id');
expect(() => cacheScopeKey({ sourceIds: ['a', '__all__'] })).toThrow('Invalid source_id');
});
});
describe('isSemanticCacheRequestSafe', () => {
test('accepts the standard cache-safe request shape', () => {
expect(isSemanticCacheRequestSafe({ limit: 20, sourceId: 'default' })).toBe(true);
});
test('accepts the production query-op shape (expandFn is folded into the knobs hash, not unsafe)', () => {
// operations.ts always passes expandFn on the default `query` op path;
// listing it unsafe would permanently disable the semantic cache for the
// flagship op. Its effect is keyed via the expansion bit instead
// (hybridSearchCached folds `expandFn ? expansion : false` into the hash).
expect(isSemanticCacheRequestSafe({
limit: 20,
offset: 0,
expansion: true,
expandFn: async (q: string) => [q],
sourceId: 'default',
})).toBe(true);
});
const unsafeRequests: HybridSearchOpts[] = [
{ offset: 10 }, { detail: 'high' }, { language: 'typescript' },
{ symbolKind: 'function' }, { types: ['note'] }, { since: '7d' },
{ until: '2026-07-18' }, { salience: 'on' }, { recency: 'strong' },
{ crossModal: 'both' }, { exclude_slugs: ['private/page'] },
];
test.each(unsafeRequests)('rejects unsupported result-shaping request %#', (opts) => {
expect(isSemanticCacheRequestSafe(opts)).toBe(false);
});
});
+4 -2
View File
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
return resolveSearchMode({ mode: 'balanced' });
}
test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => {
test('KNOBS_HASH_VERSION is 13 (12→13 email citation result-schema projection)', () => {
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
@@ -146,7 +146,9 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
// finally reaches asymmetric providers — pre-fix rows were keyed on
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
expect(KNOBS_HASH_VERSION).toBe(12);
// Email citation metadata projection: 12→13 so cached SearchResult DTOs
// written before message_id/thread_id/source_subject cannot survive.
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('flipping unified_multimodal changes the hash', () => {
+23 -8
View File
@@ -52,6 +52,21 @@ describe('cache gate end-to-end (PGLite)', () => {
let engine: PGLiteEngine;
let cache: SemanticQueryCache;
async function storeCurrent(
cacheInstance: SemanticQueryCache,
...args: Parameters<SemanticQueryCache['store']>
): Promise<void> {
const [queryText, queryEmbedding, results, meta, opts = {}] = args;
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
await cacheInstance.store(queryText, queryEmbedding, results, meta, {
...opts,
maxGenerationAtSearchStart:
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
});
}
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
@@ -90,7 +105,7 @@ describe('cache gate end-to-end (PGLite)', () => {
} as unknown as SearchResult,
];
const emb = fakeEmbedding(1);
await cache.store('alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
const hit = await cache.lookup(emb, { sourceId: 'default' });
expect(hit.hit).toBe(true);
expect(hit.results?.length).toBe(1);
@@ -102,7 +117,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'x', score: 1.0 } as unknown as SearchResult,
];
const emb = fakeEmbedding(2);
await cache.store('alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
// Update content_truth — trigger bumps p1.generation
await engine.putPage('test/p1', {
@@ -123,7 +138,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'a', score: 1.0 } as unknown as SearchResult,
];
const emb = fakeEmbedding(3);
await cache.store('alpha', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'alpha', emb, results, fakeMeta(), { sourceId: 'default' });
// Create an UNRELATED new page (different topic, not in result set).
await seedPage('test/p2', 'beta gamma');
@@ -143,7 +158,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'g', score: 1.0 } as unknown as SearchResult,
];
// Simulate a pre-v0.40.3.0 row: empty snapshot + zero bookmark.
await cache.store('gamma delta', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'gamma delta', emb, results, fakeMeta(), { sourceId: 'default' });
await engine.executeRaw(
`UPDATE query_cache
SET page_generations = '{}'::jsonb,
@@ -171,7 +186,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'q', score: 0.9 } as unknown as SearchResult,
];
const emb = fakeEmbedding(7);
await cache.store('phi chi', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'phi chi', emb, results, fakeMeta(), { sourceId: 'default' });
// Hard-delete via engine.deletePage. Pre-v0.41.19.0 the trigger
// didn't fire on DELETE so MAX(generation) didn't move and the cache
@@ -197,7 +212,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p1, slug: 'test/non-max-p1', title: 'test/non-max-p1', snippet: 'o', score: 1.0 } as unknown as SearchResult,
];
const emb = fakeEmbedding(8);
await cache.store('omega psi', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'omega psi', emb, results, fakeMeta(), { sourceId: 'default' });
// UPDATE p1 (the non-max page) with new content.
await engine.putPage('test/non-max-p1', {
@@ -218,7 +233,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'e', score: 1.0 } as unknown as SearchResult,
];
const emb = fakeEmbedding(5);
await cache.store('epsilon', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'epsilon', emb, results, fakeMeta(), { sourceId: 'default' });
// Soft-delete: UPDATE pages SET deleted_at = now() — production path
// for the user-facing `archive` command. The row-level trigger fires
@@ -245,7 +260,7 @@ describe('cache gate end-to-end (PGLite)', () => {
{ page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'e', score: 0.9 } as unknown as SearchResult,
];
const emb = fakeEmbedding(6);
await cache.store('zeta eta', emb, results, fakeMeta(), { sourceId: 'default' });
await storeCurrent(cache, 'zeta eta', emb, results, fakeMeta(), { sourceId: 'default' });
// Bump only p2.
await engine.putPage('test/p2', {
+185
View File
@@ -149,6 +149,149 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
expect(pgResults[0]?.slug).toBe(pgliteResults[0]?.slug);
});
test('email citation metadata projects identically across engines', async () => {
const slug = 'mail/example-citation';
const page = {
type: 'note' as const,
title: 'Generated page title',
compiled_truth: 'unique citation projection evidence',
timeline: '',
frontmatter: {
message_id: '<citation@example.com>',
thread_id: 'thread-example',
subject: 'Example exact email subject',
},
};
const chunks = [{
chunk_index: 0,
chunk_text: page.compiled_truth,
chunk_source: 'compiled_truth' as const,
embedding: basisEmbedding(77),
}];
await pgEngine.putPage(slug, page);
await pgEngine.upsertChunks(slug, chunks);
await pgliteEngine.putPage(slug, page);
await pgliteEngine.upsertChunks(slug, chunks);
const results = [
(await pgEngine.searchKeyword('unique citation projection evidence'))[0],
(await pgliteEngine.searchKeyword('unique citation projection evidence'))[0],
(await pgEngine.searchKeywordChunks('unique citation projection evidence'))[0],
(await pgliteEngine.searchKeywordChunks('unique citation projection evidence'))[0],
(await pgEngine.searchVector(basisEmbedding(77)))[0],
(await pgliteEngine.searchVector(basisEmbedding(77)))[0],
];
for (const result of results) {
expect(result?.message_id).toBe('<citation@example.com>');
expect(result?.thread_id).toBe('thread-example');
expect(result?.source_subject).toBe('Example exact email subject');
}
const nonEmailSlug = 'notes/generated-title-subject-gate';
const nonEmailPage = {
type: 'note' as const,
title: 'Generated page title must stay a title',
compiled_truth: 'unique non-email subject gate evidence',
timeline: '',
frontmatter: {
subject: 'Frontmatter subject without an email identity',
thread_id: 'standalone-thread-id',
},
};
const nonEmailChunks = [{
chunk_index: 0,
chunk_text: nonEmailPage.compiled_truth,
chunk_source: 'compiled_truth' as const,
}];
await pgEngine.putPage(nonEmailSlug, nonEmailPage);
await pgEngine.upsertChunks(nonEmailSlug, nonEmailChunks);
await pgliteEngine.putPage(nonEmailSlug, nonEmailPage);
await pgliteEngine.upsertChunks(nonEmailSlug, nonEmailChunks);
for (const result of [
(await pgEngine.searchKeyword('unique non-email subject gate evidence'))[0],
(await pgliteEngine.searchKeyword('unique non-email subject gate evidence'))[0],
]) {
expect(result?.message_id).toBeUndefined();
expect(result?.thread_id).toBe('standalone-thread-id');
expect(result?.source_subject).toBeUndefined();
}
const whitespaceSlug = 'mail/whitespace-message-id';
const whitespacePage = {
type: 'note' as const,
title: 'Whitespace Message-ID',
compiled_truth: 'unique whitespace message id evidence',
timeline: '',
frontmatter: {
message_id: ' \t\n ',
thread_id: 'thread-whitespace',
subject: 'Subject must remain gated',
},
};
const whitespaceChunks = [{
chunk_index: 0,
chunk_text: whitespacePage.compiled_truth,
chunk_source: 'compiled_truth' as const,
embedding: basisEmbedding(78),
}];
await pgEngine.putPage(whitespaceSlug, whitespacePage);
await pgEngine.upsertChunks(whitespaceSlug, whitespaceChunks);
await pgliteEngine.putPage(whitespaceSlug, whitespacePage);
await pgliteEngine.upsertChunks(whitespaceSlug, whitespaceChunks);
for (const result of [
(await pgEngine.searchKeyword('unique whitespace message id evidence'))[0],
(await pgliteEngine.searchKeyword('unique whitespace message id evidence'))[0],
(await pgEngine.searchKeywordChunks('unique whitespace message id evidence'))[0],
(await pgliteEngine.searchKeywordChunks('unique whitespace message id evidence'))[0],
(await pgEngine.searchVector(basisEmbedding(78)))[0],
(await pgliteEngine.searchVector(basisEmbedding(78)))[0],
]) {
expect(result?.message_id).toBeUndefined();
expect(result?.thread_id).toBe('thread-whitespace');
expect(result?.source_subject).toBeUndefined();
}
const malformedSlug = 'notes/numeric-email-frontmatter';
const malformedPage = {
type: 'note' as const,
title: 'Numeric email frontmatter',
compiled_truth: 'unique numeric email frontmatter evidence',
timeline: '',
frontmatter: {
message_id: 12345,
thread_id: 67890,
subject: 98765,
},
};
const malformedChunks = [{
chunk_index: 0,
chunk_text: malformedPage.compiled_truth,
chunk_source: 'compiled_truth' as const,
embedding: basisEmbedding(79),
}];
await pgEngine.putPage(malformedSlug, malformedPage);
await pgEngine.upsertChunks(malformedSlug, malformedChunks);
await pgliteEngine.putPage(malformedSlug, malformedPage);
await pgliteEngine.upsertChunks(malformedSlug, malformedChunks);
for (const result of [
(await pgEngine.searchKeyword('unique numeric email frontmatter evidence'))[0],
(await pgliteEngine.searchKeyword('unique numeric email frontmatter evidence'))[0],
(await pgEngine.searchKeywordChunks('unique numeric email frontmatter evidence'))[0],
(await pgliteEngine.searchKeywordChunks('unique numeric email frontmatter evidence'))[0],
(await pgEngine.searchVector(basisEmbedding(79)))[0],
(await pgliteEngine.searchVector(basisEmbedding(79)))[0],
]) {
expect(result?.message_id).toBeUndefined();
expect(result?.thread_id).toBeUndefined();
expect(result?.source_subject).toBeUndefined();
}
});
test('hard-exclude is consistent across engines', async () => {
// Both engines should hide test/ pages by default; both should opt
// them back in via include_slug_prefixes.
@@ -614,6 +757,7 @@ async function seedRelational(eng: BrainEngine) {
['people/ep-inv-b', 'person'],
['people/ep-emp-c', 'person'],
['people/ep-mentioner', 'person'],
['people/ep-quarantined', 'person'],
];
for (const [slug, type] of pages) {
await eng.putPage(slug, { type, title: slug, compiled_truth: `${slug} body`, timeline: '' });
@@ -626,7 +770,32 @@ async function seedRelational(eng: BrainEngine) {
await eng.addLink('people/ep-inv-b', 'companies/ep-widget', '', 'invested_in', 'manual');
await eng.addLink('people/ep-emp-c', 'companies/ep-widget', '', 'works_at', 'manual');
await eng.addLink('people/ep-mentioner', 'companies/ep-widget', '', 'mentions', 'mentions');
await eng.addLink('people/ep-quarantined', 'companies/ep-widget', '', 'invested_in', 'manual');
await eng.addLink('people/ep-inv-a', 'companies/ep-other', '', 'invested_in', 'manual');
await eng.executeRaw(
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb
WHERE slug = 'people/ep-quarantined' AND source_id = 'default'`,
);
await eng.executeRaw(
`INSERT INTO sources (id, name, archived, created_at)
VALUES ('ep-archived', 'ep-archived', false, NOW())`,
);
await eng.putPage(
'companies/ep-archived-widget',
{ type: 'company', title: 'Archived Widget', compiled_truth: 'Hidden company.', timeline: '' },
{ sourceId: 'ep-archived' },
);
await eng.putPage(
'people/ep-archived-investor',
{ type: 'person', title: 'Archived Investor', compiled_truth: 'Hidden person.', timeline: '' },
{ sourceId: 'ep-archived' },
);
await eng.addLink(
'people/ep-archived-investor', 'companies/ep-archived-widget', '', 'invested_in', 'manual', undefined, undefined,
{ fromSourceId: 'ep-archived', toSourceId: 'ep-archived' },
);
await eng.executeRaw(`UPDATE sources SET archived = true WHERE id = 'ep-archived'`);
}
describeBoth('Engine parity — relationalFanout', () => {
@@ -670,6 +839,22 @@ describeBoth('Engine parity — relationalFanout', () => {
expect(pg.map(r => r.slug)).not.toContain('people/ep-mentioner');
});
test('archive and quarantine visibility is identical across engines', async () => {
const visibleOpts = { direction: 'in' as const, linkTypes: ['invested_in'] };
const pgVisible = await pgEngine.relationalFanout(['companies/ep-widget'], visibleOpts);
const pgliteVisible = await pgliteEngine.relationalFanout(['companies/ep-widget'], visibleOpts);
expect(shape(pgVisible)).toEqual(shape(pgliteVisible));
expect(pgVisible.map(r => r.slug)).not.toContain('people/ep-quarantined');
const archivedOpts = {
sourceId: 'ep-archived', direction: 'in' as const, linkTypes: ['invested_in'],
};
const pgArchived = await pgEngine.relationalFanout(['companies/ep-archived-widget'], archivedOpts);
const pgliteArchived = await pgliteEngine.relationalFanout(['companies/ep-archived-widget'], archivedOpts);
expect(shape(pgArchived)).toEqual(shape(pgliteArchived));
expect(pgArchived).toEqual([]);
});
test('connects (multi-seed, both) identical across engines', async () => {
const seeds = ['companies/ep-widget', 'companies/ep-other'];
const pg = await pgEngine.relationalFanout(seeds, { direction: 'both' });
+50 -12
View File
@@ -20,11 +20,17 @@ import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
let engine: PGLiteEngine;
let chunkEmbedDim = 0;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
const dim = await (engine as any).db.query(
`SELECT atttypmod FROM pg_attribute
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'`,
);
chunkEmbedDim = (dim.rows[0] as { atttypmod: number }).atttypmod;
});
afterAll(async () => {
@@ -48,13 +54,18 @@ beforeEach(async () => {
title: 'Alice Source-A',
compiled_truth: 'Alice works on widgets in source A. Important context here.',
timeline: '',
frontmatter: {},
frontmatter: {
message_id: '<source-a@example.com>',
thread_id: 'thread-source-a',
subject: 'Source A exact subject',
},
}, { sourceId: 'default' });
await engine.upsertChunks('people/alice', [{
chunk_index: 0,
chunk_text: 'Alice works on widgets in source A. Important context here.',
chunk_source: 'compiled_truth',
token_count: 12,
embedding: Float32Array.from({ length: chunkEmbedDim }, (_, i) => i === 0 ? 1 : 0),
}], { sourceId: 'default' });
await engine.putPage('people/alice', {
@@ -62,13 +73,18 @@ beforeEach(async () => {
title: 'Alice Source-B',
compiled_truth: 'Alice works on gadgets in source B. Important context here.',
timeline: '',
frontmatter: {},
frontmatter: {
message_id: '<source-b@example.com>',
thread_id: 'thread-source-b',
subject: 'Source B exact subject',
},
}, { sourceId: 'src-b' });
await engine.upsertChunks('people/alice', [{
chunk_index: 0,
chunk_text: 'Alice works on gadgets in source B. Important context here.',
chunk_source: 'compiled_truth',
token_count: 12,
embedding: Float32Array.from({ length: chunkEmbedDim }, (_, i) => i === 1 ? 1 : 0),
}], { sourceId: 'src-b' });
await engine.putPage('people/bob', {
@@ -95,6 +111,9 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
expect(results.length).toBeGreaterThan(0);
for (const r of results) {
expect(r.source_id).toBe('default');
expect(r.message_id).toBe('<source-a@example.com>');
expect(r.thread_id).toBe('thread-source-a');
expect(r.source_subject).toBe('Source A exact subject');
}
});
@@ -103,6 +122,9 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
expect(results.length).toBeGreaterThan(0);
for (const r of results) {
expect(r.source_id).toBe('src-b');
expect(r.message_id).toBe('<source-b@example.com>');
expect(r.thread_id).toBe('thread-source-b');
expect(r.source_subject).toBe('Source B exact subject');
}
});
@@ -170,16 +192,32 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
});
test('searchVector with sourceId filters HNSW candidate pool', async () => {
// No real embeddings on the test pages; the WHERE cc.embedding IS NOT NULL
// gate filters them out. We assert the contract via an empty result
// rather than a positive match: with sourceId set, the SQL still runs
// (no type or undefined-column errors).
const synth = new Float32Array(1536).fill(0.01);
const results = await engine.searchVector(synth, { sourceId: 'src-b' });
// Either empty (no embeddings) or all from src-b. Both prove the
// filter is wired without a runtime error.
for (const r of results) {
expect(r.source_id).toBe('src-b');
const fixtures = [
{
sourceId: 'default', embeddingIndex: 0,
message_id: '<source-a@example.com>', thread_id: 'thread-source-a',
source_subject: 'Source A exact subject',
},
{
sourceId: 'src-b', embeddingIndex: 1,
message_id: '<source-b@example.com>', thread_id: 'thread-source-b',
source_subject: 'Source B exact subject',
},
];
for (const fixture of fixtures) {
const synth = Float32Array.from(
{ length: chunkEmbedDim },
(_, i) => i === fixture.embeddingIndex ? 1 : 0,
);
const results = await engine.searchVector(synth, { sourceId: fixture.sourceId });
expect(results.length).toBeGreaterThan(0);
for (const r of results) {
expect(r.source_id).toBe(fixture.sourceId);
expect(r.message_id).toBe(fixture.message_id);
expect(r.thread_id).toBe(fixture.thread_id);
expect(r.source_subject).toBe(fixture.source_subject);
}
}
});
+167 -2
View File
@@ -15,7 +15,8 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { expandAnchors } from '../../src/core/search/two-pass.ts';
import { expandAnchors, hydrateChunks } from '../../src/core/search/two-pass.ts';
import { hybridSearch } from '../../src/core/search/hybrid.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
let engine: PGLiteEngine;
@@ -84,6 +85,43 @@ describe('v0.34 W0a — multi-source isolation in two-pass retrieval', () => {
}
});
test('expandAnchors honors federated sourceIds grants', async () => {
const result = await expandAnchors(engine, [], {
walkDepth: 0,
nearSymbol: 'parseMarkdown',
sourceIds: ['source-a'],
});
const chunkIds = result.map((r) => r.chunk_id);
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
`SELECT cc.id AS chunk_id, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
);
expect(rows.length).toBeGreaterThan(0);
for (const row of rows) expect(row.source_id).toBe('source-a');
});
test('hydrateChunks honors federated sourceIds grants', async () => {
const chunkRows = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified = 'parseMarkdown'
ORDER BY p.source_id`,
[],
);
const rows = await hydrateChunks(
engine,
chunkRows.map((row) => row.id),
{ sourceIds: ['source-a'] },
);
expect(rows.length).toBeGreaterThan(0);
for (const row of rows) expect(row.source_id).toBe('source-a');
});
test('expandAnchors with nearSymbol and NO sourceId returns chunks from both sources (legacy cross-source mode preserved)', async () => {
const result = await expandAnchors(engine, [], {
walkDepth: 0,
@@ -155,6 +193,115 @@ describe('v0.34 W0a — multi-source isolation in two-pass retrieval', () => {
expect(r.source_id).toBe('source-a');
}
});
test('hybrid two-pass expansion honors federated sourceIds for direct chunk edges', async () => {
const results = await hybridSearch(engine, 'callerInA', {
walkDepth: 1,
nearSymbol: 'callerInA',
sourceIds: ['source-a'],
expansion: false,
useCache: false,
mode: 'conservative',
autocut: false,
adaptiveReturn: false,
limit: 50,
});
expect(results.length).toBeGreaterThan(0);
for (const result of results) {
expect(result.source_id).toBe('source-a');
expect(result.message_id).not.toBe('<denied@example.com>');
expect(result.thread_id).not.toBe('denied-thread');
expect(result.source_subject).not.toBe('Denied exact subject');
}
});
test('incoming direct edges cannot bypass source scope or visibility', async () => {
const chunks = await engine.executeRaw<{ id: number; source_id: string; symbol_name_qualified: string }>(
`SELECT cc.id, p.source_id, cc.symbol_name_qualified
FROM content_chunks cc JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified IN ('callerInA', 'parseMarkdown')`,
[],
);
const callerA = chunks.find(r => r.source_id === 'source-a' && r.symbol_name_qualified === 'callerInA')!.id;
const targetB = chunks.find(r => r.source_id === 'source-b' && r.symbol_name_qualified === 'parseMarkdown')!.id;
const anchor = [{
slug: 'code/src/markdown-b.ts', page_id: 0, title: '', type: 'code' as const,
chunk_text: '', chunk_source: 'compiled_truth' as const,
chunk_id: targetB, chunk_index: 0, score: 1, source_id: 'source-b', stale: false,
}];
const scoped = await expandAnchors(engine, anchor, { walkDepth: 1, sourceId: 'source-b' });
expect(scoped.map(r => r.chunk_id)).not.toContain(callerA);
await engine.executeRaw(
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb
WHERE source_id = 'source-a' AND slug = 'code/src/caller-a.ts'`,
[],
);
try {
const hidden = await expandAnchors(engine, anchor, { walkDepth: 1 });
expect(hidden.map(r => r.chunk_id)).not.toContain(callerA);
} finally {
await engine.executeRaw(
`UPDATE pages SET frontmatter = frontmatter - 'quarantine'
WHERE source_id = 'source-a' AND slug = 'code/src/caller-a.ts'`,
[],
);
}
});
test('two-pass selection and hydration reject archived, quarantined, and deleted pages', async () => {
const chunkRows = await engine.executeRaw<{ id: number; symbol_name_qualified: string }>(
`SELECT cc.id, cc.symbol_name_qualified
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.source_id = 'source-b' OR cc.symbol_name_qualified = 'callerInA'`,
[],
);
const deniedChunkId = chunkRows.find(r => r.symbol_name_qualified === 'parseMarkdown')!.id;
const callerChunkId = chunkRows.find(r => r.symbol_name_qualified === 'callerInA')!.id;
const anchor = [{
slug: 'code/src/caller-a.ts', page_id: 0, title: '', type: 'code' as const,
chunk_text: '', chunk_source: 'compiled_truth' as const,
chunk_id: callerChunkId, chunk_index: 0, score: 1, source_id: 'source-a', stale: false,
}];
const states = [
{
hide: () => engine.executeRaw(`UPDATE sources SET archived = true WHERE id = 'source-b'`, []),
restore: () => engine.executeRaw(`UPDATE sources SET archived = false WHERE id = 'source-b'`, []),
},
{
hide: () => engine.executeRaw(
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb WHERE source_id = 'source-b'`, [],
),
restore: () => engine.executeRaw(
`UPDATE pages SET frontmatter = frontmatter - 'quarantine' WHERE source_id = 'source-b'`, [],
),
},
{
hide: () => engine.executeRaw(`UPDATE pages SET deleted_at = NOW() WHERE source_id = 'source-b'`, []),
restore: () => engine.executeRaw(`UPDATE pages SET deleted_at = NULL WHERE source_id = 'source-b'`, []),
},
];
for (const state of states) {
await state.hide();
try {
const near = await expandAnchors(engine, [], { nearSymbol: 'parseMarkdown', sourceId: 'source-b' });
expect(near).toEqual([]);
const expanded = await expandAnchors(engine, anchor, { walkDepth: 1 });
expect(expanded.map(r => r.chunk_id)).not.toContain(deniedChunkId);
const hydrated = await hydrateChunks(engine, [deniedChunkId]);
expect(hydrated).toEqual([]);
} finally {
await state.restore();
}
}
});
});
// ─────────────────────────────────────────────────────────────────
@@ -197,7 +344,9 @@ async function seedTwoSourcesWithSharedSymbol(engine: PGLiteEngine): Promise<voi
// Page B: contains parseMarkdown in source-b
const pageB = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, updated_at, created_at)
VALUES ('code/src/markdown-b.ts', 'source-b', 'markdown-b.ts', 'code', 'export function parseMarkdown(s: string) { return s; }', '{}'::jsonb, NOW(), NOW())
VALUES ('code/src/markdown-b.ts', 'source-b', 'markdown-b.ts', 'code', 'export function parseMarkdown(s: string) { return s; }',
'{"message_id":"<denied@example.com>","thread_id":"denied-thread","subject":"Denied exact subject"}'::jsonb,
NOW(), NOW())
RETURNING id`,
[],
);
@@ -231,4 +380,20 @@ async function seedTwoSourcesWithSharedSymbol(engine: PGLiteEngine): Promise<voi
VALUES ($1, 'callerInA', 'parseMarkdown', 'calls', 'source-a', '{}'::jsonb)`,
[callerChunk[0]!.id],
);
// A resolved direct edge crossing into source-b. The two-pass walk must
// discard this neighbor for both scalar sourceId and federated sourceIds.
const deniedChunk = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.source_id = 'source-b' AND cc.symbol_name_qualified = 'parseMarkdown' LIMIT 1`,
[],
);
await engine.addCodeEdges([{
from_chunk_id: callerChunk[0]!.id,
to_chunk_id: deniedChunk[0]!.id,
from_symbol_qualified: 'callerInA',
to_symbol_qualified: 'parseMarkdown',
edge_type: 'calls',
}]);
}
+135
View File
@@ -203,6 +203,53 @@ describe('PGLiteEngine: Search', () => {
await engine.upsertChunks('concepts/rag', [
{ chunk_index: 0, chunk_text: 'RAG combines retrieval with generation', chunk_source: 'compiled_truth' },
]);
await engine.putPage('mail/example', {
type: 'note', title: 'Launch message',
compiled_truth: 'Launch evidence for citation metadata.',
frontmatter: {
message_id: '<launch@example.com>',
thread_id: 'thread-123',
subject: 'Example launch subject',
},
});
await engine.upsertChunks('mail/example', [
{ chunk_index: 0, chunk_text: 'Launch evidence for citation metadata', chunk_source: 'compiled_truth' },
]);
await engine.putPage('notes/generated-title', {
type: 'note', title: 'Generated page title must stay a title',
compiled_truth: 'Non-email evidence for subject gating.',
frontmatter: {
subject: 'Frontmatter subject without an email identity',
thread_id: 'standalone-thread-id',
},
});
await engine.upsertChunks('notes/generated-title', [
{ chunk_index: 0, chunk_text: 'Non-email evidence for subject gating', chunk_source: 'compiled_truth' },
]);
await engine.putPage('mail/whitespace-message-id', {
type: 'note', title: 'Whitespace message id',
compiled_truth: 'Whitespace-only email identity evidence.',
frontmatter: {
message_id: ' \t\n ',
thread_id: 'thread-whitespace',
subject: 'Subject must remain gated',
},
});
await engine.upsertChunks('mail/whitespace-message-id', [
{ chunk_index: 0, chunk_text: 'Whitespace-only email identity evidence', chunk_source: 'compiled_truth' },
]);
await engine.putPage('notes/numeric-email-frontmatter', {
type: 'note', title: 'Numeric email frontmatter',
compiled_truth: 'Numeric email identity evidence.',
frontmatter: {
message_id: 12345,
thread_id: 67890,
subject: 98765,
},
});
await engine.upsertChunks('notes/numeric-email-frontmatter', [
{ chunk_index: 0, chunk_text: 'Numeric email identity evidence', chunk_source: 'compiled_truth' },
]);
});
test('searchKeyword returns results for matching term', async () => {
@@ -211,6 +258,34 @@ describe('PGLiteEngine: Search', () => {
expect(results[0].slug).toBe('companies/novamind');
});
test('searchKeyword projects email citation identifiers from frontmatter', async () => {
const results = await engine.searchKeyword('Launch evidence');
expect(results[0].message_id).toBe('<launch@example.com>');
expect(results[0].thread_id).toBe('thread-123');
expect(results[0].source_subject).toBe('Example launch subject');
});
test('searchKeyword never promotes a non-email title or subject to source_subject', async () => {
const results = await engine.searchKeyword('Non-email evidence');
expect(results[0].message_id).toBeUndefined();
expect(results[0].thread_id).toBe('standalone-thread-id');
expect(results[0].source_subject).toBeUndefined();
});
test('searchKeyword treats whitespace-only message_id as absent', async () => {
const results = await engine.searchKeyword('Whitespace-only email identity');
expect(results[0].message_id).toBeUndefined();
expect(results[0].thread_id).toBe('thread-whitespace');
expect(results[0].source_subject).toBeUndefined();
});
test('searchKeyword rejects non-string email citation frontmatter', async () => {
const results = await engine.searchKeyword('Numeric email identity');
expect(results[0].message_id).toBeUndefined();
expect(results[0].thread_id).toBeUndefined();
expect(results[0].source_subject).toBeUndefined();
});
test('searchKeyword returns empty for non-matching term', async () => {
const results = await engine.searchKeyword('xyznonexistent');
expect(results.length).toBe(0);
@@ -232,6 +307,42 @@ describe('PGLiteEngine: Search', () => {
const results = await engine.searchVector(fakeEmbedding);
expect(results.length).toBe(0);
});
test('searchVector carries email citation metadata through the outer CTE', async () => {
const embedding = new Float32Array(CHUNK_EMBED_DIM);
embedding[0] = 1;
await engine.upsertChunks('mail/example', [
{
chunk_index: 0,
chunk_text: 'Launch evidence for citation metadata',
chunk_source: 'compiled_truth',
embedding,
},
]);
const results = await engine.searchVector(embedding);
expect(results[0].message_id).toBe('<launch@example.com>');
expect(results[0].thread_id).toBe('thread-123');
expect(results[0].source_subject).toBe('Example launch subject');
});
test('searchVector treats whitespace-only message_id as absent', async () => {
const embedding = new Float32Array(CHUNK_EMBED_DIM);
embedding[1] = 1;
await engine.upsertChunks('mail/whitespace-message-id', [
{
chunk_index: 0,
chunk_text: 'Whitespace-only email identity evidence',
chunk_source: 'compiled_truth',
embedding,
},
]);
const results = await engine.searchVector(embedding);
expect(results[0].message_id).toBeUndefined();
expect(results[0].thread_id).toBe('thread-whitespace');
expect(results[0].source_subject).toBeUndefined();
});
});
// ─────────────────────────────────────────────────────────────────
@@ -275,6 +386,19 @@ describe('PGLiteEngine: CJK keyword fallback (v0.32.7)', () => {
await engine.upsertChunks('originals/english-essay', [
{ chunk_index: 0, chunk_text: 'NovaMind builds AI agents for enterprise', chunk_source: 'compiled_truth' },
]);
await engine.putPage('mail/cjk-example', {
type: 'note', title: 'Generated CJK page title',
compiled_truth: '郵件引用識別',
frontmatter: {
message_id: '<cjk@example.com>',
thread_id: 'thread-cjk',
subject: 'Example CJK email subject',
},
});
await engine.upsertChunks('mail/cjk-example', [
{ chunk_index: 0, chunk_text: '郵件引用識別', chunk_source: 'compiled_truth' },
]);
});
test('CJK query routes to LIKE branch and finds Chinese substring', async () => {
@@ -295,6 +419,17 @@ describe('PGLiteEngine: CJK keyword fallback (v0.32.7)', () => {
expect(results[0].slug).toBe('originals/korean-essay');
});
test('CJK keyword page and chunk paths project email citation metadata', async () => {
for (const result of [
(await engine.searchKeyword('郵件引用'))[0],
(await engine.searchKeywordChunks('郵件引用'))[0],
]) {
expect(result.message_id).toBe('<cjk@example.com>');
expect(result.thread_id).toBe('thread-cjk');
expect(result.source_subject).toBe('Example CJK email subject');
}
});
test('bigram ranking: 3-hit page outranks 1-hit page', async () => {
// Add another Chinese page with only ONE occurrence of 测试.
await engine.putPage('originals/chinese-one-hit', {
+5 -3
View File
@@ -38,10 +38,12 @@ describe('quarantine marker (hides)', () => {
expect(filterOutQuarantined(pages).map((p) => p.slug)).toEqual(['a', 'c']);
});
test('QUARANTINE_FILTER_FRAGMENT is a negated JSONB existence check on p', () => {
expect(QUARANTINE_FILTER_FRAGMENT).toContain("p.frontmatter");
test('QUARANTINE_FILTER_FRAGMENT requires object-shaped JSONB and excludes the marker', () => {
expect(QUARANTINE_FILTER_FRAGMENT).toContain('p.frontmatter');
expect(QUARANTINE_FILTER_FRAGMENT).toContain("jsonb_typeof");
expect(QUARANTINE_FILTER_FRAGMENT).toContain("= 'object'");
expect(QUARANTINE_FILTER_FRAGMENT).toContain("? 'quarantine'");
expect(QUARANTINE_FILTER_FRAGMENT.startsWith('NOT (')).toBe(true);
expect(QUARANTINE_FILTER_FRAGMENT).toContain('AND NOT');
});
test('quarantineFilterFragment(alias) parameterizes the page alias; constant is the p-instance', () => {
+8
View File
@@ -281,4 +281,12 @@ describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => {
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p.id IS NULL');
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p.generation <>');
});
test('visibility seal is unconditional and uses canonical page/source predicates', () => {
expect(CACHE_GATE_WHERE_CLAUSE).toContain('jsonb_array_elements(qc.results)');
expect(CACHE_GATE_WHERE_CLAUSE).toContain('LEFT JOIN sources s_visible');
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p_visible.deleted_at IS NULL');
expect(CACHE_GATE_WHERE_CLAUSE).toContain('NOT s_visible.archived');
expect(CACHE_GATE_WHERE_CLAUSE).toContain("COALESCE(p_visible.frontmatter, '{}'::jsonb) ? 'quarantine'");
});
});
+34 -10
View File
@@ -23,6 +23,7 @@ import { resolveHardExcludes } from '../src/core/search/source-boost.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
let engine: PGLiteEngine;
let visiblePageId: number;
const conservativeHash = knobsHash(resolveSearchMode({ mode: 'conservative' }));
const balancedHash = knobsHash(resolveSearchMode({ mode: 'balanced' }));
@@ -45,6 +46,14 @@ beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
const page = await engine.putPage('cache/knobs-visible-fixture', {
type: 'note',
title: 'Knobs cache fixture',
compiled_truth: 'visible knobs cache fixture',
timeline: '',
frontmatter: {},
});
visiblePageId = page.id;
});
afterAll(async () => {
@@ -80,10 +89,25 @@ const makeResults = (label: string, n: number): SearchResult[] =>
chunk_index: i,
type: 'note' as const,
chunk_source: 'compiled_truth' as const,
page_id: i + 1,
page_id: visiblePageId,
stale: false,
}));
async function storeCurrent(
cache: SemanticQueryCache,
...args: Parameters<SemanticQueryCache['store']>
): Promise<void> {
const [queryText, queryEmbedding, results, meta, opts = {}] = args;
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
await cache.store(queryText, queryEmbedding, results, meta, {
...opts,
maxGenerationAtSearchStart:
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
});
}
describe('cacheRowId is bifurcated by knobsHash', () => {
test('same (query, source) but different knobs → different row IDs', () => {
const id1 = cacheRowId('what is the meaning of life', 'default', conservativeHash);
@@ -116,7 +140,7 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
const tokenmaxResults = makeResults('tokenmax', 50);
// Write under tokenmax knobs.
await cache.store('what is the meaning of life', emb, tokenmaxResults, {
await storeCurrent(cache, 'what is the meaning of life', emb, tokenmaxResults, {
vector_enabled: true,
detail_resolved: null,
expansion_applied: true,
@@ -137,13 +161,13 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(2);
await cache.store('q', emb, makeResults('conservative', 10), {
await storeCurrent(cache, 'q', emb, makeResults('conservative', 10), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: conservativeHash });
await cache.store('q', emb, makeResults('balanced', 25), {
await storeCurrent(cache, 'q', emb, makeResults('balanced', 25), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: balancedHash });
await cache.store('q', emb, makeResults('tokenmax', 50), {
await storeCurrent(cache, 'q', emb, makeResults('tokenmax', 50), {
vector_enabled: true, detail_resolved: null, expansion_applied: true,
}, { knobsHash: tokenmaxHash });
@@ -194,11 +218,11 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(4);
await cache.store('q', emb, makeResults('first', 5), {
await storeCurrent(cache, 'q', emb, makeResults('first', 5), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: balancedHash });
await cache.store('q', emb, makeResults('second', 7), {
await storeCurrent(cache, 'q', emb, makeResults('second', 7), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: balancedHash });
@@ -217,7 +241,7 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(5);
await cache.store('q', emb, makeResults('no-mode', 3), {
await storeCurrent(cache, 'q', emb, makeResults('no-mode', 3), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
});
@@ -251,7 +275,7 @@ describe('hard-exclude cache isolation (#2825)', () => {
// Simulate a no-exclude process writing results that include a slug the
// excluding process must never see.
const leaky = makeResults('private', 5);
await cache.store('who is alice', emb, leaky, {
await storeCurrent(cache, 'who is alice', emb, leaky, {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: noEnvHash });
@@ -269,7 +293,7 @@ describe('hard-exclude cache isolation (#2825)', () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(7);
await cache.store('who is alice', emb, makeResults('filtered', 3), {
await storeCurrent(cache, 'who is alice', emb, makeResults('filtered', 3), {
vector_enabled: true, detail_resolved: null, expansion_applied: false,
}, { knobsHash: envExcludeHash });
+137 -14
View File
@@ -22,6 +22,7 @@ import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import type { SearchResult, HybridSearchMeta } from '../src/core/types.ts';
let engine: PGLiteEngine;
let visiblePageId: number;
// Build a stable, normalized embedding. PGLite ships pgvector with 1536-dim
// support (the default); a smaller test dim won't match the column. We
@@ -60,7 +61,7 @@ function makeOrthogonalEmbedding(seed: number, dim = DIM): Float32Array {
function makeResult(slug: string): SearchResult {
return {
slug,
page_id: 1,
page_id: visiblePageId,
title: `Title for ${slug}`,
type: 'concept',
chunk_text: `chunk text for ${slug}`,
@@ -79,6 +80,26 @@ const META: HybridSearchMeta = {
intent: 'general',
};
type StoreOpts = NonNullable<Parameters<SemanticQueryCache['store']>[4]>;
async function storeCurrent(
cache: SemanticQueryCache,
queryText: string,
queryEmbedding: Float32Array,
results: SearchResult[],
meta: HybridSearchMeta,
opts: StoreOpts = {},
): Promise<void> {
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
await cache.store(queryText, queryEmbedding, results, meta, {
...opts,
maxGenerationAtSearchStart:
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
});
}
beforeAll(async () => {
// v0.36.2.0: DEFAULT_EMBEDDING_DIMENSIONS flipped to 1280 (ZE Matryoshka).
// This test hardcodes DIM=1536 in its embeddings. If another test file in
@@ -95,6 +116,14 @@ beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
const page = await engine.putPage('cache/visible-fixture', {
type: 'note',
title: 'Visible cache fixture',
compiled_truth: 'visible cache fixture',
timeline: '',
frontmatter: {},
});
visiblePageId = page.id;
});
afterAll(async () => {
@@ -141,7 +170,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
const emb = makeEmbedding(1);
const results = [makeResult('a'), makeResult('b')];
await cache.store('what is foo', emb, results, META);
await storeCurrent(cache, 'what is foo', emb, results, META);
const hit = await cache.lookup(emb);
expect(hit.hit).toBe(true);
@@ -163,7 +192,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
mag = Math.sqrt(mag);
for (let i = 0; i < DIM; i++) near[i] /= mag;
await cache.store('what is foo', base, [makeResult('a')], META);
await storeCurrent(cache, 'what is foo', base, [makeResult('a')], META);
const hit = await cache.lookup(near);
expect(hit.hit).toBe(true);
@@ -174,7 +203,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
const cache = new SemanticQueryCache(engine);
const a = makeEmbedding(1);
const b = makeOrthogonalEmbedding(2);
await cache.store('q1', a, [makeResult('a')], META);
await storeCurrent(cache, 'q1', a, [makeResult('a')], META);
const hit = await cache.lookup(b);
expect(hit.hit).toBe(false);
});
@@ -184,7 +213,7 @@ describe('SemanticQueryCache \u2014 TTL', () => {
test('stale row (past TTL) is not returned', async () => {
const cache = new SemanticQueryCache(engine, { ttlSeconds: 1 });
const emb = makeEmbedding(42);
await cache.store('q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
await storeCurrent(cache, 'q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
// Manually rewind created_at to simulate expiration.
await engine.executeRaw(
@@ -195,34 +224,128 @@ describe('SemanticQueryCache \u2014 TTL', () => {
});
});
describe('SemanticQueryCache \u2014 source isolation', () => {
test('different source_id cannot read each other\u2019s rows', async () => {
describe('SemanticQueryCache source isolation', () => {
test('different source_id cannot read each others rows', async () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(7);
await cache.store('q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
await storeCurrent(cache, 'q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
const hitB = await cache.lookup(emb, { sourceId: 'src-B' });
expect(hitB.hit).toBe(false);
const hitA = await cache.lookup(emb, { sourceId: 'src-A' });
expect(hitA.hit).toBe(true);
});
test('archiving a source invalidates its cached result without a page write', async () => {
const sourceId = 'cache-archive-src';
await engine.executeRaw(
`INSERT INTO sources (id, name, archived)
VALUES ($1, $1, false)
ON CONFLICT (id) DO UPDATE SET archived = false`,
[sourceId],
);
const page = await engine.putPage(
'cache/archive-visibility',
{
type: 'note',
title: 'Archive visibility',
compiled_truth: 'archive visibility cache canary',
timeline: '',
frontmatter: {},
},
{ sourceId },
);
const result = { ...makeResult(page.slug), page_id: page.id, source_id: sourceId };
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(71);
await storeCurrent(cache, 'archive visibility', emb, [result], META, { sourceId });
expect((await cache.lookup(emb, { sourceId })).hit).toBe(true);
await engine.executeRaw(`UPDATE sources SET archived = true WHERE id = $1`, [sourceId]);
try {
expect((await cache.lookup(emb, { sourceId })).hit).toBe(false);
} finally {
await engine.executeRaw(`UPDATE sources SET archived = false WHERE id = $1`, [sourceId]);
}
});
test('a result hard-deleted before cache snapshot cannot be served', async () => {
const page = await engine.putPage('cache/deleted-before-store', {
type: 'note',
title: 'Deleted before store',
compiled_truth: 'private cache race canary',
timeline: '',
frontmatter: {},
});
const result = { ...makeResult(page.slug), page_id: page.id, source_id: 'default' };
await engine.executeRaw(`DELETE FROM pages WHERE id = $1`, [page.id]);
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(72);
await storeCurrent(cache, 'deleted before store', emb, [result], META);
expect((await cache.lookup(emb)).hit).toBe(false);
});
test('writeback without a pre-search generation fails closed', async () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(77);
await cache.store('missing generation', emb, [makeResult('a')], META);
expect((await cache.lookup(emb)).hit).toBe(false);
expect((await cache.stats()).total_rows).toBe(0);
});
test('a concurrent page mutation prevents stale result writeback', async () => {
const clock = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
const generationAtSearchStart = Number(clock[0]?.v ?? 0);
const staleResult = { ...makeResult('cache/visible-fixture'), page_id: visiblePageId };
await engine.executeRaw(
`UPDATE pages SET compiled_truth = 'redacted after search' WHERE id = $1`,
[visiblePageId],
);
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(73);
await storeCurrent(cache, 'concurrent mutation', emb, [staleResult], META, {
maxGenerationAtSearchStart: generationAtSearchStart,
});
expect((await cache.lookup(emb)).hit).toBe(false);
});
});
describe('SemanticQueryCache \u2014 management', () => {
test('clear() wipes all rows', async () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(9);
await cache.store('q1', emb, [makeResult('a')], META);
await cache.store('q2', makeEmbedding(10), [makeResult('b')], META);
await storeCurrent(cache, 'q1', emb, [makeResult('a')], META);
await storeCurrent(cache, 'q2', makeEmbedding(10), [makeResult('b')], META);
const removed = await cache.clear();
expect(removed).toBeGreaterThanOrEqual(2);
const stats = await cache.stats();
expect(stats.total_rows).toBe(0);
});
test('source-scoped clear removes typed scalar and federated scopes containing the source', async () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(74);
await storeCurrent(cache, 'scalar-a', emb, [makeResult('a')], META, { sourceId: '["scalar","source-a"]' });
await storeCurrent(cache, 'set-ab', makeEmbedding(75), [makeResult('ab')], META, { sourceId: '["set","source-a","source-b"]' });
await storeCurrent(cache, 'scalar-b', makeEmbedding(76), [makeResult('b')], META, { sourceId: '["scalar","source-b"]' });
expect(await cache.clear({ sourceId: 'source-a' })).toBe(2);
const rows = await engine.executeRaw<{ source_id: string }>(`SELECT source_id FROM query_cache ORDER BY source_id`);
expect(rows.map(r => r.source_id)).toEqual(['["scalar","source-b"]']);
});
test('prune() deletes only stale rows', async () => {
const cache = new SemanticQueryCache(engine);
await cache.store('fresh', makeEmbedding(11), [makeResult('a')], META);
await cache.store('stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
await storeCurrent(cache, 'fresh', makeEmbedding(11), [makeResult('a')], META);
await storeCurrent(cache, 'stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
await engine.executeRaw(
`UPDATE query_cache SET created_at = now() - interval '10 seconds' WHERE query_text = 'stale'`,
);
@@ -236,7 +359,7 @@ describe('SemanticQueryCache \u2014 management', () => {
test('stats() reports fresh / stale / total / hit counters', async () => {
const cache = new SemanticQueryCache(engine);
const emb = makeEmbedding(13);
await cache.store('q', emb, [makeResult('a')], META);
await storeCurrent(cache, 'q', emb, [makeResult('a')], META);
await cache.lookup(emb); // bump hit
// Hit bump is async/fire-and-forget; give it a moment to land.
await new Promise(r => setTimeout(r, 50));
@@ -252,7 +375,7 @@ describe('SemanticQueryCache \u2014 disabled', () => {
test('disabled cache is a pure no-op on lookup', async () => {
const cache = new SemanticQueryCache(engine, { enabled: false });
const emb = makeEmbedding(99);
await cache.store('q', emb, [makeResult('a')], META);
await storeCurrent(cache, 'q', emb, [makeResult('a')], META);
// Even after a store call, lookup must miss because enabled=false.
const hit = await cache.lookup(emb);
expect(hit.hit).toBe(false);
+42
View File
@@ -36,6 +36,7 @@ beforeAll(async () => {
['people/employee-c', 'person', 'Employee C'],
['people/mentioner', 'person', 'Mentioner'],
['people/deleted-investor', 'person', 'Deleted Investor'],
['people/quarantined-investor', 'person', 'Quarantined Investor'],
];
for (const [slug, type, title] of pages) {
await eng.putPage(slug, { type: type as 'company' | 'person', title, compiled_truth: `${title} body`, timeline: '' });
@@ -50,11 +51,39 @@ beforeAll(async () => {
await eng.addLink('people/employee-c', 'companies/widget-co', '', 'works_at', 'manual');
await eng.addLink('people/mentioner', 'companies/widget-co', '', 'mentions', 'mentions');
await eng.addLink('people/deleted-investor', 'companies/widget-co', '', 'invested_in', 'manual');
await eng.addLink('people/quarantined-investor', 'companies/widget-co', '', 'invested_in', 'manual');
// investor-a also invested in other-co → widget-co and other-co connect via investor-a.
await eng.addLink('people/investor-a', 'companies/other-co', '', 'invested_in', 'manual');
// Soft-delete one investor; it must never surface.
await eng.executeRaw(`UPDATE pages SET deleted_at = now() WHERE slug = $1`, ['people/deleted-investor']);
await eng.executeRaw(
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb WHERE slug = $1`,
['people/quarantined-investor'],
);
// A complete graph inside an archived source must be invisible even when
// the caller explicitly scopes the fanout to that source.
await eng.executeRaw(
`INSERT INTO sources (id, name, archived, created_at)
VALUES ('archived-rel', 'archived-rel', false, NOW())`,
[],
);
await eng.putPage(
'companies/archived-widget',
{ type: 'company', title: 'Archived Widget', compiled_truth: 'Hidden company.', timeline: '' },
{ sourceId: 'archived-rel' },
);
await eng.putPage(
'people/archived-investor',
{ type: 'person', title: 'Archived Investor', compiled_truth: 'Hidden person.', timeline: '' },
{ sourceId: 'archived-rel' },
);
await eng.addLink(
'people/archived-investor', 'companies/archived-widget', '', 'invested_in', 'manual', undefined, undefined,
{ fromSourceId: 'archived-rel', toSourceId: 'archived-rel' },
);
await eng.executeRaw(`UPDATE sources SET archived = true WHERE id = 'archived-rel'`, []);
}, 60_000);
afterAll(async () => {
@@ -77,6 +106,19 @@ describe('relationalFanout', () => {
expect(rows.map(r => r.slug)).not.toContain('people/deleted-investor');
});
test('quarantined pages are excluded', async () => {
const rows = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', linkTypes: ['invested_in'] });
expect(rows.map(r => r.slug)).not.toContain('people/quarantined-investor');
});
test('archived sources are excluded', async () => {
const rows = await eng.relationalFanout(
['companies/archived-widget'],
{ sourceId: 'archived-rel', direction: 'in', linkTypes: ['invested_in'] },
);
expect(rows).toEqual([]);
});
test('mentions excluded by default, included on opt-in', async () => {
const off = await eng.relationalFanout(['companies/widget-co'], { direction: 'in' });
expect(off.map(r => r.slug)).not.toContain('people/mentioner');
+47 -1
View File
@@ -23,12 +23,34 @@ beforeAll(async () => {
await eng.putPage('companies/widget-co', { type: 'company', title: 'Widget Co', compiled_truth: 'A payments company.', timeline: '' });
// The investor's body deliberately NEVER mentions Widget Co — only the edge connects them.
await eng.putPage('people/alice-example', { type: 'person', title: 'Alice Example', compiled_truth: 'Alice is a seed-stage investor based in Lisbon.', timeline: '' });
await eng.putPage('people/alice-example', {
type: 'person',
title: 'Alice Example',
compiled_truth: 'Alice is a seed-stage investor based in Lisbon.',
timeline: '',
frontmatter: {
message_id: '<alice-investor@example.com>',
thread_id: 'thread-alice-investor',
subject: 'Investor profile update',
},
});
await eng.upsertChunks('people/alice-example', [{
chunk_index: 0, chunk_text: 'Alice is a seed-stage investor based in Lisbon.',
chunk_source: 'compiled_truth', embedding: new Float32Array(dim), token_count: 8,
}] satisfies ChunkInput[]);
await eng.addLink('people/alice-example', 'companies/widget-co', '', 'invested_in', 'manual');
await eng.putPage('people/numeric-example', {
type: 'person',
title: 'Numeric Example',
compiled_truth: 'A second seed-stage investor.',
timeline: '',
frontmatter: { message_id: 12345, thread_id: 67890, subject: 98765 },
});
await eng.addLink('people/numeric-example', 'companies/widget-co', '', 'invested_in', 'manual');
await eng.putPage('people/hydrate-quarantined', {
type: 'person', title: 'Hydrate Quarantined', compiled_truth: 'Must remain hidden.', timeline: '',
frontmatter: { quarantine: true, message_id: '<hidden@example.com>', subject: 'Hidden exact subject' },
});
}, 60_000);
afterAll(async () => { await eng.disconnect(); });
@@ -43,6 +65,15 @@ describe('buildRelationalArm', () => {
expect(alice!.relational_seed).toBe('companies/widget-co');
// chunk-backed page → reinforces a REAL chunk id (not synthetic 0).
expect(alice!.chunk_id).toBeGreaterThan(0);
expect(alice!.message_id).toBe('<alice-investor@example.com>');
expect(alice!.thread_id).toBe('thread-alice-investor');
expect(alice!.source_subject).toBe('Investor profile update');
const malformed = list.find(r => r.slug === 'people/numeric-example');
expect(malformed).toBeDefined();
expect(malformed!.message_id).toBeUndefined();
expect(malformed!.thread_id).toBeUndefined();
expect(malformed!.source_subject).toBeUndefined();
});
test('non-relational query is a pure no-op', async () => {
@@ -57,6 +88,21 @@ describe('buildRelationalArm', () => {
expect(list).toEqual([]);
});
test('hydrate rejects a quarantined row even if fanout returns it', async () => {
const original = eng.relationalFanout.bind(eng);
eng.relationalFanout = async () => [{
source_id: 'default', slug: 'people/hydrate-quarantined', hop: 1,
edge_count: 1, via_link_types: ['invested_in'],
path: ['companies/widget-co', 'people/hydrate-quarantined'], canonical_chunk_id: null,
}];
try {
const list = await buildRelationalArm(eng, 'who invested in widget-co');
expect(list).toEqual([]);
} finally {
eng.relationalFanout = original;
}
});
test('fail-open: fanout error returns [] + errored meta, never throws', async () => {
const original = eng.relationalFanout.bind(eng);
let captured: { errored?: boolean } = {};
+2 -2
View File
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
});
describe('KNOBS_HASH_VERSION', () => {
it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => {
expect(KNOBS_HASH_VERSION).toBe(12);
it('is 13 (12→13 invalidates cached rows without email citation metadata)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
});
});
+6 -3
View File
@@ -410,7 +410,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
// #2825: bumped 11→12 to fold the resolved hard-exclude prefix list
// (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across
// processes.
expect(KNOBS_HASH_VERSION).toBe(12);
// Email citation metadata projection: bumped 12→13 because cached
// SearchResult rows from older versions lack message_id, thread_id, and
// Message-ID-gated source_subject.
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
@@ -575,8 +578,8 @@ describe('v0.40.4 — graph_signals knob', () => {
});
describe('v0.42.3.0 — autocut knobs', () => {
test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => {
expect(KNOBS_HASH_VERSION).toBe(12);
test('KNOBS_HASH_VERSION is 13 (12→13 email citation result-schema projection)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
+102 -1
View File
@@ -27,7 +27,18 @@ function res(slug: string, score: number, title = slug): SearchResult {
describe('applyAliasHop', () => {
test('injects an absent canonical when the query matches its alias', async () => {
await engine.putPage('projects/mingtang', { type: 'note', title: 'The Mingtang', compiled_truth: 'Indoor Greek amphitheater.' });
await engine.putPage('projects/mingtang', {
type: 'note',
title: 'The Mingtang',
compiled_truth: 'Indoor Greek amphitheater.',
frontmatter: {
message_id: '<mingtang@example.com>',
thread_id: 'thread-mingtang',
subject: 'Mingtang project update',
// Internal SQL projection aliases are not trusted frontmatter fields.
source_subject: 'Spoofed projection alias',
},
});
await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']);
const organic = [res('notes/unrelated', 0.5)];
@@ -38,6 +49,85 @@ describe('applyAliasHop', () => {
expect(hit!.alias_hit).toBe(true);
expect(out[0].slug).toBe('projects/mingtang'); // injected at top-of-organic + ε
expect(hit!.score).toBeGreaterThan(0.5);
expect(hit!.message_id).toBe('<mingtang@example.com>');
expect(hit!.thread_id).toBe('thread-mingtang');
expect(hit!.source_subject).toBe('Mingtang project update');
});
test('raw frontmatter cannot forge source_subject without allowlisted subject', async () => {
await engine.putPage('mail/forged-subject', {
type: 'note',
title: 'Forged subject',
compiled_truth: 'Evidence.',
frontmatter: {
message_id: '<forged@example.com>',
source_subject: 'Forged projection alias',
},
});
await engine.setPageAliases('mail/forged-subject', 'default', ['forged mail']);
const out = await applyAliasHop(engine, [], 'forged mail', { sourceId: 'default' });
expect(out).toHaveLength(1);
expect(out[0]!.message_id).toBe('<forged@example.com>');
expect(out[0]!.source_subject).toBeUndefined();
});
test('does not inject quarantined alias pages', async () => {
await engine.putPage('notes/quarantined-alias', {
type: 'note', title: 'Quarantined alias', compiled_truth: 'Hidden.',
frontmatter: { quarantine: true, message_id: '<hidden@example.com>', subject: 'Hidden' },
});
await engine.setPageAliases('notes/quarantined-alias', 'default', ['hidden alias']);
const out = await applyAliasHop(engine, [], 'hidden alias', { sourceId: 'default' });
expect(out).toEqual([]);
});
test('does not inject aliases whose top-level frontmatter is an encoded JSON string', async () => {
const encoded = JSON.stringify({
message_id: '<encoded@example.com>',
thread_id: 'encoded-thread',
subject: 'Encoded subject',
});
await engine.executeRaw(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
VALUES ('mail/encoded-frontmatter', 'default', 'Encoded frontmatter', 'note', 'Hidden.', to_jsonb($1::text), NOW(), NOW())`,
[encoded],
);
await engine.setPageAliases('mail/encoded-frontmatter', 'default', ['encoded mail']);
const out = await applyAliasHop(engine, [], 'encoded mail', { sourceId: 'default' });
expect(out).toEqual([]);
});
test('encoded-string quarantine markers remain hidden from alias retrieval', async () => {
const encoded = JSON.stringify({ quarantine: true });
await engine.executeRaw(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
VALUES ('notes/encoded-quarantine', 'default', 'Encoded quarantine', 'note', 'Hidden.', to_jsonb($1::text), NOW(), NOW())`,
[encoded],
);
await engine.setPageAliases('notes/encoded-quarantine', 'default', ['encoded hidden']);
const out = await applyAliasHop(engine, [], 'encoded hidden', { sourceId: 'default' });
expect(out).toEqual([]);
});
test('does not inject aliases from archived sources', async () => {
await engine.executeRaw(
`INSERT INTO sources (id, name, archived, created_at)
VALUES ('archived-alias', 'archived-alias', true, NOW())`,
[],
);
await engine.executeRaw(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
VALUES ('notes/archived-alias', 'archived-alias', 'Archived alias', 'note', 'Hidden.', '{}', NOW(), NOW())`,
[],
);
await engine.setPageAliases('notes/archived-alias', 'archived-alias', ['archived alias']);
const out = await applyAliasHop(engine, [], 'archived alias', { sourceId: 'archived-alias' });
expect(out).toEqual([]);
});
test('romanization/CJK alias also resolves', async () => {
@@ -61,6 +151,17 @@ describe('applyAliasHop', () => {
test('P0 source-isolation: alias hop boosts only the aliased source, not the same slug in another source', async () => {
// The alias belongs to the src-b page only. Two same-slug results, different
// sources, both in the organic set. The hop must boost ONLY src-b's row.
await engine.executeRaw(
`INSERT INTO sources (id, name, created_at) VALUES
('src-a', 'src-a', NOW()), ('src-b', 'src-b', NOW())`,
[],
);
await engine.putPage(
'shared/page', { type: 'note', title: 'Shared A', compiled_truth: 'A' }, { sourceId: 'src-a' },
);
await engine.putPage(
'shared/page', { type: 'note', title: 'Shared B', compiled_truth: 'B' }, { sourceId: 'src-b' },
);
await engine.setPageAliases('shared/page', 'src-b', ['only in b']);
const organic = [
{ slug: 'shared/page', source_id: 'src-a', score: 0.5 } as unknown as SearchResult,
@@ -0,0 +1,81 @@
/**
* Fail-open regression for the typed cache scope key (takeover of #2875).
*
* `cacheScopeKey` rejects forged/malformed source ids (typed-key contract,
* test/cache-scope-key.test.ts). That throw is correct for direct callers,
* but inside `hybridSearchCached` an invalid scope id must degrade to
* "skip the semantic cache" the cache must never break the search hot
* path. Before the fix, the key was computed inline at the cache-lookup
* call site (outside any catch), so a forged sourceId that reached the
* cached path rejected the whole search once a query embedding was
* available.
*
* Serial file: mock.module leaks across files in a shared shard process.
*/
import { afterAll, beforeAll, describe, expect, test, mock } from 'bun:test';
const makeEmbedding = (): Float32Array => {
const arr = new Float32Array(1536);
for (let i = 0; i < 1536; i++) arr[i] = Math.sin(1 + i * 0.001);
let norm = 0;
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
norm = Math.sqrt(norm);
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
return arr;
};
// Mock the embedding module BEFORE importing hybrid.ts so the cache-lookup
// embed succeeds and the lookup call site is actually reached (the failure
// mode under test only fired once a query embedding existed).
mock.module('../../src/core/embedding.ts', () => ({
embed: async () => makeEmbedding(),
embedQuery: async () => makeEmbedding(),
}));
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
resetGateway();
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
env: { OPENAI_API_KEY: 'sk-fake' },
});
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
await engine.putPage('failopen/fixture', {
type: 'note',
title: 'Fail-open fixture',
compiled_truth: 'fail open fixture content',
timeline: '',
frontmatter: {},
});
});
afterAll(async () => {
await engine.disconnect();
resetGateway();
});
describe('hybridSearchCached cache-scope fail-open', () => {
test('a forged sourceId skips the cache instead of breaking the search', async () => {
const { hybridSearchCached } = await import('../../src/core/search/hybrid.ts');
// '__set__:a,b' is a forged legacy set-encoding — cacheScopeKey throws
// on it. The search itself must still resolve (cache silently skipped).
const results = await hybridSearchCached(engine, 'fixture', {
sourceId: '__set__:a,b',
useCache: true,
limit: 5,
});
expect(Array.isArray(results)).toBe(true);
// Nothing may have been written into the cache under a forged scope.
const rows = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM query_cache`,
);
expect(rows[0]?.n ?? 0).toBe(0);
});
});
@@ -19,7 +19,11 @@
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { hybridSearch } from '../../src/core/search/hybrid.ts';
import {
awaitPendingSearchCacheWrites,
hybridSearch,
hybridSearchCached,
} from '../../src/core/search/hybrid.ts';
import {
configureGateway,
resetGateway,
@@ -69,16 +73,31 @@ beforeAll(async () => {
// early-returns BEFORE applyReranker, so a setup that lacks embedding
// would never exercise the reranker integration.
//
// searchVector returns empty lists because chunks have NULL embeddings;
// that's fine — vectorLists is `[[]]` (length 1, not 0), so the
// keyword-only branch is skipped and the main path runs RRF + dedup +
// reranker + budget.
// Most chunks have NULL embeddings; the email fixture below carries one so
// the main path runs vector fusion + RRF + dedup + reranker + budget.
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: DIMS,
env: { OPENAI_API_KEY: 'sk-test' },
});
stubEmbeddings();
await engine.putPage('mail/vector-first', {
type: 'note',
title: 'Vector-first email',
compiled_truth: 'vector first duplicate metadata evidence',
frontmatter: {
message_id: '<vector-first@example.com>',
thread_id: 'thread-vector-first',
subject: 'Vector-first exact subject',
},
});
await engine.upsertChunks('mail/vector-first', [{
chunk_index: 0,
chunk_text: 'vector first duplicate metadata evidence',
chunk_source: 'compiled_truth',
embedding: Float32Array.from(FAKE_EMB),
}]);
});
afterAll(async () => {
@@ -105,6 +124,70 @@ describe('hybridSearch — reranker disabled (pass-through)', () => {
});
});
describe('hybridSearchCached — email metadata through vector-first fusion', () => {
test('fresh cache miss preserves metadata through vector-first RRF duplicate handling', async () => {
await engine.executeRaw(`DELETE FROM query_cache`);
const cacheStatuses: string[] = [];
const out = await hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
limit: 10,
useCache: true,
autocut: false,
graph_signals: false,
onMeta: (meta) => {
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
},
});
expect(cacheStatuses.at(-1)).toBe('miss');
const matches = out.filter(r => r.slug === 'mail/vector-first');
expect(matches).toHaveLength(1);
expect(matches[0].message_id).toBe('<vector-first@example.com>');
expect(matches[0].thread_id).toBe('thread-vector-first');
expect(matches[0].source_subject).toBe('Vector-first exact subject');
await awaitPendingSearchCacheWrites();
const cached = await hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
limit: 10,
useCache: true,
autocut: false,
graph_signals: false,
onMeta: (meta) => {
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
},
});
expect(cacheStatuses.at(-1)).toBe('hit');
const cachedMatch = cached.find(r => r.slug === 'mail/vector-first');
expect(cachedMatch?.message_id).toBe('<vector-first@example.com>');
expect(cachedMatch?.thread_id).toBe('thread-vector-first');
expect(cachedMatch?.source_subject).toBe('Vector-first exact subject');
});
test('the query-op request shape (expandFn wired) still uses the semantic cache', async () => {
// Regression: operations.ts always passes expandFn on the default `query`
// op. If expandFn were treated as cache-unsafe, cacheStatus would be
// 'disabled' here and the flagship op would never hit the cache.
await engine.executeRaw(`DELETE FROM query_cache`);
const cacheStatuses: string[] = [];
const run = () => hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
limit: 10,
useCache: true,
autocut: false,
graph_signals: false,
expansion: false,
expandFn: async (q: string) => [q],
onMeta: (meta) => {
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
},
});
await run();
expect(cacheStatuses.at(-1)).toBe('miss');
await awaitPendingSearchCacheWrites();
await run();
expect(cacheStatuses.at(-1)).toBe('hit');
});
});
describe('hybridSearch — reranker enabled (reorder)', () => {
test('rerankerFn receives a non-empty document list', async () => {
let receivedDocs: string[] = [];
+4 -2
View File
@@ -44,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs {
}
describe('KNOBS_HASH_VERSION + version invariants', () => {
test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => {
test('version is 13 (…; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825; 12→13 email citation DTO)', () => {
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
@@ -64,7 +64,9 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
// pre-fix document-side query vectors must not be served.
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
expect(KNOBS_HASH_VERSION).toBe(12);
// Email citation metadata projection: 12→13 so stale cached SearchResult
// DTOs without message_id/thread_id/source_subject cannot survive.
expect(KNOBS_HASH_VERSION).toBe(13);
});
test('hash is 16 hex chars regardless of reranker config', () => {
+24
View File
@@ -27,8 +27,25 @@ beforeEach(async () => {
const slugsOf = (m: Map<string, Array<{ slug: string; source_id: string }>>, k: string) =>
(m.get(k) ?? []).map(r => r.slug).sort();
async function seedPage(slug: string, sourceId = 'default'): Promise<void> {
if (sourceId !== 'default') {
await engine.executeRaw(
`INSERT INTO sources (id, name, archived)
VALUES ($1, $1, false)
ON CONFLICT (id) DO NOTHING`,
[sourceId],
);
}
await engine.putPage(
slug,
{ title: slug, type: 'note', compiled_truth: `# ${slug}`, timeline: '', frontmatter: {} },
{ sourceId },
);
}
describe('setPageAliases + resolveAliases', () => {
test('write then read maps alias_norm → slug', async () => {
await seedPage('projects/mingtang');
await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']);
const m = await engine.resolveAliases(['hall of light'], { sourceId: 'default' });
expect(slugsOf(m, 'hall of light')).toEqual(['projects/mingtang']);
@@ -36,6 +53,8 @@ describe('setPageAliases + resolveAliases', () => {
});
test('collision: two pages claim the same alias → both returned', async () => {
await seedPage('projects/mingtang');
await seedPage('projects/other-hall');
await engine.setPageAliases('projects/mingtang', 'default', ['the hall']);
await engine.setPageAliases('projects/other-hall', 'default', ['the hall']);
const m = await engine.resolveAliases(['the hall'], { sourceId: 'default' });
@@ -43,6 +62,8 @@ describe('setPageAliases + resolveAliases', () => {
});
test('source-scoped: alias in source A not returned for source B', async () => {
await seedPage('a/page', 'src-a');
await seedPage('b/page', 'src-b');
await engine.setPageAliases('a/page', 'src-a', ['shared name']);
await engine.setPageAliases('b/page', 'src-b', ['shared name']);
const aOnly = await engine.resolveAliases(['shared name'], { sourceId: 'src-a' });
@@ -55,6 +76,7 @@ describe('setPageAliases + resolveAliases', () => {
});
test('rewrite replaces the prior alias set (delete + insert)', async () => {
await seedPage('p/x');
await engine.setPageAliases('p/x', 'default', ['old name']);
await engine.setPageAliases('p/x', 'default', ['new name']);
const oldM = await engine.resolveAliases(['old name'], { sourceId: 'default' });
@@ -64,6 +86,7 @@ describe('setPageAliases + resolveAliases', () => {
});
test('empty alias set clears the page', async () => {
await seedPage('p/x');
await engine.setPageAliases('p/x', 'default', ['temp']);
await engine.setPageAliases('p/x', 'default', []);
const m = await engine.resolveAliases(['temp'], { sourceId: 'default' });
@@ -76,6 +99,7 @@ describe('setPageAliases + resolveAliases', () => {
});
test('idempotent re-write does not duplicate (unique triple)', async () => {
await seedPage('p/x');
await engine.setPageAliases('p/x', 'default', ['name', 'name']);
const m = await engine.resolveAliases(['name'], { sourceId: 'default' });
expect(slugsOf(m, 'name')).toEqual(['p/x']);
+1 -1
View File
@@ -305,7 +305,7 @@ describe('buildVisibilityClause (v0.26.5)', () => {
test('uses the supplied aliases verbatim', () => {
expect(buildVisibilityClause('pp', 'src')).toBe(
"AND pp.deleted_at IS NULL AND NOT src.archived AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine')",
"AND pp.deleted_at IS NULL AND NOT src.archived AND (jsonb_typeof(COALESCE(pp.frontmatter, '{}'::jsonb)) = 'object' AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine'))",
);
});
+22
View File
@@ -45,6 +45,11 @@ describe('Layer 7 (A2) — expandAnchors', () => {
title: 'src/b.ts (typescript)',
compiled_truth: 'export function b() { return c(); }',
timeline: '',
frontmatter: {
message_id: '<code-review@example.com>',
thread_id: 'thread-code-review',
subject: 'Code review notes',
},
});
await engine.upsertChunks('src-b-ts', [{
chunk_index: 0,
@@ -85,6 +90,9 @@ describe('Layer 7 (A2) — expandAnchors', () => {
{ from_chunk_id: chunkB, to_chunk_id: null,
from_symbol_qualified: 'b', to_symbol_qualified: 'c',
edge_type: 'calls' },
{ from_chunk_id: chunkA, to_chunk_id: chunkB,
from_symbol_qualified: 'a', to_symbol_qualified: 'b',
edge_type: 'imports' },
]);
});
@@ -121,6 +129,16 @@ describe('Layer 7 (A2) — expandAnchors', () => {
expect(neighbor!.score).toBeCloseTo(0.5, 2);
});
test('walkDepth=1 follows an incoming resolved edge to the opposite endpoint', async () => {
const anchors = [{
slug: 'src-b-ts', page_id: 0, title: 'b', type: 'code',
chunk_text: '', chunk_source: 'compiled_truth', chunk_id: chunkB,
chunk_index: 0, score: 1.0, stale: false, source_id: 'default',
} as never];
const expanded = await expandAnchors(engine, anchors, { walkDepth: 1 });
expect(expanded.map(e => e.chunk_id)).toContain(chunkA);
});
test('walkDepth=2 reaches grandchildren', async () => {
const anchors = [{
slug: 'src-a-ts', page_id: 0, title: 'a', type: 'code',
@@ -161,6 +179,10 @@ describe('Layer 7 (A2) — expandAnchors', () => {
const slugs = rows.map(r => r.slug);
expect(slugs).toContain('src-b-ts');
expect(slugs).toContain('src-c-ts');
const b = rows.find(r => r.slug === 'src-b-ts');
expect(b?.message_id).toBe('<code-review@example.com>');
expect(b?.thread_id).toBe('thread-code-review');
expect(b?.source_subject).toBe('Code review notes');
});
test('hydrateChunks with empty array returns []', async () => {
+49 -1
View File
@@ -48,13 +48,23 @@ describe('contentHash', () => {
});
describe('rowToPage', () => {
test('parses string frontmatter', () => {
test('rejects encoded-string frontmatter instead of double-decoding it', () => {
const page = rowToPage({
id: 1, slug: 'test', type: 'concept', title: 'Test',
compiled_truth: 'body', timeline: '',
frontmatter: '{"key":"val"}',
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
});
expect(page.frontmatter).toEqual({});
});
test('preserves object-shaped JSONB frontmatter', () => {
const page = rowToPage({
id: 1, slug: 'test', type: 'concept', title: 'Test',
compiled_truth: 'body', timeline: '',
frontmatter: { key: 'val' },
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
});
expect(page.frontmatter.key).toBe('val');
});
@@ -183,4 +193,42 @@ describe('rowToSearchResult', () => {
expect(typeof r.score).toBe('number');
expect(r.score).toBe(0.95);
});
test('projects allowlisted email identifiers when present', () => {
const r = rowToSearchResult({
slug: 'mail/example', page_id: 2, title: 'Example email', type: 'note',
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 3, chunk_index: 0,
score: 0.9, stale: false,
message_id: '<message@example.com>', thread_id: 'abc123',
source_subject: 'Example launch subject',
});
expect(r.message_id).toBe('<message@example.com>');
expect(r.thread_id).toBe('abc123');
expect(r.source_subject).toBe('Example launch subject');
});
test('does not invent email identifiers when projections are absent', () => {
const r = rowToSearchResult({
slug: 'concept/example', page_id: 3, title: 'Example', type: 'concept',
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 4, chunk_index: 0,
score: 0.8, stale: false,
source_subject: 'Generated title must not become an email subject',
});
expect(r.message_id).toBeUndefined();
expect(r.thread_id).toBeUndefined();
expect(r.source_subject).toBeUndefined();
});
test('whitespace-only message_id does not project or authorize source_subject', () => {
const r = rowToSearchResult({
slug: 'mail/whitespace-id', page_id: 4, title: 'Whitespace ID', type: 'note',
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 5, chunk_index: 0,
score: 0.7, stale: false,
message_id: ' \t\n ', thread_id: 'thread-whitespace',
source_subject: 'Must remain gated',
});
expect(r.message_id).toBeUndefined();
expect(r.thread_id).toBe('thread-whitespace');
expect(r.source_subject).toBeUndefined();
});
});