mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 17:02:19 +00:00
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3cfd0c4b42 | ||
|
|
1f10561eb1 | ||
|
|
3381dd7658 |
@@ -42,7 +42,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/reindex-search-vector.ts` — `gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`.
|
||||
- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path.
|
||||
- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToSearchResult` projects email `message_id` / `thread_id` metadata and exposes `source_subject` only when a non-empty Message-ID proves the page is an email, so generated page titles never become authoritative email subjects. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise<boolean>` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` (#1972) — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the #2084 contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack.
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`). Copies the complete source catalog FIRST (`copyMigrationSources` — every `sources` row incl. archived rows and sync/routing metadata, `ON CONFLICT (id) DO UPDATE`, `default` ordered first) so every page write has a valid `pages.source_id` FK parent and the target preserves per-source behavior; pages copy afterward, tracked in the resume manifest by composite `(source_id, slug)` key. The resume manifest is target-aware: `migrationTargetId(config)` hashes `(engine, locator)` (`database_url` for Postgres, resolved `database_path` for PGLite) and `manifestMatchesTarget` requires `schema_version === 2` plus a matching `target_id` — a legacy engine-only manifest, or one from a DIFFERENT target of the same engine kind, starts fresh instead of skipping "completed" pages the new target never received. Pinned by `test/migrate-engine-resume.test.ts` (manifest identity) + `test/e2e/migrate-engine-sources-postgres.test.ts` (source catalog lands before overlapping-slug pages, PGLite → real Postgres).
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`.
|
||||
|
||||
@@ -1636,6 +1636,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
@@ -1880,6 +1884,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
${scoreExpr} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
@@ -1905,6 +1913,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
${scoreExpr} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
@@ -2001,6 +2013,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
@@ -2113,6 +2129,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
@@ -2135,6 +2155,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
SELECT
|
||||
bpp.slug, bpp.page_id, bpp.title, bpp.type, bpp.source_id,
|
||||
bpp.effective_date, bpp.effective_date_source,
|
||||
bpp.message_id, bpp.thread_id, bpp.source_subject,
|
||||
bpp.chunk_id, bpp.chunk_index, bpp.chunk_text, bpp.chunk_source,
|
||||
bpp.score,
|
||||
CASE WHEN bpp.updated_at < (
|
||||
@@ -3186,14 +3207,17 @@ export class PGLiteEngine implements BrainEngine {
|
||||
typeFilter = `AND l.link_type = ANY($${params.length}::text[])`;
|
||||
}
|
||||
const mentionsFilter = opts?.includeMentions ? '' : `AND l.link_source IS DISTINCT FROM 'mentions'`;
|
||||
const seedVisibility = buildVisibilityClause('p', 's');
|
||||
const stepVisibility = buildVisibilityClause('p2', 's2');
|
||||
|
||||
const recurStep =
|
||||
direction === 'out'
|
||||
? `JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id`
|
||||
? `JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id JOIN sources s2 ON s2.id = p2.source_id`
|
||||
: direction === 'in'
|
||||
? `JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id`
|
||||
? `JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id JOIN sources s2 ON s2.id = p2.source_id`
|
||||
: `JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
|
||||
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`;
|
||||
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
|
||||
JOIN sources s2 ON s2.id = p2.source_id`;
|
||||
|
||||
const sql = `
|
||||
WITH RECURSIVE walk AS (
|
||||
@@ -3201,7 +3225,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
ARRAY[p.id] AS visited, ARRAY[p.slug] AS path,
|
||||
p.source_id AS seed_source, NULL::text AS last_link_type
|
||||
FROM pages p
|
||||
WHERE p.slug = ANY($1::text[]) ${seedScope} AND p.deleted_at IS NULL
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE p.slug = ANY($1::text[]) ${seedScope} ${seedVisibility}
|
||||
UNION ALL
|
||||
SELECT p2.id, p2.slug, p2.source_id, w.depth + 1,
|
||||
w.visited || p2.id, w.path || p2.slug,
|
||||
@@ -3211,7 +3236,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
WHERE w.depth < $2
|
||||
AND NOT (p2.id = ANY(w.visited))
|
||||
AND p2.source_id = w.seed_source
|
||||
AND p2.deleted_at IS NULL
|
||||
${stepVisibility}
|
||||
${mentionsFilter}
|
||||
${typeFilter}
|
||||
)
|
||||
@@ -5385,13 +5410,17 @@ export class PGLiteEngine implements BrainEngine {
|
||||
: opts?.sourceId
|
||||
? [opts.sourceId]
|
||||
: null;
|
||||
let q = `SELECT alias_norm, slug, source_id FROM page_aliases WHERE alias_norm = ANY($1::text[])`;
|
||||
let q = `SELECT pa.alias_norm, pa.slug, pa.source_id
|
||||
FROM page_aliases pa
|
||||
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE pa.alias_norm = ANY($1::text[]) ${buildVisibilityClause('p', 's')}`;
|
||||
const params: unknown[] = [aliasNorms];
|
||||
if (sources) {
|
||||
params.push(sources);
|
||||
q += ` AND source_id = ANY($2::text[])`;
|
||||
q += ` AND pa.source_id = ANY($2::text[])`;
|
||||
}
|
||||
q += ` ORDER BY alias_norm, source_id, slug`;
|
||||
q += ` ORDER BY pa.alias_norm, pa.source_id, pa.slug`;
|
||||
const { rows } = await this.db.query(q, params);
|
||||
for (const r of rows as Array<{ alias_norm: string; slug: string; source_id: string }>) {
|
||||
const list = out.get(r.alias_norm) ?? [];
|
||||
|
||||
+39
-14
@@ -1767,6 +1767,10 @@ export class PostgresEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
@@ -1794,6 +1798,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
${buildBestPerPagePoolCte('ranked_chunks')}
|
||||
SELECT slug, page_id, title, type, source_id,
|
||||
effective_date, effective_date_source,
|
||||
message_id, thread_id, source_subject,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
||||
false AS stale
|
||||
FROM best_per_page
|
||||
@@ -2065,6 +2070,10 @@ export class PostgresEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
false AS stale
|
||||
@@ -2217,6 +2226,10 @@ export class PostgresEngine implements BrainEngine {
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id, CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string' THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string' AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string' THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
1 - (cc.${col} <=> ${castSql}) AS raw_score
|
||||
FROM content_chunks cc
|
||||
@@ -2251,6 +2264,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
SELECT
|
||||
slug, page_id, title, type, source_id,
|
||||
effective_date, effective_date_source,
|
||||
message_id, thread_id, source_subject,
|
||||
chunk_id, chunk_index, chunk_text, chunk_source,
|
||||
score,
|
||||
false AS stale
|
||||
@@ -3357,15 +3371,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
const mentionsFilter = opts?.includeMentions
|
||||
? sql``
|
||||
: sql`AND l.link_source IS DISTINCT FROM 'mentions'`;
|
||||
const seedVisibility = sql.unsafe(buildVisibilityClause('p', 's'));
|
||||
const stepVisibility = sql.unsafe(buildVisibilityClause('p2', 's2'));
|
||||
|
||||
// Recursive step join differs by direction; everything else is shared.
|
||||
const recurStep =
|
||||
direction === 'out'
|
||||
? sql`JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id`
|
||||
? sql`JOIN links l ON l.from_page_id = w.id JOIN pages p2 ON p2.id = l.to_page_id JOIN sources s2 ON s2.id = p2.source_id`
|
||||
: direction === 'in'
|
||||
? sql`JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id`
|
||||
? sql`JOIN links l ON l.to_page_id = w.id JOIN pages p2 ON p2.id = l.from_page_id JOIN sources s2 ON s2.id = p2.source_id`
|
||||
: sql`JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
|
||||
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END`;
|
||||
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
|
||||
JOIN sources s2 ON s2.id = p2.source_id`;
|
||||
|
||||
const rows = await sql`
|
||||
WITH RECURSIVE walk AS (
|
||||
@@ -3373,7 +3390,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
ARRAY[p.id] AS visited, ARRAY[p.slug] AS path,
|
||||
p.source_id AS seed_source, NULL::text AS last_link_type
|
||||
FROM pages p
|
||||
WHERE p.slug = ANY(${seeds}::text[]) ${seedScope} AND p.deleted_at IS NULL
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE p.slug = ANY(${seeds}::text[]) ${seedScope} ${seedVisibility}
|
||||
UNION ALL
|
||||
SELECT p2.id, p2.slug, p2.source_id, w.depth + 1,
|
||||
w.visited || p2.id, w.path || p2.slug,
|
||||
@@ -3383,7 +3401,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
WHERE w.depth < ${depth}
|
||||
AND NOT (p2.id = ANY(w.visited))
|
||||
AND p2.source_id = w.seed_source
|
||||
AND p2.deleted_at IS NULL
|
||||
${stepVisibility}
|
||||
${mentionsFilter}
|
||||
${typeFilter}
|
||||
)
|
||||
@@ -5492,18 +5510,25 @@ export class PostgresEngine implements BrainEngine {
|
||||
: opts?.sourceId
|
||||
? [opts.sourceId]
|
||||
: null;
|
||||
const visibility = sql.unsafe(buildVisibilityClause('p', 's'));
|
||||
const rows = sources
|
||||
? await sql`
|
||||
SELECT alias_norm, slug, source_id
|
||||
FROM page_aliases
|
||||
WHERE alias_norm = ANY(${aliasNorms}::text[])
|
||||
AND source_id = ANY(${sources}::text[])
|
||||
ORDER BY alias_norm, source_id, slug`
|
||||
SELECT pa.alias_norm, pa.slug, pa.source_id
|
||||
FROM page_aliases pa
|
||||
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE pa.alias_norm = ANY(${aliasNorms}::text[])
|
||||
AND pa.source_id = ANY(${sources}::text[])
|
||||
${visibility}
|
||||
ORDER BY pa.alias_norm, pa.source_id, pa.slug`
|
||||
: await sql`
|
||||
SELECT alias_norm, slug, source_id
|
||||
FROM page_aliases
|
||||
WHERE alias_norm = ANY(${aliasNorms}::text[])
|
||||
ORDER BY alias_norm, source_id, slug`;
|
||||
SELECT pa.alias_norm, pa.slug, pa.source_id
|
||||
FROM page_aliases pa
|
||||
JOIN pages p ON p.source_id = pa.source_id AND p.slug = pa.slug
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE pa.alias_norm = ANY(${aliasNorms}::text[])
|
||||
${visibility}
|
||||
ORDER BY pa.alias_norm, pa.source_id, pa.slug`;
|
||||
for (const r of rows) {
|
||||
const a = r.alias_norm as string;
|
||||
const list = out.get(a) ?? [];
|
||||
|
||||
@@ -45,7 +45,10 @@ export const QUARANTINE_KEY = 'quarantine';
|
||||
* JSONB `?` existence, negated so we KEEP rows WITHOUT the marker.
|
||||
* `pageAlias` is engine-supplied (never user input), so no escaping needed. */
|
||||
export function quarantineFilterFragment(pageAlias: string): string {
|
||||
return `NOT (COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ? '${QUARANTINE_KEY}')`;
|
||||
const frontmatter = `COALESCE(${pageAlias}.frontmatter, '{}'::jsonb)`;
|
||||
// Fail closed on legacy/double-encoded JSONB. A top-level string can hide a
|
||||
// quarantine marker from the JSONB `?` operator and must not be searchable.
|
||||
return `(jsonb_typeof(${frontmatter}) = 'object' AND NOT (${frontmatter} ? '${QUARANTINE_KEY}'))`;
|
||||
}
|
||||
|
||||
/** The `p`-aliased instance — the common case (all 6 search call sites alias
|
||||
|
||||
+86
-11
@@ -14,6 +14,8 @@ import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts';
|
||||
import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
|
||||
import { embed, embedQuery } from '../embedding.ts';
|
||||
import { registerBackgroundWorkDrainer } from '../background-work.ts';
|
||||
import { projectEmailCitationFrontmatter } from '../utils.ts';
|
||||
import { assertValidSourceId } from '../source-id.ts';
|
||||
import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
|
||||
import { resolveHardExcludes } from './source-boost.ts';
|
||||
import {
|
||||
@@ -45,6 +47,7 @@ import {
|
||||
SemanticQueryCache,
|
||||
loadCacheConfig,
|
||||
} from './query-cache.ts';
|
||||
import { readPageGenerationClock } from './query-cache-gate.ts';
|
||||
|
||||
export const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
@@ -696,6 +699,7 @@ export async function applyAliasHop(
|
||||
score: injectScore,
|
||||
base_score: injectScore,
|
||||
alias_hit: true,
|
||||
...projectEmailCitationFrontmatter(page.frontmatter),
|
||||
} as SearchResult);
|
||||
}
|
||||
out.sort((a, b) => b.score - a.score);
|
||||
@@ -1469,6 +1473,7 @@ export async function hybridSearch(
|
||||
walkDepth,
|
||||
nearSymbol: opts?.nearSymbol,
|
||||
sourceId: opts?.sourceId,
|
||||
sourceIds: opts?.sourceIds,
|
||||
});
|
||||
// Resolve new chunk IDs (not already in fused) into full rows.
|
||||
const existingIds = new Set(fused.map(r => r.chunk_id));
|
||||
@@ -1476,7 +1481,10 @@ export async function hybridSearch(
|
||||
.filter(e => !existingIds.has(e.chunk_id))
|
||||
.map(e => e.chunk_id);
|
||||
if (newIds.length > 0) {
|
||||
const hydrated = await hydrateChunks(engine, newIds);
|
||||
const hydrated = await hydrateChunks(engine, newIds, {
|
||||
sourceId: opts?.sourceId,
|
||||
sourceIds: opts?.sourceIds,
|
||||
});
|
||||
const scoreById = new Map(expanded.map(e => [e.chunk_id, e.score]));
|
||||
for (const r of hydrated) {
|
||||
r.score = scoreById.get(r.chunk_id) ?? 0.01;
|
||||
@@ -1651,7 +1659,18 @@ export async function hybridSearchCached(
|
||||
perCall: {
|
||||
cache_enabled: opts?.useCache,
|
||||
tokenBudget: opts?.tokenBudget,
|
||||
expansion: opts?.expansion,
|
||||
// Fold EFFECTIVE expansion into the cache key: the inner hybridSearch
|
||||
// only expands when the resolved knob is on AND an expandFn is wired
|
||||
// in (`expansionAllowed && opts?.expandFn`). Without an expandFn the
|
||||
// knob can never fire, so force `false` here — otherwise a
|
||||
// no-expandFn caller under a tokenmax bundle would share a cache row
|
||||
// with the expanded results of an expandFn caller (and vice versa).
|
||||
// This is what makes `expandFn` cache-SAFE (it is deliberately NOT in
|
||||
// isSemanticCacheRequestSafe's unsafe list): its result-shaping effect
|
||||
// is fully expressed by the expansion bit of the knobs hash. The
|
||||
// production `query` op always passes expandFn, so listing it unsafe
|
||||
// would silently disable the semantic cache for the flagship op.
|
||||
expansion: opts?.expandFn ? opts?.expansion : false,
|
||||
intentWeighting: opts?.intentWeighting,
|
||||
searchLimit: opts?.limit,
|
||||
// v0.35.6.0 — floor-ratio threaded through cache resolver too so
|
||||
@@ -1728,11 +1747,23 @@ export async function hybridSearchCached(
|
||||
opts?.adaptiveReturn,
|
||||
cfgCached as unknown as Record<string, unknown> | null,
|
||||
);
|
||||
// Compute the scope key ONCE, fail-open. `cacheScopeKey` throws on a
|
||||
// malformed source id (forged encodings, invalid charset) — that rejection
|
||||
// is correct for direct callers, but inside the search hot path an invalid
|
||||
// scope must degrade to "skip the cache", never break the search itself.
|
||||
let cacheScope: string | null = null;
|
||||
try {
|
||||
cacheScope = cacheScopeKey(opts);
|
||||
} catch {
|
||||
// invalid scope id — cache lookup/store skipped below
|
||||
}
|
||||
const skipCache =
|
||||
!cache.isEnabled() ||
|
||||
(opts?.walkDepth ?? 0) > 0 ||
|
||||
Boolean(opts?.nearSymbol) ||
|
||||
isNonDefaultColumn ||
|
||||
!isSemanticCacheRequestSafe(opts) ||
|
||||
cacheScope === null ||
|
||||
adaptiveReturnOn;
|
||||
|
||||
let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss';
|
||||
@@ -1777,7 +1808,7 @@ export async function hybridSearchCached(
|
||||
}
|
||||
|
||||
if (!skipCache && queryEmbedding && cacheStatus !== 'disabled') {
|
||||
const hit = await cache.lookup(queryEmbedding, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash });
|
||||
const hit = await cache.lookup(queryEmbedding, { sourceId: cacheScope!, knobsHash: cacheKnobsHash });
|
||||
if (hit.hit && hit.results) {
|
||||
cacheStatus = 'hit';
|
||||
cacheSimilarity = hit.similarity;
|
||||
@@ -1851,6 +1882,10 @@ export async function hybridSearchCached(
|
||||
// we use a single-element box to keep the type stable.
|
||||
const innerMetaBox: { current: HybridSearchMeta | null } = { current: null };
|
||||
const userOnMeta = opts?.onMeta;
|
||||
const maxGenerationAtSearchStart =
|
||||
!skipCache && cacheStatus === 'miss' && queryEmbedding
|
||||
? await readPageGenerationClock(engine)
|
||||
: null;
|
||||
const results = await hybridSearch(engine, query, {
|
||||
...opts,
|
||||
// v0.42.20.0 (Fix 3) — share the query-embed deadline so the inner embed
|
||||
@@ -1909,11 +1944,16 @@ export async function hybridSearchCached(
|
||||
cacheStatus === 'miss' &&
|
||||
queryEmbedding &&
|
||||
results.length > 0 &&
|
||||
(innerMeta?.vector_enabled ?? false)
|
||||
(innerMeta?.vector_enabled ?? false) &&
|
||||
maxGenerationAtSearchStart !== null
|
||||
) {
|
||||
trackCacheWrite(
|
||||
cache
|
||||
.store(query, queryEmbedding, results, finalMeta, { sourceId: cacheScopeKey(opts), knobsHash: cacheKnobsHash })
|
||||
.store(query, queryEmbedding, results, finalMeta, {
|
||||
sourceId: cacheScope!,
|
||||
knobsHash: cacheKnobsHash,
|
||||
maxGenerationAtSearchStart,
|
||||
})
|
||||
.catch(() => { /* swallow */ }),
|
||||
);
|
||||
}
|
||||
@@ -1945,16 +1985,51 @@ function rrfKey(r: SearchResult): string {
|
||||
* cache only saw scalar `sourceId`; a federated query fell through to
|
||||
* `'default'` and could cross-serve an unrelated scope.
|
||||
*
|
||||
* - federated (sourceIds set) → `__set__:` + sorted, comma-joined ids
|
||||
* (order-independent; two different source-sets get distinct keys)
|
||||
* - scalar sourceId → the id itself (single-source unchanged)
|
||||
* - unscoped → `'default'` (single-source brains unchanged)
|
||||
* Keys are typed JSON tuples so arbitrary scalar text cannot collide with a
|
||||
* federated encoding. Every source id is validated before it reaches cache
|
||||
* lookup/store; this keeps internal/trusted callers on the same boundary as
|
||||
* registered sources.
|
||||
*/
|
||||
export function cacheScopeKey(opts?: { sourceId?: string; sourceIds?: string[] }): string {
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
return '__set__:' + [...opts.sourceIds].sort().join(',');
|
||||
const ids = [...new Set(opts.sourceIds)].sort();
|
||||
for (const id of ids) assertValidSourceId(id);
|
||||
return JSON.stringify(['set', ...ids]);
|
||||
}
|
||||
return opts?.sourceId ?? 'default';
|
||||
if (opts?.sourceId !== undefined) {
|
||||
assertValidSourceId(opts.sourceId);
|
||||
return JSON.stringify(['scalar', opts.sourceId]);
|
||||
}
|
||||
return JSON.stringify(['all']);
|
||||
}
|
||||
|
||||
/**
|
||||
* Semantic-cache replay is safe only when every result-shaping input is either
|
||||
* represented by the existing knobs/scope key or absent. Unsupported shapes
|
||||
* bypass lookup and writeback rather than risk cross-serving a different view.
|
||||
*/
|
||||
export function isSemanticCacheRequestSafe(opts?: HybridSearchOpts): boolean {
|
||||
if (!opts) return true;
|
||||
return !(
|
||||
(opts.offset !== undefined && opts.offset !== 0) ||
|
||||
opts.type !== undefined ||
|
||||
opts.types !== undefined ||
|
||||
opts.exclude_slugs !== undefined ||
|
||||
opts.detail !== undefined ||
|
||||
opts.language !== undefined ||
|
||||
opts.symbolKind !== undefined ||
|
||||
opts.afterDate !== undefined ||
|
||||
opts.beforeDate !== undefined ||
|
||||
opts.recencyBoost !== undefined ||
|
||||
opts.salience !== undefined ||
|
||||
opts.recency !== undefined ||
|
||||
opts.since !== undefined ||
|
||||
opts.until !== undefined ||
|
||||
opts.crossModal !== undefined ||
|
||||
opts.reranker !== undefined ||
|
||||
opts.rrfK !== undefined ||
|
||||
opts.dedupOpts !== undefined
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -756,7 +756,13 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// slugs written by a process without it, and vice versa. Same one-time
|
||||
// global cold-miss pattern as the bumps above; refills within
|
||||
// cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 12;
|
||||
//
|
||||
// bump 12→13: SearchResult now projects allowlisted email citation metadata
|
||||
// (`message_id`, `thread_id`, and Message-ID-gated `source_subject`). Cached
|
||||
// rows store the complete result DTO, so pre-projection rows must miss rather
|
||||
// than keep returning the old shape after deployment. Same one-time global
|
||||
// cold-miss pattern; refills within cache.ttl_seconds.
|
||||
export const KNOBS_HASH_VERSION = 13;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
|
||||
@@ -51,6 +51,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { buildVisibilityClause } from './sql-ranking.ts';
|
||||
|
||||
/**
|
||||
* Snapshot of (pageId, generation) pairs plus the corpus-state MAX
|
||||
@@ -71,6 +72,18 @@ export interface PageGenerationsSnapshot {
|
||||
max_generation_at_store: number;
|
||||
}
|
||||
|
||||
/** Read the global page-generation clock. Null means the safety substrate is unavailable. */
|
||||
export async function readPageGenerationClock(engine: BrainEngine): Promise<number | null> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
return Number(rows[0]?.v ?? 0);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the page-generations snapshot for a set of page_ids in one SQL
|
||||
* round trip. Used by query-cache.ts:store() at cache-write time.
|
||||
@@ -102,10 +115,7 @@ export async function buildPageGenerationsSnapshot(
|
||||
// Empty-result query: only need the Layer 1 bookmark (clock value).
|
||||
// Per D20, empty-result cache rows trust Layer 1 exclusively;
|
||||
// bumping the clock on subsequent writes correctly invalidates them.
|
||||
const rows = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0);
|
||||
snapshot.max_generation_at_store = (await readPageGenerationClock(engine)) ?? 0;
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
@@ -187,6 +197,21 @@ export const CACHE_GATE_WHERE_CLAUSE = `
|
||||
)
|
||||
)
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
-- Visibility is independent of freshness. Archiving a source does not
|
||||
-- mutate its pages or advance the page-generation clock, so this check
|
||||
-- MUST sit outside the Layer 1 / Layer 2 OR. Otherwise Layer 1 can serve
|
||||
-- a fresh-but-now-hidden cached result until TTL expiry.
|
||||
SELECT 1
|
||||
FROM jsonb_array_elements(qc.results) AS visible(result)
|
||||
LEFT JOIN pages p_visible ON p_visible.id = (visible.result->>'page_id')::int
|
||||
LEFT JOIN sources s_visible ON s_visible.id = p_visible.source_id
|
||||
WHERE NOT (
|
||||
p_visible.id IS NOT NULL
|
||||
AND s_visible.id IS NOT NULL
|
||||
${buildVisibilityClause('p_visible', 's_visible')}
|
||||
)
|
||||
)
|
||||
`;
|
||||
|
||||
/**
|
||||
|
||||
@@ -32,6 +32,7 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../types.ts';
|
||||
import { assertValidSourceId } from '../source-id.ts';
|
||||
import { buildPageGenerationsSnapshot, CACHE_GATE_WHERE_CLAUSE } from './query-cache-gate.ts';
|
||||
|
||||
/** Default cosine similarity threshold for cache hits. */
|
||||
@@ -205,9 +206,20 @@ export class SemanticQueryCache {
|
||||
queryEmbedding: Float32Array | null,
|
||||
results: SearchResult[],
|
||||
meta: HybridSearchMeta,
|
||||
opts: { sourceId?: string; ttlSeconds?: number; knobsHash?: string } = {},
|
||||
opts: {
|
||||
sourceId?: string;
|
||||
ttlSeconds?: number;
|
||||
knobsHash?: string;
|
||||
/** Global generation captured immediately before the producing search. */
|
||||
maxGenerationAtSearchStart?: number;
|
||||
} = {},
|
||||
): Promise<void> {
|
||||
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) return;
|
||||
if (
|
||||
!this.enabled ||
|
||||
!queryEmbedding ||
|
||||
queryEmbedding.length === 0 ||
|
||||
opts.maxGenerationAtSearchStart === undefined
|
||||
) return;
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
const knobsHash = opts.knobsHash ?? '';
|
||||
const ttl = clampTtl(opts.ttlSeconds ?? this.ttlSeconds);
|
||||
@@ -222,6 +234,13 @@ export class SemanticQueryCache {
|
||||
.map((r) => r.page_id)
|
||||
.filter((id): id is number => typeof id === 'number' && Number.isFinite(id));
|
||||
const snapshot = await buildPageGenerationsSnapshot(this.engine, pageIds);
|
||||
// Production search captures the clock before materializing results. If
|
||||
// any page write landed during that window, the serialized DTO may already
|
||||
// be stale; skip writeback rather than certify it with the newer snapshot.
|
||||
if (
|
||||
opts.maxGenerationAtSearchStart !== undefined &&
|
||||
snapshot.max_generation_at_store !== opts.maxGenerationAtSearchStart
|
||||
) return;
|
||||
|
||||
try {
|
||||
// v0.32.3 [CDX-4]: knobs_hash threaded into the row so concurrent
|
||||
@@ -269,10 +288,35 @@ export class SemanticQueryCache {
|
||||
async clear(opts: { sourceId?: string } = {}): Promise<number> {
|
||||
try {
|
||||
if (opts.sourceId) {
|
||||
assertValidSourceId(opts.sourceId);
|
||||
const typedScalar = JSON.stringify(['scalar', opts.sourceId]);
|
||||
const typedSetMemberPattern = `%${JSON.stringify(opts.sourceId)}%`;
|
||||
const legacySetOnly = `__set__:${opts.sourceId}`;
|
||||
const legacySetFirst = `__set__:${opts.sourceId},%`;
|
||||
const legacySetMiddle = `__set__:%,${opts.sourceId},%`;
|
||||
const legacySetLast = `__set__:%,${opts.sourceId}`;
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (DELETE FROM query_cache WHERE source_id = $1 RETURNING 1)
|
||||
`WITH deleted AS (
|
||||
DELETE FROM query_cache
|
||||
WHERE source_id = $1
|
||||
OR source_id = $2
|
||||
OR (source_id LIKE '["set",%' AND source_id LIKE $3)
|
||||
OR source_id = $4
|
||||
OR source_id LIKE $5
|
||||
OR source_id LIKE $6
|
||||
OR source_id LIKE $7
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
[opts.sourceId],
|
||||
[
|
||||
opts.sourceId,
|
||||
typedScalar,
|
||||
typedSetMemberPattern,
|
||||
legacySetOnly,
|
||||
legacySetFirst,
|
||||
legacySetMiddle,
|
||||
legacySetLast,
|
||||
],
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
}
|
||||
|
||||
@@ -29,6 +29,8 @@ import type { SearchResult, PageType, RelationalFanoutRow } from '../types.ts';
|
||||
import { createAuditWriter } from '../audit/audit-writer.ts';
|
||||
import { resolveEntitySlugWithSource } from '../entities/resolve.ts';
|
||||
import { parseRelationalQuery, type RelationalQuery, type RelationVocab } from './relational-intent.ts';
|
||||
import { projectEmailCitationMetadata } from '../utils.ts';
|
||||
import { buildVisibilityClause } from './sql-ranking.ts';
|
||||
|
||||
export interface RelationalArmOpts {
|
||||
sourceId?: string;
|
||||
@@ -108,11 +110,22 @@ async function hydrate(
|
||||
const slugs = Array.from(new Set(rows.map(r => r.slug)));
|
||||
const pageRows = await engine.executeRaw<{
|
||||
page_id: number; slug: string; source_id: string; title: string; type: string; synopsis: string | null;
|
||||
message_id: string | null; thread_id: string | null; source_subject: string | null;
|
||||
}>(
|
||||
`SELECT p.id AS page_id, p.slug, p.source_id, p.title, p.type,
|
||||
LEFT(p.compiled_truth, 240) AS synopsis
|
||||
LEFT(p.compiled_truth, 240) AS synopsis,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
|
||||
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string'
|
||||
THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
|
||||
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string'
|
||||
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject
|
||||
FROM pages p
|
||||
WHERE p.slug = ANY($1::text[]) AND p.deleted_at IS NULL`,
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE p.slug = ANY($1::text[]) ${buildVisibilityClause('p', 's')}`,
|
||||
[slugs],
|
||||
);
|
||||
const byKey = new Map<string, typeof pageRows[number]>();
|
||||
@@ -141,6 +154,7 @@ async function hydrate(
|
||||
relational_seed: seedSlug,
|
||||
relational_hop: r.hop,
|
||||
relational_path: r.path,
|
||||
...projectEmailCitationMetadata(pr),
|
||||
});
|
||||
}
|
||||
return out;
|
||||
|
||||
@@ -146,7 +146,7 @@ export function buildHardExcludeClause(slugColumn: string, prefixes: string[]):
|
||||
* responsible for joining `sources` so this alias resolves.
|
||||
*
|
||||
* @returns raw SQL fragment, e.g.
|
||||
* `AND p.deleted_at IS NULL AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine')`
|
||||
* `AND p.deleted_at IS NULL AND NOT s.archived AND (jsonb_typeof(COALESCE(p.frontmatter, '{}'::jsonb)) = 'object' AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine'))`
|
||||
*/
|
||||
export function buildVisibilityClause(pageAlias: string, sourceAlias: string): string {
|
||||
// Single source of truth for the quarantine SQL lives in quarantine.ts so
|
||||
|
||||
+91
-31
@@ -23,6 +23,8 @@
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchResult } from '../types.ts';
|
||||
import { projectEmailCitationMetadata } from '../utils.ts';
|
||||
import { buildVisibilityClause } from './sql-ranking.ts';
|
||||
|
||||
const MAX_WALK_DEPTH = 2;
|
||||
const NEIGHBOR_CAP_PER_HOP = 50;
|
||||
@@ -34,6 +36,8 @@ export interface TwoPassOpts {
|
||||
nearSymbol?: string;
|
||||
/** Filter expansion to one source. When unset, crosses sources. */
|
||||
sourceId?: string;
|
||||
/** Federated source grant. Non-empty arrays take precedence over sourceId. */
|
||||
sourceIds?: string[];
|
||||
}
|
||||
|
||||
interface ChunkWithScore {
|
||||
@@ -54,6 +58,16 @@ export async function expandAnchors(
|
||||
opts: TwoPassOpts = {},
|
||||
): Promise<ChunkWithScore[]> {
|
||||
const depth = Math.min(Math.max(opts.walkDepth ?? 0, 0), MAX_WALK_DEPTH);
|
||||
const federatedSourceIds = opts.sourceIds && opts.sourceIds.length > 0
|
||||
? opts.sourceIds
|
||||
: undefined;
|
||||
const sourceScopeSql = federatedSourceIds
|
||||
? ' AND p.source_id = ANY($2::text[])'
|
||||
: opts.sourceId
|
||||
? ' AND p.source_id = $2'
|
||||
: '';
|
||||
const sourceScopeValue = federatedSourceIds ?? opts.sourceId;
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
if (depth === 0 && !opts.nearSymbol) {
|
||||
return anchors.map(a => ({
|
||||
chunk_id: a.chunk_id,
|
||||
@@ -83,18 +97,14 @@ export async function expandAnchors(
|
||||
// filter; undefined → cross-source (matches the documented contract).
|
||||
if (opts.nearSymbol) {
|
||||
try {
|
||||
const rows = opts.sourceId
|
||||
? await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.symbol_name_qualified = $1 AND p.source_id = $2
|
||||
LIMIT 50`,
|
||||
[opts.nearSymbol, opts.sourceId],
|
||||
)
|
||||
: await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM content_chunks WHERE symbol_name_qualified = $1 LIMIT 50`,
|
||||
[opts.nearSymbol],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.symbol_name_qualified = $1${sourceScopeSql} ${visibilityClause}
|
||||
LIMIT 50`,
|
||||
sourceScopeValue === undefined ? [opts.nearSymbol] : [opts.nearSymbol, sourceScopeValue],
|
||||
);
|
||||
const baseScore = anchors.length > 0 ? anchors[0]!.score : 1.0;
|
||||
for (const r of rows) {
|
||||
if (!seen.has(r.id)) {
|
||||
@@ -135,8 +145,19 @@ export async function expandAnchors(
|
||||
const directChunkIds: number[] = [];
|
||||
const unresolvedTargets: string[] = [];
|
||||
for (const e of edges) {
|
||||
if (e.to_chunk_id != null) directChunkIds.push(e.to_chunk_id);
|
||||
else if (e.to_symbol_qualified) unresolvedTargets.push(e.to_symbol_qualified);
|
||||
if (e.to_chunk_id != null) {
|
||||
// getEdgesByChunk(direction:'both') returns incoming and outgoing
|
||||
// edges. Follow the endpoint opposite the current chunk; always
|
||||
// taking to_chunk_id makes incoming A→B edges reselect B.
|
||||
const neighborId = e.from_chunk_id === chunkId
|
||||
? e.to_chunk_id
|
||||
: e.to_chunk_id === chunkId
|
||||
? e.from_chunk_id
|
||||
: null;
|
||||
if (neighborId != null) directChunkIds.push(neighborId);
|
||||
} else if (e.to_symbol_qualified) {
|
||||
unresolvedTargets.push(e.to_symbol_qualified);
|
||||
}
|
||||
}
|
||||
// Resolve unresolved edges by looking up chunks whose
|
||||
// symbol_name_qualified matches. One batch query per frontier node.
|
||||
@@ -146,26 +167,40 @@ export async function expandAnchors(
|
||||
// boundaries silently in multi-source brains.
|
||||
if (unresolvedTargets.length > 0) {
|
||||
try {
|
||||
const resolved = opts.sourceId
|
||||
? await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.symbol_name_qualified = ANY($1::text[])
|
||||
AND p.source_id = $2
|
||||
LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
|
||||
[unresolvedTargets, opts.sourceId],
|
||||
)
|
||||
: await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM content_chunks WHERE symbol_name_qualified = ANY($1::text[]) LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
|
||||
[unresolvedTargets],
|
||||
);
|
||||
const resolved = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.symbol_name_qualified = ANY($1::text[])${sourceScopeSql} ${visibilityClause}
|
||||
LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
|
||||
sourceScopeValue === undefined ? [unresolvedTargets] : [unresolvedTargets, sourceScopeValue],
|
||||
);
|
||||
for (const r of resolved) directChunkIds.push(r.id);
|
||||
} catch {
|
||||
// best-effort
|
||||
}
|
||||
}
|
||||
|
||||
for (const tid of directChunkIds) {
|
||||
// Resolved chunk edges can cross sources even when the originating edge
|
||||
// carries an allowed source. Filter the destination page itself. A scope
|
||||
// query failure must fail closed rather than exposing an unverified row.
|
||||
let scopedChunkIds: number[] = [];
|
||||
if (directChunkIds.length > 0) {
|
||||
try {
|
||||
const scoped = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.id = ANY($1::int[])${sourceScopeSql} ${visibilityClause}`,
|
||||
sourceScopeValue === undefined ? [directChunkIds] : [directChunkIds, sourceScopeValue],
|
||||
);
|
||||
scopedChunkIds = scoped.map((row) => row.id);
|
||||
} catch {
|
||||
scopedChunkIds = [];
|
||||
}
|
||||
}
|
||||
|
||||
for (const tid of scopedChunkIds) {
|
||||
if (seen.has(tid)) continue;
|
||||
const nbScore = current.score * decay;
|
||||
seen.set(tid, { chunk_id: tid, score: nbScore, hop, source: 'neighbor' });
|
||||
@@ -188,18 +223,42 @@ export async function expandAnchors(
|
||||
export async function hydrateChunks(
|
||||
engine: BrainEngine,
|
||||
chunkIds: number[],
|
||||
opts: Pick<TwoPassOpts, 'sourceId' | 'sourceIds'> = {},
|
||||
): Promise<SearchResult[]> {
|
||||
if (chunkIds.length === 0) return [];
|
||||
const federatedSourceIds = opts.sourceIds && opts.sourceIds.length > 0
|
||||
? opts.sourceIds
|
||||
: undefined;
|
||||
const scopeSql = federatedSourceIds
|
||||
? ' AND p.source_id = ANY($2::text[])'
|
||||
: opts.sourceId
|
||||
? ' AND p.source_id = $2'
|
||||
: '';
|
||||
const params: unknown[] = [chunkIds];
|
||||
if (federatedSourceIds) params.push(federatedSourceIds);
|
||||
else if (opts.sourceId) params.push(opts.sourceId);
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
const rows = await engine.executeRaw<{
|
||||
slug: string; page_id: number; title: string; type: string; source_id: string;
|
||||
chunk_id: number; chunk_index: number; chunk_text: string; chunk_source: string;
|
||||
message_id: string | null; thread_id: string | null; source_subject: string | null;
|
||||
}>(
|
||||
`SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
|
||||
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
THEN p.frontmatter->>'message_id' END AS message_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'thread_id') = 'string'
|
||||
THEN NULLIF(p.frontmatter->>'thread_id', '') END AS thread_id,
|
||||
CASE WHEN jsonb_typeof(p.frontmatter->'message_id') = 'string'
|
||||
AND NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL
|
||||
AND jsonb_typeof(p.frontmatter->'subject') = 'string'
|
||||
THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.id = ANY($1::int[])`,
|
||||
[chunkIds],
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.id = ANY($1::int[])${scopeSql} ${visibilityClause}`,
|
||||
params,
|
||||
);
|
||||
return rows.map((r) => ({
|
||||
slug: r.slug,
|
||||
@@ -213,5 +272,6 @@ export async function hydrateChunks(
|
||||
score: 0, // two-pass caller assigns scores.
|
||||
stale: false,
|
||||
source_id: r.source_id,
|
||||
...projectEmailCitationMetadata(r),
|
||||
} as SearchResult));
|
||||
}
|
||||
|
||||
@@ -723,6 +723,12 @@ export interface SearchResult {
|
||||
*/
|
||||
effective_date?: string | null;
|
||||
effective_date_source?: string | null;
|
||||
/** RFC 5322 Message-ID projected from allowlisted email frontmatter. */
|
||||
message_id?: string;
|
||||
/** Gmail thread id projected from allowlisted email frontmatter. */
|
||||
thread_id?: string;
|
||||
/** Exact email subject, projected only when the page has a Message-ID. */
|
||||
source_subject?: string;
|
||||
/**
|
||||
* v0.40.4 graph signals — populated by applyGraphSignals when the
|
||||
* graph_signals mode-bundle knob is on. Surfaced in JSON envelope
|
||||
|
||||
+47
-2
@@ -117,7 +117,16 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
title: row.title as string,
|
||||
compiled_truth: row.compiled_truth as string,
|
||||
timeline: row.timeline as string,
|
||||
frontmatter: (typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) : row.frontmatter) as Record<string, unknown>,
|
||||
// Postgres and PGLite both decode JSONB objects. A string here means the
|
||||
// JSONB top level itself is a string (legacy double-encoding), not a driver
|
||||
// transport shape. Never decode it into trusted frontmatter.
|
||||
frontmatter: (
|
||||
row.frontmatter !== null &&
|
||||
typeof row.frontmatter === 'object' &&
|
||||
!Array.isArray(row.frontmatter)
|
||||
? row.frontmatter
|
||||
: {}
|
||||
) as Record<string, unknown>,
|
||||
content_hash: row.content_hash as string | undefined,
|
||||
// v0.29 (column added in migration v40). Old brains pre-migration return undefined.
|
||||
emotional_weight: row.emotional_weight == null ? undefined : Number(row.emotional_weight),
|
||||
@@ -162,7 +171,9 @@ export function rowToStalePage(row: Record<string, unknown>): StalePageRow {
|
||||
title: (row.title as string | null) ?? '',
|
||||
compiled_truth: (row.compiled_truth as string | null) ?? '',
|
||||
timeline: (row.timeline as string | null) ?? '',
|
||||
frontmatter: (fm == null ? {} : (typeof fm === 'string' ? JSON.parse(fm) : fm)) as Record<string, unknown>,
|
||||
frontmatter: (
|
||||
fm !== null && typeof fm === 'object' && !Array.isArray(fm) ? fm : {}
|
||||
) as Record<string, unknown>,
|
||||
updated_at: new Date(row.updated_at as string),
|
||||
// #1768: full-µs UTC string projected by the SELECT (`updated_at_iso`).
|
||||
// Fallback derives an ISO string from the Date — NEVER String(Date), which
|
||||
@@ -333,6 +344,39 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
|
||||
};
|
||||
}
|
||||
|
||||
function projectCitationMetadata(
|
||||
record: Record<string, unknown> | null | undefined,
|
||||
subjectField: 'subject' | 'source_subject',
|
||||
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
|
||||
if (!record) return {};
|
||||
const metadata: Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> = {};
|
||||
if (typeof record.message_id === 'string' && record.message_id.trim().length > 0) {
|
||||
metadata.message_id = record.message_id;
|
||||
}
|
||||
if (typeof record.thread_id === 'string' && record.thread_id.length > 0) {
|
||||
metadata.thread_id = record.thread_id;
|
||||
}
|
||||
const subject = record[subjectField];
|
||||
if (metadata.message_id && typeof subject === 'string' && subject.length > 0) {
|
||||
metadata.source_subject = subject;
|
||||
}
|
||||
return metadata;
|
||||
}
|
||||
|
||||
/** Project normalized SQL result columns. `source_subject` is an internal alias. */
|
||||
export function projectEmailCitationMetadata(
|
||||
record: Record<string, unknown> | null | undefined,
|
||||
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
|
||||
return projectCitationMetadata(record, 'source_subject');
|
||||
}
|
||||
|
||||
/** Project raw page frontmatter. Internal SQL aliases are never trusted here. */
|
||||
export function projectEmailCitationFrontmatter(
|
||||
record: Record<string, unknown> | null | undefined,
|
||||
): Partial<Pick<SearchResult, 'message_id' | 'thread_id' | 'source_subject'>> {
|
||||
return projectCitationMetadata(record, 'subject');
|
||||
}
|
||||
|
||||
export function rowToSearchResult(row: Record<string, unknown>): SearchResult {
|
||||
const result: SearchResult = {
|
||||
slug: row.slug as string,
|
||||
@@ -381,6 +425,7 @@ export function rowToSearchResult(row: Record<string, unknown>): SearchResult {
|
||||
result.effective_date_source = raw;
|
||||
}
|
||||
}
|
||||
Object.assign(result, projectEmailCitationMetadata(row));
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@@ -3,29 +3,30 @@
|
||||
*
|
||||
* A federated search reads a different graph than a single-source one, so
|
||||
* the semantic cache must key them apart. `cacheScopeKey` produces an
|
||||
* order-independent key for federated scopes and leaves single-source
|
||||
* brains on their existing key (scalar id or 'default'), so single-source
|
||||
* cache hit-rate is unchanged.
|
||||
* order-independent key for federated scopes and keeps unscoped all-source
|
||||
* reads distinct from every scalar source key.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { cacheScopeKey } from '../src/core/search/hybrid.ts';
|
||||
import { cacheScopeKey, isSemanticCacheRequestSafe } from '../src/core/search/hybrid.ts';
|
||||
import type { HybridSearchOpts } from '../src/core/search/hybrid.ts';
|
||||
|
||||
describe('cacheScopeKey', () => {
|
||||
test('unscoped → default (single-source unchanged)', () => {
|
||||
expect(cacheScopeKey(undefined)).toBe('default');
|
||||
expect(cacheScopeKey({})).toBe('default');
|
||||
test('unscoped uses a typed key and never collides with scalar default', () => {
|
||||
expect(cacheScopeKey(undefined)).toBe('["all"]');
|
||||
expect(cacheScopeKey({})).toBe('["all"]');
|
||||
expect(cacheScopeKey({})).not.toBe(cacheScopeKey({ sourceId: 'default' }));
|
||||
});
|
||||
|
||||
test('scalar sourceId → itself (single-source unchanged)', () => {
|
||||
expect(cacheScopeKey({ sourceId: 'host' })).toBe('host');
|
||||
test('scalar sourceId uses a typed key', () => {
|
||||
expect(cacheScopeKey({ sourceId: 'host' })).toBe('["scalar","host"]');
|
||||
});
|
||||
|
||||
test('federated sourceIds → order-independent set key', () => {
|
||||
const k1 = cacheScopeKey({ sourceIds: ['team-b', 'team-a', 'host'] });
|
||||
const k2 = cacheScopeKey({ sourceIds: ['host', 'team-a', 'team-b'] });
|
||||
expect(k1).toBe(k2); // order does not matter
|
||||
expect(k1).toBe('__set__:host,team-a,team-b');
|
||||
expect(k1).toBe('["set","host","team-a","team-b"]');
|
||||
});
|
||||
|
||||
test('different source-sets do NOT share a key', () => {
|
||||
@@ -39,4 +40,40 @@ describe('cacheScopeKey', () => {
|
||||
const scalar = cacheScopeKey({ sourceId: 'host' });
|
||||
expect(set).not.toBe(scalar); // a 1-element set still cannot serve a scalar read
|
||||
});
|
||||
|
||||
test('rejects forged scalar/set encodings and invalid federated ids', () => {
|
||||
expect(() => cacheScopeKey({ sourceId: '__set__:a,b' })).toThrow('Invalid source_id');
|
||||
expect(() => cacheScopeKey({ sourceIds: ['a', '__all__'] })).toThrow('Invalid source_id');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSemanticCacheRequestSafe', () => {
|
||||
test('accepts the standard cache-safe request shape', () => {
|
||||
expect(isSemanticCacheRequestSafe({ limit: 20, sourceId: 'default' })).toBe(true);
|
||||
});
|
||||
|
||||
test('accepts the production query-op shape (expandFn is folded into the knobs hash, not unsafe)', () => {
|
||||
// operations.ts always passes expandFn on the default `query` op path;
|
||||
// listing it unsafe would permanently disable the semantic cache for the
|
||||
// flagship op. Its effect is keyed via the expansion bit instead
|
||||
// (hybridSearchCached folds `expandFn ? expansion : false` into the hash).
|
||||
expect(isSemanticCacheRequestSafe({
|
||||
limit: 20,
|
||||
offset: 0,
|
||||
expansion: true,
|
||||
expandFn: async (q: string) => [q],
|
||||
sourceId: 'default',
|
||||
})).toBe(true);
|
||||
});
|
||||
|
||||
const unsafeRequests: HybridSearchOpts[] = [
|
||||
{ offset: 10 }, { detail: 'high' }, { language: 'typescript' },
|
||||
{ symbolKind: 'function' }, { types: ['note'] }, { since: '7d' },
|
||||
{ until: '2026-07-18' }, { salience: 'on' }, { recency: 'strong' },
|
||||
{ crossModal: 'both' }, { exclude_slugs: ['private/page'] },
|
||||
];
|
||||
|
||||
test.each(unsafeRequests)('rejects unsupported result-shaping request %#', (opts) => {
|
||||
expect(isSemanticCacheRequestSafe(opts)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => {
|
||||
test('KNOBS_HASH_VERSION is 13 (12→13 email citation result-schema projection)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
@@ -146,7 +146,9 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
|
||||
// finally reaches asymmetric providers — pre-fix rows were keyed on
|
||||
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
// Email citation metadata projection: 12→13 so cached SearchResult DTOs
|
||||
// written before message_id/thread_id/source_subject cannot survive.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -52,6 +52,21 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
let engine: PGLiteEngine;
|
||||
let cache: SemanticQueryCache;
|
||||
|
||||
async function storeCurrent(
|
||||
cacheInstance: SemanticQueryCache,
|
||||
...args: Parameters<SemanticQueryCache['store']>
|
||||
): Promise<void> {
|
||||
const [queryText, queryEmbedding, results, meta, opts = {}] = args;
|
||||
const rows = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
await cacheInstance.store(queryText, queryEmbedding, results, meta, {
|
||||
...opts,
|
||||
maxGenerationAtSearchStart:
|
||||
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
@@ -90,7 +105,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
} as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(1);
|
||||
await cache.store('alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
const hit = await cache.lookup(emb, { sourceId: 'default' });
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.results?.length).toBe(1);
|
||||
@@ -102,7 +117,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'x', score: 1.0 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(2);
|
||||
await cache.store('alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'alpha bravo', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// Update content_truth — trigger bumps p1.generation
|
||||
await engine.putPage('test/p1', {
|
||||
@@ -123,7 +138,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'a', score: 1.0 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(3);
|
||||
await cache.store('alpha', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'alpha', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// Create an UNRELATED new page (different topic, not in result set).
|
||||
await seedPage('test/p2', 'beta gamma');
|
||||
@@ -143,7 +158,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'g', score: 1.0 } as unknown as SearchResult,
|
||||
];
|
||||
// Simulate a pre-v0.40.3.0 row: empty snapshot + zero bookmark.
|
||||
await cache.store('gamma delta', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'gamma delta', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache
|
||||
SET page_generations = '{}'::jsonb,
|
||||
@@ -171,7 +186,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'q', score: 0.9 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(7);
|
||||
await cache.store('phi chi', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'phi chi', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// Hard-delete via engine.deletePage. Pre-v0.41.19.0 the trigger
|
||||
// didn't fire on DELETE so MAX(generation) didn't move and the cache
|
||||
@@ -197,7 +212,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p1, slug: 'test/non-max-p1', title: 'test/non-max-p1', snippet: 'o', score: 1.0 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(8);
|
||||
await cache.store('omega psi', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'omega psi', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// UPDATE p1 (the non-max page) with new content.
|
||||
await engine.putPage('test/non-max-p1', {
|
||||
@@ -218,7 +233,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'e', score: 1.0 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(5);
|
||||
await cache.store('epsilon', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'epsilon', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// Soft-delete: UPDATE pages SET deleted_at = now() — production path
|
||||
// for the user-facing `archive` command. The row-level trigger fires
|
||||
@@ -245,7 +260,7 @@ describe('cache gate end-to-end (PGLite)', () => {
|
||||
{ page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'e', score: 0.9 } as unknown as SearchResult,
|
||||
];
|
||||
const emb = fakeEmbedding(6);
|
||||
await cache.store('zeta eta', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
await storeCurrent(cache, 'zeta eta', emb, results, fakeMeta(), { sourceId: 'default' });
|
||||
|
||||
// Bump only p2.
|
||||
await engine.putPage('test/p2', {
|
||||
|
||||
@@ -149,6 +149,149 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
expect(pgResults[0]?.slug).toBe(pgliteResults[0]?.slug);
|
||||
});
|
||||
|
||||
test('email citation metadata projects identically across engines', async () => {
|
||||
const slug = 'mail/example-citation';
|
||||
const page = {
|
||||
type: 'note' as const,
|
||||
title: 'Generated page title',
|
||||
compiled_truth: 'unique citation projection evidence',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
message_id: '<citation@example.com>',
|
||||
thread_id: 'thread-example',
|
||||
subject: 'Example exact email subject',
|
||||
},
|
||||
};
|
||||
const chunks = [{
|
||||
chunk_index: 0,
|
||||
chunk_text: page.compiled_truth,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
embedding: basisEmbedding(77),
|
||||
}];
|
||||
|
||||
await pgEngine.putPage(slug, page);
|
||||
await pgEngine.upsertChunks(slug, chunks);
|
||||
await pgliteEngine.putPage(slug, page);
|
||||
await pgliteEngine.upsertChunks(slug, chunks);
|
||||
|
||||
const results = [
|
||||
(await pgEngine.searchKeyword('unique citation projection evidence'))[0],
|
||||
(await pgliteEngine.searchKeyword('unique citation projection evidence'))[0],
|
||||
(await pgEngine.searchKeywordChunks('unique citation projection evidence'))[0],
|
||||
(await pgliteEngine.searchKeywordChunks('unique citation projection evidence'))[0],
|
||||
(await pgEngine.searchVector(basisEmbedding(77)))[0],
|
||||
(await pgliteEngine.searchVector(basisEmbedding(77)))[0],
|
||||
];
|
||||
|
||||
for (const result of results) {
|
||||
expect(result?.message_id).toBe('<citation@example.com>');
|
||||
expect(result?.thread_id).toBe('thread-example');
|
||||
expect(result?.source_subject).toBe('Example exact email subject');
|
||||
}
|
||||
|
||||
const nonEmailSlug = 'notes/generated-title-subject-gate';
|
||||
const nonEmailPage = {
|
||||
type: 'note' as const,
|
||||
title: 'Generated page title must stay a title',
|
||||
compiled_truth: 'unique non-email subject gate evidence',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
subject: 'Frontmatter subject without an email identity',
|
||||
thread_id: 'standalone-thread-id',
|
||||
},
|
||||
};
|
||||
const nonEmailChunks = [{
|
||||
chunk_index: 0,
|
||||
chunk_text: nonEmailPage.compiled_truth,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
}];
|
||||
await pgEngine.putPage(nonEmailSlug, nonEmailPage);
|
||||
await pgEngine.upsertChunks(nonEmailSlug, nonEmailChunks);
|
||||
await pgliteEngine.putPage(nonEmailSlug, nonEmailPage);
|
||||
await pgliteEngine.upsertChunks(nonEmailSlug, nonEmailChunks);
|
||||
|
||||
for (const result of [
|
||||
(await pgEngine.searchKeyword('unique non-email subject gate evidence'))[0],
|
||||
(await pgliteEngine.searchKeyword('unique non-email subject gate evidence'))[0],
|
||||
]) {
|
||||
expect(result?.message_id).toBeUndefined();
|
||||
expect(result?.thread_id).toBe('standalone-thread-id');
|
||||
expect(result?.source_subject).toBeUndefined();
|
||||
}
|
||||
|
||||
const whitespaceSlug = 'mail/whitespace-message-id';
|
||||
const whitespacePage = {
|
||||
type: 'note' as const,
|
||||
title: 'Whitespace Message-ID',
|
||||
compiled_truth: 'unique whitespace message id evidence',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
message_id: ' \t\n ',
|
||||
thread_id: 'thread-whitespace',
|
||||
subject: 'Subject must remain gated',
|
||||
},
|
||||
};
|
||||
const whitespaceChunks = [{
|
||||
chunk_index: 0,
|
||||
chunk_text: whitespacePage.compiled_truth,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
embedding: basisEmbedding(78),
|
||||
}];
|
||||
await pgEngine.putPage(whitespaceSlug, whitespacePage);
|
||||
await pgEngine.upsertChunks(whitespaceSlug, whitespaceChunks);
|
||||
await pgliteEngine.putPage(whitespaceSlug, whitespacePage);
|
||||
await pgliteEngine.upsertChunks(whitespaceSlug, whitespaceChunks);
|
||||
|
||||
for (const result of [
|
||||
(await pgEngine.searchKeyword('unique whitespace message id evidence'))[0],
|
||||
(await pgliteEngine.searchKeyword('unique whitespace message id evidence'))[0],
|
||||
(await pgEngine.searchKeywordChunks('unique whitespace message id evidence'))[0],
|
||||
(await pgliteEngine.searchKeywordChunks('unique whitespace message id evidence'))[0],
|
||||
(await pgEngine.searchVector(basisEmbedding(78)))[0],
|
||||
(await pgliteEngine.searchVector(basisEmbedding(78)))[0],
|
||||
]) {
|
||||
expect(result?.message_id).toBeUndefined();
|
||||
expect(result?.thread_id).toBe('thread-whitespace');
|
||||
expect(result?.source_subject).toBeUndefined();
|
||||
}
|
||||
|
||||
const malformedSlug = 'notes/numeric-email-frontmatter';
|
||||
const malformedPage = {
|
||||
type: 'note' as const,
|
||||
title: 'Numeric email frontmatter',
|
||||
compiled_truth: 'unique numeric email frontmatter evidence',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
message_id: 12345,
|
||||
thread_id: 67890,
|
||||
subject: 98765,
|
||||
},
|
||||
};
|
||||
const malformedChunks = [{
|
||||
chunk_index: 0,
|
||||
chunk_text: malformedPage.compiled_truth,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
embedding: basisEmbedding(79),
|
||||
}];
|
||||
await pgEngine.putPage(malformedSlug, malformedPage);
|
||||
await pgEngine.upsertChunks(malformedSlug, malformedChunks);
|
||||
await pgliteEngine.putPage(malformedSlug, malformedPage);
|
||||
await pgliteEngine.upsertChunks(malformedSlug, malformedChunks);
|
||||
|
||||
for (const result of [
|
||||
(await pgEngine.searchKeyword('unique numeric email frontmatter evidence'))[0],
|
||||
(await pgliteEngine.searchKeyword('unique numeric email frontmatter evidence'))[0],
|
||||
(await pgEngine.searchKeywordChunks('unique numeric email frontmatter evidence'))[0],
|
||||
(await pgliteEngine.searchKeywordChunks('unique numeric email frontmatter evidence'))[0],
|
||||
(await pgEngine.searchVector(basisEmbedding(79)))[0],
|
||||
(await pgliteEngine.searchVector(basisEmbedding(79)))[0],
|
||||
]) {
|
||||
expect(result?.message_id).toBeUndefined();
|
||||
expect(result?.thread_id).toBeUndefined();
|
||||
expect(result?.source_subject).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('hard-exclude is consistent across engines', async () => {
|
||||
// Both engines should hide test/ pages by default; both should opt
|
||||
// them back in via include_slug_prefixes.
|
||||
@@ -614,6 +757,7 @@ async function seedRelational(eng: BrainEngine) {
|
||||
['people/ep-inv-b', 'person'],
|
||||
['people/ep-emp-c', 'person'],
|
||||
['people/ep-mentioner', 'person'],
|
||||
['people/ep-quarantined', 'person'],
|
||||
];
|
||||
for (const [slug, type] of pages) {
|
||||
await eng.putPage(slug, { type, title: slug, compiled_truth: `${slug} body`, timeline: '' });
|
||||
@@ -626,7 +770,32 @@ async function seedRelational(eng: BrainEngine) {
|
||||
await eng.addLink('people/ep-inv-b', 'companies/ep-widget', '', 'invested_in', 'manual');
|
||||
await eng.addLink('people/ep-emp-c', 'companies/ep-widget', '', 'works_at', 'manual');
|
||||
await eng.addLink('people/ep-mentioner', 'companies/ep-widget', '', 'mentions', 'mentions');
|
||||
await eng.addLink('people/ep-quarantined', 'companies/ep-widget', '', 'invested_in', 'manual');
|
||||
await eng.addLink('people/ep-inv-a', 'companies/ep-other', '', 'invested_in', 'manual');
|
||||
await eng.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb
|
||||
WHERE slug = 'people/ep-quarantined' AND source_id = 'default'`,
|
||||
);
|
||||
|
||||
await eng.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived, created_at)
|
||||
VALUES ('ep-archived', 'ep-archived', false, NOW())`,
|
||||
);
|
||||
await eng.putPage(
|
||||
'companies/ep-archived-widget',
|
||||
{ type: 'company', title: 'Archived Widget', compiled_truth: 'Hidden company.', timeline: '' },
|
||||
{ sourceId: 'ep-archived' },
|
||||
);
|
||||
await eng.putPage(
|
||||
'people/ep-archived-investor',
|
||||
{ type: 'person', title: 'Archived Investor', compiled_truth: 'Hidden person.', timeline: '' },
|
||||
{ sourceId: 'ep-archived' },
|
||||
);
|
||||
await eng.addLink(
|
||||
'people/ep-archived-investor', 'companies/ep-archived-widget', '', 'invested_in', 'manual', undefined, undefined,
|
||||
{ fromSourceId: 'ep-archived', toSourceId: 'ep-archived' },
|
||||
);
|
||||
await eng.executeRaw(`UPDATE sources SET archived = true WHERE id = 'ep-archived'`);
|
||||
}
|
||||
|
||||
describeBoth('Engine parity — relationalFanout', () => {
|
||||
@@ -670,6 +839,22 @@ describeBoth('Engine parity — relationalFanout', () => {
|
||||
expect(pg.map(r => r.slug)).not.toContain('people/ep-mentioner');
|
||||
});
|
||||
|
||||
test('archive and quarantine visibility is identical across engines', async () => {
|
||||
const visibleOpts = { direction: 'in' as const, linkTypes: ['invested_in'] };
|
||||
const pgVisible = await pgEngine.relationalFanout(['companies/ep-widget'], visibleOpts);
|
||||
const pgliteVisible = await pgliteEngine.relationalFanout(['companies/ep-widget'], visibleOpts);
|
||||
expect(shape(pgVisible)).toEqual(shape(pgliteVisible));
|
||||
expect(pgVisible.map(r => r.slug)).not.toContain('people/ep-quarantined');
|
||||
|
||||
const archivedOpts = {
|
||||
sourceId: 'ep-archived', direction: 'in' as const, linkTypes: ['invested_in'],
|
||||
};
|
||||
const pgArchived = await pgEngine.relationalFanout(['companies/ep-archived-widget'], archivedOpts);
|
||||
const pgliteArchived = await pgliteEngine.relationalFanout(['companies/ep-archived-widget'], archivedOpts);
|
||||
expect(shape(pgArchived)).toEqual(shape(pgliteArchived));
|
||||
expect(pgArchived).toEqual([]);
|
||||
});
|
||||
|
||||
test('connects (multi-seed, both) identical across engines', async () => {
|
||||
const seeds = ['companies/ep-widget', 'companies/ep-other'];
|
||||
const pg = await pgEngine.relationalFanout(seeds, { direction: 'both' });
|
||||
|
||||
@@ -20,11 +20,17 @@ import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let chunkEmbedDim = 0;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const dim = await (engine as any).db.query(
|
||||
`SELECT atttypmod FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'`,
|
||||
);
|
||||
chunkEmbedDim = (dim.rows[0] as { atttypmod: number }).atttypmod;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -48,13 +54,18 @@ beforeEach(async () => {
|
||||
title: 'Alice Source-A',
|
||||
compiled_truth: 'Alice works on widgets in source A. Important context here.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
frontmatter: {
|
||||
message_id: '<source-a@example.com>',
|
||||
thread_id: 'thread-source-a',
|
||||
subject: 'Source A exact subject',
|
||||
},
|
||||
}, { sourceId: 'default' });
|
||||
await engine.upsertChunks('people/alice', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'Alice works on widgets in source A. Important context here.',
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 12,
|
||||
embedding: Float32Array.from({ length: chunkEmbedDim }, (_, i) => i === 0 ? 1 : 0),
|
||||
}], { sourceId: 'default' });
|
||||
|
||||
await engine.putPage('people/alice', {
|
||||
@@ -62,13 +73,18 @@ beforeEach(async () => {
|
||||
title: 'Alice Source-B',
|
||||
compiled_truth: 'Alice works on gadgets in source B. Important context here.',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
frontmatter: {
|
||||
message_id: '<source-b@example.com>',
|
||||
thread_id: 'thread-source-b',
|
||||
subject: 'Source B exact subject',
|
||||
},
|
||||
}, { sourceId: 'src-b' });
|
||||
await engine.upsertChunks('people/alice', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'Alice works on gadgets in source B. Important context here.',
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 12,
|
||||
embedding: Float32Array.from({ length: chunkEmbedDim }, (_, i) => i === 1 ? 1 : 0),
|
||||
}], { sourceId: 'src-b' });
|
||||
|
||||
await engine.putPage('people/bob', {
|
||||
@@ -95,6 +111,9 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBe('default');
|
||||
expect(r.message_id).toBe('<source-a@example.com>');
|
||||
expect(r.thread_id).toBe('thread-source-a');
|
||||
expect(r.source_subject).toBe('Source A exact subject');
|
||||
}
|
||||
});
|
||||
|
||||
@@ -103,6 +122,9 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBe('src-b');
|
||||
expect(r.message_id).toBe('<source-b@example.com>');
|
||||
expect(r.thread_id).toBe('thread-source-b');
|
||||
expect(r.source_subject).toBe('Source B exact subject');
|
||||
}
|
||||
});
|
||||
|
||||
@@ -170,16 +192,32 @@ describe('v0.34.1 source-isolation regression (#861)', () => {
|
||||
});
|
||||
|
||||
test('searchVector with sourceId filters HNSW candidate pool', async () => {
|
||||
// No real embeddings on the test pages; the WHERE cc.embedding IS NOT NULL
|
||||
// gate filters them out. We assert the contract via an empty result
|
||||
// rather than a positive match: with sourceId set, the SQL still runs
|
||||
// (no type or undefined-column errors).
|
||||
const synth = new Float32Array(1536).fill(0.01);
|
||||
const results = await engine.searchVector(synth, { sourceId: 'src-b' });
|
||||
// Either empty (no embeddings) or all from src-b. Both prove the
|
||||
// filter is wired without a runtime error.
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBe('src-b');
|
||||
const fixtures = [
|
||||
{
|
||||
sourceId: 'default', embeddingIndex: 0,
|
||||
message_id: '<source-a@example.com>', thread_id: 'thread-source-a',
|
||||
source_subject: 'Source A exact subject',
|
||||
},
|
||||
{
|
||||
sourceId: 'src-b', embeddingIndex: 1,
|
||||
message_id: '<source-b@example.com>', thread_id: 'thread-source-b',
|
||||
source_subject: 'Source B exact subject',
|
||||
},
|
||||
];
|
||||
|
||||
for (const fixture of fixtures) {
|
||||
const synth = Float32Array.from(
|
||||
{ length: chunkEmbedDim },
|
||||
(_, i) => i === fixture.embeddingIndex ? 1 : 0,
|
||||
);
|
||||
const results = await engine.searchVector(synth, { sourceId: fixture.sourceId });
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
for (const r of results) {
|
||||
expect(r.source_id).toBe(fixture.sourceId);
|
||||
expect(r.message_id).toBe(fixture.message_id);
|
||||
expect(r.thread_id).toBe(fixture.thread_id);
|
||||
expect(r.source_subject).toBe(fixture.source_subject);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -15,7 +15,8 @@
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { expandAnchors } from '../../src/core/search/two-pass.ts';
|
||||
import { expandAnchors, hydrateChunks } from '../../src/core/search/two-pass.ts';
|
||||
import { hybridSearch } from '../../src/core/search/hybrid.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -84,6 +85,43 @@ describe('v0.34 W0a — multi-source isolation in two-pass retrieval', () => {
|
||||
}
|
||||
});
|
||||
|
||||
test('expandAnchors honors federated sourceIds grants', async () => {
|
||||
const result = await expandAnchors(engine, [], {
|
||||
walkDepth: 0,
|
||||
nearSymbol: 'parseMarkdown',
|
||||
sourceIds: ['source-a'],
|
||||
});
|
||||
const chunkIds = result.map((r) => r.chunk_id);
|
||||
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
|
||||
`SELECT cc.id AS chunk_id, p.source_id
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.id = ANY($1::int[])`,
|
||||
[chunkIds],
|
||||
);
|
||||
|
||||
expect(rows.length).toBeGreaterThan(0);
|
||||
for (const row of rows) expect(row.source_id).toBe('source-a');
|
||||
});
|
||||
|
||||
test('hydrateChunks honors federated sourceIds grants', async () => {
|
||||
const chunkRows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.symbol_name_qualified = 'parseMarkdown'
|
||||
ORDER BY p.source_id`,
|
||||
[],
|
||||
);
|
||||
const rows = await hydrateChunks(
|
||||
engine,
|
||||
chunkRows.map((row) => row.id),
|
||||
{ sourceIds: ['source-a'] },
|
||||
);
|
||||
|
||||
expect(rows.length).toBeGreaterThan(0);
|
||||
for (const row of rows) expect(row.source_id).toBe('source-a');
|
||||
});
|
||||
|
||||
test('expandAnchors with nearSymbol and NO sourceId returns chunks from both sources (legacy cross-source mode preserved)', async () => {
|
||||
const result = await expandAnchors(engine, [], {
|
||||
walkDepth: 0,
|
||||
@@ -155,6 +193,115 @@ describe('v0.34 W0a — multi-source isolation in two-pass retrieval', () => {
|
||||
expect(r.source_id).toBe('source-a');
|
||||
}
|
||||
});
|
||||
|
||||
test('hybrid two-pass expansion honors federated sourceIds for direct chunk edges', async () => {
|
||||
const results = await hybridSearch(engine, 'callerInA', {
|
||||
walkDepth: 1,
|
||||
nearSymbol: 'callerInA',
|
||||
sourceIds: ['source-a'],
|
||||
expansion: false,
|
||||
useCache: false,
|
||||
mode: 'conservative',
|
||||
autocut: false,
|
||||
adaptiveReturn: false,
|
||||
limit: 50,
|
||||
});
|
||||
|
||||
expect(results.length).toBeGreaterThan(0);
|
||||
for (const result of results) {
|
||||
expect(result.source_id).toBe('source-a');
|
||||
expect(result.message_id).not.toBe('<denied@example.com>');
|
||||
expect(result.thread_id).not.toBe('denied-thread');
|
||||
expect(result.source_subject).not.toBe('Denied exact subject');
|
||||
}
|
||||
});
|
||||
|
||||
test('incoming direct edges cannot bypass source scope or visibility', async () => {
|
||||
const chunks = await engine.executeRaw<{ id: number; source_id: string; symbol_name_qualified: string }>(
|
||||
`SELECT cc.id, p.source_id, cc.symbol_name_qualified
|
||||
FROM content_chunks cc JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.symbol_name_qualified IN ('callerInA', 'parseMarkdown')`,
|
||||
[],
|
||||
);
|
||||
const callerA = chunks.find(r => r.source_id === 'source-a' && r.symbol_name_qualified === 'callerInA')!.id;
|
||||
const targetB = chunks.find(r => r.source_id === 'source-b' && r.symbol_name_qualified === 'parseMarkdown')!.id;
|
||||
const anchor = [{
|
||||
slug: 'code/src/markdown-b.ts', page_id: 0, title: '', type: 'code' as const,
|
||||
chunk_text: '', chunk_source: 'compiled_truth' as const,
|
||||
chunk_id: targetB, chunk_index: 0, score: 1, source_id: 'source-b', stale: false,
|
||||
}];
|
||||
|
||||
const scoped = await expandAnchors(engine, anchor, { walkDepth: 1, sourceId: 'source-b' });
|
||||
expect(scoped.map(r => r.chunk_id)).not.toContain(callerA);
|
||||
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb
|
||||
WHERE source_id = 'source-a' AND slug = 'code/src/caller-a.ts'`,
|
||||
[],
|
||||
);
|
||||
try {
|
||||
const hidden = await expandAnchors(engine, anchor, { walkDepth: 1 });
|
||||
expect(hidden.map(r => r.chunk_id)).not.toContain(callerA);
|
||||
} finally {
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter - 'quarantine'
|
||||
WHERE source_id = 'source-a' AND slug = 'code/src/caller-a.ts'`,
|
||||
[],
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('two-pass selection and hydration reject archived, quarantined, and deleted pages', async () => {
|
||||
const chunkRows = await engine.executeRaw<{ id: number; symbol_name_qualified: string }>(
|
||||
`SELECT cc.id, cc.symbol_name_qualified
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.source_id = 'source-b' OR cc.symbol_name_qualified = 'callerInA'`,
|
||||
[],
|
||||
);
|
||||
const deniedChunkId = chunkRows.find(r => r.symbol_name_qualified === 'parseMarkdown')!.id;
|
||||
const callerChunkId = chunkRows.find(r => r.symbol_name_qualified === 'callerInA')!.id;
|
||||
const anchor = [{
|
||||
slug: 'code/src/caller-a.ts', page_id: 0, title: '', type: 'code' as const,
|
||||
chunk_text: '', chunk_source: 'compiled_truth' as const,
|
||||
chunk_id: callerChunkId, chunk_index: 0, score: 1, source_id: 'source-a', stale: false,
|
||||
}];
|
||||
|
||||
const states = [
|
||||
{
|
||||
hide: () => engine.executeRaw(`UPDATE sources SET archived = true WHERE id = 'source-b'`, []),
|
||||
restore: () => engine.executeRaw(`UPDATE sources SET archived = false WHERE id = 'source-b'`, []),
|
||||
},
|
||||
{
|
||||
hide: () => engine.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb WHERE source_id = 'source-b'`, [],
|
||||
),
|
||||
restore: () => engine.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter - 'quarantine' WHERE source_id = 'source-b'`, [],
|
||||
),
|
||||
},
|
||||
{
|
||||
hide: () => engine.executeRaw(`UPDATE pages SET deleted_at = NOW() WHERE source_id = 'source-b'`, []),
|
||||
restore: () => engine.executeRaw(`UPDATE pages SET deleted_at = NULL WHERE source_id = 'source-b'`, []),
|
||||
},
|
||||
];
|
||||
|
||||
for (const state of states) {
|
||||
await state.hide();
|
||||
try {
|
||||
const near = await expandAnchors(engine, [], { nearSymbol: 'parseMarkdown', sourceId: 'source-b' });
|
||||
expect(near).toEqual([]);
|
||||
|
||||
const expanded = await expandAnchors(engine, anchor, { walkDepth: 1 });
|
||||
expect(expanded.map(r => r.chunk_id)).not.toContain(deniedChunkId);
|
||||
|
||||
const hydrated = await hydrateChunks(engine, [deniedChunkId]);
|
||||
expect(hydrated).toEqual([]);
|
||||
} finally {
|
||||
await state.restore();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
@@ -197,7 +344,9 @@ async function seedTwoSourcesWithSharedSymbol(engine: PGLiteEngine): Promise<voi
|
||||
// Page B: contains parseMarkdown in source-b
|
||||
const pageB = await engine.executeRaw<{ id: number }>(
|
||||
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, updated_at, created_at)
|
||||
VALUES ('code/src/markdown-b.ts', 'source-b', 'markdown-b.ts', 'code', 'export function parseMarkdown(s: string) { return s; }', '{}'::jsonb, NOW(), NOW())
|
||||
VALUES ('code/src/markdown-b.ts', 'source-b', 'markdown-b.ts', 'code', 'export function parseMarkdown(s: string) { return s; }',
|
||||
'{"message_id":"<denied@example.com>","thread_id":"denied-thread","subject":"Denied exact subject"}'::jsonb,
|
||||
NOW(), NOW())
|
||||
RETURNING id`,
|
||||
[],
|
||||
);
|
||||
@@ -231,4 +380,20 @@ async function seedTwoSourcesWithSharedSymbol(engine: PGLiteEngine): Promise<voi
|
||||
VALUES ($1, 'callerInA', 'parseMarkdown', 'calls', 'source-a', '{}'::jsonb)`,
|
||||
[callerChunk[0]!.id],
|
||||
);
|
||||
|
||||
// A resolved direct edge crossing into source-b. The two-pass walk must
|
||||
// discard this neighbor for both scalar sourceId and federated sourceIds.
|
||||
const deniedChunk = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT cc.id FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.source_id = 'source-b' AND cc.symbol_name_qualified = 'parseMarkdown' LIMIT 1`,
|
||||
[],
|
||||
);
|
||||
await engine.addCodeEdges([{
|
||||
from_chunk_id: callerChunk[0]!.id,
|
||||
to_chunk_id: deniedChunk[0]!.id,
|
||||
from_symbol_qualified: 'callerInA',
|
||||
to_symbol_qualified: 'parseMarkdown',
|
||||
edge_type: 'calls',
|
||||
}]);
|
||||
}
|
||||
|
||||
@@ -203,6 +203,53 @@ describe('PGLiteEngine: Search', () => {
|
||||
await engine.upsertChunks('concepts/rag', [
|
||||
{ chunk_index: 0, chunk_text: 'RAG combines retrieval with generation', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('mail/example', {
|
||||
type: 'note', title: 'Launch message',
|
||||
compiled_truth: 'Launch evidence for citation metadata.',
|
||||
frontmatter: {
|
||||
message_id: '<launch@example.com>',
|
||||
thread_id: 'thread-123',
|
||||
subject: 'Example launch subject',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('mail/example', [
|
||||
{ chunk_index: 0, chunk_text: 'Launch evidence for citation metadata', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('notes/generated-title', {
|
||||
type: 'note', title: 'Generated page title must stay a title',
|
||||
compiled_truth: 'Non-email evidence for subject gating.',
|
||||
frontmatter: {
|
||||
subject: 'Frontmatter subject without an email identity',
|
||||
thread_id: 'standalone-thread-id',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('notes/generated-title', [
|
||||
{ chunk_index: 0, chunk_text: 'Non-email evidence for subject gating', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('mail/whitespace-message-id', {
|
||||
type: 'note', title: 'Whitespace message id',
|
||||
compiled_truth: 'Whitespace-only email identity evidence.',
|
||||
frontmatter: {
|
||||
message_id: ' \t\n ',
|
||||
thread_id: 'thread-whitespace',
|
||||
subject: 'Subject must remain gated',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('mail/whitespace-message-id', [
|
||||
{ chunk_index: 0, chunk_text: 'Whitespace-only email identity evidence', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
await engine.putPage('notes/numeric-email-frontmatter', {
|
||||
type: 'note', title: 'Numeric email frontmatter',
|
||||
compiled_truth: 'Numeric email identity evidence.',
|
||||
frontmatter: {
|
||||
message_id: 12345,
|
||||
thread_id: 67890,
|
||||
subject: 98765,
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('notes/numeric-email-frontmatter', [
|
||||
{ chunk_index: 0, chunk_text: 'Numeric email identity evidence', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('searchKeyword returns results for matching term', async () => {
|
||||
@@ -211,6 +258,34 @@ describe('PGLiteEngine: Search', () => {
|
||||
expect(results[0].slug).toBe('companies/novamind');
|
||||
});
|
||||
|
||||
test('searchKeyword projects email citation identifiers from frontmatter', async () => {
|
||||
const results = await engine.searchKeyword('Launch evidence');
|
||||
expect(results[0].message_id).toBe('<launch@example.com>');
|
||||
expect(results[0].thread_id).toBe('thread-123');
|
||||
expect(results[0].source_subject).toBe('Example launch subject');
|
||||
});
|
||||
|
||||
test('searchKeyword never promotes a non-email title or subject to source_subject', async () => {
|
||||
const results = await engine.searchKeyword('Non-email evidence');
|
||||
expect(results[0].message_id).toBeUndefined();
|
||||
expect(results[0].thread_id).toBe('standalone-thread-id');
|
||||
expect(results[0].source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('searchKeyword treats whitespace-only message_id as absent', async () => {
|
||||
const results = await engine.searchKeyword('Whitespace-only email identity');
|
||||
expect(results[0].message_id).toBeUndefined();
|
||||
expect(results[0].thread_id).toBe('thread-whitespace');
|
||||
expect(results[0].source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('searchKeyword rejects non-string email citation frontmatter', async () => {
|
||||
const results = await engine.searchKeyword('Numeric email identity');
|
||||
expect(results[0].message_id).toBeUndefined();
|
||||
expect(results[0].thread_id).toBeUndefined();
|
||||
expect(results[0].source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('searchKeyword returns empty for non-matching term', async () => {
|
||||
const results = await engine.searchKeyword('xyznonexistent');
|
||||
expect(results.length).toBe(0);
|
||||
@@ -232,6 +307,42 @@ describe('PGLiteEngine: Search', () => {
|
||||
const results = await engine.searchVector(fakeEmbedding);
|
||||
expect(results.length).toBe(0);
|
||||
});
|
||||
|
||||
test('searchVector carries email citation metadata through the outer CTE', async () => {
|
||||
const embedding = new Float32Array(CHUNK_EMBED_DIM);
|
||||
embedding[0] = 1;
|
||||
await engine.upsertChunks('mail/example', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'Launch evidence for citation metadata',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding,
|
||||
},
|
||||
]);
|
||||
|
||||
const results = await engine.searchVector(embedding);
|
||||
expect(results[0].message_id).toBe('<launch@example.com>');
|
||||
expect(results[0].thread_id).toBe('thread-123');
|
||||
expect(results[0].source_subject).toBe('Example launch subject');
|
||||
});
|
||||
|
||||
test('searchVector treats whitespace-only message_id as absent', async () => {
|
||||
const embedding = new Float32Array(CHUNK_EMBED_DIM);
|
||||
embedding[1] = 1;
|
||||
await engine.upsertChunks('mail/whitespace-message-id', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'Whitespace-only email identity evidence',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding,
|
||||
},
|
||||
]);
|
||||
|
||||
const results = await engine.searchVector(embedding);
|
||||
expect(results[0].message_id).toBeUndefined();
|
||||
expect(results[0].thread_id).toBe('thread-whitespace');
|
||||
expect(results[0].source_subject).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
@@ -275,6 +386,19 @@ describe('PGLiteEngine: CJK keyword fallback (v0.32.7)', () => {
|
||||
await engine.upsertChunks('originals/english-essay', [
|
||||
{ chunk_index: 0, chunk_text: 'NovaMind builds AI agents for enterprise', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
|
||||
await engine.putPage('mail/cjk-example', {
|
||||
type: 'note', title: 'Generated CJK page title',
|
||||
compiled_truth: '郵件引用識別',
|
||||
frontmatter: {
|
||||
message_id: '<cjk@example.com>',
|
||||
thread_id: 'thread-cjk',
|
||||
subject: 'Example CJK email subject',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('mail/cjk-example', [
|
||||
{ chunk_index: 0, chunk_text: '郵件引用識別', chunk_source: 'compiled_truth' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('CJK query routes to LIKE branch and finds Chinese substring', async () => {
|
||||
@@ -295,6 +419,17 @@ describe('PGLiteEngine: CJK keyword fallback (v0.32.7)', () => {
|
||||
expect(results[0].slug).toBe('originals/korean-essay');
|
||||
});
|
||||
|
||||
test('CJK keyword page and chunk paths project email citation metadata', async () => {
|
||||
for (const result of [
|
||||
(await engine.searchKeyword('郵件引用'))[0],
|
||||
(await engine.searchKeywordChunks('郵件引用'))[0],
|
||||
]) {
|
||||
expect(result.message_id).toBe('<cjk@example.com>');
|
||||
expect(result.thread_id).toBe('thread-cjk');
|
||||
expect(result.source_subject).toBe('Example CJK email subject');
|
||||
}
|
||||
});
|
||||
|
||||
test('bigram ranking: 3-hit page outranks 1-hit page', async () => {
|
||||
// Add another Chinese page with only ONE occurrence of 测试.
|
||||
await engine.putPage('originals/chinese-one-hit', {
|
||||
|
||||
@@ -38,10 +38,12 @@ describe('quarantine marker (hides)', () => {
|
||||
expect(filterOutQuarantined(pages).map((p) => p.slug)).toEqual(['a', 'c']);
|
||||
});
|
||||
|
||||
test('QUARANTINE_FILTER_FRAGMENT is a negated JSONB existence check on p', () => {
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain("p.frontmatter");
|
||||
test('QUARANTINE_FILTER_FRAGMENT requires object-shaped JSONB and excludes the marker', () => {
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain('p.frontmatter');
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain("jsonb_typeof");
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain("= 'object'");
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain("? 'quarantine'");
|
||||
expect(QUARANTINE_FILTER_FRAGMENT.startsWith('NOT (')).toBe(true);
|
||||
expect(QUARANTINE_FILTER_FRAGMENT).toContain('AND NOT');
|
||||
});
|
||||
|
||||
test('quarantineFilterFragment(alias) parameterizes the page alias; constant is the p-instance', () => {
|
||||
|
||||
@@ -281,4 +281,12 @@ describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => {
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p.id IS NULL');
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p.generation <>');
|
||||
});
|
||||
|
||||
test('visibility seal is unconditional and uses canonical page/source predicates', () => {
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('jsonb_array_elements(qc.results)');
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('LEFT JOIN sources s_visible');
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('p_visible.deleted_at IS NULL');
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain('NOT s_visible.archived');
|
||||
expect(CACHE_GATE_WHERE_CLAUSE).toContain("COALESCE(p_visible.frontmatter, '{}'::jsonb) ? 'quarantine'");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -23,6 +23,7 @@ import { resolveHardExcludes } from '../src/core/search/source-boost.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let visiblePageId: number;
|
||||
|
||||
const conservativeHash = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const balancedHash = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
@@ -45,6 +46,14 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const page = await engine.putPage('cache/knobs-visible-fixture', {
|
||||
type: 'note',
|
||||
title: 'Knobs cache fixture',
|
||||
compiled_truth: 'visible knobs cache fixture',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
visiblePageId = page.id;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -80,10 +89,25 @@ const makeResults = (label: string, n: number): SearchResult[] =>
|
||||
chunk_index: i,
|
||||
type: 'note' as const,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
page_id: i + 1,
|
||||
page_id: visiblePageId,
|
||||
stale: false,
|
||||
}));
|
||||
|
||||
async function storeCurrent(
|
||||
cache: SemanticQueryCache,
|
||||
...args: Parameters<SemanticQueryCache['store']>
|
||||
): Promise<void> {
|
||||
const [queryText, queryEmbedding, results, meta, opts = {}] = args;
|
||||
const rows = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
await cache.store(queryText, queryEmbedding, results, meta, {
|
||||
...opts,
|
||||
maxGenerationAtSearchStart:
|
||||
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
|
||||
});
|
||||
}
|
||||
|
||||
describe('cacheRowId is bifurcated by knobsHash', () => {
|
||||
test('same (query, source) but different knobs → different row IDs', () => {
|
||||
const id1 = cacheRowId('what is the meaning of life', 'default', conservativeHash);
|
||||
@@ -116,7 +140,7 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
const tokenmaxResults = makeResults('tokenmax', 50);
|
||||
|
||||
// Write under tokenmax knobs.
|
||||
await cache.store('what is the meaning of life', emb, tokenmaxResults, {
|
||||
await storeCurrent(cache, 'what is the meaning of life', emb, tokenmaxResults, {
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: true,
|
||||
@@ -137,13 +161,13 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(2);
|
||||
|
||||
await cache.store('q', emb, makeResults('conservative', 10), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('conservative', 10), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: conservativeHash });
|
||||
await cache.store('q', emb, makeResults('balanced', 25), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('balanced', 25), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
await cache.store('q', emb, makeResults('tokenmax', 50), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('tokenmax', 50), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: true,
|
||||
}, { knobsHash: tokenmaxHash });
|
||||
|
||||
@@ -194,11 +218,11 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(4);
|
||||
|
||||
await cache.store('q', emb, makeResults('first', 5), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('first', 5), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
await cache.store('q', emb, makeResults('second', 7), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('second', 7), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
@@ -217,7 +241,7 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(5);
|
||||
|
||||
await cache.store('q', emb, makeResults('no-mode', 3), {
|
||||
await storeCurrent(cache, 'q', emb, makeResults('no-mode', 3), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
});
|
||||
|
||||
@@ -251,7 +275,7 @@ describe('hard-exclude cache isolation (#2825)', () => {
|
||||
// Simulate a no-exclude process writing results that include a slug the
|
||||
// excluding process must never see.
|
||||
const leaky = makeResults('private', 5);
|
||||
await cache.store('who is alice', emb, leaky, {
|
||||
await storeCurrent(cache, 'who is alice', emb, leaky, {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: noEnvHash });
|
||||
|
||||
@@ -269,7 +293,7 @@ describe('hard-exclude cache isolation (#2825)', () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(7);
|
||||
|
||||
await cache.store('who is alice', emb, makeResults('filtered', 3), {
|
||||
await storeCurrent(cache, 'who is alice', emb, makeResults('filtered', 3), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: envExcludeHash });
|
||||
|
||||
|
||||
+137
-14
@@ -22,6 +22,7 @@ import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let visiblePageId: number;
|
||||
|
||||
// Build a stable, normalized embedding. PGLite ships pgvector with 1536-dim
|
||||
// support (the default); a smaller test dim won't match the column. We
|
||||
@@ -60,7 +61,7 @@ function makeOrthogonalEmbedding(seed: number, dim = DIM): Float32Array {
|
||||
function makeResult(slug: string): SearchResult {
|
||||
return {
|
||||
slug,
|
||||
page_id: 1,
|
||||
page_id: visiblePageId,
|
||||
title: `Title for ${slug}`,
|
||||
type: 'concept',
|
||||
chunk_text: `chunk text for ${slug}`,
|
||||
@@ -79,6 +80,26 @@ const META: HybridSearchMeta = {
|
||||
intent: 'general',
|
||||
};
|
||||
|
||||
type StoreOpts = NonNullable<Parameters<SemanticQueryCache['store']>[4]>;
|
||||
|
||||
async function storeCurrent(
|
||||
cache: SemanticQueryCache,
|
||||
queryText: string,
|
||||
queryEmbedding: Float32Array,
|
||||
results: SearchResult[],
|
||||
meta: HybridSearchMeta,
|
||||
opts: StoreOpts = {},
|
||||
): Promise<void> {
|
||||
const rows = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
await cache.store(queryText, queryEmbedding, results, meta, {
|
||||
...opts,
|
||||
maxGenerationAtSearchStart:
|
||||
opts.maxGenerationAtSearchStart ?? Number(rows[0]?.v ?? 0),
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
// v0.36.2.0: DEFAULT_EMBEDDING_DIMENSIONS flipped to 1280 (ZE Matryoshka).
|
||||
// This test hardcodes DIM=1536 in its embeddings. If another test file in
|
||||
@@ -95,6 +116,14 @@ beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const page = await engine.putPage('cache/visible-fixture', {
|
||||
type: 'note',
|
||||
title: 'Visible cache fixture',
|
||||
compiled_truth: 'visible cache fixture',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
visiblePageId = page.id;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -141,7 +170,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
|
||||
const emb = makeEmbedding(1);
|
||||
const results = [makeResult('a'), makeResult('b')];
|
||||
|
||||
await cache.store('what is foo', emb, results, META);
|
||||
await storeCurrent(cache, 'what is foo', emb, results, META);
|
||||
const hit = await cache.lookup(emb);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
@@ -163,7 +192,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
|
||||
mag = Math.sqrt(mag);
|
||||
for (let i = 0; i < DIM; i++) near[i] /= mag;
|
||||
|
||||
await cache.store('what is foo', base, [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'what is foo', base, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(near);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
@@ -174,7 +203,7 @@ describe('SemanticQueryCache \u2014 store + lookup', () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const a = makeEmbedding(1);
|
||||
const b = makeOrthogonalEmbedding(2);
|
||||
await cache.store('q1', a, [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'q1', a, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(b);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
@@ -184,7 +213,7 @@ describe('SemanticQueryCache \u2014 TTL', () => {
|
||||
test('stale row (past TTL) is not returned', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { ttlSeconds: 1 });
|
||||
const emb = makeEmbedding(42);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
|
||||
await storeCurrent(cache, 'q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
|
||||
|
||||
// Manually rewind created_at to simulate expiration.
|
||||
await engine.executeRaw(
|
||||
@@ -195,34 +224,128 @@ describe('SemanticQueryCache \u2014 TTL', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 source isolation', () => {
|
||||
test('different source_id cannot read each other\u2019s rows', async () => {
|
||||
describe('SemanticQueryCache — source isolation', () => {
|
||||
test('different source_id cannot read each other’s rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(7);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
|
||||
await storeCurrent(cache, 'q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
|
||||
const hitB = await cache.lookup(emb, { sourceId: 'src-B' });
|
||||
expect(hitB.hit).toBe(false);
|
||||
const hitA = await cache.lookup(emb, { sourceId: 'src-A' });
|
||||
expect(hitA.hit).toBe(true);
|
||||
});
|
||||
|
||||
test('archiving a source invalidates its cached result without a page write', async () => {
|
||||
const sourceId = 'cache-archive-src';
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived)
|
||||
VALUES ($1, $1, false)
|
||||
ON CONFLICT (id) DO UPDATE SET archived = false`,
|
||||
[sourceId],
|
||||
);
|
||||
const page = await engine.putPage(
|
||||
'cache/archive-visibility',
|
||||
{
|
||||
type: 'note',
|
||||
title: 'Archive visibility',
|
||||
compiled_truth: 'archive visibility cache canary',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
},
|
||||
{ sourceId },
|
||||
);
|
||||
const result = { ...makeResult(page.slug), page_id: page.id, source_id: sourceId };
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(71);
|
||||
|
||||
await storeCurrent(cache, 'archive visibility', emb, [result], META, { sourceId });
|
||||
expect((await cache.lookup(emb, { sourceId })).hit).toBe(true);
|
||||
|
||||
await engine.executeRaw(`UPDATE sources SET archived = true WHERE id = $1`, [sourceId]);
|
||||
try {
|
||||
expect((await cache.lookup(emb, { sourceId })).hit).toBe(false);
|
||||
} finally {
|
||||
await engine.executeRaw(`UPDATE sources SET archived = false WHERE id = $1`, [sourceId]);
|
||||
}
|
||||
});
|
||||
|
||||
test('a result hard-deleted before cache snapshot cannot be served', async () => {
|
||||
const page = await engine.putPage('cache/deleted-before-store', {
|
||||
type: 'note',
|
||||
title: 'Deleted before store',
|
||||
compiled_truth: 'private cache race canary',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
const result = { ...makeResult(page.slug), page_id: page.id, source_id: 'default' };
|
||||
await engine.executeRaw(`DELETE FROM pages WHERE id = $1`, [page.id]);
|
||||
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(72);
|
||||
await storeCurrent(cache, 'deleted before store', emb, [result], META);
|
||||
|
||||
expect((await cache.lookup(emb)).hit).toBe(false);
|
||||
});
|
||||
|
||||
test('writeback without a pre-search generation fails closed', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(77);
|
||||
|
||||
await cache.store('missing generation', emb, [makeResult('a')], META);
|
||||
|
||||
expect((await cache.lookup(emb)).hit).toBe(false);
|
||||
expect((await cache.stats()).total_rows).toBe(0);
|
||||
});
|
||||
|
||||
test('a concurrent page mutation prevents stale result writeback', async () => {
|
||||
const clock = await engine.executeRaw<{ v: number }>(
|
||||
`SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
|
||||
);
|
||||
const generationAtSearchStart = Number(clock[0]?.v ?? 0);
|
||||
const staleResult = { ...makeResult('cache/visible-fixture'), page_id: visiblePageId };
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET compiled_truth = 'redacted after search' WHERE id = $1`,
|
||||
[visiblePageId],
|
||||
);
|
||||
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(73);
|
||||
await storeCurrent(cache, 'concurrent mutation', emb, [staleResult], META, {
|
||||
maxGenerationAtSearchStart: generationAtSearchStart,
|
||||
});
|
||||
|
||||
expect((await cache.lookup(emb)).hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 management', () => {
|
||||
test('clear() wipes all rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(9);
|
||||
await cache.store('q1', emb, [makeResult('a')], META);
|
||||
await cache.store('q2', makeEmbedding(10), [makeResult('b')], META);
|
||||
await storeCurrent(cache, 'q1', emb, [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'q2', makeEmbedding(10), [makeResult('b')], META);
|
||||
const removed = await cache.clear();
|
||||
expect(removed).toBeGreaterThanOrEqual(2);
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(0);
|
||||
});
|
||||
|
||||
test('source-scoped clear removes typed scalar and federated scopes containing the source', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(74);
|
||||
await storeCurrent(cache, 'scalar-a', emb, [makeResult('a')], META, { sourceId: '["scalar","source-a"]' });
|
||||
await storeCurrent(cache, 'set-ab', makeEmbedding(75), [makeResult('ab')], META, { sourceId: '["set","source-a","source-b"]' });
|
||||
await storeCurrent(cache, 'scalar-b', makeEmbedding(76), [makeResult('b')], META, { sourceId: '["scalar","source-b"]' });
|
||||
|
||||
expect(await cache.clear({ sourceId: 'source-a' })).toBe(2);
|
||||
const rows = await engine.executeRaw<{ source_id: string }>(`SELECT source_id FROM query_cache ORDER BY source_id`);
|
||||
expect(rows.map(r => r.source_id)).toEqual(['["scalar","source-b"]']);
|
||||
});
|
||||
|
||||
test('prune() deletes only stale rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
await cache.store('fresh', makeEmbedding(11), [makeResult('a')], META);
|
||||
await cache.store('stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
|
||||
await storeCurrent(cache, 'fresh', makeEmbedding(11), [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache SET created_at = now() - interval '10 seconds' WHERE query_text = 'stale'`,
|
||||
);
|
||||
@@ -236,7 +359,7 @@ describe('SemanticQueryCache \u2014 management', () => {
|
||||
test('stats() reports fresh / stale / total / hit counters', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(13);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'q', emb, [makeResult('a')], META);
|
||||
await cache.lookup(emb); // bump hit
|
||||
// Hit bump is async/fire-and-forget; give it a moment to land.
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
@@ -252,7 +375,7 @@ describe('SemanticQueryCache \u2014 disabled', () => {
|
||||
test('disabled cache is a pure no-op on lookup', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { enabled: false });
|
||||
const emb = makeEmbedding(99);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
await storeCurrent(cache, 'q', emb, [makeResult('a')], META);
|
||||
// Even after a store call, lookup must miss because enabled=false.
|
||||
const hit = await cache.lookup(emb);
|
||||
expect(hit.hit).toBe(false);
|
||||
|
||||
@@ -36,6 +36,7 @@ beforeAll(async () => {
|
||||
['people/employee-c', 'person', 'Employee C'],
|
||||
['people/mentioner', 'person', 'Mentioner'],
|
||||
['people/deleted-investor', 'person', 'Deleted Investor'],
|
||||
['people/quarantined-investor', 'person', 'Quarantined Investor'],
|
||||
];
|
||||
for (const [slug, type, title] of pages) {
|
||||
await eng.putPage(slug, { type: type as 'company' | 'person', title, compiled_truth: `${title} body`, timeline: '' });
|
||||
@@ -50,11 +51,39 @@ beforeAll(async () => {
|
||||
await eng.addLink('people/employee-c', 'companies/widget-co', '', 'works_at', 'manual');
|
||||
await eng.addLink('people/mentioner', 'companies/widget-co', '', 'mentions', 'mentions');
|
||||
await eng.addLink('people/deleted-investor', 'companies/widget-co', '', 'invested_in', 'manual');
|
||||
await eng.addLink('people/quarantined-investor', 'companies/widget-co', '', 'invested_in', 'manual');
|
||||
// investor-a also invested in other-co → widget-co and other-co connect via investor-a.
|
||||
await eng.addLink('people/investor-a', 'companies/other-co', '', 'invested_in', 'manual');
|
||||
|
||||
// Soft-delete one investor; it must never surface.
|
||||
await eng.executeRaw(`UPDATE pages SET deleted_at = now() WHERE slug = $1`, ['people/deleted-investor']);
|
||||
await eng.executeRaw(
|
||||
`UPDATE pages SET frontmatter = frontmatter || '{"quarantine":true}'::jsonb WHERE slug = $1`,
|
||||
['people/quarantined-investor'],
|
||||
);
|
||||
|
||||
// A complete graph inside an archived source must be invisible even when
|
||||
// the caller explicitly scopes the fanout to that source.
|
||||
await eng.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived, created_at)
|
||||
VALUES ('archived-rel', 'archived-rel', false, NOW())`,
|
||||
[],
|
||||
);
|
||||
await eng.putPage(
|
||||
'companies/archived-widget',
|
||||
{ type: 'company', title: 'Archived Widget', compiled_truth: 'Hidden company.', timeline: '' },
|
||||
{ sourceId: 'archived-rel' },
|
||||
);
|
||||
await eng.putPage(
|
||||
'people/archived-investor',
|
||||
{ type: 'person', title: 'Archived Investor', compiled_truth: 'Hidden person.', timeline: '' },
|
||||
{ sourceId: 'archived-rel' },
|
||||
);
|
||||
await eng.addLink(
|
||||
'people/archived-investor', 'companies/archived-widget', '', 'invested_in', 'manual', undefined, undefined,
|
||||
{ fromSourceId: 'archived-rel', toSourceId: 'archived-rel' },
|
||||
);
|
||||
await eng.executeRaw(`UPDATE sources SET archived = true WHERE id = 'archived-rel'`, []);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -77,6 +106,19 @@ describe('relationalFanout', () => {
|
||||
expect(rows.map(r => r.slug)).not.toContain('people/deleted-investor');
|
||||
});
|
||||
|
||||
test('quarantined pages are excluded', async () => {
|
||||
const rows = await eng.relationalFanout(['companies/widget-co'], { direction: 'in', linkTypes: ['invested_in'] });
|
||||
expect(rows.map(r => r.slug)).not.toContain('people/quarantined-investor');
|
||||
});
|
||||
|
||||
test('archived sources are excluded', async () => {
|
||||
const rows = await eng.relationalFanout(
|
||||
['companies/archived-widget'],
|
||||
{ sourceId: 'archived-rel', direction: 'in', linkTypes: ['invested_in'] },
|
||||
);
|
||||
expect(rows).toEqual([]);
|
||||
});
|
||||
|
||||
test('mentions excluded by default, included on opt-in', async () => {
|
||||
const off = await eng.relationalFanout(['companies/widget-co'], { direction: 'in' });
|
||||
expect(off.map(r => r.slug)).not.toContain('people/mentioner');
|
||||
|
||||
@@ -23,12 +23,34 @@ beforeAll(async () => {
|
||||
|
||||
await eng.putPage('companies/widget-co', { type: 'company', title: 'Widget Co', compiled_truth: 'A payments company.', timeline: '' });
|
||||
// The investor's body deliberately NEVER mentions Widget Co — only the edge connects them.
|
||||
await eng.putPage('people/alice-example', { type: 'person', title: 'Alice Example', compiled_truth: 'Alice is a seed-stage investor based in Lisbon.', timeline: '' });
|
||||
await eng.putPage('people/alice-example', {
|
||||
type: 'person',
|
||||
title: 'Alice Example',
|
||||
compiled_truth: 'Alice is a seed-stage investor based in Lisbon.',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
message_id: '<alice-investor@example.com>',
|
||||
thread_id: 'thread-alice-investor',
|
||||
subject: 'Investor profile update',
|
||||
},
|
||||
});
|
||||
await eng.upsertChunks('people/alice-example', [{
|
||||
chunk_index: 0, chunk_text: 'Alice is a seed-stage investor based in Lisbon.',
|
||||
chunk_source: 'compiled_truth', embedding: new Float32Array(dim), token_count: 8,
|
||||
}] satisfies ChunkInput[]);
|
||||
await eng.addLink('people/alice-example', 'companies/widget-co', '', 'invested_in', 'manual');
|
||||
await eng.putPage('people/numeric-example', {
|
||||
type: 'person',
|
||||
title: 'Numeric Example',
|
||||
compiled_truth: 'A second seed-stage investor.',
|
||||
timeline: '',
|
||||
frontmatter: { message_id: 12345, thread_id: 67890, subject: 98765 },
|
||||
});
|
||||
await eng.addLink('people/numeric-example', 'companies/widget-co', '', 'invested_in', 'manual');
|
||||
await eng.putPage('people/hydrate-quarantined', {
|
||||
type: 'person', title: 'Hydrate Quarantined', compiled_truth: 'Must remain hidden.', timeline: '',
|
||||
frontmatter: { quarantine: true, message_id: '<hidden@example.com>', subject: 'Hidden exact subject' },
|
||||
});
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => { await eng.disconnect(); });
|
||||
@@ -43,6 +65,15 @@ describe('buildRelationalArm', () => {
|
||||
expect(alice!.relational_seed).toBe('companies/widget-co');
|
||||
// chunk-backed page → reinforces a REAL chunk id (not synthetic 0).
|
||||
expect(alice!.chunk_id).toBeGreaterThan(0);
|
||||
expect(alice!.message_id).toBe('<alice-investor@example.com>');
|
||||
expect(alice!.thread_id).toBe('thread-alice-investor');
|
||||
expect(alice!.source_subject).toBe('Investor profile update');
|
||||
|
||||
const malformed = list.find(r => r.slug === 'people/numeric-example');
|
||||
expect(malformed).toBeDefined();
|
||||
expect(malformed!.message_id).toBeUndefined();
|
||||
expect(malformed!.thread_id).toBeUndefined();
|
||||
expect(malformed!.source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('non-relational query is a pure no-op', async () => {
|
||||
@@ -57,6 +88,21 @@ describe('buildRelationalArm', () => {
|
||||
expect(list).toEqual([]);
|
||||
});
|
||||
|
||||
test('hydrate rejects a quarantined row even if fanout returns it', async () => {
|
||||
const original = eng.relationalFanout.bind(eng);
|
||||
eng.relationalFanout = async () => [{
|
||||
source_id: 'default', slug: 'people/hydrate-quarantined', hop: 1,
|
||||
edge_count: 1, via_link_types: ['invested_in'],
|
||||
path: ['companies/widget-co', 'people/hydrate-quarantined'], canonical_chunk_id: null,
|
||||
}];
|
||||
try {
|
||||
const list = await buildRelationalArm(eng, 'who invested in widget-co');
|
||||
expect(list).toEqual([]);
|
||||
} finally {
|
||||
eng.relationalFanout = original;
|
||||
}
|
||||
});
|
||||
|
||||
test('fail-open: fanout error returns [] + errored meta, never throws', async () => {
|
||||
const original = eng.relationalFanout.bind(eng);
|
||||
let captured: { errored?: boolean } = {};
|
||||
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
it('is 13 (12→13 invalidates cached rows without email citation metadata)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -410,7 +410,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// #2825: bumped 11→12 to fold the resolved hard-exclude prefix list
|
||||
// (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across
|
||||
// processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
// Email citation metadata projection: bumped 12→13 because cached
|
||||
// SearchResult rows from older versions lack message_id, thread_id, and
|
||||
// Message-ID-gated source_subject.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -575,8 +578,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
test('KNOBS_HASH_VERSION is 13 (12→13 email citation result-schema projection)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -27,7 +27,18 @@ function res(slug: string, score: number, title = slug): SearchResult {
|
||||
|
||||
describe('applyAliasHop', () => {
|
||||
test('injects an absent canonical when the query matches its alias', async () => {
|
||||
await engine.putPage('projects/mingtang', { type: 'note', title: 'The Mingtang', compiled_truth: 'Indoor Greek amphitheater.' });
|
||||
await engine.putPage('projects/mingtang', {
|
||||
type: 'note',
|
||||
title: 'The Mingtang',
|
||||
compiled_truth: 'Indoor Greek amphitheater.',
|
||||
frontmatter: {
|
||||
message_id: '<mingtang@example.com>',
|
||||
thread_id: 'thread-mingtang',
|
||||
subject: 'Mingtang project update',
|
||||
// Internal SQL projection aliases are not trusted frontmatter fields.
|
||||
source_subject: 'Spoofed projection alias',
|
||||
},
|
||||
});
|
||||
await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']);
|
||||
|
||||
const organic = [res('notes/unrelated', 0.5)];
|
||||
@@ -38,6 +49,85 @@ describe('applyAliasHop', () => {
|
||||
expect(hit!.alias_hit).toBe(true);
|
||||
expect(out[0].slug).toBe('projects/mingtang'); // injected at top-of-organic + ε
|
||||
expect(hit!.score).toBeGreaterThan(0.5);
|
||||
expect(hit!.message_id).toBe('<mingtang@example.com>');
|
||||
expect(hit!.thread_id).toBe('thread-mingtang');
|
||||
expect(hit!.source_subject).toBe('Mingtang project update');
|
||||
});
|
||||
|
||||
test('raw frontmatter cannot forge source_subject without allowlisted subject', async () => {
|
||||
await engine.putPage('mail/forged-subject', {
|
||||
type: 'note',
|
||||
title: 'Forged subject',
|
||||
compiled_truth: 'Evidence.',
|
||||
frontmatter: {
|
||||
message_id: '<forged@example.com>',
|
||||
source_subject: 'Forged projection alias',
|
||||
},
|
||||
});
|
||||
await engine.setPageAliases('mail/forged-subject', 'default', ['forged mail']);
|
||||
|
||||
const out = await applyAliasHop(engine, [], 'forged mail', { sourceId: 'default' });
|
||||
expect(out).toHaveLength(1);
|
||||
expect(out[0]!.message_id).toBe('<forged@example.com>');
|
||||
expect(out[0]!.source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('does not inject quarantined alias pages', async () => {
|
||||
await engine.putPage('notes/quarantined-alias', {
|
||||
type: 'note', title: 'Quarantined alias', compiled_truth: 'Hidden.',
|
||||
frontmatter: { quarantine: true, message_id: '<hidden@example.com>', subject: 'Hidden' },
|
||||
});
|
||||
await engine.setPageAliases('notes/quarantined-alias', 'default', ['hidden alias']);
|
||||
|
||||
const out = await applyAliasHop(engine, [], 'hidden alias', { sourceId: 'default' });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('does not inject aliases whose top-level frontmatter is an encoded JSON string', async () => {
|
||||
const encoded = JSON.stringify({
|
||||
message_id: '<encoded@example.com>',
|
||||
thread_id: 'encoded-thread',
|
||||
subject: 'Encoded subject',
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
|
||||
VALUES ('mail/encoded-frontmatter', 'default', 'Encoded frontmatter', 'note', 'Hidden.', to_jsonb($1::text), NOW(), NOW())`,
|
||||
[encoded],
|
||||
);
|
||||
await engine.setPageAliases('mail/encoded-frontmatter', 'default', ['encoded mail']);
|
||||
|
||||
const out = await applyAliasHop(engine, [], 'encoded mail', { sourceId: 'default' });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('encoded-string quarantine markers remain hidden from alias retrieval', async () => {
|
||||
const encoded = JSON.stringify({ quarantine: true });
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
|
||||
VALUES ('notes/encoded-quarantine', 'default', 'Encoded quarantine', 'note', 'Hidden.', to_jsonb($1::text), NOW(), NOW())`,
|
||||
[encoded],
|
||||
);
|
||||
await engine.setPageAliases('notes/encoded-quarantine', 'default', ['encoded hidden']);
|
||||
|
||||
const out = await applyAliasHop(engine, [], 'encoded hidden', { sourceId: 'default' });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('does not inject aliases from archived sources', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived, created_at)
|
||||
VALUES ('archived-alias', 'archived-alias', true, NOW())`,
|
||||
[],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, created_at, updated_at)
|
||||
VALUES ('notes/archived-alias', 'archived-alias', 'Archived alias', 'note', 'Hidden.', '{}', NOW(), NOW())`,
|
||||
[],
|
||||
);
|
||||
await engine.setPageAliases('notes/archived-alias', 'archived-alias', ['archived alias']);
|
||||
|
||||
const out = await applyAliasHop(engine, [], 'archived alias', { sourceId: 'archived-alias' });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
test('romanization/CJK alias also resolves', async () => {
|
||||
@@ -61,6 +151,17 @@ describe('applyAliasHop', () => {
|
||||
test('P0 source-isolation: alias hop boosts only the aliased source, not the same slug in another source', async () => {
|
||||
// The alias belongs to the src-b page only. Two same-slug results, different
|
||||
// sources, both in the organic set. The hop must boost ONLY src-b's row.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, created_at) VALUES
|
||||
('src-a', 'src-a', NOW()), ('src-b', 'src-b', NOW())`,
|
||||
[],
|
||||
);
|
||||
await engine.putPage(
|
||||
'shared/page', { type: 'note', title: 'Shared A', compiled_truth: 'A' }, { sourceId: 'src-a' },
|
||||
);
|
||||
await engine.putPage(
|
||||
'shared/page', { type: 'note', title: 'Shared B', compiled_truth: 'B' }, { sourceId: 'src-b' },
|
||||
);
|
||||
await engine.setPageAliases('shared/page', 'src-b', ['only in b']);
|
||||
const organic = [
|
||||
{ slug: 'shared/page', source_id: 'src-a', score: 0.5 } as unknown as SearchResult,
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
/**
|
||||
* Fail-open regression for the typed cache scope key (takeover of #2875).
|
||||
*
|
||||
* `cacheScopeKey` rejects forged/malformed source ids (typed-key contract,
|
||||
* test/cache-scope-key.test.ts). That throw is correct for direct callers,
|
||||
* but inside `hybridSearchCached` an invalid scope id must degrade to
|
||||
* "skip the semantic cache" — the cache must never break the search hot
|
||||
* path. Before the fix, the key was computed inline at the cache-lookup
|
||||
* call site (outside any catch), so a forged sourceId that reached the
|
||||
* cached path rejected the whole search once a query embedding was
|
||||
* available.
|
||||
*
|
||||
* Serial file: mock.module leaks across files in a shared shard process.
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, test, mock } from 'bun:test';
|
||||
|
||||
const makeEmbedding = (): Float32Array => {
|
||||
const arr = new Float32Array(1536);
|
||||
for (let i = 0; i < 1536; i++) arr[i] = Math.sin(1 + i * 0.001);
|
||||
let norm = 0;
|
||||
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
};
|
||||
|
||||
// Mock the embedding module BEFORE importing hybrid.ts so the cache-lookup
|
||||
// embed succeeds and the lookup call site is actually reached (the failure
|
||||
// mode under test only fired once a query embedding existed).
|
||||
mock.module('../../src/core/embedding.ts', () => ({
|
||||
embed: async () => makeEmbedding(),
|
||||
embedQuery: async () => makeEmbedding(),
|
||||
}));
|
||||
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-fake' },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await engine.putPage('failopen/fixture', {
|
||||
type: 'note',
|
||||
title: 'Fail-open fixture',
|
||||
compiled_truth: 'fail open fixture content',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('hybridSearchCached cache-scope fail-open', () => {
|
||||
test('a forged sourceId skips the cache instead of breaking the search', async () => {
|
||||
const { hybridSearchCached } = await import('../../src/core/search/hybrid.ts');
|
||||
// '__set__:a,b' is a forged legacy set-encoding — cacheScopeKey throws
|
||||
// on it. The search itself must still resolve (cache silently skipped).
|
||||
const results = await hybridSearchCached(engine, 'fixture', {
|
||||
sourceId: '__set__:a,b',
|
||||
useCache: true,
|
||||
limit: 5,
|
||||
});
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
// Nothing may have been written into the cache under a forged scope.
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM query_cache`,
|
||||
);
|
||||
expect(rows[0]?.n ?? 0).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -19,7 +19,11 @@
|
||||
|
||||
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { hybridSearch } from '../../src/core/search/hybrid.ts';
|
||||
import {
|
||||
awaitPendingSearchCacheWrites,
|
||||
hybridSearch,
|
||||
hybridSearchCached,
|
||||
} from '../../src/core/search/hybrid.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
@@ -69,16 +73,31 @@ beforeAll(async () => {
|
||||
// early-returns BEFORE applyReranker, so a setup that lacks embedding
|
||||
// would never exercise the reranker integration.
|
||||
//
|
||||
// searchVector returns empty lists because chunks have NULL embeddings;
|
||||
// that's fine — vectorLists is `[[]]` (length 1, not 0), so the
|
||||
// keyword-only branch is skipped and the main path runs RRF + dedup +
|
||||
// reranker + budget.
|
||||
// Most chunks have NULL embeddings; the email fixture below carries one so
|
||||
// the main path runs vector fusion + RRF + dedup + reranker + budget.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: DIMS,
|
||||
env: { OPENAI_API_KEY: 'sk-test' },
|
||||
});
|
||||
stubEmbeddings();
|
||||
|
||||
await engine.putPage('mail/vector-first', {
|
||||
type: 'note',
|
||||
title: 'Vector-first email',
|
||||
compiled_truth: 'vector first duplicate metadata evidence',
|
||||
frontmatter: {
|
||||
message_id: '<vector-first@example.com>',
|
||||
thread_id: 'thread-vector-first',
|
||||
subject: 'Vector-first exact subject',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('mail/vector-first', [{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'vector first duplicate metadata evidence',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: Float32Array.from(FAKE_EMB),
|
||||
}]);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -105,6 +124,70 @@ describe('hybridSearch — reranker disabled (pass-through)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached — email metadata through vector-first fusion', () => {
|
||||
test('fresh cache miss preserves metadata through vector-first RRF duplicate handling', async () => {
|
||||
await engine.executeRaw(`DELETE FROM query_cache`);
|
||||
const cacheStatuses: string[] = [];
|
||||
const out = await hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
|
||||
limit: 10,
|
||||
useCache: true,
|
||||
autocut: false,
|
||||
graph_signals: false,
|
||||
onMeta: (meta) => {
|
||||
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
|
||||
},
|
||||
});
|
||||
|
||||
expect(cacheStatuses.at(-1)).toBe('miss');
|
||||
const matches = out.filter(r => r.slug === 'mail/vector-first');
|
||||
expect(matches).toHaveLength(1);
|
||||
expect(matches[0].message_id).toBe('<vector-first@example.com>');
|
||||
expect(matches[0].thread_id).toBe('thread-vector-first');
|
||||
expect(matches[0].source_subject).toBe('Vector-first exact subject');
|
||||
await awaitPendingSearchCacheWrites();
|
||||
|
||||
const cached = await hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
|
||||
limit: 10,
|
||||
useCache: true,
|
||||
autocut: false,
|
||||
graph_signals: false,
|
||||
onMeta: (meta) => {
|
||||
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
|
||||
},
|
||||
});
|
||||
expect(cacheStatuses.at(-1)).toBe('hit');
|
||||
const cachedMatch = cached.find(r => r.slug === 'mail/vector-first');
|
||||
expect(cachedMatch?.message_id).toBe('<vector-first@example.com>');
|
||||
expect(cachedMatch?.thread_id).toBe('thread-vector-first');
|
||||
expect(cachedMatch?.source_subject).toBe('Vector-first exact subject');
|
||||
});
|
||||
|
||||
test('the query-op request shape (expandFn wired) still uses the semantic cache', async () => {
|
||||
// Regression: operations.ts always passes expandFn on the default `query`
|
||||
// op. If expandFn were treated as cache-unsafe, cacheStatus would be
|
||||
// 'disabled' here and the flagship op would never hit the cache.
|
||||
await engine.executeRaw(`DELETE FROM query_cache`);
|
||||
const cacheStatuses: string[] = [];
|
||||
const run = () => hybridSearchCached(engine, 'vector first duplicate metadata evidence', {
|
||||
limit: 10,
|
||||
useCache: true,
|
||||
autocut: false,
|
||||
graph_signals: false,
|
||||
expansion: false,
|
||||
expandFn: async (q: string) => [q],
|
||||
onMeta: (meta) => {
|
||||
if (meta.cache?.status) cacheStatuses.push(meta.cache.status);
|
||||
},
|
||||
});
|
||||
|
||||
await run();
|
||||
expect(cacheStatuses.at(-1)).toBe('miss');
|
||||
await awaitPendingSearchCacheWrites();
|
||||
await run();
|
||||
expect(cacheStatuses.at(-1)).toBe('hit');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch — reranker enabled (reorder)', () => {
|
||||
test('rerankerFn receives a non-empty document list', async () => {
|
||||
let receivedDocs: string[] = [];
|
||||
|
||||
@@ -44,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => {
|
||||
test('version is 13 (…; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825; 12→13 email citation DTO)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
|
||||
@@ -64,7 +64,9 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// pre-fix document-side query vectors must not be served.
|
||||
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
|
||||
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
// Email citation metadata projection: 12→13 so stale cached SearchResult
|
||||
// DTOs without message_id/thread_id/source_subject cannot survive.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
|
||||
@@ -27,8 +27,25 @@ beforeEach(async () => {
|
||||
const slugsOf = (m: Map<string, Array<{ slug: string; source_id: string }>>, k: string) =>
|
||||
(m.get(k) ?? []).map(r => r.slug).sort();
|
||||
|
||||
async function seedPage(slug: string, sourceId = 'default'): Promise<void> {
|
||||
if (sourceId !== 'default') {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, archived)
|
||||
VALUES ($1, $1, false)
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
[sourceId],
|
||||
);
|
||||
}
|
||||
await engine.putPage(
|
||||
slug,
|
||||
{ title: slug, type: 'note', compiled_truth: `# ${slug}`, timeline: '', frontmatter: {} },
|
||||
{ sourceId },
|
||||
);
|
||||
}
|
||||
|
||||
describe('setPageAliases + resolveAliases', () => {
|
||||
test('write then read maps alias_norm → slug', async () => {
|
||||
await seedPage('projects/mingtang');
|
||||
await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']);
|
||||
const m = await engine.resolveAliases(['hall of light'], { sourceId: 'default' });
|
||||
expect(slugsOf(m, 'hall of light')).toEqual(['projects/mingtang']);
|
||||
@@ -36,6 +53,8 @@ describe('setPageAliases + resolveAliases', () => {
|
||||
});
|
||||
|
||||
test('collision: two pages claim the same alias → both returned', async () => {
|
||||
await seedPage('projects/mingtang');
|
||||
await seedPage('projects/other-hall');
|
||||
await engine.setPageAliases('projects/mingtang', 'default', ['the hall']);
|
||||
await engine.setPageAliases('projects/other-hall', 'default', ['the hall']);
|
||||
const m = await engine.resolveAliases(['the hall'], { sourceId: 'default' });
|
||||
@@ -43,6 +62,8 @@ describe('setPageAliases + resolveAliases', () => {
|
||||
});
|
||||
|
||||
test('source-scoped: alias in source A not returned for source B', async () => {
|
||||
await seedPage('a/page', 'src-a');
|
||||
await seedPage('b/page', 'src-b');
|
||||
await engine.setPageAliases('a/page', 'src-a', ['shared name']);
|
||||
await engine.setPageAliases('b/page', 'src-b', ['shared name']);
|
||||
const aOnly = await engine.resolveAliases(['shared name'], { sourceId: 'src-a' });
|
||||
@@ -55,6 +76,7 @@ describe('setPageAliases + resolveAliases', () => {
|
||||
});
|
||||
|
||||
test('rewrite replaces the prior alias set (delete + insert)', async () => {
|
||||
await seedPage('p/x');
|
||||
await engine.setPageAliases('p/x', 'default', ['old name']);
|
||||
await engine.setPageAliases('p/x', 'default', ['new name']);
|
||||
const oldM = await engine.resolveAliases(['old name'], { sourceId: 'default' });
|
||||
@@ -64,6 +86,7 @@ describe('setPageAliases + resolveAliases', () => {
|
||||
});
|
||||
|
||||
test('empty alias set clears the page', async () => {
|
||||
await seedPage('p/x');
|
||||
await engine.setPageAliases('p/x', 'default', ['temp']);
|
||||
await engine.setPageAliases('p/x', 'default', []);
|
||||
const m = await engine.resolveAliases(['temp'], { sourceId: 'default' });
|
||||
@@ -76,6 +99,7 @@ describe('setPageAliases + resolveAliases', () => {
|
||||
});
|
||||
|
||||
test('idempotent re-write does not duplicate (unique triple)', async () => {
|
||||
await seedPage('p/x');
|
||||
await engine.setPageAliases('p/x', 'default', ['name', 'name']);
|
||||
const m = await engine.resolveAliases(['name'], { sourceId: 'default' });
|
||||
expect(slugsOf(m, 'name')).toEqual(['p/x']);
|
||||
|
||||
@@ -305,7 +305,7 @@ describe('buildVisibilityClause (v0.26.5)', () => {
|
||||
|
||||
test('uses the supplied aliases verbatim', () => {
|
||||
expect(buildVisibilityClause('pp', 'src')).toBe(
|
||||
"AND pp.deleted_at IS NULL AND NOT src.archived AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine')",
|
||||
"AND pp.deleted_at IS NULL AND NOT src.archived AND (jsonb_typeof(COALESCE(pp.frontmatter, '{}'::jsonb)) = 'object' AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine'))",
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -45,6 +45,11 @@ describe('Layer 7 (A2) — expandAnchors', () => {
|
||||
title: 'src/b.ts (typescript)',
|
||||
compiled_truth: 'export function b() { return c(); }',
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
message_id: '<code-review@example.com>',
|
||||
thread_id: 'thread-code-review',
|
||||
subject: 'Code review notes',
|
||||
},
|
||||
});
|
||||
await engine.upsertChunks('src-b-ts', [{
|
||||
chunk_index: 0,
|
||||
@@ -85,6 +90,9 @@ describe('Layer 7 (A2) — expandAnchors', () => {
|
||||
{ from_chunk_id: chunkB, to_chunk_id: null,
|
||||
from_symbol_qualified: 'b', to_symbol_qualified: 'c',
|
||||
edge_type: 'calls' },
|
||||
{ from_chunk_id: chunkA, to_chunk_id: chunkB,
|
||||
from_symbol_qualified: 'a', to_symbol_qualified: 'b',
|
||||
edge_type: 'imports' },
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -121,6 +129,16 @@ describe('Layer 7 (A2) — expandAnchors', () => {
|
||||
expect(neighbor!.score).toBeCloseTo(0.5, 2);
|
||||
});
|
||||
|
||||
test('walkDepth=1 follows an incoming resolved edge to the opposite endpoint', async () => {
|
||||
const anchors = [{
|
||||
slug: 'src-b-ts', page_id: 0, title: 'b', type: 'code',
|
||||
chunk_text: '', chunk_source: 'compiled_truth', chunk_id: chunkB,
|
||||
chunk_index: 0, score: 1.0, stale: false, source_id: 'default',
|
||||
} as never];
|
||||
const expanded = await expandAnchors(engine, anchors, { walkDepth: 1 });
|
||||
expect(expanded.map(e => e.chunk_id)).toContain(chunkA);
|
||||
});
|
||||
|
||||
test('walkDepth=2 reaches grandchildren', async () => {
|
||||
const anchors = [{
|
||||
slug: 'src-a-ts', page_id: 0, title: 'a', type: 'code',
|
||||
@@ -161,6 +179,10 @@ describe('Layer 7 (A2) — expandAnchors', () => {
|
||||
const slugs = rows.map(r => r.slug);
|
||||
expect(slugs).toContain('src-b-ts');
|
||||
expect(slugs).toContain('src-c-ts');
|
||||
const b = rows.find(r => r.slug === 'src-b-ts');
|
||||
expect(b?.message_id).toBe('<code-review@example.com>');
|
||||
expect(b?.thread_id).toBe('thread-code-review');
|
||||
expect(b?.source_subject).toBe('Code review notes');
|
||||
});
|
||||
|
||||
test('hydrateChunks with empty array returns []', async () => {
|
||||
|
||||
+49
-1
@@ -48,13 +48,23 @@ describe('contentHash', () => {
|
||||
});
|
||||
|
||||
describe('rowToPage', () => {
|
||||
test('parses string frontmatter', () => {
|
||||
test('rejects encoded-string frontmatter instead of double-decoding it', () => {
|
||||
const page = rowToPage({
|
||||
id: 1, slug: 'test', type: 'concept', title: 'Test',
|
||||
compiled_truth: 'body', timeline: '',
|
||||
frontmatter: '{"key":"val"}',
|
||||
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
|
||||
});
|
||||
expect(page.frontmatter).toEqual({});
|
||||
});
|
||||
|
||||
test('preserves object-shaped JSONB frontmatter', () => {
|
||||
const page = rowToPage({
|
||||
id: 1, slug: 'test', type: 'concept', title: 'Test',
|
||||
compiled_truth: 'body', timeline: '',
|
||||
frontmatter: { key: 'val' },
|
||||
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
|
||||
});
|
||||
expect(page.frontmatter.key).toBe('val');
|
||||
});
|
||||
|
||||
@@ -183,4 +193,42 @@ describe('rowToSearchResult', () => {
|
||||
expect(typeof r.score).toBe('number');
|
||||
expect(r.score).toBe(0.95);
|
||||
});
|
||||
|
||||
test('projects allowlisted email identifiers when present', () => {
|
||||
const r = rowToSearchResult({
|
||||
slug: 'mail/example', page_id: 2, title: 'Example email', type: 'note',
|
||||
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 3, chunk_index: 0,
|
||||
score: 0.9, stale: false,
|
||||
message_id: '<message@example.com>', thread_id: 'abc123',
|
||||
source_subject: 'Example launch subject',
|
||||
});
|
||||
expect(r.message_id).toBe('<message@example.com>');
|
||||
expect(r.thread_id).toBe('abc123');
|
||||
expect(r.source_subject).toBe('Example launch subject');
|
||||
});
|
||||
|
||||
test('does not invent email identifiers when projections are absent', () => {
|
||||
const r = rowToSearchResult({
|
||||
slug: 'concept/example', page_id: 3, title: 'Example', type: 'concept',
|
||||
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 4, chunk_index: 0,
|
||||
score: 0.8, stale: false,
|
||||
source_subject: 'Generated title must not become an email subject',
|
||||
});
|
||||
expect(r.message_id).toBeUndefined();
|
||||
expect(r.thread_id).toBeUndefined();
|
||||
expect(r.source_subject).toBeUndefined();
|
||||
});
|
||||
|
||||
test('whitespace-only message_id does not project or authorize source_subject', () => {
|
||||
const r = rowToSearchResult({
|
||||
slug: 'mail/whitespace-id', page_id: 4, title: 'Whitespace ID', type: 'note',
|
||||
chunk_text: 'text', chunk_source: 'compiled_truth', chunk_id: 5, chunk_index: 0,
|
||||
score: 0.7, stale: false,
|
||||
message_id: ' \t\n ', thread_id: 'thread-whitespace',
|
||||
source_subject: 'Must remain gated',
|
||||
});
|
||||
expect(r.message_id).toBeUndefined();
|
||||
expect(r.thread_id).toBe('thread-whitespace');
|
||||
expect(r.source_subject).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user