fix search: title candidate arm + gated OR fallback for lexical recall (#2956)

Pages were unreachable by their own exact titles: FTS indexed only chunk
body text while the title-weighted pages.search_vector (GIN-indexed since
its introduction) was never queried by any search path, and
websearch_to_tsquery AND-at-chunk-grain semantics meant one non-matching
token zeroed keyword recall with no fallback — long or acronym-bearing
titles (e.g. "IAWG ... AAR-LL deck") fell through to the vector arm alone
and missed.

- searchTitles (both engines): page-grain candidate arm over
  pages.search_vector (title 'A' + compiled_truth 'B' + timeline 'C'),
  ts_rank_cd ranked, representative-chunk LATERAL join, full filter
  parity with searchKeyword (visibility, soft-delete, source grants,
  hard-excludes, dates, types); fused as a weighted RRF list at the
  keyword arm's intent-effective k on all three hybrid return paths;
  fail-open with warnOncePerProcess. No schema changes — the index
  already existed, dark.
- AND->OR one-retry fallback for the keyword arm, gated behind
  SearchOpts.orFallback (only hybridSearch opts in; countMentions, link
  resolution, eval, and keyword-only MCP callers keep the strict-AND
  contract). Refused for queries carrying websearch operators (negation,
  quoted phrases). searchTitles carries its own page-grain fallback.
- Lexical arms parallelized (Promise.all) on the main path.

Verified: typecheck clean; 18 hermetic PGLite tests + 2 engine-parity e2e
cases (CI Postgres); consumer regression enrichment 18/0 +
link-extraction 127/0; independent live QA on a 10,664-page brain —
exact-title target miss -> rank 1 (exact_title_match), controls held,
negation/quoted guards proven, strict-consumer contract pinned.

Diagnosed from a 3-lane read-only diagnostic; adversarial review round
closed findings on fallback scope, Postgres test coverage, and operator
handling before this commit.

Co-authored-by: Aleksei Razsadin <razsoc.01@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Cossackx
2026-07-20 11:57:31 -07:00
committed by GitHub
co-authored by Aleksei Razsadin Claude Fable 5
parent 912407bef1
commit 184b6cb8a1
8 changed files with 784 additions and 24 deletions
+21
View File
@@ -936,6 +936,27 @@ export interface BrainEngine {
// Search
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
/**
* fix/title-retrieval-arm (D1): page-grain title candidate arm.
*
* content_chunks.search_vector never includes the page TITLE (it is
* doc_comment + symbol_name_qualified + chunk_text), so a page whose
* title tokens are absent from its body is unreachable by searchKeyword.
* This arm queries the PAGE-GRAIN DOCUMENT vector pages.search_vector —
* NOT titles alone: per trg_pages_search_vector it is title (weight 'A')
* + compiled_truth ('B') + timeline text ('C'). Ranked by ts_rank_cd,
* the 'A'-weighted title dominates, but body/timeline matches also
* produce (lower-ranked) candidates. Returns page-grain hits joined to
* ONE representative chunk per page (compiled_truth preferred, else
* lowest chunk_index) so rows are shaped like searchKeyword's output and
* can enter RRF fusion in hybridSearch.
*
* Deliberately NO query-length gating — unlike the alias hop (≤6-token
* guard) and the title-phrase re-rank boost, this arm must GENERATE
* candidates for long exact-title queries, which is exactly where
* chunk-grain AND FTS is weakest.
*/
searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
/**
* Hydrate embeddings for chunks already known by id. v0.36 (D9):
+135 -5
View File
@@ -55,7 +55,7 @@ import { GBrainError, PAGE_SORT_SQL, ENRICH_ORDER_SQL } from './types.ts';
import { finalizeLastSeen } from './chronicle/last-seen.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import {
normalizeEngineColumn,
buildVectorCastFragment,
@@ -1630,7 +1630,7 @@ export class PGLiteEngine implements BrainEngine {
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const { rows } = await this.db.query(
const keywordSql =
`WITH ranked AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
@@ -1654,10 +1654,140 @@ export class PGLiteEngine implements BrainEngine {
${buildBestPerPagePoolCte('ranked')}
SELECT * FROM best_per_page
ORDER BY score DESC, page_id ASC, chunk_id ASC
LIMIT $3 OFFSET $4`,
params
);
LIMIT $3 OFFSET $4`;
let { rows } = await this.db.query(keywordSql, params);
// D2 fix (fix/title-retrieval-arm): websearch AND semantics at chunk
// grain mean one non-co-occurring token zeroes keyword recall. When the
// strict query returns nothing, retry ONCE with OR-of-terms. Strict-AND
// results always win when non-empty (no change for working queries).
// Opt-in via SearchOpts.orFallback (Reviewer F1): only hybridSearch's
// recall arm relaxes; precision consumers (countMentions,
// link-extraction, eval) keep the strict-AND contract.
if (rows.length === 0 && opts?.orFallback) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) {
const fallbackParams = [...params];
fallbackParams[0] = orQuery;
({ rows } = await this.db.query(keywordSql, fallbackParams));
}
}
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
}
/**
* fix/title-retrieval-arm (D1): page-grain title candidate arm. See the
* BrainEngine interface doc for the full contract. Queries
* pages.search_vector (title weight 'A' dominates ts_rank_cd by
* construction) with the same page-grain filters the keyword arm applies
* (type/types/excludeSlugs/date/source scoping, hard-excludes,
* visibility), joined to one representative chunk per page. Applies the
* same ANDOR recall fallback as searchKeyword. NO query-length gate
* long exact-title queries are the case this arm exists for.
*
* CJK queries fall through to websearch FTS here (a single-token CJK
* query CAN exact-match a single-token CJK title); the richer CJK ILIKE
* fallback stays keyword-arm-only.
*/
async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
// language/symbolKind are chunk-grain code filters with no page-grain
// meaning; a code-scoped query gets no title candidates rather than
// rows that silently violate the caller's filter.
if (opts?.language || opts?.symbolKind) return [];
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const detailLow = opts?.detail === 'low';
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
const boostMap = resolveBoostMap();
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const visibilityClause = buildVisibilityClause('p', 's');
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const params: unknown[] = [query, limit, offset];
let extraFilter = '';
if (opts?.type) {
params.push(opts.type);
extraFilter += ` AND p.type = $${params.length}`;
}
if (opts?.types && opts.types.length > 0) {
params.push(opts.types);
extraFilter += ` AND p.type = ANY($${params.length}::text[])`;
}
if (opts?.exclude_slugs?.length) {
params.push(opts.exclude_slugs);
extraFilter += ` AND p.slug != ALL($${params.length}::text[])`;
}
if (opts?.afterDate) {
params.push(opts.afterDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
if (opts?.beforeDate) {
params.push(opts.beforeDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
if (opts?.sourceIds && opts.sourceIds.length > 0) {
params.push(opts.sourceIds);
extraFilter += ` AND p.source_id = ANY($${params.length}::text[])`;
} else if (opts?.sourceId) {
params.push(opts.sourceId);
extraFilter += ` AND p.source_id = $${params.length}`;
}
// Page grain — one row per page by construction, so no best_per_page
// pooling CTE is needed. The LEFT JOIN LATERAL picks the representative
// chunk (compiled_truth first, then lowest chunk_index); COALESCEs keep
// chunkless pages retrievable (the extreme D1 case: a title with no
// body) with the alias-hop row shape (chunk_id 0, empty chunk_text).
// Accepted limitations (Reviewer F5/F6): the synthetic chunkless row
// inherits the compiled-truth RRF boost and dedups on empty chunk_text;
// and detail='low' filters only the REPRESENTATIVE — pages without a
// compiled_truth chunk still surface (unlike the keyword arm's filter).
const titlesSql =
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
COALESCE(rep.id, 0) as chunk_id,
COALESCE(rep.chunk_index, 0) as chunk_index,
COALESCE(rep.chunk_text, '') as chunk_text,
COALESCE(rep.chunk_source, 'compiled_truth') as chunk_source,
ts_rank_cd(p.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
) THEN true ELSE false END AS stale
FROM pages p
JOIN sources s ON s.id = p.source_id
LEFT JOIN LATERAL (
SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_source
FROM content_chunks cc
WHERE cc.page_id = p.id
AND cc.modality = 'text'
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
ORDER BY (cc.chunk_source = 'compiled_truth') DESC, cc.chunk_index ASC
LIMIT 1
) rep ON true
WHERE p.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
${extraFilter} ${hardExcludeClause} ${visibilityClause}
ORDER BY score DESC, p.id ASC
LIMIT $2 OFFSET $3`;
let { rows } = await this.db.query(titlesSql, params);
if (rows.length === 0) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) {
const fallbackParams = [...params];
fallbackParams[0] = orQuery;
({ rows } = await this.db.query(titlesSql, fallbackParams));
}
}
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
}
+159 -5
View File
@@ -63,7 +63,7 @@ import { ConnectionManager } from './connection-manager.ts';
import { logConnectionEvent } from './connection-audit.ts';
import { validateSlug, contentHash, rowToPage, rowToStalePage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, takeHitRowToHit, isUndefinedTableError, warnOncePerProcess } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte, buildOrFallbackWebsearchQuery } from './search/sql-ranking.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
@@ -1807,10 +1807,164 @@ export class PostgresEngine implements BrainEngine {
// the GUC can never leak onto a pooled connection). Flag off → the
// wrap is identical to master's; flag on → set_config('app.scopes')
// shares the same transaction as the timeout.
const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
await tx`SET LOCAL statement_timeout = '8s'`;
return await tx.unsafe(rawQuery, params as Parameters<typeof tx.unsafe>[1]);
}, { alwaysTransaction: true });
const runKeyword = (queryText: string) =>
this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
await tx`SET LOCAL statement_timeout = '8s'`;
const boundParams = [...params];
boundParams[0] = queryText;
return await tx.unsafe(rawQuery, boundParams as Parameters<typeof tx.unsafe>[1]);
}, { alwaysTransaction: true });
let rows = await runKeyword(query);
// D2 fix (fix/title-retrieval-arm): websearch AND semantics at chunk
// grain mean one non-co-occurring token zeroes keyword recall. When the
// strict query returns nothing, retry ONCE with OR-of-terms — through
// the SAME scoped wrapper (the retry is a fresh scoped transaction, so
// RLS scope binding applies identically). Strict-AND results always win
// when non-empty (no change for working queries).
// Opt-in via SearchOpts.orFallback (Reviewer F1): only hybridSearch's
// recall arm relaxes; precision consumers (countMentions,
// link-extraction, eval) keep the strict-AND contract.
if (rows.length === 0 && opts?.orFallback) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) rows = await runKeyword(orQuery);
}
return rows.map(rowToSearchResult);
}
/**
* fix/title-retrieval-arm (D1): page-grain title candidate arm. See the
* BrainEngine interface doc for the full contract. Queries
* pages.search_vector (title weight 'A' dominates ts_rank_cd by
* construction) with the same page-grain filters the keyword arm applies
* (type/types/excludeSlugs/date/source scoping, hard-excludes,
* visibility), joined to one representative chunk per page. Applies the
* same ANDOR recall fallback as searchKeyword. NO query-length gate
* long exact-title queries are the case this arm exists for.
*/
async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
// language/symbolKind are chunk-grain code filters with no page-grain
// meaning; a code-scoped query gets no title candidates rather than
// rows that silently violate the caller's filter.
if (opts?.language || opts?.symbolKind) return [];
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const detailLow = opts?.detail === 'low';
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
const boostMap = resolveBoostMap();
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const visibilityClause = buildVisibilityClause('p', 's');
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
// — safe to interpolate into raw SQL.
const ftsLang = getFtsLanguage();
const params: unknown[] = [query];
let typeClause = '';
if (opts?.type) {
params.push(opts.type);
typeClause = `AND p.type = $${params.length}`;
}
let typesClause = '';
if (opts?.types && opts.types.length > 0) {
params.push(opts.types);
typesClause = `AND p.type = ANY($${params.length}::text[])`;
}
let excludeSlugsClause = '';
if (opts?.exclude_slugs?.length) {
params.push(opts.exclude_slugs);
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
}
// Date filters read COALESCE(effective_date, …) — upstream unified the
// Postgres keyword arm onto the PGLite effective-date-first convention
// (v0.29.1 parity); the title arm matches it for filter parity.
let afterDateClause = '';
if (opts?.afterDate) {
params.push(opts.afterDate);
afterDateClause = `AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
let beforeDateClause = '';
if (opts?.beforeDate) {
params.push(opts.beforeDate);
beforeDateClause = `AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
let sourceClause = '';
if (opts?.sourceIds && opts.sourceIds.length > 0) {
params.push(opts.sourceIds);
sourceClause = `AND p.source_id = ANY($${params.length}::text[])`;
} else if (opts?.sourceId) {
params.push(opts.sourceId);
sourceClause = `AND p.source_id = $${params.length}`;
}
params.push(limit);
const limitParam = `$${params.length}`;
params.push(offset);
const offsetParam = `$${params.length}`;
// Page grain — one row per page by construction, so no best_per_page
// pooling CTE is needed. The LEFT JOIN LATERAL picks the representative
// chunk (compiled_truth first, then lowest chunk_index); COALESCEs keep
// chunkless pages retrievable (the extreme D1 case: a title with no
// body) with the alias-hop row shape (chunk_id 0, empty chunk_text).
// Accepted limitations (Reviewer F5/F6): the synthetic chunkless row
// inherits the compiled-truth RRF boost and dedups on empty chunk_text;
// and detail='low' filters only the REPRESENTATIVE — pages without a
// compiled_truth chunk still surface (unlike the keyword arm's filter).
const rawQuery = `
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
COALESCE(rep.id, 0) as chunk_id,
COALESCE(rep.chunk_index, 0) as chunk_index,
COALESCE(rep.chunk_text, '') as chunk_text,
COALESCE(rep.chunk_source, 'compiled_truth') as chunk_source,
ts_rank_cd(p.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
false AS stale
FROM pages p
JOIN sources s ON s.id = p.source_id
LEFT JOIN LATERAL (
SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_source
FROM content_chunks cc
WHERE cc.page_id = p.id
AND cc.modality = 'text'
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
ORDER BY (cc.chunk_source = 'compiled_truth') DESC, cc.chunk_index ASC
LIMIT 1
) rep ON true
WHERE p.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
${typeClause}
${typesClause}
${excludeSlugsClause}
${afterDateClause}
${beforeDateClause}
${sourceClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY score DESC, p.id ASC
LIMIT ${limitParam}
OFFSET ${offsetParam}
`;
// Same RLS scope-binding wrapper as searchKeyword (alwaysTransaction:
// the SET LOCAL statement_timeout needs a transaction regardless of the
// GBRAIN_RLS_SCOPE_BINDING flag). The OR retry re-executes through the
// same scoped wrapper.
const runTitles = (queryText: string) =>
this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
await tx`SET LOCAL statement_timeout = '8s'`;
const boundParams = [...params];
boundParams[0] = queryText;
return await tx.unsafe(rawQuery, boundParams as Parameters<typeof tx.unsafe>[1]);
}, { alwaysTransaction: true });
let rows = await runTitles(query);
if (rows.length === 0) {
const orQuery = buildOrFallbackWebsearchQuery(query);
if (orQuery) rows = await runTitles(orQuery);
}
return rows.map(rowToSearchResult);
}
+55 -14
View File
@@ -34,6 +34,7 @@ import { normalizeAlias } from './alias-normalize.ts';
import { stampEvidence } from './evidence.ts';
import { expandAnchors, hydrateChunks } from './two-pass.ts';
import { enforceTokenBudget } from './token-budget.ts';
import { warnOncePerProcess } from '../utils.ts';
import { recordSearchTelemetry } from './telemetry.ts';
import {
weightsForIntent,
@@ -932,6 +933,11 @@ export async function hybridSearch(
// it never has to read config. Engines normalize string-or-descriptor
// via normalizeEngineColumn; the descriptor path is the strict one.
embeddingColumn: resolvedCol,
// D2 fix (fix/title-retrieval-arm, Reviewer F1): the hybrid keyword arm
// is a recall arm — opt in to the engine's AND→OR zero-recall fallback.
// Direct searchKeyword consumers (countMentions, link-extraction, eval)
// do NOT set this and keep the strict-AND contract.
orFallback: true,
};
// Track what actually ran for the optional onMeta callback (v0.25.0).
// Caller leaves onMeta undefined → these flags are computed but never
@@ -990,8 +996,31 @@ export async function hybridSearch(
const earlyModality = (opts?.crossModal && opts.crossModal !== 'auto')
? opts.crossModal
: (suggestions.suggestedModality ?? 'text');
const keywordResults: SearchResult[] =
earlyModality === 'image' ? [] : await engine.searchKeyword(query, searchOpts);
// D1 fix (fix/title-retrieval-arm): page-grain title candidate arm,
// fetched CONCURRENTLY with the keyword arm (Reviewer F7 — independent
// engine queries). The chunk FTS vector never includes the page title, so
// an exact-title query can be unretrievable by keyword — this arm queries
// pages.search_vector (title weight 'A') directly. Runs regardless of
// query token count: the alias hop (≤6-token guard) and the title-phrase
// boost are re-rank-only, so LONG exact-title queries — where strict-AND
// chunk FTS is weakest — need a candidate GENERATOR. Fail-open WITH
// SIGNAL (Reviewer F2): a SQL error (e.g. a pre-search_vector brain)
// degrades to no title candidates, but warns once per process so a
// broken engine arm cannot ship dark.
const [keywordResults, titleResults]: [SearchResult[], SearchResult[]] =
earlyModality === 'image'
? [[], []]
: await Promise.all([
engine.searchKeyword(query, searchOpts),
engine.searchTitles(query, searchOpts).catch((err: unknown) => {
warnOncePerProcess(
'search-titles-arm-failed',
`[gbrain] searchTitles arm failed (fail-open, title candidates skipped): ` +
`${err instanceof Error ? err.message : String(err)}`,
);
return [] as SearchResult[];
}),
]);
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
@@ -1069,14 +1098,16 @@ export async function hybridSearch(
if (!isAvailable('embedding', providerProbe)) {
// v0.43 — fuse the relational arm with keyword so typed-edge answers
// survive on the no-embedding-provider path (the relational win is most
// valuable exactly when vector is unavailable).
// valuable exactly when vector is unavailable). The title arm fuses here
// too — an exact-title lookup on a keyless install is precisely where
// chunk-grain keyword FTS alone fails (D1).
let noEmbedResults = keywordResults;
if (relationalList.length > 0) {
if (relationalList.length > 0 || titleResults.length > 0) {
const fk = opts?.rrfK ?? RRF_K;
noEmbedResults = rrfFusionWeighted(
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
detailResolved !== 'high',
);
const noEmbedLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
}
if (noEmbedResults.length > 0) {
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
@@ -1303,14 +1334,15 @@ export async function hybridSearch(
// post-fusion stages here too — without it, salience='on' silently
// does nothing on embed failures.
// v0.43: fuse the relational arm with keyword via RRF so typed-edge
// answers survive even when vector is unavailable.
// answers survive even when vector is unavailable. The title arm fuses
// here too (same rationale as the no-embedding-provider path — D1).
let fallbackResults = keywordResults;
if (relationalList.length > 0) {
if (relationalList.length > 0 || titleResults.length > 0) {
const fk = opts?.rrfK ?? RRF_K;
fallbackResults = rrfFusionWeighted(
[{ list: keywordResults, k: fk }, { list: relationalList, k: fk }],
detail !== 'high',
);
const fallbackLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
}
if (fallbackResults.length > 0) {
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
@@ -1375,6 +1407,15 @@ export async function hybridSearch(
{ list: keywordResults, k: keywordK },
];
// D1 fix (fix/title-retrieval-arm) — title candidate arm as a third
// weighted list. Fuses at the keyword arm's intent-effective k (same
// lexical-evidence class, no new tunable). Mirrors the keyword list's
// inclusion rules: fetch was gated on earlyModality, so no extra modality
// check here. Empty for non-matching queries → pure no-op.
if (titleResults.length > 0) {
allLists.push({ list: titleResults, k: keywordK });
}
// v0.43 — relational recall arm (fourth RRF arm), built above so it also
// contributes on the keyword-only fallback path. Neutral weight (baseRrfK):
// competes evenly with keyword/vector, not dominating. Empty for
+45
View File
@@ -206,6 +206,51 @@ export function buildBestPerPagePoolCte(candidateCte: string): string {
)`;
}
// ============================================================
// AND→OR keyword-recall fallback (fix/title-retrieval-arm, D2)
// ============================================================
/**
* Build a relaxed OR-of-terms websearch string for the keyword-arm recall
* fallback.
*
* `websearch_to_tsquery('english', query)` joins unquoted terms with `&`
* (AND). At chunk grain, one query token that doesn't co-occur in any
* single chunk zeroes keyword recall with no fallback. When the strict
* AND query returns zero rows, engines retry ONCE with the string this
* builder returns — the same tokens joined with websearch's `OR` keyword,
* which compiles to `|`.
*
* Why rebuild via websearch syntax instead of hand-assembling a tsquery:
* websearch_to_tsquery never raises on malformed input, applies the same
* stemming/stopword pipeline as the document side, and an all-stopword
* token list degrades to an empty tsquery (matches nothing) instead of a
* SQL error — the empty-tsquery guard comes free.
*
* Returns null when relaxation is pointless or unsafe:
* - fewer than 2 tokens survive tokenization (OR of one term is the same
* query as AND of one term);
* - the raw query uses websearch OPERATORS (Reviewer F3): a `-term`
* negation would be RESURRECTED as a positive OR term, and a quoted
* phrase would degrade to a bag of words — both invert caller intent,
* so operator queries get no fallback at all.
* Tokenization splits on non-alphanumeric runs (Unicode-aware). Literal
* OR/AND words are dropped so they can't be re-parsed as operators
* mid-list.
*/
export function buildOrFallbackWebsearchQuery(query: string): string | null {
// F3 operator guard: any double quote, or a dash LEADING a token
// (whitespace/start boundary — interior hyphens like "foo-bar" are fine).
if (query.includes('"') || /(^|\s)-\S/.test(query)) return null;
const tokens = query
.normalize('NFKC')
.split(/[^\p{L}\p{N}]+/u)
.filter(Boolean)
.filter(t => { const u = t.toUpperCase(); return u !== 'OR' && u !== 'AND'; });
if (tokens.length < 2) return null;
return tokens.join(' OR ');
}
// ============================================================
// v0.29.1 — Recency component SQL builder
// ============================================================
+13
View File
@@ -974,6 +974,19 @@ export interface SearchOpts {
* client) → `sourceIds`; otherwise `ctx.sourceId` (scalar) → `sourceId`.
*/
sourceIds?: string[];
/**
* fix/title-retrieval-arm (D2, Reviewer F1): opt-in AND→OR keyword-recall
* fallback. When true, `searchKeyword` retries ONCE with OR-of-terms after
* the strict websearch AND query returns zero rows (strict results always
* win when non-empty). Default false/undefined = strict-AND only — the
* pre-fix contract. hybridSearch opts in for its keyword arm; precision
* consumers (enrichment countMentions, link-extraction resolution, eval
* paths) MUST NOT set this: OR-matches would inflate mention counts and
* relax link-candidate resolution ("John Smith" matching every John and
* every Smith). `searchTitles` has its own page-grain fallback and
* ignores this flag.
*/
orFallback?: boolean;
/**
* v0.27.1 / v0.36 (D11): target column for vector search. Two shapes:
*
+60
View File
@@ -225,6 +225,66 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
expect(pgChanged || pgliteChanged).toBe(true);
});
// fix/title-retrieval-arm (Reviewer F2): the title arm must behave
// identically on both engines — including the D1 case where the title
// tokens never appear in any chunk. Without this case the Postgres
// implementation would only ever execute behind hybridSearch's fail-open
// catch and a break could ship dark on the production brain. Runs in CI
// via scripts/run-e2e.sh (docker-provisioned Postgres); skips gracefully
// when DATABASE_URL is not configured.
test('searchTitles parity: exact-title hit with title tokens absent from body', async () => {
const seed = async (eng: BrainEngine) => {
await eng.putPage('wiki/title-arm-parity', {
type: 'note',
title: 'Vermilion Icebreaker Compendium',
compiled_truth: 'A document body that never mentions those words.',
timeline: '',
});
await eng.upsertChunks('wiki/title-arm-parity', [{
chunk_index: 0,
chunk_text: 'A document body that never mentions those words.',
chunk_source: 'compiled_truth',
embedding: basisEmbedding(33),
token_count: 9,
}] satisfies ChunkInput[]);
};
await seed(pgEngine);
await seed(pgliteEngine);
const q = 'Vermilion Icebreaker Compendium';
// Premise on both engines: chunk-grain keyword cannot see the page
// (also pins the F1 contract — no orFallback flag means strict AND).
expect((await pgEngine.searchKeyword(q, { limit: 5 })).map((r: SearchResult) => r.slug))
.not.toContain('wiki/title-arm-parity');
expect((await pgliteEngine.searchKeyword(q, { limit: 5 })).map((r: SearchResult) => r.slug))
.not.toContain('wiki/title-arm-parity');
const pg = await pgEngine.searchTitles(q, { limit: 5 });
const pglite = await pgliteEngine.searchTitles(q, { limit: 5 });
expect(pg.map((r: SearchResult) => r.slug)).toContain('wiki/title-arm-parity');
expect(pglite.map((r: SearchResult) => r.slug)).toContain('wiki/title-arm-parity');
// Row-shape parity: identical representative chunk on both engines.
const pgHit = pg.find((r: SearchResult) => r.slug === 'wiki/title-arm-parity')!;
const pgliteHit = pglite.find((r: SearchResult) => r.slug === 'wiki/title-arm-parity')!;
expect(pgHit.chunk_source).toBe('compiled_truth');
expect(pgliteHit.chunk_source).toBe(pgHit.chunk_source);
expect(pgliteHit.chunk_text).toBe(pgHit.chunk_text);
});
// fix/title-retrieval-arm (Reviewer F1): the AND→OR fallback is opt-in.
// Default searchKeyword stays strict on BOTH engines; orFallback: true
// rescues the one-bad-token query identically.
test('searchKeyword orFallback parity: default strict, opt-in rescues', async () => {
const q = 'fat code thin harness zzzabsenttoken';
for (const eng of [pgEngine, pgliteEngine]) {
const strict = await eng.searchKeyword(q, { limit: 5 });
expect(strict.length).toBe(0);
const relaxed = await eng.searchKeyword(q, { limit: 5, orFallback: true });
expect(relaxed.map((r: SearchResult) => r.slug)).toContain('concepts/fat-code-thin-harness');
}
});
// v0.39.3.0 T3 — provenance write+read parity (WARN-8 + CV5).
// Both engines must write the same 4 provenance columns (source_kind,
// source_uri, ingested_via, ingested_at) on putPage AND surface them
+296
View File
@@ -0,0 +1,296 @@
/**
* fix/title-retrieval-arm — D1 title candidate arm + D2 AND→OR keyword fallback.
*
* The disease (3-lane diagnostic, 2026-07): page titles never enter the
* keyword-searchable text. content_chunks.search_vector is doc_comment +
* symbol_name_qualified + chunk_text — no title — so an exact-title query
* whose tokens are absent from the body had ZERO keyword recall, and every
* existing title mechanism (title boost, exact-match boost, alias hop) is
* re-rank-only: none can GENERATE the missing candidate. Compounding it,
* websearch_to_tsquery AND semantics at chunk grain meant one
* non-co-occurring token zeroed the whole keyword arm with no fallback.
*
* Fixes under test:
* C1 — engine.searchTitles: page-grain candidates from pages.search_vector
* (title weight 'A'), joined to one representative chunk, fused into
* hybridSearch as a keyword-class RRF list. No query-length gate.
* C2 — searchKeyword retries ONCE with OR-of-terms when strict AND
* returns zero rows; strict results always win when non-empty.
*
* Hermetic PGLite. The gateway is pinned with an EMPTY env so embedding is
* deterministically unavailable — hybridSearch takes the keyword(+title)
* no-embed path with zero network, regardless of host API keys.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
import { hybridSearch } from '../../src/core/search/hybrid.ts';
import { buildOrFallbackWebsearchQuery } from '../../src/core/search/sql-ranking.ts';
import { configureGateway } from '../../src/core/ai/gateway.ts';
let engine: PGLiteEngine;
const DIM = 1536;
beforeAll(async () => {
// Pin 1536-d (matches the preload schema default) with an EMPTY env so
// isAvailable('embedding') is false → hybridSearch never embeds.
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: DIM,
env: {},
});
engine = new PGLiteEngine();
await engine.connect({}); // in-memory
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
// Restore the preload-equivalent gateway for sibling files in this shard.
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: DIM,
env: { ...process.env },
});
});
beforeEach(async () => {
await resetPgliteState(engine);
});
/** Page whose TITLE tokens never appear in its body/chunks (the D1 shape). */
async function seedTitleOnlyPage(): Promise<void> {
await engine.putPage('projects/chronomancer', {
type: 'note',
title: 'Chronomancer Codex Ledger',
compiled_truth: 'A reference document about scheduling practices and planning.',
});
await engine.upsertChunks('projects/chronomancer', [
{
chunk_index: 0,
chunk_text: 'A reference document about scheduling practices and planning.',
chunk_source: 'compiled_truth',
},
]);
}
describe('searchTitles — D1 title candidate arm', () => {
test('exact-title query retrieves a page whose title tokens are absent from its body', async () => {
await seedTitleOnlyPage();
// Premise check: the chunk-grain keyword arm CANNOT see this page for
// this query, even with the OR fallback (no title token is in any chunk).
const kw = await engine.searchKeyword('Chronomancer Codex Ledger', { limit: 10 });
expect(kw.map(r => r.slug)).not.toContain('projects/chronomancer');
// The title arm can.
const hits = await engine.searchTitles('Chronomancer Codex Ledger', { limit: 10 });
expect(hits.map(r => r.slug)).toContain('projects/chronomancer');
const hit = hits.find(r => r.slug === 'projects/chronomancer')!;
expect(hit.title).toBe('Chronomancer Codex Ledger');
expect(hit.score).toBeGreaterThan(0);
// Shaped like a keyword-arm row: representative chunk attached.
expect(hit.chunk_text).toContain('reference document');
expect(hit.chunk_source).toBe('compiled_truth');
});
test('long 10-content-token exact-title query still retrieves (no token-count gate)', async () => {
const longTitle = 'Emerald Falcon Doctrine Quarterly Synthesis Report Alpha Bravo Charlie Delta';
await engine.putPage('reports/emerald-falcon', {
type: 'note',
title: longTitle,
compiled_truth: 'An annual planning artifact.',
});
await engine.upsertChunks('reports/emerald-falcon', [
{ chunk_index: 0, chunk_text: 'An annual planning artifact.', chunk_source: 'compiled_truth' },
]);
const hits = await engine.searchTitles(longTitle, { limit: 10 });
expect(hits.map(r => r.slug)).toContain('reports/emerald-falcon');
});
test('representative chunk prefers compiled_truth, else lowest chunk_index', async () => {
await engine.putPage('notes/mixed-chunks', {
type: 'note',
title: 'Obsidian Waterfall Registry',
compiled_truth: 'body text here',
});
await engine.upsertChunks('notes/mixed-chunks', [
{ chunk_index: 0, chunk_text: 'timeline entry text', chunk_source: 'timeline' },
{ chunk_index: 1, chunk_text: 'compiled body text', chunk_source: 'compiled_truth' },
]);
const hits = await engine.searchTitles('Obsidian Waterfall Registry', { limit: 5 });
const hit = hits.find(r => r.slug === 'notes/mixed-chunks')!;
expect(hit.chunk_source).toBe('compiled_truth');
expect(hit.chunk_index).toBe(1);
await engine.putPage('notes/timeline-only', {
type: 'note',
title: 'Cobalt Meridian Atlas',
compiled_truth: 'unrelated body',
});
await engine.upsertChunks('notes/timeline-only', [
{ chunk_index: 5, chunk_text: 'later timeline', chunk_source: 'timeline' },
{ chunk_index: 2, chunk_text: 'earlier timeline', chunk_source: 'timeline' },
]);
const tlHits = await engine.searchTitles('Cobalt Meridian Atlas', { limit: 5 });
const tlHit = tlHits.find(r => r.slug === 'notes/timeline-only')!;
expect(tlHit.chunk_index).toBe(2); // lowest index when no compiled_truth chunk
});
test('respects soft-delete visibility and source scoping', async () => {
await seedTitleOnlyPage();
// Source scope that doesn't own the page → filtered out at SQL level.
const scoped = await engine.searchTitles('Chronomancer Codex Ledger', {
limit: 10,
sourceId: 'some-other-source',
});
expect(scoped.length).toBe(0);
// Soft-deleted pages disappear (visibility clause).
await engine.softDeletePage('projects/chronomancer');
const afterDelete = await engine.searchTitles('Chronomancer Codex Ledger', { limit: 10 });
expect(afterDelete.map(r => r.slug)).not.toContain('projects/chronomancer');
});
test('respects hard-exclude slug prefixes (test/ is excluded by default)', async () => {
await engine.putPage('test/hidden-fixture', {
type: 'note',
title: 'Zanzibar Protocol Manifest',
compiled_truth: 'fixture body',
});
const hits = await engine.searchTitles('Zanzibar Protocol Manifest', { limit: 10 });
expect(hits.map(r => r.slug)).not.toContain('test/hidden-fixture');
});
});
describe('searchKeyword — D2 AND→OR fallback', () => {
async function seedQuantumPage(): Promise<void> {
await engine.putPage('notes/quantum', {
type: 'note',
title: 'Quantum Notes',
compiled_truth: 'quantum lattice harmonics resonance experiments',
});
await engine.upsertChunks('notes/quantum', [
{
chunk_index: 0,
chunk_text: 'quantum lattice harmonics resonance experiments',
chunk_source: 'compiled_truth',
},
]);
}
test('one bad token no longer zeroes keyword recall (orFallback: true rescues)', async () => {
await seedQuantumPage();
// Strict AND fails ('zzzmissingtoken' is nowhere); OR fallback rescues.
const hits = await engine.searchKeyword('quantum lattice harmonics zzzmissingtoken', {
limit: 10,
orFallback: true,
});
expect(hits.map(r => r.slug)).toContain('notes/quantum');
});
test('WITHOUT the orFallback flag the one-bad-token query returns zero (F1: strict default)', async () => {
await seedQuantumPage();
// Precision consumers (countMentions, link-extraction, eval) call
// searchKeyword without the flag — their strict-AND contract must hold.
const hits = await engine.searchKeyword('quantum lattice harmonics zzzmissingtoken', { limit: 10 });
expect(hits.length).toBe(0);
});
test('strict-AND results stay preferred: no OR dilution when AND matches', async () => {
await seedQuantumPage();
await engine.putPage('notes/partial', {
type: 'note',
title: 'Partial Overlap',
compiled_truth: 'quantum computing conference recap',
});
await engine.upsertChunks('notes/partial', [
{ chunk_index: 0, chunk_text: 'quantum computing conference recap', chunk_source: 'compiled_truth' },
]);
// All four tokens co-occur only in notes/quantum → strict AND non-empty
// → the OR retry must NOT fire (even with the flag SET), so the
// partial-overlap page stays out.
const hits = await engine.searchKeyword('quantum lattice harmonics resonance', {
limit: 10,
orFallback: true,
});
expect(hits.map(r => r.slug)).toContain('notes/quantum');
expect(hits.map(r => r.slug)).not.toContain('notes/partial');
});
test('single unmatched token returns empty (OR of one term is pointless)', async () => {
await seedQuantumPage();
const hits = await engine.searchKeyword('zzznothinghere', { limit: 10, orFallback: true });
expect(hits.length).toBe(0);
});
});
describe('buildOrFallbackWebsearchQuery — pure', () => {
test('joins tokens with OR', () => {
expect(buildOrFallbackWebsearchQuery('alpha beta')).toBe('alpha OR beta');
});
test('returns null for <2 tokens', () => {
expect(buildOrFallbackWebsearchQuery('alpha')).toBeNull();
expect(buildOrFallbackWebsearchQuery('')).toBeNull();
expect(buildOrFallbackWebsearchQuery(' ')).toBeNull();
});
test('F3: refuses queries with websearch operators (negation must not resurrect)', () => {
// A `-bar` exclusion relaxed to `foo OR bar` would MATCH the excluded
// term; a quoted phrase would degrade to a bag of words. No fallback.
expect(buildOrFallbackWebsearchQuery('foo -bar')).toBeNull();
expect(buildOrFallbackWebsearchQuery('"alpha beta" gamma')).toBeNull();
expect(buildOrFallbackWebsearchQuery('"alpha beta" -gamma')).toBeNull();
});
test('interior hyphens are not operators — still relaxed', () => {
expect(buildOrFallbackWebsearchQuery('alpha-beta gamma')).toBe('alpha OR beta OR gamma');
});
test('drops literal OR/AND words so they cannot re-parse as operators', () => {
expect(buildOrFallbackWebsearchQuery('alpha or beta')).toBe('alpha OR beta');
expect(buildOrFallbackWebsearchQuery('alpha AND beta')).toBe('alpha OR beta');
// Only operator words survive tokenization → nothing left to relax.
expect(buildOrFallbackWebsearchQuery('or and')).toBeNull();
});
});
describe('hybridSearch wiring — title arm reaches the fused result set', () => {
test('exact-title query surfaces the page through hybridSearch (keyword-only path)', async () => {
await seedTitleOnlyPage();
const results = await hybridSearch(engine, 'Chronomancer Codex Ledger', { limit: 5 });
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
});
test('long exact-title query (>=8 content tokens) surfaces through hybridSearch', async () => {
const longTitle = 'Emerald Falcon Doctrine Quarterly Synthesis Report Alpha Bravo Charlie Delta';
await engine.putPage('reports/emerald-falcon', {
type: 'note',
title: longTitle,
compiled_truth: 'An annual planning artifact.',
});
await engine.upsertChunks('reports/emerald-falcon', [
{ chunk_index: 0, chunk_text: 'An annual planning artifact.', chunk_source: 'compiled_truth' },
]);
const results = await hybridSearch(engine, longTitle, { limit: 5 });
expect(results.map(r => r.slug)).toContain('reports/emerald-falcon');
});
test('body-only queries still work (no regression from the extra arm)', async () => {
await seedTitleOnlyPage();
const results = await hybridSearch(engine, 'scheduling practices planning', { limit: 5 });
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
});
test('hybrid keyword arm still opts into the OR fallback (F1: QA-verified behavior preserved)', async () => {
await seedTitleOnlyPage();
// One bad token against body text: direct searchKeyword (no flag) finds
// nothing, but hybridSearch sets orFallback for its recall arm.
const q = 'scheduling practices zzzmissingtoken';
expect((await engine.searchKeyword(q, { limit: 5 })).length).toBe(0);
const results = await hybridSearch(engine, q, { limit: 5 });
expect(results.map(r => r.slug)).toContain('projects/chronomancer');
});
});