Compare commits

...
Author SHA1 Message Date
Garry Tan 952b237c0b fix(test): update the three remaining KNOBS_HASH_VERSION pins to 14 (#3430)
Missed in the first pass because I ran a targeted set of test files instead of
the full suite. CI shards 3, 8 and 10 caught them:

  test/search/knobs-hash-reranker.test.ts:67
  test/cross-modal-phase1.test.ts:139,149
  test/search-alias-resolved-boost.test.ts:93

Each carries the running history of why the version moved, so each gets the
13→14 rationale appended rather than just the number swapped. No pins at 13
remain anywhere in test/.
2026-07-28 12:21:29 -07:00
Garry Tan b9292ac082 fix(search): stop boosting compiled_truth at default detail (#3430)
COMPILED_TRUTH_BOOST = 2.0 is applied AFTER RRF normalization, and RRF's whole
dynamic range over a 100-deep pool is 1/60 -> 1/160 (a factor of 2.67). So a
2.0x multiplier consumes roughly three quarters of the range: break-even is
`2/(60+r) >= 1/60`, i.e. r <= 60, which means ANY boosted chunk inside the
first 60 ranks outranks an unboosted rank-1 chunk. That is a categorical
filter, not a tilt.

Measured against master's own rrfFusion, with the correct answer in a
fenced_code chunk at vector rank 0:

  compiled_truth chunks in pool | final rank | in top-20
  10                            | 10         | yes
  20                            | 20         | NO
  40                            | 40         | NO
  80                            | 59         | NO

With the boost off the answer stays at rank 0 in every case.

The gate was spelled `detail !== 'high'` -- written as though `high` were the
special case. The documented contract in src/core/operations.ts is
"low (compiled truth only), medium (default, all with dedup), high (all
chunks)", which makes LOW the special one: `low` already restricts to
compiled_truth, so a boost there is a no-op among equals, while `medium` and
`high` are both meant to see everything. So the default detail was silently
compiled-truth-only, contradicting the op's own description.

Three changes:

1. The three fusion call sites now route through a named predicate,
   `shouldBoostCompiledTruth(detail)`, returning true only for 'low'.
   Extracted rather than left inline precisely because an inline expression is
   only reachable through a full hybridSearch round trip -- which is why the
   inversion went unnoticed. The predicate is directly unit-testable.

2. KNOBS_HASH_VERSION 13 -> 14. Results are cached AFTER fusion, so rows
   ranked under the old semantics would otherwise be served under the new ones
   for the whole TTL (3600s default). One-time miss spike on upgrade.

3. test/search-compiled-truth-boost-scope.test.ts pins both the mapping and
   the arithmetic, and documents the displacement it prevents.

Verified the tests discriminate: stubbing the OLD predicate body into master
(so the failure is behavioral rather than a missing export) gives 4 fail /
3 pass; with the fix, 7 pass. typecheck clean, verify 32/32, and 144 pass /
0 fail across the search + fusion + cache suites.
2026-07-28 12:01:56 -07:00
7 changed files with 157 additions and 12 deletions
+29 -3
View File
@@ -48,6 +48,32 @@ import {
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
/**
* Which detail levels get the compiled_truth boost (#3430).
*
* ONLY `low`. The documented contract (`src/core/operations.ts`) is
* "low (compiled truth only), medium (default, all with dedup), high (all
* chunks)" — so `low` is the level that privileges compiled truth, and both
* `medium` and `high` are supposed to see everything on equal footing.
*
* This was previously spelled `detail !== 'high'`, i.e. written as though
* `high` were the special case. Because COMPILED_TRUTH_BOOST is applied AFTER
* RRF normalization, and RRF's whole range over a 100-deep pool is 1/60 → 1/160,
* a 2.0x multiplier is not a tilt — break-even is `2/(60+r) >= 1/60`, so any
* boosted chunk inside the first 60 ranks outranks an unboosted rank-1 chunk.
* At the default detail that made search categorically compiled-truth-only:
* a page whose answer lived in a `fenced_code` chunk returned the prose chunk,
* and the code chunk fell out of the window entirely.
*
* Extracted as a named predicate rather than left inline at three call sites so
* the detail→boost mapping is directly testable. An inline expression can only
* be covered through a full `hybridSearch` round trip, which is why the
* original inversion went unnoticed.
*/
export function shouldBoostCompiledTruth(detail: string | null | undefined): boolean {
return detail === 'low';
}
const pendingCacheWrites = new Set<Promise<unknown>>();
/**
@@ -1169,7 +1195,7 @@ export async function hybridSearch(
const noEmbedLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) noEmbedLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) noEmbedLists.push({ list: relationalList, k: fk });
noEmbedResults = rrfFusionWeighted(noEmbedLists, detailResolved !== 'high');
noEmbedResults = rrfFusionWeighted(noEmbedLists, shouldBoostCompiledTruth(detailResolved));
}
if (noEmbedResults.length > 0) {
await runPostFusionStages(engine, noEmbedResults, postFusionOpts);
@@ -1413,7 +1439,7 @@ export async function hybridSearch(
const fallbackLists = [{ list: keywordResults, k: fk }];
if (titleResults.length > 0) fallbackLists.push({ list: titleResults, k: fk });
if (relationalList.length > 0) fallbackLists.push({ list: relationalList, k: fk });
fallbackResults = rrfFusionWeighted(fallbackLists, detail !== 'high');
fallbackResults = rrfFusionWeighted(fallbackLists, shouldBoostCompiledTruth(detail));
}
if (fallbackResults.length > 0) {
await runPostFusionStages(engine, fallbackResults, postFusionOpts);
@@ -1500,7 +1526,7 @@ export async function hybridSearch(
// arms BEFORE fusion so the compiled-truth authority boost skips them.
await stampUnverifiedExtractions(engine, allLists.flatMap((l) => l.list));
let fused = rrfFusionWeighted(allLists, detail !== 'high');
let fused = rrfFusionWeighted(allLists, shouldBoostCompiledTruth(detail));
// Cosine re-scoring before dedup so semantically better chunks survive.
// v0.36 (D9): hydrate from the active embedding column so rescore happens
+1 -1
View File
@@ -766,7 +766,7 @@ export function attributeKnob<K extends keyof ModeBundle>(
// written between the #3391 stale-fix (which changes which chunks count as
// current) and the operator's migration run. Same one-time global cold-miss
// pattern as the bumps above.
export const KNOBS_HASH_VERSION = 13;
export const KNOBS_HASH_VERSION = 14;
/**
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
+3 -2
View File
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
return resolveSearchMode({ mode: 'balanced' });
}
test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => {
test('KNOBS_HASH_VERSION is 14 (cross-modal still appended; 13→14 compiled_truth boost scope #3430)', () => {
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
@@ -146,7 +146,8 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
// finally reaches asymmetric providers — pre-fix rows were keyed on
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
expect(KNOBS_HASH_VERSION).toBe(13);
// #3430: 13→14 compiled_truth boost no longer applies at detail=medium.
expect(KNOBS_HASH_VERSION).toBe(14);
});
test('flipping unified_multimodal changes the hash', () => {
+2 -2
View File
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
});
describe('KNOBS_HASH_VERSION', () => {
it('is 13 (12→13 embedding-provider migration invalidates rows written against the prior embedding space, #3390)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
it('is 14 (13→14 compiled_truth boost no longer applies at detail=medium, so pre-fix rankings must be unreachable, #3430)', () => {
expect(KNOBS_HASH_VERSION).toBe(14);
});
});
@@ -0,0 +1,112 @@
/**
* #3430: the compiled_truth boost must not apply at `detail=medium`.
*
* `COMPILED_TRUTH_BOOST = 2.0` is applied AFTER RRF score normalization. RRF's
* entire dynamic range over a 100-deep pool is 1/60 → 1/160 (a factor of 2.67),
* so a 2.0x multiplier consumes roughly three quarters of it. Break-even is
* `2/(60+r) >= 1/60`, i.e. r <= 60 — so ANY boosted chunk in the first 60 ranks
* outranks an unboosted rank-1 chunk. That is a categorical filter, not a tilt:
* a page whose actual answer is in a `fenced_code` chunk returns the prose
* chunk instead, and the code chunk leaves the result window entirely.
*
* The gate was written as `detail !== 'high'` — "high is special" — but the
* documented contract in `src/core/operations.ts` is:
*
* low (compiled truth only), medium (default, all with dedup), high (all chunks)
*
* which makes LOW the special one. `low` already restricts to compiled_truth,
* so a boost there is a no-op among equals; `medium` and `high` are both
* supposed to see everything. Hence `detail === 'low'`.
*
* These tests pin the arithmetic, not the constant — they would still fail if
* someone reintroduced a boost at medium with a different multiplier or behind
* a score floor, which is why they assert final RANK rather than score.
*/
import { describe, test, expect } from 'bun:test';
import { rrfFusion, RRF_K, shouldBoostCompiledTruth } from '../src/core/search/hybrid.ts';
import { KNOBS_HASH_VERSION } from '../src/core/search/mode.ts';
import type { SearchResult } from '../src/core/types.ts';
function chunk(slug: string, chunkSource: string): SearchResult {
return { slug, chunk_source: chunkSource, chunk_text: 'x', title: slug, score: 0 } as unknown as SearchResult;
}
/** One vector arm: the correct answer at rank 0, then `n` compiled_truth chunks. */
function poolWithAnswerFirst(n: number): SearchResult[] {
const list = [chunk('code/answer', 'fenced_code')];
for (let i = 0; i < n; i++) list.push(chunk(`prose/p${i}`, 'compiled_truth'));
return list;
}
function rankOfAnswer(results: SearchResult[]): number {
return results.findIndex((r) => r.slug === 'code/answer');
}
describe('#3430: the detail→boost mapping itself', () => {
// These are the assertions that actually FAIL on master. The rrfFusion tests
// below pin the arithmetic but pass either way, because they pass the boost
// flag explicitly — they cannot see how hybridSearch decides it. This is the
// wiring.
test('ONLY detail=low boosts compiled_truth', () => {
expect(shouldBoostCompiledTruth('low')).toBe(true);
expect(shouldBoostCompiledTruth('medium')).toBe(false);
expect(shouldBoostCompiledTruth('high')).toBe(false);
});
test('an absent detail does not boost — medium is the documented default', () => {
// Callers that omit detail get medium semantics, so the unset case must
// match medium, not low. A `!== 'high'` spelling gets this backwards.
expect(shouldBoostCompiledTruth(undefined)).toBe(false);
expect(shouldBoostCompiledTruth(null)).toBe(false);
});
test('an unrecognized detail value does not boost', () => {
// Fail-open toward showing everything rather than silently filtering.
expect(shouldBoostCompiledTruth('')).toBe(false);
expect(shouldBoostCompiledTruth('LOW')).toBe(false);
expect(shouldBoostCompiledTruth('detailed')).toBe(false);
});
test('the cache version was bumped so pre-fix rankings are unreachable', () => {
// Results are cached AFTER fusion, so rows written under the old boost
// semantics would otherwise be served under the new ones for the whole TTL.
// 13 was the pre-fix value.
expect(KNOBS_HASH_VERSION).toBeGreaterThanOrEqual(14);
});
});
describe('#3430: compiled_truth boost scope', () => {
test('boost OFF (detail=medium/high) keeps the vector-ranked answer at rank 0', () => {
// The regression this file exists for. Pre-fix, medium passed applyBoost=true
// and the answer landed at rank n — outside a 20-result window for n >= 20.
for (const n of [10, 20, 40, 80]) {
const fused = rrfFusion([poolWithAnswerFirst(n)], RRF_K, false);
expect(rankOfAnswer(fused), `n=${n}: answer must stay first without the boost`).toBe(0);
}
});
test('boost ON demonstrates the categorical displacement it causes', () => {
// Documents WHY the boost cannot be on at medium. Not an endorsement of
// these numbers — a characterization of the mechanism, so a future reader
// sees the cost rather than re-deriving it.
const observed = [10, 20, 40].map((n) => ({
n,
rank: rankOfAnswer(rrfFusion([poolWithAnswerFirst(n)], RRF_K, true)),
}));
// Displacement scales with pool composition: the answer is pushed back by
// roughly one position per boosted chunk ahead of the break-even rank.
for (const { n, rank } of observed) {
expect(rank, `n=${n}: boosted chunks should displace the answer`).toBeGreaterThan(0);
}
// And past ~20 compiled_truth chunks it leaves a default-size window.
expect(observed.find((o) => o.n === 20)!.rank).toBeGreaterThanOrEqual(20);
});
test('with the boost off, compiled_truth still wins when the vector arm ranks it first', () => {
// Guard against over-correcting: removing the boost must not penalize
// compiled_truth, only stop privileging it.
const list = [chunk('prose/answer', 'compiled_truth'), chunk('code/other', 'fenced_code')];
const fused = rrfFusion([list], RRF_K, false);
expect(fused[0].slug).toBe('prose/answer');
});
});
+6 -3
View File
@@ -413,7 +413,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
// #3390/#3391: bumped 12→13 for the embedding-provider migration wave —
// legacy callers hash prov=default before AND after a provider swap, so
// pre-migration cache rows must become unreachable on upgrade.
expect(KNOBS_HASH_VERSION).toBe(13);
// v0.42.67.x bumped 13→14: the compiled_truth boost no longer applies at
// detail=medium (#3430). Cached rows were ranked under the old semantics,
// so they must become unreachable rather than be served under the new ones.
expect(KNOBS_HASH_VERSION).toBe(14);
});
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
@@ -578,8 +581,8 @@ describe('v0.40.4 — graph_signals knob', () => {
});
describe('v0.42.3.0 — autocut knobs', () => {
test('KNOBS_HASH_VERSION is 13 (12→13 embedding-migration wave, #3390/#3391)', () => {
expect(KNOBS_HASH_VERSION).toBe(13);
test('KNOBS_HASH_VERSION is 14 (13→14 compiled_truth boost scope fix, #3430)', () => {
expect(KNOBS_HASH_VERSION).toBe(14);
});
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
+4 -1
View File
@@ -64,7 +64,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
// pre-fix document-side query vectors must not be served.
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
expect(KNOBS_HASH_VERSION).toBe(13);
// #3430: 13→14 — the compiled_truth boost no longer applies at
// detail=medium. Results are cached after fusion, so rows ranked under
// the old boost semantics must not be served under the new ones.
expect(KNOBS_HASH_VERSION).toBe(14);
});
test('hash is 16 hex chars regardless of reranker config', () => {