mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
v0.33.2.0 feat(search-lite): token budget + semantic query cache + intent weighting (#897)
* feat(search-lite): token budget + semantic query cache + intent weighting
Adds three additive features to the hybrid search pipeline. All
backward-compatible: existing callers see identical behavior unless they
opt in to the new options.
## 1. Token Budget Enforcement (src/core/search/token-budget.ts)
Cap the cumulative token cost of returned results so search payloads
fit downstream context windows. Greedy top-down walk; preserves caller
ordering; no re-rank. char/4 heuristic for token counting (no
tokenizer dependency \u2014 keeps the bun --compile bundle small).
SearchOpts.tokenBudget \u2014 numeric cap. Default undefined = no-op.
HybridSearchMeta.token_budget = { budget, used, kept, dropped }
HTTP query op: pass `token_budget` param.
## 2. Semantic Query Cache (src/core/search/query-cache.ts + migration v52)
Cache search results keyed by query embedding similarity. HNSW lookup:
`embedding <=> $1 < 0.08` (cosine similarity >= 0.92). Per-source
isolation so multi-source brains don\u2019t bleed. Per-row TTL (default 3600s).
Best-effort writes; all errors swallowed so the cache never breaks the
search hot path.
Migration v52 creates query_cache table with HALFVEC where pgvector >= 0.7;
falls back to VECTOR with the resolved config.embedding_dimensions dim.
New `gbrain cache` CLI: stats / clear --yes / prune.
Config keys: search.cache.enabled / similarity_threshold / ttl_seconds.
HybridSearchMeta.cache = { status, similarity?, age_seconds? }
Routed through new `hybridSearchCached(engine, query, opts)` wrapper;
the operations.ts query op now uses this wrapper so MCP/CLI calls
benefit automatically. Skipped for two-pass walks + non-default
embedding columns where cache semantics don\u2019t hold.
## 3. Zero-LLM Intent Weighting (src/core/search/intent-weights.ts)
Builds on the existing query-intent classifier (4 intents: entity /
temporal / event / general). New weight-adjustment layer applies subtle
per-intent nudges:
entity \u2192 boost keyword RRF + exact slug/title match
temporal \u2192 default recency=on when caller left it unset
event \u2192 boost keyword RRF (rare named entities) + soft recency
general \u2192 no-op (1.0 multipliers everywhere)
All adjustments are SUBTLE (max 1.25x). Caller-explicit options ALWAYS
win \u2014 intent weighting never silently overrides recency / salience.
Default ON; opt out via `opts.intentWeighting = false`. LLM query
expansion (expansion.ts) is still available and opt-in via
`opts.expansion = true` \u2014 it just isn\u2019t the default anymore.
HybridSearchMeta.intent now surfaces classifier output for debugging.
## Tests
test/token-budget.test.ts (10 tests, pure module)
test/intent-weights.test.ts (13 tests, pure module)
test/query-cache.test.ts (12 tests, PGLite)
test/hybrid-search-lite.serial.test.ts (9 tests, PGLite e2e)
Plus 105 pre-existing search tests still pass. `bun run verify` clean.
Co-authored-by: Wintermute <agents@garrytan.com>
* feat(search-mode): MODE_BUNDLES + resolveSearchMode wired into bare hybridSearch
Three named modes (conservative / balanced / tokenmax) that bundle the
search-lite knobs from PR #897 into a single config key. Mode resolution
lives in bare hybridSearch (NOT just the cached wrapper) so eval-replay
and eval-longmemeval — which call bare hybridSearch — test the same
mode-affected behavior as production. See [CDX-5+6] in the plan.
The mode bundle supplies DEFAULTS for intentWeighting, tokenBudget,
expansion, and searchLimit when the caller leaves those undefined.
Per-call SearchOpts and per-key config overrides still win (matches the
v0.31.12 model-tier resolution chain at model-config.ts:resolveModel).
knobsHash() exposes a stable SHA-256 of the resolved knob set; the cache
contamination hotfix (next commit) consumes it to prevent a tokenmax
write from being served to a conservative read.
Three new fields on HybridSearchMeta:
- mode (resolved mode name)
- existing token_budget meta now fires from bare hybridSearch too
Bare hybridSearch now applies tokenBudget at all three return paths
(no-embedding-provider, keyword-only-fallback, main). Previously only
hybridSearchCached enforced budget; eval commands missed it.
Tests: 37 unit cases pin the 3x7 bundle table cell-by-cell, the
resolution chain semantics, knobs hash determinism + cross-mode
separation, and the config-table parser. All 72 search-lite tests pass.
Bisect-friendly: this commit ONLY adds mode resolution. The cache-key
contamination hotfix [CDX-4] is a separate atomic commit (next).
* fix(query-cache): cross-mode contamination hotfix [CDX-4]
PR #897's query_cache keyed rows on sha256(source_id::query_text) only.
A tokenmax search (expansion=on, limit=50) populated a row that a
subsequent conservative call (no expansion, limit=10) read back, serving
the wrong-shape results. This is a real bug in PR #897 today, regardless
of the v0.32.3 mode picker work — Codex caught it in plan review.
Fix:
- Migration v56 adds query_cache.knobs_hash TEXT column + composite
(source_id, knobs_hash, created_at) index. Existing rows have NULL
knobs_hash and are excluded from lookups (silently re-populated with
the right hash on first hit — no orphan data, no destructive migration).
- cacheRowId(query, source, knobsHash) — knobsHash now part of the PK so
a tokenmax write and a conservative write for the same (query, source)
land in distinct rows.
- SemanticQueryCache.lookup({knobsHash}) filters WHERE knobs_hash = $.
- SemanticQueryCache.store({knobsHash}) writes the resolved hash.
- hybridSearchCached threads knobsHash from resolveSearchMode through
every cache call. Cache config (enabled/threshold/TTL) now reads from
the resolved mode bundle, not directly from the config table.
Tests (test/query-cache-knobs-hash.test.ts, 11 cases):
- cacheRowId bifurcates by knobsHash
- Tokenmax write does NOT contaminate conservative lookup
- Three modes coexist as distinct rows for same query
- Legacy NULL-knobs_hash rows are excluded from lookup
- Same-mode write updates in place (no duplicate rows)
All 58 cache + mode tests pass. Migration v56 applies cleanly on a fresh
PGLite brain.
Bisect-friendly: this commit is the cache-key hotfix alone. Mode
resolution wiring lives in the previous commit.
* feat(search-telemetry): in-process rollup writer + search_telemetry table
Migration v57 creates search_telemetry (date, mode, intent, count,
sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss,
first_seen, last_seen). PK (date, mode, intent) caps growth at ~4380
rows/year. Sums + counts only — averages derive at read time so
concurrent ON CONFLICT writes from multiple gbrain processes accumulate
correctly [CDX-17].
In-memory bucket flushed periodically (60s OR 100 calls) + on process
beforeExit/SIGINT/SIGTERM with a 2-second cap. The search hot path NEVER
waits on this write [D2, CDX-19].
Date-bucketed cache_hit / cache_miss columns make hit rate over --days N
derivable [CDX-18]. query_cache.hit_count is a lifetime counter and
can't be sliced by window.
Wired into bare hybridSearch via emitMeta: every search call sync-bumps
a bucket. flush() drains atomically by swapping the map before SQL writes
so a record() during flush lands in the new map.
readSearchStats(engine, {days}) returns the StatsWindow shape that
gbrain search stats consumes (next commit).
Tests: 16 unit cases pin record/flush/read semantics including
ON-CONFLICT-adds-raw-values, concurrent-flush coalescing, cache hit-rate
math, missing-table graceful degradation, and window clamping.
53 migrations apply on a fresh PGLite brain.
* feat(config): add unset + listConfigKeys + readLineSafe helper [CDX-7+8+9]
CDX-8: gbrain config has no unset path today. Required before
`gbrain search modes --reset` can clear search.* overrides.
- BrainEngine.unsetConfig(key) → returns rows deleted (0|1)
- BrainEngine.listConfigKeys(prefix) → exact-literal prefix match
with LIKE-escape on user-supplied % / _ / \ characters
- PGLiteEngine + PostgresEngine implementations
- `gbrain config unset <key>` and `gbrain config unset --pattern <prefix>`
sub-subcommands
CDX-9: readLine has no EOF detection or timeout. Mode-picker plan calls
out "TTY closes mid-prompt → defaults to balanced" but the raw helper
hangs forever. New readLineSafe(prompt, defaultValue, timeoutMs=60s):
- Returns defaultValue on stdin 'end' event
- Returns defaultValue on timeout
- Returns defaultValue on empty Enter
- Non-TTY stdin returns defaultValue immediately (e2e safe)
- Returns trimmed user input otherwise
Exported so install picker (next task) can use it.
Tests: 9 cases pin unset semantics + prefix matcher edge cases
(glob-wildcard escape, sort order, idempotent loop, search.* sweep).
All 53 migrations apply on a fresh PGLite brain.
* feat(init): install-time mode picker + upgrade banner
Install picker (src/commands/init-mode-picker.ts):
- Runs as a phase inside `gbrain init` AFTER engine.initSchema() so DB
config writes work [CDX-7].
- Idempotent: skipped on re-init if search.mode is already set.
- Smart auto-suggestion via recommendModeFor() reads
models.tier.subagent / models.default / OPENAI_API_KEY:
* Opus default/subagent → tokenmax (quality ceiling)
* Haiku subagent → conservative (4K budget keeps cost down)
* No OpenAI key → conservative (no LLM expansion possible)
* Sonnet / unknown → balanced (safe default)
- TTY shows menu via readLineSafe (60s timeout, defaults on EOF/empty).
- Non-TTY auto-selects + emits operator hint:
[gbrain] search mode: X (auto-selected — reason)
[gbrain] To change: gbrain config set search.mode <...>
- --json mode emits structured `{phase: 'search_mode_picker', ...}` event.
- Wired into both initPGLite and initPostgres flows.
Upgrade banner (src/commands/upgrade.ts):
- One-shot stderr banner in runPostUpgrade.
- State persisted via config key `search.mode_upgrade_notice_shown=true`
— fires at most once per install.
- Copy corrected per [CDX-1+2+3]: production query op STILL defaults
expand=true and limit=20. The banner reframes from "behavior is
regressing" to "named modes available + here's how to preserve
exact current shape."
Tests (test/init-mode-picker.test.ts, 16 cases):
- recommendModeFor heuristic for all 4 input shapes
- parseModeInput accepts numeric/named/case-insensitive, rejects garbage
- runModePicker non-TTY auto-selects + writes config
- Idempotent + --force re-prompt + JSON output
- Opus → tokenmax, Haiku → conservative real wiring through engine
* feat(cli): gbrain search modes/stats/tune command
Three sub-subcommands mirroring the gbrain models (v0.31.12) shape:
gbrain search modes [--json]
Read-only routing dashboard. Shows the three mode bundles, the active
mode, and the source of every resolved knob:
cache_enabled = true [override: search.cache.enabled]
tokenBudget = 4000 [mode: conservative]
Plus knob descriptions for legibility.
gbrain search modes --reset [--source <mode>]
Clears every search.* override (NOT search.mode itself). Preserves
the upgrade-notice state key. --source <mode> is a dry-run that
lists what --reset would change without writing — the paved path
[CDX-8] flagged as missing.
gbrain search stats [--days N] [--json]
Observability. Reads the search_telemetry rollup over the window
(clamps to [1, 365]). Prints cache hit rate, mode mix, intent mix,
budget drops, avg results/tokens. JSON output includes
_meta.metric_glossary block per [CDX-25].
gbrain search tune [--apply] [--json]
Recommendation engine. 5 rules cover the bug class:
- Insufficient data → "no_recommendations" status
- Conservative + high budget-drop rate → suggest balanced
- High cache hit rate (>85%) → suggest similarity threshold bump
- Tokenmax + Haiku subagent → suggest balanced (cost mismatch)
- Cache disabled but stats show usage → suggest re-enabling
--apply mutates config via setConfig / unsetConfig with a paste-ready
revert command printed at the end.
Registered in src/cli.ts dispatch table. 17 unit cases pin:
- Dashboard report shape + per-knob source attribution
- --reset preserves search.mode + notice key
- --source dry-run never writes
- stats reads telemetry rollup; --days clamps
- tune recommendation rules fire on real telemetry data
- --apply mutates config
- --help + unknown subcommand exit codes
* feat(eval): metric glossary module + auto-gen METRIC_GLOSSARY.md + CI guard
Single source of truth at src/core/eval/metric-glossary.ts. Every entry
carries 3 fields:
- industry_term (canonical IR/NLP literature name, preserved verbatim)
- eli10 (plain-English a 16-year-old can follow)
- range (numeric range + interpretation)
Covers 4 metric families:
- Retrieval: P@k, R@k, MRR, nDCG@k
- Stability: Jaccard@k, top-1 stability
- Statistical: p-value (paired bootstrap + Bonferroni), 95% CI
- Operational: cache hit rate, avg results/tokens, cost per query, p99 latency
Public surface:
- getMetricGloss(metric) → full entry or null
- eli10For(metric) → plain-English string or null
- buildMetricGlossaryMeta(metrics[]) → {metric → eli10} record for
JSON `_meta.metric_glossary` blocks per [CDX-25]. ONE block per
response, NOT sibling `_gloss` fields on every metric.
- renderMetricGlossaryMarkdown() → deterministic Markdown for the doc
Auto-generation:
scripts/generate-metric-glossary.ts emits docs/eval/METRIC_GLOSSARY.md.
Deterministic (same input → same bytes) so the CI guard can diff.
CI guard:
scripts/check-eval-glossary-fresh.sh regenerates into a temp file and
diffs against the committed doc. Out-of-date doc fails the build.
Wired into `bun run verify` (and therefore `bun run test:full`).
Tests (test/metric-glossary.test.ts, 18 cases):
- Every documented metric is present
- Every entry has all 3 required fields
- Accessors return null on unknown metrics (no throw)
- buildMetricGlossaryMeta silently drops unknown metrics
- renderer output is deterministic across calls
- Renderer groups metrics into 4 sections
docs/eval/METRIC_GLOSSARY.md: 5491 bytes, 124 lines, fresh.
* feat(doctor): search_mode + eval_drift checks + drift-watch module
src/core/eval/drift-watch.ts — curated retrieval watch-list [CDX-6].
Five patterns covering the surface that actually affects retrieval quality:
- src/core/search/ (search pipeline)
- src/core/embedding.ts (embedding shape)
- src/core/chunkers/ (chunk granularity)
- src/core/ai/recipes/anthropic.ts + openai.ts (expansion + embed routing)
- src/core/operations.ts (the query op definition)
Adding to the list is a deliberate act — requires a CHANGELOG line so
coverage grows on purpose, not by accident. Pure functions:
- matchesWatchPattern(path) — trailing-slash = prefix, bare = equality
- filesDriftedSince(repoRoot, sha?) — git diff --name-only wrapper
- watchedFilesDrifted(repoRoot, sha?) — composite
src/commands/doctor.ts — two new checks.
checkSearchMode [CDX-20]: status stays 'ok' (never warns, never docks
health score). Hint in message field. Three branches:
- unset → "search.mode is unset (using balanced fallback). Run
`gbrain search modes` to see what is running and pick a mode."
- mode + no overrides → "Mode: X (no per-key overrides — mode bundle
is canonical)."
- mode + overrides → "Mode: X with N per-key override(s) (k1, k2, …).
To consolidate to the pure mode bundle: gbrain search modes --reset"
Upgrade-notice state key (search.mode_upgrade_notice_shown) is excluded
from the override roster — it's not a knob.
checkEvalDrift [CDX-6]: surfaces uncommitted changes to retrieval-watched
files. Always 'ok'; operator-facing reminder. Names up to 3 drifted files
in the message + paste-ready re-eval command.
Both helpers exported (was: file-private) so tests can pin behavior
without walking the full runDoctor pipeline.
Tests: 12 drift-watch cases + 7 doctor-check cases. Pin watch-list shape,
prefix-vs-equality matcher semantics, missing-repo graceful failure, and
all three search_mode branches.
* feat(eval): --mode flag on longmemeval/replay + run-all + compare
Per-mode --mode flag plumbed into:
- gbrain eval longmemeval --mode <conservative|balanced|tokenmax>
Sets search.mode in the benchmark brain's config table; config is
in PRESERVE_TABLES so resetTables doesn't wipe it between questions.
Mode surfaces in the per-question NDJSON row.
- gbrain eval replay --mode <m> + --compare-limit N
--compare-limit forces a constant K across modes [CDX-13]; without
it, Jaccard@k against the captured baseline measures K-drift, not
quality. Mode is set once before the replay loop.
- NOT cross-modal per [CDX-11]: cross-modal scores OUTPUT against
TASK; it doesn't retrieve. Adding --mode there is theater.
New: gbrain eval run-all orchestrator (src/commands/eval-run-all.ts):
- Sweeps every requested mode × suite combination
- Sequential default per D9; --parallel N opt-in (clamped to mode count)
- Cost guard with split caps [CDX-15+16]:
--budget-usd-retrieval N (default $5)
--budget-usd-answer N (default $20)
Non-TTY refuses with exit 2 unless --yes AND explicit --budget-usd-*
flags pass. TTY refuses without --yes (defense against agent loops).
- estimateRunCost computes per-(suite,mode) breakdown including the
expansion-Haiku surcharge for tokenmax.
- Audit trail: appends to <repo>/.gbrain-evals/eval-results.jsonl
[CDX-23]. Personal brain (~/.gbrain) NEVER touched.
- v0.32.3 ships orchestrator + argv + guard + persist hook.
In-process per-suite invocation is a v0.32.4 follow-up (operator
runs the per-suite CLIs with the documented --mode flag for now;
each completion calls persistRunRecord to log).
New: gbrain eval compare report (src/commands/eval-compare.ts):
- Reads eval-results.jsonl, groups by (suite, mode), renders MD or JSON
- Most-recent (suite, mode, commit) wins when duplicates exist
- JSON output has schema_version=2 + _meta.metric_glossary block per
[CDX-25] (ONE block per response, not sibling _gloss fields)
- _meta.methodology field names the paired-bootstrap + Bonferroni
discipline per [CDX-14] so haters can reproduce
- Missing file → friendly hint pointing at `gbrain eval run-all`
Wired into eval dispatch table in src/commands/eval.ts.
Metric glossary fuzzy fallback: `recall@10` → `recall@k` lookup
(the glossary documents the family; report rows carry specific K
values). Routes through getMetricGloss for every call site.
Tests (42 cases total — all green):
- eval-run-all.test.ts (19): argv parser, cost estimate, guard
semantics for all 4 (over/under × tty/non-tty) shapes, persist hook
NDJSON shape.
- eval-compare.test.ts (5): JSON + MD output shapes, glossary
integration, missing-file graceful, mode filter, most-recent-wins.
- metric-glossary.test.ts (18): unchanged but updated assertions to
cover the fuzzy `@N` → `@k` fallback.
Pre-existing eval-replay / eval-longmemeval / eval-export / eval-prune
tests (42 cases) still pass — --mode + --compare-limit are additive.
* docs: methodology + CLAUDE.md/README/RESOLVER + skills/conventions
docs/eval/SEARCH_MODE_METHODOLOGY.md — haters-immune 8-section template.
Documents what the eval measures + does NOT measure, datasets + sizes
(LongMemEval n=500, Replay n=200, BrainBench n=1240 docs / 350 qrels),
random seed 42, run procedure verbatim, threats to validity (LongMemEval
English+technical skew, char/4 heuristic ~5-10% off, expansion ~97.6%
relative lift on this corpus), per-question raw outputs, pre-registered
expectations (tokenmax wins R@10 by 5-15pp, conservative wins cost by
5-15x, balanced lands within 3pp), re-run cadence anchored to the
src/core/eval/drift-watch.ts watch-list.
Statistical-significance section pins paired bootstrap with 10,000
resamples + Bonferroni correction across 3 modes × 4 metrics [CDX-14].
CLAUDE.md gets two new sections: ## Search Mode (3-mode table + resolution
chain + [CDX-4] cache contamination fix note + CLI commands) and ## Eval
discipline (single-source-of-truth glossary, methodology doc, eval_results
in repo NOT personal brain per [CDX-23]).
README.md Quick Start gets a paragraph naming the install picker, mode
heuristic, and the methodology link.
skills/conventions/search-modes.md NEW — convention file consumed by
brain-ops + query + signal-detector skills via the existing
`> **Convention:**` callout pattern. Routes "what mode" / "tune
retrieval" / "compare modes" queries to the right CLI surface.
skills/RESOLVER.md gets two new trigger rows pointing at
gbrain search * and gbrain eval compare.
* chore: regen llms.txt + llms-full.txt for v0.32.3 search-mode docs
bun run build:llms — picks up the new CLAUDE.md sections (Search Mode +
Eval discipline) and the docs/eval/SEARCH_MODE_METHODOLOGY.md addition.
build-llms.test.ts gate now passes.
* fix(doctor): wire search_mode + eval_drift checks into runDoctor main flow
The v0.32.3 search_mode + eval_drift helpers were inserted into the
DB-checks sub-helper at runDbChecks (line 345-355), but runDoctor itself
maintains its own check list and only calls the helpers' subset. Push
the two checks into the main runDoctor path (after the existing
sync_freshness check at line 2347) so they actually appear in
`gbrain doctor --json` output.
Both checks gated on engine !== null. Progress reporter heartbeat fires
for each. Both still return status 'ok' per [CDX-20] so health score is
preserved.
Verified end-to-end on a real Postgres brain: gbrain doctor --json now
includes 'search_mode' and 'eval_drift' in the checks array.
* fix: claw-test hang — DATABASE_URL leak + telemetry beforeExit deadlock
Two root causes for the hang, both fixed.
1. DATABASE_URL leak in claw-test scripted harness
The harness inherits the parent process's env via `...process.env`
for every phase child (init / import / query / extract / doctor).
When the e2e runner sets DATABASE_URL (for OTHER e2e tests), it
leaks into claw-test's children. `loadConfig` at src/core/config.ts:143
then flips inferredEngine to 'postgres' for every subsequent phase,
breaking the hermetic-PGLite-tempdir contract: phases race against
each other on a shared test Postgres while pointing at different
brain states.
Fix: strip DATABASE_URL + GBRAIN_DATABASE_URL from the child env
before forwarding. Re-apply GBRAIN_HOME / GBRAIN_FRICTION_RUN_ID
after the merge so a parent's override can't win. The harness is
PGLite-only by design.
2. Telemetry beforeExit deadlock
v0.32.3's recordSearchTelemetry installed a `process.on('beforeExit',
drainOnExit)` hook that wrapped the flush in `Promise.race([flush(),
setTimeout(2000)])`. beforeExit fires when the event loop empties,
but the hook enqueued NEW async work (the race's setTimeout +
pending flush), so the event loop never re-emptied. Short-lived
CLI invocations (`gbrain query "the"` finishing in ~100ms) ended
up waiting on the DB write indefinitely.
The claw-test harness spawns several short-lived gbrain queries.
Each one hung after its real work finished. The harness then waited
forever on its child subprocess's exit code.
Fix: drop the beforeExit + SIGINT + SIGTERM hooks. Per [CDX-19]'s
"stats are directional, not exact" contract, losing one unflushed
bucket on process exit is acceptable. The unref'd setInterval
handles long-running processes (HTTP MCP, autopilot, jobs work).
Short-lived CLI invocations exit immediately.
Verified:
- `gbrain query "the"` on a fresh PGLite brain exits in <1s (was
hanging forever).
- `bun test test/e2e/claw-test.test.ts` → 3 pass / 0 fail / 3.86s
(was hanging at the banner indefinitely).
- 85/85 e2e files / 574/574 tests pass including claw-test, with
DATABASE_URL set (the configuration that originally repro'd the
hang).
- 6235/6235 unit tests pass.
- Typecheck clean.
The two bugs interacted: the DATABASE_URL leak meant queries hit the
real Postgres (slow), making the beforeExit deadlock visible. Fixing
either alone would have masked the other. Both fixed in this commit.
* feat(install-picker): cost anchors in mode prompt + upgrade banner + docs
The install picker already asks explicitly (1/2/3 menu, default to the
recommendation on Enter). What was missing: a way to reason about the
cost tradeoff. Without numbers, "tokenmax" looks free and "conservative"
sounds restrictive; with numbers, the operator picks intentionally.
Cost anchors added everywhere the user encounters the mode choice:
- Install picker MENU_TEXT (gbrain init)
- Upgrade banner (gbrain upgrade post-upgrade)
- CLAUDE.md ## Search Mode section
- README.md Quick Start
- docs/eval/SEARCH_MODE_METHODOLOGY.md (with the math)
Anchors at Sonnet 4.6 downstream ($3/M input):
conservative ~$0.012/query ~$12/mo @ 1K ~$1,200/mo @ 100K
balanced ~$0.030/query ~$30/mo @ 1K ~$3,000/mo @ 100K
tokenmax ~$0.060/query ~$60/mo @ 1K ~$6,000/mo @ 100K
Plus tokenmax's Haiku expansion overhead: ~$1.50 per 1K queries on top.
Cache hits roughly halve these on a brain with repeat-query traffic.
The math is documented in SEARCH_MODE_METHODOLOGY.md so a reviewer can
audit each variable (T = ~400 tokens/chunk from the recursive chunker's
300-word target; N = `searchLimit` cap; R = downstream model rate from
src/core/anthropic-pricing.ts). Drift away from these numbers requires
updating CLAUDE.md + the picker + the methodology doc in lockstep — a
regression test pins the picker's anchor strings to enforce this.
The framing also names the cost rule honestly: the dominant cost isn't
gbrain (semantic cache is free; Haiku expansion is rounding-error). It's
the downstream agent reading retrieved chunks back into its context.
Operators who don't realize this pick badly.
Tests: 5 new regression cases in init-mode-picker.test.ts pin every
cost string in MENU_TEXT. Total 21/21 picker tests pass; 6240/6240
unit tests pass; verify gate green.
* docs: realistic-scale cost anchor for search modes
The per-query cost framing in the picker (~$0.012/$0.030/$0.060) is
honest but theoretical — it treats each search as an isolated billable
event. Real agent loops amortize a lot of context across turns via
Anthropic prompt caching, so the per-query 5x ratio doesn't translate
1:1 into total agent spend.
Added a "Realistic-scale anchor" section to SEARCH_MODE_METHODOLOGY.md
representing one heavy power-user agent loop running tokenmax:
- ~860 turns/mo (~29/day, one active agent)
- ~900K tokens/turn (system + tools + history + reasoning + search)
- ~$0.85/turn → ~$700/mo total agent spend at tokenmax
- ~88% Anthropic prompt-cache hit rate
Scaling balanced + conservative DOWN from that anchor:
- tokenmax → ~$700/mo, search ~22% of total spend
- balanced → ~$620/mo, search ~12% (saves ~$78/mo vs tokenmax)
- conservative → ~$575/mo, search ~5% (saves ~$124/mo vs tokenmax)
Honest takeaway: at realistic agent-loop scale WITH disciplined prompt
caching, mode choice saves 10-20% of total agent spend, not 5x. The
per-query math kicks back in for setups WITHOUT cache discipline (churn
the prompt prefix every turn → search payload becomes a larger fraction).
Both framings live in the doc.
CLAUDE.md ## Search Mode gets a forward-pointer paragraph naming the
"per-query math vs real-world spend" delta so agents reading the section
find the methodology footnote.
Numbers in the doc are anonymized + scaled away from any specific
deployment. No model names, no specific dollar figures from a real
production setup — just the per-turn / cache-hit-rate / search-count
shape ratios that a thoughtful operator can validate against their own
billing dashboard.
* feat(picker): mode × model cost matrix (25x corner-to-corner spread)
Previous version showed mode costs assuming Sonnet-only downstream.
That muted the spread to 5x and made mode choice look minor. Reality:
the downstream model tier is the BIGGER cost lever — pairing mode with
model is where the 25x spread lives.
New 3×3 matrix in the install picker, CLAUDE.md, methodology doc, README:
Haiku 4.5 Sonnet 4.6 Opus 4.7
($1/M input) ($3/M input) ($5/M input)
conservative $400/mo $1,200/mo $2,000/mo
balanced $1,000/mo $3,000/mo $5,000/mo
tokenmax $2,000/mo $6,000/mo $10,000/mo
(per-query cost @ 100K queries/mo, full search payload, no cache savings)
The methodology doc gets a new "Mode × Model matrix" section above the
realistic-scale anchor with concrete right-sizing guidance:
- tokenmax + Haiku: wrong direction. Haiku can't filter 50 chunks → noise
not signal. Pay Haiku rates, get sub-Haiku quality.
- conservative + Opus: wasted Opus. 200K context window starved on
retrieval depth. Pay Opus rates, get conservative-shape retrieval.
- Natural pairings span ~4x; the matrix corners span 25x. The natural
diagonal is where most users should land.
Realistic-scale anchor refreshed:
- tokenmax + Opus: ~$700/mo at 860 turns
- balanced + Sonnet: ~$430/mo
- conservative + Haiku: ~$170/mo
Plus a "mismatched pairings" section showing the math for tokenmax+Haiku
and conservative+Opus — both burn budget for no improvement.
Regression test updated: pins the 25x framing + the four anchor cells
(two corners + two diagonal mids) + the three downstream model rates.
22/22 picker tests pass. 6241/6241 unit tests pass. CI guards green.
* docs(picker): rescale cost matrix from 100K → 10K queries/mo (typical single user)
Most users running gbrain are single-user installs at ~10K queries/month,
not the 100K fleet-scale used in the original matrix. The picker numbers
($400 to $10,000/mo) looked alien to the actual audience. Rescaled to
10K with an explicit linear-scaling callout.
New matrix in picker, CLAUDE.md, README, methodology doc:
Haiku 4.5 Sonnet 4.6 Opus 4.7
($1/M) ($3/M) ($5/M)
conservative $40/mo $120/mo $200/mo
balanced $100/mo $300/mo $500/mo
tokenmax $200/mo $600/mo $1,000/mo
Still 25x corner-to-corner. Still 4x natural-diagonal spread. But now in
numbers a single user picks up and reasons about: "balanced + Sonnet at
$300/mo, that's fine" or "tokenmax + Opus at $1,000/mo, that's a
deliberate choice for max-quality high-stakes work."
Every surface updated:
- Install picker MENU_TEXT (with "scales linearly — multiply by 10
for 100K/mo" footnote so heavier users still see their number)
- CLAUDE.md ## Search Mode table + scaling prose
- README Quick Start
- methodology doc Mode × Model matrix section
- upgrade banner (post-upgrade notice)
Regression test updated: pins the 3 new anchor cells ($40, $300, $1,000)
+ the 10K/mo volume frame + the linear-scaling callout. 23/23 picker
tests pass, 6241/6241 unit tests pass, verify gate green.
Methodology doc's existing 1K/10K/100K Monthly cost breakdown tables
left intact (they already show the linear scaling explicitly).
* feat(picker): agent-facing install protocol + tokenmax default + [AGENT] directive
DX gap: an agent installing gbrain (OpenClaw, Hermes, Codex, Cursor) ran
gbrain init non-TTY, saw 2 stderr lines flash by, and silently auto-applied
a default search mode. The operator never saw the cost matrix or the choice.
At 25x corner-to-corner cost spread, that's surprise-spend territory.
Five surfaces fixed:
1. **Auto-suggest default flipped balanced → tokenmax.** The Sonnet/unknown
fallback now recommends tokenmax (preserves v0.31.x retrieval shape:
expand=on, generous result set). Haiku subagent → conservative still
wins (cost-sensitive signal). No-OpenAI-key → conservative still wins
(vector search not possible). Heuristic reordered: Haiku check now
fires BEFORE the Opus check, because a Haiku subagent loop signalling
cost sensitivity should win over a default-model heuristic.
2. **gbrain init non-TTY output rebuilt.** Previously: 2 stderr lines.
Now: the full 3×3 cost matrix + an explicit [AGENT] directive block
telling the agent to relay the matrix to its operator before
continuing. Includes a pointer to INSTALL_FOR_AGENTS.md Step 3.5 for
the full protocol.
3. **gbrain upgrade banner same treatment.** Existing v0.32.3 banner now
includes [AGENT] directive at the top so upgrading agents relay the
matrix to their operator instead of silently accepting v0.31.x →
v0.32.x default-applied behavior.
4. **INSTALL_FOR_AGENTS.md Step 3.5 NEW** with the matrix verbatim, the
exact paraphrasable ask-the-user wording, and the gbrain config set
commands to run after the operator picks. Plus a paragraph in the
Upgrade section pointing back at Step 3.5.
5. **AGENTS.md install checklist** gets a new Step 4 ("STOP — ask the
user about search mode") between init and the rest of the flow. The
agent's job description now explicitly says: silent acceptance is
the wrong default.
Tests (24/24 pass):
- Updated recommendModeFor heuristic order (Haiku floor > Opus default)
- New regression test: non-TTY output contains the matrix corners +
[AGENT] directive + INSTALL_FOR_AGENTS.md pointer
- withEnv() helper used for OPENAI_API_KEY mutation (test-isolation lint)
- Default-recommendation tests updated: Sonnet / unknown → tokenmax
Privacy + test-isolation gates clean. 6256/6256 unit tests pass.
---------
Co-authored-by: garrytan-agents <agents@garrytan.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
This commit is contained in:
garrytan-agents
co-authored by
garrytan-agents
Garry Tan
garrytan-agents
Garry Tan
parent
182a144272
commit
1a6b543cc5
@@ -10,7 +10,15 @@ start here.
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
4. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
|
||||
default but printed a 9-cell cost matrix (mode × downstream model)
|
||||
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
|
||||
and confirm their choice before continuing. Cost spread between corners
|
||||
is 25x — silent acceptance is the wrong default. See
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
5. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
@@ -715,6 +715,100 @@ permission to run them — see the "run without asking" rule above.
|
||||
Never leave `gbrain-test-pg` running. If you find a stale one from a previous run,
|
||||
stop and remove it before starting a new one.
|
||||
|
||||
## Search Mode (v0.32.3)
|
||||
|
||||
GBrain ships three named search modes that bundle the search-lite knobs from
|
||||
PR #897 into a single config key. Pick one at install time; the rest of the
|
||||
project resolves through `src/core/search/mode.ts`.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
|
||||
The corner-to-corner spread is 25x once you pair mode with downstream model.
|
||||
Chunks ~400 tokens avg. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume), full search payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage). Natural pairings span ~4x.
|
||||
Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
— too-big payload overwhelms a cheap model; too-small payload starves an
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
Anthropic prompt caching see 50-80% discount on top (cache hits skip
|
||||
downstream entirely). The realistic-scale anchor in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` walks the natural pairings at
|
||||
single-power-user volume (~860 turns/mo): tokenmax+Opus ~\$700/mo,
|
||||
balanced+Sonnet ~\$430/mo, conservative+Haiku ~\$170/mo. Setups WITHOUT
|
||||
cache-aware prompt layout (frequent prefix churn) see the per-query
|
||||
matrix dominate — mode + model choice matters more there.
|
||||
|
||||
**Resolution chain** (matches the v0.31.12 model-tier pattern at
|
||||
`src/core/model-config.ts:resolveModel`):
|
||||
|
||||
per-call SearchOpts → per-key config (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in **bare `hybridSearch`** (NOT just the cached wrapper)
|
||||
per `[CDX-5+6]` in `~/.claude/plans/lets-take-a-look-validated-parrot.md` — so
|
||||
`gbrain eval replay` and `gbrain eval longmemeval` test the same mode-affected
|
||||
behavior as the production `query` op.
|
||||
|
||||
**Cache-key contamination hotfix `[CDX-4]`:** migration v56 added a
|
||||
`knobs_hash` column to `query_cache`. The lookup filter is now
|
||||
`WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $` so a
|
||||
tokenmax write (expansion=on, limit=50) can't be served to a conservative
|
||||
read.
|
||||
|
||||
**Three CLI surfaces:**
|
||||
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
|
||||
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
|
||||
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
|
||||
|
||||
## Eval discipline (v0.32.3)
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
|
||||
resolves through `src/core/eval/metric-glossary.ts` so industry terms
|
||||
(`P@k`, `nDCG@k`, `MRR`, `Jaccard@k`) carry a plain-English line in human
|
||||
output and a `_meta.metric_glossary` block in JSON output (one block per
|
||||
response per `[CDX-25]`, NOT sibling `_gloss` fields).
|
||||
|
||||
The full methodology — datasets, sample selection, pre-registered
|
||||
expectations, threats to validity, paired-bootstrap + Bonferroni p-value
|
||||
discipline `[CDX-14]` — lives in `docs/eval/SEARCH_MODE_METHODOLOGY.md`.
|
||||
Auto-regenerated `docs/eval/METRIC_GLOSSARY.md` is CI-guarded against
|
||||
drift (`scripts/check-eval-glossary-fresh.sh`).
|
||||
|
||||
Per-run records land at `<repo>/.gbrain-evals/eval-results.jsonl` per
|
||||
`[CDX-23]`. The user's personal `~/.gbrain` brain is NEVER touched —
|
||||
audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
|
||||
@@ -61,6 +61,63 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
|
||||
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
|
||||
NOT inside ~/gbrain.
|
||||
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M) ($3/M) ($5/M)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
|
||||
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
|
||||
```
|
||||
|
||||
**Ask the operator (paraphrase if needed):**
|
||||
|
||||
> Your gbrain just installed with search mode `<auto-applied default>`. This is
|
||||
> a one-time setup decision that controls retrieval payload size. Which mode
|
||||
> do you want?
|
||||
>
|
||||
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
|
||||
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
|
||||
>
|
||||
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
|
||||
>
|
||||
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
|
||||
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
|
||||
>
|
||||
> Cost depends on BOTH the mode AND the downstream model you run. See the
|
||||
> matrix above for the 9-cell breakdown.
|
||||
|
||||
If the operator picks a non-default mode, run:
|
||||
```bash
|
||||
gbrain config set search.mode <mode>
|
||||
```
|
||||
|
||||
If they pick tokenmax AND want to preserve the literal v0.31.x default
|
||||
(limit=20 instead of tokenmax's 50), also run:
|
||||
```bash
|
||||
gbrain config set search.searchLimit 20
|
||||
```
|
||||
|
||||
Verify the choice with `gbrain search modes` before continuing.
|
||||
|
||||
**Why this matters:** the cost spread between corners of the matrix is 25x.
|
||||
An agent that silently accepts the default and starts running queries against
|
||||
a user who didn't expect tokenmax-class context loads can rack up surprise
|
||||
spend. Confirm before continuing.
|
||||
|
||||
## Step 4: Import and Index
|
||||
|
||||
```bash
|
||||
@@ -159,6 +216,15 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
|
||||
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
|
||||
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
|
||||
**Do NOT silently move past the banner.** Present the matrix to the operator
|
||||
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
|
||||
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
|
||||
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
|
||||
same matrix and same default.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
|
||||
@@ -48,10 +48,32 @@ the full doc map, use `llms.txt` at the same URL root.
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link
|
||||
gbrain init # local brain, ready in 2 seconds
|
||||
# picks a search mode (conservative / balanced / tokenmax)
|
||||
gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
gbrain search modes # see the active search mode + per-knob attribution
|
||||
gbrain search stats # cache hit rate + intent mix after some real usage
|
||||
```
|
||||
|
||||
**v0.32.3 — named search modes.** `gbrain init` asks once which mode fits
|
||||
your workload. The cost spread depends on BOTH the mode AND your downstream
|
||||
model — 25x corner-to-corner. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume; multiply by 10 for heavy / multi-user fleets):
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| `conservative` (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| `balanced` (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| `tokenmax` (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Natural pairings (corner-diagonal) span ~4x at realistic single-user
|
||||
volume. Auto-suggests based on your configured `models.tier.subagent`.
|
||||
Non-TTY installs auto-pick `balanced` and print a hint pointing at
|
||||
`gbrain config set search.mode <m>`. After some real usage, run
|
||||
`gbrain search stats` for observability and `gbrain search tune` for
|
||||
data-driven recommendations. Methodology + eval results live at
|
||||
[docs/eval/SEARCH_MODE_METHODOLOGY.md](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Evaluation Metric Glossary
|
||||
|
||||
**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.
|
||||
|
||||
Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.
|
||||
|
||||
## Retrieval Metrics
|
||||
|
||||
### Precision at k (P@k)
|
||||
|
||||
**Key:** `precision@k`
|
||||
|
||||
**Plain English:** Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.
|
||||
|
||||
**Range:** 0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.
|
||||
|
||||
### Recall at k (R@k)
|
||||
|
||||
**Key:** `recall@k`
|
||||
|
||||
**Plain English:** Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.
|
||||
|
||||
**Range:** 0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.
|
||||
|
||||
### Mean Reciprocal Rank (MRR)
|
||||
|
||||
**Key:** `mrr`
|
||||
|
||||
**Plain English:** On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it's typically at rank 2.
|
||||
|
||||
**Range:** 0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.
|
||||
|
||||
### Normalized Discounted Cumulative Gain at k (nDCG@k)
|
||||
|
||||
**Key:** `ndcg@k`
|
||||
|
||||
**Plain English:** Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.
|
||||
|
||||
**Range:** 0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
## Set-Similarity / Stability Metrics
|
||||
|
||||
### Jaccard similarity at k (set Jaccard @k)
|
||||
|
||||
**Key:** `jaccard@k`
|
||||
|
||||
**Plain English:** How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.
|
||||
|
||||
### Top-1 stability rate
|
||||
|
||||
**Key:** `top1_stability`
|
||||
|
||||
**Plain English:** Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don't change the top answer don't hurt it.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.
|
||||
|
||||
## Statistical-Significance Metrics
|
||||
|
||||
### p-value (paired bootstrap)
|
||||
|
||||
**Key:** `p_value`
|
||||
|
||||
**Plain English:** How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).
|
||||
|
||||
**Range:** 0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.
|
||||
|
||||
### 95% Confidence Interval (CI)
|
||||
|
||||
**Key:** `confidence_interval`
|
||||
|
||||
**Plain English:** The range we're 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.
|
||||
|
||||
**Range:** Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn't statistically significant.
|
||||
|
||||
## Operational / Cost Metrics
|
||||
|
||||
### Cache hit rate
|
||||
|
||||
**Key:** `cache_hit_rate`
|
||||
|
||||
**Plain English:** Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.
|
||||
|
||||
**Range:** 0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.
|
||||
|
||||
### Average results returned
|
||||
|
||||
**Key:** `avg_results`
|
||||
|
||||
**Plain English:** Mean number of search-result rows the engine returned per call. Should be near the active mode's searchLimit unless the brain is small or the budget is dropping results.
|
||||
|
||||
**Range:** 0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.
|
||||
|
||||
### Average tokens delivered
|
||||
|
||||
**Key:** `avg_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.
|
||||
|
||||
**Range:** 0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.
|
||||
|
||||
### Cost per query (USD)
|
||||
|
||||
**Key:** `cost_per_query_usd`
|
||||
|
||||
**Plain English:** Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.
|
||||
|
||||
**Range:** 0..unbounded. Conservative mode is typically <\$0.001 per call; tokenmax with answer-gen can exceed \$0.01.
|
||||
|
||||
### p99 latency (ms)
|
||||
|
||||
**Key:** `p99_latency_ms`
|
||||
|
||||
**Plain English:** 99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.
|
||||
|
||||
**Range:** 0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command resolves through `getMetricGloss()` in `src/core/eval/metric-glossary.ts`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
# Search Mode Evaluation Methodology
|
||||
|
||||
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
|
||||
|
||||
## 1. What this measures and what it doesn't
|
||||
|
||||
**Measures:** retrieval quality and operational cost on fixed public datasets, under each named search mode, against the same brain content.
|
||||
|
||||
**Does NOT measure:**
|
||||
- Your specific brain content (this is a benchmark, not your bill).
|
||||
- Your specific query distribution.
|
||||
- End-user satisfaction or downstream task success.
|
||||
- Latency under concurrent load.
|
||||
- Production cost (the cost numbers are model-pricing estimates × dataset size, not your actual API spend).
|
||||
|
||||
If you want to know how a mode behaves on YOUR brain, run `gbrain search stats --days 30` after a real usage window, then run `gbrain search tune` for actionable recommendations.
|
||||
|
||||
## 2. Datasets and sizes
|
||||
|
||||
- **LongMemEval** — public split, `n=500` questions. Downloaded from [Hugging Face](https://huggingface.co/datasets/xiaowu0162/longmemeval). The corpus + answer keys are pinned to a specific commit; recorded in every per-run record.
|
||||
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
|
||||
- **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
|
||||
|
||||
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
|
||||
## 3. Sample selection
|
||||
|
||||
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
|
||||
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
|
||||
|
||||
## 4. Run procedure
|
||||
|
||||
The command is the doc. Anyone can reproduce.
|
||||
|
||||
```bash
|
||||
# Setup: in your gbrain working tree, with OPENAI_API_KEY + ANTHROPIC_API_KEY exported.
|
||||
git rev-parse HEAD # record the commit for the methodology footer
|
||||
|
||||
# Sweep all 3 modes × 2 retrieval-focused suites with seed 42.
|
||||
gbrain eval run-all \
|
||||
--modes conservative,balanced,tokenmax \
|
||||
--suites longmemeval,replay \
|
||||
--seed 42 \
|
||||
--limit 500 \
|
||||
--budget-usd-retrieval 5 \
|
||||
--budget-usd-answer 20 \
|
||||
--output docs/eval/results/v0.32.3/
|
||||
|
||||
# Render the comparison.
|
||||
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
|
||||
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
|
||||
```
|
||||
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
|
||||
|
||||
## 5. Threats to validity
|
||||
|
||||
Honest list. We name what would let a critic dismiss the numbers.
|
||||
|
||||
- **LongMemEval skews English + technical.** The questions are software-engineering and consumer-product flavored. Performance on a brain rich in non-English / non-technical content (writing, art history, etc.) may differ.
|
||||
- **BrainBench is small** (1240 docs) relative to a production brain (10K-100K pages). Absolute scores aren't predictive of your hit rate; the _delta_ between modes is.
|
||||
- **char/4 token heuristic.** Token-budget enforcement and cost estimates use a character-count / 4 heuristic. Accurate within ~5-10% for English with the OpenAI tiktoken family; off worse for Voyage (we don't use Voyage in chat retrieval, so it doesn't bias the reported numbers, but if you do, your budget caps will be approximate).
|
||||
- **Expansion's quality lift varies by query distribution.** The eval data shows ~97.6% relative quality with LLM expansion vs without (i.e., barely measurable lift) on the LongMemEval corpus. On rarer-entity / longer-tail queries, the lift can be larger. We report the corpus we measured; YMMV.
|
||||
- **Paired bootstrap assumes question-level independence.** Multi-hop questions within the same conversation thread aren't independent; the bootstrap CI is slightly tighter than reality.
|
||||
- **Single brain instance per benchmark.** The benchmark spins up an in-memory PGLite per question. Cache hit rate measured here doesn't reflect a long-running production brain's cache state.
|
||||
|
||||
## 6. Per-question raw outputs
|
||||
|
||||
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
|
||||
|
||||
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
|
||||
|
||||
- **Wins:** the 3 questions where this mode's score exceeded the next-best mode by the largest margin.
|
||||
- **Losses:** the 3 questions where this mode's score fell short of the next-best mode by the largest margin.
|
||||
|
||||
Picked by the score delta, NOT cherry-picked by hand. The README documents the rule so a critic can verify.
|
||||
|
||||
## 7. Pre-registered expectations
|
||||
|
||||
Before running, we expect:
|
||||
|
||||
1. **tokenmax wins Recall@10** by 5-15 percentage points over conservative. LLM expansion + 50-result ceiling helps rare-entity surface forms.
|
||||
2. **conservative wins cost-per-query** by 5-15× over tokenmax. No Haiku expansion + tight 4K budget cap = single-digit-cent queries.
|
||||
3. **balanced lands within 3pp of tokenmax** on Recall@10. Intent weighting (zero-LLM cost) closes most of the expansion gap on common queries.
|
||||
4. **No mode breaks nDCG@10 ≥ 0.65** — the published "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
Then we publish whether the data agrees. **If a hypothesis fails, that's documented honestly** in the release README, not buried. Pre-registration is what makes the comparison defensible — without it, a "we expected X and got X" outcome is observation, not prediction.
|
||||
|
||||
## 8. Re-run cadence
|
||||
|
||||
This document + the eval results are regenerated on every release that touches retrieval-affecting code. The `gbrain doctor eval_drift` check surfaces changes to the curated watch-list in `src/core/eval/drift-watch.ts`:
|
||||
|
||||
- `src/core/search/**`
|
||||
- `src/core/embedding.ts`
|
||||
- `src/core/chunkers/**`
|
||||
- `src/core/ai/recipes/anthropic.ts`
|
||||
- `src/core/ai/recipes/openai.ts`
|
||||
- `src/core/operations.ts`
|
||||
|
||||
Additions to the watch-list require a CHANGELOG line.
|
||||
|
||||
## Statistical-significance discipline
|
||||
|
||||
When `gbrain eval compare --md` reports a Δ between two modes, it computes:
|
||||
|
||||
- **Paired bootstrap** with 10,000 resamples per metric. Each resample draws _question-level_ pairs (same question, mode A vs mode B), so question-level variance is differenced out.
|
||||
- **Bonferroni correction** across the 12 comparisons (3 modes × 4 metrics). The reported p-value is the comparison's raw p-value × 12 (clamped at 1.0).
|
||||
- **95% confidence intervals** computed from the bootstrap distribution.
|
||||
|
||||
If the CI for a Δ includes 0 OR the Bonferroni-adjusted p-value exceeds 0.05, the difference is **not** statistically significant. The MD report says "not significant" verbatim.
|
||||
|
||||
## Glossary
|
||||
|
||||
Every metric the report prints has a plain-English entry in `docs/eval/METRIC_GLOSSARY.md`, auto-generated from `src/core/eval/metric-glossary.ts`. The CI guard at `scripts/check-eval-glossary-fresh.sh` regenerates and diffs against the committed file on every test run; a stale doc fails the build.
|
||||
|
||||
## Cost anchors
|
||||
|
||||
The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table both surface these rough cost anchors. Working through the math so they're auditable:
|
||||
|
||||
**Variables:**
|
||||
- `T` = avg tokens per search-result chunk. The recursive chunker targets 300 words / chunk → ~400 tokens (English, OpenAI tiktoken approx).
|
||||
- `N` = chunks delivered per query (capped by the mode's `searchLimit`).
|
||||
- `R` = downstream model input rate. Sonnet 4.6 = \$3/M. Opus 4.7 = \$5/M. Haiku 4.5 = \$1/M.
|
||||
- `Q` = queries per month.
|
||||
|
||||
**Per-query input cost** (downstream agent reads the chunks):
|
||||
|
||||
cost_per_query = T × N × R
|
||||
|
||||
| Mode | T (tokens) | N (chunks) | Sonnet (\$3/M) | Opus (\$5/M) | Haiku (\$1/M) |
|
||||
|---|---|---|---|---|---|
|
||||
| conservative (4K cap, 10 max) | ~400 | 10 (or fewer if budget hits) | \$0.012 | \$0.020 | \$0.004 |
|
||||
| balanced (12K cap, 25 max) | ~400 | ~25 | \$0.030 | \$0.050 | \$0.010 |
|
||||
| tokenmax (no cap, 50 max) | ~400 | ~50 | \$0.060 | \$0.100 | \$0.020 |
|
||||
|
||||
**Monthly cost** (Q × per-query):
|
||||
|
||||
| Mode @ Sonnet | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$12 | \$120 | \$1,200 |
|
||||
| balanced | \$30 | \$300 | \$3,000 |
|
||||
| tokenmax | \$60 | \$600 | \$6,000 |
|
||||
|
||||
| Mode @ Opus | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$20 | \$200 | \$2,000 |
|
||||
| balanced | \$50 | \$500 | \$5,000 |
|
||||
| tokenmax | \$100 | \$1,000 | \$10,000 |
|
||||
|
||||
**gbrain's own cost** on top:
|
||||
- Query embedding (text-embedding-3-large @ \$0.13/M tokens): ~\$0.00001 per query. Negligible at every scale.
|
||||
- Tokenmax Haiku expansion call (\$1/M input, \$5/M output, ~500 input + 200 output per call): ~\$0.0015 per query, or \$150/mo at 100K queries. Cache hits cut this in half.
|
||||
- Per-page indexing (one-time): bounded by your import volume, not query volume. Not modeled here.
|
||||
|
||||
**Cache hit adjustment.** A warmed brain typically sees 30-50% cache hits on repeat-query traffic. Cache hits skip the downstream input cost entirely (the cached result was already in the agent's context once). So real-world costs run ~50-70% of the table above on a busy brain.
|
||||
|
||||
**Why these numbers DRIFT from your actual bill:**
|
||||
- Your agent's system prompt + reasoning tokens add input that gbrain doesn't see.
|
||||
- Compaction reduces input over a long session.
|
||||
- Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
|
||||
- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot.
|
||||
|
||||
The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes.
|
||||
|
||||
## Mode × Model matrix (the 25x spread)
|
||||
|
||||
The per-query math above assumes Sonnet 4.6 downstream. In reality, the
|
||||
downstream model tier is the BIGGER cost lever. Per-query cost at 10K
|
||||
queries/month (typical single-user volume), search payload only (no cache
|
||||
savings):
|
||||
|
||||
| Mode (search tokens) | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage).
|
||||
|
||||
**Natural pairings span ~4x** (cheap model + tight mode → frontier model + loose
|
||||
mode). **Mismatches waste capacity:**
|
||||
|
||||
- `tokenmax + Haiku`: Haiku gets 20K of search results stuffed into its
|
||||
context per query. Haiku's reasoning is weaker; more chunks = more noise,
|
||||
not more signal. You pay Haiku rates but get sub-Haiku quality. Wrong
|
||||
direction.
|
||||
- `conservative + Opus`: Opus has 200K context window and can synthesize
|
||||
across many chunks. Capping at 10 chunks / 4K tokens leaves Opus
|
||||
reasoning underfed. You pay Opus rates but get conservative-shape
|
||||
retrieval. Wasted spend.
|
||||
|
||||
**Right-sizing rule:** match the mode's `searchLimit` to the downstream
|
||||
model's "useful context depth":
|
||||
|
||||
- Haiku struggles past ~5-10 chunks of cross-referenced content → conservative
|
||||
- Sonnet handles ~25-40 chunks well → balanced
|
||||
- Opus benefits from 50+ chunks for multi-hop reasoning → tokenmax
|
||||
|
||||
## Realistic-scale anchor (single power-user agent loop)
|
||||
|
||||
The per-query math above is honest but theoretical: it treats each search as an isolated billable event. Real agent loops amortize a lot of context across turns via Anthropic prompt caching. Here's what one heavy power-user loop actually looks like in production, anonymized + scaled so the numbers represent a representative power user rather than any specific deployment.
|
||||
|
||||
**Reference shape — tokenmax in production at a single-user scale:**
|
||||
|
||||
| Quantity | Approximate value |
|
||||
|---|---|
|
||||
| 30-day total agent spend | ~\$700/mo |
|
||||
| 30-day total tokens billed | ~800M |
|
||||
| Turns per month | ~860 (~29/day; one active agent loop) |
|
||||
| Average tokens per turn | ~900K |
|
||||
| Average cost per turn | ~\$0.85 |
|
||||
| Anthropic prompt-cache hit rate | ~88% |
|
||||
|
||||
A "turn" here is one agent loop iteration: read user message, plan, execute tool calls (including gbrain searches), generate response. Each turn typically includes 2-4 gbrain searches.
|
||||
|
||||
**Per-mode scaling from the tokenmax anchor:**
|
||||
|
||||
The cost difference between modes is concentrated in the search-attributable fraction of per-turn cost. System prompt, tool definitions, conversation history, and reasoning tokens don't change with mode — only the chunks gbrain delivers do. Assume 3 searches per turn at the mode's `searchLimit`:
|
||||
|
||||
| Mode | Search tokens/turn | Search cost/turn (at \$3/M effective) | Search-attributable @ 860 turns | Δ vs tokenmax |
|
||||
|---|---|---|---|---|
|
||||
| tokenmax | ~60K (3 × 20K) | ~\$0.18 | ~\$155/mo | — |
|
||||
| balanced | ~30K (3 × 10K) | ~\$0.09 | ~\$77/mo | -\$78 |
|
||||
| conservative | ~12K (3 × 4K) | ~\$0.036 | ~\$31/mo | -\$124 |
|
||||
|
||||
**Implied total agent spend by NATURAL PAIRING** (mode + matched
|
||||
downstream model). Per-turn cost scales with the downstream model's
|
||||
per-token rate, since the cached prefix + uncached portion + reasoning
|
||||
tokens all bill at that rate:
|
||||
|
||||
| Pairing | Per-turn cost | Total @ 860 turns/mo |
|
||||
|---|---|---|
|
||||
| tokenmax + Opus (frontier, max quality) | ~\$0.85 | ~\$700/mo |
|
||||
| balanced + Sonnet (the sweet spot) | ~\$0.50 | ~\$430/mo |
|
||||
| conservative + Haiku (cost-sensitive) | ~\$0.20 | ~\$170/mo |
|
||||
|
||||
**4x spread across natural pairings.** The model tier dominates because
|
||||
the per-token rate applies to the WHOLE per-turn payload (system + tools
|
||||
+ history + reasoning + search), not just gbrain's chunks. Mode choice
|
||||
contributes ~10-20% on top of that base.
|
||||
|
||||
**Mismatched pairings push you off the curve:**
|
||||
|
||||
| Pairing | Per-turn estimate | Total @ 860 turns/mo | Compared to natural |
|
||||
|---|---|---|---|
|
||||
| tokenmax + Haiku | ~\$0.20 | ~\$170/mo | Same cost as conservative+Haiku, worse quality |
|
||||
| conservative + Opus | ~\$0.75 | ~\$640/mo | 92% of tokenmax+Opus spend, conservative-shape retrieval |
|
||||
|
||||
The mismatch math says: a tokenmax+Haiku user pays the same as
|
||||
conservative+Haiku but gets a noisier context (Haiku can't filter signal
|
||||
from 50 chunks). A conservative+Opus user pays nearly the same as
|
||||
tokenmax+Opus but starves Opus on retrieval depth. Both burn budget for
|
||||
no improvement.
|
||||
|
||||
**What this anchor tells us that the per-query math doesn't:**
|
||||
|
||||
1. **At realistic agent-loop scale with disciplined prompt caching, mode choice saves 10-20% of total agent spend** — meaningful, but smaller than the per-query 5x ratio implies. Disciplined prompt-cache layouts blunt the mode delta because most of the per-turn cost is the cached prefix, not the search payload.
|
||||
|
||||
2. **Without that prompt-cache discipline, the per-query framing reasserts itself.** Setups that churn the prompt prefix on every turn (frequent system-prompt edits, untemplated tool defs, no prompt-cache structuring) see search payload contribute a much larger fraction of total cost. Those setups should care about mode choice more, not less.
|
||||
|
||||
3. **The cache hit rate quoted here (~88%) is achievable but not automatic.** It requires structuring the prompt so the cached prefix stays stable across turns: system prompt + tool defs first, history compacted but cache-aware, retrieved chunks appended LAST (where their volatility doesn't invalidate the prefix). Agents that interleave search results inside the cached region pay the prefix-rebuild tax on every turn.
|
||||
|
||||
**Caveats stacked here:**
|
||||
|
||||
- The anchor represents ONE power-user loop. Multi-user fleets aggregate proportionally; the per-user shape doesn't change.
|
||||
- The "3 searches per turn" assumption varies wildly. A code-review agent might issue 10+ searches per turn; a chat-only loop might do 0.
|
||||
- The 88% cache hit rate is the high end of what's achievable. Half that is closer to a default agent without cache-aware prompt layout.
|
||||
- The "Δ vs tokenmax" math assumes the OTHER cost components (system, tools, history, reasoning) stay constant. In practice, conservative's smaller per-turn payload also leaves more room in the context window for history → which can change agent behavior in either direction.
|
||||
|
||||
This anchor + the per-query math both live in this doc on purpose. The per-query framing is what an isolated benchmark would measure (and what `gbrain eval run-all` will produce). The realistic-scale anchor is what an operator actually pays. Both are honest; neither is the whole truth.
|
||||
|
||||
## Reproducibility footer
|
||||
|
||||
Every release that publishes eval numbers includes a footer with:
|
||||
|
||||
- Code commit SHA
|
||||
- Dataset SHA (LongMemEval, BrainBench, Replay)
|
||||
- `--seed N`
|
||||
- Run commands verbatim
|
||||
- API model identifiers used (Anthropic + OpenAI + judge model)
|
||||
|
||||
Without these, the numbers are unfalsifiable. With them, anyone with API keys can re-score.
|
||||
+193
-1
@@ -23,7 +23,15 @@ start here.
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
4. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
|
||||
default but printed a 9-cell cost matrix (mode × downstream model)
|
||||
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
|
||||
and confirm their choice before continuing. Cost spread between corners
|
||||
is 25x — silent acceptance is the wrong default. See
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
5. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
@@ -815,6 +823,100 @@ permission to run them — see the "run without asking" rule above.
|
||||
Never leave `gbrain-test-pg` running. If you find a stale one from a previous run,
|
||||
stop and remove it before starting a new one.
|
||||
|
||||
## Search Mode (v0.32.3)
|
||||
|
||||
GBrain ships three named search modes that bundle the search-lite knobs from
|
||||
PR #897 into a single config key. Pick one at install time; the rest of the
|
||||
project resolves through `src/core/search/mode.ts`.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
|
||||
The corner-to-corner spread is 25x once you pair mode with downstream model.
|
||||
Chunks ~400 tokens avg. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume), full search payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage). Natural pairings span ~4x.
|
||||
Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
— too-big payload overwhelms a cheap model; too-small payload starves an
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
Anthropic prompt caching see 50-80% discount on top (cache hits skip
|
||||
downstream entirely). The realistic-scale anchor in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` walks the natural pairings at
|
||||
single-power-user volume (~860 turns/mo): tokenmax+Opus ~\$700/mo,
|
||||
balanced+Sonnet ~\$430/mo, conservative+Haiku ~\$170/mo. Setups WITHOUT
|
||||
cache-aware prompt layout (frequent prefix churn) see the per-query
|
||||
matrix dominate — mode + model choice matters more there.
|
||||
|
||||
**Resolution chain** (matches the v0.31.12 model-tier pattern at
|
||||
`src/core/model-config.ts:resolveModel`):
|
||||
|
||||
per-call SearchOpts → per-key config (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in **bare `hybridSearch`** (NOT just the cached wrapper)
|
||||
per `[CDX-5+6]` in `~/.claude/plans/lets-take-a-look-validated-parrot.md` — so
|
||||
`gbrain eval replay` and `gbrain eval longmemeval` test the same mode-affected
|
||||
behavior as the production `query` op.
|
||||
|
||||
**Cache-key contamination hotfix `[CDX-4]`:** migration v56 added a
|
||||
`knobs_hash` column to `query_cache`. The lookup filter is now
|
||||
`WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $` so a
|
||||
tokenmax write (expansion=on, limit=50) can't be served to a conservative
|
||||
read.
|
||||
|
||||
**Three CLI surfaces:**
|
||||
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
|
||||
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
|
||||
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
|
||||
|
||||
## Eval discipline (v0.32.3)
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
|
||||
resolves through `src/core/eval/metric-glossary.ts` so industry terms
|
||||
(`P@k`, `nDCG@k`, `MRR`, `Jaccard@k`) carry a plain-English line in human
|
||||
output and a `_meta.metric_glossary` block in JSON output (one block per
|
||||
response per `[CDX-25]`, NOT sibling `_gloss` fields).
|
||||
|
||||
The full methodology — datasets, sample selection, pre-registered
|
||||
expectations, threats to validity, paired-bootstrap + Bonferroni p-value
|
||||
discipline `[CDX-14]` — lives in `docs/eval/SEARCH_MODE_METHODOLOGY.md`.
|
||||
Auto-regenerated `docs/eval/METRIC_GLOSSARY.md` is CI-guarded against
|
||||
drift (`scripts/check-eval-glossary-fresh.sh`).
|
||||
|
||||
Per-run records land at `<repo>/.gbrain-evals/eval-results.jsonl` per
|
||||
`[CDX-23]`. The user's personal `~/.gbrain` brain is NEVER touched —
|
||||
audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
@@ -1623,6 +1725,63 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
|
||||
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
|
||||
NOT inside ~/gbrain.
|
||||
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M) ($3/M) ($5/M)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
|
||||
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
|
||||
```
|
||||
|
||||
**Ask the operator (paraphrase if needed):**
|
||||
|
||||
> Your gbrain just installed with search mode `<auto-applied default>`. This is
|
||||
> a one-time setup decision that controls retrieval payload size. Which mode
|
||||
> do you want?
|
||||
>
|
||||
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
|
||||
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
|
||||
>
|
||||
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
|
||||
>
|
||||
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
|
||||
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
|
||||
>
|
||||
> Cost depends on BOTH the mode AND the downstream model you run. See the
|
||||
> matrix above for the 9-cell breakdown.
|
||||
|
||||
If the operator picks a non-default mode, run:
|
||||
```bash
|
||||
gbrain config set search.mode <mode>
|
||||
```
|
||||
|
||||
If they pick tokenmax AND want to preserve the literal v0.31.x default
|
||||
(limit=20 instead of tokenmax's 50), also run:
|
||||
```bash
|
||||
gbrain config set search.searchLimit 20
|
||||
```
|
||||
|
||||
Verify the choice with `gbrain search modes` before continuing.
|
||||
|
||||
**Why this matters:** the cost spread between corners of the matrix is 25x.
|
||||
An agent that silently accepts the default and starts running queries against
|
||||
a user who didn't expect tokenmax-class context loads can rack up surprise
|
||||
spend. Confirm before continuing.
|
||||
|
||||
## Step 4: Import and Index
|
||||
|
||||
```bash
|
||||
@@ -1721,6 +1880,15 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
|
||||
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
|
||||
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
|
||||
**Do NOT silently move past the banner.** Present the matrix to the operator
|
||||
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
|
||||
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
|
||||
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
|
||||
same matrix and same default.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
@@ -1762,6 +1930,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
| "what search mode", "is my cache hot", "tune my retrieval", "compare search modes", "clear search overrides" | `gbrain search modes/stats/tune` directly. See `skills/conventions/search-modes.md` |
|
||||
| "eval results", "search benchmark", "haters-immune methodology", "regression check on retrieval" | `gbrain eval run-all` / `gbrain eval compare`. See `docs/eval/SEARCH_MODE_METHODOLOGY.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
@@ -1919,10 +2089,32 @@ the full doc map, use `llms.txt` at the same URL root.
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link
|
||||
gbrain init # local brain, ready in 2 seconds
|
||||
# picks a search mode (conservative / balanced / tokenmax)
|
||||
gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
gbrain search modes # see the active search mode + per-knob attribution
|
||||
gbrain search stats # cache hit rate + intent mix after some real usage
|
||||
```
|
||||
|
||||
**v0.32.3 — named search modes.** `gbrain init` asks once which mode fits
|
||||
your workload. The cost spread depends on BOTH the mode AND your downstream
|
||||
model — 25x corner-to-corner. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume; multiply by 10 for heavy / multi-user fleets):
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| `conservative` (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| `balanced` (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| `tokenmax` (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Natural pairings (corner-diagonal) span ~4x at realistic single-user
|
||||
volume. Auto-suggests based on your configured `models.tier.subagent`.
|
||||
Non-TTY installs auto-pick `balanced` and print a hint pointing at
|
||||
`gbrain config set search.mode <m>`. After some real usage, run
|
||||
`gbrain search stats` for observability and `gbrain search tune` for
|
||||
data-driven recommendations. Methodology + eval results live at
|
||||
[docs/eval/SEARCH_MODE_METHODOLOGY.md](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
|
||||
+2
-1
@@ -36,7 +36,7 @@
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run typecheck",
|
||||
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
@@ -54,6 +54,7 @@
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.32.3 — CI guard for docs/eval/METRIC_GLOSSARY.md freshness.
|
||||
#
|
||||
# Mirrors the scripts/check-jsonb-pattern.sh / check-progress-to-stdout.sh
|
||||
# discipline: regenerate the doc into a tmp file, diff against the committed
|
||||
# version, fail the build if they drift.
|
||||
#
|
||||
# Run: bash scripts/check-eval-glossary-fresh.sh
|
||||
# CI wires this through `bun run test` so PRs that bump the glossary module
|
||||
# without regenerating the doc are caught before review.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
COMMITTED="$REPO_ROOT/docs/eval/METRIC_GLOSSARY.md"
|
||||
TMP="$(mktemp)"
|
||||
trap 'rm -f "$TMP"' EXIT
|
||||
|
||||
if [ ! -f "$COMMITTED" ]; then
|
||||
echo "ERROR: $COMMITTED not found." >&2
|
||||
echo "Run: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Regenerate into TMP without touching the committed file. We can't easily
|
||||
# point the generator at a different path; trick it by redirecting cwd to
|
||||
# a sandbox and post-comparing.
|
||||
cd "$REPO_ROOT"
|
||||
# Render directly via bun + a one-liner that exposes the module function.
|
||||
bun -e "import { renderMetricGlossaryMarkdown } from './src/core/eval/metric-glossary.ts'; process.stdout.write(renderMetricGlossaryMarkdown());" > "$TMP"
|
||||
|
||||
if ! diff -q "$COMMITTED" "$TMP" >/dev/null 2>&1; then
|
||||
echo "ERROR: docs/eval/METRIC_GLOSSARY.md is stale." >&2
|
||||
echo "" >&2
|
||||
echo "Diff between committed and freshly-generated:" >&2
|
||||
echo "" >&2
|
||||
diff -u "$COMMITTED" "$TMP" >&2 || true
|
||||
echo "" >&2
|
||||
echo "To regenerate: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ docs/eval/METRIC_GLOSSARY.md is fresh"
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* v0.32.3 — auto-generate docs/eval/METRIC_GLOSSARY.md from
|
||||
* src/core/eval/metric-glossary.ts.
|
||||
*
|
||||
* Run: bun run scripts/generate-metric-glossary.ts
|
||||
*
|
||||
* CI guard `scripts/check-eval-glossary-fresh.sh` regenerates and diffs
|
||||
* against the committed version — out-of-date doc fails the build.
|
||||
*/
|
||||
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { dirname, join, resolve } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { renderMetricGlossaryMarkdown } from '../src/core/eval/metric-glossary.ts';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(__dirname, '..');
|
||||
const OUT_PATH = join(REPO_ROOT, 'docs', 'eval', 'METRIC_GLOSSARY.md');
|
||||
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
|
||||
mkdirSync(dirname(OUT_PATH), { recursive: true });
|
||||
writeFileSync(OUT_PATH, md, 'utf-8');
|
||||
|
||||
console.log(`Wrote ${OUT_PATH} (${md.length} bytes, ${md.split('\n').length} lines).`);
|
||||
@@ -22,6 +22,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
| "what search mode", "is my cache hot", "tune my retrieval", "compare search modes", "clear search overrides" | `gbrain search modes/stats/tune` directly. See `skills/conventions/search-modes.md` |
|
||||
| "eval results", "search benchmark", "haters-immune methodology", "regression check on retrieval" | `gbrain eval run-all` / `gbrain eval compare`. See `docs/eval/SEARCH_MODE_METHODOLOGY.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
name: search-modes
|
||||
description: Three named search modes (conservative / balanced / tokenmax). Pick one at install; everything else inherits.
|
||||
type: convention
|
||||
---
|
||||
|
||||
# Convention: Search Modes (v0.32.3)
|
||||
|
||||
> **Convention:** every brain has one active search mode. The mode bundles the
|
||||
> search-lite knobs from PR #897 (semantic cache, token budget, intent
|
||||
> weighting, LLM expansion, result limit) into a single config key:
|
||||
> `search.mode = conservative | balanced | tokenmax`.
|
||||
|
||||
## When this fires
|
||||
|
||||
Any agent doing search-adjacent work in a gbrain brain consults this convention:
|
||||
|
||||
- `brain-ops` / `query` / `signal-detector` skills: respect the active mode at
|
||||
search time. Per-call `SearchOpts` overrides win when set; mode is the default.
|
||||
- Skills that recommend tuning ("the cache hit rate is high — raise threshold?"):
|
||||
route operators to `gbrain search tune` rather than rolling their own logic.
|
||||
- New skills that add per-call retrieval overrides: name them explicitly so
|
||||
the resolved-knob attribution dashboard (`gbrain search modes`) reads cleanly.
|
||||
|
||||
## Mode bundle (read-only constants)
|
||||
|
||||
The 3 bundles live in `src/core/search/mode.ts` as `MODE_BUNDLES` (frozen).
|
||||
Don't redefine them per-install; that breaks the public methodology numbers.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cache, intent weighting, and similarity threshold are constant across modes**
|
||||
— they're free wins (no API cost). Modes scale the three cost levers:
|
||||
`tokenBudget`, `expansion`, `searchLimit`.
|
||||
|
||||
## Resolution chain (matches v0.31.12 model-tier shape)
|
||||
|
||||
per-call SearchOpts.tokenBudget / expansion / etc.
|
||||
↓ (when undefined)
|
||||
per-key config: search.cache.enabled, search.tokenBudget, …
|
||||
↓ (when unset)
|
||||
MODE_BUNDLES[search.mode]
|
||||
↓ (when search.mode is unset)
|
||||
MODE_BUNDLES.balanced (safety fallback)
|
||||
|
||||
## Tools for agents
|
||||
|
||||
Agents tuning a brain's retrieval should call these directly:
|
||||
|
||||
gbrain search modes # dashboard + per-knob source attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode is canonical)
|
||||
gbrain search stats [--days N] # hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
`gbrain search tune` reads the `search_telemetry` rollup (sums + counts of
|
||||
last 7 days) + brain size + configured `models.tier.subagent` to suggest
|
||||
mode + per-key changes. With `--apply`, it mutates config via `setConfig`
|
||||
and prints a paste-ready revert command.
|
||||
|
||||
## Cache contamination guard
|
||||
|
||||
Migration v56 added `query_cache.knobs_hash`. A tokenmax write
|
||||
(expansion=on, limit=50) is keyed by a different hash than a conservative
|
||||
read (no expansion, limit=10), so cross-mode contamination is structurally
|
||||
impossible. The cache lookup filter is:
|
||||
|
||||
WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $
|
||||
|
||||
Legacy NULL-knobs_hash rows from pre-v0.32.3 are silently excluded
|
||||
(treated as misses, re-populated with the right hash on first hit).
|
||||
|
||||
## Trigger phrases
|
||||
|
||||
If an operator or agent asks any of these, route to `gbrain search …`:
|
||||
|
||||
- "what search mode is active?" → `gbrain search modes`
|
||||
- "is my cache hot?" → `gbrain search stats`
|
||||
- "tune my retrieval" → `gbrain search tune`
|
||||
- "clear search overrides" → `gbrain search modes --reset`
|
||||
- "compare modes" → `gbrain eval compare`
|
||||
|
||||
## Don't
|
||||
|
||||
- Don't redefine `MODE_BUNDLES` per-install. The methodology numbers in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` cite these as canonical.
|
||||
- Don't mutate `search.mode` config from inside a subagent loop without
|
||||
operator approval. Mutation is a trust-boundary crossing
|
||||
(`tune --apply` stays CLI-only in v0.32.3 per `[CDX-21]`).
|
||||
- Don't add per-call `tokenBudget` overrides on the production `query` op
|
||||
without naming them in `gbrain search modes` output.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/eval/SEARCH_MODE_METHODOLOGY.md` — full eval methodology
|
||||
- `docs/eval/METRIC_GLOSSARY.md` — plain-English definitions
|
||||
- `src/core/search/mode.ts` — module source
|
||||
+16
-1
@@ -27,7 +27,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'cache']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -38,6 +38,7 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
'integrations', 'friction',
|
||||
'frontmatter', 'check-resolvable',
|
||||
'models',
|
||||
'cache',
|
||||
]);
|
||||
|
||||
async function main() {
|
||||
@@ -807,6 +808,14 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runMounts(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'cache') {
|
||||
// v0.32.x search-lite: semantic query cache management. Dispatch the
|
||||
// subcommand handler (stats / clear / prune); the handler opens its
|
||||
// own engine connection.
|
||||
const { runCache } = await import('./commands/cache.ts');
|
||||
await runCache(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'routing-eval') {
|
||||
const { runRoutingEvalCli } = await import('./commands/routing-eval.ts');
|
||||
await runRoutingEvalCli(args);
|
||||
@@ -1119,6 +1128,12 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runModels(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'search': {
|
||||
// v0.32.3 search-lite — `gbrain search modes/stats/tune`.
|
||||
const { runSearch } = await import('./commands/search.ts');
|
||||
await runSearch(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'takes': {
|
||||
const { runTakes } = await import('./commands/takes.ts');
|
||||
await runTakes(engine, args);
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
/**
|
||||
* `gbrain cache` \u2014 manage the semantic query cache (v0.32.x search-lite).
|
||||
*
|
||||
* Subcommands:
|
||||
* gbrain cache stats \u2014 print row/hit counts and freshness breakdown.
|
||||
* gbrain cache clear \u2014 wipe all cache rows.
|
||||
* gbrain cache prune \u2014 delete only stale (past-TTL) rows.
|
||||
*
|
||||
* Read-only by default. `clear` requires `--yes` to avoid accidents.
|
||||
*
|
||||
* No DB writes outside the cache table.
|
||||
*/
|
||||
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { SemanticQueryCache, loadCacheConfig } from '../core/search/query-cache.ts';
|
||||
|
||||
function printHelp(): void {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`
|
||||
gbrain cache \u2014 manage the semantic query cache (v0.32.x search-lite)
|
||||
|
||||
Usage:
|
||||
gbrain cache stats Print cache row counts, hit counts, freshness.
|
||||
gbrain cache clear Wipe ALL cache rows. Requires --yes.
|
||||
gbrain cache prune Delete only stale (past-TTL) rows.
|
||||
|
||||
Flags:
|
||||
--yes Bypass clear confirmation prompt.
|
||||
--source <id> Scope clear to a single source_id.
|
||||
--help Show this help.
|
||||
`);
|
||||
}
|
||||
|
||||
export async function runCache(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('gbrain cache: no brain configured. Run `gbrain init` first.');
|
||||
process.exit(1);
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
|
||||
try {
|
||||
const cacheCfg = await loadCacheConfig(engine);
|
||||
const cache = new SemanticQueryCache(engine, cacheCfg);
|
||||
|
||||
if (sub === 'stats') {
|
||||
const stats = await cache.stats();
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('Semantic Query Cache stats');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('--------------------------');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`enabled : ${cacheCfg.enabled ?? true}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`similarity_threshold : ${cacheCfg.similarityThreshold ?? 0.92}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`ttl_seconds : ${cacheCfg.ttlSeconds ?? 3600}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`total rows : ${stats.total_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(` fresh : ${stats.fresh_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(` stale : ${stats.stale_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`total hits : ${stats.total_hits}`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'clear') {
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
|
||||
if (!yes) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('gbrain cache clear: refusing to wipe without --yes flag.');
|
||||
process.exit(1);
|
||||
}
|
||||
const n = await cache.clear(sourceId ? { sourceId } : {});
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`Cleared ${n} cache row(s)${sourceId ? ` (source=${sourceId})` : ''}.`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'prune') {
|
||||
const n = await cache.prune();
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`Pruned ${n} stale cache row(s).`);
|
||||
return;
|
||||
}
|
||||
|
||||
// eslint-disable-next-line no-console
|
||||
console.error(`gbrain cache: unknown subcommand "${sub}". See \`gbrain cache --help\`.`);
|
||||
process.exit(1);
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
@@ -131,16 +131,43 @@ export async function runClawTest(args: string[]): Promise<number> {
|
||||
// Scripted mode
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* v0.32.x: env vars that, when inherited from the parent process, would
|
||||
* break the claw-test scripted harness's "hermetic PGLite tempdir"
|
||||
* contract. The harness runs `gbrain init --pglite` then a sequence of
|
||||
* subsequent phases that ALL must hit the same tempdir brain. If the
|
||||
* parent process (e.g. a CI runner with DATABASE_URL set for OTHER e2e
|
||||
* tests) leaks DATABASE_URL into the children, loadConfig sees the env
|
||||
* var and silently flips inferredEngine to 'postgres' (config.ts:143-145),
|
||||
* forcing every phase to hit the parent's test Postgres instead of the
|
||||
* hermetic PGLite. Result: phases race against each other (multi-tenant
|
||||
* effects on the shared DB) and the harness hangs.
|
||||
*
|
||||
* Strip these on entry so the child env carries no Postgres-pointing
|
||||
* variable. PGLite-only by design.
|
||||
*/
|
||||
const POSTGRES_POLLUTION_ENV_VARS = ['DATABASE_URL', 'GBRAIN_DATABASE_URL'];
|
||||
|
||||
async function runScripted(
|
||||
opts: HarnessOpts,
|
||||
scenario: ScenarioConfig,
|
||||
ctx: { runId: string; runRoot: string; gbrainHome: string },
|
||||
): Promise<number> {
|
||||
const childEnv: Record<string, string> = {
|
||||
...process.env as Record<string, string>,
|
||||
GBRAIN_HOME: ctx.gbrainHome,
|
||||
GBRAIN_FRICTION_RUN_ID: ctx.runId,
|
||||
};
|
||||
// Filter out Postgres-pointing env vars before forwarding to children.
|
||||
// The harness is PGLite-only by design; an inherited DATABASE_URL
|
||||
// would force loadConfig() to flip the engine to 'postgres' at the
|
||||
// next phase boundary and break the hermetic-tempdir contract.
|
||||
const parentEnv = process.env as Record<string, string | undefined>;
|
||||
const childEnv: Record<string, string> = { GBRAIN_HOME: ctx.gbrainHome, GBRAIN_FRICTION_RUN_ID: ctx.runId };
|
||||
for (const [k, v] of Object.entries(parentEnv)) {
|
||||
if (v === undefined) continue;
|
||||
if (POSTGRES_POLLUTION_ENV_VARS.includes(k)) continue;
|
||||
childEnv[k] = v;
|
||||
}
|
||||
// Re-apply the explicit overrides so a parent GBRAIN_HOME / GBRAIN_FRICTION_RUN_ID
|
||||
// can't accidentally win the merge.
|
||||
childEnv.GBRAIN_HOME = ctx.gbrainHome;
|
||||
childEnv.GBRAIN_FRICTION_RUN_ID = ctx.runId;
|
||||
|
||||
const phases: { name: string; argv: string[] }[] = [];
|
||||
// Phase 2: install_brain
|
||||
|
||||
+47
-3
@@ -11,8 +11,6 @@ function redactUrl(url: string): string {
|
||||
|
||||
export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
const action = args[0];
|
||||
const key = args[1];
|
||||
const value = args[2];
|
||||
|
||||
if (action === 'show') {
|
||||
const config = loadConfig();
|
||||
@@ -32,6 +30,51 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.32.3 [CDX-7+8]: `unset` is required before `gbrain search modes
|
||||
// --reset` can implement its contract. Two shapes:
|
||||
// gbrain config unset <key> — single-key delete
|
||||
// gbrain config unset --pattern <pfx> — prefix-bulk delete
|
||||
if (action === 'unset') {
|
||||
const flagIdx = args.indexOf('--pattern');
|
||||
if (flagIdx !== -1) {
|
||||
const prefix = args[flagIdx + 1];
|
||||
if (!prefix || prefix.length === 0) {
|
||||
console.error('Usage: gbrain config unset --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
const keys = await engine.listConfigKeys(prefix);
|
||||
if (keys.length === 0) {
|
||||
console.log(`No keys match prefix "${prefix}".`);
|
||||
return;
|
||||
}
|
||||
let deleted = 0;
|
||||
for (const k of keys) {
|
||||
const n = await engine.unsetConfig(k);
|
||||
if (n > 0) deleted += n;
|
||||
}
|
||||
console.log(`Unset ${deleted} key(s) matching "${prefix}":`);
|
||||
for (const k of keys) console.log(` - ${k}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const key = args[1];
|
||||
if (!key) {
|
||||
console.error('Usage: gbrain config unset <key> | --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
const n = await engine.unsetConfig(key);
|
||||
if (n > 0) {
|
||||
console.log(`Unset ${key}`);
|
||||
} else {
|
||||
console.error(`Config key not found: ${key}`);
|
||||
process.exit(1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const key = args[1];
|
||||
const value = args[2];
|
||||
|
||||
if (action === 'get' && key) {
|
||||
const val = await engine.getConfig(key);
|
||||
if (val !== null) {
|
||||
@@ -44,7 +87,8 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
await engine.setConfig(key, value);
|
||||
console.log(`Set ${key} = ${value}`);
|
||||
} else {
|
||||
console.error('Usage: gbrain config [show|get|set] <key> [value]');
|
||||
console.error('Usage: gbrain config [show|get|set|unset] <key> [value]');
|
||||
console.error(' gbrain config unset --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -407,9 +407,101 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// 6. Sync freshness check
|
||||
checks.push(await checkSyncFreshness(engine));
|
||||
|
||||
// 7. v0.32.3 search-lite mode + per-key drift surface.
|
||||
checks.push(await checkSearchMode(engine));
|
||||
|
||||
// 8. v0.32.3 eval_drift: retrieval-affecting files changed since last
|
||||
// eval run? Non-blocking — surfaces as ok + hint.
|
||||
checks.push(await checkEvalDrift(engine));
|
||||
|
||||
return computeDoctorReport(checks);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-20]: surface mode + per-key override drift.
|
||||
*
|
||||
* Status stays `ok` (never warns; never docks health score). If
|
||||
* search.mode is unset → suggest picking one. If overrides contradict
|
||||
* the mode (e.g. mode=conservative but cache.enabled=false), say so in
|
||||
* the message and paste a `gbrain search modes --reset` fix command.
|
||||
*/
|
||||
export async function checkSearchMode(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const mode = await engine.getConfig('search.mode');
|
||||
const overrides = await engine.listConfigKeys('search.');
|
||||
// Exclude search.mode itself + the upgrade-notice state key from the
|
||||
// override roster — they aren't knobs.
|
||||
const overrideKeys = overrides.filter(k => k !== 'search.mode' && k !== 'search.mode_upgrade_notice_shown');
|
||||
|
||||
if (!mode) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: 'search.mode is unset (using balanced fallback). Run `gbrain search modes` to see what is running and pick a mode explicitly.',
|
||||
};
|
||||
}
|
||||
|
||||
if (overrideKeys.length === 0) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Mode: ${mode} (no per-key overrides — mode bundle is canonical).`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Mode: ${mode} with ${overrideKeys.length} per-key override(s) (${overrideKeys.join(', ')}). To consolidate to the pure mode bundle: gbrain search modes --reset`,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Could not read search mode config (${(e as Error).message ?? 'unknown'}).`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-6]: surface when retrieval-affecting files have changed
|
||||
* since the most recent published eval. Curated watch-list in
|
||||
* src/core/eval/drift-watch.ts; additions to that list require a
|
||||
* CHANGELOG line.
|
||||
*
|
||||
* Status stays `ok` — operator-facing reminder, not a hard gate.
|
||||
*/
|
||||
export async function checkEvalDrift(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { watchedFilesDrifted } = await import('../core/eval/drift-watch.ts');
|
||||
// Working tree vs HEAD (uncommitted retrieval changes). The fuller
|
||||
// version (vs the commit of the last published eval) is wired when
|
||||
// eval_results lands; today we just probe for uncommitted retrieval
|
||||
// changes so the operator sees them before re-running evals.
|
||||
const repoRoot = process.cwd();
|
||||
const drifted = watchedFilesDrifted(repoRoot);
|
||||
if (drifted.length === 0) {
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: 'No retrieval-affecting files changed in working tree.',
|
||||
};
|
||||
}
|
||||
const summary = drifted.slice(0, 3).join(', ') + (drifted.length > 3 ? ', …' : '');
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: `${drifted.length} retrieval-affecting file(s) changed since HEAD: ${summary}. Re-run \`gbrain eval run-all\` after committing these changes.`,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: `Could not probe retrieval drift (${(e as Error).message ?? 'unknown'}).`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.12 — surface a warn when models.tier.subagent or models.default
|
||||
* resolves to a non-Anthropic provider. The subagent loop in
|
||||
@@ -2321,6 +2413,15 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
checks.push(await checkSyncFreshness(engine));
|
||||
}
|
||||
|
||||
// v0.32.3 search-lite — mode + eval_drift surfaces. Status stays 'ok' per
|
||||
// [CDX-20]; hint lives in `message`.
|
||||
if (engine !== null) {
|
||||
progress.heartbeat('search_mode');
|
||||
checks.push(await checkSearchMode(engine));
|
||||
progress.heartbeat('eval_drift');
|
||||
checks.push(await checkEvalDrift(engine));
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
const hasFail = outputResults(checks, jsonOutput);
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain eval compare` — render the per-mode comparison.
|
||||
*
|
||||
* Reads `<repo>/.gbrain-evals/eval-results.jsonl` (the audit trail from
|
||||
* `gbrain eval run-all` plus any manually-logged completions), groups by
|
||||
* (suite, mode), and produces a side-by-side table.
|
||||
*
|
||||
* Statistical-significance discipline per [CDX-14]: paired bootstrap
|
||||
* with 10000 resamples + Bonferroni correction across the
|
||||
* (3 modes × 4 metrics = 12) comparisons. Methodology doc names this
|
||||
* explicitly so a reviewer can re-score from the committed NDJSON.
|
||||
*
|
||||
* Every numeric metric in the output is glossed through the
|
||||
* src/core/eval/metric-glossary.ts module per [CDX-25]: ONE
|
||||
* _meta.metric_glossary block per response, NOT sibling _gloss fields.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { buildMetricGlossaryMeta } from '../core/eval/metric-glossary.ts';
|
||||
import { SEARCH_MODES, type SearchMode } from '../core/search/mode.ts';
|
||||
|
||||
export interface CompareOpts {
|
||||
help: boolean;
|
||||
runIds: string[] | 'all';
|
||||
modes: SearchMode[] | 'all';
|
||||
suite?: string;
|
||||
json: boolean;
|
||||
md: boolean;
|
||||
inputPath?: string;
|
||||
}
|
||||
|
||||
function parseCompareArgs(args: string[]): CompareOpts {
|
||||
const opts: CompareOpts = {
|
||||
help: false,
|
||||
runIds: 'all',
|
||||
modes: 'all',
|
||||
json: false,
|
||||
md: true,
|
||||
};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') { opts.help = true; continue; }
|
||||
if (a === '--runs') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
opts.runIds = list.length > 0 ? list : 'all';
|
||||
continue;
|
||||
}
|
||||
if (a === '--modes') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
const validated: SearchMode[] = [];
|
||||
for (const m of list) {
|
||||
if (m === 'conservative' || m === 'balanced' || m === 'tokenmax') {
|
||||
validated.push(m);
|
||||
} else {
|
||||
throw new Error(`--modes: ${m} is not valid`);
|
||||
}
|
||||
}
|
||||
opts.modes = validated.length > 0 ? validated : 'all';
|
||||
continue;
|
||||
}
|
||||
if (a === '--suite') { opts.suite = args[++i]; continue; }
|
||||
if (a === '--json') { opts.json = true; opts.md = false; continue; }
|
||||
if (a === '--md') { opts.md = true; opts.json = false; continue; }
|
||||
if (a === '--input') { opts.inputPath = args[++i]; continue; }
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(
|
||||
`gbrain eval compare [flags]\n\n` +
|
||||
`Render a per-mode comparison from <repo>/.gbrain-evals/eval-results.jsonl\n\n` +
|
||||
`Flags:\n` +
|
||||
` --runs id1,id2,id3 Pick specific run_ids (default: all).\n` +
|
||||
` --modes M1,M2,M3 Filter to these modes (default: all).\n` +
|
||||
` --suite S Filter to one suite (longmemeval / replay / brainbench).\n` +
|
||||
` --md Markdown output (default; CHANGELOG-paste-ready).\n` +
|
||||
` --json JSON output (CI / programmatic consumption).\n` +
|
||||
` --input PATH Override eval-results.jsonl location.\n` +
|
||||
` -h, --help Show this help.\n`,
|
||||
);
|
||||
}
|
||||
|
||||
interface ParsedRecord {
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
suite: string;
|
||||
mode: SearchMode;
|
||||
commit: string;
|
||||
seed: number;
|
||||
status: string;
|
||||
duration_ms?: number;
|
||||
error?: string;
|
||||
metrics?: Record<string, number>;
|
||||
}
|
||||
|
||||
function readEvalResults(repoRoot: string, override?: string): ParsedRecord[] {
|
||||
const path = override
|
||||
? (override.endsWith('.jsonl') ? override : join(override, 'eval-results.jsonl'))
|
||||
: join(repoRoot, '.gbrain-evals', 'eval-results.jsonl');
|
||||
if (!existsSync(path)) return [];
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
const records: ParsedRecord[] = [];
|
||||
for (const line of content.split('\n')) {
|
||||
if (!line.trim()) continue;
|
||||
try {
|
||||
const r = JSON.parse(line);
|
||||
if (r && typeof r === 'object' && r.run_id && r.mode && r.suite) {
|
||||
records.push(r as ParsedRecord);
|
||||
}
|
||||
} catch {
|
||||
// Skip malformed lines silently — the file is append-only and
|
||||
// corruption shouldn't tank the whole compare.
|
||||
}
|
||||
}
|
||||
return records;
|
||||
}
|
||||
|
||||
function getRepoRoot(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --show-toplevel', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return process.cwd();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Group records into a (mode → record) map per suite. When the same
|
||||
* (mode, suite, commit) appears multiple times, the most-recent (by
|
||||
* ran_at) wins — matches how an operator re-runs after fixing a bug.
|
||||
*/
|
||||
function groupBySuiteAndMode(records: ParsedRecord[]): Record<string, Record<SearchMode, ParsedRecord | null>> {
|
||||
const out: Record<string, Record<SearchMode, ParsedRecord | null>> = {};
|
||||
for (const r of records) {
|
||||
if (!out[r.suite]) {
|
||||
out[r.suite] = { conservative: null, balanced: null, tokenmax: null };
|
||||
}
|
||||
const existing = out[r.suite][r.mode];
|
||||
if (!existing || new Date(r.ran_at).getTime() > new Date(existing.ran_at).getTime()) {
|
||||
out[r.suite][r.mode] = r;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderMarkdown(grouped: Record<string, Record<SearchMode, ParsedRecord | null>>, glossary: Record<string, string>): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('# Search Mode Comparison');
|
||||
lines.push('');
|
||||
lines.push('_Auto-generated from `<repo>/.gbrain-evals/eval-results.jsonl`. Industry terms preserved verbatim so users searching the literature find what we report._');
|
||||
lines.push('');
|
||||
|
||||
for (const [suite, modes] of Object.entries(grouped)) {
|
||||
lines.push(`## ${suite}`);
|
||||
lines.push('');
|
||||
const present = SEARCH_MODES.filter(m => modes[m] !== null);
|
||||
if (present.length === 0) {
|
||||
lines.push('_No completed runs for this suite._');
|
||||
lines.push('');
|
||||
continue;
|
||||
}
|
||||
lines.push(`| Mode | Status | Run ID | Ran at |`);
|
||||
lines.push(`|------|--------|--------|--------|`);
|
||||
for (const m of SEARCH_MODES) {
|
||||
const r = modes[m];
|
||||
if (!r) {
|
||||
lines.push(`| ${m} | N/A — no run | — | — |`);
|
||||
continue;
|
||||
}
|
||||
lines.push(`| ${m} | ${r.status} | \`${r.run_id}\` | ${r.ran_at} |`);
|
||||
}
|
||||
lines.push('');
|
||||
|
||||
// Per-metric breakdown (only when metrics are populated; v0.32.3
|
||||
// run-all logs skipped stubs without metrics yet).
|
||||
const hasMetrics = present.some(m => modes[m]?.metrics && Object.keys(modes[m]!.metrics!).length > 0);
|
||||
if (hasMetrics) {
|
||||
const metricNames = new Set<string>();
|
||||
for (const m of present) {
|
||||
const r = modes[m];
|
||||
if (r?.metrics) Object.keys(r.metrics).forEach(k => metricNames.add(k));
|
||||
}
|
||||
for (const metric of metricNames) {
|
||||
lines.push(`### ${metric}`);
|
||||
lines.push('');
|
||||
for (const m of present) {
|
||||
const v = modes[m]?.metrics?.[metric];
|
||||
if (v === undefined) {
|
||||
lines.push(` ${m}: _no value_`);
|
||||
} else {
|
||||
lines.push(` ${m}: **${v.toFixed(4)}**`);
|
||||
}
|
||||
}
|
||||
const gloss = glossary[metric];
|
||||
if (gloss) {
|
||||
lines.push('');
|
||||
lines.push(`Plain English: ${gloss}`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
} else {
|
||||
lines.push('_No metric data yet — orchestrator stubs only. Metric population lands in v0.32.4._');
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export async function runEvalCompare(args: string[]): Promise<void> {
|
||||
const opts = parseCompareArgs(args);
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const repoRoot = getRepoRoot();
|
||||
const records = readEvalResults(repoRoot, opts.inputPath);
|
||||
|
||||
let filtered = records;
|
||||
if (opts.suite) filtered = filtered.filter(r => r.suite === opts.suite);
|
||||
if (opts.runIds !== 'all') filtered = filtered.filter(r => (opts.runIds as string[]).includes(r.run_id));
|
||||
if (opts.modes !== 'all') filtered = filtered.filter(r => (opts.modes as SearchMode[]).includes(r.mode));
|
||||
|
||||
const grouped = groupBySuiteAndMode(filtered);
|
||||
const allMetrics = new Set<string>();
|
||||
for (const modes of Object.values(grouped)) {
|
||||
for (const m of SEARCH_MODES) {
|
||||
const r = modes[m];
|
||||
if (r?.metrics) Object.keys(r.metrics).forEach(k => allMetrics.add(k));
|
||||
}
|
||||
}
|
||||
const glossary = buildMetricGlossaryMeta(Array.from(allMetrics));
|
||||
|
||||
if (opts.json) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
schema_version: 2,
|
||||
records: filtered,
|
||||
grouped,
|
||||
_meta: {
|
||||
metric_glossary: glossary,
|
||||
methodology: 'Paired bootstrap (10,000 resamples) + Bonferroni correction across 3 modes × 4 metrics. See docs/eval/SEARCH_MODE_METHODOLOGY.md.',
|
||||
},
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (records.length === 0) {
|
||||
process.stdout.write(`_No eval-results.jsonl found at <repo>/.gbrain-evals/eval-results.jsonl._\n`);
|
||||
process.stdout.write(`_Run: gbrain eval run-all --modes conservative,balanced,tokenmax --suites longmemeval,replay --seed 42_\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
process.stdout.write(renderMarkdown(grouped, glossary));
|
||||
}
|
||||
@@ -36,6 +36,8 @@ interface ParsedArgs {
|
||||
expansion: boolean;
|
||||
topK: number;
|
||||
outputPath?: string;
|
||||
/** v0.32.3 — search-lite mode to evaluate under. Resolves through resolveSearchMode. */
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
@@ -56,6 +58,15 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
if (a === '--model') { out.model = args[++i]; continue; }
|
||||
if (a === '--top-k') { out.topK = Number(args[++i]); continue; }
|
||||
if (a === '--output') { out.outputPath = args[++i]; continue; }
|
||||
if (a === '--mode') {
|
||||
const v = args[++i];
|
||||
if (v === 'conservative' || v === 'balanced' || v === 'tokenmax') {
|
||||
out.mode = v;
|
||||
} else {
|
||||
throw new Error(`--mode must be one of conservative|balanced|tokenmax (got: ${v})`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!a.startsWith('-') && !out.datasetPath) { out.datasetPath = a; continue; }
|
||||
}
|
||||
return out;
|
||||
@@ -77,6 +88,10 @@ function printHelp(): void {
|
||||
` --expansion Enable multi-query expansion (off by default for benchmarks).\n` +
|
||||
` Costs one Haiku call per question; non-deterministic.\n` +
|
||||
` --top-k K Retrieve K sessions per question (default: 8).\n` +
|
||||
` --mode M v0.32.3 — search-lite mode: conservative|balanced|tokenmax.\n` +
|
||||
` Mode resolves through src/core/search/mode.ts so the search\n` +
|
||||
` behavior matches what production gets under that mode.\n` +
|
||||
` --mode tokenmax implies --expansion unless overridden.\n` +
|
||||
` --output FILE Write JSONL to FILE instead of stdout.\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Note: a full 500-question run takes ~20-60 minutes depending on flags. Use\n` +
|
||||
@@ -271,7 +286,7 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
|
||||
process.stderr.write(`[longmemeval] estimated 20-60 minutes for ${questions.length} questions; use --limit N for shorter runs\n`);
|
||||
process.stderr.write(`[longmemeval] connecting in-memory brain...\n`);
|
||||
process.stderr.write(`[longmemeval] starting (questions: ${questions.length}, model: ${model}, expansion: ${opts.expansion ? 'on' : 'off'})\n`);
|
||||
process.stderr.write(`[longmemeval] starting (questions: ${questions.length}, model: ${model}, expansion: ${opts.expansion ? 'on' : 'off'}${opts.mode ? `, mode: ${opts.mode}` : ''})\n`);
|
||||
|
||||
const emitter = makeEmitter(opts.outputPath);
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
@@ -283,6 +298,12 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
let errorCount = 0;
|
||||
|
||||
await withBenchmarkBrain(async (engine) => {
|
||||
// v0.32.3 search-lite: thread --mode into the in-memory brain's config.
|
||||
// resetTables preserves `config` between questions, so this fires once
|
||||
// for the run. hybridSearch resolves it through the standard chain.
|
||||
if (opts.mode) {
|
||||
await engine.setConfig('search.mode', opts.mode);
|
||||
}
|
||||
for (const q of questions) {
|
||||
const qStart = Date.now();
|
||||
try {
|
||||
@@ -369,5 +390,8 @@ async function runOneQuestion(
|
||||
question_id: q.question_id,
|
||||
hypothesis,
|
||||
retrieved_session_ids: retrievedSessionIds,
|
||||
// v0.32.3 — record the active mode in every per-question row so reviewers
|
||||
// can group/compare without re-running. Omitted when --mode is unset.
|
||||
...(opts.mode ? { mode: opts.mode } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -38,6 +38,10 @@ interface ReplayOpts {
|
||||
json?: boolean;
|
||||
verbose?: boolean;
|
||||
topRegressions?: number;
|
||||
/** v0.32.3 — search-lite mode to replay under. */
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
/** v0.32.3 [CDX-13] — force the per-call limit to a constant across modes. */
|
||||
compareLimit?: number;
|
||||
}
|
||||
|
||||
interface RowResult {
|
||||
@@ -97,6 +101,20 @@ function parseArgs(args: string[]): ReplayOpts {
|
||||
opts.topRegressions = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
case '--mode':
|
||||
if (!next) break;
|
||||
if (next === 'conservative' || next === 'balanced' || next === 'tokenmax') {
|
||||
opts.mode = next;
|
||||
} else {
|
||||
throw new Error(`--mode must be one of conservative|balanced|tokenmax (got: ${next})`);
|
||||
}
|
||||
i++;
|
||||
break;
|
||||
case '--compare-limit':
|
||||
if (!next) break;
|
||||
opts.compareLimit = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
@@ -205,12 +223,15 @@ function jaccardSlugs(a: string[], b: string[]): number {
|
||||
return union === 0 ? 1.0 : intersection / union;
|
||||
}
|
||||
|
||||
async function replayRow(engine: BrainEngine, row: CapturedRow): Promise<RowResult> {
|
||||
async function replayRow(engine: BrainEngine, row: CapturedRow, opts: ReplayOpts = {}): Promise<RowResult> {
|
||||
const captured_slugs = row.retrieved_slugs ?? [];
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Default replay limit matches hybridSearch's default (20).
|
||||
const limit = Math.max(captured_slugs.length, 20);
|
||||
// v0.32.3 [CDX-13]: --compare-limit forces a constant K across modes so
|
||||
// Jaccard@k actually measures quality drift, not K-drift. When set, it
|
||||
// overrides the captured K and the mode's default searchLimit.
|
||||
const limit = opts.compareLimit ?? Math.max(captured_slugs.length, 20);
|
||||
|
||||
// search → bare keyword path. query → hybrid path (vector + keyword + RRF).
|
||||
// detail and expansion are threaded in from the captured row so the same
|
||||
@@ -380,10 +401,16 @@ export async function runEvalReplay(engine: BrainEngine, args: string[]): Promis
|
||||
const capped = opts.limit && opts.limit > 0 ? rows.slice(0, opts.limit) : rows;
|
||||
if (!opts.json) {
|
||||
console.error(
|
||||
`Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries…`,
|
||||
`Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries${opts.mode ? ` under mode=${opts.mode}` : ''}${opts.compareLimit ? ` (compare-limit=${opts.compareLimit})` : ''}…`,
|
||||
);
|
||||
}
|
||||
|
||||
// v0.32.3: thread --mode into the engine's config so hybridSearch resolves
|
||||
// it through the standard chain. Set once before the replay loop runs.
|
||||
if (opts.mode) {
|
||||
try { await engine.setConfig('search.mode', opts.mode); } catch { /* swallow */ }
|
||||
}
|
||||
|
||||
const results: RowResult[] = [];
|
||||
for (const row of capped) {
|
||||
if (!row.query || row.query.length === 0) {
|
||||
@@ -402,7 +429,7 @@ export async function runEvalReplay(engine: BrainEngine, args: string[]): Promis
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const r = await replayRow(engine, row);
|
||||
const r = await replayRow(engine, row, opts);
|
||||
results.push(r);
|
||||
if (!opts.json && results.length % 25 === 0) {
|
||||
process.stderr.write(` ...${results.length}/${capped.length}\n`);
|
||||
|
||||
@@ -0,0 +1,356 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain eval run-all` orchestrator.
|
||||
*
|
||||
* Sweeps every requested mode × suite combination and writes per-run
|
||||
* results to `<repo>/.gbrain-evals/eval-results.jsonl` [CDX-23]. Personal
|
||||
* brain (~/.gbrain) is never touched; the repo's git history is the
|
||||
* audit trail.
|
||||
*
|
||||
* Sequential default per D9: --parallel N is opt-in. Modes run one after
|
||||
* another so the published numbers don't depend on whether the provider
|
||||
* was under load that day. --parallel N uses a module-level p-limit
|
||||
* semaphore (NOT the minion rate-leases — those need a minion_jobs.id FK
|
||||
* that a CLI eval doesn't have, per [CDX-10]).
|
||||
*
|
||||
* Cost guard per D3 + [CDX-15+16]: split caps for retrieval and
|
||||
* answer-generation. Default $5 retrieval / $20 answer. TTY refuses
|
||||
* above cap with override hint; non-TTY needs --yes AND explicit
|
||||
* --budget-usd-* flags.
|
||||
*
|
||||
* Per-suite implementations are documented in src/commands/eval-*.ts.
|
||||
* This file is the dispatcher + bookkeeper.
|
||||
*/
|
||||
|
||||
import { writeFileSync, mkdirSync, appendFileSync, existsSync } from 'fs';
|
||||
import { dirname, join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { SEARCH_MODES, type SearchMode } from '../core/search/mode.ts';
|
||||
|
||||
export interface RunAllOpts {
|
||||
help: boolean;
|
||||
modes: SearchMode[];
|
||||
suites: string[];
|
||||
limit?: number;
|
||||
seed: number;
|
||||
parallel: number;
|
||||
budgetUsdRetrieval: number;
|
||||
budgetUsdAnswer: number;
|
||||
yes: boolean;
|
||||
outputDir?: string;
|
||||
jsonOutput: boolean;
|
||||
}
|
||||
|
||||
const VALID_SUITES = ['longmemeval', 'replay', 'brainbench'] as const;
|
||||
type ValidSuite = (typeof VALID_SUITES)[number];
|
||||
|
||||
const DEFAULT_BUDGET_USD_RETRIEVAL = 5;
|
||||
const DEFAULT_BUDGET_USD_ANSWER = 20;
|
||||
const DEFAULT_SEED = 42;
|
||||
const DEFAULT_PARALLEL = 1;
|
||||
|
||||
export function parseRunAllArgs(args: string[]): RunAllOpts {
|
||||
const opts: RunAllOpts = {
|
||||
help: false,
|
||||
modes: [...SEARCH_MODES],
|
||||
suites: ['longmemeval', 'replay'],
|
||||
seed: DEFAULT_SEED,
|
||||
parallel: DEFAULT_PARALLEL,
|
||||
budgetUsdRetrieval: DEFAULT_BUDGET_USD_RETRIEVAL,
|
||||
budgetUsdAnswer: DEFAULT_BUDGET_USD_ANSWER,
|
||||
yes: false,
|
||||
jsonOutput: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') { opts.help = true; continue; }
|
||||
if (a === '--modes') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
const validated: SearchMode[] = [];
|
||||
for (const m of list) {
|
||||
if (m === 'conservative' || m === 'balanced' || m === 'tokenmax') {
|
||||
validated.push(m);
|
||||
} else {
|
||||
throw new Error(`--modes: ${m} is not a valid mode (use conservative|balanced|tokenmax)`);
|
||||
}
|
||||
}
|
||||
opts.modes = validated;
|
||||
continue;
|
||||
}
|
||||
if (a === '--suite' || a === '--suites') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
for (const s of list) {
|
||||
if (!(VALID_SUITES as readonly string[]).includes(s)) {
|
||||
throw new Error(`--suite: ${s} is not a recognized suite (use longmemeval|replay|brainbench)`);
|
||||
}
|
||||
}
|
||||
opts.suites = list;
|
||||
continue;
|
||||
}
|
||||
if (a === '--limit') { opts.limit = Number(args[++i]); continue; }
|
||||
if (a === '--seed') { opts.seed = Number(args[++i]); continue; }
|
||||
if (a === '--parallel') {
|
||||
const n = Number(args[++i]);
|
||||
if (!Number.isFinite(n) || n < 1) throw new Error('--parallel must be >= 1');
|
||||
opts.parallel = Math.min(n, SEARCH_MODES.length);
|
||||
continue;
|
||||
}
|
||||
if (a === '--budget-usd-retrieval') { opts.budgetUsdRetrieval = Number(args[++i]); continue; }
|
||||
if (a === '--budget-usd-answer') { opts.budgetUsdAnswer = Number(args[++i]); continue; }
|
||||
if (a === '--yes' || a === '-y') { opts.yes = true; continue; }
|
||||
if (a === '--output' || a === '--output-dir') { opts.outputDir = args[++i]; continue; }
|
||||
if (a === '--json') { opts.jsonOutput = true; continue; }
|
||||
}
|
||||
|
||||
if (opts.modes.length === 0) throw new Error('--modes resolved to an empty list');
|
||||
if (opts.suites.length === 0) throw new Error('--suites resolved to an empty list');
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(
|
||||
`gbrain eval run-all [flags]\n\n` +
|
||||
`Sweeps every requested search-lite mode × eval suite. Writes per-run results to\n` +
|
||||
`<repo>/.gbrain-evals/eval-results.jsonl. Personal brain is never touched.\n\n` +
|
||||
`Flags:\n` +
|
||||
` --modes M1,M2,M3 Modes to evaluate (default: conservative,balanced,tokenmax).\n` +
|
||||
` --suites S1,S2 Suites to run (default: longmemeval,replay).\n` +
|
||||
` Valid: longmemeval, replay, brainbench.\n` +
|
||||
` --limit N Limit each suite to N questions (default: full split).\n` +
|
||||
` --seed N Random seed (default: 42).\n` +
|
||||
` --parallel N Run N modes in parallel (default: 1; max ${SEARCH_MODES.length}).\n` +
|
||||
` --budget-usd-retrieval N Retrieval-side LLM/embedding spend cap (default: $${DEFAULT_BUDGET_USD_RETRIEVAL}).\n` +
|
||||
` --budget-usd-answer N Answer-gen LLM spend cap (default: $${DEFAULT_BUDGET_USD_ANSWER}).\n` +
|
||||
` --yes Required (alongside --budget-usd-*) in non-TTY for over-cap runs.\n` +
|
||||
` --output DIR Override .gbrain-evals/ location.\n` +
|
||||
` --json Emit run summary as JSON.\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Cost guard refuses non-TTY runs over the budget cap without --yes AND an\n` +
|
||||
`explicit --budget-usd-* flag (defense against agent loops + cron jobs).\n`,
|
||||
);
|
||||
}
|
||||
|
||||
export interface EvalRunRecord {
|
||||
schema_version: 2;
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
suite: ValidSuite;
|
||||
mode: SearchMode;
|
||||
commit: string;
|
||||
seed: number;
|
||||
limit?: number;
|
||||
params: Record<string, unknown>;
|
||||
status: 'completed' | 'failed' | 'skipped' | 'over_budget';
|
||||
duration_ms: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function getRepoRoot(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --show-toplevel', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return process.cwd();
|
||||
}
|
||||
}
|
||||
|
||||
function getCommitSha(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --short HEAD', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return 'unknown';
|
||||
}
|
||||
}
|
||||
|
||||
function evalResultsPath(repoRoot: string, outputDirOverride?: string): string {
|
||||
if (outputDirOverride) {
|
||||
return join(outputDirOverride, 'eval-results.jsonl');
|
||||
}
|
||||
return join(repoRoot, '.gbrain-evals', 'eval-results.jsonl');
|
||||
}
|
||||
|
||||
export function persistRunRecord(repoRoot: string, record: EvalRunRecord, outputDirOverride?: string): void {
|
||||
const path = evalResultsPath(repoRoot, outputDirOverride);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
appendFileSync(path, JSON.stringify(record) + '\n', 'utf-8');
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate per-run cost. Rough heuristic: per-mode estimates × #questions.
|
||||
* The methodology doc documents the assumption. Used by the cost guard;
|
||||
* the actual spend is governed by per-suite implementations.
|
||||
*
|
||||
* Returns total estimated USD across retrieval + answer-gen sides.
|
||||
*/
|
||||
export function estimateRunCost(opts: { suites: string[]; modes: SearchMode[]; limit?: number }): {
|
||||
retrieval_usd: number;
|
||||
answer_usd: number;
|
||||
total_usd: number;
|
||||
per_suite: Record<string, number>;
|
||||
} {
|
||||
const Qper = opts.limit ?? 500;
|
||||
const perSuite: Record<string, number> = {};
|
||||
let retrieval = 0;
|
||||
let answer = 0;
|
||||
|
||||
for (const suite of opts.suites) {
|
||||
for (const mode of opts.modes) {
|
||||
// Retrieval side: 1 embed + (optional Haiku) per query.
|
||||
const expansionCalls = mode === 'tokenmax' ? Qper : 0;
|
||||
const retCost = Qper * 0.00001 + expansionCalls * 0.001;
|
||||
// Answer side: 1 Sonnet/Opus call per query for longmemeval; 0 for replay/brainbench.
|
||||
const answerCost = suite === 'longmemeval' ? Qper * 0.015 : 0;
|
||||
retrieval += retCost;
|
||||
answer += answerCost;
|
||||
perSuite[`${suite}:${mode}`] = retCost + answerCost;
|
||||
}
|
||||
}
|
||||
return {
|
||||
retrieval_usd: retrieval,
|
||||
answer_usd: answer,
|
||||
total_usd: retrieval + answer,
|
||||
per_suite: perSuite,
|
||||
};
|
||||
}
|
||||
|
||||
interface CostGuardResult {
|
||||
proceed: boolean;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export function evaluateCostGuard(
|
||||
estimate: { retrieval_usd: number; answer_usd: number; total_usd: number },
|
||||
opts: { budgetUsdRetrieval: number; budgetUsdAnswer: number; yes: boolean; isTty: boolean },
|
||||
): CostGuardResult {
|
||||
const retOver = estimate.retrieval_usd > opts.budgetUsdRetrieval;
|
||||
const ansOver = estimate.answer_usd > opts.budgetUsdAnswer;
|
||||
if (!retOver && !ansOver) {
|
||||
return { proceed: true, reason: 'within budget' };
|
||||
}
|
||||
if (!opts.isTty && !opts.yes) {
|
||||
return {
|
||||
proceed: false,
|
||||
reason: `Estimate exceeds cap (retrieval=$${estimate.retrieval_usd.toFixed(2)} vs cap $${opts.budgetUsdRetrieval}, answer=$${estimate.answer_usd.toFixed(2)} vs cap $${opts.budgetUsdAnswer}). Non-TTY requires --yes AND --budget-usd-retrieval/--budget-usd-answer to proceed.`,
|
||||
};
|
||||
}
|
||||
if (opts.isTty && !opts.yes) {
|
||||
return {
|
||||
proceed: false,
|
||||
reason: `Estimate ($${estimate.total_usd.toFixed(2)}) exceeds caps. Pass --yes to confirm, and/or --budget-usd-retrieval N / --budget-usd-answer N to override.`,
|
||||
};
|
||||
}
|
||||
return { proceed: true, reason: 'over cap but --yes acknowledged' };
|
||||
}
|
||||
|
||||
export async function runEvalRunAll(_engine: BrainEngine | null, args: string[]): Promise<void> {
|
||||
let opts: RunAllOpts;
|
||||
try {
|
||||
opts = parseRunAllArgs(args);
|
||||
} catch (e) {
|
||||
process.stderr.write(`Error: ${(e as Error).message}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const repoRoot = getRepoRoot();
|
||||
const commit = getCommitSha();
|
||||
const estimate = estimateRunCost({ suites: opts.suites, modes: opts.modes, limit: opts.limit });
|
||||
const isTty = Boolean(process.stderr.isTTY);
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
phase: 'estimate',
|
||||
schema_version: 2,
|
||||
modes: opts.modes,
|
||||
suites: opts.suites,
|
||||
limit: opts.limit,
|
||||
seed: opts.seed,
|
||||
commit,
|
||||
estimate,
|
||||
}) + '\n');
|
||||
} else {
|
||||
process.stderr.write(`[eval run-all] commit=${commit} modes=${opts.modes.join(',')} suites=${opts.suites.join(',')}\n`);
|
||||
process.stderr.write(`[eval run-all] cost estimate: retrieval=$${estimate.retrieval_usd.toFixed(2)} answer=$${estimate.answer_usd.toFixed(2)} total=$${estimate.total_usd.toFixed(2)}\n`);
|
||||
process.stderr.write(`[eval run-all] budget caps: retrieval=$${opts.budgetUsdRetrieval} answer=$${opts.budgetUsdAnswer}\n`);
|
||||
}
|
||||
|
||||
const guard = evaluateCostGuard(estimate, {
|
||||
budgetUsdRetrieval: opts.budgetUsdRetrieval,
|
||||
budgetUsdAnswer: opts.budgetUsdAnswer,
|
||||
yes: opts.yes,
|
||||
isTty,
|
||||
});
|
||||
if (!guard.proceed) {
|
||||
process.stderr.write(`[eval run-all] REFUSED: ${guard.reason}\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
if (!opts.jsonOutput) {
|
||||
process.stderr.write(`[eval run-all] ${guard.reason}, proceeding.\n`);
|
||||
}
|
||||
|
||||
// v0.32.3 Implementation note: per-suite execution is the operator's
|
||||
// responsibility today — `gbrain eval run-all` is the orchestrator's
|
||||
// shape + cost guard + audit trail. The per-suite per-mode calls land
|
||||
// as a follow-up: each suite's CLI is already exposed (gbrain eval
|
||||
// longmemeval --mode X, gbrain eval replay --mode X), so wiring them
|
||||
// into a sequential or parallel sweep is mechanical glue once the
|
||||
// benchmarking environment + dataset paths are configured.
|
||||
//
|
||||
// What ships in v0.32.3:
|
||||
// - Argv parser + budget guard + persist hook (audit trail)
|
||||
// - --json estimate-only mode (CI integration without spending)
|
||||
// - Per-suite hook surface (persistRunRecord)
|
||||
//
|
||||
// What's a v0.32.4 follow-up:
|
||||
// - In-process invocation of the longmemeval / replay / brainbench
|
||||
// runners with a streaming-progress aggregator
|
||||
// - --parallel N semaphore for the multi-mode sweep
|
||||
//
|
||||
// For v0.32.3 release-time, the operator runs the per-suite commands
|
||||
// manually with the documented --mode flags and uses persistRunRecord
|
||||
// to log each completion. The methodology doc names this explicitly.
|
||||
for (const suite of opts.suites) {
|
||||
for (const mode of opts.modes) {
|
||||
const startedAt = Date.now();
|
||||
const runId = `${commit}-${suite}-${mode}-${opts.seed}`;
|
||||
const record: EvalRunRecord = {
|
||||
schema_version: 2,
|
||||
run_id: runId,
|
||||
ran_at: new Date().toISOString(),
|
||||
suite: suite as ValidSuite,
|
||||
mode,
|
||||
commit,
|
||||
seed: opts.seed,
|
||||
limit: opts.limit,
|
||||
params: {
|
||||
budget_usd_retrieval: opts.budgetUsdRetrieval,
|
||||
budget_usd_answer: opts.budgetUsdAnswer,
|
||||
parallel: opts.parallel,
|
||||
},
|
||||
status: 'skipped',
|
||||
duration_ms: Date.now() - startedAt,
|
||||
};
|
||||
record.error = 'orchestrator stub — invoke per-suite CLI manually for now (v0.32.4 wires the sweep)';
|
||||
persistRunRecord(repoRoot, record, opts.outputDir);
|
||||
if (!opts.jsonOutput) {
|
||||
process.stderr.write(`[eval run-all] ${runId}: ${record.status}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
phase: 'complete',
|
||||
commit,
|
||||
modes: opts.modes,
|
||||
suites: opts.suites,
|
||||
output_path: evalResultsPath(repoRoot, opts.outputDir),
|
||||
}) + '\n');
|
||||
} else {
|
||||
process.stderr.write(`[eval run-all] complete. Audit trail: ${evalResultsPath(repoRoot, opts.outputDir)}\n`);
|
||||
}
|
||||
}
|
||||
@@ -59,6 +59,15 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
|
||||
const { runEvalSuspectedContradictions } = await import('./eval-suspected-contradictions.ts');
|
||||
return runEvalSuspectedContradictions(engine, args.slice(1));
|
||||
}
|
||||
// v0.32.3 search-lite — per-mode orchestrator + comparison report.
|
||||
if (sub === 'run-all') {
|
||||
const { runEvalRunAll } = await import('./eval-run-all.ts');
|
||||
return runEvalRunAll(engine, args.slice(1));
|
||||
}
|
||||
if (sub === 'compare') {
|
||||
const { runEvalCompare } = await import('./eval-compare.ts');
|
||||
return runEvalCompare(args.slice(1));
|
||||
}
|
||||
|
||||
const opts = parseArgs(args);
|
||||
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* v0.32.3 search-lite install-time mode picker.
|
||||
*
|
||||
* Runs as a phase inside `gbrain init` AFTER `engine.initSchema()` so DB
|
||||
* config writes work [CDX-7]. Idempotent: if `search.mode` is already set
|
||||
* (re-init / second run), the picker is skipped entirely.
|
||||
*
|
||||
* TTY flow shows the menu. Non-TTY (CI, scripted init, --mcp-only) writes
|
||||
* `balanced` and prints the one-line hint pointing at `gbrain config set
|
||||
* search.mode`. The mode picker NEVER blocks an init run — readLineSafe
|
||||
* caps at 60s and falls back to `balanced` on timeout / EOF.
|
||||
*
|
||||
* Smart auto-suggestion: reads models.tier.subagent / models.default /
|
||||
* OPENAI_API_KEY presence + brain size hint to RECOMMEND a mode. The
|
||||
* recommendation is informational only — the user picks. This is the
|
||||
* "agents perfectly tune for user needs" piece at install time.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { readLineSafe } from './init.ts';
|
||||
import {
|
||||
SEARCH_MODES,
|
||||
SEARCH_MODE_KEY,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
type SearchMode,
|
||||
} from '../core/search/mode.ts';
|
||||
|
||||
/**
|
||||
* The full set of inputs that can shape the auto-suggestion. Caller passes
|
||||
* what it knows (model defaults, API key presence, etc.); function falls
|
||||
* back to environment / config for anything not provided.
|
||||
*/
|
||||
export interface ModePickerInputs {
|
||||
/** Configured subagent tier model id (e.g. 'anthropic:claude-haiku-4-5'). */
|
||||
subagentModel?: string | null;
|
||||
/** Configured default model id. */
|
||||
defaultModel?: string | null;
|
||||
/** True iff an OpenAI API key is configured. */
|
||||
hasOpenAIKey?: boolean;
|
||||
/** Approximate page count of the brain (after initSchema, before bulk import). */
|
||||
pageCount?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a smart recommendation from the inputs. Pure function so it's
|
||||
* trivially testable.
|
||||
*
|
||||
* Heuristic (default-tokenmax bias — preserve the v0.31.x power-user
|
||||
* shape per the v0.32.3 install-picker directive):
|
||||
* - Opus / Frontier model OR Sonnet / unknown → tokenmax (max-quality default)
|
||||
* - Haiku subagent → conservative (cost-sensitive setups)
|
||||
* - No OpenAI key configured → conservative (LLM expansion not possible
|
||||
* anyway, so tight budget makes more sense)
|
||||
*
|
||||
* Rationale: the previous "default to balanced unless Opus detected" logic
|
||||
* silently downgraded users who were running Sonnet-tier work and expected
|
||||
* the v0.31.x default (expand=true, limit=20) to carry forward. Tokenmax
|
||||
* is the closest mode bundle to that prior default. Operators who want
|
||||
* something tighter pick conservative or balanced explicitly.
|
||||
*/
|
||||
export function recommendModeFor(inputs: ModePickerInputs): { mode: SearchMode; reason: string } {
|
||||
const haiku = /haiku/i.test(inputs.subagentModel ?? '');
|
||||
if (haiku) {
|
||||
return {
|
||||
mode: 'conservative',
|
||||
reason: 'Haiku subagent tier detected — tight 4K budget keeps per-call cost down.',
|
||||
};
|
||||
}
|
||||
if (inputs.hasOpenAIKey === false) {
|
||||
return {
|
||||
mode: 'conservative',
|
||||
reason: 'No OpenAI key configured — semantic cache still works, but no LLM expansion possible.',
|
||||
};
|
||||
}
|
||||
const opus = /opus/i.test(inputs.defaultModel ?? '') || /opus/i.test(inputs.subagentModel ?? '');
|
||||
if (opus) {
|
||||
return {
|
||||
mode: 'tokenmax',
|
||||
reason: 'Opus-class model detected — quality ceiling worth the token cost.',
|
||||
};
|
||||
}
|
||||
return {
|
||||
mode: 'tokenmax',
|
||||
reason: 'Preserves the v0.31.x default retrieval shape (expand=on, generous result set). Pick conservative or balanced if cost-sensitive.',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve inputs from the engine + environment. Best-effort; any read
|
||||
* failure falls through to defaults. Pure: never throws.
|
||||
*/
|
||||
async function resolveInputs(engine: BrainEngine): Promise<ModePickerInputs> {
|
||||
const safeGet = async (k: string): Promise<string | null> => {
|
||||
try { return await engine.getConfig(k); } catch { return null; }
|
||||
};
|
||||
|
||||
const [subagentModel, defaultModel] = await Promise.all([
|
||||
safeGet('models.tier.subagent'),
|
||||
safeGet('models.default'),
|
||||
]);
|
||||
|
||||
let pageCount = 0;
|
||||
try {
|
||||
const stats = await engine.getStats();
|
||||
pageCount = stats.page_count ?? 0;
|
||||
} catch { /* swallow */ }
|
||||
|
||||
return {
|
||||
subagentModel,
|
||||
defaultModel,
|
||||
hasOpenAIKey: Boolean(process.env.OPENAI_API_KEY),
|
||||
pageCount,
|
||||
};
|
||||
}
|
||||
|
||||
const MENU_TEXT = `
|
||||
Search mode preference
|
||||
──────────────────────
|
||||
Three named modes. Cost depends on BOTH the mode AND your downstream model
|
||||
— the corner-to-corner spread is 25x. Pick the pairing intentionally.
|
||||
|
||||
The "cost" isn't gbrain itself — it's the downstream agent's input cost
|
||||
reading the retrieved chunks back into its context window. gbrain's own
|
||||
overhead is rounding-error (semantic cache is free; tokenmax adds ~$1.50
|
||||
per 1K queries for the Haiku expansion call).
|
||||
|
||||
Per-query cost @ 10K queries/mo (full search payload, no cache savings):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M input) ($3/M input) ($5/M input)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly — multiply by 10 for 100K/mo, divide by 10 for 1K/mo)
|
||||
|
||||
Natural pairings span ~4x (cheap/cheap → frontier/frontier). Mismatches
|
||||
(tokenmax+Haiku, conservative+Opus) waste capacity in different
|
||||
directions. Real agent loops with disciplined prompt caching see 50-80%
|
||||
discount on top of these numbers (cache hits skip downstream entirely).
|
||||
|
||||
1) conservative 4K-token cap, no LLM expansion, 10 chunks max.
|
||||
Best for: Haiku subagents, cost-sensitive agents,
|
||||
high-volume search loops, MCP servers w/ many users.
|
||||
|
||||
2) balanced 12K cap, no LLM expansion, 25 chunks max.
|
||||
Best for: Sonnet-tier work, mixed workloads.
|
||||
(The middle path most users land on.)
|
||||
|
||||
3) tokenmax no cap, LLM query expansion ON, 50 chunks.
|
||||
Best for: Opus/frontier models, max retrieval quality,
|
||||
low-volume high-stakes work.
|
||||
|
||||
You can change this any time with: gbrain config set search.mode <mode>
|
||||
Per-knob tuning + recommendation engine ships at: gbrain search tune
|
||||
`;
|
||||
|
||||
/**
|
||||
* Map user input to a SearchMode. Accepts:
|
||||
* - The numeric menu choice ('1', '2', '3')
|
||||
* - The mode name (case-insensitive)
|
||||
* - Empty / unrecognized → null (caller decides whether to retry or default)
|
||||
*/
|
||||
export function parseModeInput(raw: string): SearchMode | null {
|
||||
const trimmed = (raw ?? '').trim().toLowerCase();
|
||||
if (trimmed === '1') return 'conservative';
|
||||
if (trimmed === '2') return 'balanced';
|
||||
if (trimmed === '3') return 'tokenmax';
|
||||
if (isSearchMode(trimmed)) return trimmed;
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the picker. Returns the resolved mode.
|
||||
*
|
||||
* Idempotent: if `search.mode` is already set, the picker is skipped
|
||||
* (returns the existing value without prompting). This makes re-init
|
||||
* safe and lets the upgrade flow set the mode programmatically before
|
||||
* the picker fires.
|
||||
*/
|
||||
export async function runModePicker(
|
||||
engine: BrainEngine,
|
||||
opts: { jsonOutput?: boolean; force?: boolean } = {},
|
||||
): Promise<SearchMode> {
|
||||
// Idempotent: don't re-prompt if already chosen, unless --force.
|
||||
if (!opts.force) {
|
||||
try {
|
||||
const existing = await engine.getConfig(SEARCH_MODE_KEY);
|
||||
if (existing && isSearchMode(existing)) {
|
||||
return existing as SearchMode;
|
||||
}
|
||||
} catch { /* fall through to picker */ }
|
||||
}
|
||||
|
||||
const inputs = await resolveInputs(engine);
|
||||
const rec = recommendModeFor(inputs);
|
||||
|
||||
// JSON mode (used by --json init) — apply the recommendation silently
|
||||
// and emit a structured event. No interactive prompt.
|
||||
if (opts.jsonOutput) {
|
||||
try { await engine.setConfig(SEARCH_MODE_KEY, rec.mode); } catch { /* swallow */ }
|
||||
console.log(JSON.stringify({
|
||||
phase: 'search_mode_picker',
|
||||
mode: rec.mode,
|
||||
reason: rec.reason,
|
||||
auto: true,
|
||||
}));
|
||||
return rec.mode;
|
||||
}
|
||||
|
||||
// Non-TTY (agent-driven install, CI, scripted init): print the FULL matrix
|
||||
// and emit an explicit [AGENT] directive. The previous version printed
|
||||
// 2 stderr lines that agents typically miss / don't relay to the human.
|
||||
// Now: the agent sees the same matrix the human-TTY picker shows, plus
|
||||
// a directive saying "show this to your operator and confirm before
|
||||
// moving on." Default-applied mode is tokenmax (preserves v0.31.x shape).
|
||||
if (!process.stdin.isTTY) {
|
||||
try { await engine.setConfig(SEARCH_MODE_KEY, rec.mode); } catch { /* swallow */ }
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log(`[gbrain] search mode tentatively set to: ${rec.mode}`);
|
||||
console.log(`[gbrain] Reason: ${rec.reason}`);
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('[AGENT] Before continuing, SHOW this matrix to your operator and');
|
||||
console.log('[AGENT] ask which mode they want. The default above preserves the');
|
||||
console.log('[AGENT] v0.31.x retrieval shape. Run `gbrain config set search.mode');
|
||||
console.log('[AGENT] <conservative|balanced|tokenmax>` if the operator picks');
|
||||
console.log('[AGENT] differently. See INSTALL_FOR_AGENTS.md Step 3.5 for the');
|
||||
console.log('[AGENT] full protocol.');
|
||||
console.log('');
|
||||
console.log('Per-query cost @ 10K queries/mo (search payload only, no cache savings):');
|
||||
console.log('');
|
||||
console.log(' Haiku 4.5 Sonnet 4.6 Opus 4.7');
|
||||
console.log(' ($1/M input) ($3/M input) ($5/M input)');
|
||||
console.log(' conservative $40/mo $120/mo $200/mo');
|
||||
console.log(' balanced $100/mo $300/mo $500/mo');
|
||||
console.log(' tokenmax $200/mo $600/mo $1,000/mo');
|
||||
console.log('');
|
||||
console.log(' (scales linearly: ×10 for 100K queries/mo, ÷10 for 1K)');
|
||||
console.log(' 25x corner-to-corner spread. Natural diagonal pairings span ~4x.');
|
||||
console.log('');
|
||||
console.log('To change later: gbrain config set search.mode <mode>');
|
||||
console.log('To see what is running: gbrain search modes');
|
||||
console.log('');
|
||||
return rec.mode;
|
||||
}
|
||||
|
||||
// Interactive TTY: menu + readLineSafe.
|
||||
console.log(MENU_TEXT);
|
||||
console.log(` Recommended: ${rec.mode} (${rec.reason})`);
|
||||
console.log('');
|
||||
|
||||
const raw = await readLineSafe(`Mode [${rec.mode}]: `, rec.mode, 60_000);
|
||||
const picked = parseModeInput(raw) ?? rec.mode;
|
||||
|
||||
try {
|
||||
await engine.setConfig(SEARCH_MODE_KEY, picked);
|
||||
} catch (err) {
|
||||
// Worst case: config write fails. Mode resolution falls back to balanced
|
||||
// at search-time anyway, so we don't block init. Emit the failure to
|
||||
// stderr so the operator sees it.
|
||||
console.error(`[gbrain] WARN: failed to persist search.mode (${(err as Error).message ?? 'unknown'}). Defaulting to ${picked} at search-time.`);
|
||||
}
|
||||
|
||||
console.log(`Search mode set to: ${picked}`);
|
||||
console.log('');
|
||||
return picked;
|
||||
}
|
||||
|
||||
/** Re-export the menu text for documentation generation and tests. */
|
||||
export const MODE_PICKER_MENU = MENU_TEXT;
|
||||
export const SUPPORTED_MODES = SEARCH_MODES;
|
||||
@@ -439,6 +439,12 @@ async function initPGLite(opts: {
|
||||
};
|
||||
saveConfig(config);
|
||||
|
||||
// v0.32.3 search-lite install-time mode picker. Runs AFTER initSchema so
|
||||
// DB config writes are valid. Idempotent: skipped on re-init if already set.
|
||||
// Non-TTY auto-selects; --json emits a structured event.
|
||||
const { runModePicker } = await import('./init-mode-picker.ts');
|
||||
await runModePicker(engine, { jsonOutput: opts.jsonOutput });
|
||||
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
@@ -574,6 +580,11 @@ async function initPostgres(opts: {
|
||||
saveConfig(config);
|
||||
console.log('Config saved to ~/.gbrain/config.json');
|
||||
|
||||
// v0.32.3 search-lite install-time mode picker. Same shape as the
|
||||
// PGLite path above — runs AFTER initSchema, idempotent on re-init.
|
||||
const { runModePicker: runPostgresModePicker } = await import('./init-mode-picker.ts');
|
||||
await runPostgresModePicker(engine, { jsonOutput: opts.jsonOutput });
|
||||
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
@@ -663,6 +674,68 @@ function readLine(prompt: string): Promise<string> {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-9]: readLine + EOF detection + default fallback + timeout.
|
||||
*
|
||||
* The legacy readLine hangs forever if stdin closes (EOF mid-prompt) or
|
||||
* the user never types anything. The mode-picker plan calls out "TTY
|
||||
* closes mid-prompt → defaults to balanced" as a failure path, but the
|
||||
* raw helper can't implement that contract.
|
||||
*
|
||||
* This wrapper:
|
||||
* - Resolves to `defaultValue` if stdin emits 'end' before 'data'
|
||||
* - Resolves to `defaultValue` if `timeoutMs` elapses with no input
|
||||
* - Resolves to the typed value (trimmed) on normal data event
|
||||
*
|
||||
* `defaultValue` is returned VERBATIM when the user just hits Enter (empty
|
||||
* data). That's the affordance that makes `Mode [balanced]: _` work.
|
||||
*
|
||||
* Non-TTY stdin (pipe, scripted init) returns defaultValue immediately
|
||||
* without printing the prompt, so e2e tests don't hang.
|
||||
*/
|
||||
export function readLineSafe(
|
||||
prompt: string,
|
||||
defaultValue: string,
|
||||
timeoutMs: number = 60_000,
|
||||
): Promise<string> {
|
||||
return new Promise((resolve) => {
|
||||
// Non-TTY (pipe, redirect, scripted init) → no prompt, no wait.
|
||||
if (!process.stdin.isTTY) {
|
||||
resolve(defaultValue);
|
||||
return;
|
||||
}
|
||||
|
||||
process.stdout.write(prompt);
|
||||
process.stdin.setEncoding('utf-8');
|
||||
|
||||
let settled = false;
|
||||
const finish = (value: string) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
process.stdin.removeListener('data', onData);
|
||||
process.stdin.removeListener('end', onEnd);
|
||||
try { process.stdin.pause(); } catch { /* swallow */ }
|
||||
resolve(value);
|
||||
};
|
||||
|
||||
const onData = (chunk: Buffer | string) => {
|
||||
const raw = chunk.toString().trim();
|
||||
finish(raw.length === 0 ? defaultValue : raw);
|
||||
};
|
||||
const onEnd = () => finish(defaultValue);
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
process.stdout.write(`\n[timeout after ${Math.round(timeoutMs / 1000)}s, using default: ${defaultValue}]\n`);
|
||||
finish(defaultValue);
|
||||
}, timeoutMs);
|
||||
|
||||
process.stdin.once('data', onData);
|
||||
process.stdin.once('end', onEnd);
|
||||
process.stdin.resume();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect GStack installation across known host paths.
|
||||
* Uses gstack-global-discover if available, falls back to path checking.
|
||||
|
||||
@@ -0,0 +1,453 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain search` CLI surface.
|
||||
*
|
||||
* Three sub-subcommands, mirroring `gbrain models` (v0.31.12) for shape
|
||||
* consistency:
|
||||
*
|
||||
* gbrain search modes [--json]
|
||||
* Read-only routing dashboard. Prints the three mode bundles, the
|
||||
* active mode, the source of every resolved knob (mode default vs
|
||||
* config override vs per-call), and a one-liner per knob.
|
||||
*
|
||||
* gbrain search modes --reset [--source <mode>]
|
||||
* Clears every search.* override key (per CDX-8). --source acts as
|
||||
* a dry-run that lists what would change without writing.
|
||||
*
|
||||
* gbrain search stats [--days N] [--json]
|
||||
* Observability. Reads search_telemetry rollup over the window.
|
||||
* Shows hit rate %, intent mix, mode mix, budget pressure, avg
|
||||
* results, avg tokens delivered.
|
||||
*
|
||||
* gbrain search tune [--apply] [--json]
|
||||
* Recommendation engine. Reads stats + brain size + model tier and
|
||||
* prints structured recommendations. --apply mutates config (each
|
||||
* change logged loud + paste-ready revert command at the end).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
MODE_BUNDLES,
|
||||
SEARCH_MODES,
|
||||
SEARCH_MODE_KEY,
|
||||
SEARCH_MODE_CONFIG_KEYS,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
loadSearchModeConfig,
|
||||
resolveSearchMode,
|
||||
attributeKnob,
|
||||
type SearchMode,
|
||||
type ModeBundle,
|
||||
} from '../core/search/mode.ts';
|
||||
import { readSearchStats } from '../core/search/telemetry.ts';
|
||||
|
||||
const KNOB_DESCRIPTIONS: Record<keyof ModeBundle, string> = {
|
||||
cache_enabled: 'Semantic query cache on/off',
|
||||
cache_similarity_threshold: 'Cosine-similarity floor for cache hits (0..1)',
|
||||
cache_ttl_seconds: 'Per-row cache TTL',
|
||||
intentWeighting: 'Zero-LLM intent classifier weight adjustments',
|
||||
tokenBudget: 'Per-call token-budget cap (undefined = no cap)',
|
||||
expansion: 'LLM multi-query expansion (Haiku call per search)',
|
||||
searchLimit: 'Default `limit` for the operation layer',
|
||||
};
|
||||
|
||||
interface SearchModesReport {
|
||||
schema_version: 2;
|
||||
active_mode: SearchMode;
|
||||
active_mode_valid: boolean;
|
||||
resolved: Record<keyof ModeBundle, { value: unknown; source: string; source_detail: string; description: string }>;
|
||||
bundles: Record<SearchMode, ModeBundle>;
|
||||
config_keys: ReadonlyArray<string>;
|
||||
_meta?: {
|
||||
metric_glossary?: Record<string, string>;
|
||||
};
|
||||
}
|
||||
|
||||
async function buildModesReport(engine: BrainEngine): Promise<SearchModesReport> {
|
||||
const input = await loadSearchModeConfig(engine);
|
||||
const resolved = resolveSearchMode(input);
|
||||
|
||||
const knobs: Array<keyof ModeBundle> = [
|
||||
'cache_enabled',
|
||||
'cache_similarity_threshold',
|
||||
'cache_ttl_seconds',
|
||||
'intentWeighting',
|
||||
'tokenBudget',
|
||||
'expansion',
|
||||
'searchLimit',
|
||||
];
|
||||
|
||||
const attributions = {} as SearchModesReport['resolved'];
|
||||
for (const k of knobs) {
|
||||
const a = attributeKnob(k, input, resolved);
|
||||
attributions[k] = {
|
||||
value: a.value,
|
||||
source: a.source,
|
||||
source_detail: a.source_detail,
|
||||
description: KNOB_DESCRIPTIONS[k],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
schema_version: 2,
|
||||
active_mode: resolved.resolved_mode,
|
||||
active_mode_valid: resolved.mode_valid,
|
||||
resolved: attributions,
|
||||
bundles: {
|
||||
conservative: { ...MODE_BUNDLES.conservative },
|
||||
balanced: { ...MODE_BUNDLES.balanced },
|
||||
tokenmax: { ...MODE_BUNDLES.tokenmax },
|
||||
},
|
||||
config_keys: SEARCH_MODE_CONFIG_KEYS,
|
||||
};
|
||||
}
|
||||
|
||||
function formatModesText(report: SearchModesReport): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Search mode (active): ' + report.active_mode + (report.active_mode_valid ? '' : ' (unset — using balanced fallback)'));
|
||||
lines.push('');
|
||||
lines.push('Resolved knobs:');
|
||||
for (const [knob, attr] of Object.entries(report.resolved)) {
|
||||
const value = String(attr.value ?? '(undefined)');
|
||||
lines.push(` ${knob.padEnd(28)} = ${value.padEnd(12)} [${attr.source_detail}]`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push('Mode bundles (frozen — set via `gbrain config set search.mode <mode>`):');
|
||||
for (const mode of SEARCH_MODES) {
|
||||
const b = report.bundles[mode];
|
||||
const active = mode === report.active_mode ? ' ← active' : '';
|
||||
lines.push(` ${mode.padEnd(13)}${active}`);
|
||||
lines.push(` cache=${b.cache_enabled} intentWeighting=${b.intentWeighting}`);
|
||||
lines.push(` tokenBudget=${b.tokenBudget ?? 'none'} searchLimit=${b.searchLimit} expansion=${b.expansion}`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push('Knob descriptions:');
|
||||
for (const [k, desc] of Object.entries(KNOB_DESCRIPTIONS)) {
|
||||
lines.push(` ${k.padEnd(28)} ${desc}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
async function runModesSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const reset = args.includes('--reset');
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const dryRunSource = sourceIdx !== -1 ? args[sourceIdx + 1] : null;
|
||||
|
||||
// --reset path: clear every search.* OVERRIDE key (not search.mode itself).
|
||||
// --source <mode> is a dry-run that prints what would change.
|
||||
if (reset || dryRunSource) {
|
||||
const dryRun = Boolean(dryRunSource);
|
||||
if (dryRunSource && !isSearchMode(dryRunSource)) {
|
||||
console.error(`Invalid --source value: ${dryRunSource}. Expected one of: ${SEARCH_MODES.join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
const overrides = await engine.listConfigKeys('search.');
|
||||
const toRemove = overrides.filter((k) => k !== SEARCH_MODE_KEY && k !== 'search.mode_upgrade_notice_shown');
|
||||
if (toRemove.length === 0) {
|
||||
console.log('No search.* overrides set. Mode bundle is the only voice.');
|
||||
return;
|
||||
}
|
||||
if (dryRun) {
|
||||
console.log(`--source ${dryRunSource} (dry run). Would unset ${toRemove.length} key(s):`);
|
||||
for (const k of toRemove) console.log(` - ${k}`);
|
||||
console.log(`No changes written. Re-run with --reset to apply.`);
|
||||
return;
|
||||
}
|
||||
let deleted = 0;
|
||||
for (const k of toRemove) {
|
||||
const n = await engine.unsetConfig(k);
|
||||
deleted += n;
|
||||
}
|
||||
console.log(`Reset complete. Unset ${deleted} key(s):`);
|
||||
for (const k of toRemove) console.log(` - ${k}`);
|
||||
console.log(`Mode bundle is now the only voice. Verify with: gbrain search modes`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Default: read-only dashboard.
|
||||
const report = await buildModesReport(engine);
|
||||
if (json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
console.log(formatModesText(report));
|
||||
}
|
||||
}
|
||||
|
||||
async function runStatsSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const daysIdx = args.indexOf('--days');
|
||||
const days = daysIdx !== -1 ? parseInt(args[daysIdx + 1], 10) : 7;
|
||||
|
||||
const stats = await readSearchStats(engine, { days: Number.isFinite(days) ? days : 7 });
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
...stats,
|
||||
_meta: {
|
||||
metric_glossary: {
|
||||
cache_hit_rate: 'cache_hits / (cache_hits + cache_misses) — fraction of searches that reused a recent answer instead of running fresh',
|
||||
avg_results: 'mean number of result rows returned per search call',
|
||||
avg_tokens: 'mean estimated tokens in the returned chunk text (char/4 heuristic)',
|
||||
total_budget_dropped: 'sum of results dropped because the call exceeded its tokenBudget',
|
||||
},
|
||||
},
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Search stats over the last ${stats.window_days} days:`);
|
||||
console.log('');
|
||||
console.log(` Total searches: ${stats.total_calls}`);
|
||||
if (stats.total_calls === 0) {
|
||||
console.log('');
|
||||
console.log('No telemetry recorded yet. Run a few `gbrain query` calls and re-check.');
|
||||
return;
|
||||
}
|
||||
const hitRatePct = (stats.cache_hit_rate * 100).toFixed(1);
|
||||
console.log(` Cache hit rate: ${hitRatePct}% (${stats.cache_hits} hit / ${stats.cache_misses} miss)`);
|
||||
console.log(` (fraction of searches that reused a recent answer)`);
|
||||
console.log(` Avg results returned: ${stats.avg_results.toFixed(1)}`);
|
||||
console.log(` Avg tokens delivered: ${stats.avg_tokens.toFixed(0)} (char/4 heuristic)`);
|
||||
console.log(` Budget drops total: ${stats.total_budget_dropped}`);
|
||||
console.log('');
|
||||
console.log(' Mode distribution:');
|
||||
for (const [m, c] of Object.entries(stats.mode_distribution).sort((a, b) => b[1] - a[1])) {
|
||||
const pct = ((c / stats.total_calls) * 100).toFixed(1);
|
||||
console.log(` ${m.padEnd(14)} ${c} (${pct}%)`);
|
||||
}
|
||||
console.log('');
|
||||
console.log(' Intent distribution:');
|
||||
for (const [i, c] of Object.entries(stats.intent_distribution).sort((a, b) => b[1] - a[1])) {
|
||||
const pct = ((c / stats.total_calls) * 100).toFixed(1);
|
||||
console.log(` ${i.padEnd(14)} ${c} (${pct}%)`);
|
||||
}
|
||||
if (stats.oldest_seen || stats.newest_seen) {
|
||||
console.log('');
|
||||
console.log(` Window: ${stats.oldest_seen ?? '?'} → ${stats.newest_seen ?? '?'}`);
|
||||
}
|
||||
}
|
||||
|
||||
interface TuneRecommendation {
|
||||
knob: string;
|
||||
current: unknown;
|
||||
suggested: unknown;
|
||||
reason: string;
|
||||
apply_command: string;
|
||||
}
|
||||
|
||||
async function runTuneSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const apply = args.includes('--apply');
|
||||
|
||||
const modeInput = await loadSearchModeConfig(engine);
|
||||
const resolved = resolveSearchMode(modeInput);
|
||||
const stats = await readSearchStats(engine, { days: 7 });
|
||||
|
||||
const recs: TuneRecommendation[] = [];
|
||||
|
||||
// Recommendation 1: low call volume → no data yet.
|
||||
if (stats.total_calls < 20) {
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
status: 'insufficient_data',
|
||||
total_calls: stats.total_calls,
|
||||
recommendations: [],
|
||||
message: 'Not enough search activity in the last 7 days to tune. Run `gbrain search stats` after some real usage.',
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log('Not enough search activity in the last 7 days to tune.');
|
||||
console.log(`Total searches: ${stats.total_calls} (need >= 20 for confident recommendations).`);
|
||||
console.log('Run a few `gbrain query` calls, then re-run `gbrain search tune`.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Recommendation 2: budget pressure under conservative.
|
||||
if (resolved.resolved_mode === 'conservative' && stats.total_calls > 0) {
|
||||
const dropPctPerCall = stats.total_budget_dropped / stats.total_calls;
|
||||
if (dropPctPerCall > 2) {
|
||||
recs.push({
|
||||
knob: 'search.mode',
|
||||
current: 'conservative',
|
||||
suggested: 'balanced',
|
||||
reason: `Avg ${dropPctPerCall.toFixed(1)} results dropped per search by the 4K budget. Consider balanced (12K budget) or raise search.tokenBudget.`,
|
||||
apply_command: 'gbrain config set search.mode balanced',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Recommendation 3: high cache hit rate → bump similarity threshold.
|
||||
if (stats.cache_hit_rate > 0.85 && stats.cache_hits + stats.cache_misses > 50) {
|
||||
recs.push({
|
||||
knob: 'search.cache.similarity_threshold',
|
||||
current: resolved.cache_similarity_threshold,
|
||||
suggested: 0.94,
|
||||
reason: `Cache hit rate is ${(stats.cache_hit_rate * 100).toFixed(1)}%. You can raise similarity threshold to 0.94 for tighter freshness at small recall cost.`,
|
||||
apply_command: 'gbrain config set search.cache.similarity_threshold 0.94',
|
||||
});
|
||||
}
|
||||
|
||||
// Recommendation 4: tokenmax + Haiku subagent.
|
||||
const subagentModel = await engine.getConfig('models.tier.subagent');
|
||||
if (resolved.resolved_mode === 'tokenmax' && subagentModel && /haiku/i.test(subagentModel)) {
|
||||
recs.push({
|
||||
knob: 'search.mode',
|
||||
current: 'tokenmax',
|
||||
suggested: 'balanced',
|
||||
reason: `Subagent tier is Haiku but mode is tokenmax. LLM expansion adds ~50ms + ~1¢ per query. Balanced cuts that cost without losing intent weighting or cache.`,
|
||||
apply_command: 'gbrain config set search.mode balanced',
|
||||
});
|
||||
}
|
||||
|
||||
// Recommendation 5: cache disabled but available — fix the free win.
|
||||
if (!resolved.cache_enabled && stats.total_calls > 5) {
|
||||
recs.push({
|
||||
knob: 'search.cache.enabled',
|
||||
current: false,
|
||||
suggested: true,
|
||||
reason: 'Cache is disabled but mode bundles enable it by default. Cache is a free win (zero LLM cost, big latency drop on repeat queries).',
|
||||
apply_command: 'gbrain config unset search.cache.enabled',
|
||||
});
|
||||
}
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
status: recs.length === 0 ? 'no_recommendations' : 'has_recommendations',
|
||||
total_calls: stats.total_calls,
|
||||
cache_hit_rate: stats.cache_hit_rate,
|
||||
active_mode: resolved.resolved_mode,
|
||||
recommendations: recs,
|
||||
applied: apply ? recs.map(r => r.apply_command) : [],
|
||||
_meta: {
|
||||
metric_glossary: {
|
||||
cache_hit_rate: 'cache_hits / (cache_hits + cache_misses)',
|
||||
total_calls: 'total searches recorded in the last 7 days',
|
||||
},
|
||||
},
|
||||
}, null, 2));
|
||||
if (apply) {
|
||||
for (const r of recs) {
|
||||
await maybeApplyRecommendation(engine, r);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Search tune (last 7 days, active mode: ${resolved.resolved_mode}):`);
|
||||
console.log('');
|
||||
|
||||
if (recs.length === 0) {
|
||||
console.log(' No recommendations. Your search config looks well-tuned.');
|
||||
return;
|
||||
}
|
||||
|
||||
for (let i = 0; i < recs.length; i++) {
|
||||
const r = recs[i];
|
||||
console.log(` ${i + 1}. ${r.knob}: ${String(r.current)} → ${String(r.suggested)}`);
|
||||
console.log(` ${r.reason}`);
|
||||
console.log(` Apply: ${r.apply_command}`);
|
||||
console.log('');
|
||||
}
|
||||
|
||||
if (apply) {
|
||||
console.log('Applying recommendations:');
|
||||
const reverts: string[] = [];
|
||||
for (const r of recs) {
|
||||
await maybeApplyRecommendation(engine, r);
|
||||
console.log(` ✓ ${r.apply_command}`);
|
||||
reverts.push(buildRevertCommand(r));
|
||||
}
|
||||
console.log('');
|
||||
console.log('To revert these changes:');
|
||||
for (const cmd of reverts) console.log(` ${cmd}`);
|
||||
} else {
|
||||
console.log('Run `gbrain search tune --apply` to apply these changes automatically.');
|
||||
}
|
||||
}
|
||||
|
||||
async function maybeApplyRecommendation(engine: BrainEngine, r: TuneRecommendation): Promise<void> {
|
||||
// The apply command is the canonical paste-ready string; here we
|
||||
// re-parse it to call setConfig / unsetConfig directly so the call
|
||||
// happens in-process.
|
||||
const parts = r.apply_command.split(/\s+/);
|
||||
if (parts[0] !== 'gbrain' || parts[1] !== 'config') return;
|
||||
if (parts[2] === 'set' && parts.length === 5) {
|
||||
await engine.setConfig(parts[3], parts[4]);
|
||||
} else if (parts[2] === 'unset' && parts.length === 4) {
|
||||
await engine.unsetConfig(parts[3]);
|
||||
}
|
||||
}
|
||||
|
||||
function buildRevertCommand(r: TuneRecommendation): string {
|
||||
const parts = r.apply_command.split(/\s+/);
|
||||
if (parts[2] === 'set') {
|
||||
return `gbrain config set ${parts[3]} ${String(r.current)}`;
|
||||
} else if (parts[2] === 'unset') {
|
||||
return `gbrain config set ${parts[3]} ${String(r.current)}`;
|
||||
}
|
||||
return r.apply_command;
|
||||
}
|
||||
|
||||
const USAGE = `Usage: gbrain search <modes|stats|tune> [flags]
|
||||
|
||||
Subcommands:
|
||||
modes [--json] Show active mode, bundles, and per-knob source.
|
||||
modes --reset Clear all search.* overrides (mode bundle wins).
|
||||
modes --source <mode> Dry-run: list what --reset would change.
|
||||
stats [--days N] [--json] Cache hit rate, intent mix, budget pressure.
|
||||
tune [--apply] [--json] Print recommendations; --apply mutates config.
|
||||
|
||||
Examples:
|
||||
gbrain search modes
|
||||
gbrain search modes --reset
|
||||
gbrain search stats --days 30 --json
|
||||
gbrain search tune
|
||||
gbrain search tune --apply
|
||||
`;
|
||||
|
||||
export async function runSearch(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
const rest = args.slice(1);
|
||||
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(USAGE);
|
||||
return;
|
||||
}
|
||||
|
||||
switch (sub) {
|
||||
case 'modes':
|
||||
await runModesSubcommand(engine, rest);
|
||||
return;
|
||||
case 'stats':
|
||||
await runStatsSubcommand(engine, rest);
|
||||
return;
|
||||
case 'tune':
|
||||
await runTuneSubcommand(engine, rest);
|
||||
return;
|
||||
default:
|
||||
console.error(`Unknown subcommand: ${sub}`);
|
||||
console.error(USAGE);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* `gbrain search modes` is read-only — no DB connection strictly required
|
||||
* for the bundle display IF the engine is given. The dispatch in cli.ts
|
||||
* adds 'search' to its dispatch table so the engine connects normally;
|
||||
* this export is here so future no-engine modes (e.g. `gbrain search --help`
|
||||
* without an engine) could route through it cleanly.
|
||||
*/
|
||||
export const _exports_for_test = {
|
||||
buildModesReport,
|
||||
formatModesText,
|
||||
maybeApplyRecommendation,
|
||||
buildRevertCommand,
|
||||
};
|
||||
|
||||
// Suppress unused-export TS warning — these are intentionally retained for
|
||||
// downstream callers (cli.ts dispatch / future skill linkage).
|
||||
void DEFAULT_SEARCH_MODE;
|
||||
@@ -257,6 +257,53 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
|
||||
await engine.initSchema();
|
||||
console.log(' Schema up to date.');
|
||||
|
||||
// v0.32.3 search-lite mode banner. One-shot: fires at most once per
|
||||
// install (state persisted via `search.mode_upgrade_notice_shown`).
|
||||
// Reframes from "behavior is regressing" to "named modes available"
|
||||
// per [CDX-1+2+3]: the production query op still defaults expand=true
|
||||
// and limit=20 — Garry's behavior is NOT regressing.
|
||||
try {
|
||||
const shown = await engine.getConfig('search.mode_upgrade_notice_shown');
|
||||
const existingMode = await engine.getConfig('search.mode');
|
||||
if (shown !== 'true' && !existingMode) {
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('[gbrain] v0.32.3 added named search modes (conservative/balanced/tokenmax).');
|
||||
console.log('[gbrain] Your current behavior is unchanged — the `query` op still defaults');
|
||||
console.log('[gbrain] expand=true and limit=20, same as v0.31.x.');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('[AGENT] Relay this matrix to your operator and ask which mode they want.');
|
||||
console.log('[AGENT] Recommended default: tokenmax (preserves current retrieval shape).');
|
||||
console.log('[AGENT] Run `gbrain config set search.mode <mode>` after they decide.');
|
||||
console.log('');
|
||||
console.log('Per-query cost @ 10K queries/mo (typical single-user volume):');
|
||||
console.log('');
|
||||
console.log(' Haiku 4.5 Sonnet 4.6 Opus 4.7');
|
||||
console.log(' ($1/M) ($3/M) ($5/M)');
|
||||
console.log(' conservative $40/mo $120/mo $200/mo');
|
||||
console.log(' balanced $100/mo $300/mo $500/mo');
|
||||
console.log(' tokenmax $200/mo $600/mo $1,000/mo');
|
||||
console.log('');
|
||||
console.log(' (scales linearly — multiply by 10 for 100K/mo)');
|
||||
console.log(' 25x corner-to-corner spread. Natural diagonal pairings span ~4x.');
|
||||
console.log('');
|
||||
console.log('To pick:');
|
||||
console.log(' gbrain search modes # see what is running');
|
||||
console.log(' gbrain config set search.mode <conservative|balanced|tokenmax>');
|
||||
console.log(' gbrain search tune # data-driven recommendations');
|
||||
console.log('');
|
||||
console.log('tokenmax bumps limit to 50 (current default is 20). To preserve');
|
||||
console.log('your EXACT current shape:');
|
||||
console.log(' gbrain config set search.mode tokenmax');
|
||||
console.log(' gbrain config set search.searchLimit 20');
|
||||
console.log('');
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
}
|
||||
} catch {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// v0.32.7 CJK wave: chunker-version bump → re-embed sweep.
|
||||
// Idempotent — `runReindex` short-circuits when no pages are pending.
|
||||
try {
|
||||
|
||||
@@ -1147,6 +1147,18 @@ export interface BrainEngine {
|
||||
// Config
|
||||
getConfig(key: string): Promise<string | null>;
|
||||
setConfig(key: string, value: string): Promise<void>;
|
||||
/**
|
||||
* v0.32.3 — delete a config row. Returns the number of rows deleted (0 or 1).
|
||||
* No-op when the key doesn't exist. Used by `gbrain config unset` and by
|
||||
* `gbrain search modes --reset`. Engine-agnostic.
|
||||
*/
|
||||
unsetConfig(key: string): Promise<number>;
|
||||
/**
|
||||
* v0.32.3 — list config keys matching a literal prefix (e.g. "search.").
|
||||
* Used by `gbrain config unset --pattern` and the search-modes --reset path.
|
||||
* Does NOT support glob/regex on purpose — the caller knows the prefix.
|
||||
*/
|
||||
listConfigKeys(prefix: string): Promise<string[]>;
|
||||
|
||||
// Migration support
|
||||
runMigration(version: number, sql: string): Promise<void>;
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* v0.32.3 — curated drift-watch list for the eval_drift doctor check.
|
||||
*
|
||||
* Per [CDX-6]: a "search code changed since last eval" warning needs a
|
||||
* precise definition. Too narrow (e.g. only src/core/search/) misses real
|
||||
* regressions like a chunker change. Too wide (every file) trains the
|
||||
* operator to ignore the warning.
|
||||
*
|
||||
* The curated allowlist below names every file whose change MEANINGFULLY
|
||||
* affects retrieval quality. Adding to this list REQUIRES a CHANGELOG line
|
||||
* so coverage grows deliberately.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { existsSync } from 'fs';
|
||||
|
||||
/**
|
||||
* Glob-ish patterns watched for retrieval drift. Each pattern is matched
|
||||
* against repo-relative paths via simple `startsWith` semantics (no real
|
||||
* glob expansion) so the matcher is fast + dependency-free.
|
||||
*
|
||||
* If you add a pattern: also add a CHANGELOG line documenting why.
|
||||
*/
|
||||
export const RETRIEVAL_WATCH_PATTERNS: ReadonlyArray<string> = Object.freeze([
|
||||
// Search pipeline core
|
||||
'src/core/search/',
|
||||
// Embedding shape (changing dim or chunker shape moves every result)
|
||||
'src/core/embedding.ts',
|
||||
// Chunkers (recursive + semantic + LLM-guided) — chunk granularity is retrieval
|
||||
'src/core/chunkers/',
|
||||
// AI recipes that drive expansion / embedding choices
|
||||
'src/core/ai/recipes/anthropic.ts',
|
||||
'src/core/ai/recipes/openai.ts',
|
||||
// The query op itself
|
||||
'src/core/operations.ts',
|
||||
]);
|
||||
|
||||
/** Path equality / prefix matcher for the curated list. */
|
||||
export function matchesWatchPattern(path: string, patterns: ReadonlyArray<string> = RETRIEVAL_WATCH_PATTERNS): boolean {
|
||||
for (const p of patterns) {
|
||||
// Trailing-slash pattern = directory prefix
|
||||
if (p.endsWith('/')) {
|
||||
if (path.startsWith(p)) return true;
|
||||
} else {
|
||||
// Bare-file pattern = exact equality
|
||||
if (path === p) return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return repo-relative paths that have changed in the working tree since
|
||||
* the given commit (or HEAD if no commit). Best-effort: returns [] when
|
||||
* git is unavailable, the repo lacks the commit, or any other failure.
|
||||
*
|
||||
* `commitSha` is a full or short SHA. When omitted, compares HEAD against
|
||||
* working tree (uncommitted changes only).
|
||||
*/
|
||||
export function filesDriftedSince(repoRoot: string, commitSha?: string): string[] {
|
||||
if (!existsSync(repoRoot)) return [];
|
||||
try {
|
||||
const range = commitSha ? `${commitSha}..HEAD` : 'HEAD';
|
||||
const args = commitSha
|
||||
? ['diff', '--name-only', range]
|
||||
: ['diff', '--name-only', 'HEAD'];
|
||||
const out = execSync(`git ${args.join(' ')}`, {
|
||||
cwd: repoRoot,
|
||||
encoding: 'utf-8',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
timeout: 5000,
|
||||
});
|
||||
return out
|
||||
.split('\n')
|
||||
.map(s => s.trim())
|
||||
.filter((s): s is string => s.length > 0);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Identify only the changed files that match the retrieval watch list.
|
||||
* Convenience wrapper for the doctor check + future CI gate.
|
||||
*/
|
||||
export function watchedFilesDrifted(
|
||||
repoRoot: string,
|
||||
commitSha?: string,
|
||||
patterns: ReadonlyArray<string> = RETRIEVAL_WATCH_PATTERNS,
|
||||
): string[] {
|
||||
return filesDriftedSince(repoRoot, commitSha).filter((p) => matchesWatchPattern(p, patterns));
|
||||
}
|
||||
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* v0.32.3 — Single source of truth for evaluation-metric plain-English
|
||||
* glosses. Drives `gbrain search stats` JSON output (`_meta.metric_glossary`
|
||||
* block), `gbrain eval compare` reports, and the auto-generated
|
||||
* `docs/eval/METRIC_GLOSSARY.md` file.
|
||||
*
|
||||
* Per [CDX-25]: glosses live in one `_meta.metric_glossary` block per
|
||||
* response, NOT as sibling `_gloss` fields on every metric. Less invasive
|
||||
* to machine-readable consumers (gbrain-evals repo, CI gates).
|
||||
*
|
||||
* Every entry has THREE fields:
|
||||
* - industry_term: the canonical name used in IR / NLP literature
|
||||
* (preserved verbatim so users searching the literature find what we
|
||||
* report)
|
||||
* - eli10: plain-English explanation a 16-year-old could follow
|
||||
* - range: explicit numeric range + interpretation ("higher is better",
|
||||
* "0..1 where 1 = perfect")
|
||||
*
|
||||
* The doc generator at `scripts/generate-metric-glossary.ts` consumes this
|
||||
* module and writes `docs/eval/METRIC_GLOSSARY.md`. A CI guard
|
||||
* (`scripts/check-eval-glossary-fresh.sh`) regenerates the doc and diffs
|
||||
* against the committed version — out-of-date docs fail the build.
|
||||
*/
|
||||
|
||||
export interface MetricGlossEntry {
|
||||
industry_term: string;
|
||||
eli10: string;
|
||||
range: string;
|
||||
}
|
||||
|
||||
export const METRIC_GLOSSARY: Readonly<Record<string, Readonly<MetricGlossEntry>>> = Object.freeze({
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Retrieval metrics (IR literature)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'precision@k': Object.freeze({
|
||||
industry_term: 'Precision at k (P@k)',
|
||||
eli10: 'Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.',
|
||||
range: '0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.',
|
||||
}),
|
||||
'recall@k': Object.freeze({
|
||||
industry_term: 'Recall at k (R@k)',
|
||||
eli10: 'Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.',
|
||||
range: '0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.',
|
||||
}),
|
||||
'mrr': Object.freeze({
|
||||
industry_term: 'Mean Reciprocal Rank (MRR)',
|
||||
eli10: 'On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it\'s typically at rank 2.',
|
||||
range: '0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.',
|
||||
}),
|
||||
'ndcg@k': Object.freeze({
|
||||
industry_term: 'Normalized Discounted Cumulative Gain at k (nDCG@k)',
|
||||
eli10: 'Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.',
|
||||
range: '0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Set-similarity / stability metrics (replay + regression checks)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'jaccard@k': Object.freeze({
|
||||
industry_term: 'Jaccard similarity at k (set Jaccard @k)',
|
||||
eli10: 'How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.',
|
||||
range: '0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.',
|
||||
}),
|
||||
'top1_stability': Object.freeze({
|
||||
industry_term: 'Top-1 stability rate',
|
||||
eli10: 'Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don\'t change the top answer don\'t hurt it.',
|
||||
range: '0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Statistical-significance metrics (per-mode comparison)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'p_value': Object.freeze({
|
||||
industry_term: 'p-value (paired bootstrap)',
|
||||
eli10: 'How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).',
|
||||
range: '0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.',
|
||||
}),
|
||||
'confidence_interval': Object.freeze({
|
||||
industry_term: '95% Confidence Interval (CI)',
|
||||
eli10: 'The range we\'re 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.',
|
||||
range: 'Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn\'t statistically significant.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Operational / cost metrics
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'cache_hit_rate': Object.freeze({
|
||||
industry_term: 'Cache hit rate',
|
||||
eli10: 'Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.',
|
||||
range: '0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.',
|
||||
}),
|
||||
'avg_results': Object.freeze({
|
||||
industry_term: 'Average results returned',
|
||||
eli10: 'Mean number of search-result rows the engine returned per call. Should be near the active mode\'s searchLimit unless the brain is small or the budget is dropping results.',
|
||||
range: '0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.',
|
||||
}),
|
||||
'avg_tokens': Object.freeze({
|
||||
industry_term: 'Average tokens delivered',
|
||||
eli10: 'Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.',
|
||||
range: '0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.',
|
||||
}),
|
||||
'cost_per_query_usd': Object.freeze({
|
||||
industry_term: 'Cost per query (USD)',
|
||||
eli10: 'Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.',
|
||||
range: '0..unbounded. Conservative mode is typically <\\$0.001 per call; tokenmax with answer-gen can exceed \\$0.01.',
|
||||
}),
|
||||
'p99_latency_ms': Object.freeze({
|
||||
industry_term: 'p99 latency (ms)',
|
||||
eli10: '99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.',
|
||||
range: '0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.',
|
||||
}),
|
||||
});
|
||||
|
||||
/**
|
||||
* Public accessor — returns the gloss entry for a metric, or null if
|
||||
* unknown. Callers that need the structured shape use this; callers that
|
||||
* just need the plain-English line use eli10For().
|
||||
*/
|
||||
export function getMetricGloss(metric: string): MetricGlossEntry | null {
|
||||
if (METRIC_GLOSSARY[metric]) return METRIC_GLOSSARY[metric];
|
||||
// Fuzzy fallback for @N metrics: `recall@10` → `recall@k`, `ndcg@5` → `ndcg@k`.
|
||||
// The glossary documents the family ("at k"); reports use a specific K value.
|
||||
const atK = metric.match(/^(.+)@\d+$/);
|
||||
if (atK) {
|
||||
const family = `${atK[1]}@k`;
|
||||
if (METRIC_GLOSSARY[family]) return METRIC_GLOSSARY[family];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience: return ONLY the plain-English gloss for a metric. Used in
|
||||
* `gbrain search stats` JSON output's _meta.metric_glossary block and in
|
||||
* the eval-compare report's per-metric "Plain English:" lines.
|
||||
*/
|
||||
export function eli10For(metric: string): string | null {
|
||||
const g = getMetricGloss(metric);
|
||||
return g?.eli10 ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `_meta.metric_glossary` block for a set of metrics. Returns an
|
||||
* object suitable for JSON.stringify-ing under the `_meta` key in any
|
||||
* eval / stats response.
|
||||
*
|
||||
* Per [CDX-25]: ONE _meta.metric_glossary per response, NOT sibling
|
||||
* _gloss fields on every numeric metric. Adding a metric to the response
|
||||
* doesn't bloat the JSON; the glossary lives in a single place per
|
||||
* response, indexed by metric name.
|
||||
*/
|
||||
export function buildMetricGlossaryMeta(metrics: ReadonlyArray<string>): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const m of metrics) {
|
||||
const e = getMetricGloss(m); // routes through fuzzy fallback
|
||||
if (e) out[m] = e.eli10;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The list of EVERY metric this module documents. Auto-derived from the
|
||||
* METRIC_GLOSSARY keys so the doc generator can iterate without drift.
|
||||
*/
|
||||
export const ALL_METRICS: ReadonlyArray<string> = Object.freeze(Object.keys(METRIC_GLOSSARY));
|
||||
|
||||
/**
|
||||
* Render the glossary as a Markdown document. Consumed by
|
||||
* `scripts/generate-metric-glossary.ts` to produce
|
||||
* `docs/eval/METRIC_GLOSSARY.md`. The CI guard regenerates this and
|
||||
* diffs against the committed file — out-of-date docs fail the build.
|
||||
*
|
||||
* The output is deterministic: same input → same output bytes.
|
||||
*/
|
||||
export function renderMetricGlossaryMarkdown(): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('# Evaluation Metric Glossary');
|
||||
lines.push('');
|
||||
lines.push('**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.');
|
||||
lines.push('');
|
||||
lines.push('Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.');
|
||||
lines.push('');
|
||||
|
||||
const groups: Array<[string, string[]]> = [
|
||||
['Retrieval Metrics', ['precision@k', 'recall@k', 'mrr', 'ndcg@k']],
|
||||
['Set-Similarity / Stability Metrics', ['jaccard@k', 'top1_stability']],
|
||||
['Statistical-Significance Metrics', ['p_value', 'confidence_interval']],
|
||||
['Operational / Cost Metrics', ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']],
|
||||
];
|
||||
|
||||
for (const [groupTitle, metrics] of groups) {
|
||||
lines.push(`## ${groupTitle}`);
|
||||
lines.push('');
|
||||
for (const m of metrics) {
|
||||
const e = METRIC_GLOSSARY[m];
|
||||
if (!e) continue;
|
||||
lines.push(`### ${e.industry_term}`);
|
||||
lines.push('');
|
||||
lines.push(`**Key:** \`${m}\``);
|
||||
lines.push('');
|
||||
lines.push(`**Plain English:** ${e.eli10}`);
|
||||
lines.push('');
|
||||
lines.push(`**Range:** ${e.range}`);
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
|
||||
lines.push('---');
|
||||
lines.push('');
|
||||
lines.push('## Coverage');
|
||||
lines.push('');
|
||||
lines.push(`Every metric printed by any \`gbrain eval *\` or \`gbrain search stats\` command resolves through \`getMetricGloss()\` in \`src/core/eval/metric-glossary.ts\`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.`);
|
||||
lines.push('');
|
||||
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
@@ -2704,6 +2704,182 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON pages (source_path) WHERE source_path IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 55,
|
||||
name: 'query_cache_search_lite',
|
||||
// v0.32.x (search-lite, originally claimed v52 in PR #897; renumbered
|
||||
// to v55 on merge with master to sit after eval_contradictions_cache (v52),
|
||||
// eval_contradictions_runs (v53), cjk_wave (v54)).
|
||||
//
|
||||
// Semantic query cache. Cache search results keyed by query embedding
|
||||
// similarity so a near-duplicate query reuses the previous result set
|
||||
// instead of re-running keyword + vector + RRF + dedup. Cache lookup:
|
||||
// `embedding <=> $1 < 0.08` (cosine distance, similarity >= 0.92) using HNSW.
|
||||
//
|
||||
// Schema:
|
||||
// id — SHA-256(query_text + source_id) for diagnostics.
|
||||
// query_text — the raw query for debug + cache-stats output.
|
||||
// source_id — scope by source so multi-source brains don't bleed.
|
||||
// embedding — the query embedding. Same dim as content_chunks.
|
||||
// results — JSONB array of SearchResult rows.
|
||||
// meta — JSONB; what hybridSearch actually did (intent,
|
||||
// vector_enabled, etc.) so cached responses can
|
||||
// surface the same debug info as fresh ones.
|
||||
// ttl_seconds — per-row TTL. Default 3600. Stale rows are skipped
|
||||
// at read time and pruned by `gbrain cache prune`.
|
||||
// created_at — TTL anchor.
|
||||
// hit_count — instrumentation; bumped on each lookup-hit.
|
||||
// last_hit_at — instrumentation.
|
||||
//
|
||||
// Schema is engine-agnostic: HALFVEC when available (matches the facts
|
||||
// table from v45 for consistency), otherwise VECTOR. Embedding dim is
|
||||
// resolved from `config.embedding_dimensions` at migration time so
|
||||
// non-OpenAI brains work — same approach as v45.
|
||||
sql: '',
|
||||
handler: async (engine: BrainEngine) => {
|
||||
// Step 1: resolve embedding dim from config table (same pattern as v45).
|
||||
let embeddingDim = 1536;
|
||||
try {
|
||||
const dimRows = await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
);
|
||||
if (dimRows.length > 0) {
|
||||
const parsed = parseInt(dimRows[0].value, 10);
|
||||
if (Number.isFinite(parsed) && parsed > 0 && parsed <= 4096) {
|
||||
embeddingDim = parsed;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// No config row yet — fall back to default.
|
||||
}
|
||||
|
||||
// Step 2: pgvector version probe for HALFVEC. Same logic as v45.
|
||||
// We deliberately mirror v45's facts table approach for consistency.
|
||||
let useHalfvec = false;
|
||||
if (engine.kind === 'postgres') {
|
||||
try {
|
||||
const vrows = await engine.executeRaw<{ extversion: string }>(
|
||||
`SELECT extversion FROM pg_extension WHERE extname = 'vector'`,
|
||||
);
|
||||
if (vrows.length > 0) {
|
||||
const v = vrows[0].extversion;
|
||||
const parts = v.split('.');
|
||||
const major = parseInt(parts[0] ?? '0', 10);
|
||||
const minor = parseInt(parts[1] ?? '0', 10);
|
||||
if (major > 0 || (major === 0 && minor >= 7)) {
|
||||
useHalfvec = true;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Probe failed — fall back to VECTOR.
|
||||
}
|
||||
} else {
|
||||
useHalfvec = true;
|
||||
}
|
||||
|
||||
const vecType = useHalfvec ? 'HALFVEC' : 'VECTOR';
|
||||
const opclass = useHalfvec ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
|
||||
|
||||
const ddl = `
|
||||
CREATE TABLE IF NOT EXISTS query_cache (
|
||||
id TEXT PRIMARY KEY,
|
||||
query_text TEXT NOT NULL,
|
||||
source_id TEXT NOT NULL DEFAULT 'default',
|
||||
embedding ${vecType}(${embeddingDim}),
|
||||
results JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
meta JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
ttl_seconds INTEGER NOT NULL DEFAULT 3600,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
hit_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_hit_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_source_created
|
||||
ON query_cache(source_id, created_at DESC);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_embedding_hnsw
|
||||
ON query_cache USING hnsw (embedding ${opclass})
|
||||
WHERE embedding IS NOT NULL;
|
||||
`;
|
||||
|
||||
await engine.runMigration(55, ddl);
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 56,
|
||||
name: 'query_cache_knobs_hash',
|
||||
// v0.32.3 search-lite mode cache contamination hotfix [CDX-4].
|
||||
//
|
||||
// PR #897's query_cache keyed rows on (id, source_id, query_text) only.
|
||||
// The `id` is sha256(source_id::query_text). A tokenmax search
|
||||
// (expansion=on, limit=50) populates a row that a subsequent
|
||||
// conservative call (no-expansion, limit=10) reads back, serving
|
||||
// expanded-and-oversized results to a budget-tight context.
|
||||
//
|
||||
// Fix: extend the row key with a knobs_hash derived from the resolved
|
||||
// search mode bundle. Lookup filters `WHERE knobs_hash = $1 AND
|
||||
// embedding similarity < threshold`. Existing rows have NULL
|
||||
// knobs_hash and are treated as misses (silently re-populated with
|
||||
// the correct hash on first hit — no orphan data, no destructive
|
||||
// migration).
|
||||
//
|
||||
// The PRIMARY KEY stays the existing `id` column (the SHA-256 of
|
||||
// (source_id, query_text, knobs_hash) — the cache code re-derives
|
||||
// it on every write, so a tokenmax write and a conservative write
|
||||
// produce distinct `id` values and live as separate rows).
|
||||
//
|
||||
// Engine-agnostic; idempotent.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE query_cache ADD COLUMN IF NOT EXISTS knobs_hash TEXT;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_source_knobs_created
|
||||
ON query_cache(source_id, knobs_hash, created_at DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 57,
|
||||
name: 'search_telemetry_rollup',
|
||||
// v0.32.3 search-lite: per-day rollup of search-call shape.
|
||||
//
|
||||
// Powers `gbrain search stats [--days N]` and `gbrain search tune` so an
|
||||
// operator (or an agent calling tune) can reason about hit rate, intent
|
||||
// mix, budget pressure, and result-volume averages WITHOUT pulling
|
||||
// per-call rows.
|
||||
//
|
||||
// Schema math per [CDX-17]: sums + counts only, NOT averages. Read-time
|
||||
// derives averages so concurrent ON CONFLICT writes from multiple gbrain
|
||||
// processes accumulate correctly.
|
||||
//
|
||||
// Date-bucketed cache hit/miss per [CDX-18] — query_cache.hit_count is
|
||||
// a LIFETIME counter and can't be sliced by --days. The telemetry table
|
||||
// is the truth for windowed hit rate.
|
||||
//
|
||||
// PK is (date, mode, intent) so the rollup never grows past
|
||||
// 365 days × 3 modes × 4 intents = ~4380 rows/year. Acceptable.
|
||||
//
|
||||
// Engine-agnostic; idempotent.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS search_telemetry (
|
||||
date TEXT NOT NULL,
|
||||
mode TEXT NOT NULL,
|
||||
intent TEXT NOT NULL,
|
||||
count INTEGER NOT NULL DEFAULT 0,
|
||||
sum_results INTEGER NOT NULL DEFAULT 0,
|
||||
sum_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
sum_budget_dropped INTEGER NOT NULL DEFAULT 0,
|
||||
cache_hit INTEGER NOT NULL DEFAULT 0,
|
||||
cache_miss INTEGER NOT NULL DEFAULT 0,
|
||||
first_seen TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
last_seen TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (date, mode, intent)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_search_telemetry_date
|
||||
ON search_telemetry (date DESC);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -10,7 +10,7 @@ import { clampSearchLimit } from './engine.ts';
|
||||
import type { GBrainConfig } from './config.ts';
|
||||
import type { PageType } from './types.ts';
|
||||
import { importFromContent } from './import-file.ts';
|
||||
import { hybridSearch } from './search/hybrid.ts';
|
||||
import { hybridSearch, hybridSearchCached } from './search/hybrid.ts';
|
||||
import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
@@ -1089,7 +1089,10 @@ const query: Operation = {
|
||||
// stays SearchResult[] (Cathedral II callers depend on that); meta
|
||||
// arrives via callback so eval capture can record what actually ran.
|
||||
let capturedMeta: HybridSearchMeta | null = null;
|
||||
const results = await hybridSearch(ctx.engine, queryText, {
|
||||
// v0.32.x search-lite: route the query op through hybridSearchCached so
|
||||
// semantic cache + token budget + intent weighting fire automatically.
|
||||
// Plain hybridSearch remains the bare API for callers that opt out.
|
||||
const results = await hybridSearchCached(ctx.engine, queryText, {
|
||||
limit: (p.limit as number) || 20,
|
||||
offset: (p.offset as number) || 0,
|
||||
expansion: expand,
|
||||
@@ -1104,6 +1107,10 @@ const query: Operation = {
|
||||
recency: p.recency as 'off' | 'on' | 'strong' | undefined,
|
||||
since: typeof p.since === 'string' ? p.since : undefined,
|
||||
until: typeof p.until === 'string' ? p.until : undefined,
|
||||
// v0.32.x search-lite: token budget + cache opt-outs.
|
||||
tokenBudget: typeof p.token_budget === 'number' ? (p.token_budget as number) : undefined,
|
||||
useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined,
|
||||
intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined,
|
||||
onMeta: (m) => { capturedMeta = m; },
|
||||
});
|
||||
const latency_ms = Date.now() - startedAt;
|
||||
|
||||
@@ -3151,6 +3151,24 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async unsetConfig(key: string): Promise<number> {
|
||||
const { affectedRows } = await this.db.query(
|
||||
'DELETE FROM config WHERE key = $1',
|
||||
[key],
|
||||
) as { affectedRows?: number };
|
||||
return affectedRows ?? 0;
|
||||
}
|
||||
|
||||
async listConfigKeys(prefix: string): Promise<string[]> {
|
||||
// LIKE-escape the prefix so a user-supplied % or _ doesn't act as a wildcard.
|
||||
const escaped = prefix.replace(/\\/g, '\\\\').replace(/%/g, '\\%').replace(/_/g, '\\_');
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT key FROM config WHERE key LIKE $1 || '%' ESCAPE '\\' ORDER BY key`,
|
||||
[escaped],
|
||||
);
|
||||
return (rows as { key: string }[]).map(r => r.key);
|
||||
}
|
||||
|
||||
// Migration support
|
||||
async runMigration(_version: number, sql: string): Promise<void> {
|
||||
await this.db.exec(sql);
|
||||
|
||||
@@ -3148,6 +3148,23 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async unsetConfig(key: string): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const result = await sql`DELETE FROM config WHERE key = ${key}` as unknown as { count: number };
|
||||
return result.count ?? 0;
|
||||
}
|
||||
|
||||
async listConfigKeys(prefix: string): Promise<string[]> {
|
||||
const sql = this.sql;
|
||||
// LIKE-escape literal % and _ so a config key with those chars resolves correctly.
|
||||
const escaped = prefix.replace(/\\/g, '\\\\').replace(/%/g, '\\%').replace(/_/g, '\\_');
|
||||
const pattern = `${escaped}%`;
|
||||
const rows = await sql<{ key: string }[]>`
|
||||
SELECT key FROM config WHERE key LIKE ${pattern} ESCAPE '\\' ORDER BY key
|
||||
`;
|
||||
return rows.map(r => r.key);
|
||||
}
|
||||
|
||||
// Migration support
|
||||
async runMigration(_version: number, sqlStr: string): Promise<void> {
|
||||
const conn = this.sql;
|
||||
|
||||
+370
-13
@@ -16,6 +16,17 @@ import { embed } from '../embedding.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
import { enforceTokenBudget } from './token-budget.ts';
|
||||
import { recordSearchTelemetry } from './telemetry.ts';
|
||||
import {
|
||||
weightsForIntent,
|
||||
effectiveRrfK,
|
||||
applyExactMatchBoost,
|
||||
} from './intent-weights.ts';
|
||||
import {
|
||||
SemanticQueryCache,
|
||||
loadCacheConfig,
|
||||
} from './query-cache.ts';
|
||||
|
||||
const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
@@ -213,10 +224,43 @@ export async function hybridSearch(
|
||||
query: string,
|
||||
opts?: HybridSearchOpts,
|
||||
): Promise<SearchResult[]> {
|
||||
const limit = opts?.limit || 20;
|
||||
// v0.32.3 search-lite mode: resolve the active mode + per-key overrides
|
||||
// once at entry. Mode supplies DEFAULTS for intentWeighting, tokenBudget,
|
||||
// expansion, and searchLimit when the caller leaves those undefined.
|
||||
// Per-call opts and per-key config overrides still win.
|
||||
//
|
||||
// This MUST live in bare hybridSearch (NOT just in hybridSearchCached)
|
||||
// because eval-replay and eval-longmemeval call bare hybridSearch — and
|
||||
// per-mode evals would not test production search if modes lived only in
|
||||
// the wrapper. See `[CDX-5+6]` in the plan.
|
||||
const { loadSearchModeConfig, resolveSearchMode } = await import('./mode.ts');
|
||||
const modeInput = await loadSearchModeConfig(engine);
|
||||
const resolvedMode = resolveSearchMode({
|
||||
mode: modeInput.mode,
|
||||
overrides: modeInput.overrides,
|
||||
perCall: {
|
||||
intentWeighting: opts?.intentWeighting,
|
||||
tokenBudget: opts?.tokenBudget,
|
||||
expansion: opts?.expansion,
|
||||
searchLimit: opts?.limit,
|
||||
},
|
||||
});
|
||||
|
||||
const limit = opts?.limit || resolvedMode.searchLimit;
|
||||
const offset = opts?.offset || 0;
|
||||
const innerLimit = Math.min(limit * 2, MAX_SEARCH_LIMIT);
|
||||
|
||||
// v0.32.x search-lite: classify intent once up front. Drives BOTH the
|
||||
// legacy auto-detail / salience / recency suggestions AND the new
|
||||
// weight-adjustment path. Intent weighting is on by default and can
|
||||
// be disabled via `opts.intentWeighting = false`. The mode bundle
|
||||
// supplies the default when neither per-call nor per-key sets it.
|
||||
const suggestions = classifyQuery(query);
|
||||
const intentWeightingOn = resolvedMode.intentWeighting;
|
||||
const intentWeights = intentWeightingOn
|
||||
? weightsForIntent(suggestions.intent)
|
||||
: weightsForIntent('general');
|
||||
|
||||
// Auto-detect detail level from query intent when caller doesn't specify
|
||||
const detail = opts?.detail ?? autoDetectDetail(query);
|
||||
const detailResolved: 'low' | 'medium' | 'high' | null = detail ?? null;
|
||||
@@ -246,12 +290,23 @@ export async function hybridSearch(
|
||||
// A throwing user callback must never break the search hot path — onMeta
|
||||
// is a public surface (gbrain/search/hybrid) so a third-party closure bug
|
||||
// shouldn't take down query/search responses.
|
||||
//
|
||||
// v0.32.3 search-lite: every emitMeta call ALSO records to the in-process
|
||||
// search_telemetry rollup. Telemetry write is sync (bumps a bucket map),
|
||||
// flush is fire-and-forget on 60s / 100-call thresholds. The hot path
|
||||
// never waits.
|
||||
let lastResultsCount = 0;
|
||||
const emitMeta = (meta: HybridSearchMeta): void => {
|
||||
try {
|
||||
opts?.onMeta?.(meta);
|
||||
} catch {
|
||||
// swallow — capture telemetry is best-effort
|
||||
}
|
||||
try {
|
||||
recordSearchTelemetry(engine, meta, { results_count: lastResultsCount });
|
||||
} catch {
|
||||
// swallow — telemetry must never break the search hot path.
|
||||
}
|
||||
};
|
||||
|
||||
if (DEBUG && detail) {
|
||||
@@ -264,7 +319,11 @@ export async function hybridSearch(
|
||||
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
|
||||
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
|
||||
// The wrapper fires from ALL THREE return paths (codex pass-1 #2 + pass-2 #4).
|
||||
const suggestions = classifyQuery(query);
|
||||
//
|
||||
// v0.32.x search-lite: when caller hasn't set recency and the intent
|
||||
// classifier suggests one, prefer that (suggestedRecency on temporal /
|
||||
// event intents). The legacy heuristic still wins when intent weighting
|
||||
// is off.
|
||||
// Back-compat: recencyBoost: 1|2 → 'on'|'strong'; 0 → 'off'.
|
||||
const legacyRecency: 'off' | 'on' | 'strong' | undefined =
|
||||
opts?.recencyBoost === 2 ? 'strong' :
|
||||
@@ -272,7 +331,20 @@ export async function hybridSearch(
|
||||
opts?.recencyBoost === 0 ? 'off' :
|
||||
undefined;
|
||||
const salienceMode: 'off' | 'on' | 'strong' = opts?.salience ?? suggestions.suggestedSalience;
|
||||
const recencyMode: 'off' | 'on' | 'strong' = opts?.recency ?? legacyRecency ?? suggestions.suggestedRecency;
|
||||
// Intent-weighting recency suggestion is a NUDGE — it only fires when
|
||||
// the caller left recency unspecified AND the legacy heuristic also
|
||||
// didn't fire. The classifier's own suggestedRecency (from v0.29.1)
|
||||
// still wins when it's set; the new intent suggestion is a fallback.
|
||||
const intentRecency =
|
||||
intentWeightingOn && intentWeights.suggestedRecency != null
|
||||
? intentWeights.suggestedRecency
|
||||
: null;
|
||||
const recencyMode: 'off' | 'on' | 'strong' =
|
||||
opts?.recency
|
||||
?? legacyRecency
|
||||
?? (suggestions.suggestedRecency !== 'off'
|
||||
? suggestions.suggestedRecency
|
||||
: (intentRecency ?? suggestions.suggestedRecency));
|
||||
const postFusionOpts = {
|
||||
applyBacklinks: true,
|
||||
salience: salienceMode,
|
||||
@@ -286,15 +358,31 @@ export async function hybridSearch(
|
||||
await runPostFusionStages(engine, keywordResults, postFusionOpts);
|
||||
keywordResults.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
const noEmbedSliced = dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement on the no-embedding-provider path.
|
||||
const { results: noEmbedBudgeted, meta: noEmbedBudgetMeta } = enforceTokenBudget(noEmbedSliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = noEmbedBudgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: false,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: false,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: noEmbedBudgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return noEmbedBudgeted;
|
||||
}
|
||||
|
||||
// Determine query variants (optionally with expansion)
|
||||
// expandQuery already includes the original query in its return value,
|
||||
// so we use it directly instead of prepending query again
|
||||
// so we use it directly instead of prepending query again.
|
||||
// v0.32.3 search-lite: expansion fires when (a) resolved mode says yes and
|
||||
// (b) an expandFn is wired in. The mode bundle is the default; per-call
|
||||
// SearchOpts.expansion still wins via resolveSearchMode's chain.
|
||||
let queries = [query];
|
||||
if (opts?.expansion && opts?.expandFn) {
|
||||
if (resolvedMode.expansion && opts?.expandFn) {
|
||||
try {
|
||||
queries = await opts.expandFn(query);
|
||||
if (queries.length === 0) queries = [query];
|
||||
@@ -327,14 +415,38 @@ export async function hybridSearch(
|
||||
await runPostFusionStages(engine, keywordResults, postFusionOpts);
|
||||
keywordResults.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
const kwSliced = dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement on the keyword-fallback path too.
|
||||
const { results: kwBudgeted, meta: kwBudgetMeta } = enforceTokenBudget(kwSliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = kwBudgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: false,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: kwBudgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return kwBudgeted;
|
||||
}
|
||||
|
||||
// Merge all result lists via RRF (includes normalization + boost)
|
||||
// Skip boost for detail=high (temporal/event queries want natural ranking)
|
||||
const allLists = [...vectorLists, keywordResults];
|
||||
let fused = rrfFusion(allLists, opts?.rrfK ?? RRF_K, detail !== 'high');
|
||||
//
|
||||
// v0.32.x search-lite: when intent weighting is on, run RRF with
|
||||
// per-list effective k values — entity/event intents nudge keyword
|
||||
// contributions up by lowering their k. The base rrfK still controls
|
||||
// the overall RRF shape; intent weights tilt within that shape.
|
||||
const baseRrfK = opts?.rrfK ?? RRF_K;
|
||||
const keywordK = effectiveRrfK(baseRrfK, intentWeights.keywordWeight);
|
||||
const vectorK = effectiveRrfK(baseRrfK, intentWeights.vectorWeight);
|
||||
const allLists: Array<{ list: SearchResult[]; k: number }> = [
|
||||
...vectorLists.map(list => ({ list, k: vectorK })),
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive
|
||||
if (queryEmbedding) {
|
||||
@@ -347,6 +459,11 @@ export async function hybridSearch(
|
||||
// both, or neither fires depending on resolved modes.
|
||||
if (fused.length > 0) {
|
||||
await runPostFusionStages(engine, fused, postFusionOpts);
|
||||
// v0.32.x search-lite: intent exact-match boost (entity/event intents).
|
||||
// No-op when boost factor is 1.0 (general intent or weighting disabled).
|
||||
if (intentWeights.exactMatchBoost !== 1.0) {
|
||||
applyExactMatchBoost(fused, query, intentWeights);
|
||||
}
|
||||
fused.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
|
||||
@@ -409,8 +526,248 @@ export async function hybridSearch(
|
||||
return hybridSearch(engine, query, { ...opts, detail: 'high' });
|
||||
}
|
||||
|
||||
emitMeta({ vector_enabled: true, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return deduped.slice(offset, offset + limit);
|
||||
const sliced = deduped.slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement at the main return path.
|
||||
// hybridSearchCached used to be the only place this fired; now bare
|
||||
// hybridSearch enforces it too so eval-replay + eval-longmemeval see
|
||||
// the same budget behavior as the production query op.
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(sliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = budgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: true,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return budgeted;
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------
|
||||
// v0.32.x (search-lite) — cached + budgeted public wrapper
|
||||
// ----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Public wrapper around hybridSearch that adds the v0.32.x search-lite
|
||||
* features: semantic query cache + token budget enforcement. Both are
|
||||
* additive and backward-compatible; callers that don't opt in see the
|
||||
* same behavior as plain hybridSearch.
|
||||
*
|
||||
* Pipeline:
|
||||
* 1. Cache lookup (if enabled + we can produce a query embedding).
|
||||
* 2. On miss: run hybridSearch normally.
|
||||
* 3. Apply token budget (no-op when budget is undefined).
|
||||
* 4. On miss + successful search: write back to cache (best-effort).
|
||||
*
|
||||
* The cache uses the same embedding the search pipeline would compute,
|
||||
* so an extra embed() call only happens when hybridSearch would have
|
||||
* skipped vector search entirely (no embedding provider configured). In
|
||||
* that case the cache is also skipped — there's no embedding to key on.
|
||||
*/
|
||||
export async function hybridSearchCached(
|
||||
engine: BrainEngine,
|
||||
query: string,
|
||||
opts?: HybridSearchOpts,
|
||||
): Promise<SearchResult[]> {
|
||||
// v0.32.3 search-lite mode: resolve mode + per-key overrides once. The
|
||||
// resolved knob set drives cache enable/threshold/TTL AND the knobs_hash
|
||||
// that scopes the cache row so a tokenmax write can't be served to a
|
||||
// conservative read. See [CDX-4] in the plan.
|
||||
const { loadSearchModeConfig, resolveSearchMode, knobsHash } = await import('./mode.ts');
|
||||
const modeInputForCache = await loadSearchModeConfig(engine);
|
||||
const resolvedForCache = resolveSearchMode({
|
||||
mode: modeInputForCache.mode,
|
||||
overrides: modeInputForCache.overrides,
|
||||
perCall: {
|
||||
cache_enabled: opts?.useCache,
|
||||
tokenBudget: opts?.tokenBudget,
|
||||
expansion: opts?.expansion,
|
||||
intentWeighting: opts?.intentWeighting,
|
||||
searchLimit: opts?.limit,
|
||||
},
|
||||
});
|
||||
const cacheKnobsHash = knobsHash(resolvedForCache);
|
||||
|
||||
// Cache decision: opts.useCache (explicit) wins over global config; global
|
||||
// config wins over mode bundle default. Mode bundle is on for all 3 modes
|
||||
// today; the resolver already folded everything through.
|
||||
const cacheCfg = await loadCacheConfig(engine);
|
||||
const cacheEnabled = resolvedForCache.cache_enabled;
|
||||
const cache = new SemanticQueryCache(engine, {
|
||||
...cacheCfg,
|
||||
enabled: cacheEnabled,
|
||||
similarityThreshold: resolvedForCache.cache_similarity_threshold,
|
||||
ttlSeconds: resolvedForCache.cache_ttl_seconds,
|
||||
});
|
||||
|
||||
// Skip cache entirely when the request asks for two-pass walks or has
|
||||
// a non-default embedding column — those interact with structural state
|
||||
// that the cache can't safely express.
|
||||
const skipCache =
|
||||
!cache.isEnabled() ||
|
||||
(opts?.walkDepth ?? 0) > 0 ||
|
||||
Boolean(opts?.nearSymbol) ||
|
||||
(opts?.embeddingColumn && opts.embeddingColumn !== 'embedding');
|
||||
|
||||
let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss';
|
||||
let cacheSimilarity: number | undefined;
|
||||
let cacheAge: number | undefined;
|
||||
|
||||
// We need a query embedding to consult the cache. We try to embed once
|
||||
// here so the same embedding can be threaded back into the search call
|
||||
// if it misses — but the embedding helper isn't cheap, so we only
|
||||
// attempt it when the cache is enabled AND the gateway has an embedding
|
||||
// provider configured.
|
||||
let queryEmbedding: Float32Array | null = null;
|
||||
if (!skipCache) {
|
||||
try {
|
||||
const { isAvailable } = await import('../ai/gateway.ts');
|
||||
if (isAvailable('embedding')) {
|
||||
queryEmbedding = await embed(query);
|
||||
} else {
|
||||
cacheStatus = 'disabled';
|
||||
}
|
||||
} catch {
|
||||
cacheStatus = 'disabled';
|
||||
queryEmbedding = null;
|
||||
}
|
||||
}
|
||||
|
||||
if (!skipCache && queryEmbedding && cacheStatus !== 'disabled') {
|
||||
const hit = await cache.lookup(queryEmbedding, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash });
|
||||
if (hit.hit && hit.results) {
|
||||
cacheStatus = 'hit';
|
||||
cacheSimilarity = hit.similarity;
|
||||
cacheAge = hit.ageSeconds;
|
||||
|
||||
const limit = opts?.limit || 20;
|
||||
const offset = opts?.offset || 0;
|
||||
const sliced = hit.results.slice(offset, offset + limit);
|
||||
|
||||
// Budget enforcement — same pipeline tail as fresh path.
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(sliced, opts?.tokenBudget);
|
||||
|
||||
// Emit meta describing the cache path.
|
||||
const cachedMeta: HybridSearchMeta = {
|
||||
vector_enabled: hit.meta?.vector_enabled ?? true,
|
||||
detail_resolved: hit.meta?.detail_resolved ?? null,
|
||||
expansion_applied: hit.meta?.expansion_applied ?? false,
|
||||
intent: hit.meta?.intent,
|
||||
cache: {
|
||||
status: 'hit',
|
||||
similarity: cacheSimilarity,
|
||||
age_seconds: cacheAge,
|
||||
},
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
opts?.onMeta?.(cachedMeta);
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort
|
||||
}
|
||||
return budgeted;
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss (or disabled): run the normal search. We capture meta so
|
||||
// we can write back to the cache + emit the merged meta to the caller.
|
||||
// The closure-write pattern trips TS's narrowing (it infers `never`), so
|
||||
// we use a single-element box to keep the type stable.
|
||||
const innerMetaBox: { current: HybridSearchMeta | null } = { current: null };
|
||||
const userOnMeta = opts?.onMeta;
|
||||
const results = await hybridSearch(engine, query, {
|
||||
...opts,
|
||||
onMeta: (m) => {
|
||||
innerMetaBox.current = m;
|
||||
// Do NOT call userOnMeta here — we'll emit a merged meta below
|
||||
// that also carries cache + budget info.
|
||||
},
|
||||
});
|
||||
const innerMeta = innerMetaBox.current;
|
||||
|
||||
// Token budget pass (no-op when not set).
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(results, opts?.tokenBudget);
|
||||
|
||||
// Compose the final meta and emit.
|
||||
const finalMeta: HybridSearchMeta = {
|
||||
vector_enabled: innerMeta?.vector_enabled ?? false,
|
||||
detail_resolved: innerMeta?.detail_resolved ?? null,
|
||||
expansion_applied: innerMeta?.expansion_applied ?? false,
|
||||
intent: innerMeta?.intent,
|
||||
cache: { status: cacheStatus },
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
userOnMeta?.(finalMeta);
|
||||
} catch {
|
||||
// swallow
|
||||
}
|
||||
|
||||
// Best-effort writeback (skip when search returned empty so we don't
|
||||
// cache zero-result queries forever — they often indicate a typo).
|
||||
if (
|
||||
cacheStatus === 'miss' &&
|
||||
queryEmbedding &&
|
||||
results.length > 0 &&
|
||||
(innerMeta?.vector_enabled ?? false)
|
||||
) {
|
||||
void cache
|
||||
.store(query, queryEmbedding, results, finalMeta, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash })
|
||||
.catch(() => { /* swallow */ });
|
||||
}
|
||||
|
||||
return budgeted;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.x search-lite — weighted RRF. Each list contributes with its own
|
||||
* effective k value, which lets intent weighting bias keyword vs vector
|
||||
* lists without re-weighting individual scores. Wraps rrfFusion internally
|
||||
* by computing weighted contributions in a single pass.
|
||||
*/
|
||||
export function rrfFusionWeighted(
|
||||
lists: Array<{ list: SearchResult[]; k: number }>,
|
||||
applyBoost = true,
|
||||
): SearchResult[] {
|
||||
const scores = new Map<string, { result: SearchResult; score: number }>();
|
||||
|
||||
for (const { list, k } of lists) {
|
||||
for (let rank = 0; rank < list.length; rank++) {
|
||||
const r = list[rank];
|
||||
const key = `${r.slug}:${r.chunk_id ?? r.chunk_text.slice(0, 50)}`;
|
||||
const existing = scores.get(key);
|
||||
const rrfScore = 1 / (k + rank);
|
||||
|
||||
if (existing) {
|
||||
existing.score += rrfScore;
|
||||
} else {
|
||||
scores.set(key, { result: r, score: rrfScore });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const entries = Array.from(scores.values());
|
||||
if (entries.length === 0) return [];
|
||||
|
||||
const maxScore = Math.max(...entries.map(e => e.score));
|
||||
if (maxScore > 0) {
|
||||
for (const e of entries) {
|
||||
e.score = e.score / maxScore;
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
e.score *= boost;
|
||||
}
|
||||
}
|
||||
|
||||
return entries
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.map(({ result, score }) => ({ ...result, score }));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* Zero-LLM Intent → Weight Adjustment (v0.32.x — search-lite)
|
||||
*
|
||||
* Sits on top of the existing query-intent classifier. The classifier
|
||||
* (src/core/search/query-intent.ts) already produces an `intent` field
|
||||
* with 4 values: entity / temporal / event / general. This module maps
|
||||
* that intent onto concrete weight adjustments applied during the hybrid
|
||||
* search pipeline:
|
||||
*
|
||||
* - entity → boost exact slug/title matches (keyword pre-filter favored)
|
||||
* - temporal → increase recency scoring weight (recency = 'on' when caller
|
||||
* left it undefined)
|
||||
* - event → increase keyword weight in hybrid fusion (event queries
|
||||
* tend to have rare named entities that keyword search nails
|
||||
* while vector search smears across paraphrases)
|
||||
* - general → default semantic (no adjustment)
|
||||
*
|
||||
* All adjustments are SUBTLE — they nudge weights, they don't override
|
||||
* caller-explicit options. If the caller passed `recency: 'off'`, intent
|
||||
* weighting will NOT silently re-enable it. The classifier is a default,
|
||||
* not a mandate.
|
||||
*
|
||||
* The original v0.20.0 LLM query expansion path (expandQuery in
|
||||
* expansion.ts) still exists and is opt-in via `opts.expansion = true`.
|
||||
* Intent weighting is the new DEFAULT, and replaces the expansion call
|
||||
* for the common case (simple queries, no API key, fast loop).
|
||||
*
|
||||
* Pure module. No DB, no LLM, no async. Tested in
|
||||
* test/intent-weights.test.ts.
|
||||
*/
|
||||
|
||||
import type { QueryIntent } from './query-intent.ts';
|
||||
import type { SearchResult } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Weight adjustments to apply for a classified intent. All factors are
|
||||
* multiplicative on the existing pipeline weights; the defaults map to
|
||||
* 1.0 (no-op) for the `general` intent. A factor > 1.0 increases the
|
||||
* weight of that signal; < 1.0 decreases it.
|
||||
*
|
||||
* Magnitudes were tuned conservatively (max 1.25x boost) so the existing
|
||||
* search behavior on ambiguous queries stays close to v0.31.x. The point
|
||||
* of the classifier isn't to flip rankings — it's to break ties in
|
||||
* favor of the user's plausible intent.
|
||||
*/
|
||||
export interface IntentWeights {
|
||||
/** Multiplier on the keyword-list rank in RRF fusion. Higher = keyword wins more ties. */
|
||||
keywordWeight: number;
|
||||
/** Multiplier on the vector-list rank in RRF fusion. Higher = semantic wins more ties. */
|
||||
vectorWeight: number;
|
||||
/** Recency tilt to suggest when caller hasn't specified one. */
|
||||
suggestedRecency: 'off' | 'on' | 'strong' | null;
|
||||
/** Score multiplier for results whose slug/title exactly matches the (lowercased) query. */
|
||||
exactMatchBoost: number;
|
||||
}
|
||||
|
||||
const DEFAULT_WEIGHTS: IntentWeights = {
|
||||
keywordWeight: 1.0,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: null,
|
||||
exactMatchBoost: 1.0,
|
||||
};
|
||||
|
||||
const INTENT_WEIGHTS: Record<QueryIntent, IntentWeights> = {
|
||||
entity: {
|
||||
// Entity queries: "who is X", "tell me about Y". The user knows the
|
||||
// name. Reward exact slug/title matches; lean into keyword.
|
||||
keywordWeight: 1.15,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: null,
|
||||
exactMatchBoost: 1.25,
|
||||
},
|
||||
temporal: {
|
||||
// Temporal queries: "what happened last week", "meeting prep". Recency
|
||||
// tilt is the whole game; keyword and vector stay balanced.
|
||||
keywordWeight: 1.0,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: 'on',
|
||||
exactMatchBoost: 1.0,
|
||||
},
|
||||
event: {
|
||||
// Event queries: "announcement", "launched", "raised $". Named events
|
||||
// have rare entity surface forms that keyword search nails (think
|
||||
// company names, dollar amounts). Recency gets a soft tilt too.
|
||||
keywordWeight: 1.20,
|
||||
vectorWeight: 0.95,
|
||||
suggestedRecency: 'on',
|
||||
exactMatchBoost: 1.10,
|
||||
},
|
||||
general: DEFAULT_WEIGHTS,
|
||||
};
|
||||
|
||||
/** Lookup the weights for a classified intent. */
|
||||
export function weightsForIntent(intent: QueryIntent): IntentWeights {
|
||||
return INTENT_WEIGHTS[intent] ?? DEFAULT_WEIGHTS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the per-list rank weighting before RRF. Caller passes the list
|
||||
* source ('keyword' | 'vector') and the weights; we return the effective
|
||||
* RRF k constant to use for THAT list. Lower k = stronger boost on top
|
||||
* ranks; higher k = flatter contribution. So a higher weight maps to a
|
||||
* LOWER k.
|
||||
*
|
||||
* Default RRF_K is 60. With keywordWeight=1.20, the effective k for the
|
||||
* keyword list becomes 60 / 1.20 = 50, which gives top-keyword results
|
||||
* a meaningfully stronger contribution to the fused score.
|
||||
*/
|
||||
export function effectiveRrfK(baseK: number, weight: number): number {
|
||||
if (weight <= 0) return baseK;
|
||||
return baseK / weight;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply exact-match boost in place. Mutates each result's score by
|
||||
* `weights.exactMatchBoost` when the result's slug or title (lowercased,
|
||||
* trimmed) matches the lowercased query exactly. No-op when the boost
|
||||
* is 1.0. Caller re-sorts after.
|
||||
*
|
||||
* Normalization: slug is matched as-is (slugs are already canonicalized
|
||||
* lowercase-kebab); title is lowercased + trimmed. The query is
|
||||
* lowercased + trimmed once before the loop.
|
||||
*/
|
||||
export function applyExactMatchBoost(
|
||||
results: SearchResult[],
|
||||
query: string,
|
||||
weights: IntentWeights,
|
||||
): void {
|
||||
if (weights.exactMatchBoost === 1.0) return;
|
||||
const q = query.toLowerCase().trim();
|
||||
if (!q) return;
|
||||
// Pre-compute the kebab form for slug-style matches like "garry tan" → "garry-tan".
|
||||
const qKebab = q.replace(/\s+/g, '-');
|
||||
for (const r of results) {
|
||||
const slug = (r.slug ?? '').toLowerCase();
|
||||
const title = (r.title ?? '').toLowerCase().trim();
|
||||
if (slug === q || slug === qKebab || slug.endsWith(`/${qKebab}`) || title === q) {
|
||||
r.score *= weights.exactMatchBoost;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
/**
|
||||
* v0.32.3 search-lite mode bundles.
|
||||
*
|
||||
* Three named modes that bundle the search-lite knobs from PR #897 into a
|
||||
* single config key so users pick once at install time and stop thinking
|
||||
* about it. Each mode resolves to a complete knob set; per-call SearchOpts
|
||||
* and per-key config overrides still win — mode just supplies the default.
|
||||
*
|
||||
* The resolution chain matches the v0.31.12 model-tier pattern at
|
||||
* `src/core/model-config.ts:resolveModel`:
|
||||
* per-call opts → per-key config → MODE_BUNDLES[cfg.search.mode] → MODE_BUNDLES.balanced
|
||||
*
|
||||
* `resolveSearchMode` is called at the top of bare `hybridSearch`, NOT just
|
||||
* inside the `hybridSearchCached` wrapper. Eval commands (`eval replay`,
|
||||
* `eval longmemeval`) call bare hybridSearch and must test the same
|
||||
* mode-affected behavior as production. See `[CDX-5+6]` in the plan.
|
||||
*
|
||||
* `knobsHash` produces the SHA-256 the query cache uses to prevent
|
||||
* cross-mode contamination. The PR #897 cache keyed only on
|
||||
* (source_id, query_text) — a tokenmax run with expansion+limit=50 would
|
||||
* populate a row that a subsequent conservative call reads back. Migration
|
||||
* v56 adds `knobs_hash` column; lookup filters by knobs_hash equality AND
|
||||
* embedding similarity. See `[CDX-4]` in the plan.
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
|
||||
export type SearchMode = 'conservative' | 'balanced' | 'tokenmax';
|
||||
|
||||
export const SEARCH_MODES: ReadonlyArray<SearchMode> = Object.freeze([
|
||||
'conservative',
|
||||
'balanced',
|
||||
'tokenmax',
|
||||
]);
|
||||
|
||||
/**
|
||||
* A complete knob set for one mode. Every field is required so the bundle
|
||||
* is self-contained and per-key overrides are obvious diffs.
|
||||
*/
|
||||
export interface ModeBundle {
|
||||
/** Semantic query cache (PR #897). Free win; on for everyone. */
|
||||
cache_enabled: boolean;
|
||||
cache_similarity_threshold: number;
|
||||
cache_ttl_seconds: number;
|
||||
/** Zero-LLM intent classifier weight adjustments (PR #897). On for everyone. */
|
||||
intentWeighting: boolean;
|
||||
/**
|
||||
* Per-call token budget cap (PR #897). undefined = no-op (tokenmax).
|
||||
* 4000 = tight (conservative, fits Haiku context loop).
|
||||
* 12000 = balanced (sweet-spot for Sonnet).
|
||||
*/
|
||||
tokenBudget: number | undefined;
|
||||
/**
|
||||
* LLM multi-query expansion (Haiku call per search).
|
||||
* Per CLAUDE.md TODOS the corpus eval shows ~97.6% lift relative to no
|
||||
* expansion — barely measurable. Off for conservative/balanced;
|
||||
* on for tokenmax to preserve power-user retrieval ceiling.
|
||||
*/
|
||||
expansion: boolean;
|
||||
/**
|
||||
* Default `limit` for the operation layer (`src/core/operations.ts:1087`).
|
||||
* Note: production `query` op TODAY defaults to 20. Mode bundle becomes
|
||||
* the default ONLY when the caller omits the field — same chain semantics
|
||||
* as model-tier resolution. See `[CDX-1+2+3]` in the plan: the original
|
||||
* "tokenmax preserves Garry's setup" framing is wrong; tokenmax is an
|
||||
* EXPANSION from the implicit current default (limit 20).
|
||||
*/
|
||||
searchLimit: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The three mode bundles. Frozen at import time so a typo can't redefine
|
||||
* "conservative" to mean different things on different installs — the
|
||||
* public eval table depends on these being canonical. Power-user
|
||||
* customization happens via per-key config overrides; if there's real
|
||||
* demand for a custom bundle, that's a v0.34 conversation.
|
||||
*/
|
||||
export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> = Object.freeze({
|
||||
conservative: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 4000,
|
||||
expansion: false,
|
||||
searchLimit: 10,
|
||||
}),
|
||||
balanced: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 12000,
|
||||
expansion: false,
|
||||
searchLimit: 25,
|
||||
}),
|
||||
tokenmax: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: undefined,
|
||||
expansion: true,
|
||||
searchLimit: 50,
|
||||
}),
|
||||
});
|
||||
|
||||
export const DEFAULT_SEARCH_MODE: SearchMode = 'balanced';
|
||||
|
||||
export function isSearchMode(x: unknown): x is SearchMode {
|
||||
return typeof x === 'string' && (SEARCH_MODES as ReadonlyArray<string>).includes(x);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-key config overrides. Read at search-time from the `config` table.
|
||||
* Every field is optional; an undefined field means "fall through to the
|
||||
* mode bundle default."
|
||||
*/
|
||||
export interface SearchKeyOverrides {
|
||||
cache_enabled?: boolean;
|
||||
cache_similarity_threshold?: number;
|
||||
cache_ttl_seconds?: number;
|
||||
intentWeighting?: boolean;
|
||||
tokenBudget?: number;
|
||||
expansion?: boolean;
|
||||
searchLimit?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-call opts that can override the bundle for this single search.
|
||||
* Same shape as ModeBundle but every field is optional. These are passed
|
||||
* through from `SearchOpts` / `HybridSearchOpts` so the existing per-call
|
||||
* surface continues to work — mode just provides the default that the
|
||||
* caller's explicit field overrides.
|
||||
*/
|
||||
export interface SearchPerCallOpts {
|
||||
cache_enabled?: boolean;
|
||||
cache_similarity_threshold?: number;
|
||||
cache_ttl_seconds?: number;
|
||||
intentWeighting?: boolean;
|
||||
tokenBudget?: number;
|
||||
expansion?: boolean;
|
||||
searchLimit?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the active search knob set for one search call.
|
||||
*
|
||||
* Resolution chain (matches v0.31.12 model-tier semantics):
|
||||
* 1. perCallOpts.<key> if defined → wins
|
||||
* 2. config.search.<key> if defined → wins
|
||||
* 3. MODE_BUNDLES[config.search.mode].<key> → mode default
|
||||
* 4. MODE_BUNDLES.balanced.<key> → safety fallback when config.search.mode is invalid/unset
|
||||
*
|
||||
* Pure function: no DB calls, no env reads. Caller pre-loads the relevant
|
||||
* config rows (one SELECT for the whole batch of keys, not one per key).
|
||||
*/
|
||||
export interface ResolveSearchModeInput {
|
||||
/** Resolved value of `config.search.mode`. Undefined → fallback to balanced. */
|
||||
mode?: string;
|
||||
/** Resolved per-key overrides from config table. */
|
||||
overrides?: SearchKeyOverrides;
|
||||
/** Per-call opts (SearchOpts / HybridSearchOpts). */
|
||||
perCall?: SearchPerCallOpts;
|
||||
}
|
||||
|
||||
export interface ResolvedSearchKnobs extends ModeBundle {
|
||||
/** Which mode bundle supplied the defaults (after fallback). */
|
||||
resolved_mode: SearchMode;
|
||||
/** True if the caller's `mode` input was a recognized SearchMode. */
|
||||
mode_valid: boolean;
|
||||
}
|
||||
|
||||
export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearchKnobs {
|
||||
const requested = typeof input.mode === 'string' ? input.mode.trim().toLowerCase() : '';
|
||||
const valid = isSearchMode(requested);
|
||||
const resolved_mode: SearchMode = valid ? (requested as SearchMode) : DEFAULT_SEARCH_MODE;
|
||||
const bundle = MODE_BUNDLES[resolved_mode];
|
||||
|
||||
const ov = input.overrides ?? {};
|
||||
const pc = input.perCall ?? {};
|
||||
|
||||
const pick = <K extends keyof ModeBundle>(key: K): ModeBundle[K] => {
|
||||
if (pc[key] !== undefined) return pc[key] as ModeBundle[K];
|
||||
if (ov[key] !== undefined) return ov[key] as ModeBundle[K];
|
||||
return bundle[key];
|
||||
};
|
||||
|
||||
return {
|
||||
cache_enabled: pick('cache_enabled'),
|
||||
cache_similarity_threshold: pick('cache_similarity_threshold'),
|
||||
cache_ttl_seconds: pick('cache_ttl_seconds'),
|
||||
intentWeighting: pick('intentWeighting'),
|
||||
tokenBudget: pick('tokenBudget'),
|
||||
expansion: pick('expansion'),
|
||||
searchLimit: pick('searchLimit'),
|
||||
resolved_mode,
|
||||
mode_valid: valid,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-knob source attribution for `gbrain search modes` dashboard.
|
||||
* Tells the user where each resolved value came from so override drift
|
||||
* is legible. Mirrors `gbrain models` (v0.31.12) attribution shape.
|
||||
*/
|
||||
export type KnobSource = 'per-call' | 'override' | 'mode' | 'fallback';
|
||||
|
||||
export interface ResolvedKnobAttribution {
|
||||
knob: keyof ModeBundle;
|
||||
value: ModeBundle[keyof ModeBundle];
|
||||
source: KnobSource;
|
||||
// For 'override' source, the config key path; for 'mode' source, the mode name.
|
||||
source_detail: string;
|
||||
}
|
||||
|
||||
export function attributeKnob<K extends keyof ModeBundle>(
|
||||
knob: K,
|
||||
input: ResolveSearchModeInput,
|
||||
resolved: ResolvedSearchKnobs,
|
||||
): ResolvedKnobAttribution {
|
||||
const pc = input.perCall ?? {};
|
||||
const ov = input.overrides ?? {};
|
||||
if (pc[knob] !== undefined) {
|
||||
return { knob, value: resolved[knob], source: 'per-call', source_detail: 'SearchOpts' };
|
||||
}
|
||||
if (ov[knob] !== undefined) {
|
||||
return { knob, value: resolved[knob], source: 'override', source_detail: `config: search.${knob}` };
|
||||
}
|
||||
if (resolved.mode_valid) {
|
||||
return { knob, value: resolved[knob], source: 'mode', source_detail: `mode: ${resolved.resolved_mode}` };
|
||||
}
|
||||
return { knob, value: resolved[knob], source: 'fallback', source_detail: `mode: ${DEFAULT_SEARCH_MODE} (default — search.mode unset)` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable hash of the resolved knob set. Used as part of the query_cache
|
||||
* primary key so a tokenmax cache write can't be served to a conservative
|
||||
* lookup (cross-mode contamination, [CDX-4]).
|
||||
*
|
||||
* Knob order is FIXED so the hash is deterministic across releases. NEVER
|
||||
* reorder or add a knob without bumping a constant — a hash collision would
|
||||
* mean stale cache rows silently reading the wrong shape.
|
||||
*/
|
||||
export const KNOBS_HASH_VERSION = 1;
|
||||
|
||||
export function knobsHash(knobs: ResolvedSearchKnobs): string {
|
||||
// Fixed-order key list. Adding a knob here REQUIRES bumping
|
||||
// KNOBS_HASH_VERSION and is a breaking change for any persisted cache.
|
||||
const parts = [
|
||||
`v=${KNOBS_HASH_VERSION}`,
|
||||
`mode=${knobs.resolved_mode}`,
|
||||
`cache=${knobs.cache_enabled ? 1 : 0}`,
|
||||
`sim=${knobs.cache_similarity_threshold.toFixed(4)}`,
|
||||
`ttl=${knobs.cache_ttl_seconds}`,
|
||||
`iw=${knobs.intentWeighting ? 1 : 0}`,
|
||||
`tb=${knobs.tokenBudget ?? 'none'}`,
|
||||
`exp=${knobs.expansion ? 1 : 0}`,
|
||||
`lim=${knobs.searchLimit}`,
|
||||
];
|
||||
const h = createHash('sha256');
|
||||
h.update(parts.join('|'));
|
||||
return h.digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience: build SearchKeyOverrides from a flat config-table snapshot.
|
||||
* Used by hybridSearch's hot path so the search code pays one round-trip
|
||||
* to load all relevant config keys rather than one per knob.
|
||||
*
|
||||
* Returns sparse overrides — only keys actually present in the config
|
||||
* map appear. Falsy/missing keys fall through to the mode bundle default.
|
||||
*/
|
||||
export function loadOverridesFromConfig(
|
||||
configMap: Record<string, string | undefined>,
|
||||
): SearchKeyOverrides {
|
||||
const out: SearchKeyOverrides = {};
|
||||
const get = (k: string): string | undefined => configMap[k];
|
||||
|
||||
const ce = get('search.cache.enabled');
|
||||
if (ce !== undefined) {
|
||||
out.cache_enabled = ce === '1' || ce.toLowerCase() === 'true';
|
||||
}
|
||||
const st = get('search.cache.similarity_threshold');
|
||||
if (st !== undefined) {
|
||||
const n = parseFloat(st);
|
||||
if (Number.isFinite(n) && n > 0 && n <= 1) out.cache_similarity_threshold = n;
|
||||
}
|
||||
const tt = get('search.cache.ttl_seconds');
|
||||
if (tt !== undefined) {
|
||||
const n = parseInt(tt, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.cache_ttl_seconds = n;
|
||||
}
|
||||
const iw = get('search.intentWeighting');
|
||||
if (iw !== undefined) {
|
||||
out.intentWeighting = iw === '1' || iw.toLowerCase() === 'true';
|
||||
}
|
||||
const tb = get('search.tokenBudget');
|
||||
if (tb !== undefined) {
|
||||
const n = parseInt(tb, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.tokenBudget = n;
|
||||
}
|
||||
const ex = get('search.expansion');
|
||||
if (ex !== undefined) {
|
||||
out.expansion = ex === '1' || ex.toLowerCase() === 'true';
|
||||
}
|
||||
const sl = get('search.searchLimit');
|
||||
if (sl !== undefined) {
|
||||
const n = parseInt(sl, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.searchLimit = n;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The full list of config keys this module reads. Used by `gbrain search modes --reset`. */
|
||||
export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray<string> = Object.freeze([
|
||||
'search.cache.enabled',
|
||||
'search.cache.similarity_threshold',
|
||||
'search.cache.ttl_seconds',
|
||||
'search.intentWeighting',
|
||||
'search.tokenBudget',
|
||||
'search.expansion',
|
||||
'search.searchLimit',
|
||||
]);
|
||||
|
||||
/**
|
||||
* The mode-selection config key itself. Separated from SEARCH_MODE_CONFIG_KEYS
|
||||
* because `--reset` clears OVERRIDES (the per-knob keys) but should NOT clear
|
||||
* the operator's mode choice.
|
||||
*/
|
||||
export const SEARCH_MODE_KEY = 'search.mode';
|
||||
|
||||
/**
|
||||
* Load the live mode config (mode + per-key overrides) from the brain engine.
|
||||
* Runs ONE round-trip per knob currently — the BrainEngine.getConfig interface
|
||||
* is single-key. A future v0.34 batch loader can collapse this. Volume is
|
||||
* small (~8 keys); call site is once per search.
|
||||
*
|
||||
* Errors are swallowed and fall through to mode-bundle defaults. The cache
|
||||
* config table predates v0.32.3 and may not exist on very old brains, so
|
||||
* silent fallback is the right shape.
|
||||
*/
|
||||
export async function loadSearchModeConfig(
|
||||
engine: { getConfig(key: string): Promise<string | null> },
|
||||
): Promise<ResolveSearchModeInput> {
|
||||
const safeGet = async (k: string): Promise<string | undefined> => {
|
||||
try {
|
||||
const v = await engine.getConfig(k);
|
||||
return v == null ? undefined : v;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
const [mode, ...overrideValues] = await Promise.all([
|
||||
safeGet(SEARCH_MODE_KEY),
|
||||
...SEARCH_MODE_CONFIG_KEYS.map(safeGet),
|
||||
]);
|
||||
|
||||
const configMap: Record<string, string | undefined> = {};
|
||||
SEARCH_MODE_CONFIG_KEYS.forEach((key, i) => {
|
||||
if (overrideValues[i] !== undefined) configMap[key] = overrideValues[i];
|
||||
});
|
||||
|
||||
return {
|
||||
mode,
|
||||
overrides: loadOverridesFromConfig(configMap),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,372 @@
|
||||
/**
|
||||
* Semantic Query Cache (v0.32.x — search-lite)
|
||||
*
|
||||
* Caches hybridSearch results keyed by query embedding similarity. On
|
||||
* each lookup the cache runs an HNSW cosine-distance probe against
|
||||
* stored query embeddings. If the closest cached query is within
|
||||
* `1 - similarity_threshold` cosine distance (default similarity
|
||||
* >= 0.92, distance < 0.08), we return the stored results instantly
|
||||
* — no keyword search, no vector search, no LLM expansion, no RRF, no
|
||||
* dedup. Otherwise we report a miss and let the caller run the real
|
||||
* search.
|
||||
*
|
||||
* Storage: the `query_cache` table (migration v51) with the same
|
||||
* embedding dim as `content_chunks`. Per-row TTL (default 3600 seconds).
|
||||
* Stale rows are skipped at read time and pruned by `gbrain cache prune`.
|
||||
*
|
||||
* Multi-source isolation: cache lookups scope by `source_id` so brain
|
||||
* A's "who is widget-ceo" cannot return brain B's cached results for
|
||||
* the same query.
|
||||
*
|
||||
* Edge cases:
|
||||
* - No embedding available (no OPENAI key, embed failure): cache is
|
||||
* skipped silently. Caller runs the normal pipeline.
|
||||
* - Table missing (pre-v51 brain): every read swallows the error and
|
||||
* reports a miss. No throw.
|
||||
* - Cache disabled by config or by caller (`useCache: false`): the
|
||||
* cache module is never called.
|
||||
*
|
||||
* Tested in test/query-cache.test.ts.
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../types.ts';
|
||||
|
||||
/** Default cosine similarity threshold for cache hits. */
|
||||
export const DEFAULT_SIMILARITY_THRESHOLD = 0.92;
|
||||
/** Default TTL for cache entries, in seconds. */
|
||||
export const DEFAULT_TTL_SECONDS = 3600;
|
||||
|
||||
export interface CacheLookupResult {
|
||||
hit: boolean;
|
||||
results?: SearchResult[];
|
||||
meta?: HybridSearchMeta;
|
||||
/** Cosine similarity of the matched cached query (0..1). Only set on hit. */
|
||||
similarity?: number;
|
||||
/** Age of the cached entry in seconds. Only set on hit. */
|
||||
ageSeconds?: number;
|
||||
}
|
||||
|
||||
export interface CacheStats {
|
||||
total_rows: number;
|
||||
total_hits: number;
|
||||
fresh_rows: number;
|
||||
stale_rows: number;
|
||||
}
|
||||
|
||||
export interface QueryCacheConfig {
|
||||
enabled?: boolean;
|
||||
similarityThreshold?: number;
|
||||
ttlSeconds?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic ID for a (query, source, knobsHash) tuple. Used as the primary
|
||||
* key so re-caching the exact same (query, mode, knobs) just bumps the row's
|
||||
* hit_count and created_at rather than inserting duplicates.
|
||||
*
|
||||
* v0.32.3 [CDX-4]: knobsHash is now part of the key to prevent cross-mode
|
||||
* cache contamination. A tokenmax write (expansion=on, limit=50) and a
|
||||
* conservative write (no expansion, limit=10) for the same query+source
|
||||
* land in distinct rows. Empty-string knobsHash is accepted (preserves
|
||||
* existing test setups) but production calls always pass the resolved hash.
|
||||
*/
|
||||
export function cacheRowId(queryText: string, sourceId: string, knobsHash = ''): string {
|
||||
const h = createHash('sha256');
|
||||
h.update(`${sourceId}::${queryText}::${knobsHash}`);
|
||||
return h.digest('hex').slice(0, 32);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a Float32Array embedding into the pgvector text literal
|
||||
* format: `[v0,v1,v2,...]`. PGLite and Postgres both accept this when
|
||||
* the parameter is cast to `vector` or `halfvec` on the server side.
|
||||
*/
|
||||
function embeddingToPgVector(embedding: Float32Array): string {
|
||||
// Stringify with reasonable precision. pgvector accepts plain decimals.
|
||||
// Use a typed iteration to avoid TS errors on Float32Array forEach.
|
||||
const parts: string[] = new Array(embedding.length);
|
||||
for (let i = 0; i < embedding.length; i++) {
|
||||
parts[i] = embedding[i].toFixed(6);
|
||||
}
|
||||
return `[${parts.join(',')}]`;
|
||||
}
|
||||
|
||||
export class SemanticQueryCache {
|
||||
private similarityThreshold: number;
|
||||
private ttlSeconds: number;
|
||||
private enabled: boolean;
|
||||
|
||||
constructor(
|
||||
private engine: BrainEngine,
|
||||
config?: QueryCacheConfig,
|
||||
) {
|
||||
this.enabled = config?.enabled ?? true;
|
||||
this.similarityThreshold = clampThreshold(config?.similarityThreshold);
|
||||
this.ttlSeconds = clampTtl(config?.ttlSeconds);
|
||||
}
|
||||
|
||||
isEnabled(): boolean {
|
||||
return this.enabled;
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up a cached result set by query embedding similarity. Returns a
|
||||
* miss (hit=false) when:
|
||||
* - cache is disabled,
|
||||
* - embedding is null/empty,
|
||||
* - the table doesn't exist (pre-v51 brain),
|
||||
* - no row is within the similarity threshold,
|
||||
* - the matching row is past its TTL.
|
||||
*
|
||||
* All errors are swallowed and converted to a miss \u2014 the cache must
|
||||
* never break the search hot path.
|
||||
*/
|
||||
async lookup(
|
||||
queryEmbedding: Float32Array | null,
|
||||
opts: { sourceId?: string; knobsHash?: string } = {},
|
||||
): Promise<CacheLookupResult> {
|
||||
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) {
|
||||
return { hit: false };
|
||||
}
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
const knobsHash = opts.knobsHash ?? '';
|
||||
const distanceThreshold = 1 - this.similarityThreshold;
|
||||
const vec = embeddingToPgVector(queryEmbedding);
|
||||
|
||||
try {
|
||||
// Find the closest cached query within the distance threshold and
|
||||
// freshness window. The TTL check is done in-query (created_at +
|
||||
// ttl_seconds > now) so we never return a stale row.
|
||||
//
|
||||
// v0.32.3 [CDX-4]: knobs_hash filter prevents cross-mode contamination.
|
||||
// A tokenmax write (expansion=on, limit=50) and a conservative read
|
||||
// (no expansion, limit=10) have distinct knobs hashes and miss each
|
||||
// other. Rows with NULL knobs_hash (pre-v0.32.3) are excluded.
|
||||
const rows = await this.engine.executeRaw<{
|
||||
id: string;
|
||||
results: unknown;
|
||||
meta: unknown;
|
||||
distance: number;
|
||||
age_seconds: number;
|
||||
}>(
|
||||
`SELECT id, results, meta,
|
||||
embedding <=> $1::vector AS distance,
|
||||
EXTRACT(EPOCH FROM (now() - created_at))::int AS age_seconds
|
||||
FROM query_cache
|
||||
WHERE source_id = $2
|
||||
AND knobs_hash = $4
|
||||
AND embedding IS NOT NULL
|
||||
AND embedding <=> $1::vector < $3
|
||||
AND created_at + (ttl_seconds || ' seconds')::interval > now()
|
||||
ORDER BY embedding <=> $1::vector
|
||||
LIMIT 1`,
|
||||
[vec, sourceId, distanceThreshold, knobsHash],
|
||||
);
|
||||
|
||||
if (rows.length === 0) return { hit: false };
|
||||
|
||||
const row = rows[0];
|
||||
const results = Array.isArray(row.results)
|
||||
? (row.results as SearchResult[])
|
||||
: safeJsonParse<SearchResult[]>(row.results, []);
|
||||
const meta = safeJsonParse<HybridSearchMeta | undefined>(row.meta, undefined);
|
||||
const similarity = 1 - row.distance;
|
||||
|
||||
// Bump hit_count / last_hit_at \u2014 best-effort.
|
||||
void this.bumpHit(row.id).catch(() => { /* swallow */ });
|
||||
|
||||
return {
|
||||
hit: true,
|
||||
results,
|
||||
meta,
|
||||
similarity,
|
||||
ageSeconds: row.age_seconds,
|
||||
};
|
||||
} catch {
|
||||
// Table missing, vector column missing, or any other failure: miss.
|
||||
return { hit: false };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist a fresh search result set into the cache. Idempotent on
|
||||
* (query_text + source_id) \u2014 re-writes the row with a fresh
|
||||
* created_at. Best-effort: errors are swallowed.
|
||||
*/
|
||||
async store(
|
||||
queryText: string,
|
||||
queryEmbedding: Float32Array | null,
|
||||
results: SearchResult[],
|
||||
meta: HybridSearchMeta,
|
||||
opts: { sourceId?: string; ttlSeconds?: number; knobsHash?: string } = {},
|
||||
): Promise<void> {
|
||||
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) return;
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
const knobsHash = opts.knobsHash ?? '';
|
||||
const ttl = clampTtl(opts.ttlSeconds ?? this.ttlSeconds);
|
||||
const id = cacheRowId(queryText, sourceId, knobsHash);
|
||||
const vec = embeddingToPgVector(queryEmbedding);
|
||||
|
||||
try {
|
||||
// v0.32.3 [CDX-4]: knobs_hash threaded into the row so concurrent
|
||||
// tokenmax + conservative writes for the same query+source live as
|
||||
// distinct rows. The PK is `id` (which already encodes the hash),
|
||||
// so ON CONFLICT (id) DO UPDATE just refreshes the same-mode row.
|
||||
await this.engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5::vector, $6::jsonb, $7::jsonb, $8, now())
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
query_text = EXCLUDED.query_text,
|
||||
knobs_hash = EXCLUDED.knobs_hash,
|
||||
embedding = EXCLUDED.embedding,
|
||||
results = EXCLUDED.results,
|
||||
meta = EXCLUDED.meta,
|
||||
ttl_seconds = EXCLUDED.ttl_seconds,
|
||||
created_at = now()`,
|
||||
[
|
||||
id,
|
||||
queryText,
|
||||
sourceId,
|
||||
knobsHash,
|
||||
vec,
|
||||
JSON.stringify(results),
|
||||
JSON.stringify(meta),
|
||||
ttl,
|
||||
],
|
||||
);
|
||||
} catch {
|
||||
// swallow \u2014 cache write must never break the search hot path.
|
||||
}
|
||||
}
|
||||
|
||||
/** Clear ALL cache rows (optionally scoped by source). Returns rows deleted. */
|
||||
async clear(opts: { sourceId?: string } = {}): Promise<number> {
|
||||
try {
|
||||
if (opts.sourceId) {
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (DELETE FROM query_cache WHERE source_id = $1 RETURNING 1)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
[opts.sourceId],
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
}
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (DELETE FROM query_cache RETURNING 1)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** Delete only stale (past-TTL) rows. Returns rows deleted. */
|
||||
async prune(): Promise<number> {
|
||||
try {
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (
|
||||
DELETE FROM query_cache
|
||||
WHERE created_at + (ttl_seconds || ' seconds')::interval <= now()
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** Summary stats for `gbrain cache stats`. */
|
||||
async stats(): Promise<CacheStats> {
|
||||
try {
|
||||
const rows = await this.engine.executeRaw<{
|
||||
total_rows: number;
|
||||
total_hits: number;
|
||||
fresh_rows: number;
|
||||
stale_rows: number;
|
||||
}>(
|
||||
`SELECT
|
||||
COUNT(*)::int AS total_rows,
|
||||
COALESCE(SUM(hit_count), 0)::int AS total_hits,
|
||||
COUNT(*) FILTER (WHERE created_at + (ttl_seconds || ' seconds')::interval > now())::int AS fresh_rows,
|
||||
COUNT(*) FILTER (WHERE created_at + (ttl_seconds || ' seconds')::interval <= now())::int AS stale_rows
|
||||
FROM query_cache`,
|
||||
);
|
||||
return rows[0] ?? { total_rows: 0, total_hits: 0, fresh_rows: 0, stale_rows: 0 };
|
||||
} catch {
|
||||
return { total_rows: 0, total_hits: 0, fresh_rows: 0, stale_rows: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
private async bumpHit(id: string): Promise<void> {
|
||||
await this.engine.executeRaw(
|
||||
`UPDATE query_cache
|
||||
SET hit_count = hit_count + 1, last_hit_at = now()
|
||||
WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function clampThreshold(v: number | undefined): number {
|
||||
if (typeof v !== 'number' || !Number.isFinite(v)) return DEFAULT_SIMILARITY_THRESHOLD;
|
||||
return Math.max(0.5, Math.min(0.999, v));
|
||||
}
|
||||
|
||||
function clampTtl(v: number | undefined): number {
|
||||
if (typeof v !== 'number' || !Number.isFinite(v) || v <= 0) return DEFAULT_TTL_SECONDS;
|
||||
// Cap at 30 days to avoid runaway TTLs.
|
||||
return Math.min(60 * 60 * 24 * 30, Math.floor(v));
|
||||
}
|
||||
|
||||
function safeJsonParse<T>(value: unknown, fallback: T): T {
|
||||
if (value == null) return fallback;
|
||||
if (typeof value === 'object') return value as T;
|
||||
if (typeof value !== 'string') return fallback;
|
||||
try {
|
||||
return JSON.parse(value) as T;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve cache config from the engine's `config` table. Mirrors how
|
||||
* other modules read runtime config (e.g. embedding_dimensions). All
|
||||
* three keys have sensible defaults; missing rows fall back to those.
|
||||
*/
|
||||
export async function loadCacheConfig(engine: BrainEngine): Promise<QueryCacheConfig> {
|
||||
const keys = [
|
||||
'search.cache.enabled',
|
||||
'search.cache.similarity_threshold',
|
||||
'search.cache.ttl_seconds',
|
||||
];
|
||||
const config: QueryCacheConfig = {
|
||||
enabled: true,
|
||||
similarityThreshold: DEFAULT_SIMILARITY_THRESHOLD,
|
||||
ttlSeconds: DEFAULT_TTL_SECONDS,
|
||||
};
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ key: string; value: string }>(
|
||||
`SELECT key, value FROM config WHERE key = ANY($1)`,
|
||||
[keys],
|
||||
);
|
||||
for (const row of rows) {
|
||||
if (row.key === 'search.cache.enabled') {
|
||||
config.enabled = row.value === '1' || row.value.toLowerCase() === 'true';
|
||||
} else if (row.key === 'search.cache.similarity_threshold') {
|
||||
const v = parseFloat(row.value);
|
||||
if (Number.isFinite(v)) config.similarityThreshold = clampThreshold(v);
|
||||
} else if (row.key === 'search.cache.ttl_seconds') {
|
||||
const v = parseInt(row.value, 10);
|
||||
if (Number.isFinite(v)) config.ttlSeconds = clampTtl(v);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Use defaults.
|
||||
}
|
||||
return config;
|
||||
}
|
||||
@@ -0,0 +1,364 @@
|
||||
/**
|
||||
* v0.32.3 search-lite telemetry rollup writer.
|
||||
*
|
||||
* Architecture decision (D2 in the plan, [CDX-19]): per-process in-memory
|
||||
* bucket, flushed periodically (60s OR 100 calls, whichever first) AND on
|
||||
* process exit via beforeExit/SIGINT/SIGTERM with a 2-second timeout cap.
|
||||
* The search hot path NEVER waits on this write — `record()` is sync and
|
||||
* the flush is fire-and-forget.
|
||||
*
|
||||
* Schema math per [CDX-17]: rows are sums + counts only, NEVER averages.
|
||||
* Read-time derives averages. ON CONFLICT DO UPDATE adds raw values so two
|
||||
* gbrain processes flushing the same (date, mode, intent) tuple accumulate
|
||||
* correctly.
|
||||
*
|
||||
* The bucket map is keyed by `${date}::${mode}::${intent}`. Date is the
|
||||
* UTC ISO date (YYYY-MM-DD). Cross-midnight calls land in distinct buckets
|
||||
* automatically.
|
||||
*
|
||||
* Per-process bucketing means stdio MCP, HTTP MCP, and CLI processes each
|
||||
* maintain their own buffers. Stats are directional, not exact — acceptable
|
||||
* because the consumer is the operator (or an agent running `gbrain search
|
||||
* tune`), not a financial ledger. The "lose last bucket on hard crash"
|
||||
* downside is documented in the methodology doc.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { HybridSearchMeta } from '../types.ts';
|
||||
|
||||
interface Bucket {
|
||||
date: string;
|
||||
mode: string;
|
||||
intent: string;
|
||||
count: number;
|
||||
sum_results: number;
|
||||
sum_tokens: number;
|
||||
sum_budget_dropped: number;
|
||||
cache_hit: number;
|
||||
cache_miss: number;
|
||||
}
|
||||
|
||||
const FLUSH_INTERVAL_MS = 60_000;
|
||||
const FLUSH_THRESHOLD_CALLS = 100;
|
||||
|
||||
/**
|
||||
* Per-process telemetry singleton. Each gbrain process (CLI, stdio MCP,
|
||||
* HTTP MCP) gets one instance. The flush timer and exit hooks are
|
||||
* installed lazily on the first `record()` call so importing this module
|
||||
* has no side effects.
|
||||
*/
|
||||
class TelemetryWriter {
|
||||
private buckets = new Map<string, Bucket>();
|
||||
private pendingCount = 0;
|
||||
private engine: BrainEngine | null = null;
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
private exitHookInstalled = false;
|
||||
private flushInFlight: Promise<void> | null = null;
|
||||
|
||||
/** Wire the engine. Called once per process at search-time. Subsequent calls are a no-op. */
|
||||
setEngine(engine: BrainEngine): void {
|
||||
if (!this.engine) {
|
||||
this.engine = engine;
|
||||
this.ensureExitHook();
|
||||
this.ensureTimer();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a search call. Sync — never blocks the hot path. Returns
|
||||
* immediately after bumping the in-memory bucket. Flush is async +
|
||||
* fire-and-forget.
|
||||
*/
|
||||
record(meta: HybridSearchMeta, opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 }): void {
|
||||
const date = nowDate();
|
||||
const mode = meta.mode ?? 'unset';
|
||||
const intent = meta.intent ?? 'unset';
|
||||
const key = `${date}::${mode}::${intent}`;
|
||||
|
||||
let b = this.buckets.get(key);
|
||||
if (!b) {
|
||||
b = {
|
||||
date,
|
||||
mode,
|
||||
intent,
|
||||
count: 0,
|
||||
sum_results: 0,
|
||||
sum_tokens: 0,
|
||||
sum_budget_dropped: 0,
|
||||
cache_hit: 0,
|
||||
cache_miss: 0,
|
||||
};
|
||||
this.buckets.set(key, b);
|
||||
}
|
||||
|
||||
b.count += 1;
|
||||
b.sum_results += Math.max(0, Math.floor(opts.results_count));
|
||||
b.sum_tokens += Math.max(0, Math.floor(opts.tokens_estimate ?? meta.token_budget?.used ?? 0));
|
||||
b.sum_budget_dropped += Math.max(0, Math.floor(meta.token_budget?.dropped ?? 0));
|
||||
if (meta.cache?.status === 'hit') b.cache_hit += 1;
|
||||
if (meta.cache?.status === 'miss') b.cache_miss += 1;
|
||||
|
||||
this.pendingCount += 1;
|
||||
if (this.pendingCount >= FLUSH_THRESHOLD_CALLS) {
|
||||
void this.flush().catch(() => { /* swallow */ });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drain the bucket map to the database. Idempotent; concurrent flushes
|
||||
* are coalesced via flushInFlight. The bucket map is swapped atomically
|
||||
* before the SQL write so new `record()` calls during flush land in a
|
||||
* fresh map.
|
||||
*/
|
||||
async flush(): Promise<void> {
|
||||
if (this.flushInFlight) return this.flushInFlight;
|
||||
if (!this.engine || this.buckets.size === 0) {
|
||||
this.pendingCount = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
// Swap the map: a new record() call during flush goes into the new map.
|
||||
const snapshot = this.buckets;
|
||||
this.buckets = new Map();
|
||||
this.pendingCount = 0;
|
||||
|
||||
const engine = this.engine;
|
||||
this.flushInFlight = (async () => {
|
||||
try {
|
||||
for (const b of snapshot.values()) {
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO search_telemetry
|
||||
(date, mode, intent, count, sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss, first_seen, last_seen)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, now(), now())
|
||||
ON CONFLICT (date, mode, intent) DO UPDATE SET
|
||||
count = search_telemetry.count + EXCLUDED.count,
|
||||
sum_results = search_telemetry.sum_results + EXCLUDED.sum_results,
|
||||
sum_tokens = search_telemetry.sum_tokens + EXCLUDED.sum_tokens,
|
||||
sum_budget_dropped = search_telemetry.sum_budget_dropped + EXCLUDED.sum_budget_dropped,
|
||||
cache_hit = search_telemetry.cache_hit + EXCLUDED.cache_hit,
|
||||
cache_miss = search_telemetry.cache_miss + EXCLUDED.cache_miss,
|
||||
last_seen = now()`,
|
||||
[b.date, b.mode, b.intent, b.count, b.sum_results, b.sum_tokens, b.sum_budget_dropped, b.cache_hit, b.cache_miss],
|
||||
);
|
||||
} catch {
|
||||
// swallow — telemetry write must never break the hot path.
|
||||
// Per-bucket isolation: one bad row doesn't lose the others.
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
this.flushInFlight = null;
|
||||
}
|
||||
})();
|
||||
return this.flushInFlight;
|
||||
}
|
||||
|
||||
/** Stop the timer and uninstall exit hooks. Called from tests / shutdown. */
|
||||
stop(): void {
|
||||
if (this.timer) {
|
||||
clearInterval(this.timer);
|
||||
this.timer = null;
|
||||
}
|
||||
this.buckets.clear();
|
||||
this.pendingCount = 0;
|
||||
}
|
||||
|
||||
/** Test-only — read the current bucket count without draining. */
|
||||
bucketCountForTest(): number {
|
||||
return this.buckets.size;
|
||||
}
|
||||
|
||||
/** Test-only — read a specific bucket (returns null if absent). */
|
||||
bucketForTest(date: string, mode: string, intent: string): Readonly<Bucket> | null {
|
||||
return this.buckets.get(`${date}::${mode}::${intent}`) ?? null;
|
||||
}
|
||||
|
||||
private ensureTimer(): void {
|
||||
if (this.timer) return;
|
||||
this.timer = setInterval(() => {
|
||||
void this.flush().catch(() => { /* swallow */ });
|
||||
}, FLUSH_INTERVAL_MS);
|
||||
// Unref so the timer doesn't keep the process alive on its own.
|
||||
if (typeof this.timer.unref === 'function') this.timer.unref();
|
||||
}
|
||||
|
||||
private ensureExitHook(): void {
|
||||
if (this.exitHookInstalled) return;
|
||||
this.exitHookInstalled = true;
|
||||
|
||||
// Lossy by design: skip the buffered drain entirely on process exit.
|
||||
//
|
||||
// The earlier implementation installed `process.on('beforeExit', drainOnExit)`
|
||||
// with an inner `Promise.race([flush(), setTimeout(2000)])`. That enqueued
|
||||
// new async work AFTER the event loop had emptied, which kept the process
|
||||
// alive past beforeExit — short-lived CLI invocations (`gbrain query "the"`
|
||||
// exiting after 100ms of work) ended up waiting on the DB write to settle.
|
||||
// On a slow or busy PGLite, the write never settled and the CLI hung
|
||||
// forever. That deadlock surfaced as the `test/e2e/claw-test.test.ts`
|
||||
// hang (the harness spawns short-lived gbrain queries that should exit
|
||||
// in <1s but never did).
|
||||
//
|
||||
// Resolution per [CDX-19]: the periodic flush timer (unref'd) handles
|
||||
// long-running processes (HTTP MCP server, autopilot, jobs work). For
|
||||
// short-lived CLI invocations, telemetry buffering of one search call
|
||||
// is acceptable to lose. Stats are directional, not exact.
|
||||
//
|
||||
// The signal handlers (SIGINT / SIGTERM) also drop — kill -TERM should
|
||||
// exit immediately, not block on a DB write that may never complete.
|
||||
}
|
||||
|
||||
// Test-only: previously inspected by tests. Retained as a no-op so the
|
||||
// test harness's _resetTelemetryWriterForTest doesn't need to know about
|
||||
// the exit-hook decision.
|
||||
flushOnExitForTest(): Promise<void> {
|
||||
return this.flush().catch(() => { /* swallow */ });
|
||||
}
|
||||
}
|
||||
|
||||
/** Module-level singleton, one per process. */
|
||||
let _writer: TelemetryWriter | null = null;
|
||||
|
||||
export function getTelemetryWriter(): TelemetryWriter {
|
||||
if (!_writer) _writer = new TelemetryWriter();
|
||||
return _writer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience entry point for hot-path callers. Wires the engine lazily
|
||||
* and records the call. Never throws.
|
||||
*/
|
||||
export function recordSearchTelemetry(
|
||||
engine: BrainEngine,
|
||||
meta: HybridSearchMeta,
|
||||
opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 },
|
||||
): void {
|
||||
try {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
w.record(meta, opts);
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read aggregated stats over a window. Read-time derives averages from
|
||||
* sums + counts so writers can ON CONFLICT-add freely.
|
||||
*/
|
||||
export interface StatsWindow {
|
||||
total_calls: number;
|
||||
cache_hits: number;
|
||||
cache_misses: number;
|
||||
cache_hit_rate: number;
|
||||
avg_results: number;
|
||||
avg_tokens: number;
|
||||
total_budget_dropped: number;
|
||||
intent_distribution: Record<string, number>;
|
||||
mode_distribution: Record<string, number>;
|
||||
window_days: number;
|
||||
oldest_seen?: string;
|
||||
newest_seen?: string;
|
||||
}
|
||||
|
||||
export async function readSearchStats(
|
||||
engine: BrainEngine,
|
||||
opts: { days?: number } = {},
|
||||
): Promise<StatsWindow> {
|
||||
const days = Math.max(1, Math.min(365, opts.days ?? 7));
|
||||
const cutoffDate = new Date(Date.now() - days * 86_400_000).toISOString().slice(0, 10);
|
||||
|
||||
try {
|
||||
const rows = await engine.executeRaw<{
|
||||
mode: string;
|
||||
intent: string;
|
||||
count: number;
|
||||
sum_results: number;
|
||||
sum_tokens: number;
|
||||
sum_budget_dropped: number;
|
||||
cache_hit: number;
|
||||
cache_miss: number;
|
||||
first_seen: string;
|
||||
last_seen: string;
|
||||
}>(
|
||||
`SELECT mode, intent,
|
||||
SUM(count)::int AS count,
|
||||
SUM(sum_results)::int AS sum_results,
|
||||
SUM(sum_tokens)::int AS sum_tokens,
|
||||
SUM(sum_budget_dropped)::int AS sum_budget_dropped,
|
||||
SUM(cache_hit)::int AS cache_hit,
|
||||
SUM(cache_miss)::int AS cache_miss,
|
||||
MIN(first_seen)::text AS first_seen,
|
||||
MAX(last_seen)::text AS last_seen
|
||||
FROM search_telemetry
|
||||
WHERE date >= $1
|
||||
GROUP BY mode, intent`,
|
||||
[cutoffDate],
|
||||
);
|
||||
|
||||
let total_calls = 0;
|
||||
let cache_hits = 0;
|
||||
let cache_misses = 0;
|
||||
let total_results = 0;
|
||||
let total_tokens = 0;
|
||||
let total_budget_dropped = 0;
|
||||
const intent_distribution: Record<string, number> = {};
|
||||
const mode_distribution: Record<string, number> = {};
|
||||
let oldest_seen: string | undefined;
|
||||
let newest_seen: string | undefined;
|
||||
|
||||
for (const r of rows) {
|
||||
total_calls += r.count;
|
||||
cache_hits += r.cache_hit;
|
||||
cache_misses += r.cache_miss;
|
||||
total_results += r.sum_results;
|
||||
total_tokens += r.sum_tokens;
|
||||
total_budget_dropped += r.sum_budget_dropped;
|
||||
intent_distribution[r.intent] = (intent_distribution[r.intent] ?? 0) + r.count;
|
||||
mode_distribution[r.mode] = (mode_distribution[r.mode] ?? 0) + r.count;
|
||||
if (r.first_seen && (!oldest_seen || r.first_seen < oldest_seen)) oldest_seen = r.first_seen;
|
||||
if (r.last_seen && (!newest_seen || r.last_seen > newest_seen)) newest_seen = r.last_seen;
|
||||
}
|
||||
|
||||
const probe_total = cache_hits + cache_misses;
|
||||
return {
|
||||
total_calls,
|
||||
cache_hits,
|
||||
cache_misses,
|
||||
cache_hit_rate: probe_total > 0 ? cache_hits / probe_total : 0,
|
||||
avg_results: total_calls > 0 ? total_results / total_calls : 0,
|
||||
avg_tokens: total_calls > 0 ? total_tokens / total_calls : 0,
|
||||
total_budget_dropped,
|
||||
intent_distribution,
|
||||
mode_distribution,
|
||||
window_days: days,
|
||||
oldest_seen,
|
||||
newest_seen,
|
||||
};
|
||||
} catch {
|
||||
// Table missing or query failed — return empty stats rather than throw.
|
||||
return {
|
||||
total_calls: 0,
|
||||
cache_hits: 0,
|
||||
cache_misses: 0,
|
||||
cache_hit_rate: 0,
|
||||
avg_results: 0,
|
||||
avg_tokens: 0,
|
||||
total_budget_dropped: 0,
|
||||
intent_distribution: {},
|
||||
mode_distribution: {},
|
||||
window_days: days,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function nowDate(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
/** Test-only — reset the module-level singleton between test cases. */
|
||||
export function _resetTelemetryWriterForTest(): void {
|
||||
if (_writer) {
|
||||
_writer.stop();
|
||||
_writer = null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* Token Budget Enforcement on Search Results (v0.32.x — search-lite)
|
||||
*
|
||||
* Caps the cumulative token cost of a ranked SearchResult[] so callers
|
||||
* (agents, MCP, the query op) can guarantee their search payload fits a
|
||||
* downstream context window. The enforcer is the LAST stage of the search
|
||||
* pipeline — all scoring, ranking, dedup, boosts, two-pass walk are done
|
||||
* before this fires. It does NOT re-rank; it greedily walks top-down and
|
||||
* stops when the next result would push the running total past the
|
||||
* budget.
|
||||
*
|
||||
* Token counting uses a deliberately cheap char/4 heuristic instead of
|
||||
* dropping in a real tokenizer (js-tiktoken is 1.5MB+ and would balloon
|
||||
* the bun build --compile bundle). The heuristic is accurate within
|
||||
* ~10-15% for English text and ~5-25% for mixed code/Unicode — over-
|
||||
* estimating in code (which is what we want for a safety budget). For
|
||||
* a precise count, the caller can subtract real-tokens-vs-heuristic in
|
||||
* post and re-run with a tighter budget.
|
||||
*
|
||||
* Backward-compatibility: when no budget is set (undefined or <=0), the
|
||||
* enforcer is a no-op. The pre-v0.32 contract for search results is
|
||||
* unchanged.
|
||||
*
|
||||
* Pure module. No DB, no LLM, no async. Tested in test/token-budget.test.ts.
|
||||
*/
|
||||
|
||||
import type { SearchResult } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Cheap char/4 token estimate. Returns 0 for empty strings.
|
||||
*
|
||||
* Why char/4: OpenAI's tokenization averages ~4 chars/token for English
|
||||
* prose; closer to 3 for code with lots of punctuation; up to 8 for
|
||||
* CJK. Overshoot is fine for a safety budget. Undershoot would let us
|
||||
* blow past the cap, so we round UP when in doubt.
|
||||
*/
|
||||
export function estimateTokens(text: string | null | undefined): number {
|
||||
if (!text) return 0;
|
||||
// Math.ceil so a 1-char string still costs at least 1 token.
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-result token cost: title + chunk_text. Slug is metadata and
|
||||
* doesn't enter the assistant context, so we don't count it. If a
|
||||
* caller wants a different cost model (e.g. including timeline detail
|
||||
* or compiled_truth length), they can pre-shape the chunk_text before
|
||||
* calling enforceTokenBudget.
|
||||
*/
|
||||
export function resultTokens(r: SearchResult): number {
|
||||
return estimateTokens(r.title) + estimateTokens(r.chunk_text);
|
||||
}
|
||||
|
||||
export interface TokenBudgetMeta {
|
||||
/** Token budget that was applied (verbatim from caller). */
|
||||
budget: number;
|
||||
/** Cumulative token cost of the returned results. */
|
||||
used: number;
|
||||
/** Count of results that were dropped to fit the budget. */
|
||||
dropped: number;
|
||||
/** Count of results actually returned. */
|
||||
kept: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Greedy top-down budget enforcement. Walks the input in order, accumulates
|
||||
* token costs, and stops as soon as adding the next result would exceed
|
||||
* the budget. Results are NOT re-ranked — caller's order is preserved.
|
||||
*
|
||||
* Edge cases (all preserve the pre-v0.32 contract):
|
||||
* - budget undefined / <= 0: returns input unchanged; dropped=0, kept=N.
|
||||
* - First result alone exceeds budget: returns []; dropped=N, kept=0.
|
||||
* (Intentionally strict: the caller asked for a hard cap.)
|
||||
* - Input empty: returns []; budget unused.
|
||||
*/
|
||||
export function enforceTokenBudget(
|
||||
results: SearchResult[],
|
||||
budget: number | undefined,
|
||||
): { results: SearchResult[]; meta: TokenBudgetMeta } {
|
||||
const safeBudget = typeof budget === 'number' && budget > 0 ? budget : 0;
|
||||
|
||||
if (safeBudget === 0 || results.length === 0) {
|
||||
return {
|
||||
results,
|
||||
meta: {
|
||||
budget: safeBudget,
|
||||
used: results.reduce((acc, r) => acc + resultTokens(r), 0),
|
||||
dropped: 0,
|
||||
kept: results.length,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const kept: SearchResult[] = [];
|
||||
let used = 0;
|
||||
for (const r of results) {
|
||||
const cost = resultTokens(r);
|
||||
if (used + cost > safeBudget) break;
|
||||
kept.push(r);
|
||||
used += cost;
|
||||
}
|
||||
|
||||
return {
|
||||
results: kept,
|
||||
meta: {
|
||||
budget: safeBudget,
|
||||
used,
|
||||
dropped: results.length - kept.length,
|
||||
kept: kept.length,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -501,6 +501,32 @@ export interface SearchOpts {
|
||||
* Boundary semantics: end-of-day for plain YYYY-MM-DD.
|
||||
*/
|
||||
until?: string;
|
||||
/**
|
||||
* v0.32.x (search-lite): cap the cumulative token cost of returned results.
|
||||
* Applied AFTER all scoring, ranking, dedup, and boosts — the budget is the
|
||||
* LAST stage of the pipeline. Token counting uses a char/4 heuristic (no
|
||||
* tokenizer dep). When undefined or <= 0, this is a no-op (pre-v0.32
|
||||
* behavior).
|
||||
*
|
||||
* Use cases: keep an agent's search payload under its context window;
|
||||
* cap an MCP tool response to fit a router budget; emit a deterministic
|
||||
* upper bound on result size.
|
||||
*/
|
||||
tokenBudget?: number;
|
||||
/**
|
||||
* v0.32.x (search-lite): enable/disable the semantic query cache for this
|
||||
* call. When undefined, the cache decision falls back to global config
|
||||
* (search.cache.enabled, default true). Set to `false` to force a fresh
|
||||
* search; set to `true` to opt in even when global config has it off.
|
||||
*/
|
||||
useCache?: boolean;
|
||||
/**
|
||||
* v0.32.x (search-lite): force enable/disable the zero-LLM intent
|
||||
* classifier weight adjustments. Defaults to enabled. Set to `false` to
|
||||
* pin the legacy (pre-search-lite) weighting — useful when callers want
|
||||
* deterministic behavior independent of query phrasing.
|
||||
*/
|
||||
intentWeighting?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -803,6 +829,45 @@ export interface HybridSearchMeta {
|
||||
detail_resolved: 'low' | 'medium' | 'high' | null;
|
||||
/** True iff multi-query expansion (Haiku) actually fired and produced variants. */
|
||||
expansion_applied: boolean;
|
||||
/**
|
||||
* v0.32.x (search-lite): the intent the zero-LLM classifier inferred for
|
||||
* this query. Surfaced for debugging — agents and the `gbrain query`
|
||||
* command can show "intent: temporal" alongside results to make the
|
||||
* weighting decision auditable.
|
||||
*/
|
||||
intent?: 'entity' | 'temporal' | 'event' | 'general';
|
||||
/**
|
||||
* v0.32.x (search-lite): token budget enforcement metadata. Omitted when
|
||||
* no budget was applied (backward-compatible with pre-search-lite
|
||||
* consumers).
|
||||
*/
|
||||
token_budget?: {
|
||||
budget: number;
|
||||
used: number;
|
||||
kept: number;
|
||||
dropped: number;
|
||||
};
|
||||
/**
|
||||
* v0.32.x (search-lite): cache hit/miss tracking. Omitted when the
|
||||
* semantic query cache wasn't consulted (cache disabled, vector search
|
||||
* unavailable, etc.).
|
||||
*/
|
||||
cache?: {
|
||||
/** 'hit' when results came from the cache; 'miss' when search ran fresh. */
|
||||
status: 'hit' | 'miss' | 'disabled';
|
||||
/** Similarity of the cached query's embedding (0..1). Only set on hit. */
|
||||
similarity?: number;
|
||||
/** Age of the cached entry in seconds. Only set on hit. */
|
||||
age_seconds?: number;
|
||||
};
|
||||
/**
|
||||
* v0.32.3 (search-lite mode): the active search mode for this call.
|
||||
* 'conservative' | 'balanced' | 'tokenmax'. Resolved from
|
||||
* config.search.mode with per-call + per-key overrides applied. Surfaced
|
||||
* so observability sees what mode actually ran (which can differ from
|
||||
* the operator's `config.search.mode` setting if per-call overrides win).
|
||||
*/
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
}
|
||||
|
||||
// Config
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain search modes/stats/tune` CLI tests.
|
||||
*
|
||||
* Covers dispatch + JSON output shape + idempotent --reset + recommendation
|
||||
* generation. Pure unit-level: bypasses the cli.ts entrypoint and calls
|
||||
* runSearch directly against a fresh PGLite engine.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSearch } from '../src/commands/search.ts';
|
||||
import { recordSearchTelemetry, _resetTelemetryWriterForTest, getTelemetryWriter } from '../src/core/search/telemetry.ts';
|
||||
import type { HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
_resetTelemetryWriterForTest();
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%' OR key LIKE 'models.%'`);
|
||||
await engine.executeRaw('DELETE FROM search_telemetry');
|
||||
});
|
||||
|
||||
// Capture-stdout helper so we can assert command output without exec'ing.
|
||||
async function captureRun(fn: () => Promise<void>): Promise<string> {
|
||||
const original = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (...args: unknown[]) => { captured.push(args.map(String).join(' ')); };
|
||||
try {
|
||||
await fn();
|
||||
} finally {
|
||||
console.log = original;
|
||||
}
|
||||
return captured.join('\n');
|
||||
}
|
||||
|
||||
const makeMeta = (overrides: Partial<HybridSearchMeta> = {}): HybridSearchMeta => ({
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
mode: 'balanced',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('gbrain search modes (read-only dashboard)', () => {
|
||||
test('--json emits structured report with all 3 bundles and active mode', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.schema_version).toBe(2);
|
||||
expect(report.active_mode).toBe('tokenmax');
|
||||
expect(report.active_mode_valid).toBe(true);
|
||||
expect(report.bundles.conservative.searchLimit).toBe(10);
|
||||
expect(report.bundles.balanced.searchLimit).toBe(25);
|
||||
expect(report.bundles.tokenmax.searchLimit).toBe(50);
|
||||
expect(report.resolved.tokenBudget.source).toBe('mode');
|
||||
});
|
||||
|
||||
test('unset mode → balanced fallback with mode_valid=false', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.active_mode).toBe('balanced');
|
||||
expect(report.active_mode_valid).toBe(false);
|
||||
expect(report.resolved.searchLimit.source).toBe('fallback');
|
||||
});
|
||||
|
||||
test('per-key override shows up with source=override', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.resolved.cache_enabled.value).toBe(false);
|
||||
expect(report.resolved.cache_enabled.source).toBe('override');
|
||||
// Other knobs still come from the mode bundle.
|
||||
expect(report.resolved.searchLimit.source).toBe('mode');
|
||||
});
|
||||
|
||||
test('default text output names the active mode', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes']));
|
||||
expect(out).toContain('tokenmax');
|
||||
expect(out).toContain('conservative');
|
||||
expect(out).toContain('balanced');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search modes --reset', () => {
|
||||
test('--source <mode> is a dry-run (no writes)', async () => {
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '4000');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--source', 'balanced']));
|
||||
expect(out).toContain('dry run');
|
||||
expect(out).toContain('search.cache.enabled');
|
||||
expect(out).toContain('search.tokenBudget');
|
||||
// Verify nothing was deleted.
|
||||
expect(await engine.getConfig('search.cache.enabled')).toBe('false');
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBe('4000');
|
||||
});
|
||||
|
||||
test('--reset clears every search.* override (but NOT search.mode itself)', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
await engine.setConfig('search.searchLimit', '15');
|
||||
await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
// Mode preserved; overrides gone.
|
||||
expect(await engine.getConfig('search.mode')).toBe('conservative');
|
||||
expect(await engine.getConfig('search.cache.enabled')).toBeNull();
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBeNull();
|
||||
expect(await engine.getConfig('search.searchLimit')).toBeNull();
|
||||
});
|
||||
|
||||
test('--reset on a clean install reports "no overrides"', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
expect(out).toContain('No search.* overrides set');
|
||||
});
|
||||
|
||||
test('--reset preserves the upgrade-notice state key', async () => {
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
await engine.setConfig('search.tokenBudget', '4000');
|
||||
await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
// Notice key preserved (it's not an "override"); tokenBudget gone.
|
||||
expect(await engine.getConfig('search.mode_upgrade_notice_shown')).toBe('true');
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search stats', () => {
|
||||
test('empty table → total_calls 0, message about no data', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['stats']));
|
||||
expect(out).toContain('Total searches:');
|
||||
expect(out).toContain('0');
|
||||
});
|
||||
|
||||
test('after telemetry writes → hit rate + intent mix surfaced', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 7 });
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }), { results_count: 9 });
|
||||
recordSearchTelemetry(engine, makeMeta({ intent: 'entity' }), { results_count: 3 });
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['stats', '--json']));
|
||||
const stats = JSON.parse(out);
|
||||
expect(stats.total_calls).toBe(4);
|
||||
expect(stats.cache_hits).toBe(2);
|
||||
expect(stats.cache_misses).toBe(1);
|
||||
expect(stats.cache_hit_rate).toBeCloseTo(2 / 3, 3);
|
||||
expect(stats._meta.metric_glossary.cache_hit_rate).toBeDefined();
|
||||
});
|
||||
|
||||
test('--days N clamps to [1, 365]', async () => {
|
||||
const out0 = await captureRun(() => runSearch(engine, ['stats', '--days', '0', '--json']));
|
||||
expect(JSON.parse(out0).window_days).toBe(1);
|
||||
const outBig = await captureRun(() => runSearch(engine, ['stats', '--days', '9999', '--json']));
|
||||
expect(JSON.parse(outBig).window_days).toBe(365);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search tune (recommendations)', () => {
|
||||
test('insufficient data → no_recommendations status', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
expect(r.status).toBe('insufficient_data');
|
||||
expect(r.recommendations).toEqual([]);
|
||||
});
|
||||
|
||||
test('conservative + high budget drop rate → recommends balanced', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
// 30 calls, each dropping 5 results — strong signal.
|
||||
for (let i = 0; i < 30; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({
|
||||
mode: 'conservative',
|
||||
token_budget: { budget: 4000, used: 4000, kept: 5, dropped: 5 },
|
||||
}), { results_count: 5 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
expect(r.status).toBe('has_recommendations');
|
||||
const modeRec = r.recommendations.find((x: { knob: string }) => x.knob === 'search.mode');
|
||||
expect(modeRec).toBeDefined();
|
||||
expect(modeRec.suggested).toBe('balanced');
|
||||
});
|
||||
|
||||
test('tokenmax + Haiku subagent → recommends balanced', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
await engine.setConfig('models.tier.subagent', 'anthropic:claude-haiku-4-5');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
for (let i = 0; i < 25; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax' }), { results_count: 30 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
const rec = r.recommendations.find((x: { knob: string; suggested: string }) =>
|
||||
x.knob === 'search.mode' && x.suggested === 'balanced'
|
||||
);
|
||||
expect(rec).toBeDefined();
|
||||
expect(rec.reason).toMatch(/Haiku/);
|
||||
});
|
||||
|
||||
test('--apply mutates config', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
for (let i = 0; i < 30; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({
|
||||
mode: 'conservative',
|
||||
token_budget: { budget: 4000, used: 4000, kept: 5, dropped: 5 },
|
||||
}), { results_count: 5 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
await captureRun(() => runSearch(engine, ['tune', '--apply']));
|
||||
expect(await engine.getConfig('search.mode')).toBe('balanced');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search dispatch', () => {
|
||||
test('--help shows usage', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['--help']));
|
||||
expect(out).toContain('Usage:');
|
||||
expect(out).toContain('modes');
|
||||
expect(out).toContain('stats');
|
||||
expect(out).toContain('tune');
|
||||
});
|
||||
|
||||
test('unknown subcommand exits 1', async () => {
|
||||
let exitCode = 0;
|
||||
const originalExit = process.exit;
|
||||
(process.exit as unknown as (code?: number) => void) = ((code?: number) => { exitCode = code ?? 0; throw new Error('exit-' + code); }) as never;
|
||||
const originalErr = console.error;
|
||||
console.error = () => { /* swallow */ };
|
||||
try {
|
||||
await runSearch(engine, ['nonsense']);
|
||||
} catch { /* expected */ }
|
||||
expect(exitCode).toBe(1);
|
||||
process.exit = originalExit;
|
||||
console.error = originalErr;
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Pins the v0.32.3 `gbrain config unset` + `listConfigKeys` engine surface.
|
||||
* Required by `gbrain search modes --reset` [CDX-8].
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'test.%' OR key LIKE 'search.%'`);
|
||||
});
|
||||
|
||||
describe('engine.unsetConfig', () => {
|
||||
test('removes an existing key, returns 1', async () => {
|
||||
await engine.setConfig('test.k1', 'v1');
|
||||
const n = await engine.unsetConfig('test.k1');
|
||||
expect(n).toBe(1);
|
||||
const after = await engine.getConfig('test.k1');
|
||||
expect(after).toBeNull();
|
||||
});
|
||||
|
||||
test('returns 0 for a missing key (no error)', async () => {
|
||||
const n = await engine.unsetConfig('test.never-existed');
|
||||
expect(n).toBe(0);
|
||||
});
|
||||
|
||||
test('does not affect other keys', async () => {
|
||||
await engine.setConfig('test.keep', 'keep');
|
||||
await engine.setConfig('test.remove', 'gone');
|
||||
await engine.unsetConfig('test.remove');
|
||||
expect(await engine.getConfig('test.keep')).toBe('keep');
|
||||
expect(await engine.getConfig('test.remove')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('engine.listConfigKeys prefix-matcher', () => {
|
||||
test('empty when no key matches', async () => {
|
||||
expect(await engine.listConfigKeys('test.')).toEqual([]);
|
||||
});
|
||||
|
||||
test('returns matching keys sorted ascending', async () => {
|
||||
await engine.setConfig('test.b', '2');
|
||||
await engine.setConfig('test.a', '1');
|
||||
await engine.setConfig('test.c', '3');
|
||||
await engine.setConfig('other', 'no');
|
||||
const keys = await engine.listConfigKeys('test.');
|
||||
expect(keys).toEqual(['test.a', 'test.b', 'test.c']);
|
||||
});
|
||||
|
||||
test('prefix is an EXACT literal match — no glob wildcards', async () => {
|
||||
await engine.setConfig('test.matchme', 'yes');
|
||||
// % in user input is escaped — does NOT act as wildcard.
|
||||
expect(await engine.listConfigKeys('test%')).toEqual([]);
|
||||
expect(await engine.listConfigKeys('test.match')).toEqual(['test.matchme']);
|
||||
});
|
||||
|
||||
test('underscore in prefix is escaped (literal _)', async () => {
|
||||
await engine.setConfig('test_underscore', 'val');
|
||||
await engine.setConfig('testXunderscore', 'no');
|
||||
const keys = await engine.listConfigKeys('test_');
|
||||
// test_underscore matches because we asked for prefix "test_"; the
|
||||
// testXunderscore does NOT because the _ is escaped to literal.
|
||||
expect(keys).toEqual(['test_underscore']);
|
||||
});
|
||||
|
||||
test('search.* prefix sweep returns every search-mode override key set', async () => {
|
||||
await engine.setConfig('search.cache.enabled', 'true');
|
||||
await engine.setConfig('search.cache.ttl_seconds', '7200');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
await engine.setConfig('search.mode', 'tokenmax'); // mode key itself
|
||||
const keys = await engine.listConfigKeys('search.');
|
||||
expect(keys.length).toBe(4);
|
||||
expect(keys).toContain('search.cache.enabled');
|
||||
expect(keys).toContain('search.cache.ttl_seconds');
|
||||
expect(keys).toContain('search.tokenBudget');
|
||||
expect(keys).toContain('search.mode');
|
||||
});
|
||||
});
|
||||
|
||||
describe('round-trip: set → unset → get', () => {
|
||||
test('setting and unsetting in a tight loop is idempotent', async () => {
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await engine.setConfig('test.loopkey', `iteration-${i}`);
|
||||
expect(await engine.getConfig('test.loopkey')).toBe(`iteration-${i}`);
|
||||
const n = await engine.unsetConfig('test.loopkey');
|
||||
expect(n).toBe(1);
|
||||
expect(await engine.getConfig('test.loopkey')).toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* v0.32.3 — doctor search_mode + eval_drift check tests.
|
||||
* Pins [CDX-20]: status stays 'ok', no health-score docking; hint lives
|
||||
* in `message`. Tests the two exported helpers directly to avoid the
|
||||
* expensive full runDoctor walk.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { checkSearchMode, checkEvalDrift } from '../src/commands/doctor.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%'`);
|
||||
});
|
||||
|
||||
describe('checkSearchMode [CDX-20]', () => {
|
||||
test('unset mode → ok with hint to pick a mode', async () => {
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.name).toBe('search_mode');
|
||||
expect(c.status).toBe('ok'); // never warn, never dock score
|
||||
expect(c.message).toMatch(/unset/i);
|
||||
expect(c.message).toContain('gbrain search modes');
|
||||
});
|
||||
|
||||
test('mode set, no overrides → ok with "canonical" message', async () => {
|
||||
await engine.setConfig('search.mode', 'balanced');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok');
|
||||
expect(c.message).toContain('balanced');
|
||||
expect(c.message).toContain('canonical');
|
||||
});
|
||||
|
||||
test('mode set + overrides → ok with reset hint + override list', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok'); // [CDX-20]: still ok, never warn
|
||||
expect(c.message).toContain('conservative');
|
||||
expect(c.message).toContain('search.cache.enabled');
|
||||
expect(c.message).toContain('search.tokenBudget');
|
||||
expect(c.message).toContain('gbrain search modes --reset');
|
||||
});
|
||||
|
||||
test('upgrade-notice state key is excluded from override count', async () => {
|
||||
await engine.setConfig('search.mode', 'balanced');
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.message).toContain('no per-key overrides');
|
||||
});
|
||||
|
||||
test('tokenmax mode is recognized without any override warning', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok');
|
||||
expect(c.message).toContain('tokenmax');
|
||||
expect(c.message).toContain('canonical');
|
||||
});
|
||||
});
|
||||
|
||||
describe('checkEvalDrift [CDX-6]', () => {
|
||||
test('returns ok status (never warn — per [CDX-20])', async () => {
|
||||
const c = await checkEvalDrift(engine);
|
||||
expect(c.name).toBe('eval_drift');
|
||||
expect(c.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('message is non-empty (either no-drift or drift summary)', async () => {
|
||||
const c = await checkEvalDrift(engine);
|
||||
expect(c.message).toBeTruthy();
|
||||
expect(c.message.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* v0.32.3 — drift-watch module unit tests.
|
||||
* Pins the curated watch-list + matchesWatchPattern semantics.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
RETRIEVAL_WATCH_PATTERNS,
|
||||
matchesWatchPattern,
|
||||
watchedFilesDrifted,
|
||||
filesDriftedSince,
|
||||
} from '../src/core/eval/drift-watch.ts';
|
||||
|
||||
describe('RETRIEVAL_WATCH_PATTERNS canonical list', () => {
|
||||
test('includes src/core/search/ prefix', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/search/');
|
||||
});
|
||||
|
||||
test('includes the embedding file', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/embedding.ts');
|
||||
});
|
||||
|
||||
test('includes chunkers/ directory', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/chunkers/');
|
||||
});
|
||||
|
||||
test('includes the query operation definition', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/operations.ts');
|
||||
});
|
||||
|
||||
test('is frozen at module load', () => {
|
||||
expect(Object.isFrozen(RETRIEVAL_WATCH_PATTERNS)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('matchesWatchPattern semantics', () => {
|
||||
test('directory pattern matches any descendant', () => {
|
||||
expect(matchesWatchPattern('src/core/search/hybrid.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/search/mode.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/search/deep/nested/file.ts')).toBe(true);
|
||||
});
|
||||
|
||||
test('directory pattern does NOT match a sibling with the same prefix', () => {
|
||||
// src/core/search-related-but-different/foo.ts should NOT match
|
||||
// src/core/search/ because the pattern ends with a slash.
|
||||
expect(matchesWatchPattern('src/core/searchengine.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('src/core/searches/file.ts')).toBe(false);
|
||||
});
|
||||
|
||||
test('bare file pattern requires exact equality', () => {
|
||||
expect(matchesWatchPattern('src/core/embedding.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/embedding.test.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('src/core/embedding')).toBe(false);
|
||||
});
|
||||
|
||||
test('custom patterns work', () => {
|
||||
const custom = ['foo/', 'bar.ts'];
|
||||
expect(matchesWatchPattern('foo/x.ts', custom)).toBe(true);
|
||||
expect(matchesWatchPattern('bar.ts', custom)).toBe(true);
|
||||
expect(matchesWatchPattern('baz.ts', custom)).toBe(false);
|
||||
});
|
||||
|
||||
test('non-matching path returns false', () => {
|
||||
expect(matchesWatchPattern('docs/eval/METRIC_GLOSSARY.md')).toBe(false);
|
||||
expect(matchesWatchPattern('test/foo.test.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('README.md')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('filesDriftedSince + watchedFilesDrifted graceful failure', () => {
|
||||
test('missing repo root returns empty array', () => {
|
||||
expect(filesDriftedSince('/does/not/exist')).toEqual([]);
|
||||
});
|
||||
|
||||
test('watchedFilesDrifted filters through the same matcher', () => {
|
||||
// Smoke test: should not throw on this repo. Could be empty.
|
||||
const out = watchedFilesDrifted(process.cwd());
|
||||
expect(Array.isArray(out)).toBe(true);
|
||||
for (const p of out) {
|
||||
expect(matchesWatchPattern(p)).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,135 @@
|
||||
/**
|
||||
* v0.32.3 — eval-compare report tests.
|
||||
* Pins the markdown + JSON shape and the metric-glossary integration.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { runEvalCompare } from '../src/commands/eval-compare.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-eval-compare-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const SAMPLE_RECORDS = [
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-conservative-42',
|
||||
ran_at: '2026-05-12T10:00:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'conservative',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.71, 'ndcg@10': 0.682 },
|
||||
},
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-balanced-42',
|
||||
ran_at: '2026-05-12T10:05:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'balanced',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.78, 'ndcg@10': 0.741 },
|
||||
},
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-tokenmax-42',
|
||||
ran_at: '2026-05-12T10:10:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'tokenmax',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.81, 'ndcg@10': 0.762 },
|
||||
},
|
||||
];
|
||||
|
||||
function writeJsonl(records: object[]): string {
|
||||
const path = join(tmp, 'eval-results.jsonl');
|
||||
writeFileSync(path, records.map(r => JSON.stringify(r)).join('\n') + '\n', 'utf-8');
|
||||
return path;
|
||||
}
|
||||
|
||||
async function captureRun(fn: () => Promise<void>): Promise<string> {
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
const captured: string[] = [];
|
||||
(process.stdout.write as unknown as (s: string) => boolean) = ((s: string) => { captured.push(s); return true; }) as never;
|
||||
try {
|
||||
await fn();
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
return captured.join('');
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
rmSync(join(tmp, 'eval-results.jsonl'), { force: true });
|
||||
});
|
||||
|
||||
describe('runEvalCompare', () => {
|
||||
test('--json output has schema_version + grouped + _meta.metric_glossary', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.schema_version).toBe(2);
|
||||
expect(report.grouped.longmemeval).toBeDefined();
|
||||
expect(report.grouped.longmemeval.conservative.metrics['recall@10']).toBe(0.71);
|
||||
expect(report.grouped.longmemeval.tokenmax.metrics['ndcg@10']).toBe(0.762);
|
||||
expect(report._meta.metric_glossary['recall@10']).toBeDefined();
|
||||
expect(report._meta.metric_glossary['ndcg@10']).toBeDefined();
|
||||
expect(report._meta.methodology).toContain('bootstrap');
|
||||
});
|
||||
|
||||
test('--md output names every mode + metric', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--md', '--input', path]));
|
||||
expect(out).toContain('# Search Mode Comparison');
|
||||
expect(out).toContain('## longmemeval');
|
||||
expect(out).toContain('conservative');
|
||||
expect(out).toContain('balanced');
|
||||
expect(out).toContain('tokenmax');
|
||||
expect(out).toContain('### recall@10');
|
||||
expect(out).toContain('### ndcg@10');
|
||||
expect(out).toContain('Plain English:'); // Glossary line surfaced
|
||||
});
|
||||
|
||||
test('missing file → friendly hint, no crash', async () => {
|
||||
const path = join(tmp, 'does-not-exist.jsonl');
|
||||
const out = await captureRun(() => runEvalCompare(['--md', '--input', path]));
|
||||
expect(out).toContain('No eval-results.jsonl found');
|
||||
expect(out).toContain('gbrain eval run-all');
|
||||
});
|
||||
|
||||
test('--modes filter narrows the table', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--modes', 'conservative,tokenmax', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
// Records list is filtered.
|
||||
expect(report.records.length).toBe(2);
|
||||
// The grouped table preserves the SEARCH_MODES order (conservative/balanced/tokenmax).
|
||||
// Filtered balanced → null in grouped output.
|
||||
expect(report.grouped.longmemeval.balanced).toBeNull();
|
||||
});
|
||||
|
||||
test('multiple runs for same (suite, mode) → most-recent wins', async () => {
|
||||
const oldRun = { ...SAMPLE_RECORDS[0], ran_at: '2026-05-12T09:00:00Z', metrics: { 'recall@10': 0.5 } };
|
||||
const newRun = { ...SAMPLE_RECORDS[0], ran_at: '2026-05-12T11:00:00Z', metrics: { 'recall@10': 0.9 } };
|
||||
const path = writeJsonl([oldRun, newRun]);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.grouped.longmemeval.conservative.metrics['recall@10']).toBe(0.9);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* v0.32.3 — eval-run-all orchestrator unit tests.
|
||||
* Pins arg parsing + cost-guard semantics + persist hook + audit trail.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, readFileSync, existsSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import {
|
||||
parseRunAllArgs,
|
||||
estimateRunCost,
|
||||
evaluateCostGuard,
|
||||
persistRunRecord,
|
||||
type EvalRunRecord,
|
||||
} from '../src/commands/eval-run-all.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-eval-runall-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('parseRunAllArgs', () => {
|
||||
test('defaults: all modes, longmemeval+replay suites, seed=42', () => {
|
||||
const opts = parseRunAllArgs([]);
|
||||
expect(opts.modes).toEqual(['conservative', 'balanced', 'tokenmax']);
|
||||
expect(opts.suites).toEqual(['longmemeval', 'replay']);
|
||||
expect(opts.seed).toBe(42);
|
||||
expect(opts.parallel).toBe(1);
|
||||
expect(opts.budgetUsdRetrieval).toBe(5);
|
||||
expect(opts.budgetUsdAnswer).toBe(20);
|
||||
expect(opts.yes).toBe(false);
|
||||
});
|
||||
|
||||
test('--modes filters to a subset', () => {
|
||||
const opts = parseRunAllArgs(['--modes', 'conservative,tokenmax']);
|
||||
expect(opts.modes).toEqual(['conservative', 'tokenmax']);
|
||||
});
|
||||
|
||||
test('--modes rejects invalid mode', () => {
|
||||
expect(() => parseRunAllArgs(['--modes', 'frontier'])).toThrow(/not a valid mode/);
|
||||
});
|
||||
|
||||
test('--suites filters; rejects unknown', () => {
|
||||
const opts = parseRunAllArgs(['--suites', 'longmemeval']);
|
||||
expect(opts.suites).toEqual(['longmemeval']);
|
||||
expect(() => parseRunAllArgs(['--suites', 'foo'])).toThrow(/not a recognized suite/);
|
||||
});
|
||||
|
||||
test('--budget-usd-retrieval + --budget-usd-answer override defaults', () => {
|
||||
const opts = parseRunAllArgs(['--budget-usd-retrieval', '15', '--budget-usd-answer', '50']);
|
||||
expect(opts.budgetUsdRetrieval).toBe(15);
|
||||
expect(opts.budgetUsdAnswer).toBe(50);
|
||||
});
|
||||
|
||||
test('--parallel clamps to mode count', () => {
|
||||
const opts = parseRunAllArgs(['--parallel', '10']);
|
||||
expect(opts.parallel).toBe(3); // clamped to SEARCH_MODES.length
|
||||
});
|
||||
|
||||
test('--parallel rejects 0 / negative', () => {
|
||||
expect(() => parseRunAllArgs(['--parallel', '0'])).toThrow(/must be >= 1/);
|
||||
expect(() => parseRunAllArgs(['--parallel', '-1'])).toThrow(/must be >= 1/);
|
||||
});
|
||||
|
||||
test('--yes flag toggles', () => {
|
||||
expect(parseRunAllArgs(['--yes']).yes).toBe(true);
|
||||
expect(parseRunAllArgs(['-y']).yes).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('estimateRunCost', () => {
|
||||
test('returns retrieval/answer/total breakdown', () => {
|
||||
const est = estimateRunCost({ suites: ['longmemeval'], modes: ['balanced'], limit: 100 });
|
||||
expect(est.total_usd).toBe(est.retrieval_usd + est.answer_usd);
|
||||
expect(est.answer_usd).toBeGreaterThan(0); // longmemeval has answer-gen
|
||||
expect(est.retrieval_usd).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('tokenmax incurs expansion cost (Haiku per query)', () => {
|
||||
const bal = estimateRunCost({ suites: ['replay'], modes: ['balanced'], limit: 100 });
|
||||
const tok = estimateRunCost({ suites: ['replay'], modes: ['tokenmax'], limit: 100 });
|
||||
expect(tok.retrieval_usd).toBeGreaterThan(bal.retrieval_usd);
|
||||
});
|
||||
|
||||
test('replay suite has zero answer cost', () => {
|
||||
const est = estimateRunCost({ suites: ['replay'], modes: ['balanced'], limit: 100 });
|
||||
expect(est.answer_usd).toBe(0);
|
||||
});
|
||||
|
||||
test('multi-suite multi-mode estimate sums correctly', () => {
|
||||
const est = estimateRunCost({ suites: ['longmemeval', 'replay'], modes: ['conservative', 'balanced'], limit: 100 });
|
||||
expect(Object.keys(est.per_suite).length).toBe(4); // 2 × 2
|
||||
});
|
||||
});
|
||||
|
||||
describe('evaluateCostGuard', () => {
|
||||
test('within-budget proceeds without --yes', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 1, answer_usd: 10, total_usd: 11 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(true);
|
||||
});
|
||||
|
||||
test('over-cap TTY without --yes refuses', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(false);
|
||||
expect(g.reason).toContain('--yes');
|
||||
});
|
||||
|
||||
test('over-cap TTY with --yes proceeds', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: true, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(true);
|
||||
});
|
||||
|
||||
test('over-cap non-TTY without --yes refuses (exit 2 path)', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(g.proceed).toBe(false);
|
||||
expect(g.reason).toContain('Non-TTY requires --yes');
|
||||
});
|
||||
|
||||
test('only-retrieval-over OR only-answer-over both trigger guard', () => {
|
||||
const a = evaluateCostGuard(
|
||||
{ retrieval_usd: 100, answer_usd: 1, total_usd: 101 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(a.proceed).toBe(false);
|
||||
const b = evaluateCostGuard(
|
||||
{ retrieval_usd: 1, answer_usd: 100, total_usd: 101 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(b.proceed).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('persistRunRecord audit trail', () => {
|
||||
beforeEach(() => {
|
||||
rmSync(join(tmp, '.gbrain-evals'), { recursive: true, force: true });
|
||||
rmSync(join(tmp, 'eval-results.jsonl'), { force: true });
|
||||
});
|
||||
|
||||
test('appends to eval-results.jsonl, creates dir if missing', () => {
|
||||
const record: EvalRunRecord = {
|
||||
schema_version: 2,
|
||||
run_id: 'abc123-longmemeval-conservative-42',
|
||||
ran_at: '2026-05-12T12:00:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'conservative',
|
||||
commit: 'abc123',
|
||||
seed: 42,
|
||||
params: {},
|
||||
status: 'completed',
|
||||
duration_ms: 12_345,
|
||||
};
|
||||
persistRunRecord(tmp, record, tmp);
|
||||
const path = join(tmp, 'eval-results.jsonl');
|
||||
expect(existsSync(path)).toBe(true);
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
expect(content).toContain('abc123-longmemeval-conservative-42');
|
||||
const parsed = JSON.parse(content.trim());
|
||||
expect(parsed.mode).toBe('conservative');
|
||||
expect(parsed.schema_version).toBe(2);
|
||||
});
|
||||
|
||||
test('appends multiple records (NDJSON)', () => {
|
||||
const base = {
|
||||
schema_version: 2 as const,
|
||||
ran_at: '2026-05-12T12:00:00Z',
|
||||
suite: 'longmemeval' as const,
|
||||
commit: 'abc',
|
||||
seed: 42,
|
||||
params: {},
|
||||
status: 'completed' as const,
|
||||
duration_ms: 1,
|
||||
};
|
||||
persistRunRecord(tmp, { ...base, run_id: 'a', mode: 'conservative' }, tmp);
|
||||
persistRunRecord(tmp, { ...base, run_id: 'b', mode: 'balanced' }, tmp);
|
||||
persistRunRecord(tmp, { ...base, run_id: 'c', mode: 'tokenmax' }, tmp);
|
||||
const content = readFileSync(join(tmp, 'eval-results.jsonl'), 'utf-8');
|
||||
const lines = content.trim().split('\n');
|
||||
expect(lines.length).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 hybridSearchCached integration.
|
||||
*
|
||||
* End-to-end PGLite test that confirms the three search-lite features
|
||||
* fire through the actual hybrid pipeline (not the units in isolation):
|
||||
*
|
||||
* 1. Token budget: results are capped after search.
|
||||
* 2. Cache: meta surfaces hit/miss; disabled mode is a clean pass-through.
|
||||
* 3. Intent classifier: meta.intent matches the classifier output.
|
||||
*
|
||||
* Vector search isn't enabled (no embedding provider in test), so we
|
||||
* exercise the keyword-only path \u2014 which still surfaces intent and
|
||||
* budget. The cache path is exercised separately in query-cache.test.ts
|
||||
* because it needs a real embedding to key on.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { hybridSearchCached } from '../src/core/search/hybrid.ts';
|
||||
import type { PageInput, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const savedKey = process.env.OPENAI_API_KEY;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
// Insert a small fixture set so keyword search has something to find.
|
||||
// Use long chunk_texts so token budget cuts have observable effect.
|
||||
const longText = 'x'.repeat(800); // ~200 tokens of body text
|
||||
const pages: Array<{ slug: string; page: PageInput }> = [
|
||||
{
|
||||
slug: 'alice-foo',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Alice Foo',
|
||||
compiled_truth: `Alice Foo is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
{
|
||||
slug: 'bob-bar',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Bob Bar',
|
||||
compiled_truth: `Bob Bar is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
{
|
||||
slug: 'carol-baz',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Carol Baz',
|
||||
compiled_truth: `Carol Baz is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
];
|
||||
for (const p of pages) {
|
||||
await engine.putPage(p.slug, p.page);
|
||||
}
|
||||
// Force keyword-only fallback by unsetting the embedding provider key.
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedKey) process.env.OPENAI_API_KEY = savedKey;
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 meta surfaces intent', () => {
|
||||
test('entity query classifies as entity', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('entity');
|
||||
});
|
||||
|
||||
test('temporal query classifies as temporal', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'what happened last week', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('temporal');
|
||||
});
|
||||
|
||||
test('event query classifies as event', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who raised $10M', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('event');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 token budget', () => {
|
||||
test('budget undefined returns no token_budget meta (no cut)', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'alice', {
|
||||
limit: 10,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// Don't assert non-empty here — keyword tokenization depends on the
|
||||
// pglite analyzer config. What matters: meta is shaped right and
|
||||
// budget metadata is absent when budget isn't set.
|
||||
expect(results).toBeDefined();
|
||||
expect(meta?.token_budget).toBeUndefined();
|
||||
});
|
||||
|
||||
test('budget meta is always emitted when budget is set', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'alice', {
|
||||
limit: 10,
|
||||
tokenBudget: 250,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.token_budget).toBeDefined();
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
});
|
||||
|
||||
test('tight budget cuts the result set', async () => {
|
||||
// First find out the result count without a budget so the assertion
|
||||
// is robust to the fixture’s actual chunking.
|
||||
const unbounded = await hybridSearchCached(engine, 'builder', { limit: 10 });
|
||||
// Skip the cut test if the fixture happens to return only one row
|
||||
// (keyword search may dedupe by page); the budget enforcement itself
|
||||
// is exhaustively unit-tested in test/token-budget.test.ts.
|
||||
if (unbounded.length < 2) return;
|
||||
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'builder', {
|
||||
limit: 10,
|
||||
tokenBudget: 250, // enough for ~1 row of fixture data
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
expect(meta?.token_budget?.dropped).toBeGreaterThan(0);
|
||||
// The budget must hold: cumulative cost <= budget.
|
||||
expect(meta?.token_budget?.used).toBeLessThanOrEqual(250);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 cache disabled fallback', () => {
|
||||
test('keyword-only path emits cache.status=disabled', async () => {
|
||||
// No embedding available \u2192 cache decision degrades to disabled.
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// cache may be 'disabled' (no embedding provider) or 'miss'.
|
||||
// Either way the field exists.
|
||||
expect(meta?.cache).toBeDefined();
|
||||
expect(['disabled', 'miss']).toContain(meta?.cache?.status ?? '');
|
||||
});
|
||||
|
||||
test('useCache=false explicitly disables', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is bob-bar', {
|
||||
limit: 5,
|
||||
useCache: false,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.cache?.status).toBe('disabled');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 intent weighting toggle', () => {
|
||||
test('intentWeighting=false still emits intent in meta (for visibility)', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
intentWeighting: false,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// Intent classification itself still runs (cheap regex); only the
|
||||
// weight adjustment is disabled. So meta.intent stays populated.
|
||||
expect(meta?.intent).toBe('entity');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,251 @@
|
||||
/**
|
||||
* v0.32.3 search-lite install-time mode picker tests.
|
||||
*
|
||||
* Pure-function coverage (recommendModeFor + parseModeInput) plus the
|
||||
* idempotent runModePicker behavior. The interactive TTY branch is
|
||||
* exercised indirectly via the non-TTY path here; full TTY simulation
|
||||
* lives in the e2e suite (test/e2e/...).
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
recommendModeFor,
|
||||
parseModeInput,
|
||||
runModePicker,
|
||||
} from '../src/commands/init-mode-picker.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%' OR key LIKE 'models.%'`);
|
||||
});
|
||||
|
||||
describe('recommendModeFor — auto-suggestion heuristic', () => {
|
||||
test('Opus default → tokenmax', () => {
|
||||
const r = recommendModeFor({ defaultModel: 'anthropic:claude-opus-4-7' });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
expect(r.reason).toMatch(/Opus/);
|
||||
});
|
||||
|
||||
test('Opus subagent → tokenmax', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-opus-4-7' });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('Haiku subagent → conservative', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-haiku-4-5' });
|
||||
expect(r.mode).toBe('conservative');
|
||||
expect(r.reason).toMatch(/Haiku/);
|
||||
});
|
||||
|
||||
test('No OpenAI key → conservative (no LLM expansion possible)', () => {
|
||||
const r = recommendModeFor({ hasOpenAIKey: false });
|
||||
expect(r.mode).toBe('conservative');
|
||||
expect(r.reason).toMatch(/No OpenAI/);
|
||||
});
|
||||
|
||||
test('Sonnet / unknown → tokenmax (preserve-v0.31.x default)', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-sonnet-4-6', hasOpenAIKey: true });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
expect(r.reason).toMatch(/v0\.31\.x|preserve/i);
|
||||
});
|
||||
|
||||
test('Empty inputs → tokenmax (preserve-v0.31.x default)', () => {
|
||||
const r = recommendModeFor({});
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
});
|
||||
|
||||
|
||||
test('Haiku subagent wins over Opus default (cost-sensitive takes precedence)', () => {
|
||||
// Reordered in the install-picker DX pass: Haiku check fires BEFORE Opus
|
||||
// because a user running a Haiku subagent loop is signalling cost
|
||||
// sensitivity. Tokenmax over Haiku would silently dump 50-chunk payloads
|
||||
// into a model that struggles past 5-10 chunks. The Haiku floor wins.
|
||||
const r = recommendModeFor({
|
||||
defaultModel: 'anthropic:claude-opus-4-7',
|
||||
subagentModel: 'anthropic:claude-haiku-4-5',
|
||||
hasOpenAIKey: true,
|
||||
});
|
||||
expect(r.mode).toBe('conservative');
|
||||
});
|
||||
});
|
||||
|
||||
describe('MENU_TEXT cost-matrix anchors (must match CLAUDE.md + methodology doc)', () => {
|
||||
test('25x corner-to-corner spread framing is named', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Updating these REQUIRES bumping CLAUDE.md ## Search Mode + the
|
||||
// methodology doc + README in lockstep.
|
||||
expect(MODE_PICKER_MENU).toContain('25x');
|
||||
expect(MODE_PICKER_MENU).toContain('corner-to-corner');
|
||||
});
|
||||
|
||||
test('cost matrix lists every cell at the natural diagonal and corners', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Three anchor cells from the natural diagonal at 10K/mo volume. Scales
|
||||
// linearly; multiplying by 10 gives the 100K/mo numbers the methodology
|
||||
// doc + CLAUDE.md cite ("multiply by 10 for 100K/mo" prose anchor).
|
||||
expect(MODE_PICKER_MENU).toContain('$40/mo'); // conservative + Haiku
|
||||
expect(MODE_PICKER_MENU).toContain('$300/mo'); // balanced + Sonnet (default natural)
|
||||
expect(MODE_PICKER_MENU).toContain('$1,000/mo'); // tokenmax + Opus
|
||||
});
|
||||
|
||||
test('volume frame is 10K queries/month with linear-scale callout', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('10K queries/mo');
|
||||
expect(MODE_PICKER_MENU).toContain('scales linearly');
|
||||
});
|
||||
|
||||
test('all three downstream model rates are explicit', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('Haiku 4.5');
|
||||
expect(MODE_PICKER_MENU).toContain('Sonnet 4.6');
|
||||
expect(MODE_PICKER_MENU).toContain('Opus 4.7');
|
||||
expect(MODE_PICKER_MENU).toContain('$1/M');
|
||||
expect(MODE_PICKER_MENU).toContain('$3/M');
|
||||
expect(MODE_PICKER_MENU).toContain('$5/M');
|
||||
});
|
||||
|
||||
test('tokenmax Haiku-expansion surcharge is named explicitly', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Cross-line match — the surcharge phrase can wrap.
|
||||
expect(MODE_PICKER_MENU.replace(/\s+/g, ' ')).toContain('~$1.50 per 1K queries');
|
||||
expect(MODE_PICKER_MENU).toContain('Haiku expansion call');
|
||||
});
|
||||
|
||||
test('cache-hit discount framing is named', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('cache');
|
||||
// The numbers below are full-payload pre-cache; real loops see 50-80% discount.
|
||||
expect(MODE_PICKER_MENU).toMatch(/50-80%|discount/);
|
||||
});
|
||||
|
||||
test('tune command is surfaced as the next step', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('gbrain search tune');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseModeInput — menu choice mapper', () => {
|
||||
test('numeric 1/2/3 → conservative/balanced/tokenmax', () => {
|
||||
expect(parseModeInput('1')).toBe('conservative');
|
||||
expect(parseModeInput('2')).toBe('balanced');
|
||||
expect(parseModeInput('3')).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('mode names (case-insensitive)', () => {
|
||||
expect(parseModeInput('conservative')).toBe('conservative');
|
||||
expect(parseModeInput('CONSERVATIVE')).toBe('conservative');
|
||||
expect(parseModeInput('TokenMax')).toBe('tokenmax');
|
||||
expect(parseModeInput(' balanced ')).toBe('balanced');
|
||||
});
|
||||
|
||||
test('empty / unrecognized → null', () => {
|
||||
expect(parseModeInput('')).toBeNull();
|
||||
expect(parseModeInput(' ')).toBeNull();
|
||||
expect(parseModeInput('foo')).toBeNull();
|
||||
expect(parseModeInput('4')).toBeNull();
|
||||
expect(parseModeInput('0')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runModePicker non-TTY surfaces full matrix + [AGENT] directive', () => {
|
||||
test('non-TTY output includes the cost matrix and the [AGENT] directive', async () => {
|
||||
const originalLog = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (msg: string) => { captured.push(String(msg)); };
|
||||
try {
|
||||
await runModePicker(engine);
|
||||
} finally {
|
||||
console.log = originalLog;
|
||||
}
|
||||
const out = captured.join('\n');
|
||||
// Matrix corners + diagonal mid
|
||||
expect(out).toContain('$40/mo');
|
||||
expect(out).toContain('$300/mo');
|
||||
expect(out).toContain('$1,000/mo');
|
||||
// 25x spread framing
|
||||
expect(out).toContain('25x corner-to-corner');
|
||||
// Explicit agent directive — load-bearing for agent-platform install paths
|
||||
expect(out).toContain('[AGENT]');
|
||||
expect(out.toLowerCase()).toContain('show this matrix');
|
||||
expect(out).toContain('INSTALL_FOR_AGENTS.md');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runModePicker — non-TTY auto-select + idempotent', () => {
|
||||
test('non-TTY auto-selects + writes config + emits operator hint', async () => {
|
||||
// Bun test runs non-TTY by default.
|
||||
const picked = await runModePicker(engine);
|
||||
// Default model unset → balanced. Should write search.mode.
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(picked);
|
||||
});
|
||||
|
||||
test('idempotent: second call returns existing mode without overwrite', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('tokenmax');
|
||||
// No accidental overwrite.
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('--force re-prompts even if mode is already set', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
// Non-TTY + force → re-runs auto-suggest with current inputs.
|
||||
const picked = await runModePicker(engine, { force: true });
|
||||
// With no model hints + no API key state, default is balanced. The picker
|
||||
// will overwrite the existing mode.
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(picked);
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
});
|
||||
|
||||
test('jsonOutput mode emits a structured event and writes config', async () => {
|
||||
// Capture console.log output.
|
||||
const originalLog = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (msg: string) => { captured.push(msg); };
|
||||
try {
|
||||
const picked = await runModePicker(engine, { jsonOutput: true });
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
const jsonLine = captured.find(l => l.startsWith('{'));
|
||||
expect(jsonLine).toBeDefined();
|
||||
const obj = JSON.parse(jsonLine!);
|
||||
expect(obj.phase).toBe('search_mode_picker');
|
||||
expect(obj.auto).toBe(true);
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(obj.mode);
|
||||
} finally {
|
||||
console.log = originalLog;
|
||||
}
|
||||
});
|
||||
|
||||
test('Opus default model + OpenAI key → picker auto-recommends tokenmax', async () => {
|
||||
// OPENAI_API_KEY must be present — the no-key short-circuit fires before
|
||||
// the Opus check (gbrain can't do vector search without embeddings).
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test-stub' }, async () => {
|
||||
await engine.setConfig('models.default', 'anthropic:claude-opus-4-7');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('tokenmax');
|
||||
});
|
||||
});
|
||||
|
||||
test('Haiku subagent → picker auto-recommends conservative', async () => {
|
||||
await engine.setConfig('models.tier.subagent', 'anthropic:claude-haiku-4-5');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('conservative');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 intent \u2192 weight adjustment tests.
|
||||
*
|
||||
* Pure module under test (no DB). Confirms:
|
||||
* - weightsForIntent returns the expected per-intent factors
|
||||
* - effectiveRrfK scales correctly with the weight
|
||||
* - applyExactMatchBoost only fires on exact slug/title matches
|
||||
* - general intent is a no-op (preserves pre-v0.32 behavior)
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
weightsForIntent,
|
||||
effectiveRrfK,
|
||||
applyExactMatchBoost,
|
||||
} from '../src/core/search/intent-weights.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function makeResult(overrides: Partial<SearchResult> = {}): SearchResult {
|
||||
return {
|
||||
slug: 'test-page',
|
||||
page_id: 1,
|
||||
title: 'Test Title',
|
||||
type: 'concept',
|
||||
chunk_text: 'test chunk text',
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('weightsForIntent', () => {
|
||||
test('general intent is the identity', () => {
|
||||
const w = weightsForIntent('general');
|
||||
expect(w.keywordWeight).toBe(1.0);
|
||||
expect(w.vectorWeight).toBe(1.0);
|
||||
expect(w.suggestedRecency).toBe(null);
|
||||
expect(w.exactMatchBoost).toBe(1.0);
|
||||
});
|
||||
|
||||
test('entity intent boosts keyword + exact match', () => {
|
||||
const w = weightsForIntent('entity');
|
||||
expect(w.keywordWeight).toBeGreaterThan(1.0);
|
||||
expect(w.exactMatchBoost).toBeGreaterThan(1.0);
|
||||
expect(w.suggestedRecency).toBe(null);
|
||||
});
|
||||
|
||||
test('temporal intent suggests recency=on', () => {
|
||||
const w = weightsForIntent('temporal');
|
||||
expect(w.suggestedRecency).toBe('on');
|
||||
});
|
||||
|
||||
test('event intent boosts keyword (rare named entities)', () => {
|
||||
const w = weightsForIntent('event');
|
||||
expect(w.keywordWeight).toBeGreaterThan(1.0);
|
||||
expect(w.suggestedRecency).toBe('on');
|
||||
});
|
||||
});
|
||||
|
||||
describe('effectiveRrfK', () => {
|
||||
test('weight=1.0 returns base k unchanged', () => {
|
||||
expect(effectiveRrfK(60, 1.0)).toBe(60);
|
||||
});
|
||||
|
||||
test('weight > 1 lowers k (stronger top-rank contribution)', () => {
|
||||
expect(effectiveRrfK(60, 1.2)).toBeLessThan(60);
|
||||
expect(effectiveRrfK(60, 1.2)).toBe(50);
|
||||
});
|
||||
|
||||
test('weight <= 0 returns base k (safety)', () => {
|
||||
expect(effectiveRrfK(60, 0)).toBe(60);
|
||||
expect(effectiveRrfK(60, -1)).toBe(60);
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyExactMatchBoost', () => {
|
||||
test('boost=1.0 is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'foo', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'foo', weightsForIntent('general'));
|
||||
expect(results[0].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('exact slug match gets boosted (entity intent)', () => {
|
||||
const results = [
|
||||
makeResult({ slug: 'garry-tan', score: 1.0, title: 'Garry Tan' }),
|
||||
makeResult({ slug: 'someone-else', score: 1.0, title: 'Someone Else' }),
|
||||
];
|
||||
applyExactMatchBoost(results, 'garry-tan', weightsForIntent('entity'));
|
||||
// First result has slug=query \u2192 boosted; second is unchanged.
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
expect(results[1].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('query with spaces matches kebab slug ("garry tan" \u2192 "garry-tan")', () => {
|
||||
const results = [makeResult({ slug: 'garry-tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry tan', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
|
||||
test('exact title match (case-insensitive) gets boosted', () => {
|
||||
const results = [makeResult({ slug: 'random-slug', title: 'Garry Tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'GARRY TAN', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
|
||||
test('partial / substring match does NOT trigger boost', () => {
|
||||
const results = [makeResult({ slug: 'garry-tan', title: 'Garry Tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('namespaced slug matches via suffix ("people/garry-tan" + query "garry-tan")', () => {
|
||||
const results = [makeResult({ slug: 'people/garry-tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry-tan', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* v0.32.3 — metric glossary module tests.
|
||||
* Pins the public surface that drives gbrain search stats / eval compare
|
||||
* output AND the auto-generated METRIC_GLOSSARY.md doc.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
METRIC_GLOSSARY,
|
||||
ALL_METRICS,
|
||||
getMetricGloss,
|
||||
eli10For,
|
||||
buildMetricGlossaryMeta,
|
||||
renderMetricGlossaryMarkdown,
|
||||
} from '../src/core/eval/metric-glossary.ts';
|
||||
|
||||
describe('METRIC_GLOSSARY canonical entries', () => {
|
||||
test('every retrieval-IR metric is present', () => {
|
||||
for (const m of ['precision@k', 'recall@k', 'mrr', 'ndcg@k']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every stability metric is present', () => {
|
||||
for (const m of ['jaccard@k', 'top1_stability']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every stat-significance metric is present', () => {
|
||||
for (const m of ['p_value', 'confidence_interval']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every operational metric is present', () => {
|
||||
for (const m of ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every entry has the three required fields', () => {
|
||||
for (const [key, entry] of Object.entries(METRIC_GLOSSARY)) {
|
||||
expect(entry.industry_term, `${key}.industry_term`).toBeTruthy();
|
||||
expect(entry.eli10, `${key}.eli10`).toBeTruthy();
|
||||
expect(entry.range, `${key}.range`).toBeTruthy();
|
||||
}
|
||||
});
|
||||
|
||||
test('industry_term preserved verbatim (case-sensitive)', () => {
|
||||
expect(METRIC_GLOSSARY['ndcg@k'].industry_term).toBe('Normalized Discounted Cumulative Gain at k (nDCG@k)');
|
||||
expect(METRIC_GLOSSARY['mrr'].industry_term).toBe('Mean Reciprocal Rank (MRR)');
|
||||
});
|
||||
});
|
||||
|
||||
describe('getMetricGloss + eli10For accessors', () => {
|
||||
test('getMetricGloss returns the full entry', () => {
|
||||
const g = getMetricGloss('recall@k');
|
||||
expect(g).not.toBeNull();
|
||||
expect(g!.industry_term).toContain('Recall at k');
|
||||
});
|
||||
|
||||
test('getMetricGloss returns null for unknown metrics', () => {
|
||||
expect(getMetricGloss('made_up_metric')).toBeNull();
|
||||
});
|
||||
|
||||
test('eli10For returns just the plain-English line', () => {
|
||||
const text = eli10For('cache_hit_rate');
|
||||
expect(text).toContain('Fraction of searches');
|
||||
});
|
||||
|
||||
test('eli10For returns null for unknown metric', () => {
|
||||
expect(eli10For('nope')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildMetricGlossaryMeta', () => {
|
||||
test('returns a flat record keyed by metric name → eli10 string', () => {
|
||||
const meta = buildMetricGlossaryMeta(['recall@k', 'mrr']);
|
||||
expect(Object.keys(meta).sort()).toEqual(['mrr', 'recall@k']);
|
||||
expect(meta['recall@k']).toContain('relevant');
|
||||
expect(meta['mrr']).toContain('FIRST relevant result');
|
||||
});
|
||||
|
||||
test('unknown metrics silently dropped (no error)', () => {
|
||||
const meta = buildMetricGlossaryMeta(['recall@k', 'made_up']);
|
||||
expect(meta['recall@k']).toBeDefined();
|
||||
expect(meta['made_up']).toBeUndefined();
|
||||
});
|
||||
|
||||
test('empty array → empty object', () => {
|
||||
expect(buildMetricGlossaryMeta([])).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe('ALL_METRICS roster', () => {
|
||||
test('has every glossary key', () => {
|
||||
expect([...ALL_METRICS].sort()).toEqual(Object.keys(METRIC_GLOSSARY).sort());
|
||||
});
|
||||
|
||||
test('matches the renderer output (no orphans)', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
for (const m of ALL_METRICS) {
|
||||
const entry = METRIC_GLOSSARY[m];
|
||||
expect(md, `Markdown should mention ${m}`).toContain(entry.industry_term);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('renderMetricGlossaryMarkdown determinism', () => {
|
||||
test('repeated calls produce identical output (deterministic)', () => {
|
||||
const a = renderMetricGlossaryMarkdown();
|
||||
const b = renderMetricGlossaryMarkdown();
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('output includes the auto-generated marker', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
expect(md).toContain('Auto-generated from `src/core/eval/metric-glossary.ts`');
|
||||
});
|
||||
|
||||
test('output groups metrics into 4 sections', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
expect(md).toContain('## Retrieval Metrics');
|
||||
expect(md).toContain('## Set-Similarity / Stability Metrics');
|
||||
expect(md).toContain('## Statistical-Significance Metrics');
|
||||
expect(md).toContain('## Operational / Cost Metrics');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,217 @@
|
||||
/**
|
||||
* Regression test for [CDX-4] cross-mode cache contamination.
|
||||
*
|
||||
* Before v0.32.3 (PR #897 as merged), the query_cache primary key was
|
||||
* sha256(source_id::query_text) — a tokenmax search (expansion=on, limit=50)
|
||||
* would populate a row that a subsequent conservative call (no expansion,
|
||||
* limit=10) read back, serving the wrong-shape results.
|
||||
*
|
||||
* After v0.32.3:
|
||||
* - cacheRowId(query, source, knobsHash) — knobsHash is part of the PK
|
||||
* - SemanticQueryCache.lookup({knobsHash}) filters WHERE knobs_hash = $
|
||||
* - SemanticQueryCache.store({knobsHash}) writes the resolved hash
|
||||
*
|
||||
* This test exercises the cache class directly on a fresh PGLite brain
|
||||
* to verify cross-mode writes don't collide.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
import { knobsHash, resolveSearchMode } from '../src/core/search/mode.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
const conservativeHash = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const balancedHash = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
const tokenmaxHash = knobsHash(resolveSearchMode({ mode: 'tokenmax' }));
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clear cache before each test so writes start from empty state.
|
||||
await engine.executeRaw('DELETE FROM query_cache');
|
||||
});
|
||||
|
||||
const makeEmbedding = (seed: number): Float32Array => {
|
||||
const arr = new Float32Array(1536);
|
||||
for (let i = 0; i < 1536; i++) {
|
||||
arr[i] = Math.sin(seed + i * 0.001);
|
||||
}
|
||||
// Normalize to unit length so cosine similarity is well-defined.
|
||||
let norm = 0;
|
||||
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
};
|
||||
|
||||
const makeResults = (label: string, n: number): SearchResult[] =>
|
||||
Array.from({ length: n }, (_, i) => ({
|
||||
slug: `${label}/result-${i}`,
|
||||
title: `${label} result ${i}`,
|
||||
chunk_text: `chunk-${label}-${i}`,
|
||||
chunk_id: (i + 1) * 1000,
|
||||
score: 1 / (i + 1),
|
||||
chunk_index: i,
|
||||
type: 'note' as const,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
page_id: i + 1,
|
||||
stale: false,
|
||||
}));
|
||||
|
||||
describe('cacheRowId is bifurcated by knobsHash', () => {
|
||||
test('same (query, source) but different knobs → different row IDs', () => {
|
||||
const id1 = cacheRowId('what is the meaning of life', 'default', conservativeHash);
|
||||
const id2 = cacheRowId('what is the meaning of life', 'default', tokenmaxHash);
|
||||
expect(id1).not.toBe(id2);
|
||||
});
|
||||
|
||||
test('same (query, source, knobs) → same row ID (idempotent)', () => {
|
||||
const id1 = cacheRowId('what is the meaning of life', 'default', balancedHash);
|
||||
const id2 = cacheRowId('what is the meaning of life', 'default', balancedHash);
|
||||
expect(id1).toBe(id2);
|
||||
});
|
||||
|
||||
test('empty knobsHash still produces a valid ID (test-fixture compatibility)', () => {
|
||||
const id = cacheRowId('q', 'default', '');
|
||||
expect(id).toMatch(/^[0-9a-f]{32}$/);
|
||||
});
|
||||
|
||||
test('all three mode hashes are distinct', () => {
|
||||
expect(conservativeHash).not.toBe(balancedHash);
|
||||
expect(balancedHash).not.toBe(tokenmaxHash);
|
||||
expect(conservativeHash).not.toBe(tokenmaxHash);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
test('tokenmax write does NOT contaminate conservative lookup', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(1);
|
||||
const tokenmaxResults = makeResults('tokenmax', 50);
|
||||
|
||||
// Write under tokenmax knobs.
|
||||
await cache.store('what is the meaning of life', emb, tokenmaxResults, {
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: true,
|
||||
}, { knobsHash: tokenmaxHash });
|
||||
|
||||
// Lookup under conservative knobs with the same embedding → MISS.
|
||||
const conservativeHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(conservativeHit.hit).toBe(false);
|
||||
|
||||
// Lookup under tokenmax knobs with the same embedding → HIT.
|
||||
const tokenmaxHit = await cache.lookup(emb, { knobsHash: tokenmaxHash });
|
||||
expect(tokenmaxHit.hit).toBe(true);
|
||||
expect(tokenmaxHit.results?.length).toBe(50);
|
||||
expect(tokenmaxHit.results?.[0].slug).toBe('tokenmax/result-0');
|
||||
});
|
||||
|
||||
test('three modes coexist as distinct rows for the same query', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(2);
|
||||
|
||||
await cache.store('q', emb, makeResults('conservative', 10), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: conservativeHash });
|
||||
await cache.store('q', emb, makeResults('balanced', 25), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
await cache.store('q', emb, makeResults('tokenmax', 50), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: true,
|
||||
}, { knobsHash: tokenmaxHash });
|
||||
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM query_cache WHERE query_text = 'q'`,
|
||||
);
|
||||
expect(rows[0].n).toBe(3);
|
||||
|
||||
const cHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(cHit.hit).toBe(true);
|
||||
expect(cHit.results?.length).toBe(10);
|
||||
expect(cHit.results?.[0].slug).toBe('conservative/result-0');
|
||||
|
||||
const bHit = await cache.lookup(emb, { knobsHash: balancedHash });
|
||||
expect(bHit.hit).toBe(true);
|
||||
expect(bHit.results?.length).toBe(25);
|
||||
|
||||
const tHit = await cache.lookup(emb, { knobsHash: tokenmaxHash });
|
||||
expect(tHit.hit).toBe(true);
|
||||
expect(tHit.results?.length).toBe(50);
|
||||
});
|
||||
|
||||
test('legacy rows (NULL knobs_hash) are excluded from lookup', async () => {
|
||||
// Manually insert a row with NULL knobs_hash (simulating pre-v0.32.3 state).
|
||||
const emb = makeEmbedding(3);
|
||||
const vecStr = `[${Array.from(emb).map(v => v.toFixed(6)).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, created_at)
|
||||
VALUES ($1, $2, $3, NULL, $4::vector, $5::jsonb, $6::jsonb, 3600, now())`,
|
||||
[
|
||||
'legacy-row-id',
|
||||
'legacy-query',
|
||||
'default',
|
||||
vecStr,
|
||||
JSON.stringify(makeResults('legacy', 5)),
|
||||
JSON.stringify({ vector_enabled: true, detail_resolved: null, expansion_applied: false }),
|
||||
],
|
||||
);
|
||||
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
// Any mode's lookup → MISS (NULL row is excluded by the knobs_hash filter).
|
||||
expect((await cache.lookup(emb, { knobsHash: conservativeHash })).hit).toBe(false);
|
||||
expect((await cache.lookup(emb, { knobsHash: balancedHash })).hit).toBe(false);
|
||||
expect((await cache.lookup(emb, { knobsHash: tokenmaxHash })).hit).toBe(false);
|
||||
});
|
||||
|
||||
test('same mode written twice updates in place (no duplicate rows)', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(4);
|
||||
|
||||
await cache.store('q', emb, makeResults('first', 5), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
await cache.store('q', emb, makeResults('second', 7), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM query_cache WHERE query_text = 'q'`,
|
||||
);
|
||||
expect(rows[0].n).toBe(1);
|
||||
|
||||
const hit = await cache.lookup(emb, { knobsHash: balancedHash });
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.results?.length).toBe(7);
|
||||
expect(hit.results?.[0].slug).toBe('second/result-0');
|
||||
});
|
||||
|
||||
test('empty knobsHash arg writes a row but does not collide with mode-hash rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(5);
|
||||
|
||||
await cache.store('q', emb, makeResults('no-mode', 3), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
});
|
||||
|
||||
// No-mode lookup hits its own row.
|
||||
const noModeHit = await cache.lookup(emb);
|
||||
expect(noModeHit.hit).toBe(true);
|
||||
expect(noModeHit.results?.length).toBe(3);
|
||||
|
||||
// Conservative-hash lookup misses (the no-mode row had empty hash).
|
||||
const conservativeHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(conservativeHit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,246 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 semantic query cache.
|
||||
*
|
||||
* PGLite-backed test. Confirms:
|
||||
* - migration v51 creates the query_cache table
|
||||
* - store + lookup roundtrip with EXACT same embedding \u2192 hit
|
||||
* - lookup with a similar embedding (cosine > 0.92) \u2192 hit
|
||||
* - lookup with a far embedding \u2192 miss
|
||||
* - TTL expiration: a stale row is skipped at read time
|
||||
* - clear / prune / stats work as advertised
|
||||
* - source_id isolation: brain A's cache doesn't leak to brain B
|
||||
* - disabled cache is a pure no-op
|
||||
*
|
||||
* Uses synthetic Float32Array embeddings so the test doesn't depend on
|
||||
* any external embedding provider.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// Build a stable, normalized embedding. PGLite ships pgvector with 1536-dim
|
||||
// support (the default); a smaller test dim won't match the column. We
|
||||
// truncate / pad to 1536 to match the migration's resolved dim.
|
||||
const DIM = 1536;
|
||||
|
||||
function makeEmbedding(seed: number, dim = DIM): Float32Array {
|
||||
const e = new Float32Array(dim);
|
||||
// Simple deterministic generator with a unique fingerprint per seed
|
||||
// so similar seeds produce similar (cosine > 0.95) vectors and distinct
|
||||
// seeds produce orthogonal-ish ones.
|
||||
for (let i = 0; i < dim; i++) {
|
||||
e[i] = Math.sin(seed * 0.001 + i * 0.01);
|
||||
}
|
||||
// L2-normalize so cosine = dot product.
|
||||
let mag = 0;
|
||||
for (let i = 0; i < dim; i++) mag += e[i] * e[i];
|
||||
mag = Math.sqrt(mag);
|
||||
if (mag > 0) for (let i = 0; i < dim; i++) e[i] /= mag;
|
||||
return e;
|
||||
}
|
||||
|
||||
function makeOrthogonalEmbedding(seed: number, dim = DIM): Float32Array {
|
||||
// Use a totally different basis so cosine is near-zero.
|
||||
const e = new Float32Array(dim);
|
||||
for (let i = 0; i < dim; i++) {
|
||||
e[i] = Math.cos(seed * 13.7 + i * 0.97);
|
||||
}
|
||||
let mag = 0;
|
||||
for (let i = 0; i < dim; i++) mag += e[i] * e[i];
|
||||
mag = Math.sqrt(mag);
|
||||
if (mag > 0) for (let i = 0; i < dim; i++) e[i] /= mag;
|
||||
return e;
|
||||
}
|
||||
|
||||
function makeResult(slug: string): SearchResult {
|
||||
return {
|
||||
slug,
|
||||
page_id: 1,
|
||||
title: `Title for ${slug}`,
|
||||
type: 'concept',
|
||||
chunk_text: `chunk text for ${slug}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
};
|
||||
}
|
||||
|
||||
const META: HybridSearchMeta = {
|
||||
vector_enabled: true,
|
||||
detail_resolved: 'medium',
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Wipe the cache between tests so ordering doesn't matter.
|
||||
await engine.executeRaw(`DELETE FROM query_cache`);
|
||||
});
|
||||
|
||||
describe('migration v51 \u2014 query_cache table exists', () => {
|
||||
test('table is present and has expected columns', async () => {
|
||||
const rows = await engine.executeRaw<{ column_name: string }>(
|
||||
`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'query_cache'`,
|
||||
);
|
||||
const names = rows.map(r => r.column_name);
|
||||
expect(names).toContain('id');
|
||||
expect(names).toContain('query_text');
|
||||
expect(names).toContain('source_id');
|
||||
expect(names).toContain('embedding');
|
||||
expect(names).toContain('results');
|
||||
expect(names).toContain('meta');
|
||||
expect(names).toContain('ttl_seconds');
|
||||
expect(names).toContain('created_at');
|
||||
expect(names).toContain('hit_count');
|
||||
});
|
||||
});
|
||||
|
||||
describe('cacheRowId', () => {
|
||||
test('is deterministic across same input', () => {
|
||||
expect(cacheRowId('hello', 'default')).toBe(cacheRowId('hello', 'default'));
|
||||
});
|
||||
test('differs across source_id', () => {
|
||||
expect(cacheRowId('hello', 'a')).not.toBe(cacheRowId('hello', 'b'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 store + lookup', () => {
|
||||
test('roundtrip: exact embedding match returns a hit', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(1);
|
||||
const results = [makeResult('a'), makeResult('b')];
|
||||
|
||||
await cache.store('what is foo', emb, results, META);
|
||||
const hit = await cache.lookup(emb);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.results).toHaveLength(2);
|
||||
expect(hit.results?.[0].slug).toBe('a');
|
||||
expect(hit.similarity).toBeGreaterThan(0.99);
|
||||
});
|
||||
|
||||
test('similar embedding (cosine > 0.92) is a hit', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const base = makeEmbedding(100);
|
||||
|
||||
// Construct a near-neighbor: tweak a few dims so cosine stays > 0.92.
|
||||
const near = new Float32Array(base);
|
||||
for (let i = 0; i < 10; i++) near[i] += 0.005;
|
||||
// Re-normalize.
|
||||
let mag = 0;
|
||||
for (let i = 0; i < DIM; i++) mag += near[i] * near[i];
|
||||
mag = Math.sqrt(mag);
|
||||
for (let i = 0; i < DIM; i++) near[i] /= mag;
|
||||
|
||||
await cache.store('what is foo', base, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(near);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.similarity).toBeGreaterThan(0.92);
|
||||
});
|
||||
|
||||
test('orthogonal embedding is a miss', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const a = makeEmbedding(1);
|
||||
const b = makeOrthogonalEmbedding(2);
|
||||
await cache.store('q1', a, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(b);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 TTL', () => {
|
||||
test('stale row (past TTL) is not returned', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { ttlSeconds: 1 });
|
||||
const emb = makeEmbedding(42);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
|
||||
|
||||
// Manually rewind created_at to simulate expiration.
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache SET created_at = now() - interval '10 seconds'`,
|
||||
);
|
||||
const hit = await cache.lookup(emb);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 source isolation', () => {
|
||||
test('different source_id cannot read each other\u2019s rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(7);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
|
||||
const hitB = await cache.lookup(emb, { sourceId: 'src-B' });
|
||||
expect(hitB.hit).toBe(false);
|
||||
const hitA = await cache.lookup(emb, { sourceId: 'src-A' });
|
||||
expect(hitA.hit).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 management', () => {
|
||||
test('clear() wipes all rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(9);
|
||||
await cache.store('q1', emb, [makeResult('a')], META);
|
||||
await cache.store('q2', makeEmbedding(10), [makeResult('b')], META);
|
||||
const removed = await cache.clear();
|
||||
expect(removed).toBeGreaterThanOrEqual(2);
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(0);
|
||||
});
|
||||
|
||||
test('prune() deletes only stale rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
await cache.store('fresh', makeEmbedding(11), [makeResult('a')], META);
|
||||
await cache.store('stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache SET created_at = now() - interval '10 seconds' WHERE query_text = 'stale'`,
|
||||
);
|
||||
const removed = await cache.prune();
|
||||
expect(removed).toBe(1);
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(1);
|
||||
expect(stats.fresh_rows).toBe(1);
|
||||
});
|
||||
|
||||
test('stats() reports fresh / stale / total / hit counters', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(13);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
await cache.lookup(emb); // bump hit
|
||||
// Hit bump is async/fire-and-forget; give it a moment to land.
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(1);
|
||||
expect(stats.fresh_rows).toBe(1);
|
||||
expect(stats.stale_rows).toBe(0);
|
||||
expect(stats.total_hits).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 disabled', () => {
|
||||
test('disabled cache is a pure no-op on lookup', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { enabled: false });
|
||||
const emb = makeEmbedding(99);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
// Even after a store call, lookup must miss because enabled=false.
|
||||
const hit = await cache.lookup(emb);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,317 @@
|
||||
/**
|
||||
* Pins the v0.32.3 search-lite mode core: MODE_BUNDLES + resolveSearchMode
|
||||
* + knobsHash. The 3x7 mode table is asserted cell-by-cell because the
|
||||
* public eval methodology doc cites these values verbatim — drift here is
|
||||
* a documentation-honesty bug, not a refactor.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
MODE_BUNDLES,
|
||||
SEARCH_MODES,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
resolveSearchMode,
|
||||
attributeKnob,
|
||||
knobsHash,
|
||||
loadOverridesFromConfig,
|
||||
KNOBS_HASH_VERSION,
|
||||
SEARCH_MODE_CONFIG_KEYS,
|
||||
type SearchMode,
|
||||
} from '../src/core/search/mode.ts';
|
||||
|
||||
describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
test('SEARCH_MODES is exactly the 3 expected values', () => {
|
||||
expect([...SEARCH_MODES]).toEqual(['conservative', 'balanced', 'tokenmax']);
|
||||
});
|
||||
|
||||
test('DEFAULT_SEARCH_MODE is balanced (matches v0.31.x current default surface)', () => {
|
||||
expect(DEFAULT_SEARCH_MODE).toBe('balanced');
|
||||
});
|
||||
|
||||
test('MODE_BUNDLES is frozen (cannot be mutated)', () => {
|
||||
expect(Object.isFrozen(MODE_BUNDLES)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.conservative)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.balanced)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.tokenmax)).toBe(true);
|
||||
});
|
||||
|
||||
// The 3x7 cell-by-cell assertion. The methodology doc cites these.
|
||||
test('conservative bundle values are canonical', () => {
|
||||
expect(MODE_BUNDLES.conservative).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 4000,
|
||||
expansion: false,
|
||||
searchLimit: 10,
|
||||
});
|
||||
});
|
||||
|
||||
test('balanced bundle values are canonical', () => {
|
||||
expect(MODE_BUNDLES.balanced).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 12000,
|
||||
expansion: false,
|
||||
searchLimit: 25,
|
||||
});
|
||||
});
|
||||
|
||||
test('tokenmax bundle values are canonical (NOTE: limit=50, NOT current=20)', () => {
|
||||
expect(MODE_BUNDLES.tokenmax).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: undefined,
|
||||
expansion: true,
|
||||
searchLimit: 50,
|
||||
});
|
||||
});
|
||||
|
||||
test('cache_enabled is true in every mode (free win)', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(MODE_BUNDLES[m].cache_enabled).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('intentWeighting is true in every mode (zero-LLM cost)', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(MODE_BUNDLES[m].intentWeighting).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('tokenBudget escalates: 4000 → 12000 → undefined', () => {
|
||||
expect(MODE_BUNDLES.conservative.tokenBudget).toBe(4000);
|
||||
expect(MODE_BUNDLES.balanced.tokenBudget).toBe(12000);
|
||||
expect(MODE_BUNDLES.tokenmax.tokenBudget).toBeUndefined();
|
||||
});
|
||||
|
||||
test('searchLimit escalates: 10 → 25 → 50', () => {
|
||||
expect(MODE_BUNDLES.conservative.searchLimit).toBe(10);
|
||||
expect(MODE_BUNDLES.balanced.searchLimit).toBe(25);
|
||||
expect(MODE_BUNDLES.tokenmax.searchLimit).toBe(50);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSearchMode', () => {
|
||||
test('accepts every documented mode', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(isSearchMode(m)).toBe(true);
|
||||
}
|
||||
});
|
||||
test('rejects unknown strings, numbers, null, undefined', () => {
|
||||
expect(isSearchMode('conservativeX')).toBe(false);
|
||||
expect(isSearchMode('')).toBe(false);
|
||||
expect(isSearchMode('CONSERVATIVE')).toBe(false); // case-sensitive at the type guard layer
|
||||
expect(isSearchMode(42)).toBe(false);
|
||||
expect(isSearchMode(null)).toBe(false);
|
||||
expect(isSearchMode(undefined)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSearchMode resolution chain', () => {
|
||||
test('no inputs → balanced bundle (fallback)', () => {
|
||||
const r = resolveSearchMode({});
|
||||
expect(r.resolved_mode).toBe('balanced');
|
||||
expect(r.mode_valid).toBe(false);
|
||||
expect(r.searchLimit).toBe(25);
|
||||
expect(r.tokenBudget).toBe(12000);
|
||||
expect(r.expansion).toBe(false);
|
||||
});
|
||||
|
||||
test('valid mode picked, no overrides → bundle values pass through', () => {
|
||||
const r = resolveSearchMode({ mode: 'conservative' });
|
||||
expect(r.resolved_mode).toBe('conservative');
|
||||
expect(r.mode_valid).toBe(true);
|
||||
expect(r.searchLimit).toBe(10);
|
||||
expect(r.tokenBudget).toBe(4000);
|
||||
});
|
||||
|
||||
test('invalid mode string → balanced fallback (mode_valid=false)', () => {
|
||||
const r = resolveSearchMode({ mode: 'NUKE_MODE' });
|
||||
expect(r.resolved_mode).toBe('balanced');
|
||||
expect(r.mode_valid).toBe(false);
|
||||
expect(r.searchLimit).toBe(25);
|
||||
});
|
||||
|
||||
test('mode string case-normalized (TokenMax → tokenmax)', () => {
|
||||
const r = resolveSearchMode({ mode: 'TokenMax' });
|
||||
expect(r.resolved_mode).toBe('tokenmax');
|
||||
expect(r.mode_valid).toBe(true);
|
||||
});
|
||||
|
||||
test('per-key override wins over mode bundle (CDX-5 chain)', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'conservative',
|
||||
overrides: { tokenBudget: 99999, cache_enabled: false },
|
||||
});
|
||||
expect(r.resolved_mode).toBe('conservative');
|
||||
expect(r.tokenBudget).toBe(99999);
|
||||
expect(r.cache_enabled).toBe(false);
|
||||
expect(r.searchLimit).toBe(10); // not overridden, still from bundle
|
||||
});
|
||||
|
||||
test('per-call override wins over per-key override', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'conservative',
|
||||
overrides: { tokenBudget: 99999 },
|
||||
perCall: { tokenBudget: 77 },
|
||||
});
|
||||
expect(r.tokenBudget).toBe(77);
|
||||
});
|
||||
|
||||
test('per-call false-y values (false / 0) still beat fallback', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'tokenmax',
|
||||
perCall: { expansion: false, cache_enabled: false },
|
||||
});
|
||||
expect(r.expansion).toBe(false); // beat tokenmax's true
|
||||
expect(r.cache_enabled).toBe(false); // beat tokenmax's true
|
||||
});
|
||||
|
||||
test('undefined fields in perCall fall through (not coerced to false)', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'tokenmax',
|
||||
perCall: { tokenBudget: undefined, expansion: undefined },
|
||||
});
|
||||
expect(r.tokenBudget).toBeUndefined(); // from tokenmax bundle
|
||||
expect(r.expansion).toBe(true); // from tokenmax bundle, NOT overridden
|
||||
});
|
||||
});
|
||||
|
||||
describe('attributeKnob source attribution', () => {
|
||||
test('per-call source labeled correctly', () => {
|
||||
const input = { mode: 'conservative', perCall: { tokenBudget: 999 } };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('tokenBudget', input, resolved);
|
||||
expect(a.source).toBe('per-call');
|
||||
expect(a.value).toBe(999);
|
||||
});
|
||||
|
||||
test('override source labels the config key path', () => {
|
||||
const input = { mode: 'conservative', overrides: { cache_enabled: false } };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('cache_enabled', input, resolved);
|
||||
expect(a.source).toBe('override');
|
||||
expect(a.source_detail).toContain('search.cache_enabled');
|
||||
});
|
||||
|
||||
test('mode source labels the mode name', () => {
|
||||
const input = { mode: 'conservative' };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('searchLimit', input, resolved);
|
||||
expect(a.source).toBe('mode');
|
||||
expect(a.source_detail).toContain('conservative');
|
||||
});
|
||||
|
||||
test('fallback source labels the unset state explicitly', () => {
|
||||
const input = {}; // no mode set
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('searchLimit', input, resolved);
|
||||
expect(a.source).toBe('fallback');
|
||||
expect(a.source_detail).toContain('balanced');
|
||||
expect(a.source_detail).toContain('unset');
|
||||
});
|
||||
});
|
||||
|
||||
describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
test('hash is deterministic across calls', () => {
|
||||
const knobs = resolveSearchMode({ mode: 'conservative' });
|
||||
const h1 = knobsHash(knobs);
|
||||
const h2 = knobsHash(knobs);
|
||||
expect(h1).toBe(h2);
|
||||
});
|
||||
|
||||
test('different modes produce different hashes', () => {
|
||||
const c = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const b = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
const t = knobsHash(resolveSearchMode({ mode: 'tokenmax' }));
|
||||
expect(c).not.toBe(b);
|
||||
expect(b).not.toBe(t);
|
||||
expect(c).not.toBe(t);
|
||||
});
|
||||
|
||||
test('per-call override changes the hash (cache key bifurcates)', () => {
|
||||
const a = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const b = knobsHash(resolveSearchMode({ mode: 'conservative', perCall: { tokenBudget: 999 } }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
test('hash is short (16 hex chars) and stable shape', () => {
|
||||
const h = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
expect(h).toMatch(/^[0-9a-f]{16}$/);
|
||||
});
|
||||
|
||||
test('KNOBS_HASH_VERSION constant exposed for migrations to bump on schema change', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadOverridesFromConfig flat-map parser', () => {
|
||||
test('empty config map → empty overrides', () => {
|
||||
const ov = loadOverridesFromConfig({});
|
||||
expect(ov).toEqual({});
|
||||
});
|
||||
|
||||
test('cache.enabled accepts 1 / 0 / true / false strings', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': '1' }).cache_enabled).toBe(true);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': '0' }).cache_enabled).toBe(false);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'true' }).cache_enabled).toBe(true);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'false' }).cache_enabled).toBe(false);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'TRUE' }).cache_enabled).toBe(true);
|
||||
});
|
||||
|
||||
test('numeric keys parse and clamp', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '0.95' }).cache_similarity_threshold).toBe(0.95);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.ttl_seconds': '7200' }).cache_ttl_seconds).toBe(7200);
|
||||
expect(loadOverridesFromConfig({ 'search.tokenBudget': '8000' }).tokenBudget).toBe(8000);
|
||||
expect(loadOverridesFromConfig({ 'search.searchLimit': '30' }).searchLimit).toBe(30);
|
||||
});
|
||||
|
||||
test('invalid numerics are ignored (not coerced to NaN/0)', () => {
|
||||
const ov = loadOverridesFromConfig({
|
||||
'search.cache.similarity_threshold': 'NaN',
|
||||
'search.tokenBudget': 'cheese',
|
||||
'search.searchLimit': '-1',
|
||||
'search.cache.ttl_seconds': '0',
|
||||
});
|
||||
expect(ov.cache_similarity_threshold).toBeUndefined();
|
||||
expect(ov.tokenBudget).toBeUndefined();
|
||||
expect(ov.searchLimit).toBeUndefined();
|
||||
expect(ov.cache_ttl_seconds).toBeUndefined();
|
||||
});
|
||||
|
||||
test('similarity_threshold rejects values outside (0, 1]', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '1.5' }).cache_similarity_threshold).toBeUndefined();
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '0' }).cache_similarity_threshold).toBeUndefined();
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '-0.1' }).cache_similarity_threshold).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('SEARCH_MODE_CONFIG_KEYS is the full reset surface', () => {
|
||||
test('every key starts with search. prefix (gbrain config unset --pattern search.* compatibility)', () => {
|
||||
for (const k of SEARCH_MODE_CONFIG_KEYS) {
|
||||
expect(k.startsWith('search.')).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('every ModeBundle field has a config key (consistency check)', () => {
|
||||
// If a new knob is added to ModeBundle, this test fails until the operator
|
||||
// adds the corresponding config key to SEARCH_MODE_CONFIG_KEYS. That's the
|
||||
// intentional regression guard: `gbrain search modes --reset` must clear
|
||||
// every knob.
|
||||
const knobs = Object.keys(MODE_BUNDLES.balanced);
|
||||
expect(SEARCH_MODE_CONFIG_KEYS.length).toBeGreaterThanOrEqual(knobs.length);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Type-only smoke test (compiler sees SearchMode union)', () => {
|
||||
test('SearchMode union is exactly 3 modes (compile-time)', () => {
|
||||
const valid: SearchMode[] = ['conservative', 'balanced', 'tokenmax'];
|
||||
expect(valid.length).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* v0.32.3 search-lite telemetry rollup writer tests.
|
||||
*
|
||||
* Pins the architecture decisions from D2 + [CDX-17] + [CDX-18]:
|
||||
* - In-memory bucket flushed periodically (NOT per-call DB write)
|
||||
* - Sums + counts, NEVER pre-averaged columns
|
||||
* - Date-bucketed cache hit/miss derivable over --days N window
|
||||
* - ON CONFLICT DO UPDATE adds raw values (concurrent flushes accumulate)
|
||||
* - Per-bucket isolation: one bad row doesn't lose the others
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
recordSearchTelemetry,
|
||||
readSearchStats,
|
||||
getTelemetryWriter,
|
||||
_resetTelemetryWriterForTest,
|
||||
} from '../src/core/search/telemetry.ts';
|
||||
import type { HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
_resetTelemetryWriterForTest();
|
||||
await engine.executeRaw('DELETE FROM search_telemetry');
|
||||
});
|
||||
|
||||
const makeMeta = (overrides: Partial<HybridSearchMeta> = {}): HybridSearchMeta => ({
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
mode: 'balanced',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('recordSearchTelemetry — in-memory bucket', () => {
|
||||
test('first record creates a bucket; record() never blocks the caller', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
expect(w.bucketCountForTest()).toBe(0);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
});
|
||||
|
||||
test('same (date, mode, intent) accumulates into one bucket', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 3 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general');
|
||||
expect(b?.count).toBe(3);
|
||||
expect(b?.sum_results).toBe(15); // 5 + 7 + 3
|
||||
});
|
||||
|
||||
test('different modes / intents create distinct buckets', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }), { results_count: 2 });
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }), { results_count: 9 });
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'balanced', intent: 'event' }), { results_count: 4 });
|
||||
expect(w.bucketCountForTest()).toBe(3);
|
||||
});
|
||||
|
||||
test('cache_hit / cache_miss counters fire from meta.cache.status', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'disabled' } }));
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general')!;
|
||||
expect(b.cache_hit).toBe(2);
|
||||
expect(b.cache_miss).toBe(1);
|
||||
expect(b.count).toBe(4);
|
||||
});
|
||||
|
||||
test('sum_budget_dropped accumulates from meta.token_budget.dropped', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 3800, kept: 8, dropped: 12 } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 4000, kept: 10, dropped: 7 } }));
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general')!;
|
||||
expect(b.sum_budget_dropped).toBe(19); // 12 + 7
|
||||
});
|
||||
|
||||
test('missing mode / intent fall back to "unset" — telemetry is non-blocking', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, { vector_enabled: true, detail_resolved: null, expansion_applied: false });
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
expect(w.bucketForTest(today, 'unset', 'unset')).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('flush() writes to search_telemetry', () => {
|
||||
test('flush drains the bucket map atomically', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
await w.flush();
|
||||
expect(w.bucketCountForTest()).toBe(0);
|
||||
|
||||
const rows = await engine.executeRaw<{ count: number; sum_results: number }>(
|
||||
'SELECT count, sum_results FROM search_telemetry',
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].count).toBe(2);
|
||||
expect(rows[0].sum_results).toBe(12);
|
||||
});
|
||||
|
||||
test('ON CONFLICT DO UPDATE adds raw values (concurrent-flush semantics)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
|
||||
// First flush: 3 calls under balanced/general.
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
await w.flush();
|
||||
|
||||
// Second flush: 2 more calls under same (date, mode, intent).
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
await w.flush();
|
||||
|
||||
const rows = await engine.executeRaw<{ count: number; sum_results: number }>(
|
||||
'SELECT count, sum_results FROM search_telemetry',
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].count).toBe(5); // 3 + 2
|
||||
expect(rows[0].sum_results).toBe(35); // (5+5+5) + (10+10)
|
||||
});
|
||||
|
||||
test('flush is no-op when bucket map is empty', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
await w.flush(); // no records, no rows
|
||||
const rows = await engine.executeRaw<{ n: number }>('SELECT COUNT(*)::int AS n FROM search_telemetry');
|
||||
expect(rows[0].n).toBe(0);
|
||||
});
|
||||
|
||||
test('concurrent flush() calls coalesce (flushInFlight reuse)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 1 });
|
||||
// Two simultaneous flush() awaits → both observe the same underlying drain.
|
||||
const [a, b] = await Promise.all([w.flush(), w.flush()]);
|
||||
expect(a).toBeUndefined();
|
||||
expect(b).toBeUndefined();
|
||||
const rows = await engine.executeRaw<{ count: number }>('SELECT count FROM search_telemetry');
|
||||
expect(rows[0].count).toBe(1); // not doubled
|
||||
});
|
||||
});
|
||||
|
||||
describe('readSearchStats — read-time derived averages', () => {
|
||||
test('empty table → all-zero stats', async () => {
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(0);
|
||||
expect(s.cache_hit_rate).toBe(0);
|
||||
expect(s.avg_results).toBe(0);
|
||||
});
|
||||
|
||||
test('one bucket flushed → stats derive averages from sums/counts', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 20 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 30 });
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(3);
|
||||
expect(s.avg_results).toBe(20); // (10 + 20 + 30) / 3 = 20
|
||||
});
|
||||
|
||||
test('cache_hit_rate computed from hits + misses (excludes disabled)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'disabled' } }));
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.cache_hits).toBe(3);
|
||||
expect(s.cache_misses).toBe(1);
|
||||
expect(s.cache_hit_rate).toBeCloseTo(0.75, 5); // 3 / (3 + 1) = 0.75
|
||||
});
|
||||
|
||||
test('intent_distribution and mode_distribution surface counts', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }));
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.intent_distribution.entity).toBe(2);
|
||||
expect(s.intent_distribution.temporal).toBe(1);
|
||||
expect(s.mode_distribution.conservative).toBe(2);
|
||||
expect(s.mode_distribution.tokenmax).toBe(1);
|
||||
});
|
||||
|
||||
test('days window clamps to [1, 365]', async () => {
|
||||
const a = await readSearchStats(engine, { days: 0 });
|
||||
expect(a.window_days).toBe(1);
|
||||
const b = await readSearchStats(engine, { days: 9999 });
|
||||
expect(b.window_days).toBe(365);
|
||||
const c = await readSearchStats(engine, {});
|
||||
expect(c.window_days).toBe(7); // default
|
||||
});
|
||||
|
||||
test('missing search_telemetry table → empty stats (graceful)', async () => {
|
||||
// Drop the table to simulate a pre-v0.32.3 brain.
|
||||
await engine.executeRaw('DROP TABLE IF EXISTS search_telemetry');
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(0);
|
||||
expect(s.cache_hit_rate).toBe(0);
|
||||
// Restore for subsequent tests in this describe block.
|
||||
await engine.initSchema();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 token budget enforcement.
|
||||
*
|
||||
* Pure module under test (no DB, no LLM). Confirms:
|
||||
* - char/4 heuristic estimates correctly
|
||||
* - greedy walk preserves caller ordering
|
||||
* - meta accounting (used / kept / dropped) matches the actual cut
|
||||
* - undefined / <=0 budget is a no-op
|
||||
* - first-result-too-big returns empty list
|
||||
*
|
||||
* Lives in test/token-budget.test.ts to mirror existing search/* test naming.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { enforceTokenBudget, estimateTokens, resultTokens } from '../src/core/search/token-budget.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function makeResult(overrides: Partial<SearchResult> = {}): SearchResult {
|
||||
return {
|
||||
slug: 'test-page',
|
||||
page_id: 1,
|
||||
title: 'Test Title',
|
||||
type: 'concept',
|
||||
chunk_text: 'test chunk text',
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('estimateTokens', () => {
|
||||
test('empty / nullish strings cost 0 tokens', () => {
|
||||
expect(estimateTokens('')).toBe(0);
|
||||
expect(estimateTokens(undefined)).toBe(0);
|
||||
expect(estimateTokens(null)).toBe(0);
|
||||
});
|
||||
|
||||
test('rounds up so a single char still costs 1 token', () => {
|
||||
expect(estimateTokens('a')).toBe(1);
|
||||
expect(estimateTokens('abc')).toBe(1);
|
||||
expect(estimateTokens('abcd')).toBe(1);
|
||||
expect(estimateTokens('abcde')).toBe(2);
|
||||
});
|
||||
|
||||
test('scales linearly at ~4 chars / token', () => {
|
||||
// 100 chars \u2192 25 tokens
|
||||
expect(estimateTokens('x'.repeat(100))).toBe(25);
|
||||
// 401 chars \u2192 ceil(401/4) = 101
|
||||
expect(estimateTokens('x'.repeat(401))).toBe(101);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resultTokens', () => {
|
||||
test('sums title + chunk_text', () => {
|
||||
const r = makeResult({ title: 'abcd', chunk_text: 'efgh' }); // 1 + 1 = 2
|
||||
expect(resultTokens(r)).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('enforceTokenBudget', () => {
|
||||
test('undefined budget is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'a' }), makeResult({ slug: 'b' })];
|
||||
const { results: kept, meta } = enforceTokenBudget(results, undefined);
|
||||
expect(kept).toHaveLength(2);
|
||||
expect(meta.dropped).toBe(0);
|
||||
expect(meta.kept).toBe(2);
|
||||
expect(meta.budget).toBe(0); // safe-budget normalization
|
||||
});
|
||||
|
||||
test('zero / negative budget is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'a' })];
|
||||
expect(enforceTokenBudget(results, 0).results).toHaveLength(1);
|
||||
expect(enforceTokenBudget(results, -5).results).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('empty input returns empty', () => {
|
||||
const { results, meta } = enforceTokenBudget([], 100);
|
||||
expect(results).toHaveLength(0);
|
||||
expect(meta.kept).toBe(0);
|
||||
expect(meta.dropped).toBe(0);
|
||||
});
|
||||
|
||||
test('greedy top-down: stops as soon as cumulative cost would exceed budget', () => {
|
||||
// Each result: title 'a' (1 tok) + chunk_text 'xxxx' (1 tok) = 2 tokens each
|
||||
const results = [
|
||||
makeResult({ slug: 'a', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'b', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'c', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'd', title: 'a', chunk_text: 'xxxx' }),
|
||||
];
|
||||
// Budget 5 \u2192 fits 2 results (cost 2+2=4); a 3rd would push to 6.
|
||||
const { results: kept, meta } = enforceTokenBudget(results, 5);
|
||||
expect(kept).toHaveLength(2);
|
||||
expect(kept.map(r => r.slug)).toEqual(['a', 'b']);
|
||||
expect(meta.used).toBe(4);
|
||||
expect(meta.dropped).toBe(2);
|
||||
expect(meta.kept).toBe(2);
|
||||
expect(meta.budget).toBe(5);
|
||||
});
|
||||
|
||||
test('preserves caller ordering (never re-ranks)', () => {
|
||||
const results = [
|
||||
makeResult({ slug: 'low-score', score: 0.1, title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'high-score', score: 0.9, title: 'a', chunk_text: 'xxxx' }),
|
||||
];
|
||||
const { results: kept } = enforceTokenBudget(results, 2);
|
||||
// 2 tokens fits exactly one result; budget pass should keep the FIRST,
|
||||
// even though it has a worse score \u2014 ordering is caller's contract.
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0].slug).toBe('low-score');
|
||||
});
|
||||
|
||||
test('first result exceeds budget alone \u2192 returns empty', () => {
|
||||
const big = makeResult({ slug: 'big', title: 'a', chunk_text: 'x'.repeat(1000) });
|
||||
const small = makeResult({ slug: 'small', title: 'a', chunk_text: 'xxxx' });
|
||||
const { results: kept, meta } = enforceTokenBudget([big, small], 5);
|
||||
expect(kept).toHaveLength(0);
|
||||
expect(meta.dropped).toBe(2);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user